Noise2Score: Tweedie’s Approach to Self-Supervised Image Denoising without Clean Images
Authors: Kwanyoung Kim
Jong Chul Ye
Venue: NeurIPS 2021 Poster (2021)
Keywords: Self-supervised image denoising
Tweedie's formula
The score function
Noises from general exponential family

Abstract: Recently, there has  been extensive research interest in training  deep networks to denoise images without clean reference.
However, the representative approaches such as Noise2Noise, Noise2Void, Stein's unbiased risk estimator (SURE), etc.  seem to differ from one another and it is difficult to find the coherent mathematical structure. To address this, here we present a novel approach, called Noise2Score, which reveals a missing link in order to unite these seemingly different approaches.
Specifically, we  show that   image denoising  problems  without clean images can be addressed by finding the mode of the posterior distribution and that the Tweedie's formula offers an explicit solution through the score function (i.e. the gradient of loglikelihood). Our method then uses the  recent finding that  the score function  can be stably estimated from the noisy images using the amortized residual denoising autoencoder, the method of which is closely related to Noise2Noise or Nose2Void. Our Noise2Score approach is so universal  that the same network training can be used to remove noises from images that are corrupted by any exponential family distributions and noise parameters. Using extensive  experiments with Gaussian, Poisson, and Gamma noises, we show  that  Noise2Score significantly outperforms the state-of-the-art self-supervised denoising methods in the benchmark data set such as (C)BSD68, Set12, and Kodak, etc. 


Noise2Score: Tweedie's Approach to Self-Supervised Image Denoising without Clean Images
Kwanyoung Kim
Jong Chul Ye1.
Department of Bio and Brain Engineering
Kimm Jaechul Graduate School of Al 3 Deptartment of Mathematical Sciences Korea Advanced Institute of Science and Technology (KAIST) {cubeyoung, jong-yeu ye) ac.kr
Abstract
Recently. there has been extensive research interest in training deep networks to denoise images without clean reference. However, the representative approaches such as Noise2Noise, Noise2 Stein's 's unbiased risk estimator (SURE), etc. seem differ from one another and it is difficult find the coherent mathematical structure. To address this, here we present a novel approach, called Noise2 which reveals a missing link in order unite these seemingly different approaches. Specifically, we show that image denoising problems without clean images can be addressed by finding the mode the posterior distribution and that the Tweedie's s formula offers an explicit solution through the score function (i.e. the gradient of loglikelihood). Our method then uses the recent finding that the score function can be stably estimated from the noisy images using the amortized residual denoising autoencoder, the method which is closely related to Noise2Noise or Nose2Void Our Noise2Score approach is universal that the same network training be used to remove noises from images that are corrupted by any exponential family distributions and noise parameters. Using extensive experiments with Gaussian, Poisson, and Gamma noises, we show that Noise2Score significantly outperforms the state-of-the-arr self-supervised denoising methods in the benchmark data set such as (C)BSD68, Set12, and Kodak, etc.
Introduction
Bayesian inference, which derives the posterior probability using a prior probability and a likelihood function for the observed data, has been an important tool in statistics. This approach has been used extensively for image denoising from early ages to the modern era of deep learning. For example, in the recent unsupervised deep learning approach for image denoising using Stein's 's risk estimate (SURE) M the unknown Bayesian risk is replaced by the SURE that can be calculated from the noisy measurement that deep neural network training is performed by minimizing it. Unfortunately, this method is sensitive to hyper-parament parameters, and the neural network must be retrained if the underlying noise model varies (2]. On the other hand, there has been increased research interest in image denoisers that can be trained by minimizing variants of empirical risks that are not associated with clean data. Noise2Noise B3 was the first representative approach that does not require clean data. Unfortunately, multiple noisy versions the same images are necessary for training. To address this, self-supervised learning approaches such Noise2Void 44) Noise2Self (5), etc. have been developed in order to use only a single noisy image. This class of approaches, which we will call Noise2X throughout the paper, especially important for practical applications, where noiseless clean images or multiple noisy realization of the same image are difficult or impossible to collect.
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

(a) 0' arg min Ey- (b()))
(b) @" arg min oo-loRlly ou]l?
youPy
a
y
Fer (3)
y
Re- (z)
('))
Tweedle's Formula
Figure 1: Overall reconstruction flow of (a) supervised learning. SURE and Noise2X, where the target and the distance measure d(---) are uniquely determined by each algorithm, and (b) Noise2Score, where the first step is the estimation of the score function Í' using neural network training, which is followed by Tweedie's formula to obtain the final denoising result.
Though SURE and Noise2X hold promise for practical applications, one the fundamental questions is how these seemingly different approaches are related and whether there is coherent mathematical theory that can be leveraged to further improve the performance. Although we are aware of a preliminary prior work [6] attempting to unite subset of them, there is largely lack of a principled method of designing image denoiser without clean data.
In this article, we try to approach this open problem from Bayesian statistics in completely different route. Instead of minimizing different forms of empirical risks or SURE, one of the most important contributions of this paper is the discovery of the importance of classical result from Bayesian statistics the Tweedie's formula 7 which provide an explicit way of computing the posterior mean of canonical parameters from the noisy measurements corrupted with exponential family noises. Specifically, we show that the Tweedie's formula provides a unified approach for image denoising from any exponential family noises through score function (i.e. the gradient of the loglikelihood). Therefore, the self self-supervised image denoising problem without clean image can be reduced to the problem of estimating the score function.
This change of perspective has many important theoretical implications as well as the flexibility with regard to the implementation of the algorithm. In fact, the score function estimation problem has been an important research topic in Bayesian statistics and machine learning I- In particular, Alain and Bengio [I0] showed that the minimization of the denoising autoencoder (DAE) objective function provides an explicit way of approximating the score function. This result was further extended using the amortized residual denoising autoencoder (AR-DAE) for numerical stability and accuracy [[] Interestingly, the training method for DAE or its variants is strikingly similar to that of Noise2X. Therefore, by combining this with the Tweedie's 's formula, we can obtain novel unified framework of Bayesian approach for self supervupeee image denoising. The conceptual difference and similarity of method to the existing approaches illustrated in Fig, [a))). As the first step of our algorithm is similar to Noise2X, we call our method Noise2Score.
2Scoe is so powerful that it can be used to deal with any exponential family noises. Moreover, in contrast to the SURE approach, which the network must be retrained using different loss function depending on the noise model and parameters that the identical neural network training step is used regardless of noise models. This property can be exploited to extend Noise2Score to blind setups in which noise parameters are unknown and should be estimated with minimal complexity. In addition to the novel theoretical findings mentioned above, our empirical results using additive Gaussian, Poisson and Gamma noise models have shown that Noise2Score significantly outperforms SURE and Noise2X under similar experimental conditions.
[ [2], Noise2Score is universal in the sense
Related Works
Here we give a brief overview of the existing deep learning approaches to denoise images. As shown below, their main differences arise from the different choices of the distance metric d(:,.) in Fig.
2.1 Image denoising using supervised learning
Let X' and y denotes the spaces of clean and noisy images with the probability measure Px and Py, respectively. When the paired ground-truth data æ is available for noisy measurement y, i.c.

---

= (V), then the supervised training can be carried out by minimizing the following loss function:
Cxup (+) Byy-yy d(x, Fe(y)): where d(x, Fe(()) | (v)ll
(1)
where (y) is deep neural network parameterized by θ with input and here we use 12 distance metric to simplify the explanation. Although this approach has significantly improved performance compared to the classical denoising algorithms [3] the supervised learning approach has drawback
as the large number of paired clean images are necessary. In real world applications, the acquisition for paired clean images are difficult or even impossible.
2.2 Noise2X for image denoising without clean images
To address this issue, various forms of the denoisers that can trained without clean images have been proposed, which includes Noise2Noise []. Noise2 [ Noise2Self [14, Noise2Same [5]. etc. These approaches are usually based on the variants of loss functions that do not depend on clean images:
2 (0) = p (e()), where d(v': Fo(u)) = 11 - Fo(
(2)
where the target image 1/ is related to y in unique ways depending on algorithms. For example, in Noise2Noise (3). y another noisy realization of the same underlying clean images. However, Noise2Noise is not practical since multiple noisy realizations of the same image are not usually available in practice. Noise2Void (14) generates y' by adding blind spots on y. In Noise2Self 14. so-called J-invariant mask J is added on the input y, and y/ is defined as the image the
complementary mask. All of these changes of the target images are intended to prevent the network from converging to trivial identity mapping.
2.3 Image denoising using Stein's 's Unbiased Risk Estimate (SURE)
Rather than using the variants of a loss function as in Noise2X, Soltanayev al proposed training scheme using the loss function from Stein's 's unbiased risk estimate (SURE) 2 which has additional regularization term. Specifically, the loss function for the SURE denoiser defined by
İSURE (e) Py d(y, Fe(y)), d(y, Fe(y)) == 1 Fe + 2o²div,,/'et))
(3)
Here, the second term in d(y, Fe()) of ( is the regularization term to prevent the network from converging to a trivial identity mapping, where div, denotes the divergence with respect to Due to the difficulty of calculating the divergence term, the authors in (T] relied on MonteCarlo SURE [15] which calculates the divergence term using MonteCarlo simulation. This introduces additional hyperparameters, on which the final results critically depend. Yet another limitation of SURE image denoiser is that the loss function is determined by the noise model. For example. in [12], the authors derived Poisson Unbiased risk estimator (PURE) to train deep CNN for dealing with Poisson noise, whose loss function is completely different from (3). Even under the same Gaussian noise models, if the noise variance 2² varies, the neural network Fe must be trained again, which causes additional
computational burden. This differs from Noise2X, which uses the same loss function regardless of the noise models.
3 Main Contribution: Noise2Score
In contrast to the existing approaches shown in Fig. I our denoising approach is composed of two steps as illustrated in Fig. (bb,, which is inspired by the results from Bayesian statistics. In the following, we explain the details of each step.
3.1 Tweedie's formula of the posterior mean for image denoising
Suppose that the noisy measurement y is given by
y + w, where w 02
(4)
where x is the clean image, and (/ o² denotes the independent and identically distributed (i.i.d) Gaussian distribution with zero mean and variance o². Then, Tweedie's formula calculates the posterior expectation of given y as [16]
E|z|y] + +z' where ('()) loggg(())
(5)
3

---

where p(y) denotes the marginal distribution of y. For the image corrupted by Gaussian noises, the posterior mean is the minimum mean square error (MMSE) estimator. so that the denoised image can be obtained using (5), as long as we know (v()) the score function [17].
Tweedie's formula exponential family of probability distributions are defined as those distributions whose density have the following general form:
extended for general class of exponential family distribution 77. Specifically,
p(yln) = po(y) exp (1) T(y) (p(1))
(6)
where the superscript denotes the transpose operation. Here 7) is a canonical (vector) parameter of the family, Ty) is a (vector) function of (p(n) the cumulant generating function which makes p(yln) integrate I, and (y(y) the density up to a scale factor when 1) = Exponential family distribution covers large class of important distributions such as the Gaussian, binomial, multinomial, Poisson, gamma, and beta distributions, as well as many others. Table summarizes the corresponding parameters for each exponential family distribution.
Then, using the Bayes' rule, the posterior density of n given y is given by
p()lv) exp (1) (Tv) X(v)) (((n)e-(()) where M(y)
P(V) Po
log
where p(y) and p(n)) denote the marginal distribution of y and r), respectively. This implies that the posterior density is again an exponential family distribution, and the mode of the posterior distribution can be obtained by finding the maximum of p(nly). Specifically, by computing the gradient of log p(nly) with respect to y and setting it to zero, the posterior estimate of the canonical parameter should satisfy the following equality:
(T)
X'(x) = v, log Po()) + V logp(y) = (v(() + +1y)
(8)
where ('(v) V, log p(y) and (y) V, log (yy) are score functions, and T' V Twy). In particular, if Tyy) = y. we can obtain the following closed form solution for the posterior mean:
f Enn w] (v) (v) + ('()
In the following, we derive denoising algorithms using Tweedie's 's formulation for several representa- tive distributions.
Additive Gaussian noise For Gaussian distribution, in the first row of Table 1 we have
x
(v( =
V2TO e
20
(10)
Accordingly,
4 ²² o²
E[yly]
+ +yy),, where $ :: [[xy]
(11)
since (1()) = y//02. Therefore, the posterior mean is given by
== + o 21
(12)
which is equal to (5) Even with the different parameterization of Gaussian as shown in the second row of Table we can arrive at the same result using 8
Table 1: Tweedie's 'ss formula of exponential family distribution for image denoising.
Distribution
ply
7
T(y)
Poly)
Gaussian
2
y + (M)
Virao
Viewe"
Gaussian
ViTa
[M
0
(
Poisson
log (x)
log (v #
(y + #) exp((
Gamma(a, !)
(M) e-
Dog M. [w]
Bermoulli
21 2)
log
0
FID ((v)
Exponential
00 220

---

Poisson noise
For the case of Poisson noises, we have
(o(y)
1 y! (o()) = +1
v2r(++1)
-log (リ+z)
(13)
where (yy denotes the gamma function and the last approximation comes from we have
Accordingly,
1
- log(()) log y y 2 + (v)
(14)
This leads to the following posterior estimate of the image
= y 1 exp(('()))
(15)
This corresponds Tweedie's formula for Poisson case [6]
This result can be generalized to the low photon count sensing scenario, where the sensor measurement be described by (18):
(z, z ~ Poisson (2r
(16)
where = a random variable that follows the Poisson distribution in Table T (> is the gain of the acquisition process that is related to the noise level. Then, we have
o
(= 2!'
log
(17)
Therefore, we have
1 + exp((((2)) 2
(()
2
y
(18)
Gamma noise application. Specifically, the image corrupted with the speckle noise is represented by
Gamma noise distribution can be used to model the speckle noises in various imaging
g y = xn, p(n; a 3) = r(a)
n exp(-Bn)
(19)
where p(n; a; 3) denotes the Gamma distribution with (a. B) parameters. Here, (a, B) are the parameters which determine the noise level of Gamma noise distribution. For example, for the k-look measurement case, = = k [19]. Then, the probability p(ylær) can be obtained by replacing n in p(n;o, a, () with y/x, and the resulting density function be bound in Table Then, easily show that
- -3/x
log y y/
Po(y) 1, 1 ==
T(y)
(20)
Using 8
we have
B
By
((())
= (a 1) vl((2)
(21)
Similar derivations for other exponential family distribution be found in Table 1
3.2 Score function estimation
So far, our derivation assumes the prior knowledge of the score function. practice, this should be estimated, so here we describe how the score function can be estimated using neural network.
Historically, Hyvärinen et al g was the first derive a remarkable implicit score matching objective that no longer requires having an explicit score target but is nevertheless equivalent to the original problem that minimizes the expected quadratic distance between the model function and the score function of data. With the advance of the Denoising Auto Encoder (DAE) (20) the author of [ observed that the minimization of the DAE objective function with the residual form is related the score matching between the model and perturbed data. This observation was rigorously analyzed
5

---

by Alain and Bengio [0], who showed that as the perturbed noise becomes sufficiently small, DAE leads to the score function. More specifically, in the denoising autoencoder (DAE), the following loss is minimized:
E (D) =
E
lly - Fe + ux))
(22)
y-PP
(0
(0,60)
According to 10 the optimal DAE Fe- (1)) can be represented by
Fe-
og (v) + o(o2),
(23)
where (1) is the score function defined in ( and o(-) denotes the small "o" notation. In other words, for sufficiently small Oa. we can approximate the score function as:
(ly Fe- (v) y
(24)
One of the downsides of the score function estimation using DAE is that the score function estimate in (24) is numerically unstable as oa + O. To address the numerical instability and reduce the approximation error, Lim et al [I] recently proposed so-called the amortized residual DAE (AR-
DAE), Specifically, AR-DAE is trained by minimizing the following the objective function:
L (AR-DAE(O))
E y-PPy w=N()
||u + Ra + ull
(25)
where Rg from the residual form of the DAE:
Fe oa (v) +
(26)
By plugging 26 in (25.. can obtain the original DAE cost in using we have
22 up scale factor. Furthermore,
((y) FB- y Re(y)
(27)
Therefore, the neural network trained with the AR-DAE (55 is direct and stable way of estimating the score function. Therefore, we employ the AR-DAE to estimate the score function from the noisy measurement as the first step of our method (see Fig. Tb).
3.3 Relation Noise2X and SURE
Although our 2oise2Score was derived from a completely different perspective, it turns out that has very important connections to Noise2X and SURE.
Specifically, if the noisy image y corrupted by additive Gaussian, using Tweedie's fomula for Gaussian noises in Table I the denoised image can be approximated by
== 2 = Fe- (v)
(28)
where we use (27 Recall that Fe- (3) is the neural nework trained with DAE loss function in (22)
The DAE training with (22 is basically inserting noises into the images and then find the mapping that removes the noises. In terms of adding extra noise to a noisy image, our method closely related to Noisier2Noise (21). However, in the training phase of Noisier2Noise, the main assumption is that noise from the same noise statistic as the original noisy image should be injected into the noisy images so that the neural networks learn to reduce the noise from the noisy input image. In our Noise2Score training method, however, Gaussian noise added to noisy images at different noise levels regardless of the noise statistics to estimate the score function rather than noise. In addition, in contrast to Noisier2Noise, our method has a post-processing step with the Tweedie's 's formula, which is determined by the noise model such as Poisson, Gamma, etc.
Yet another important connection its relation to SURE. Using the residual representation 26 the SURE cost function in 3 can be equivalently represented by
(S) = EysPr {lly - Fe (v)l + 20 Fo
Ey-Py (o) + 20 Rovy)) + 20 diiii)
(29)

---

where (y)(y) is the dimension of the vector y, which is constant. Up to the scaling factor, this cost function is identical the implicit score matching objective by Hyvärinen et al [8, which is given by
(I) Euyr { [[
+ div, Vo(v)
(30)
where (ey is the score function estimate parameterized by O Therefore, using the residual transform in (26). the SURE objective is nothing but to find the score function, which corresponds to the first step of our Noise2Score. Furthermore, due to the use of (2)) SURE is not optimal for noise models other than Gaussian. Although there are variants of SURE for other noise type [22], the cost function of this method varies depending our method uses the same cost function for neural network training. but only the post-po post-processing step differs depending
the type of noise. However, as discussed in detail later,
the type of noise. Moreover, considering that the modern research trend of the score function estimation has quickly evolved from the implicit score matching to DAE or AR-DAE due to the numerical instability and difficulty of computing the divergence term in (30). can easily expect that the SURE denoiser also suffers from similar limitations.
3.4 Universal neural estimation for blind denosing problems
Recall that Noise2Score is composed of two steps: the score function estimation by minimizing the AR-DAE loss function, after which the final denoised image is obtained by applying Tweedie's s formula shown in Table | This decoupling allows Noise2Score to have unique advantages compared to the conventional approaches.
Specifically, one of the most important advantages of Noise2Score is that AR-DAE training is universal in the sense that the same loss function is used regardless of noise models and parameters. More specifically, in (25). the model and noise parameter dependency is only through the sampling y from Py. This sampling step can be performed by random patch cropping from the input images, which procedure is identical regardless of Py being Gaussian, Poisson, Gamma, etc.
Yet another advantage that this property can be utilized to estimate the unknown noise parameters without retraining neural networks. For example, in the case of Gaussian denoising in Table [ we have the following form of the denoised images:
a(o) = y + o²i((y)
(31)
If the noise parameter is unknown, then we can find the optimal o value by solving the optimization problem
arg min Q ()(G))
(32)
where Q(-) is an image quality penalty such as total variation (TV). contrast to the SURE, where the a dependency exists in the loss function in (3) and the neural network must be trained with a new value, the optimization problem in (32p is computationally negligible since the precalculated score function estimate ('(y) is used regardless of different values. similar on-the-fly fly parameter estimation can be used for Gamma noise model. As shown in (2I). the parameters (a, 3) can be adjusted using the precomputed score function estimate I' (ty) Even for the Poisson noise model, the same trained neural network is used to compute the score function for each input y/ς. Therefore, our Noise2Score has advantages for the blind application where existing methods cannot be used due to the difficulty of generating samples by changing the noise level parameters without retraining.
4 Experimental Results
Dataset and Implementation detail We evaluated the proposed method for gray-scale and color images in the four benchmark datasets: the gray-scale image dataset contains BSD68 (23 and Set The color image dataset contains RGB natural images CBSD68 (23], Kodak dataset. We adopt DIV2K (24. and (C)BSD400 dataset as train data set. We generated the synthetic noise images for each noise distribution. In order to evaluate the proposed method fairly with comparison methods, the same modified U-Net generator 3 is used for all methods. The total epoch was set to 100 and the Adam optimizer (25] was used to train the network. The learning rate was initialized to 2x10-1 for first 50 epoch, and after 50 epoch the learning rate was decayed to 2 x 10-5, The proposed method
7

---

Table 2: Quantitative comparison for various noise model using various methods in terms of PSNR(dB) when the noise parameters are known or unknown (N2V: Vois,2Void, N2S: Noise2Self, N2N: Noise2Noise, SL. supervised learning, Anscombe: Anscombe method for BM3D).
Noise type
Known parameters
Unknown parameters
Gaussian
Dataset
BM3D
N2S
SURE
Ours
N2N
Ours
N2N
BSD68
28.59
26.77
28.28
28.78
29.12
29.18
29.20
28.95
28.98
28.98
Sct12
29.96
27.56
29.16
29.13
30.13
30.33
30.36
30.08
30.08
30.08
25
CBSD68
30.56
29.22
30.05
30.23
30.85
31.10
31.10
30.78
30.91
30.91
Kodak
31.68
30.02
30.53
30.75
31.89
32.20
32.20
31.78
31.96
31.96
BSD68
25.62
24.34
25.61
25.80
26.21
26.27
26.30
25.81
25.86
25.98
Set12
26.33
24.68
26.19
26.23
27.16
27.20
27.20
26.65
26.65
26.72
CBSD68
27.38
25.13
27.05
26.24
27.75
27.94
27.95
27.32
27.66
27.68
Kodak
27.02
25.75
28.01
26.93
28.83
29.07
29.10
28.13
28.70
28.71
Poisson
Dataset
Anscombe
N2V
N2S
PURE
Ours
N2N
SL
Ours
N2N
SL
BSD68
30.51
28.73
29.76
30.16
30.81
30.89
30.98
30.63
30.91
30.93
Set
31.20
30.06
30.47
30.69
31.58
31.70
31.79
31.42
31.67
31.70
Ç 0.01
CBSD68
32,40
31.85
31.04
32.30
32.61
33.01
33.01
32.23
32,94
32.95
Kodak
33.13
32.98
32.24
33.01
33,41
33.91
33,91
32.96
33.85
33.87
BSD68
26.77
26.12
26.54
24.76
27.12
27.23
27.25
26.77
27.19
27.20
Sct12
27.54
27.21
27.53
25.07
27.86
28.02
28.04
27.76
28.01
28.03
<= 0.05
CBSD68
28.33
28.37
28.32
26.68
29.23
29.27
28.23
29.07
Kodak
29.31
29.56
29.52
26.38
29.71
30.33
28.98
30.13
30.25
Gamma
Dataset
N2V
N2S
Ours
N2N
SL
Ours
N2N
SL
BSD68
29.32
30.49
32.67
32.87
32.93
32.54
32.83
32.88
Sct12
30.54
30.71
33.01
33.21
31.72
32.89
33.19
33.21
100
CBSD68
31.11
30.54
33.82
35.45
35.53
33.50
35.33
35.33
Kodak
31.96
31.60
34.22
36.26
36.41
33.82
36.16
36.16
BSD68
26.98
29.25
30.53
31,10
31.16
30.42
30.64
30.64
Sct12
27.36
29.67
30.87
31.68
31.72
30.83
31.58
31.67
== 50
CBSD68
30.51
30.19
31.05
33.52
33.53
30.93
33.40
33.41
Kodak
31.38
31.03
31.34
34.49
34.57
31.32
34.39
34.40
was implemented in Py (26] with NVidia GeForce GTX 1080-Ti. The network training took about 10 hours. The more detail of implementation
described in Supplementary Material. To deal with blind noise case, as for the quality metric Q(-) we use the TV norm for the Gaussian noises and its combination with data fidelity term for the Poisson and Gamma noise. The details of the quality metric and the discussion on the accuracy of the blind parameter estimation provided in Supplementary Material.
Gaussian noise Using additive Gaussian noise, our method was compared to (C)BM3D B Noise2Void, Noise2Self, SURE, Noise2Noise and supervised learning shown in Table 2. As expected, the supervised learning and Noise2Noise using multiple noisy realization were the best, but these methods are not practical as explained before. Among the other self-supervised learning approaches, our Noise2Score provide the best performance for all dataset. With the comparison with the supervised learning and Noise2Noise, the performance difference of our method only margin. Even for blind-noise cases where the noise levels are randomly sampled from Gaussian distribution with [5, 55], Noise2Score with the on-the-fly parameter estimation performed comparable the supervised learning approach. These results indicate the superior performance of our method. The qualitative comparison Fig. show that Noise2Score retains the better image details and provides much visually pleasing results compared to other top two self- saf--superviee learning approaches.
Poisson noise For the Poisson noise case, BM3D and SURE were replaced with Anscombe (BM3D+VST) [27] and PURE [12), respectively, indicated in Table 2, as they are optimized for Possion noise model. Again, our Noise2Score results in significant performance gain compared to other approaches, and the results are comparable to supervised learning approach and Noise2Noise. In addition, to perform experiments when the noise parameters are unknown, we performed experiments with the noise levels randomly sampled between Ç [0.001,0.1]. We found that proposed method

---

Noisy
SURE
N25
Ours
Gaussian (a 50)
15.26
27.13
28.14
28.71
Noisy
GT
PURE
N2S
Our
Poisson ( =000))
18.01
25.19
25.52
26 .4
Noisy
GT
N2V
N2S
Ours
Gamma (k 100)
26.06
30.72
30.23
32.91
Figure 2: Qualitative comparison using CBSD68 dataset. Top Gaussian noise with = 50. Middle: Poisson noise with Ç = 0.05. Bottom: Gamma noise with ie 100. White numbers the lower left part of the images indicate the PSNR values in dB. Noisy: noisy input, GT: ground-truth image.
produce even higher PSNR than the results of other existing methods even with known parameters, and also provide comparable results the supervised learning approaches. The qualitative comparison in Fig. with the other two best approach showed that our proposed method provides much cleaner images.
Gamma noise As the extension of BM3D and SURE are available for Gamma noises, only four comparison methods was adopted for comparison as indicated in Table Here, we set = 3 = Again, Noise2Score produced best results among self-supervised learning approaches, and provided comparable results to Noise2Noise and supervised learning approaches. Even for the blind noise case, where the noise parameter & randomly selected in k E [40, 120]. Noise2Score provided the better PSNR results compared to the other self-supervised learning method even with known parameters, and produces comparable results to supervised learning approaches. The qualitative comparison in Fig. 2 confirm that our Noise2Score provides best reconstruction results.
Conclusion
In this work, we provided a novel Bayesian framework for self-supervised image denoising without clean data, which surpasses SURE, PURE, Noise2X, etc. Our novel innovation came from the Tweedie's s formula, which provides explicit representation of denoise images through the score function. By combining with the score-function estimation using AR-DAE, our Noise2Score can be applied to image denoising problem from any exponential family noises. Furthermore, an identical neural network training can be universally used regardless of the noise models, which leads to the noise parameter estimation with minimal complexity. The links to SURE and existing Noise2X were also explained, which clearly showed why our method is better generalization.

---

Limitation and negative societal impacts
As negative societal impact, the failure of image denoising methods could produce side effects. For example, removing both the noise and the texture of the medical images could lead to misdiagnoisis. While the proposed Noise2Score has merits, there are also limitations. In real environments, the prior knowledge of noise distribution may not be available, and the noise model could not be modeled by exponential family noises. Therefore, future extension to such scenario would be beneficial.
Acknowledgment
This research was funded by the National Research Foundation (NRF) of Korea grant NRF- 2020R1 1 A2B5503001980,, ETRI (Electronics and Telecommunications Research Institute)'s internal funds [21 YR2500, Development of Digital Biopsy Core Technology for high-precision Diagnosis and Therapy of Senile Disease], and KAIST Key Research Institute (Interdisciplinary Research Group) Project. This work was also supported by Institute of Information & Communications Technology Planning & Evaluation (IITP) grant funded by the Korea government (MSIT) (N0.2019-0-00075, Artificial Intelligence Graduate School Program (KAIST)).
References
[1] Shakarim Soltanayev and Se Young Chun. Training and Retining Deep Learning Based Denoisers without Ground Truth Data. arXiv preprint arXiv:1803.00 0131, 2018.
[2] Charles M Stein. Estimation of the mean of a multivariate normal distribution. The annuls of Statistics, pages 1355-115, 1981.
[3] Jaakko Lehtinen, Jacob Munkberg. Jon Hasselgren, Samuli Laine, Tero Karras, Miika Aittala, and Timo Aila. Noise2Noise: Learning image restoration without clean data. arXiv preprint 1XXiv:8803.04189 2018.
[4] Alexander Krull, Tim-Oliver Buchholz, and Florian Jug. Noise2Void--earning denoising from single noisy images. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2129-200 213992217,222 2019.
[5] Yaochen Xie, Zhengyang Wang, and Shuiwang Ji. Noise2Same: Optimizing self-supervised bound for image denoising. arXiv preprint arXiv:20/0.11977 2020.
[6] Magauiya Zhussip, Shakarim Soltanayev, and Se Young Chun. Extending Stein's unbiased risk estimator to train deep denoisers with correlated pairs of noisy images. arXiv preprint 1rXiv:1902.022 2019.
[7] Bradley Efron. Tweedie's 's formula and selection bias. Journal of the American Statistical Association, 106(496):1602116 2011.
[8] Aapo Hyvärinen and Peter Dayan. Estimation of non-normalized statistical models by score matching. Journal of Machine Learning Research, 6(4), 2005,
[9] Pascal Vincent. A connection between matching and denoising autoencoders. Neural computation, 23(7):1661 2011.
[10] Guillaume Alain and Yoshua Bengio. What regularized auto-encoders learn from the data- getta-gnnn distribution. The Journal of Machine Learning Research, 15(1):3563-3593 2014.
[11] Jae Hyun Lim, Aaron Courville, Christopher Pal, and Chin-Wei Huang. AR-DAE: Towards Unbiased Neural Entropy Gradient Estimation. In International Conference on Machine Learning, pages 6061-6077 PMLR, 2020.
[12] Kwanyoung Kim, Shakarim Soltanayev, and Se Young Chun. Unsupervised Training of Denoisers for Low-Dose CT Reconstruction Without Full-Dose Ground Truth. IEEE Journal of Selected Topics in Signal Processing, 14(6):112
2020.
[13] Kostadin Dabov, Alessandro Foi, Vladimir Katkovnik, and Karen Egiazarian. Image denoising with mlock-matching and 3D filtering. In Image Processing: Algoritims and Systems, Neural Networks, and Machine Learning, volume 6064, page 606414. International Society for Optics and Photonics, 2006.
10

---

[14] Joshua Batson and Loic Royer. Noise2Self: Blind denoising by self-supervision In International Conference Machine Learning, pages 524-533. PMLR. 2019.
[15] Sathish Ramani, Thierry Blu, and Michael Unser. Monte-carlo SURE: A black-box optimization of regularization parameters for general denoising algorithms. IEEE Transactions image processing. 17(9):1500-13 2008.
[16] H Robbins. An empirical Bayes approach statistics. In Proc. 3rd Berkeley Symp. Math. Statist. Probab., 1956, volume pages 157 177-163, 1956.
[17] Robert J Serfling, Approximation theorems of mathematical statistics, volume 162. John Wiley & Sons, 2009.
[18] Yoann Le Montagner, Elsa D Angelini, and Jean-Christophe Olivo-Marin. An unbiased risk estimator image denoising in the presence of mixed Poisson Gaussian noise. IEEE Transactions on Image processing, 23(3): (213):1255-111) 1268, 2014.
[19] José M BBoucas-Dias and Mário Figueiredo. Multiplicative noise removal using variable splitting and constrained optimization. JEEE Trunsactions on Image Processing, 19(7):1200-1730, 2010.
[20] Pascal Vincent, Hugo Larochelle, Isabelle Lajoie, Yoshua Bengio, Pierre-Antoine Manzagol, and Léon Bottou. Stacked denoising autoencoders: Learning useful representations in a deep network with a local denoising criterion. Journal of machine learning research, 11(12), 2010.
[21] Nick Moran, Dan Schmidt, Yu Zhong, and Patrick Coady. Noisier2noise: Learning to denoise from unpaired noisy data. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 12064- 12072, 2020.
[22] Martin Raphan and Eero P Simoncelli. Least squares estimation without priors supervision. Neural computation, 23(2)3774 320-4400,444444 2011.
[23] David Martin, Charless Fowlkes, Doron Tal, and Jitendra Malik. A database of human segmented natural images and its application evaluating segmentation algorithus and measuring ecological statistics. In Proceedings Eighth IEEE International Conference on Computer Vision. ICCV 2001, volume 2, pages 416-423. IEEE, 2001.
[24] Radu Timofte, Shuhang Gu, Jiqing Wu, Luc Van Gool, Lei Zhang, Ming-Hsual Yang, Muhammad Haris, et al. NTIRE 2018 challenge on Single Image super-Resolution Methods and Results. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR) Workshops, June 2018.
[25] Diederik P Kingma and Jimmy Ba. Adam: method for stochastic optimization. arXiv preprint arXiv: 1412.690,, 2014.
[26] Adam Paszke, Sam Gross, Soumith Chintala, Gregory Chanan, Edward Yang. Zachary Vito, Zeming Lin, Alban Desmaison, Luca Antiga, and Adam Lerer. Automatic diffcrentiation in pytorch. 2017.
[27] Markku Makitalo and Alessandro Foi. Optimal inversion of the Anscombe transformation in low-count Poisson image denoising, IEEE transactions on Image Processing, 20(1):99-100 2010.


Reviews:
[{"review_id": "vcSAOinSolp", "reviewer": "Reviewer_yu4N", "summary": "The paper proposes a new approach based on Tweedie's formula to perform self-supervised image denoising. The proposed approach has two steps: (1) learn the score function or gradient of density of the noisy images, and (2) use Tweedie's formula to compute a denoised image using the learned score function. The proposed approach builds on multiple relevant work in the literature, and is well motivated. The model is tested on Gaussian, Poisson and Gamma noise removal tasks on natural images and achieves competitive results. ", "questions": "", "limitations": "", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper is well organized, and clearly written. The experiments are well thought and supports the main claim of the paper. Please find my specific comments below:\n\n1. **Missing baseline in Table 2**. The authors have done a thorough job of comparison, however, I think one of the relevant method is missing. N2V and N2S do not exploit noise type while learning the denoising function, and in general suffer from the drawback that they do not use the value of central pixel. Several post-processing schemes have been introduced to rectify this and incorporate the value of the central pixel back into the denoised output by assuming that the noise type is known. Given that the proposed method exploits the noise type, I think it is unfair to leave such methods out of comparison. For example, [High-Quality Self-Supervised Deep Image Denoising](https://arxiv.org/abs/1901.10277), Laine et. al, NeurIPS 2019 incorporates blind-spot in N2V and N2S into the architecture and has a post-processing scheme for Poisson and Gaussian noise. This post-processing scheme (again based on a Bayesian view) is shown to provide non-trivial improvement in denoising performance. When the noise type is unknown, Noise2Same [5] is a better baseline than N2V and N2S, since it promotes the use of center pixel. \n\n2. **Relation with Noise2X**. I appreciate the authors providing context and illuminating the relationship of their method with existing methods in the literature. However, I think the relation with Noise2Void discussed in lines 191-198 is force fitting a narrative. The main idea behind masking schemes in Noise2Void and Noise2Self is to discourage the use of central pixel in the denoising function, and it looks like this is a non-trivial interpretation to get from score matching perspective. In my understanding, the practical version of masking used in Noise2Void, pixel values are replaced with a random pixel from the neighbourhood. However, the proposed method maybe more releted to Noiser2Noise (see below). Also, one of the other strong points claimed by the authors is the university of the score matching cost function for different noise types. But the relation is explained only using SURE. I think there is no easy interpretation for PURE or other noise types, which maybe worth highlighting in the paper for an informed reader. \n\n3. **Noise2X and SURE beiing suboptimal for non-Gaussian**. In my understanding, Noise2Void or other blind-spot methods only make the assumption that the noise is pixel-wise independent. They are not even optimal for Gaussian denoising. SURE is derived explicitly assuming that the noise distribution is Gaussian, there are variants of SURE for most known noise types. See [Least squares estimation without priors or supervision. Raphan and Simoncelli. 2011](http://www.cns.nyu.edu/pub/eero/raphan10.pdf)\n\n4. **Noiser2Noise**: I think the method is more related to [Noiser2Noise, Moran et. al, CVPR 2019](https://arxiv.org/pdf/1910.11908.pdf). The training process, and the estimation formula is very related. Readers may benefit if the authors include this in related work, and discuss more about it. It appears to me that Noiser2Noise essentially has the same idea derived without referring Tweedie's formula and score matching.  \n\n5. **Real data**. This is completely optional, but I would also appreciate if the authors tried to apply this method to real dataset. I understand that the method assumes known noise distributions, but some of the real datasets, like [fluorescence microscopy dataset](https://arxiv.org/abs/1812.10366), [electron-microscopy dataset](https://arxiv.org/pdf/2010.12970.pdf), or [DND and SIDD](https://arxiv.org/pdf/2003.07761.pdf) have real noise which can be modeled fairly accurately. Even if the authors choose not to include experiments on these datasets, they can discuss these datasets as potential real applications of their method. \n\n**Minor nitpicking:**\nLines 221-234 makes it sounds like the proposed method makes it much easier to perform blind-denoising. However, for baselines like N2S or N2V or [Laine et. al](https://arxiv.org/abs/1901.10277) performing blind-denoising is easier than the proposed approach since they do not have to estimate the $\\sigma$ parameter. For SURE, one can just train for the whole range of $\\sigma$ (by using the ground-truth $\\sigma$ value), and apply the trained network directly to the test image. Thus the baseline methods do not require a separate $\\sigma$ estimation step. While I don't consider this as a major limitation, I think the authors should either tone done the claim, or acknowledge that competing methods can also perform blind-denoising just as/more easily. \n\nOverall I enjoyed reading the paper. Thanks for the great work. \n\n\n================== **Post-rebuttal** =================\n\nThank you for a detailed response to my comments. I'm happy with the responses, and I maintain my score. ", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "A major practical limitation, as the authors outlined is the assumption of a known noise type belonging to exponential distribution (and a way to estimate noise parameters at test time). ", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "6"}, "extra_text": {}}, {"review_id": "NGDXf3uJ_tI", "reviewer": "Reviewer_1non", "summary": "The authors how many recent self-supervised denoising methods can be related to Tweedie's formula and based on this insight they propose a novel method that can work with different noise distributions and outperform previous apporaches.", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Overall the paper is well written, even though some concepts can be hard to follow if the reader is not familiar with Tweedie's formula. The theoretical insight linking the formula to recent methods in the self-supervised denoising literature is interesting and novel and brings the authors to an original formulation of a denoising method. My main concern is the lack of discussion and comparisons with the work by Laine et al. \"High-Quality Self-Supervised Deep Image Denoising\", NeurIPS 2019, which showed how the various Noise2X methods are limited and also formulate a Bayesian framework leading to the estimation of the posterior mean of the image. While the two experimental settings are not directly comparable, it is seems that the work by Laine et al. already had performance competitive or superior to the one shown by the proposed method. The authors should address these points.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Societal impact has been discussed.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "1"}, "extra_text": {}}, {"review_id": "7ifCGir5_Rr", "reviewer": "Reviewer_kPzN", "summary": "This paper considers the problem of image denoising using deep learning. Traditionally, deep denoisers are trained in a supervised way using pairs of noisy and ground-truth images. Recent methods, such as Noise2Noise and those based on SURE, are used to train deep denoisers without ground truth.\n\nThis paper proposes Noise2Score as an alternative to existing methods. The idea of Noise2Score is to pre-train a denoising autoencoder (DAE) and use it as an estimator of the score function within the Tweedie’s formula. The key observation is that the pre-training stage of DAE doesn't explicitly need to know the true noise statistics. On other hand, the Tweedie’s formula enables the adaptation of the DAE to specific noise statistics.\n\nThe main conceptual contribution of the paper is that it establishes a formal link between the usage of DAEs for approximating score functions and image denoising. The empirical results show the competitive performance of Noise2Score against several well known image denoising methods, including Noise2Noise and SURE.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths: The key idea of Noise2Score—based on combining the DAE training with the Tweedie’s formula—is original in the context of denoising without ground truth. The method seems to perform well compared to related methods on simulations with several types of noise.\n\nMajor Issue 1: The current manuscript is vague on important theoretical/conceptual issues. (1) We know the theoretical conditions for both Noise2Noise and SURE to work, but not for Noise2Score. As an example, what are the assumptions necessary for the DAE to well approximate the score function? (2) How close should the noise statistics be to the one used in the Tweedie’s formula at the inference stage? If I believe the noise is AWGN and use Line 1 in Table 1, but the true noise is Poisson, how will this influence the result? (3) What is expected to happen if the training samples for p(y) in DAE training are not matched to the actual distribution of noise. (4) The noise estimation in (32) seems to use traditional denoising loss functions based on TV. Is there a reason one cannot leverage similar ideas used to develop Noise2Score? (5) The manuscript needs editing for clarity. The symbol sigma is overloaded as the noise variance and as a sampling parameter in (25). It is worth also mentioning that unlike Noise2Score, Noise2Noise does not need the knowledge of the statistical noise distribution at the inference stage.\n\nMajor Issue 2: Missing important details on empirical validation. (1) How was the UNet for Noise2Score trained. Was there only one Unet trained for all experiments or several? (2) How were the noisy images/patches sampled for training UNet? Were they picked equally from different distributions (Poisson, AWGN) or only from one? Was the training using only one level of noise or multiple? (3) Why wasn’t Noise2Noise tested on unknown parameters? Noise2Noise doesn’t need the knowledge of the parameter at the inference stage.\n\nSummary: This paper is proposing an intriguing alternative to Noise2Noise and related methods. However, the paper falls short in terms of fully explaining the concept and its theoretical properties. Additionally, empirical validation overlooks important details on training, which makes it difficult to understand the potential of Noise2Score to actually work.\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "No issues with this.", "review_text": "", "extra_scores": {"ethical_concerns": "No issues with this.", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3 hours"}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposes a new method, Noise2Score, for denoising images without clean references at training time. The proposed approach has two steps. First a denoising autoencoder (DAE) is used to learn the score function in a noise agnostic fashion. Second, the denoised image is computed through the estimated score function by exploiting Tweedie’s formula (which can be used with any exponential family noises). Empirical evaluation shows the benefits of the proposed approach (comparing to a large number of relevant baselines)

All reviewers found the proposed method interesting and novel. Furthermore, its connection with the recent literature in the subject (e.g. Noise2X) is a nice feature that will inform future research on the topic. 

In the original review, several clarifications were needed. The author provided a detailed response which clarified many of the raised concerns. After the rebuttal, all reviewers recommend accepting the paper. Reviewers kPzN and 1non updated their score (to 6 and 7 respectively). 

In sum, the work provides an interesting contribution and the current empirical evaluation is sufficient for accepting the paper. The AC encourages the authors to incorporate the detailed clarifications provided to the reviewers. 

Decision: Accept (Poster)