Title: Unsupervised Image Denoising with Score Function
Venue: NeurIPS 2023
Accepted: yes
Keywords: unsupervised learning

image denoising

score function
Abstract: Though achieving excellent performance in some cases, current unsupervised learning methods for single image denoising usually have constraints in applications. In this paper, we propose a new approach which is more general and applicable to complicated noise models. Utilizing the property of score function, the gradient of logarithmic probability, we define a solving system for denoising. Once the score function of noisy images has been estimated, the denoised result can be obtained through the solving system. Our approach can be applied to multiple noise models, such as the mixture of multiplicative and additive noise combined with structured correlation. Experimental results show that our method is comparable when the noise model is simple, and has good performance in complicated cases where other methods are not applicable or perform poorly.

Unsupervised Image Denoising with Score Function

Abstract

Though achieving excellent performance in some cases, current unsupervised learning methods for single image denoising usually have constraints in applications. In this paper, we propose a new approach which is more general and applicable to complicated noise models. Utilizing the property of score function, the gradient of logarithmic probability, we define a solving system for denoising. Once the score function of noisy images has been estimated, the denoised result can be obtained through the solving system. Our approach can be applied to multiple noise models, such as the mixture of multiplicative and additive noise combined with structured correlation. Experimental results show that our method is comparable when the noise model is simple, and has good performance in complicated cases where other methods are not applicable or perform poorly.

1 Introduction

Image denoising [4, 20, 26] has been studied for many years. Suppose x is a clean image, y is a noisy image of x , and p ( y | x ) is the noise model. Supervised learning methods try to train a model representing a mapping from y to x . Due to the difficulty of collecting paired clean and noisy images in practice, methods in unsupervised learning manner are focus of research. Noise2Noise [12] is the first to use pairs of two different noisy images constructed from the same clean image to train a denoising model. Strictly speaking, Noise2Noise is not an unsupervised learning method. Collecting noisy pairs is still difficult. Despite all this, many other methods [2, 10, 11, 24, 25] are inspired from Noise2Noise or borrow the idea behind of it. These methods achieve good performance in some simple noise models.

The main drawback of current methods is the constraint of application. Once the noise model is complicated, they either are not applicable or have poor performance. Noisier2Noise [16] can only handle additive noise and pepper noise. Recorrupted-to-Recorrupted [17] is limited to Gaussian noise. Neighbor2Neighbor [7] requires that the noise model is pixel-wise independent and unbiased ( i.e. E [ y | x ] = x ). Noise2Score [9] applies Tweedie's Formula to image denoising and provides a unified framework for those noise models that follow exponential family distributions. However, the practical noise model will be more complicated. It may contain both multiplicative and additive noise, even combining with structural correlation.

In this paper, we propose a new unified approach to handle more noise models. The key of our approach is a theoretical property about score function, ∇ y log p ( y ) , which is shown in Proposition 3.1. This property indicates that the score function of y is the average of the score function of y | x under the posterior distribution p ( x | y ) . Based on it, we define a system that the score function of y equals to the score function of y | x , which turns out to be an equation about x since y is known. We

estimate

solve

s(y) =Vylogp(ylx)

scorefunction

equation

denoised result

discover that the solution of this equation can be regarded as the denoised result of y . Implementing our approach practically contains two steps: the first is to estimate the score function of y and the second is to solve the system defined above according to the specific noise model. The overall flow is shown in Figure 1. All the model training is the estimation of score function, which is represented by a neural network. We adapt the amortized residual denoising autoencoder (AR-DAE) [13] to figure out it. More details can be seen in Section 3.3.

Our approach is so powerful that as long as the score function of y | x is known and the system is solvable, any kind of noise model is applicable. It means our approach is even able to solve sophisticated noise models such as mixture noise. Another advantage of our approach is that regardless of noise models, the training process of the score function neural network is identical. Therefore, once the assumption of the noise model does not hold or parameters of noise are corrected, we only change the equation system to be solved and resolve it without training the model again. In summary, our main contribution are: (1) We propose a general unsupervised approach for image denoising, which is based on the score function. (2) Experimental results show that our approach is competitive for simple noise models and achieves excellent performance for complicated noise models where other unsupervised methods is invalid.

2 Related Works

Here we briefly review the existing deep learning methods for image denoising. When the pairs of clean images and noisy images are available, the supervised training [26] is to minimize E x , y [d ( x , f ( y ; θ ))] , where f ( · ; θ ) is a neural network and d( · , · ) is a distance metric. Though supervised learning has great performance, the difficult acquisition for training data hampers its application in practice.

To avoid the issues of acquisition for paired clean and noisy images, unsupervised 1 approaches are proposed to use noisy image pairs to learn image denoising, which is started from Noise2Noise (N2N) [12]. While N2N can achieve comparable results with supervised methods, collecting noisy image pairs from real world is still intracTable Motivated by N2N, the followed-up works try to learn image denoising with individual noisy images. Mask-based unsupervised approaches [2] design mask schemes for denoising on individual noisy images and then they train the network to predict the masked pixels according to noisy pixels in the input receptive field. Noise2Void [10] also proposes the blind-spot network (BSN) to avoid learning the identity function. Noisier2Noise [16], Noisy-AsClean [25] and Recorrupted-to-Recorrupted [17] require a single noisy realization of each training sample and a statistical model of the noise distribution. Noisier2Noise first generates a synthetic noisy sample from the statistical noise model, adds it to the already noisy image, and asks the network to predict the original noisy image from the doubly noisy image. Besides the blind-spot network design, Self2Self [19] is proposed on blind denoising to generate paired data from a single noisy image by applying Bernoulli dropout. Recently, Neighbor2Neighbor [7] proposes to create subsampled paired images based on the pixel-wise independent noise assumption and then a denoising network is trained on generated pairs, with additional regularization loss for better performance. Noise2Score [9] is another type of unsupervised methods that can deal with any noise model which follows an exponential family distribution. It utilize Tweedie's Formula [21, 6] and the estimation of score function to denoise noisy images.

3 Method

In this section, we provide a more detailed description of the proposed approach. The organization is as follows: In Section 3.1 we review Noise2Score, introduce the basic theory and present the relationship between our method and Noise2Score. After that, we derive the specific algorithms for different examples of noise models in Section 3.2. Finally, we describe the method for estimating the score function ∇ y log p ( y ) in Section 3.3. All the proofs are in Supplementary Material.

3.1 Basic Theory

3.1.1 Noise2Score

Suppose the noise model follows some exponential family distribution and has the following form:

where T ( y ) and ( x ) have the same dimensions, and a ( x ) and b ( y ) are both scalar functions. One of the properties of an exponential family distribution is that:

According to Bayesian Formula, we have the following derivation:

Therefore, p ( x | y ) is also an exponential family distribution. According to Eq. 2, we can derive a conclusion about ∇ y log p ( y ) (the score function of y ), Tweedie's Formula, that

In Noise2Score, the denoised result ˆ x of y is the solution of following equation:

Since Noise2Score is derived from Tweedie's Formula, it is limited to the noise models of exponential family distributions.

3.1.2 Proposed Approach

In this work, the proposed approach generalized Noise2Score to non-exponential family distributions. We begin with the following proposition.

Proposition 3.1 Let y ∼ y and x ∼ x where x and y are two random variable, then the equation below holds:

The proof of Proposition 3.1 is in Supplementary Material. In the task of image denoising, suppose y ∈ R d is the noisy image and x ∈ R d is the clean image where d represents the number of image pixels. p ( x | y ) represents the posterior distribution and p ( y | x ) denotes the noise model. When p ( y | x ) follows some exponential family distribution, it is easy to verify that the right part of Eq. 6 equals to the right part of Tweedie's Formula (Eq. 4). Therefore, the equation (Eq. 5) to be solved in Noise2Score in essence can be written as:

We use ˆ x rather than x to indicate that ˆ x is the unknown variable to be solved in the equation. Hence, we propose a new unsupervised denoising method that given noisy image y , the denoised result can be obtained by solving Eq. 7 . When the noise model follows some exponential family distribution, our method is the same as Noise2Score. The specific conclusions of Gaussian, Gamma and Poisson noise can be seen in Supplementary Material. However, different from Noise2Score, our approach is applicable to some complex non-exponential family distributions as long as the score function of y and noise model p ( y | x ) are known. For convenience, let s ( y ) = ∇ y log p ( y ) and f ( x , y ) = ∇ y log p ( y | x ) . Our approach is consist of two steps as shown in Figure 1. The general algorithm is illustrated in Algorithm 1.

The following theorem explains why the solution of Eq. 7 is a good denoised result of y .

Algorithm 1 The general denoising process

Input:

Output: ˆ x , the solution of Eq. 7.

1: Estimate s ( y ) .

2: Solve s ( y ) = f ( x , y ) .

Theorem 3.1 Given y , suppose the Hessian matrix f ( x , y ) is bounded, f ( x , y ) is invertible with respect to x , and its inverse function is Lipschitz continuous. Denote ˆ x as the solution of Eq. 7, then,

where Cov[ x | y ] is the covariance matrix and C is a constant.

The proof of Theorem 3.1 is in Supplementary Material. Theorem 3.1 shows the lower noise level, the smaller the error between ˆ x and E x | y [ x ] , which guarantees the effectiveness of proposed method.

Next, in Section 3.2 we will derive the specific algorithm for solving x in Eq. 7 under different noise models, including additive Gaussian noise in Section 3.2.1, multiplicative noise in Section 3.2.2 and mixture noise in Section 3.2.3. Estimating score function s ( y ) will be introduced in Section 3.3.

3.2 Application Examples

3.2.1 Additive Gaussian Noise

Suppose the noise model is y = x + ϵ where ϵ follows a multi-variable Gaussian distribution with mean of 0 and covariance matrix of Σ denoted by N ( 0 , Σ ) , i.e.

Then, we derive that f ( x , y ) = ∇ y log p ( y | x ) = -Σ -1 ( y -x ) . We consider the following four kinds of Σ : (1) Σ = σ 2 I ; (2) Σ = σ 2 A ⊤ A ; (3) Σ = Σ ( x ) = diag ( a x + b 1 ) 2 ; (4) Σ = Σ ( x ) = A ⊤ diag ( a x + b 1 ) 2 A . In the second and fourth cases, A usually represents a convolution transform which describes the correlation between adjacent pixels.

For the first and second cases, Σ is a constant matrix. By solving Eq. 7, we have

While in the third and fourth cases, Σ is related to x and Eq. 10 is a fixed point equation. Therefore, we use an iterative trick to solve it as shown in Algorithm 2.

Algorithm 2 An iterative trick to solve x = Σ ( x ) s ( y ) + y

Input: noisy image y , s ( y ) , the parameters of Σ ( · ) and the number of iterations n . Output: ˆ x , the solution.

1: Initial value of ˆ x is set as y .

2: for i = 1 , ..., n do

3: ˆ x ← Σ (ˆ x ) s ( y ) + y .

4: end for

3.2.2 Multiplicative Noise

Firstly, we discuss three types of multiplicative noise model, Gamma, Poisson and Rayleigh noise. Then, we consider the situation where the convolution transform A exists.

Gamma Noise is constructed from Gamma distribution, G ( α, β ) : p ( x ; α, β ) = β α Γ( α ) x α -1 e -βx , and is defined as y = η ⊙ x , where η i ∼ G ( α, α ) , α > 1 , and ⊙ means component-wise multiplication, i.e.

Then, we derive that f ( x , y ) = ∇ y log p ( y | x ) = α -1 y -α x . Here, the division is component-wise division. In the residual part of this paper, we neglect such annotation if it is not ambiguous. By solving Eq. 7, we have

Poisson Noise is constructed from Poisson distribution, P ( λ ) : Pr( x = k ) = λ k k ! e -λ , k = 0 , 1 , · · · , and is defined as y = 1 λ η , η i ∼ P ( λx i ) , λ > 0 , i.e.

Then, we derive that f ( x , y ) = ∇ y log Pr ( y | x ) = λ log ( λ x ) -λ log ( λ y + 1 2 ) . By solving Eq. 7, we have

Rayleigh Noise is constructed from Rayleigh distribution, R ( σ ) : p ( x ; σ ) = x σ 2 exp { -x 2 2 σ 2 } , and is defined as y = ( η + 1 ) ⊙ x , η i ∼ R ( σ ) , σ > 0 , i.e.

Then, we derive that f ( x , y ) = ∇ y log p ( y | x ) = 1 y -x -y -x σ 2 x 2 . Solving Eq. 7 directly is not easy. Here we provide an iterative algorithm to solve it. It is illustrated in Algorithm 3.

Algorithm 3 An iterative method to solve Eq. 7 in the case of Rayleigh noise

Input: noisy image y , s ( y ) , the parameter of Rayleigh noise σ and the number of iterations n . Output: ˆ x , the solution of Eq. 7.

1: Initial value of ˆ x is set as y .

2: for i = 1 , ..., n do

3: Compute b = σ 2 s ( y ) ⊙ ˆ x . √

5: ˆ x ← y t + 1 .

4: Compute t = 1 2 ( -b + b ⊙ b +4 σ 2 1 ) .

6: end for

Now, we consider the situation where the convolution transform A exists. Suppose the noise model is represented by y = Az , z = N ( x ) , where N ( x ) can be any multiplicative noise model discussed above. Then, we have ∇ y log p y ( y | x ) = A -1 , ⊤ ∇ z log p z ( A -1 y | x ) . To avoid confusion, we use subscripts to distinguish different distribution. Therefore, we can apply Algorithm 4 to solve Eq. 7, which is shown as follows.

Algorithm 4 The general framework to solve Eq. 7 for correlated multiplicative noise model

Input: y , s ( y ) , A and ˜ f ( x , z ) = ∇ z log p z ( z | x ) . Output: ˆ x , the solution of Eq. 7.

1: Computing ˜ s = A ⊤ s ( y ) .

2: Computing z = A - 1 y .

3: Solve ˜ s = ˜ f ( x , z ) by the corresponding Algorithm

3.2.3 Mixture Noise

In this paper, the mixture noise model is composed of a multiplicative noise and an additive Gaussian noise. We denote it as y = z + ϵ , ϵ ∼ N ( 0 , σ 2 I ) , z = A N ( x ) , where N ( x ) is any multiplicative

noise model that can be solved by our approach and A is either a convolution transform or identity matrix. It is easy to derive that p y ( y | x ) = ∫ p y ( y | z ) p z ( z | x ) d z . Generally speaking, p y ( y | x ) has not an explicit analytical form. In this paper, we assume that the additive Gaussian noise is far smaller than the multiplicative noise. Thus, we utilize Taylor expansion to approximate p y ( y | x ) . We have the following conclusion:

where ¯ z = E [ z | y ] . Then, we can further derive that

The full and rigorous derivations of Eq. 16 and Eq. 17 are in Supplementary Material. Applying Eq. 10 in Section 3.2.1, we have ¯ z = y + σ 2 s ( y ) . Thus, the equation to be solve is s ( y ) = f ( x , ¯ z ) . The full denoising process is illustrated in Algorithm 5.

Algorithm 5 The full denoising process for mixture noise y = z + ϵ

Input:

y , s ( y ) .

Output: ˆ x , the solution of s ( y ) = f ( x , ¯ z ) .

1: Computing ¯ z = y + σ 2 s ( y ) .

2: Solve s ( y ) = f ( x , ¯ z ) through the corresponding algorithm discussed in Section 3.2.2.

3.3 Estimation of Score Function

So far, we assume that the score function of y , s ( y ) , is known. However, it is usually unknown and should be estimated from the dataset of noisy images { y } . We use the same method, the amortized residual Denoising Auto Encoder (AR-DAE) [13] discussed in Noise2Score. Suppose s ( · ; θ ) is a neural network used to represent the score function of y . The following objective function is used to train the model:

where σ a is a fixed value. Given σ a , the optimal model s ( y ; θ ∗ ) that minimizes L is the score function of perturbed y , y + σ a u . In other words, we can approximate the score function s ( y ) by using a sufficiently small σ a . Related analysis can also be seen in [8, 23, 1]. During the training process, the value of σ a will be decreasing gradually to a very small value. This progressive process is helpful to numerically stabilize the model training. The only model training is to estimate s ( y ) by s ( · ; θ ) , which is served as the first step of our approach. After the score function model is trained, we apply the denoising algorithms given in Section 3.1 to obtain denoised results and no more training is required.

4 Experiment

Dataset and Implementation Details We evaluate the proposed method for color images in the three benchmark datasets containing RGB natural images: Kodak dataset, CBSD68 [15] and CSet9. DIV2K [22] and CBSD500 dataset [3] are used as training datasets. The synthetic noise images for each noise model are generated and fixed through the training process. For the sake of fair comparison, we use the same modified U-Net [5] for all methods. When training, we randomly clip the training images to patches with the resolution of 128 × 128 . AdamW optimizer [14] is used to train the network. We train each model for 5000 steps with the batch size of 32. To reduce memory, we utilize the tricks of cumulative gradient and mixed precision training. The learning rate is initialized to 1 × 10 -4 for first 4000 steps and it is decreased to 1 × 10 -5 for final 1000 steps. All the models are implemented in PyTorch [18] with NVidia V100. The pixel value range of all clean images is [0 , 255] and the parameters of noise models are built on it. Noisy images will be scaled when fed into the network. When an iterative algorithm is needed to solve Eq. 7, we set the number of iterations as 10 . The more details of implementation are described in Supplementary Material.

Baseline and Comparison Methods We use supervised learning with MSE loss as the baseline model. Noisier2Noise and Neighbor2Neighbor are used as comparison methods. Since our approach

is identical to Noise2Score when the noise model follows exponential family distributions, we do not compare to it through metrics. Because Noisier2Noise can only be applied to additive noise, we do not train models by Noisier2Noise for other noise models. Though Neighbor2Neighbor is not suitable for some noise models from the perspective of theoretical analysis, we still train corresponding models and report its results. Table 1 shows the comparison of application range for different methods, including additive Gaussian noise, multiplicative noise and mixture noise. Based on it our experiments are conducted. Only supervised learning and our approach can handle all noise models listed in Table 1.

Parameters of Noise Models Here, we emphasize that for all noise models in our experiments, A is set as a 3 × 3 convolution transform with the kernel of

if it is used. The additive Gaussian noise in every mixture noise model is set as N ( 0 , 100 I ) . Other parameters will be given later. Finally, all parameters are assumed to be known in our experiments.

Additive Gaussian Noise Using additive Gaussian noise, we consider four kinds of noise models with different Σ corresponding from No. 1 to No. 4 in Table 1. Our method is compared to supervised learning, Noisier2Noise and Neighbor2Neighbor as shown in Table 2. For the first two noise models σ is 25 , and for the rest a = 0 . 98 and b = 25 . As expected, supervised learning performs best. In the cases without A Neighbor2Neighbor is the best among three other unsupervised learning methods. However, in the cases with A Neighbor2Neighbor performs very poorly. Our approach outperform other unsupervised learning methods in the second noise model and is competitive in the whole.

Multiplicative Noise We consider the combination of three various multiplicative noise model (Gamma, Poisson and Rayleigh) and a convolution transform A . They are corresponding from No. 5 to No. 10 in Table 1. Our method is compared to supervised learning and Neighbor2Neighbor and the results are shown in Table 3. Because Noisier2Noise can not address such multiplicative noise models, we neglect it. Though the noise is not pixel-wise independent when the convolution transform exists,

we can execute its inverse transform on y so that the requirement of pixel-wise independence is satisfied for Neighbor2Neighbor. Therefore, tackling noise models with a convolution transform is equivalent to the situations without A . That is why we do not provide corresponding metrics result for Neighbor2Neighbor in Table 3. We set α as 26 for the Gamma noise, λ as 0 . 2 for the Poisson noise, and σ as 0 . 3 for the Rayleigh noise. When the noise model is based on Gamma or Poisson noise, it is unbiased, i.e. E [ y | x ] = x . In these cases, Neighbor2Neighbor is better than ours. However, when the noise model is based on Rayleigh noise which is biased our approach still has excellent performance while Neighbor2Neighbor is poor.

Mixture Noise We also consider the combination of three various multiplicative noise model (Gamma, Poisson and Rayleigh) and a convolution transform A . For each one, additive Gaussian noise with Σ = 100 I is added to construct mixture noise models. They are corresponding from No. 11 to No. 16 in Table 1. Because of the same reason discussed before, our method is compared to supervised learning and Neighbor2Neighbor, and Noisier2Noise is neglected. The experimental

33.5

33.0

32.5

32.0

PSNR

31.5

31.0

30.5

Poisson + Conv + Gaussian

30.0

Rayleigh + Gaussian

0.00

0.02

0.04

0.06

0.08

0.10

results are shown in Table 4. Due to the additive Gaussian Noise, Neighbor2Neighbor are not able to handle the cases with A through the inverse convolution transform. The correlation of noise hampers the performance of Neigh2bor2Neighbor. Except the first noise model in Table 4, our approach all outperforms Neighbor2Neighbor and achieve excellent performance near to supervised learning.

Robustness Evaluation To apply our approach, the parameters of noise models have to be known beforehand. Therefore their precision may impact on the performance. We choose No.14 and No.15 noise models in Table 1 as examples to show the robustness to parameters' precision. Suppose k is one of parameters, we disturb it by ( a +1) k where a ∼ N (0 , r 2 ) and r is called the disturbance rate. The larger r , the less precise the noise model. As r is increasing, the PSNR of the denoised result is shown in Figure 2. When r = 0 . 1 , the PSNR reduces about 1 dB, which displays the robustness.

5 Conclusion

In this paper, we propose a new approach for unsupervised image denoising. The key part is Proposition 3.1. Based on it, we construct an equation, Eq. 7 about the clean image x and the noisy image y . After the score function of y is estimated, the denoised result can be obtained by solving the equation. Our approach can be applied to many different noise model as long as Eq. 7 is solvable. The denoising performance is competitive for simple noise models and excellent for complicated ones. We hope that this work is helpful to address sophisticated image denoising problems in practice.

Noisy

GT

SL

Nb2Nb

Ours

Noisy

GT

SL

Nr2N

Ours

Noisy

GT

SL

Nb2Nb

Ours

Noisy.

Nb2Nb

urs

Noisy

Nb2Nb

Ours

References

Guillaume Alain and Yoshua Bengio. What regularized auto-encoders learn from the datagenerating distribution. The Journal of Machine Learning Research , 15(1):3563-3593, 2014.

Joshua Batson and Loic Royer. Noise2self: Blind denoising by self-supervision. In International Conference on Machine Learning , pages 524-533. PMLR, 2019.

Lalit Chaudhary and Yogesh Yogesh. A comparative study of fruit defect segmentation techniques. In 2019 International Conference on Issues and Challenges in Intelligent Computing Techniques (ICICT) , volume 1, pages 1-4. IEEE, 2019.

Kostadin Dabov, Alessandro Foi, Vladimir Katkovnik, and Karen Egiazarian. Image denoising with block-matching and 3d filtering. In Image processing: algorithms and systems, neural networks, and machine learning , volume 6064, pages 354-365. SPIE, 2006.

Prafulla Dhariwal and Alexander Nichol. Diffusion models beat gans on image synthesis. Advances in Neural Information Processing Systems , 34:8780-8794, 2021.

Bradley Efron. Tweedie's formula and selection bias. Journal of the American Statistical Association , 106(496):1602-1614, 2011.

Tao Huang, Songjiang Li, Xu Jia, Huchuan Lu, and Jianzhuang Liu. Neighbor2neighbor: Selfsupervised denoising from single noisy images. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 14781-14790, 2021.

Aapo Hyvärinen and Peter Dayan. Estimation of non-normalized statistical models by score matching. Journal of Machine Learning Research , 6(4), 2005.

Kwanyoung Kim and Jong Chul Ye. Noise2score: tweedie's approach to self-supervised image denoising without clean images. Advances in Neural Information Processing Systems , 34:864-874, 2021.

Alexander Krull, Tim-Oliver Buchholz, and Florian Jug. Noise2void-learning denoising from single noisy images. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 2129-2137, 2019.

Alexander Krull, Tomáš Viˇ car, Mangal Prakash, Manan Lalit, and Florian Jug. Probabilistic noise2void: Unsupervised content-aware denoising. Frontiers in Computer Science , 2:5, 2020.

Jaakko Lehtinen, Jacob Munkberg, Jon Hasselgren, Samuli Laine, Tero Karras, Miika Aittala, and Timo Aila. Noise2noise: Learning image restoration without clean data. In International Conference on Machine Learning , page 2965-2974. PMLR, 2018.

Jae Hyun Lim, Aaron Courville, Christopher Pal, and Chin-Wei Huang. Ar-dae: towards unbiased neural entropy gradient estimation. In International Conference on Machine Learning , pages 6061-6071. PMLR, 2020.

Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. arXiv preprint arXiv:1711.05101 , 2017.

David Martin, Charless Fowlkes, Doron Tal, and Jitendra Malik. A database of human segmented natural images and its application to evaluating segmentation algorithms and measuring ecological statistics. In Proceedings Eighth IEEE International Conference on Computer Vision. ICCV 2001 , volume 2, pages 416-423. IEEE, 2001.

Nick Moran, Dan Schmidt, Yu Zhong, and Patrick Coady. Noisier2noise: Learning to denoise from unpaired noisy data. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 12064-12072, 2020.

Tongyao Pang, Huan Zheng, Yuhui Quan, and Hui Ji. Recorrupted-to-recorrupted: unsupervised deep learning for image denoising. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 2043-2052, 2021.

Adam Paszke, Sam Gross, Soumith Chintala, Gregory Chanan, Edward Yang, Zachary DeVito, Zeming Lin, Alban Desmaison, Luca Antiga, and Adam Lerer. Automatic differentiation in pytorch. 2017.

Yuhui Quan, Mingqin Chen, Tongyao Pang, and Hui Ji. Self2self with dropout: Learning self-supervised denoising from single image. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 1890-1898, 2020.

Sathish Ramani, Thierry Blu, and Michael Unser. Monte-carlo sure: A black-box optimization of regularization parameters for general denoising algorithms. IEEE Transactions on image processing , 17(9):1540-1554, 2008.

H Robbins. An empirical bayes approach to statistics, proceedings of the third berkeley symposium on mathematical statistics and probability, 1956.

Radu Timofte, Shuhang Gu, Jiqing Wu, and Luc Van Gool. Ntire 2018 challenge on single image super-resolution: Methods and results. In Proceedings of the IEEE conference on computer vision and pattern recognition workshops , pages 852-863, 2018.

Pascal Vincent. A connection between score matching and denoising autoencoders. Neural computation , 23(7):1661-1674, 2011.

Xiaohe Wu, Ming Liu, Yue Cao, Dongwei Ren, and Wangmeng Zuo. Unpaired learning of deep image denoising. In European conference on computer vision , pages 352-368. Springer, 2020.

Jun Xu, Yuan Huang, Ming-Ming Cheng, Li Liu, Fan Zhu, Zhou Xu, and Ling Shao. Noisy-asclean: Learning self-supervised denoising from corrupted image. IEEE Transactions on Image Processing , 29:9316-9329, 2020.

Kai Zhang, Wangmeng Zuo, Yunjin Chen, Deyu Meng, and Lei Zhang. Beyond a gaussian denoiser: Residual learning of deep cnn for image denoising. IEEE transactions on image processing , 26(7):3142-3155, 2017.

Peer reviews:
comments: SUMMARY:
The paper introduces an extension of Noise2Score for unsupervised image denoising and demonstrates its efficiency on several noise models, including non-exponential family distributions.

STRENGTHS:
The extension of Noise2Score allows to deal with different kinds of noise, as opposed to prior works that only focus on exponential family distributions. The paper is clearly-written, with a fair amount of examples and experiments.

WEAKNESSES:
The contribution is relatively incremental compared to Noise2Score, and is based on an approximation (equation (7) replaces equation (6) in the resolution) that is not really discussed. Even though Theorem 3.1 gives some upper bound on the distance between the estimate and the conditional expectation, it would have been interesting to get insights on the constant $C$ and on the covariance of $x$ given $y$ in practice, in particular with complex noise models the paper aims to deal with.

QUESTIONS:
* If possible, could the authors give numerical details on the approximation error between equations (6) and (7) in practical cases that differ from exponential family distributions, and elaborate on the hypotheses (f invertible, inverse function Lipschitz continuous, and bounded Hessian) ?

* Could the authors exhibit ill-posed cases for which this approximation wouldn't be accurate, and show the implication on the denoising algorithm ?

LIMITATIONS:
See remarks above.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-06-18

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents a self-supervised learning algorithm for image denoising which only requires the score of the noisy image distribution to perform denoising. The algorithm builds on ideas of a recent paper Noise2Score, extending the family of noise distributions that can be handled via an application of Fisher's identity.\n\nSTRENGTHS:\n- The paper presents a principled way of performing self-supervised denoising with a large family of noise distributions, which go beyond the exponential family. The previous Noise2Score only handles distributions belonging to the exponential family.\n- The paper demonstrates the good performance of the proposed approach on a large number of denoising experiments.\n\nWEAKNESSES:\n- The authors do not compare with SURE-based methods, which can handle mixture distributions, such as mixed Poisson Gaussian noise, e.g., see \"An unbiased risk estimator for image denoising in the presence of mixed Poisson–Gaussian noise\" by LeMontagner et al.\n\n- While the idea is theoretically stimulating, I don't see a lot of practical applications for the family of distributions where SURE or Noise2Score would fail. It would be good if the authors could specify some.\n\n- Some statements such as \"Our approach is so powerful that\" should be toned down. In particular, note that the approach cannot handle *any* noise distribution. See for example the discussion on Binomial noise in \"Least squares estimation without priors or supervision\" by Raphan and Simoncelli.\n\nQUESTIONS:\nIf I'm not wrong, the main formula in Proposition 3.1 is just Fisher's identity? It would be good to name this accordingly and cite some references.\n\nIn Theorem 3.1, what does \"f(x,y) is invertible\" means? Invertible with respect to all variables? There is also a typo in Lipschitz continuous.\n\nLIMITATIONS:\nThe limitations are not discussed in detail. It would be good to specify that the method strongly relies on the knowledge of the noise distribution, whereas other methods such as Neighbor2Neighbor do not.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-04"}

comments: SUMMARY:
The following work attempts to solve a single image denoising task in an unsupervised fashion. The authors propose to predict the score function and then denoise the input noisy image by solving the system of equations. Moreover, the proposed approach is a generalization of the Noise2Score method, which can work for complex non-exponential family distribution. Extensive experimental results demonstrate comparable and superior results in complex cases over existing unsupervised denoising methods.

STRENGTHS:
+ Proposed extension of Noise2Score to images with complex noise distribution is very appealing and substantially increases the chances to apply the method to real-world scenarios.
+ Extensive experimental results on additive, multiplicative, and mixed noises.

WEAKNESSES:
- Proposed method is highly susceptible to noise parameter estimations according to Figure 2.
- Moreover, one needs to understand the noise distribution type (Gamma Noise, Poisson Noise, etc.) of an arbitrary input image to estimate particular parameters of the distribution. Then those estimations are further used to solve Eq. 7.
- It would be better to apply the proposed method to real image denoising given an estimated noise parameters. For example, raw image denoising (e.g. SIDD, DnD).

QUESTIONS:
Suggestions:
•	It would be better to move some of the details in the implementation part. For example, AR-DAE, which is used for score function estimation.
•	Typos (e.g. line 114)
•	I would suggest to add some visual examples in the Supplementary (if the main paper does not have enough space).

LIMITATIONS:
not applicable
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-04

comments: SUMMARY:
This paper proposes a general unsupervised image denoising approach by solving system with an estimated score function. The proposed system can be applied on multiple noise models by changing the equation system rather than retraining the model.

STRENGTHS:
1)	Compared to the Noise2Score limited to the noise models of exponential family distributions, the proposed method unlocks the limitation and generalizes to the non-exponential family distributions.
2)	For different noise model, the proposed method only requires to modify the equation system to be solved and keeps the trained score function estimator unchanged.
3)	The applications on the additive noise, multiplicative noise, and mixture noise have been proved in the manuscript and the supplementary materials.

WEAKNESSES:
1)	Qualitative comparison should be provided in the supplementary materials.
2)	The Noise2Score can be applied to the unknown noise parameters, and whether the proposed methods can achieve it? Please provide the discussion and experiments.

QUESTIONS:
The overall idea presented in this paper is interesting and the results are promising. Since I am not very sure about the application of the unknown noise parameters, I may defer my recommendation after authors' response.

LIMITATIONS:
N.A.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-06

comments: SUMMARY:
This paper proposes a unsupervised image denoising method based on score function. Compared with Noise2Score, it discards the noise distribution on exponential family, and derives the solutions for additive Gaussion noise, multiplicative noise, and mixture noise. The experiments verify the effectiveness of the proposed method.

STRENGTHS:
1. The  proposed method is theoretically sound.
2. It is a general unsupervised method, being able to deal with many kinds of noise types.
3. The synthetic experimental results showcase that it is superior or comparable to recent SoTA method.

WEAKNESSES:
1. The proposed method is built on the core equation of Eq. (7).  The detailed mathematical derivation of this equation is not clearly presented. One of the main contributions of this work is that it dose not rely on the exponential family distribution. However, according to my understanding, Eq. (7) still follows the exponential family distribution. The writing of this part should be further improved.

2. The introduction section states that "Another advantage of our approach is that regardless of noise models, the training process of the score function neural network is identical."  In my opinion, it still requires to individually train the models on various noise types.

3. The most limitation of this work is its practicality. It depends on the specific parameters of the noise distribution, thus cannot to deal with real-word denoising task. The experiments all focus on the synthetic datasets.

QUESTIONS:
See weakness.

LIMITATIONS:
As explained in the weakness, this work cannot handle the blind or real-world denoising task.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 4
DATE: 2023-07-23