SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Yutong Xie
yutongxie_research@163.com
Mingze Yuan
mzyuan@pku.edu.cn
Bin Dong
dongbin@bicmr.pku.edu.cn
Quanzheng Li
Massachusetts General Hospital and Harvard Medical School
li.quanzheng@mgh.harvard.edu
Though achieving excellent performance in some cases, current unsupervised learning methods for single image denoising usually have constraints in applications. In this paper, we propose a new approach which is more general and applicable to complicated noise models. Utilizing the property of score function, the gradient of logarithmic probability, we define a solving system for denoising. Once the score function of noisy images has been estimated, the denoised result can be obtained through the solving system. Our approach can be applied to multiple noise models, such as the mixture of multiplicative and additive noise combined with structured correlation. Experimental results show that our method is comparable when the noise model is simple, and has good performance in complicated cases where other methods are not applicable or perform poorly.
Image denoising [4, 20, 26] has been studied for many years. Suppose  $\mathbf{x}$  is a clean image,  $\mathbf{y}$  is a noisy image of  $\mathbf{x}$ , and  $p(\mathbf{y} \mid \mathbf{x})$  is the noise model. Supervised learning methods try to train a model representing a mapping from  $\mathbf{y}$  to  $\mathbf{x}$ . Due to the difficulty of collecting paired clean and noisy images in practice, methods in unsupervised learning manner are focused on research. Noise2Noise [12] is the first to use pairs of two different noisy images constructed from the same clean image to train a denoising model. Strictly speaking, Noise2Noise is not an unsupervised learning method. Collecting noisy pairs is still difficult. Despite all this, many other methods [2, 10, 11, 24, 25] are inspired from Noise2Noise or borrow the idea behind of it. These methods achieve good performance in some simple noise models.
The main drawback of current methods is the constraint of application. Once the noise model is complicated, they either are not applicable or have poor performance. Noisier2Noise [16] can only handle additive noise and pepper noise. Recorrupted-to-Recorrupted [17] is limited to Gaussian noise. Neighbor2Neighbor [7] requires that the noise model is pixel-wise independent and unbiased (i.e.  $\mathbb{E}[y\mid x] = x$ ). Noise2Score [9] applies Tweedie's Formula to image denoising and provides a unified framework for those noise models that follow exponential family distributions. However, the practical noise model will be more complicated. It may contain both multiplicative and additive noise, even combining with structural correlation.
In this paper, we propose a new unified approach to handle more noise models. The key of our approach is a theoretical property about score function,  $\nabla_{\pmb{y}}\log p(\pmb{y})$ , which is shown in Proposition 3.1. This property indicates that the score function of  $\pmb{y}$  is the average of the score function of  $\pmb{y} \mid \pmb{x}$  under the posterior distribution  $p(\pmb{x} \mid \pmb{y})$ . Based on it, we define a system that the score function of  $\pmb{y}$  equals to the score function of  $\pmb{y} \mid \pmb{x}$ , which turns out to be an equation about  $\pmb{x}$  since  $\pmb{y}$  is known. We
Figure 1: The overall flow of our approach. The first step is to estimate the score function  $s(\pmb{y})$  and the second step is to solve an equation.
discover that the solution of this equation can be regarded as the denoised result of  $\mathbf{y}$ . Implementing our approach practically contains two steps: the first is to estimate the score function of  $\mathbf{y}$  and the second is to solve the system defined above according to the specific noise model. The overall flow is shown in Figure 1. All the model training is the estimation of score function, which is represented by a neural network. We adapt the amortized residual denoising autoencoder (AR-DAE) [13] to figure out it. More details can be seen in Section 3.3.
Our approach is so powerful that as long as the score function of  $\mathbf{y} \mid \mathbf{x}$  is known and the system is solvable, any kind of noise model is applicable. It means our approach is even able to solve sophisticated noise models such as mixture noise. Another advantage of our approach is that regardless of noise models, the training process of the score function neural network is identical. Therefore, once the assumption of the noise model does not hold or parameters of noise are corrected, we only change the equation system to be solved and resolve it without training the model again. In summary, our main contribution are: (1) We propose a general unsupervised approach for image denoising, which is based on the score function. (2) Experimental results show that our approach is competitive for simple noise models and achieves excellent performance for complicated noise models where other unsupervised methods is invalid.
Here we briefly review the existing deep learning methods for image denoising. When the pairs of clean images and noisy images are available, the supervised training [26] is to minimize  $\mathbb{E}_{\boldsymbol{x},\boldsymbol{y}}[\mathrm{d}(\boldsymbol{x},f(\boldsymbol{y};\theta))]$ , where  $f(\cdot ;\theta)$  is a neural network and  $\mathrm{d}(\cdot ,\cdot)$  is a distance metric. Though supervised learning has great performance, the difficult acquisition for training data hampers its application in practice.
To avoid the issues of acquisition for paired clean and noisy images, unsupervised<sup>1</sup> approaches are proposed to use noisy image pairs to learn image denoising, which is started from Noise2Noise (N2N) [12]. While N2N can achieve comparable results with supervised methods, collecting noisy image pairs from real world is still intracTable Motivated by N2N, the followed-up works try to learn image denoising with individual noisy images. Mask-based unsupervised approaches [2] design mask schemes for denoising on individual noisy images and then they train the network to predict the masked pixels according to noisy pixels in the input receptive field. Noise2Void [10] also proposes the blind-spot network (BSN) to avoid learning the identity function. Noisier2Noise [16], Noisy-As-Clean [25] and Recorrupted-to-Recorrupted [17] require a single noisy realization of each training sample and a statistical model of the noise distribution. Noisier2Noise first generates a synthetic noisy sample from the statistical noise model, adds it to the already noisy image, and asks the network to predict the original noisy image from the doubly noisy image. Besides the blind-spot network design, Self2Self [19] is proposed on blind denoising to generate paired data from a single noisy image by applying Bernoulli dropout. Recently, Neighbor2Neighbor [7] proposes to create subsampled paired images based on the pixel-wise independent noise assumption and then a denoising network is trained on generated pairs, with additional regularization loss for better performance. Noise2Score [9] is another type of unsupervised methods that can deal with any noise model which follows an exponential family distribution. It utilize Tweedie's Formula [21, 6] and the estimation of score function to denoise noisy images.
1Sometimes self-supervised is used.
In this section, we provide a more detailed description of the proposed approach. The organization is as follows: In Section 3.1 we review Noise2Score, introduce the basic theory and present the relationship between our method and Noise2Score. After that, we derive the specific algorithms for different examples of noise models in Section 3.2. Finally, we describe the method for estimating the score function  $\nabla_{\pmb{y}}\log p(\pmb{y})$  in Section 3.3. All the proofs are in Supplementary Material.
3.1 Basic Theory
3.1.1 Noise2Score
Suppose the noise model follows some exponential family distribution and has the following form:
$$
p (\boldsymbol {y} \mid \boldsymbol {x}) = b (\boldsymbol {y}) \exp \left\{H (\boldsymbol {x}) ^ {\top} T (\boldsymbol {y}) - a (\boldsymbol {x}) \right\}, \tag {1}
$$
where  $T(\pmb{y})$  and  $(\pmb{x})$  have the same dimensions, and  $a(\pmb{x})$  and  $b(\pmb{y})$  are both scalar functions. One of the properties of an exponential family distribution is that:
$$
H ^ {\prime} (\boldsymbol {x}) ^ {\top} \mathbb {E} [ T (\boldsymbol {y}) \mid \boldsymbol {x} ] = a ^ {\prime} (\boldsymbol {x}). \tag {2}
$$
According to Bayesian Formula, we have the following derivation:
$$
p (\boldsymbol {x} \mid \boldsymbol {y}) = \frac {p (\boldsymbol {x}) p (\boldsymbol {y} \mid \boldsymbol {x})}{p (\boldsymbol {y})} = p (\boldsymbol {x}) e ^ {- a (\boldsymbol {x})} \exp \left(T (\boldsymbol {y}) ^ {\top} H (\boldsymbol {x}) + \log \frac {b (\boldsymbol {y})}{p (\boldsymbol {y})}\right). \tag {3}
$$
Therefore,  $p(\boldsymbol{x} \mid \boldsymbol{y})$  is also an exponential family distribution. According to Eq. 2, we can derive a conclusion about  $\nabla_{\boldsymbol{y}} \log p(\boldsymbol{y})$  (the score function of  $\boldsymbol{y}$ ), Tweedie's Formula, that
$$
\nabla_ {\boldsymbol {y}} \log p (\boldsymbol {y}) = \nabla_ {\boldsymbol {y}} \log b (\boldsymbol {y}) + T ^ {\prime} (\boldsymbol {y}) ^ {\top} \mathbb {E} [ H (\boldsymbol {x}) \mid \boldsymbol {y} ]. \tag {4}
$$
In Noise2Score, the denoised result  $\hat{\pmb{x}}$  of  $\pmb{y}$  is the solution of following equation:
$$
\nabla_ {\boldsymbol {y}} \log p (\boldsymbol {y}) = \nabla_ {\boldsymbol {y}} \log b (\boldsymbol {y}) + T ^ {\prime} (\boldsymbol {y}) ^ {\top} H (\hat {\boldsymbol {x}}). \tag {5}
$$
Since Noise2Score is derived from Tweedie's Formula, it is limited to the noise models of exponential family distributions.
3.1.2 Proposed Approach
In this work, the proposed approach generalized Noise2Score to non-exponential family distributions. We begin with the following proposition.
Proposition 3.1 Let  $\mathbf{y} \sim \mathbf{y}$  and  $\mathbf{x} \sim \mathbf{x}$  where  $\mathbf{x}$  and  $\mathbf{y}$  are two random variables, then the equation below holds:
$$
\nabla_ {\boldsymbol {y}} \log p (\boldsymbol {y}) = \int p (\boldsymbol {x} \mid \boldsymbol {y}) \nabla_ {\boldsymbol {y}} \log p (\boldsymbol {y} \mid \boldsymbol {x}) d \boldsymbol {x} \tag {6}
$$
The proof of Proposition 3.1 is in Supplementary Material. In the task of image denoising, suppose  $\mathbf{y} \in \mathbb{R}^d$  is the noisy image and  $\mathbf{x} \in \mathbb{R}^d$  is the clean image where  $d$  represents the number of image pixels.  $p(\mathbf{x} \mid \mathbf{y})$  represents the posterior distribution and  $p(\mathbf{y} \mid \mathbf{x})$  denotes the noise model. When  $p(\mathbf{y} \mid \mathbf{x})$  follows some exponential family distribution, it is easy to verify that the right part of Eq. 6 equals to the right part of Tweedie's Formula (Eq. 4). Therefore, the equation (Eq. 5) to be solved in Noise2Score in essence can be written as:
$$
\nabla_ {\boldsymbol {y}} \log p (\boldsymbol {y}) = \nabla_ {\boldsymbol {y}} \log p (\boldsymbol {y} \mid \hat {\boldsymbol {x}}). \tag {7}
$$
We use  $\hat{x}$  rather than  $x$  to indicate that  $\hat{x}$  is the unknown variable to be solved in the equation. Hence, we propose a new unsupervised denoising method that given noisy image  $\pmb{y}$ , the denoised result can be obtained by solving Eq. 7. When the noise model follows some exponential family distribution, our method is the same as Noise2Score. The specific conclusions of Gaussian, Gamma and Poisson noise can be seen in Supplementary Material. However, different from Noise2Score, our approach is applicable to some complex non-exponential family distributions as long as the score function of  $\pmb{y}$  and noise model  $p(\pmb{y} \mid \pmb{x})$  are known. For convenience, let  $s(\pmb{y}) = \nabla_{\pmb{y}} \log p(\pmb{y})$  and  $f(\pmb{x}, \pmb{y}) = \nabla_{\pmb{y}} \log p(\pmb{y} \mid \pmb{x})$ . Our approach is consistent with two steps as shown in Figure 1. The general algorithm is illustrated in Algorithm 1.
The following theorem explains why the solution of Eq. 7 is a good denoised result of  $\pmb{y}$ .
Algorithm 1 The general denoising process
Input: noisy image  $\mathbf{y}$  and  $f(x, y)$ .
Output:  $\hat{x}$ , the solution of Eq. 7.
1: Estimate  $s(y)$ .
2: Solve  $\boldsymbol{s}(\boldsymbol{y}) = \boldsymbol{f}(\boldsymbol{x},\boldsymbol{y})$ .
Theorem 3.1 Given  $\mathbf{y}$ , suppose the Hessian matrix  $\mathbf{f}(\mathbf{x},\mathbf{y})$  is bounded,  $\mathbf{f}(\mathbf{x},\mathbf{y})$  is invertible with respect to  $\mathbf{x}$ , and its inverse function is Lipschitz continuous. Denote  $\hat{\mathbf{x}}$  as the solution of Eq. 7, then,
$$
\left\| \hat {\boldsymbol {x}} - \mathbb {E} _ {\boldsymbol {x} \mid \boldsymbol {y}} [ \boldsymbol {x} ] \right\| _ {2} \leq C \operatorname {T r} (\operatorname {C o v} [ \boldsymbol {x} \mid \boldsymbol {y} ]) + o (\operatorname {T r} (\operatorname {C o v} [ \boldsymbol {x} \mid \boldsymbol {y} ])), \tag {8}
$$
where  $\operatorname{Cov}[\pmb{x} \mid \pmb{y}]$  is the covariance matrix and  $C$  is a constant.
The proof of Theorem 3.1 is in Supplementary Material. Theorem 3.1 shows the lower noise level, the smaller the error between  $\hat{x}$  and  $\mathbb{E}_{\boldsymbol{x}|\boldsymbol{y}}[\boldsymbol{x}]$ , which guarantees the effectiveness of proposed method.
Next, in Section 3.2 we will derive the specific algorithm for solving  $\pmb{x}$  in Eq. 7 under different noise models, including additive Gaussian noise in Section 3.2.1, multiplicative noise in Section 3.2.2 and mixture noise in Section 3.2.3. Estimating score function  $s(\pmb{y})$  will be introduced in Section 3.3.
3.2 Application Examples
3.2.1 Additive Gaussian Noise
Suppose the noise model is  $\pmb{y} = \pmb{x} + \pmb{\epsilon}$  where  $\pmb{\epsilon}$  follows a multi-variable Gaussian distribution with mean of  $\mathbf{0}$  and covariance matrix of  $\pmb{\Sigma}$  denoted by  $\mathcal{N}(\mathbf{0},\pmb{\Sigma})$ , i.e.
$$
p (\boldsymbol {y} \mid \boldsymbol {x}) = \frac {\exp \left\{- \frac {1}{2} (\boldsymbol {y} - \boldsymbol {x}) ^ {\top} \boldsymbol {\Sigma} ^ {- 1} (\boldsymbol {y} - \boldsymbol {x}) \right\}}{\sqrt {2 \pi} ^ {d} | \boldsymbol {\Sigma} | ^ {\frac {1}{2}}} \tag {9}
$$
Then, we derive that  $\pmb{f}(\pmb{x},\pmb{y}) = \nabla_{\pmb{y}}\log p(\pmb{y}\mid \pmb{x}) = -\pmb{\Sigma}^{-1}(\pmb{y} - \pmb{x})$ . We consider the following four kinds of  $\pmb{\Sigma}$ : (1)  $\pmb{\Sigma} = \sigma^{2}\pmb{I}$ ; (2)  $\pmb{\Sigma} = \sigma^{2}\pmb{A}^{\top}\pmb{A}$ ; (3)  $\pmb{\Sigma} = \pmb{\Sigma}(\pmb{x}) = \mathrm{diag}(a\pmb{x} + b\mathbf{1})^{2}$ ; (4)  $\pmb{\Sigma} = \pmb{\Sigma}(\pmb{x}) = \pmb{A}^{\top}\mathrm{diag}(a\pmb{x} + b\mathbf{1})^{2}\pmb{A}$ . In the second and fourth cases,  $\pmb{A}$  usually represents a convolution transform which describes the correlation between adjacent pixels.
For the first and second cases,  $\pmb{\Sigma}$  is a constant matrix. By solving Eq. 7, we have
$$
\hat {\boldsymbol {x}} = \boldsymbol {\Sigma} \boldsymbol {s} (\boldsymbol {y}) + \boldsymbol {y}. \tag {10}
$$
While in the third and fourth cases,  $\Sigma$  is related to  $x$  and Eq. 10 is a fixed point equation. Therefore, we use an iterative trick to solve it as shown in Algorithm 2.
Algorithm 2 An iterative trick to solve  $\pmb{x} = \pmb{\Sigma}(\pmb{x})\pmb{s}(\pmb{y}) + \pmb{y}$
Input: noisy image  $\mathbf{y}$ ,  $s(\mathbf{y})$ , the parameters of  $\boldsymbol{\Sigma}(\cdot)$  and the number of iterations  $n$ .
Output:  $\hat{x}$ , the solution.
1: Initial value of  $\hat{x}$  is set as  $y$ .
2: for  $i = 1, \dots, n$  do
3:  $\hat{\pmb{x}}\gets \pmb {\Sigma}(\hat{\pmb{x}})\pmb {s}(\pmb {y}) + \pmb{y}.$
4: end for
3.2.2 Multiplicative Noise
Firstly, we discuss three types of multiplicative noise model, Gamma, Poisson and Rayleigh noise. Then, we consider the situation where the convolution transform  $\mathbf{A}$  exists.
Gamma Noise is constructed from Gamma distribution,  $\mathcal{G}(\alpha, \beta)$ :  $p(x; \alpha, \beta) = \frac{\beta^{\alpha}}{\Gamma(\alpha)} x^{\alpha - 1} e^{-\beta x}$ , and is defined as  $\boldsymbol{y} = \boldsymbol{\eta} \odot \boldsymbol{x}$ , where  $\eta_i \sim \mathcal{G}(\alpha, \alpha)$ ,  $\alpha > 1$ , and  $\odot$  means component-wise multiplication, i.e.
$$
p (\boldsymbol {y} \mid \boldsymbol {x}) = \prod_ {i = 1} ^ {d} \frac {\alpha^ {\alpha}}{\Gamma (\alpha)} \left(\frac {y _ {i}}{x _ {i}}\right) ^ {\alpha - 1} \exp \left\{- \frac {\alpha y _ {i}}{x _ {i}} \right\} \cdot \frac {1}{x _ {i}}. \tag {11}
$$
Then, we derive that  $\pmb{f}(\pmb{x},\pmb{y}) = \nabla_{\pmb{y}}\log p(\pmb{y}\mid \pmb{x}) = \frac{\alpha - 1}{\pmb{y}} -\frac{\alpha}{\pmb{x}}$ . Here, the division is component-wise division. In the residual part of this paper, we neglect such annotation if it is not ambiguous. By solving Eq. 7, we have
$$
\hat {\boldsymbol {x}} = \frac {\alpha \boldsymbol {y}}{\alpha - 1 - \boldsymbol {y} \odot \boldsymbol {s} (\boldsymbol {y})}. \tag {12}
$$
Poisson Noise is constructed from Poisson distribution,  $\mathcal{P}(\lambda)$ :  $\operatorname*{Pr}(x = k) = \frac{\lambda^k}{k!} e^{-\lambda}$ ,  $k = 0,1,\dots$ , and is defined as  $\pmb{y} = \frac{1}{\lambda}\pmb{\eta}$ ,  $\eta_{i}\sim \mathcal{P}(\lambda x_{i})$ ,  $\lambda >0$ , i.e.
$$
\Pr (\boldsymbol {y} \mid \boldsymbol {x}) = \prod_ {i = 1} ^ {d} \frac {\left(\lambda x _ {i}\right) ^ {\lambda_ {i}} y _ {i}}{\left(\lambda y _ {j}\right) !} e ^ {- \lambda x _ {i}}. \tag {13}
$$
Then, we derive that  $\pmb{f}(\pmb{x},\pmb{y}) = \nabla_{\pmb{y}}\log \operatorname*{Pr}(\pmb{y}\mid \pmb{x}) = \lambda \log (\lambda \pmb{x}) - \lambda \log \left(\lambda \pmb{y} + \frac{1}{2}\right)$ . By solving Eq. 7, we have
$$
\hat {\boldsymbol {x}} = \left(\boldsymbol {y} + \frac {1}{2 \lambda}\right) \odot \exp \left\{\frac {\boldsymbol {s} (\boldsymbol {y})}{\lambda} \right\}, \tag {14}
$$
Rayleigh Noise is constructed from Rayleigh distribution,  $\mathcal{R}(\sigma)$ :  $p(x;\sigma) = \frac{x}{\sigma^2}\exp \left\{-\frac{x^2}{2\sigma^2}\right\}$ , and is defined as  $\boldsymbol {y} = (\boldsymbol {\eta} + \mathbf{1})\odot \boldsymbol {x},\eta_{i}\sim \mathcal{R}(\sigma),\sigma >0,i.e.$
$$
p (\boldsymbol {y} \mid \boldsymbol {x}) = \prod_ {i = 1} ^ {d} \frac {1}{x _ {i}} \frac {y _ {i} - x _ {i}}{x _ {i} \sigma^ {2}} \exp \left\{- \frac {\left(y _ {i} - x _ {i}\right) ^ {2}}{2 x _ {i} ^ {2} \sigma^ {2}} \right\}. \tag {15}
$$
Then, we derive that  $\pmb{f}(\pmb{x},\pmb{y}) = \nabla_{\pmb{y}}\log p(\pmb{y}\mid \pmb{x}) = \frac{1}{\pmb{y} - \pmb{x}} -\frac{\pmb{y} - \pmb{x}}{\sigma^2\pmb{x}^2}$ . Solving Eq. 7 directly is not easy. Here we provide an iterative algorithm to solve it. It is illustrated in Algorithm 3.
Algorithm 3 An iterative method to solve Eq. 7 in the case of Rayleigh noise
Input: noisy image  $\mathbf{y}$ ,  $s(\mathbf{y})$ , the parameter of Rayleigh noise  $\sigma$  and the number of iterations  $n$ .
1: Initial value of  $\hat{x}$  is set as  $\pmb{y}$ .
2: for  $i = 1,\dots,n$  do
3: Compute  $\pmb {b} = \sigma^2\pmb {s}(\pmb {y})\odot \hat{\pmb{x}}$
4: Compute  $\pmb{t} = \frac{1}{2}\left(-\pmb{b} + \sqrt{\pmb{b} \odot \pmb{b} + 4\sigma^2\mathbf{1}}\right)$ .
5:  $\hat{\pmb{x}}\gets \frac{\pmb{y}}{t + 1}$
6: end for
Now, we consider the situation where the convolution transform  $\mathbf{A}$  exists. Suppose the noise model is represented by  $\mathbf{y} = \mathbf{A}\mathbf{z}$ ,  $\mathbf{z} = N(\mathbf{x})$ , where  $N(\mathbf{x})$  can be any multiplicative noise model discussed above. Then, we have  $\nabla_{\mathbf{y}}\log p_{\mathbf{y}}(\mathbf{y}|\mathbf{x}) = \mathbf{A}^{-1,\top}\nabla_{\mathbf{z}}\log p_{\mathbf{z}}\left(\mathbf{A}^{-1}\mathbf{y}|\mathbf{x}\right)$ . To avoid confusion, we use subscripts to distinguish different distribution. Therefore, we can apply Algorithm 4 to solve Eq. 7, which is shown as follows.
Algorithm 4 The general framework to solve Eq. 7 for correlated multiplicative noise model
Input:  $y, s(y), A$  and  $\tilde{f}(x, z) = \nabla_z \log p_z(z \mid x)$ .
1: Computing  $\tilde{s} = A^{\top}s(y)$
2: Computing  $z = A^{-1}y$ .
3: Solve  $\tilde{s} = f(x, z)$  by the corresponding Algorithm
3.2.3 Mixture Noise
In this paper, the mixture noise model is composed of a multiplicative noise and an additive Gaussian noise. We denote it as  $\pmb{y} = \pmb{z} + \pmb{\epsilon}, \epsilon \sim \mathcal{N}(0, \sigma^2\pmb{I}), z = \pmb{A}\pmb{N}(\pmb{x})$ , where  $N(\pmb{x})$  is any multiplicative
noise model that can be solved by our approach and  $\mathbf{A}$  is either a convolution transform or identity matrix. It is easy to derive that  $p_{\mathbf{y}}(\boldsymbol{y} \mid \boldsymbol{x}) = \int p_{\mathbf{y}}(\boldsymbol{y} \mid \boldsymbol{z}) p_{\mathbf{z}}(\boldsymbol{z} \mid \boldsymbol{x}) \mathrm{d}\boldsymbol{z}$ . Generally speaking,  $p_{\mathbf{y}}(\boldsymbol{y} \mid \boldsymbol{x})$  has not an explicit analytical form. In this paper, we assume that the additive Gaussian noise is far smaller than the multiplicative noise. Thus, we utilize Taylor expansion to approximate  $p_{\mathbf{y}}(\boldsymbol{y} \mid \boldsymbol{x})$ . We have the following conclusion:
$$
p _ {\mathbf {y}} (\boldsymbol {y} \mid \boldsymbol {x}) \approx p _ {\mathbf {z}} (\bar {\boldsymbol {z}} \mid \mathbf {x}) + \nabla_ {\boldsymbol {z}} p _ {\mathbf {z}} (\bar {\boldsymbol {z}} \mid \boldsymbol {x}) ^ {T} (\boldsymbol {y} - \bar {\boldsymbol {z}}), \tag {16}
$$
where  $\bar{\pmb{z}} = \mathbb{E}\left[\pmb {z}\mid \pmb {y}\right]$ . Then, we can further derive that
$$
\nabla_ {\boldsymbol {y}} \log p _ {\boldsymbol {y}} (\boldsymbol {y} \mid \boldsymbol {x}) \approx \nabla_ {\boldsymbol {z}} \log p _ {\boldsymbol {z}} (\bar {\boldsymbol {z}} \mid \boldsymbol {x}). \tag {17}
$$
The full and rigorous derivations of Eq. 16 and Eq. 17 are in Supplementary Material. Applying Eq. 10 in Section 3.2.1, we have  $\bar{z} = y + \sigma^2 s(y)$ . Thus, the equation to be solve is  $s(y) = f(x,\bar{z})$ . The full denoising process is illustrated in Algorithm 5.
Algorithm 5 The full denoising process for mixture noise  $y = z + \epsilon$
Input:  $y, s(y)$ .
Output:  $\hat{x}$ , the solution of  $s(y) = f(x, \bar{z})$ .
1: Computing  $\bar{z} = \pmb{y} + \sigma^2\pmb{s}(\pmb{y})$
2: Solve  $s(\pmb{y}) = \pmb{f}(\pmb{x},\bar{\pmb{z}})$  through the corresponding algorithm discussed in Section 3.2.2.
3.3 Estimation of Score Function
So far, we assume that the score function of  $\pmb{y}$ ,  $s(\pmb{y})$ , is known. However, it is usually unknown and should be estimated from the dataset of noisy images  $\{\pmb{y}\}$ . We use the same method, the amortized residual Denoising Auto Encoder (AR-DAE) [13] discussed in Noise2Score. Suppose  $s(\cdot; \theta)$  is a neural network used to represent the score function of  $\pmb{y}$ . The following objective function is used to train the model:
$$
L = \mathbb {E} _ {\boldsymbol {y}, \boldsymbol {u} \sim \mathcal {N} (\boldsymbol {0}, \boldsymbol {I})} \| \boldsymbol {u} + \sigma_ {a} \boldsymbol {s} (\boldsymbol {y} + \sigma_ {a} \boldsymbol {u}; \theta) \| _ {2} ^ {2}, \tag {18}
$$
where  $\sigma_{a}$  is a fixed value. Given  $\sigma_{a}$ , the optimal model  $s(\pmb{y};\theta^{*})$  that minimizes  $L$  is the score function of perturbed  $\pmb{y}, \pmb{y} + \sigma_{a}\pmb{u}$ . In other words, we can approximate the score function  $\pmb{s}(\pmb{y})$  by using a sufficiently small  $\sigma_{a}$ . Related analysis can also be seen in [8, 23, 1]. During the training process, the value of  $\sigma_{a}$  will be decreasing gradually to a very small value. This progressive process is helpful to numerically stabilize the model training. The only model training is to estimate  $\pmb{s}(\pmb{y})$  by  $\pmb{s}(\cdot;\hat{\theta})$ , which is served as the first step of our approach. After the score function model is trained, we apply the denoising algorithms given in Section 3.1 to obtain denoised results and no more training is required.