Noise2Score: Tweedie’s Approach to Self-Supervised Image Denoising without Clean Images

1 Introduction . Bayesian inference , which derives the posterior probability using a prior probability and a likelihood function for the observed data , has been an important tool in statistics . This approach has been used extensively for image denoising from early ages to the modern era of deep learning . For example , in the recent unsupervised deep learning approach for image denoising using Stein ’ s risk estimate ( SURE ) [ 1 ] , the unknown Bayesian risk is replaced by the SURE that can be calculated from the noisy measurement so that deep neural network training is performed by minimizing it . Unfortunately , this method is sensitive to hyper-parameters , and the neural network must be retrained if the underlying noise model varies [ 2 ] . On the other hand , there has been increased research interest in image denoisers that can be trained by minimizing variants of empirical risks that are not associated with clean data . Noise2Noise [ 3 ] was the first representative approach that does not require clean data . Unfortunately , multiple noisy versions of the same images are necessary for training . To address this , self-supervised learning approaches such as Noise2Void [ 4 ] , Noise2Self [ 5 ] , etc . have been developed in order to use only a single noisy image . This class of approaches , which we will call Noise2X throughout the paper , are especially important for practical applications , where noiseless clean images or multiple noisy realization of the same image are difficult or impossible to collect . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . Though SURE and Noise2X hold promise for practical applications , one of the fundamental questions is how these seemingly different approaches are related and whether there is a coherent mathematical theory that can be leveraged to further improve the performance . Although we are aware of a preliminary prior work [ 6 ] attempting to unite a subset of them , there is largely a lack of a principled method of designing image denoiser without clean data . In this article , we try to approach this open problem from Bayesian statistics in a completely different route . Instead of minimizing different forms of empirical risks or SURE , one of the most important contributions of this paper is the discovery of the importance of classical result from Bayesian statistics - the Tweedie ’ s formula [ 7 ] which provide an explicit way of computing the posterior mean of canonical parameters from the noisy measurements corrupted with exponential family noises . Specifically , we show that the Tweedie ’ s formula provides a unified approach for image denoising from any exponential family noises through a score function ( i.e . the gradient of the loglikelihood ) . Therefore , the self-supervised image denoising problem without clean image can be reduced to the problem of estimating the score function . This change of perspective has many important theoretical implications as well as the flexibility with regard to the implementation of the algorithm . In fact , the score function estimation problem has been an important research topic in Bayesian statistics and machine learning [ 8–10 ] . In particular , Alain and Bengio [ 10 ] showed that the minimization of the denoising autoencoder ( DAE ) objective function provides an explicit way of approximating the score function . This result was further extended using the amortized residual denoising autoencoder ( AR-DAE ) for numerical stability and accuracy [ 11 ] . Interestingly , the training method for DAE or its variants is strikingly similar to that of Noise2X . Therefore , by combining this with the Tweedie ’ s formula , we can obtain a novel unified framework of Bayesian approach for self-supervised image denoising . The conceptual difference and similarity of our method to the existing approaches are illustrated in Fig . 1 ( a ) ( b ) . As the first step of our algorithm is similar to Noise2X , we call our method Noise2Score . Noise2Score is so powerful that it can be used to deal with any exponential family noises . Moreover , in contrast to the SURE approach , in which the network must be retrained using a different loss function depending on the noise model and parameters [ 1 , 12 ] , Noise2Score is universal in the sense that the identical neural network training step is used regardless of noise models . This property can be exploited to extend Noise2Score to blind setups in which noise parameters are unknown and should be estimated with minimal complexity . In addition to the novel theoretical findings mentioned above , our empirical results using additive Gaussian , Poisson and Gamma noise models have shown that Noise2Score significantly outperforms SURE and Noise2X under similar experimental conditions . 2 Related Works . Here we give a brief overview of the existing deep learning approaches to denoise images . As shown below , their main differences arise from the different choices of the distance metric d ( · , · ) in Fig . 1 . 2.1 Image denoising using supervised learning . Let X and Y denotes the spaces of clean and noisy images with the probability measure PX and PY , respectively . When the paired ground-truth data x is available for a noisy measurement y , i.e . x : = x ( y ) , then the supervised training can be carried out by minimizing the following loss function : ` sup ( Θ ) = Ey∼PY d ( x , FΘ ( y ) ) , where d ( x , FΘ ( y ) ) : = ‖x− FΘ ( y ) ‖2 ( 1 ) where FΘ ( y ) is a deep neural network parameterized by Θ with input y and here we use l2 distance metric to simplify the explanation . Although this approach has significantly improved performance compared to the classical denoising algorithms [ 13 ] , the supervised learning approach has a drawback as the large number of paired clean images are necessary . In real world applications , the acquisition for paired clean images are difficult or even impossible . 2.2 Noise2X for image denoising without clean images . To address this issue , various forms of the denoisers that can be trained without clean images have been proposed , which includes Noise2Noise [ 3 ] , Noise2Void [ 4 ] , Noise2Self [ 14 ] , Noise2Same [ 5 ] , etc . These approaches are usually based on the variants of loss functions that do not depend on clean images : ` Noise2X ( Θ ) = Ey∼PY d ( y′ , FΘ ( y ) ) , where d ( y′ , FΘ ( y ) ) : = ‖y′ − FΘ ( y ) ‖2 ( 2 ) where the target image y′ is related to y in unique ways depending on algorithms . For example , in Noise2Noise [ 3 ] , y′ is another noisy realization of the same underlying clean images . However , Noise2Noise is not practical since multiple noisy realizations of the same image are not usually available in practice . Noise2Void [ 14 ] generates y′ by adding blind spots on y . In Noise2Self [ 14 ] , so-called J-invariant mask J is added on the input y , and y′ is defined as the image on the complementary mask . All of these changes of the target images are intended to prevent the network from converging to a trivial identity mapping . 2.3 Image denoising using Stein ’ s Unbiased Risk Estimate ( SURE ) . Rather than using the variants of a loss function as in Noise2X , Soltanayev et al [ 1 ] proposed a training scheme using the loss function from Stein ’ s unbiased risk estimate ( SURE ) [ 2 ] which has additional regularization term . Specifically , the loss function for the SURE denoiser is defined by ` SURE ( Θ ) = Ey∼PY d ( y , FΘ ( y ) ) , d ( y , FΘ ( y ) ) : = ‖y − FΘ ( y ) ‖2 + 2σ2divyFΘ ( y ) ( 3 ) Here , the second term in d ( y , FΘ ( y ) ) of ( 3 ) is the regularization term to prevent the network from converging to a trivial identity mapping , where divy denotes the divergence with respect to y . Due to the difficulty of calculating the divergence term , the authors in [ 1 ] relied on MonteCarlo SURE [ 15 ] which calculates the divergence term using MonteCarlo simulation . This introduces additional hyperparameters , on which the final results critically depend . Yet another limitation of SURE image denoiser is that the loss function is determined by the noise model . For example , in [ 12 ] , the authors derived Poisson Unbiased risk estimator ( PURE ) to train deep CNN for dealing with Poisson noise , whose loss function is completely different from ( 3 ) . Even under the same Gaussian noise models , if the noise variance σ2 varies , the neural network FΘ must be trained again , which causes additional computational burden . This differs from Noise2X , which uses the same loss function regardless of the noise models . 3 Main Contribution : Noise2Score . In contrast to the existing approaches shown in Fig . 1 ( a ) , our denoising approach is composed of two steps as illustrated in Fig . 1 ( b ) , which is inspired by the results from Bayesian statistics . In the following , we explain the details of each step . 3.1 Tweedie ’ s formula of the posterior mean for image denoising . Suppose that the noisy measurement y is given by y = x+ w , where w ∼ N ( 0 , σ2I ) ( 4 ) where x is the clean image , and N ( 0 , σ2I ) denotes the independent and identically distributed ( i.i.d ) Gaussian distribution with zero mean and variance σ2 . Then , Tweedie ’ s formula calculates the posterior expectation of x given y as [ 16 ] E [ x|y ] = y + σ2l′ ( y ) , where l′ ( y ) = ∇y log p ( y ) ( 5 ) where p ( y ) denotes the marginal distribution of y . For the image corrupted by Gaussian noises , the posterior mean is the minimum mean square error ( MMSE ) estimator , so that the denoised image can be obtained using ( 5 ) , as long as we know l′ ( y ) - the score function [ 17 ] . Tweedie ’ s formula was extended for general class of exponential family distribution [ 7 ] . Specifically , exponential family of probability distributions are defined as those distributions whose density have the following general form : p ( y|η ) = p0 ( y ) exp ( η > T ( y ) − ϕ ( η ) ) ( 6 ) where the superscript > denotes the transpose operation . Here η is a canonical ( vector ) parameter of the family , T ( y ) is a ( vector ) function of y , ϕ ( η ) the cumulant generating function which makes p ( y|η ) integrate to 1 , and p0 ( y ) the density up to a scale factor when η = 0 . Exponential family distribution covers a large class of important distributions such as the Gaussian , binomial , multinomial , Poisson , gamma , and beta distributions , as well as many others . Table 1 summarizes the corresponding parameters for each exponential family distribution . Then , using the Bayes ’ rule , the posterior density of η given y is given by [ 7 ] : p ( η|y ) = exp ( η > T ( y ) − λ ( y ) ) [ p ( η ) e−ϕ ( η ) ] , where λ ( y ) = log ( p ( y ) p0 ( y ) ) ( 7 ) where p ( y ) and p ( η ) denote the marginal distribution of y and η , respectively . This implies that the posterior density is again an exponential family distribution , and the mode of the posterior distribution can be obtained by finding the maximum of p ( η|y ) . Specifically , by computing the gradient of log p ( η|y ) with respect to y and setting it to zero , the posterior estimate of the canonical parameter η̂ should satisfy the following equality : η̂ > T ′ ( y ) = λ′ ( y ) = −∇y log p0 ( y ) +∇y log p ( y ) = −l′0 ( y ) + l′ ( y ) ( 8 ) where l′ ( y ) : = ∇y log p ( y ) and l′0 ( y ) : = ∇y log p0 ( y ) are score functions , and T ′ ( y ) = ∇yT ( y ) . In particular , if T ( y ) = y , we can obtain the following closed form solution for the posterior mean : η̂ : = E [ η|y ] = λ′ ( y ) = −l′0 ( y ) + l′ ( y ) ( 9 ) In the following , we derive denoising algorithms using Tweedie ’ s formulation for several representative distributions . Additive Gaussian noise For Gaussian distribution , in the first row of Table 1 , we have p0 ( y ) = 1√ 2πσ e− y2 2σ2 , η = x σ2 ( 10 ) Accordingly , E [ η|y ] = x̂ σ2 = y σ2 + l′ ( y ) , where x̂ : = E [ x|y ] ( 11 ) since l′0 ( y ) = y/σ 2 . Therefore , the posterior mean is given by x̂ = y + σ2l′ ( y ) ( 12 ) which is equal to ( 5 ) . Even with the different parameterization of Gaussian as shown in the second row of Table 1 , we can arrive at the same result using ( 8 ) . Poisson noise For the case of Poisson noises , we have p0 ( y ) = 1 y ! , l′0 ( y ) = − ∇yΓ ( y + 1 ) Γ ( y + 1 ) ' − log ( y + 1 2 ) ( 13 ) where Γ ( y ) denotes the gamma function and the last approximation comes from [ 7 ] . Accordingly , we have η̂ = log ( x̂ ) = log ( y + 1 2 ) + l′ ( y ) ( 14 ) This leads to the following posterior estimate of the image x̂ = ( y + 1 2 ) exp ( l′ ( y ) ) ( 15 ) This corresponds to Tweedie ’ s formula for Poisson case [ 16 ] . This result can be generalized to the low photon count sensing scenario , where the sensor measurement can be described by [ 18 ] : y = ζz , z ∼ Poisson ( x/ζ ) ( 16 ) where z is a random variable that follows the Poisson distribution in Table 1 and ζ > 0 is the gain of the acquisition process that is related to the noise level . Then , we have p0 ( z ) = 1 z ! , η = log ( x ζ ) ( 17 ) Therefore , we have x̂ ζ = ( z + 1 2 ) exp ( l′ ( z ) ) −→ x̂ = ( y + ζ 2 ) el ′ ( yζ ) ( 18 ) Gamma noise Gamma noise distribution can be used to model the speckle noises in various imaging application . Specifically , the image corrupted with the speckle noise is represented by y = xn , n ∼ p ( n ; α , β ) = β α Γ ( α ) nα−1 exp ( −βn ) ( 19 ) where p ( n ; α , β ) denotes the Gamma distribution with ( α , β ) parameters . Here , ( α , β ) are the parameters which determine the noise level of Gamma noise distribution . For example , for the k-look measurement case , α = β = k [ 19 ] . Then , the probability p ( y|x ) can be obtained by replacing n in p ( n ; α , β ) with n = y/x , and the resulting density function can be bound in Table 1 . Then , we can easily show that po ( y ) = 1 , η = [ α− 1 −β/x ] , T ( y ) = [ log y y ] ( 20 ) Using ( 8 ) , we have α− 1 y − β x = l′ ( y ) =⇒ x̂ = βy ( α− 1 ) − yl′ ( y ) ( 21 ) Similar derivations for other exponential family distribution can be found in Table 1 .

Summary: The paper proposes a new approach based on Tweedie's formula to perform self-supervised image denoising. The proposed approach has two steps: (1) learn the score function or gradient of density of the noisy images, and (2) use Tweedie's formula to compute a denoised image using the learned score function. The proposed approach builds on multiple relevant work in the literature, and is well motivated. The model is tested on Gaussian, Poisson and Gamma noise removal tasks on natural images and achieves competitive results.