An Unsupervised Deep Learning Approach for Real-World Image Denoising

1 INTRODUCTION . Noise always exists during the process of image acquisition and its removing is important for image recovery and vision tasks , e.g. , segmentation and recognition . Specifically , the noisy image y is modeled as y = x+ n , where x denotes the clean image , n denotes the corrupted noise and image denoising aims at recovering x from y . Over the past two decades , this problem has been extensively explored and many works have been proposed . Among these works , one typical kind of model assumes that the image is corrupted by additive white Gaussian noise ( AWGN ) , i.e. , n ∼ N ( 0 , σ2I ) where N ( 0 , 1 ) is the standard Gaussian distribution . Representative Gaussian denoising approaches include block matching and 3D filtering ( BM3D ) ( Dabov et al. , 2007b ) , non-local mean method ( NLM ) ( Buades et al. , 2005 ) , KSVD ( Aharon et al. , 2006 ) and weighted nuclear norm minimization ( WNNM ) ( Gu et al. , 2014 ) , which perform well on AWGN noise removal . However , the AWGN assumption seldom holds in practical applications as the noise is accumulated during the whole imaging process . For example , in typical CCD or CMOS cameras , the noise depends on the underlying context ( daytime or nighttime , static or dynamic , indoor or outdoor , etc . ) and the camera settings ( shutter speed , ISO , white balance , etc. ) . In Figure 1 , two real noisy images captured by Samsung Galaxy S6 Edge and Google Pixel smartphones are chosen from Smartphone Image Denoising Dataset ( SIDD ) ( Abdelhamed et al. , 2018 ) and three 40×40 patches are chosen for illustration of noisy distribution . It is clear that real noise distribution is content dependent and noise in each patch has different statistical properties which can be non-Gaussian . Due to the violation of the AWGN assumption , the performance of ∗Corresponding author . the Gaussian denoiser deteriorates significantly ( Figure 1 ( d ) ) . Thus , it is crucial to characterize the noise distribution and adapt the noise models to the denoiser in real-world image denoising . In recent years , deep learning based methods have achieved remarkable performance with careful architecture design , good training strategies , a large number of noisy and clean image pairs . However , there are two main drawbacks of these approaches from the perspective of practical applications . One is the high dependency on the quality and the size of the training dataset . Collecting such image pairs is time-consuming and requires much of human efforts , especially when the labeling needs deep domain knowledge such as medical or seismic images . The very recent deep learning methods including Noise2Noise ( N2N ) ( Lehtinen et al. , 2018 ) , Noise2Void ( N2V ) ( Krull et al. , 2019a ) and Noise2Self ( N2S ) ( Batson & Royer , 2019 ) have relaxed the dataset requirement and can be trained on organized/un-organized noisy and noisy image pairs . Nevertheless , to guarantee the performance , these networks need to be pre-trained with a large number of images to cover sufficiently many local patterns , and thus they are not cost-effective . Therefore , to reduce the dependency of the training set , single-image based image denoising approaches deserved to be studied and have both practical and scientific value . It is worth mentioning that a recent unsupervised learning work ( Ulyanov et al. , 2018 ) uses a deep image prior to the general image recovery problem but its denoising results are inferior to some typical Gaussian denoisers , e.g. , BM3D . The other drawback is the generalization ability of a trained network . When the noisy distribution is complicated and not contained in the training set , the results of the deep learning method can be deteriorated significantly , even worse than non-learning based methods . To alleviate this problem , some recent works are proposed by further consideration of noise estimation in the network design , e.g. , Guo et al . ( 2019 ) ; Yue et al . ( 2019 ) ; Zhang et al . ( 2017 ) . Despite their good performance in blind Gaussian denoising ( Guo et al. , 2019 ; Zhang et al. , 2017 ) and real-world denoising problem ( Yue et al. , 2019 ) , a large number of noisy and clean image pairs are needed and the generalization problem remains when the imaging system is complicated . Very recently , a single image based method has been proposed in ( Quan et al. , 2020 ) by developing a novel dropout technique for image denoising . Thus , unsupervised deep learning approaches with accurate noise models are important for solving real-world image denoising problems , yet current solutions are unsatisfactory . Such approach deserves to be studied and is a challenging problem as it needs a good combination of traditional methods and deep learning based methods such that the benefits of both methods are fully explored . 1.1 THE SUMMARY OF IDEAS AND CONTRIBUTIONS . Motivated by the above analysis , the goal of this paper is to propose an unsupervised deep learning method that boosts the performance of existing Gaussian denoisers when solving real-world image denoising problems . The basic idea is to find a latent image z associated with the input noisy image y such that z|x satisfies the AWGN assumption , and thus we can obtain the clean image x from z by any existing Gaussian denoiser . To find the appropriate latent representation , we propose a neural network ( NN ) based approach that builds up the mapping between the noisy image y and latent image z with an encoder-decoder structure . By applying the Gaussian denoiser in the latent space , we alternatively update the NN and the denoised image which does not need the other training samples . Figure 2 illustrates the workflow of the proposed approach . Also , this idea can be formulated under the classical maximum a posterior ( MAP ) framework which consists of a likelihood term and a prior term . Building a proper likelihood term requires an accurate estimation of the noise distribution . Although the accurate noise distribution is difficult to get , an evidence lower bound ( ELBO ) can be analytically derived for approximating the likelihood from below using variational auto-encoder ( VAE ) ( Kingma & Welling , 2013 ) . This ELBO term gives the loss function for the encoder and decoder networks that are maps between noisy image and latent image . From the above derivation , we arrive at a model min x , F , G f ( x , y , F , G ) +R ( x ) , ( 1 ) where x is the clean image , y is the input noisy image , F , G are decoder and encoder maps parameterized by NNs , f is the loss from ELBO and R ( x ) is the regularization term . Model ( 1 ) can be minimized by the alternative direction method of multiplier ( ADMM ) which alternatively updates networks and the clean image estimation x . Using Plug-and-Play technique ( Venkatakrishnan et al. , 2013 ) , updating x can be replaced by any Gaussian denoiser . Thus , by fully exploiting the benefits of deep neural networks and classic denoising schemes , the real-world image denoising can be improved by a large margin as shown in Figure 1 ( e ) . More importantly , training the proposed networks only uses the noisy image itself and does not need any pre-training . In summary , we list our main contributions as follows . • We propose an effective approach that combines the deep learning method with traditional methods for unsupervised image denoising . Thanks to the great expressive power of deep neural networks , the complex noise distribution is mapped into a latent space in which the AWGN assumption tends to hold , and thus better results are obtained by applying existing Gaussian denoiser for latent images . • Instead of a heuristic loss design , the proposed NN approximates the likelihood in the classic Bayesian framework , which gives clear interpretations of each loss term . Meanwhile , compared to many existing deep learning methods , our model is only trained on a single image , which significantly reduces the burden of data collection . • Extensive numerical experiments on real-world noisy image datasets have shown that the NN boosts the performance of the existing denoisers including NLM , BM3D and DnCNN . In particular , the results of NN+BM3D are competitive with some supervised deep learning approaches such as DnCNN+ , FFDNet+ , CBDNet . 2 RELATED WORK . There are numerous works in image denoising . Here , we review the works related to the nonlearning and learning approaches for real-world image denoising . 2.1 NON-LEARNING BASED APPROACHES . Non-learning approaches are mainly based on the MAP framework that contains a data fidelity and a regularization term . Many works have been proposed to improve the regularize term , e.g. , sparsity based methods ( Rudin et al. , 1992 ; Perona & Malik , 1990 ) , low rank prior ( Dong et al. , 2012b ; Gu et al. , 2014 ) and non-local methods ( Buades et al. , 2005 ; Dabov et al. , 2007a ) . Among these methods , BM3D ( Dabov et al. , 2007a ) is one of the top methods . There are several other works related to the construction of data fidelity by modeling the complex noise distribution , e.g. , Lebrun et al . ( 2015a ) ; Nam et al . ( 2016 ) ; Xu et al . ( 2017 ) ; Zhu et al . ( 2016 ) . The correlated Gaussian distribution ( Lebrun et al. , 2015a ) and Mixture of Gaussian ( Zhu et al. , 2016 ; Nam et al. , 2016 ) are used to approximate the unknown noise distribution . In Xu et al . ( 2017 ) , different noise statistics are estimated in different channels without the consideration of the content dependent noise . Recently , Amini et al . ( 2020 ) proposed a Gaussianization method for gray scale OCT images . However , this method is not applicable for real-world image denoising tasks as natural images are colorful and the noise distribution is much more complicated than that in OCT images . Overall , due to the complexity of real-world noise , the performance of these approaches is unsatisfactory and needs to be improved . 2.2 LEARNING BASED APPROACHES . The learning based approaches can be classified into two groups : single-image based methods and dataset based methods . Typical single-image based approaches are sparse coding methods Aharon et al . ( 2006 ) ; Bao et al . ( 2015 ) ; Xu et al . ( 2018b ) . In Xu et al . ( 2018b ) , the noise in each channel is estimated and followed by a weighted sparse coding scheme . In recent years , as the appearance of real image denoising datasets including CC ( Nam et al. , 2016 ) , PolyU ( Xu et al. , 2018a ) , DND ( Plotz & Roth , 2017 ) and SIDD ( Abdelhamed et al. , 2018 ) , deep neural networks including Guo et al . ( 2019 ) ; Yu et al . ( 2019 ) ; Zhang et al . ( 2017 ) ; Zhou et al . ( 2019 ) ; Yue et al . ( 2019 ) have shown promising results on these datasets . However , these networks require many noisy/clean training pairs which limit their practical applications especially when the labeling work needs domain experts . Recently , the deep learning approaches ( Krull et al. , 2019a ; b ; Batson & Royer , 2019 ; Laine et al. , 2019 ; Lehtinen et al. , 2018 ) are proposed and trained with organized or unorganized noisy image pairs . To guarantee a satisfactory performance , these methods still need many training pairs such that sufficiently many local patterns are covered . Compared to the above deep learning approaches , our method is a single image based method which does not need training samples or pre-training from other datasets .

Summary: The paper proposes an unsupervised approach for denoising for which works on a single noisy image. The central idea is to use a neural network to map the noisy image to a latent image space where the noisy distribution follows Gaussian (like Variance Stabliziation Transform) and then use a off-the-shelf Gaussian denoiser on this image. The paper shows good numerical results.