An Unsupervised Deep Learning Approach for Real-World Image Denoising

1 INTRODUCTION . Noise always exists during the process of image acquisition and its removing is important for image recovery and vision tasks , e.g. , segmentation and recognition . Specifically , the noisy image y is modeled as y = x+ n , where x denotes the clean image , n denotes the corrupted noise and image denoising aims at recovering x from y . Over the past two decades , this problem has been extensively explored and many works have been proposed . Among these works , one typical kind of model assumes that the image is corrupted by additive white Gaussian noise ( AWGN ) , i.e. , n ∼ N ( 0 , σ2I ) where N ( 0 , 1 ) is the standard Gaussian distribution . Representative Gaussian denoising approaches include block matching and 3D filtering ( BM3D ) ( Dabov et al. , 2007b ) , non-local mean method ( NLM ) ( Buades et al. , 2005 ) , KSVD ( Aharon et al. , 2006 ) and weighted nuclear norm minimization ( WNNM ) ( Gu et al. , 2014 ) , which perform well on AWGN noise removal . However , the AWGN assumption seldom holds in practical applications as the noise is accumulated during the whole imaging process . For example , in typical CCD or CMOS cameras , the noise depends on the underlying context ( daytime or nighttime , static or dynamic , indoor or outdoor , etc . ) and the camera settings ( shutter speed , ISO , white balance , etc. ) . In Figure 1 , two real noisy images captured by Samsung Galaxy S6 Edge and Google Pixel smartphones are chosen from Smartphone Image Denoising Dataset ( SIDD ) ( Abdelhamed et al. , 2018 ) and three 40×40 patches are chosen for illustration of noisy distribution . It is clear that real noise distribution is content dependent and noise in each patch has different statistical properties which can be non-Gaussian . Due to the violation of the AWGN assumption , the performance of ∗Corresponding author . the Gaussian denoiser deteriorates significantly ( Figure 1 ( d ) ) . Thus , it is crucial to characterize the noise distribution and adapt the noise models to the denoiser in real-world image denoising . In recent years , deep learning based methods have achieved remarkable performance with careful architecture design , good training strategies , a large number of noisy and clean image pairs . However , there are two main drawbacks of these approaches from the perspective of practical applications . One is the high dependency on the quality and the size of the training dataset . Collecting such image pairs is time-consuming and requires much of human efforts , especially when the labeling needs deep domain knowledge such as medical or seismic images . The very recent deep learning methods including Noise2Noise ( N2N ) ( Lehtinen et al. , 2018 ) , Noise2Void ( N2V ) ( Krull et al. , 2019a ) and Noise2Self ( N2S ) ( Batson & Royer , 2019 ) have relaxed the dataset requirement and can be trained on organized/un-organized noisy and noisy image pairs . Nevertheless , to guarantee the performance , these networks need to be pre-trained with a large number of images to cover sufficiently many local patterns , and thus they are not cost-effective . Therefore , to reduce the dependency of the training set , single-image based image denoising approaches deserved to be studied and have both practical and scientific value . It is worth mentioning that a recent unsupervised learning work ( Ulyanov et al. , 2018 ) uses a deep image prior to the general image recovery problem but its denoising results are inferior to some typical Gaussian denoisers , e.g. , BM3D . The other drawback is the generalization ability of a trained network . When the noisy distribution is complicated and not contained in the training set , the results of the deep learning method can be deteriorated significantly , even worse than non-learning based methods . To alleviate this problem , some recent works are proposed by further consideration of noise estimation in the network design , e.g. , Guo et al . ( 2019 ) ; Yue et al . ( 2019 ) ; Zhang et al . ( 2017 ) . Despite their good performance in blind Gaussian denoising ( Guo et al. , 2019 ; Zhang et al. , 2017 ) and real-world denoising problem ( Yue et al. , 2019 ) , a large number of noisy and clean image pairs are needed and the generalization problem remains when the imaging system is complicated . Very recently , a single image based method has been proposed in ( Quan et al. , 2020 ) by developing a novel dropout technique for image denoising . Thus , unsupervised deep learning approaches with accurate noise models are important for solving real-world image denoising problems , yet current solutions are unsatisfactory . Such approach deserves to be studied and is a challenging problem as it needs a good combination of traditional methods and deep learning based methods such that the benefits of both methods are fully explored . 1.1 THE SUMMARY OF IDEAS AND CONTRIBUTIONS . Motivated by the above analysis , the goal of this paper is to propose an unsupervised deep learning method that boosts the performance of existing Gaussian denoisers when solving real-world image denoising problems . The basic idea is to find a latent image z associated with the input noisy image y such that z|x satisfies the AWGN assumption , and thus we can obtain the clean image x from z by any existing Gaussian denoiser . To find the appropriate latent representation , we propose a neural network ( NN ) based approach that builds up the mapping between the noisy image y and latent image z with an encoder-decoder structure . By applying the Gaussian denoiser in the latent space , we alternatively update the NN and the denoised image which does not need the other training samples . Figure 2 illustrates the workflow of the proposed approach . Also , this idea can be formulated under the classical maximum a posterior ( MAP ) framework which consists of a likelihood term and a prior term . Building a proper likelihood term requires an accurate estimation of the noise distribution . Although the accurate noise distribution is difficult to get , an evidence lower bound ( ELBO ) can be analytically derived for approximating the likelihood from below using variational auto-encoder ( VAE ) ( Kingma & Welling , 2013 ) . This ELBO term gives the loss function for the encoder and decoder networks that are maps between noisy image and latent image . From the above derivation , we arrive at a model min x , F , G f ( x , y , F , G ) +R ( x ) , ( 1 ) where x is the clean image , y is the input noisy image , F , G are decoder and encoder maps parameterized by NNs , f is the loss from ELBO and R ( x ) is the regularization term . Model ( 1 ) can be minimized by the alternative direction method of multiplier ( ADMM ) which alternatively updates networks and the clean image estimation x . Using Plug-and-Play technique ( Venkatakrishnan et al. , 2013 ) , updating x can be replaced by any Gaussian denoiser . Thus , by fully exploiting the benefits of deep neural networks and classic denoising schemes , the real-world image denoising can be improved by a large margin as shown in Figure 1 ( e ) . More importantly , training the proposed networks only uses the noisy image itself and does not need any pre-training . In summary , we list our main contributions as follows . • We propose an effective approach that combines the deep learning method with traditional methods for unsupervised image denoising . Thanks to the great expressive power of deep neural networks , the complex noise distribution is mapped into a latent space in which the AWGN assumption tends to hold , and thus better results are obtained by applying existing Gaussian denoiser for latent images . • Instead of a heuristic loss design , the proposed NN approximates the likelihood in the classic Bayesian framework , which gives clear interpretations of each loss term . Meanwhile , compared to many existing deep learning methods , our model is only trained on a single image , which significantly reduces the burden of data collection . • Extensive numerical experiments on real-world noisy image datasets have shown that the NN boosts the performance of the existing denoisers including NLM , BM3D and DnCNN . In particular , the results of NN+BM3D are competitive with some supervised deep learning approaches such as DnCNN+ , FFDNet+ , CBDNet . 2 RELATED WORK . There are numerous works in image denoising . Here , we review the works related to the nonlearning and learning approaches for real-world image denoising . 2.1 NON-LEARNING BASED APPROACHES . Non-learning approaches are mainly based on the MAP framework that contains a data fidelity and a regularization term . Many works have been proposed to improve the regularize term , e.g. , sparsity based methods ( Rudin et al. , 1992 ; Perona & Malik , 1990 ) , low rank prior ( Dong et al. , 2012b ; Gu et al. , 2014 ) and non-local methods ( Buades et al. , 2005 ; Dabov et al. , 2007a ) . Among these methods , BM3D ( Dabov et al. , 2007a ) is one of the top methods . There are several other works related to the construction of data fidelity by modeling the complex noise distribution , e.g. , Lebrun et al . ( 2015a ) ; Nam et al . ( 2016 ) ; Xu et al . ( 2017 ) ; Zhu et al . ( 2016 ) . The correlated Gaussian distribution ( Lebrun et al. , 2015a ) and Mixture of Gaussian ( Zhu et al. , 2016 ; Nam et al. , 2016 ) are used to approximate the unknown noise distribution . In Xu et al . ( 2017 ) , different noise statistics are estimated in different channels without the consideration of the content dependent noise . Recently , Amini et al . ( 2020 ) proposed a Gaussianization method for gray scale OCT images . However , this method is not applicable for real-world image denoising tasks as natural images are colorful and the noise distribution is much more complicated than that in OCT images . Overall , due to the complexity of real-world noise , the performance of these approaches is unsatisfactory and needs to be improved . 2.2 LEARNING BASED APPROACHES . The learning based approaches can be classified into two groups : single-image based methods and dataset based methods . Typical single-image based approaches are sparse coding methods Aharon et al . ( 2006 ) ; Bao et al . ( 2015 ) ; Xu et al . ( 2018b ) . In Xu et al . ( 2018b ) , the noise in each channel is estimated and followed by a weighted sparse coding scheme . In recent years , as the appearance of real image denoising datasets including CC ( Nam et al. , 2016 ) , PolyU ( Xu et al. , 2018a ) , DND ( Plotz & Roth , 2017 ) and SIDD ( Abdelhamed et al. , 2018 ) , deep neural networks including Guo et al . ( 2019 ) ; Yu et al . ( 2019 ) ; Zhang et al . ( 2017 ) ; Zhou et al . ( 2019 ) ; Yue et al . ( 2019 ) have shown promising results on these datasets . However , these networks require many noisy/clean training pairs which limit their practical applications especially when the labeling work needs domain experts . Recently , the deep learning approaches ( Krull et al. , 2019a ; b ; Batson & Royer , 2019 ; Laine et al. , 2019 ; Lehtinen et al. , 2018 ) are proposed and trained with organized or unorganized noisy image pairs . To guarantee a satisfactory performance , these methods still need many training pairs such that sufficiently many local patterns are covered . Compared to the above deep learning approaches , our method is a single image based method which does not need training samples or pre-training from other datasets .

Summary: The key idea of this paper is to transform a real-world noisy image into a latent image space using an encoder neural network(NN), where the noise is hopefully white Gaussian in the latent image space so that existing Gaussian denoisers can be used in the optimization steps. The proposed methods are reported to have improved performance compared to original Gaussian denoisers combined in their methods. One specific proposed combination, i.e., NN+BM3D shows better results than other unsurprised denoising approaches and is competitive with a supervised network. The idea of this paper is interesting and the presentation of the major idea is relatively clear.