GAN2GAN: Generative Noise Learning for Blind Denoising with Single Noisy Images

1 INTRODUCTION . Image denoising is one of the oldest problems in image processing and low-level computer vision , yet it still attracts lots of attention due to the fundamental nature of the problem . A vast number of algorithms have been proposed over the past several decades , and recently , the CNN-based methods , e.g. , Cha & Moon ( 2019 ) ; Zhang et al . ( 2017 ) ; Tai et al . ( 2017 ) ; Liu et al . ( 2018 ) , became the throne-holders in terms of the PSNR performance . The main approach of the most CNN-based denoisers is to apply the discriminative learning framework with ( clean , noisy ) image pairs and known noise distribution assumption . While being effective , such framework also possesses a couple of limitations that become critical in practice ; the assumed noise distribution may be mismatched to the actual noise in the data or obtaining the noise-free clean target images is not always possible or very expensive , e.g. , medical imaging ( CT or MRI ) or astrophotographs . Several attempts have been made to resolve above issues . For the noise uncertainty , the so-called blind training have been proposed . Namely , a denoiser can be trained with a composite training set that contains images corrupted with multiple , pre-defined noise levels or distributions , and such blindly trained denoisers , e.g. , DnCNN-B in Zhang et al . ( 2017 ) , were shown to alleviate the mismatch scenarios to some extent . However , the second limitation , i.e. , the requirement of clean images for building the training set , still remains . As an attempt to address this second limitation , Lehtinen et al . ( 2018 ) recently proposed the Noise2Noise ( N2N ) method . It has been shown that a denoiser , which has a negligible performance loss , can be trained without the clean target images , as long as two independent noisy image realizations for the same underlying clean image are available . Despite its ∗Corresponding author ( E-mail : tsmoon @ snu.ac.kr ) effectiveness , the requirement of the two independently realized noisy image pair for a single clean image , which may hardly be available in practice , is a critical limiting factor for N2N . In this paper , we consider a setting in which neither of above approach is applicable , namely , the pure unsupervised blind denoising setting where only single distinct noisy images are available for training . Namely , nothing is known about the noise other than it being zero-mean , additive , and independent of the clean image , and neither the clean target images for blind training nor the noisy image pairs for N2N training is available . While some recent work , e.g. , Krull et al . ( 2019 ) ; Batson & Royer ( 2019 ) ; Laine et al . ( 2019 ) , took the self-supervised learning ( SSL ) approach for the same setting , we take a generative learning approach . The crux of our method is to first learn a Wasserstein GAN ( Arjovsky et al. , 2017 ) -based generative model that can 1 ) learn and simulate the noise in the given noisy images and 2 ) generate rough , initially denoised images . Using such generative model , we then synthesize noisy image pairs by corrupting each of the initially denoised images with the simulated noise twice and use them to train a CNN denoiser as in the N2N training ( i.e. , Noisy N2N ) . We further show that iterative N2N training with refined denoised images can significantly improve the final denoising performance . We dubbed our method as GAN2GAN ( Generated-Artifical-Noise to Generated-Artificial-Noise ) and show that the denoiser trained with our method can achieve ( sometimes , even outperform ) the performance of the standard supervised-trained or N2N-trained blind denoisers for the white Gaussian noise case . Furthermore , for mixture/correlated noise or real-world noise in microscopy/CT images , for which the exact distributions are hard to know a priori , we show our denoiser significantly outperforms those standard blind denoisers , which are mismatch-trained with white Gaussian noise , as well as other baselines that operate in the same condition as ours : the SSL baseline , N2V ( Krull et al. , 2019 ) , and a more conventional BM3D ( Dabov et al. , 2007 ) . 2 RELATED WORK . Several works have been proposed to overcome the limitation of the vanilla supervised learning based denoising . As mentioned above , Noise2Self ( N2S ) ( Batson & Royer , 2019 ) and Noise2Void ( N2V ) ( Krull et al. , 2019 ) recently applied self-supervised learning ( SSL ) approach to train a denoiser only with single noisy images . Their settings exactly coincide with ours , but we show later that our GAN2GAN significantly outperforms them . More recently , Laine et al . ( 2019 ) improved N2V by incorporating specific noise likelihood models with Bayesian framework , however , their method required to know the exact noise model and could not be applied to more general , unknown noise settings . Similarly , Soltanayev & Chun ( 2018 ) proposed SURE ( Stein ’ s Unbiased Risk Estimator ) based denoiser that can also be trained with single noisy images , but it worked only with the Gaussian noise . Their work was extended in Zhussip et al . ( 2019 ) , but it required noisy image pairs as in N2N as well as the Gaussian noise constraint . Chen et al . ( 2018 ) devised GCBD method to learn and generate noise in the given noisy images using W-GAN Arjovsky et al . ( 2017 ) and utilized the unpaired clean images to build a supervised training set . Our GAN2GAN is related to Chen et al . ( 2018 ) , but we significantly improve their noise learning step and do not use the clean data at all . Table 1 summarizes and compares the settings among the above mentioned recent baselines . We clearly see that only our GAN2GAN and N2V do not utilize any “ sidekicks ” that other methods use . Additionally , there are recently published papers on blind image denoising but these also have a difference with ours . Anwar & Barnes ( 2019 ) ; Zhang et al . ( 2018 ) suggest effective CNN architectures for denoising , however , they only consider the setting in which clean images are necessary for training . Zamir et al . ( 2020 ) considers the denoising of specific camera settings , and it also requires clean sRGB images as well as the knowledge of the noise level . Thus , it can not be applied to the complete blind setting as ours , in which no information on the specific noise distribution or clean images is available . More classical denoising methods are capable of denoising solely based on the single noisy images by applying various principles , e.g. , filtering-based Buades et al . ( 2005 ) ; Dabov et al . ( 2007 ) , optimization-based Elad & Aharon ( 2006 ) ; Mairal et al . ( 2009 ) , Wavelet-based Donoho & Johnstone ( 1995 ) , and effective prior-based Zoran & Weiss ( 2011 ) . Those methods typically are , however , computationally intensive during the inference time and can not be trained from a separate set of noisy images , which limits their denoising performance . Another line of recent work worth mentioning is the deep learning-based priors or regularizers , e.g. , Ulyanov et al . ( 2018 ) ; Yeh et al . ( 2018 ) ; Lunz et al . ( 2018 ) , but their PSNRs still fell short of the supervised trained CNN-based denoisers . 3 MOTIVATION . In order to develop the core intuition for motivating our method , we first consider a simple , singleletter Gaussian noise setting . Let Z = X +N be the noisy observation of X ∼ N ( 0 , σ2X ) , corrupted by the N ∼ N ( 0 , σ2N ) . It is well known that the minimum MSE ( MMSE ) estimator of X given Z is f∗MMSE ( Z ) = E ( X|Z ) = σ2X σ2X+σ 2 N Z . We now identify the optimality of N2N in this setting . N2N Assume that we have two i.i.d . copies of the noise N : N1 and N2 . Then , let Z1 = X +N1 and Z2 = X +N2 be the two independent noisy observation pairs of X . The N2N in this setting corresponds to obtaining the MMSE estimator of Z2 given Z1 , fN2N ( Z1 ) , argmin f E ( Z2 − f ( Z1 ) ) 2 = E ( Z2|Z1 ) = E ( X +N2|Z1 ) ( a ) = E ( X|Z1 ) = σ2X σ2X + σ 2 N Z1 , ( 1 ) in which ( a ) follows fromN2 being independent ofZ1 . Note ( 1 ) has the exact same form as f∗MMSE ( Z ) , hence , estimating X with fN2N ( Z ) also achieves the MMSE , in line with ( Lehtinen et al. , 2018 ) . “ Noisy ” N2N Now , consider the case in which we again have the two i.i.d . N1 and N2 , but the noisy observations are of a noisy version of X . Namely , let X ′ = X +N0 , in which N0 ∼ N ( 0 , σ20 ) , and denote Z ′1 = X ′ +N1 and Z ′2 = X ′ +N2 as the noisy observation pairs . Then , we can define a “ Noisy ” N2N estimator as the MMSE estimator of Z ′2 given Z ′ 1 , fNoisy N2N ( Z ′ 1 , y ) , argmin f E ( Z ′2 − f ( Z ′1 ) ) 2 = E ( X ′|Z ′1 ) = σ2X ( 1 + y ) σ2X ( 1 + y ) + σ 2 N Z ′1 , ( 2 ) in which we denote y , σ20/σ 2 X and assume that 0 ≤ y < 1 . Note clearly ( 2 ) coincides with ( 1 ) when y = σ20 = 0 . Following N2N , ( 2 ) is essentially estimating X ′ based on Z ′ = X ′ + N . An interesting subtle question is what happens when we use the mapping fNoisy N2N ( Z , y ) for estimating X given Z = X +N , not X ′ given Z ′ . Our theorem below , of which proof is in the Supplementary Material ( S.M . ) , shows that for a sufficiently large σ20 , fNoisy N2N ( Z , y ) gives a better estimate of X than X ′ . Theorem 1 Consider the single-letter Gaussian setting and fNoisy N2N ( Z , y ) obtained in ( 2 ) . Also , assume 0 < y < 1 . Then , there exists some y0 s.t . ∀y ∈ ( y0 , 1 ) , E ( X − fNoisy N2N ( Z , y ) ) 2 < σ20 . Theorem 1 provides a simple , but useful , intuition that motivates our method ; if simulating the noise in the images is possible , we may carry out the N2N training iteratively , provided that a rough noisy estimate of the clean image is initially available . Namely , we can first simulate the noise to generate noisy observation pairs of the initial noisy estimate , then do the Noisy N2N training with them to obtain a denoiser that may result in a better estimate of the clean image when applied to the actual noisy image subject to denoising ( as in Theorem 1 ) . Then , we can refine the estimates by iterating the Noisy N2N training with the generated noisy observation pairs of the previous step ’ s estimate of the clean image , until convergence . To check whether above intuition is valid , we carry out a feasibility experiment . Figure 1 shows the denoising results on BSD68 ( Roth & Black , 2009 ) for Gaussian noise with σ = 25 . The blue line is the PSNR of the N2N model trained with noisy observation pairs of the clean images in the BSD training set , serving as an upper bound . The orange line , in contrast , is the PSNR of the Noisy N2N1 model that is trained with the noisy observation pairs of the noisy estimates for the clean images , which were set to be another Gaussian noise-corrupted training images . The standard deviations ( σ0 ) of the Gaussian for generating the noisy estimates are given in the horizontal axis , and the corresponding PSNRs of the estimates are given in the parentheses . Although Noisy N2N1 clearly lies much lower than the N2N upper bound , we note its PSNR is still higher than that of the initial noisy estimates , which is in line with Theorem 1 . Now , if we iterate the Noisy N2N with the previous step ’ s denoised images ( i.e. , Noisy-N2N2/Noisy-N2N3 for second/third iterations , respectively ) , we observe that the PSNR significantly improves and approaches the ordinary N2N for most of the initial σ0 values . Thus , we observe the intuition from Theorem 1 generalizes well to the image denoising case in an ideal setting , where the noise can be perfectly simulated , and the initial noisy estimates are Gaussian corrupted versions . The remaining question is whether we can also obtain similar results for the blind image denoising setting . We show our generative model-based approach in details in the next section .

Summary: This paper proposed a new method for blind image denoising using a "GAN2GAN" network. Different from previous work noise2noise, the proposed method only needs single noisy images to train the network, without noisy pairs. Given a noisy dataset, a GAN generator is trained using the real noisy but smooth patches. With the noise generator, a pair of generated noise samples are added on the noisy image content to train a denoiser. This is trained iteratively, so that the image content is better and better after cleaned by the trained denoiser. The final results on synthetic and real data showed improvements over the baselines.