GAN2GAN: GENERATIVE NOISE LEARNING FOR BLIND DENOISING WITH SINGLE NOISY IMAGES
Year: 2021

Abstract:
We tackle a challenging blind image denoising problem, in which only single distinct noisy images are available for training a denoiser, and no information about noise is known, except for it being zero-mean, additive, and independent of the clean image. In such a setting, which often occurs in practice, it is not possible to train a denoiser with the standard discriminative training or with the recently developed Noise2Noise (N2N) training; the former requires the underlying clean image for the given noisy image, and the latter requires two independently realized noisy image pair for a clean image. To that end, we propose GAN2GAN (GeneratedArtificial-Noise to Generated-Artificial-Noise) method that first learns a generative model that can 1) simulate the noise in the given noisy images and 2) generate a rough, noisy estimates of the clean images, then 3) iteratively trains a denoiser with subsequently synthesized noisy image pairs (as in N2N), obtained from the generative model. In results, we show the denoiser trained with our GAN2GAN achieves an impressive denoising performance on both synthetic and real-world datasets for the blind denoising setting; it almost approaches the performance of the standard discriminatively-trained or N2N-trained models that have more information than ours, and it significantly outperforms the recent baseline for the same setting, e.g., Noise2Void, and a more conventional yet strong one, BM3D. The official code of our method is available at https://github.com/csm9493/GAN2GAN.

heading: 1 INTRODUCTION
text: Image denoising is one of the oldest problems in image processing and low-level computer vision, yet it still attracts lots of attention due to the fundamental nature of the problem. A vast number of algorithms have been proposed over the past several decades, and recently, the CNN-based methods, e.g., Cha & Moon (2019); Zhang et al. (2017); Tai et al. (2017); Liu et al. (2018), became the throne-holders in terms of the PSNR performance. The main approach of the most CNN-based denoisers is to apply the discriminative learning framework with (clean, noisy) image pairs and known noise distribution assumption. While being effective, such framework also possesses a couple of limitations that become critical in practice; the assumed noise distribution may be mismatched to the actual noise in the data or obtaining the noise-free clean target images is not always possible or very expensive, e.g., medical imaging (CT or MRI) or astrophotographs.
Several attempts have been made to resolve above issues. For the noise uncertainty, the so-called blind training have been proposed. Namely, a denoiser can be trained with a composite training set that contains images corrupted with multiple, pre-defined noise levels or distributions, and such blindly trained denoisers, e.g., DnCNN-B in Zhang et al. (2017), were shown to alleviate the mismatch scenarios to some extent. However, the second limitation, i.e., the requirement of clean images for building the training set, still remains. As an attempt to address this second limitation, Lehtinen et al. (2018) recently proposed the Noise2Noise (N2N) method. It has been shown that a denoiser, which has a negligible performance loss, can be trained without the clean target images, as long as two independent noisy image realizations for the same underlying clean image are available. Despite its
∗Corresponding author (E-mail: tsmoon@snu.ac.kr)
effectiveness, the requirement of the two independently realized noisy image pair for a single clean image, which may hardly be available in practice, is a critical limiting factor for N2N.
In this paper, we consider a setting in which neither of above approach is applicable, namely, the pure unsupervised blind denoising setting where only single distinct noisy images are available for training. Namely, nothing is known about the noise other than it being zero-mean, additive, and independent of the clean image, and neither the clean target images for blind training nor the noisy image pairs for N2N training is available. While some recent work, e.g., Krull et al. (2019); Batson & Royer (2019); Laine et al. (2019), took the self-supervised learning (SSL) approach for the same setting, we take a generative learning approach. The crux of our method is to first learn a Wasserstein GAN (Arjovsky et al., 2017)-based generative model that can 1) learn and simulate the noise in the given noisy images and 2) generate rough, initially denoised images. Using such generative model, we then synthesize noisy image pairs by corrupting each of the initially denoised images with the simulated noise twice and use them to train a CNN denoiser as in the N2N training (i.e., Noisy N2N). We further show that iterative N2N training with refined denoised images can significantly improve the final denoising performance. We dubbed our method as GAN2GAN (Generated-Artifical-Noise to Generated-Artificial-Noise) and show that the denoiser trained with our method can achieve (sometimes, even outperform) the performance of the standard supervised-trained or N2N-trained blind denoisers for the white Gaussian noise case. Furthermore, for mixture/correlated noise or real-world noise in microscopy/CT images, for which the exact distributions are hard to know a priori, we show our denoiser significantly outperforms those standard blind denoisers, which are mismatch-trained with white Gaussian noise, as well as other baselines that operate in the same condition as ours: the SSL baseline, N2V (Krull et al., 2019), and a more conventional BM3D (Dabov et al., 2007).

heading: 2 RELATED WORK
text: Several works have been proposed to overcome the limitation of the vanilla supervised learning based denoising. As mentioned above, Noise2Self (N2S) (Batson & Royer, 2019) and Noise2Void (N2V) (Krull et al., 2019) recently applied self-supervised learning (SSL) approach to train a denoiser only with single noisy images. Their settings exactly coincide with ours, but we show later that our GAN2GAN significantly outperforms them. More recently, Laine et al. (2019) improved N2V by incorporating specific noise likelihood models with Bayesian framework, however, their method required to know the exact noise model and could not be applied to more general, unknown noise settings. Similarly, Soltanayev & Chun (2018) proposed SURE (Stein’s Unbiased Risk Estimator)based denoiser that can also be trained with single noisy images, but it worked only with the Gaussian noise. Their work was extended in Zhussip et al. (2019), but it required noisy image pairs as in N2N as well as the Gaussian noise constraint. Chen et al. (2018) devised GCBD method to learn and generate noise in the given noisy images using W-GAN Arjovsky et al. (2017) and utilized the unpaired clean images to build a supervised training set. Our GAN2GAN is related to Chen et al. (2018), but we significantly improve their noise learning step and do not use the clean data at all. Table 1 summarizes and compares the settings among the above mentioned recent baselines. We clearly see that only our GAN2GAN and N2V do not utilize any “sidekicks” that other methods use.
Additionally, there are recently published papers on blind image denoising but these also have a difference with ours. Anwar & Barnes (2019); Zhang et al. (2018) suggest effective CNN architectures for denoising, however, they only consider the setting in which clean images are necessary for training. Zamir et al. (2020) considers the denoising of specific camera settings, and it also requires clean sRGB images as well as the knowledge of the noise level. Thus, it cannot be applied to the complete blind setting as ours, in which no information on the specific noise distribution or clean images is available.
More classical denoising methods are capable of denoising solely based on the single noisy images by applying various principles, e.g., filtering-based Buades et al. (2005); Dabov et al. (2007), optimization-based Elad & Aharon (2006); Mairal et al. (2009), Wavelet-based Donoho & Johnstone (1995), and effective prior-based Zoran & Weiss (2011). Those methods typically are, however, computationally intensive during the inference time and cannot be trained from a separate set of noisy images, which limits their denoising performance. Another line of recent work worth mentioning is the deep learning-based priors or regularizers, e.g., Ulyanov et al. (2018); Yeh et al. (2018); Lunz et al. (2018), but their PSNRs still fell short of the supervised trained CNN-based denoisers.

heading: 3 MOTIVATION
text: In order to develop the core intuition for motivating our method, we first consider a simple, singleletter Gaussian noise setting. Let Z = X +N be the noisy observation of X ∼ N (0, σ2X), corrupted by the N ∼ N (0, σ2N ). It is well known that the minimum MSE (MMSE) estimator of X given Z is f∗MMSE(Z) = E(X|Z) =
σ2X σ2X+σ 2 N Z. We now identify the optimality of N2N in this setting.
N2N Assume that we have two i.i.d. copies of the noise N : N1 and N2. Then, let Z1 = X +N1 and Z2 = X +N2 be the two independent noisy observation pairs of X . The N2N in this setting corresponds to obtaining the MMSE estimator of Z2 given Z1,
fN2N(Z1) , argmin f
E(Z2 − f(Z1))2 = E(Z2|Z1) = E(X +N2|Z1) (a) = E(X|Z1) = σ2X σ2X + σ 2 N Z1, (1)
in which (a) follows fromN2 being independent ofZ1. Note (1) has the exact same form as f∗MMSE(Z), hence, estimating X with fN2N(Z) also achieves the MMSE, in line with (Lehtinen et al., 2018).
“Noisy” N2N Now, consider the case in which we again have the two i.i.d. N1 and N2, but the noisy observations are of a noisy version of X . Namely, let X ′ = X +N0, in which N0 ∼ N (0, σ20), and denote Z ′1 = X ′ +N1 and Z ′2 = X ′ +N2 as the noisy observation pairs. Then, we can define a “Noisy” N2N estimator as the MMSE estimator of Z ′2 given Z ′ 1,
fNoisy N2N(Z ′ 1, y) , argmin f E(Z ′2 − f(Z ′1))2 = E(X ′|Z ′1) =
σ2X(1 + y)
σ2X(1 + y) + σ 2 N
Z ′1, (2)
in which we denote y , σ20/σ 2 X and assume that 0 ≤ y < 1. Note clearly (2) coincides with (1) when y = σ20 = 0. Following N2N, (2) is essentially estimating X ′ based on Z ′ = X ′ + N . An interesting subtle question is what happens when we use the mapping fNoisy N2N(Z, y) for estimating X given Z = X +N , not X ′ given Z ′. Our theorem below, of which proof is in the Supplementary Material (S.M.), shows that for a sufficiently large σ20 , fNoisy N2N(Z, y) gives a better estimate of X than X ′.
Theorem 1 Consider the single-letter Gaussian setting and fNoisy N2N(Z, y) obtained in (2). Also, assume 0 < y < 1. Then, there exists some y0 s.t. ∀y ∈ (y0, 1), E(X − fNoisy N2N(Z, y))2 < σ20 .
Theorem 1 provides a simple, but useful, intuition that motivates our method; if simulating the noise in the images is possible, we may carry out the N2N training iteratively, provided that a rough noisy estimate of the clean image is initially available. Namely, we can first simulate the noise to generate noisy observation pairs of the initial noisy estimate, then do the Noisy N2N training with them to obtain a denoiser that may result in a better estimate of the clean image when applied to the actual noisy image subject to denoising (as in Theorem 1). Then, we can refine the estimates by iterating the Noisy N2N training with the generated noisy observation pairs of the previous step’s estimate of the clean image, until convergence.
To check whether above intuition is valid, we carry out a feasibility experiment. Figure 1 shows the denoising results on BSD68 (Roth & Black, 2009) for Gaussian noise with σ = 25. The blue line is the PSNR of the N2N
model trained with noisy observation pairs of the clean images in the BSD training set, serving as an upper bound. The orange line, in contrast, is the PSNR of the Noisy N2N1 model that is trained with the noisy observation pairs of the noisy estimates for the clean images, which were set to be another Gaussian noise-corrupted training images. The standard deviations (σ0) of the Gaussian for generating the noisy estimates are given in the horizontal axis, and the corresponding PSNRs of the estimates are given in the parentheses. Although Noisy N2N1 clearly lies much lower than the N2N upper bound, we note its PSNR is still higher than that of the initial noisy estimates, which is in line with Theorem 1. Now, if we iterate the Noisy N2N with the previous step’s denoised images (i.e., Noisy-N2N2/Noisy-N2N3 for second/third iterations, respectively), we observe that the PSNR significantly improves and approaches the ordinary N2N for most of the initial σ0 values. Thus, we observe the intuition from Theorem 1 generalizes well to the image denoising case in an ideal setting, where the noise can be perfectly simulated, and the initial noisy estimates are Gaussian corrupted versions. The remaining question is whether we can also obtain similar results for the blind image denoising setting. We show our generative model-based approach in details in the next section.

heading: 4 MAIN METHOD: THREE COMPONENTS OF GAN2GAN
text: To concretely describe our method, we first set the notations. We assume the noisy image Z is generated by Z = x+N, in which x denotes the underlying clean image and N denotes the zeromean, additive noise that is independent of x. For training a denoiser, we do not assume either the distribution or the covariance of N is known. Moreover, we assume only a database of n distinct noisy images, D = {Z(i)}ni=1, is available for learning a denoiser. A CNN-based denoiser is denoted as X̂φ(Z) with φ being the model parameter, and we use the standard quality metrics, PSNR/SSIM, for evaluation. Our method consists of three parts; 1) smooth noisy patch extraction, 2) training a generative model, and 3) iterative GAN2GAN training of X̂φ(Z), each of which we elaborate below.

heading: 4.1 SMOOTH NOISY PATCH EXTRACTION
text: The first step is to extract the noisy image patches from D that correspond to smooth, homogeneous areas. Our extraction method is similar to that of the GCBD proposed in (Chen et al., 2018), but we make a critical improvement. The GCBD determines a patch p (of pre-determined size) is smooth if it satisfies the following for all of its smaller sub-patches, qj , with some hyperparameters µ, γ ∈ (0, 1):
|E(qj)− E(p)| ≤ µE(p), |V(qj)− V(p)| ≤ γV(p), (3)
in which E(·) and V(·) are the empirical mean and variance of the pixel values.
While (3) works for extracting smooth patches to some extent, as we show in Figure 2(b), it does not rule out choosing patches with high-frequency repeating patterns, which are far from being smooth. Thus, we instead use the 2D discrete wavelet transform (DWT) for a new extraction rule; namely, we determine p is smooth if its four sub-band decompositions obtained by DWT, {Wk(p)}4k=1, satisfy
1
4 4∑ k=1 ∣∣∣σ̂(Wk(p))− E[σ̂W (p)]∣∣∣ ≤ λE[σ̂W (p)], (4) in which σ̂(·) is the empirical standard deviation of the wavelet coefficients, E[σ̂W (p)] , 1 4 ∑4 k=1 σ̂(Wk(p)), and λ ∈ (0, 1) is a hyperparameter. This rule is much simpler than (3), which
has to be evaluated for all the sub-patches, {qj}. Once N patches are extracted from D using (4), we subtract each patch with its mean pixel value, and obtain a set of “noise” patches, N = {n(j)}Nj=1. Such subtraction is valid since all the pixel values should be close to their mean in a smooth patch, and the noise is assumed to be zero-mean, additive.
Figure 2 compares the rules (3) and (4) by showing the quality of the “noise” patches extracted from 1,000 Gaussian-corrupted images . The two plots in Figure 2(a) show the normalized histograms of the empirical standard deviations, σ̂, of the extracted patches when true σ = {25, 50}, respectively. We clearly observe that while the σ̂’s for (4) are mostly concentrated on true σ, those of (3) have much higher variation. In addition, Figure 2(b) visualizes the randomly sampled patches of which σ̂’s were above the 90-th percentile among the extracted patches for each rule (when σ = 25). Again, it is obvious that (3) also may result in selecting the patches with high-frequency patterns, whereas (4) is much more effective for extracting accurate noise patches. Later, we show (in Figure 5) that such improved quality of the noise patches by our (4) plays an essential role; namely, our pure unsupervised learning based denoiser using (4) even outperforms the clean target image based denoiser in (Chen et al., 2018) using (3).

heading: 4.2 TRAINING A W-GAN BASED GENERATIVE MODEL
text: Equipped withD = {Z(i)}ni=1 and the extracted noise patchesN = {n(j)}Nj=1, we train a generative model, which can learn and simulate the noise as well as generate initial noisy estimates of the clean images, hence, realize the Noisy N2N training explained in Section 3. As shown in Figure 3, our model has three generators, {gθ1 , gθ2 , gθ3}, and two critics, {fw1 , fw2}, in which the subscripts stand for the model parameters. The loss functions associated with the components of our model are:
Ln(θ1,w1) , En [ fw1(n)]− Er[fw1(gθ1(r)) ] (5)
LZ(θ1,θ2,w2) , EZ [ fw2(Z) ] − EZ,r [ fw2(gθ2(Z) + gθ1(r)) ] (6)
Lcyc(θ2,θ3) , EZ [ ‖z − gθ3(gθ2(Z))‖1 ] . (7)
The loss (5) is a standard W-GAN (Arjovsky et al., 2017) loss for training the first generator-critic pair, (gθ1 , fw1), of which gθ1 learns to generate the independent realization of the noise mimicking the patches inN = {n(j)}Nj=1, taking the random vector r ∼ N (0, I) as input. The second loss (6) links the two generators, gθ1 and gθ2 , with the second critic, fw2 . The second generator gθ2 is intended to generate the estimate of the underlying clean patch for Z, i.e., coarsely denoise Z, and the critic fw2 determines how close the distribution of the generated noisy image, gθ2(Z) + gθ1(r), is to the that of Z1. Our intuition is, if gθ1 can realistically simulate the noise, then enforcing gθ2(Z) + gθ1(r) to mimick Z would result in learning a reasonable initial denoiser gθ2 . One important detail regarding gθ2 is its final activation must be the sigmoid function for stable training. The third loss (7), which
1We assume gθ2 implicitly has the cropping step for Z such that the dimension of gθ2(Z) and gθ1(r) match.
resembles the cycle loss in (Zhu et al., 2017), imposes the encoder-decoder structure between gθ2 and gθ3 , hence, helps gθ2 to compress the most redundant part of Z, i.e., the noise, and carry out the initial denoising. Once the losses are defined, training the generators and critics are done in an alternating manner, as in the training of W-GAN (Arjovsky et al., 2017), to approximately solve
min θ1,θ2,θ3 max w1,w2
[ αLn(θ1,w1)+ βLZ(θ1,θ2,w2) + γLcyc(θ2,θ3) ] , (8)
in which (α, β, γ) are hyperparameters to control the trade-offs between the loss functions. The pseudo algorithm for training a generative model is given in Algorithm 1. There are a couple of subtle points for training with the overall objective (8), and we describe the full details on model architectures and hyperparameters in the S.M.
Algorithm 1 Training a generative model, all experiments in this paper used the defaults values, ncritic = 5, nepoch = 30, m = 64, αg = 4e−4, αcritic = 5e−5, α = 5, β = 1, γ = 10
1: Require D, λ 2: N ← NoisePatchExtraction(D, λ) 3: for epGAN ← 1, nepoch do 4: Sample {n(i)}mi=1 ∼ N , {r(i)}mi=1 ∼ N(0, I), {Z(i)}mi=1 ∼ D 5: for epcritic ← 1, ncritic do 6: gw1 ← ∇w1 [Ln(θ1, w1)], gw2 ← ∇w2 [LZ(θ1, θ2, w2)] 7: w1 ← Clip(w1 + αcritic ·Adam(w1, gw1),−c, c) 8: w2 ← Clip(w2 + αcritic ·Adam(w2, gw2),−c, c) 9: end for
10: gθ1 , gθ2 , gθ3 ← ∇θ1,θ2,θ3 [ αLn(θ1, w1) + βLZ(θ1, θ2, w2) + γLcyc(θ2, θ3) ] 11: θ1 ← θ1−αg ·Adam(θ1, gθ1) , θ2 ← θ2−αg ·Adam(θ2, gθ2) , θ3 ← θ3−αg ·Adam(θ3, gθ3) 12: end for 13: return θ1, θ2

heading: 4.3 ITERATIVE GAN2GAN TRAINING OF A DENOISER
text: With our generative model, we then carry out the iterative Noisy N2N training as described in Section 3, with the generated noisy images. Namely, given each Z(i) ∈ D, we generate the pair
(Ẑ (i) 11 , Ẑ (i) 12 ) , (gθ2(Z (i)) + gθ1(r (i) 11 ), gθ2(Z (i)) + gθ1(r (i) 12 )), (9)
in which r(i)11 , r (i) 12 ∈ R128 are i.i.d. ∼ N (0, I). In contrast to the ideal case in Section 3, each generated image in (9) is a noise-corrupted version of gθ2(Z (i)), in which the corruption is done by the simulated noise gθ1(r). Denoting the set of such pairs as D̂1 = {(Ẑ (i) 11 , Ẑ (i) 12 )}ni=1, a denoiser
X̂φ(Z) is trained by minimizing LG2G(φ, D̂1) , 1n ∑n i=1(Ẑ (i) 11 − X̂φ(Ẑ (i) 12 ))
2. In LG2G(·), we only use the generated noisy images and do not use the actual observed Z(i), hence, we dubbed our training as GAN2GAN (G2G) training. Now, denoting the learned denoiser as G2G1 (with parameter φ1), we can iterate the G2G training. For the j-th iteration (with j ≥ 2), we generate(
Ẑ (i) j1 , Ẑ (i) j2 ) , ( X̂φj−1(Z (i)) + gθ1(r (i) j1 ), X̂φj−1(Z (i)) + gθ1(r (i) j2 ) ) , (10)
for each Z(i) and denote the resulting set of the pairs as D̂j . Note in (10), we update the noisy estimate of the clean image with the output of G2Gj−1. Then, the new denoiser G2Gj is obtained by computing φj , argminφ LG2G(φ, D̂j), where the minimization is done via warm-starting from φj−1. In our experiments, we show the sequence, G2Gj≥1, successively refines the denoising quality and significantly improves the initial noisy estimate, similarly as in Figure 1. Moreover, we identify the benefit of the iterative G2G training becomes greater when noise is more sophisticated; i.e., for synthetic noise, the performance of G2Gj≥1 converges after 1∼3 iterations, whereas for the real-world microscopy noise, the performance keeps increasing until larger number of iterations.

heading: 5 EXPERIMENTAL RESULTS

heading: 5.1 DATA AND EXPERIMENTAL SETTINGS
text: Data & training details In synthetic noise experiments, we always used the noisy training images from BSD400 (Martin et al., 2001). For evaluation, we used the standard BSD68 (Roth & Black,
2009) as a test set. For real-noise experiment, we experimented on two data sets: the WF set in the microscopy image datasets in (Zhang et al., 2019) and the reconstructed CT dataset. For both datasets, we trained/tested on each (Avg = n) and each dose level, respectively, which corresponds to different noise levels. For the generative model training, the patch size used for D and N was 96× 96, and n and N were set to 20, 000 (BSD) and 40, 000 (microscopy), respectively. For the iterative G2G training, the patch size for D was 120× 120 and n = 20, 500, and in every mini-batch, we generated new noisy pairs with gθ1 as in the noise augmentation of (Zhang et al., 2017). The architecture of G2Gj was set to 17-layer DnCNN in (Zhang et al., 2017). We put full details on training, model architectures and hyperparameters as well as the software platforms in the S.M. Baselines The baselines were BM3D (Dabov et al., 2007), DnCNN-B (Zhang et al., 2018), N2N (Lehtinen et al., 2018), and N2V (Krull et al., 2019). We reproduced and trained DnCNN-B, N2N and N2V using the publicly available source codes on the exactly same training data as our iterative G2G training. For DnCNN-B and N2N, which use either clean targets or two independent noisy image copies, we used 20-layers DnCNN model with composite additive white Gaussian noise with σ ∈ [0, 55]. N2V considers the same setting as ours and uses the exact same architecture as G2Gj ; more details on N2V are also given in the S.M. We could not compare with the scheme in (Laine et al., 2019), since their code cannot run beyond white Gaussian noise case in our experiments and they had an unfair advantage: they newly generate noisy images by corrupting given clean images for every mini-batch whereas we assume the given noisy images are fixed once for all. It is known that such noise augmentation significantly can increase the performance, and their code could not run in our setting in which the noisy images are fixed once given. As an upper bound, we implemented N2C(Eq.(4)), denoting a 17-layer DnCNN trained with clean target images in BSD400 and their noisy counterpart, which is corrupted by our gθ1 learned with (4).

heading: 5.2 DENOISING RESULTS ON SYNTHETIC NOISE
text: White Gaussian noise Table 2 shows the results on BSD68 corrupted by white Gaussian noise with different σ’s. Several variations of our G2G, gθ2 and the G2G iterates, G2Gj≥1, are shown for two different training data versions for learning the generative model. Firstly, we clearly observe the
iterative G2G training is very effective; namely, it significantly improves the initial noisy estimate gθ2 , particularly when the quality of the initial estimate is not good enough. This result confirms the result of Figure 1 indeed carries over to the blind denoising setting with our method. Secondly, we note G2G1 already considerably outperforms N2V, which is trained with the exact same model architecture and dataset. Finally, the performance of G2G3 is very strong; it outperforms BM3D, which knows true σ, and even sometimes outperforms the blindly trained DnCNN-B and N2N, which is trained with the same BSD400 dataset, but with more information. This somewhat counter-intuitive result is possible since our G2Gj accurately learns the correct noise level in the image, while DnCNN-B and N2N are trained with the composite noise levels, σ ∈ [0, 55]. Mixture and correlated noise Table 3 shows the results on mixture and correlated noise beyond white Gaussian. Note our G2Gj does not assume any distributional or correlation structure of the noise, hence, it can still run as long as the assumption on the noise holds. In the table, the G2G results
are for (BSD) as specified above. Moreover, DnCNN-B and N2N are still blindly trained with the mismatched white Gaussian noise. For mixture noise, we tested with two cases. Case A corresponds to the same setting as given in (Chen et al., 2018), i.e., 70% ∼ N (0, 0.12), 20% ∼ N (0, 1), and
10% ∼ Unif[−s, s] which means the random variable that is uniformly distributed between [−s, s] with s = 15, 25. For case B, we tested with larger variances, i.e., 70% Gaussian N(0, 152), 20% Gaussian N(0, 252), and 10% Uniform [−s, s] with s = 30, 50. For correlated noise, we generated the following noise for each `-th pixel,
N` = ηM` + (1− η) ( 1√ |NB`| ∑ m∈NB` Mm ) , ` = 1, 2, . . .
in which {M`} are white Gaussian N (0, σ2), NB` is the k × k neighborhood patch except for the pixel `, and η is a mixture parameter. We set η = 1/ √ 2 such that the marginal distribution of N` is also N (0, σ2) and set k = 16. Note in this case, N` has a spatial correlation, and we tested with σ = 15, 25. From the table, we first note that DnCNN-B and N2N suffer from serious performance degradation for both mixture and correlated noises due to noise mismatch, and the conventional BM3D outperforms them for some cases (e.g., Case A for mixture noise). However, we note our G2G2 can still denoise very well after just two iterations and outperforms all the baselines for all noise types. Note N2V seriously suffers and is not comparable to ours. Finally, N2C(Eq.(4)) is a sound upper bound for all noise types, confirming the correctness of the extraction rule (4).

heading: 5.3 DENOISING RESULTS ON REAL NOISE
text: We also test our method on the real-world noise. While some popular real noise is known to have source-dependent characteristics, there are also cases in which the noise is source-independent and pixel-wise correlated, which satisfies the assumption of our method. We tested on two such datasets, the Wide-Focal (WF) set in the microscopy image dataset (Zhang et al., 2019) and a Reconstructed CT dataset. A more detailed description and analysis on these two datasets are in S.M. The WF
and Reconstructed CT data has 5 sets (Avg = 1, 2, 4, 8, 16) and 4 sets (Dose=25, 50, 75, 100) with different noise levels, respectively. We did not exploit the fact that the images are multiple noisy measurements of a clean image, which enables employing N2N, but treated them as noisy images of distinct clean images. Figure 4(a) and 4(b) shows the PSNR of all methods for each dataset, respectively, averaged over all sets. The baselines were DnCNN-B, BM3D and N2V. We note BM3D estimated noise σ using the method in Chen et al. (2015). We iterated until G2G3 and N2C(Eq.(4)) was an upper bound for each set. We clearly observe that the performance of G2Gj significantly improves (over gθ2) as the iteration continues. In results, G2G3 becomes significantly better than DnCNN-B and N2V as well as BM3D, still one of the strongest baselines for real-world noise denoising when no clean target images are available, for both datasets. We report more detailed experimental results (including SSIM) on both datasets in S.M. Moreover, the inference time for BM3D is about 4.5∼5.0 seconds per image since a noise estimation has to be done for each image separately, whereas that for G2Gj is only 4 ms (on GPU), which is another significant advantage of our method. Figure 4(c) shows the visualizations on the WF, and we give more examples in the S.M.

heading: 5.4 ABLATION STUDY
text: Noise patch extraction Here, we evaluate the effect of the noisy patch extraction rules (3) and (4) in the final denoising performance. Figure 5 compares the PSNR of N2C(GCBD Eq.(3)), a re-implementation of (Chen et al., 2018), N2C(Ours Eq.(4)) and the best G2G, for each dataset.
We note neither source code nor training data of (Chen et al., 2018) is publicly available, and the PSNR in (Chen et al., 2018) could not be reproduced (with the exact same η and γ as in (Chen et al., 2018)). From the figure, we clearly observe the significant gap between N2C(Our Eq.(4)) and N2C(GCBD Eq.(3)), particularly when the noise is not white Gaussian. Moreover, our pure unsupervised G2G with (4) even outperforms N2C(GCBD Eq.(3)) that utilizes the clean target images, confirming the quality
difference shown in Figure 2(b) significantly affects learning noise and a denoiser.
Generative model and iterative G2G training Figure 6(a) shows the PSNRs of gθ2 on BSD68/Gaussian(σ = 25) trained with three variations; “No LZ” for no fw2 , “No Lcyc” for no (7) and gθ3 , and “No sigmoid” for no sigmoid activation at the output layer of gθ2 . We confirm that our proposed architecture achieves the highest PSNR for gθ2 , the sigmoid activation and fw2 are essential, and the cylce loss (7) is also important. Achieving a decent PSNR for gθ2 is beneficial for saving the number of G2G iterations and achieving high final PSNR. More detailed analyses on the generative model architecture are in the S.M. Figure 6(b) and 6(c) show the effect of the quality of
the initial estimate for the iterative G2G training. From Figure 1, one may ask whether gθ2 is indeed necessary, since even when σ0 ≈ σ, the iterating the Noisy N2N can mostly achieve the upper bound. Hence, for samples of synthetic and real microscopy data, we evaluate how G2Gj performs when the iteration simply starts with Z. Figure 6(b) shows a somewhat surprising result that for synthetic noises, starting from Z achieves essentially the same performance as starting from gθ2 with a couple more G2G iterations. However, for real microscopy noise case in Figure 6(c), WF(Avg= 1) in which starting from Z achieves far lower performance than starting from gθ2 , justifying our generative model for attaining the initial noisy estimate.

heading: 6 CONCLUDING REMARK
text: Motivated by a novel observation on Noisy N2N, we proposed a novel GAN2GAN method, which can tackle the challenging blind image denoising problem solely with single noisy images. Our method showed impressive denoising performance that even sometimes outperform the methods with more information as well as VST+BM3D for real noise denoising. As a future work, we plan to extend our framework to more explicitly handle the source-dependent real-world noise.

heading: ACKNOWLEDGMENT
text: This work was supported in part by NRF Mid-Career Research Program [NRF-2021R1A2C2007884] and IITP grant [No.2019- 0-01396, Development of framework for analyzing, detecting, mitigating of bias in AI model and training data], funded by the Korean government (MSIT).

Review summaries:
This paper proposes a framework to train a network to remove noises, which are zero-mean, additive and independent of the clean image, with only noisy images and without knowing the noise statistics. They mathematically prove that a network, which is trained from pairs of images generated by adding simulated noises into the noisy image, can remove noises from the input noisy image. The proposed framework can remove the noises from the input noisy image. Then, it adds simulated noises into the denoised image to generate a pair of images and train the next network to further remove noises and it does this process iteratively. The proposed framework follows GCBD and utilizes flat textureless regions to train a network to simulate noises and proposes a wavelet based method to effectively distinguish flat regions from the ones that contain high-frequency repeating patterns. The experimental results show that the proposed method has good performance under simulated Gaussian noises as well as WT and CT datasets.

This paper addresses a challenging task of blind image denoising where a single noisy image is provided with assumption that it is zero mean, additive and independent from the original image content. This is mostly the real-world scenario. Different from the recent N2N training, the authors propose a GAN2GAN based method since this blind setting cannot be trained by N2N. N2N or deterministic training needs explicit or implicit knowledge of clean image in order to be trained whereas the GAN2GAN method does not, leading to more realistic and efficient training. This method first attempts to simulate noise given the noisy image, generate rough and noisy estimates of the clean image, and iteratively train a denoiser with synthetic noisy pairs from the generator. For blind denoising, this work produces impressive results for synthetic and real world blind denoising. 

This paper proposed a new method for blind image denoising using a "GAN2GAN" network. Different from previous work noise2noise, the proposed method only needs single noisy images to train the network, without noisy pairs. Given a noisy dataset, a GAN generator is trained using the real noisy but smooth patches. With the noise generator, a pair of generated noise samples are added on the noisy image content to train a denoiser. This is trained iteratively, so that the image content is better and better after cleaned by the trained denoiser. The final results on synthetic and real data showed improvements over the baselines.