Learning to Generate Realistic Noisy Images via Pixel-level Noise-aware  Adversarial Training

1 Introduction . Image denoising is an important yet challenging problem in low-level vision . It aims to restore a clean image from its noisy counterpart . Traditional approaches concentrate on designing a rational maximum a posteriori ( MAP ) model , containing regularization and fidelity terms , from a Bayesian perspective [ 1 ] . Some image priors like low-rankness [ 2 , 3 , 4 ] , sparsity [ 5 ] , and non-local similarity [ 6 , 7 ] are exploited to customize a better rational MAP model . However , these hand-crafted methods are inferior in representing capacity . With the development of deep learning , image denoising has witnessed significant progress . Deep convolutional neural network ( CNN ) applies a powerful learning model to eliminate noise and has achieved promising performance [ 8 , 9 , 10 , 11 , 12 , 13 , 14 ] . These deep CNN denoisers rely on a large-scale dataset of real-world noisy-clean image pairs . Nonetheless , collecting even small datasets is extremely tedious and labor-intensive . The process of acquiring real-world noisy-clean image pairs is to take hundreds of noisy images of the same scene and average them to get the clean image . To get more image pairs , researchers try to synthesize noisy images . In particular , there are two common settings for synthesizing noisy images . As shown in Fig . 1 ( a1 ) , setting1 directly adds the additive white Gaussian noise ( AWGN ) with the clean RGB image . For a long time , single image denoising [ 15 , 16 , 17 , 18 , 19 , 10 ] is performed with setting1 . Nevertheless , fundamentally different from AWGN , real camera noise is generally more sophisticated and signaldependent [ 20 , 21 ] . The noise produced by photon sensing is further affected by the in-camera signal processing ( ISP ) pipeline ( e.g. , Gama correction , compression , and demosaicing ) . Models ∗Haoqian Wang is the corresponding author , email : wanghaoqian @ tsinghua.edu.cn 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . trained with setting1 are easily over-fitted to AWGN and fail in real noise removal . Setting2 is based on ISP-modeling CNN [ 22 ] and Poisson-Gaussian [ 21 , 23 ] noise model that modeling photon sensing with Poisson and remaining stationary disturbances with Gaussian has been adopted in RAW denoising . As shown in Fig . 1 ( a2 ) , setting2 adds a Poisson-Gaussian noise with the clean RAW image and then passes the result through a pre-trained RAW2RGB CNN to obtain the RGB noisy counterpart . Notably , when the clean RAW image is unavailable , a pre-trained RGB2RAW CNN is utilized to transform the clean RGB image to its RAW counterpart [ 22 ] . However , setting2 has the following drawbacks : ( i ) The noise is assumed to obey a hand-crafted probability distribution . However , because of the randomness and complexity of real camera noise , it ’ s difficult to customize a hand-crafted probability distribution to model all the characteristics of real noise . ( ii ) The ISP pipeline is very sophisticated and hard to be completely modeled . The RAW2RGB branch only learns the mapping from the clean RAW domain to the clean RGB space . However , the mapping from the Poisson-Gaussian noisy RAW domain to the real noisy RGB space can not be ensured . ( iii ) The ISP pipelines of different devices vary significantly , which results in the poor generality and robustness of ISP modeling CNNs . Thus , whether noisy images are synthesized with setting1 or 2 , there still remains a discrepancy between synthetic and real noisy datasets . We notice that GAN utilizes the internal information of the input image and external information from other images when modeling image priors . Hence , we propose to use GAN to adaptively learn the real noise distribution . GAN is firstly introduced in [ 24 ] and has been proven successful in image synthesis [ 25 , 26 , 27 ] and translation [ 26 , 27 ] . Subsequently , GAN is applied to image restoration and enhancement , e.g. , super resolution [ 28 , 29 , 30 ] , style transfer [ 27 , 31 ] , enlighten [ 32 , 33 ] , deraining [ 34 ] , dehazing [ 35 ] , image inpainting [ 36 , 37 ] , image editing [ 38 , 39 ] , and mobile photo enhancement [ 40 , 41 ] . Although GAN is widely applied in low-level vision tasks , few works are dedicated to investigating the realistic noise generation problem [ 42 ] . Chen et al . [ 43 ] propose a simple GAN that takes Gaussian noise as input to generate noisy patches . However , as in general , this GAN is image-level , i.e. , it treats images as samples and attempts to approximate the probability distribution of real-world noisy images . This image-level GAN neglects that each pixel of a real noisy image is a random variable and the real noise is spatio-chromatically correlated , thus results in coarse learning of the real noise distribution . To alleviate the above problems , this work focuses on learning how to generate realistic noisy images so as to augment the training data for real denoisers . To begin with , we propose a simple yet reasonable noise model that treats each pixel of a real noisy image as a random variable . This noise model splits the noise generation problem into two sub-problems : image domain alignment and noise domain alignment . Subsequently , to tackle these two sub-problems , we propose a novel Pixel-level Noise-aware Generative Adversarial Network ( PNGAN ) . During the training procedure of PNGAN , we employ a pre-trained real denoiser to map the generated and real noisy images into a nearly noise-free solution space to perform image domain alignment . Simultaneously , PNGAN establishes a pixel-level adversarial training that encourages the generator to adaptively simulate the real noise distribution so as to conduct the noise domain alignment . In addition , for better real noise fitting , we present a lightweight yet efficient CNN architecture , Simple Multi-scale Network ( SMNet ) as the generator . SMNet repeatedly aggregates multi-scale features to capture rich auto-correlation , which provides more sufficient spatial representations for noise simulating . Different from general image-level GAN , our discriminator is pixel-level . The discriminator outputs a score map . Each position on the score map indicates how realistic the corresponding noisy pixel is . With this pixellevel noise-aware adversarial training , the generator is encouraged to create solutions that are highly similar to real noisy images and thus difficult to be distinguished . In conclusion , our contributions can be summarized into four points : ( 1 ) We formulate a simple yet reasonable noise model . This model treats each noisy pixel as a random variable and then splits the noisy image generation into two parts : image and noise domain alignment . ( 2 ) We propose a novel framework , PNGAN . It establishes an effective pixel-level adversarial training to encourage the generator to favor solutions that reside on the manifold of real noisy images . ( 3 ) We customize an efficient CNN architecture , SMNet learning rich multi-scale auto-correlation for better noise fitting . SMNet serves as the generator in PNGAN costing only 0.8M parameters . ( 4 ) Qualitative validation shows that noise generated by PNGAN is highly similar to real noise in terms of intensity and distribution . Quantitative experiments demonstrate that a series of denoisers finetuned with the generated noisy images achieve SOTA results on four real denoising benchmarks . 2 Proposed Method . As shown in Fig . 1 , the pipeline of using PNGAN to perform data augmentation consists of three phases . ( a ) is the synthesizing phase . ( a1 ) and ( a2 ) are two common synthetic settings . In this phase , we produce the synthetic noisy image from its clean RGB or RAW counterpart . ( b ) is the training phase of PNGAN . The generator G adopts the synthetic image as input . Which synthetic setting is selected is controlled by the switch . By using a pre-trained real denoiser Dd , PNGAN establishes a pixel-level noise-aware adversarial training between the generator G and discriminator D so as to simultaneously conduct image and noise domain alignment . Dd is set as RIDNet [ 44 ] in this work . ( c ) is the finetuning phase . Firstly , in ( c1 ) , the generator creates extended fake noisy-clean image pairs . Secondly , in ( c2 ) , the fake and real data are jointly utilized to finetune a series of real denoisers . 2.1 Pixel-level Noise Modelling . Real camera noise is sophisticated and signal-dependent . Specifically , in the real camera system , the RAW noise produced by photon sensing comes from multiple sources ( e.g. , short noise , thermal noise , and dark current noise ) and is further affected by the ISP pipeline . Besides , illumination changes and camera movement inevitably lead to spatial pixel misalignment and color or brightness deviation . Hence , hand-designed noise models based on mathematical assumptions are difficult to accurately and completely describe the properties of real noise . Different from previous methods , we don ’ t base our noise model on any mathematical assumptions . Instead , we use CNN to implicitly simulate the characteristics of real noise . We begin by noting that when taking multiple noisy images of the same scene , the noise intensity of the same pixel varies a lot . Simultaneously , affected by the ISP pipeline , the real noise is spatio-chromatically correlated . Thus , the correlation between different pixels of the same real noisy image should be considered . In light of these facts , we treat each pixel of a real noisy image as a random variable and formulate a simple yet reasonable noise model : Irn [ i ] = Îclean [ i ] +N [ i ] , Dd ( Irn ) [ i ] = Îclean [ i ] , 1 ≤ i ≤ H ×W , ( 1 ) where Îclean ∈ RH×W×3 is the predicted clean counterpart of Irn , it ’ s denoised by Dd . Each N [ i ] is a random noise variable with unknown probability distribution . Therefore , each Irn [ i ] can also be viewed as a distribution-unknown random variable . Now we aim to design a framework to generate a fake noisy image Ifn ∈ RH×W×3 such that the probability distribution of Ifn [ i ] and Irn [ i ] is as close as possible . Please note that the mapping learned by Dd is precisely from Irn to Îclean . If the constant in Eq . ( 1 ) is set as the clean image Iclean ∈ RH×W×3 , the subsequent domain alignment will introduce unnecessary errors and eventually lead to inaccurate results . 2.2 Pixel-level Noise-aware Adversarial Training . Our goal is to generate realistic noisy images . According to the noise model in Eq . ( 1 ) , we split this problem into two sub-problems : ( i ) Image domain alignment aims to align Îclean [ i ] . ( ii ) Noise domain alignment targets at modeling the distribution of N [ i ] . To handle the sub-problems , PNGAN establishes a novel pixel-level noise-aware adversarial training between G and D in Fig . 1 ( b ) . Image Domain Alignment . A very naive strategy to construct both image and noise domain alignment is to directly minimize the distance of Ifn and Irn . However , due to the intrinsic randomness , complexity , and irregularity of real noise , directly deploying L1 loss between Ifn and Irn is unreasonable and drastically damages the quality of Ifn . Besides , as analyzed in Sec . 2.1 , each pixel of Irn is a distribution-unknown random variable . This indicates that such a naive strategy challenges the training and may easily cause the non-convergence issue . Therefore , the noise interference should be eliminated while constructing the image domain alignment . To this end , we feed Ifn and Irn into Dd to obtain their denoised versions and then perform L1 loss between Ifn and Irn : L1 = H×W∑ i=1 ∣∣∣∣Dd ( Ifn ) [ i ] −Dd ( Irn ) [ i ] ∣∣∣∣1 = H×W∑ i=1 ∣∣∣∣Dd ( Ifn ) [ i ] − Îclean [ i ] ∣∣∣∣1 . ( 2 ) By using Dd , we can transfer Irn and Ifn into a nearly noise-free solution space . The value of Îclean is relatively stable . Therefore , minimizing L1 can encourage G to favor solutions that after being denoised by Dd converge to Îclean . In this way , the image domain alignment is constructed . Noise Domain Alignment . Becasue of the complexity and variability of real noise , it ’ s hard to completely seperate N [ i ] from Irn [ i ] in Eq ( 1 ) . Fortunately , we note that on the basis of constructing the image domain alignment of Îclean [ i ] , the noise domain alignment of N [ i ] is equivalent to the distribution estimation of Irn [ i ] . Additionally , as the real noise is signaldependent , the alignment between Ifn [ i ] and Irn [ i ] is more beneficial to capture the correlation between noise and scene . We denote the distribution of Irn [ i ] as Pdata ( xi ) , some real noisy pixel samples of Irn [ i ] as { x1i , x2i , ... , xmi } such that xki ∼ Pdata ( xi ) , and the distribution of Ifn [ i ] as PG ( xi ; θG ) . Here θG is the parameter of G. Then we formulate the noise domain aligment into a maximum likelihood estimation problem : θ∗G = arg max θG H×W∑ i=1 m∑ k=1 logPG ( x k i ; θG ) = arg max θG Ei [ Exki [ logPG ( x k i ; θG ) ] ] , ( 3 ) where E means taking the average value . To approach this upper bound as close as possible , we present D and establish the pixel-level adversarial traininig between G and D. The architecture of D is shown in Fig . 2 . D consists of 4 convolutional ( conv ) layers and utilizes LeakyReLU activation ( α = 0.2 ) . General discriminator treats a image as a sample and outputs a score indicating how realistic the image is . Instead , D is a pixel-level classifier . D adopts the fake and real noisy images as input in a mini-batch and outputs a score map P ∈ RH×W for each image . Specifically , the information of P [ i ] ∈ [ 0 , 1 ] is the probability value indicating how realistic PG ( xi ; θG ) is . G aims to generate more realistic Ifn [ i ] to fool D while D targets at distinguishing Ifn [ i ] from Irn [ i ] . According to Eq . ( 3 ) , we formulate the adversarial training between G and D as a min-max problem : min θG max θD Ei [ EIrn [ log ( D ( Irn ; θD ) [ i ] ) ] ] + Ei [ EIfn [ log ( 1−D ( Ifn ; θD ) [ i ] ) ] ] , ( 4 ) where EIrn and EIfn respectively represent the operation of taking the average for all fake and real data in the mini-batch . As analyzed in [ 45 ] , to make GANs analogous to divergence minimization and produce sensible predictions based on the a priori knowledge that half of the samples in the mini-batch are fake , we utilize the recently proposed relativistic discriminator [ 45 ] as follow : D ( Irn ; θD ) = σ ( CD ( Irn ) ) , DRa ( Irn , Ifn ) = σ ( CD ( Irn ) − EIfn ( CD ( Ifn ) ) ) , D ( Ifn ; θD ) = σ ( CD ( Ifn ) ) , DRa ( Ifn , Irn ) = σ ( CD ( Ifn ) − EIrn ( CD ( Irn ) ) ) , ( 5 ) where DRa denotes the relativistic discriminator , σ means the Sigmoid activation , and CD represents the non-transformed discriminator output . DRa estimates the probability that real data is more realistic than fake data and also directs the generator to create a fake image that is more realistic than real images . The loss functions of D and G are then defined in a symmetrical form : LD = −Ei [ EIrn [ log ( DRa ( Irn , Ifn ) [ i ] ) ] + EIfn [ log ( 1−DRa ( Ifn , Irn ) [ i ] ) ] ] , LG = −Ei [ EIrn [ log ( 1−DRa ( Irn , Ifn ) [ i ] ) ] + EIfn [ log ( DRa ( Ifn , Irn ) [ i ] ) ] ] . ( 6 ) During the training procedure , we fix D to train G and fix G to train D iteratively . Minimizing LG and LD alternately allows us to train a generative model G with the goal of fooling the pixel-level discriminator D that is trained to distinguish fake noisy images from real noisy images . This pixellevel noise-aware adversarial training scheme encourages G to favor perceptually natural solutions that reside on the manifold of real noisy images so as to construct the noise domain alignment .

Summary: The authors propose a pixel-level noise-aware generative adversarial network (PNGAN). They improve the denoising performance by generating more realistic noisy images.  Quantitative experiments prove that the performance of the existing denoising model can be significantly improved after being trained with the synthetic realistic noisy images.