Learning to Generate Realistic Noisy Images via Pixel-level Noise-aware  Adversarial Training
Authors: Yuanhao Cai
Xiaowan Hu
Haoqian Wang
Yulun Zhang
Hanspeter Pfister
Donglai Wei
Venue: NeurIPS 2021 Poster (2021)
Keywords: Applications
Computer Vision
Low-level Vision
Computational Photography
Image Restoration
Real Image Denoising
Realistic Noisy Image Generation
Real Noise Removal

Abstract: Existing deep learning real denoising methods require a large amount of noisy-clean image pairs for supervision. Nonetheless, capturing a  real noisy-clean dataset is an unacceptable expensive and cumbersome procedure. To alleviate this problem, this work investigates how to generate realistic noisy images. Firstly, we formulate a simple yet reasonable noise model that treats each real noisy pixel as a random variable. This model splits the noisy image generation problem into two sub-problems: image domain alignment and noise domain alignment. Subsequently, we propose a novel framework, namely Pixel-level Noise-aware Generative Adversarial Network (PNGAN). PNGAN employs a pre-trained real denoiser to map the fake and real noisy images into a nearly noise-free solution space to perform image domain alignment. Simultaneously, PNGAN establishes a pixel-level adversarial training to conduct noise domain alignment. Additionally, for better noise fitting, we present an efficient architecture Simple Multi-scale Network (SMNet) as the generator. Qualitative validation shows that noise generated by PNGAN is highly similar to real noise in terms of intensity and distribution. Quantitative experiments demonstrate that a series of denoisers trained with the generated noisy images achieve state-of-the-art (SOTA) results on four real denoising benchmarks.

Learning to Generate Realistic Noisy Images via Pixel-level Noise-aware Adversarial Training
Yuanhao Cai 1.2 Xiaowan Hu 1,2 Haoqian Wang 1.2,* Yulun Zhang Hanspeter Pfister 4, Donglai Wei Shenzhen International Graduate School, Tsinghua University, 2 Shenzhen Institute of Future Media Technology.
3 ETH Zürich, HHavard University, 5 Boston College
Abstract
Existing deep learning real denoising methods require large amount of noisy- clean image pairs for supervision. Nonetheless, capturing real noisy-clean dataset is an unacceptable expensive and cumbersome procedure. To alleviate this problem, this work investigates how generate realistic noisy images. Firstly, we formulate a simple yet reasonable noise model that treats each real noisy pixel as random variable. This model splits the noisy image generation problem into two sub- problems: image domain alignment and noise domain alignment. Subsequently, we propose a novel framework, namely Pixel-level Noise- aware Generative Adver- sarial Network (PNGAN). PNGAN employs a pre- pre-trained real denoiser to map the fake and real noisy images into a nearly noise-free solution space to perform image domain alignment. Simultaneously, PNGAN establishes pixel-level adversarial training to conduct noise domain alignment. Additionally, for better noise fitting, we present efficient architecture Simple Multi-scale Network (SMNet) as the generator. Qualitative validation shows that noise generated by PNGAN is highly similar to real noise in terms of intensity and distribution. Quantitative experiments demonstrate that a series of denoisers trained with the generated noisy images achieve state-of-the-art (SOTA) results on four real denoising benchmarks.
Introduction
Image denoising is an important yet challenging problem in low-level vision. aims to restore clean image from its noisy counterpart. Traditional approaches concentrate on designing a rational maximum a posteriori (MAP) model, containing regularization and fidelity terms, from a Bayesian perspective [1]. Some image priors like low-rankness [2, 4], sparsity [5], and non-local similarity [6, 7] are exploited to customize a better rational MAP model. However, these hand-crafted methods are inferior in representing capacity. With the development of deep learning, image denoising has witnessed significant progress. Deep convolutional neural network (CNN) applies a powerful learning model to eliminate noise and has achieved promising performance [8, 9, 10, 11, 12, 13, 14]. These deep CNN denoisers rely on a large-scale dataset of real-world noisy-cleam image pairs. Nonetheless, collecting small datasets is extremely tedious and labor- inttrninte The process of acquiring real-world clesy-clean image pairs is to take hundreds of noisy images of the same scene and average them to get the clean image. To get more image pairs, researchers try synthesize noisy images.
In particular, there are two common settings for synthesizing noisy images. As shown Fig. (al), settingl directly adds the additive white Gaussian noise (AWGN) with the clean RGB image. For long time, single image denoising [15, 16, 17, 18, 19, 10] performed with setting 1. Nevertheless, fundamentally different from AWGN, real camera noise is generally more sophisticated and signal- dependent[20, 21J. The noise produced by photon sensing is further affected by the in-camera signal processing (ISP) pipeline (e.g., Gama correction, compression, and demosaicing). Models
Haoqian Wang is the corresponding author, email: Langhaoqian@tsinnghuun
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

trained with settingl are easily over-fitted to AWGN and fail in real noise removal. Setting2 is based on ISP-modeling CNN [22] and Gaasssinn [21, 23] noise model that modeling photon sensing with Poisson and remaining stationary disturbances with Gaussian has been adopted in RAW denoising. As shown in Fig. 1 (a2); setting2 adds a GaassnnG noise with the clean RAW image and then passes the result through pre-trained RAW2RGB CNN to obtain the RGB noisy counterpart. Notably, when the clean RAW image is unavailable, a pre-trained RGB2RAW CNN is utilized to transform the clean RGB image to its RAW counterpart [22]. However, setting2 has the following drawbacks: (i) The noise is assumed to obey a hand-crafted probability distribution. However, because of the randomness and complexity of real camera noise, it's difficult to customize a hand-crafted probability distribution to model all the characteristics of real noise. (ii) The ISP pipeline is very sophisticated and hard to be completely modeled. The RAW2RGB branch only learns the mapping from the clean RAW domain to the clean RGB space. However, the mapping from the Poisson-Gaussiar noisy RAW domain the real noisy RGB space can not be ensured. (iii) The ISP pipelines of different devices vary significantly, which results in the poor generality and robustness of ISP modeling CNNs. Thus, whether noisy images synthesized with setting1 or 2, there still remains a discrepancy between synthetic and real noisy datasets. We notice that GAN utilizes the internal information of the input image and external information from other images when modeling image priors. Hence, we propose to use GAN to adaptively learn the real noise distribution.
GAN is firstly introduced in [24] and has been proven successful in image synthesis [25, 26, 27] and translation [26, 27]. Subsequently. GAN is applied to image restoration and enhancement, e.g., super resolution [28, 20]. style transfer [27,31], enlighten [32, 33], deraining [34], dehazing [35], image inpainting [36, 37], image editing [38, 391, and mobile photo enhancement [40, 41J. Although
GAN is widely applied in low-level vision tasks, few works are dedicated to investigating the realistic noise generation problem [42]. Chen et al. [43] propose a simple GAN that takes Gaussian noise as input to generate noisy patches. However, as in general, this GAN is image-level, i.e., it treats images as samples and attempts to approximate the probability distribution of real-world noisy images. This image-level GAN neglects that each pixel of a real noisy image is a random variable and the real noise is caatii-cromatically correlated, thus results in coarse learning of the real noise distribution.
To alleviate the above problems, this work focuses learning how generate realistic noisy images so as to augment the training data for real denoisers. To begin with, we propose a simple yet reasonable noise model that treats each pixel of a real noisy image as a random variable. This noise model splits the noise generation problem into two sub-problems: image domain alignment and noise domain alignment. Subsequently, to tackle these two uub-problems, we propose novel Pixel-level Noise-aware Generative Adversarial Network (PNGAN). During the training procedure of PNGAN, we employ a pre-trained real denoiser map the generated and real noisy images into nearly noise-free solution space to perform image domain alignment. Simultaneously, PNGAN establishes a pixel-level adversarial training that encourages the generator to adaptively simulate the real noise distribution so as to conduct the noise domain alignment. In addition, for better real noise fitting, we present a lightweight yet efficient CNN architecture, Simple Multi-scale Network (SMNet) as the generator. SMNet repeatedly aggregates multi-scale features to capture rich auto-correlation, which provides more sufficient spatial representations for noise simulating. Different from general image-level GAN, our discriminator is pixel-level. The discriminator outputs score map. Each position on the score map indicates how realistic the corresponding noisy pixel is. With this pixel- level noise-aware adversarial training, the generator is encouraged to create solutions that are highly similar to real noisy images and thus difficult to be distinguished.
In conclusion, our contributions can be summarized into four points:
(1) We formulate simple yet reasonable noise model. This model treats each noisy pixel random variable and then splits the noisy image generation into two parts: image and noise domain alignment.
(2) We propose a novel framework, PNGAN. It establishes an effective pixel-level adversarial training to encourage the generator to favor solutions that reside on the manifold of real noisy images.
(3) We customize an efficient CNN architecture, SMNet learning rich multi-scale auto-correlation for better noise fitting. SMNet serves as the generator in PNGAN costing only 0.8M parameters.
(4) Qualitative validation shows that noise generated by PNGAN is highly similar to real noise in terms of intensity and distribution. Quantitative experiments demonstrate that series of denoisers finetuned with the generated noisy images achieve SOTA results on four real denoising benchmarks.
2

---

(a) Syathesizing Phase
(b) Training Phase
(c)
Finetuning Phase
(a1) Synthetic Seningt
(b11 Image Domain Aligment
(c1) Fake Ime Pairs Generating
(Pre- Prre-trained RIDNet)
Da
Lype
Imn
Clon RAW or RGB
Generator
Fake Image Pain
AWGN
Switch o
G
Iyn
Pubase-Canolin
Far Suste
Generator
DN
fn
D
Any Denolser
Dicciminator
Image Real Denoiser Finctuning
Synthetic Setting2
Domaln
Figure 1: The pipeline of using PNGAN to perform data augmentation. It divided into: (a) symthesizing phase, (b) training phase, and (c) finetuning phase. Please refer to the text (Sec. 2) for more detailed descriptions.
2 Proposed Method
As shown in Fig. I, the pipeline of using PNGAN to perform data augmentation consists of three phases. (a) is the synthesizing phase. (al) and (a2) are two common synthetic settings. In this phase, we produce the synthetic noisy image from its clean RGB or RAW counterpart. (b) is the training phase of PNGAN. The generator G adopts the synthetic image as input. Which synthetic setting is selected is controlled by the switch. By using pre-trained real denoiser Da, PNGAN establishes a pixel-level noise-aware adversarial training between the generator G and discriminator simultaneously conduct image and noise domain alignment. Da is set as RIDNet [44] in this work. (c) is the finetuning phase. Firstly, in (cl), the generator creates extended fake noisy-clean image pairs. Secondly, in (c2), the fake and real data are jointly utilized to finetune a series of real denoisers.
so as to
2.1 Pixel-level Noise Modelling
Real camera noise is sophisticated and signal-dependent. Specifically, in the real camera system, the RAW noise produced by photon sensing comes from multiple sources (e.g., short noise, thermal noise, and dark current noise) and is further affected by the ISP pipeline. Besides, illumination changes and camera movement inevitably lead to spatial pixel misalignment and color or brightness deviation. Hence, hand-designed noise models based on mathematical assumptions are difficult to accurately and completely describe the properties of real noise. Different from previous methods, we don' 't base our noise model on any mathematical assumptions. Instead, we use CNN to implicitly simulate the characteristics of real noise. We begin by noting that when taking multiple noisy images of the same scene, the noise intensity of the same pixel varies a lot. Simultaneously, affected by the ISP pipeline, the real noise is spatio-chree chromatically correlated. Thus, the correlation between different pixels of the same real noisy image should be considered. In light of these facts, we treat each pixel of a real noisy image random variable and formulate a simple yet reasonable noise model:
I [ = icteam [i] N Da(Lrn)( = Iciean [t]. H
where Ietean R.. Hxxx3 is the predicted clean counterpart of Iyn, it's denoised by Da. Each N|i is a random noise variable with unknown probability distribution. Therefore, each I, [i] can also be viewed as
crtrbbuttonuunnnoooe random variable. Now we aim to design a framework to generate fake noisy image fn E # such that the probability distribution of [ and I,,, [i] is as
close as possible. Please note that the mapping learned by D is precisely from I,n to Letean If the constant in Eq. (1) is set as the clean image Ieteon E IR x x3 the subsequent domain alignment will introduce unnecessary errors and eventually lead to inaccurate results.
2.2 Pixel-level Noise-aware Adversarial Training
Our goal is to generate realistic noisy images. According to the noise model in Eq. (1), we split this problem into two sub-problems: (i) Image domain alignment aims to align Ietean [i]. (ii) Noise domain alignment targets at modeling the distribution of N To handle the sub-problems, PNGAN establishes a novel pixel-level noisc-aware ad training between G and D in Fig. (b).
3

---

Image Domain Alignment. A very naive strategy to construct both image and noise domain align- ment to directly minimize the distance of fn and I, However, due the intrinsic randomness, complexity, and irregularity of real noise, directly deploying Li loss between I fn and I,n is unrea- sonable and drastically damages the quality of fn- Besides, as analyzed Sec. 2.1, each pixel of In is distribution-unke unkknown random variable. This indicates that such naive strategy challenges the training and may easily cause the non-convergence issue. Therefore, the noise interference should be eliminated while constructing the image domain alignment. To this end, we feed L fn and I into Da to obtain their denoised versions and then perform Li loss between In and Lypi
HxW
HxW Σ (|Da(I,))B) - fetean [ill
- [|Da([ L=1
- Da Da(L.
(2)
By using Da, we can transfer I and I fn into a nearly noise-free solution space. The value of Lelean is relatively stable. Therefore, minimizing Lı can encourage G to favor solutions that after being denoised by Da converge to Lelean. In this way, the image domain alignment is constructed.
Noise Domain Alignment. Becasue of the complexity and variability of real noise, it's hard to completely seperate N from [ in Eq (1). Fortunately, we note that on the basis of constructing the image domain alignment of
CORD
LeakyReLU
Batch
Sigmoid
Jayer
Norm
Fake Seare Map
Fake Kolay
kind4s1
khn128c1
kch22ifet
kintet
Lelean[i] [i]. the noise domain alignment of N[ is equivalent to the distribution estimation of [, Additionally, as the real noise is signal- dependent. the alignment between I [i] and Inn [i] is more beneficial to capture the correla- tion between noise and scene. We denote the distribution of [, as Patta (x:z), some real noisy pixel samples of I,n [i] as {art, :?.,., .. such that a Panza (x;), and the distribution of I as Pa(æ;; 0g). Here Oa is the parameter of G. Then we formulate the noise domain aligment into a maximum likelihood estimation problem:
Red Score Map
Real Hoisy
I kernel size n- number of channels
s- s-stride
Figure 2: Architecture of discriminator.
HxW
0g arg MALE PC
Σ
Σ
logPa(x;; 0a) arg max E, Exg [1ogPo(af; 0a)] (3)
i=1
where E means taking the average value. To approach this upper bound as close possible, present and establish the pixel-level adversarial traininig between and D. The architecture
of D is shown in Fig. 2. D consists of 4 convolutional (conv) layers and utilizes LeakyReLU activation (a = 0.2). General discriminator treats a image as a sample and outputs a score indicating how realistic the image is. Instead, D is pixel-level classifier. D adopts the fake and real noisy images as input in a mini-batch and outputs a score map P E # for each image. Specifically,
the information of [ E [0, 1] is the probability value indicating how realistic Pa(a; 0G) is. G aims to generate more realistic y [i] to fool D while D) targets at distinguishing y from [,][i]. According to Eq. (3), we formulate the adversarial training between G and D min-max problem:
min max E, [EI.. [(o.( [ + EE, [E1,, [1og(1 D (b)(
(4)
0g 6p
where Er., and E1,, respectively represent the operation of taking the average for all fake and real data in the mini-batch. As analyzed in [45], make GANs analogous to divergence minimization and produce sensible predictions based on the a priori knowledge that half of the samples in the mini-batch are fake, we utilize the recently proposed relativistic discriminator [45] as follow:
( D( 0D) (C Dra fm) = (CD((I..) - (CD((
(5)
D 0p) (CDD(In))) DRa (I Imn) = (CDD(I,,) EI.. (CD((I))
where Daa denotes the relativistic discriminator, a means the Sigmoid activation, and Cp represents the non-transformed discriminator output. Dna estimates the probability that real data more realistic than fake data and also directs the generator to create a fake image that is more realistic than real images. The loss functions of D and G are then defined in symmetrical form:
LD EE, Et, [1og(D RR (Irn fn) [t]) EI,,, [1og(1 DRa (n [m)]]
(6)
LG -E, Er... [1og(1 - (Iam [+)) + Er,, [log( (D R. (1,, Ipm) [t]]

---

Syntheelt
SRG
SRG
Take Rowe
Map
GAP
Pooling
Average
f Suza Concatenate
Upssmpling
Dowssampling
X Multphhattion
(a) SMNet
MAB
FCA
FCA
SRG
FCA
4.2
GAP Frac
MAR
MAB
(b) Simple Residual Group
(c) Multi-scale Attention Block
(d) Fast Channel Attention
Figure 3: Details of the generator. (a) the architecture of SMNet. (b) depicts the components of
SRG. (c) shows the details of MAB. MAB is equipped with FCA, which is illustrated in (d).
During the training procedure, we fix D train G and fix G to train D iteratively, Minimizing Lo and LD alternately allows us to train a generative model G with the goal of fooling the pixel-level discriminator D that is trained to distinguish fake noisy images from real noisy images. This pixel- level awise- adversarial training scheme encourages G to favor perceptually natural solutions that reside on the manifold of real noisy images so to construct the noise domain alignment.
2.3 Noisy Image Generating
In Sec. 2.1, we denote the probability distribution of f= as Pa(x;; Now we customize a light- yet efficient CÑN architecture, SMNet as G to generate Po (zy; 0g). this section, we firstly introduce the input setting of G' and subsequently detail the architecture of SMNet.
Input Setting. We aim to generate realistic noisy image from its clean counterpart. A naive setting to directly adopt the clean image as the input to generate the noisy image. However, this naive setting is not in line with the fact. When we repeatedly feed the same clean image pre-trained G, G outputs completely the same noisy images. In contrast, when taking multiple pictures in the real world, the real noisy images vary a lot in the intensity of each pixel. This is caused by many factors (e.g., photon sensing noise, ISP pipelines, and illumination conditions). Hence, the naive input setting containing no distribution is unreasonable. We review that the general GANs sample from an initial random distribution (usually Gaussian) to generate fake image. Hence, the input of G should contain random distribution so as to generate multiple noisy images of the same scene. We note that the two common synthetic settings meet this condition. Therefore, we utilize the two common settings to produce the synthetic image and then adopt the synthetic image as the input of G. Subsequently, we propose a light-weight yet efficient architecture, SMNet for better real noise fitting.
SMNet Architecture. The architecture of SMNet is shown in Fig. (a). SMNet involves Simple Residual Groups (SRG) and each SRG contains n Multi-scale Attention Blocks (MAB). The synthetic input syn E IR ** x3 continuously undergoes a conv layer fi, SRGs, and conv layer fz, then adds with a long identity mapping for efficient residual learning to eventually generate the fake noisy counterpart fn R. HxWx5 This process can be formulated as:
1 Layn [2(S. s)), ES,1: = sF,,s,) Fs. = (I
where Sy denotes the Jen SRG, 1 S t-1 The components of SRG are depicted in Fig. 3 (b). We define the input feature of the joh. SRG as Fs, e #xxXCC and its channel as C. Fs, continuously undergoes conv layer, MABs, and conv layer to add with an identity mapping:
F F5 = Fs, + (F Ma
F M = ME(F Mp)
F MM = Fsy,
(8)
where Mc denotes the high MAB of the juh SRG, < 1: < 1. MAB is the basic building block and the most significant component SMNet. The details of MAB are depicted in Fig. 3 (c). We customize MAB with the following motivations: (i) Multi-scale feature fusion can increase the receptive field and multi-resolution contextual information can cover rich auto-correlation, which provides more sufficient spatial representations for noise fitting. (ii) The noise level decreases as the scale increases and nonlinear sampling operations can increase the richness of the mapping in the potential space of real noise. Therefore, we exploit parallel multi-resolution branch aggregation from top to bottom and bottom to top to facilitate the learning of complex real noise. (iii) Specifically, during the feature downsampling, general downsample operation damages the image information, resulting pixel discontinuity and jagged artifact. To alleviate these issues, we exploit Shift- Invariant Downsample [46] that copes with the discontinuity by using continuous pooling and
5

---

SIDD [47]
DND [48]
PolyU [40]
Nam [49]
Methods
PSNR 1
SSIM
SSIM
Methods
PSNR
SSIM
PSNR t
SSIM
DrCNN-B [10]
23,66
0.583
32.43
0.790
RDN [12]
37,94
0,946
38.16
0.956
CBDNet [50]
33.28
0.868
38.06
0.942
FFDNet+ [19]
38.17
0.951
38.81
0.957
RIDNet [44]
38.71
0.914
39.26
0.953
TWSC [5]]
38.68
0.958
38.96
0.962
AINDNet [52]
39.15
0.955
39.53
0,956
CBDNet
38.74
0,961
39.08
0.969
VDN [53]
39.23
0.955
39.38
0.952
RIDNet [44]
38.86
0.962
39.20
0,973
CycleISP [22]
39,52
0.957
39.56
0,956
VDN [53]
39.04
0.965
39.68
0,976
MPRNet [54]
39.71
0.958
39.80
0.954
MPRNet [54]
39.07
0.969
39.41
0.974
MIRNet [55]
39.72
0.959
39.88
0.956
MIRNet [55]
39.18
0.973
39.57
0.979
RIDNet* (Ours)
39,25
0.956
39.55
0.955
RIDNett (Ours)
39.54
0.971
39.69
0.975
MPRNet (Ours)
40,06
0.960
40.18
0.961
MPRNet (Ours)
40.48
0.982
40,72
0,984
MIRNet (Ours)
40.25
MIRNet (Ours)
40.55
0.986
Table 1: Comparison on four henchmarks. denotes denoisers finetuned with images generated by PNGAN.
filtering operation, preserving rich cross-correlation information between original and downsampled images. (iv) To efficiently capture continuous channel correlation and avoid information loss, we use the ID channel attention module, Fast Channel Attention (FCA) instead of the general 2D convolution attention module. The input feature, F asg E IR XXXXC is fed into three parallel multi-scale paths:
Fl Mr FCA(F ug))
F Mr S (FCAI MR))
F Mie
(up(FCC(fSaa (FM())) (9)
where FCA denotes Fast Channel Attention. Phip denotes conv layer after bilinear interpolation up- sampling, is the scale factor. füp is similarly defined. faid means Invarianv Downsample [46], 2 is also the scale factor. Sad is similarly defined. Subsequently, the output feature is derived by:
(F (R) Mgg
Mg f(FF)
F
Mr Fr,
(10)
My,
where / represents the last cony layer, [' denotes the concatenating operation. The architecture of FCA is shown in Fig. 3 (d). We define the input feature as Fa, then FCA can be formulated as:
FCA(F2) = Fa a1+o(fiDch (L+o((D((AAP((E))
(11)
where a represents the Sigmoid activation function, GAP means global average pooling along the spatial wise, fine denotes I-Dimension Convolution. In this work, we set t = 3. 2, and C' 64.
2.4 Overall Training Objective
In addition to the aforementioned losses, we employ perceptual loss function that assesses a solution with respect to perceptually relevant characteristics (e.g.. the structural contents and detailed textures):
Cp ||VGG([L,) VGG(L,))ll: Ifa Da(Tyn), I = Da(I,.),
(12)
where VGG denotes the last feature map of VGG16 [56]. Eventually, the training objective is:
L = L1 + >p Lp + Ana (ED + LG),
(13)
where Ay and ARa are two hyper- hyper-parameters controlling the importance balance. The proposed PNGAN framework is end-to-end trained by minimizing L. Note that the parameters in Da and VGG16 are fixed. Each mini-batch training procedure is divided into two steps: (i) Fix D and train G. (ii) Fix G and train D. This pixel-level adversarial training scheme promotes D the ability to distinguish fake noisy images from real noisy images and allows G to learn to create the solutions that are highly similar to real camera noisy images and thus difficult to be classified by D.
3 Experiment
3.1 Experiment Setup
Datasets. We first use SIDD [47] train set to train Da. Then we fix Da to train G on the same set. Subsequently, G uses clean images from DIV2K [57], Flickr2K [58], BSD68 [59], Kodak24 [60]. and Urban 100 [61] to generate realistic noisy-clean image pairs. We use the generated data and SIDD train set jointly to finetune real denoisers and evaluate them on four real denoising benchmarks: SIDD [47], DND [48], PolyU [40], and Nam [49]. The images in SIDD [47] are collected using five smartphone cameras in 10 static scenes. There are 320 image pairs for training and 1,280 image patch pairs for validation. DND [48] composes 50 noisy-clean image pairs captured by 4 consumer

---

SITMEE MEETNE
SITEEC SER SCTPOHT MISINAEYROOO
SXEMCC SAT
cemting
orttngg PA.
sartingg APRICAN
660
60
SIDD
DND
PolyU
Nam
Figure 4: Domain discrepancy comparisons. We use the metrie, Maximum Mean Discrepancy (MMD) to measure the domain discrepancy between synthetic and real noisy datasets, PNGAN generating and real noisy datasets. Under both settingl and 2, the discrepancy decreases significantly when PNGAN is applied.
SIDD [47) $2
DF2K
[8)]
Methods
S1
PNGAN
S2+PNGAN
Real
S1
S1 + PNGAN
S2
S2+PNGAN
RIDNet
22.55
37.92
(+15.37)
36.13
38.71
(+2.58)
38.69
22.55
32.10
(+9.55)
33.98
38.14
(+4.16)
MPRNet
22.86
38.52
(+15.66)
36,52
39.53
(+3.01)
39.45
22.85
32.82
(49.97)
34.19
38.61
(+4.42)
MIRNet
22.83
38.76
(+15,93)
36.55
39.57
(+3.02)
39.58
23.08
32.34
(+9.26)
34.26
38.72 (+4.46)
Table 2: Training denoisers with different data from scratch. PSNR is reported. S1,2 = syntheicc setting 1,2.
cameras. 1,000 patches at size 512x512 are cropped from the collected images. PolyU [40] consists of 40 real camera noisy images. Nam [49] is composed of real noisy images of 11 static scenes.
Implementation Details. We set the hyper-parameter X, = 6x 10-3, Ana 8 10-4, For synthetic settingl, we set the noise intensity, on = 50. For synthetic setting2, we directly exploit CycleISP to generate the synthetic noisy input. All the sub-modules (Da. G. and D) are trained with the Adam [62] optimizer (31 = 0.9 and 31 = 0.9999) for 7 10 iterations. The initial learning rate is set to 2x 10 4, The cosine annealing strategy [63] is employed to steadily decrease the learning rate from the initial value to 10 during the training procedure. Patches at size 128x 128 cropped from training images are fed into the models. The batch size is set as 8. The horizontal and vertical flips are performed for data augmentation. All the models are trained on RTX8000 GPUs. In the finetuning phase, the learning rate is set xx10-9 other settings remain unchanged.
3.2 Quantitative Results
Domain Discrepancy Validation. We use the widely applied metric, Maximum Mcan Discrepancy (MMD) [64] to measure the domain discrepancy between synthetic and real-world noisy images, PNGAN generating. and real noisy images on four real noisy benchmarks. For DND, we derive a pseudo clean version by denoising the real noisy counterparts with a pre-trained MIRNet [55]. Then we use the pseudo clean version to synthesize noisy images. The results are depicted as a histogram in Fig. 4. For settingl. I, the domain discrepancy decreases by 74%, 75%, 44%, and 43% on SIDD, DND, PolyU, and Nam when PNGAN is exploited. For setting2, the discrepancy decreases by 64%, 67%, 46%, and 44%. These results demonstrate that PNGAN can narrow the discrepancy between synthetic and real noisy datasets. Please refer to the supplementary for detailed calculation process.
Comparison with SOTA Methods. We use the generated noisy-clean image pairs (setting2) to finetune a series of denoisers. We compare our models with SOTA algorithms on four real denoising datasets: SIDD, DND, PolyU, and Nam. The results are reported in Tab. 1. * denotes denoisers finetuned with image pairs generated by PNGAN. We have the following observations: (i) Our denoisers outperform SOTA methods by a large margin. Specifically. MPRNet R and MIRNet* exceed the recent best method MIRNet by 0.34 and 0.35 dB on SIDD, 0.30 and 0.37 dB on DND. RIDNet MPRNet*, and MIRNet* surpass the best performers by 0.36. 1.30, and 1.37 dB on PolyU and 0.01, 1.04, and 1. 10 dB on Nam. (ii) Compared with the counterparts that are not finetuned, our models achieve a significant promotion. In particular, RIDNet is 0.54, 0.29, 0.68, and 0.49 dB higher than RIDNet on SIDD, DND, PolyU, and Nam. MPRNet achieves 0.35, 0.38, 1.41, and 1.31 dB gain than MPRNet on SIDD, DND, PolyU, and Nam. MIRNet is improved by 0.35, 0.37, 1.37, and 1.21 dB. This evidence clearly suggests the high similarity between PNGAN generating and real noisy images. Denoisers adapted with our fake image pairs generalize better across different benchmarks.
Train from Scratch. For more strong comparisons, we use the fake noisy images generated from clean SIDD train and DF2K Flikerk) respectively to train denoisers from scratch. The PSNR results evaluated on SIDD test are listed in Tab. 2. All models are trained with the same experiment schedule except the training data. can be observed: (i) On SIDD train, when PNGAN is applied to settingl, 1, denoisers are promoted by 15.65 dB and only ~ 0.84 dB lower than those
7

---

Clean
Settingl
Setting?
Setting?
Figure 5: Visual comparisons of noisy images on SIDD, DND, PolyU, and Nam. Please zoom in.
PNGAN Component
Generator Architecture S1D
Mathods
Baseline1
D2
D
Cp
Boseline2
Multi-scale
RIDNet
14.54
35.37
(+20.83)
37.49
(+2.12)
37,92
(+0.43)
35.62
37.01
(+1.39)
37.23 (+0.22)
37,92 (10.99)
MPRNet
14.25
36.26
(122.01)
38.27 (+2.01)
38.52
(40.25)
36.28
37.47
(+1.19)
37.86 (+0.39)
38.52 (10.6))
MIRNet
13:57
36.15
(+22.58)
38.28 (+2.13)
38.76 (40.48)
36.39
37.66 (+1.27)
37.89 (+0.23)
38.76 (1087)
Table 3: Ablation study of PNGAN component and the noise generator architecture. PSNR is reported.
trained with real data (SIDD train set). While applying PNGAN to setting2 (CycleISP), denoisers are improved by ~ 2.87 dB. Surprisingly, in this case, denoisers achieve almost the same performance as those trained with real data. The relative error is 0.2%. (ii) To validate the generality of PNGAN, also adopt synthetic DF2K noisy-clean image pairs train denoisers. As shown in the right part of Tab. 2, when PNGAN is applied to setting I, denoisers are promoted by ~ 9.59 dB. While applying PNGAN to setting2, denoisers are improved by 4.35 dB and only 0.75 dB lower than those trained with SIDD real train set. These results convincingly demonstrate: (i) The generated noise is highly similar to the real noise especially when PNGAN is applied to synthetic setting2. (ii) PNGAN can significantly narrow the domain discrepancy between synthetic and real-world noise.
3.3 Qualitative Results
Visual Examinations of Noisy Images. To intuitively evaluate the generated noisy images, we provide visual comparisons of noisy images on the four real noisy datasets, shown in Fig. 5. Note that the clean image of DND is pseudo, denoised from its noisy version by MIRNet. The left part depicts noisy images from SIDD, DND, PolyU, and Nam (top to down). The right part exhibits the patches cropped by the yellow bboxes, from left to right: clean, synthetic settingl, setting? (CycleISP), PNGAN generating, and real noisy images. can be seen from the zoom-in patches: (i) Noisy images synthesized by settingl is signal-independent. The distribution and intensity remain unchanged across diverse scenes, indicating the characteristics AWGN fundamentally differ from those of the real noise. (ii) Noisy images generated by PNGAN are closer to the real noise than those synthesized by setting2 visually. Noise synthesized by setting2 shows randomness that is obviously inconsistent with the real noise in terms of intensity and distribution. While PNGAN can model upatio-chromatice through the same camera pipeline, different shooting conditions lead to the diversity of real noise. It's unreasonable for the noise synthesized by CycleISP to show nearly uniform fitting to different input images. In contrast, PNGAN adaptively simulate This adaptability allows PNGAN to show robust performance across different real noisy datasets.
correlated and non- non-Gaussian noise more accurately. (iii) Even if passing
sophisticated and rhalsstealistic models.
Visual Comparison of Denoised Images. We compare the visual results of denoisers before and after being finetuned (denoted with *) with the generated data in Fig. 4. We observe that models finetuned with the generated data are more effective in real noise removal. Furthermore, they are capable of preserving the structural content, textural details, and spatial smoothness of the homogeneous regions. In contrast, original models either yield over-smooth images sacrificing fine textural details and structural content or introduce redundant blotchy texture and chroma artifacts.
3.4 Ablation Study
Break-down Ablations. We perform break-down ablations to evaluate the effects of PNGAN components and SMNet architecture. We select setting synthesize the noisy input from SIDD train set. Then we use the generated data only to train the denoisers from scratch and evaluate them on SIDD test. The PSNR results are reported in Tab. (i) Firstly, ,, set as SMNet to validate the effects of PNGAN components. We start from Basclinel, no discriminator is used and the C₁ loss is

---

PolyU
SIDD
DND
KRU FRO
Gins
Beyontl
0.911 Naley
31.8 0.361 RIONat
3344 8.675
ILM4
6,438
34,90
Naky
MPRNN
Nets
MIRNa
RIDNeI
RIDNe
NO
Beyondl
of Beyond
CHI
CH
themaaie
08.75 RID
PSNR SSIM GroundTrath
40.21 /0.994 MPRN
PSNR SSIM
3L09 8367
PSNR
ILMAN MPNNN
5442 BLRNK
Ground
MIRN
Grauddraath
MPRNN
Figure 6: Visual results of denoisers before and after being finetuned with fake data. Please zoom in.
Parametrr
Analriis
SIDD (47)
PulyU
(40) SSIM
Nam [19] SSIM
5SIM
PSNR
SSIM
Non:
58.71
0.9914
38.86
01562
39:20
0.973
38.76
0.930
39.32
019557
38.01
(.94)
38.34
01958
38.92
0.955
20%
39:29
0.957
38:45
0.959
38.87
0.970
39.00
0.958
40%
59.72
0.956
39
0.566
39.26
0.973
39.20
0.950
60%
39.26
0.956
39.54
0,971
39.69
1,975
39.35
0.961
80%
39.23
0.955
39.56
0.972
39.72
0.976
29.33
0.960
10078
39.21
0.955
39.57
0.972
39.73
0.976
39.33
0,950
Figure 7: Parameter analysis of Ap. ARs and on-
Table 4: Analysis of the finetuning data ratio g.
directly performed between In and Irn in Eq. (2). Denoisers trained with the generated data collapse dramatically, implying the naive strategy mentioned in Sec. 2.2 is unfeasible. When D& is applied, the denoisers are promoted by 21.81 dB on average. In addition, the PSNR and SSIM between the denoised counterparts of generated and real noisy images are 39.14 14 dB and 0.928 on average respectively. This evidence indicates that Da successfully conducts the image domain alignment as mentioned in Sec. 2.2. Subsequently, we use image-level D with stride conv layers to classify whether the whole generated image is real. Nonetheless, the performance of denoisers remains almost unchanged. After deploying D, the models are improved by 2.09 dB, suggesting that the pixel-level noise model is more in line with real noise scenes and benefits generating more realistic noisy images. When Cp is used, the denoisers gain slight improvement by about 0.39 dB, indicating Lp facilitates yielding more vivid results. (ii) Secondly, we only change the architecture of G to study the effects of its components. We start from Baseline2 that doesn't 't exploit multi-scale feature fusion, SID, and FCA. When we add two different scale branches and use bilinear interpolation to downsample and upsample, denoisers trained with the generated images are promoted by about 1.28 dB. After applying SID and FCA, the denoisers further gain 0.28 and 0.74 dB improvement on average. These results convincingly demonstrate the superiority of the proposed SMNet in real-world noise fitting.
Parameter Analysis. We adopt RIDNet as the baseline to perform parameter analysis. We firstly validate the effects of Ap ARa in Eq. (13), and the noise intensity of setting i.c., on. We change the parameters, train G, use G to generate realistic noisy images from clean images of SIDD train set, train RIDNet with the generated data, and evaluate its performance analyzing one parameter, we fix the others at their optimal values. The PSNR results are shown in Fig. 7. The optimal setting is A 6x 10-3, ARa = 8 10-4, and on 40 or 50. Secondly, evaluate the effect of the ratio of finetuning data. We denote the ratio of extended training data (setting2) to SIDD real noisy training data q. We change the value of q. finetuned the original RIDNet, and test on three real denoising datasets: SIDD, PolyU, and Nam. The results are listed in Tab. 4. When q = 0, all the finetuning data comes from SIDD train set, RIDNet achieves the best performance on SIDD. However, its performance on PolyU and Nam degrades drastically due to the domain discrepancy between different real noisy datasets. We gradually increase the value of q to study its effects. The average performance on the three datasets yields the maximum when g 60%.
SIDD test set. When
4 Conclusion
Too much research focuses on designing a CNN architecture for real noise removal. In contrast, this work investigates how to generate more realistic noisy images so as to boom the denoising performance. We first formulate noise model that treats each noisy pixel as random variable. Then we propose a novel framework PNGAN to perform the image and noise domain alignment. For better noise fitting, we customize efficient architecture, SMNet as the generator. Experiments show that noise generated by PNGAN is highly similar to real noise terms of intensity and distribution. Denoisers finetuned with the generated data outperform SOTA methods on real denoising datasets.

---

Acknowledgement
This work is jointly supported by the NSFC fund (61831014), in part by the Shenzhen Sci- ence and Technology Project under Grant (ZDYBH20190000000,, JCYI20018000811502220000 CJG1ZD202006117022200001)4
References
[]] C. M. Bishop, "Pattern recognition and machine learning," in springer, 2006.
[2] S. Gu, L. Zhang, W. Zuo, and X. Feng, "Weighted nuclear norm minimization with application to image denoising," " in CVPR, 2014."
[3] Zhu, G. Chen, and P.-A. Heng, "From noise modeling blind image denoising," in CVPR, 2016.
[4] J. Xu, L. Zhang, and D. Zhang, "A trilateral weighted sparse coding scheme for real-world image denoising," " in ECCV, 2018.
[5] M. Aharon, M. Elad, and A. Bruckstein, "K-svd: An algorithm for designing overcomplete dictionaries for sparse representation" " TSP, 2006. 1
[6] A. Buades, Coll, and J. Morel, "A non-local algorithm for image denoising." in CVPR, 2005.
[7] M. Maggioni, V. Katkovnik, K. Egiazarian, and A. Foi, "Nonlocal transform-domain filter for volumetric data denoising and reconstruction," TIP, 2013. 1
[8] H. C. Burger, C. Schuler, and S. Harmeling, "Image denoising: Can plain neural networks compete with bm3d?," in CVPR, 2012.
[9] S. Lefkimmiatis, "Non-local color image denoising with convolutional neural networks," in CVPR, 2017. 1
[10] K. Zhang, W. Zuo, Y. Chen, D. Meng, and L. Zhang, "Beyond gaussian denoiser: Residual learning of deep cnn for image denoising," TIP, 2017. 1,6
[11] K. Zhang, W. Zuo, Gu, and L. Zhang, "Learning deep enn denoiser prior for image restoration." in CVPR, 2017.
[12] Y. Zhang, Y. Tian, Y. Kong, B. Zhong, and Y. Fu, "Residual dense network for image restoration," TPAMI, 2020. 6
[13] Y. Cai, Z. Wang, Z. Luo, B. Yin, A. Du H. Wang, X. Zhou, E. Zhou, X. Zhang. and J. Sun,
"Learning delicate local representations for multi-person pose estimation," " in ECCV, 2020.
[14] Z. Luo, Z. Wang. Y. Cai, Wang, Wang, Y. Huang, E. Zhou, Tan, and Sun, "Efficient human pose estimation by learning deeply aggregated representations," in ICME, 2021. 1
[15] Y. Chen, W. Yu, and T. Pock, "On learning optimized reaction diffusion processes for effective
image restoration,"" " in CVPR, 2015. 1
[16) K. Dabov, A. Foi, V. Katkovnik, and K. Egiazarian, "Image denoising by sparse 3-d transform- domain collaborative filtering," TIP, 2007. 1
[17] X. Hu, Y. Cai. Z. Liu, H. Wang. and Y. Zhang, "Multi Mcule selective feedback network with dual loss for real image denoising," in IJCAI, 2021.
[18] h. Xiaowan, M. Ruijun, I.. Zhihong, C. Yuanhao, 7. Xiaole, 7. Yulun, and W. Haoqian, "Pseudo 3d auto-correlation network for real image denoising," CVPR, 2021.
[19] K. Zhang, W. Zuo, and L. Zhang, "Ffdnet: Toward fast and flexible solution for enn-based image denoising," TIP, 2018. 1,
[20] K. Ma, Z. Duanmu, Q. Wu, Z. Wang, H. Yong, H. Li, and L.. Zhang., "Waterloo exploration database: New challenges for image quality assessment models," TIP, 2014.
[21] A. Foi. M. Trimeche, V. Katkovnik. and K. O. Egiazarian, "Practical poissonian poissonian-gaussian gaussiin noise modeling and fitting for single-image raw-data," TIP, 2008. 1,
[22] S. W. Zamir, Arora, Khan, M. Hayat, F. S. Khan, M.-H. Yang, and Shao, "Cycleisp:
Real image restoration via improved data synthesis," in CVPR, 2020. 2, 6
10

---

[23] Y. Hwang, J.-S. Kim, and I.-S. Kweon, "Difference-basee image noise modeling using skellam distribution," TPAMI, 2012. 2
[24] I. Goodfellow, J. Pouget-Abadie, M. Mirza, B. Xu, D. Warde-Farley, S. Ozair, A. Courville, and Y. Bengio, "Generative udversarial nets," in NeurIPS, 2014. 2
[25] X. Gong, S. Chang, Y. Jiang, and Z. Wang, "Autogan: Neural architecture search for generative adversarial networks," in ICCV, 2019. 2
[26] P. Isola, J.-Y. Zhu, T. Zhou, and A. A. Efros, "Image-to-image translation with conditional
adversarial networks," " in CVPR, 2017.
[27] J.-Y. Zhu, T. Park, P. Isola, and A. Efros, "Unpaired image-to-image translation using cycle-consistent adversarial networkss," in ICCV, 2017. 2
[28] C. Ledig, L. Theis, F. Huszar, J. Caballero, A. Cunningham, A. Acosta, A. Aitken, A. Tejani,
J. Totz, Z. Wang, and W. Shi, "Photo-realistic single image super-resolution using generative
adversarial network," " in CVPR, 2017. 2
[29] X. Wang, K. Yu, S. Wu, J. Gu, Y. Liu, C. Dong, Y. Qiao, and C. C. Loy, "Esrgan: Enhanced super-resolution generative adversarial networks," in ECCVW, 2018. 2
[30] X. Hu, H. Wang, Y. Cai, X. Zhao, and Y. Zhang. "Pyramid orthogonal attention network based on dual self-similarity for accurate mr image super-resolution, in ICME, 2021.
[31] C. Li and M. Wand, "Combining markov random fields and convolutional neural networks for image synthesis," in CVPR, 2016. 2
[32] Y. Jiang, X. Gong, D. Liu, Y. Cheng, C. Fang, X. Shen, J. Yang, P. Zhou, and Z. Wang, "Enlightengan: Deep light enhancement without paired supervision." " T7P, 2021.
[33] W. Yang. S. Wang, Y. Fang, Y. Wang, and J. Liu, "From fidelity to perceptual quality: A semi-supervised approach for low-light image enhancement," in CVPR, 2020.
[34] R. Qian, R. T. Tan, W. Yang, J. Su, and J. Liu, "Attentive generative adversarial network for raindrop removal from single image," in CVPR, 2018. 2
[35] R. Li, J. Pan, Z. Li, and J. Tang. "Single image dehazing via conditional generative adversarial network," in CVPR, 2018. 2
[36] C. Zhennn T.-J. Cham, and J. Cai, "Pluralistic image completion," in CVPR, 2019.
[37] Yu, Z. Lin, J. Yang. X. Shen, X. Lu, and T. S. Huang. "Generative image inpainting with contextual attention," in CVPR, 2018. 2
[38] S. Yang, Z. Wang, Z. Wang, N. Xu, J. Liu, and 7. Guo, "Controllable artistic text style transfer via shape-matching gan," in ICCV, 2019. 2
[39] S. Yang. Z. Wang. J. Liu, and Z. Guo, "Deep plastic surgery: Robust and controllable image editing with human-drawn sketches," in ECCV, 2020.
[40] Y. Yuan, S. Liu, J. Zhang, Y. Zhang, C. Dong, and L.. Lin, "Unsupervised image super-resolution using cycle-in-cycle generative adversarial networks," in CVPRW, 2018. 2, 6, 7,9
[41] Y.-S. Chen, Y.-. Wang. M.-H. Kao, and Y.-Y. Chuang, "Deep photo enhancer: Unpaired learning for image enhancement from photographs with gans," " in CVPR, 2018. 2
[42] 7. Yue, Q Zhao, I.. Zhang, and D. Meng, "Dual adversarial network: Toward real-world noise removal and noise generation," in ECCV, 2020.
[43] Chen, J. Chen, H. Chao, and M. Yang. "Image blind denoising with generative adversarial network based noise modeling." in CVPR, 2018. 2
[44] S. Anwar and N. Barnes, "Real image denoising with feature attention," in ICCV, 2019. 3, 6
[45] A. Jolicocur-Martincau, "The relativistic discriminator: a key element missing from standard gan," in ICLR, 2019. 4
[46] R. Zhang, "Making convolutional networks shift-invariant again," in ICML., 2019. 5, 6
[47] A. Abdelhamed. S. Lin, and M. S. Brown, "A high-quality denoising dataset for smartphone cameras," " in CVPR, 2018 6.
[48] Plotz and S. Roth, "Benchmarking denoising algorithms with real photographs," in CVPR, 2017. 6
11

---

[49] S. Nam, Y. Hwang. Y. Matsushita, and S. J. Kim, "A holistic approach to cross-channel image noise modeling and its application image denoising," in CVPR, 2016. 6, 7,
[50] S. Guo, Yan, K. Zhang, W. Zuo, and L. Zhang, "Toward convolutional blind denoising of real photographs," in CVPR, 2019. 6
[51] Xu, L. Zhang, and D. Zhang, "A trilateral weighted sparse coding scheme for real-world
image denoising," in ECCV, 2018. 6
[52] Y. Kim, J. W. Soh, G. Y. Park, and N. I. Cho, "Transfer learning from synthetic to real-noise denoising with adaptive instance normalization," in CVPR, 2020. 6
[53] Z. Yue, H. Yong. Q Zhao, D. Meng. and L. Zhang, "Variational denoising network: Toward blind noise modeling and removal," in NeurlPS, 2019. 6
[54] S. W. Zamir, A. Arora, S. Khan, M. Hayat, F. Khan, M.-H. Yang, and L. Shao, "Multi-stage progressive image restoration," in CVPR, 2021.
[55] S. W. Zamir, A. Arora, S. Khan, M. Hayat, F. S. Khan, M.-H. Yang, and L. Shao, "Learning
enriched features for real image restoration and enhancement," in ECCV, 2020. 6,
[56] A. Z. Karen Simonyan, "Very deep convolutional networks for large-scale image recognition."
in ICLR, 2015.
[57] R. Timofte, E. Agustsson, L. Van Gool, M.-H. Yang, and L. Zhang, "Ntire 2017 challenge on
single image super-resolution: Methods and results," in CVPRW, 2017. 6,
[58] B. Lim, S. Son, H. Kim, S. Nah, and K. Mu Lee, "Enhanced deep residual networks for single image super-resolution," " in CVPRW, 2017. 6,7
[59] S. Roth and M. Black, "Fields of experts: A framework for learning image priors," in CVPR, 2015. 6
[60] O. Russakovsky, J. Deng, H. Su, Krause, S. Satheesh, Ma, Z. Huang, A. Karpathy,
A. Khosla, and Michael Bernstein, "Imagenet large scale visual recognition challenge," in IJCV, 2015. 6
[61] J.-B. Huang, A. Singh, and N. Ahuja, "Single image super-resolution from transformed self- exemplars," in CVPR, 2015. 6
[62]
D. P. Kingma and J. L. Ba, "Adam: A method for stochastic optimization," in ICLR, 2015. 7 I. Loshchilov and F. Hutter, "Sgdr: Stochastic gradient descent with warm restarts," in ICLR,
[63]
2017. 7
[64] A. Gretton, K. M. Borgwardt, M. J. Rasch, B. Schölkopf, and A. Smola, "A kernel two-sample test," in JMLR, 2012.
12


Reviews:
[{"review_id": "lSUYWmO7cCA", "reviewer": "Reviewer_oDMH", "summary": "In this paper, the authors proposed a novel way to generate real-world noise, and presented the reliability of generating noise data sets on a large number of existing methods. Overall, the lack of real noisy images limits the performance of the fully supervised denoising network, and the proposed method contributes a reasonable solution.  ", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The authors propose a simple but novel idea (PNGAN) for real image denoising. In my opinion, the motivation is reasonable by treating each noisy pixel as a random variable and then splitting noisy image generation into image and noise domain alignment. Compared with the previously proposed image-level GAN, the novel pixel-level PNGAN can obtain better predictions through the score map. The pixel-level adversarial training encourages the generator to pursue better solutions. In the experiments, ablation studies demonstrate the effects of PNGAN components, such as the proposed discriminator, pixel-level vs. image-level, components in the generator. Meanwhile, compared to existing works, the proposed PNGAN also achieved promising results. \n\nHowever, I still have the following concerns:\n\n(i) The novelty of the generator in Figure 3 is limited. Multi-scale attention and stacking of residual blocks are common components. But the use of fast 1D convolution is innovative.\n\n(ii)It seems that Gaussian Poisson noise is better than Gaussian noise. The argument in favor of the used initial noise distribution should be provided.\n\n(iii) What I am concerned about is whether the proposed PNGAN has good generalization?\n(iv) The font in Figure 4 is too small, you can use abbreviations instead.\n(v) In Fig 1 (a2), when the metadata of the image is unavailable, how does the PNGAN create noisy images using Setting2?\n(vi) In Line 321-324 of the main paper, the authors claim that the image domain alignment is successfully conducted. It would be more convincing if more results are provided.\n\n\n\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "I think this method may have certain limitations in generalization, such as how to deal with unlearned real noise patterns in real-world applications.", "review_text": "", "extra_scores": {"ethical_concerns": "None", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "8"}, "extra_text": {}}, {"review_id": "aC0MMI-7ta", "reviewer": "Reviewer_7ouL", "summary": "This paper proposes a new method to simulate the real-world noisy images, which seems further improve the performance of current deep denoiser. A slight and effective generator is elaborately designed, and may facilitates the research of noise generation.\nBesides, the proposed method combines lots of existing deep learning technologies, and surpasses the current SoTA by a large margin.", "questions": "", "limitations": "", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "In summary, this work presents one noisy image generation method, and validates its effectiveness by comprehensive experiments to some extent.\n\n1) As fas as I know, this is the first work to employ the pre-trained deep denoiser $D_d$ as a regularizer to constrain the generated noisy images. It is interesting, but not a general regularizer. In other words, I think such strategy will pull the generated noisy image distribution to that of the specific training data used to train $D_d$, which may not be desired by us in some cases.\n\n2) The experiments only involve the comparison with other methods on the denoising task. For the noise generation task, this work does not cite and compare with some important works, such as:\n\n     \"Unprocessing Images for Learned Raw Denoising\", CVPR 2019.\n\n     \"Dual Adversarial Network: Toward Real-World Noise Removal and Noise Generation\", ECCV, 2020\n\n3) In Table 4, the generator is trained on SIDD training dataset. After finetuning jointly on the SIDD training and the generated dataset, the performance on SIDD testing dataset has a little drop. So, the trained generator does not exactly capture the noise distribution of the training dataset (SIDD), even though with complex loss function?  \n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The paper has addressed the limitations and potential negative societal impact in the supplementary material.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4"}, "extra_text": {}}, {"review_id": "UjeNttlzqFj", "reviewer": "Reviewer_NrP3", "summary": "The authors propose a pixel-level noise-aware generative adversarial network (PNGAN). They improve the denoising performance by generating more realistic noisy images.  Quantitative experiments prove that the performance of the existing denoising model can be significantly improved after being trained with the synthetic realistic noisy images.", "questions": "", "limitations": "", "rating": 8, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "+ This work has many advantages, such as:\n\nThe authors solve certain limitations in the current real denoising task. For example, the domain difference between synthetic noise and real noise is alleviated. As a better noise data enhancement method, PNGAN reduces the difficulty of obtaining real noise image pairs.\n\nThe proposed method uses GAN to generate pixel-level synthetic noise, which is consistent with real-world noise. It can be used for effective training data augmentation.\n\nVisually, PNGAN produces results that are closer to real noise. The quantitative experiments in Table 1 also show that the generated fake noisy images can be directly used for the training of the existing network, and the performance is improved by a large margin.\n\nThe paper is well-organized and clearly written, and the detailed source code is provided.\n\n-\tHowever, I still have the following concerns:\n\nThere is no quantitative evaluation of model parameters and interface speed in the paper.\n\nFor disordered and random noise, how to achieve Noise Domain Alignment in line140~175?\n\nWhat are the differences and advantages of Pixel-level Noise Modelling and Image-level Noise Modeling?\n\nFigure 4 shows that different data sets perform differently in domain discrepancy. Please analyze the reasons.\n\nFinally, the method PNGAN is novel and its effectiveness is demonstrated by extensive ablation study and main comparisons. The authors can further improve the paper by trying to address the concerns.\n\n-------------------------------------------------\nAll my concerns are well addressed in the rebuttal, and I would like to keep my initial rating.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "There are some limitations that almost all deep learning methods will have, including the input dependence and performance limitations. The detailed limitations of the PNGAN are explained in the supplementary. This work has no limitations and potential negative social impact.", "review_text": "", "extra_scores": {"ethical_concerns": "No.", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "16"}, "extra_text": {}}, {"review_id": "E2x4l-qjgky", "reviewer": "Reviewer_HLgB", "summary": "This paper proposed a method to generate fake noisy-clean image pairs, which are expected to well simulate the real training pairs, in an adversarial training fashion, in order to augment the training set for a deep denoiser.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The idea of adopt the adversarial training strategy to generate more training data is reasonable, and has been shown to be effective in previous studies. This paper has done good job in making this strategy also work in denoising problem, and shown promising results in experiments. The paper is also well-written and easy to follow. \n\nThe major problem of this work is that it misses discussion and comparison with a related study that addressed the similar problem. Specifically, Yue et al. proposed in [1] a dual adversarial training framework, which also aims to generate noisy-clean image pairs to augment the training set for real image denoising. The authors should discuss and empirically compare with this method. Besides, Yue et al. have also proposed two metrics to assess the performance of a real noisy image generator, and it is better to also consider these metrics in this work.\n\nA minor problem: $\\hat{I}_{clean}$  in the first equation of (1) should be  without \"^\" sign.\n\n[1] Zongsheng Yue, Qian Zhao, Lei Zhang and Deyu Meng. Dual adversarial network: Toward real noise removal and noise generation. ECCV, 2020.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The authors did not mention the limitations of their method. However, in my opinion, the major limitation is that it can not guarantee that the generated noisy images follow the same distribution of the real noisy images, although I know it is a very difficult problem. ", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "8 hours."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposes a novel idea to synthesize real world noisy images for image denoising.
The main idea is to treat each noisy pixel as a random variable and then split noisy image generation into image domain alignment and noise domain alignment. Based on the idea, a GAN method is proposed to perform the pixel level adversarial training. The idea is interesting and novel, and the effectiveness of the proposed method is demonstrated in the experiments by comparing it with SOTA methods and detailed ablation studies. All the reviewers are also impressed by the idea and the promising experimental results. However, a major drawback of the work is the lack of theoretical guarantee on the distribution gap between the real noisy images and the generated noisy images. Given that this is the first work that moves towards real world noisy image generation,  I recommend acceptance given the novelty and the practical significance of the proposed work.

Decision: Accept (Poster)