sRGB Real Noise Modeling via Noise-Aware Sampling with Normalizing Flows
Authors: Dongjin Kim
Donggoo Jung
Sungyong Baik
Tae Hyun Kim
Venue: ICLR 2024 poster (2024)
Keywords: sRGB real noise modeling
Normalizing flow
Low-level vision
Primary area: representation learning for computer vision, audio, language, and other modalities

Abstract: Noise poses a widespread challenge in signal processing, particularly when it comes to denoising images. Although convolutional neural networks (CNNs) have exhibited remarkable success in this field, they are predicated upon the belief that noise follows established distributions, which restricts their practicality when dealing with real-world noise. To overcome this limitation, several efforts have been taken to collect noisy image datasets from the real world. Generative methods, employing techniques such as generative adversarial networks (GANs) and normalizing flows (NFs), have emerged as a solution for generating realistic noisy images. Recent works model noise using camera metadata, however requiring metadata even for sampling phase. In contrast, in this work, we aim to estimate the underlying camera settings, enabling us to improve noise modeling and generate diverse noise distributions. To this end, we introduce a new NF framework that allows us to both classify noise based on camera settings and generate various noisy images. Through experimental results, our model demonstrates exceptional noise quality and leads in denoising performance on benchmark datasets.

Published as a conference paper at ICLR 2024
SRGB REAL NOISE MODELING VIA NOISE-AWARE SAMPLING WITH NORMALIZING FLOWS
Dongjin Kim"!, Donggoo Jung"? Sungyong Baik), Tae Hyun Kim
Dept. of Computer Science', Dept. of Artificial Intelligence", Dept. of Data Science
Hanyang University {dong j inkkim, dgjung, dsybaik, Seehyunkim/Jhanyaage
ABSTRACT
Noise poses a widespread challenge in signal processing, particularly when it comes to denoising images. Although convolutional neural networks (CNNs) have exhibited remarkable success in this field, they are predicated upon the belief that noise follows established distributions, which restricts their practicality when dealing with real-world noise. To overcome this limitation, several efforts have been taken to collect noisy image datasets from the real world. Generative meth- ods, employing techniques such generative adversarial networks (GANs) and normalizing flows (NFs), have emerged as a solution for generating realistic noisy images. Recent works model noise using camera metadata, however requiring metadata even for sampling phase. In contrast, in this work, we aim to estimate the underlying camera settings, enabling us to improve noise modeling and gen- erate diverse noise distributions. To this end, we introduce a new NF framework that allows us both classify noise based on camera settings and generate various noisy images. Through experimental results, our model demonstrates exceptional noise quality and leads denoising performance on benchmark datasets.
INTRODUCTION
Noise is one of the most common and undesired artifacts in the field of signal processing. There- fore, denoising has been studied for many decades across many research areas, among which image denoising is task to restore clean image by removing noise from the input image. Recently, there have been dramatic improvements in image denoising through the development neural networks (CNN) (Zhang et al., 2017; Anwar & Barnes, 2019; Chen et al., 2021) trained in supevised manner. Despite their impressive achievements, these supervision-baase simple assumption that added noise follows known distri-
convolutional
algorithms have
critical limitations in that they rely butions, such as Gaussian and Poisson. Such an assumption hinders the supervised algorithms from generalizing real-world noises that do not follow the well-known, simple distributions.
To enable supervised approaches in handling real-world noise, there have been several attempts to collect real-world datasets consisting of pairs of clean images and their corrupted versions with real noise (Plotz & Roth, 2017; Nam et al., 2016; Xu et al., 2018; Anaya & Barbu, 2018; Ab- delhamed al., 2018; 2020). Among them, one of the widely used datasets for training is the SIDD (Abdelhamed et al., 2018) dataset, which consists of real-world noisy images captured by various smartphone cameras, considering different ISO, shutter speed, and aperture settings. Never- theless, obtaining such images requires a significant amount of time and resources, which restricts the availability of large-scale real noisy datasets.
To alleviate these constraints, generative methods have been proposed. Specifically, numerous gen- erative approaches employ generative adversarial networks (GAN) or normalizing flows (NF) to leam the distribution of real noise and synthesize realistic noisy images (Yue et al., 2020; Zamir et al., 2020; A Abdelhamed et al., 2019; Jang et al., 2021; Chang et al., 2020; Kousha et al., 2022).
In this study, we introduce novel framework called NAFlow, short for Noise-A ware Normalizing Flow, designed to learn and effectively manage the diverse noise distributions originating from various camera settings.
Equal contribution. *Corresponding author.

---

Published as a conference paper at ICLR 2024
Unlike previous models such as NeCA (Fu et al., 2023) that use multiple different noise models deal with different noise distribu- tions, we only single uni- fied model to learn such complex noise distributions. Additionally, we present Noise-Awant ware Sampling (NAS) algorithm, which empowers our noise generation model to exploit the Gaussian mixture model (GMM) derived from diverse noise distributions. Importantly, NAFlow mitigate the necessity for metadata (e.g., ISO, camera model) associated with input noisy images for sampling. Furthermore, we recognize the importance of spatial noise correlation sRGB space, and thus propose multi-scale noise embedding technique to facilitate the synthesis of the spatially correlated noise. Through various experiments, NAFlow has demonstrated its outstanding noise quality. Additionally, when utilized for denoising tasks with generated noise, it has achieved a state-of-the-a level of performance on SIDD benchmark.
Table I: Comparison between sRGB noise modeling networks.
Key features
Flow-sRGB
NeCA
Ours
Sampling without metadata
X
Single unified model
/
Correlated noise generation
X
2 RELATED WORK
Noisy Image Generation. Due the lack of real-noise datasets, there are several attempts gen- erate realistic noisy images. DANet (Yue et al., 2020) learns the joint distribution of the clean and noisy image pair to map the noisy image to the clean one, enabling simultaneous noise genera- tion and elimination. C2N (Jang et al., 2021) is trained with unpaired clean and noisy images and generates synthetic noisy images with adversarial loss. CycleISP (Zamir et al., 2020) presents a CNN-based framework and exploits camera image pipelines on both the sRGB and RAW spaces. CA-NoiseGAN (Chang et al., 2020) encodes camera-dependent noise information using contrastive learning and generates realistic noise conditioned on that information. PNGAN (Cai et al., 2021) splits the noisy image generation problem into image-domain and noise-domain alignment prob- lems and carries out pixel-level noise modeling. Moreover, there some generative models based on conditional NF rather than GANs since NF can be trained easily and generate more diverse im- ages (Lugmayr et al., 2020; Abdelhamed al., 2019; Kingma & Dhariwal, 2018; Wang al., 2022; Liang et al., 2021; Kousha et al., 2022). NoiseFlow (Abdelhamed et al., 2019) learns noise distri- bution of RAW images captured by smartphone cameras conditioned on the smartphone type and gain setting (e.g., ISO), and Flow-sRGB (Kousha et al., 2022) extends the capability of NoiseFlow to tackle denoising in sRGB space. However, the existing noise modeling methods lack modeling spatially correlated noise. To address this issue, NeCA (Fu et al., 2023) explicitly learns the neigh- boring correlation of the noise within the dataset of SIDD (Abdelhamed et al., 2018). Although NeCA closes the distribution gap between synthetic and real noise in the sRGB space, it requires multiple different noise models to deal with different real-world noise.
In this work, we extend the capabilities of the NF-based framework as generative classifier as well as the ability to generate a variety of samples, as discussed in previous studies (Mackowiak et al., 2021: Izmailov et al., 2020; Ardizzone al., 2020). In particular, we employ GMM to leam multiple different real-world noise distributions according to ISO levels, camera models, which mainly affect the noise distribution in camera imaging pipeline (Ramanath et al., 2005; Gow al., 2007). These learned distributions enable effective processing and generation of real input noisy images. For clarity, Tab. illustrates the differences between recent noise modeling methods and our approach. Our objective is to sampling realistic noisy that considers noise correlation during the sampling phase without any need of metadata, while employing only a single unified model.
3 BACKGROUND
3.1 NOISE MODELING
In this subsection, we begin by explaining the process of image noise acquisition, and introduce several noise modeling methods. Noisy images can be represented as a combination of a clean image and noise using the following equation:
y = 2 + n2,
2

---

Published as a conference paper at ICLR 2024
Training
(b) Nuise-aware sampling
Cleen images (Ic.))
Whe Fou
of Auscon
O
image
X(AEE
A0ss Souitll
POLTAER
Natuum Scuand)
Laite:
Notey images (I))]
LLL
Noxy image
Later
Tstern
Synthric
image (P)
Figure I: Overview of proposed framework. (a) The training procedure of our NAFlow, which uses conditional normalizing flow network f to map noisy images to Gaussian distributions on the latent space, where each Gaussian distribution corresponds to the camera configuration used for capturing images. (b) The pipeline of inference procedure NAS. We use the Gaussian mixture model to obtain a more accurate latent representation without metadata of a given input noisy images.
where æ and y represent the clean and noisy images, respectively, and n represents the added noise.
Real-world image noise n. arises from inherent constraints associated with camera sensors and is further introduced during the post-imaging process. This real-world noise-include photon noise, read noise, and quantization quantization-occuus occurs during the initial capture of a RAW-RGB image. Subsequently, RAW-RGB image undergoes non-lineat imaging processes (such demosaicking, gamma cor- rection, and tone mapping), transforming it from scene- based RAW-RGB color space to screen- based sRGB color space. Due the application of multiple non-linear operations to both a clean image and a noise, modeling sRGB noise becomes more challenging problem. To address this problem, we aim transform the complex real-world noise distributions into several simple distri- butions to leverage the diverse properties of real-world noise by adopting normalizing flows.
3.2 NORMALIZING FLOW
Normalizing flows (NF) belong to a class of generative models that use invertible transformations to map simple distribution, typically Gaussian distribution, more complex target distribution. The invertible mapping allows NF to directly optimize negative log likelihood (NLL). Due to the advantageous combination of simplicity and effectiveness offered by NF (Abdelhamed al., 2019; Kousha et al., 2022), we use NF for modeling noise distributions.
Composed of sequence of invertible transformations, our NF aims learn an invertible mapping f + x, where and are the original (image) and transformed data (latent), respectively. Specifically, the change of variables formula is employed to compute the probability density of the latent with respect to the original data as follows:
p(x) = p(=) fe
(2)
Where, detD) represents the determinant of the Jacobian matrix of the transformation Df((r), taking into account how the volume changes during the mapping. The training objective the NLL loss of the observed data æ, which is minimized by training the model parameters A.
PROPOSED METHOD
In this section, we start with the description of our NAFlow, novel based framework that allows for learning multiple distinctive Gaussian distributions within the latent space, thereby enabling the effective modeling of diverse real noise distributions (Sec. 4.1). Next, we present our NAS algorithm, which uses the trained NF not only a noise generator but also as a noise classifier during the inference phase (Sec. 4.2). contrast to previous works (Kousha et al., 2022; Fu et al., 2023) that require the meta-data of target noise even during inference, we reduce the dependency through our NAS algorithm. Specifically, our NAS algorithm measures similarities between the input noise and several learned real- world noise distributions. And then, synthesizes noise that follows a similar distribution to the input noise. Finally, we present multi-scale noise embedding strategy capture spatially correlated noise features within the sRGB input image. (Sec. 4.3).
real-world
3

---

Published as a conference paper at ICLR 2024
(o) 56 00100
(b) G4 00400
(d) 01000
(d) N6 (32200
(e) GP 06400
Figure 2:
Noise visualization from diverse device settings (e.g,, smartphone model, ISO). Each noise has unique distribution due the distinct characteristics of each image signal processor.
4.1 LEARNING MULTIPLE DISTINCT NOISE DISTRIBUTIONS
In this work, we aim to generate realistic noisy images to solve the real-world denoising problem. To generate such synthetic noisy images, it is required capture distributions of real noisy im- ages as in (Yue et al., 2020; Jang et al., 2021; Fu et al., 2023). To do so, we learn the complex distribution noisy images in the SIDD dataset (Abdelhamed et al., 2018), which consists of real noisy images with corresponding meta-information and clean counterparts. Note that the real-world noise distribution is unknown and varies depending on the camera model, sensor type, and camera settings (Ramanath et al., 2005; Gow et al., 2007). Furthermore, distinct noise correlation conditioning on each camera configuration (e-g,, smartphone model, ISO) in the SIDD. This observation motivates
shown in Fig. 2, observe
model multiple camen-configuraioo-spcciii normal distributions latent space to handle various real-world noise.
Following the usual settings in the field of noise generation (Yue et al., 2020; Kousha et al., 2022; Fu et al., 2023), we assume that ground truth clean images are given, and we use the clean images conditional information, where this conditional information allows the NF learn condition- specific noise distribution. Specifically, in our work, we use the conditional NF (Ardizzone et al., 2019) which takes ground truth clean image æ for conditioning as:
Ze = (e(: - ye = fo (Ze;z),
(3)
where denotes real noisy image taken with camera configuration and our camera configuration consists of smartphone model and ISO setting (e.g., iPhone7 with ISO 1600). The noisy image undergoes mapping to latent variable ze through the invertible conditional normalizing flow f with parameters and ground truth clean image æ is used for conditioning the NF.
Then, we learn the noise distributions using camera intggurrtion-speciie normal distributions
through a single NF as follows:
Mfpe Ee),
(4)
where Mes E. denotes the mean and covariance matrix of the specific normal distribution for the configuration and are trainable parameters in this work. Then, we can compute the conditional probability density function of ye given x as:
p(ve(z) (Zci Mer Ee) fo
where Dfe(Ve: x) denotes Jacobian of fe. To handle multiple different distributions using a single NE, we train the NF by minimizing the negative log-likelihood (NLL) loss function as follows:
(N).1(0) = logu(ue(=)
(6)
Σ (logN(zzjj Ex) IogldetD
where denotes the number of camera configurations available in the SIDD dataset.
Through the trained conditional NF, we can synthesize noisy images which have similar distri- bution to ye by taking random samples from A(ue, Σ.) as an input of the fø given ground truth clean image the condition. Fig. 1(a) illustrates the proposed framework.
4.2 NOISE-AWARE IMAGE GENERATION
Unlike our NF training procedure, we propose to generate noisy images without metadata (e.g., ISO, smartphone manufacturer) given target noisy input y. In particular, we aim synthesize
4

---

Published as a conference paper at ICLR 2024
1kt Ouar
Iss. Quee
1xd Dane
Phas
Phant
Image
Scale level
Aftrne
Actnemn
Invetile
Invettile
Invottll
Is 1000
Dandiood
Latent
Noiey Image
Embedding Block
Transition Conditional Flow Block =K
Figure 3: Overview of NAFlow.
noisy images whose noise distributions are similar to the noise distribution of the given imput im- age. To do so, utilize the capability of the NF generative classifier (Mackowiak et al., 2021; Izmailov et al., 2020). Specifically, we obtain latent & from the noisy input through the trained NF and measure the likelihood of that latent z for each learned Gaussian distribution (i.e. NI W (2,, . N2: pC, ).)). Recall that we use the GMM in latent space to approximate target noise using a combination of distinct noise information given these likelihoods. Thus, using this set of C' likelihood values, we can mix the learned C' distinct noise distributions and generate input specefie images from the mixture model. The proposed generation process and algorithm are illustrated in Fig. ((b) and Alg. 1.
Algorithm Noise-Aware Sampling (NAS)
Input: clean condition z, noisy input y
1: = x)
2: Sample ((, Σ). where jt Σ Mazpa E.) b (ai
Me. Σ Σ
Nssuee. Ex) Σ
N (Fittel (ic)) E.
return fo (²;x)
In our experiments, we demonstrate that the proposed NAS algorithm allows us to synthesize real-world noisy images without metadata with the aid of our noise-aware generation mechanism.
4.3 MULTI-SCALE NOISE EMBEDDING
Taking into account spatial noise correlation in sRGB image a critical consideration (Zhou al., 2020; Lee et al., 2022; Fu et al., 2023). Specifically, majority of digital cameras employ a Bayer filter array, where each pixel in the image sensor can only record the intensity of light for one the three primary colors (e.g.. red, green, or blue), necessitating the interpolation of color information to generate a comprehensive full-color image. In the other words, the process demosaicking on the Bayer filter induces noise correlation among adjacent pixels (Jin et al., 2020). Moreover, recent study (Zhang et al., 2023) has computed the size spatially correlated noise areas within different regions of the image, and found considerable variance in this area size, ranging from a single pixel to over 25 pixels. Inspired by this noteworthy observation, our NF architecture incorporates a multi- scale noise embedding strategy involving squeeze operations effectively address different scales of noise correlation. Please note that our approach is the first NF-based attempt to model real-world noise considering noise correlation through the multi-scale embedding approach.
We illustrate our proposed NAFlow in Fig. 3. The overall architecture of our proposed NAFlow draws inspiration from several prior works (Dinh et al., 2017; Kingma Dhariwal, 2018; Ardiz- zone et al., 2019; Lugmayr et al., 2020). For a clearer description, we provide brief descriptions of each component of NF Appx. A. The architecture of our model network is designed with three scale-level blocks (L 3) effectively manage diverse scales of noise. The initial process- ing step involves encoding the noisy image through an embedding block, which including invertible 1x1 convolution (Kingma & Dhariwal, 2018) and ActNorm (Kingma & Dhariwal, 2018). Each scale-level block is responsible for accommodating different noise scales and comprises squeeze operation, single transition block, and two conditional flow blocks (K 2). It is worth noting that
5

---

Published as a conference paper at ICLR 2024
the squeeze operation reduces the spatial resolution of the imput features by half, crucial step for addressing multi-scale noise. Additionally, we have incorporated an extra transition block follow- ing the squeeze operation to mitigate checkerboard artifacts in the reconstructed image, stemming from pixel re-ordering (Lugmayr et al., 2020). We opt for K = 2 and bypass conditional flow blocks following the embedding block transform all channels in the feature each level, ensur- ing the network's s efficiency. This choice is motivated by the fact that affine coupling can transform only half of the feature channels conditioned on the remaining half (see Appx. A.I). Meanwhile, each conditional block comprises a conditional affine coupling (Ardizzone et al., 2019), an affine injector (Lugmayr et al., 2020), and one transition block. Notably, both the affine injector and con- ditional affine coupling blocks are conditioned on clean images, enabling them to model noise that contingent upon the underlying clean signals.
5 EXPERIMENTS
To evaluate the capabilities of our noise generation model, we conduct two main experiments. First, we assess the quality of the generated noise quantitatively and qualitatively and investigate noise correlation through visualization (Sec 5.2, Sec 5.3, and Sec 5.4). Second, we examine the effective- ness of the generated noise in real-world denoising tasks by training a denoising network using the generated noisy images (Sec 5.5). Finally, we conduct ablation studies show superiority of our proposed NAFlow, NAS, and multi-scale noise embedding (Sec 5.6).
5.1 EXPERIMENTAL. SETUP
Implementation Details. All the networks optimized using Adam optimizer (Kingma & Ba, 2014). For training NAFlow, we minimize the LNLL loss in Eq. 6 with initial learning rate le- 4 which is reduced by half at 50k, 75k, 90k during 100k iterations. We use randomly cropped patches (160 x 160) and the mini-batch size of 8 for training. For a denoising network, we use the DnCNN (Zhang al., 2017). Training is conducted with the identical experimental configuration as described in Fu et al. (2023); with 300 epochs, a learning rate 10-3, and a batch size of 8.
Dataset. To train NAFlow, we use SIDD (Abdelhamed et al., 2018) dataset, which has 34 different camera configurations (i.e., C = 34). Specifically, we use noisy-clean image pairs captured with five different smartphone cameras: Google Pixel (GP), iPhone 7 (IP), Samsung Galaxy 56 Edge (S6). Motorola Nexus 6 (N6), and LG G4 (G4). To evaluate the noise generation, we use
split which comprises 320
The training of the denoiser Sec. 5.5 also uses the same dataset, but instead of real noisy data, it uses noisy data generated by each noise modeling method. Furthermore, to measure the real denoising performance, we use SIDD-Benchmark dataset.
To evaluate the quality of the generated noise, we use two metrics: Kullback-Leeie Di- vergence (KLD) and Average Kullhack-Leible Divergence (AKLD) (Yue et al., 2020). In addition, we adapt PSNR and SSIM metrics to evaluate the denoising performance.
Metrics.
5.2 NOISE GENERATION WITH SIDD
Compared Baselines. We compare our model with several noise generation models, including C2N (Jang et al., 2021), Flow-sRGB (Kousha et al., 2022), and NeCA-S, NeCA-W (Fu et al., 2023). We measure noise quality for each camera in the SIDD- Validaiin to assess device-specific charac- teristics as in (Fu et al., 2023). Following the usual settings as in (Kousha et al., 2022; Fu et al., 2023), to ensure consistency between the training and validation sets, we ensure that both sets con- tain the same ISO levels. We use official weight parameters to evaluate the compared methods. Please note that C2N differs from other models in that it trains noise distribution an unpaired manner, utilizing additional clean dataset and noisy dataset such BSD500 and DND (Plotz & Roth, 2017) during training.
Tab. 2 shows the noise quality in terms of KLD and AKLD conducted on of five different cameras. Compared to other noise modeling methods, our NAFlow consistently demonstrates superior performance in terms of average KLD and average AKLD. An important point to note is that our NAFlow shows a performance gain of 0.0035/0.013 in terms of KLD/AKLD compared to NeCA-W (Fu et al., 2023).
Results.

---

Published as a conference paper at ICLR 2024
Table Quantitative results of synthetic noise on SIDD-Validatio. The results are computed with KLDJ and AKLDJ. In the case of NeCA-S and NeCA-W (Fu et., 2023), they require five different models for noise generation, one for each smartphone camera. In particular, NeCA W requires three sub-models (GENet", NPNet", NCNet") for each camera. Best results are highlighted in bold.
Camera
Metries
C2N
Flow-sRGB
NeCA-S
NeCA-W
NAFlow
KLD
0.1660
0.0507
0,4025
0.0242
0,0254
G4
AKLD
0.2007
0.1504
1.6803
0.1524
0.1367
KLD
0.1315
0.0781
0.1713
0.0432
0.0352
GP
AKLD
0.1968
0.1797
0.7379
0.1273
0,1180
KLD
0.0581
0,5128
0.3424
0.0410
0.0339
IP
AKLD
0.2929
1.7490
1,2924
0.1145
0.1522
KLD
0.3524
0.2026
0,2830
0.0206
0,0309
N6
AKLD
0.2919
0.2469
1,0998
0.1304
0.1108
KLD
0.4517
0.3735
0.1724
0.0302
0.0272
S6
AKLD
0.4190
0.2641
0,4646
0.1933
0.1355
KLD
0.2129
0.2435
0,2743
0.0342
0.0305
Average
AKLD
0,2802
0,5180
1.0470
0.1436
0.1306
# Params
2.2M
6K
5 * (7.8M")
5 (263K
42K 7.8M")
1.1M
NeCA.W
Real
Noiey
Noise
RL- 0745
KL-- 0.063
XI 0.034
0.86
0.01
0.15
GP 10000
(b) IP 01600
Visualization of synthetic noisy images with different Camera configura- tions (Camera ISO) on SIDD- Validation. From left to right: C2N, NeCA-W, Ours (NAFlow), and Real noisy images.
Figure 4:
Note that NeCA-W is trained separately for each camera model, five differently trained NeCA-W models may produce more realistic noise for certain camera models. In contrast, our model uses single unified model with a much smaller number of parameters for all camera models. Nevertheless, our model shows comparable performance to NeCA-W: W. Moreover, Fig. 4 compares the generated noise and noisy images. notable observation is the strong similarity between the noise generated by NAFlow and real-world noise. This indicates that NAFlow excels at synthesizing realistic noise.
5.3 NOISE GENERATION WITH SIDD+
Generating noise for different camera configurations that are not included the SIDD dataset is a challenging task, most conventional noise generation models synthesize noise using the metadata available in the SIDD. During training, our NAFlow also uses metadata from SIDD (Abdelhamed al., 2018) learn multi-distribution, but unlike other models (Kousha et al., 2022; Fu al., 2023), NAFlow can generate noise using only noisy-clean image pairs without relying on the meta- data during the inference phase. Consequently, NAFlow can generate noise that is similar within the training distribution even for the camera configuration not included in the SIDD dataset. To validate this advantage, we use the SIDD+ (Abdelhamed al., 2020) dataset, which is captured with smart- phones similar to SIDD but with smartphones not used to capture images in the SIDD dataset. Tab. shows the accuracy of synthetic noise using the clean-noise image pairs on the SIDD+ (Abdelhamed et al., 2020). For the pre-trained NeCA-S and NeCA-W models, the challenge is to generate noise for camera configurations which are not available in the SIDD training dataset. Therefore, we show the best KLD values among their five different camera models. In Tab. has the highest performance on SIDD+. This shows that NAFlow is able to generate comparable synthetic noisy images in the absence of metadata.
that NAFlow

---

Published as a conference paper at ICLR 2024
Table 3:
Quantitative results of synthetic noise from the dataset (SIDD+) not used for training the generative model. The results are computed in terms KI..J. Best result is highlighted in bold.
10
64 Rcal
Figure 5: Noise correlation map on N6_03200 from dif- ferent methods, Ours shows the highest similarity with the correlation map Real (d).
Metric C2N
NeCA-S
NeCA-W
NAFlow
KLD1
0.3973
0.2620
0.0629
0.0494
Table 4: Quantitative results of denoising performance on SIDD Benchmark in terms of PSNR1 and SSIM All methods are trained under the same dataset conditions (synthetic noisy-cleat pairs). Note that Real Noise are trained with real noisy-clean pairs. The best result is highlighted in bold.
Metries
C2N
NoiseFlow
Flow
NeCA-S
NeCA-W
NAFlow
Real Noise
PSNR
33.76
33.81 0.894
34.74
36.10
36,82 0.932
37.22 0.935
37.63
SSIM
0.901
0.912
0.927
0.935
5.4
NOISE CORRELATION VISUALIZATION
It is important to model real-world noise by considering noise correlation (Fu et al., 2023). In Fig. 5, we compute correlation between the noise in the center pixel and the neighbour noises following the procedure in (Lee et al., 2022; Wang et al., 2023) and compare the noise correlation of our NAFlow with other noise generating networks. Our network generates spatially correlated noise that is most closely resembles the noise correlation in the Real compared to C2N and NeCA-W.
5.5 APPLICATION: SRGB DENOISING
To evaluate the efficacy of the noise generating models and compare their performance in real-world denoising, we train conventional denoising network using synthetic noisy images from C2N (Jang et al., 2021), NoiseFlow (Abdelhamed et al., 2019), Flow-sRGB (Kousha et al., 2022), NeCA-S (Fu al., 2023), NeCA-W (Fu et al., 2023), and our NAFlow. To be specific, DnCNN is trained with pairs of clean and synthetic noisy images, and denoising performance of DnCNN (Zhang et al., 2017) is compared Tab. 4. The last column (Real Noise) in Tab. 4 shows the denoising result by DnCNN trained with clean and real noisy image pairs, indicating the upper bound result.
As a result, when using synthetic noisy generated with our NAFlow, DrCNN shows significantly better denoising performance compared other noise generating methods. Specifically, NAFlow outperforms the current state-of-the-art method NeCA-W by 0.39 dB in PSNR. Note that our NAFlow does require metadata separately trained network for each camera configuration. Comparing the performance with the upper bound, we acknowledge that the PSNR value still lags behind. However, is worth noting that we significantly reduce the PSNR gap and achieve equiv- alent performance in terms of SSIM. We also provide denoising results from the differently trained DnCNN Fig. 6. Our synthetic noise dataset leads to visually more pleasing denoising results, showing higher robustness against remaining noise artifacts compared to the other baseline models.
5.6 ABLATION STUDY
Effect of noise-aware sampling. As in Alg. 1, our model explores the learned latent space with multiple different noise distributions, and generates input noise-specific noisy images. To evaluate the effectiveness of our noise-aware sampling algorithm, we provide an ablation study result by comparing different sampling methods in Tab. 5. First, the noisy input images are mapped to la- tents through our NF model. Then we measure the likelihood value of the latents for each normal distribution NHee E.), and generate new noisy images using the normal distributions with high likelihood values. In Tab. 5, (Rank-k) indicates the accuracy of the synthetic noise from the normal distribution that has the k-th best likelihood value. we expect, image samples generated from normal distribution with higher likelihood value give better results, which shows the performance of our NF as a generative classifier.
8

---

Published as a conference paper at ICLR 2024
Table Ablation study on the effective NAS. We rank the Table 6: Two conditional NFs (fe) camera configurations classified by NAFlow and generate trained under single latent distribution the noise from these e and measure the metrics in the SIDD- and multiple latent distributions are Validation.
compared.
Metrics
Rank-1
Rank-2
Rank-3
Rank-4
Rank-5
Random
All
Latent distribation
KLD
AKLD
KLDJ
0.0294
0.0312
0.0408
0.0422
0.0613
0,4700
0.0291
Single normal distribation
0.0544
0.1657
AKLDI
0.1302
0.1308
0.1375
0.1463
0.1676
0.4486
0.1243
Multiple distinct distributions
0.0291
0.1293
250 M4YT
10722-007
PV-NoD T4N
Figure Denoising results in terms of PSNRT and SSIM on Validaaaiottion from DrCNN trained on each method. For more qualitative results, please refer to the appendix.
Furthermore, see that our NAFlow outperforms naive Rank-k sampling, and Random sampling which generates new noisy images from randomly se- lected normal distribution in the latent.
Table 7: KLD score conditioning on learnable parameters of Gaus- sian distribution in Eq. 4.
Table 8: KLD score depending on different number scale- level blocks L in Fig. 3.
Training μ Training KLD
Scale-level KLD
X
X
0.0544
L
0.0349
X
0.0449
0.0313
0.0340
= 33
0.0291
0.0291
= N/A
Effeet of learning mul- tiple distributions. Our NAFlow trained by assuming C' distinct latent distributions, and the latent sampling mechanism presented in Alg. allows data. To show the superiority generate input noise-award noisy images without additional meta- our generation mechanism considering multiple distinct distribu- tions, train the NF (fo) by assuming single distribution (i.e., μlc 0, Σ I for all c) and com- pare the accuracy of the generated noisy images in terms of KLD. In Tab. 6, we see that NAS based on multiple different distributions renders more accurate noisy images on the SIDD-Van Validatio..
Trainable parameters of Gaussian distribution. NAFlow leverages multiple Gaussian models with both trainable mean and covariance the latent space, effectively addressing the variety of real-world noise modeling as described in Eq. 4. In Tab. 7, we conduct ablation study to assess the effectiveness of each parameter of the Gaussian distribution within the latent space. Our findings indicate that the NAFlow with learnable both mean and covariance of Gaussian distribution outper- forms the other models. This indicates that the richer expressive power of the Gaussian distribution is crucial in accurately modeling the various real-world noise.
Impact multi-scale noise embedding. In our flow architecture, we integrate a multi-scale noise embedding approach to handle spatial noise correlations of varying areas. As shown in Tab. 8, an increase the scale-level leads to better KLD scores on the Valddaaaitt,on, indicating our model's 's ability to adeptly learn multi-scale noise characteristics. However, could not investigate the higher scale level factor (e.g., L 2 4) due to unstable training.
6 CONCLUSION
In this study, we developed a novel NAFlow framework to address the complex and diverse noise distributions that arise from a variety of camera settings. In addition, we introduced the Nwise-Aware Sampling (NAS) algorithm. which builds on the foundation of the NAFlow framework. NAS allows our noise generation model use Gaussian mixture models derived from multiple noise distribu- tions to synthesize realistic sRGB images with noise. A notable advantage of our method is that eliminates the need to input noisy image metadata during inference. In addition, we recognize the importance of considering noise correlations with different scales. Consequently, we present multi-scale noise modeling approach that effectively addresses noise correlation across a range of scales. The experimental results demonstrate the significant performance improvement that our al- gorithm brings to both the noise modeling and the denoising network, highlighting the effectiveness of our proposed methods.

---

Published as a conference paper at ICLR 2024
ACKNOWLEDGMENTS
This work was supported by Institute of Information & communications Technology Planning & Evaluation (IITP) grant funded by the Korea MMVI) (No.202--0-00156, Fundamental research on continual meta-learning for quality enhancement of casual videos and their 3D meta- verse transformation) and Institute of Information & communications Technology Planning & Eval- uation (ITTP) grant funded by the Korea governmen((MSTT) (No. RS2023-00220688, Artificial telligence for prediction of structure-based protein interaction reflecting physicochemical principles, 30%)
REFERENCES
Abdelrahman Abdelhamed, Stephen Lin, and Michael S Brown. A high-quality denoising dataset for smartphone cameras. In CVPR, 2018.
Abdelrahman Abdelhamed, Marcus A Brubaker, and Michael Brown. Noise flow: Noise modeling with conditional normalizing flows. In ICCV, 2019.
Abdelrahman Abdelhamed, Mahmoud Afif,, Radu Timofte, and Michael S Brown. Ntire 2020
challenge on real image denoising: Dataset, methods and results. In CVPR, 2020.
Josue Anaya and Adrian Barbu, Renoir-a dataset for real low-light image noise reduction. Journal of Visual Communication and Image Representation, 51:144-155: 2018.
Saeed Anwar and Nick Barnes. Real image denoising with feature attention. In /CCV, 2019.
Lynton Ardizzone, Carsten Lüth, Jakob Kruse, Carsten Rother, and Ullrich Köthe. Guided image generation with conditional invertible neural networks. arXiv preprint 1907.02997 2019.
Lynton Ardizzone, Radek Mackowiak, Carsten Rother, and Ullrich Köthe. Training normalizing flows with the information bottleneck for competitive generative classification. NeurIPS, 2020.
Yuanhao Cai, Xiaowan Hu, Haoqian Wang, Yulun Zhang, Hanspeter Pfister, and Donglai Wei. Learning to generate realistic noisy images via pixel-level noise-aware adversarial training. NewrlPS, 2021.
Ke-Chi Chang, Ren Wang, Hung-Jin Lin, Yu-Lun Liu, Chia-Ping Chen, Yu-Lin Chang, and Hwann- Tzong Chen. Learning comera-award noise models. In ECCV, 2020.
Liangyu Chen, Xin Lu, Jie Zhang, Xiaojie Chu, and Chengpeng Chen. Hinet: Half instance normal- ization network for image restoration. In CVPR, 2021.
Laurent Dinh, David Krueger, and Yoshua Bengio. NICE: non non-linear independent components esti- mation. In ICLR, 2015.
Laurent Dinh, Jascha Sohl-Dickstein, and Samy Bengio. Density estimation using real NVP. In ICLR, 2017.
Zixuan Fu, Lanqing Guo, and Bihan Wen. srgb real noise synthesizing with neighboring correlation- aware noise model. In CVPR, pp. 1683-1691,
Ryan D Gow, David Renshaw, Keith Findlater, Lindsay Grant, Stuart J McLeod, John Hart, and Robert L Nicol. A comprehensive tool for modeling cmos image-sensor-nonn performance. IEEE T-ED, 54(6): 1321-1399,,, 2007.
Pavel Izmailov, Polina Kirichenko, Mare Finzi, and Andrew Gordon Wilson. Semi-supervised learn ing with normalizing flows. In ICML, 2020.
Geonwoon Jang, Wooseok Lee, Sanghyun Son, and Kyoung Mu Lee. C2n: Practical generative noise modeling for real-world denoising. In ICCV, 2021.
Qiyu Jin, Gabriele Facciolo, and Jean-Michel Morel. A review of an old dilemma: Demosaicking first, denoising first? In CVPRW, pp. 514-515, 2020.
10

---

Published as a conference paper at ICLR 2024
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6990, 2014.
Durk P Kingma and Prafulla Dhariwal. Glow: Generative flow with invertible 1x1 convolutions. NeurlPS, 2018.
Shayan Kousha, Ali Maleky, Michael S Brown, and Marcus A Brubaker. Modeling srgb camera noise with normalizing flows. In CVPR, pp. 17463-17471, 2022.
Wooseok Lee, Sanghyun Son, and Kyoung Mu Lee. Ap-bsn: Self-supervised denoising for real- world images via asymmetric pd and blind-spot network. In CVPR, 2022.
Jingyun Liang, Andreas Lugmayr, Kai Zhang, Martin Danelljan, Lue Van Gool, and Radu Timo- fte. Hierarchical conditional flow: A unified framework for image super-resolution and image rescaling. In ICCV, 2021.
Andreas Lugmayr, Martin Danelljan, Luc Van Gool, and Radu Timofte. Srflow: Learning the super- resolution space with normalizing flow. In ECCV, 2020.
Radek Mackowiak, Lynton Ardizzone, Ullrich Kothe, and Carsten Rother. Generative classifiers as basis for trustworthy image classification. In CVPR, 2021.
Seonghyeon Nam, Youngbae Hwang, Yasuyuki Matsushita, and Seon Joo Kim. A holistic approach to cross-channel image noise modeling and its application to image denoising. In CVPR, 2016.
Tobias Plotz and Stefan Roth. Benchmarking denoising algorithms with real photographs. CVPR, 2017.
Rajeev Ramanath, Wesley E Snyder, Youngjun Yoo, and Mark S Drew. Color image processing pipeline. IEEE Signal processing magazine, 22(1):34 43, 2005.
Wenzhe Shi, Jose Caballero, Ferenc Huszár, Johannes Totz, Andrew P Aitken, Rob Bishop, Daniel Rueckert, and Zehan Wang. Real-time single image and video super-resolution using an efficient sub-pixel convolutional neural network. In CVPR, 2016.
Yufei Wang, Renjie Wan, Wenhan Yang, Haoliang Li, Lap-Pui Chau, and Alex Kot. Low-light image enhancement with normalizing flow. In AAAI, 2022.
Zichun Wang, Ying Fu, Liu, and Yulun Zhang. Lg-bpn: Local and global Wind-pptch network for self-supervised real-world denoising. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 18156-18165, 2023.
Valentin Wolf, Andreas Lugmayr, Martin Danelljan, Luc Van Gool, and Radu Timofte. Deflow: Learning complex image degradations from unpaired data with conditional flows. In CVPR, 2021.
Jun Xu, Hui Li, Zhetong Liang, David Zhang, and Lei Zhang. Real-world noisy image denoising: A new benchmark. arXiv preprini 1rXiv:/804.022 2018.
Zongsheng Yue, Qian Zhao, Lei Zhang, and Deyu Meng. Dual adversarial network: Toward real- world noise removal and noise generation. In ECCV, 2020.
Syed Waqas Zamir, Aditya Arora, Salman Khan, Munawar Hayat, Fahad Shahbaz Khan, Ming- Hsuan Yang, and Ling Shao. Cycleisp: Real image restoration via improved data synthesis. In CVPR, 2020.
Dan Zhang, Fangfang Zhou, Yuwen Jiang, and Zhengming Fu. Mm-bsn: Self-supervised image denoising for real-world with multi-mask based on blind-spot network. In CVPRW, pp. 4188- 1197,2023.
Kai Zhang, Wangmeng Zuo, Yunjin Chen, Deyu Meng, and Lei Zhang. Beyond gaussian denoiser: Residual learning deep enn for image denoising. IEEE TIP, 26(7):3142-311 2017.
Yuqian Zhou, Jianbo Jiao, Haibin Huang, Yang Wang, Jue Wang, Honghui Shi, and Thomas Huang. When awgn- based denoiser meets real noises. In Proceedings the AAAI Conference on Artificial Intelligence, volume 34, pp. 13074-13081 2020.

---

Published as a conference paper at ICLR 2024
A
APPENDIX
A.1
NAFLOW ARCHITECTURE DETAIL
Affine Injector
Conditional Affine Coupling
Ixl Conv
Ixl Conv
Ixl Canv
Conv
3x3 Cony
Ixl Cony
1xl Conv
h
y
XA
XB
YB
XA
Split
Concat
Figure 7: Forward pass of Conditional Affine Coupling and Affine Injector.
The overall architecture of our proposed NAFlow draws inspiration from several prior works (Dinh et al., 2017; Kingma & Dhariwal, 2018; Ardizzone et al., 2019; Lugmayr et al., 2020).
Conditional Affine Coupling (Ardizzone layer (Ardizzone et al., 2019) is an extension of the concept of an affine coupling layer as orig- inally proposed in the RealNVP (Dinh et al., 2017). The distinguishing feature of a conditional affine coupling layer is that it depends on conditional input to perform an affine transformation. Given an input and conditional input h, an affine coupling layer splits into two parts: 21, Z2- Note that the conditional input h. is downsampled clean image by the pixel downsampling opera- tor in our NAFlow. One part Iı, remains unchanged, while the other part æ2, undergoes an affine transformation. The transformation can be defined as follows:
al., 2019): The conditional affine coupling
y1 21
1/2 22 c (x1 h)) + (x1:)))
where fs and Se arbitrary neural networks to extract scale and translation factors perform affine transformations, and @ indicates element-wise multiplication. In Fig. 7, we compose a few layers of 1x1 convolution and 3x3 convolution design the neural networks maintain the simplicity and efficiency of our NAFlow. And the inverse transformation can be defined as follows:
1/1
F2 (y2 f((11; h)) @ (x1
where Ø stands for element-wise division. Finally, the log determinant of this transformation can be calculated (xr
Affine Injector (Lugmayr et al., 2020): To further increase the influence of the conditional input on the mapping performed by NF, affine injector is introduced to transform input features solely based on conditional input features. We adopt this layer to inject clean signal information on each scale block in Fig. 3 because learning signal dependence is the important factor in modelling real- world noise. The difference between Affine Injector and Conditional Affine Coupling is that it uses the output features of neural networks, which only use the conditional input h, follows:
V/ I1
= x2 c exp(f.(h) + (h).
And for the inverse process of the layer, each dlement-wise multiplication and addition operation is replaced by an element-wis wise division and subtraction operation.
Invertible 1x1 Convolution (Kingma & Dhariwal, 2018): The operation is introduced to generalize the channel-wise permutation (Dinh et al., 2017: 2015) and fully invertible since independently multiplies each spatial coordinate of the input feature. We employ LU decomposition (Kingma & Dhariwal, 2018) to factorize the weight matrix of the 1 x Convolution to reduce the computation of its determinant and enhance model training stability (Wolf et al., 2021).
12

---

Published as a conference paper at ICLR 2024
Actnorm (Kingma & Dhariwal, 2018): The operation normalizes the input feature along the channel axis using learned affine transformation parameters.
Having the same structure the pixel downsampling operator (Shi et al., 2016), the squeeze operation is invertible and reduces the resolution the input feature in half by moving each spatial x 2 neighborhood of pixels into the channel axis and allowing for learning broader spatial correlations. To follow the naming convention previous NF works, we refer to the operation as Squeeze if it used in NF module.
Squeeze:
A.2 QUALITATIVE RESULTS OF NOISE GENERATION
In Fig. 8, we present additional visualization results of noise sampling using our method (NAFlow), as well as comparisons to existing methods (Jang et al., 2021; Fu et al., 2023). Based on the noise (bottom of each row in Fig. 8), we can confirm that NAFlow is the most similar to the real noise. the case of C2N (Jang et al., 2021), it shows similar results in terms KLD for low-light images, but it does not consider noise correlation. NeCA (Fu et al., 2023) demonstrates superior results compared to C2N, however is clear that ours (NAFlow) outperforms existing methods in terms of noise correlation and KIL metric.
A.3 QUALITATIVE RESULTS OF DENOISING PERFORMANCE
We present additional denoising results using DrCNN (Zhang et al., 2017) denoiser. We also provide denoising results from C2N (Jang et al., 2021) and NeCA (Fu et al., 2023) for comparison. The DrCNN denoisers and related test code in C2N and NeCA use pretrained official weight parameters and code from the official GitHub repositories.
C2N (Jang et al., 2021): https//gittub.comm/mww//CC2
NeCA (Fu al., 2023): https///iihub..com//uuu 1/sRGB-Ress Laa-RG--Rel-Nois--S--- Real-Noise-Synthesiizg
Based on the visualization in Fig. 9, it is apparent that NAFlow (Ours) surpasses other models in regards to PSNR and exhibits the highest similarity the clean image when compared.
A.4 NOISY GENERATION FROM METADATA
We aim to generate noise without metadata, but it is possible to generate noise using only clean im- ages learned from known distributions for each camera configuration. In Fig. 10, images generated from clean images using three cameras (GP, N6, S6) and five different ISO (100, 400, 800, 1600, 3200) settings are provided. We can observe that even at the same ISO level, slightly different noise generated depending the camera (sensor) model.
A.5 LIMITATION OF THE SIDD DATASET
The limited range of smartphone model types and lack of scene diversity in the SIDD (Abdelhamed et al., 2018) dataset makes it challenging to generalize to real-world noise modeling. Consequently, real-world noise modeling studies face difficulties in utilizing SIDD for their methods, and there is a need for more large-scale real-world noise datasets for better understanding of real-world noise.
Furthermore, the sRGB images in the SIDD dataset acquired using the simplified RAW-to-sRGB pipeline' and can produce the distribution gap from noise in real-world images. Therefore, these limitations should be addressed in future research.
13

---

Published as a conference paper at ICLR 2024
Noisy
Noise
RL. 0194
2688
B.25
KL 9.92
KL- 0235
XLL 0224
Noisy
Noise
8008
2007
K
SOLD
SOLO
SOLD
SOLD
Noisy
Noise
KL
3050
2C23
JL.
12. 0.00
Noisy
Noise
C2N
NeCA-W
Ours
C2N
NeCA-W
Ours
Real
Figure 8: Visualization of synthetic noise images on SIDD Validation. The images are arranged from left right in the following order: C2N, NeCA-W, Ours (NAFlow), and Real noisy image. For more detail, noise images are also provided at the bottom of each row.
14

---

Published as a conference paper at ICLR 2024
25N8=25.87
PSNR= 26.93
PSNR =32.37
COLOR C
COLOR D
COLOR
COLLOR
COLOR
PSNR-39:32
PSNR= 39.78
PSNR =41.97
library
the CoD ROM d CDD.RO ROM
CD ROM dd the CD-ROM d the CD-ROM
colos and lighting
coins and lighting
colos and lighting
colnr and lighting
coler and lighting
screen to world
screen to world
sereen to world
screen to world
screen-to-world
world- to scress
wil PSNR=27 83
PSNR= 32.04
PSNR #33.75
world-to-sca screen
PSNR=25.78
PSNR= 29.07
PSNR =29.87
(a) Noisy
(b) C2N
(c) NeCA-W
(d) Ours
(e) Clean
Figure 9: Denoising results on from DnCNN trained on each method.
ISO
100
400
800
1600
3200
N6
Figure 10: Generation results using camera configuration of NAFlow. The images is obtained from SIDD+ (Abdelhamed et al., 2020).
15


Reviews:
[{"review_id": "E9zRdNyYMp", "reviewer": "Reviewer_YHPu", "summary": "This paper proposes a system to generate realistic sRGB noise with noiseflow. The idea is to encode noise from multiple cameras into the same common latent space. For an unseen camera, a sample noise is encoded and a new noise is draw from a gaussian mixture model of learned noise. Multi-scale encoding schemes are employed to model long range spatial correlation of noise. Experiments show that a denoising network trained on generated noise works better than comparison techniques. Ablation study is provided and shows the importance of each of the proposed components.", "questions": "Please see my weakness section.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "I think the technical contribution of this work is solid. The proposed method works well with multiple camera models without the need for retraining. With different proprietary processing, noises from different cameras can have very different characteristics. The idea to encode different noise into different gaussian in a common latent space is convincing, and the result seems to suggest that this works well. The comparison was done convincingly to account for differences in how the comparison methods should be applied (for example, the best value was reported for the NeCA models where it needed to be trained per-camera).", "weaknesses": "Despite being successful in learning noise from SIDD dataset, I find the noise being generated to be far from realistics. Modern cellphone cameras apply heavy denoising in the chrominance channel that chromatic noise is never visible. Examples shown throughout the paper (e.g. in figure 4) shows heavy chromatic noise. Further, SIDD dataset only contain images from 10 scenes, so there is not a lot of diversity in the data. This limits how much generalization we can expect to real world cameras. While this is largely a limitation of the SIDD dataset, I think it is very important for the author to acknowledge this point in their manuscript.\n\nAs an alternative, it may be interesting for the authors to look at other dataset. There is a technical report by Jaroensri et al. (“Generating Training Data for Denoising Real RGB Images via Camera Pipeline Simulation”) that provides a dataset of raw-processed pairs that, in my opinion, is the most realistic among the raw-processed datasets available. The author should also consider applying processing from raw-only datasets such as those from Chen et al (“Learning to See in the Dark”) to achieve a more realistic noise distribution. Having included more dataset (especially in evaluation) could also be helpful in providing signal for generalization to the real world.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "wtm8ugx3tz", "reviewer": "Reviewer_EqxC", "summary": "This paper proposes a sRGB real noise modeling based on noise-aware sampling and normalized flow. Specifically, the proposed method includes (1) a new NAFlow framework with the Noise-Awar algorithm (NAS), which allows the use of Gaussian mixture models from multiple noise distributions to synthesize real sRGB images with noise without the need for metadata. (2) a multi-scale modeling method to capture the noise correlations with different scales.", "questions": "See Weaknesses", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The paper looks technically sound and describes the algorithm clearly.\n2. Experimental results demonstrate the advantages of NAFlow compared to some previous methods.\n3. The Noise-Aware Sampling (NAS) algorithm eliminates the need to input noisy image metadata during inference which is novel to me.", "weaknesses": "1.This paper combines the normalized flow generation of Flow-sRGB with the multi-scale spatial correlation of NeCA. In my opinion, the author draws more on the flow-normalized noise generation method of Flow-SRGB and further uses multi-scale noise modeling to improve performance. I want to know the effect of adding multi-scale noise modeling on Flow-sRGB, and related ablation experiments should be added.\n2.Writing errors, such as “NECA-W”, ” NAflow”, et al.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "BUVZkYLSBi", "reviewer": "Reviewer_s3bG", "summary": "This work proposes a normalizing flow-based framework NAFlow for realistic noisy image generation. The generated images are used to adapt real denoisers and boost their performance. With the proposed Noise-Aware sampling algorithm, NAFFow learns and effectively manages the diverse noise distributions originating from various camera settings.", "questions": "What are the advantages of your method when compared with other NF image denoising algorithms? Show experiments to support this.", "limitations": "", "rating": 8, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "(i) The idea of using the Normalizing Flow to model the in-camera setting distribution is interesting. Previous methods mainly focus on designing a generative adversarial network for noise modeling.\n\n(ii) The results on self-supervised image denoising are good and solid.\n\n(iii) The presentation is good and easy to follow.\n\n(vi) The appendix provides lots of visual results to show the advantages of the proposed method.\n\n(v) The ablation study is sufficient to demonstrate the effectiveness of the approaches.", "weaknesses": "(i) The motivation is unclear. Why using Normalizing Flow is not explained. Because using GAN, Auto-encoder, DeFusion Model, etc., can also model the noise distribution. So, what's the difference?\n\n(ii) The paper proposes a normalizing flow (NF) for noise generation. However, normalizing flow for noise generation has been proposed in NoiseFlow. What is the difference and your contributions?\n\n(iii) The proposed method has some technical drawbacks.\n\n[1]  The normalizing flow-based framework requires a statistic, i.e., the mean of the specific normal distribution for the configuration c. This is a very strong priori condition. In real application, this statistic may not be given.\n\n[2] The proposed method is trained on SIDD, even the pre-trained denoiser. However, the noise distribution varies across different datasets (like DND, PloyU, Nam) since they are captured by different hardware. The training may make the noise generator overfit on the SIDD dataset and not fit well with other noise distributions. Then how to solve this issue? \n\n\n(iv) The experiments need to be improved.\n\n[1] Experiments on other noisy image datasets should be provided to resolve my concerns in (iii) [2].\n\n[2] The comparisons should also be improved. The metric KLD may not be suitable for noise generation. Why not follow the metric PSNR Gap in DANet or Maximum Mean Discrepancy (MMD) to measure the domain gap?\n\n[3] The adaptation now is conducted in a self-supervised manner. What about its performance in a fully supervised manner? And what about its performance on the SOTA denoising methods like Restormer? Since some noise generation methods like PNGAN can still improve the performance of Restormer.\n\n[4] The comparison between the proposed NAFlow and other SOTA noise generation algorithms should be added in the main paper, including DANet, PNGAN, GDANet, CycleISP, C2N, etc.\n\n(v) Reproducibility: The network and the method (Meta-train and Fast Adaptation) are pretty complicated. It is difficult for other researchers to reproduce the whole method. Since the code and models are not provided, the reproducibility cannot be checked.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "oDVsDQTng9", "reviewer": "Reviewer_VvGJ", "summary": "The paper introduces a new normalizing flow framework (i.e., NAFlow) for noise modeling and synthesis. For noise modeling, NAFlow learns to map noise of various smartphone types and gain settings to different Gaussian distributions. For noise synthesis, NAFlow uses the Gaussain mixture model of learnt distributions to generate accurate yet diverse noisy images. Experiment results on real-world denoising datasets show the superiority of NAFlow over existing methods.", "questions": "1. (This question is not critical to rating) What are the applications of noise synthesis and modeling methods? These methods apply paired noisy-clean images for training, then synthesize new pairs from clean images for training denoiser. However, the denoiser trained with synthesized pairs shows inferior performance than trained with original real paires. Could the authors explain the applications of proposed method?", "limitations": "", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The idea of learning the noise distribution implicitly without metadata is reasonable. It is practical to model the noise of devices that is hard to accquire metadata, such as smartphones.\n2. The proposed NAFlow outperforms baseline methods by a large margin on both noise modeling and noise synthesis.\n3. The paper is well organized and written.", "weaknesses": "1. The comparison with C2N is unfair. C2N is trained with only noisy images, while other methods are trained with paired images. This training setting should be pointed out in the experiments.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)