Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 7: Good paper, accept
review_content: summary: In this paper, the authors proposed a novel way to generate real-world noise, and presented the reliability of generating noise data sets on a large number of existing methods. Overall, the lack of real noisy images limits the performance of the fully supervised denoising network, and the proposed method contributes a reasonable solution.  
main_review: The authors propose a simple but novel idea (PNGAN) for real image denoising. In my opinion, the motivation is reasonable by treating each noisy pixel as a random variable and then splitting noisy image generation into image and noise domain alignment. Compared with the previously proposed image-level GAN, the novel pixel-level PNGAN can obtain better predictions through the score map. The pixel-level adversarial training encourages the generator to pursue better solutions. In the experiments, ablation studies demonstrate the effects of PNGAN components, such as the proposed discriminator, pixel-level vs. image-level, components in the generator. Meanwhile, compared to existing works, the proposed PNGAN also achieved promising results. 

However, I still have the following concerns:

(i) The novelty of the generator in Figure 3 is limited. Multi-scale attention and stacking of residual blocks are common components. But the use of fast 1D convolution is innovative.

(ii)It seems that Gaussian Poisson noise is better than Gaussian noise. The argument in favor of the used initial noise distribution should be provided.

(iii) What I am concerned about is whether the proposed PNGAN has good generalization?
(iv) The font in Figure 4 is too small, you can use abbreviations instead.
(v) In Fig 1 (a2), when the metadata of the image is unavailable, how does the PNGAN create noisy images using Setting2?
(vi) In Line 321-324 of the main paper, the authors claim that the image domain alignment is successfully conducted. It would be more convincing if more results are provided.




limitations_and_societal_impact: I think this method may have certain limitations in generalization, such as how to deal with unlearned real noise patterns in real-world applications.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This paper proposed a method to generate fake noisy-clean image pairs, which are expected to well simulate the real training pairs, in an adversarial training fashion, in order to augment the training set for a deep denoiser.
main_review: The idea of adopt the adversarial training strategy to generate more training data is reasonable, and has been shown to be effective in previous studies. This paper has done good job in making this strategy also work in denoising problem, and shown promising results in experiments. The paper is also well-written and easy to follow. 

The major problem of this work is that it misses discussion and comparison with a related study that addressed the similar problem. Specifically, Yue et al. proposed in [1] a dual adversarial training framework, which also aims to generate noisy-clean image pairs to augment the training set for real image denoising. The authors should discuss and empirically compare with this method. Besides, Yue et al. have also proposed two metrics to assess the performance of a real noisy image generator, and it is better to also consider these metrics in this work.

A minor problem: $\hat{I}_{clean}$  in the first equation of (1) should be  without "^" sign.

[1] Zongsheng Yue, Qian Zhao, Lei Zhang and Deyu Meng. Dual adversarial network: Toward real noise removal and noise generation. ECCV, 2020.
limitations_and_societal_impact: The authors did not mention the limitations of their method. However, in my opinion, the major limitation is that it can not guarantee that the generated noisy images follow the same distribution of the real noisy images, although I know it is a very difficult problem.
review_title: null

final_score: rating: 8: Top 50% of accepted NeurIPS papers, clear accept
review_content: summary: The authors propose a pixel-level noise-aware generative adversarial network (PNGAN). They improve the denoising performance by generating more realistic noisy images.  Quantitative experiments prove that the performance of the existing denoising model can be significantly improved after being trained with the synthetic realistic noisy images.
main_review: + This work has many advantages, such as:

The authors solve certain limitations in the current real denoising task. For example, the domain difference between synthetic noise and real noise is alleviated. As a better noise data enhancement method, PNGAN reduces the difficulty of obtaining real noise image pairs.

The proposed method uses GAN to generate pixel-level synthetic noise, which is consistent with real-world noise. It can be used for effective training data augmentation.

Visually, PNGAN produces results that are closer to real noise. The quantitative experiments in Table 1 also show that the generated fake noisy images can be directly used for the training of the existing network, and the performance is improved by a large margin.

The paper is well-organized and clearly written, and the detailed source code is provided.

-	However, I still have the following concerns:

There is no quantitative evaluation of model parameters and interface speed in the paper.

For disordered and random noise, how to achieve Noise Domain Alignment in line140~175?

What are the differences and advantages of Pixel-level Noise Modelling and Image-level Noise Modeling?

Figure 4 shows that different data sets perform differently in domain discrepancy. Please analyze the reasons.

Finally, the method PNGAN is novel and its effectiveness is demonstrated by extensive ablation study and main comparisons. The authors can further improve the paper by trying to address the concerns.

-------------------------------------------------
All my concerns are well addressed in the rebuttal, and I would like to keep my initial rating.
limitations_and_societal_impact: There are some limitations that almost all deep learning methods will have, including the input dependence and performance limitations. The detailed limitations of the PNGAN are explained in the supplementary. This work has no limitations and potential negative social impact.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This paper proposes a new method to simulate the real-world noisy images, which seems further improve the performance of current deep denoiser. A slight and effective generator is elaborately designed, and may facilitates the research of noise generation.
Besides, the proposed method combines lots of existing deep learning technologies, and surpasses the current SoTA by a large margin.
main_review: In summary, this work presents one noisy image generation method, and validates its effectiveness by comprehensive experiments to some extent.

1) As fas as I know, this is the first work to employ the pre-trained deep denoiser $D_d$ as a regularizer to constrain the generated noisy images. It is interesting, but not a general regularizer. In other words, I think such strategy will pull the generated noisy image distribution to that of the specific training data used to train $D_d$, which may not be desired by us in some cases.

2) The experiments only involve the comparison with other methods on the denoising task. For the noise generation task, this work does not cite and compare with some important works, such as:

     "Unprocessing Images for Learned Raw Denoising", CVPR 2019.

     "Dual Adversarial Network: Toward Real-World Noise Removal and Noise Generation", ECCV, 2020

3) In Table 4, the generator is trained on SIDD training dataset. After finetuning jointly on the SIDD training and the generated dataset, the performance on SIDD testing dataset has a little drop. So, the trained generator does not exactly capture the noise distribution of the training dataset (SIDD), even though with complex loss function?  

limitations_and_societal_impact: The paper has addressed the limitations and potential negative societal impact in the supplementary material.
review_title: null

Meta-review: This paper proposes a novel idea to synthesize real world noisy images for image denoising.
The main idea is to treat each noisy pixel as a random variable and then split noisy image generation into image domain alignment and noise domain alignment. Based on the idea, a GAN method is proposed to perform the pixel level adversarial training. The idea is interesting and novel, and the effectiveness of the proposed method is demonstrated in the experiments by comparing it with SOTA methods and detailed ablation studies. All the reviewers are also impressed by the idea and the promising experimental results. However, a major drawback of the work is the lack of theoretical guarantee on the distribution gap between the real noisy images and the generated noisy images. Given that this is the first work that moves towards real world noisy image generation,  I recommend acceptance given the novelty and the practical significance of the proposed work.

Decision: Accept (Poster)