Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The paper proposes and evaluates a training method for neural network-based image denoising when raw images and corresponding processed (e.g. demosaiced) rgb images are available. It leverages the known observation that SNR increases when spatially downscaling an image, and thus realistic low and high-noise images can be generated at lower resolution. The method consists of several carefully chosen ingredients, e.g. (1) a scheme for estimating parameters of a Poisson-Gaussian noise model on raw images, (2) a scheme for estimating parameters of a signal processing pipeline converting raw to rgb, (3) augmentations using the estimated parameters, (4) supervised training on noisy/clean low-resolution rgb image pairs simulated from raw images. The method is evaluated for two different neural networks and compared to sota methods that have not been trained or real (hard to get) noisy/clean images. In this comparison the proposed method performs preferably.
strength_and_weaknesses: Strengths of the paper are in the well-designed, easy to understand ingredients of the method and the improved denoising performance wrt. competitors. It is overall well-structured and presents clear experiments. It also provides a rich overview on related work.

Main weaknesses are language issues that sometimes lead to unclear train of thoughts within sentences or paragraphs (see examples below).

clarity,_quality,_novelty_and_reproducibility: Minor hiccups in argumentation exist, e.g.
P6L19 claims ‘To avoid learning identity mapping from the input y to the output \hat{y}, we first adopt image scale augmentation (SA) that changes the image resolution.’ Comparing Figure 4, there is no path from y to \hat{y} allowing to learn identity, even without SA. The next sentence then claims ‘[…] SA randomly samples the scaling factor to prevent the encoder from learning noises’. It should be written clearly, for what purpose SA has been introduced and it would be informative to see, if performance of the method really drops when removing it.

Clarity is mainly reduced due to language issues. E.g. the paper frequently violates the rule that the definite article (the) is used before a noun to indicate that the identity of the noun is known to the reader or before unique nouns. It is somewhat distracting, when this rule is not kept, as it leads the reader to consider if they overlooked a previous mentioning.
Other examples of language issues, notation:

P1L5: ‘Intuitively, downscaling easily removes high-frequency noises than natural textures.’ The word ‘more’ is missing, allowing to use comparison by ‘than’: ‘Intuitively, downscaling removes high-frequency noise more easily than natural textures.’ Same on P5L16: ‘This phenomenon indicates that the true signal z is robust to the pixel variances through downscaling than real noises.’

P1L-10: ‘The first line of works synthesizes the realistic noise from clean images […]’. It remains unclear, how noise can be synthesized from clean images, and if this is really meant. Presumably noisy images are generated, where the noise not necessarily depends on the clean image.

P1L15: ‘synthetic raw images’: no synthetic raw images are used in the paper. Only rgb images are simulated.
P5, paragraph ‘Optimization’: the notation $a \times e^y$ is unusual. Either use $ae^y$ or $a \times 10^y$ .

Novelty: The noise parameter estimation, as well as the estimation of the raw2rgb converter are new, the underlying models are not but this is also not claimed. Using down-sampling for noise reduction in order to then add noise to produce noisy/clean image pairs seems to be new. All simple but effective. However, the observation, that down-sampling increases SNR is not new and obvious in Fourier space, as high-frequency regions have lower SNR in typical images. This observation should therefore be presented less pronounced.

Reproducibility: While the method is clear overall, details of implementation are not. Especially exact network architectures remain unclear. A table, figure, or code would help.

Results in Tables 2 and 3 seem to be inconsistent, as numbers presented for NERDS+D in SIDD (with full augmentation in Table 3) are different.


summary_of_the_review: Overall a paper with merits due to its simple but novel and effective content. However, it suffers from language issues. This brings the paper well into the range of being acceptable, but keeps it from being a ‘good paper’ overall.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: In this paper a learning-based camera denoising framework is introduced. Noisy raw/RGB image pairs are used for training, while the method takes a noisy RGB image only at test time for denoising. The authors designed a framework that models the raw sensor noise and ISP transformed noise separately. 

Instead of relying on paired noisy/clean images for training, the authors exploited the use of image downsampling and optimization-based noise estimation and synthesis to create pseudo noisy/clean image pairs at lower resolution, purely from noisy images. For estimating the Poisson-Gaussian coefficients for the raw sensor noise, a re-parameterization trick is applied and an optimization problem is solved such that the pixel variances can match between synthesized low-res noisy image and the original full-res noisy input. While P-G parameters are often provided by camera manufacturers in metadata, they may not be accurate and the authors proved that the proposed approach leads to more accurate estimation than prior arts. 

For the RAW2RGB conversion, this ISP process is posed as a style transfer problem, where the style is learned from the raw-RGB noisy image pair (at original resolution) and is used to convert the raw image to RGB.  

Once the P-G model (N) and RAW2RGB conversion network T are determined, a given denoiser network can be trained on the synthesized low-res noisy/clean image pairs with a number of data augmentation tricks. The overall objective function can be seen in Eq (8).

Thorough qualitative, quantitive, as well as ablation studies have been conducted on various datasets, demonstrating the effectiveness of the framework, and justifying the design decisions.  


strength_and_weaknesses: Strength:
- Creative way of synthesizing raw noisy/clean image pairs based on the observation that true image texture is more robust to pixel variance through downscaling than real noises. 
- Poses the ISP estimation as a style encoding and transfer problem. This, in combination with the innovation above, greatly relax the dataset requirements on real noisy/clean image pairs, ISPs, metadata etc. which are not always available or accurate especially at test time. 
- While key proposals such as pseudo clean image generation via downsampling may seem to be empirical, the authors made an attempt to justify this with data. The ablation study also seems adequate. 
- NERDS outperforms previous works with simpler denoiser network, demonstrating the effectiveness of the noisy/clean image pair synthesis, style disentanglement-based ISP estimation, as well as the rich data augmentation. 

Weaknesses:
- The work still relies on raw/RGB noisy image pairs for training, although only RGB noisy image is required for denoising at test time. I would recommend that the authors rethink the title, "NERDS: A GENERAL FRAMEWORK TO TRAIN CAMERA DENOISERS FROM SINGLE NOISY IMAGES", as the claim on training from single noisy images may be ambiguous and potentially misleading (it actually requires both raw and RGB noisy images for training and testing ideally, thus the "s" in "images"). 
- While the authors have made an attempt to explain why the pseudo clean image (with residual noises) works in practice due to various observations / hypothesis, it may add more insights if ablation study can be conducted to assess the relative contribution from factors such as blurring strengths, downscaling factor, augmentation for noise parameter, scale/intensity, and style etc.
- Although it is claimed by the authors that PG parameters in metadata may not be accurate, and that NERDS-raw does a better job, the audience may still appreciate more justification on this. For example, what is the impact of inaccurate PG parameters on the overall RGB denoiser performance? 
- On reproducibility, it would be helpful if the authors could clarify if the PG optimization, ISP network T training, and denoiser D training are done subsequently or jointly.  
- I recommend that the authors consider referencing the supplementary material in the main paper, as the audience may walk away with many questions just by reading the main paper only. 
clarity,_quality,_novelty_and_reproducibility: The reviewer would like to appreciate the quality and novelty of this paper. 

On clarity and reproducibility:
- See my comments in the previous question on optimization / model training procedure. 
- One question that remains is the modern ISPs may contain denoising block. As the RAW2RGB process is learned from real noisy raw/RGB image pairs, it is possible that some degree of denoising is already learned implicitly in T, which potentially may impact the denoising strength of D. 
- Generalizability: please clarify if at test time, D needs to be fine-tuned / retrained for device specific noise levels and ISPs. Based on Sec A.1.3 it looks like this is required and time-consuming to get the best denoising performance, meaning that we will also need to access the noisy RGB/raw image pair at test time training. If so, please clarify if the comparison in table 2 was done fairly, i.e. did the other methods have access to training set at test time? 



summary_of_the_review: While minor clarity issues remain, NERDS is a novel and effective framework for camera denoising. The main drawback (and overselling) though, seems to be the claim on requiring single noisy images only -- in practice, both the raw and RGB noisy images are required to get the best performance out of the denoiser. Please consider making the problem setting and contribution more concrete and transparent. Please also consider referencing some key justifications in the supplementary material from the main paper, as the readers may otherwise believe some design decisions lack sufficient justification (empirical, theoretic). As such, I recommend a borderline accept (6) for the work, and lean towards score 7 if these can be addressed in the revision.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper presents a novel method for training image denoising techniques, called NERDS.  The paper makes the observation that a noisy image can be downsampled to produce a pseudo-clean image, as the downsampling operation removes noise.  Based on the noise statistics between the psudeo-clean image and the original image, a noise model can be estimated in a learnable way using a reparametrization trick.  The paper also models the “black box” of an ISP disentangling style, and thereby is able to generate large amounts noisy/pseudo-clean pairs using the model and data augmentation.  These pairs are then used for training denoising methods like DnCNN.  Experimental results show the method outperforms state of the art methods.
strength_and_weaknesses: Denoising of images is a fundamental problem in computer vision and has seen considerable attention in the last decade particularly with the advent of deep learning.  However, as argued by the paper, supervised methods require paired noisy/clean data which is difficult to obtain.  The paper addresses this problem in an appealing way by only requiring noisy images – which are easily acquired.  A clean counterpart isn’t needed, which greatly simplifies data collection for training a denoiser.  This approach is used to synthesize noisy RGB images.

Strengths:

1.	The paper’s approach that only requires noisy RGB images as input.  This is appealing as capturing paired data often involves tedious work.  The method is able to produce noisy RGB images for RGB denoisers.  This reviewer appreciates the approach of trying to generate pairs from noisy images. 

2.	The reparametrization trick to estimate the noise parameters, while simple, was interesting and something new for this reviewer.  This could be relevant for other researchers working in noise modelling and estimation. 


3.	The modelling of the ISP, whilst not entirely new, was useful in this paper to produce noisy RGB images for training an RGB denoiser.  The disentangling of style was appreciated by this reviewer.

4.	The experimental results compare the method to a variety of recent methods, with the method producing convincing results.  

Weaknesses:

1.	This reviewer is concerned about the downsampling technique to create “clean” images.  Clearly the images aren’t clean – for example in Figure 1, the “clean” image in part (d) is much noisier than the clean image in part (a) of the figure.  So there is remaining noise after downsampling.  The paper doesn’t really discuss this issue, although the supplementary material provides some context.  Downsampling can’t remove the image noise because it applies a low pass filter (averaging filter) to the data.  So the noise is reduced through the averaging, but not removed.  Therefore, this reviewer would prefer if the in the main paper, instead of calling it a “clean” image, the downsampled image was called a “pseudo-clean” image as done in the supplementary.  More importantly, it would be helpful if the paper could comment on the impact of this remaining noise in the downsampled image – does it impact the results negatively?  Would downsampling by a factor of 4 help more than by a factor of 2?  This reviewer was somewhat surprised to see the results in Table 1, it does appear NERDS can produce a good fit despite the remaining noise.  Perhaps the author rebuttal could comment on this point. 

2.	Noise is harder to remove in the RGB domain than the RAW domain as the statistics are simpler in the RAW domain before the noise becomes more spatially correlated due to demosacing and other operations.  Why develop a denoiser to run in the RGB domain? Conceivably the method could be used to denoise in the RAW domain.  Relatedly, it could be interesting to compare the approach to the Brooks et al. paper from CVPR 2019.


3.	The reviewer is also worried about the domain gap between a “black box” ISP and the ISP that is used in the paper.  Clearly there may be a domain gap as the ISP used in the paper is only an approximation to a “black box” ISP.  Does this domain gap cause any limitations in the method?  Perhaps the proposed method does not generalize well to some phone models.

4.	Why call it NERDS?  Is this an acronym for something?

clarity,_quality,_novelty_and_reproducibility: Generally the paper is clearly written and of good quality.  There are some small issues listed below.  

The main originality comes from the noise modelling approach through downsampling and noise model fitting.  This appears to be a good advance although the reviewer has some concerns about remaining noise articulated above.

It wasn’t clear if code will be released should the paper be accepted.  Perhaps the author rebuttal could comment on this.

Smaller issues: 

•	Figure 1 caption, extra period, please remove

•	Page 2, please replace “an other” to “another”


•	Page 2, footnote, please replace “Sonny” with “Sony”

•	Page 3, please replace “RELATED WORKS” with “RELATED WORK” as work is already plural


•	Page 2, please replace “test noises” with “test noise” as noise is not countable.  This issue is also on Page 4, “unseen noises” with “unseen noise”; Page 6 “learning noises” with “learning noise”

•	Page 4, it’s true modern smartphones provide multiple image styles; note however these can be applied even without user edits.  The camera will recognise the scene (e.g. beach) and optimise the ISP for that scene.  Therefore the qualifier “with user edits” is not needed. 


•	Why are the results so poor for SCUNet in Table 2?

summary_of_the_review: In summary, this paper is addressing a fundamental computer vision problem of removing noise from captured images.  It brings some new ideas in terms of how to model the noise from a noisy image.  Whilst the formulation might not be perfect, it provides a practical approximate way to estimate the noise which is appealing as it can be used to generate noisy/clean pairs for image denoising.  The results are appealing as the method outperforms papers recently appearing in the literature.  Overall I think the paper has merit but it’s let down a bit by overstating assumptions and remaining gaps (e.g. residual noise in the “clean” image, differences between real and “blackbox” ISPs).  Still this reviewer believes there are some interesting ideas here.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This paper proposes a method for real image denoising. Specifically, it firstly estimates Gaussian and Poisson noise parameters. Then, it learns a style encoder to map a raw image into rgb one. In this way, it can synthesize noisy-clean image pairs to train the denoising network. The provided experiments show the proposed method performs better than existing methods.
strength_and_weaknesses: Strength:
The provided experimental results show it outperforms better than other methods.

Weaknesses:
1. Even though Table 1 shows the proposed method can estimate noise parameters accurately and Figure 7 shows downscaling with blurring can make the variance difference between clean and noisy image closer, there lacks theoretical proof why the proposed method can accurately estimate noise parameters and analyzing the possible deviations in the proposed method. According to Figure 7, the variance of noisy images is still higher than clean ones. In this way, the estimated noise parameters based on Eq. 3 should be smaller.
2. In Figure 7, why will downsampling the clean image without blurring not change the variance of the image obviously? What is the downsampling method used? Why will downsampling 2 times have a smaller variance in (a) and (c)?
3. It is not clear whether the noise parameters are estimated for every image or for the whole dataset. According to my understanding, the proposed method estimates fixed Gaussian and Poisson parameters for the whole dataset. This is not correct. Since these parameters depend on the sensor gain or ISO. In this way, the experiment in Table 1 seems unfair. Since other methods estimate noise level image by image while the proposed one estimates it according to the whole dataset.
4. Also, using reparameterization to estimate noise parameters may be too complicated. Why not directly search all possible Gaussian and Poisson parameters and find the settings to minimize Eq. 3?
5. It seems the blur kernel and downsampling rate are important for the noise level estimation. Can the authors demonstrate how the proposed method selects the settings? More experiments are needed.
6. For isp estimation, the proposed method seems to use spatially-invariant linear operators. It may be not that accurate. In the camera isp, the processing is spatially variant e.g. lens shading, local tone mapping, contrast enhancement, etc. In addition, the number of parameters in camera isp is much larger than 3 which is the number of style parameters in the proposed method. I want to know the accuracy of the generated rgb images compared with the ground truth.
clarity,_quality,_novelty_and_reproducibility: There are many concerns in the novelty of the proposed method. Please see the weaknesses above.
summary_of_the_review: Even though the provided experimental results are better than other methods, there exist plenty of concerns regarding noise level estimation as well as isp estimation. The authors should address them completely.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: The paper proposes a framework to train denoisers based on pairs of raw and RGB images. The method only relies on noisy images. The paper shows that the method enables training CNN-based denoisers well, relative to training with other self-supervised losses that do not rely on having seen clean images for training. 

Strength:
The paper provides a well-designed framework for self-supervised learning of denoisers, based on interesting ideas such as the particular synthesis of raw/clean image pairs.


Weaknesses:
- Some language issues, concrete examples are given by R1. Most have been fixed. 
- Both raw and RGB noisy image pairs are required at training, while only noisy RGB images are used at inference. This was initially not made sufficiently clear but has been clarified better in the revised version. I recommend to also explain early on in the paper what raw and RGB noisy image pairs are, and how they are obtained in practice, this is not sufficiently clear in the revised version.
- The paper does not provide a proof stating under which assumptions the estimation of the noise parameters (e.g., the variance) is sound. Adding such as proof would add value to the paper, since it would illustrate under what assumptions the method is sound and when it is not.


justification_for_why_not_higher_score: The paper introduces an effective method that relies on new ideas and a variety of carefully selected old ones. However, since many elements of the method (such as the downsampling idea) are well known in the community, since the method actually relies on using image raw+RGB image pairs (which is a slight weakness in terms of the significance of the setup considered), and since the paper could be better written and presented, this is not a paper for a spotlight or an award. 


justification_for_why_not_lower_score: The paper provides a well-designed framework for self-supervised learning of denoisers, based on interesting ideas such as the particular synthesis of raw/clean image pairs, and shows with well-designed comparisons and ablation studies that the method works well.
Decision: Accept: poster