Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 5: Marginally below the acceptance threshold
review_content: summary: The paper proposes an adaptive denoising method that improves the performance of the test images by learning multiplicative scaling parameters for each channel in the CNN network. The paper also presents GainTunning test time adaptation of CNN, by performing the experiments where CNN is trained on synthetic data and adapted to real transmission-electron-microscope images.
main_review: - It would be better if the paper presents qualitative analysis on real noisy images
- Could authors explain why the improvements are minor in the table3 and table 4
- It would be better if the paper presents changes in runtime and parameter comparisons due to GainTuning.
- Is there any specific reason for why in the GainTunning only multiplicative scaling parameters are used why not other types of parameters? 
limitations_and_societal_impact: None
review_title: null

final_score: rating: 4: Ok but not good enough - rejection
review_content: summary: The paper considers image denoising in the problem setting recently proposed in the literature, i.e., fine-tuning a pretrained-model on a test image in an unsupervised fashion. The method modifies the optimization procedure of the fine-tuning. 
main_review: I think there are two issues. One is the lack of explanation as to why the proposed method should work. The other is insufficient experimental comparisons. 

The study extends existing studies with their optimization procedures. The extension is to confine the parameters optimized in the fine-tuning stage to the multiplicative scalars of the channels of the layers. The authors state that the existing methods suffer from overfit and employ several remedies to avoid it. They then claim that optimizing much fewer parameters leads to better results than the existing methods.

Two questions then arise. Although it is reasonable that optimizing fewer parameters will contribute to avoiding overfitting, it is unclear why the method should work. Why does optimizing the “gaining parameters” help generalize to novel input distribution? Why isn’t the “regularization” too strong? Without a reasonable explanation, the paper is better regarded as a report of an empirical finding. 

Another question is concerned with the experimental validation. GainTuning shares the application scenario (i.e., either scratch-training a model or fine-tuning a pretrained-model on a test image without label) with many recent methods. Then, it will be necessary to compare against the best-performing method w/o GainTuning. For instance, only Self2Self is considered in the experiments of Fig.3 of Table SM4 etc., but there are several others in the category of self-supervised training a model on an input image. Table SM4 shows the method is worse than Self2Self; it will be worse than more recent methods.

Moreover, we may regard the method as a regularization method. As the authors also state (ll87), the previous studies employ early stopping [52] or a special architecture [56]. There will be many more standard regularization methods that can be performed during the fune-tuning. It may also be necessary to compare with them. 

limitations_and_societal_impact: We first need clarify the contribution of the study. See above.
review_title: null

final_score: rating: 4: Ok but not good enough - rejection
review_content: summary: The goal of the paper is to adapt pretrained denoisers to denoise new images; novel noise distribution or data distribution. The authors, inspired by self-supervised denoisers and methods using channel manipulation, propose to fine-tune the gain of CNN denoiser channels to the test image. This fine tuning is tried out with multiple known losses.
main_review: Originality:
The work is a combination of known techniques, however with limited novelty. To some extent it differs from other methods by the set of parameters it chooses to learn at test time.

Quality:
The submission is technically sound but lacks theoretical analysis/justification and somewhat in terms of the experimental results provided in the main manuscript. Some experimental details also need to be added (see comments).

Clarity:
The submission is clearly written and well organized but could benefit from extended details around the ‘gain’ parameters that are manipulated for various methods. There are also a few typos (see comments).

Significance:
The submission does address a difficult task but the importance of the results is limited and the advancement of the state of the art not demonstrable. 

For in-distribution data, the majority of improvements are smaller than 0.03dB-0.04dB on the BSD68 dataset which is the main benchmark for the BSD400 training set. And such an improvement would be expected since the method is an incremental fine-tuning over pre-trained models.

For out-of-distribution noise, simply using the original denoiser pretrained blindly (full range of noise levels up to 100) is better than using the proposed method (which also requires some extra test time optimization). And similarly using the bias free model of [35] is also better.

For out-of-distribution data, the chosen settings are not very relevant. Urban/natural images are easily obtainable and AWGN is easy to synthesize, so models can be easily pretrained on these distributions. Piecewise constant images as well. The improvements here are more interesting, up to 0.71dB for natural images to scanned documents for instance, however, please add the information on the pretrained model that was used and how it was fine-tuned.

The application to TEM is the most interesting one in my opinion as it comes close to what can be needed in real applications, but it is lightly addressed in the main manuscript. Conclusions are drawn based on one shown visual example and we have no quantitative measures.

Is GainTuning done exactly the same way for all experiments? For all denoiser networks?

Can you provide more results on real image denoising where ground-truth cannot be easily obtained (unlike AWGN)? For instance microscopy imaging.

Lastly, as the authors rightfully mention in their limitations, choosing to fine-tune the gains, or any other subset of parameters, is quite arbitrary, and because the authors provide little support for why they made that choice, the theoretical justification is less strong. How many parameters should be fine-tuned and where is the limit between overfitting and adaptation, is in fact a key question. The whole concept of a “prior” learned from a distribution is questioned by methods that adapt to test images and this should not be taken lightly when merging “prior-based methods” and “self-supervised” methods that work at test time. It would be interesting if the authors look into denoising (or any image restoration) methods that study blind denoising, and priors in denoising. Mixing this with self-supervised techniques is not straight-forward from a theoretical ground. As it stands, the paper exploits an interesting idea but develops it in a somewhat empirical manner, which to me is the main weakness next to the experiments provided in the main manuscript.

Comments:

Figure 4 (Sec 5.2 & 5.3), all your comparisons are against a “pre-trained CNN” denoiser, please specify exactly which denoiser it is, and how it was trained, this information is critical there.

Typo: Figure 4 caption “on test imageS with noise outside”

Typo: L201: “(see Figure 4 and)” there is something missing, you say comparison to pretrained denoisers (plural) while there is only one in Fig 4, so I assume the ‘and’ should refer to SM there.

Typo: L277 “for different architectureS”

Sec 5.2 Why is the ‘gold-standard’ a blind model, rather than a model trained on the same noise level as the test noise level for instance?

limitations_and_societal_impact: Adequately addressed.
review_title: null

final_score: rating: 5: Marginally below the acceptance threshold
review_content: summary: This paper proposes a method to finetune the pre-trained denoising network, which is trained in a supervised manner, in an unsupervised way to adapt the network to the tested noisy image. To avoid overfitting, they propose GainTuning which only adjusts each channel in the convolutional layers. The experimental results show that the proposed method can improve the denoising performance of existing supervised trained networks.
main_review: The idea is simple and easy to understand. However, I have the following concerns:
1. I think one crucial usage for training a denoising network in an unsupervised way is for the noises whose distributions are unknown. For Gaussian noises with unknown noise levels, we can simply train a network to cover a large range of variance and achieve good performance which is shown in the first table in Fig. 4. However, almost all the experiments are for removing Gaussian noises except for the atomic-resolution transmission electron microscope image. There is plenty of real nature image denoising datasets currently, the authors should validate their method in these datasets.
2. The improvement of the proposed method is marginal. In Fig. 3, the differences between 'Pre-trained' with 'GainTunning' are at most 0.12dB and most of the improvements are less than 0.1dB. As to Fig.4, 'Gaintunning' does not perform better than 'Trained on [0,100]'. Even though 'Gaintunning' performs better than 'Pre-trained', the comparison is unfair. Since 'Pre-trained' is trained with smaller noise variance.
3. To avoid overfitting during finetune, the authors propose to use GainTunning. However, there are many other ways to only adjust fewer parameters in the network. For example, we can only finetune some layers in the network. Can the authors explain why GainTunning is the best choice? Also, can the authors provide some visual results to demonstrate how Gain Tunning works after finetuning?
4. Sec. 4 is not relevant to this paper. These losses are proposed by existing arts. Also, the authors do not analyze the pros and cons of these losses in the proposed method.
5. In Ln. 177, should '(3)' be '(2)'?
limitations_and_societal_impact: yes
review_title: null

Meta-review: All four reviewers agreed that the paper was not currently above threshold for acceptance, with a high indicator of confidence in their assessments. 

In addition to the reviewer comments, one reviewer suggested in a note during discussion three areas for improvement that my be helpful for the authors: "Especially: (1) The authors provide some experiments to show the proposed 'Gaintunning' performs better than 'only finetune some layers'. The authors do not explain the reason. (2) The network trained from a larger range of noises levels can perform better than the proposed method. This reduces the contribution of the proposed method. (3) The authors provide additional experiments to remove Poisson noises when pretrained on Gaussian noises. But the proposed method will be more useful if it can work well for real noises."
Decision: Accept (Poster)