Adaptive Denoising via GainTuning
Authors: Sreyas Mohan
Joshua L. Vincent
Ramon Manzorro
Peter Crozier
Carlos Fernandez-Granda
Eero P Simoncelli
Venue: NeurIPS 2021 Poster (2021)
Keywords: denoising
adaptation
out of distribution
generalization
gain

Abstract: Deep convolutional neural networks (CNNs) for image denoising are typically trained on large datasets. These models achieve the current state of the art, but they do not generalize well to data that deviate from the training distribution. Recent work has shown that it is possible to train denoisers on a single noisy image. These models adapt to the features of the test image, but their performance is limited by the small amount of information used to train them. Here we propose "GainTuning'', a methodology by which CNN models pre-trained on large datasets can be adaptively and selectively adjusted for individual test images. To avoid overfitting, GainTuning optimizes a single multiplicative scaling parameter (the “Gain”) of each channel in the convolutional layers of the CNN. We show that GainTuning improves state-of-the-art CNNs on standard image-denoising benchmarks, boosting their denoising performance on nearly every image in a held-out test set. These adaptive improvements are even more substantial for test images differing systematically from the training data, either in noise level or image type. We illustrate the potential of adaptive GainTuning in a scientific application to transmission-electron-microscope images, using a CNN that is pre-trained on synthetic data. In contrast to the existing methodology, GainTuning is able to faithfully reconstruct the structure of catalytic nanoparticles from these data at extremely low signal-to-noise ratios. 

Adaptive Denoising via GainTuning
Sreyas Mohan', Joshua L. Vincent", Ramon Manzorro', Peter A. Crozier 2
Carlos Granddl 1.3 Eero P. Simoncelli 1,3,4
Center For Data Science, NYU, Sccooo for Engineering of Matter, Transport and Energy, ASU CCurant Institute of Mathematical Sciences, NYU *Center for Neural Science, NYU and Flatiron Institute, Simons Foundation
Abstract
Deep convolutional neural networks (CNNs) for image denoising are typically trained on large datasets. These models achieve the current state of the art, but they do not generalize well to data that deviate from the training distribution. Recent work has shown that is possible to train denoisers on a single noisy image. These models adapt the features of the test image, but their performance is limited by the small amount of information used to train them. Here we propose "Gain' "aain"" a methodology by which CNN models pre-trained on large datasets be adaptively and selectively adjusted for individual test images. To avoid overfitting. Gain Tuning optimizes single multiplicative scaling parameter (the "Gain") of each channel in the convolutional layers of the CNN. We show that GainTuning improves state-of-the-art CNNs standard image-denoising benchmarks, boosting their denoising performance on nearly every image in a held-out test set. These adaptive improvements are even more substantial for test images differing systematically from the training data, either in noise level or image type. We illustrate the potential of adaptive GainTuning in a scientific application to transmission microscope images, using a CNN that is pre-trained on synthetic data. In contrast to the existing methodology. GainTuning able faithfully reconstruct the structure of catalytic nanoparticles from these data at extremely low signal-to- signal-to-noise ratios.
Introduction
Like many problems in image processing, the recovery of signals from noisy measurements has been revolutionized by the development of convolutional neural networks (CNNs)
66
8
67 These
models are typically trained on large databases of images, either in a supervised 37 66
68
67]
noisy test images. This approach achieves state-of of-the-arrrr- performance when the test images and the training data belong to the same distribution. However, when this is not the case, the performance of these models is often substantially degraded [59 37 68) This is an important limitation for many practical applications, in which it is challenging (or even impossible) gather training dataset that comparable in noise and signal content to the images encountered at test time. Overcoming this limitation requires adaptation to the test data.
or unsupervised fashion 62 27 29). Once trained, these solutions are evaluated
recent unsupervised method (Self2Self) has shown that CNNs can be trained exclusively on individual test images, producing impressive results 46) Despite this, the performance of Self2Self limited by the small amount of available training information, and generally inferior to CNN models trained on large databases.
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

(a) training on
(b) Adaptive training image
(c) Combined pee training adupiive training
image
image
Test image
Training data
Training data
CNN
CNN
Denoised image
Denoised
image
Denoised image
Figure 1: Proposed denoising paradigm. (a) Typically, CNNs trained large dataset and evaluated directly on a test image. (b) Recent unsupervised methods perform training on a single test image. (c) We propose GuinTuning, a framework which bridges the gap between both of these paradigms: CNN pre-trained on a large training database is adapted to the test image.
In this work, we propose GainTuning, a framework to bridge the gap between models pre-trained on large datasets, and models trained exclusively on test images. In the spirit of two recent meth- ods [55 [59. GainTuning adapts pre-trained CNN models to individual test images by minimizing an unsupervised denoising cost function, thus fusing the generie capabílities obtained from the training data with specific refinements matched to the structure of the test data. Rather than adapt the full parameter set (filter weights and additive constants) the test image, GainTuning instead optimizes single multiplicative scaling parameter (the "Gain") for each channel within each layer of the CNN. The dimensionality of this reduced parameter set is small fraction (+s 0.1 % our examples) that of the full parameter set. We demonstrate through extensive examples that this prevents overfitting to the test data. The Guinnuning procedure is general, and can applied to any CNN denoising model, regardless of the architecture pre-true eettttaining process.
Our contributions. GainTuning provides a novel method for adapting CNN denoisers trained on large datasets to single test image. Gain'l Tuinn improves state-of-the-ard CNNs on standard image- denoising benchmarks, boosting their denoising performance on nearly every image held-out test sets. Performance improvements are even more substantial when the test images differ systematically from the training data. We showcase this ability through controlled experiments in which we vary the distribution of the noise and image structure of the test data. Finally, we evaluate Gain real scientific-imaging application where adaptivity is crucial: denoising transmission-electre mieroscope data at extremely low signal-to-noise ratios. As shown in Figure 2 both CNNs pre-trained on simulated images and CNNs trained only on the test data produce denoised images with substantial artefacts. In contrast, Gain achieves effective denoising, accurately revealing the atomic structure in the real data.
Related Work
Denoising via deep learning. the last five years, CNN-based methods have clearly outperformed
previous state-of-the-ard denoising methods 13 53
5
55
14 21
[T]
Denoising CNNs are typically
trained in a supervised fashion, minimizing mean squared error (MSE) over a large database of
example ground-truth clean images and their noisy counterparts [66
37
Unsupervised methods
have also been developed, which do not rely on ground-truth images. There are two main strategies to achieve this: use of an empirical Bayes objective, such Stein's unbiased risk estimator (SURE) [I3
33 4
36 55 (561, and architectural "blind-spot" methods detailed description).
(see Section for a more
Generalization to mnt-o--distribution noise. Previous studies have shown that CNN denoisers fail to generalize when the noise encountered at test time differs from that of the training data 68 37 Ref. 377 proposes the use of a modified CNN architecture without additive bias terms, which able to generalize to noise with variance well beyond that encountered in the training set. Here, we show

---

(b) Unsupervised training only on (a)
(c) Supervised training on simulated datu [3]
(d) GainTuniig CNN trained on data
(e) Estimated reference
(a) Noisy image
image
Figure 2: Denoising results for real-world data. (a) An experimentally eeeeriuenttaly-aquireeddd acquired resoucinn transmission electron microscope image of a Ce02-supported Pt nanoparticle. The image has a very low signal noise ratio (PSNR of xz 3dB). (b) Denoised image obtained using Self2Self [46]. which fails to reconstruct three atoms (blue arrow, second row). (c) Denoised image obtained via a CNN trained a simulated dataset, where the pattern of the supporting atoms is not recovered faithfully (third row), (d) Denoised image obtained by adapting the CNN in (c) to the noisy test image in (a) using GainTuning. Both the nanoparticle and the support are recovered without artefacts. (e) Reference image, estimated by averaging 40 different noisy images of the same nanoparticle.
that augmenting generic architecture with GainTunnig yields comparable performance removing bias.
Generalization to -ittr--distribution images. In order to adapt CNNs to operate on test data with characteristics differing from the training set, recent publications propose fine-tuning the networks using an additional training dataset that is more aligned with the test data (59
[8] This is form However, it often
of transfer learning, a popular technique in classification problems challenging to obtain relevant additional training data. Here, we show that GainTuning can adapt CNN denoisers to novel test images.
12 34
Feature normalization. Normalization techniques such batch normalization (BN) 23 are standard component of deep CNNs. BN consists two stages: (1) centering and normalizing the features corresponding each channel, (2) scaling and shifting the normalized features using two learned parameters per channel (a scaling factor and a shift). The scaling parameter is analogous to the gain parameter introduced in GainTuning. However, in BN this parameter is adjusted during training and fixed during test time, whereas GainTuuiin Tuning adjusts it adaptively. for each test image.
Gain normalization. Motivated by gain control properties observed in biological sensory neurons [5] adaptive local normalization of response gains has been previously applied in object recognition [24 density estimation [1] and compression [2]. In contrast to these approaches, which adjust gains based on local responses, GainTTuning adjusts global gain for each channel by optimizing an unsupervised objective function.
Adapting CNN denoisers test data. Two recent publications have developed methods of adapting CNN denoisers to test data [56] [59] Ref. [56) include the noisy test images in the training set. In a recent extension, the authors fine-tune a pre-trained CNN on single test image using the SURE cost function 55 Ref. [59] does the same using a novel cost function based on noise resampling (see Section for a detailed description). As shown in Section (ectionE]ffinettt ffine-tuning the full set of CNN parameters using only a single test image can lead to overfitting. Ref. (55 avoids this using early stopping, selecting the number of fine-tuning steps beforehand. Ref. [59] uses a specialized architecture with reduced number of parameters. Here, we show that several unsupervised cost functions can be used to perform adaptation without overfitting, as long as we only optimize a subset of the model parameters (specifically, the gain of each channel).
3

---

Adjustment of channel parameters to improve generalization in other tasks. Adjustment of channel parameters, such as gains and biases, has been shown to improve generalization multiple machine-learning tasks, such as the vision-language problems 4 [il. image generation 77 style transfer [[7]. and image restoration [20] In these methods, the adjustment carried out while
training the model by minimizing supervised cost function. In image classification, recent studies
have proposed performing adaptive normalization 225 11 [1
and optimization [6]] of channel parameters during test time, in the same spirit as Gain
3 Proposed Methodology: Gain
In this section we describe the GainTTuning framework. Let fo be CNN denoiser parameterized by weight and bias parameters, 0. We assume that we have available training database and a test image Yiest that we aim to denoise. First, the networks parameters are optimized on the training database
0 pre-trained arg min
Σ
Spre-tranning (v. fo()).
Ertraining database
The cost function frr training used for pre-training can be supervised, if the database contains clean and noisy examples, or unsupervised, if it only contains noisy data.
direct method of adapting the pre-trained CNN the test data is to finetune all the parameters, as is done in all prior work on test-time adaptation [59 55 18. Unfortunately this can lead to overfitting the test data (see Section minimize the unsupervised cost function without denoising the noisy test data effectively. This can be avoided some extent by employing CNN architectures with a small number of parameters [59], or by only optimizing for short time ("early stopping") [55]. Unfortunately, using a CNN with reduced parameters can limit performance (see Section (5) and it is unclear how to choose single criterion for early stopping that can operate correctly for all test images. Here, we propose a different strategy: tuning a single parameter (the gain) in each channel of the CNN. GainTuning can be applied to any pre-trained CNN.
Due the large number of degrees of freedom, the model able to
We denote the gain parameter of the dh channel of the the 1 layer as 7/,, c], and the conventional parameters of that channel by pre-trrinee [I (a vector containing the filter weights). The adapted Gain parameters are the product of these:
aaiin (1) = c[1,c] c) Pre trained [l.
We estimate the gains by minimizing an unsupervised loss that only depends on the noisy image:
arg min
LaainTuning (ress GainTaning (?))
(3)
The final denoised image S (9) (y (Vies). Section 4 describes several possible choices for the cost function Lnain Since we use only one scalar parameter per channel. the adjustment performed by GainTuning is very oiw-dimesiional (= 0.1% of the dimensionality of 6). This makes optimization quite efficient, and prevents overfitting (see Section[E) Further, in Section E we show that performing TainTuning provides better performance when compared to fine-tuning only the last few layers of the pre-trained network.
4
Cost Functions for GainTunning
critical element of Gain Tuning is the use of an unsupervised cost function, which is minimized in order to adapt the pre-trained CNN to the test data. Here, describe three different choices, cach of which are effective for the Guinnuning framework, but which have different benefits and limitations.
Blind-spot loss. This loss measures the ability of the denoiser to reproduce the noisy observation, while excluding the identity solution. To achieve this, the CNN must estimate the jth pixel y of the noisy image y as function of the other pixels V[j)e, excluding the pixel itself. As long as the noise degrades pixels independently, the network to learn a nontrivial denoising function that exploits the relationships between pixels arising from the underlying clean image(s). The resulting loss can be written as
Lolind-spox (y, 0) = [(fo(y())) 31
(4)

---

Here the expectation is over the data distribution and the selected pixel. This "blind spot" can be enforced through architecture design [29], or by masking G (see also 46 and 622 for related approaches). The blind-spot loss has a key property that makes it very powerful in practical applications: it makes no assumption about the noise distribution beyond pixel-wise independence. When combined with Gain Mainnuniig it achieves effective denoising of real electron-microsoope data at very low SNRs (see Figure 2
and Section 5.4 F.5
Stein's ' Unbiased Risk Estimator (SURE). Let x be an N-dimensional ground-truth random vector x and let y := x + n be a corresponding noisy observation, where n ~ (O I. SURE provides an expression for the MSE between x and a denoised estimate fo(y), which only depends on the noisy observation y:
20 N
N
a(f(y))a) 0y
E
x
fo(y))?
= EE byy fo(y))1 -2
LSURE (,,), 0). (5)
The last term in Equation S is the divergence of fe. which can be approximated using Monte Carlo techniques 47 (Section D). The divergence is the sum of the partial derivatives of each denoised pixel with respect to the corresponding input pixel. Intuitively, penalizing it forces the denoiser to not rely heavily on the jth noisy pixel to estimate the jth clean pixel. This is similar to the blind-spot strategy, with the added benefit that the jth noisy pixel is not ignored completely. To further illustrate this connection, consider linear convolutional denoising function fy(y) 0 @ y. where the center-indexed parameter vector is = [0_k; k, Oktl,..... 0..............1..... 0x-1, cost function (Equation reduces to
The SURE
Em N
yy 0 @ yl3 - o² 20²00
(6)
The SURE loss equals the MSE between the denoised output and the noisy image, with penalty on the "self" pixel. As this penalty is increased, the self pixel will be ignored, so the loss tends towards the blind-spot cost function. When integrated into the Gain framework, the SURE loss is limited to additive Gaussian noise, for which it outperforms the blind-spot loss. Extensions of SURE to many other stochastic observation models have been developed 49 and may offer alternative objectives for GainTuning.
Noise Resampling. Ref. (59 introduced novel procedure for adaptation which we call noise resampling. Given a pre-trained denoiser fe and a test image y. first one obtains an initial denoised image by applying fe to y, & := ff (y). This denoised image is then artificially corrupted & by simulating noise from the same distribution as the data of interest to create synthetic noisy examples. Finally, the denoiser is fine-tuned by minimizing the MSE between & and the synthetic examples. If we assume additive noise, the resulting loss is of the form
Lauiss resampling (y, O) E,, [1l(f((*+]] - 1
Noise resampling is reminiscent of Refs. 40 63 which add noise to an already noisy image. When integrated in the GainTuning framework, we find the noise-resampling loss results in effective denoising in the case of additive Gaussian noise, although it generally underperforms the SURE loss.
5 Experiments and Results
We performed three different types of experiment to evaluate the performance of GainTuning In- distribution (test examples held out from the training set); out-o--distribuuti noise (noise level or distribution of test examples differs from training set); and out-of-distr signal (test images differ in features or context from the training set). We also apply GainTuning to real data from transmission electron microscope.
Our experiments make use of four datasets: The BSD400 natural image database 34 with test sets Set12 and Se168 (66), the Urban100 images of urban environments [22] the IUPR dataset of scanned documents 4 and a set of synthetic piecewise constant images 31 (see Section B). We demonstrate the broad applicability of GainTuning by using it in conjunction with multiple architectures for image denoising: DnCNN (66. BFCNN 571, UNet [50) and Blind-spot net
29 (see Section A Finally, we compare our results to several benchmarks: (1) models trained on the training database,
(3) CNN models adapted by fine-tuning all parameters (as opposed to just the gains), (3) model trained only on the test data, (4) LIDIA, specialized architecture and adaptation strategy proposed
in 59
We provide details training and optimization in Section C
5

---

Set12
BSD68
Model
30
40
60
30
40
50
GainTuning
DrCNN
Pre-trained
29.52
28.21
27.19
28.19
27.16
26.27
GainTuning
29.62
28,30
27.29
28.47
27.23
26.33
Imorovement PSNR
Pre-trained
29.34
28.05
27.05
28.27
27.05
26.15
UNet
Gain Tuning
29.46
28.15
27.13
28.34
27.12
26.22
Baseline
Pre-trained
29.46
27.95
26.58
28.24
26.91
25.74
LIDIA
Adapted
29.50
26.95
28.23
26.97
26.02
Sel12SeIr
29.21
27.80
26.58
27.83
26.67
25.73
0.0
Improvement
6.1
0.2
0.3
Figure 3: Gain achieves state-of-the-art performance. (Left) The average PSNR on two test set of generic natural images improves after Gain using SURE loss function for different architectures across multiple noise levels. The CNNs are trained on generie natural images (BSD400). (Right) Histograms of improvement in PSNR achieved by DrCNN over test images from Set12 (top) and BSD68 (bottom) 30.
5.1 GainTuniig surpasses state-of-the-art performance for in-distribution data
Experimental set-up. We use BSD4000, standard natural-image benchmark, corrupted with Gaus- sian white noise with standard deviation a sampled uniformly from [0, 55] (relative to pixel intensity range 00, 255)). Following [66), we evaluate performance on two independent test sets: Set12 and BSD68, corrupted with Gaussian noise with a E {30, 40,50), 50).
Comparison to pre-trained CNNs. GainTunnig consistently improves the performance of pre- trained CNN models. Figure shows this for two different models, DnCNN [66] and UNet [50] (see also Section F.T. The SURE loss outperforms the blind-spot loss, and is slightly better than noise resampling Table 7. The same holds for other architectures, as reported in Section E. On average the improvement is modest, but for some images it is quite substantial (up to 0.3 dB in PSNR for 30, see histogram in Figure 5
Comparison to other baselines. GainTuning outperforms fine-tuning based on optimizing all the parameters for different architectures and loss functions (see Section DD. Gain'Tnning clearly outperforms a Self2Self model, which is trained exclusively on the test data (Figure 3 It also
outperforms the specialized architecture and adaptation process introduced in [59] with a larger gap in performance for higher noise levels.
5.2 GainTuning generalizes to new noise distributions
Experimental set-up. The same sct-up as Section 5.1 is used, except that the test sets are corrupted with Gaussian noise with {70, 80) (both beyond the training range of a E [0, 55]).
Comparison to pre-trained CNNs. Pre-trained CNN denoisers fail to generalize in this setting. Gain Tuning consistently improves their performance (see Figure
The SURE loss again outperforms the blind-spot loss, and is slightly better than noise resampling (see
Section [.2).
The same holds for other architectures, as reported in Section F.2 The improvement
in performance for all images is substantial (up to 12 dB in PSNR for 80, see histogram in
Figure
Comparison to other baselines. GainTuning achieves comparable performance to gold-standard CNN trained with supervision at all noise levels (Figure 4. GainTuning matches the performance of a bias-free CNN [ specifically designed to generalize to out-of-distribution noise (Figure 4 GainTuning outperforms fine-tuning based on optimizing all the parameters for different architectures and loss functions (see Section
E) Gain GainTuning clearly outperforms Self2Self model trained
exclusively on the test data (Section F22. and the LIDIA adaptation method (59)
Gaussian to Poisson generalization: Section F22 and Figure show that GainTuning can effectively
adapt a CNN pre-trained for Gaussian noise removal to restore images corrupted with Poisson noise as well.

---

Out-of-distribution test noise
Tost set
Trained E [o, 55]
Bias Free
Trained on
20%
BSD a 70
Pre trained
Gaintuming
Model Ri
E [o, 100]
Fraction
70
22.45
25.54
25.59
25.50
10%
Set12
BSD68
70
22.15
24.89
24.87
24.88
0%
0.0
2.0
4.0
6.0
8.0
10.0
12.0
18.72
24.14
24.38
24.36
Improvement PSNR
Out-of--istributioo test image
Fraction Deta
40%
Training dain
Test data
Pre: trained
Gaintuning
30%
(b)
20%
(c)
Piocewise constant
Natural images
27.31
28.60
10%
(b)
Natural images
Urban images
28.35
28.79
0%
(c)
Natural images Scanned documents
30,02
30.73
0.0
0.5
1.0
2.0 2.5
3.0 3.5
Improvement in PSNR
Figure 4: Tuning generalizes to out-of-distribution data. Average performance of a CNN trained denoise noise levels E [0, 55] improves significantly test image with noise outside the training range, a = 70, 80 (top) and on images with different characteristics than training data (bottom) after GainTuning. Capability of GainTuning
generalize to out-of-distribution noise is comparable to that of Bias-Free CNN 3 which is an architecture explicitly designed to generalize noise levels outside the training range, and that of denoiser trained with supervision at all noise levels. (Right) Histogram showing improvement in performance for each image in the test set. The improvement substantial across most images, reaching nearly 12dB improvement in one example. For these examples, the denoiser was DrCNN (with additive bias terms) and the Gainng loss function was SURE. See Section for experiments with other CNN architectures and loss functions.
5.3 GainTuning generalizes to out-of-distribution image content
Experimental set-up. We evaluate the performance of GainTuning on test images that have different characteristics from the training images. We perform the following controlled experiments:
(a) Simulated piecewise constant images + Natural images. We pre-train CNN denoisers on simulated piecewise constant images. These images consists of constant regions (of different intensities values) with the boundaries having varied shapes such circle and lines with different orientations (see Section B for some examples). Piecewise constant images provide a crude model for natural images B5 B1I We use GainTuning to adapt a CNN trained on this dataset to generic natural images (Set12). This experiment demonstrates the ability of Gain GainTuning Tuning to adapt from a simple simulated dataset to a significantly more complex real dataset.
(b) Generic natural images + Images with high self-similarity. We apply GainTuning to adapt a CNN trained on generic natural images to images in Urban100 dataset. Urban 100 consists images of buildings and other structures typically found in an urban setting, which contain substantially
repeating/periodic structure (see Section B than generic natural images.
(c) Generic natural images Images of scanned documents. We apply GainTuning to adapt a CNN trained on generic natural images to images of scanned documents in IUPR dataset (see Section
B
All CNNs were trained for denoising Gaussian white noise with standard deviation o e [0, 55] and 30.
evaluated at
Comparison to pre-trained CNNs. GainTuning consistently improves the performance of pre- trained CNNs in all the three experiments. Figure Showss shows this for DnCNN when Gain'Tunigg is based on SURE loss. We obtain similar results with other architectures (see Section F3 In experiment (a). all test images show substantial improvements over the pre-trained results (average increase of roughly 1. 1.3dB, and best case more than dB, at a 30). We observe similar trends for experiments
(b) and (c) as well, with improvements being better on an average for experiment (c). Note that we obtain similar performance increases when both image and noise are distriiiiitiiitt as discussed
in Section !
7

---

Comparison to other baselines. In experiment (a), GainTuning outperforms methods that optimize all parameters over different architectures and loss functions (Section E However, Self2Self trained only on test data outperforms Gain Tuninngin this case, because the test images contain content that
differs substantially from the training images. Self2Self provides the strongest form of adaptation,
since it is trained exclusively on the test image, whereas the denoising properties of
are partially due to the pretraining (see Sections 7 FF3). p. We did not evaluate LIDIA [59] for this experiment. For experiments (b) and (c), training all parameters clearly outperforms Gain Guning for case (b), but has similar performance for (c). GainTuning outperforms LIDIA on experiments (b) and (c). Self2Self trained exclusively on test data outperforms Gain TiiinTTiingnnn LIDIA) on (b) and (c)
(see Sections
5.4 Application to Electron microscopy
Scientific motivation. Transmission electron microscopy (TEM) is a popular imaging technique in
Recent advancements in TEM enable to image high frame rates [ These images can for example capture the dynamic, atomic-level rearrangements of catalytic systems [57]
materials science [54] [X
15
32 9 which is critical to advance our understanding of functional materials. Acquiring image series at such high temporal resolution produces data severely degraded by shot noise. Consequently, there is an acute need for denoising in this domain.
30
The need for adaptive denoising. Ground-truth images are not available TEM, because measuring at high SNR is often impossible. Prior work has addressed this by using simulated training data 33
60
whereas others have trained CNNs directly on noisy real data
Dataset. We use the training set of 5583 simulated images and the test set of 40 real TEM images from
35 60
The data correspond a catalytic platinum nanoparticle on CeO₂ support (Section B).
Comparison to pre-trained CNN. A CNN 2 pre-trained on the simulated data fails to reconstruct the pattern of atoms faithfully (green box in Figure (c), (c)). GainTuning applied to this CNN using the blind-spot loss correctly recovers this pattern (green box in Figure 2 (d). (e)) reconstructing the small oxygen atoms in the CeO2 support. GainTuuning with noise resampling failed to reproduce the support pattern (probably because it is absent from the initial denoised estimate) (Section F.5
Comparison to other baselines. GainTuning clearly outperforms Self2Self, which is trained exclu- sively on the real data. The denoised image from Self2Self shows missing atoms and substantial artefacts (see Section
[:5). We also compare Gain dataset to blind-spot methods using the
40 test frames [29 52). GainTuning clearly outperforms these methods (see Section F.5). Finally,
Gain MainTunnng outperforms fine-tuning based on optimizing all the parameters, which overfits heavily (see Section E)
6 Analysis
In this section, we perform qualitative analysis of the properties of GainTuning.
Which images benefit most from Gain TainnTuning adaptation? Section G.1 shows the images in the different test datasets for which GainTunnig achieves the most and the least improvement in PSNR. The result is quite consistent over multiple architectures: the improvement in performance achieved by Gain is larger if the test image contains highly repetitive patterns. This makes intuitive sense; the repetitions effectively provide multiple examples from which to learn these patterns during the unsupervised refinement.
Generalization via Gain SainnTuning. Section G.2 shows that GainTuning can achieve generalization to images that are similar to the test image used for adaptation.
How does Maainguning adapt to out-of-distribution noise? Generalization to distributio noise provides unique opportunity to understand how Gain modifies the denoising function. Ref. (37 shows that the first-order Taylor approximation of denoising CNNs trained on multiple noise levels tend to have negligible constant term, and that the growth of this term is the primary culprit for the failure of these models when tested on new noise levels. GainTuning reduces the amplitude of this constant term, facilitating generalization (See Section G.3 for more details).
8

---

(a) Noisy imuge
(b) Trained piecewise constant
(c) After GainTTuning
Difference b/w and
Noisy
Before GT
Filter bef.
After GT
Filter
Noisy
Before GT
Filter bef
After GT
Filter af.
Figure 5: Adaptation to new image content. (Top) A Bias-free CNN 37 pre-trained on piecewise
constant images applied natural test image (a) oversmooths the image and blurs the details (b), but is able to recover more detail after applying GainTuning using SURE loss function (c). (Bottom) The CNN estimates a denoised pixel (colored pixel at the center each image) as linear combination of the noisy input pixels. The weighting functions (filters) of pre-trained CNN are dispersed, consistent with the training set. However, after GainTuning, the weighting functions are more precisely targeted to the local features, resulting in better recovery of details in the denoised image (c).
How does GainTuning adapt out-of-disttibuutio images? Figure 5 shows the result of applying a Bias-free CNN (37] trained on piccewise-constan images to natural images. Due to its learned prior, the CNN averages over large areas, ignoring fine textures. This is apparent in the equivalent linear filters obtained from a local linear approximation of the denoising function (37]. After TainTuning the model is better able to preserve the fine features, which is reflected in the equivalent filters (see Section G.4 for more details).
7 Limitations
As shown in Section 5 GainTuning improves the state of the art on benchmark datasets, adapts well to out-of-distribution noise and image content, and outperforms all existing methods application to real world the parameters that are optimized test time. However, this constraint also limits the potential improvement in performance one can achieve, as seen when fine-tuning for test images from the 000 and TUPR datasets, each of which contain many images with highly repetitive structure. In these cases, we observe that fine-tune fine-tuning all parameters, and even training only on the test data using Self2Self can outperform GainTuning. This raises the question of how to effectively leverage training datasets for such images.
data. A crucial component in the success of Gain Tuning is restricting
In addition, when the pre-trained denoiser is highly optimized, and the test image is within distribution, GainTuning occasionally causes a slight degradation performance. This is atypical (3 occurrences in 412 GainTuning experiments using DnCNN and SURE), and the decreases are quite small (maximum PSNR degradation of about 0.02dB, compared to maximum improvement of nearly 12dB; Figure
8 Conclusions
We've introduced GainTuning an adaptive denoising methodology for adaptively fine-tuning pre-trained CNN denoiser on individual test images. The method, which general enough to be used with any denoising CNN, improves the performance of state-of-the-art CNNs standard

---

denoising benchmarks, and provides even more substantial improvements when the test data differ systematically from the training data, either in noise level, noise type, or image type. We demonstrate the potential of adaptive denoising in scientific imaging through an application to electron microscopy. Here, Tainnn able to jointly exploit synthetic data and test-time adaptation to reconstruct meaningful structure (the atomic configuration of nanoparticle and its support), which cannot be recovered through alternative approaches. A concrete challenge for future research is to combine the unsupervised denoising strategy of Self2Self, which relies heavily on dropout and ensembling, with pre-trained models. More generally, is of interest to explore whether Guiniuning can provide benefits for other processinn tasks.
Finally, we would like to comment on the potential negative societal outcomes of our work. The training of CNN models on large computational clusters contributes to carbon emissions, and therefore global warming. We hope that these effects may offset to some extent by the potential applications of these approaches to tackle challenges such as global warming. In particular, the catalytic system studied in this work representative of catalysts used in clean energy conversion and environmental remediation
33 65 42
Acknowledgments and Disclosure of Funding
We gratefully acknowledge financial support from the National Science Foundation (NSF): NSF NRT HDR Award 1922658 partially supported SM. NSF CBET 1604971 supported JLV and PAC, and NSF OAC-1940263 supported RM and PAC. NSF DAC-I940099 and OAC-2103922 supported CFG. Funding from the Simons Foundation supported SM and EPS. Thanks to ASU Research Computing and NYU HPC for high performance computing resources, and the John M. Cowley Center for High Resolution Electron Microscopy at Arizona State University.
References
[1] BALLÉ, J., LAPARRA, V., AND SIMONCELLI, F. P. Density modeling of images using generalized normalization transformation. In Int'l Conf Learning Representations (ICLR) (San Juan, Puerto Rico, May 2016). Available at http://arxxw.orr/abbe httpa//wwi..ogg//////////// wtt.a////11....1..11
[2] BALLÉ, J., LAPARRA, V., AND SIMONCELLI, E. P. End-to-end optimized image compression. In Int'l Conf Learning Representations (ICLR) (Toulon, France, April 2017). Available http://arxiv.org/abaa wrwp:////o/aabb/111....111
[3] BATSON, J., AND ROYER, L. Noise2self: Blind denoising by self-supervision. In Proceedings of the 36th International Conference on Machine Learning (2019), pp. 524-533.
[4] BUKHARI, S. S., SHAFAIT, F., AND BREUEL, T. M. The iupr dataset of camera-captured document images. In International Workshop on Camera- Baased Document Analysis and Recognition (2011), 1), Springer, pp. 164-171.
[5] CARANDINI, M., AND HEEGER, J. Normalization as a canonical neural computation. Nature Reviews Neuroscience 13, 1 (2012), 51-62.
[6] CHANG, S. G., Yu, B., AND VETTERLI, M. Adaptive wavelet thresholding for image denoising and compression. IEEE Trans. Image Processing 9, 9 (2000), 1532-1546.
[7] CHEN, T., LUCIC, M., HOULSBY, N., AND GELLY, S. On self modulation for generative adversarial networks. arXiv preprint 1rXiv:1810.0136. (2018).
[8] CHEN, Y., AND POCK, T. Trainable nonlinear reaction diffusion: A flexible framework for fast and effective image restoration. IEEE transactions on pattern analysis and machine intelligence 39, 6 (2016), 1256-1272.
[9] CROZIER, P. A., LAWRENCE, E. L., VINCENT, J. L., AND LEVIN, B. D. Dynamic restructuring during processing: approaches to higher temporal resolution. Microscopy and Microanalysis 25, S2 (2019), 1464-1465.
[10] DABOV, K., FOT, A., KATKOVNIK, V., AND EGIAZARIAN, K. Image denoising by sparse 3-d transform- domain collaborative filtering. IEFE Transactions on Image Processing (2017), 2080-2095.
[11] DE VRIES. H., STRUB, F., MARY, J., LAROCHELLE, H., PIETQUIN, O., AND COURVILLE, A. Modulat- ing early visual processing by language. arXiv preprint 17700:1707.00683 (2017).
10

---

[12] DONAHCE. J., JIA, Y., VINYALS, O., HOFFMAN, J.. ZHANG, N., TZENG, E., AND DARRELL, T. Decaf: A deep convolutional activation feature for generic visual recognition. In International conference on machine learning (2014), PMLR, pp. 647-655.
[13] DONOHO. D., AND JOHNSTONE, Adapting to unknown smoothness via wavelet shrinkage. J American Stat Assoe 90, 432 (December 1995).
[14] ELAD, M., AND AHARON, M. Image denoising via sparse and redundant representations over learned dictionaries. IEEE Trans. Image processing 15, 12 (2006), 3736 3745.
[15] ERCIUS, P., JOHNSON, I., BROWN, H., PELZ. P., Hsu, S.-L.. DRANEY. B., FONG, E., GOLDSCHMIDI. The 4d camera 87 khz frame-rate detector for counted 4d-stem
A., JOSEPH, J., LBE, J., AND experiments. Microscopy and Microanalysis (2020), 1-3.
[16] FARUQI, A., AND MCMULLAN, i
imaging detectors for electron microscopy. Nuclear Instruments and Methods Physics Research Section A: Accelerators, Spectrometers, Detectors and Associated Equipment 878 (2018), 180 - 190. Radiation Imaging Techniques and Applications.
[17] GHIASI, G., LEE, H., KUDLUR, M., DUMOULIN, V., AND SHLENS, J. Exploring the structure of real-time, arbitrary neural artistic stylization network. arXiv preprint arXiv: 775.068888 (2017).
[18] GONG, K., GUAN, J., LIU, C.-C., AND QI, J. Pet image denoising using a deep neural network through fine tuning. IEEE Transactions on Radiation and Plasma Medicul Sciences 3, (2018). 153-161.
[19] GUO, H., SAUTET, P., AND ALEXANDROVA, N. Reagent-triggered isomerization of fluxional cluster catalyst via dynamic coupling. The Journal of Physical Chemistry Letters /1, 8 (2020), 3089-3094, PMID: 32227852.
[20] HE, J., DONG, C., AND QIAO, Y. Modulating image restoration with continual levels via adaptive feature modification layers. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (2019), pp.
[21] HEL-OR, Y., AND SHAKED, D. A discriminative approach for wavelet denoising. IEEE Trans. Image Processing (2008).
[22] HUANG, J.-B.. SINGH. A., AND AHUJA, N. Single image super-resolution from transformed self- exemplars. In Proceedings of the IEEE conference computer vision and pattern recognition (2015), pp. 5197-5266.
[23] IOFFE, S., AND SZEGEDY, C. Batch normalization: Accelerating deep network training by reducing internal covariate shift. arXiv preprint 1502::10000316 (2015).
[24] JARRETT, K., KAVUKCUOGLU, K., RANZATO, M., AND LECUN, Y. What is the best multi-stage architecture for object recognition? In 2009 IEEE 12th international conference computer vision (2009),
IEEE, pp. 2146-2153.
[25] KAKU, A., MOHAN, S., PARNANDI, A., SCHAMBRA, H., FERNANDEZ-GRANDA, C. Be like water: Robustness to extraneous variables via adaptive feature normalization. arXiv preprint 2rX10:2000.00009 (2020).
[26] KINGMA, P., AND BA, Adam: A method stochastic optimization, arXiv preprint urXiv:1412.600 (2014).
[27] KRULL, A., BUCHHOLZ, T.-O., AND JUG, Noise2void learning denoising from single noisy images. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (2019). pp. 2124-2132.
[28] KRULL, A., VICAR, T., AND JUG, F. Probabilistic noise2void: Unsupervised content- denoising. arXiv preprint 1rXiv:I906.006 00651 (2019).
[29] LAINE, S., KARRAS, T., LEHTINEN, J., AND AILA, T. High-quality self-supervised deep image denoising. In Advances in Neural Information Processing Systems 32 (2019), pp.
[30] LAWRENCE, E. L., LEVIN, B. D., MILLER, B. K., AND CROZIER, P. A. Approaches to exploring spatio- temporal surface dynamics in nanoparticles with in situ transmission electron microscopy. Microscopy and Microanalysis 26, (2020), 94.
[31] LEE, A. B., MUMFORD, D.. AND HUANG. J. Occlusion models for natural images: A statistical study of scale-invariant dead leaves model. International Journal Computer Vision 41, (2001), 35-59.
11

---

[32] LEVIN, B. D., LAWRENCE, E. L.. AND CROZIER. P. A. Tracking the picoscale spatial motion of atomic columns during dynamic structural change. Utramiccoxcopyy 213 (2020). 112978.
[33] LUISIER, F., BLU, T., AND UNSER, M. sure approach image demoising: Interscale orthonormal wavelet thresholding. IEEE Transactions on Image Processing 16 (2007), 593-606.
[34] MARTIN, D., FOWLKES, C., TAL, D., AND MALIK. J. A database of human segmented natural images and its application to evaluating segmentation algorithms and mcasuring ccological statistics. In Proc. 8th Int'l Conf. Computer Vision (July 2001), vol. 2, pp. 416-423.
[35] MATHERON, G. Random sets and integral geometry.
[36] METZLER, C. A., MOUSAVI, A., HECKEL, R., AND BARANIUK, R. G. Unsupervised learning with stein's unbiased risk estimator, arXiv preprint arXiv: 1053/ (2018).
[37] MOHAN, S., KADKHODAIE, Z., SIMONCELLI, E. P. AND FERNANDEZ- GRANDA, C. Robust and interpretable blind image denoising via bias-free convolutional neural networks. In Proceedings of the International Conference on Learning Representations (2020).
[38] MOHAN, S., MANZORRO, R., VINCENT, J. L.. TANG, B., SHETH, D. Y., SIMONCELLI, E. P., MATTE-
SON, D. S., CROZIER, P. A.. AND FERNANDEZ- GRANDA, C. Deep denoising for scientitic discovery: case study in electron microscopy. arXiv preprint ar 20v:20000 12970 (2020).
[39] MONTINI, T., MELCHIONNA, M., MONAL, M., AND FORNASIERO, P. Fundamentals and catalytic applications of cco2-based materials. Chemical reviews 116, 10 (2016), 5987-6041.
[40] MORAN, N., SCHMIDT, D., ZHONG, Y., AND COADY, P. Noisier2noise: Learning to denoise from unpaired noisy data. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition (2020), pp. 22064-12072
[41] NADO, Z., PADHY, S., SCULLEY, D., D'AMOUR, A., LAKSHMINARAYNNAK, B., AND SNOEK, J. Evaluating prediction-time batch normalization for robustness under covariate shift. arXiv preprint arXiv:2006. 10963 (2020).
[42] NIE, Y., LI, L., AND WEI, Z. Recent advancements in and pt-free catalysts for oxygen reduction reaction. Chemical Society Reviews 44, 8 (2015), 2168-2201.
[43] PEREZ, E., STRUB, F., DE VRIES, H., DUMOULIN, V., AND COURVILLE, A. Film: Visual reasoning with a general conditioning layer. In Proceedings of the AAAI Conference on Artificial Intelligence (2018), vol. 32.
[44] PITKOW, X. Exact feature probabilities in images with occlusion. Journal of vision 10, 14 (2010), 42-42.
[45] PORTILLA, J., STRELA, V., WAINWRIGHT, M. J., AND SIMONCELLJ, F. P. Image denoising using scale mixtures of gaussians in the wavelet domain. IEFE Trans. Image Processing /2, 11 (2003).
[46] QUAN, Y., CHEN, M., PANG. T., AND J1, H. Self2self with dropout: Learning self-supe denoising from single image. In 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) (2020), pp. 1887-1895.
[47] RAMANI, S.. BLU, T., AND UNSER. M. Monte-carlo sure: A black-box optimization of regularization parameters for general denoising algorithms. IEEE Transactions image processing 17, 9 (2008). 1540-1554.
[48] RAPHAN, M., AND SIMONCELLL, P. Optimal denoising in redundant representations. IEEE Trans Image Processing 17,8 8 (Aug 2008), 1342-1352.
[49] RAPHAN, M., AND SIMONCELLI, E. P. Least squares estimation without priors or supervision. Neural Computation 23, 2 (Feh 2011), 374-420. Published online, Nov 2010.
[50] RONNEBERGER, O., FISCHER, P., AND BROx, T. U-net: Convolutional networks for biomedical image segmentation. Medical Image Computing and Computer-Assiste Intervention, Springer, LNCS 935/ (2015), 234-241.
[51] SCHNEIDER, S., RUSAK, E., ECK, L., BRINGMANN, O., BRENDEL, W., AND BETHGE, M. Improving robustness against common corruptions by covariate shift adaptation. Advances in Neural Information Processing Systems 33 (2020).
12

---

[52] SHETH, D. Y., MOHAN, S., VINCENT. J. L.,, MANZORRO, R., CROZIER, P. A., KHAPRA, M. M.. SIMONCELLI, E. P., AND FERNANDEZ-GRANDA, C. Unsupervised deep video denoising. arXiv preprint urXiv:0011.50455 (2020).
[53] SIMONCELLI, E. P., AND ADELSON. E. H. Noise removal via Bayesian wavelet coring. In Proc 3rd IEEE Int'l Conf on Image Proc (Lausanne, Sep 16-19 1996), vol. 1, IEEE Sig Proc Society, pp. 379-382.
[54] SMITH, D. CHAPTER I:
Characterization of nanomaterials using transmission electron microscopy. 37 ed. No. 37 RSC Nanoscience and Nanotechnology. Royal Society of Chemistry, Jan. 2015, pp. 1-29.
[55] SOLTANAYEV, S., AND CHUN. S. Y. Training and refining deep learning based denoisers without ground truth data, arXiv preprint wrXiv:1803.012 (2018).
[56] SOLTANAYEV, S., AND CHUN, S. Y. Training deep learning based denoisers without ground truth data. In Advances in Neural Information Processing Systems (2018). vol. 31.
[57] SUN, G., ALEXANDROVA, A. N., AND SAUTET, P. Structural rearrangements of subnanometer cu oxide clusters govern catalytic oxidation. ACS Catalysis 10, 9 (2020), 5309-5317.
[58] TAO, F., AND CROZIER, P. Atomic-scale observations of catalyst structures under reaction conditions and during catalysis. Chemical Reviews 116, (Mar. 2016), 3487-3539.
[59] VAKSMAN, G., ELAD, M., AND MILANFAR, P. Lidia: Lightweight learned image denoising with instance adaptation. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (2020), pp. 524-525.
[60] VINCENT, L.,, MANZORRO, R., MOHAN, S., TANG, B., SHETH, D. Y., SIMONCELLI, E. P., MATTE- SON, D. S.. TERNANDEZ-GRANDEE C., AND CROZIER, P. A. Developing and evaluating deep neural network-based denoising for nanoparticle tem images with ultra-low signal-to-noise.
[61] WANG, D., SHELHAMER F., LIU, S., OLSHAUSEN, B., DARRELI,, T., BERKELEY, U., AND RESEARCH,
A. tent: fully test-time adaptation by entropy minimization. In International Conference on Learning Representations (2021). vol. 4. p.
[62] XIE, Y., WANG, Z., AND J1, S. Noise2same: Optimizing self-supervised bound for image denoising. Advances in Neural Information Processing Systems 33 (2020).
[63] Xu, J., HUANG, Y., CHENG, M.-M., LIU, L., ZHU, F., Xu, Z., AND SHAO, L. Noisy-as-clean: Learning self-supervised denoising from corrupted image. IELE Transactions Image Processing 29 (2020), 9316-9329.
[64] YOSINSKI, J., CLUNE, J.. BENGIO. Y.. AND LIPSON, H. How transferable are features in deep neural networks? arXiv preprint arXiv:1411.178 (2014).
[65] YU, W., POROSOFF, M. D., AND CHEN, J. G. Review of pt-based bimetallic catalysis: from model surfaces to supported catalysts. Chemical reviews 112, (2012), 5780-5817.
[66] ZHANG, K., ZUO, W., CHEN, Y.. MENG, D., AND ZHANG, L. Beyond gaussian denoiser: Residual learning of deep enn for image denoising. IEEE Transactions on Image Processing (2017). 3142-3155.
[67] ZHANG, K., ZUO, W., AND ZHANG, L. Fídnet: Toward fast and flexible solution for enn-based image denoising. IEFE Transactions on Image Processing 27, (2018), 4608-4622.
[68] ZHANG, X., LU. Y., LIU, J., AND DONG, B. Dynamically unfolding recurrent restorer: A moving endpoint control method for image restoration. arXiv preprint IrxXiv11055.0770 (2018).
13

---

Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's contribu-
tions and scope? [Yes] All claims are supported by extensive empirical experiments
(b) Did you describe the limitations your work? [Yes] see Section 7
(c) Did you discuss any potential negative societal impacts of your work? [Yes] Section
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [N/A]
(b) Did you include complete proofs of all theoretical results? [N/A]
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material URL)? [Yes] We include the main code. Main data and models are public.
(h) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes] see Section
(c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)? [Yes] We report histogram of improvement in the main paper, and box plot and raw data in supplementary material, instead of just summary statisties.
(d) Did you include the total amount of compute and the type of resources used (c.g., type of GPUs, internal cluster, or cloud provider)? [Yes] see Section C
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators? [Yes]
(b) Did you mention the license of the assets? [N/A] see Section B
(c) Did you include any new assets either in the supplemental material or as URL? [Yes] Code will be made available at https///gihub..om//
mohan/gaintunin
(d) Did you discuss whether and how consent was obtained from people whose data you're 'r us- ing/curating? [N/A]
(c) Did you discuss whether the data you are using/curating contains personally identifiable informa- tion or offensive content? [N/A]
If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if applicable?
[N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent participant compensation? [N/A]
14


Reviews:
[{"review_id": "qTXoHzTzpkW", "reviewer": "Reviewer_2yuX", "summary": "This paper proposes a method to finetune the pre-trained denoising network, which is trained in a supervised manner, in an unsupervised way to adapt the network to the tested noisy image. To avoid overfitting, they propose GainTuning which only adjusts each channel in the convolutional layers. The experimental results show that the proposed method can improve the denoising performance of existing supervised trained networks.", "questions": "", "limitations": "", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The idea is simple and easy to understand. However, I have the following concerns:\n1. I think one crucial usage for training a denoising network in an unsupervised way is for the noises whose distributions are unknown. For Gaussian noises with unknown noise levels, we can simply train a network to cover a large range of variance and achieve good performance which is shown in the first table in Fig. 4. However, almost all the experiments are for removing Gaussian noises except for the atomic-resolution transmission electron microscope image. There is plenty of real nature image denoising datasets currently, the authors should validate their method in these datasets.\n2. The improvement of the proposed method is marginal. In Fig. 3, the differences between 'Pre-trained' with 'GainTunning' are at most 0.12dB and most of the improvements are less than 0.1dB. As to Fig.4, 'Gaintunning' does not perform better than 'Trained on [0,100]'. Even though 'Gaintunning' performs better than 'Pre-trained', the comparison is unfair. Since 'Pre-trained' is trained with smaller noise variance.\n3. To avoid overfitting during finetune, the authors propose to use GainTunning. However, there are many other ways to only adjust fewer parameters in the network. For example, we can only finetune some layers in the network. Can the authors explain why GainTunning is the best choice? Also, can the authors provide some visual results to demonstrate how Gain Tunning works after finetuning?\n4. Sec. 4 is not relevant to this paper. These losses are proposed by existing arts. Also, the authors do not analyze the pros and cons of these losses in the proposed method.\n5. In Ln. 177, should '(3)' be '(2)'?", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "yes", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "2h"}, "extra_text": {}}, {"review_id": "ke8Nn9TMHq8", "reviewer": "Reviewer_9H2a", "summary": "The paper proposes an adaptive denoising method that improves the performance of the test images by learning multiplicative scaling parameters for each channel in the CNN network. The paper also presents GainTunning test time adaptation of CNN, by performing the experiments where CNN is trained on synthetic data and adapted to real transmission-electron-microscope images.", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "- It would be better if the paper presents qualitative analysis on real noisy images\n- Could authors explain why the improvements are minor in the table3 and table 4\n- It would be better if the paper presents changes in runtime and parameter comparisons due to GainTuning.\n- Is there any specific reason for why in the GainTunning only multiplicative scaling parameters are used why not other types of parameters? ", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "None", "review_text": "", "extra_scores": {"ethical_concerns": "None", "needs_ethics_review": "No", "ethics_review_area": ["I don’t know"], "time_spent_reviewing": "3"}, "extra_text": {}}, {"review_id": "hRLtwu9JGrO", "reviewer": "Reviewer_bRMP", "summary": "The paper considers image denoising in the problem setting recently proposed in the literature, i.e., fine-tuning a pretrained-model on a test image in an unsupervised fashion. The method modifies the optimization procedure of the fine-tuning. ", "questions": "", "limitations": "", "rating": 4, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "I think there are two issues. One is the lack of explanation as to why the proposed method should work. The other is insufficient experimental comparisons. \n\nThe study extends existing studies with their optimization procedures. The extension is to confine the parameters optimized in the fine-tuning stage to the multiplicative scalars of the channels of the layers. The authors state that the existing methods suffer from overfit and employ several remedies to avoid it. They then claim that optimizing much fewer parameters leads to better results than the existing methods.\n\nTwo questions then arise. Although it is reasonable that optimizing fewer parameters will contribute to avoiding overfitting, it is unclear why the method should work. Why does optimizing the “gaining parameters” help generalize to novel input distribution? Why isn’t the “regularization” too strong? Without a reasonable explanation, the paper is better regarded as a report of an empirical finding. \n\nAnother question is concerned with the experimental validation. GainTuning shares the application scenario (i.e., either scratch-training a model or fine-tuning a pretrained-model on a test image without label) with many recent methods. Then, it will be necessary to compare against the best-performing method w/o GainTuning. For instance, only Self2Self is considered in the experiments of Fig.3 of Table SM4 etc., but there are several others in the category of self-supervised training a model on an input image. Table SM4 shows the method is worse than Self2Self; it will be worse than more recent methods.\n\nMoreover, we may regard the method as a regularization method. As the authors also state (ll87), the previous studies employ early stopping [52] or a special architecture [56]. There will be many more standard regularization methods that can be performed during the fune-tuning. It may also be necessary to compare with them. \n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "We first need clarify the contribution of the study. See above.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4"}, "extra_text": {}}, {"review_id": "B03sw6ZbC1R", "reviewer": "Reviewer_YCi9", "summary": "The goal of the paper is to adapt pretrained denoisers to denoise new images; novel noise distribution or data distribution. The authors, inspired by self-supervised denoisers and methods using channel manipulation, propose to fine-tune the gain of CNN denoiser channels to the test image. This fine tuning is tried out with multiple known losses.", "questions": "", "limitations": "", "rating": 4, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Originality:\nThe work is a combination of known techniques, however with limited novelty. To some extent it differs from other methods by the set of parameters it chooses to learn at test time.\n\nQuality:\nThe submission is technically sound but lacks theoretical analysis/justification and somewhat in terms of the experimental results provided in the main manuscript. Some experimental details also need to be added (see comments).\n\nClarity:\nThe submission is clearly written and well organized but could benefit from extended details around the ‘gain’ parameters that are manipulated for various methods. There are also a few typos (see comments).\n\nSignificance:\nThe submission does address a difficult task but the importance of the results is limited and the advancement of the state of the art not demonstrable. \n\nFor in-distribution data, the majority of improvements are smaller than 0.03dB-0.04dB on the BSD68 dataset which is the main benchmark for the BSD400 training set. And such an improvement would be expected since the method is an incremental fine-tuning over pre-trained models.\n\nFor out-of-distribution noise, simply using the original denoiser pretrained blindly (full range of noise levels up to 100) is better than using the proposed method (which also requires some extra test time optimization). And similarly using the bias free model of [35] is also better.\n\nFor out-of-distribution data, the chosen settings are not very relevant. Urban/natural images are easily obtainable and AWGN is easy to synthesize, so models can be easily pretrained on these distributions. Piecewise constant images as well. The improvements here are more interesting, up to 0.71dB for natural images to scanned documents for instance, however, please add the information on the pretrained model that was used and how it was fine-tuned.\n\nThe application to TEM is the most interesting one in my opinion as it comes close to what can be needed in real applications, but it is lightly addressed in the main manuscript. Conclusions are drawn based on one shown visual example and we have no quantitative measures.\n\nIs GainTuning done exactly the same way for all experiments? For all denoiser networks?\n\nCan you provide more results on real image denoising where ground-truth cannot be easily obtained (unlike AWGN)? For instance microscopy imaging.\n\nLastly, as the authors rightfully mention in their limitations, choosing to fine-tune the gains, or any other subset of parameters, is quite arbitrary, and because the authors provide little support for why they made that choice, the theoretical justification is less strong. How many parameters should be fine-tuned and where is the limit between overfitting and adaptation, is in fact a key question. The whole concept of a “prior” learned from a distribution is questioned by methods that adapt to test images and this should not be taken lightly when merging “prior-based methods” and “self-supervised” methods that work at test time. It would be interesting if the authors look into denoising (or any image restoration) methods that study blind denoising, and priors in denoising. Mixing this with self-supervised techniques is not straight-forward from a theoretical ground. As it stands, the paper exploits an interesting idea but develops it in a somewhat empirical manner, which to me is the main weakness next to the experiments provided in the main manuscript.\n\nComments:\n\nFigure 4 (Sec 5.2 & 5.3), all your comparisons are against a “pre-trained CNN” denoiser, please specify exactly which denoiser it is, and how it was trained, this information is critical there.\n\nTypo: Figure 4 caption “on test imageS with noise outside”\n\nTypo: L201: “(see Figure 4 and)” there is something missing, you say comparison to pretrained denoisers (plural) while there is only one in Fig 4, so I assume the ‘and’ should refer to SM there.\n\nTypo: L277 “for different architectureS”\n\nSec 5.2 Why is the ‘gold-standard’ a blind model, rather than a model trained on the same noise level as the test noise level for instance?\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Adequately addressed.", "review_text": "", "extra_scores": {"ethical_concerns": "no", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3h"}, "extra_text": {}}]

Author rebuttal:

Meta-review: All four reviewers agreed that the paper was not currently above threshold for acceptance, with a high indicator of confidence in their assessments. 

In addition to the reviewer comments, one reviewer suggested in a note during discussion three areas for improvement that my be helpful for the authors: "Especially: (1) The authors provide some experiments to show the proposed 'Gaintunning' performs better than 'only finetune some layers'. The authors do not explain the reason. (2) The network trained from a larger range of noises levels can perform better than the proposed method. This reduces the contribution of the proposed method. (3) The authors provide additional experiments to remove Poisson noises when pretrained on Gaussian noises. But the proposed method will be more useful if it can work well for real noises."
Decision: Accept (Poster)