Adaptive Denoising via GainTuning
Authors: Sreyas Mohan
Joshua L. Vincent
Ramon Manzorro
Peter Crozier
Carlos Fernandez-Granda
Eero P Simoncelli
Venue: NeurIPS 2021 Poster (2021)
Keywords: denoising
adaptation
out of distribution
generalization
gain

Abstract: Deep convolutional neural networks (CNNs) for image denoising are typically trained on large datasets. These models achieve the current state of the art, but they do not generalize well to data that deviate from the training distribution. Recent work has shown that it is possible to train denoisers on a single noisy image. These models adapt to the features of the test image, but their performance is limited by the small amount of information used to train them. Here we propose "GainTuning'', a methodology by which CNN models pre-trained on large datasets can be adaptively and selectively adjusted for individual test images. To avoid overfitting, GainTuning optimizes a single multiplicative scaling parameter (the “Gain”) of each channel in the convolutional layers of the CNN. We show that GainTuning improves state-of-the-art CNNs on standard image-denoising benchmarks, boosting their denoising performance on nearly every image in a held-out test set. These adaptive improvements are even more substantial for test images differing systematically from the training data, either in noise level or image type. We illustrate the potential of adaptive GainTuning in a scientific application to transmission-electron-microscope images, using a CNN that is pre-trained on synthetic data. In contrast to the existing methodology, GainTuning is able to faithfully reconstruct the structure of catalytic nanoparticles from these data at extremely low signal-to-noise ratios. 

Adaptive Denoising via GainTuning
Sreyas Mohan', Joshua L. Vincent", Ramon Manzorro', Peter A. Crozier 2
Carlos Granddl 1.3 Eero P. Simoncelli 1,3,4
Center For Data Science, NYU, Sccooo for Engineering of Matter, Transport and Energy, ASU CCurant Institute of Mathematical Sciences, NYU *Center for Neural Science, NYU and Flatiron Institute, Simons Foundation
Abstract
Deep convolutional neural networks (CNNs) for image denoising are typically trained on large datasets. These models achieve the current state of the art, but they do not generalize well to data that deviate from the training distribution. Recent work has shown that is possible to train denoisers on a single noisy image. These models adapt the features of the test image, but their performance is limited by the small amount of information used to train them. Here we propose "Gain' "aain"" a methodology by which CNN models pre-trained on large datasets be adaptively and selectively adjusted for individual test images. To avoid overfitting. Gain Tuning optimizes single multiplicative scaling parameter (the "Gain") of each channel in the convolutional layers of the CNN. We show that GainTuning improves state-of-the-art CNNs standard image-denoising benchmarks, boosting their denoising performance on nearly every image in a held-out test set. These adaptive improvements are even more substantial for test images differing systematically from the training data, either in noise level or image type. We illustrate the potential of adaptive GainTuning in a scientific application to transmission microscope images, using a CNN that is pre-trained on synthetic data. In contrast to the existing methodology. GainTuning able faithfully reconstruct the structure of catalytic nanoparticles from these data at extremely low signal-to- signal-to-noise ratios.
Introduction
Like many problems in image processing, the recovery of signals from noisy measurements has been revolutionized by the development of convolutional neural networks (CNNs)
66
8
67 These
models are typically trained on large databases of images, either in a supervised 37 66
68
67]
noisy test images. This approach achieves state-of of-the-arrrr performance when the test images and the training data belong to the same distribution. However, when this is not the case, the performance of these models is often substantially degraded [59 37 68) This is an important limitation for many practical applications, in which it is challenging (or even impossible) gather training dataset that comparable in noise and signal content to the images encountered at test time. Overcoming this limitation requires adaptation to the test data.
or unsupervised fashion 62 27 29). Once trained, these solutions are evaluated
recent unsupervised method (Self2Self) has shown that CNNs can be trained exclusively on individual test images, producing impressive results 46) Despite this, the performance of Self2Self limited by the small amount of available training information, and generally inferior to CNN models trained on large databases.
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

(a) training on
(b) Adaptive training image
(c) Combined pee training adupiive training
image
image
Test image
Training data
Training data
CNN
CNN
Denoised image
Denoised
image
Denoised image
Figure 1: Proposed denoising paradigm. (a) Typically, CNNs trained large dataset and evaluated directly on a test image. (b) Recent unsupervised methods perform training on a single test image. (c) We propose GuinTuning, a framework which bridges the gap between both of these paradigms: CNN pre-trained on a large training database is adapted to the test image.
In this work, we propose GainTuning, a framework to bridge the gap between models pre-trained on large datasets, and models trained exclusively on test images. In the spirit of two recent meth- ods [55 [59. GainTuning adapts pre-trained CNN models to individual test images by minimizing an unsupervised denoising cost function, thus fusing the generie capabílities obtained from the training data with specific refinements matched to the structure of the test data. Rather than adapt the full parameter set (filter weights and additive constants) the test image, GainTuning instead optimizes single multiplicative scaling parameter (the "Gain") for each channel within each layer of the CNN. The dimensionality of this reduced parameter set is small fraction (+s 0.1 % our examples) that of the full parameter set. We demonstrate through extensive examples that this prevents overfitting to the test data. The Guinnuning procedure is general, and can applied to any CNN denoising model, regardless of the architecture pre-true eettttaining process.
Our contributions. GainTuning provides a novel method for adapting CNN denoisers trained on large datasets to single test image. Gain'l Tuinn improves state-of-the-ard CNNs on standard image- denoising benchmarks, boosting their denoising performance on nearly every image held-out test sets. Performance improvements are even more substantial when the test images differ systematically from the training data. We showcase this ability through controlled experiments in which we vary the distribution of the noise and image structure of the test data. Finally, we evaluate Gain real scientific-imaging application where adaptivity is crucial: denoising transmission-electre mieroscope data at extremely low signal-to-noise ratios. As shown in Figure 2 both CNNs pre-trained on simulated images and CNNs trained only on the test data produce denoised images with substantial artefacts. In contrast, Gain achieves effective denoising, accurately revealing the atomic structure in the real data.
Related Work
Denoising via deep learning. the last five years, CNN-based methods have clearly outperformed
previous state-of-the-ard denoising methods 13 53
5
55
14 21
[T]
Denoising CNNs are typically
trained in a supervised fashion, minimizing mean squared error (MSE) over a large database of
example ground-truth clean images and their noisy counterparts [66
37
Unsupervised methods
have also been developed, which do not rely on ground-truth images. There are two main strategies to achieve this: use of an empirical Bayes objective, such Stein's unbiased risk estimator (SURE) [I3
33 4
36 55 (561, and architectural "blind-spot" methods detailed description).
(see Section for a more
Generalization to mnt-o--distribution noise. Previous studies have shown that CNN denoisers fail to generalize when the noise encountered at test time differs from that of the training data 68 37 Ref. 377 proposes the use of a modified CNN architecture without additive bias terms, which able to generalize to noise with variance well beyond that encountered in the training set. Here, we show

---

(b) Unsupervised training only on (a)
(c) Supervised training on simulated datu [3]
(d) GainTuniig CNN trained on data
(e) Estimated reference
(a) Noisy image
image
Figure 2: Denoising results for real-world data. (a) An experimentally eeeeriuenttaly-aquireeddd acquired resoucinn transmission electron microscope image of a Ce02-supported Pt nanoparticle. The image has a very low signal noise ratio (PSNR of xz 3dB). (b) Denoised image obtained using Self2Self [46]. which fails to reconstruct three atoms (blue arrow, second row). (c) Denoised image obtained via a CNN trained a simulated dataset, where the pattern of the supporting atoms is not recovered faithfully (third row), (d) Denoised image obtained by adapting the CNN in (c) to the noisy test image in (a) using GainTuning. Both the nanoparticle and the support are recovered without artefacts. (e) Reference image, estimated by averaging 40 different noisy images of the same nanoparticle.
that augmenting generic architecture with GainTunnig yields comparable performance removing bias.
Generalization to -ittr--distribution images. In order to adapt CNNs to operate on test data with characteristics differing from the training set, recent publications propose fine-tuning the networks using an additional training dataset that is more aligned with the test data (59
[8] This is form However, it often
of transfer learning, a popular technique in classification problems challenging to obtain relevant additional training data. Here, we show that GainTuning can adapt CNN denoisers to novel test images.
12 34
Feature normalization. Normalization techniques such batch normalization (BN) 23 are standard component of deep CNNs. BN consists two stages: (1) centering and normalizing the features corresponding each channel, (2) scaling and shifting the normalized features using two learned parameters per channel (a scaling factor and a shift). The scaling parameter is analogous to the gain parameter introduced in GainTuning. However, in BN this parameter is adjusted during training and fixed during test time, whereas GainTuuiin Tuning adjusts it adaptively. for each test image.
Gain normalization. Motivated by gain control properties observed in biological sensory neurons [5] adaptive local normalization of response gains has been previously applied in object recognition [24 density estimation [1] and compression [2]. In contrast to these approaches, which adjust gains based on local responses, GainTTuning adjusts global gain for each channel by optimizing an unsupervised objective function.
Adapting CNN denoisers test data. Two recent publications have developed methods of adapting CNN denoisers to test data [56] [59] Ref. [56) include the noisy test images in the training set. In a recent extension, the authors fine-tune a pre-trained CNN on single test image using the SURE cost function 55 Ref. [59] does the same using a novel cost function based on noise resampling (see Section for a detailed description). As shown in Section BectionE]ffinette ffine-tuning the full set of CNN parameters using only a single test image can lead to overfitting. Ref. (55 avoids this using early stopping, selecting the number of fine-tuning steps beforehand. Ref. [59] uses a specialized architecture with reduced number of parameters. Here, we show that several unsupervised cost functions can be used to perform adaptation without overfitting, as long as we only optimize a subset of the model parameters (specifically, the gain of each channel).
3

---

Adjustment of channel parameters to improve generalization in other tasks. Adjustment of channel parameters, such as gains and biases, has been shown to improve generalization multiple machine-learning tasks, such as the vision-language problems 4 [il. image generation 77 style transfer [[7]. and image restoration [20] In these methods, the adjustment carried out while
training the model by minimizing supervised cost function. In image classification, recent studies
have proposed performing adaptive normalization 225 11 [1
and optimization [6]] of channel parameters during test time, in the same spirit as Gain
3 Proposed Methodology: Gain
In this section we describe the GainTTuning framework. Let fo be CNN denoiser parameterized by weight and bias parameters, 0. We assume that we have available training database and a test image Yiest that we aim to denoise. First, the networks parameters are optimized on the training database
0 pre-trained arg min
Σ
Spre-tranning (v. fo()).
Ertraining database
The cost function frr training used for pre-training can be supervised, if the database contains clean and noisy examples, or unsupervised, if it only contains noisy data.
direct method of adapting the pre-trained CNN the test data is to finetune all the parameters, as is done in all prior work on test-time adaptation [59 55 18. Unfortunately this can lead to overfitting the test data (see Section minimize the unsupervised cost function without denoising the noisy test data effectively. This can be avoided some extent by employing CNN architectures with a small number of parameters [59], or by only optimizing for short time ("early stopping") [55]. Unfortunately, using a CNN with reduced parameters can limit performance (see Section (5) and it is unclear how to choose single criterion for early stopping that can operate correctly for all test images. Here, we propose a different strategy: tuning a single parameter (the gain) in each channel of the CNN. GainTuning can be applied to any pre-trained CNN.
Due the large number of degrees of freedom, the model able to
We denote the gain parameter of the dh channel of the the 1 layer as 7/,, c], and the conventional parameters of that channel by pre-trrinee [I (a vector containing the filter weights). The adapted Gain parameters are the product of these:
aaiin (1) = c[1,c] c) Pre trained [l.
We estimate the gains by minimizing an unsupervised loss that only depends on the noisy image:
arg min
LaainTuning (ress GainTaning (?))
(3)
The final denoised image S (9) (y (Vies). Section 4 describes several possible choices for the cost function Lnain Since we use only one scalar parameter per channel. the adjustment performed by GainTuning is very oiw-dimesiional (= 0.1% of the dimensionality of 6). This makes optimization quite efficient, and prevents overfitting (see Section[E) Further, in Section E we show that performing TainTuning provides better performance when compared to fine-tuning only the last few layers of the pre-trained network.
4
Cost Functions for GainTunning
critical element of Gain Tuning is the use of an unsupervised cost function, which is minimized in order to adapt the pre-trained CNN to the test data. Here, describe three different choices, cach of which are effective for the Guinnuning framework, but which have different benefits and limitations.
Blind-spot loss. This loss measures the ability of the denoiser to reproduce the noisy observation, while excluding the identity solution. To achieve this, the CNN must estimate the jth pixel y/ of the noisy image y as function of the other pixels V[j)e, excluding the pixel itself. As long as the noise degrades pixels independently, the network to learn a nontrivial denoising function that exploits the relationships between pixels arising from the underlying clean image(s). The resulting loss can be written as
Lolind-spox (y, 0) = [(fo(y())) 31
(4)

---

Here the expectation is over the data distribution and the selected pixel. This "blind spot" can be enforced through architecture design [29], or by masking G (see also 46 and 622 for related approaches). The blind-spot loss has a key property that makes it very powerful in practical applications: it makes no assumption about the noise distribution beyond pixel-wise independence. When combined with Gain Mainnuniig it achieves effective denoising of real electron-microsoope data at very low SNRs (see Figure 2
and Section 5.4 F.5
Stein's ' Unbiased Risk Estimator (SURE). Let x be an N-dimensional ground-truth random vector x and let y := x + n be a corresponding noisy observation, where n ~ (O I. SURE provides an expression for the MSE between x and a denoised estimate fo(y), which only depends on the noisy observation y:
20 N
N
a(f(y))a) 0y
E
x
fo(y))?
= EE byy fo(y))1 -2
LSURE (,,), 0). (5)
The last term in Equation S is the divergence of fe. which can be approximated using Monte Carlo techniques 47 (Section D). The divergence is the sum of the partial derivatives of each denoised pixel with respect to the corresponding input pixel. Intuitively, penalizing it forces the denoiser to not rely heavily on the jth noisy pixel to estimate the jth clean pixel. This is similar to the blind-spot strategy, with the added benefit that the jth noisy pixel is not ignored completely. To further illustrate this connection, consider linear convolutional denoising function fy(y) 0 @ y. where the center-indexed parameter vector is = [0_k; k, Oktl,..... 0..............1..... 0x-1, cost function (Equation reduces to
The SURE
Em N
yy 0 @ yl3 - o² 20²00
(6)
The SURE loss equals the MSE between the denoised output and the noisy image, with penalty on the "self" pixel. As this penalty is increased, the self pixel will be ignored, so the loss tends towards the blind-spot cost function. When integrated into the Gain framework, the SURE loss is limited to additive Gaussian noise, for which it outperforms the blind-spot loss. Extensions of SURE to many other stochastic observation models have been developed 49 and may offer alternative objectives for GainTuning.
Noise Resampling. Ref. (59 introduced novel procedure for adaptation which we call noise resampling. Given a pre-trained denoiser fe and a test image y. first one obtains an initial denoised image by applying fe to y, & := ff (y). This denoised image is then artificially corrupted & by simulating noise from the same distribution as the data of interest to create synthetic noisy examples. Finally, the denoiser is fine-tuned by minimizing the MSE between & and the synthetic examples. If we assume additive noise, the resulting loss is of the form
Lauiss resampling (y, O) E,, [1l(f((*+]] - 1
Noise resampling is reminiscent of Refs. 40 63 which add noise to an already noisy image. When integrated in the GainTuning framework, we find the noise-resampling loss results in effective denoising in the case of additive Gaussian noise, although it generally underperforms the SURE loss.
5 Experiments and Results
We performed three different types of experiment to evaluate the performance of GainTuning In- distribution (test examples held out from the training set); out-o--distribuuti noise (noise level or distribution of test examples differs from training set); and out-of-distr signal (test images differ in features or context from the training set). We also apply GainTuning to real data from transmission electron microscope.
Our experiments make use of four datasets: The BSD400 natural image database 34 with test sets Set12 and Se168 (66), the Urban100 images of urban environments [22] the IUPR dataset of scanned documents 4 and a set of synthetic piecewise constant images 31 (see Section B). We demonstrate the broad applicability of GainTuning by using it in conjunction with multiple architectures for image denoising: DnCNN (66. BFCNN 571, UNet [50) and Blind-spot net
29 (see Section A Finally, we compare our results to several benchmarks: (1) models trained on the training database,
(3) CNN models adapted by fine-tuning all parameters (as opposed to just the gains), (3) model trained only on the test data, (4) LIDIA, specialized architecture and adaptation strategy proposed
in 59
We provide details training and optimization in Section C
5

---

Set12
BSD68
Model
30
40
60
30
40
50
GainTuning
DrCNN
Pre-trained
29.52
28.21
27.19
28.19
27.16
26.27
GainTuning
29.62
28,30
27.29
28.47
27.23
26.33
Imorovement PSNR
Pre-trained
29.34
28.05
27.05
28.27
27.05
26.15
UNet
Gain Tuning
29.46
28.15
27.13
28.34
27.12
26.22
Baseline
Pre-trained
29.46
27.95
26.58
28.24
26.91
25.74
LIDIA
Adapted
29.50
26.95
28.23
26.97
26.02
Sel12SeIr
29.21
27.80
26.58
27.83
26.67
25.73
0.0
Improvement
6.1
0.2
0.3
Figure 3: Gain achieves state-of-the-art performance. (Left) The average PSNR on two test set of generic natural images improves after Gain using SURE loss function for different architectures across multiple noise levels. The CNNs are trained on generie natural images (BSD400). (Right) Histograms of improvement in PSNR achieved by DrCNN over test images from Set12 (top) and BSD68 (bottom) 30.
5.1 GainTuning surpasses state-of-the-art performance for in-distribution data
Experimental set-up. We use BSD4000, standard natural-image benchmark, corrupted with Gaus- sian white noise with standard deviation a sampled uniformly from [0, 55] (relative to pixel intensity range 00, 255)). Following [66), we evaluate performance on two independent test sets: Set12 and BSD68, corrupted with Gaussian noise with a E {30, 40,50), 50).
Comparison to pre-trained CNNs. GainTunnig consistently improves the performance of pre- trained CNN models. Figure shows this for two different models, DnCNN [66] and UNet [50] (see also Section F.T. The SURE loss outperforms the blind-spot loss, and is slightly better than noise resampling Table 7. The same holds for other architectures, as reported in Section E. On average the improvement is modest, but for some images it is quite substantial (up to 0.3 dB in PSNR for 30, see histogram in Figure 5
Comparison to other baselines. GainTuning outperforms fine-tuning based on optimizing all the parameters for different architectures and loss functions (see Section DD. Gain'Tnning clearly outperforms a Self2Self model, which is trained exclusively on the test data (Figure 3 It also
outperforms the specialized architecture and adaptation process introduced in [59] with a larger gap in performance for higher noise levels.
5.2 GainTuning generalizes to new noise distributions
Experimental set-up. The same sct-up as Section 5.1 is used, except that the test sets are corrupted with Gaussian noise with {70, 80) (both beyond the training range of a E [0, 55]).
Comparison to pre-trained CNNs. Pre-trained CNN denoisers fail to generalize in this setting. Gain Tuning consistently improves their performance (see Figure
The SURE loss again outperforms the blind-spot loss, and is slightly better than noise resampling (see
Section [.2).
The same holds for other architectures, as reported in Section F.2 The improvement
in performance for all images is substantial (up to 12 dB in PSNR for 80, see histogram in
Figure
Comparison to other baselines. GainTuning achieves comparable performance to gold-standard CNN trained with supervision at all noise levels (Figure 4. GainTuning matches the performance of a bias-free CNN [ specifically designed to generalize to out-of-distribution noise (Figure 4 GainTuning outperforms fine-tuning based on optimizing all the parameters for different architectures and loss functions (see Section
E) Gain GainTuning clearly outperforms Self2Self model trained
exclusively on the test data (Section F22. and the LIDIA adaptation method (59)
Gaussian to Poisson generalization: Section F22 and Figure show that GainTuning can effectively
adapt a CNN pre-trained for Gaussian noise removal to restore images corrupted with Poisson noise as well.

---

Out-of-distribution test noise
Tost set
Trained E [o, 55]
Bias Free
Trained on
20%
BSD a 70
Pre trained
Gaintuming
Model Ri
E [o, 100]
Fraction
70
22.45
25.54
25.59
25.50
10%
Set12
BSD68
70
22.15
24.89
24.87
24.88
0%
0.0
2.0
4.0
6.0
8.0
10.0
12.0
18.72
24.14
24.38
24.36
Improvement PSNR
Out-of--istributioo test image
Fraction Deta
40%
Training dain
Test data
Pre: trained
Gaintuning
30%
(b)
20%
(c)
Piocewise constant
Natural images
27.31
28.60
10%
(b)
Natural images
Urban images
28.35
28.79
0%
(c)
Natural images Scanned documents
30,02
30.73
0.0
0.5
1.0
2.0 2.5
3.0 3.5
Improvement in PSNR
Figure 4: Tuning generalizes to out-of-distribution data. Average performance of a CNN trained denoise noise levels E [0, 55] improves significantly test image with noise outside the training range, a = 70, 80 (top) and on images with different characteristics than training data (bottom) after GainTuning. Capability of GainTuning
generalize to out-of-distribution noise is comparable to that of Bias-Free CNN 3 which is an architecture explicitly designed to generalize noise levels outside the training range, and that of denoiser trained with supervision at all noise levels. (Right) Histogram showing improvement in performance for each image in the test set. The improvement substantial across most images, reaching nearly 12dB improvement in one example. For these examples, the denoiser was DrCNN (with additive bias terms) and the Gainng loss function was SURE. See Section for experiments with other CNN architectures and loss functions.
5.3 GainTuning generalizes to out-of-distribution image content
Experimental set-up. We evaluate the performance of GainTuning on test images that have different characteristics from the training images. We perform the following controlled experiments:
(a) Simulated piecewise constant images + Natural images. We pre-train CNN denoisers on simulated piecewise constant images. These images consists of constant regions (of different intensities values) with the boundaries having varied shapes such circle and lines with different orientations (see Section B for some examples). Piecewise constant images provide a crude model for natural images B5 B1I We use GainTuning to adapt a CNN trained on this dataset to generic natural images (Set12). This experiment demonstrates the ability of Gain GainTuning Tuning to adapt from a simple simulated dataset to a significantly more complex real dataset.
(b) Generic natural images + Images with high self-similarity. We apply GainTuning to adapt a CNN trained on generic natural images to images in Urban100 dataset. Urban 100 consists images of buildings and other structures typically found in an urban setting, which contain substantially
repeating/periodic structure (see Section B than generic natural images.
(c) Generic natural images Images of scanned documents. We apply GainTuning to adapt a CNN trained on generic natural images to images of scanned documents in IUPR dataset (see Section
B
All CNNs were trained for denoising Gaussian white noise with standard deviation o e [0, 55] and 30.
evaluated at
Comparison to pre-trained CNNs. GainTuning consistently improves the performance of pre- trained CNNs in all the three experiments. Figure Showss shows this for DnCNN when Gain'Tunigg is based on SURE loss. We obtain similar results with other architectures (see Section F3 In experiment (a). all test images show substantial improvements over the pre-trained results (average increase of roughly 1. 1.3dB, and best case more than dB, at a 30). We observe similar trends for experiments
(b) and (c) as well, with improvements being better on an average for experiment (c). Note that we obtain similar performance increases when both image and noise are distriiiiitiiitt as discussed
in Section c
7

---

Comparison to other baselines. In experiment (a), GainTuning outperforms methods that optimize all parameters over different architectures and loss functions (Section E However, Self2Self trained only on test data outperforms Gain Tuninngin this case, because the test images contain content that
differs substantially from the training images. Self2Self provides the strongest form of adaptation,
since it is trained exclusively on the test image, whereas the denoising properties of
are partially due to the pretraining (see Sections 7 FF3). p. We did not evaluate LIDIA [59] for this experiment. For experiments (b) and (c), training all parameters clearly outperforms Gain Guning for case (b), but has similar performance for (c). GainTuning outperforms LIDIA on experiments (b) and (c). Self2Self trained exclusively on test data outperforms Gain Gain TiiinTTiingnnn LIDIA) on (b) and (c)
(see Sections
5.4 Application to Electron microscopy
Scientific motivation. Transmission electron microscopy (TEM) is a popular imaging technique in
Recent advancements in TEM enable to image high frame rates [ These images can for example capture the dynamic, atomic-level rearrangements of catalytic systems [57]
materials science [54] [X
15
32 9 which is critical to advance our understanding of functional materials. Acquiring image series at such high temporal resolution produces data severely degraded by shot noise. Consequently, there is an acute need for denoising in this domain.
30
The need for adaptive denoising. Ground-truth images are not available TEM, because measuring at high SNR is often impossible. Prior work has addressed this by using simulated training data 33
60
whereas others have trained CNNs directly on noisy real data
Dataset. We use the training set of 5583 simulated images and the test set of 40 real TEM images from
35 60
The data correspond a catalytic platinum nanoparticle on CeO₂ support (Section B).
Comparison to pre-trained CNN. A CNN 2 pre-trained on the simulated data fails to reconstruct the pattern of atoms faithfully (green box in Figure (c), (c)). GainTuning applied to this CNN using the blind-spot loss correctly recovers this pattern (green box in Figure 2 (d). (e)) reconstructing the small oxygen atoms in the CeO2 support. GainTuuning with noise resampling failed to reproduce the support pattern (probably because it is absent from the initial denoised estimate) (Section F.5
Comparison to other baselines. GainTuning clearly outperforms Self2Self, which is trained exclu- sively on the real data. The denoised image from Self2Self shows missing atoms and substantial artefacts (see Section
[:5). We also compare Gain dataset to blind-spot methods using the
40 test frames [29 52). GainTuning clearly outperforms these methods (see Section F.5). Finally,
Gain MainTunnng outperforms fine-tuning based on optimizing all the parameters, which overfits heavily (see Section E)
6 Analysis
In this section, we perform qualitative analysis of the properties of GainTuning.
Which images benefit most from Gain TainnTuning adaptation? Section G.1 shows the images in the different test datasets for which GainTunnig achieves the most and the least improvement in PSNR. The result is quite consistent over multiple architectures: the improvement in performance achieved by Gain is larger if the test image contains highly repetitive patterns. This makes intuitive sense; the repetitions effectively provide multiple examples from which to learn these patterns during the unsupervised refinement.
Generalization via Gain SainnTuning. Section G.2 shows that GainTuning can achieve generalization to images that are similar to the test image used for adaptation.
How does Maainguning adapt to out-of-distribution noise? Generalization to distributio noise provides unique opportunity to understand how Gain modifies the denoising function. Ref. (37 shows that the first-order Taylor approximation of denoising CNNs trained on multiple noise levels tend to have negligible constant term, and that the growth of this term is the primary culprit for the failure of these models when tested on new noise levels. GainTuning reduces the amplitude of this constant term, facilitating generalization (See Section G.3 for more details).
8

---

(a) Noisy imuge
(b) Trained piecewise constant
(c) After GainTTuning
Difference b/w and
Noisy
Before GT
Filter bef.
After GT
Filter
Noisy
Before GT
Filter bef
After GT
Filter af.
Figure 5: Adaptation to new image content. (Top) A Bias-free CNN 37 pre-trained on piecewise
constant images applied natural test image (a) oversmooths the image and blurs the details (b), but is able to recover more detail after applying GainTuning using SURE loss function (c). (Bottom) The CNN estimates a denoised pixel (colored pixel at the center each image) as linear combination of the noisy input pixels. The weighting functions (filters) of pre-trained CNN are dispersed, consistent with the training set. However, after GainTuning, the weighting functions are more precisely targeted to the local features, resulting in better recovery of details in the denoised image (c).
How does GainTuning adapt out-of-disttibuutio images? Figure 5 shows the result of applying a Bias-free CNN (37] trained on piccewise-constan images to natural images. Due to its learned prior, the CNN averages over large areas, ignoring fine textures. This is apparent in the equivalent linear filters obtained from a local linear approximation of the denoising function (37]. After TainTuning the model is better able to preserve the fine features, which is reflected in the equivalent filters (see Section G.4 for more details).
7 Limitations
As shown in Section 5 GainTuning improves the state of the art on benchmark datasets, adapts well to out-of-distribution noise and image content, and outperforms all existing methods application to real world the parameters that are optimized test time. However, this constraint also limits the potential improvement in performance one can achieve, as seen when fine-tuning for test images from the 000 and TUPR datasets, each of which contain many images with highly repetitive structure. In these cases, we observe that fine-tune fine-tuning all parameters, and even training only on the test data using Self2Self can outperform GainTuning. This raises the question of how to effectively leverage training datasets for such images.
data. A crucial component in the success of Gain Tuning is restricting
In addition, when the pre-trained denoiser is highly optimized, and the test image is within distribution, GainTuning occasionally causes a slight degradation performance. This is atypical (3 occurrences in 412 GainTuning experiments using DnCNN and SURE), and the decreases are quite small (maximum PSNR degradation of about 0.02dB, compared to maximum improvement of nearly 12dB; Figure
8 Conclusions
We've introduced GainTuning an adaptive denoising methodology for adaptively fine-tuning pre-trained CNN denoiser on individual test images. The method, which general enough to be used with any denoising CNN, improves the performance of state-of-the-art CNNs standard

---

denoising benchmarks, and provides even more substantial improvements when the test data differ systematically from the training data, either in noise level, noise type, or image type. We demonstrate the potential of adaptive denoising in scientific imaging through an application to electron microscopy. Here, Tainnn able to jointly exploit synthetic data and test-time adaptation to reconstruct meaningful structure (the atomic configuration of nanoparticle and its support), which cannot be recovered through alternative approaches. A concrete challenge for future research is to combine the unsupervised denoising strategy of Self2Self, which relies heavily on dropout and ensembling, with pre-trained models. More generally, is of interest to explore whether Guiniuning can provide benefits for other processinn tasks.
Finally, we would like to comment on the potential negative societal outcomes of our work. The training of CNN models on large computational clusters contributes to carbon emissions, and therefore global warming. We hope that these effects may offset to some extent by the potential applications of these approaches to tackle challenges such as global warming. In particular, the catalytic system studied in this work representative of catalysts used in clean energy conversion and environmental remediation
33 65 42
Acknowledgments and Disclosure of Funding
We gratefully acknowledge financial support from the National Science Foundation (NSF): NSF NRT HDR Award 1922658 partially supported SM. NSF CBET 1604971 supported JLV and PAC, and NSF OAC-1940263 supported RM and PAC. NSF DAC-I940099 and OAC-2103922 supported CFG. Funding from the Simons Foundation supported SM and EPS. Thanks to ASU Research Computing and NYU HPC for high performance computing resources, and the John M. Cowley Center for High Resolution Electron Microscopy at Arizona State University.
References
[1] BALLÉ, J., LAPARRA, V., AND SIMONCELLI, F. P. Density modeling of images using generalized normalization transformation. In Int'l Conf Learning Representations (ICLR) (San Juan, Puerto Rico, May 2016). Available at http://arxxw.orr/abbe wttpa//wwi..ogg//////////// wtt.a////11....1..11
[2] BALLÉ, J., LAPARRA, V., AND SIMONCELLI, E. P. End-to-end optimized image compression. In Int'l Conf Learning Representations (ICLR) (Toulon, France, April 2017). Available http://arxiv.org/abaa wrwp:////o/aabb/111....111
[3] BATSON, J., AND ROYER, L. Noise2self: Blind denoising by self-supervision. In Proceedings of the 36th International Conference on Machine Learning (2019), pp. 524-533.
[4] BUKHARI, S. S., SHAFAIT, F., AND BREUEL, T. M. The iupr dataset of camera-captured document images. In International Workshop on Camera- Baased Document Analysis and Recognition (2011), 1), Springer, pp. 164-171.
[5] CARANDINI, M., AND HEEGER, J. Normalization as a canonical neural computation. Nature Reviews Neuroscience 13, 1 (2012), 51-62.
[6] CHANG, S. G., Yu, B., AND VETTERLI, M. Adaptive wavelet thresholding for image denoising and compression. IEEE Trans. Image Processing 9, 9 (2000), 1532-1546.
[7] CHEN, T., LUCIC, M., HOULSBY, N., AND GELLY, S. On self modulation for generative adversarial networks. arXiv preprint 1rXiv:1810.0136. (2018).
[8] CHEN, Y., AND POCK, T. Trainable nonlinear reaction diffusion: A flexible framework for fast and effective image restoration. IEEE transactions on pattern analysis and machine intelligence 39, 6 (2016), 1256-1272.
[9] CROZIER, P. A., LAWRENCE, E. L., VINCENT, J. L., AND LEVIN, B. D. Dynamic restructuring during processing: approaches to higher temporal resolution. Microscopy and Microanalysis 25, S2 (2019), 1464-1465.
[10] DABOV, K., FOT, A., KATKOVNIK, V., AND EGIAZARIAN, K. Image denoising by sparse 3-d transform- domain collaborative filtering. IEFE Transactions on Image Processing (2017), 2080-2095.
[11] DE VRIES. H., STRUB, F., MARY, J., LAROCHELLE, H., PIETQUIN, O., AND COURVILLE, A. Modulat- ing early visual processing by language. arXiv preprint 17700:1707.00688 (2017).
10

---

[12] DONAHCE. J., JIA, Y., VINYALS, O., HOFFMAN, J.. ZHANG, N., TZENG, E., AND DARRELL, T. Decaf: A deep convolutional activation feature for generic visual recognition. In International conference on machine learning (2014), PMLR, pp. 647-655.
[13] DONOHO. D., AND JOHNSTONE, Adapting to unknown smoothness via wavelet shrinkage. J American Stat Assoe 90, 432 (December 1995).
[14] ELAD, M., AND AHARON, M. Image denoising via sparse and redundant representations over learned dictionaries. IEEE Trans. Image processing 15, 12 (2006), 3736 3745.
[15] ERCIUS, P., JOHNSON, I., BROWN, H., PELZ. P., Hsu, S.-L.. DRANEY. B., FONG, E., GOLDSCHMIDI, The 4d camera 87 khz frame-rate detector for counted 4d-stem
A., JOSEPH, J., LEE, J., AND experiments. Microscopy and Microanalysis (2020), 1-3.
[16] FARUQI, A., AND MCMULLAN, D
imaging detectors for electron microscopy. Nuclear Instruments and Methods Physics Research Section A: Accelerators, Spectrometers, Detectors and Associated Equipment 878 (2018), 180 - 190. Radiation Imaging Techniques and Applications.
[17] GHIASI, G., LEE, H., KUDLUR, M., DUMOULIN, V., AND SHLENS, J. Exploring the structure of real-time, arbitrary neural artistic stylization network. arXiv preprint arXiv: 775.068888 (2017).
[18] GONG, K., GUAN, J., LIU, C.-C., AND QI, J. Pet image denoising using a deep neural network through fine tuning. IEEE Transactions on Radiation and Plasma Medicul Sciences 3, (2018). 153-161.
[19] GUO, H., SAUTET, P., AND ALEXANDROVA, N. Reagent-triggered isomerization of fluxional cluster catalyst via dynamic coupling. The Journal of Physical Chemistry Letters /1, 8 (2020), 3089-3094, PMID: 32227852.
[20] HE, J., DONG, C., AND QIAO, Y. Modulating image restoration with continual levels via adaptive feature modification layers. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (2019), pp.
[21] HEL-OR, Y., AND SHAKED, D. A discriminative approach for wavelet denoising. IEEE Trans. Image Processing (2008).
[22] HUANG, J.-B.. SINGH. A., AND AHUJA, N. Single image super-resolution from transformed self- exemplars. In Proceedings of the IEEE conference computer vision and pattern recognition (2015), pp. 5197-5266.
[23] IOFFE, S., AND SZEGEDY, C. Batch normalization: Accelerating deep network training by reducing internal covariate shift. arXiv preprint 10XXv:1502.03100 (2015).
[24] JARRETT, K., KAVUKCUOGLU, K., RANZATO, M., AND LECUN, Y. What is the best multi-stage architecture for object recognition? In 2009 IEEE 12th international conference computer vision (2009),
IEEE, pp. 2146-2153.
[25] KAKU, A., MOHAN, S., PARNANDI, A., SCHAMBRA, H., FERNANDEZ-GRANDA, C. Be like water: Robustness to extraneous variables via adaptive feature normalization. arXiv preprint 2rX10:2000.00009 (2020).
[26] KINGMA, P., AND BA, Adam: A method stochastic optimization, arXiv preprint urXiv:1412.600 (2014).
[27] KRULL, A., BUCHHOLZ, T.-O., AND JUG, Noise2void learning denoising from single noisy images. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (2019). pp. 2124-2132.
[28] KRULL, A., VICAR, T., AND JUG, F. Probabilistic noise2void: Unsupervised content- denoising. arXiv preprint 1rXiv:I906.006 00651 (2019).
[29] LAINE, S., KARRAS, T., LEHTINEN, J., AND AILA, T. High-quality self-supervised deep image denoising. In Advances in Neural Information Processing Systems 32 (2019), pp.
[30] LAWRENCE, E. L., LEVIN, B. D., MILLER, B. K., AND CROZIER, P. A. Approaches to exploring spatio- temporal surface dynamics in nanoparticles with in situ transmission electron microscopy. Microscopy and Microanalysis 26, (2020), 94.
[31] LEE, A. B., MUMFORD, D.. AND HUANG. J. Occlusion models for natural images: A statistical study of scale-invariant dead leaves model. International Journal Computer Vision 41, (2001), 35-59.
11

---

[32] LEVIN, B. D., LAWRENCE, E. L.. AND CROZIER. P. A. Tracking the picoscale spatial motion of atomic columns during dynamic structural change. Utramiccoxcopyy 213 (2020). 112978.
[33] LUISIER, F., BLU, T., AND UNSER, M. sure approach image demoising: Interscale orthonormal wavelet thresholding. IEEE Transactions on Image Processing 16 (2007), 593-606.
[34] MARTIN, D., FOWLKES, C., TAL, D., AND MALIK. J. A database of human segmented natural images and its application to evaluating segmentation algorithms and mcasuring ccological statistics. In Proc. 8th Int'l Conf. Computer Vision (July 2001), vol. 2, pp. 416-423.
[35] MATHERON, G. Random sets and integral geometry.
[36] METZLER, C. A., MOUSAVI, A., HECKEL, R., AND BARANIUK, R. G. Unsupervised learning with stein's unbiased risk estimator, arXiv preprint arXiv: 1053/ (2018).
[37] MOHAN, S., KADKHODAIE, Z., SIMONCELLI, E. P. AND FERNANDEZ- GRANDA, C. Robust and interpretable blind image denoising via bias-free convolutional neural networks. In Proceedings of the International Conference on Learning Representations (2020).
[38] MOHAN, S., MANZORRO, R., VINCENT, J. L.. TANG, B., SHETH, D. Y., SIMONCELLI, E. P., MATTE-
SON, D. S., CROZIER, P. A.. AND FERNANDEZ- GRANDA, C. Deep denoising for scientitic discovery: case study in electron microscopy. arXiv preprint ar 20v:20000 12970 (2020).
[39] MONTINI, T., MELCHIONNA, M., MONAL, M., AND FORNASIERO, P. Fundamentals and catalytic applications of cco2-based materials. Chemical reviews 116, 10 (2016), 5987-6041.
[40] MORAN, N., SCHMIDT, D., ZHONG, Y., AND COADY, P. Noisier2noise: Learning to denoise from unpaired noisy data. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition (2020), pp. 22064-12072
[41] NADO, Z., PADHY, S., SCULLEY, D., D'AMOUR, A., LAKSHMINARAYNNAK, B., AND SNOEK, J. Evaluating prediction-time batch normalization for robustness under covariate shift. arXiv preprint arXiv:2006. 10963 (2020).
[42] NIE, Y., LI, L., AND WEI, Z. Recent advancements in and pt-free catalysts for oxygen reduction reaction. Chemical Society Reviews 44, 8 (2015), 2168-2201.
[43] PEREZ, E., STRUB, F., DE VRIES, H., DUMOULIN, V., AND COURVILLE, A. Film: Visual reasoning with a general conditioning layer. In Proceedings of the AAAI Conference on Artificial Intelligence (2018), vol. 32.
[44] PITKOW, X. Exact feature probabilities in images with occlusion. Journal of vision 10, 14 (2010), 42-42.
[45] PORTILLA, J., STRELA, V., WAINWRIGHT, M. J., AND SIMONCELLJ, F. P. Image denoising using scale mixtures of gaussians in the wavelet domain. IEFE Trans. Image Processing /2, 11 (2003).
[46] QUAN, Y., CHEN, M., PANG. T., AND J1, H. Self2self with dropout: Learning self-supe denoising from single image. In 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) (2020), pp. 1887-1895.
[47] RAMANI, S.. BLU, T., AND UNSER. M. Monte-carlo sure: A black-box optimization of regularization parameters for general denoising algorithms. IEEE Transactions image processing 17, 9 (2008). 1540-1554.
[48] RAPHAN, M., AND SIMONCELLL, P. Optimal denoising in redundant representations. IEEE Trans Image Processing 17,8 8 (Aug 2008), 1342-1352.
[49] RAPHAN, M., AND SIMONCELLI, E. P. Least squares estimation without priors or supervision. Neural Computation 23, 2 (Feh 2011), 374-420. Published online, Nov 2010.
[50] RONNEBERGER, O., FISCHER, P., AND BROx, T. U-net: Convolutional networks for biomedical image segmentation. Medical Image Computing and Computer-Assiste Intervention, Springer, LNCS 935/ (2015), 234-241.
[51] SCHNEIDER, S., RUSAK, E., ECK, L., BRINGMANN, O., BRENDEL, W., AND BETHGE, M. Improving robustness against common corruptions by covariate shift adaptation. Advances in Neural Information Processing Systems 33 (2020).
12

---

[52] SHETH, D. Y., MOHAN, S., VINCENT. J. L.,, MANZORRO, R., CROZIER, P. A., KHAPRA, M. M.. SIMONCELLI, E. P., AND FERNANDEZ-GRANDA, C. Unsupervised deep video denoising. arXiv preprint urXiv:0011.50455 (2020).
[53] SIMONCELLI, E. P., AND ADELSON. E. H. Noise removal via Bayesian wavelet coring. In Proc 3rd IEEE Int'l Conf on Image Proc (Lausanne, Sep 16-19 1996), vol. 1, IEEE Sig Proc Society, pp. 379-382.
[54] SMITH, D. CHAPTER I:
Characterization of nanomaterials using transmission electron microscopy. 37 ed. No. 37 RSC Nanoscience and Nanotechnology. Royal Society of Chemistry, Jan. 2015, pp. 1-29.
[55] SOLTANAYEV, S., AND CHUN. S. Y. Training and refining deep learning based denoisers without ground truth data, arXiv preprint wrXiv:1803.012 (2018).
[56] SOLTANAYEV, S., AND CHUN, S. Y. Training deep learning based denoisers without ground truth data. In Advances in Neural Information Processing Systems (2018). vol. 31.
[57] SUN, G., ALEXANDROVA, A. N., AND SAUTET, P. Structural rearrangements of subnanometer cu oxide clusters govern catalytic oxidation. ACS Catalysis 10, 9 (2020), 5309-5317.
[58] TAO, F., AND CROZIER, P. Atomic-scale observations of catalyst structures under reaction conditions and during catalysis. Chemical Reviews 116, (Mar. 2016), 3487-3539.
[59] VAKSMAN, G., ELAD, M., AND MILANFAR, P. Lidia: Lightweight learned image denoising with instance adaptation. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (2020), pp. 524-525.
[60] VINCENT, L.,, MANZORRO, R., MOHAN, S., TANG, B., SHETH, D. Y., SIMONCELLI, E. P., MATTE- SON, D. S.. TERNANDEZ-GRANDEE C., AND CROZIER, P. A. Developing and evaluating deep neural network-based denoising for nanoparticle tem images with ultra-low signal-to-noise.
[61] WANG, D., SHELHAMER F.,, LIU, S., OLSHAUSEN, B., DARRELI,, T., BERKELEY, U., AND RESEARCH,
A. tent: fully test-time adaptation by entropy minimization. In International Conference on Learning Representations (2021). vol. 4. p.
[62] XIE, Y., WANG, Z., AND J1, S. Noise2same: Optimizing self-supervised bound for image denoising. Advances in Neural Information Processing Systems 33 (2020).
[63] Xu, J., HUANG, Y., CHENG, M.-M., LIU, L., ZHU, F., Xu, Z., AND SHAO, L. Noisy-as-clean: Learning self-supervised denoising from corrupted image. IELE Transactions Image Processing 29 (2020), 9316-9329.
[64] YOSINSKI, J., CLUNE, J.. BENGIO. Y.. AND LIPSON, H. How transferable are features in deep neural networks? arXiv preprint arXiv:1411.178 (2014).
[65] YU, W., POROSOFF, M. D., AND CHEN, J. G. Review of pt-based bimetallic catalysis: from model surfaces to supported catalysts. Chemical reviews 112, (2012), 5780-5817.
[66] ZHANG, K., ZUO, W., CHEN, Y.. MENG, D., AND ZHANG, L. Beyond gaussian denoiser: Residual learning of deep enn for image denoising. IEEE Transactions on Image Processing (2017). 3142-3155.
[67] ZHANG, K., ZUO, W., AND ZHANG, L. Fídnet: Toward fast and flexible solution for enn-based image denoising. IEFE Transactions on Image Processing 27, (2018), 4608-4622.
[68] ZHANG, X., LU. Y., LIU, J., AND DONG, B. Dynamically unfolding recurrent restorer: A moving endpoint control method for image restoration. arXiv preprint I (2018).
13

---

Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's contribu-
tions and scope? [Yes] All claims are supported by extensive empirical experiments
(b) Did you describe the limitations your work? [Yes] see Section 7
(c) Did you discuss any potential negative societal impacts of your work? [Yes] Section
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [N/A]
(b) Did you include complete proofs of all theoretical results? [N/A]
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material URL)? [Yes] We include the main code. Main data and models are public.
(h) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes] see Section
(c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)? [Yes] We report histogram of improvement in the main paper, and box plot and raw data in supplementary material, instead of just summary statisties.
(d) Did you include the total amount of compute and the type of resources used (c.g., type of GPUs, internal cluster, or cloud provider)? [Yes] see Section C
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators? [Yes]
(b) Did you mention the license of the assets? [N/A] see Section B
(c) Did you include any new assets either in the supplemental material or as URL? [Yes] Code will be made available at https///gihub..om//
mohan/gaintunin
(d) Did you discuss whether and how consent was obtained from people whose data you're 'r us- ing/curating? [N/A]
(c) Did you discuss whether the data you are using/curating contains personally identifiable informa- tion or offensive content? [N/A]
If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if applicable?
[N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent participant compensation? [N/A]
14


Reviews:
[{"review_id": "_prMVIWrlH1", "reviewer": "Reviewer_aT3f", "summary": "In this paper a new method is proposed to selectively fine-tune the existing neural network for denoising using the test image. In order to achieve this the authors propose to use additional trainable gain parameters for each channel in each layer and optimize these parameters during fine-tuning. The authors found that by doing this the network is well adapted to the target test images even if the noise level and image type are novel to the original neural network.", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "In this paper a new method is proposed to selectively fine-tune the existing neural network for denoising using the test image. In order to achieve this the authors propose to use additional trainable gain parameters for each channel in each layer and optimize these parameters during fine-tuning. The authors found that by doing this the network is well adapted to the target test images even if the noise level and image type are novel to the original neural network.\n\nThe idea of the paper and the application of such method in the test stage are interesting. The analysis of the proposed method in the paper is relevant. The experiments seem to supported the claimed contribution. While there're merits, I have several concerns of this paper. First of all, the authors claim that the method works when the noise level and image type are novel to the original network. However, in practice the situation in the test images could be much more complex. It is not uncommon that the test images' noise pattern is novel. It is also not uncommon that the noise exists in the images are not uniform. This is particularly common when the image is not in the raw domain (after ISP's nonlinear operators).  I wonder if the method would still work in these common and important situations. If the method has difficulty in deal with these practical situations, the value of using such method against any novel test image during test time is limited. Secondly, because there're limited number of the parameters to tune during test time, I suspect the original network needs to be big and powerful so that it may offer sufficient flexibility to the proposed approach. This may bring significant difficulty to network compression and quantization if one was to deploy such technology in real life. I believe this issue is relevant to such an application oriented paper.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "There should be no potential negative societal impact of their work", "review_text": "", "extra_scores": {"ethical_concerns": "None", "needs_ethics_review": "No", "ethics_review_area": ["I don’t know"], "time_spent_reviewing": "3"}, "extra_text": {}}, {"review_id": "Uk6ocJnFR1p", "reviewer": "Reviewer_sYYW", "summary": "In this work the authors propose a new approach to adapt pre-trained denoising model to new test scenarios not seen at train time both in terms of noise distribution and image content. The proposed solution is fairly simple and straightforward: a gain parameter is estimated per channel per layer for the denoising neural network model at test time for each image. These gain parameters are estimated using the following unsupervised cost functions on the test image: Blind-spot loss, SURE loss, Noise Resampling [56].", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "**Originality**\n\nThe proposed idea is interesting and a relatively simple way to manipulate the output from a denoising model. There is a relationship with feature modulation techniques such as AdaFM and “Cfsnet:  Toward  a  controllable  featurespace for image restoration” that is not sufficiently discussed. \nIn addition to this “Interactive Multi-Dimension Modulation with Dynamic Controllable Residual Learning for Image Restoration” (ECCV 2020) should be discussed. It adopts similar feature modulation to target multiple degradation. \n\nThe optimization process is also related to some blind super resolution methods such as:\n\n-Blind Super-Resolution With Iterative Kernel Correction\n\n-Blind image super-resolution with spatially variant degradations,\n\nIn the case of denoising unsupervised cost functions could be directly used and a separate network is not needed, which allows to potentially better target “out of distribution” noise levels.\n\n\n**Quality**\n\nThe method achieves good results and the authors demonstrate a wide variety of scenarios where the proposed model performs better than pre-trained models and blind denoising methods overfitted to the test data. There are however a few elements that needs explanations and more details:\n\n-The authors only demonstrate results on gray scale images. Is there a reason to not investigate denoising for color images?\n\n-The proposed noise distribution is relatively simple (Gaussian). More complex noise distribution should be investigated in particular signal dependent models? It is not clear if the model would be adapted to those situations.\n\n-Still related to the noise adaptation, it is not clear if it would be possible to have locally different denoising gains? \n\n-Authors should discuss the link with some existing denoising method that takes into account the noise level as conditioning, such as “FastDVDnet: Towards Real-Time Deep Video Denoising Without FlowEstimation”. In addition, in this case a similar optimization strategy could be used to estimate noise levels; This seems like a reasonable baseline to demonstrate the interest of the gain.\n\n**Clarity**\n\nThe paper is clear and easy to follow. All the experiments are sufficiently well described and the supplemental material provides even more details.\n\n\n**Significance**\n\nThe paper presents an interesting idea and denoising is one of the most common problems, however a few questions need to be answered to appropriately judge the significance. \n\n**After rebuttal**\n\nThe authors answered most of my questions and concerns in their rebuttal. I would insist they include discussion with respect to the different papers that were missing in the first version. If possible, I would also advise including experimental comparisons against methods like FastDVDnet.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The authors discuss the limitations of the model and point to the restricted optimization space which can limit the possible improvements. However I think the experiments did not investigate enough more complex noise models and it would be important to address such issues.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4"}, "extra_text": {}}, {"review_id": "TlzOS_2aS7N", "reviewer": "Reviewer_YFNi", "summary": "This paper proposes a new approach for image denoising, which combines pretraining a neural network on a dataset, and then fine tuning it on the noisy test image to be denoised. The key new feature is to tune only a small set of parameters, called gains, one for each channel within each layer in the network. These gains multiply the corresponding channels. \nThe benefit of the approach is illustrated in the task of removing white Gaussian noise. The method is shown to improve performance both in the standard denoising setting, and in cases where the noise distribution or the image distribution (or both) in the test set differ from the train set.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The idea of combining external supervision with internal learning, is not new, and has been already explored in several contexts, including super-resolution and denoising. However, the particular way proposed in this paper seems to be new, and is quite interesting. It makes a lot of sense to restrict the degrees of freedom in the fine-tuning stage, and doing so by controlling channel gains is shown to lead to good results. However, I do have several concerns.\n\n*Generalization to out-of-distribution noise*\n\nA major claim in the paper is that the proposed method generalizes well to out-of-distribution noise. But the method is still inferior to [35] in this task (Fig. 4). The authors explain this by referring to [35] as \"specifically designed to generalize to out-of-distribution noise\". But all that [35] really do is remove the additive biases within all the layers of the network. They showed that this turns any denoising network into one that generalizes well to different noise-levels. What's bothering here is that the method in the current manuscript also uses no biases, as can be learned from the supplementary (SM 1.1). So the authors can't blame the poorer performance w.r.t. [35], on any specialized architecture. It seems to me that the authors' experiments prove the exact opposite of what they claim. Namely, GainTuning rather *impairs* the ability of a bias-free network to generalize well to out-of-distribution noise. Indeed, [35], which uses a bias free network but without GainTuning, performs better than GainTuning. \n\n*Experiments*\n\n- It's not clear whether the UNet architecture also uses no biases. This should be stated and emphasized in the paper. \n\n- It's also not clear which architecture was used in which experiment (DnCNN or UNet or something else). This is very important for assessing the comparisons.\n \n-The three fine-tuning losses: A whole section (Sec. 4) is devoted to explaining about three possible losses for fine-tuning, but no comparison between them is provided in the main text (only in the supp.).  Line 189 refers to Table 2 for such a comparison, but no such table exists. Also, it's not clear which loss was eventually used in each experiment (for example in Figs. 3-5).\n\n- Visualization in Fig. 5: The visualization of the filter coeff. in Fig. 5 should be explained. I understand this is taken from [35], which only makes sense for bias-free networks, but it should be explained in the current paper for completeness.\n\n\n*Exposition*\n\n-SURE: The terminology used is a bit inconsistent with the original concept. SURE refers to what's inside the second expectation in (5), namely it's without the expectation. And this is also the loss that's being optimized in practice in this paper (without the expectation), because there is only a single y. Stein's theorem shows that the expectation over y of SURE is the expectation over y of $||x-f(y)||^2$, for any given $x$.\n\n-Eq. (7): Extra left parentheses. \n\n-Line 201: Missing ref. in \"(Figure 4 and)\"\n\n\n**---------------- Post Rebuttal ----------------**  \nMy main concern had to do with the biases in the networks. This has been clarified in the rebuttal. I therefore recommend acceptance.\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The authors have addressed limitations and societal impact.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "5"}, "extra_text": {}}, {"review_id": "FzikGZf3G1j", "reviewer": "Reviewer_uD7E", "summary": "The authors present a novel training method for denoising that combines pre-training with unsupervised/selfsupervised finetuning on the test image.\nTo avoid overfitting during finetuning , the authors adjust not the weights of the network but only single scalar variables for each feature channel in each layer of the network.\nThey evaluate their method on different on different datasets and with different self-supervised/unsupervised loss functions and achieve remarkable results.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "========================\nStrengths\n========================\n\nAdaptation to test data that is different from the training set is an important problem.\n\nThe results are convincing.\n\nI like the fact that the method is viewed as a general approach, and applied with different established network architectures and loss functions.\nThis is in contrast to many papers that present a new training scheme coupled with a modified architecture, making it difficult to disentangle the advantages of each component.\n\nI appreciate the extensive experiments documented in the supplementary material.\n\nThe paper is well written and easy to follow.\n\n========================\nWeaknesses\n========================\n\nThere are some open questions for me regarding the assessment of the potential and limitations of the method:\nIn line 277 the authors write: \n\"GainTuning is more beneficial for images with highly repetitive patterns\".\nYet, in the Limitations sections they state that for images containing repetitive structure, finetuning all parameters outperforms the method.\nCan the authors please clarify this?\nIt makes intuitive sense that images with repeating structures are less prone to overfitting. Shouldn't Gain tuning be most beneficial for images that do not contain repeating structures? \n\nIt is important for the paper to show that overfitting is real problem.\nSupplementary Figures SM2 and SM3 discuss this, but it looks like overfitting mainly seems to be a problem of SURE and occurs to a much smaller extent with Noise Resampling. Why is this the case? Why is Blind spot denoising missing from the analysis in Supplementary Figures SM2 and SM3?\n\n\n========================\nMinor Points\n========================\n\nTable 2 is referenced in line 189, but I cannot seem to find it in the paper.\nIt would actually be interesting to have these results.\n\nI think [1] is highly related and could be discussed.\n[1]: Wang, Yina, et al. \"Image denoising for fluorescence microscopy by self-supervised transfer learning.\" bioRxiv (2021).\n\n\n========================\nFinal Recommendation\n========================\n\nI think the paper presents a solid contribution to an important topic.\nGiven an adequate response and clarifications regarding the mentioned weaknesses I am happy to support it.\n\n\n========================\nPost Rebuttal\n========================\n\nI am happy with the answers provided by the authors and I recommend accepting the paper.\nI believe my questions regarding overfitting and the performance of SURE and Noise Resampling are sufficiently answered and I very much appreciate the additional Figure, which I hope can be included in the supplementary material.\n ", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "I do not see any negative societal impacts. Regarding limitations, please see Main Review.", "review_text": "", "extra_scores": {"ethical_concerns": "I do not have any ethical concerns regarding the paper.", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "5"}, "extra_text": {}}]

Author rebuttal:

Meta-review:   To solve the image denoising problem, this paper proposed to tune a single gain parameter for each channel on a single test image. The tuning is done via minimizing an unsupervised loss. Compared to non-fine-tuning methods, tuning can provide better adaptation to out-of-distribution images. Compared to other tuning-methods, this approach is designed to avoid the overfitting issue of fine-tuning all hyperparameters. The idea is simple and natural, and this work has presented many experiments to show that it outperforms existing methods. After rebuttal, all reviewers think that the rebuttal has addressed their most concerns, and agree with the acceptance (though some are on the borderline). Thus I recommend acceptance.  

  Note that one reviewer requested to add experimental comparison with conditioned models such as DVDnet. The goal of the reviewer is to understand the difference with the works that study image restoration by providing info about the degradation to the restoration neural-net. It will be good to see which method is better. 
Decision: Accept (Poster)