Fully Unsupervised Diversity Denoising with Convolutional Variational Autoencoders
Authors: Mangal Prakash
Alexander Krull
Florian Jug
Venue: ICLR 2021 Poster (2021)
Keywords: Diversity denoising
Unsupervised denoising
Variational Autoencoders
Noise model

Abstract: Deep Learning based methods have emerged as the indisputable leaders for virtually all image restoration tasks. Especially in the domain of microscopy images, various content-aware image restoration (CARE) approaches are now used to improve the interpretability of acquired data. Naturally, there are limitations to what can be restored in corrupted images, and like for all inverse problems, many potential solutions exist, and one of them must be chosen. Here, we propose DivNoising, a denoising approach based on fully convolutional variational autoencoders (VAEs), overcoming the problem of having to choose a single solution by predicting a whole distribution of denoised images. First we introduce a principled way of formulating the unsupervised denoising problem within the VAE framework by explicitly incorporating imaging noise models into the decoder. Our approach is fully unsupervised, only requiring noisy images and a suitable description of the imaging noise distribution. We show that such a noise model can either be measured, bootstrapped from noisy data, or co-learned during training. If desired, consensus predictions can be inferred from a set of DivNoising predictions, leading to competitive results with other unsupervised methods and, on occasion, even with the supervised state-of-the-art. DivNoising samples from the posterior enable a plethora of useful applications. We are (i) showing denoising results for 13 datasets, (ii) discussing how optical character recognition (OCR) applications can benefit from diverse predictions, and are (iii) demonstrating how instance cell segmentation improves when using diverse DivNoising predictions.

Published as a conference paper at ICLR 2021
FULLY UNSUPERVISED DIVERSITY DENOISING WITH CONVOLUTIONAL VARIATIONAL AUTOENCODERS
Mangal Prakash
Alexander Krull
Center for Systems Biology Dresden
School of Computer Science University of Birmingham Birmingham, UK f. w.w.fkrullee www.klblllaammmm ac. ac.uk
Max-Planck Institute (CBG) Dresden, Germany
prakashempi-cbg. de
Florian Jug
Center for Systems Biology Dresden Max-Planchs Institute (CBG) Dresden, Germany
Fondazione Human Technopole, Milano, Italy jug@mpi-ooe -cbg. de, florian. jug@fht.or
ABSTRACT
Deep Learning based methods have emerged as the indisputable leaders for virtually all image restoration tasks. Especially in the domain of microscopy images, various content-aware image restoration (CARE) approaches are now used to improve the interpretability of acquired data. Naturally, there are limitations to what can be restored in corrupted images, and like for inverse problems, many potential solutions exist, and one of them must be chosen. Here, we propose DIVNOISING, denoising approach based on fully convolutional variational autoencoders ( (VAEs), overcoming the problem of having to choose single solution by predicting a whole distribution of denoised images. First we introduce principled way of formulating the unsupervised denoising problem within the VAE framework by explicitly incorporating imaging noise models into the decoder. Our approach is fully unsupervised, only requiring noisy images and a suitable description of the imaging noise distribution. We show that such a noise model can either be measured, bootstrapped from noisy data, or co-learned during training. If desired, consensus predictions can be inferred from set of DIVNOISING predictions, leading to competitive results with other unsupervised methods and, on occasion, even with the supervised state-of-the-e DIvNOISING samples from the posterior enable a plethora of useful applications. We are (i) showing denoising results for 13 datasets, (di) discussing how optical character recognition (OCR) applications can benefit from diverse predictions, and are (iit) demonstrating how instance cell segmentation improves when using diverse DIVNOISING predictions.
INTRODUCTION
The goal of scientific image analysis is to analyze pixel-data and measure the properties of objects of interest in images. Pixel intensities are subject to undesired noise and other distortions, motivating an initial preprocessing step called image restoration. Image restoration is the task of removing unwanted noise and distortions, giving us clean images that are closer to the true but unknown signal.
In the past years, Deep Learning (DL) has enabled tremendous progress in image restoration (Mao al.) 2016) Zhang et al.) 20176] Zhang al., [2077] Weigert et al.) 2018). Supervised DL methods use corresponding pairs of clean and distorted images to learn mapping between the two quality levels. The utility of this approach is especially pronounced for microscopy image data of biological samples (Weigert et al. 2017; 2018 [Ouyang et al.] 2018 Wang et al. 2019), where quantitative downstream analysis is essential. More recently, unsupervised content-award image restoration
Shared first authors.
1 Shared last authors.

---

Published as a conference paper at ICLR 2021
Noiey Image (0
Latent Space
Sample (s')
Noise Model
Noisy Image (X)
TRAINING
d eyen
Caa()) f00000 (x)
Vu(x)
sule')
ano to
fu.. mr)
Σ locaee (rull (K))
eshaius
xt ualals)
observation
Encoder
Decuder
pu(()))
N(H.or
d eveh d eyeh
d eyen
d even d even and to .nd to
INVERENC
[o(x) Jr. or)
ou(x") **
and to ano to
and to
evhaus eshails
eknaus
exhaiis exhans
Noiry Image (X)
Space
Sample (s')
Sample (s5)
Sample (s*)
Figure 1: Training and prediction/inference with DIVNOISING. (top) A DivNoising VAE can be trained fully unsupervised. using only noisy data and (measured, bootstrapped, co-learned) pixel noise model you (ars) (s)) (see main text for details). (bottom) After training, the encoder can used to sample multiple z" go (z)x), giving rise to diverse denoised samples s² These samples can further be used to infer consensus point estimates such as MMSE or MAP solution.
(CARE) methods (Lehtinen et al., 2018 Krull et al. 2019 Batson & Royer 2019 Buchholz et al.) 2019) have emerged. They can, enabled by sensible assumptions about the statistics of imaging noise, learn mapping from noisy clean images, without ever seeing clean data during training. Some of these methods additionally include a probabilistic model of the imaging noise (Krull et al. 2020
Laine et al., 2019 Prakash et al. 2020: Khademi al. 2020)p to further improve their performance. Note that such denoisers can directly be trained on a given body of noisy images.
All existing approaches have common flaw: distortions degrade some of the information content in images, generally making it impossible to fully recover the desired clean signal with certainty. Even an ideal method cannot know which of many possible clean images really has given rise the degraded observation at hand. Hence, any restoration method has to make a compromise between possible solutions when predicting a restored image.
Generative models, such as VAEs, are a canonical choice when distribution over set of variables needs to be learned. Still, so far VAEs have been overlooked as method to solve unsupervised image denoising problems. This might also be due to the fact that vanilla VAEs (Kingma & Welling 2014 Rezende et al. 2014) show sub-par performance on denoising problems (see Section (6)
Here we introduce DIVNOISI principled approach to incorporate explicit models of the imaging noise distribution in the decoder of a VAE. Such noise models can be either measured or derived (bootstrapped) from the noisy image data alone (Krull et al. 2020 Prakash et al. 2020). Additionally we propose a way to co-learn a suitable noise model during training, rendering DIVISNGG fully unsupervised. We show on 13 datasets that fully convolutional VAEs, trained with our proposed DIVNOISING framework, yield competitive results, in 8 cases actually becoming the new state-of- the-art (see Fig. 2 and Table I Still, the key benefit of DIVNOISING is that the method does not need to commit to single prediction. but is instead capable of generating diverse samples from an approximate posterior of possible true signals. (Note that point estimates can still be inferred if desired, as shown in Fig. 4 Other unsupervised denoising methods only provide a single solution (point estimate) of that posterior (Krull et al. 2019; Lehtinen al. 2018 Batson & Royer 2019) or predict an independent posterior distribution of intensities per pixel (Krull et al., | Laine et al.) 2019. Prakash et al.) 2020 Khademi et al. 2020). Hence, DIVNOISING is the first method that learns to approximate the posterior over meaningful structures in a given body of images.
We believe that DIVNOISING will be hugely beneficial for computational biology applications in biomedical imaging, where noise is typically unavoidable and huge datasets need to be processed on a daily basis. Here, DIVNOIIING enables unsupervised diverse SOTA denoising while requiring only comparatively little computational resources, rendering our approach particularly practical.
Finally, we discuss the utility of diverse denoising results for OCR and showcase it for a ubiquitous analysis task in biology the instance segmentation of cells in microscopy images (see Fig. (5). Hence, DIvNOISING has the potential to be useful for many real-world applications and will not only generate state-of-the-art (SOTA) restored images, but also enrich quantitative downstream processing.
2

---

Published as a conference paper at ICLR 2021
2 RELATED WORK
Classical Denoising. The denoising problem has been addressed by variety of filtering ap- proaches. Arguably some the most prominent ones are Non-Local Means (Buades et al. 2005) and BM3D (Dabov et al., 2007), which implement sophisticated non-local filtering scheme. A comprehensive survey and in-depth discussion of such methods can be found in [Milanfar] 2012).
DL Based Denoising. Deep Learning methods which directly learn mapping from a noisy image to its clean counterpart (see e.g. /Zhng et af. (017a) and (Weigert et al. (018)) have outperformed classical denoising methods in recent years. Two well known contributions are the seminal works by Zhang et al. (2017a) and later by Weigert et al. (0188) More recently, a number of unsupervised variations have been proposed, and in Section [ we have deseribed their advantages and disadvantages in detail. One additional interesting contribution was made by Ullyanov et al. (2018), introducing quite different kind of unsupervised restoration approach. Their method, Deep Image Prior, trains a network separately for each noisy input image in the training set, making this approach computationally rather expensive. Furthermore, training has be stopped after a suitable but a priori unknown number of training steps.
Recently, Quan et al. (2020) proposed an interesting method called SELF2SELF which trains a U-NET like architecture requiring only single noisy images. The key idea of this approach is to use blind spot masking, similar to Krull et al. (2019), together with dropout (Srivastava et al., (0014). which avoids overfitting and allows sampling of diverse solutions. Similar to DIVNOISING. the single denoised result is obtained by averaging many diverse predictions. Diverse results obtained via dropout are generally considered to capture the so called epistemic or model uncertainty (Gal & Ghahramani 2016 Lakshminarayaaan et al. 2017). i.e. the uncertainty arising from the fact that we have a limited amount of training data available. In contrast, DIv NOISING combines a VAE and a model of the imaging noise to capture what is known as aleatoric or data uncertainty (Böhm eet al., 2019]] [Sensoy et al.] 2020), i.e. the unavoidable uncertainty about the true signal resulting from noisy measurements. Like in |Uyanov et al. (2018), also SELF2SELF trains separately on each image that has be denoised. While this renders the method universally applicable, it is computationally prohibitive when applied to large datasets. The same is true for real time applications such as facial denoising. DIVNOISING, on the other hand, is trained only once on a given body of data. Afterwards, it can be efficiently applied to new images. A detailed comparison of SELF2SELF and Drv NOISING in terms of denoising performance, run time and GPU memory requirements can be found in Appendix 14fd Appendix T
Denoising (Variational) Autoencoders. Despite the suggestive name, denoising variational autoen- coders (Im et al. 2017) are not solving denoising problems. Instead, this method proposes to add noise to the input data in order to boost the quality of encoder distributions. This, in turn, can lead to stronger generative models. Other methods also follow similar approach to improve overall performance of autoencoders (Vincent et al. 20088] (0110) Jiao et al. 2020).
VAES for Diverse Solution Sampling. Although not explored in the context of unsupervised de- noising, VAEs are designed to sample diverse solutions from trained posteriors. The probabilistic U-NET (Kohl et al., 2018 2019) uses conditional VAEs to learn a conditional distribution over seg- mentations. Baumgartner et al. (2019) improve the diversity of segmentation samples by introducing hierarchy of latent variables model segmentations multiple resolutions. Unlike NIVNOIIG both methods rely on paired training data. Nazabal et al. (2020) employ VA to learn the distribution of incomplete and heterogeneous data in a fully unsupervised manner. Babaeizadeh al. (2017)
build upon VAE style framework to predict multiple plausible future frames of videos conditioned on given context frames. A variational inference approach was used by Balakrishnan et al. (2019) to generate multiple deprojected samples for images and videos collapsed in either spatial or temporal dimensions. Unlike all these approaches, we address the uncertainty introduced by common imaging noise and show how denoised samples can improve downstream processing.
3 THE DENOISING TASK
Image restoration is the task of estimating a clean signal s (s1...., from a corrupted obser- vation x = (I)...., N,), where s; and 2;, refer to the respective pixel intensities. The corrupted is thought to be drawn from probability distribution px)())s), which we call the observation
3

---

Published as a conference paper at ICLR 2021
likelihood the noise model. In this work we focus on restoring images that suffer from insufficient illumination and detector/camera imperfections. Contrary to existing methods, DIVNOISING is designed to capture the inherent uncertainty of the denoising problem by learning suitable posterior distribution. Formally, the posterior we are interested in is p(s) x ())x)(x(s)))( components: the prior distribution p(s) of the signal as well as the observation likelihood (NM((1s) we introduced above. While the prior is a highly complex distribution, the likelihood p(xis) of given imaging system (camera/miiroocoop)) can be described analytically (Krull et al., 2020).
and depends on two
Models of Imaging Noise. The noise model is usually thought to factorize as a product of pixels, implying that the corruption, given the underlying signal, is occurring independently in each pixel as
NV.
p( x|s)
[ (N.(ls)).)
This assumption is known to hold true for Poisson shot noise and camera readout noise (Zhang et al. 2019 Krull et al. 2020 Prakash et al. 2020). We will refer the probability PNM(2.|s;) of observing particular noisy value Iú at pixel given clean signal s; as the pixel noise model. Various types of pixel noise models have been proposed, ranging from physies based analytical models (Zhang et al.
2019: Luisier al., 2010 Foi et al. 2008) to simple histograms (Krull et al.. 2220)) In this work, follow the Gaussian Mixture Model (GMM) based noise model description of (Prakash et al., 2020). The parameters of noise model can be estimated whenever pairs xx's') of corresponding noisy and clean calibration images are available (Krull et al. 2020). The signal s' + +E -ーー x" can then be 37
computed by averaging these noisy observations (Prakash et al.) (020)) In a case where no calibration data can be acquired, s' can be estimated by a bootstrapping approach (Prakash et al.) 2020). Later, we additionally show how suitable noise model can be co-learned during training.
THE VARIATIONAL AUTOENCODER (VAE)
We want to briefly introduce the VAE approach introduced by Kingma & Welling (2014). A more complete introduction to the topic can be found in (Doersch) 2016] & Welling (2019). VAEs
are generative models, capable of learning complex distributions over images x, such as hand written digits (Kingma & Welling 2014) or faces (Huang et al. 2018). To achieve this, VAES use a latent variable z with a fixed (usually a unit normal distribution) prior p(z) and describe
pe(x)=
po (z()))))(z)dz.
(2)
Like conventional autoencoders, they consist of two components: A decoder network gz(z) that takes a point in latent space and maps a distribution pe (x zz) image space and an encoder network fu(x), which takes an observed image and maps it to a distribution q0 (z(z)x) x) in latent space. By & and 0, we denote network parameters of the encoder and decoder, respectively.
Note that the decoder alone (together with suitable prior p(z)) is sufficient to completely describe the generative model in Eq. It usually modelled to factorize over pixels
(x(x = [ (z.
where po (2) is normal distribution, with its mean and variance predicted by the decoder network network ge (x). The encoder distribution is modelled in a similar fashion, factorizing over the dimensions of the latent space.
Training for VAES consists of adjusting the parameters 0 to make sure that Eq. 2 fits the distri- bution of training images encoder by jointly optimizing é and 0 to minimize the loss Lx,)(x) = CR (x) + 2 (x)((x, where po log lx) and [XL(() is the KL di- vergence KIL (cx(z))) While ckL can be computed analytically, the expected value in cB (x) is approximated by drawing a single sample z' from g,y (z(xx) and using the reparametrization trick by Kingmaa& & Welling (2014) for gradient computation.
x. Kingma et al. show that this can be achieved with the help of the
CR (x)0 [ lx|2)(x(2)]
Exate(x) Σ>
4

---

Published as a conference paper at ICLR 2021
Suyenisol CARE
DoNNeiing Sumple
DirNusiegt Suauple 2
Dwriuiing MMSE
Grouad tuds
Noayy arput
legut stop
Siampple 3- Samplee
FULPNE FULPNY PE-PEV Dewillg
Convlltui
W25 ebaooe
MONNT
9
3
7
9 9
8
7
3
7
dep aep
tlep
Hop
dep dep
BedD
Figure Qualitative denoising results. We compare two DIVNOISING samples, MMSE estimate (derived by averaging 1000 sampled images), and results by the supervised CARE baseline. The diversity between individual samples is visualized in the column of difference images. (See Appendix A.9 for additional images of DIVNOISING results.)
DIVNOISING
In DIvNOISING, we build on the VAE setup but interpret from a denoising-specific perspective.
We assume that images have been created from clean signal via known noise model, i.e., x ~ Px)(x To account for this within the VAE setup, we replace the generic normal distribution pixel intensities in Eq. with known noise model PNM (x|s) (see Eq. We get pe px(2))
(x|s) 11, (=;));)):), with the decoder now predicting the signal gz) = s. Together with p(2) and the noise model, the decoder now describes full joint model for all three variables, including the signal:
vo(z,x,s) PNM (xon(x(s)po((l2)p(z))
where we assume that (x|s, z) PNM (x|s). For given z as for standard Vs,s, the decoder describes a distribution over noisy images p(x|z). The corresponding clean signal s in contrast, is deterministically defined. Hence, Po(s|z) is a Dirac distribution centered at (z)...
Training. Considering Eq.
the reconstruction loss becomes
CR
(xe(z)x) - -lggpasss (z)) Apart from this modification, we can follow the stan- dard VAE training procedure, just as described in Section Since we have only modified how the decoder distribution modeled, we can assume that the traiming procedure still produces (i) a model describing the distribution of our training data, while (dx) making
that the encoder distribution well approximates the distribution of the latent variable given the image. A complete derivation of the DIVNOISING loss (from probability model perspective) can be found in Appendix A.12
5

---

Published as a conference paper at ICLR 2021
Figure 3: Sensibility of Noise Models. For each predicted signal intensity (x-axis), we show the variance of noisy observations (y- axis). The plot is generated from experiments on the Convallaria dataset. The dashed red line shows the true noise distribution (mea- sured from pairs of noisy and clean calibration data). This true distribution, as well as the noise model created via bootstrapping, and the noise model we co-learned with DIVNOISING, show sim- ple (approximately) lincar relationships between signal intensities and noise variance. Such relationship is known to coincide with the physical reality of Paisson noise (shot noise) (Zhang al. 2019). The implicitly learned noise model of the vanilla VAE has to independently predict the noise variance for each pixel. Its predictions clearly deviate from the true linear relationship. See Appendix (A.13 A..13 for results on BiolD Face dataset and more details.
(GT)
Se10
Booetaepol (Jom maka data)
Co-kamed (VAD)
Pixel make variance
Co-komed (DedNoking)
2010
Ix10-
1000
2010 Pisel signal intensity
X00
Fully Unsupervised
Unsup (PNM requ.)
Supervised CARE
Dataset
N2V
Vanilla VAE
DivNoising
PN2V
DivNoising
FU-NNY
Convallaria
35.03-10.037
36.5710.003
36.7810.007
36.4710.031
36.90-10.004
36.71
Buvistraped
36.7010.012
36.010
Mouse Act.
33.39:00.014
33.46+00.188
33.82+0.006
0.086+000118
33.99+0.0004
34.20+0.021
Mouse Nuc.
35.84=0.015
35.84+0.023
36.05+0.052
0.008
36.26+000077
36.58+0.011
Ch.0 bwgb
34.59-0.041
33.02+F0.147
34.24+0.006
34.13++0.002 35. 0.609
Ch. (ogg))
32.11+0.030
31.36+00001
32.22+0.021
32.22-+-0013
32.88+00021
W25
Ch.2 (argl)
35.04:10.033
33.72110.87
35.2410.028
32.7910.085
35.18110.020 30.90
Ch.0 (avgtó)
39.01
39.2740.192
39.4510.036
39.3610.103
39.63-1.0007 42.05
Ch.1 (ovg16)
37.91
38.33+0.021
38.4140.018
0..11
38.39+0.007 3..64
Ch.2 (oogtt)
40.30=+0.023
40.24400.03
40.56+0.019
40.36
40.41 42.02
Denoneer
Mouse
33.84
34.06110.003
34.0610.0005
34.19
34.13110.003 10.011
Flywing
24.7910.033
24.88100.045
24.9210.016
20.05
25.02:10.024 25.79110.014
Mouse s&p
32.98+0.020
23.62+0.084
35.10+0.030
29.67+0.079
36.21 37.03+0.016
BiolD Face
30.04
32.5810.0222 33.0210.020 33.7610.079 33.1210.039 35.06
Table I: Quantitative results. For all experiments,
compare all results in terms of mean Peak Signal-to- Noise Ratio (PSNR in dB) and 11 standard error over 5 runs. Overall best performance indicated by being underlined, best unsupervised method in bold, and best fully unsupervised method in italic. For many datasets, DIVNOISING is the unsupervised SOTA, typically not being far behind the supervised CARE results.
Prediction. While we can use the trained VAE to generate images from pe (x) (see Appendix A.5). here we mainly interested in denoising. Hence, we desire access the posterior p(s) x), i.e. the distribution of possible clean signals s given a noisy observation x. Assuming the encoder and decoder sufficiently well trained, samples s from approximate posterior can be obtained by (i) feeding the noisy image x into our encoder, (ii) drawing samples z² ~ go (z(x), and (iiz) decoding the samples via the decoder to get * (z*).
Inference. Given set of posterior samples s for a noisy image x, we can infer different consensus estimates (point estimates). We can, for example, approximate the MMSE estimate (see Fig. 2). by averaging many samples s². Alternatively, we can attempt find the maximum posteriori (MAP) estimate, i.e. the most likely signal given the noisy observation x, by finding the mode of the posterior distribution. For this purpose, we iteratively use the mean shift algorithm (Cheng 1995) with decreasing bandwidth to find the mode of our sample set (see Fig. 4 and Appendix(A.4 A.4
Fully Unsupervised DivNoising. So far we explained our setup under the assumption that the noise model can either be measured with paired calibration images, or bootstrapped from noisy data (Prakash et al. (020). Here, we propose yet another alternative approach of co-learning the noise model directly from noisy data during training. More concretely, this is enabled by a simple modification to the DIVNOISING decoder. We assume that the noise at each pixel follows a normal distribution with its variance being a linear function of s, i.e., o² as, + 6. Linearity is motivated by noise properties in low-light settings (Faraji & MacLean 2006 Jezierska et al., 2011). The learnable network parameters and b are co-optimized during training. Since variances cannot be negative, we additionally constrain the predicted values for of to be positive (see Appendix A.3 for details).
6

---

Published as a conference paper at ICLR 2021
Denoising with Vanilla VAEs. While not originally intended for denoising tasks, we were curious to see how vanilla VAEs perform when applied to these problems. Just like fully unsupervised DIVNOISING, also the vanilla VAE does not require noise model. It does, instead, directly predict per-pixel mean and variance (see Section 4 leaving the possibility open that the right values could be learned. However, here the decoder is not restricted to make each pixel's variance function of predicted signal. We investigate the denoising performance of the vanilla VAE in Section (ection[Jand show in Fig. D that the predicted variances significantly diverge from ground truth noise distributions.
Signal Prior in NIYSINGG Classical denoising methods often explicitly model the image/signal
prior p(s) e.g. as smoothness priors (Grimson & Grimson 1981 Li 1994), non-local similarity priors (Buades et al.) [005] Dabov et al. (007). sparseness priors (Tibshirani 1996) etc., assum- ing specific properties of the images at hand. They effectively assign the same probability to all images/signals sharing e.g. the same level of smoothness. However, the true distribution p(s) of clean signals (e.g. for a particular experimental setup in a fluorescence microscope) gener- ally more complex. Instead of explicitly modelling p(s), DIVNOISING only implicitly describes Po(s) = [P z p(z)dz as integral over all possible values of z. We recall that the prior p(z) is assumed to be the unit Gaussian distribution and the conditional distribution pe (s) z) is learned by the decoder network as the Dirac distribution centered at go (z). Depending on its parameters 8, the network will implement the function differently, leading different po))). This implicit distribution is quite powerful and can capture complex structures. See Appendix A.5 for samples obtained from this signal prior for different datasets.
different po(s) z), and ultimately to
6 DATA, EXPERIMENTS, RESULTS
We quantitatively evaluated the performance of DIVNOISING on 13 publicly available datasets (see Appendices A. |.2 for data details), 9 of which are subject to high levels of intrinsic (real world) noise. To 4 others we synthetically added noise, hence giving us full knowledge about the nature of the added noise.
Denoising Baselines. We choose tfae-oof-the-ar baseline methods to compare against DIVNOISING, namely, the supervised CARE (Weigert et al. 2018) and the unsupervised methods NOISE2 VOID (N2V) (Krull al. 2019 and Probabilistic NoIsE2VoID (PN2V) (Krull et al. 2019). All baselines use the available implementations of (Krull et al. 2020) and, as long as not specified otherwise, make use of a depth 3 U-NET with input channel and 64 channels in the first layer. As an additional baseline, we choose vanilla VA with the same network architecture as DIVNOISING, but predicting per pixel mean and variance independently. Training is performed using the ADAM (Kingma & Ba 2015) optimizer for 200 epochs with 10 steps per epoch with batch size of 4 and virtual batch size of 20 for N2V and CARE and a batch size of 1 and a virtual batch size of 20 for PN2V, an initial learning rate of 0.001, and the same basic learning rate scheduler in (Krull al. 2020 All baselines use on the fly data augmentation (flipping and rotation) during training.
Training Details. In all experiments we use rather small, fully convolutional VAE networks, with either 200k or 713k parameters (see Appendix (A.3). For all experiments on intrinsically noisy microscopy data, validation and test set splits follow the ones described in the respective publication. In contrast to the synthetically noisy data, no apriori noise model is known for microscopy datasets. For these datasets, we used GMM-based noise models (Prakash et al.. [2200] Khademi al. 2020). which are measured from calibration images, as well as co-learned noise models. For the W2S datasets, no dedicated calibration samples to create noise models are available. Hence, for this dataset, we use the available clean ground truth images and all noisy observations of the training data to learn a GMM- noise model. All GMM noise models use Gaussians and coefficients each. Find more training details in Appendix A.3
Denoising Results. In Table D. we report denoising performance of all experiments we conducted in terms of peak signal-to-noise noise ratio (PSNR) with respect to available ground truth images. The DIVNOISING results (using the MMSE estimate from 1000 averaged samples) are typically either on par or even beyond the denoising quality reached by the baselines in the 'fully unsupervised' category, as well as the "unsupervised with noise model' category.
Note that sampling is very efficient. For all presented experiments sampling 1000 images consistently took less than 7 seconds (see Table Appendix [A.15)oo for precise sampling times). The effect of averaging different number of samples is explored in Appendix A.8
7

---

Published as a conference paper at ICLR 2021
Input
MMSK Estimate
Clussers Posterion
MAP Estimate
Ground Truth
hing ob whio ob whic
wad ounme higgger
ob whic
egged hung hung hing egged i Degged
hunger Birhghung hinng hungor hunger
Figure 4: Exploring the learned posterior. The MMSE estimate (average of 10k samples) shows faintly overlaid letters as consequence of ambiguities in noisy input. Among these samples from the posterior, we use mean shift clustering (on smaller crops) to identify diverse and likely points in the posterior. We show 9 such cluster centers in particular order. We also obtain an approximate MAP estimate (see Supplementary Material), which has most artifacts of the MMSE solution removed.
Low SNR
DivNoising Sample + Soç
DivNoising Sample + Sog.
DivNoising Sample Seg.
DivNoising Sample 4 Seg.
Consensus/ Fasion
DivNoising MMSE Seg.
High SNR Seg.
+ Seg..
Methools
PIC
Ave.
Figure 5: DIYNOISING enables downstream segmentation. We show input images (upper row) and results of fixed (untrained) segmentation pipeline (lower row). Cells that were segmented incorrectly (merged or split) are indicated in magenta. While segmentations of the noisy raw data are of very poor quality, sampled DIVNOISING results give rise to much better and diverse solutions (cols. 2-5). We then use two label fusion methods to find consensus segmentations (col. 6), which are even outperforming segmentation results on high SNR (GT) images. Quantitative results are presented in Appendix Fig. 11
DIVNOISING MMSE typically, expected, slightly behind the performance of the fully supervised baseline CARE (Weigert et al. 2018). Additionally, on FU-PN2V Convallaria we have demonstrated that a suitable noise model for DIVNOISING can created via bootstrapping (Prakash al. 2020, Khademi et al., 2020). We also compare against Deep Image Prior (DIP) on DenoiSeg Flywing dataset as has smallest number of test images and DIP has to be trained for each image. DIP achieves PSNR of 24.67 + 0.050dB compared to 25.02 + 0.024dB with DIvNOISING.
Due to the extensive computational requirements of SELF2SELF, we cannot run the method on all images in any of our dataset. Instead, we run it on single, randomly selected images from the FU-PN2V Convallaria, FU-PN2V Mouse actin, FU-PN2V Mouse nuclei, and W2S Ch. (avgl) datasets. We compare SELF2SELF to DIVNOISING when (i) trained the same randomly chosen image from the respective dataset, and (ii) when NIVSING was trained on the entire dataset and applied the respective randomly selected image. Within generous time limit of 10 hours for training per image, DIV still outperforms SELF2SELF in measured PSNR performance while requiring about 7 times less GPU memory (see Appendix A.14 Appendix Table 2 Note that the application of 1ELF2SELF to an entire dataset containing 100 images would require 1000 hours of cumulative training time, while an overall 10 hour training of DIVNOISING dataset is sufficient to denoise all contained images. The performance on the natural image benchmark dataset BSD68 (Roth& & Black 2005) is shown in Fig. and discussed in Appendix A0 Additional qualitative results for all datasets can be found in Appendix A.9 A discussion on the accuracy of the posterior modeled by DIVNOISING can be found in Appendix A.1I
the entire
A.
Downstream Processing: OCR. In Fig. 4 we show how Optical Character Recognition (OCR) applications might benefit from diverse denoising. While regular denoising approaches predict poor compromises that would never be seen in clean text, DIVSSIIG can generate diverse set of rather plausible denoised solutions. While our MAP estimates clean up most such problems, occasional mistakes cannot be avoided, e.g. changing "hunger" to "hungor" (see Fig. 4 Diverse denoising solutions obtained by clustering typically correspond to plausible alternative interpretations. It stands to reason that OCR systems can benefit from having access to diverse interpretations.
Downstream Processing: Instance Cell Segmentation. We demonstrate how diverse denoised images generated with DIVNOISING can help segment all cells in the DenoiSeg Flywing data.
8

---

Published as a conference paper at ICLR 2021
While methods to generate diverse segmentations do exist (Kohl et al. 2018 2019), they require ground truth segmentation labels during training. In contrast, we use a simple and fast downstream segmentation pipeline c(x) based on local thresholding and skelctonization (see Appendix AA6 for details) and apply it to individual samples (s.....s) predicted by DIVNOISING to derive segmentations (c (c!....c ...... We explore two label fusion methods to combine the individual results and obtain an improved segmentation. We do: (i) use Consensus (BIC) (Emre Akbas et al. 2018)
and (ii) create a pixel-wise average of (c* ... .c)... followed by again applying our threshold based segmentation procedure on this average, calling it Consensus (Avg).
For comparison, we also segment (2) the low SNR input images, (ti) the original high SNR images. and (dii) the MMSE solutions of DIvNOISING. Figure and Appendix Fig. [1 show all results of our instance segmentation experiments. is important to note that segmentation from even a single DIV prediction outperforms segmentations on the low SNR image data quite substantially. We observe that label fusion methods can, by utilizing multiple samples, outperform the MMSE estimate, with Consensus (Arg) giving the best overall results (see Appendix Fig, ITp.
7 DISCUSSION AND CONCLUSION
We have introduced DIV NOISING, novel unsupervised denoising paradigm that allows us, for the first time, to generate diverse and plausible denoising solutions, sampled from a learned posterior. We have demonstrated that the quality of denoised images is highly competitive, typically outperforming the unsupervised state-of titte-of-hhearrt thet and at times even improving on supervised results
DIVNOISING uses a lightweight fully convolutional architecture. The success of Deep Image Prior (Ulyanov et al. (2018) shows that convolutional neural networks are inherently suitable for image denoising. Yokota et al. (2019) reinforce this idea and Tachella et al. (2020) additionally hypothesize that possible reason for the success of convolutional networks is their similarity to non-local patch based filtering techniques. However, the overall performance of DIVIINIG is not merely a consequence of its convolutional architecture. We believe that the novel and explicit modeling of imaging noise in the decoder plays an essential role. This becomes evident when comparing our results to other convolutional baselines (including Deep Image Prior and fully convolutional VAEs), which do not perform as well as DIVNOISING on any of the datasets we used. Additionally, we observe that incorrect noise models consistently lead inferior results (see Appendix A.7].
We find that NOVSIIG is suited particularly well for microscopy data or other applications on limited image domain. In its current form it works less well on collections of natural images (see Appendix 10) This might not be very surprising, as we are training a generative model for our image data and would not expect to be capturing the tremendously diverse domain of natural photographic images with the comparatively tiny networks used in our experiments (see Appendix A..3). For microscopy data, instead, the diversity between datasets can be huge. Images of the same type of sample, acquired using the same experimental setup, however, contain many resembling structures of lesser overall diversity (they are from a limited image domain). Nevertheless, the stunning results we achieve suggest that DIVNOISING will also find application in other areas where low SNR limited domain image data has to be analyzed. Next to microscopy, we can think of astronomy, medical imaging, or limited domain natural images such faces or street scenes. Additionally, follow up research will explore larger and improved network architectures, able to capture more complex DIVNOISING posteriors datasets covering larger image domains.
While we constrained ourselves to the standard per-pixel noise models in this paper, the DIVNOISING approach could in principle also work with more sophisticated higher level image degradation models, as long as they can be prohabilistically described. This might include diffraction, blur, or even compression and demosaicing artefacts.
Maybe most importantly, DIVNOISING can not only produce competitive and diverse results, but these results can also be leveraged for downstream processing. We have seen that cell segmentation can be improved and that clustering our results provides us with meaningful alternative interpretations of the same data (see Fig. (4. We believe that this is highly promising direction for many applications. as it provides us with a way to account for the uncertainty introduced by the imaging process. We are looking forward to see how DIVNOISING will be applied and extended by the community, showcasing the true potential and limitations of this approach.
SSpervvisdd methods using perfect GT will outperform DIVNOISING, hut GT data is times not perfect.

---

Published as a conference paper at ICLR 2021
REFERENCES
BiolD Face Database Face Detection Dataset
facedb.
URL htt.taa//ww.bio.....cco
facedb/
Mohammad Babacizadeh, Chelsea Finn, Dumitru Erhan, Roy H Campbell, and Sergey Levine. Stochastic variational video prediction. arXiv preprint arXiv: /710. /252, 2017.
Guha Balakrishnan, Adrian V Dalca, Amy Zhao, John V Guttag, Fredo Durand, and William Freeman. Visual deprojection: Probabilistic recovery of collapsed dimensions. In Proceedings of the IEEE International Conference on Computer Vision, pp. 171- 2019.
Joshua Batson and Loic Royer. Noise2self: Blind denoising by self-supervision, 2019.
Christian F Baumgartner, Kerem C. Tezcan, Krishna Chaitanya, Andreas M Hötker, Urs J Muehlemat- ter, Khoschy Schawkat, Anton S Becker, Olivio Donati, and Ender Konukoglu. Phiseg: Capturing uncertainty in medical image segmentation. In International Conference Medical Image Computing and Computer-Assister Intervention, pp. 119-127. Springer, 2019.
Vanessa Böhm, François Lanusse, and Uroš Seljak. Uncertainty quantification with generative models. arXiv preprint arXiv: 19/0./9004/000000 2019.
Samuel R Bowman, Luke Vilnis, Oriol Vinyals, Andrew M Dai, Rafal Jozefowiez, and Samy Bengio. Generating sentences from continuous space. arXiv preprint
2015.
Antoni Buades, Bartomeu Coll, and J-M Morel. A non-local algorithm for image denoising. In 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR (OS). volume 2, pp. 60-65. IEEE, 2005.
Tim-Oliver Buchholz, Alexander Krull, Réza Shahidi, Gaia Pigino, Gáspár Jékely, and Florian Jug. Content-aware image restoration for electron microscopy. In Gaia Pigino and Thomas Müller- Reichert (eds.), Stree-Dimensional Electron Microscopy, pp. 277-289. Academic Press, July 2019.
Tim-Oliver Buchholz, Mangal Prakash, Deborah Schmidt, Alexander Krull, and Florian Jug. De- noiseg: joint denoising and segmentation. In European Conference on Computer Vision, pp. 324-337. Springer, 2020.
Yizong Cheng. Mean shift, mode seeking, and clustering. IEEE transactions on pattern analysis and machine intelligence, 17(8):90-779, 1995.
Tarin Clanuwat, Mikel Bober-Irizat Asanobu Kitamoto, Alex Lamb, Kazuaki Yamamoto, and David Ha. Deep learning for classical japanese literature. arXiv preprint 18120///11.0///1 2018.
Kostadin Dabov, Alessandro Foi, Vladimir Katkovnik, and Karen Egiazarian. Image denoising by sparse 3-d transform-domatt collaborative filtering. IEEE Transactions on image processing, 16 (8):2080-2095, 2007.
Carl Doersch. Tutorial variational autoencoders. arXiv preprint arXiv: 2016.
Cem Emre Akbas, Vladimír Ulman, Martin Maska, Florian Jug, and Michal Kozubek. Automatic fusion of segmentation and tracking labels. In Proceedings of the European Conference on Computer Vision (ECCV), pp. 0-0, 2018.
Hilda Faraji and W James MacLean. Ced noise removal in digital images. IEEE Transactions on image processing, 15(9):2676-268) 2006.
Alessandro Foi, Mejdi Trimeche, Vladimir Katkovnik, and Karen Egiazarian. Practical poissonian- gaussian noise modeling and fitting for single-image raw-data. IEEE Transactions on Image Processing. 17(10):1777-1544 2008
Yarin Gal and Zoubin Ghahramani. Dropout bayesian approximation: Representing model uncertainty deep learning. In international conference on machine learning, pp. 1050-1059, 2016.
10

---

Published as a conference paper at ICLR 2021
William Eric Leifur Grimson and WEL Grimson. From images surfaces: A computaational study of the human early visual system, volume 4. MIT press Cambridge, MA, 1981.
Irina Higgins, Loic Matthey, Arka Pal, Christopher Burgess, Xavier Glorot, Matthew Botvinick, Shakir Mohamed, and Alexander Lerchner. beta-vae: vae: Learning basic visual concepts with a constrained variational framework. Iclr, 2(5):6, 2017.
Huaibo Huang, Ran He, Zhenan Sun, Tieniu Tan, et al. Introvae: Introspective variational autoen- coders for photographic image synthesis. In Advances in neural information processing systems, pp. 52-63, 2018.
Daniel Im Jiwoong Im, Sungjin Ahn, Roland Memisevic, and Yoshua Bengio. Denoising criterion for variational auto-encoding framework. In Thirty-First AAAI Conference on Artificial Intelligence, 2017.
Paul Jaccard. Érude comparative de la distribution florale dans une portion des alpes et des jura. Bull Soc Vaudoise Sci Nat, 37:547-579, 1901.
Anna Jezierska, Caroline Chaux, -han-Chritt Pesquet, and Hugues Talbot. An em approach for
noise modeling. 2011 19th European Signal Processing Conference, pp. 2244-2248. IEEE, 2011.
Jianbo Jiao. Linchao Bao, Yunchao Wei, Shengfeng He, Honghui Shi, Rynson Lau, and Thomas Huang. Laplacian denoising autoencoder. arXiv preprint 2XXv:20003.16223 2020.
Wesley Khademi. Sonia Rao, Clare Minnerath, Guy Hagen, and Jonathan Ventura. Self-supervised poisson-gaussian denoising. arXiv preprint arXiv:2002.0955 2020.
Diederik Kingma and Jimmy Ba. Adam: A method for stochastic optimization. 3rd international conference on learning representations, iclr 2015. arXiv preprint arXiv: 1412.6980 9, 2015.
Diederik P. Kingma and Max Welling, Auto-encoding variational bayes. In Yoshua Bengio and Yann LeCun (eds.), 2nd International Conference on Learning Representations ICLR 2014, Banff, AB, Canada, April 14-16, 2014, Conference Track Proceedings, 2014. URL http:
//arxiv. www.v.orr/aaa abs / 1312. 6114
Diederik Kingma and Max Welling. An introduction to variational autoencoders. Foundations and T in Machine Learning, 12(4):307-39, 2019. ISSN 1935-8237. doi: 10.1561/2200000056. /ttppp////.......oo//000.1111//1 0tt.
URL http: doi
Simon Kohl, Bernardino Komera-Paredes, Clemens Meyer, Jeffrey De Fauw, Joseph R Ledsam, Klaus Maier-Hein, SM Ali Eslami, Danilo Jimenez Rezende, and Olaf Ronneberger. probabilistic u-net for segmentation of ambiguous images. In Advances in Neural Information Processing Systems, pp. 6965-6975, 2018
Simon AA Kohl, Bernardino Romera-Paredes, Klaus H Maier-Hein, Danilo Jimenez Rezende, SM Es- lami, Pushmeet Kohli, Andrew Zisserman, and Olaf Ronneberger. hierarchical probabilistic u-net for modeling multi-scale ambiguities. arXiv preprint arXiv: 1905. 2019.
Alexander Krull, Tim-Oliver Buchholz, and Florian Jug. Goise2void-learning denoising from single noisy images. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition, pp. 22129-21337 2019.
Alexander Krull, Tomas Vicar, Mangal Prakash, Manan Lalit, and Florian Jug. Probabilistic Vois: Unsupervised Content-Aware Denoising. Front. Comput. Sci., 2:60, February 2020.
Samuli Laine, Tero Karras, Jaakko Lehtinen, and Timo Aila. High-quality self-supervised deep image denoising. Advances in Neural Information Processing Systems, pp. 6968-6978, 2019.
Balaji Lakshminarayanan, Alexander Pritzel, and Charles Blundell. Simple and scalable predictive uncertainty estimation using deep ensembles. In Advances in neural information processing systems, pp. 6402-6413, 2017.
11

---

Published as a conference paper at ICLR 2021
Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient- learning applied to document recognition. Proceedings of the IEEE, (111)):227822222 1998.
Jaakko Lehtinen, Jacob Munkberg. Jon Hasselgren, Samuli Laine, Tero Karras, Miika Aittala, and Timo Aila. Noise2noise: Learning image restoration without clean data. arXiv preprint arXiv:/803.04J0 2018.
Stan Z Li. Markov random field models in computer vision. In European conference computer vision, pp. 361-370. Springer, 1994.
Tsung-Yi Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In European conference on computer vision, pp. 740-755. Springer, 2014.
Florian Luisier, Thierry Blu, and Michael Unser. Image denoising in mixed poisson-gaussicn noise. IEEE Transactions image processing, 20(3):696-708, 2010.
Xiaojiao Mao, Chunhua Shen, and Yu-Bin Yang. Image restoration using very deep convolutional pecoder-decodee networks with symmetric skip connections. In D. D. Lee, M. Sugiyama. U. V. Luxburg. I. Guyon, and R. Gamnett (eds.), Advances Neural Information Processing Systems 29, pp. 2802-28 10. 2016.
Richard Marsh. The beetle. Broadview Press, 2004.
Peyman Milanfar. A tour of modern image filtering: New insights and methods, both practical and theoretical. IEEE signal processing magazine, 30(1):106-128, 2012.
Alfredo Nazabal, Pablo M Olmos, Zoubin Ghahramani, and Isabel Valera. Handling incomplete heterogeneous data using vaes. Pattern Recognition, pp. 107501, 2020.
Wei Ouyang, Andrey Aristov, Mickaël Lelek, Xian Hao, and Christophe Zimmer. Deep learning massively accelerates super-resolution localization microscopy. Nat Biotechnol, 27:2672, April 2018.
Mangal Prakash, Manan Lalit, Pavel Tomancak, Alexander Krul, and Florian Jug. Fully unsupervised probabilistic noise2void. In 2020 IEEE 17th International Symposium on Biomedical Imaging (ISBI), pp. 154-158. IEEE, 2020.
Yuhui Quan, Mingqin Chen, Tongyao Pang, and Hui Ji. Self2self with dropout: Learning self- supervised denoising from single image. Proceedings the IEEE/CVF Conference Computer Vision and Patlern Recognition, pp. 1890-1898, 2020.
Danilo Jimenez Rezende, Shakir Mohamed, and Daan Wierstra. Stochastic backpropagation and approximate inference deep generative models. arXiv preprint 441.v:1401.4082, 2014.
Stefan Roth and Michael Black. Fields of experts: framework for learning image priors. In 2005 IEEE Computer Society Conference Computer Vision and Pattern Recognition (CVPR'0H '05), volume 2. pp. 860-867. IEEE, 2005.
Johannes Schindelin, Ignacio Arganda-Carmn Carreras, Erwin Frise, Verena Kaynig, Mark Longair, Tobias Pietzsch, Stephan Preibisch, Curtis Rueden, Stephan Saalfeld, Benjamin Schmid, Yves Tinevez, Daniel James White, Volker Hartenstein, Kevin Eliceiri, Pavel Tomancak, and Albert Cardona. Fiji: an open-source platform for biological-image analysis. Nature Methods, 9(7): 676-682, July 2012.
Murat Sensoy, Lance Kaplan, Federico Cerutti, and Maryam Saleki. Uncertainty-aware deep classifiers using generative models. arXiv preprint Sit::2006.04/8.. 2020.
Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. Dropout: simple way prevent neural networks from overfitting. The journal of machine learning research, 19(1)11929-1958 2014.
Julián Tachella, Junqi Tang, and Mike Davies. Cnn denoisers as non-local filters: The neural tangent denoiser. arXiv preprint arXiv:2006.02379 2020.
12

---

Published as a conference paper at ICLR 2021
Robert Tibshirani. Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society: Series B (Methodological), 58(1):267-28) 1996.
Dmitry Ulyanov, Andrea Vedaldi, and Victor Lempitsky. Deep image prior. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 9446-9454, 2018.
Comellis Joost Van Rijsbergen. Information retrieval. Citeseer, 1979.
Pascal Vincent, Hugo Larochelle, Yoshua Bengio, and Pierre-Antoing Manzagol. Extracting and composing robust features with denoising autoencoders. In Proceedings of the 25th international conference on Machine learning, pp. 1096-1103, 2008.
Pascal Vincent, Hugo Larochelle, Isabelle Lajoie, Yoshua Bengio, Pierre-Antoine Manzagol, and
Léon Bottou. Stacked denoising autoencoders: Learning useful representations in deep network with a local denoising criterion. Journal of machine learning research. 11(12), 2010.
Hongda Wang, Yair Rivenson, Yiyin Jin, Zhensong Wei, Ronald Gao, Harun Günaydın, Laurent A Bentolila, Comert Kural, and Aydogan Ozean. Deep learning enables cross-modality super- resolution in fluorescence microscopy. Nature Publishing Group, (6(1):103-111) January 2019.
Martin Weigert, Loic Royer, Florian Jug, and Gene Myers. Isotropic reconstruction of 3D fluorescence microscopy images using convolutional neural networks. arXiv, April 2017.
Martin Weigert, Uwe Schmidt, Tobias Boothe, Andreas Müller, Alexandr Dibrov, Akanksha Jain, Benjamin Wilhelm, Deborah Schmidt, Coleman Broaddus, Siān Culley, al. Content-aware image restoration: pushing the limits of fluorescence microscopy. Nature methods, 15(12):0000-1097, 2018.
Tatsuya Yokota, Hidekata Hontani, Qibin Zhao, and Andrzej Cichocki. Manifold modeling in embed- ded space: A perspective for interpreting" deep image prior". arXiv preprint arXiv:1908. 02995, 2019.
K. Zhang, W. Zuo, Y. Chen, D. Meng, and L.. Zhang. Beyond a gaussian denoiser: Residual learning of deep enn for image denoising. IEEE Transactions Image Processing, 26(7):3142-3155 2017.
Kai Zhang, Wangmeng Zuo, Yunjin Chen, Deyu Meng, and Lei Zhang. Beyond a gaussian denoiser: Residual learning of deep enn for image denoising. IEEE Transactions on Image Processing, 26 (7):3142-3155, 2017a.
Kai Zhang, Wangmeng Zuo. Shuhang Gu, and Lei Zhang. Learning deep enn denoiser prior for image restoration. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), July 2017b.
Yide Zhang, Yinhao Zhu, Evan Nichols, Qingfei Wang, Siyuan Zhang, Cody Smith, and Scott Howard. A poisson-gaussian denoising dataset with real fluorescence microscopy images. In CVPR, 2019.
Ruofan Zhou, Majed Helou, Daniel Sage, Thierry Laroche, Ame Seitz, and Sabine Süsstrunk. W2s: A joint denoising and super-resolution dataset. arXiv preprint arXiv:2003.05961, 2020.
13

---

Published as a conference paper at ICLR 2021
A APPENDIX
A.1 INTRINSICALLY NOISY MICROSCOPY DATA
We use public microscopy datasets which show realistic levels of noise, introduced by the respective optical imaging setups. The FU-PN2V Convallaria (Krull et al.) 2020 Prakash et al. 2020) data, consists of 100 noisy calibration images (intended to generate noise model), and 100 images of size 1024 x 1024 showing a noisy Convallaria section. The FU-PN2V Mouse nuclei (Prakash et al. 2020) data is composed of 500 noisy calibration images and 200 noisy images of size 512 x 512 showing labeled cell nuclei. The FU-PN2V Mouse actin (Prakash et al. 2020) data from the same source consists of 100 noisy calibration images and 100 noisy images of size 1024 x 1024 of the same sample, but labeled for the protein actin. Finally, we use all 3 channels of 2 noise levels (avg1 and 16g of the W2S (Zhou et al.) 2020) data. For each channel, corresponding high quality (ground truth) images are available. Each channel's training and test sets consist of 80 and 40 images, respectively. All images are 512 x 512 pixels in size.
A.2 DATA EXPOSED TO SYNTHETIC NOISE
We use the well known MNIST (LeCun al. 1998) well the KMNIST (Clanuwat al. 2018)
dataset showing 28 x 28 images of handwritten digits and phonetic letters of hiragana, respectively. Both datasets contain 60000 training examples and 10000 test examples. Onto both datasets added pixel-wise independent Gaussian noise with ja = 0 and o = 140. As a third text-based dataset we rendered the freely available eBook "The Beetle" (Marsh 2004) and extracted 40800 image patches of size 128 x 128. We separated 34680 patches for training and 6120 patches for validation, and added pixel-wise independent Gaussian noise with μr = 0 and o 255. Additionally, we use three datasets from microscopy. The DenoiSeg Mouse (Buchholz et al. 2020) data, showing cell nuclei in the developing mouse skull, consists of 908 training and 160 validation images of size 128 x 128, with additional 67 images of size 256 x 256 for testing. Two noisy datasets were created with this data, one by exposing all images to pixel-wise independent Gaussian noise with μ = 0 and o = 20 and another one by first applying poisson noise with À = followed by adding gaussian noise with u = 0 and a = 10 followed by randomly changing 3% of pixels to either 0 or 255. This dataset is called Mouse s&p in Table 1 The DenoiSeg Flywing (Buchholz et al. 2020) data is showing membrane labeled cells in a fly wing, consisting of 1428 training and 252 validation patches of size 128 x 128, with additional 42 images of size 512 512 for testing. We exposed this data pixel-wise independent Gaussian noise with μ 0 and o = 70 to create a synthetic low SNR version. All original datasets are 8-bit. Lastly, we randomly select 500 images of size 384 286 from BiolD Face recognition database (noa) and corrupt them with pixel-wise independent Gaussian noise with = 0 and a 15. We use 340, 60 and 100 images for training, validation and test respectively.
A.3 TRAINING AND NETWORK DETAILS
Here, we provide additional details about the network architecture and training parameters used throughout the main manuscript. For all DIVNOISING experiments, we use rather lightweight depth 2 and depth 3 VAE architectures (see Appendix Figs. [ and 7 respectively). All networks use a single input channel and 32 feature channels in the first network layer except for the network trained on mouse s&p dataset which uses 96 feature channels in the first network layer. We use two 3 x 3 convolutions (with padding 1), each followed by ReI.U activation, followed by 2 x 2 max pooling layer. After each such downsampling step, we double the number of feature channels. For all experiments we use network architecture of depth 2 (with 2 down/upsampling steps). The only exceptions are our experiments on DenoiSeg Flywing and eBook data, for which we use a depth architecture (with 3 down/upsampling steps). In total, our depth networks have only around 2004: parameters and depth 3 networks have around 700k parameters.
While we generally use VAE bottleneck of 64 latent space feature dimensions for each pixel of the image (after encoding), for the small 28 x 28 MNIST and KMNIST images we use only 8 such latent space dimensions.
14

---

Published as a conference paper at ICLR 2021
We consistently use 8-fold data augmentation (rotation and flipping) in all experiments. All networks are trained with a batch size of 32 and an initial learning rate of 0.001. The learning rate multiplied by 0.5 if the validation loss does not decrease for 30 epochs.
For all datasets other than MNIST and KMNIST. we extract training patches of size 128 x 128, and separate 15% of all patches for validation. We set the maximum number of epochs such that approximately 22 million steps are performed, and in each epoch the entire training data being fed. Training is terminated if the validation loss does decrease by least 10-6 over 100 epochs.
For DenoiSeg Flywing we observed KL vanishing and solved it via Annealing within the first 15 epochs (Bowman al. 2013).
The fully unsupervised DIVNOISING decoder directly predicts the signal and the noise variance per pixel where the variance constrained linearly depend on the signal (See Section 5 To avoid numerical problems and ensure that the predicted variance always remains positive, we allow the user set a minimum allowed variance/standard deviation minn / and enforce this by clamping the predicted values. Note that a viable choice for this parameter depends on the intensity range of the dataset. We use the following values: For all FU-PN2V datasets amin 50, for DenoiSeg Flywing and DenoiSeg Mouse datasets amin 3, for DenoiSeg Mouse s&sp dataset aiin 1, for BiolD Face dataset minn 15, for W2S avg / datasets aiin 25 and for W2S avg 16 datasets amin 3.
Run Time and Hardware Requirements.
DIVNOISING using light weight fully convolutional networks (see Appendix Figs. and 7 runs on relatively cheap computational budget. Our depth 2 networks trained for all experiments requires about 1.8 GB GPU memory and our depth 3 networks roughly GB GPU memory on a NVIDIA TITAN Xp GPU. The training time varied from 5 12 hours average depending on the dataset.
15

---

Published as a conference paper at ICLR 2021
Input (1, 128, 128)
Convolution (3, (32, 128,
Convolution (3, 3) (32. 128, 128)
Maxpooling (2, 2) (32, 64, 64)
Convolution (3.3) (64, 64, 64)
Convolution (3,3) (64, 64, 64)
Maxpooling (2, 2)
(64, 32, 32)
Convolution (3.3)
(64, 32)
Convolution (3.3) (64,32, 32)
Z (64, 32, 32) Reparametrization trick
Convolation (3, (64, 32, 32)
Convolution (3,3) (64,, 32, 32)
Transposed Convolution (3. (64, 64, 64)
Convolation (3, 3) (32, 64, 64)
Convolution (3.3) (32, 64, 64)
Transposed Convolution (3, 3) (32, 128, 128)
Convolution (3. (1. 128, 128)
Figure 6: The fully convolutional architecture used for depth 2 networks. We show the depth DIVNOISING network architecture used for FU-PN2V Convallaria, FU-PN2V Mouse nuclei, FU- PN2V Mouse actin, all W2S channels and DenoiSeg Mouse datasets. These networks count about 200 parameters and have a GPU memory footprint of approximately 1.8GB on a NVIDIA TITAN Xp.
16

---

Published as a conference paper at ICLR 2021
Inpur 125, 1280
Convolation (3, (32, 128)
Convolation (3, 3) (32, 128, 128)
Maspooling (32,64,
Convolution (3.3) (64, 64)
Maxpooling (2,2) (64, 32)
Convolation 3) (128, 32, 12)
Comvolution (3.3) (128,32,32)
Maxpooling
Convolation (3.3) (64,
Convolution (64.
z. (64, 16, 16) Reperunciizatonn
Convolution (3,3) 3) (128,16, 16)
Convolation (128, 16)
Transposed Convolution (3, 32,
Convolution (24,32.32)
Convolution (3.3) (2,
Transoosed Convolution 3)
Convolution (3.3) (32, (32.64,66 64)
(3.3)
Transposed Convelation (3,3) 128.
Comvolution 128)
Figure 7: The fully convolutional architecture used for depth networks. We show the depth 3 DIvNOISING network architecture used for DenoiSeg Flywing and eBook datasets. These networks count about 700k parameters and have a GPU memory footprint of approximately 5GB on NVIDIA TITAN Xp.
17

---

Published as a conference paper at ICLR 2021
A.4 CLUSTERING OF SOLUTIONS AND DERIVING THE MAP ESTIMATE
Here we provide additional details on how the cluster centers and the approximate MAP estimate of Fig. 4 (see main text) were found. We first drew 10000 sampled images from the approximate posterior as described in Section 4 of the main text. We then performed mean shift (Cheng. 1995) clustering (using the existing scipy implementation)
the cropped image region shown in the figure. We set bandwidth of 800 and the the maximum number of iterations to 20, and used the 100 first samples of DIVNOISING as seeds. We finally show of the resulting cluster centers in the figure.
To produce the MAP estimate, we employ a similar strategy. In order to find the mode of the sampled distribution efficiently, we assume that dependencies in the predicted samples should be local. This assumption is valid, since our network only has only finite receptive field for each predicted pixel. Hence, we apply mean shift algorithm on locally overlapping regions. We use window size of 10 10 pixels with an overlap of 3 pixels in and y. On each such region, the mean shift algorithm is executed repeatedly with decreasing bandwidth, always using the latest result as new seed. We start by using the sample mean as seed and with an initial bandwidth of 200. After each iteration the bandwidth is decreased by a factor of 0.9, until it drops below 100.
Similar results should also be achievable by applying mean shift algorithm the entire image. But since samples will differ at any location in the image, this global approach would require an excessively large number of DIVNOISING samples.
A.5 GENERATING IMAGES WITH DIVNOISING MODELS
Just as with a vanilla VAE (see Section 4 in the main text), we can use a trained DIVNOISING VAE synthesise images of structures resembling the training data. To achieve this, we sample from the normal distribution z ~ p(z) and process each sample with the decoder network s* ga (z" We show such generated images in comparison to real crops from the test data in Appendix Figs. 8 I0 We see that the images appear most plausible for local structures, indicating that the small networks we use in this work are not capable of capturing larger structural features in the given data.
18

---

Published as a conference paper at ICLR 2021
Gemerahed Real 36266
Real
Genertted 33-33
Gemerahed 60x04
Gener 20.00
Figure 8: Generating synthetic images with the DIVNOISING VAE for the FU-PN2V Conval- laria dataset Krull et al. (2020); Prakash et al. (2020). DIvNOISING can also be used to generate images by sampling from the unit normal distribution p(z) and then using the decoder to produce an image. Here, compare generated images and randomly cropped real images. We show images of different resolutions to see how well the VAE captures structures at different scales. The VAE we for denoising is only able realistically capture small local structures. Note that the network use is quite shallow (see Appendix Fig. 6
19

---

Published as a conference paper at ICLR 2021
Rext 30x6
Gent 10:18
Rarl 3233
Gene Denerateed 33-33
Gent chvee
Daral GOO8E
Cente othedd s0v00
Figure 9: Generating synthetic images with the DIVNOISING VAE for the DenoiSeg Fly- wing Buchie al. (2020) dataset. DIVNOISING can also be used to generate images by sampling from the unit normal distribution p(z) and then using the decoder to produce an image. Here, we compare generated images and randomly cropped real images. We show images of different reso- lutions to see how well the VAE captures structures at different scales. Note that the network (see Appendix Fig. 7 we use is a bit deeper compared to Supplementary Fig. g This VAE captures larger structures a little better but struggles to produce crisp high frequency structures. This is likely consequence of the increased depth of the used network.
20

---

Published as a conference paper at ICLR 2021
Seel 7503
9
4
2
Gemeratad 2003
2
3
6
9
Generated 3703
9
3
74
4
Genmrrted GOOEE
DOCOS
Gamenoteed 2002200
Figure 10: Generating synthetic images with the DIVNOISING VAE for the MNIST LeCun et al. 1998) dataset. DIV can also be used to generate images by sampling from the unit normal distribution p(z) and then using the decoder to produce an image. Here, we compare generated images and random ground truth images. Our fully convolutional architecture allows generate images of different sizes (despite all input images being only of size 28 x 28).
21

---

Published as a conference paper at ICLR 2021
0.50
0.725
0.200
0.700
0.750
0675
Avsage Prociim
01000
0.275
0.30
0.50
0.750
0.655
0625
0.725
0.500
0400
0.900
High NNA
0.000
0.300
0.400
0.300
0.30
0,200
512
1034
Number of samples
Number
Number samples
Figure 11: DIVNOISING enables downstream segmentation. Evaluation of segmentation results (using the Fl score (Van Rijsbergen 1979). Jaccard score [Jaccard] 1901) Average Precision (Lin et 2014). x x-axis we plot the number of DIv NOISING samples used. The performance of BIC is only evaluated up to 100 samples because we limited run- time to 30 minutes). Remarkably, Consensus (Avg) using only 30 DIVNOISING segmentation labels, outperforms segmentations obtained from high SNR images.
A.6 INSTANCE CELL SEGMENTATION
Here, we provide additional details regarding the downstream segmentation task described in Section of the main text. We used the first 21 images in the test set of DenoiSeg Flywing for our analysis.
Given imput image, segmentation pipeline consists of (i) generating segmentation masks using local thresholding with a mean filter of radius 15, followed by (ii) skeletonizing the space between these masks, followed by (izi) connected component analysis to obtain instance segmentation.
Using this pipeline, we generated segmentation for the noisy (low SNR) images, ground truth (high SNR) images, as well for the DIVNOIIING MMSE estimate (obtained by averaging 1000 sampled denoised images).
We also apply the above described pipeline for each of the 1000 NIVNOIIING samples separately serve as input for the two label fusion methods, namely (i) Consensus (BIC), and (ii) Consensus (Avg). For the latter label fusion method we skip the connected component analysis and directly average the thresholded and skeletonized images. To obtain the final result, we again apply the full segmentation pipeline described above to this average image.
All segmentations were obtained with the open source image analysis software Fiji (Shiinellnn et al. 2012).
The quantitative results illustrating the benefit of diverse segmentation for label fusion methods is shown in Appendix Fig.
A.7 THE RELATIVE IMPORTANCE OF THE KL Loss COMPONENT
We generalize DIVNOISING training loss weighted combination of modified recon- struction loss (see Section in the main text) and KL divergence loss, where the two loss components weighted equally. Following the exposition in (Higgins al. (207)), we explore the effect of
weighting the KI. loss component during training with factor B. Our modified training loss thus becomes
Lo.0 (x) = (x) (x))(())
(5)
where setting 3 gives our DIVNOISING setup described in Section the main text. Note that increasing or reducing B3, i.e. changing the relative importance of the reconstruction loss, equivalent to using wider or narrower noise model, such as Gaussian noise model with larger smaller standard deviation a. We can thus interpret above results as the effect of using mismatched noise model that is either too wide too narrow.
Effect of 3 on Denoising Quality. We investigated the effect of on the denoising ability of DIVNOISING network with the DenoiSeg Flywing dataset. As illustrated in Appendix Fig. 12h, gives the optimal results for the MMSE estimate (obtained by averaging 1000 samples). Both regimes, 3 > 1 and 1, yield sub-par denoising performance.
Effect of 3 on Diversity of Denoised Samples. We introduce a simple new metric, called standard deviation PSNR, quantify the diversity of denoised results obtained as function of 3. For a given noisy image x and given a set of denoised samples S,, we compute the PSNR of each sample E S with respect the corresponding ground truth image This yields vector of PSNR values where
22

---

Published as a conference paper at ICLR 2021
2 (SN a;), for Standard deviation PSNR for the noisy image is then defined as the standard deviation elements in the vector v. Appendix Fig. 12p reports the average standard deviation PSNR obtained for 42 test images of the DenoiSeg Flywing dataset. The higher the beta, the higher is the standard deviation PSNR indicating higher diversity. Qualitative results presented in 1, there is increased diversity the bigger image scales
Appendix Fig. show that with (c.g. diverse predictions of cell membranes), and generated denoised images appear smoother than reduces diversity and introduces grainy artefacts, thereby yielding poor reconstructions. Note that / = 1 gives the best results in terms of PSNR of MMSE while maintaining a fair level of diversity.
those observed in real data, Setting
23

---

Published as a conference paper at ICLR 2021
A.8 How DOES NOISE AFFECT THE DIVERSITY OF DIVNOISING SAMPLES?
We quantified how the diversity of DIVNOISING samples changes with the amount of noise present in the original dataset. Increased level of noise introduces additional uncertainty about the true signal, hence we would expect this to lead to increasingly diverse samples.
To test this hypothesis, choose the DenoiSeg Flywing dataset and inject pixel wise independent gaussian noise of mean 0 and standard deviations
30, 50 and 70. We report the standard deviation PSNR diversity metric, introduced in Appendix Section A..7 for all three noise levels. As demonstrated in Appendix Fig. 122 the higher the noise level, the more diverse the DIVNOISING samples become, thereby confirming our hypothesis.
Number samples averaged
100
500
4.00
70:17
31
27.11
22.17
007-
(relative of
00K-
2,00
21.91
25.25
25.05
23.0
2108
21.94
Shi PSNR
1,00
22:36
14:81
23.05
23.06
25.15
13.00
23
PSNK
0.75
042-
0.0-
0.50
24.27
000
fag
Pa02 Pa0.3
0.00
Figure 12: Analyzing the denoising quality and diversity of DIVNOISING samples with differ- ent factors for the DenoiSeg Flywing dataset. (a) The heatmap shows how the quality (PSNR in db) of DIVNOISING MMSE estimate changes with averaging increasingly larger number of samples (numbers shown for run). Unsurprisingly, the more samples are averaged, the better the results get. We also investigate the effect weighting the KL loss term with a factor 3 (Supplementary Eq, 5) on the quality of reconstruction. We observe that the usual VAE setup with / gives the best results in terms of reconstruction quality. Increasing 3 > leads to higher diversity at the expense of poor reconstruction (see Appendix Fig. [3) (b) We quantify the denoising diversity achieved with different (3 values in terms of standard deviation PSNR (see Appendix section A.7 for details on the metric). We report the average standard deviation of PSNRs over all test images for different values of B and observe that the higher 3 values increase the diversity. (c) We also investigate the effect of noise on the diversity of denoised DIVNOISING samples by adding pixel wise independent zero mean Gaussian noise of standard deviations 30. 50 and 70. The higher the noise, the more ambiguous the noisy input images are, thus leading higher diversity.
24

---

Published as a conference paper at ICLR 2021
MMSE (B 4)
MMSE (B-
MMSE (B 1)
MMSE (B 0.75)
MMSE
NoIKy RapAK
Groind Igeth
Sample
Sample 2
Sample
Sample
Sample
Sample
Sample
Sample
Sample 3
B-0075
8-0.50
Figure 13: Qualitative analysis of the effect of weighting KL loss term with factor B for De- noiSeg Flywing dataset. (a) We show the DIVNOISING MMSE estimate obtained by averaging 1000 samples for all considered 3 values (Supplementary Eq. 5). We observe that the reconstruc-
tion quality suffers on either increasing 3 > or decreasing 3 < 1. Best results (with respect to PSNR) are obtained with 3 as demonstrated in Fig. 12h. (b) For each value, we show three randomly chosen DIVNOISING samples as well as difference images. Increasing 3 >> 1, allows the NIVNOIIING network generate structurally very diverse denoised solutions, while typically leading to textural smoothing. Decreasing generates DIVNOISING samples with overall much reduced structural diversity, introducing reconstruction artefacts/structures at smaller scales.
25

---

Published as a conference paper at ICLR 2021
A.9 ADDITIONAL RESULTS
More Qualitative Results. In addition to the qualitative results presented Fig. 2 in the main text, here present more results for each considered dataset in Appendix Figs. A.9 24
Noney impur
DivNasing
MMSE
Ground frush
SASSE
Ureed FAR
Inpe:
NMSSE
Surple1
2erple
Striphi
Sumple 1
Sungle
Sumple
2
Sucple 1- S-Sorkk
Sayple
Sumple
Seeple
Sample Sompple
Figure 14: Additional qualitative results for the DenoiSeg Mouse Buchholz et al. (2020) dataset.
Here, we show qualitative results for two cropped regions (green and cyan). The MMSE estimate was produced by averaging 1000 sampled images. We choose samples display to illustrate the diversity of NIVNOIII results.
26

---

Published as a conference paper at ICLR 2021
Ingit
DirNohinn! MMST
Surete1
Sumple
Suropke3
Samph
Saxelc
Suxple
Sarpa I- 2
Sargle 1- 3
Surgle 2- 3arp43
Surppee 2- Sargle
Sumete 1- Serple
Surgle 2- Ntmplee
Additional qualitative results for the DenoiSeg Flywing Buchholz et al. (020 dataset. Here, we show qualitative results for two cropped regions (green and cyan). The MMSE estimate was produced by averaging 1000 sampled images. We choose 3 samples to display to illustrate the diversity of DIVNOISING results.
Figure 15:
27

---

Published as a conference paper at ICLR 2021
Noney Input
MMSE
DD BOROOO 8
O
Sumple
Surppe
Sansk
Saxple
Suxple
Sarpa 1- 2
Surple Sorrake3
Surgle 2- Sarpes
Strple: 2- Sungue
Sumete 1. Serple
Sungle 2- Stmplee
Figure 16:
Additional qualitative results for the FU-PN2V Convallaria Krull al. (2020);
Prakash et al. (000)) dataset. Here, we show qualitative results for two cropped regions (green and cyan). The MMSE estimate was produced by averaging 1000 sampled images. We choose samples to display to illustrate the diversity of DIVNOISING results.
28

---

Published as a conference paper at ICLR 2021
Nony Inpur
DivNosing MMSE
Screte
Sumple
Sumppe
Sansk
Saxplc
Suxple
Sampa1 2
Surple 1- Sorrake3
Surple 2- Sarpe
Sumple 1- Surgle
Sumete 1- Serple
Surque 2 Nimppee
Figure 17: Additional qualitative results for the FU-PN2V Mouse nuclei Prakash et 2020 dataset. Here, we show qualitative results for two cropped regions (green and cyan). The MMSE estimate was produced by averaging 1000 sampled images. We choose 3 samples to display to illustrate the diversity of DIVNOISING results.
29

---

Published as a conference paper at ICLR 2021
Nony Input
DivNossing MMSE
Surete
Sample
Surppi3
Sample
Saxple
Suxple
Sample- 2
Sarple 1- S
Surgle 2- 5
Surple 1- Surgue
Sunete I. Sarple
Surgle Nomple
Figure 18: Additional qualitative results for the FU-PN2V Mouse actin Prakash 2020 dataset. Here, we show qualitative results for two cropped regions (green and cyan). The MMSE estimate was produced by averaging 1000 sampled images. We choose 3 samples to display to illustrate the diversity of DIVNOISING results.
30

---

Published as a conference paper at ICLR 2021
Nonyy Input
DivNosing MMSE
Screte1
Sumple
Surppk
Surrpl
Saxplc
Surple
Sampa 1- 2
Sarple 1- Serzak
Surgle 2- Sarpe
Surple T- Surgue
Sunvde 1. Serple
Surque 2- Nimple
Additional qualitative results for the W2S Zhou al. 2020) dataset (ch. avgl). Here, we show qualitative results for two cropped regions (green and cyan). The MMSE estimate was produced by averaging 1000 sampled images. We choose samples display to illustrate the diversity of DIVNOISING results.
Figure 19:
31

---

Published as a conference paper at ICLR 2021
Nonyy Inpux
DivNoising MMSE
Sprete1
Sumple
Sumppe
Samrp
Saxple
Saxple
Sample- 2
Sarple 1- Sorrzke
Surple 2- Sarpe
Surple 2- Sangie
Surete 1. Serple
Sarqle Nimppe
Additional qualitative results for the W2S Zhou al. (020) dataset (ch. avgl). Here, we show qualitative results for two cropped regions (green and cyan). The MMSE estimate was produced by averaging 1000 sampled images. We choose samples display to illustrate the diversity of DIVNOISING results.
Figure 20:
32

---

Published as a conference paper at ICLR 2021
Nony Input
DivNosing MMSE
Surete
Sumple
Surapk
Samrk
Sumple
Surpl 1- 2umple
Surple 1- Sorrsle
Surgle 2- Sarpe
Surppi: 1- Sangle
Sumete 1. Serple
Surqle 2- Stmpple
Additional qualitative results for the W2S Zhou al. 2020) dataset (ch. avgl). Here, we show qualitative results for two cropped regions (green and cyan). The MMSE estimate was produced by averaging 1000 sampled images. We choose samples display to illustrate the diversity of DIVNOISING results.
Figure 21:
33

---

Published as a conference paper at ICLR 2021
DivNeising
MASE
1
O
7
4
5
9
9
0
6
2
6
3
3
Ground Iruhh
MUSE
leper
5
5
2
2
Sugge
S
Serpl
Senfle
Surple 2
Sxeple
8
5
5
2
2
Sarpk 2
Sapple 1- 3tack3
Sa 3urp
Sunpl: Sarple
Sumelc 1- Sarple
Sargle2 Sunpk
Figure 22: Additional qualitative results for the MNIST LeCun et al. (1998) dataset. Here, we show qualitative results for two cropped regions (green and cyan). The MMSE estimate produced by averaging 1000 sampled images. We choose 3 samples to display to illustrate the diversity of DIVNOISING results.
L
34

---

Published as a conference paper at ICLR 2021
DivNocine
h
55
R
Z
co
MASSE
Z
Surete1
Sumple
Sumppe
Sanck
Saxelc
Surple
47
Sappl 2
Sarple 1- 3
Surgl 2- Sarpe
Surple: : Surpue
Surete Serple
Surple Stmple
Figure 23: Additional qualitative results for the KMNIST Clanuwat et al. (0188 dataset. Here, we show qualitative results for two cropped regions (green and cyan). The MMSE estimate was produced by averaging 1000 sampled images. We choose samples to display illustrate the diversity of DIVNOISING results.
35

---

Published as a conference paper at ICLR 2021
MMSE
I was a se. Aworse
just bel just be
Grond
was was
orseorse
Surete
Sumple
Simple
Sample
Saxple
Sample
orseprsearse
was was whs
Sangle 1- 2
Surple 1- Soreyle
Secple 3- Sarpe
Sceple 1- Sangle 2
Sapple 1- Serple
Sanple Sorple
irsca CAACOLL ge
Figure 24: Additional qualitative results for the eBook Marsh (2004) dataset. Here, we show qualitative results for two cropped regions (green and cyan). The MMSE estimate was produced by averaging 1000 sampled images. We choose 3 samples to display to illustrate the diversity of DIVNOIINNG results.
36

---

Published as a conference paper at ICLR 2021
Nonsy input
DivNoising MMSE
Graund trush
Txpur
MUSE
Teper
MMSE
Sungle
Sveple
Soople
Senyle
Sargle 3
Seeple
Serple 2ample
Sumple Starak
Sample Sump
Sumple Sample
Sumplc Sarple
Surgle Sumple
Figure 25: Additional qualitative results for the BiolD Face noa dataset. Here, we show qualitative results for cropped regions (green and cyan). The MMSE estimate produced by averaging 1000 sampled images. We choose samples to display to illustrate the diversity of DIVNOISING results.
37

---

Published as a conference paper at ICLR 2021
A.10 RESULTS ON NATURAL IMAGES
We investigated the denoising performance of DIVNOISING network on the natural images benchmark dataset BSD (Roth & Black 2005) and show our results in Appendix Fig. 26 where the input has been corrupted with Gaussian noise of a = 25. With our depth 2 network having 96 feature channels in the first network layer, achieve PSNR of 27.45 dB while baseline gives 27.71 dB. As discussed in the main text, this does not come as surprise since our DIVNOISING network is comparatively small and asked to learn complete generative model of the entire data domain (see main text and Appendix Figs. g TO tremendous diversity present natural images is challenging, and likely the reason why other architectures solving problems posed the domain of natural images
unsupervised NOISE2VOID
Learning such model for the
much larger than our networks are. Future versions of DIVNOISING will address this issue by using more expressive architectures. However, DIvNOISING already gives access to clean samples from the true (data) posterior (see Appendix Fig. 26).
38

---

Published as a conference paper at ICLR 2021
DIVNAising
Graund
Sempte
Sumple
Somple:
Sengle
Seeplc
Sxxple
Sepple S
Sasple i- Strgke
Sample2- 3
Sanple L Senple
Sengle 1- Sarple)
Sasple Sungle3
Figure 26: Qualitative results for the BSD68 dataset Roth & Black (2005). Our relatively small DIVNOISING networks fail to capture the ample structural diversity present in natural photographic images thereby exhibiting sub-par performance. However, diversity adequately small image scales (with respect to the used network's 's capabilities) can still be observed, as demonstrated with the different samples and the difference images corresponding to the green and cyan insets. We are confident that future work on DIVNOISING with larger networks and different network architecurrestraining schedules will expand the capabilities of this method to capture more complex image domains.
39

---

Published as a conference paper at ICLR 2021
A.11 How ACCURATE IS THE DIVNOISING MODEL AND THE APPROXIMATE POSTERIOR?
Upon close inspection, we find that the images sampled by DIVNOISING exhibit various imperfec- tions, making clear that they are in fact only samples from approximate posterior.
For example, we find that DIVNOISING samples are often smoother than real images, see e.g. network architecture (see also Appendix
Appendix Figs. 15 and 22 We attribute this problem Section A.9 For instance, U-NET based supervised denoiser can make use of skip connections to propagate high frequency information. But DIVNOISING VAEs have pipe all information through the downsampled latent variable bottleneck.
Another common artefact in sampled images the presence of faint overlayed structures in the background (see Suppl. Fig. 24). Note that this artefact is less pronounced than in the MMSE estimate (where we expect such artefacts).
We believe that most of these remaining issues will be solved/reduced by using more sophisticated network architectures and refined training schedules.
A.12 DERIVATION OF DIVNOISING LOss FUNCTION FROM PROBABILITY MODEL PERSPECTIVE
Here, we want to provide more formal derivation of why our loss function can used to train the VAE as desired. We follow similar line of argument as has been laid out for the standard VAE by Doersch in (Doersch, 2016).
In framework, assume that the observed data is generated from some underlying latent variable z through some clean signal s via a known noise model (x|s). This process of data generation is depicted as graphical model shown in Appendix Fig. 27
Z
S
X
(o(sl))
Peud (Xis)
Figure 27: Graphical model of the data generation process.
The decoder describes a full joint model for all three variables:
po (z x, s) = p(x, s/z)p(z) = p(x[s, ())o(s)))p(2)
(6)
In the assumed graphical model in (Appendix Fig. 27 x is conditionally independent of z given s. Formally, this implies that
p(x|s, z) PNM (x|s).
Using Supp. Eq. we can reformulate Supp. Eq. 6
Pe pe x, = (xM(xisspp(sizz)pte)
(8)
To train the generative model from Appendix Fig. 27 we try to adjust the parameters 0 to maximize the likelihood observing our training data x. This means that we need maximize
(x(x)=
( ge(z))p(z)dz.
(9)
However, computing the integral in Supp. Eq. is intractable due to the high dimensionality of z. In our particular model, would need integrate over 64 dimensions for each pixel for all our datasets except MNIST and KMNIST datasets where we would need to integrate over 8 dimensions for each pixel. An alternative to computing the integral would be approximate it by sampling large number of values z z² ..., zK from p(z) and computing pe (x) R R > PNM (x|s = go(z²*).. However, since (x = go )z))) will be very close to for almost all x² this would require K to be a very large number for each image in training set.
40

---

Published as a conference paper at ICLR 2021
Following the idea introduced in (Kingma & Welling 2014), overcome this problem by instead using am encoder to describe an auxiliary distribution go (z]z)) x). The encoder can take a noisy image and yield distribution over values, which in turn likely produce x under the generative model. We want the encoder distribution go(2)x to approximate the true underlying distribution (x(2))) (z)xx),, it is implicitly descrihed by graphical model. From Bayes theorem, xx(z)x) factorizes as
po(z|x) Pe(x|z)p(z))
(10)
[x
The decoder in DIVNOISING setup is deterministic function of z. i.e., go(z) = Hence, we can reformulate Supp. Eq. a5
[NM(x|s= = g0)
Pe (2/x)
(1I)
po(x)
We can describe the quality of the encoder distribution, i.e. how well it approximates the true pe (z/x) via the KL. divergence
KL ([(g(xxx))lpp((xxx) - -f (z(xx log po(z dz. (z(z)x)
(12)
Substituting Supp. Eq. 11 in Supp.
Eq.
we get
PNM x go(z))p(2) dz
KL (lx(2)))lloozzlxx
(z(2))) log
po ( x))
PNM [x|s 9o (z) p(()
(z(2)x) [lgg
log po (x)]
(z
S
PNM [ (z( (z) )r(z)
(z(2)) log
dz + (o(z))) log)o(x)dz
9 (z
PNM (x|s gz (z))
dz logpo(x) f (z)2)))
(z(z)x) log
(z(x)
Since {go(2(x)dz = 1, we get
S (xo(2(x)) log PNM(x(s= = go(z))p(z) (x0))(s==0(z))((2) (z/x)
KI (10(2)x))lpo(2xx))
dz + Po(x)
This implies
DogPo(X) (x) = 96 90(2/x) log (x|(x (z(2))((z) dz + KL (cx(2)x))loo((xx) (x(z)x)
(13)
= ELBO + KL (po
where ELBO is the Evidence Lower Bound also introduced (Kingma & Welling 2019) in the context of standard VAEs and here ELBO = S to (z/x) xx) log (xon(x(s-gu(z))nt dz. Note that the KI. divergence term in Supp. Eq. 13 is always greater than equal to 0 and hence, ELBO is lower
(x(x)x)
bound for log Pe (x), i.e., log Pe x ELBO. follows from Supp. Eq. 13 that
ELBO px KIL (z)
(14)
Supp. Eq. 14 implies that maximizing ELBO with respect to @ and 0 maximizes log po(p))) and minimizes RT (1s(2|x) (x)))))))) the goals seek to achieve. Hence,
41

---

Published as a conference paper at ICLR 2021
(f [0(2))) log px|( = (0(2))p((2) dz
max ELBO- max
(o(2|x)
+ ( log
p(z)
max
[o(z))))oggggil 90(2))
dz
90 (z(x))
( [o(2)x) log PNM (x go(z)) KL ((oo2//x))]l)()
max
max (x) PNM go(2))] KLL (gx(z(x)]lp(z)))
Maximizing the ELBO is equivalent minimizing the negative ELBO, thus giving us the DIVNOISING loss function
(x) =
log (x - 90(2)) ] + KL (1=(2)x))lp(z))
(15)
where the expected value is approximated in each iteration by drawing single sample from ds (z) (2
Note that the first term the summation in Supp. Eq. [T5is the same as described Section 5 5n the main text whereas the second term in the summation is the same used in the standard VAE loss.
A.13 COMPARISON OF PREDICTED VARIANCES BY VARIOUS METHODS
Unsupervised DIVNOISING and vanilla VAEs are both trained fully unsupervised, learning to predict per pixel noise models. Learning good noise model is essential for good denoising performance as evident from Table T Here, we compare the noise models and variance maps predicted for two datasets by unsupervised DIVNOISING and vanilla VAEs.
BiolD Face dataset. This dataset has been synthetically corrupted with Gaussian noise of μ = 0 and 15.
DiNNosingg dencised
VAB
voriance
Figure 28: Comparison of noise models and variance maps predicted by the vanilla VAE and DIVNOISING. (a) For each predicted signal intensity (x-axis) we show the variance of noisy observations (y-axis). The plot generated from experiments on the BiolD Face dataset. The dashed red line shows the true noise distribution (Gaussian noise with o² 225). The noise model created via bootstrapping, and the noise model we co-learned with DIVNOIIING, correctly show (approximately) constant values across all signal intensities. The implicitly learned noise model of the vanilla VAE has to independently predict the noise variance for each pixel. Its predictions clearly deviate from the true constant noise variance. (b) We visually compare the denoising results and show how the predicted variance varies across the image. While the variance predicted by the implicitly co-learned vanilla VAE model varies depending the image content, the variance predicted by the co-learned NOISING model correctly remains flat.
42

---

Published as a conference paper at ICLR 2021
Convallaria dataset. This dataset is intrinsically noisy and the noise distribution resembles the shot noise and read out noise characteristics as typical for images acquired under low light settings.
a)
Input
VAL denoised
DivNoising denoised
VAL variance
DevNoisine variance
Monardd
2004
08
69
08
10
08
100
300 x0i
Prodi intoady
Figure 29: Comparison of noise models and variance maps predicted by the vanilla VAE and DIVNSIGGGG. (a) For each predicted signal intensity (x-axis), we show the variance of noisy observations (y-axis). The plot is generated from experiments the Convallaria dataset. The dashed red line shows the true noise distribution (measured from pairs of noisy and clean calibration data). We compare the noise model co-learned by the vanilla VAE and DIVNOISING with ground truth noise model and a noise model bootstrapped from noisy data alone as described in Prakash et al. (2020). Clearly, the noise model learnt by unsupervised DIVNOISING a much better approximation to the ground truth noise model compared to the noise models learned/obtained by other methods. (b) We visually compare the denoising results and show how the predicted variance varies across the image. As a consequence of the approximately linear relationship between signal and noise variance, both variance images closely resemble the denoised results. However, the result of the vanilla VAE additionally contains artifacts.
A.14 QUANTITATIVE COMPARISON OF DIVNOISING WITH SELF2SELF
Since SELF2SELF is trained per image, leading to prohibitive computation times on our test sets, we randomly chose single images for four of our datasets (FU-PN2V Convallaria, FU-PN2V Mouse actin, FU-PN2V Mouse nuclei and W2S Ch. / (avgl)) which contain real-world noise.
We compare the performance of SELF2SELF trained on single images with the performance of DIVNOISING when trained on (i) the same single image as SELF2SELF, and (ti) the entire body of available noisy data in the respective dataset. All trained networks are then applied to the selected single images. Note that SELF2SELF is run with its default settings.
Since SELF2SELF training is computationally expensive even for single image, we decided to limit training time to 10 hours per input on a NVIDIA TITAN Xp GPU. We monitored its performance by periodically computing the PSNR (every 3000 training steps), showing that even after 10 hours, SE.FF2EFLF is not yet fully converged. Table shows all results we obtained. can be seen that DIVNOISING, when trained
the full dataset, leads consistently better performance, while DIVNOIIING trained on single images leads to comparable results in a fraction of training time and using significantly less GPU memory.
43

---

Published as a conference paper at ICLR 2021
Datasets
PSNR (dB)
Run time (hours)
GPU memory (GB)
S2S
DivN.
DivN.
S2S
DivN.
DivN.
S2S
DivN.
DivN.
was FUPNN
Convallaria
36.23
36.42
36.94
10
0.44
10
11
1.5
1.5
Mouse actin
33.15
33.80
33.99
10
1.09
10
11
1.5
1.5
Mouse nuclei
36.21
35.99
36.46
10
0.16
7
11
1.5
1.5
Ch.0 laegl)
31.40
31.81
31.59
0.48
3.75
11
1.5
1.5
Table 2: Comparison of Self2Self with DivNoising. We train Sclf2Self (S2S) on a random single image per dataset and compare with NOVISII trained on same single image (DivN. ') and DIVNOISING trained with the full dataset (DivN. "). All methods are tested on the selected single image. Overall best method is indicated in bold. For all datasets, DIVNOISING leads best performance while being orders of magnitude faster and needing significantly less GPU memory.
A.15 SAMPLING TIME DURING PREDICTION
During prediction, in order to obtain diverse results, to compute the MMSE or MAP estimates, we need to sample multiple denoised images from the trained DIVNOIGING posterior. reports the time (in seconds) needed for sampling 1000 denoised images. For all datasets holds that sampling 1000 denoised images requires less than seconds.
Datasets
time (sec)
Convallaria
3.3790.059
FU-PN2V
Mouse Act.
6.40+:0.552 sec
Mouse Nue.
1.91 sec
Ch.0 (avgl)
3.427-0.074 sec
Ch.1 (avgl)
3.37:0.068 sec
Ch.2 (avgl)
3.38+00053
W2S
Ch.0 (avg16)
3.39+00000 sec
Ch.1 (avg16)
3.40+0.062
Ch.2 (avg16)
3.404 sec
Mouse
1.17 sec
DenoiSeg
Flywing
3.91 sec
Mouse s&p
2.82+ sec
BioID Face
2.0940.046 see
Table 3: Sampling times with NOISING. Average time (- SD) needed sample 1000 denoised images from trained DIVNOIIING network (evaluated all test images of the respective dataset).
44


Reviews:
[{"review_id": "hF48vV86MjP", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes a new method of noise removal using convolutional VAE.  An observed image with noise is input to VAE, and after the expression $z$ in the latent space, the noise removed image is finally output.  After that, it is possible to generate a pseudo noisy observation image according to the noise model.  The noise model part is flexibly designed using the Gaussian mixture model.  In the training, VAE and noise model can be learned at the same time. Since VAE is a generative model, and a clean denoising image can be obtained by averaging a large number of candidates of clean images, $s$, sampled from the periphery of the latent space representation $z$.\n\nThe advantages of the proposed method are that it is Fully unsupervised and that the noise model can be learned at the same time.\n\nThe paper itself is interesting, and the proposed method itself is good as one of the image processing methods, but there is a lack in the explanation part.  In this paper, the noise model $p (x|s)$ is well discussed, but I think that the discussion about the image prior, $p(s)$, for clean image is weak. As explained in p3 of the paper, $p(s|x) \\propto p (x | s) p (s)$ is an important formula. In conventional image processing, $p(s)$ has been interpreted as smoothness or non-local similarity. In this method, it is treated as $p (s | z) p (z)$, but I think we should discuss a little more about what this is.\n\nIn the proposed method, there is no special assumption about the prior knowledge of the image, and it seems that it is normally distributed in the latent space. In this case, a large amount of data or some help of a network structure is needed to remove noise. The use of the convolutional structure in the proposed method is an indispensable element in the explanation of $p(s)$.\n\nThe noise reduction effect of the convolutional structure itself is well known after the study of Deep Image Prior, and recent research [a, b] also discusses the relationship between convolutional neural networks and non-local similarity. These may help the discussion about $p(s)$.\n\n[a] Yokota, Tatsuya, et al. \"Manifold Modeling in Embedded Space: A Perspective for Interpreting\" Deep Image Prior \".\" ArXiv preprint arXiv: 1908.02995 (2019).\n[b] Tachella, Julián, Junqi Tang, and Mike Davies. \"CNN Denoisers As Non-Local Filters: The Neural Tangent Denoiser.\" ArXiv preprint arXiv: 2006.02379 (2020).\n\n-------------\nAs a result of the feedback, the part I was concerned  became clear, so I would like to raise the score.\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "Px2-lv-rPBg", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This manuscript proposed an interesting method to infer the distribution of denoised images for a given input instead of a single denoised image. Then, the final denoise image was generated using MMSE (averaging multiple denoised images) or using MAP (finding the mode of the posterior distribution).\n\n1) Unfortunately, this manuscript missed one very important relevant recent work:\n[Quan2020] Quan et al., Self2Self With Dropout: Learning Self-Supervised Denoising From Single Image, CVPR 2020.\nThis work also infers the distribution of denoised images instead of a single denoised image and yielded state-of-the-art results for both synthetic noise, real noise, salt-and-pepper noise and inpainting. \nThe ways of implementing distribution generations seem similar (\"a principled approach to incorporate explicit models of the imaging noise distribution in the decoder of a VAE\" in this manuscript and also similar structure was implemented in the decoder part of the network in [Quan2020]) except that the proposed method here utilized random sampling in the latent space while [Quan2020] used dropout to implement it. In my view, [Quan2020] and the proposed method share some common aspects that can undermine an important contribution of this manuscript in its current form, and thus this manuscript must explain the advantages and novelty of the proposed method over [Quan2020] properly. \n\nEven though the proposed method with MAP is still new, for now [Quan2020] seems to have advantages over the proposed method such as (a) Self2Self requires a single noisy image while the proposed method requires noisy dataset, thus Self2Self can cover the case in this manuscript while the proposed method may not be able to cover the case of [Quan2020] with a single noisy image, (b) Self2Self yielded state-of-the-art performance on well known datasets such as Set9, BSD68, thus the readers can know that Self2Self is one of the state-of-the-art denoising methods. Even though the proposed method was evaluated on 13 datasets, none of them are used to evaluate denoisers in general. Therefore, it is hard to justify that the current proposed method also achieved state-of-the-art performance. (c) Self2Self was evaluated on both color, gray-scale images with synthetic noises as well as real noise (PolyU) while the proposed method was evaluated only on gray-scale images with real noise.\n\nIn addition, the way of implementing the generation network for samples was already proposed in [Kohl2018] for segmentation task. Thus, the contribution of the proposed method could be weakened substantially according to [Kohl2018], too, even though this manuscript nicely demonstrated that this sampling method in [Kohl2018] worked well for denoising problems.\n\n2) Table 1 shows 13 datasets, but it is hard to use them to compare with other denoiser results in other literature. Many denoising works are using common datasets such as Set9, BSD68 (for synthetic noise), and often PolyU (for real noise) and it is helpful to have these results (at least in the appendix) for the comparison purpose with other previous works. Moreover, most images that were used in this manuscript seem to have specific structures (cells, characters), so I am not convinced that the proposed method will be generalized to regular images and color images. The proposed method has noise estimation components (co-learned, bootstrapped) that may be sensitive to underlying ground truth image structures and properties. Showing some results for common datasets (with complicated textures and patterns) as well as some synthetic noise with different noise levels (ideal cases) will convince readers that the proposed method can be generalized to other cases. The proposed method involves noise estimation and it seems to work well with texts, faces and microscopy imaging. However, these results do not seem to be easily extended to other imaging cases. \n\nIn addition, it seems that generating 1000 images for averaging will be computationally expensive. Thus, in Table 1, it will be fair to indicate computation time for training / testing per image.\n\n\n\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "M5SQwAT8fyT", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes a modified VAE for unsupervised image denoising. Unlike existing methods only predicting a single denoised image, the proposed one can generate diverse and plausible results. The experimental results show that this method can outperform existing unsupervised denoising methods. I have several concerns about this paper.\n\n(1) The modifications relative to Kingma & Welling (2014) are marginal. The proposed method more likes an application of Kingma & Welling (2014).\n\n(2) According to Eq. 1, the noises are pixel independent which does not hold for most of the applications e.g. images after demosaicing or ISP denoising. How can the proposed method deal with it?\n\n(3) VAE framework seems only to work for small and constrained images. I am wondering about the denoising results of the proposed method for high-resolution nature images?\n\n(4) At the beginning of Sec. 5, the authors claim that the noise model should be known. How does the proposed method deal with the noises without the noise model for example images after demosaicing or ISP denoising?\n-------------------------------------------------------------------------------------------------------------------------------------\n\nI appreciate that this paper has some merits. But I lower my rating because of the limited usage of the proposed method.\nIt seems it cannot handle high-resolution nature images. Most of the experiments use constrained images e.g. biomedical images or images with small resolution.\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "-g8JIhYEBqN", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper devises a novel unsupervised denoising paradigm, DIVNOISING, that allows us, for the first time, to generate diverse and plausible denoising solutions, sampled from a learned posterior. This approach only requires noisy images and a suitable description of the imaging noise distribution, providing a new perspective for the image denoising field. It has demonstrated that the quality of denoised images is highly competitive, typically outperforming the unsupervised state-of-the-art, and at times even improving on supervised results. This paper is well-written and good-organized. However, the reviewer has the following concerns.\n\nQ1. In DIVNOISING, noisy images are been created from a clean signal $s$ via a known noise model, i.e., $x\\sim p_{NM}(x|s)$. One main practical concern is that is there any assumptions about this noise model and which type of noises are mainly considered to be removed?\n\nQ2. The authors have demonstrated the excellent performance of DIVNOISING on several datasets, especially on microscopy datasets. Whether this denoiser can be applied to other noisy scenarios, such as a real color noisy image or MRI.\n\nQ3. Open the source codes.\n", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: A simple but sensible idea to improve VAE with good experimental results.
Decision: Accept (Poster)