Interpretable Unsupervised Diversity Denoising and Artefact Removal
Authors: Mangal Prakash
Mauricio Delbracio
Peyman Milanfar
Florian Jug
Venue: ICLR 2022 Spotlight (2022)
Keywords: Interpretable Unsupervised Image Restoration
Diversity Image Restoration
Unsupervised Image Denoising
Unsupervised Artefact Removal

Abstract: Image denoising and artefact removal are complex inverse problems admitting multiple valid solutions. Unsupervised diversity restoration, that is, obtaining a diverse set of possible restorations given a corrupted image, is important for ambiguity removal in many applications such as microscopy where paired data for supervised training are often unobtainable. In real world applications, imaging noise and artefacts are typically hard to model, leading to unsatisfactory performance of existing unsupervised approaches. This work presents an interpretable approach for unsupervised and diverse image restoration. To this end, we introduce a capable architecture called Hierarchical DivNoising (HDN) based on hierarchical Variational Autoencoder. We show that HDN learns an interpretable multi-scale representation of artefacts  and we leverage this interpretability to remove imaging artefacts commonly occurring in microscopy data. Our method achieves state-of-the-art results on twelve benchmark image denoising datasets while providing access to a whole distribution of sensibly restored solutions.
Additionally, we demonstrate on three real microscopy datasets that HDN removes artefacts without supervision, being the first method capable of doing so while generating multiple plausible restorations all consistent with the given corrupted image.

Published as a conference paper at ICLR 2022
INTERPRETABLE UNSUPERVISED DIVERSITY DENOISING AND ARTEFACT REMOVAL
Mangal Prakash 1.2
Mauricio Delbracio* 3
Peyman Milanfar
Florian Jug 1,2,4
1Center for Systems Biology Dresden
Max Planck Institute (CBG)
Google Research
'Fondazione Human Technopole
prakash@mpi cbg de, {mde lbra, milanfar @8google. com, florian. juge org
ABSTRACT
Image denoising and artefact removal are complex inverse problems admitting mul- tiple valid solutions. Unsupervised diversity restoration, that is, obtaining diverse set of possible restorations given a corrupted image, is important for ambiguity removal in many applications such as microscopy where paired data for supervised training are often unobtainable. In real world applications, imaging noise and artefacts typically hard to model, leading unsatisfactory performance of existing unsupervised approaches. This work presents an interpretable approach for unsupervised and diverse image restoration. To this end, we introduce capable architecture called HIERARCHICAL DIVNOISING (HDN) based on hierarchical Variational Autoencoder. We show that HDN learns an interpretable multi-scale representation of artefacts and we leverage this interpretability to remove imaging artefacts commonly occurring in microscopy data. Our method achieves state- of-the of-the-art results on twelve benchmark image denoising datasets while providing access to a whole distribution of sensibly restored solutions. Additionally, we demonstrate three real microscopy datasets that HDN removes artefacts without supervision, being the first method capable of doing so while generating multiple plausible restorations all consistent with the given corrupted image.
INTRODUCTION
Deep Learning (DL) based methods are currently the state-of outte-of-thheart (SOTA) for image denoising and artefact removal with supervised methods typically showing best performance (Zhang al. 2017, Weigert et al. 2017 Chen et al. 2018 Delbracio et al. 2020). However, in order to be trained. supervised methods need either paired high and low quality images (Zhang et al. [017 Weigert et al.
2017 Delbracio al. 2020) or pairs of low quality images (Lehtinen et al. 2018 Buchholz et al. 2019). Both requirements are often not, or at least not efficiently satisfiable, hindering application of these methods many practical applications, mainly in microscopy and biomedical imaging. Hence, unsupervised methods that can be trained on noisy images alone Krull et al. 2019); Batson & Royer (2019) al. (2020); Quan et al. (2020) present an attractive alternative. Methods additionally
using models of imaging noise have also been proposed (Krull et al.) 2020 Laine et al. 2019 Prakash
al. 2019b) and can further boost the denoising performance of unsupervised models.
Still, unsupervised methods suffer from three drawbacks: (t) they fail to account for the inherent uncertainty present in corrupted image and produce only single restored solution (i.e. point estimation) (ii) they typically show weaker overall performance than their supervised counterparts, and (iti) they are, by design, limited to pixel-wise noise removal and cannot handle structured noises or other image artefacts (spatially correlated noise).
Recently, the first of these drawbacks was addressed by DIVNOISING (DN) (Prakash et al. 2021 which proposed a convolutional Variational Autoencoder (VAE) architecture for unsupervised de- noising and generates diverse denoised solutions, giving users access to samples from a distribution of sensible denoising results. But DN exhibits poor performance on harder (visually more com- plex and varied) datasets, e.g. diverse sets natural images. Additionally, it does not improve on
'This is also true for supervised methods.

---

Published as a conference paper at ICLR 2022
the performance achieved with other unsupervised denoisers existing microscopy benchmark datasets (Prakash et al., 2021), where supervised methods, whenever applicable, lead to best results. Last but not least, DN does not address the problem of artefact removal (structured noise removal).
We hypothesize that the performance of methods like DN is limited by the used VAE architecture, which can neither capture longer range dependencies, nor the full structural complexity of many practically relevant datasets. Although more expressive hierarchical VAE (HVAE) architectures have been known for some time in the context of image generation (Sønderby et al. 2016 Maaløe et al.) 2019; Vahdat & Kautz 2020; Child (02T). so far they have never been applied to image restoration or denoising. Besides, well performing HVAE architectures are computationally very expensive. with training times easily spanning many days or even weeks on multiple modern GPUs (Vahdat & Kautz 2020 Child (02T). Hence, it remains unexplored if more expressive HVAE architectures can indeed improve unsupervised image restoration tasks and if they can do so efficiently enough to justify their application in practical use-cases.
Contributions. In this paper, we first introduce new architecture for unsupervised diversity denoising called HIBRARCHICAL DIVNOISING (HDN), inspired by ladder VA (Sonderby et al. 2016). but introducing a number of important task-specific modifications. We show that HDN is considerably more expressive than DN (see Fig. 2). leading to much improved denoising results (see Fig. D while still not being excessively computationally expensive (on a common GB Tesla P100 GPU training still only takes about day). Most importantly though, HDN leads to SOTA results on natural images and biomedical image data, outperforming popular unsupervised denoising baselines and considerably closing the gap to supervised results (see Table D.
Additionally, we investigate the application of diversity denoising to the removal of spatially correlated structured noise in microscopy images, an application of immense practical impact. More specifically, we show that HDN, owing to its hicrarchical architecture, enforces an interpretable decomposition of learned representations and we devise effective approach to remove structured noise in microscopy data by taking full advantage of this interpretable hierarchical representation. We showcase this on multiple wallworld datasets from diverse microscopy modalities, demonstrating that our method is an important step towards interpretable image restoration and is immediately applicable to many real-world datasets as they are acquired in the life-sciences on daily basis.
Related Work. Apart from the DL based works already discussed carlier, classical methods such as Non-Local Means (Buades et al., (005) and BM3D (Dabov et al., 2007) have also found widespread use for denoising applications. A detailed review of classical denoising methods can be found in Milanfar (2012). An interesting unsupervised DL based image restoration method is Deep Image Prior (DIP) (Ulynnov et 2018) which showed that convolutional neural networks (CNNs) can be used to restore corrupted images without supervision when training is stopped at an apriori unknown time before convergence. Methods such as DIP (or other methods that require training for each input image separately, e.g. SELF2SELF (Quan et al. (2200) are, however, computationally demanding when applied to entire datasets. Diversity pixel denoising based on VAEs was introduced by DN (Prakash et al. 2021), and similar approach using GANs by Ohayon et al. (202T).
For structured noise removal, Broaddus al. (220)) extended Krull et al. 2019 to specifically remove structured line artefacts that can arise in microscopy data. Their method needs exact apriori knowledge about the size of artefacts, which cannot span more than some connected pixels. |Dorta et al.) (2018) model structured noises as the uncertainty in the VAE decoder represented by structured Gaussian distribution with non-diagonal covariance matrix which is learned separately by another network. Jancsary et al. (2012) learn structured noise model in regression tree field framework. Unlike these approaches, we take an interpretability- first perspective for artefact removal and remove artefacts even without having/learning any structured noise model.
2 THE IMAGE RESTORATION TASK
The task of image restoration involves the estimation of a clean and unobservable signal ($1, 82,..., from a corrupted reconstruction x = (2,, ,N), where and r;, refer to the respective pixel intensities in the image domain. In general, the reconstructed image comes from solving an inverse imaging problem giving by the forward model,
y A(s) e,
2

---

Published as a conference paper at ICLR 2022
where A is the forward operator (tomography, blurring, sub-sampling, etc.), is noise in the measure- ments typically assumed iid, and y is the noisy measurements. An image reconstruction algorithm is needed to recover estimation x of s from the noisy measurements y.
Typically, the image reconstruction obtained through an optimization formulation, where we seek a solution that fits the observed values and is compatible with some image prior R,
x arg min [A(s') yl" AR(s'),
where s' is the auxiliary variable for the signal being optimized for and À > 0 is related to the level of confidence of the prior R. There exists an extensive amount of work defining image priors (Rudin
it al. 1992 Golub et al. 1999; Bruckstein et al., 2009 Zoran & Weiss 2011: Romano et al. 2017
Ulyanov al. 2018).
Without loss of generality, we can decompose the reconstructed image as x S + n, where n is the residual (noise) between the ideal image and the reconstructed one. Generally, the noise n on the reconstruction x is composed of pixel-noise (such as Poisson Gaussian noise) and multi-pixel artefacts or structured noise that affect groups of pixels in correlated ways. Such artefacts arise through dependencies that are introduced by the adopted reconstruction technique and the domain- specific inverse problem (e.g. tomography, microscopy, or ISP in an optical camera). For example, consider the case where the reconstruction is done using Tikhonov regularization R(s) [s]?, in the particular case of linear operator. In this case, the solution to Eq. 2 is x
(A² A A A n,
11))11"" A y. Thus,
x (A" A + AI) A" As (A" A
XI)
(3)
where
I)s + (AT A + 11)-11A-
(4)
The reconstructed image is affected by colored noise (term in Eq. 4 depending on c), and also by structured perturbation introduced by the reconstruction method (term in Eq. ddepending on s). This structured perturbation appears even in the case when the measurements are noiseless. Accurately modeling the noise/artefacts on the reconstructed image is therefore challenging even in the simplest case where the reconstruction is linear, showing that structured noise removal is non-trivial.
In Section 5 we deal with image denoising where noise contribution n, to each pixel z; assumed to be conditionally independent given the signal 8;, i.e. p(n[s) {)) [.n, (Krull et al. 2019). In many practical applications, including the ones presented in Section 6 this assumption does not hold true, and the noise n is referred to structured noise/artefacts.
3 NON-HIERARCHICA VAE BASED DIVERSITY DENOISERS
Recently, non- hierarchical VAE based architectures (Kingma & Welling 2014 Rezende et al. 2014 Prakash et al. (2021) were explored in the context of image denoising allowing key advantage of providing samples & drawn from a posterior distribution of denoised images. More formally, the denoising operation of these models fp can be expressed by fx(() s p(s|x), where & are the model parameters. In this section, we provide a brief overview of these models.
Vanilla VAES. VAEs are generative encoder-decoder models, capable learning latent representa- tion of the data and capturing distribution over inputs x (Kingma & Welling 2019 Rezende et al.) 2014). The encoder maps input x to conditional distribution go ( (z|x) x) in latent space. The decoder, go (), takes sample from 4 x/xx and maps it to distribution pe (x z)) in image space. Encoder and decoder are neural networks, jointly trained to minimize the loss
(5) with the second term being the KL-divergence between the encoder distribution (z(z))) and prior distribution p(z) (usually a unit Normal distribution). The network parameters the encoder and decoder are given by o and 6, respectively. The decoder usually factorizes over pixels as
[x,o(x) (x)) log po (x/z)] + KL (go(z)x)\lp(z)) (x) + (x)(,),
Pa
I (x,
with po(zy(z) (z) being a Normal distribution predicted by the decoder, x; representing the intensity of pixel and N being the total number of pixels in image The encoder distribution is modeled in similar way, factorizing over the dimensions of the latent space z. Performance of vanilla VAEs on pixel-noise removal tasks is analyzed and compared to DN in Prakash et al, (2021).
3

---

Published as a conference paper at ICLR 2022
DIVNOISING (DN). DN (Prakash al. 2021 is VAE based method for unsupervised pixel noise removal that incorporates am explicit pixel wise noise model PN x x|s) in the decoder. More formally, the generic Normal distribution over pixel intensities of Eq. is replaced with known noise model PNM (x|s) which factorizes as product of pixels, i.e. PNM (x|s) [Л (x:)) and the decoder learns mapping from z directly to the space of restored images, i.e. 9o (z(z) = s. Therefore,
pe(x|z) (NM((x(9e(z))
The loss of DN hence becomes
(x.0(x)= = (xlpx)
(2/)x)
log ((z))l90(z))) + KL (g0())))lp(zz) (x) + (x).
(8)
Intuitively, the reconstruction loss in Eq. measures how likely the generated image is given the noise model, while the KL loss incentivizes the encoded distribution to be unit Normal. DN is the current SOTA for many unsupervised denoising benchmarks (particularly working well for microscopy images), but performs poorly for complex domains such as natural image denoising (Prakash et al.) 2021.
HIERARCHICAL DIVNOISING
Here we introduce new diversity denoising setup called HIERARCIICCAL DIVNSISI (HDN) built on the ideas of hierarchical VA (Sonderby et al. 2016 Maaloe et al. 2019 Vahdat & Kautz 2020: Child 2021 HDN splits its latent representation over n > hierarchically dependent
stochastic latent variables z {Z1, ....... ). As in (Child 2021). is the bottom-mos most latent variable (see Appendix Fig. 5 seeing the input essentially unaltered form. The top-most latent variable is 2n. receiving conditioning back down the hierarchy. The architecture consists of two paths: a bottom-up path and top-down path implemented by neural networks with parameters & and 0, respectively. The bottom-up up path extracts features from a given input at different scales, while the top-down path processes latent variables different scales top-to- to-bottom conditioned on the latent representation of the layers above.
-1 times downsampled input and starting cascade of latent variable
The top-down network performs a downward pass to compute a hicrarchical prior po(z) which factorizes as
(z po (...)
po (z, (zy>s),
with each factor pa (z)z/)))>)) being learned multivariate Normal distribution with diagonal covari- ance, and Zjsi referring to all zj with j > i.
Given noisy imput x, the encoder distribution 4s (z/xx is computed in two steps. First the bottom-up network performs a deterministic upward pass and extracts representations from noisy input x at each layer. Next, the representations extracted by the top-down network during the downward pass merged with results from the bottom-up network before inferring the conditional distribution
(x(2)x) go (z, (x) I Go. 0z.
(10)
The conditional distribution pe p=(x(z) described in Eq. 7 The generative model along with the prior (z) and noise model PNM (x|s) describes the joint distribution
pa(z, x, s) PNM (x)(()))
(11)
Considering Eq. 7 and 9 the denoising loss function for HDN thus becomes
(x,0(x) = LR, (x) + KKL (0o((z//))ll0 (z.)))
Σ dlo (qo,e (a,.((/)))sss ())o [z]=)))))}}
(12)
with C (x) being the same as in Eq. 8 Similar to DN, the reconstruction loss in Eq. 12 measures how likely the generated image is under the given noise model, while the KL loss IS calculated between the conditional prior and approximate posterior at each layer.
Following Vahdat & Kautz 2020); Child (2021),
gated residual blocks in the encoder and decoder. Additionally, as proposed in Maalge et al. (201);; Liévin et al.] (2019). our generative path uses skip connections which enforce conditioning all layers above. We also found that
4

---

Published as a conference paper at ICLR 2022
Seperitoo
Unterdiicall Sergle
liknrrakal Sangle 2
bangli Suagpie
Vartance Map
Uiternticall SIMSE
Gread Ini
Noty
Sxpar
1004
Derradi Tooffves Corbedl
Nia
Maa
JUAANN Candiar
Naks
BEAD Fane
Kegi
A
1
A
2
1
17
MANST
3
8
8
7
3
Figure 1: Qualitative pixel-noise removal with HIERARCHICAL DIVNOISING (HDN). For subset of the
datasets we lested (rows), we show denoising results for one representative input image. Columns show, from to right, the imput image and an interesting crop region, results of CARE, two randomly chosen HDN samples, a difference map between those samples, the per- per-pixel variance of 100 diverse samples, the MMSE estimate derived by averaging these 100 samples, and the ground truth image.
employing batch normalization (loffe & Szegedy) 2013) additionally boosts performance. To avoid KL-vanishing (Bowman et al., (2155), we use the so called freeed free-bits approach (Kingma et al. 2016 Chen et al. 2016), which defines threshold the KL term in Eq. 12 and then only uses this term if its value lies above the threshold. A schematic of our fully convolutional network architecture is shown in Appendix A.1
Noise Model. Our noise model factorizes as pNM(x|s) = II PNM (x, (1))8)), implying that the corrup- tion occurs independently per pixel given the signal, which is only true for pixel-noises (such as Gaussian and Poisson). Pixel noise model can easily be obtained from either calibration images (Krull al., 2020 Prakash et al. 2019b) or via bootstrapping from noisy data (Prakash al. 2019b; 2021). Interestingly, despite using such pixel noise models, we will later see that HDN can also be used to remove structured noises (see Section
5 DENOISING WITH HDN
To evaluate our unsupervised diversity denoising method, we used publicly available denoising datasets from different image domains, including natural images, microscopy data, face images, and hand-written characters. Some of these datasets are synthetically corrupted with pixel-noise while others are intrinsically noisy. Examples of noisy images for each dataset are shown in Fig. Appendix A.I.1 describes all datasets in detail.
We compare HDN on all datasets against (i) 7 unsupervised baseline methods, i.e. BM3D Daboy et al. 2007), Deep Image Prior (DIP) (Ulynnvv et al. 2018). SELF2SELF (S2S) (Quan et al. 2020).
NoISE2 VOld (N2V) (Krull et al. 2019). Probabilistic NOISE2VOI (PN2V) (Krull al. 2020
NoIsE2SAME AME (Xe et al.) 2020), and NOISINGG (DN) (Prakash et al. 2021): and ( against
2 supervised methods, namely NOISE2NOISE (N2N) (Lehtinen et al. 2018) and CARE (Weigert
5

---

Published as a conference paper at ICLR 2022
Non DL
Single-image DL
Multi Multi-image DL
Diversity DL
Supervised
Dataset
BM3D
DIP
S2$
N2V
N2Same
PN2V
HVAE
DN
HDN
N2N CARE
Convallaria
35.45
35.73
36.46
36.47
34.11
36.90
37.39
36.85
36.71
Confocal Mice
37.95
37.56
37.96
38.17
31.62
37.82
38.28
38.19
38.39
2 Photon Mice
33.81
33.42
33.58
33.67
25.96
33.61
34.00
34.33
34.35
Mouse Actin
33.64
33.39
32.55
33.86
27.24
33.99
34.12
34.60
34.20
Mouse Nuclci
36,20
35.84
36.20
36.35
32.62
36.26
36.87
37.33
36.58
Flywing
23.45
24.67
24.79
22.81
24.85
25.33
25.02
25.59
25.67
25.79
BSD68
28.56
27.96
28.61
27.70
27.95
28.46
27.20
27.42
28.82
28.86
29.07
Set12
29.94
28.60
29.51
28.92
29.35
29.61
27.81
28.24
29.95
30.04
30.36
BioID Faces
33.91
32.34
34.05
33.76
31.65
33.12
34.59
35.04
35,06
CelchA HQ
33.28
30.80
31.82
33.01
31.45
31.41
33.54
33.39
33.57
Kanji
20.45
19.95
20.28
19.40
20.44
19.47
20.72
20.56
20.64
MNIST
15.82
19.04
18.79
13.87
20.52
19.06
20.87
20.29
20.43
Table I: Quantitative evaluation of pixel-noise removal several datasets. We compare results in terms of mean Peak signal-to-noise ratio (PSNR, in dB) over runs. Note, PSNR is logarithmic measure. Best performing method is indicated with underline, best performing unsupervised method is shown bold. Diversity methods allow to generate an arbitrary number of plausible denoised reconstructions given the single observation. For all datasets, our HIERARCHICAL DIVNOISING (HDN) is the new unsupervised SOTA method.
latent
PSNR
res. blocks PSNR
Gating
BN Gen. skips
Free bits
PSNR
1
27.81
28.49
28.58
2
28,46
2
28.57
28.37
3
28.63
3
28.66
28.61
4 28.75
4 28.67
28.66
5
28.83
5 28.82
28.82
6 28.82
6 28.82
(c) Ablation of other units
(a) Latent variables ablation (b) Res. blocks ablation
(6 latent layer, res. blocks/layer)
(5 res. blocks/layer)
(6 latent variables)
Table 2: Ablation studies on BSD68 denoising with HDN. We find that performance usually improves with (a) increasing the number of latent layers. (b) increasing the number of residual blocks per latent layer. (c) incorporating gating blocks, batch norm. skip connections bottom-up path and free-bits thus validating the importance each component and validating our careful design. PSNR results (in dB) are shown.
al. 2018). Note that we are limiting the evaluation of DIP and S2S 3 and 2 datasets respectively because of their prohibitive training times on full datasets. All training parameters for N2V, PN2V,
DN, N2N and CARE are as described in their respective publications (Krull et al. 2019 Prakash et al. 2021; Krull et al., 2020; Goncharova et al.] 2020). NOISE2S AME, SELF2SELF and DIP are
trained using the default parameters mentioned Xie et al. (2020), Quan et al. (0220) and Ulynov
ct al. 2018), respectively.
HDN Training and Denoising Results. All but one HDN networks make use of 6 stochastic latent The one exception holds for the small MNIST images, for which we only use 3 latent variables. For all datasets, training takes between 12 to 24 hours on single Nvidia P100 GPU requiring only about GB GPU memory. Full training details can be found in Appendix A. and detailed description of hardware requirement and training and inference times is reported in Appendix| AA.9 All denoising results are quantified in terms of Peak- Seak-Signal-to-Noiss Ratio (PSNR) against available ground truth (GT) in Table PSNR is logarithmic measure and must be interpreted as such. Given a noisy image, HDN can generate an arbitrary number of plausible denoised images. We therefore approximated the conditional expected mean (i.e., MMSE estimate) by averaging 100 independently generated denoised samples per noisy input. Fig. T shows qualitative denoising results.
variables 21,... Zg.
For datasets with higher noise and hence more ambiguity such DenoiSeg Flywing, the generated samples are more diverse, accounting for the uncertainty in the noisy input. Note also that each generated denoised sample appears less smooth than the MMSE estimate (see e.g. BSD68 BiolD results in Fig. T.. Most importantly, HDN outperforms all unsupervised baselines on all datasets and even supersedes the supervised baselines on 3 occasions. A big advantage of fully unsupervised approach like HDN is that it does not require any paired data for training. Additional results using the SSIM metric confirming this trend and are given Appendix A.5
Practical utility of diverse denoised samples from HDN. Additional to the ability to generate diverse denoised samples, we can aggregate these samples into point estimates such as the pixel-wise median (Appendix A.6) MAP estimates (Appendix A.7p. Additionally, the diversity of HDN

---

Published as a conference paper at ICLR 2022
Flywing
Convalaria
Kanji
120 128477
704 25
1e0 12 HIIN band 1:000
HONN SAAA
プ
スラスガ
a
HON conrro
X
Beck 14KA
Figure 2: Accuracy of the HDN generative model. We show real images (bottom row), unconditionally generated images from HIERARCHICAL DIVNOISING (middle row) and those from DIVNOISING (top row)
different resolutions for the DenoiSeg Flywing, FU-PN2V Convallaria and Kanji datasets. The generated images from our method look much more realistic compared to those generated by DIVNOISING, thereby validating the fact that our method is much more expressive and powerful compared to DIVNOISING.
Unsupervised
Supervised
N2V
PN2V 29.43
Struct N2V 30.02
DN 31.09
HDN (Ours)
HDN3 (Ours)
CARE 31.56
Convct.onvvllliaa
29.33
29.96
31.36
Table 3: Quantitative striping artefacts removal. On the real- worldworld microscopy data from Broaddus al. (020) (Struct. Convallaria), the proposed HDNN6 produces statc-of-the-ar results in terms of PSNR (in dB), outperforming all unsupervised haselines. Note that PSNR is logarithmic measure.
samples is indicative of the uncertainty in the raw data, with more diverse samples pointing at greater uncertainty in the input images (see Appendix 1..1).. On top of all this, in Appendix [A.8 we showcase practical example of an instance segmentation task which greatly benefits from diverse samples being available.
Comparison of HDN with Vanilla Hierarchical VAE (HVAE) and DN. Here, we compare the performance of HDN with Vanilla HVAE and DN, conceptually the two closest baselines. Vanilla HVAE can be used for denoising and does not need noise model. We employed HVAE with the same architecture as HDN and observe (see Table I that HDN is much better for all 12 datasets.
Arguably, the superior performance of HDN can be attributed to the explicit use of a noise model. However, Table also shows that HDN is leading to much improved results than DN, indicating that the noise model is not the only factor. The carefully designed architecture of HDN contributing by learning a posterior of clean images that better fits the true clean data distribution than the one learned by DN (see also Fig. 2). A detailed description of the architectural differences between DN and HDN are provided in Appendix AI
The ablation studies presented in Table additionally demonstrate that all architectural components of HDN contribute the overall performance. Also note that the careful design of these architec- tural blocks allows HDN to yield superior performance while still being computationally efficient
(requiring only around GB on a common Tesla P100 GPU, training for only about 24 hours). Other
hierarchical VAE models proposed for SOTA image generation (Vahdat & Kautz 2020 Child 2021 need be trained for days or even weeks on multiple GPUs.
Accuracy of the HDN generative model. Since both HDN and DN are fully convolutional, can also use them for arbitrary size unconditioned image generation (other than denoising). For the DN setup, we sample from the unit normal Gaussian prior and for HDN, learned priors as discussed in Section and then generate results using the top-down network. Note that in either case, the images are generated by sampling the latent code from the prior without conditioning on an input image. We show a random selection of generated images for DenoïSeg Flywing, FU-PN2V Convallaria and Kanji datasets in Fig. 2 (and more results in Appendix (A.2). When comparing generated images from both setups with real images from the respective datasets, is rather evident that HDN has learned a fairly accurate posterior of clean images, while DN has failed to do so.
sample from the
7

---

Published as a conference paper at ICLR 2022
Represcitation luyer
Repissotatiin Reprssstatiin Reyresentiion Repressiittin Reprcestatiio layer
layer
kayer
layer'
Figure 3: Visualizing what HIER HIERARCHICAL DIVNOISING encodes at different latent layers. Here we inspect the contrihution of each latent layer in our n = latent layer hierarchy when trained with data containing structured noise. To inspect contribution of latent variable z; fix the latent variables of layers > i, draw k = 6 conditional samples for layer and take the generating then total of & denoised samples (columns). This repeated every latent variable layer (rows). Please see Appendix (.444xxAAfffr details. An interesting observation is that structures that resemble the structured line artefacts are only visible in layers and 2, thus motivating the proposed HDNg-6 network (sec Tables and
of the conditional distributions at layers m <
and
INTERPRETABLE STRUCTURED NOISE REMOVAL WITH HDN
HDN is using a pixel noise model that does not capture any property of structured noises (artefacts). Here we show and analyze the ability of HDN to remove structured noise in different imaging modalities, even if artefacts affect many pixels in correlated ways. To our knowledge, we are the first to show this, additionally looking at through the lens of interpretability, without utilizing prior knowledge about structured noise and without learning/emploiing a structured noise model.
Additional to pixel-noise, microscopy images can be subject to certain artefacts (see Fig. 4 and Ap- cendix6ooor a detailed characterization of such artefacts). Hence, we applied the HDN setup described in Section[[] 6 stochastic latent layers such confocal microscopy dataset from Broaddus et al. (2020), subject to intrinsic pixel-noise as well as said artefacts. In order to apply HDN, we learnt a pixel-wise GMM noise model as mentioned in Section
4 We observed that HDN did not remove these striping patterns (see Appendix Fig. and Table 3). presumably because the hierarchical latent space of HDN is expressive enough to encode all real image features and artefacts contained in the training data.
However, the hierarchical nature of HDN lends an interpretable view of the contributions of individual latent variables 21...., to the restored image. We use the trained HDN network to visualize the image aspects captured hierarchy levels striping artefacts are almost exclusively captured in and 22, the two bottom-most levels of the hierarchy and corresponding to the highest resolution image features. Motivated by this finding, we modified our HDN setup to compute the conditional distribution qs.0 (z, م x) of Eq. [0 to only use information from the top-down pass, neglecting the encoded input that usually received from bottom-up computations.
in Fig. 3 Somewhat surprisingly, we find that
Since we want to exclude artefacts which are captured in Z1 and Zy we apply these modifications only to these two layers and keep all operations for layers to 6 unchanged. We denote this particular setup with HDN listing all layers that operate without alteration. As can be seen in Table the trained HDN 3- network outperforms all other unsupervised methods by rather impressive margin
8

---

Published as a conference paper at ICLR 2022
HDN
HDN2-6
HDN3-6
HDN4-6
PSNR 29.96
31.24
31.36
28.13
Table 4: Performance of HDN networks with "deactivated" latent layers. We show results for HDN setups (in terms of PSNN in dB) with different layers of the original HDN architecture "deactivated" (see main text and Appendix [A.4 for more details). In line with our observations Fig.
3
the HDN3-6 setup leads to the best
performance on the data of [Broaddus el al. 2020). which we used here.
Ours
Input
HDN,, Sample
HDN, Sample 2
HDN,, MMSE
Ground
Convalluria Spining Disk
Three Photon Meuse
?
Does not exist
Spining Disk
?
MSC
Does exist
Figure 4: Qualitative structured noise results microscopy datasets. For three microscopy datasets coming from different modalities (rows), we show artefact removal results for one representative crop input images (leftmost column). We illustrate the nature of the structured artefacts for cach dataset by indicating the spatial autocorrelation of the noise on top of the raw images in the first column. See Appendix for a more detailed description. Note that N2V recovers structured noise and fails to remove it. We show two randomly chosen HDN samples which successfully remove structured noise, the HDNs MMSE estimate derived by averaging 100 restored samples and ground truth image.
and is now removing the undesired artefacts (see the third, fourth and fifth columns in first row in Fig. 4 for this dataset). To validate our design further, we also computed results by "deactivating" only the bottom-mot layer (HDN2-o), and the bottom-most three layers (HDN All results are shown in Table The best results are obtained, as expected, with HDN3-6-
Furthermore, we demonstrate the applicability of our proposed HDN3-s setup for removing mi- croscopy artefacts for two more real-world datasets: (i) mouse cells imaged with three-photon microscopy and (it) mesenchymal stem cells (MSC) imaged with spinning disk confocal micro- scope. We present the qualitative results for all three datasets in Fig. 4 Note that ground truth for the spinning disk MSC images not acquired and for the three-photon data is technically unobtainable, illustrating how important our unsupervised HDN3-6 setup is for microscopy image data analysis.
7 CONCLUSIONS AND FUTURE WORK
In this work, we have looked at unsupervised image denoising and artefact removal with emphasis on generating diverse restorations that can efficiently be sampled from posterior of clean images, learned from noisy observations. We first introduced HIERARCHICAL DIVNOISING (HDN), a new and carefully designed network utilizing expressive hierarchical latent spaces and leading SOTA results on 12 unsupervised denoising benchmarks. Additionally, we showed that the representations learned by different latent layers of HDN are interpretable, allowing us to leverage these insights to devise modified setups for structured noise removal in real-world microscopy datasets.
Selective "deactivation" of latent layers will find a plethora of practical applications. Microscopy data is typically diffraction limited, and the pixel-resolution of micrographs typically exceeds the optical resolution of true sample structures. other words, true image signal is blurred by a point spread function (PSF), but imaging artefacts are often not. Hence, the spatial scale of the true signal in microscopy data is not the same as the spatial scale of many structured microscopy noises and the method we proposed will likely lead to excellent results.
Our work takes an interpretability- have shown, method leads to SOTA results for pixel-wise noise removal and for structured noise removal in microscopy, the situation is more complicated for structured noises in other data domains (see Appendix A. for details) and is slated for future work.
perspective in the context of artefact removal. While, as we

---

Published as a conference paper at ICLR 2022
The authors would like to thank Alexander Krull University of Birmingham and Sander Dieleman at DeepMind for insightful discussions, the Prevedel Lab at EMBL Heidelberg for helpful discussions and providing the Three Photon Mouse dataset and the Tabler Lab MPI-CBG for providing the Spinning Disc MSC dataset. Funding was provided from the care hudget of the Max-Planck Institute of Molecular Cell Biology and Geneties (MPI-CBG), the MPT for Physies of Complex Systems, the Human Technopole, and the BMBF under code 03100102 (de.NBI) and the DFG under code JU3110/1-1 (FiSS). We also want to thank the Scientific Computing Facility at MPI-CBG giving us access to their computing resources.
ACKNOWLEDGEMENTS,
REFERENCES
BioID Face Database
Face Detection Dataset
facedb. URL httts::/www.bbodd.
Eacedb/
Joshua Batson and Loic Royer. Noise2self: Blind denoising by self-supervision, 2019.
Samuel R Bowman, Luke Vilnis, Oriol Vinyals, Andrew M Dai, Rafal Jozefowicz, and Samy Bengio. Generating sentences from a continuous space. arXiv preprint arXiv: 2015.
C. Broaddus, A. Krull, M. Weigert, U. Schmidt, and G. Myers. Removing structured noise with self-supervised blind-spot networks. In 2020 IEEE 17th International Symposium on Biomedical Imaging (ISBI). pp. 159-163, 2020.
Alfred M Bruckstein, David L Donoho, and Michael Elad. From sparse solutions of systems of equations to sparse modeling of signals and images. SIAM review, 51(1):34-81, 2009.
Antoni Buades, Bartomeu Coll, and J-M Morel. A non-local algorithm for image denoising. In 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'05, (05), volume 2, pp. 60-65. IEEE, 2005.
Tim-Oliver Buchholz, Mareike Jordan, Gaia Pigino, and Florian Jug. Cryo-care: Content-aware image restoration for cryo-transmission electron microscopy data. In 2019 IEEE 16th International Symposium on Biomedical Imaging (ISBI 2019), pp. 502-506. IEEE, 2019.
Tim-Oliver Buchholz, Mangal Prakash, Alexander Krull, and Florian Jug. Denoiseg: Joint denoising and segmentation. arXiv preprint arXiv:2005.02987 2020.
Chen Chen, Qifeng Chen, Jia Xu, and Vladlen Koltun. Learning to see in the dark. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2018.
Xi Chen, Diederik P Kingma, Tim Salimans, Yan Duan, Prafulla Dhariwal, John Schulman, llya Sutskever, and Pieter Abbeel. Variational lossy autoencoder. arXiv preprint arXiv: 161.02331.. 2016.
Rewon Child. Very deep (vae)s generalize autoregressive models and can outperform them on images. In International Conference on Learning Representations, 2021. URL http::tttt/ponreeeeeeee net/ forum? R==
Tarin Clanuwat, Mikel Bober-Irizar, Asanobu Kitamoto, Alex Lamb, Kazuaki Yamamoto, and David
Ha. Deep learning for classical japanese literature. arXiv preprint 1rXiv:1812.007111 2018.
Kostadin Dabov, Alessandro Foi, Vladimir Katkovník, and Karen Egiazarian, Image denoising by sparse 3-d transform-domain collaborative filtering. IEEE Transactions on image processing, 16 (8):2080-2095, 2007.
Mauricio Delbracio, Hossein Talebi, and Peyman Milanfar. Projected distribution loss for image
enhancement. arXiv preprint arXiv:20/2.09288 2020.
Garoe Dorta, Sara Vicente, Lourdes Agapito, Neill DF Campbell, and Ivor Simpson. Structured uncertainty prediction networks. In Proceedings of the IEEÉ Conference on Computer Vision and Pattern Recognition, pp. 5477-5485, 2018.
10

---

Published as a conference paper at ICLR 2022
Gene H. Golub. Per Christian Hansen, and Dianne P O'LLear.. Tikhonov regularization and total least squares. SIAM journal on matrix analysis and applications, 21(1):185-194, 1999.
Anna Goncharova, Alf Honigmann, Florian Jug, and Alexander Krull. Improving blind spot denoising for microscopy. In European Conference on Computer Vision, pp. 380-393. Springer, 2020.
Sergey loffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In International conference on machine learning, pp. 448-456. PMLR, 2015.
Jeremy Janesary. Sebastian Nowozin, Toby Sharp, and Carsten Rother. Regression tree fields an efficient, non-parametric approach image laheling problems. In 2012 IEEE Conference Computer Vision and Pattern Recognition, pp. 2376-2383 IEEE, 2012.
Tero Karras, Timo Aila, Samuli Laine, and Jaakko Lehtinen. Progressive growing of gans for improved quality, stability, and variation. arXiv preprint arXiv: 10196, 2017.
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. 3rd international conference on learning representations, iclr 2015. arXiv preprint arXiv: 1412,6994 9, 2015.
Diederik P. Kingma and Max Welling. Auto-encoding variational bayes. In Yoshua Bengio and Yann LeCun (eds.), 2nd International Conference Learning Representations, ICLR 2014 Banff. AB, Canada, April 14-16, 2014, Conference Track Proceedings, 2014. URL http:
//arxiv.org/aba 1312. 6114
Diederik P. Kingma and Max Welling. An introduction variational autoencoders. Foundations and Trendso in Machine Learning, 12(4):307-392, 2019. ISSN 1935-8237. doi: 10.1561/22000000556 URL http: /x doi. www.or///.o 10. 1561 2200000056
Diederik P Kingma, Tim Salimans, Rafal Jozefowicz, Xi Chen, Ilya Sutskever, and Max Welling. Improving variational inference with inverse autoregressive flow. arXiv preprint arXiv: 04934, 2016.
Alexander Krull, Tim-Oliver Buchholz, and Florian Jug. Noise2void-learning denoising from single noisy images. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 2129-2137, 2019.
Alexander Krull, Tomas Vicar, Mangal Prakash, Manan Lalit, and Florian Jug. Probabilistic Noise2Void: Unsupervised Content-Aware Denoising. Front. Comput. Sci., 2:60, February 2020.
Samuli Laine, Tero Karras, Jaakko Lehtinen, and Timo Aila. High-quality self-supervised deep image denoising. Advances in Neural Information Processing Systems, pp. 6968-6975 2019.
Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient-based learning applied document recognition. Proceedings of the IEEE, 86(1 (1:1)))227722222 1998.
Jaakko Lehtinen, Jacob Munkberg. Jon Hasselgren, Samuli Laine, Tero Karras, Miika Aittala, and Timo Aila. Noise2noise: Learning image restoration without clean data. arXiv preprint 1803:01803.04189 2018.
Valentin Liévin, Andrea Dittadi, Lars Maaløe, and Ole Winther. Towards hierarchical discrete variational autoencoders. 2019.
Tsung-Yi Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In European conference on computer vision, pp. 740-755. Springer, 2014.
Lars Maaløe, Marco Fraccaro, Valentin Liévin, and Ole Winther. Biva: A very deep hierarchy of latent variables for generative modeling. arXiv preprint
102, 2019.
Richard Marsh. The beetle. Broadview Press, 2004.
Peyman Milanfar. A tour of modern image filtering: New insights and methods, both practical and theoretical. IEEE signal processing magazine, 30(1):106-12) 2012.
11

---

Published as a conference paper at ICLR 2022
Guy Ohayon, Theo Adrai, Gregory Vaksman, Michael Elad, and Peyman Milanfar. High perceptual quality image denoising with a posterior sampling egan. arXiv preprint arXiv:2103.0419, 2021.
Mangal Prakash, Tim-Oliver Buchholz, Manan Lalit, Pavel Tomancak, Florian Jug, and Alexander Krull. Leveraging self-supervised denoising for image segmentation, 2019a.
Mangal Prakash, Manan Lalit, Pavel Tomancak, Alexander Krull, and Florian Jug. Fully unsupervised probabilistic noise2void. arXiv preprint arXiv:1911.1222 122291,,,, 2019b.
Mangal Prakash, Alexander Krull, and Florian Jug. Fully unsupervised diversity denoising with convolutional variational autoencoders. In Intermational Conference on Learning Representations, 2021. URL /URL:ttps:///paee /ttE::::/ppenree.ee..tteeeeeee http://EEorrmaaaaaaaaaCCCCCSM jP
Yuhui Quan, Mingqin Chen, Tongyao Pang, and Hui Ji. Self2self with dropout:
Learning self-
supervised denoising from single image. In Proceedings the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 1890-1898, 2020.
Danilo Jimenez Rezende, Shakir Mohamed, and Daan Wierstra. Stochastic backpropagation and approximate inference in deep generative models. arXiv preprint arXiv: 3003v::401.40855 2014.
Yaniv Romano, Michael Elad, and Peyman Milanfar. The little engine that could: Regularization by denoising (red). SIAM Journal on Imaging Sciences, (0(4):1804-1844) 2017.
Stefan Roth and Michael Black. Fields of experts: framework for learning image priors. In 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR 005), volume 2, pp. 860-867. IEEE, 2005.
Leonid Rudin, Stanley Osher, and Emad Fatemi. Nonlinear total variation based noise removal algorithms. Physica D: nonlinear phenomena, 60(1-4):259-2660 1992.
Tim Salimans, Andrej Karpathy, Xi Chen, and Diederik P Kingma. Pixelenn++: Improving the pixelenn with discretized logistic mixture likelihood and other modifications. arXiv preprint arXiv: 2017.
Casper Kaae Sønderby, Tapani Raiko, Lars Maaløe, Søren Kaae Sønderby, and Ole Winther. Ladder variational autoencoders. arXiv preprint arXiv:1602.02282 2016.
Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. Dropout: simple way prevent neural networks from overfitting. The journal of machine learning research, (((1):192-1195))) 2014.
Vladimír Ulman, Martin Maška, Klas EG Magnusson, Olaf Ronneberger, Carsten Haubold, Nathalic Harder, Pavel Matula, Petr Matula, David Svoboda, Miroslav Radojevic, et al. An objective comparison of cell- tracking algorithms. Nature methods, 14112):1141, 2017.
Dmitry Ulyanov, Andrea Vedaldi, and Victor Lempitsky. Deep image prior. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition, pp. 9446-945. 2018.
Arash Vahdat and Jan Kautz. Nvae: A deep hierarchical variational autoencoder. arXiv preprint arXiv:2007.0389. 2020.
Martin Weigert, Loic Royer, Florian Jug, and Gene Myers. Isotropic reconstruction of 3D fluorescence microscopy images using convolutional neural networks. arXiv, April 2017.
Martin Weigert, Uwe Schmidt, Tobias Boothe, Andreas Müller, Alexandr Dibrov, Akanksha Jain, Benjamin Wilhelm, Deborah Schmidt, Coleman Broaddus, Siân Culley, et al. Content-aware image restoration: pushing the limits of fluorescence mieroscopy. Nature methods, 15(12):1090-15 1097, 2018.
Yaochen Xic, Zhengyang Wang, and Shuiwang Ji. Noise2same: Optimizing a self-supervised bound for image denoising. arXiv preprint arXiv:2010. /971, 2020.
Kai Zhang, Wangmeng Zuo, Yunjin Chen, Deyu Meng, and Lei Zhang. Beyond gaussian denoiser: Residual learning of deep enn for image denoising. IEEE Transactions on Image Processing. 26 (7):3142-3155, 2017.
12

---

Published as a conference paper at ICLR 2022
Yide Zhang, Yinhao Zhu, Evan Nichols, Qingfei Wang, Siyuan Zhang, Cody Smith, and Scott Howard. A poisson-gausea denoising dataset with real fluorescence microscopy images. In CVPR, 2019.
Daniel Zoran and Yair Weiss. From learning models of natural image patches whole image restoration. In 20/1 International Conference on Computer Vision, pp. 479-486. IEEE, 2011.
13

---

Published as a conference paper at ICLR 2022
A APPENDIX
A.I DATASETS AND TRAINING DETAILS
Canri
Coar2D
Proilleg
blck
Canr20
Rondial
Mock
Mas Pooling
Merge
Can2D
Tipakons
Ss Reodsal black
Coeetti
Werge back
Sonn book
lapdians block
ta Kenstal blcc
Onn2D
Starn Mock
BXU
EUU
Resceea Mook
Spli
CoeddD
Treddoe blcc
Benitill Neck
Nock
Figure 5: HIERARCHICAL DIVNOISING (HDN) network architecture. The network architecture used for MNIST dataset is shown. For all other datascts, stochastic latent groups are used instead of the 3 groups shown here.
A.1.1 DATASETS CORRUPTED WITH PIXEL-WISE NOISE
We consider six microscopy datasets namely, the (i) FU-PN2V Convallaria dataset from Krull al. (2020); Prakash al. (2019b) which shows 1024 1024 images of a intrinsically real-world
noisy Convallaria sections, the (ii FU-PN2V Mouse Nuclei dataset from Prakash et al. (b).
showing 512 512 images of a mouse skull nuclei, (iii) FU-PN2V Mouse Actin dataset from Prakash
et al. (201b)), showing 1024 x 1024 images of actin labeled sample, (iv) Two Photon Mice dataset
from Zhang et al. 2019 9 of size 512 512, (v) Confocal Mice dataset from Zhang et al. 2011 of
size 512 x 512, and DenoiSeg Flywing dataset from Buchholz et al. (2020). showing 5 512 images developing Flywing. Following Prakash et al.] (2021), ) we synthetically corrupted DenoiSeg Flywing images with zero mean Gaussian noise of standard deviation 70 while the other five microscopy datasets are real-world intrinsically noisy images. The Two Photon Mice and Confocal Mice datasets are available in multiple noise levels starting from highest noise level to the lowest noise level. We present results for the highest noise level (most noisy images) for both these datasets.
The train and validation splits for these datasets are as described in the respective publication. For the FU-PN2V Convallaria dataset, the test consists of 100 images of size 512 512, for the FU-PN2V
14

---

Published as a conference paper at ICLR 2022
Mouse Actin dataset, the test set consists of 100 images of size 1024 x 512, for the FU-PN2V Mouse Nuclei dataset the test set has 200 images of size 512 x 256 while the test set for DenoiSeg Flywing dataset consists of 42 images of size 512 x 512. Both Two Photon Mice and Confocal Mice datasets consist of 50 test images each of size 512 x 512.
From the domain of natural images, consider two grayscale test datasets namely, (vii) the popular BSD68 dataset from (Roth & Black 2003) containing 68 natural images of different sizes, and (uitz) the Ser12 dataset from (Zhang et al., 2017) containing 12 images of either size 512 x 512 or 256 256. Both datasets have been synthetically corrupted with zero mean Gaussian noise of standard deviation 25, For training and validation, we use a set of 400 natural images of size 180 x 180 as released by Zhang et al. (2017) and has later also been used in Krull et al. (2199)
Next, we chose to use (iz) the popular MNIST dataset (LeCun et al. 1998) showing 28 28 images of digits, and (z:) the Kuzushiji-Kan KanussiiiiKaajii dataset (Clanuwat et al. 2018 8 showing 64 x 64 images of Kanji characters. Both datasets have been synthetically corrupted with zero mean Gaussian noise of standard deviation 140. For the Kanji dataset, we randomly select 119360 images for training and 14042 images for validation. The test set contains 100 randomly selected images not present in either training or validation sets. For the MNIST dataset, the publicly available data has two splits containing 60000 training images and 10000 test images. We further divide the training images into training and validation sets containing 541000 and 6000 images, respectively. We take random subset of 100 images from the test data and use this as our test set to evaluate all methods. The noisy train, validation and test images for Kanji and MNIST' data will be released publicly.
Finally, we selected two datasets of faces, namely, (z1) the BiolD Faces dataset (noa) showing 286 x 384 images of different persons under different lighting conditions, and (aii)) the CelebA HQ dataset at 256 x 256 image resolution containing faces of celebrities from Karras et al. (2017 2 The
CelebA HQ dataset has RGB images and we take only the red channel images for our experiments.
Both facial datasets have been synthetically corrupted with zero mean Gaussian noise of standard
deviation 15. For the BiolD Faces dataset, we choose 340 images in the training set and 60 images in the validation set while the test set contains 100 additional images. For the CelebA HQ dataset, the train and validation sets contain 27000 and 1500 images, respectively, while the test set contains 100 additional images. We will release the noisy versions of these datasets publicly.
A.1.2 DATASETS CORRUPTED WITH STRUCTURED NOISE/ARTHFACTS
We consider three real-world intrinsically noisy microscopy datasets namely, the (i) Struct. Con- vallaria dataset from Broaddus et al. (020) which shows 1024 x 1024 images of a intrinsically real-world noisy Convallaria sections imaged with spinning disk confocal microscope, the (ti) Three Photon Mouse dataset showing intravital calcium imaging data (256 x 256 images) of astrocytes ex- pressing GCaMP6s, acquired the cortex of an anesthesized mouse at 814um depth below the brain surface with three photon microscopy. and (izi) Spinning disk MSC dataset showing 1024 x 1024 images of mouse Mesenchymal Stem Cells imaged with a spinning disk confocal microscope.
For the Struct. Convallaria dataset, the test set consists of 100 images of size 512 x 512, for the Three Photon Mouse dataset, the test set consists of 500 images of size 256 x 256 while for the MSC dataset the test set has 100 images of size 512 512. All these datasets are intriniscally noisy containing both pixel noise and structured noise/artefacts.
Description of structured noise/artefacts in microscopy images. Structured noise/artefacts are common in microscopy and arise due to various reasons (e.g.cament camera/deetecor imperfectionss. However, spatial correlations of such artefacts are quite variable, rendering unsupervised artefact removal non- trivial. To better characterize structured noises in our datasets, we compute the spatial autocorrelation of the noise (See Appendix Fig. 6).
For the Convallaria dataset subject to structured noise, we obtain the residual noise image by subtracting the ground truth from the raw image data. Since for the three photon mouse and spinning disk MSC datasets no ground truth images are available, we estimate the noise at background areas (empty areas) of the data, assuming that such regions only contain noise. For reference, we also show the spatial autocorrelation for Gaussian noise (non-structured pixel-wise noise) in the first column
TThe full dataset can be downloaded from the Google Drive link at https: :/gi twwb..u.cco.o suvojit- (ooojitt0x55aaaa 0x55005555aa- ce hata--goaat..tet-daee
15

---

Published as a conference paper at ICLR 2022
Pixel independent noise
Convallaria structured moise
3 Photon Rat Structured Noise
MSC strietured noise
Dataser
Noise spatial autocorelation
Figure 6: Illustration of structured noise/artefacts microscopy images. better illustrate structured noise in the considered datasets, we take the noise (first row) and compute their spatial autocorrelation (second row). For structured Convallaria dataset, the noise is obtained by subtracting the ground truth image with noisy image. For the three photon mouse and spinning disk MSC datasets, ground truth images hence, noise estimated at background regions (empty areas) of the images, which supposedly only contain noise. For reference, we also show the spatial autocorrelation of Gaussian noise (pixel independent noise) in first column. For cach case, we show the autocorrelation of noise for one pixel in the center (shown in red). For all three real microscopy datasets, there is spatial correlation between noise unlike the Gaussian noise case where the noise is independent per pixel. Also that the spatial extent of the structured noise different for different datasets. Still, our proposed HDN3-6 makes no assumption on prior knowledge spatial extent of these artefacts and successfully removes them for all considered datasets whereas other methods such as NOISE2 VOID only remove pixel noise and recover structured noise (see Fig. 4 in main text).
available
of Appendix Fig. 6. It can be seen that for all real-world microscopy datasets the noise spatially correlated unlike pixel-wise Gaussian noise.
A.1.3 TRAINING DETAILS OF HIERARCHICAL DIVNOISING
Our HDN network is fully convolutional and consists of multiple hicrarchical stochastic latent variables. For experiments
network with 6 hierarchical latent variables, only for the MNIST dataset we have only used hierarchy of height 3. Each latent group has H; x W; x 32 dimensions where H, and W; represent the dimensionality of the feature maps layer i, The initial number of filters is set to 64 and we utilize batch-normalization (Ioffe & Szegedy] 2015) and dropout (Srivastava et al. 2014) with probability of 0.2. Both top-down and bottom-up networks have 5 residual blocks between each latent group. A schematic of the residual blocks and the full network architecture is shown in Fig.
We use an initial learning rate of 0.0003 and always train for 200000 steps using Adamax (Kingma & Ba 2015). During training, we extract random patches from the training data (128 x 128 patches for BiolD) Faces and natural image datasets, 256 x 256 patches for CelebA HQ. 28 x 28 patches for MNIST, and 64 x 64 patches for all other datasets). batch size of 16 was used for BioID Faces and BSDD68 while batch size of was used for the CelebA HQQ dataset. For all other datasets, a batch size 64 was used. To prevent KL vanishing (Bowman et al. 2015), we use the free bits approach as described in Kingma et al. (2016); Chen et al. (2016). For experiments all datasets other than the MNIST dataset, we set the value of free bits to 1.0, while for the MNIST dataset a value of 0.5 was used.
16

---

Published as a conference paper at ICLR 2022
A.1.4 ARCHITECTURE OF HIERARCHICAL DIVNOISING
Figure shows our HIERARCHICAL DIVNOISING architecture with 3 stochastic latent variables, as used for experiments on the MNIST data. For all other experiments our architecture is the same, but has 6 latent variable groups instead of 3. Please note that we have used the exact same architecture for pixel-wise and structured noise removal experiments. Our networks with 6 latent groups need around GB of GPU memory and were trained on Tesla P100 GPU. The training time until convergence varies from roughly 12 hours to 1 day depending on the dataset.
A.2 How ACCURATE IS THE HIERARCHICAL DIVNOISING POSTERIOR?
In Section 5 in main text, we qualitatively compared the learned posterior of HIERARCHICAL DIVNOISING with DIVNOISING. Since both methods fully convolutional, can generate images of arbitrary size. Expanding on what was discussed in Section main text, here we show more generated images for multiple datasets. For the DIVNOISING setup, we sample from the unit normal Gaussian prior and for Hierarchical DIVNOISING, we sample from the learned priors as discussed in Section 3 and then generate results using the top-down network. Note that in either case, the images are generated by sampling the latent code from the prior without conditioning on an input image. Below, we show a random selection of generated images for DenoiSeg Flywing, FU-PN2V Convallaria, Kanji, MNIST and natural images datasets in Fig. Fig. Fig. and Fig. TO respectively at different image resolutions.
When comparing generated images from both setups with real images from the respective datasets, is rather evident that HIERARCHICAL DIVNOISING learns a much more accurate posterior of clean images than DIVNOISING. For the MNIST, Kanji, FU-PN2V Convallaria and DenoiSeg Flywing datasets, the images generated by HIERARCHICAL DIVNOISING have a very realistic appearance. For the natural image dataset the situation is rather different. Generated natural images do not have the same quality as instances of the training dataset do, offering lots of room for future improvement. Hence, it appears that learning a suitable latent space encoding of clean natural images is much harder than for the other datasets.
A.3 QUALITATIVE RESULTS FOR STRUCTURED NOISE EXPERIMENTS WITH HDN
Following what was discussed in Section here we show qualitative results for microscopy striping artefacts removal experiments for HDN (Fig. [T]. Clearly, HDN is too expressive and captures the undesirable striping patterns and fails to remove them. Both the individual samples from HDN and the MMSE estimate (averaging 100 HDN samples) clearly show striping patterns. Owing this, the restoration performance of HDN rather poor (see Table Hence, we have proposed HDN to overcome this drawback of HDN (see Section and Table 3 in main text for more details) which establishes a new state-of-the-art.
A.4 VISUALIZING PATTERNS ENCODED BY HDN LATENT LAYERS AND TARGETED DEACTIVATION
In this section present more details regarding Fig, in main text pertaining to visualizing what the individual levels of a HDN learned to represent when trained on Struct. Convallaria dataset containing structured noise. The idea for this way of inspecting Ladder VAE is not new and can also be found in a GitHub repo") authored by a member of the team around Liévin et al. (001
More specifically, want to visualize what our 6-layered HDN model has learned in Section 6 of the main text. We inspect layer i by performing the following steps:
We draw a sample from all latent variables for layers j >
We then draw samples from layer i (conditioned on the samples we drew just before).
For each of these 6 samples, we take the mean of the conditional distributions layers m < i, i.e. we do not sample for these layers.
We generate total of 6 images given the latent variable samples and means from above.
17

---

Published as a conference paper at ICLR 2022
Real 128k 128
Real 64x64
IDD (ours)) 1200 22
DXN 126k 128
DXN 64004
Figure 7: Accuracy of the learned posterior. We show real images (top row), generated images from HIERARCHICAL DIVNOISING (middle row) and generated images from DIVNOISING (bottom row) at different resolutions for the DenoiSeg Flywing dataset. The images from our method look much more realistic compared to those generated by DIVNOISING, thereby validating the fact that our method learns a much better posterior distribution of clean images compared to DIVNOISING.
Real 128 22
Real 640064
HDX (UBrrs) 128k 224
HDN
DN 128K-122 124
DN 646044
Figure 8: Accuracy of the learned posterior. We show real images (top row), generated images from HIERARCHICAL DIVNOISING (middle row) and generated images from DIVNOISING (bottom row) at different resolutions for the FU-PN2V Convallaria dataset. The images from our method look much more realistic compared to those generated by DIVNOISING, thereby validating the fact that our method learns much better posterior distribution of clean images compared to NOISINGG
18

---

Published as a conference paper at ICLR 2022
Kanji
MNIST
8
5
6
Real
6
5
8
(ours))
DN PANO
2
Figure Accuracy of the learned posterior. We show real images (top row), generated images from HIERARCHICAL DIVNOISING (middle row) and generated images from DIvNOISING (bottom row) for the Kanji dataset in first half and the MNIST dataset in second half. The images from method look much more realistic compared to those generated DIVNOISING, thereby validating the fact that method learns much better posterior distribution of clean images compared to DIVNOIIING.
HDN 128
(onrr))
DN 64464
DN 128
Figure 10: Accuracy of the learned posterior. We show generated images from HIFRARCHICAL NIVNSIG (top row) generated images from DIVNOISING (bottom row) at different resolutions for natural images. The images from our method look much more realistic compared to those generated by DIVNOISING, although there seems to be plenty of room for improvement. The generated images by HDN vaguely resemble structures such rock, clouds, etc. More meaningful structures like people, animals, etc. not generated at all. This indicates that our method is not powerful enough capture the tremendous diversity natural image datascts but still gives SOTA denoising performance on natural image benchmarks (see Table in main text and Appendix Table 5).
19

---

Published as a conference paper at ICLR 2022
Input
HDN Sample
HDN Sample
HDN
Ground Truth
Spining Disk Convallaria
Figure 11: Qualitative results of structured noise experiments with HDN. As explained the main text, HDN owing to its high expressivity captures striping artefacts and fails to remove them, thereby leading to inferior restoration performance (see Table B). Hence, we propose the modified HDN3-6 for microscopy structured noise removal (see Fig. andd Table 3 main text) which establishes new stale-of-the-art.
The results of this procedure on the HDN network used Section can be seen in Fig. in main text.
By visually inspecting Fig. 3 in main text one can observe that the bottom two layers (layers 1 and 2) learned fine details, including the line artefacts would like remove from input images. The other layers (layers 3 to 6), on the other hand, either learn smoother global structures or learn such abstracted concepts that the used visualization method is not very informative.
Still, the created visuals help us to understand why the full HDN leads to worse results for removing microscopy line artefacts than a DN setup did (see Table in main text). Owing to the high expressivity of the HDN model, the structured noise can actually be encoded in the hierarchical latent spaces and the artefacts will therefore be faithfully be represented the otherwise denoised resulting image.
Motivated by these observations, we proposed a simple modification to HDN that has then led to SOTA results on the given benchmark dataset (see again ij the main text). We showed that by "deactivating" layers 1 and 2, i.e. the ones that show artefact-like patterns in Fig. B the modified HDN model restored the data well and removed pixel noises well as the undesired line artefacts (structured noise).
We also computed results by "deactivating" only the bottom-most layer (HDN2-6). the lowest two layers as described the main text (HDN3-6), and the bottom-mosi three layers (HDN4-6). All results are shown in terms of peak signal-to-noise ratio (PSNR) in Table in main text. The best results are obtained with HDN3-6 setup.
A.5 PIXEL-WISE DENOISING IN TERMS OF SSIM
While we have measured pixel-denoisin performance the main text in terms of PSNR, here we quantify results also in terms of Structural Similarity (SSIM), Table Our proposed HDN setup outperforms all unsupervised baselines for all datasets except for Set12 dataset where it is the second best. Notably, HDN even surpasses SSIM results of supervised methods on datasets.
A.6 PIXEL-WISE MEDIAN ESTIMATES
We compute the pixel-wise median estimates (computed by taking the per-pixel median for 100 denoised samples from HDN) and compare them to MMSE estimates in terms of PSNR (dB). The results are reported in Table 6 MMSE estimates are marginally better than pixel-wise median estimates for all datasets.
A.7 MAP ESTIMATES
We show the utility of having diverse denoised samples by computing MAP estimates and qualitatively comparing them to MMSE estimates. We present the case for the MNIST dataset (LeCun et al. 1998)
(Fig. 12) and the eBook dataset (Marsh] 2004) (Fig. [3). both also used in Prakash et al. 202
In order to compute the MAP estimate, we follow the procedure of Prakash et al. (2021) involving a mean-shift clustering approach with 100 ddiverse HDN samples. The MMSE estimates represent
20

---

Published as a conference paper at ICLR 2022
Input
HDN MMSE (Ours)
HDN MAP (Ours)
Ground Truth
4
4
4
333
Figure 12: Comparison of HDN MMSE and MAP estimates for two representative MNIST inputs. We show noisy input images (first column). HIERARCHICAL DIVNOISING MMSE estimates computed from 100 samples (second column), HIERARCHICAL DIVNOISING MAP estimates (third columm) computed using a mean-shift clustering approach with the same 100 samples and ground truth images (fourth column). Note that the MMSE estimate is a compromise hetween all possible denoised solutions and hence the obtained MMSE results have faint overlays suggesting compromise between the digits 9 and 4 (first row) and the digits 3 and 8 (second row) while the MAP estimate is committing to one interpretation.
Input
MMSE oh whic egggd
HDDN MMNE (Ours)
MNP
(Oers)
Ground
ob who pegge hunger
ob whic ob whic ob whic
i lgggd
pegged i begged
hunger
hunger
hungor hunger hunger
ob whic ob whic ob whic ob whic ob which
Figure 13: Comparison of HDN MMSE and MAP estimates for eBook dataset. We show noisy input image (first column). DIVNOISING MMSE estimate taken from Prakash et al. (202I) (second column), our HIERARCHICAL NOVSING MMSE estimates computed from 100 samples (third column), DIVNOISING MAP estimate taken from Prakash et al. () (fourth column), our HIERARCHICAL DIVNOISING estimate (fifth column) and ground truth mmage (sixth column). Note that the MMSE estimate is compromise between all possible denoised solutions and hence the obtained MMSE results have faint overlays suggesting compromise between different possible letters while the MAP estimate commits to one interpretation letters.
21

---

Published as a conference paper at ICLR 2022
Non DL Single-image DL
Multi-image DL
Diversity DL
Supervised
Dataset
BM3D
DIP
S2S
N2V
N2Same
PN2V
HVAE
DN
HDN
N2N
CARE
SSIM
Convallaria
0,939
0,954
0.963
0,959
0.911
0.966
0.966
0.963
0,962
Confocal Mice
0.963
0,963
0.965
0.965
0.693
0.964
0.966
0.966
0.967
2 Photon Mice
0.924
0.919
0.918
0.920
0.355
0.918
0.920
0.924
0.906
Mouse Actin
0.855
0.850
0.818
0.867
0.653
0.877
0.889
0.890
0.882
Mouse Nuclci
0,952
0,952
0.958
0.955
0.852
0.958
0.962
0.967
0,959
Flywing
0.721
0.843
0.482
0.821
0.850
0.842
0.852
0.865
0.866
BSD68
0.801
0.774
0.803
0.778
0.776
0.801
0,764
0.754
0.812
0.812
0.827
Set12
0.850
0.772
0.832
0.828
0.818
0.831
0,800
0.787
0.837
0.844
0.860
BioID Faces
0.909
0.865
0.907
0.891
0.770
0.886
0.920
0.920
0.919
CelebA HQ
0,904
0.851
0.884
0.895
0.825
0.874
0.918
0.912
0,914
Kanji
0,504
0.678
0.431
0.576
0.839
0.664
0.844
0.891
0.888
MNIST
0.521
0.707
0.475
0.455
0,868
0.759
0.874
0.869
0.866
PSNR
Convallaria
35.45
35.73
36.46
36.47
34.11
36.90
37.39
36.85
36.71
Confocal Mice
37.95
37.56
37.96
38.17
31.62
37.82
38.28
38.19
38.39
2 Photon Mice
33.81
33.42
33.58
33.67
25.96
33.61
34.00
34.33
34.35
Mouse Actin
33.64
33.39
32.55
33.86
27.24
33,99
34.12
34.60
34.20
Mouse Nuclei
36.20
35.84
36.20
36.35
32.62
36.26
36.87
37.33
36.58
Flywing
23.45
24.67
24.79
22.81
24.85
25.33
25.02
25.59
25.67
25.79
BSD68
28.56
27.96
28.61
27.70
27.95
28.46
27.20
27.42
28.82
28.86
29.07
Set12
29.94
28.60
29.51
28.92
29.35
29.61
27.81
28.24
29.95
30.04
30,36
BioID Faces
33.91
32.34
34.05
33.76
31.65
33.12
34.59
35.04
35.06
CelebA HQ
33.28
30.80
31.82
33.01
31.45
31.41
33.54
33.39
33.57
Kanji MNIST 15.82
20.45
19.95
20.28
19.40
20.44
19.47
20.72
20.56
19.04
18.79
13.87
20.52
19.06
20.87
20.29
20.43
Table 5: Quantitative pixel-noise removal with HDN. For all conducted experiments, compare results terms of Structural Similarity (SSIM) and PSNR. The best performing method is indicated by being under- lined, best performing unsupervised method is shown bold. For all one dataset, our HIERARCHICAL DIVNOISING is the new unsupervised SOTA method, even superseding the competing supervised baselines on 1 of the used datasets.
Datasets
HDN MMSE
HDN Median Estimate
Convallaria
37.39
37.28
Confocal Mice
38.28
38.25
2 Photon Mice
34.00
33.91
Mouse Actin
34.12
34.09
Mouse Nuclei
36.87
36.83
Flywing
25.59
25.57
BSD68
28.82
28.50
Set12
29.95
29.85
BioID Faces
34.59
34.55
CelebA HQ
33.54
33.51
Kanji
20.72
20.67
MNIST
20.87
20.78
Table 6: Comparison of HIERARCHICAL DIVNOISING MMSE estimate HIERARCHICAL NOISSNGGGG pixel-wise median estimate. The numbers represent PSNR values. Both the MMSE estimate and median estimates are computed from 100 HDN denoised samples.
a compromise between all possible denoised solutions and hence show faint overlays of diverse interpretations. Note that the MAP estimate gets rid of such faint overlays by committing the most likely interpretation. For the sake of comparison, we have also showed our MAP estimate to that obtained with Div reported in Prakash et al. (021).
22

---

Published as a conference paper at ICLR 2022
0.90
0.775
0.750
Avenge Prosisico
0.85
0.725
HON
0.700
DN OOnOOTT
0.75
0.675
0.650
SNK
0.40
0.45
0.35
0.40
0.30
2
16
32
64
8
16
32
64
Number off sssmpllss
Number of samples
Figure 14: Application of diverse samples for instance segmentation. We use the diverse denoised samples from HDN obtain diverse segmentation results using the procedure presented in PPraaash al. (2021). We then the diverse segmentation results input consensus algorithm which leverages the diversity present in the samples obtain a higher quality segmentation estimate than any of the individual diverse segmentation maps. The segmentation performance is compared against available ground truth in terms of Average Precision (Lin et al., (2014) and SEG score (Ulman et al. 2017). Higher scores represent better segmentation. The segmentation performance corresponding different number of denoised samples improves with increasing number samples
HDN MMSE estimate obtained by averaging for obtaining denoised MMSE estimate. Clearly, the segmentation results obtained with consensus method using our HDN samples outperforms even segmentation on high SNR images using as little as 3 diverse segmentation samples.
A.8 DOWNSTREAM APPLICATION: INSTANCE SEGMENTATION OF CELLS
Here we present a downstream application of having diverse denoising results. We choose the same instance cell segmentation application for the DenoiSeg Flywing (Buchholz et al. 2020) dataset as also presented in Prakash et al. (2021). For fair comparison to DN (Prakash et al.) |02 I). we also use the same unsupervised segmentation methodology as described Prakash et al. (2019a) (locally thresholding images with mean filter of radius 15 followed by skeletonization and connected component analysis) to obtain instance cell segmentations. Thus, we obtain diverse segmentation maps corresponding diverse denoised solutions.
We then used the label fusion method (Consensus (Avg.). )) as described in Prakash et al. (02 which takes the diverse segmentation maps and estimates a higher quality segmentation compared to any individual segmentation. We measure the segmentation performance against available ground truth segmentation in terms of Average Precision (Lin et al. 2014) and SEG score (Ulman et al.) 2017) with higher scores representing better segmentation.
For reference, we also show segmentation result when the same segmentation method is applied directly to noisy low SNR and ground truth high SNR images. The segmentation performance on our HDN MMSE estimate obtained by averaging different number of denoised samples improves with increasing number of samples used for obtaining the denoised MMSE estimate. Clearly, the segmentation results obtained with the deployed consensus method using our HDN samples outperform even segmentation high SNR images. This is true starting with as little 3 diverse segmentation samples. The performance of the same consensus algorithm with the diverse samples obtained with NOVSING (DN) is also shown and is much inferior compared to results.
A.9 HARDWARE REQUIREMENTS TRAINING AND INFERENCE TIMES
Hardware Requirements. The training for all models presented in this work were performed single Nvidia Tesla P100 GPU requiring only GB of GPU memory. Hence, our models can be trained relatively cheaply on commonly available GPU.
Number of Trainable Parameters and Training Times. All our networks have 7.301 million train- able parameters except for the network used for MNIST dataset which has 3.786 million trainable parameters. Thus, networks have rather small computational footprint compared to many commonly used generative network architectures such as PixeICNN+ (Salimans et al. (017).
BIVA (Maalee et al., 2019), NVAE (Vahdat & Kautz 2020 and VDVAE (Child 2021) with 53
23

---

Published as a conference paper at ICLR 2022
million, 103 million, 268 million and 115 million trainable parameters, respectively (numbers taken from Child (2021)).
Depending on the dataset, the training time for our models vary between 12 - 24 hours in total single Nvidia Tesla P100 GPU. In comparison, the training of baseline DN (Prakash et al. 202T) takes around 5 12 hours while the supervised CARE (Weigert et al.] 2018) method takes around 6 hours for training on the same hardware. Thus, our training times are adding only a small factor compared over DN but we show that we get much improved results (see Fig. [ and Table I in main text). Compared to CARE, our method is unsupervised and hence we do not need high and low quality image pairs for training. In fact, our performance is almost on par with supervised CARE results on most datasets and even outperform CARE on some of them (see Table inn main text). An additional advantage of our method over CARE our ability to generate multiple plausible diverse denoised solutions for any given noisy image which other supervised methods like CARE (or even other unsupervised methods except DN) cannot.
Prediction/Sampligg Times. For any dataset, to sample one denoised image with our HDN model, we need at most 0.34 seconds, thus making HDN attractive for practical purposes. The sampling time for one denoised image for all considered datasets shown in Table 7 Given the samples, the MMSE inference takes 0.000012 seconds only. Hence, computing the MMSE estimate takes almost the same time as computing 100 samples. With DN, the MMSE inference time is around seconds for the largest dataset (compared to 34 seconds with HDN). Although the MMSE inference time with HDN is slightly higher, the performance of HDN is far better compared to DN for all datasets (see Table I and Table B while still being fast enough for almost all practical purposes. Additionally, HDN allows for interpretable structured noise removal unlike DN.
Datasets
Time (1 sample)
Time (100 samples)
Convallaria (512 x 512)
0.17 sec
17 sec
Mouse Actin (1024 x 512)
0.34 sec
34 sec
Mouse Nuclei (512 x 256)
0.11 sec
11 sec
Flywing (512 x 512)
0.17 sec
17 sec
Confocal Mice (512 x 512)
0.17 sec
17 sec
2 Photon Mice (512 x 512)
0.17 sec
17 sec
BSD68 (481 x 321)
0.12 sec
12 sec
Set12 (512 512 256 x 256)
0.13
13 sec
BioID Faces (384 x 284)
0.10 sec
10 sec
CelebA HQ (256 x 256)
0.07
7
Kanji (64 x 64)
0.04 sec
4 sec
MNIST (28 28)
0.02 see
sec
Struct. Convallaria (512 x 512)
0.17 sec
17 sec
Struct. Mouse (256 x 256)
0.17 sec
17 sec
Struct. MSC (512 x 512)
0.08 sec
8 sec
Table 7: Sampling times with HIERARCHICAL DIVNOISING. Average time (evaluated over all test images) needed to sample denoised image and 100 denoised images from trained HIERARCHICAL DIVNOISING network. The size of test images is also shown.
A. 10 LIMITATIONS HDN FOR STRUCTURED NOISE REMOVAL
As discussed in Sec. in the main text, we expect that our proposed technique of selectively "deactivating" bottom- up layers will be useful for commonly occurring spatial artefacts in microscopy images and thus has wide practical applicability. Microscopy data is diffraction limited and the pixel-resolution of micrographs typically exceeds the optical resolution of true structures. In such situations, the true image signal is blurred by a point spread function (PSF), but imaging artefacts are often not. Hence, the spatial scale of the true signal in microscopy data is not the same as the spatial scale of many structured microscopy noises and HDN captures these structured noises mostly in the bottom two layers. This allows our HDN setup to disentangle the artefacts from true signals in different hierarchical levels, subsequently allowing us to perform selective bottom-ur up "deactivation" " technique (as described in the main text) to remove said artefacts. However, if the spatial scale of
24

---

Published as a conference paper at ICLR 2022
15-
Sid per pixel
10
30
50
70
Noise level
Figure 15: Measuring sample diversity with increasing noise levels on imput images. We synthetically corrupt DenoiSeg Flywing dataset with different levels of Gaussian noise = 30, 50, 70) and train different HDN networks for these noise levels. For each of these networks, we sampled 100 HDN denoised solutions corresponding to each noisy image in the test set. For the 100 samples, we computed the standard deviation per pixel and report the average standard deviation over all test images. The higher the noise level, the greater the standard deviation per pixel is. This is expected because more noisy data is more ambiguous and hence HDN accounts for this uncertainty in the increasingly noisy input images by generating more diverse samples.
the image artefacts is at the same scale as that of true signals, our HDN setup will not be able to disentangle the artefacts across scales and hence will not be applicable in such cases.
A.11 ARCHITECTURAL DIFFERENCES BETWEEN HDN AND DN
As noted in Sec. f and illustrated in Fig. 5 our HDN architecture is significantly different from DN. HDN uses several architectural components not present DN which ultimately contribute to significant performance boosts.
The foremost difference is that DN uses a single layer latent space VAE architecture whereas we employ multi-layer hierarchical ladder VAE for increased expressivity (sce Table and Fig. 2 Unlike DN, our HDN models have hierarchical stochastic latent variables the bottom-up and top- down networks with each layer consisting of 5 gated residual blocks. As shown in Table increasing the number of latent variables as well as the number of gated residual blocks per layer increases performance till certain point, after which the performance more or less saturates. Additionally, unlike DN, we find that employing batch normalization (Toffe & Szegedy 2015) significantly enhances performance by about 0.5 dB PSNR on BSD68 (see Table Furthermore, the generative path of our model conditions on all layers above by using skip connections which yield a performance gain of around 0.2 dB PSNR on BSD68 dataset. Finally, stabilize network training and avoid the common problem of posterior collapse (Bowman et al.) (015) in VAEs, we use the free-bits approach used in Kingma et al. (2016). This is a significant difference from DN which uses KL annealing (Bowman et al. 2015) to avoid this problem. Note that the public code of DN does not use KL-annealing by default, but only switches it on if posterior collapse happens multiple times (default is set to 20 times in the publicly available implementation). However, we always employ the free-bits approach and we never encounter posterior collapse for any dataset tested so far. In fact, we notice that free-bits also brings in performance improvements (around 0.15 dB BSD68 dataset) in addition to stabilizing the training.
Thus, architecturally there is significant difference between HDN and DN and all the architectural modifications play an important role in achieving a better overall performance of HDN over DN (see Table TT.
A.12 HOW DOES NOISE ON IMAGES AFFECT THE SAMPLE DIVERSITY OF HDN?
Noise introduces ambiguity in the data and hence, more noisy images are more ambiguous. Thus, we expect that HDN samples will be more diverse if input images are more noisy. To verify this, we corrupted the DenoiSeg Flywing dataset with three levels of Gaussian noise (a 30, 50, 70) and trained three HDN models, respectively. During prediction, for each noisy image in the test set, we sampled 100 denoised images and compared the per pixel standard deviation. We report the average
25

---

Published as a conference paper at ICLR 2022
of the standard deviations over all test images in Appendix Fig. 15. As expected, the higher the noise on the imput datasets, the greater the per pixel standard deviation of the 100 denoised samples, indicating higher data uncertainty at higher levels of noise.
26


Reviews:
[{"review_id": "p1wFaCi9dhX", "reviewer": "Reviewer_kqm1", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n\nThe proposed model is a novel unsupervised technique that is competitive or outperforms the state of the art using a multiscale variational autoencoder framework. I believe that this is a very interesting contribution. In addition, the paper provides extensive experiments and ablation analysis that provide interesting insights into the properties of the model. Also, the paper is written clearly.\n\nWeaknesses:\n\nIn the abstract and introduction, the authors imply that their method produces meaningful uncertainty quantification because it enables one to sample possible restored images, but at the end of the day they use an average as the final denoised result in order to compare to other methods. The stochastic outputs are not evaluated. It would be very interesting (and would back up the tone of the abstract and introduction) if the authors could at least provide an example where the stochastic inputs are useful in some way. This is not a condition for publication, I think the method is interesting in any case. ", "summary_of_the_paper": "The paper proposes a hierarchical VAE model that produces good denoising results. In addition, the model can be adapted to perform removal of more structured artefacts. ", "summary_of_the_review": "The authors show that a multiscale VAE can produce good unsupervised denoising results, provide meaningful analysis, and show that their technique can be adapted to remove structured artefacts. I think this is a meaningful contribution, and therefore I recommend publication.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "c4Gq_eyvDw", "reviewer": "Reviewer_iaBg", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\nThe paper shows one can improve the performance of DN by introducing the hierarchical representation of the latent variables introduced in VAEs. In addition, the paper shows that the artifact removal can be realized by the analysis of the visualized image patterns corresponding to the latent variables in each layer: By deactivating the layers of latent variables at which the artifacts are encoded in the restoration, one can remove artifacts even though they are spatially structured perturbation of pixel values.\n\nWeaknesses:\nArtifacts targeted in the paper are not well characterized. It is not sure if the structured noises would be encoded in the first and/or the second layer(s) of the latent variables independently from the input images. The limitation of the performance of the proposed artifact reduction should be experimentally or theoretically explained more clearly. When the deactivation is not applied, would the HDN always generate similar artifacts independently from the input images or from the network architectures? Is it sure if the artifacts are always enced by the latent variables of layers near the bottom? If the artifacts change depending on the input images as described by Eq.(3) and are encoded by the layers near the top, can the proposed method still reconstruct high quality images even when the layers near the top are deactivated?　It is not clearly demonstrated by the current paper whether the proposed method will show high performance of artifact removal against variety of input images.\n\nBy the way, if the reviewer understands correctly, the artifacts handled by the proposed method are not generated in the process of imaging but essentially by the reconstruction from given images. The noises denoted by e in Eq.(1) does not include artifacts (spatially correlated noises) and the noises denoted by n in Eq.(3) include the artifacts. Denoting both e and n as \"noise\" makes it more difficult to understand the proposed method. Artifacts, structured noises, can be generated even in the process of imaging. It is recommended to denote e and n in different words. It is also recommended to clearly describe that only the artifacts generated in the reconstruction process are handled by the proposed method when describing the related works of structured noise removal.", "summary_of_the_paper": "The paper proposes an image restoration method that can not only reduce pixelwise noises but also remove artifacts in resultant images. Introducing the idea of hierarchical representation of latent variables analogous to VAEs, the proposed method improves the denoising performance of DivNoising (DN). In addition, the authors propose a method for artifacts removal based on the analysis of the image components represented by the latent variables at each layer. The method requires no pairs of noisy images and the corresponding noise-free signals for training but requires the probability distribution of pixel values conditioned by the corresponding signal strength. The experimental results demonstrate the proposed method outperforms DNs.", "summary_of_the_review": "This paper improves  Prakash et al.(2021). The reported facts that the hierarchical representation of the latent variables improve the performance and that the artifacts are encoded at the layers near the bottom would have worth sharing. As for the artifact removal, on the other hand, it is not so clear how effective the proposed method is for variety of images and hence the reviewer downgrades the rating.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "bu7dBcSmtTT", "reviewer": "Reviewer_3vAh", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "This paper has the following strong points\n\n- First work to use hierarchical Variational Autoencoder for unsupervised image denoising task\n- Yields state-of-the-art results on unsupervised image denoising\n- Reconstruction results are more interpretable as compared to competing ones\n- Could remove structured noise in some special cases of imaging\n\nThe only negative that I found concerning this work is that\n\nLack of discussion on training and inference efficiency comparisons of the proposed method with others\nRecommendation: As I have stated under the strong points, this paper brings in multiple contributions in terms of the first attempt of its kind while yielding state-of-the-art results. Therefore I rate this work as novel enough to get accepted.\n\nI have the following questions for the authors.\n\nThere has been no discussion on how much inferior the training and test time of the proposed method as opposed to the competing ones such as DivNoising or even other fully supervised methods. Also, how much time does it take to find the MMSE estimate with 100 samples? I would like authors to have a detailed comparison along these aspects.\nDivNoising has reported the MAP estimate to be best performing. What is the plausibility of such a MAP estimate with the proposed HDN framework? I do not find any discussion regarding this possibility.\nI would like the authors to address the below mentioned points in the final version of their paper.\n\nSection 6 points out to “line artefacts in Fig. 4”. However for a reader, it is hard to understand what these artefacts are. It would be better to give a description or highlight such artefacts for improved revelation of the same. Also, providing additional visualization of the ground truth image can further improve clarity.\n\nSection 7 says “the situation is more complicated for structured noises in other data domains and is slated for future work.” However, it would have been more informative if authors were to show examples of structured noises in these other domains, and how the proposed method works in these cases. This can help shed more light on the limitations of the proposed method and also for future research directions.", "summary_of_the_paper": "This work extends the work of DivNoising (Prakash et al., 2021) to build an interpretable approach for unsupervised image restoration. The proposed method, namely Hierarchical DivNosing (HDN), is based on the well-studied concept of hierarchical Variational Autoencoder. This work is the first to use hierarchical Variational Autoencoder for the task of unsupervised image denoising. In doing so, HDN achieves state-of-the-art unsupervised image denoising results on twelve benchmark image datasets. The method is also shown to remove structured artifacts on three real microscopy datasets.", "summary_of_the_review": "This work advances the state-of-the-art in terms of use-case novelty and performance improvement. Henceforth, the contributions are significant enough for the acceptance\n", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "5ST-v2KwGpa", "reviewer": "Reviewer_5Ptv", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "For the most part, the authors make claims that are well-supported by either the evidence presented in the paper or by the cited research. The presentation of the work, including the use of language, formulas, and graphics is self-consistent and favors smooth reading. The results discussed in the paper make it clear that the Hierarchical DivNoising is superior to all relevant baselines, sometimes even surpassing the performance of supervised methods.\n\nAlthough overall the paper is very comprehensive and thorough, there are two main things that are worth considering:\n1. Section 3, Non-hierarchical VAE Based Diversity Denoisers, discusses the most notable competitors for HDN method, namely classical VAEs algorithms and DivNoising. Later in Section 4, the authors introduce HDN, describing in detail its architecture, loss function, types of extracted representations. The main differences (from the architectural point of view) between HDN and DN need to be explicitly written out either in this or later sections to help the reader assess the novelty of the discussed approach. Also, the authors did not discuss the difference in the number of trainable parameters that can influence both the training time as well as the performance of the model. \n2. Hardware requirements. Multiple times across the paper, the authors highlight the low computational cost of the HDN method compared to other VAE-based approaches. Yet, to the best of my knowledge, no specific data on training times i.e. time for one epoch, inference time on the same size data, total training time have been presented in the text. \n\nA few less critical, yet still relevant comments:\n1. Nor all variables used in equations are explained or defined. For example, variable s' from the equation (2), N, xi, and i in equation (6). It is important that every element is properly defined to help the reader follow the main idea. \n2. Bottom-up, top-down network definitions are missing. In the first paragraph of Section 4, the authors introduce two new terms that have not been defined anywhere above in the text, namely bottom-up and top-down networks. These terms should be defined and explained before being used, otherwise, they might cause confusion.\n3. Some equations would benefit from short intuitive explanations. Equations such as (8) and (12) would be easier to comprehend if they would be supported by short and more intuitive explanations. \n4. Structured noise patterns are not clearly visible in Figure 4, which shows the results of removing structured noise. As input images also contain pixel-noise it is hard to observe the structured noise, except for N2V where it is possible to see it. Is it possible to make an example that very clearly illustrates the structured noise in the original images? ", "summary_of_the_paper": "Reviewed work presented a new and very promising approach to unsupervised denoising titled Hierarchical DivNoising. Hierarchical DivNoising is based on earlier published DivNoising architecture. The authors compared the performance of their HDN approach to the closest state-of-the-art unsupervised as well as supervised denoising methods on an impressive set of twelve datasets. In all datasets, the presented method has outperformed other unsupervised alternatives, while sometimes even improving on the performance of supervised counterparts. The authors also show that the layers of the proposed architecture encode interpretable levels of abstraction, which can be used to one's own advantage e.g. to get rid of the structured noise often present on the microscopy images. ", "summary_of_the_review": "The reviewed paper makes claims that appear to be correct, supported by empirical evidence, theoretical insights, and prior literature. The proposed method is based on existing technology but extends in a number of ways, which makes the novelty of the paper significant. The potential impact of this research seems to be substantial, provided that denoising is an important problem in many applications related to computer vision. Overall, this is sound work, which seems to significantly improve the current state-of-the-art.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Meta-review: A multi-scale hierarchical variational autoencoder based technique is developed for unsupervised image denoising and artefact removal. The method is shown to achieve state of the art performance on several datasets. Further, the multi-scale latent representation leads to an interpretable visualization of the denoising process.

The reviewers unanimously recommend acceptance.
Decision: Accept (Spotlight)