Tomographic Auto-Encoder: Unsupervised Bayesian Recovery of Corrupted Data
Authors: Francesco Tonolini
Pablo Garcia Moreno
Andreas Damianou
Roderick Murray-Smith
Venue: ICLR 2021 Poster (2021)
Keywords: Missing value imputation
variational inference
variational auto-encoders

Abstract: We propose a new probabilistic method for unsupervised recovery of corrupted data. Given a large ensemble of degraded samples, our method recovers accurate posteriors of clean values, allowing the exploration of the manifold of possible reconstructed data and hence characterising the underlying uncertainty. In this set-ting, direct application of classical variational methods often gives rise to collapsed densities that do not adequately explore the solution space.  Instead, we derive our novel reduced entropy condition approximate inference method that results in rich posteriors.  We test our model in a data recovery task under the common setting of missing values and noise, demonstrating superior performance to existing variational methods for imputation and de-noising with different real data sets. We further show higher classification accuracy after imputation, proving the advantage of propagating uncertainty to downstream tasks with our model.

Published as a conference paper at ICLR 2021
TOMOGRAPHIC AUTO-ENCODER: UNSUPERVISED
BAYESIAN RECOVERY OF CORRUPTED DATA
Francesco Tonolini
Pablo Garcia Moreno
School Computing Science
Amazon
University of Glasgow
London, UK
Glasgow, UK 2402432t ac..kk
morepabl @ama zon Co. uk
Andreas Damianou
Roderick Murray-Smith
Amazon
School of Computing Science University Glasgow Glasgow, UK roderice www.raaaa--athhggaaao...........
London, UK
co. .o.uk
ABSTRACT
We propose new probabilistic method for unsupervised recovery of corrupted data. Given large ensemble of degraded samples, our method recovers accurate posterions of clean values, allowing the exploration of the manifold of possible reconstructed data and hence characterising the underlying uncertainty. In this set- ting, direct application of classical variational methods often gives rise to collapsed densities that do not adequately explore the solution space. Instead, we derive our novel reduced entropy condition approximate inference method that results in rich posteriors. We test our model in data recovery task under the common setting missing values and noise, demonstrating superior performance to existing variational methods for imputation and de-noising with different real data sets. We further show higher classification accuracy after imputation, proving the advantage of propagating uncertainty to downstream tasks with our model.
INTRODUCTION
Data sets are rarely clean and ready to use when first collected. More often than not, they need to undergo some form of pre-processing before analysis, involving expert human supervision and manual adjustments (Zhou et al.) 2017; Chu et al. 2016). Filling missing entries, correcting noisy samples, filtering collection artefacts and other similar tasks are consuming stages in the data modeling process and pose an enormous obstacle to machine learning at scale (Munson) 2012). Traditional data cleaning methods rely
of the most costly and time
degree of supervision in the form of clean dataset or some knowledge collected from domain experts. However, the exponential increase of the data collection and storage rates recent years, makes any supervised algorithm impractical in the context of modern applications that consume millions or billions of datapoints. In this paper, we introduce a novel variational framework perform automated data cleaning and recovery without any example of clean data or prior signal assumptions.
The Tomographic auto-encoder (TAE), named in analogy with standard tomography. Tomographic techniques for signal recovery aim at reconstructing target signal, such as 3D image, by algorith- mically combining different incomplete measurements, such as 2D images from different view points, subsets of image pixels or other projections (Geyer et al. 2015). The TAE extends this concept to the reconstruction data manifolds; our target signal is a clean data set, where corrupted data is interpreted incomplete measurements. Our aim to combine these reconstruct the clean data.
More specifically, we are interested in performing Bayesian recovery, where we do not simply transform degraded samples into clean ones, but recover probabilistic functions, with which we can generate diverse clean signals and capture uncertainty. Uncertainty is considerably important when cleaning data. If we over-confident about specific solutions, emror easily ignored and passed

---

Published as a conference paper at ICLR 2021
Corrupted Data Set
Posterions
Possible Recoverios
Corrupted
Data
(1) VAB Recovery 1n Space
TAE Recovery Space
3i g bn)
Draw
32., nM ((a)
TAB
g(r bu)
Draw
7
23. ~ g(22)1)) Us)
Draw
Oberrved Corrupted
Hidden Clean Data
Variational Posterior
* Cormuttd Point
Hidden Poins to Be Recovered
Recovered
(b)
Figure 1: (a) Example of Bayesian recovery from corrupted data with a Tomographic (TAE) on corrupted MNIST. The TAE recovers posterior probability densities ((xl)))) for each corrupted sample y/- We can draw from these to explore different possible clean solutions. (b) Two dimensional Bayesian recovery experiment. (i) Observed set of corrupted data Y, with the point we are inferring from yi highlighted. (ii) Ground truth hidden clean data with the target point x; highlighted. along with the posterior q<>ly)) reconstructed by a VAE. (iii) Posterior ( ys) recovered with our TAE. While the VAE posterior collapses to single point, the TAE reconstructs rich posterior that adjusts to the data manifold.
on to downstream tasks. For instance, the example of figure Ta), some corrupted observations consistent with multiple digits. If we were to impute single possibility for each sample, the true underlying solution may be ignored early on in the modeling pipeline and the digit will be consistently mis-classified. If remain adequately uncertain in any subsequent processing task.
instead able recover accurate probability densities, we can
Several variational auto-encoder (VAE) models have been proposed for applications that can be considered special cases of this problem (Im et al.) 2017 Nazabal et al. 2018 Ainsworth et al. 2018 and, in principle, they are capable of performing Bayesian reconstruction. However, we show that surrogating variational inference (VI) in a latent space with VAEs results in collapsed distributions that do not explore the different possibilities of clean samples, but only return single estimates. The TAE performs approximate VI in the space of recovered data instead, through our reduced entropy condition method. The resulting posteriors adequately explore the manifold of possible clean samples for each corrupted observation and, therefore, adequately capture the uncertainty of the task.
In our experiments we focus on data recovery from noisy samples and missing entries. This is one of the most common data corruption settings being encountered in wide range of domains with different types of data (White et al. 2011; Kwak Kim 2017). By testing our approach this prevalent scenario. can closely compare with recently proposed VAE approaches (Nazabal et al. 2018 Dalca et al., 2019 Mattei & Frellsen 2019). We show how the existing VAE models exhibit the posterior collapse problem while the TAE produces rich posteriors that capture the underlying uncertainty. We further test TAFS on classification subsequent to imputation, demonstrating superior performance existing methods in these downstream tasks. Finally, automated missing values imputation on raw depth maps from the NYU rooms data set.
TAE perform
METHOD
In order to frame the problem and understand the issues with standard variational methods in this context, we view the task from signal reconstruction prospective. The final scope of a Bayesian data recovery method is to build and train a parametrie probability density function (PDF) (x)y)) which takes inputs corrupted samples and generates different possible corresponding clean data x ~ q(x) y) through sampling. There are two aspects we need to design: i) the structure of this conditional PDF and ii) the way it will be trained to perform the recovery task.
Regarding the former, as natural data often lies on highly non-linear manifolds, we need the condi- tional PDF to capture complicated modalities, e.g. the distribution of plausible images consistent with one of the corrupted observations in figure aa)). A suitable recovery PDF q(zly) needs to be able to capture such complexity. A natural choice achieve high capacity and tractability is to
2

---

Published as a conference paper at ICLR 2021
(a) Data Recovery LVM
(b) VAB for Data Recovery
(c) Tomographic
Posterior <(ol))
Empirical Prior
Prior
p(a]
p(
p(zls,)
Q(=
p(alz)
q(=ly)
garla.
Corrupted Data
Latent
Clean Data
Posterior LVM
Posterior LVM
Variable
p(ylar)
ply x))
Figure 2: Training LVMs for data recovery. (a) Structure of the reconstruction LVM used to infer approximate posteriors g( (y) of clean data æ from corrupted observations y/ as conditional inputs. (b) Training of g<<ly) using VAE. A prior in the latent space is introduced as regulariser, however no explicit regularisation is imposed in æ. (c) Training of g(>ly) using our TAE model. An empirical p( (()ppp((()))))))))) 2p ddp is instead introduced in clean data space
prior p(x)
construct (<<lv) as a conditional latent variable model (LVM). Conditional LVM neural networks have achieved efficient and expressive variational inference many recovery settings, capturing complex solution spaces in high dimensional problems. such as image reconstruction (Nguyen jtt al. 2017: Mirza Osindero 2014, Adler Ōktem 2018). The conditional LVM consists of first conditional distribution g( mapping input corrupted data y to latent variables z. and second inference (z)2 y) mapping latent variables to output clean data æ. The resulting PDF is q(al)) f q(2 (=) (2 y)d2, where both (==ly) and q(x) 2, y) simple distributions, such as isotropic Gaussians, whose moments are inferred by neural networks taking the respective conditional arguments as inputs. Figure aa) shows graphical model for the conditional LVM.
While the choice of structure is fairly straightforward, the main difficulty lies in training the recovery LVM in the absence of clean ground truths x. In the supervised case, several established methods exist; the observed distributions of clean data x conditioned on paired observations y can be matched by parametric ones through VAE GAN training strategy (Sohn al. 2015 Adler & Öktem 2018: Tonolini et al.) 2020). However, we are instead interested in the unsupervised situation. where we only have corrupted data Y {yI:N} and functional form for the corrupted data likelihood p(y]æ)) e.g., missing values and additive noise. Training a conditional LVM to fit posteriors without any ground truth examples x rather challenging, as we do not have data encode from, in the case of VAE architectures, adversarially compare with, in the case of GAN models.
2.1 VAES AND THE POSTERIOR COLLAPSE PROBLEM
Variational auto-encoders (VAEs) have been proposed for several problems within this definition of unsupervised reconstruction (Dalca et al. 2019 Im et al. 2017: Ainsworth et al.) 2018). These methods lead to good single estimates of the underlying targets. However, they easily over-fit their posteriors resulting in collapsed PDFs d(xly). Put differently, they are often unable to explore different possible solutions to the recovery problem and return single estimates instead. Figure I(biii) shows this pathology in two dimensional experiment.
The reason for this can be explained considering what the reconstruction LVM (=]gy) is and how it is trained when directly employing VAE in the unsupervised recovery scenario. The VAE encodes latent vectors 2 from corrupted observations y with an encoder q(2ly) and reconstructs clean data æ with decoder p(x[2). These two functions constitute the reconstruction LVM g(øly) [q(=|g)p(z|2)dd As we do not have clean ground truths x, data likelihood is maximised by mapping reconstructed clean samples back corrupted samples y with corruption process likelihood p(y|z), e.g. zeroing out missing entries, to maximise reconstruction of the observations Concurrently, regularisation the latent space induced with a user defined prior p(2) (e.g. a unit Gaussian). The resulting lower bound to be maximised during training can be expressed as follows:
Lv Egta/)) p)rl( KI(g(>/W)llp(e))
where the observations likelihood is 2) = p(x| p(z|=)p()|z)) and some cases, such as for missing values and additive noise, it is analytical. A derivation is given in supplementary A.1.
3

---

Published as a conference paper at ICLR 2021
Viewing the VAE training from a signal reconstruction prospective, where our reconstruction model is ((xlg) = q( p()l) zz|2]dz we can see that we are not introducing any prior directly on the hidden targets 2, but only in the LVM latent space 2. While regularising only in may be computationally desirable, if the decoder p(x]z) is of sufficient capacity, the model can learn to collapse regularised distributions in = to single estimates in x, failing to capture uncertainty. In fact, this is induced by the objective function of equation D the model finds broad distributions in the latent space (z(1y), which minimise the KL divergence with p(z), but the generator p(z[z) can learn to collapse them back to single maximum likelihood solutions in x, maximising By(=)s) log p(z]=))((])) ))d This effect may be counteracted by reducing the capacity of p(z]zz or the dimensionality of z, but doing so also reduces the capacity of the reconstruction model (arly) resulting in undesirable coupling between regularisation and posterior capacity.
2.2 SEPARATING POSTERIOR AND PRIOR: THE TOMOGRAPHIC AUTO-ENCODER
The premise of our model to address the aforementioned problem is simple: Introduce prior p(æ) in the hidden clean signal space. In particular, we propose to use an empirical prior, having itself the form of an LVM p(z) inference in clean data space x, instead of surrogating it to the reconstruction function's 's latent space z. By doing so, we can control the capacity of the prior p(x) induce regularisation independently of the capacity of our reconstruction model ((xly) = ((=l)))(<(), y). For this framework, We can formulate the following ELBO:
In this way, we perform approximate variational
CAAE = (g(ayy)) log p(vl()) + (a(()))) (Bu(sa)a)) (og((112p) - K (L (p){p) + +H ((()lv))
The above ELBO is derived detail supplementary section A.2. The main technical challenge and focus of this paper is how to compute and maximise the self entropy of the approximate posterior H (q(l)))) as this conditional distribution is an LVM of the form ((xly) ) = (2)1)))(<)) y).
Reduced Entropy Condition:
Direct computation of the entropy of LVM model g<>ly)
S. (=<11))((<)2,, d)dz is intractable in the general case. Titsias & Ruiz (201)) proposed an approximate inference method to compute the gradient of the LVM s entropy for variational inference in latent spaces. However, this involves multiple samples to be drawn and evaluated with the LVM, which is expected to scale in complexity the dimensionality and capacity of the target distribution increase.
In our case, we aim to approximately compute and optimise the entropy (gg(ly)) for a distribution capturing natural data, which can be high-dimensional and lie complicated manifolds. In order to maintain efficiency in the entropy estimation, we propose a new strategy; we identify a class of LVM posteriors for which the entropy reduces tractable form and then approximately constrain the posterior to such a class in our optimisation. Our main result is summarized in the following theorem:
Theorem If q(=la_))
((=lg) = Bö(z g(x, v)), where 5(.) is the Dirac Delta function, B is real positive parameter and g(ø, y) is a deterministic function, then H(g(zl)) H(q(zl))) - Ex H (q(z)2, v)).
We detail the proof in supplementary A.3. Theorem sttates that if the posterior over latent variables q(2|1, y) is infinitely more localised than the latent conditional g(>ly), then the LVM entropy H (q(xl)) has the tractable form given above. This condition imposes the LVM posterior to present non non-overlapping conditionals g(z|2, y) for different latent variables 2, but does not impose any explicit restriction to the capacity of the model. We can also formulate the condition as follows:
((2/2, v) C, C - oc.
Eg log
g(2ly)
The proof is provided in supplementary section A.4. To train our posterior g (zll)) we aim to maximise the ELBO LTAB with the reduced entropy, while enforcing the condition of equation? [
arg max Eytely) log p(yla)) + Eycel)) (Eg(sll)) (gg(((/=p) - -K (T((((,)
+H(((()))) + Eycely) H (g(x)2, y)), s.t. Ey(r,)1)) log (11/0, v)
(3)
C, C' + cc.
9(2 ()
4

---

Published as a conference paper at ICLR 2021
While the ELBO is now amenable to stochastic optimization, the constraint is intractable since C' - oc and the posterior ((1/2,)) y) is intractable.
Relaxed Constraint: To render the constraint tractable, we firstly relax C' to be positive hyper- parameter. The higher the value of C, the more localised q( (x, y) is imposed to compared to (=ly)) and the closest the reduced entropy is to the true one.
To address the intractability of the posterior (=|2z, y). we employ a variational approximation with a parametric function r(222, 2. y). In fact, for any valid probability density r(z[a, y), we can prove that
g( [[2]0,8)
((>ly) 2 Bg(,,)l)) log r(2(2,))
E =(x,))y)) log
(4)
((=lv)
The proof is given in supplementary section A.5. The above bound implicates the following:
r(2(2,8) = C = (g(t,)1)) log ((2/2, v) <<>ly)
(((+)))))) log
C.
((=ly)
This means that imposing the condition with a parametric distribution r(= lx, v), which trained along with the rest of the model, ensures deviation from the set condition only by excess. As the exact condition is met only at (y(z)=)y) log g(al=) oc, we can never relax the constraint more than already set by the finite value of C.
((2(M)
The TAE Objective Function: Having defined a tractable ELBO and tractable condition, we need to perform the constrained optimisation
arg max
(x(s)) log p(ylx) (a (B(()))))) log (ogp(a)z
r(2 2, v)
(5)
+H(((=l)) + Bo(aal) (1((1)), v)), s.l. (y(t))))) log
C.
g(2(g)
We use the commonly adopted penalty function method (Zangwill 1967 Phuong et al, 2018) and relax equation 5 to an unconstrained optimisation with the of positive hyper-parameter A:
arg max (all)) log plyl()) ()(())) (Ba((s))) log p(x) =p) K KT(g(=plia [p(p)))]
- A (g(z,sl)) log r r(210, v)
(6)
(g(zlv)) + (d
o
( y)
To train the model, we perform the maximisation of equation 6 using the ADAM optimiser. Once the generate diverse reconstructions from corrupt observation yı by sampling
model is trained, from the posterior ((r(y;). Details of our optimisation are reported in supplementary B.1. We describe how handle parameters of the corruption process p(yl=:) in supplementary B.2.
RELATED WORK
3.1 SUPERVISED BAYESIAN RECONSTRUCTION
The reconstruction of posterior densities from incomplete measurements has been recently investi- gated supervised situations, where examples of clean data are available. In particular, conditional generative models were demonstrated with high dimensional data (Parmar et al., 2018). These methods work by exploiting an LVM generate diverse realisations of targets conditioned associ- ated observations (Isola et al. 2017 Nguyen et al. 2017). Both conditional generative adversarial networks (CGANs) (Mirza& & Osindero 2014] Isola et al. 2017) and conditional VAEs (CVAEs) (Sohn et al.) 2015 Nguyen et al. 2017) have been studied in this context. In both cases, the samples generated by conditioning on an observation can be interpreted as samples from the corresponding conditional posterior densities.
These approaches proved successful a range of recovery tasks: reconstruction of images with missing groups of pixels (Nguyen et al. 2017), super-resolution (Parmar et al. 2018). medical computed tomography reconstructions (Adler & Öktem 2018) and semi supervised situations, where examples of clean data and conditions are available in different amounts (Kingma et al. Dentonn (et al.) 2016) Tonolini et al. 2020). Other works reconstruct manifolds of solutions from observations only, but be considered supervised, as they exploit pre-trained generators (Anirudh et al. 2018).
5

---

Published as a conference paper at ICLR 2021
These works make the important observation that when learning recover data from corrupted or partial observations, there is not a single right solution, but many differently likely ones. We aim to extend this ability to completely unsupervised scenarios, where clean data examples are available.
3.2 UNSUPERVISED BAYESIAN RECONSTRUCTION
Reconstructing posteriors in the unsupervised case is largely still an open problem. However, several tasks that fall within this definition have been recently approached with Bayesian machine learning methods. Arguably the most investigated is de-noising. Several works solve this problem by exploiting the natural tendency of neural networks to regularise outputs (Lehtinen et al. 2018 Kruil et al. 2019a]b) b). Other methods build LVMs that explicitly model the noise process in their decoder, retrieving clean samples upon encoding and generation (Im et al., 2017 7 Creswell & Bharath 2018).
A second notable example is that of missing value imputation. Corrupted data corresponds to samples with missing entries. Recent works have explored the use of LVMs to perform imputation, both with GANs (Li et al., 2019; Yoon et al., 2018 Luo et al.) 2018) and VAEs (Nazabal et al. 2018 Mattei & Frellsen 2019 Ma et al.) 2018). .. In the former, the discriminator of the GAN is trained to distinguish real values from imputed ones, such that the generator is induced to synthesise realistic imputations. In the latter, the encoder of a VAE maps incomplete samples to a latent space, to then generate complete samples. Successful unsupervised Bayesian missing value imputation has also been demonstrated with neural processes, where a global latent representation is learned to generate input-output models used impute in each example (Garnelo et al. 2018).
Finally, Bayesian LVM methods have been used on other unsupervised tasks that can be cast as special cases of data recovery problems. Amongst these, we find Multi-view generation (Shang et al.) 2017 Ainsworth et al., 2018), where the target clean data includes all views for each samples, but the observed data only presents subsets. Blind source separation can also be cast as a recovery problem and has been approached with GANs and VAEs (Kameoka et al.) 2018 Hoshen (019).
These models proved be successful at reconstructing data in their specific domain. However, in our work, we show how exploiting a standard VAE inference structure, similarly to several of the aforementioned methods, often leads to posteriors of clean data that collapse on single estimates, sacrificing the probabilistic capability of LVMs.
3.3 POSTERIOR COLLAPSE IN VARIATIONAL INFERENCE
The posterior collapse problem we approach with the reduced entropy condition method presented in this paper has some analogy with the latent posterior collapse encountered when using implicit distributions in variational inference to obtain flexible recognition models. The main issue in training these models successfully without collapse is the computation of density rations between the latent prior and the implicit variational posterior. This problem is analogous to the difficulty in estimating the LVM entropy in our method. Yin & Zhou (018) proposed to use a further lower bound on the ELBO and add term encouraging diversity to avoid collapse. This term is obtained by drawing K Gaussian components from the LVM posterior and computing the KL divergence of an individual component with the mixture distribution, i.e. the sum of the drawn Gaussians. TTissas & Ruiz (2019) build on this work by deriving an unbiased estimator for the ELBO gradient, instead of using a surrogate lower bound.
These methods rely on estimating the LVM posterior through sampling and aggregating K explicit distribution components. This was demonstrated to work well for posteriors in artificial latent space by drawing only a few components. However, in our data recovery setting, we need to capture the posteriors in clean data space, rather than latent space. Posteriors capturing the uncertainty in natural data are expected to be much more complex and higher dimensional, leading to the number K of drawn Gaussians needed approximate the true LVM with these methods to become rather large, making optimisation inefficient or even intractable in extreme cases. The reduced entropy condition method we derive in this paper avoids the posterior collapse without having estimate the LVM posterior through sampling and is therefore specially suited for the data recovery setting, where we are required to capture posteriors in clean data space.

---

Published as a conference paper at ICLR 2021
(a)
Recovery Examples
(b) Mean Inference
Probabilistic
Ground
Trui hm
Corrupt
Datm
Mean luf.
Draws from Posterior
Performance
Performance
6
6
6
6
6
MVAE ---. TAE
MVAE ---- TAE
MVAB
4
9
)
9
7
4
24
2000
1
7
7
7
7
7
PSNR (IB)
S. 5/5
E.BB
1600
20
6
+
6
6
6
4
1200
TME
4
9
7 4
9 4
16
800
!
1
0.2 0.4 0.6
0.2
0.4 0.6
0.8
Ratio of Observed Entrics
Ratio Observed Entrics
Figure 3: MNIST data recovery from missing entries and noise. (a) Recoveries using an MVAE and our TAE, showing average reconstruction and samples from the trained posteriors. (b) PSNR between ground truths and mean reconstruction. (c) ELBO assigned by the recovered posteriors to the ground truth data. The mean inference performance is very similar for the two models (PSNR values), while the probabilistic performance (ELBO values) is significantly higher for our TAE model. We can evidence of this difference in the reconstruction examples. The MVAE and TAE return similarly adequate mean solutions, but the MVAE posterior's draws all very similar, suggesting that the posterior has collapsed on a particular reconstruction. Contrarily, the posteriors returned by the TAE explore different possible solutions that are consistent with the associated corrupted observation.
Table 1: Bayesian recovery from noisy data with different percentages of missing entries. Table shows the ELBO assigned by the retrieved posteriors to the ground truth clean data. Our TAE model consistently returns higher ELBO values compared to the competing variational methods, as is able to retrieve rich posteriors that adequately sample the solution space. More values in supp. D.3.
MNIST
Fashion-MNIST
UCI HAR
50%
80%
50%
80%
50%
80%
MVAE
870 + 6
803 + 15
757 + 1
723 + 7
585 + 4
471 + 10
MIWAE
917 = 4
780 + 6
800 = 7
766 - 8
613 6
584=
TAE
1719 7
1536 + 14
1326 + 7
1094 = 13
1014
854 52
EXPERIMENTS
4.1
POSTERIOR RECOVERY
We corrupt the MNIST dataset (Deng 2012) by introducing missing values and additive Gaussian noise on the observed entry. We then train both a missing value imputation VAE (MVAE), analogous to those presented in (Nazabal al.) 20188) and (Dalca et al. 2019). and our TAE model with the corrupted data sets. The VAE and TAE are constructed such that the structure of their posteriors q(xlg). i.c the functions mapping corrupted data to distributions of clean data at test time, exactly the same. this way, we can ensure that differences performance are due to the variational inference method employed and not the choice of posterior model. The resulting variational posteriors are used to perform data recovery from the corrupted samples. Fig. 3[a) shows examples of mean reconstruction and posterior draws. See analogous experiments for grouped missings in suppl. D.2.
We evaluate the accuracy of mean reconstruction at different ratios of observed entries by measuring the peak signal noise ratio (PSNR) between the ground truth data and mean recoveries (Figure 3(b)). To evaluate probabilistic performance we approximately measure the likelihood assigned by the recovered posteriors the ground truth data through a reconstruction ELBO, by training a new inference function with the clean ground truths, but leaving the posterior fixed, as is common for evaluating ELBOs in unsupervised settings (Cremer et al. 2018 Mattei & Frellsen, 2018 2019). A detailed description of this approach is given in supplementary C.3. We also carry out analogous experiments testing de-non de-noising and missing value imputation separately. THese results are reported in supplementary D.4 and D.5. Results shown in figure B(c). We further evaluate our TAE with Fashion- Fashion-MNIST - 28 x 28 gcey-scale images of clothing (Xiao et al. 2017). and the UCI HAR dataset, which consists of filtered accelerometer signals from mobile phones worn by different people
7

---

Published as a conference paper at ICLR 2021
Comrupt Data y
Draws from MVAE Posterior
Draws From TAE Posterior
Classification Probability
p(claselp)
MIVAE TAF
piolosels)
XIVAE TAE
Trolen
Figure 4: Propagating uncertainty classification task. Draws from the MVAE posterior are all very similar to each other. As result, the imputed images are almost always classified in the same way and the uncertainty of the task underestimated. The TAE posterior explores varied possible solutions to the recovery task. These can be recognised as different classes, resulting in less concentrated distributed probabilities that better reflect the associated uncertainty.
during common activities (Anguita et al. 2012). before, we test the recovery these data sets from version affected by missing values and additive noise. In addition to the MVAE baseline, compared against the recently proposed missing values importance weighted auto encoder (MIWAE) (Mattei & Frellsen 2019). which optimises importance weighted ELBO in place of the standard one. For each model and settings we compute the ELBO assigned to the ground truth data. Results shown in Table 2] Experimental details in Sec. C of suppl. mat.
4.2 DOWNSTREAM TASKS
To investigate the ad advantage of capturing complex uncertainties with our TAE model, we are interested in testing performance downstream tasks. We classification performance on subsets of the MNIST and Fashion-MNIST data sets, after recovery with our TAE. With both sets, we consider situations in which 10.000 examples are available, but corrupted with missing entries and noise. 1, 000 of these are labelled with one of 10 possible classes and we wish to classify the remaining 9, 000. To do so, we first train the TAE model on the full set, then use recovered posteriors to
generate multiple possible cleaned data for the labelled sub-set and use them to train classifier.
MNIST
Fashion-MNIST
0.9
0.8
TAE MVAE
fluttication Accaries
0.8
Raw
MVAR
Checifecatio Ancursey
0.7
0.7
0.0
0.6
0.5
0.4
0.2
01.5
0.8
0.9
0.2
3.5
0.9
Ratio of Missing Eatries
Ratio of Missing Entries
Figure 5: Classification accuracy after imputation. Classifying using TAE imputations gives advantage in this downstream task over using raw corrupted data and MVAE imputations, especially when the number of missing entries is high. This because the MVAE collapses on single imputations, while the TAE generates diverse samples for each corrupted observation. The TAE classifier trains with data augmentations consistent with observed corrupted images, instead of single estimates.
To perform classification on the 9, 000 remaining examples, we generate multiple possible cleaned data with the variational posteriors. Then, for each posterior sample, we perform classification and histogram the results. Examples are shown figure 4 To evaluate the performance, we take the class with the largest histogram the inferred one. We repeat this experiment for different ratios of
8

---

Published as a conference paper at ICLR 2021
missing values and several repetitions, varying the subsets of labelled and unlabelled data be used. Classification accuracy results are shown in figure
4.3 MISSING VALUES THE NYU DEPTII MAPS
As final practical application, we use convolutional version of our TAE perform structured missing value imputation depth maps of indoors rooms collected with Kinect depth sensor. Miss- ing entries are very common in depth maps recorded with such structured light sensors (Scharstein & Szeliski 2003). We raw depth data from the NYU rooms dataset, commonly used to test various computer vision systems (Silberman & Fergus 2011 Silberman et al. 2012 Dollár & Zitnick 2013) Chang et al. 2018). p) A large portion of the set avai lable only raw data, which presents missing entries. These are especially concentrated around objects' edges and reflecting surfaces, breaking the common assumption of missing at random, making this task particularly challenging. We train
TAE with a subset of this raw data set to perform imputation. Examples of results are shown in figure
Additional examples are shown in supplementary section D.6.
Observed Image
Modian Hittrrd
Recovered
Recovered
Recovered
Degth STD (m)
6.5
0.3
6.0
5.5
0.1
0.5
Depth STD (m)
6.0
83
0.1
5.2
Dapth STD (m)
0.0
Figure 6: Unsupervised missing value imputation with our TAE on raw depth maps from the NYU rooms data set, compared with median filter approach and the standard MVAE. Missing pixels in the observed images are in white. The median filter results overly smoothed images and is unable fill pixels that surrounded by large missing areas. The MVAE returns adequate reconstructions, however, over-fits to inaccurate solutions in certain locations, returning low uncertainty. The TAE returns good reconstructions and assigns high uncertainty to locations where reconstructions most inaccurate, as shown by the marginal standard deviations.
5 CONCLUSION
We presented tomographic auto-encoders; variational inference method for recovering posterior distributions of clean data from corrupted data set alone. We derive the reduced entropy condition method; a novel inference strategy that results in rich distributions of clean data given corrupted observations, thereby capturing the uncertainty of the task, while standard variational methods often collapse single answers. In experiments, we demonstrate this capability and show the advantage of capturing uncertainty with the TAE downstream tasks, outperforming the state-of-the. tiee-o-fthher- art VAE based recovery methods.
ACKNOWLEDGEMENTS
E.T. and R.M-S. acknowledge funding support from Amazon and EPSRC grants EP/M01326X/1, EP/T00097X/1 (QuantIC UK Quantum Technology Hub in Quantum Enhanced Imaging). R.M- S. acknowledges funding support from EP/R018634/1 Mlosed-Loop Data Science for Complex, Computationally and Intensiee Analytics).

---

Published as a conference paper at ICLR 2021
REFERENCES
Jonas Adler and Ozan Öktem. Deep Bayesian inversion. arXiv preprint arXiv:1811.05912 2018.
Samuel K. Ainsworth, Nicholas J. Foti, and Emily B. Fox. Disentangled VAE representations for multi-aspect and missing data, arXiv preprint 1806 2018
D. Anguita, A. Ghio, L. Oneto, X. Parra, and J. L. Reyes-Ortiz Human activity recognition on smartphones using multiclass hardware-friendly support vector machine. In innPro Proceedings the International Workshop on Ambient Assisted Living, pp. 216-223. Springer, 2012.
Rushil Anirudh, Jayaraman J. Thiagarajan, Bhavya Kailkhura, and Timo Bremer. An unsupervised
approach to solving inverse problems using generative adversarial networks. arXiv preprint arXiv:1805.07281 2018.
Angel Chang, Angela Dai, Thomas Allen Funkhouser, Maciej Halber, Matthias Niebner, Manolis Savva, Shuran Song, Andy Zeng, and Yinda Zhang. Matterport3D: Learning from RGB-D data in indoor environments. In 7th IEEE International Conference 3D Vision, 3DV 2017, pp. 667-676. Institute of Electrical and Electronics Engineers Inc., 2018.
Xu Chu, Ihab F. Ilyas, Sanjay Krishnan, and Jiannan Wang. Data cleaning: Overview and emerging challenges. In Proceedings of the 2016 International Conference on Management of Data, pp. 2201-2206. ACM, 2016.
Chris Cremer, Xuechen Li, and David Duvenaud. Inference suboptimality in variational autoencoders. In Proc. 35th Inter. Conference on Machine Learning, PMLR 80, 2018.
Antonia Creswell and Anil Anthony Bharath. Denoising adversarial autoencoders. IEEE Transactions on neural networks and learning systems, 30(4):968-99)) 994,,944- 2018.
Adrian V. Dalca, John Guttag, and Mert Sabuncu. Unsupervised data imputation via variational inference of deep subspaces. arXiv preprint arXiv: 0et::0903333333 2019.
Li Deng. The MNIST database of handwritten digit images for machine learning research [best of the web]. IEEE Signal Processing Magazine, 29(6):141-42,, 2012.
Emily Denton, Sam Gross, and Rob Fergus. Semi-supervised learning with context-conditional generative adversarial networks. arXiv preprint arXiv:1611.06430 2016.
Piotr Dollár and C. Lawrence Zitnick. Structured forests for fast edge detection. In Proceedings of the IEEE international conference on computer vision, pp. 1841-1848, 2013.
Marta Garnelo, Dan Rosenbaum, Christopher Maddison, Tiago Ramalho, David Saxton, Murray Shanahan, Yee Whye Teh, Danilo Jimenez Rezende, and SM Ali Eslami. Conditional neural processes. In ICML, 2018
Lucas L. Geyer, U. Joseph Schoepf, Felix G Meinel, John W. Nance Jr, Gorka Bastarrika, Jonathon A. Leipsic, Narinder S. Paul, Marco Rengo, Andrea Laghi, and Carlo N. De Cecco. State of the art: iterative CT reconstruction techniques. Radiology, 276(2):339-357, 2015.
Yedid Hoshen. Towards unsupervised single-channel blind source separation using adversarial pair unmix-and-remix. In ICASSP 2019-2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 3272-3276. IEEE, 2019.
Daniel Im Jiwoong Im, Sungjin Ahn, Roland Memisevic, and Yoshua Bengio. Denoising criterion for variational auto-encoding framework. In Thirty-First AAAI Conference on Artificial Intelligence, 2017.
Phillip Isola, Jun-Yan Zhu, Tinghui Zhou, and Alexei A. Efros. Image-to-image translation with conditional adversarial networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 1125-1134, 2017.
Hirokazu Kameoka, Li Li, Shota Inoue, and Shoji Makino. Semi-blind source separation with multichannel variational autoencoder. arXiv preprint arXiv: 00892, 2018.
10

---

Published as a conference paper at ICLR 2021
Durk P. Kingma, Shakir Mohamed, Danilo Jimenez Rezende, and Max Welling. Semi-supervise learning with deep generative models. In Advances in neural information processing systems, pp. 3581-3589, 2014.
Alexander Krull, Tim-Olive Buchholz, and Florian Jug. Noise2void-learming denoising from single noisy images. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 2129 2137, 2019a.
Alexander Krull, Tomas Vicar, and Florian Jug. Probabilistic Noise2 unsupervised content- aware denoising. arXiv preprint arXivv/9066.0 0066,0 2019b.
Sang Kyu Kwak and Jong Hae Kim. Statistical data preparation: management of missing values and outliers. Korean journal of anesthesiology, 70(4):407, 2017.
Jaakko Lehtinen, Jacob Munkberg, Jon Hasselgren, Samuli Laine, Tero Karras, Miik Aittala, and Timo Aila. Noise2noise: Learning image restoration without clean data. arXiv preprint arXiv: 189, 2018.
Steven Cheng Cheng-Xian with generative adversarial networks. arXiv preprint arXiv:1902.09599 2019.
Xian Li, Bo Jiang, and Benjamin Marlin. Misgan: Learning from incomplete data
Yonghong Luo, Xiangrui Cai, Ying Zhang, Jun Xu, et al. Multivariate time series imputation with generative adversarial networks. In Advances in Neural Information Processing Systems, pp. 1596-1607, 2018.
Chao Ma, Sebastian Tschiatschek, Konstantina Palla, José Miguel Hernández-Lobato. Sebastian Nowozin, and Cheng Zhang. Eddi: Efficient dynamic discovery of high-value information with partial vae. arXiv preprint arXiv:/800 / 142, 2018.
Alerre-lee Mattei and Jes Frellsen. Leveraging the exact likelihood deep latent variable models. In Advances in Neural Information Processing Systems, pp. 3855-3866, 2018.
Pierre-Alexandre Mattei and Jes Frellsen. MIWAE: deep generative modelling and imputation of incomplete data sets. In International Conference on Machine Learning, pp. 1413-4423, 2019.
Mchdi Mirza and Simon Osindero. Conditional generative adversarial nets. arXiv preprint
arXiv:/411 1784, 2014.
M. Arthur Munson. A study the importance of and time spent different modeling steps. ACM SIGKDD Explorations Newsletter, 13(2):65-71, 2012.
Alfredo Nazabal, Pablo M. Olmos, Zoubin Ghahramani, and Isabel Valera. Handling incomplete heterogeneous data using VAEs. arXiv preprint 187.00380. 2018.
Anh Nguyen, Jeff Clune, Yoshua Bengio, Alexey Dosovitskiy, and Jason Yosinski. Plug & play generative networks: Conditional iterative generation images in latent space. In CVPR, volume pp. 7, 2017.
Niki Parmar, Ashish Vaswani, Jakob Uszkoreit, Lukasz Kaiser, Noam Shazeer, Alexander Ku, and Dustin Tran. Image transformer. arXiv preprint arXiv:1802.05740 2018.
Mary Phuong, Max Welling, Nate Kushman, Ryota Tomioka, and Sebastian Nowozin. The mutual autoencoder: Controlling information in latent code representations. In ICLR, 2018.
Daniel Scharstein and Richard Szeliski. High-accuracy stereo depth maps using structured light. In 2003 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2003. Proceedings., volume pp. I-I. IEEE, 2003.
Chao Shang, Aaron Palmer, Jiangwen Sun, Ko-S Shin Chen, Jin Lu. and Jinbo Bi. VIGAN: Missing view imputation with generative adversarial networks. In 2017 IEEE International Conference on Big Data (Big Data), pp. 766-775. IEEE, 2017.
N. Silberman and R. Fergus. Indoor scene segmentation using structured light sensor. In Proceedings of the International Conference on Computer Vision Workshop on 3D Representation and Recognition, 2011.
11

---

Published as a conference paper at ICLR 2021
Nathan Silberman, Derek Hoiem, Pushmeet Kohli, and Rob Fergus. Indoor segmentation and support inference from RGB-D images. In European conference on computer vision, pp. 746-760 Springer, 2012.
Kihyuk Sohn. Honglak Lee, and Xinchen Yan. Learning structured output representation using deep conditional generative models. In Advances in Neural Information Processing Systems, pp. 3483-3491, 2015.
Michalis Titsias and Francisco Ruiz. Unbiased implicit variational inference. In The 22nd International Conference Artificial Intelligence and Statistics, pp. 167-176, 2019.
Francesco Tonolini, Jack Radford, Alex Turpin, Daniele Faccio, and Roderick Murray-Smith. Vari- ational inference for computational imaging inverse problems. Journal of Machine Learning Research, 21(179):1-46, 2020.
Ian R. White, Patrick Royston, and Angela M Wood. Multiple imputation using chained equations: issues and guidance for practice. Statistics in medicine, 30(4):377-399, 2011.
H. Xiao, K. Rasul, and R. Vollgraf. Fashion-MNIST: a novel image dataset for benchmarking machine learning algorithms. 1708. 2017.
2017.
Mingzhang Yin and Mingyuan Zhou. Semi-implicit variational inference. In International Conference Machine Learning, pp. 5660-5669, 2018.
Jinsung Yoon, James Jordon, and Mihaela Van Der Schaar. Gain: Missing data imputation using generative adversarial nets. arXiv preprint arXiv:1806.0299 0000002000002, 2018.
Willard I. Zangwill. Non-linear programming via penalty functions. Management science, 13(5): 344-358, 1967.
Lina Zhou, Shimei Pan, Jianwu Wang, and Athanasios Vasilakos. Machine learning on big data: Opportunities and challenges. Neurocomputing, 237:350-361, 2017.
12

---

Published as a conference paper at ICLR 2021
Tomographic Auto-Encoder Supplementary material
A PROOFS AND DERIVATIONS
A.1 DERIVATION OF VAE ELBO FOR DATA RECOVERY
We aim to maximise the log likelihood of the observed corrupted data y
log log / [ p=2)0(=)=)dd p(y)))))
p(x)
We introduce variational distribution in both clean data space and latent space g(v, zly) and define a lower hound as
logp(y) 2 J. ((x, =ly) log (x, =ly)
p(z)p(zlz)dz dzdæ
(8)
/ =lv) =ly) log plgg(y)))dda
To obtain the VAE ELBO used data recovery settings, the choice of the variational posterior ((x, =ly) = (<=])))(a(z) The ELBO can then be simplified to give
p(=)p(x|a)dz dzdr (zlu)p(<1zz)
logp(y) 2
((<1))p(z)))
logg(l/)))
p(alz)dr (=(1)) log p(z)dz dz
(9)
((>ly)
/ (o log plyl
For situations in which the observations' likelihood f p(x ((yp))(()d)))))))))))) has a closed form, such as additive noise and missing entries, we can define tighter bound to the likelihood by moving the integral in æ in the second term inside the logarithm
(<>ly) log p(z)dz dz ((>1y)
logp(v) p(v)
+ (>ly)) log /
(10)
dz
(L(g(=ll)) (x))) (gtt) log p(y|z)dæ.
Because p(z|=) simplifies in the KL term, this ELBO avoids variational inference in the space of clean data æ.
A.2 DERIVATION OF TAE ELBO
In TAE model defined separate LVMs for prior and posterior. To distinguish between the posterior latent variable and the prior latent variable, we name the former and the latter žp. The likelihood we aim maximise is
log p(y) = log [
p(zp)p(z0) (dz (d)..
(11)
p(x)
13

---

Published as a conference paper at ICLR 2021
Similarly to the VAE ELBO case, define variational posterior 2x1)))11))))))) to find lower bound
logp(y) 2 J. g(0, =plu) log p(2p)p(<)) ddz, dzpdæ ((0, zply)
(12)
+ / g(x, zplu) log p dz,dr.
However, in our model we do make the assumption that the variational posterior has the special form described in section A.1 and instead set it to have the form q(z, <plv) ((<l)))((<p)) separating posterior inference from observations y to clean data and inference of prior latent variables Zp. The resulting lower bound is
logp(y) 2 J. ((xl)))((p)))) log p(zp)p(al(z)) dzpda + Jol ((>l)))(()))))) log log(yla)dddd ((<lv))(Zppo))
=s ((xlv) / ((<pl>) log pl )p(p) zp)
dzp dx + Jal ((pplz) dzpg(aly) log logllu)>)d
(<pp|z)
J a(zpl=)dzz (()l))) log (()l))da
Eg v) [Eq(zpl=) (ogp(<|zp) (L(((=)l))ll((x)))) + (a(( logp(u/>) + (g(xly))
(13)
A.3 PROOF OF THROREM 1
(<1[x,)) Bō(z g(x,4))
g y) g(< ((<1(1,))
0, V0,
((=ly)
<(aly) q(=1l1)
g<<|2,8) g(zlk',)) = 0, V0,
(14)
q(rl)) q(zl))
q<>2, ()](=l2/ = 0, V2,
<<r|=', u) = 0, Vr q( 2, n) #
Using the result of equation 14 as the following:
derive the form of the entropy H(g(x)y) for this special case
H(g(xlv)) /. T/ (2) log S. q(2 a(='lu)g(ale) w)de dx
[ ((=lo)a())))))))) log I (('l)))((a)2' ())d
((21l))
dadæ
3/40
eg
(15)
((= ())(=> log (a(==l/)((<))))).> v)) drdz
((21))((()) 2) log f ((zl))a(<12) v) log q(0|2, y)drdz
J (((=l)) log (og((=/)dd ((>lv) /. g(012, v) log ql0|2, y)dd
((((l))) + E H(g(z)2,)).
14

---

Published as a conference paper at ICLR 2021
A.4 PROOF OF THE EQUIVALENCE BETWEEN CONDITIONS
proof of necessary condition:
Bg(e,=ly) log g(2(x,y)
(<>ly) ((rl2,)) log g(2(2, drd=
((=ly)
g(=ly)
(212, v) log g(=|2,)) v) dedæ
((11))
(==ly)
((<1))
(<|,, v) log (22| æ dddd
[. ((xll))
(</|,, v) log (=lly)dedr
/ ((<lv)) (<2|>, v) log (212, y)dzdr
(16)
q(x|2 y)dx / (<<|)) log g( k2)d
B((ll) [ ((2 v) log ((2|,, v)dz
H(g(slew))
(<>lv) log dzll)dd2.
-(((tl)))
If the above expression tends infinity, either 11 (1(22x v)) -o (q{()ly) + co, meaning that either q(2|x, 3) - Delta function, or q(2l)) - uniform. Either condition implies s(a|z.v)
((2lu)
Bo(= g(x, v))
proof of sufficient condition:
[g(,)) log g(2|x, y)
q(<l)) / q<>|,, v) log g(2|2, v)
dadæ g(>1y)
((=ly)
(17)
/ u(>l)) f ((=l)) (<= g(2|=. v)
log /<2/,, v) dada.
((=ly)
((=lg)
Now we set g(sla)) ((2l9)
B8(= g(x,
/ ((>lv) / q(elw) - g(r, u)) log Bö(z g(x, dzdd
/ ((cl)))(((z.<)))) log B 8(g(2, v) g(z, u)) dx.
(18)
+00, Var
B6(2 g(2,, v)) is sufficient condition for Ex(a,,)yy) log g(ala.g) Therefore, ((*l))
g(slw.))
a(*1x) +
15

---

Published as a conference paper at ICLR 2021
A.5
PROOF OF EQUATION 6
E (y(x,)lv) log (<1(2,8)
g(a,, = zly) log y)dada
g(=ly)
a(ar= = log (
a(alv) /. (< y) log (<]|,, y)dzdr
g(2,2 2 log ((=l))dddz
(19)
> /. a(xlv) /. ([2|2, v) log r(z|o, y)dzdr
f g(2, ( log q(= dzde
=E ((, log r(=(,,))
g(22g)
Where the inequality derives from the positivity of the KL divergence (L(g(z)2, ())rr()),,)))
B
ALGORITHM
B. DETAILS TRAINING
As detailed in section 3.2, to train our variational posterior q(rly), we maximise through gradient ascent the TAE ELBO with the reduced entropy penalty function
arg max
Eg(r)p) log x)) + Ea(= v) (2g())))) log (0gp(()))))))))))))))))))))))p KK(g(E,
Prior
ELBO, 2x(x)
(20)
+ (q(2lb)) + Eg(zly) H (g(2) v)) (g((a))))) log r(2/2, v)
0
(<<1y)
All expectations in the above expression are computed and optimised by sampling the corresponding conditional distributions using the re- parametrisation trick characteristic of VAEs.
p( )p(2) ddpp is training entirely with samples from the posterior LVM, which is also training, the model can easily obtain high values for the prior ELBO by generating collapsed samples x with the posterior and get stuck in an unfavourable local minimum. TO avoid this, we employ a warm up strategy. We define positive parameter y that multiplies the expectation of the prior ELBO and the entropy Hxx
Because the prior LVM p(z)
arg max
By(rll) log pgg(()))) + BBg(l))) Be(xplz) log pogp((aa zp) K (L(g(pl=))l((zp))
Prior ELBO, 2p(x)
(21)
(gtt,=)p)) log r(=(=r,)) C
+ H(g(zlv)) + ( H (g(x12,8))
q(21y)
The value of y is initially set to zero. After a set number iterations it is linearly increased reach one and kept constant for the remaining training iterations.
B.2 COMPLETE OBJECTIVE FUNCTION
Observation Parameters: the general case, the corruption process p(y[z), mapping clean data degraded samples is controlled by parameters that differ from sample to sample. We can distinguish these into observed parameters & and unobserved parameters B. For example, in the of missing values and noise, the indexes of missing entries in each sample often observed parameters, while the noise level is an unobserved parameter. The complete form of the corruption likelihood for clean sample 2; is then plyla 0i Br).
16

---

Published as a conference paper at ICLR 2021
Objective Function:
With the parameters conditionals deseribed in subsection 3.2.4 and explicitly showing the parameters to be optimised, the objective function we maximise is the following
(l log logp(y/z, a, arg max
0,0
+ (u [B (=)))) log log K Kolgossollla
+ H (a (zly, a)) + yB (#lv.a) H (g0z (212.
(22)
ro. (2(2))
(x. log
C
To1 (z)y,
(0(2, B) =
1 qs (21 a) (z) y, aos (812,
(o(xly)a)
J_ 4o, (zly, gg (z]2,, y, a.)z, (z(2, xly. a) = le (zly, (o (zlz, a), = {dis} are
the parameters of the inference models and are the parameter of the prior model.
B.3
PSEUDO-CODE
B.4 POSTERIOR COLLAPSE IN VARIATIONAL INFERENCE
The posterior collapse problem we approach with the reduced entropy condition method presented in this paper has some analogy with the latent posterior collapse encountered when using implicit distributions in variational inference to obtain fiexible recognition models. The main issue in training these models successfully without collapse is the computation of density rations between the latent prior and the implicit variational posterior. This problem is analogous to the difficulty in estimating the LVM entropy in method. Yin & Zhou (018) proposed to a further lower bound on the ELBO and add a term encouraging diversity to avoid collapse. This term is obtained by drawing K Gaussian components from the LVM posterior and computing the KL divergence of an individual component with the mixture distribution, i.e. the sum of the drawn Gaussians, Titsias & Ruiz (2019) build on this work by deriving an unbiased estimator for the ELBO gradient, instead of using surrogate lower bound.
These methods rely on estimating the LVM posterior through sampling and aggregating explicit distribution components. This was demonstrated to work well for posteriors in artificial latent space by drawing only a few components. However, in our data recovery setting, we need to capture the posteriors in clean data space, rather than latent space. Posteriors capturing the uncertainty in natural data are expected to be much more complex and higher dimensional, leading to the number K of drawn Gaussians needed to approximate the true LVM with these methods to become rather large, making optimisation inefficient or even intractable in extreme cases. The reduced entropy condition method we derive in this paper avoids the posterior collapse without having estimate the LVM posterior through sampling is therefore specially suited for the data recovery setting, where we are required to capture posteriors in clean data space.
C EXPERIMENTAL DETAILS
C.I MODELS' ARCHITECTURES
In all experiments we carry out comparing our TAE with competitive methods, we make the in- dependence assumption q<<|z,)) v) = g<x|=), consequentially making r(>|x; jr, y) r(z|z). In this way, the reconstruction posterior LVMs (<rl)) we compare between TAE, MVAE and MIWAE all present identical structure and differences performance are result of the model constructed to train them alone. However, we note that, unlike the two competing method, the TAE is not formally limited this choice and can infer conditionals q(rl2,, y) the general case. We hereafter detail the architecture used for all quantitative experiments of section 4.1 and 4.2.
MVAE and MIWAE models: The MVAE model is built as an LVM having a unit Gaus- sian prior in the latent space p(=) = =N(2;; 0, 1) and isotropic Gaussian clean data likelihood p(z|2) = N(x;; Mu, of), where the moments Hæ and 0² are outputs of a neural network having as input the latent variables 2. Because only observe corrupted data y, rather than clean data x, the recognition model q(2ly) conditioned on observed corrupted data y and also has the form of an isotropic Gaussian ((=ly) (2; (2z;; o?), where the moments fl and o² outputs of
17

---

Published as a conference paper at ICLR 2021
Algorithm 1 Training the TAE Model
Inputs: Corrupted observations Y [vi:N]: }: Observed Parameters A {a1.N} initial model parameters, {0(D) "60)) ): user-defined posterior latent dimensionality, J; user-defined prior latent dimensionality. Jp: user-defined condition strength X; user-defined condition parameter C; user- defined latent prior p(pp); user-defined initial warm-up coefficient To: user-defined final warm-up coefficient "y, warm-up start Nwo; warm-up end Nwf; user-defined number of iterations, Niter.
1: y (R=0) + %
2: for the k k'th iteration in [0 Niter 1]
3: for the i'th observation
4:
2, 9,,00 aq)
5:
24 9000) (z Vis Q4)
6:
B
~ 4 (3) 24, y2. ass)
7:
pp 40g (zp (ia)
((() - p(yi Ti, 33)
8:
p(A) +- log Pgra ( (p())
9:
K (k) DKL (g ( (zpl=s))lp(<p)
10:
Hz, (k) + H Hl (k) (2|3kt az))
11:
Hx (k) H (q (1) (xl 21 V/s a:))
13:
R log (2) I1. V4, a.)
14:
- log 40 (zilyi,, a:)
15:
end
16:
F(1)_ = Ei
E + +x(k) (kk)= k!²) + Hx [0)
(k) -d)
17:
+Hz
18:
1(++1), ((++)) arg max
19:
if > Nuo and & < Nosf
20:
7 [k+1) + 700) + (78 70)/ (N Nww)
21:
else
22:
7 (k+1) 700
23:
end
24: end
neural network having as input the corrupted observations y. The corrupt data likelihood p(y|z) is obtained by simply selecting the likelihood p(z|2) over the observed indexes, i.e. for missing values corruption the integral ply 2) p(x) =))((() da simply masks out the unobserved entries. The model is then trained by maximising the ELBO of equation [ The MIWAE is built with the same structure, but instead of optimising the MVAE ELBO of equation T an importance weighted lower bound is maximised, described in Mattei & Frellsen 2019). The precise architectures used for the neural networks are described for the different experiments throughout the rest of this section. One important point notice that, in each experiments, the structures of p(æ|2) and 7(>ly) are chosen such that the resulting reconstruction model after training, i.e. the model taking as input test corrupt observation y and generating clean samples is identical for the TAE and the two tested competing models. That is qg(2)1p (=())((()=))d for the MVAE and MIWAE have identical structure to J (((z)))q(()2)))) for the TAE. this way, performance differences can be attributed solely to the difference in inference strategy and not reconstruction model's 's capacity.
18

---

Published as a conference paper at ICLR 2021
Posteriors structure: The posterior parametric components 9o, (zly, a) and goz (z]22) (Poa (2]=) in the case of the MVAE and MIWAE). 9o, (zly, a)) consists in a fully connected two layers neural network with leaky ReLu non-lincitt non-lincarities, taking as input concatenated corrupted observations y and binary mask that labels the missing entries a and returning as output a vector of latent means and a vector of latent log variances. The two intermediate deterministic layers have 400 hidden units, while the latent space 2 is 20-dimensional.
goz (z]]z), and P+z (2|2z) in the case of the MVAE and MIWAE, are similarly constructed, consisting in fully connected two layers neural network with leaky ReLu non-linearities, taking as input latent variables 2 and returning vector of means and vector of log variances of clean samples z. The two intermediate deterministic layers have 400 hidden units.
The TAE prior encoder qds (z)la)) has the same general structure as the posterior encoder, with two fully connected layers and leaky ReLu non-linearities, taking as input generated clean data x and returning as outputs vector of latent means and vector of latent log variances for the prior latent variable 2p. As this model has less capacity than the posterior LVM, the two deterministic hidden layers have 50 hidden units each and the latent variables Ep are 5-dimensional.
TAE Prior LVM Structure:
po(2)zp)) is similarly constructed, consisting in a fully connected two layers neural network with leaky ReLu non-linearities, taking input latent variables 2p and returning vector of means and vector of log variances of clean samples æ. The two intermediate deterministic layers have 50 hidden units.
Approximate Latent Posterior Structure: The approximate latent posterior r(2[z) has the same structure as the posterior encoder, consisting in fully connected two layers neural network with leaky ReLu non-linearities, taking as input generated clean data x and returning as outputs a vector of latent means and vector of latent log variances. The two intermediate deterministic layers have 400 hidden units.
Convolutional TAE Structure: For the imputation of NYU missing data we use convolutional conditionals in our model, instead of fully connected ones. In this version. we do not make the independence assumption, using q(>|2, 3) and r(2|z, y). q(2 y, a) takes concatenated y and a and passes them through 4 recurrent convolutional layers with filters of size 3 x 3 and 5 channels, each time wwwnssamllnn by two. the last layer is mapped to means and standard deviation of latent images 2, which are 1 1/32 of the original size in each axis and have 10 channels, through two convolutional filter banks with strieds x 1. (]<|2, y, a)) is built to mirror this structure, with the addition of accepting inputs from y and a. Three recurrent transpose convolutional layers with 3 x 3 filters, 5 channels and 2 x 2 upsampling each map Z to a deterministic layer with 1/2 x 1/2 of the original images size. concatenated y and a are mapped to the same size with single convolutional layer, downsampling it by 1/2 x 1/2 and 5 channels. The two are concatenated and the resulting layer is finally upsampled to inferred clean image x by last convolution with filter bank. All non-linearities are Elu.
The prior networks are built in similar way, but with shallower structures to give less capacity. (<2p (x:) passes x through 2 convolutional layers, each with down-sampling of 4 x 4 and 5 channels. as before, mens and standard deviations of latent images zp are generated from this last layers with 2 x 2 down-sampling and, in this case, 5 channels. The prior generator p(z[zz) is built to exactly mirror this structure. r(2 (r, y. a)) has the same structure as (2ly,, a),, with the only difference being that it accepts as input concatenated x, y and a.
C.2 EXPERIMENTAL CONDITIONS
All posterior recovery experiments, with each of the three data sets tested, performed samples that have been re-scaled from 0 to 1. In all cases, the sets are injected with additive Gaussian noise having standard deviation 0.1. Subsequently, random binary masks are generated to block out some entries, resulting in missing values. The proportion of missing entries in the masks was set as described in the main body in each case.
Posterior Recovery:
Experiments were repeated with generated binary masks 5 times. The means and error bars shown in figure and the uncertainty reported in table were computed from these. The MIWAE was trained with 20 weights per sample. After training, all posteriors g(>ly) have identical structure and
19

---

Published as a conference paper at ICLR 2021
tested in the same way, by training inference network on the test set to compute the ELBO values.
Classification Experiments: The TAE models for the MNIST and Eashion-MNISS experiments were trained in the conditions described above. In each case, a random subset of 10, 000 samples is taken from the corrupted set and the TAE and MVAE models are trained with it. A random subset of 1, 000 of these is selected and ground-truth lables for these samples are made available.
A classifier consisting in single fully connected layer with leaky ReLu non-linearity is trained to perform classification on this subset. For each stochastic training iteration of this classifier, we generate samples associated with the corrupted observations and provide the associated labels. After the classifier is trained. we test classification performance on the remaining 9, 000 examples, by running the train classifier 400 times per sample, each time generating clean data from corrupted observation with the TAE and the MVAE. The histograms shown in figure 5 are built by aggregating the resulting classification.
The above procedure is repeated 15 times. The resulting means and standard deviations of the tested classification performance are shown in figure
Training Conditions: Plyper-parameters of optimisation for the models were cross validated with the MNIST data set at a proportion of missing entries of 0.9. Hyper-parameters common to all models were determined by obtaining best performance with the MVAE model. Hyper- parameters specific to the TAE model were obtained by fixing the common parameters and cross validating these. The resulting optimal hyper parameters were then used in all other experiments of section 4.1 and 4.2, including those with different data sets. Common parameters are as follows: 500, 000 iterations with the ADAM optimiser in Tensorflow, initial training of 2 and batch size of 20. The hyper-parameters specific to the TAE are instead: initially set to 0.01 and then linearly increased to between 50, 000 and 100, 000 iterations, À and C 10. All experiments were performed using& a TitanX GPU.
NYU Rooms Experiments: For these experiments, take a subset of 3612 depth maps from the NYU raw data set. We slightly crop these in one dimension to be 480 x 608 images. The convolutional TAE and MVAE to obtain the results of figure 7, were trained for 100, 000 iterations using the ADAM optimiser in Tensorflow, with a batch size of 20 images and an initial training rate of 2 x 10 10-2, For the warm up. we initially set y 0.01 and linearly increase it to 1 between 10, 000 and 20, 000 iterations. For these experiments, À = 2 and C = 15.
C.3 EVALUATION ELBO
To evaluate the probabilistic performance of our method compared to others, we compute an evaluation ELBO which relies test ground truths. After each model is trained unsupervisedly, a posterior of the form g( (y) = <<>>))<(<<)>)) p(a|2). Given the test set of paired clean and corrupted samples s, and yt, we construct new parametric recognition model, which encodes latent distributions from ground-truths gn (z(2))) We then optimise the following:
obtain where for the MVAE and MIWAF, g(x|2) =
arg max E (x,()))) log g(2,12) + (L(qq
(23)
The above is a conditional VAE ELBO with conditional prior (=ll)) and is a lower bound to the test likelihood we are interested in (Bxl) Note that we optimise over only, therefore the new recognition model ((2(z) is the only one which is affected by this optimisation and the components of our reconstruction model g(((y) and g<x|=) remain the same as trained with the unsupervised training set. As a result, this new optimisation only tightens the bound, rather than maximising the likelihood, which we want evaluate trained previously.
D ADDITIONAL EXPERIMENTS
D.1 C' AND À CROSS- VALIDATIO
C' and À in equation 22 are hyper- parameters of our inference algorithm and need to be user defined. In our experiments, determine the optimal values by cross-validation, described in section C.
20

---

Published as a conference paper at ICLR 2021
We report in figure 7 cross validation study where measure the TAE ELBO for MNIST with 90% missing values and additive noise.
Cross Validation over C and X
2
1400
5
1200
e
10
3
20
40
400
0.5
1
2
4
8
16
32
64
X
Model Collapsed
ELBO with VAE
Figure 7: ELBO for MNIST with 90% missing values and additive noise as a function of chosen hyper-parameters and (in log scale). The performance of TAE exceeds that of a standard VAE approach over broad range of values. If the values are too large, the model collapses during optimisation, making such situation easy to diagnose.
As shown in figure the performance of TAEs is robust to variations in hyper- parameters C' and À over a broad range of values. They also have an intuitive meaning that helps in their selection. In practice, C controls the final value of localisation and is desirable to be as high as stability of the optimisation allows. controls how fast we are imposing the model to approach C.
D.2 STRUCTURED MISSINGS
We test TAE in situation analogous to that shown in figure 4 of section 4, but with structured missing values instead of randomly missing ones. For each sample in MNIST, we only make visible small 10 10 pixels window, randomly placed each example, while the rest of the images remain hidden. In addition, the values in the observed window are subject to additive Gaussian noise, similarly to the dissing-at-random case. Reconstructions with the comparative MVAE and our TAE are shown in B
Similarly to the missing-at-randoon case, the MVAE collapses on single solutions, giving draws from the posterior that are all very similar to each other. Contrarily, the TAE gives more variation in the possible solutions exploring more appropriately the uncertainty the solution space. The MVAE ELBO over the clean data for this problem is 428, while the TAE one is 638. The performance improvement provided by the TAE is analogous to that observed with missing at at-random experiments.
21

---

Published as a conference paper at ICLR 2021
Ground
Corrupt Data
Mean Inf.
Truths
Draws from Posterior
1
2
2
2
7
2
MVAE
0
0
0
0
0
4
4
4
4
4
7
7
7
/
TAE
O
0
0
0
O
4
4
9
9
9
Figure 8: Examples of Bayesian reconstructions with MVAE and TAE on structured missing values. the MVAE returns good mean reconstructions, but its posteriors collapse on single solutions, giving draws that are very similar to each other. The TAE returns posteriors which more broadly explore the different possible clean samples associated with the corrupted observations, giving more variation in the posterior's 's draws.
D.3 MORE ELBO EVALUATIONS
Table 2: ELBO assigned by the retrieved posteriors to the ground truth clean data.
MVAE
MIWAE
TAE 1831 + 8
MNIST,
883 +
940 +
20% missing
MNIST,
870 + 6
917 + 4
1719 + 7
50% missing
MNIST,
803 + 15
780 =
1536 + 14
80% missing
Aoshion-MNISTT 20% missing
775 +
815 + 4
1407 + 24
Fashion-MN MNIST, 50% missing
757
+
1326
723 +
766 -
1094 13
80% missing
UCIHAR,
611 + 3
628 + 10
1039 + 11
20% missing
UCI HAR.
585 + 4
613 + 6
1014 + 6
50% missing
UCI HAR,
471
+ 10
584 + 8
854 52
80% missing
D.4 IMPUTATION WITHOUT NOISE
We carry out experiments on MNIST analogous to those shown in figure 3 but in the absence of noise, in order to test performance imputation alone. Each tested ratio of observed entries is
22

---

Published as a conference paper at ICLR 2021
repeated three times, re-generating the patterns of missings each repeat in order to obtain error bars. Results are shown in figure I0
(b) Mean Inference
(c)
Probabilistic
Performance
Performance
MVAE ... TAE
MVAE ... TAE
2550
2600
PSNRi
ELBO
2450
2400
17
Ratio Observed Entries
Ratio Observed Entries
Figure 9: Missing value imputation performance on MNIST in the absence of noise. As in the noisy case, the PSNR values between the MVAE and the TAE very similar. The TAE presents significantly superior ELBO values at low ratios of observed entries, but in this case, the gap is reduced as more entries are observed. This is because in the noiseless case, the solution space when most entries are observed is much more localised than in the noisy case, and therefore the MVAE collapsed posteriors do not fail as much capture it.
D.5 DE-NOISING
We carry out experiments MNIST analogous to those shown in figure 3 but testing fully observed images at different levels noise. Each tested ratio of observed entries is repeated three times, re-generating the patterns of missings each repeat in order to obtain error bars. Results are shown in figure
(b) Mean Inference Performance
(c) Probabillstic
Performance
MVAE ... TAF
MVAE
TAE
2500
2000
(S)
22.0
ELBO
1500
21.8
1000
21.6
21.4
500
0.66
0.00
0.10
0.22
0.14
0.16
0.18
0.20
0.06
0.00
0.10
0.12
0.14
0.16
0.10
0.20
Noise Standard Deviation
Noise Standard Deviation
Figure 10: De-noising performance on MNIST. in the missing value imputtion case, the MV and 7 AE perform very similarly in their mean reconstructions, but the TAE presents significantly better performance in capturing the distributions of clean solutions, as the test ELBO values are higher.
23

---

Published as a conference paper at ICLR 2021
D.6 NYU ROOMS RECOVERY EXAMPLES
Image
Median Fitteeed
MVAE Recovered
MNAE
Depth
1 1: t Dapth/m)
STO
5
(m))
Dejtth [m)
Degith [mm)
Figure 11: Unsupervised missing value imputation with our TAE raw depth maps from the NYU rooms data set, compared with a median filter approach and the standard MVAE. Missing pixels in the observed images in white. The median filter results in overly smoothed images and is unable to fill pixels that are surrounded by large missing areas. The MVAE returns adequate reconstructions, however, over-fits in certain locations and its uncertainty is largely over-estimated. The TAE returns good reconstructions and assigns high uncertainty to locations where reconstruction is most inaccurate, shown by the marginal standard deviations.
24


Reviews:
[{"review_id": "YNsZ87I9iKV", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Practical datasets often come with corruptions, such as missing items or noisy observations, thus needs models enable to recover the corrupted data automatically. This paper presents the tomographic auto-encoder (TVAE), which conducts inference over the data space $x$. Because the prior regularization acts over the data space, TVAE is enforced to generate diverse samples from the corrupted observations. Empirically, the paper demonstrates that TVAE can indeed generate diverse samples and can achieve superior test ELBO compared to the previous baselines.\n\nMissing data imputation is one important problem in machine learning, given the imperfection of the practical data. Specifically, TVAE focuses on two properties: 1) unsupervised learning, which resolves the intractability of labelling large datasets 2) diverse recoveries, which attempts to generate multiple possible samples instead of collapsing onto one possibility. Towards these two properties, TVAE proposes to conduct inference over the data space $x$, whose prior directly prevents from collapsing.  To resolve the intractability of the entropy $H(q(x|y))$, TVAE identifies the *reduced entropy condition* and transforms the intractable ELBO maximization into a constrained optimization problem. The resulting model demonstrates superior performances compared to baselines, in terms of generating diverse samples. I think this work makes an important contribution.\n\nThis paper is well written.\n\nConditional neural process (Garnelo et. al., 2018a) and neural process (Garnelo et. al., 2018b) are another stream of models for missing data imputation, which are missed by the paper.\n\nRegarding the reduced entropy condition, though it ensures that the entropy is decomposed into more tractable forms, will enforcing this condition limit the expressiveness of the encoder $q(x|  y)$ ?", "extra_scores": {}, "extra_text": {}}, {"review_id": "SzkLvzTD-HO", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Review:\n\nThis paper proposes a novel approach to handle the recovery of dirty data in fully unsupervised scenarios. The corrupted data considers both missing data and noisy samples. They derive a VAE model with a novel reduced entropy condition inference method that results in richer posteriors. This is a very challenging problem, since the model cannot use clean examples as part of their training procedure.\n\nQuestions/Comments:\n\n- It feels to me that the main focus of this paper is on missing data rather than in other types of corruption. I get this impression mainly from the experimental section and methods used for comparison, well known for handling missing data. Something I would have appreciated in this work is to observe the performance of the authors method in scenarios handling missing data and corrupted data separately. Nonetheless, it is interesting to see experiments with both effects combined, which is not so common in the literature.\n\n- Would it be possible to get results beyond maximizing the ELBO? The ELBO contains the reconstruction term of the images, but also many other terms. In the end, in a missing data imputation model, it is interesting to get an idea of how good the reconstruction of the images is. I can see from figures 3 and 4 that they should be good, but having a different metric might be helpful. I find it surprising that in Table 1 the ELBO of TAE is almost double of the other methods, while the average reconstruction of MVAE for example does not look that much different in figure 3 from the ground truth, and it is pretty similar to TAE.\n\nSummary:\n\nThe paper is well written and the idea is novel as far as I know. The notation is clear and the proofs in the appendix look sensible. The experimental section showcases several scenarios where they compare to unsupervised generative models to handle missing data. The analysis of corrupted data (outside of missing data) seems a bit lacking.", "extra_scores": {}, "extra_text": {}}, {"review_id": "AHvRLtRYqFf", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes a Tomographic auto-encoder (TAE) for unsupervised recovery of corrupted data. More specifically, TAE takes a Bayesian approach to recover the posterior distribution of a clean image conditioned on an observed corrupted image and thus effectively modeling uncertainty in data recovery. The paper argues that a naive application of VAE is not effective due to the latent variable collapse, and proposes an alternative model where hierarchical latent variable models are used for both prior and variational posterior. Some tricks are introduced to facilitate the stochastic gradient variational inference. \n\nI think the paper is tackling an important problem and I advocate the use of a VAE-like model for uncertainty modeling. The paper is clearly written with helpful figures. The experimental results, at least compared to the baseline (MVAE), looks promising. I like the way the methods are compared using the downstream task.\n\nHowever, I'm not sure whether a new model should be developed besides the existing approaches. The authors state that the problem of a vanilla VAE with hierarchical latent structure (latent code $z$ - clean image $x$) is that it is prone to latent variable collapse.  This is true, but there are plenty of existing works (partially) resolving this problem. The most relevant approach I can think of is the semi-implicit variational inference [1] for which a hierarchical latent variable model is used for the variational distribution similar to the setting considered in this paper. [1] proposes a theoretically guaranteed solution to prevent a trivial case where the lower-level latent variable ($z$) completely collapses into a point mass, and I think this can directly be applied for the problem considered in this paper. Also, as [2] pointed out that one can consider using a more expressive prior distribution for latent code $z$ to combat latent variable collapse. For instance, flow-based models may be employed for both prior and variational posterior. The proposed design also makes sense, but I don't think they are more expressive than the models listed above. If that is the case, there should be a special factor making TAE specifically well-suited for the problem at hand - the recovery of corrupted images - but I failed to find such a thing. Is there any reason that the existing models other than the vanilla VAE cannot be considered?\n\nAlso, the baseline (MVAE) is not clearly described, so it is not clear how MVAE was actually implemented. I recommend giving a more detailed description of the baseline at least in the appendix.\n\nReferences\n[1] Yin and Zhou, Semi-implicit variational inference, ICML 2018.\n[2] Chen et al., Variational lossy autoencoder, ICLR 2016.", "extra_scores": {}, "extra_text": {}}, {"review_id": "5qBJW-gB-Vf", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary: The paper proposes a method for reconstructing noise-free data instances without assuming any ground truths for this, using a variant of auto-encoder that avoids posterior collapse by utilising a newly proposed reduced entropy condition. The problem itself is important and the proposed method seems to offer good empirical performance for the task, outperforming a good selection of recent methods. The paper is well written with ample citations for relevant work, and provides sufficient technical details.\n\nReasons for score: A good paper with no obvious flaws; I do not have concrete improvement suggestions.\n\nAdditional comments: The key concept of empirical prior and its use for regularising the autoencoder is good and intuitive, and I could not spot any obvious theoretical or practical issues with the idea. In the end the theoretical development results in fairly simple modification for the standard VAE objective that can be directly trained using standard algorithms. This is both a strength (the approach is easy to implement and can be plugged into existing models) and a minor weakness in terms of technical depth. Nevertheless, the justification for the loss is well explained in the paper.\n\nThe empirical experiments are sufficiently comprehensive and show good empirical performance in a range of tasks, against reasonable comparison methods. As a minor comment, I would recommend using a more consistent style for the figures (e.g. Fig 3 and Fig 5 use very different visual style).", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: Summary: 
The authors propose a Bayesian approach to data cleaning, implemented
via a variational auto-encoder. They argue that a common problem in
this context are posteriors that overfit by
concentrating on a low-dimensional subset and introduce an
optimization target intended to discourage that behavior.

Discussion: 
Arguably the main concern brought up in the reviews was how
much novelty there is in addressing latent variable posterior
collapse, solutions for which have been proposed. The authors were able to clarify that this was due to a
misunderstanding (the collapse they address is not in latent space),
and the reviewer considers the matter resolved.


Recommendation: 
I recommend publication. The reviewers are all
positive, agree that the method is interesting, and seems novel. The
writing is clear, and remaining doubts have been addressed in the discussion.

Decision: Accept (Poster)