Improving Variational Autoencoder Estimation from Incomplete Data with Mixture Variational Families
Authors: Vaidotas Simkus
Michael U. Gutmann
Venue: Accepted by TMLR ()

Abstract: We consider the task of estimating variational autoencoders (VAEs) when the training data is incomplete. We show that missing data increases the complexity of the model’s posterior distribution over the latent variables compared to the fully-observed case. The increased complexity may adversely affect the fit of the model due to a mismatch between the variational and model posterior distributions. We introduce two strategies based on (i) finite variational-mixture and (ii) imputation-based variational-mixture distributions to address the increased posterior complexity. Through a comprehensive evaluation of the proposed approaches, we show that variational mixtures are effective at improving the accuracy of VAE estimation from incomplete data.

Published in Transactions on Machine Learning Research (06/2024)
Improving Variational Autoencoder Estimation from Incomplete Data with Mixture Variational Families
Vaidotas Simkus
Sridotas.simkussee uk michael.gutmanne www.enaummmmmmmm.....m.mm wed.
Michael U. Gutmann
School of Informatics
University of Edinburgh
Reviewed on OpenReview: openrevvee.. net/ formm? id= LLVMIu2fry
Abstract
We consider the task of estimating variational autoencoders (VAEs) when the training data incomplete. We show that missing data increases the complexity of the model's posterior distribution over the latent variables compared to the fully-observed case. The increased complexity may adversely affect the fit of the model due to a mismatch between the vari- ational and model posterior distributions. We introduce two strategies based on (i) finite variational-mixtuure and (ii) imputation-based variational-mixture distributions to address the increased posterior complexity. Through comprehensive evaluation the proposed approaches, we show that variational mixtures are effective at improving the accuracy of VAE estimation from incomplete data.
1 Introduction
Deep latent. variable models, as introduced by Kingma & Welling (2013); Rezende et al. (2014); Goodfellow et al. (2014); Sohl-Dickstein et al. (2015); Krishnan et al. (2016); Dinh et al. (2017), have emerged as a predominant approach to model real-world data. The models excel in capturing the intricate nature data by representing it within well-structured latent space. However, they lypically require large amounts of fully-observed data at training time, while practitioners many domains often only have access to incomplete data sets.
In this paper focus the class of variational autoencoders (VAEs, Kingma & Welling, 2013; Rezende et al., 2014) and investigate the implications of incomplete training data on model estimation. Our contri- butions are as follows:
We show that data missingness can add significant complexity to the model posterior of the latent variables, hence requiring more flexible variational families compared to scenarios with fully-observed data (section 3).
We propose finite variational-mixxure approaches to deal with the increased complexity due to missingness for both standard and inportance-weighted ELBOs (section 4.1).
We further propose imputation-based variational-mixture approach, which decouples model esti- mation from data missingness problems, and result, improves model estimation when using the standard ELBO (section 4.2).
We evaluate the proposed methods for VAE estimation on synthetic and realistic data sets with missing data (section 6).
The proposed methods achieve better or similar estimation performance compared to existing methods that do not use variational mixtures. Moreover, the mixtures formed by the variational families that used

---

Published in Transactions on Machine Learning Research (06/2024)
in the fully-observed case, which allows us to seamlessly re-use the inductive biases from the well-studied scenarios with fully-observed data (see e.g. Miao et al., 2022, for the importance of inductive biases in VAFs).
2 Background: Standard approach for VAEs estimation from incomplete data
We consider the situation where some part of the training data-points might be missing. We denote the observed and missing parts of the i-th data-point x' by the and wiis: respectively, where x is D-dimensional and the dimensions of at and aims must add to D. This split into observed and missing componnents corresponds to a missingness pattern m E {0, 1] with m) if the j-th dimension is observed and m² 0 if the dimension is missing. The missingness pattern m' is generally different for each data-point and is a realisation of random variable that follows a typically unknown missingness distribution p" (m x²). We make the common assumption that the missingness distribution does not depend on the missing variables, which is known as the ignorable missingness or diising-at-randonn assumption (MAR, c.g. Little & Rubin, 2002, Section 1.3).² The MAR assumption allows us to ignore the missingness pattern m when fitting a model po of the true distribution p" (x) from incomplete data (see e.g. Seaman et al., 2013, Theorem 1), as
well as when performing multiple imputation of the missing data (see e.g. van Buuren, 2018, Section 2.2.6).
The VAE model with parameters 0 is typically specified using decoder distribution (0 2), parametrised using a neural network, and a prior (=) over the latents that either be fixed or learnt. A principled approach to handling incomplete training data is then to marginalise the missing variables from the likelihood po (z), which yields the marginal likelihood
po (2:tt)))) = f pe (2'tt w'mss) darmis /ʃ po pa I'mis (z) dz de = f pe pe =))o(z) dz, (1)
where the inner integral po (o Tmis z) dærmss is often computationally tractable in VAEs due slandard assumptions, such as the conditional independence of æ given x or the use of the Gaussian family for the dccoder Pa (a: | z). Similar to existing work, we also make the assumption that the marginalisation of the missing variables is tractable. However, the marginal likelihood above remains intractable to compute as a consequence of the integral over the latents z.
Due to the intractable integral, VAEs are typically fitted via a variational evidence lower-bound (ELBO)
log)o(y)) 2 (x/() log po 2)))(z) (0(2 9) = log po()) DKI. y) po ()),
(2)
where refers to of in the fully-observed case, and to Thys in the incomplete-data case, and q (z is an (amortised) variational distribution with parameters & that is shared for all data-points in the data set (Gershman & Goodman, 2014). The amortised distribution is parametrised using a neural network (the encoder), which takes the data-point y as the input and predicts the distributional parameters of the variational family. Morcover, when the data incomplete, f.c. y E sharing of the encoder for any pattern of missingness is often achieved by fixing the input dimensionality of the encoder twice the size of x and providing ( and m' as the inputs, ? where 7(-) is a function that takes the incomplete data-point Babs and produces a vector of length D with the missing dimensions set to zero3 (Nazábal et al., 2020; Mattci & Frellsen, 2019).
y)
From eq. (2), we see that the training objective for incomplete and fully-observed data has the same form, and therefore may scem that fitting VAEs from incomplete data would be similarly difficult the fully- observed case. However, as we will see next, data missingness can make model estimation much harder than in the complete data case.
2

---

Published in Transactions on Machine Learning Research (06/2024)
p x)
p z
Xoha
2(z x)
B le à 19(z) xx]
Figure 1: Nlustration of the posterior complezity due to missing dala. Each colour represents a different data-
point x". First: the model posterior pe(= x) under complete data æ. Second: the model posterior pe(= Fobs) under incomplete data Bobs. Third: variational approximation 2 (z x)) of the complete-data posterior po x). Fourth: an imputation-mixture variational approximation Bpo
( (2 Tobs, Fmiis]] )] of the incomplete posterior po(= dohn). In these figures, we use VAE with Gaussian variational, prior, and decoder distributions fitted on complete data, then the incomplete data-points Tobs are obtained by randomly masking 50% of the values from the complete data-points x.
3 Implications of incomplete data for VAE estimation
The decomposition of the ELBO in eq. (2) emphasises that accurate cstimation of the VAE model requires the variational distribution (z(x Tobs) to accurately approximate the model posterior pe(z Toba). While it might appear that the marginalisation of the missing variables in eq- (1) comes at no cost since the ELBO maintains the same form as in the complete case, we here illustrate that his is not the case.
In the two left- columns of fig. 1 we illustrate the model posteriors po .) under fully-observed data x and partially-observed data Lobs. We discover that the model posteriors po(z x)), which exhibited a certain regularity in the complete-data scenario, have become irregular multimodal distributions po(z Zobs) in the case of incomplete data.5 Hence, accurate estimation of VA from incomplete data may require more flexible variational families than in the fully-observed case: while family may sufficiently well approximate the model posterior in the fully-observed case, it may no longer be sufficiently flexible in the incomplete data case. We provide a further explanation when this situation may occur in appendix A. As a result of the mismatch between the model posterior po Fabe) and the variational distribution ( Fabx), the incomplete-data KL divergence term DKL. (1 Wobs) || po Tobs)) in eq. (2) may be large compared to the analogous KL term DKL DKL(9g(5 (= | x)) (= x)) in the complete-data case, subsequently introducing a bias to the fit of the model.
In the two right-most columns of fig. 1 we illustrate the variational approximations of the aforementioned model posterior distributions, pe (z x) and pe( Robs). The first of the two plots shows the complete- data variational distribution (z x)) obtained after training, which well-approximates the model posterior [z æ). In the second of the two plots, we construct the incomplete-data posterior approximation as follows: Tobss #miss]] R (po ( Labs, Fmis)], which well-approxinates the model posterior p=(z Robs) too. Taken together, the two plots show that if the variational family used in the fully-observed case well-approximates the model posterior, i.c. 90(2 æ) R po(z æ), then the (ys (4, (Fobs 4miss]] will also be a good approximation of the incomplete- data posterior Po Fabs). This observation suggests that we can work with the same variational family in both the fully-observed and incomplete data scenarios if we adopt a mixture approach. In the rest of
po(z Wohs) = (poo (oonil [po(= (Po(
impulation-mixiun Epa (2mue
While there is some historical disparity between MAR assumptions in the statistics literature, we can here adopt the weakest MAR assumption, also known as the realised MAR (Seaman al., 2013).
2 Alternative encuder architectures, such as, permutation-invariant networks (Ma et al., 2019) are also used.
a 'Equivalent to setting the missing dinensions the empirical mean for zero-centered data.
'In fig. use VAE with Gaussian variational, prior, and decoder distributions fitted complete data,
A related phenomenan, called posterior inconsistency, has been recently reported concurrent work by Sudak & Tschi- atschek (2023), relating pe (= aoha) and po(z uube where is subset the observed dimensions (see section 5).
3

---

Published in Transactions on Machine Learning Research (06/2024)
this paper, we investigate opportunities to improve VAE estimation from incomplete data by constructing variational mixture approximations of the incomplete-data posterior.
4 Fitting VAEs from incomplete data using mixture variational families
We propose working with mixture variational families to mitigate the increase in posterior complexity and to improve the estimation accuracy of VAEs when the training data are incomplete. This allows us to use families of distributions for the mixture components that are known to work well when the data is fully-observed, and use the mixtures handle the increased posterior complexity due to missing data.
We propose two approaches for constructing variational mixtures. In section 4.1 specify g6(2 Tobs) mintte-ixtture distribution that can be learnt directly using the reparametrisation trick. In section 4.2 we investigate an imputation-based variational-mixture to approximate ppo(#mi (aare) 99 (z Tobs: Ømis))
Detailed evaluation of the proposed methods is provided in section 6.
4.1 Using finite mixture variational distributions to fit VAEs from incomplete data
In section 3 we saw that the imputation-mixture Eng (m (99(2 Tobs, Tmiis) ] is good approximation of the incomplete-dala posterior (a (bbs)) and would thus be a suitable variational distribution ( Boha). However, estimation of the imputation distribution (T Toba) is generally intractable for VAEs (Rezende et al., 2014; Mattei & Frellsen, 2018a; Simkus & Gutmann, 2023). Hence, we here consider a more tractable approach and specify the variational distribution 90 (z Tobs) in terms of fiito-mixture distribution:
(x(2 Wobs)) = E ( Doba) Tobs);
where (k(k (obs) is a categorical distribution over the components k E {l...., K} and each component distribution d/ Tobs) belongs to any reparametrisable distribution family. Both qs (k Fobs)) and x (z Wobs) are amortised using an encoder network, similar section 2.
The "reparametrisation trick" is typically used in VAEs to efficiently optimise the parameters & of the variational distribution (0(2 Dobs). This requires that the random variable can be parametrised as a learnable differentiable transformation t(c; Tobs, o) of another random variable that follows distribution with no learnable parameters. However, reparametrising mixture-families requires extra care: sampling the mixture (²(= wobs) in eq. (3) is typically done via ancestral sampling by first drawing k ~ 9g(k Tobs) and then z d (= Toba), but the sampling of the categorical distribution 4g(k aabs)) is non-diflerentiabll, making the direct application of the "reparametrisation trick" generally infeasible.
To make the fitting of VAEs using mixture- variiaiona distributions feasible we consider two objectives basod on the variational ELBO (Kingma Welling, 2013; Rezende et al., 2014):
CFL.O(EEtaa (xg(x((aan)) [log w(2)], and
(4)
K OSELBO(Aaaa) Σ (T Doba Bug (= [log w(2)),
(5)
k=1
where wl(z) Po Po(orobs:
(6)
9s (z Dobss)
The first objective LELBO corresponds to the standard ELBO, while LSELBO is the stratified ELBO (Roeder et al., 2017, Section 4; Morningstar et al., 2021). When working with LELBO, due to the mixture varia- tional family, we will need to optimise & with implicit reparametrisation (Figurnov et al., 2019). Implicit reparametrisation mixture distributions requires that the component distributions of Tobs) can factorised using the chain rule, i.e. o dlob)) = Ia at (za Zed, Fohs), and that we have access to the CDF (or other standardisation function) of each factor (z4 #<d. Fobs). However, the chain rule require- ment be difficult to satisfy for some highly flexible variational families, such normalising flows (e.g.
4

---

Published in Transactions on Machine Learning Research (06/2024)
Papamakarios et al., 2021), and finding the (conditional) CDF of the factors can also be hard if not already known in closed form. Consequently, LELBO with implicit reparametrisation may not be usable with all distribution families as components of the variational mixture.
The second objective LSELBO, on the other hand, samples the mixture distribution with stratified sampling," 6 which avoids the non-differentiabiiiyy of sampling q (k Tabs), and as a result allows us to use any family of reparametrisable distributions as the mixture components.
The importance-weiggted ELBO (IWELBO, Burda et al., 2015) is often used as alternative to the standard ELBO as it can be made tighter. We here also consider an ordinary version, LIWELBO, and stratified version, LSIEeBBO (Shi et al., 2019, Appendix A; Morningstar et al., 2021):
(oob)) =
Eix, 3/=1 -9 ( (alacabe))
log Σ w(
and
(7)
f=1
CSSEELBO (Fota) =
log
Σ Uo
(Ic
Fohs)
1 Σ w
7
w(2))
(8)
where I is the number of importance samples in LIWELBO and the number of samples per-mixlureecomponeni in LSIWELBO.
When the number of mixturre-componets is K K=1 1 the lower-bounds in eqs. (4-5) and eqs. (7-8) correspond to the MVAE and MIWAE bounds in Mattei & Frellsen (2019) which are among the most popular bounds for fitting VAES from incomplete data. However, as K > the proposed bounds can be tighter due an increased flexibility of the variational distribution go (z @oohs) (Morningstar et al., 2021, Appendix A), which potentially mitigates the problems caused by the missing data (see section 3). Finally, the importance- weighted bounds in egs. (7) and (8) maintain the asymptotic consistency guarantees of Burda et al. (2015) and approaches the true marginal log-likelihood log pe (Kohs) as K I + xo, allowing for more accurate estimation of the model with increasing computational budget.
We denote the four methods based on eqs. (4), (5), (7) and (8) by MissVAE, MissSVAE, MissIWAE, and MissSIWAE respectively.
4.2 Using imputation- mixttrre distributions to fit VAEs from incomplete data
In section 4.1, we jointly dealt with both the inference of the latents = (section 2) and the posterior complexity increase due to missing data (section 3) by learning finte-mixture variational distribution. Here, we propose second "decomposed" approach to deal with the complexities of missing data.
Intuitively, if we had an oracle that were able to generate impulations of the missing data from the ground truth conditional distribution p" (*mis Robe), then the VAE estimation task would reduce to the case of complete-data. This suggests that an effective strategy is to decompose the task of model estimation from incomplete data into two (iterative) tasks: (i) data imputation and (ûi) model estimation, akin to the Monte Carlo EM algorithm (Wei & Tanner, 1990; Dempster et al., 1977). However, access to the oracle p" (Emis Fobs) is unrealistic and the exact sampling of po(#mis Toba), as required in EM, is generally intractable. To address this, we resort to (i) approximate but computationally cheap conditional sampling methods for VAEs to generate imputations (Rezende et al., 2014; Mattei & Frellsen, 2018a; Simkus &
"Stratified sampling of mixture distributions typically draws an equal number of samples from each component and weighs the samples by the component probabilities (k(k) | (obs) when estimating expectations. It is commonly used to reduce Monte Carlo variance (Robert Casella, 2004).
7In multimodal-domall VAE literature, Shi et al. (2019) proposed a looser bound related to LSIWELBO:
K
SSWELBO (Poba)) 2 Zl (TWEEBB((FOL)) (Toba)) Eas(k (k(k | (13) (4l
log
Σ m(()))
(
T
j=1
and empirically showed that may alleviate potential mixture collapse a subset. of the mixture components. Therefore, the looser bound may useful when variational mixture collapse is abserved.
5

---

Published in Transactions on Machine Learning Research (06/2024)
Gutmann, 2023) and (ii) separate learning objectives for the model po and the variational distribution 9 to compensate for potential sampling errors. We call the proposed approach DeMissVAE (decomposed approach for handling missing data in VAEs).
We construct the variational distribution g0 Iobs) for an incomplete data-point Tobs using a completed-data variational distribution (0 Babss Piis) and an (approximate) imputation distribution
f'(miis Toba) z po (mmis Tobs):
( Tobs) B (Mtmial motoa) [g Tobs: &mis)]
(9)
The intuition for this construction comes from the decomposition of the model posterior pe(= Tobs) po (mate)
(po() [po(z Tobs, Fmis)]. Assuming that the complotod-data variational distribution 9 Bobs, Bmis) well-represents the model posterior po(z Lohs, Lmiss; and that the imputation distribution (2 Dobs) draws plausible imputations of the missing variables, then f6,r=(2 Tabs) will reasonably represent p= (obs)) (see the two right-most columns of fig. 1). A more technical justification is provided below, in the paragraph after eq. (13). In contrast to section 4.1 we here use a continuous mixture variational distribution, which more flexible than a finite-mixture distribution, albeit at an extra computational cost. due sampling the (approximate) imputations (see appendix D).
We now derive the DeMiss objectives for fitting the generative model (x and the completed-data variational distribution Us Tobs, Emis); see appendix C for more in-depth treatment.
Objective for pa z). With the variational distribution in eq. (9), we derive an ELBO on the marginal log-like liellikei similar to eq. (2), to learn the parameters of the generative model:
po (oobs x)
log pa (Doba) 2 E l (20is 2g (#1 (emis)
log
E ( (mob))
9gs Dobss Emis)
E ( (Coo 2o (#l ammte)
[log po poo(anoo (I 2)] H [9. (z Dohs)]
(10)
Coust.
This bound can be further decomposed into log- likelihood and KL divergence terms
(ovi Φ, 0, (*) + H [= [orrs)] = log (m (K..(y @obs)) po(z (oba)),
(11)
which means that if gp Tobs) (= Foba) then maximising eq. (10) w.r.t. 0 performs approximate maximum-likelihon estimation. Importantly, the missing variables Emis marginalised-out, which adds robustness to the potential sampling errors in I (3 Robs).
Objective for (p(2 æ). We obtain the objective for learning the variational distribution o x)) by marginalising the missing variables Emis from the complete-data ELBO in eq. (2) and then lower- bounding the integral using f ('(miss Tobs) (sce appendix B):
pol Tobs 3 Tmiss 2) I (z Dohss xmiss)
log Pa([Ebbb
2 (r ( 20 Lolo miis)
log
+ H (2 [ohaa]
(12)
Coust. w.r.t.
LLve (p.0.
This lower-bound can also be decomposed into the log- likeelhld term and two KL. divergence terms
co LLMva(Aoha 0,0,5' + H ( (amis Dobs]] = log (a DKL. (20mis mobss) ||
po (Xmiis Dohss) ))
E [ mmore)
DKL (gpl Lobs: Ømis) pe (z Tobss @ Emiss ))] (13)
which means that the bound maximisod w.r.t. o iff (/(2 (z Therefore, using the above objective to fit 9g corresponds directly the completo-data case, and hence avoids having approximate complex posteriors that arise due to missing data (see 3).
oobs: #mis) = Po(z Tobss Fmis) for all Emis-
6

---

Published in Transactions on Machine Learning Research (06/2024)
If 90 (z | Tobs: Fmis) pz Tobs Fmis) for all Tmis: then maximising either of the bounds in eqs. (10) or (12) w.r.t. the imputation distribution f' ( Robs) would correspond to setting f ( Toba) po (Tiis Tobs). However, diroctly learning imputation distribution f" [obs) po (Fmis | Fobs) is challenging (Simkus et al., 2023, Section 2.2). This motivates using sampling methods to approximate the optimal imputation distribution f" [ohss) R (Tmis Tobx) with samples. We draw samples from (Z Tota) using (cheap) approximate conditional sampling methods for VA to obtain K imputations 1K and then use them to approximate the expectations w.r.t. (I (obs)) in the above objectives. farin Fg
We discuss the implementation of the algorithm in detail appendix D.
Finally, we note that the Levi and Ci objectives in eqs. (10) and (12) are based on the standard ELBO. Extensions to the importance-weighted ELBO might improve the method further by increasing the flexibility of the variational posterior. However, unlike the standard ELBO used in eq. (10) where the density of the imputation-basee variational-mixture f Toba) can be dropped, IWELBO requires computing the density of the proposal distribution f (= Toha), which is generally intractable. We hence leave this direction for future work.
5 Related work
Fitting VAEs from incomplete data. Since the seminal works of Kingma & Welling (2013) and Rezende et al. (2014), VAFs have been widely used for density estimation from incomplete data and various down- stream tasks, primarily due to the computationally efficient marginalisation of the model in eq. (1). Vedantam et al. (2017) and Wu & Goodman (2018) explored the use of product-of-experts variational distributions, drawing inspiration from findings in the factor analysis case with incomplete data (Williams et al., 2018). Mattei & Frellsen (2019) used the apportance-wiightee ELBO (Burda et al., 2015) for training VA on incomplete training data sets. Ma al. (2019) proposed the use of permutation invariant neural networks to parametrise the encoder network instead of relying on zero-masking. Nazábal et al. (2020) introduced hierarchical priors to handle incomplete heterogeneous training data. Simkus et al. (2023) proposed a general-purpent approach that is applicable to VAEs, not requiring the decoder distribution to be casily marginalisable. Here, we further develop the understanding of VAEs in the presence missing values in the training data set, and propose pariational-mixturre. as a natural approach to improve VAE estimation from incomplete data, building upon the motivation from imputation-mixtunn discussed in section 3.
Variational mixture distributions. Mixture distributions have found widespread application in vari- ational inference and VAE literature. Roeder et al. (2017) introduced the stratified ELBO corresponding to eq. (5). In the context of VAEs in multimodal domains, Shi et al. (2019, Appendix A) introducedd the stratified IWELBO corresponding to eq. (8), but opted to use looser bound instead, see footnote 7. These bounds were subsequently rediscovered by Morningstar et al. (2021) and Kviman et al. (2023), who inves- tigated their use for VAE estimation in fully-observed data scenarios. Figurnov el al. (2019) introduced implicit reparametrisation, enabling gradient estimation for ancestrally-samplee mixtures, allowing the es- timation of variational mixtures using cqs. (4) and (7). Here, we build on this prior work, asscrting that variatiionll-mixtures are well-suited for handling the posterior complexity increase due to missing data (see section 3). Moreover, the imputation-mixture distribution used in DeMiss is a novel type of variational mixtures specifically designed for incomplete data scenarios.
Posterior complexity increase due to missing data. Concurrent to this study, Sudak & Tschiatschek (2023) have brought attention to phenomenon related to the increase in posterior complexity due to incomplete data, discussed in section 3. They noted that, for any Zobs and Tohs) where is subset of the observed dimensions, the model posteriors pe(= Øobs) and po @obs)(u should exhibit a strong dependency. However, because of the approximations in the variational posterior (sce c.g. Cremer et al., 2018; Zhang et al., 2021), the variational approximations (z Tomx) and (x @obela) may not consistently capture this dependency, They refer to the lack of dependency between (x(= xobs) and 90(2z &aberw), compared to pe(= xobas) and pe(= Tobetw); as postcrior inconsistency. Focusod on improving downstream task performance, they introduce regularisation into the VAE training objective to address posterior inconsistency. In contrast. to their work, we compare the fully-observed and incomplete-data posteriors, po(z æ)) and
7

---

Published in Transactions on Machine Learning Research (06/2024)
Method
Po objective
ls objective
# of components
Mixture sampling
MVAEt
cq. (4)
eq. (4)
K ==l
MissVAE
eq- (4)
eq. (4)
K >
Ancestral
MissSVAR
eq- (5)
eq. (5)
K K>
Stratified
MIWAEI
eq. (7)
eq. (7)
K = 1
MissIWAE
eq. (7)
(7)
K >
Ancestral
MissSTWAE
eq. (8)
eq. (8)
K> 1
Stratified
DcMissVAE
eq. (10)
eq. (12)
K>1
Conditional VAE
Table 1: Summary of the proposed and baseline methods. The non-mixture baselines (1) are based on Mattei & Frellsen (2019) and the other methods are proposed in this paper. Morcover, the mothods using ancestral sampling require implicit reparametrisation (Figurnov et al., 2019), whereas the other methods work with the standard reparametrisation trick.
pa Tobs), respectively. Observing that the incomplete-data posterior pe Toba) can be expressed as a mixture fully-observed posteriors po( æ)) that is, Po(z Tobb)) = Ep (pmmil//))))))) [po(2 æ)], we propose using pariationalmmixure to improve the match belween the variational and model posteriors when dealing with incomplete data in order to improve model estimation performance.
Marginalised variational bound. In the standard ELBO derivation for incomplete data in eq. (2) the missing variables are first marginalised (collapsed) from the likelihood, and then variational ELBO is established. This approach is sometimes referred to as collapsed variational inference (CVT). In contrast, in the derivation of the DeMiss encoder objective in eq. (12) we swap the order of marginalisation and variational inference. Specifically, we start with the variational ELBO on completed-data, and then marginalisc the missing variables (sce appendix B). This approach bears similarity to the marginalisod variational bound (MVB, KI-corrected bound) in exponential-conjugate variational inference literature (King & Lawrence, 2006; has been preferred over CVI due to improved convergence and guarantees that, for appropriately formulated conjugate models, MVB is analytically tractable in cases where CVI is not (Hensman et al., 2012, Section 3.3). While MVB remains intractable the VAE setting with incomplete data, similar to how the standard ELBO is intractable fully-observed case, we find the motivation behind MVB and DeMiss VAE to be similar.
& Titsias, 2011; Hensman et al., 2012). In these works, MVB
6 Evaluation
We here evaluate the proposed methods, MiE, MissSVAE, MissIWAE, MissSIWAE (section 4.1), and DeMissVAE (section 4.2), on synthetic and real-world data, and compare them to the popular methods MVAE and MIWAE that do not use mixture variational distributions (Mattei & Frellsen, 2019). The methods are summarised in table and the code implementation is available at https://github. com/ rsinkus/demiss-vae.
6.1 Mixture-of of- Gaussians data with a 2D latent VAE
Evaluating ligelikelihood on held-out data is generally intractable for VAFs due to the intractable integral in eq. (1). We hence here choose a VAE with 2D latent space, where numerical integration can be used estimate the log-likelihood of the model accurately (see appendix E.1 for more details). We fit the model incomplete data drawn from mixture-of Gaussians distribution. By introducing uniform missingness of 50% in the ceatuure-ffGaussiane allows us to verify the efficacy of mixture--vriational distributions when the posteriors are multi-modal due to missing data.
data we introduce multi-modality in the latent space (see fig. 1), which
Results are shown fig. 2. We first note that the stratified MissSVAE approach performed better than ViE that uses ancestral sampling. The reason for this likely that stratified sampling reduces Monte Carlo variance of the gradients w.r.t. o and hence enables better fit of the variational distribution 9s(2)
8

---

Published in Transactions on Machine Learning Research (06/2024)
MVAE
Mis
MisSVVAE
MIWAET
Miss
MisSIWAE
DeMiss VAE
-275
Nileeibbood
-3.00
-3.00
-3.0
2=
Z-15
1-
K-5 26-255
K= K=157=151
K K-252-20
K=5 7=
K=25 Z=1
1-15
25
K=51=6
K=15
K=25 15
K=5
K=15 1=1
K= 1=]
K=5
K-15
K=25
K=15
Figure 2: Log-likelihood on held out data evaluated by numerically inlegruting the lalent variables. VAFs were fitted on mixture-of- mixture-ofGGausiaan Gaussian data with 50% missingness. Each model is fitted with a computational budget of 5/15/25 samples from the variational distribution. The box plots show 1st and 3rd quartiles, the black lines are the medians, the dashed lines are the means, and the whiskers show the data range over 5 independent runs. MVAE and MIWAF (f) are baseline methods by Mattei & Frellsen (2019). The other five methods are proposed in this paper.
MVAE
MIWAR
DeMisVAE
Vis
MisIWAE MisSSTAAE
MisSVAE
GAS
POWER
HEPMASS
MINIBOONE
20
10.0
-15
Log-likeliboon
7.5
22
-2
-1
5.0
25
2.5
30
0.0
20%
50%
80%
20%
80%
20%
80%
20%
80%
Miseingness
Missingness
Missingness
Missingness
Figure 3: Estimate of the test log-likelihood using the IWELBO with 50000, on four UCI data sets. Each data set was rendered incomplete by applying uniform missingness of 20/50/80% The curves show average performance over 5 independent runs of the algorithms and the intervals show the 90% centered interval.
Wobs) (see a further investigation in appendix F.1.1). In line with this intuition, the MissVAE results exhibit significantly larger variance than MissSVAE. Similarly, we observe that the stratified MissSIWAE approach performed better than MAS. Importantly, we see that the use of mixture variational distributions in MissSVAE and MissSTWAE improve the model fit over the MVAE and MIWAE basclines that do not use mixtures deal with the increased posterior complexity due to missingness. Finally, observe that DeMissVAB is capable of achieving comparable performance to MIWAE and MissSIWAE, despite using looser ELBO bound, which shows that the decomposed approach to handling data missingness can be used to achieve an improved fit of the model.
In appendix F,1.2, we analyse the model and variational posteriors of the learnt models. We observe that the mixture approaches bettor-approximate the incompleto-data posteriors, compared the approaches that do not use variational-mixttue better-behaved when fitted using the decomposed approach in DeMiss VAF.
Morcover, we also observe that the structure of the latent space is
9

---

Published in Transactions on Machine Learning Research (06/2024)
MNIST
Omniglot
-90
-115
Log-likelihoon
-120
-94
-126
96
MVAE
Miss Miss VAE
MissVVE
MIWAE
MissIWE
MisS
DeMiss VAL
MVAE
Miss VAE
MisSVAE
MIWAE
MissIWAE
MISSSWAE
DeMissVAR VAE.
Figure 4: Estimate of the test log-like log-likel using the TWELBO with 1 1000, MNIST and Omniglot data sets. Each image in the training data set was missing 2 out of 4 random quadrants. The box plots show 1st and 3rd quartiles, the black lines are the medians, the dashed lines are the means, and the whiskers show the data range over 5 independent runs.
6.2 Real-world UCI data sets
We here evaluate the proposed methods on real-world data sets from the UCI repository (Dua Graff, 2017: Papamakarios et al., 2017). We train VAE model with ResNet architecture on incomplete data sets with 20/50/ /80% uniform missingness (see appendix E.2 for more details). We then estimate the log- ligliiieldhood on complete test data set using the IWELBO bound with / = 50K importance samples. 8 For additional metrics see appendix F.2.
The results are shown in fig. 3. We first note that, similar to before, the stratified MissSVAF approach performed better than VAE which uses ancestral sampling. Importantly, we observe that using mixture variational distributions in MissSVAE improves the fit of the model over MVAE (with the exception on non-mixture variational distributions. Furthermore, the gains in model
the Miniboone data set.) that accuracy typically increase with data missingness, which verifies that MissSVAE performs better because it handles the increased posterior complexity due to missing data better (see fig. 1). Next, observe that the performance of MIWAE, MissIWAE, and MissSIWAE is similar, although we can note small improvement by using MissWWAF and MissSIWAF in large missingness settings. We observe only a relatively small difference between the IWAE methods because the use of importance weighted bound already corresponds to using more flexible comi-mmpiciitly defined variational distribution (Cremer et al., 2017), which here seems be sufficient deal with the complexities arising due to missingness. Finally, we note that DeMiss results in-between MissSVAE and MIWAE. This verifies that the decomposed approach result, can improve the fit of the model. Nonetheless, DeMiss VAE is
be used to
deal with data. missinguess and, surpassed by the IWAE methods, which is likely due to using the ELBO in DeMiss versus IWELBO in IWAE methods that can tighten the bound more effectively.
6.3 MNIST and Omniglot data sets
In this section we evaluate the proposed mothods on binarised MNIST (Garris et al., 1991) and Omniglot (Lake el al., 2015) data sets of handwrilten characters. We a VAE model with a convolutional ResNet encoder and decoder networks (see appendix E.3 for more details). The data made incomplete by masking out of 4 quadrants of an image random. Similar to the previous section, we estimate the log-likelihood
8 As oc. IWELBO approaches log pe (a:). Moreover, as suggested by Mattel Frellsen (2018h), improve the estimate held-out data fine-tune the encoder complete data before estimating the log- likellihood..
10

---

Published in Transactions on Machine Learning Research (06/2024)
a complete test data set using the IWELBO bound with I == 1000 importance samples. In appendix F.3 we report additional results with varying dimensionality of the latent variables.
On the MNIST data set we see that MVAE < MissVAE < MissSVAE similar to the previous results but MIWAE < MissSIWAE < MissWWAE. This suggests that MissIWAE, which uses ancestral sampling, was able to tighten the bound more effectively compared to stratified MissSTWAF, and was able to fit the variational distribution 9,(2 Fobs) well despite the potentially larger variance w.r.t. Φ. Moreover, we also see that MVAE < DeMissVAE < MIWAE, which further verifies that the decomposed approach is able to handle the data missingness well.
On the Omniglot data we observe that the mixture approaches perform similarly to MVAE and MIWAE, which do not use mixture variational distributions. This suggests that either the posterior multi-modality is less prominent in the Omniglot data set or that due to the reverse KL optimisation of the variational distri- bution all mixture components have degenerated to a single mode. Finally, DeMiss slightly outperforms MVAE, MissVAE, and MisSVVE, but is surpassed by the auporraanc-weeihhee. approaches.
Interestingly, in this evaluation the stratified approaches (MissVVAE and MissSIWAE) were outperformed by the approaches using standard ELBO and implicit reparamctrisation (MissVAE and MissIWAE). This suggests that the performance of each approach can be data- and model-dependent and hence both should be evaluated when possible.
7 Discussion
Handling missing data is key challenge in modern machine learning, as many real-world applications involve incomplete data. In the context of variational autoencoders, we have shown that incomplete data increases the complexity of the latent variables' posterior distribution. Therefore, accurately fitting models from incomplete data requires more flexible variational families than in the complete-data case. We stipulated that variational-mixturee are a natural approach for handling missing data. One benefit is that it allows us to work with the same variational families as in the fully-observed case, which enables the transfer of useful known inductive biases (Miao et al., 2022) from the fully-observed to the incomplete data scenario.
Subsequently, we have introduced two methodologies grounded in variational mixtures. First, proposed using finite variational mixtures with the standard and weightte ELBOs using ancestral and stratified sampling of the mixtures. Second, we have proposed a novel "decomposed" variational-mittie approach, that uses cost-effective yet often coarse conditional sampling methods for VAÉs to generate im- putations and ELBO-based objectives that are robust to the sampling errors.
Our evaluation shows that using variational mixtures can improve the fit of VAFs when dealing with incom- plete data, surpassing the performance of models without variational mixtures. Moreover, our observations indicate that, although stratified sampling of the finite mixtures often yields better results compared to ancestral sampling, the effectiveness of these methods can be data- and model-deependent and hence both approaches should be evaluated when possible, Our results further indicate that variational mixtures provide relatively little improvement with the IWELBO-based methods compared the ELBO-based methods. We believe that this is mainly because IWELBO can be seen to be working with semi-implicitly defined varia- tional distributions that are flexible enough to handle the posterior complexity increase due to missing data. Alternatively, this may be related to an observation by Shi et al. (2019, Appendix A) that the Reverse-KL formulation of the importance weeigten bound may lead to situations where the mixture components collapse to single mode. Hence, future direction would be to investigate alternative formulations of importance- weighted bounds that avoid the modc-secking nature (Bornschein & Bengio, 2015; Mnih & Rezende, 2016; Wan et al., 2020). Furthermore, we note that the decomposed approach in DeMissVAE outperforms all ELBO-based methods but falls short of surpassing IWELBO- based methods. These results point towards promising rescarch avenues, suggesting potential improvements in VAE model estimation from incomplete data. Future directions include extending the DeMiss VAE approach to incorporate IWELBO-based objec- tives and developing improved cost-effective conditional sampling methods for VA
11

---

Published in Transactions on Machine Learning Research (06/2024)
Budget
Variational families
Latent structure"
Evaluation rank
Method
MoG
UCI
MNIST+ Omniglot
V small
limited any
well-behaved well-behaved
5
MissSVAE medium
4
MissIWAE medium
limited
potentially irregular
2
MissSIWAE medium any
potentially irregular
1
2
DeMissVAE medium/higg any
well-behaved'*
2 3
Table 2: A coorse summarr of advantages and disadvantages of the proposed methods. Budget:
mmall/medium// high depending on the number of latent samples required or whether conditional sampling of VAEs is needed. Variational families: which families distributions can be used as mixture components- any reparamctrisable families, or limitod familics, as discussed in section 4.1. Latent structure: methods with potential to learn irregular latent spaces may have decreased downstream performance in certain lasks. We have found (+) that DeMiss VAE is able to achieve the most well-behaved latent structures the MoG data in appendix F.1.2. Please note (*) that the learnt latent structure will depend on the chosen model architecture. Evaluation rank: the rank of the proposed methods in the evaluations in sections 6.1 to 6.3.
The choice between the proposed methods for fitting VAEs from incomplete data depends on various factors such compatational budget, variational families, model accuracy goals, and the specific requirements of downstream tasks, discussed next and summarised in table 2.
Computational and memory budget. The standard ELBO with ancestral sampling is the most suitable method for small computational and memory budgets, since the objective can be estimated using a single latent sample for cach data point. On the other hand, methods using stratified sampling or the importance- weighted ELBO require multiple latent samples for each data-point and hence may only be used if the memory and compute budget. allows. Moreover, for fixed budget, stratified approaches may limit the number of components K that may be used. Lastly, akin to the standard ELBO, the DeMissVAE objectives can be estimated using a single latent sample, but the approach incurs extra cost in sampling the imputations.
Variational families. While the stratified and DeMissVAF. approaches can use any reparametrisable dis- tribution family for the mixture components, the ancestral sampling methods require the use of implicil reparametrisation (Figurnov et al., 2019) and as a result may not work with all distribution families (see discussion in section 4.1).
Model accuracy. Stratified sampling of mixtures can improve the model accuracy, compared to ancestral sampling, by reducing Monte Carlo gradient variance. Additionally, methods using the importance- weighted ELBO, compared to the standard ELBO, are oflen able to tighten the bound more ef- fectively by using multiple importance samples, leading to improved model accuracy. DeMissVAE performance lies in between the standard ELBO and importance-weighted ELBO approaches. Al- though the introduced DeMissVAE objectives exhibit robustness to some imputation distribution error, improved model accuracy can often be achieved by improving the accuracy of imputations by using larger budget for the imputation step.
Latent structure. Different downstream tasks may prefer distinct latent structures, for example, condi- tional generation from unconditional VAEs is often easier if the latent space is well-structured (Engel et al., 2018). To this end, observations in appendix F.1.2 show that
et al., 2017; the latent space of DeMissVAE bchaves well, and is comparable to a model fitted with complete data. This characteristic makes il preferable for downstream tasks requiring well-struct ured latent spaces. On the other hand, as noted by Burda et al. (2015, Appendix C) and Cremer et al. (2018, Scction 5.4), the use of importance-weightee ity due to missing data may make the latent space less regular, compared to model trained on fully-observed data set, which potentially decreases the model's performance on downstream tasks.
ELBO to mitigate the incrcased posterior complex-
12

---

Published in Transactions on Machine Learning Research (06/2024)
Finally, step back to note that this paper is focused the class of variational autoencoder models, subset of the broader family of deep latent variable models (DIVMs). Much like VA DIVMs usually aim to efficiently represent the intricate nature of data through a well-structured latent space, implicitly defined by a learnable generative process. Building on our findings in VAEs, where incomplete data led to an increased complexity in the posterior distribution compared to the fully-observed case, we conjecture that a similar effect may occur within the wider family of DLVMs, affecting the fit of the model. We therefore believe that there substantial scope to explore the implications of incomplete data in other DLVM classes, particularly focusing on the effects of marginalisation on latent space representations and the associated generative processes. Investigating decomposed approaches, similar to DeMiss or Monte Carlo EM (Wei & Tanner, 1990), presents promising avenues for further research in this direction.
References
Jöng Bornschein and Yoshua Bengio. Reweighted Sake-Sleep. In International Conference on Learning Representations (ICLR), April 2015. doi: 448550/ar..i.1411.. (Cited on 11)
Yuri Burda, Roger Grosse, and Ruslan Salakhutdinov. Importance Weighted Autoencoders. In International Conference on Learning Representations (ICLR), San Juan, Puerto Rico, September 2015. (Cited on 5, 7, 12, 17, 20, 25)
Chris Cremer, Quaid Morris, and David Duvenaud. Reinterpreting Seporlaace-Weighieee Autoencoders. In ICLR Workshop, February 2017. (Cited on 10, 17, 25, 27)
Chris Cremer, Xuechen Li, and David Duvenaud. Inference Suboptimality in Variational Autoencoders. In International Conference on Machine Learning (ICML), May 2018. (Cited on 7, 12, 25)
Arthur P. Dempster, Nan M. Laird, and Donald B. Rubin. Maximum Likelihood from Incomplete Data Via the EM Algorithm. Journal of the Royal Statistical Society: Series B (Methodological), 39(1):1 22, 1977. (Cited on 5)
doi:
Laurent Dinh, Jascha Sohl-Dickstein, and Samy Bengio. Density estimation using Real NVP. In International Conference Learning Representations (ICLR), February 2017. (Cited on 1)
Dheeru Dua and Casey Graff. UCI Machine Learning Repository, 2017. (Cited on 10, 23)
Jesse Engel, Matthew Hoffman, and Adam Roberts. Latent Constraints: Learning to Generate Conditionally from Unconditional Generative Models. In International Conference on Learning Representations (ICLR), December 2017. (Cited 12)
Michael Figurnov, Shakir Mohamed, and Andriy Mnih. Implicit. Reparameterization Gradients. In Advances in Neural Information Processing Systems (NeurIPS), January 2019. (Cited on 4, 7, 8, 12)
Michael D. Garris, R. A. Wilkinson, and Charles L. Wilson. Methods for enhancing neural network hand- written character recognition. In International Joint Conference on Neural Networks (IJCNN) volume 1, pp. 695 700, Seattle, WA, USA, 1991. IEEE. ISBN 978-0-7803-0164-1. doi: 10.1109 10.1109/IJCNN, (Cited on 10)
Samuel J. Gershman and Noah D. Goodman. Amortized Inference in Probabilistic Reasoning. In Annual Meeting of the Cognitive Science Society, volume 36, 2014. (Cited on 2)
Rafacl Bombaz-Bmm,arlli, Jennifer N. Wei, David Duvenaud, José Miguel Hernández-Lobato, Benjamín Sânchez-Lengeling, Demnis Sheberla, Jorge Aguilera-lparraguirre, Timothy D. Hirzel, Ryan P. Adams, and Alán Aspuru-Guzik. Automatic Chemical Design Using a Data-Driven Continuous Representation of Molecules, ACS Central Science, 4(2):268 276, February 2018 ISSN 2374-7943. doi: / 7b00572. (Cited on 12)
Ian Goodfellow, Jean Pouget-Abadic, Mchdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative Adversarial Networks. In Advances in Neural Information Processing Systems (NeurIPS), June 2014. (Cited on 1)
13

---

Published in Transactions on Machine Learning Research (06/2024)
James Hensman, Magnus Rattray, and Neil D. Lawrence. Fast Variational Inference in the Conjugate Exponential Family. In Advances in Neural Information Processing Systems (NeurIPS), December 2012. (Cited on 8)
Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. GANs Trained by a Two Time-Scale Update Rule Converge to Local Nash Equilibrium. In Adoances in Newral Information Processing Systems (NeurIPS), 2017. (Cited on 28)
Niels Bruun Ipsen, Pierre-Alexandre Mattci, and Jes Frellsen. Not-MIWAE: Deep Generative Modelling with Missing not at Random Data. In International Conference on Leurning Representations (ICLR), June 2020. (Cited on 17)
Nathaniel J. King and Neil D. Lawrence. Fast Variational Inference for Gaussian Process Models Through KL-Correction. In European Conference on Machine Learning (ECML), 2006. (Cited on 8)
Diederik P. Kingma and Max Welling. Auto-Encoding Variational Bayes. In International Conference on Learning Represendations (ICLR), December 2013. (Cited 1, 4,
Rahul G. Krishnan, Uri Shalit, and David Sontag. Structured Inference Networks for Nonlinear State Space Models. In AAAI Conference Artificial Intelligence, December 2016. doi: 10.88500/arXi..1600.96666 (Cited on 1)
Oskar Kviman, Ricky Molén, Alexandra Hotti, Semih Kurt, Victor Elvira, and Jens Lagergren. Cooperation in the Latent Space: The Benefits of Adding Mixture Components in Variational Autoencoders. In Inter- national Conference on Machine Learning (ICML), July 2023. doi: 10. 4..4885//rXiiv2200 2tt..2229.1155. (Cited on 7)
Brenden M. Lake, Ruslan Salakhutdinov, and Joshua B. Tenenbaum. Human-level concept learning through probabilistic program induction. Science, 2015. doi: 111126/ciiene.aaa
(Cited on 10, 23)
Miguel Lázaro-Gredilla and Michalis K. Titsias. Variational heteroscedastic Gaussian process regression. International Conference on Machine Learning (ICML), June 2011. (Cited on 8)
Roderick A. Little and Donald B. Rubin. Statistical Analysis with Missing Data: Second Edition. Wiley- Interscience, 2002. ISBN 0-471-18386-5. (Cited on 2)
Chao Ma and Cheng Zhang. Identifiable Generative Models for Missing Not Random Data Imputation. Tn Advances in Neural Information Processing Systems (NewrIPS), October 2021. (Cited on 17)
Chao Ma, Sebastian Tschiatschek, Konstantina Palla, José Miguel Hernández-Lobato Sebastian Nowozin, and Cheng Zhang. EDDI: Efficient dynamic discovery of high-value information with partial VAE. In International Conference on Machine Learning (ICML), pp. 7483 1504, 2019. ISBN 9781510886988. (Cited on 3, 7)
Mattci and Jes Frellsen. Leveraging the Exact Likelihood of Deep Latent Variable Models. In Advances in Neural Information Processing Systems (NeurIPS), February 2018a. (Ciled on 4, 5, 20, 23)
Phero-AAlxandie Mattei and Jes Frellsen. Refit your Encoder when New Data Comes by. In Workshop on Bayesian Deep Learning at Neural Information Processing Systems (NeurIPS), pp. 4, Montreal, Canada, 2018b. (Cited on 10)
Plerre-Alexandre Mattei and Jes Frellsen. MIWAF: Deep Generative Modelling and Imputation of Incomplete Data Sets. In International Conference on Machine Learning (ICML), 2019. (Cited on 2, 5, 7, 8, 9, 17)
Xiao-Li Meng. On the Rate of Convergence of the ECM Algorithm. The Annals of Statistics, 22(1):326-2) 323333, March 1994. ISSN 0090-5364, 2168-8966. doi: 10. /01214//0s/1117635377 (Cited on 18)
14

---

Published in Transactions on Machine Learning Research (06/2024)
Ning Miao, Emile Mathieu, N. Siddharth, Yee Whye Tch, and Tom Rainforth. On Incorporating Inductive Biases into VAFs. In International Conference on Learning Representations (ICLR), February 2022. doi: 10.48550/ 1..48500/arXiv. Xiv
13746. (Cited on 2, 11)
Andriy Mnih and Danilo J. Rezende. Variational inference for Monte Carlo objectives. fes, stati, June 2016. (Cited on 11)
Warren Morningstar, Sharad Vikram, Cusuh Ham, Andrew Gallagher, and Joshua Dillon. Automatic Differ- entiation Variational Inference with Mixtures. In International Conference on Artificial Intelligence and Statislies (AISTATS), pp. 3250 3258. PML.R, March 2021. (Cited on 4, 5, 7)
Alfredo Nazábal, Pablo M. Olmos, Zoubin Ghahramani, and Isabel Valera. Handling Incomplete Heteroge- neous Data using VAEs. Pattern Recognition, 107, 2020. ISSN 0031-3203. doi: 10.1016/j. 2020. 107501. (Cited on 2, 7)
George Papamakarios, Theo Pavlakou, and lain Murray. Masked Autoregressive Flow for Density Estimation. Advances in Newral Information Processing Systems (NeurIPS), 30, 2017. (Cited on 10, 23)
George Papamakarios, Eric Nalisnick, Danilo Jimenez Rezende, Shakir Mohamed, and Balaji Lakshmi- narayanan. Normalizing Flows for Probabilistic Modeling and Inference. Journal of Machine Learning Research, 22(57):1 64, 2021. (Cited on 5)
Tom Rainforth, Adam R. Kosiorek, Tuan Anh Le, Chris J. Maddison, Maximilian Igl, Frank Wood, and Yee Whye Teh. Tighter Variational Bounds are Not Necessarily Better. In International Conference on Machine Learning (ICML), March 2019. (Cited on 24)
Sashank J. Reddi, Satyen Kale, and Sanjiv Kumar. On the convergence of Adam and beyond. In International Conference on Learning Representations (ICLR), pp. 1-23, 2018. (Cited on 22, 23)
Danilo Jimenez Rezende, Shakir Mohamed, and Daan Wierstra. Stochastic Backpropagation and Approxi- mate Inference. In International Conference on Machine Learning (ICML), Beijing, China, 2014. (Cited on 1, 4, 5, 20, 23)
Christian P. Robert and George Casella. Monte Carlo Statistical Methods. Springer, 2004. ISBN 0-387- 21239-6. (Cited on 5)
Geoffrey Roeder, Yuhuai Wu, and David K. Duvenaud. Sticking the Landing: Simple, Lower-Variance Gradient Estimators for Variational Inference. In Advances in Newral Information Processing Systems (NeurIPS), volume 30, 2017. (Cited on 4, 7, 22, 23)
Shaun Seaman, John Galati, Dan Jackson, and John Carlin. What Is Meant by "Missing at Random "? Statistical Science, 28(2):257 268, May 2013. ISSN 0883-4237, 2168-8745. doi: 10.1214/13-ST4415. (Cited on 2, 3)
Yuge Shi, N. Siddharth, Brooks Paige, and Philip Torr. Variational Mixture-of-Expert Autoencoders for Multi-Modal Deep Generative Models. In Advances in Neural Information Processing Systems (NewrIPS), 2019. (Cited on 5, 7, 11)
Vaidotas Simkus and Michael U. Gutmann. Conditional Sampling of Variational Autoencoders via Iterated Approximate Ancestral Sampling. Transactions on Machine Learning Research, 2023. ISSN 2835-8856. (Cited on 4, 5, 19, 20, 23)
Vaidotas Simkus, Benjamin Rhodes, and Michael U. Gutmann. Variational Gibbs Inference for Statistical Model Estimation from Incomplete Data. Journal of Machine Learning Research, 24(196):1- 2023. ISSN 1533-7928. (Cited on 7, 21)
Jascha Sohh-Dickstein, Eric A. Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep Unsupervisod Learn- ing using Nonequiibrrum Thermodynamies. In International Conference on Machine Learning (ICML), November 2015. doi: 10.48550/arxiv.15503. (Cited on
15

---

Published in Transactions on Machine Learning Research (06/2024)
Timur Sudak and Sebastian Tschiatschck. Posterior Consistency for Missing Data in Variational Autoen- coders. In Europenn, Conference on Machine Learning and Principles and Practice of Knowledge Discovery in Databases (ECML-PKDD), October 2023. doi: 10.48550/ 231.1..66110. (Cited on 3, 7)
Tijmen Tieleman. Training restricted Boltzmann machines using approximations to the likelihood gradient. In International Conference Machine Learning (ICML), pp. 1064-1071, 2008. ISBN 9781605582054. doi: 10.1145/1390156 1399016611902000 (Cited on 21)
George Tucker, Dieterich Lawson, Shixiang Gu, and Chris J. Maddison. Doubly Reparameterized Gradient Estimators for Monte Carlo Objectives. In International Conference Learning Represenlations (ICLR); November 2018. (Cited on 22)
Stef van Buuren. Flexible Imputation of Missing Data. CRC Press LLC, 2 edition, 2018 ISBN 978-1-138- 58831-8. (Citod on 2)
Ramakrishna Vedantam, lan Fischer, Jonathan Huang, and Kevin P. Murphy, Generative Models of Visually Grounded Imagination. International Conference on Learning Representations (ICLR), May 2017. (Cited
on 7)
Neng Wan, Dapeng Li, and Naira Hovakimyan. F-Divergence Variational Inference. In Advances in Newral Information Processing Systems, volume 33, pp. 17370-17373) Curran Associates, Inc., 2020. (Cited on 11)
Greg C. G. Wei and Martin A. Tanner. A Monte Carlo Implementation of the EM Algorithm and the Poor Man's Data Augmentation Algorithms. Journal of the American Statistical Association, 85(411):699 704, September 1990. doi: 10.1080/0000 011.0//11..99..11111111919.1
(Cited on 5, 13)
Christopher K. I. Williams, Charlie Nash, and Alfredo Nazábal. Autoencoders and Probabilistic Inference with Missing Data: An Exact Solution for The Factor Analysis Case. arXiv preprint, 1801.03851, January 2018. (Cited
Mike Wu and Noah D. Goodman. Multimodal Generative Models for Scalable Yeakly-Supervise Learning. In NeurIPS 2018, February 2018. (Cited on 7)
Laurent Younes. On the convergence of markovian stochastic algorithms with rapidly decreasing ergodicity rates. Stochastics and Stochastic Reports, 05(3-4):177-2255 February 1999. ISSN 1045-1129, doi: 10.1080/ 174425099086341799 (Cited on 21)
Minglian Zhang, Peter Hayes, and David Barber. Generalization Gap in Amortized Inference. In Workshop on Bayesian Deep Learning at Neural Information Processing Systems (NeurIPS), pp. 6, 2021. (Cited on 7)
16

---

Published in Transactions on Machine Learning Research (06/2024)
A Posterior complexity due to missing information
The complexity increase of the model posterior due to missing data, shown in fig. 1, explains why flexible variational distributions (Burda et al., 2015; Cremer et al., 2017) have been preferred when litting VA from incomplete data (Mattei & Frellsen, 2019; Ipsen et al., 2020; Ma & Zhang, 2021). We here define the increase of the posterior complexity via the expected Kullback-Loible (KL) divergence as follows
pa(= Coba: (miss) I(z, Tmis Robs).
(x)( (Po(z a)) po(z [obe)]] Ep (mx) (=)()) log
pe Tobs)
As shown above the expected KL divergence equals the (conditional) mutual information (MI) between the latents and the missing variables miis-
The mutual information interpretation allows us to reason when a more flexible variational family may be necessary to accurately estimate VAEs from incomplete data. Specifically, when the MI is small then the two posterior distributions, p= x) and po(z (Bohs) are similar, which case a simple variational distribution may work sufficiently well. This situation might appear when the observed oobs and unobserved Fmis variables highly related and miss provides little additional information about over just Bobs: for example, when random pixels of an image strong relationship between neighbouring pixels. On the other hand, when the MT is high then Emis provides significant additional information about & over just Tobs, in which case more flexible variational family may needed, for example, when the pixels of an image are masked in blocks such that it introduces significant uncertainty about what missing.
masked it is "easy" to infer the complete image due to
B DeMissVAE: Encoder objective derivation
The standard (completc-data) ELBO in eq. (2) gives the incquality
Po (Dbss Triis: 2) ( Dobs: Fmis)
log (Dobs Emis) En ( Fmis)
log
which, together with the identity
log pe(Zobo) log / exp (log pa (oobs Emiis) damiss
yields
] dæmis:
log po Fmiss 2) [os (0(= Robs, xmiss)
log pe (Tobe) 2 log
exp
Eg Baglzlmotn (autn mmii))
As the integral on the r.h.s. is intractable, we lower-bound it using the imputation distribution f' Babs) and Jensen's 's inequality
/
ff (xmma Doba) Koba
Po Tmiss x)
log pe (Foba) 2 log|
exp
Bgg (=1 (aoeee log
dæmis
f (2mis
Io
Eobss Lmmis)
Po (oobs x Baiss 2)
log By (oob))
exp (- log (2 mobx)) exp
E Bgg (2t
log
gos (z) Babss Fruis
Pe (Fobs mmas 2) Tobes Emis ) (xmiss Zobe)
log ( (ooka)
exp Egg (Eloall (mmis)
log q
Where notation is suppressed due MAR assumption. In of missing-not-at-raaaee (MNAR) assumption there would be an additional dependency
17

---

Published in Transactions on Machine Learning Research (06/2024)
po oobss Fmiss 2)
2 ( (2mulaaaab Wobal gg Wobe pense)
log Los (2 Dobs Fmmss) ) I (Tmis Tobs)
po (oobs wmis 2) I (= Tabss Tmiis)
( ( (zlpooba
log
+ H [ (20is i Enbas)
def L
LMVB (Robn:
Coust. w.r.t. +
C DeMissVAE: Motivating the separation of objectives
_ The two DeMiss VAE objectives Levi and LLMV in eqs. (10) and (12) correspond to valid lower-bounds on log irrespective of f' Tobs). Morcover, both of them are tight at f' Dobs) = Po (T Wobs) and 9,(2 Tobs: Lmis) = pa (z Bohs; Tmis). So, a natural question is why do we prefer Levi to learn po and LiMVR to learn 9r. ?
Why use fevi in eq. (10) over LMMV in eq. (12) to learn po (x))? Maximisation of the objective C LLMV in iteration t w.r.t. O would have to compromise between maximising the log- aogllikeihood log pe (obb)) and keeping the other two KT, divergence terms in eq. (13) low. Specifically, the compromise between max- imising log Po (Tobx) and keeping DKL (I Toba) || po (T Tabw)) low is equivalent to the compromise in the EM algorithm, which known to affect the convergence of the model (Meng, 1994). Moreover, if (z Tobe) * (T Tobs) then minimising the (f Tobe) || po Øoba)) will fit the model (a)) to the biased samples from (a Tobs)) On the other hand, in L the missing variables mmis are marginalised from the model, therefore it avoids the compromise with DKI((f' (obs)) pe (2mis ))ha)) and the potential bias of the imputation distribution f'( Dohx) affects the model only via the latents & ~ (0 Robs), increasing the robustness to sub-optimal imputations.
Why use G.MVB in eq. (12) over Levi in eq. (10) to learn 9g(2 | f'(xmla Tobs) = po amis Dobs) then the bound is tightened when g0(= / Tobsi @mii)) = pe(= Tobs mmia) for all Ømis; which is the same optimal 4s if we used LLMVB. But, there is also at least one more possible optimal p
æ)?? In the case of LevI. if
solution (z Tobs, Fmiss) (= Eabs), which ignores the imputations and corresponds to the optimal solution of the standard approach in section 2, and thus it means that the optimum is (partially) unidentifiable and can make optimisation of 9, using LevT difficult. Moreover, if (* (obs) A pe po (Xmia Kobs) then in order to minimise DKL (9p.9 Tora) || p (obaa) w.r.t. & the variational distribution 9 Tobs Tmis) would have to compensate for the inaccuracies of ['( | (obss) by adjusting the probability mass over the latents z, such that ( Robs: #miss is correct on average, i.c. Ip.fi(2 Dota)) = EE 100 Tobs. mmiss] = (z Toba). These two issues make optimising via Levi such that 9 Tobe: Amia) pe Zobs: amis) difficult. On the other hand, in LLMVB the optimal 94 is always at g0 Tobss Dmis)) = (z Tobs, 2mit), irrospective of the imputation distribution f' f'(mmis Lobe); hence the CIAVE objective
in eq. (12) is well-defined and more robust to inaccuracies of f (æmis Fobs) for the optimisation of 9 (z Babs; F.nis).
In fig. 5 we verify the efficacy of DeMissVAE via a control study on small VAE model po( with 2D latent space fitted on incomplete samples from a ground truth mixture-of Gaussian (MoG) distribution p (x)) We evaluate fitting the VAE using only G in eq. (10) (CVI-VAE, blue), only LIMVR in eq. (12) VAE, yellow), and using the proposed two-objective approach (DeMissVAE, green). In the left-most figure we evaluate the three methods where we represent the imputation distribution f' Tobs) Po ( Robs) using rejection sampling, which corresponds to the optimal imputation distribution w.r.t. DRT.(f'(x... @ohn) || (Dmis (nw)) = O. We see that the proposed approach (green) dominates over the other two control methods (blue and yellow), and importantly that marginalisation the missing variables in DeMissVAE (green) improves the model accuracy compared to an EM-type handling of the missing variables (yellow). Furthermore, in the remaining two figures we investigate the sensitivity of the methods to the accuracy of imputations in f((mmis Tabs). In Oracle 1 we start with the ground-truth conditional
18

---

Published in Transactions on Machine Learning Research (06/2024)
CVI-VAE
MVB-VAE
DeMiss
Po Xmia
Xohs
Oracle
Oracle 2
Log- likeeikkeioodd
-3.5
-40
0.01
0.6
0.111
JSD( (Xxmis Xelw) p [Xenis Xota))
JSD (Xmia Xrha) 10" (Xinia Xolw))
Figure 5: A control study on a VAE model with 2D latent space (see additional details in appendir E. ezamining the sensitivity of the proposed method (DeMiss VAE, green) and two control methods (blue and yellow) to the accuracy of the imputation distribution f (" Tobs). Left: f'(mmms/ [obs) = pe (2 Tobs) represented using rejection sampling. Center: an oracle imputation function that gets progressively "wider" from left-to-right. of the figure. Right: an oracle imputation distribution that towards the right of the figure more significantly oversamples low-probability posterior modes. The lieeiiold is computed on a held- out test data set by numerically integrating the 2D latent space of the VAE. The horizontal axis on the two right-most figures shows the Jensen- -haennnn divergence between the imputation distribution and the ground-truth conditional p" ( Robs).
("mii Toba) and, along the x-axis of the figure, investigate how the methods perform when the imputation distribution becomes "wider": first interpolating from p (armis | fots) to an independent unconditional distribution [ldesd<(m) p" (xa) and then further towards and independent Gaussian distribution. And Oracle 2 we investigate what happens when the sampler "oversamples" posterior modes: we interpolate the imputation distribution from p" (mmis/ Øoba) to t Ee p" ( Tobs, c), where is the component of the mixture distribution with total of C components. As we see in the figure, the proposed DeMissVAF approach (green) performs similar or better than the MVB-VAE (yellow) and CVI-VAE (blue) control methods, with an exception when the (2(mmis Dobs) are extremely inaccurate (last two points on the middle figure) which is expected since 90.r=(z Dobe)) in eq. (9) can be arbitrarily far from po(z Fobs) when 9 Tobe: Fmis) = (e Tobs: {miss but f (" Tobs) & po (Imis/ Zobs).
Finally, in fig. we investigate what happens if we used only L&vi in eq. (10) or LLMVE in eq. (12) to the VAE model, in contrast the two separate objectives for encoder and decoder DeMissVAF. We use the LAIR sampling method (Simkus & Gutmann, 2023) detailed in appendix D obtain approximate samples from f'(xmis Tobs) (2iis Tobs) (x Tobs). And, we observe that DeMissVAE achieves better fit of the model, in line with our motivation in this section.
D DeMissVAE: Implementing the training procedure
& MAMissVAE requires optimising two objectives Lev1 and LLMVB in eqs. (10) and (12) and drawing (approx- imate) samples to represent f' Tobs) 20 pe (miss Loba). Our aim is to implement this efficiently to minimise redundant computation.
The algorithm starts with a randomly-initialisedd target VAE model po(a 2), an amortised variational dis- tribution 9 x)) and an incomplete data
(a And then, to represent the imputation distribution (x Wats), K imputations { 1K are generated for each a E D using some simple imputation function such as sampling the marginal empirical distributions of the missing variables. The algorithm then itcrates between the following two stops:
Imputation. Update the K imputations {Iihh 1K k=1 representing samples from the imputation distribution (w Bobss; ; such that f1 Bobs) is "closer" to po (Mmis a.bs)) For this, we
19

---

Published in Transactions on Machine Learning Research (06/2024)
CVI-VAE
AIV--VAE
DeMissVAR
-6
-28
Log- lieliikeiooodd
-300
-32
-3.4
-38
-18
-40
K=5
K-15
K=25
K=5
K=15-
K-25
K=5
K-15-
K=25
Figure 6: A control study on VAE model with 2D latent space (see additional details in appendir E.1), inwestigating the importance of the lwo-objectàve approach in DeMiss VA (green) and two control methods (blue and yellow). In CVI-VAE (blue) we fit both the encoder and decoder using eq. (10), and in MVB-VAE (yellow) we fit both the encoder and decoder using eq. (12). The log-liiclihood is computed on held-out test data by numerically integrating the 2D latent space of the VAE.
Algorithm Shared computation the DeMiss VAE learning objectives
Input: parameters 0 and Φ, number of latent samples L, completed data-point (x) Fmis))
tha Encoder (arg x misi ②)
ik
Compute parameters of the variational distribution
21,..., 21 q(2 oos: Disi 1b5k)
Sample latents z
+ Decoder 0) for VI E [1,
Compute parameters of the generative distribution
4: def (
m.... 7L):
Procedure for estimating eq. (10)
return + Σ log p0g ZG, m))
def CIMVA (z6th 21...., 21, ....., 71):
Procedure for estimating eq. (12)
7: return 7, Et log p( Joos x 21; m1) log 9(2 xo sk 2b*h)
return (2. Z1 m...... mc). LLMVE( (zbth 22....., ZL ....., 71)
use cheap approximate iterative sampling methods such as pseudo-Gibbs (Rezende et al., 2014, Appendix F), liith-olls-iithnn-iibbss (MWG, Mattei & Frellsen, 2018a), or latent-adaptive impor- tance resampling (LAIR, Simkus & Gutmann, 2023). Moreover, since the model and the variational distributions are initialised randomly, we skip the imputation step during the first epoch over the data.
c Parameter update. Update the parameters using stochastic gradient ascent on Ce and LMVB eqs. (10) and (12) with the imputations from I (((miss Robs).
Efficient parameter update. While the two objectives for Pe and ds in eqs. (10) and (12) are different, major part of the computation can be shared, as shown in algorithm 1. As usual, the objectives are approximated using Monte Carlo averaging and require only one evaluation of the generative model, including the encoder, decoder, and prior, for each completed data-point (#) =the) Therefore, only backpropagation needs to be performed separately and the overall per-ileration computational cost. of optimising the two objectives is about 1.67 times the cost of a fully-observed VAE optimisation (instead of times if implemented naïvely). 10
Efficient imputation. To make the imputation step efficient, the imputation distribution f' Fobs) is "persitent" between iterations, that is, the imputation distribution from the previous iteration
10 of backpropagation ahout 2 times the cost forward pass (Burda al., 2015).
20

---

Published in Transactions on Machine Learning Research (06/2024)
( Tobs)) is used to initialise the iterative approximate VAE sampler iteration t." Moreover, iteration of pseudo-Gibbs, MWG, or LAIR samplers uses the same quantities the objectives Levi and LMVB in eqs. (10) and (12), and hence the cost of iteration of the sampler in the imputation step can be shared with the cost of computation of the learning objectives. However, it is important to note that the accuracy of imputations affects the accuracy of the estimated model, and hence better estimation can be achieved by increasing the computational budget for imputation or using better imputation methods.
Persistent samplers have heen used in the past increase efficiency of maximum-llae -ikelluumlikelhhodd estimation methods (Younes, 1999; Tieleman, 2008; Simkus al., 2023).
21

---

Published in Transactions on Machine Learning Research (06/2024)
E
Experiment details
In this appendix we provide additional details on the experiments.
E.1 Mixture-of ofaerr--f-GGuusian data with 2D latent VAE
We generated a random 5D mixture-of- muiuuuae-o-GGussian model with 15 components by sampling the mixture Gaussians covariance matrices from the inverse Wishart distribution W- = D, V I), means from the Gaussian distribution N(, = 0 = 3) and the component probabilities from Dirichlet distribution Dir(a = 1) (uniform). The model was then standardised to have zero mean and standard deviation of one. The pairwise marginal densities of the generated distribution is visualised in fig. 7 showing highly-compexx and multimodal distribution, and the generated parameters and data used in this paper are available in the shared code repository. We simulated a 20K sample data set used to fit the VAEs.
dim=1
dia-=2
dim-=3
dim=
cirr-O
dim
dim-2
dim-3
Figure 7: The pairwise marginals of the ground-truth Mixture-of-Gaussians dist ion.
We then fitted a VAE model with 2-dimensional latent space using diagonal Gaussian encoder and decoder distributions, and fixed standard Normal prior. For the decoder and encoder networks we used fully- connected residual neural networks with residual blocks, 200 hidden dimensions, and ReLU activations. To optimise the model parameters we have used AMSGrad optimiser (Reddi et al., 2018) with lcarning rale of 10-3 for total of 500 epochs.
The hyperameters are listed in table 3, note that the total number of samples was the same for all methods (i.c. 5/15/25). Morcover, we have used gradient variance for all methods." 12
(STL) gradicnts (Rooder et al., 2017) to reduce
We have also evaluated the rembly--eparmmettiied gradients (DReG, Tucker al., 2018) for IWAE methods but found STL perform similar better.
22

---

Published in Transactions on Machine Learning Research (06/2024)
Method
2
Mixture sampling
MVAE
5/15/25
1
Miss
5/15 5/15/25
Ancestral
MissSVAR
1
5/15/25 1
Stratified
MIWAE
1
5/15/25 5/15/25
MissIWAE
1
5/15/25
Ancestral
MissSTWAE
5/15/25
Stratified
LAIR (1 iteration, 0)
DeMiss
5/15/25
(Simkus & Gutmann, 2023)
Table 3: Method hoperpparameters on MoG data.
E.2 UCI data sets
We fit the VAEs four data sets from the UCI repository (Dua & Graff, 2017) with the preprocessing of (Papamakarios et al., 2017). The VAE model uses diagonal Gaussian encoder and decoder distributions regularised such that the standard deviation 2 10 (Mattei & Frellsen, 2018a), and a fixed standard Normal prior. The latent space 16-dimensional, except for the MINBBOONE where 32 dimensions were used.
The encoder and decoder networks used fully-connected residual neural nelworks with residual blocks (except for on the MINIBOONE dataset where blocks were used in the encoder) with 256 hidden dimen- sionality, and ReLU activations. A dropout of 0.5 was used on the MINIBOONE dataset. The parameters were optimised using AMSGrad optimiser (Reddi et al., 2018) with learning rate of 10 and cosine learning rate schedule for total of 200K iterations (or 22K iterations on MINIBOONE)) As before, STT gradients (Roeder et al., 2017) were used to reduce the variance for all methods. DeMiss used the LAIR sampler (Simkus & Gutmann, 2023) with K = 5 R = 1 and 10 ilerations. Moreover we have used gradient norm clipping to stabilise DeMissVAE with the maximum norm set to 1 (except for POWER dataset where we set it 0.5).
E.3 MNIST and Omniglot data sets
We fit a VAE on statically binarisod MNIST and Omniglot data sets (Lake et al., 2015) downsampled to 28x28 pixels. The VAE uses diagonal Gaussian decoder distributions regularised such that the standard deviation 2 10 (Mattei & Frellsen, 2018a), a fixed standard Normal prior, and a Bernoulli decoder distribution. The latent space is 50-dimensional.
For both MNIST and Omniglot we have used convolutional ResNet neural networks for the encoder and decoder with 4 residual blocks, ReLU activations, and dropout probability of 0.3. For MNIST, the encoder the residual block hidden dimensionalitics were 32, 64, 128, 256, and for the decoder they were 12,,34,3 For Omniglot, the encoder the residual block hidden dimensionalitics were 64, 128, 256, 512, and for the decoder they were 256, 128,64,64. We used AMSGrad optimiser (Reddi et al., 2018) with 10-4 learning rate, cosine learning rate schedule, and STL gradients (Roeder et al., 2017) for 500 epochs for MNIST and 200 epochs for Omniglot.
For MVAF, we use 5 latent samples and for MIWAF we use 5 importance samples. For MissVAR we use K = 5 mixture components and sample 5 latent samples. For MissSVAE we use K = 5 mixture components and sample 1 sample from each component, for a total of 5 samples. For MissIWAE we use K = 5 components and sample 5 importance samples, for MissSIWAE we use K = 5 components and sample sample from each component. For DeMiss we use K imputations and update them using single step of pscudo-Gibbs (Rezende et al., 2014).
23

---

Published in Transactions on Machine Learning Research (06/2024)
F Additional figures
In this appendix we provide additional figures for the experiments in this paper.
F.1
Mixtura-of-Gaussians data with a latent VAE
In this section we show additional analysis on the Cextuure-f-GGussianne data, supplementing the results section 6.1.
F.1.1 Analysis of gradient variance with ancestral and stratified sampling
In section 6.1 we observed that the model estimation performance depend whether ancestral sampling (with implicit reparametrisation) or stratified sampling is used to approximate the expectations in eqs. (4);
(5), (7) and (8), corresponding to Miss VAE/
and MissSVAE/ WA,,,,, respectively.
We analyse the signal-to-noise ratio (SNR) of the gradients w.r.t. & and for the two approaches, which is defined as follows (Rainforth al., 2019)
SNR(Ø) = E [ (}) and SNR(0) = o (()
E [
where A(-) denotes the gradient estimate, and of.] is the standard deviation of random variable. We estimate the SNR by computing the expectation and standard deviation over the entire training epoch.
The SNR for & and 0 is plotted in fig. 8. We observe that the stratified approaches (MissSVAE and MissSIWAE) generally have higher SNR. This possibly the reason why MissVVAE and MissSIWAE have achieved better model accuracy than the ancestral approaches (MissVAE and MissIWAE) in section 6.1.
Ms&VAE
MisSVVRR
Miss/WAF.
MissSIWAR.
1200
0100
SNR
000
6000
0.00
130
.000
300
2000
Training epoch
Training epoch
Figure 8: Signal-to-noise ratio (SNR, higher is better) of the gradients w.r.t. encoder parameters & (left) and decoder parometers 0 (right). For all methods we used a budget of 5 samples from the variational distribution (see appendix E.1 for more details). We show the median SNR over independent runs and a 90% confidence interval.
F.1.2 Analysis of the model posteriors
In figs. to 11 visualise the model posteriors with complete and incomplete data, pe(= x)) and (e Bobs), respectively, and the variational distribution (z(z |) that was used to fit the model via the variational ELBO. For each method we have used budget of 25 samples from the variational distribution during training (additional details are in appendix 1. Each figure shows the posteriors for 5 training data-points using distinct colours.
Figure 9 shows MVAE, VAE, and MissSVAE model posteriors po(z x)) and pe(= Fobs), well the variational distribution (s(2) Dobss; which approximates the incompleteddata posterior. As motivated
24

---

Published in Transactions on Machine Learning Research (06/2024)
in section 3 observe that the Gaussian posterior in MVAE (first row) is not sufficiently flexible to ap- proximate the complex incomplete-data posteriors po(= Tohs). On the other hand, the mixture-variations approaches, VAE (second row) and MissSVAE (third row), are able to well-approximate the incomplete- data posteriors.
Figure 10 shows MIWAF, MissIWAF,, and MisSSTWFF, model posteriors po(= | .) and po(z | Roha), as well as the variational proposal 9 (z Tobs) and the importance- weighted semi-implicit distribution g (0.0,1-25(2 (= Lohs) that arises from sampling the variational proposal 9/(2 Tohs) and re-sampling using importance weights w(z) Pe(Zobs <)/ I Tobs) (Cremer et al., 2017). Similar to the MVAE case above, the variational proposal (2 Tobe)) in MIWAE (first row) is quite far from the model posterior po(z Tobn), but the sepooraanceweighlle bound in eq. (7) is able to re-weigh the samples to sufficiently-wll match the model posterior, as shown in the fourth column. However, as efficiency of importance sampling depends on the discrepancy between the proposal and the target distributions, we can expect that more flexible variational distributions may improve the performance of the importance-weighted ELBO mothods. Importantly, we show that the variational-mixtuu approaches, MissIWAE (second row) and MissSIWAE (third row), are able to adapt the variational proposals to the incomplete-data posteriors well, and result achieve better efficiency than MIWAE.
Figure 11 shows DeMissVAF model posteriors pa(= x) and pa(z | Tohx), the variational distribution (0(2 x), which approximates the complete-data posterior, and the iuputation-mixtune fe Tobs) approximated using the 25 imputations in f' | (oba)) at the end of training. We observe similar behaviour to fig. 1, where the complete data posteriors pe æ)) are close to Gaussian but the incomplete-data posteriors po Tobs) are irregular. As we show in scction 6.1, DeMissVAE is capable of fitting the model well by learning the completed-dala variational distribution (0(2 | x) (third column) and using the imputation- mixture in eq. (9) to approximate the incomplete data posterior po Tobs). Moreover, we observe that the imputation-mixture f (z Dabs) (fourth column) captures only one of the modes of the model postcrior po Bobs). This is a result of using a small imputation sampling budget, that is, using only a single iteration of LAIR to update the imputations (see more details in appendix D), and hence better accuracy can be achieved by trading-off some computational cost to obtain better imputations that would ensure a better representation of the imputation distribution. Nonetheless, as observed in fig. 2, DeMissVAE achieves good model accuracy despite potentially sub-optimal imputations, further signifying the importance of the two learning objectives for the encoder and decoder distributions in section 4.2 and appendix C.
Interestingly, by comparing the complete-data posteriors po(z x) (first column) in figs. 9 to 11, ob- serve that they are slightly more irregular than in the complete case in fig. 1, except for DeMissVAE whose posteriors are nearly Gaussian. The irregularity is stronger for the importancc-weighted ELBO-based meth- ods in fig. 10. This is in line with the observation by Burda et al. (2015, Appendix C) and Cremer et al. (2018, Section 5.4) that VAEs trained with more flexible variational distributions tend to learn a more complex model posterior. This means that using the importance-wer weighted bounds, and to a lesser extent the finite pariational-mixture approaches from section 4.1, to fit VAEs on incomplete data may result in worse-structured latent spaces, compared to models fitted on complete data. On the other hand, we observe that DeMiss learns better-structured latent space, with the posterior pa(= x) close to Gaussian, that is comparable to the complete case. This suggests that the decomposed approach in DeMissVAE may be important in cases where the latent space needs to be regular, at the additional cost of obtaining missing data imputations (see appendix D).
25

---

Published in Transactions on Machine Learning Research (06/2024)
pix
piz
Xala
g/z
MIVAL
Mass XIE
MisSVAE
Figure 9: Posterior distributions of MVAE, Miss VAE, and MissSVAE. First column: the model posterior Pz x) under complete data a. Second column: the model posterior pe(z Tobs) under incomplete data Bohs- Third column: variational approximation (2 Dobs) of the incomplete posterior (z(2 Labs) obtained at the end of training.
26

---

Published in Transactions on Machine Learning Research (06/2024)
x
p Nuha
gia
X.w
0/-25 12
Xxta)
MIWAE
MissWAE
MisSSWAA
Figure 10: Posterior distributions of MIWAF, MisslWAF, E, and MissSIWAE. First column: the model pos- terior pe(z x) under complcte data z. Second column: the model posterior po | æobs) under incomplete data Cohs- Third column: variational proposal 4g(2 2 Wohs) for an incomplete data-point Dohs obtained at the end of training. Fourth column: importanco-weightet variational distribution 40r=25 (z Fobs) for incomplete data-point Lobs obtained after re-weighting samples from (x(2 Dobs) (Cremer et al., 2017).
piz
Xala]
q(z x)
golz Xain)
DeMiss VAE
Figure 11: Posterior distributions of DeMiss VAE. First: the model posterior pe( x)) under complete data
æ. Second: the model posterior Po(z Tobs) under incomplete data Lobs- Third: variational approximation (z(2 æ)) of the complete-data posterior po(z æ) obtained the end of training. Fourth: the variational
imputation-mixttue distribution in eq. (9) using the imputation distribution f'(amis Tobs) obtained at the end of training, approximated using a Monte Carlo average with 25 imputations.
27

---

Published in Transactions on Machine Learning Research (06/2024)
F.2 UCI data sets
In fig. 12 we plot the Fréchet inception distance (FID, Heusel et al., 2017) versus training iteration on the UCI datasets. The results closely mimic the log-likelihood results in section 6.2. Importantly, observe that using mixture variational distributions becomes more important as the missingness fraction increases, causing the posterior distributions to be more complex.
MVAE
MIWAE
DeMisVAE
MaWVAR
MisIWAR
McSSVAE
Missingness
20%
50%
80%
112
GAS
182
10-12
10"1-
00039
150000
200000
5006
10000
12000
330000
56000
000001
13000n
200000
102-
107
57
10"1
POWER
11
10.
10/
:211
102
cane
000.0
180000
1040.0
20.000
58800
10000
130000
230.00
000
00000
130030
20000
501
HEPMASS
10-1
11P
50030
UX0099
150000
200000
33000
100K0
13000
23000
50000
500091
150030
20000
112
MINIBONNE
21A
11/
10-3
1990
1500
3300
1000
10000
10000
20000
95.0
10300
1900
2000
Iteration
Exranion
Iteration
Figure 12: FID (lower is better) between the model and the complete test data versus training iterations. The FID is computed using features of the last encoder layer of an independent VAE model trained on complete data. Lines show the median of 5 independent runs and the intervals show 90% confidence.
28

---

Published in Transactions on Machine Learning Research (06/2024)
F.3 MNIST and Omniglot data sets, latent dimensionality
Rather than handling the posterior complexity due to missingness with varialional mixtures, an alternative approach may be to increase the capacity of the model by making the latent space dimensionality larger. In fig. 13 we plot the estimated test log-likelihood against latent variable dimonsionality.
On MNIST data, the effect from increasing the latent space is small, with the exception DeMissVAB. The reason why DeMiss VAE improves quite significantly with latent space dimensionality may be more due easier sampling than larger capacity of the model. This again highlights that DeMissVAE, when used with efficient sampling methods, can be an efficient approach to handle data missingness.
On Omniglot data, we observe a small improvement for MIWAE, MVAE, and the stratified MissSIWAE, while the other methods either remained at about the same accuracy or declined. However, there is no significant change from the results section 6.3.
Hence, while increasing latent dimensionality generally increases the capacity of the model, cnabling the modelling of more complex distributions, overall observe that it provides minimal effect for dealing with data missingness.
MVAE
MIWAE
DAM
Mis
MisiWAE
MisSVAE
MisSIWAE
MNIST
Omniglot
55
50
-120
Trog:Blothhord
-84
-06
-6
-136
Latent dineensionality
Latent diinensiounlity
Figure 13: Estimate of the test logliikel against latent dimensionality MNIST and Omniglot data sets. The log-likelihood was estimated on complete test data using the IWELBO with 7 = 1000. The curves show median performance over 5 independent runs of the methods and the intervals show the 90% centered interval.
29


Reviews:
[{"review_id": "o7kPBOCvCL", "reviewer": "Reviewer_ATvF", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strength:\n\n- learning latent variable model under the situation where the observed data is incomplete is an interesting task\n- the two proposed methods are reasonable, though some of the details need to be further justified\n\n\nWeakness:\n\n- The relationship between the hierarchical latent variable model needs to be further justified.\n\nThe missing piece of the data itself can be the latent variable. In this aspect, we can either treat $\\tilde{z} = [z, x_{mis}]$ and use the vanilla latent variable model framework like VAE for training. Or we can treat it in a hierarchical way, i.e., modeling both $p(x_{mis}|x_{obs})$ and $p(z|x_{obs}, x_{mis})$ and the factorization here incorporates the domain knowledge on the different roles of the two missing pieces ($x_{mis}$ and $z$). \n\n- Regarding the mixture model, the optimization technique needs further justification.\n\nIn the mixture model formulation, if the number of mixtures is not large, why can't we simply do explicit integration (i.e., summation over K)? I'm not sure if the sampling is ever needed here. In this way we can also avoid back-propagating through non-differentiable categorical distribution.\n\n- The experiments are mostly on toy datasets. Some more interesting datasets should be presented (e.g., coco with missing image segmentations, or imagenet with missing class labels)\n\nOr some more interesting tasks where the missing data itself is not artificial. \n\n- Stronger baseline methods need to be included.\n\nConceptually, the limitation of the assumption on the variational posterior (i.e., factorized gaussian) limits the expressiveness of the approximation. If we can include some baselines that have better expressiveness in the parameterization of variational posterior, then that would be more convincing.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper proposed two methods to improve the expressiveness when learning the latent variable model under the incomplete data. The two methods are: 1) a mixture model with finite number of mixtures to parameterize the variational posterior; 2) an imputation model that completes the observation first, and then parameterize the variational posterior given the completed data. The paper has performed experiments on synthetic data and real-world UCI dataset and minist and omniglot dataset. Compared to some existing baselines, the paper has shown better results.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Please see the suggestions in the weakness section.", "broader_impact_concerns": "no concerns.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "JKrLj1qqA1", "reviewer": "Reviewer_xSek", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "STRENGTHS: Many modern methods are discussed and analysed. In simulations, high dimensional problems can be addressed. The paper seems carefully written, and I did not find typos. \n\nWEAKNESSES: The paper is numerically-oriented. There are no long proofs or theoretical guarantees. In my opinion this is not really a problem, since a paper can be numerically-oriented in essence. \nIn the various figures, the performance benefit compared to the baseline MIWAE is not clear (Figs 2 and 3).", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper provides a methodological/numerical contribution to variational auto encoders under missing data. It provides 5 new methods based on mixtures and imputation methods. The new methods are justified heuristically in details. They are compared to two existing methods in a wide range of simulations. The related literature is carefully discussed.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Perhaps developing the benefit compared to MIWAE.\n\nOn page 3, \"(2) may not minimized\". Could it be phrased as \"(2) cannot be set to zero\"? It seems any function, such as (2), can be minimised with respect to its parameters, even if the minimal value is not zero?", "broader_impact_concerns": "I do not have broader impact concerns.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "sILXWUPCeh", "reviewer": "Reviewer_WraX", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Overall I think the paper is worth accepting to TMLR and that some readers may find it useful (ideally with some clarifications).** The method itself is not new (using a mixture of variational posteriors, data imputation before inference) but the specific discussion of the difficulty of variational inference with missing data is interesting. In particular, it raises questions as to the appropriate way to handle missingness, and whether it is sufficient to just \"scale up the posterior family\" or if additional structure can be leveraged---particularly when the form of such structure may be available to domain experts ahead of time.\n\n**My two main suggestions are to improve the clarity of the writing at certain points and to expand on the somewhat limited experiments.**\n\n## Regarding writing clarity, there are quite a few segments that could be made much easier for readers to follow:\n\nSection 2 should be much clearer about a lot of definitions, variable domains, etc. For example,\n* In the first paragraph, what is $D$? I guess it must be the data dimension? It looks like it's not defined until the end of the section.\n* Is $m^i$ itself observed? The paper describes methods which feed $m^i$ as input, but then later in the paper describes the function $f_t$ as imputing conditioned on only $x^i_{obs}$, presumably without $m^i$.\n* Similarly, how can it be that MAR is equivalent to $p(m^i \\mid x^i) = p(m^i \\mid x^i_{obs})$? Wouldn't $p(m^i \\mid x^i_{obs})$ have zero mass on any $m^i$ with 0 in the dimensions of $x^i_{obs}$? If the $x^i_{obs}$ are the values in $x^i$ which are observed, then they can only be defined *after* $m^i$ is chosen, no? This confusion could be addressed by more precisely specifying these definitions.\n\nThe setup for Figure 1 is not explained **at all**. What each of the plots depicts is clear, but how are these results generated?\n\n* What are the domains/distributions of $x$ and $z$? What is the missingness distribution? Does it match the \"mixture\" assumption made by the method in the rightmost plot? How would it look when it does not?\n* At the beginning of Section 4.1 the authors write: “In section 3 we saw that a good approximation of the incomplete data posterior $p_\\theta(z \\mid x^i_{obs})$ would be the imputation-mixture”. Isn't this circular reasoning? The rightmost plot of Figure 1 is being used to motivate a new method... which is the method being depicted in Figure 1? Ultimately this seems to just be saying \"using a variational posterior family which gets closer to the true posterior can do a better job\", which is obvious...\n\nSimilarly to the first sentence of 4.1, the first sentence after Eq (9) is extremely speculative---there is *some* reasoning offered closer to the end of that section, these should be connected. **Throughout the paper, whenever a claim is made about what should or shouldn't work and why, it's up to the authors to offer justification.**\n\n## Regarding the somewhat limited experiments\n\nIt's not too surprising that the existing MIWAE is equal/better in most cases, since as I mentioned the objectives discussed in this work don't do anything explicitly new, instead applying existing ideas to missingness (a worthwhile application!). However, I am somewhat disappointed that the experiments are so simplistic. There are a lot of ways this could be improved, off the top of my head:\n\n* Why not use more complex real-world data, instead of, e.g., GAS/MNIST/Omniglot?\n* Why are all the missingness patterns so basic (e.g. very often *explicitly* inducing mixture posteriors, such as randomly masking two quadrants of an image), rather than more interesting distributions?\n* Why not evaluate in settings when the data is *not* MAR, to determine how important that assumption is to each of the methods?\n* Does the relative ranking of these methods change if you use a more complex (i.e., larger) encoder/decoder? Explicitly structuring the posterior is of course helpful, but simply choosing a more flexible family could also potentially work. How do these trade off? (In particular, were the experiments on more complex data I would understand if this were computationally costly; but for such simple data, these types of experiments would still be pretty trivial).\n\nNot all of these suggestions would need to be implemented, of course, but these experiments feel a bit like just checking boxes rather than truly exploring the interesting problem of variational inference under missingness. I think investigating a bit more deeply would make the paper much more informative overall.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper studies variational inference with a VAE when parts of the observations are missing at random. They point out that this can make the correct form of the posterior more complex, necessitating more suitable variational forms. They consider mixtures of variational posteriors and posteriors which first estimate and sample imputations before doing standard variational inference.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "See above", "broader_impact_concerns": "None", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The two methods introduced in the paper have not been considered before for VAEs with incomplete data, to the knowledge of the reviewers and myself. Therefore, we believe that the paper is a valuable addition to the literature, despite its limited experiments.

The initial submission lacked some details, but the paper's writing has been improved during the rebuttal. However, I feel there is still room for improvement. In particular, the update of the imputation distribution is very unclear to me. On page 20, the authors mention methods such as pseudo-Gibbs (Rezende et al., 2014), Metropolis-within-Gibbs (MWG, Mattei & Frellsen, 2018a), and latent-adaptive importance resampling (LAIR, Simkus & Gutmann, 2023), but I did not understand their relevance since these are sampling algorithms. Furthermore, the target distribution is not specified. Therefore, I require that the authors significantly clarify this part of the paper before final acceptance.
Decision: Accept (poster)