Conditional Sampling of Variational Autoencoders via Iterated Approximate Ancestral Sampling
Authors: Vaidotas Simkus
Michael U. Gutmann
Venue: Accepted by TMLR ()

Abstract: Conditional sampling of variational autoencoders (VAEs) is needed in various applications, such as missing data imputation, but is computationally intractable. A principled choice for asymptotically exact conditional sampling is Metropolis-within-Gibbs (MWG). However, we observe that the tendency of VAEs to learn a structured latent space, a commonly desired property, can cause the MWG sampler to get “stuck” far from the target distribution. This paper mitigates the limitations of MWG: we systematically outline the pitfalls in the context of VAEs, propose two original methods that address these pitfalls, and demonstrate an improved performance of the proposed methods on a set of sampling tasks.

Published in Transactions on Machine Learning Research (11/2023)
Conditional Sampling of Variational Autoencoders via Iterated Approximate Ancestral Sampling
Vaidotas Simkus
cridotas.simkussee ac. uk michael.gutmane www.ggttmaammee.......m wed.
Michael U. Gutmann
School of Informatics
University of Edinburgh
Reviewed on OpenReview: openrevvee.. net/ formm? id= I5sJ6UU6JJ
Abstract
Conditional sampling of variational autoencoders (VAEs) is nceded in varions applications, such as missing data imputation, but is computationally intractable. A principled choice for asymptotically exact conditional sampling is Metropolis- within-Giibiei (MWG). However, we observe that the tendency of VAžs learn a structured latent space, a commonly desired property, cause the MWG sampler to get "stuck" far from the target distribution. This paper miligales the limilations of MWG: we systematically outline the pitsalls in the context of VAEs, propose two original methods that address these pitfalls, and demonstrate an improved performance of the proposed methods on a set of sampling tasks.
1 Introduction
Conditional sampling of modern deep probabilistic models is an important but generally intractable prob- lem. Variational autoencoders (VAEs, Kingma & Welling, 2013; Rezende et al., 2014) are a family of deep probabilistic models that capture the complerity of real-world data distributions via structured latent space. The impressive modelling capability and the usefulness of the structured latent space make VAEs model of choice in broad range of domains from healthcare (Han et al., 2019) and chemistry (Gómez-Boo Bombarelli et al., 2018) to images (Child, 2021) and audio (van den Oord et al., 2017). Ancestral sampling can be used for efficient unconditional sampling of VAEs, but many downstream tasks, for example, prediction or missing data imputation (e.g- Goodfellow al., 2016, Chapter 5.1.1), instead require condilional sampling. However, for Vs this intractable, and hence approximate methods are needed.
A canonical approximate method is Markov chain Monte Carlo (MCMC, c.g. Barber, 2017, Chapter 27.4) but the general lack of knowledge about the learnt VAE may make tuning, for example, picking a good proposal distribution, and hence successfully using MCMC samplers challenging. To make sampling easier, an approach called Metropolis-wittinn-ibbbb (MWG, Mattei & Frellsen, 2018) re-uses the encoder, an auxiliary component from the training of the VAE, to construct a suitable proposal distribution Hastings-type algorithm (Metropolis al., 1953; Hastings, 1970). The simplicity of MWG and its asymptotic convergence guarantees make it a compelling choice for conditional sampling of VAEs.
a Metropolis-
While structured latent space is often desirable property of VAEs, enabling the modelling of complex distributions, we notice that this latent structure can cause the Markoe chains of MWG to get "stuck" hence impeding conditional sampling. In this paper we
Detail the potential pitfalls of Metropolis within-Giiss in the context of VAEs (section 3).
Propose a modification of MWG, called adaptive Metpeeddd-Merrools-wtii------ee (AC-MWG, section 4.1), that mitigates the outlined pitfalls and prove its convergence.
Introduce an alternative sampling method, called latent-adaptive importance resampling (LAIR, section 4.2), which demonstrates an improved sampling performance in our experiments.

---

Published in Transactions on Machine Learning Research (11/2023)
Evaluate the samplers set of conditional sampling tasks: (semi-)synthetic, where sampling from the ground truth conditional distributions is computationally tractable, and real-world missing data imputation tasks, where the ground truth distribution is not available.
With the proposed methods we address the conditional sampling problem of VAEis, a key challenge to downstream application of this flexible family of models. Our methods build and improve upon the limitations of MWG enabling more accurate use of VAFs in important tasks like missing data imputation.
2 Background: Conditional sampling of VAEs
We here describe the conditional sampling problem and the existing Gibbs-like methods that have been used to draw conditional samples.
2.1 Problem and assumptions
Given a pre-trained variational autoencoder, whose generative model we denote as p(x; x) = p(æ x))(();
where x (Fabs; Fmis) are the visible and z are the latent variables, we would like to sample:
p Bmiss dz dz pl p(Dobs))
= f
p(mmis| Koba)
p( p( Tosaa z)p(z @oohs dz.
(1)
The variables amis and Tobs are respectively the target/ target/missingg /missing and conditioning observedd variables. This choice of notation is motivated by the correspondence between conditional sampling and probabilistic im- putation of missing data (Rubin, 1987: 1996). Unlike unconditional generation, ancestral sampling of p(aris FFbs)) generally intractable since the posterior distribution p(z @bbb)) is not accessible and hence approximations are required.
/obserrven
In the rest of the paper assume that the generative model is such that computation of p( =) and sampling of p(amis Dobs: z) is tractable. This is typically the case for most VAE architectures due to conditional independence assumptions (i.e. Iy IL æzj z for all Vj) or the use of Gaussian family for the decoder distributtion p(æ: z). Moreover, we assume that the encoder distribution, or the amortised variational posterior, q(= x) (Gershman & Goodman, 2014) which approximates the model posterior p( a)) available.?
2.2 Pseudo-Gibbs (Rezende et al., 2014)
Rezende et al. (2014, Appendix F) have proposed a procedure related to Gibbs sampling (Geman & Geman, 1984), also called pseudo-Gibbe Gibbs (Heckerman et al., 2000; Mattei & Frellsen, 2018), that due to its generality and simplicity has been regularly used for missing data imputation with VAËs (e.g. Rezende et al., 2014; Li et al., 2016; 2017; Rezende et al., 2018; Boquet et al., 2019). Starting with some random imputations x the procedure iteratively samples latents zt 7(> Fohs: ) and imputations ar'ia p(# Fobs z)).² This iterative procedure generates Markov chain that subject to some conditions on the closeness of the variational posterior q(z Lobs, Emis) and the intractable model postcrior p(z Lobar Fmis)) converges asymptotically in t to a distribution that approximately follows p( Tobs) (Rezende et al., 2014, Proposition F.1). The sampler corresponds to an exact Gibbs sampler z Tobs: Fmiss = p(z Babs, Fmiis)
However, the equality q(> Fabs; @mis) ) = p(> Tobs Ømis) generally does not hold due to at least one of the following issues: insufficient flexibility of the variational distributional family, amortisation gap, or inference generalisation gap (Cremer et al., 2018; Zhang et al., 2021). Hence, pseudo-Gibbs sampling may produce sub-optinal samples even in the asymptotic limit or completely fail to converge due to incompatibility of g(= Lobs, miis) and p(mmis Rohs, z).
Equation (1) corresponds directly to missing data imputation with missing-at-random (MAR) missingness pattern. 2The variational posterior is typically available after fitting the VAE complete data using standard variational Bayes (Rezende al., 2014; Kingma et al., 2014), 4), or can be fitted afterwards using real generated complete data set. SSuperseript represents the sampler iteration.
2

---

Published in Transactions on Machine Learning Research (11/2023)
2.3 Metropolis wwiithin-Gibbs (Mattei & Frellsen, 2018)
Mattei & Frellsen (2018, Section 3.2) have proposed a simple modification of the pseudo Gibbs sampler that can asymptotically in generate exact samples from p(amia Robs). The method incorporates Metropolis- Hastings accept-reject step (Metropolis et al., 1953; Hastings, 1970) to correct for the mismatch between q( Tobs: mmis) and p( Zobs: Tmis) followed by sampling from p(Zmis| Tobs, 2), hence yielding a sampler in the Setrrpolis-within-GGe iteration it generates the proposal sample z ~ q(>
(MWG) family (Gelman & Rubin, 1992 Section 4.4). Specifically, at each
Robs: almi)) and accepts it as x' = with probability
(26 2 11 x = min {1, p( mis 3)p(3)
g(='11 Tohs, x pmmi
(2)
D( a/iis 2 p)(=t11)) q(z Tobs: 2mmss
If the proposal E is rejected, the latent sample from the previous iteration is used, so that x' = ==-1 Given z", a new imputation ahis is then sampled as in standard Gibbs sampling: a' p(aEmis Rohss z¹). By incorporating the Metropolis Hastings acceptance step, the pseudo-Gibbs sampler is transformed into an asymptotically exact MCMC sampler with p(Imiss 2 Tobs) as stationary distribution oven if q(> Bobs, Fmis) + p(= oobs: miis).
Importantly, as noted by the authors, the asymptotic exactness of MWG comes, compared to the pseudo- Gibbs sampler, at little additional computational cost in cach iteration: the quantities required for computing p' are also computed in the aseudo-Gibbs sampler, except for the often cheap prior evaluations p(z).
In summary, MWG has several desirable properties which make it an attractive choice for conditional sam pling of VAEs: (i) it provides theoretical guarantees of convergence to the correct conditional distribution, (ii) it is simple to implement, and (iii) its per-iteration computational cost is relatively small, i.. one stan- dard evaluation of a VAE, and comparable to the cost of pscudo-Gibbs. However, as we will see next. MWG is not free of important pitfalls.
3 Pitfalls of Gibbs-like samplers for VAEs
Although the Gibbs-like samplers from sections 2.2 and 2.3 are often used to conditionally sample from a VAE model, the structure of the latent space can cause poor non-asymptotic sampling behaviour. We here detail, in a form of three pitfalls, how this structure can affect the aforementioned samplers. While the reported pitfalls are related to the known limitations of the classical Gibbs (Geman & Geman, 1984) and Metropolis-withe within-Gibbs samplers (Gelman & Rubin, 1992), we here work out their significance in the context of VA In fig. 1 we excmplify these pitfalls in an archetypical scenario using a synthotic 2-dimensional VAE model (for details about the model see appendix C.1). 4 The proposed methods in the following section, AC-MWG (section 4.1) and T.AIR (section 4.2), provide remedies for the reported pitfalls.
Pitfall I. Strong relationship between the latents and the visibles can cause poor mixing. We often train VAEs to learn structured latent space that captures the complexity of the data. This is typically achieved by using a decoder with simple, often conditionallyinddeeeneee, distribution. For example, to fit binarised MNIST data set well with a Bernoulli decoder distribution p(x x) = II Bernoulli(z, z), the digits in the image space must be well-represented in the latent space and the variance of the decoder must be nearly 0, otherwise the model would produce noisy samples due to random "flips" of the pixels. Hence, in VAEs with simple decoders the complexity of modelling the visibles æ is often converted to learning a complex structure in the latent space along with a near -eteemmtriiiiii mapping between the latents z and the visibles æ as given by the decoder p(= =)) But this strong, near-deterministic, relationship can substantially inhibit the convergence and mixing properties of sampler like Metropolis Writhin-Gibbar This is because the proposed samples z ~ g(z) will be rejected with a high probability if the conditional distribution p( Loba, ž) x p(Zobar æiis| 2) places little density / /ass on the previous value of &mis = = as a small value of p(Zobe: 1-1 ž) will make the Metropolis-Hastinng
acceptance probability in eq. (2) small. This small acceplance probability leads to Markov chains that get "stuck" " a mode and prevents the sampler
We note that the variational distribution q(z | 2) in this section constructed be slightly wider than the model conditional p(= a) to differentiate the different modes of failure.
3

---

Published in Transactions on Machine Learning Research (11/2023)
Pitfall
Pitfall II
Pitfal III
4-
0.8
True VAE
3-
Psendo-Git
MWG
0.4
p( 7 (ota))
2
Tmis
Tuiss
Figure 1: Pitfalls of Gibbs-like samplers for VAE models. (The figure is best viewed in colour.) Each panel corresponds to distinct sampling problem, where the the observed variable Joobe (2o, is, from left to right, æ1 = 0, To = 0, and æ1 1. The line plots show the ground truth density p(zmis Fobs) (bluc) and the density of the samples obtained from the two Gibbs-like methods, pseudo-Gibbs (orange) and MWG (pink). The contour plot shows the conditional joint density p(Zmis, Iobs) of the VAE model over the missing variable Tmis (bottom axis) and the latent (right axis), and the dashed green curve shows the expectod value of z given Zg and T1- Both samplers were initialised with the same state and run for 50k iterations. Left: MWG fails to mix between nearby modes (in the space of z; right axis) due to high rejection probability in eq. (2). Center: both eseudo-Gibbe and MWG fail to find modes that are far apart (in the space of 2; right axis) due to narrow proposal distribution. (We note that MWG and pseudo-Gibbs lines overlap in this plot.) Right: poor initialisation may leave MWG "stuck" lar from the target distribution. Appendix D.1 contains an additional view of the pitfalls.
from moving to nearby modes that are close in the latent space. We illustrate this pitfall in fig. 1 (left). In this example, MWG (pink) fails to mix between the modes that are close in the space of latents. This failure occurs despite the proposal distribution generating samples from the neighbouring modes because such proposed samples are rejected by the Metropolis-Haan Hasting step. On the other hand, pseudo-Gibbs (orange) can mix between the modes since it does use the Metropolis Hastings step.
HaatopolisHHastingss
Pitfall II. The encoder distribution generates proposals that are insufficiently exploratory. A further complication of the structured latent space is illustrated fig. (center). Here, the modes of the target distribution are sparsely dispersed in the latent space. In this example, we see that both MWG (pink) and pseudo-Gibbs (orange) fail to find distant modes. This because the proposal distribution, as given by the encoder that approximates the model posterior p(= æ), too "narrow" to propose values from the alternative modes. For example, given the upper-half of MNIST image of number "8", it may not be possible to tell if the completed image should be "8" "9", representing two modes of imputations. If the latent space representation of "8" and "g" are sufficiently far, then an encoder conditioned on a current imputation state, for example, Loba U - = "9", is unlikely to propose ă that would decode into Emia in the alternative mode, that is, Lobs U Emis = "8". On the other hand, even the proposal distribution were wide enough to propose jumps to distant modes, MWG would still reject such proposals with high probability due to pitfall and thus prevent effective exploration.
Pitfall III. Poor initialisation can cause sampling of the wrong mode. As noted by Mattei & Frellsen (2018) MWG for VAEs is extremely sensitive to initialisation, and to alleviate this they suggest initialising by first sampling using pseudo-Gibbe before switching to MWG. But, deciding when to stop the "warm-up" not easy, and poor initialisation can make MWG get stuck. Morcover, initialisation via an (approximate) MAP using stochastic gradient ascent may also suffer from the multimodality issues described above. In fig. (right) demonstrate case where MWG (pink) fails due to poor initialisation.
4

---

Published in Transactions on Machine Learning Research (11/2023)
The limitations of Gibbs-like samplers described in pitfalls I-III motivate our development of improved samplers. Interestingly, despite pseudo-Gibbs being theoretically inferior to MWG, we have seen in this section that pseudo-Gibbe Gibbs can under some conditions perform better than MWG (fig. 1). In the following sections we propose two different methods that, like pseudo-Gibbs and MWG, utilise the encoder of the VAE to propose transitions in the latent space, whilst mitigating pitfalls I-III and having stronger theoretical guarantces than the simple pseudo-Gibbs method.
4 Remedies
The Metropolis-within Gibbls (MWG) sampler for conditional sampling of VAEs has several desirable prop- erties (see section 2.3). However, discussed in the previous section, the Gibbs-like sampler can have poor non-asymptotic performance. In this section we propose two methods for conditional sampling of VAEs inspired by MWG that also mitigate its potential pitfalls (section 3). The key idea of the proposed methods is akin to ancestral sampling of eq. (1); first, the methods approximately sample the intractable postcrior over the latents p(z | Fnbs), improve this approximation ileratively, and then sample from the decoder distribution p( Toba, z) conditional on the produced latent samples. In section 4.1 we propose a few simple modifications to the MWG sampler and demonstrate pitfalls of MWG. In section 4.2 propose an alternative method based adaptive importance sampling and likewise demonstrate on synthetic example how it mitigates the pitfalls of MWG. Detailed evaluation of the proposed methods is provided in section 5 and the code to reproduce the experiments is available at https: hopps://gihhll www./vsinus/vveeooooooooo saaeenitionnnlsmmplaa
synthetic example how this mitigates the
4.1 Adaptive collapsed- Metropoliss-within-Giba
We propose several modifications to the MWG sampler from section 2.3 to mitigate the pitfalls outlined in section 3. The proposed sampler is summarised in algorithm
First, to improve exploration and reduce the effects of poor initialisation (see pitfalls 11 and III) we introduce prior- variational mixture proposal
qe (z Toba; mmis) (1 - e)q(= Bobs: mmis) + ep(z),
(3)
where q(2 Pobs: Fmis) the variational encoder distribution, p(z) is the prior distribution of the VAE, and (0, 1) the probability to sample from the prior. Clearly this modification alone would not resolve the pitfalls of MWG, since proposals & sampled from the prior p(z) would be rejected with high probability the Metropolis- Hastings step due to disagreement with the current imputation a in eq. (2).
Hence, we next propose changing the target distribution of the Metropolis Hastings step from p(z Bohs, mmix) to p(z Wohn), such that a good proposal á would not be rejected due to a disagreement with an imputation a-1 (see pitfall 1). The modified Metropolis Hastings acceptance probability is defined as
p' x'="="=1, ( [mis) = min {1, p(zoba x'))(z") qe (x Bobs- Emis p(Eobs 2z 2z-1)p(((11 ge (zt Bobsr Tmis)
(4)
z²) corresponds to reducing the conditioning (or collapsing) in Gibbs samplers which is a common approach to improve mixing and convergence (van Dyk & Park, 2008; van Dyk & Jiao, 2015). In our case, if the optimal proposal distribution p(> Tobs) were known, the sampler would become a standard ancestral sampler and would be maximally efficient, i.c. it would draw an independent. sample at each iteration. Moreover, rather than using the imputation f-11 from the previous iteration to condition the proposal distribution, in MWG, we are here going to re-sample random imputation Eiis from an available set of historical imputations Hi-1 that is updated adaptively with iterations t.
Marginalising the missing variables Amis out of the likelihood p( Amis
We now combine the proposed changes in eqs. (3) and (4) to introduce the algorithm called adaptive collapsed- Metropolisswithinn-Gii (AC-MWG), which can be seen as an instance of the class of adaptive independent
6Our mixture proposal related the small-world proposal Guan et (2006), which has been shown to improve performance complicated heterogeneous and multimodal distributions.
5

---

Published in Transactions on Machine Learning Research (11/2023)
Algorithm Adaptive
Input: VAE model p(æ, z), variational posterior q(>
Emis: Bobs), mixture prob. c, and data-point Tobs Initialise imputation history Sample the initial values
1: Hg = Ø
2: (z mmis) ~ p(=)p((miis Tobs,
3: for t 1 to do
4: mis ~ Uniform (Himis)
> Choose random omis from the history Sample proposal value á
B ~ (c(z Bobs: Fmis)
p² - -2'(,,2'1) (miss)
Calculate acceptance probability using eq. (4) Accept & with probability p²
if p., with Uniform(O, 1) then
8:
x' =
9:
Hmiss faiis 14-1
10:
else
D. Reject & with probability p
11:
x' = x1-1
12: Wimis = Heil
13:
end if
14:
miis paa Bobss 2²)
Sample Emis
15: end for
return {( )²), (I z²) z²²}}
b Return all samples
Metropolis Hastings algorithms (Holden et al., 2009). Assume we start with an initial latent state x and an imputation history H = { such that x" and @ are mutually independent (for example, 2 and aiis are generated via independent short runs of psoudo-Gibbe Gibbs, see section 2.2, or LAIR, see section 4.2). Then a single iteration of the sampler as follows:
Proposal sampling. First, a historical sample kmis is re-sampled uniformly at random from the available imputation history H We then use the proposal distribution from eq. (3) to sample a single proposal é.
2. Metropolis Hastingg acceptance. The proposed sample ă is then either accepted as x' z with probability p²(z, x"-1, x"=1; @iis) in eq. (4) or rejected leaving z' =
3. Imputation sampling. The imputation witi is updated by sampling the conditional a p( Tobs, 2").
4. Adaptation (history update). The available history Hins is updated as follows: if a new á has been accepted then all imputations { [[=1 up to step 1-1 are made available the next { otherwise it is left unchanged H = H
iteration, iee H {o 1+=]
Step 4 of the sampler constructs the available history Ha-1 for the next iteration such that does not contain imputations that depend on the current state zt-1, which ensures that the proposed values are indepen- dent of z and thus guarantees that the stationary distribution of the independent remains correct as the history H changes (Roberts & Rosenthal, 2007; Holden et al., 2009). However, the dependence on the sample history mis makes AC-MWG non-Mar Markovian, and hence convergence needs to be verified. Adapting proofs by Holden et al. (2009), we prove in appendix A that the Markov chain of AC-MWG correctly converges to the stationary distribution p(z, Dmis Dobs)) with probability arbitrarily close to 1 as the number of iterations grows.
Finally, we note that the per-iteration computational cost of AC-MWG and MWG (section 2.3) are nearly the same. The differences are: re-sampling Emis from the history H which should be negligible compared to the cost of evaluating the model, and marginalising the missing variables from the likelihood p( 22) f p( Rmis z) dæ which is often free if the standard conditional independence assumption holds.
In this paper, re-sample åmis from all the past samples in the available history H1-1 however other strategics might be devised improve the computational and convergence properties of the algorithm (see e.g. Holden et al., 2009; Martino al., 2018). For example, by using shorter window of past samples instead of the full length of the history.
6

---

Published in Transactions on Machine Learning Research (11/2023)
Pitfall
Pitfall II
Pitfal III
0.8
Trwe VAE
Pseudo-Gii Gibbs
MWG
AC-MWG
0.4
p(Zmies 2 (aabx)
2
Tmis
Tuiss
Figure 2: The proposed AC-MWG sampler (yellow) with 0.01 2D VAE sampling problems, in fig. 1. (The figure is best viewed in colour.) AC-MWG (yellow) samples the target distribution (blue) more accurately than MWG (pink) and pscudo-Gibbs (orange). All three samplers were initialised with the same state and run for 50k iterations.
4.1.1 Verification of AC-MWG on synthetic VAE
We verify the proposed AC-MWG method on the synthetic VAE example in section 3 (see additional details in appendix C.1). The results are shown in fig. 2 (see also additional figures in appendix D.1). With the proposed modifications, AC-MWG samples the target distribution more accurately by exploring modes that are close in the latent space (left) due to the modified acceptance probability in eq. (4), as well distant modes (center) due to the modified proposal distribution in eq. (3). The modified method is also less sensitive perform ablation studies in appendices D.2 and D.4 to further
to poor initialisation (right). Moreover, validate that both modifications, the mixture proposal in eq. (3) and the collapsed- Gibbs target in eq. (4), are key to the performance of the method.
4.2 Latent- adaptive importance resampling
Instead MCMC, we can sample from eq. (1) via importance resampling (IR, see appendix for details on standard importance resampling and Chopin & Papaspillippulos, 2020, for a comprehensive introduction). However, like MCMC, the efficiency of IR significantly depends on the choice of the proposal distribution. Our goal in this section is to design an adoptive importance resampling method that elficiently samples p(#mis Wabs) of joint VAE model p(z), and we achieve this by constructing an adaptive proposal distribution q' Bobs) using the encoder distribution g(z Bobs, 2mis). The proposed method is summarised in algorithm 2.
As for AC-MWG, we aim to promote exploration and reduce the effects of poor initialisation (see pitfalls II and IIT). We thus start with the prior-variational mixture proposal (z Fobs: Fmiss) from eq. (3) and use it to construct the following adaplive mixture proposal distribution Bobs),
K
d'(z| Robss) = E (Wtis (aars)) [āc(z)
Tobss Fmuis)
with
Fobs)
(0-1)) (1
K
where f2 (obs) an imputation distribution represented as a mixture Dirac masses at particles (t-11,))
[t Tk=11 which we will use to adapt the proposal distribution at each iteration t. We further rewrite the proposal by insorting the definition of f'(mmis @obs) and (e Cobs. Emis), and re- ro-parametrise it by where R is a non-negative integer, to obtain
setting e K+n
R
K
R Σ P=
(z Toba) K+N R
Σ
q(z Cobs 2 Wiss
(B-1,k)
p(z)
(5)

---

Published in Transactions on Machine Learning Research (11/2023)
Algorithm 2 Latent-adaplive importance resampling
Input: VAE model p(æ, z), variational posterior q( x)) data-point Zobs, number of imputation particles
K, number of iterations T (0.K)
x (0.1)
..... mmis ~ Ep(z) (p( Wohss
Sample the initial imputation particle values
for t to do
3:
z((A)) q(> Tobs: mis
(t-1,b) for vk E {I,...,KK K}
Draw sample for each particle. > Draw R. prior proposals.
4:
((t,Kr)) ~ p(z)) for Vr E (l....,RR)
w((((A)) qp p(a
5:
for Vk E {l,.... ......,KK K + R$ w(sll1h))
> Unnormalised importance weights.
6:
for vk E {l..... ...... K + R} ((((1))) = En m((())))
v Normalise importance weights.
2(1.1) g.... x(1,K) Multinomial((E(A.) (( k=1 R+
Resample 2(1,k) from the proposed set.
8:
x (r.k)
mis p(ammis Bohs: z(1,b) for vk E {l...., Kj.
D Updale imputation particles.
9: end for
10: w ((((k)))
EjT ((3(T))) for vk E {l,..... K + R} and Vt E (I,..., > Re-norm. all proposals.
11: 2' ~ Multinomial (12(0)), (( (TT for Vi (I.....,T... T.K) b Resample proposals from all iter.
12: mii p(ammis Bobs, = yT.K
for Vi E (1,..., T K}.
Sample imputations
return 1=
The above proposal can be interpreted to have total of K + R components of which K components depend on the imputation particles {m (=[(t=1.)) ]==1, Ik=1, which encourage exploitation, and are "replenishing" prior components p(z), which encourage exploration and mitigate particle collapse. Moreover, we sample the proposal distribution using stratified sampling (Robert & Casella, 2004; Owen, 2013, Section 9.12; Elvira et al., 2019, Appendix A), well-known variance-reduction technique that draws one sample from each of the K + R components.
Using the mixture proposal distribution in eq. (5) we now introduce the new algorithm that we call latent- adaptive importance resampling (T.IR)) which belongs to the class of adaptive importance sampling al- gorithms (ATS) of Elvira Martino (2022, Section 4). The algorithm starts with K imputation particles (0.4) {= lk=1 that may come from a simple distribution such as the empirical marginals, another multiple im- putation method, or simply the unconditional marginal of the VAE p(#miss) An iteration t of the algorithm then performs the following three steps:
1. Proposal sampling. Sample the proposal distribution (x Øobz) in eq. (5) using stratified sam-
pling. That is, for each particle æmis (f-1,A) draw a sample g(t.k) from the proposal q(> Tober w (t-1,6))
and draw R proposals g(t,K+r) from the prior p(z), for a total of K + R proposals.
2. Weighting. Compute the unnormalised importance weights w()(1.))) 78
w w(2(h)))- = p(zobe ((1.A)
(6)
9¹ (z((Ak) Boba)
3. Adaptation.
3.1. Resample a set (2(tk)) K to the weights w(z(t.k)).
kkz1 with replacement from the proposal set {z(t.)) K+R Tk=l proportionally
3.II. Update the imputation particles far (t.8) KKm by sampling p(Zmis Lobs, 2) conditional on cach E (=((,)) Je from step 3.I.
We here determiiistiicmiixture MIS (DM-MIS) weights but alternative weighting schemes can also be used that enable more fine-grained control of the cost-variance trade-off, see Elvira al. (2019, Scction 7.2).
8 By marginalising the variables Pmis the numerator of the weights address pitfall similar to eq. (4) of AC-MWG. 9 Alternative resampling schemes may also be used, see Chopin & Papaspillopoulls (2020, Section 9.4).
8

---

Published in Transactions on Machine Learning Research (11/2023)
Each iteration / at step 3.II. (accordingly, line 8 of algorithm 2) produces (approximate) samples (f.R) k mis Jk=1 from the target distribution { Tobs), since any iteration of the algorithm corresponds to standard importance resampling and hence inherits its properties (Cappé et al., 2004), see appendix B. In particular, the sampler monotonically approaches the target distribution as the number of proposed samples K + R tends to infinity. Hence, the algorithm may be used in settings where the target distribution changes across iterations& t, for instance, when fitting a model from incomplete data via a Monte Carlo EM (Wei & Tanner, 1990; Simkus et al., 2023).
However, unlike MCMC methods, finite set of samples at any iteration t are not generally guaranteed to convergence to the target distribution as t grows large (Cappé et al., 2004; Douc et al., 2007). In particular, for finite sample sizes, K + R < oo, the sampler bias is of the order O ((Rt)) (Owen, 2013; Paananen et al., 2021) at any iteration 1. and depends on the disparity between the proposal and the target distributions. To improve the approximation, after the algorithm completes all T iterations, we can use samples from all iterations & E ( to construct more accurate estimator (Cappé et al., 2004).
Draw final samples after completing all iterations.
4.I. Re-normalise the weights of g((.) over all iterations E {o....,T) to obtain w(z(kk)
T) and all k (l,..., K + R}
(((((.)))
E wi
4.II. Resample T K samples x² with replacement from the set (=(()))))))))))) I==1k= using the weights
w(z(t-)) from the previous step. 4.III. Sample imputations {amiss T via ancestral sampling by sampling p( Bobs: z) conditional on each # E { [x1 from step 4.11.
The advantage of resampling from the re-weighted full sequence of samples is that the bias of the self- normalised importance sampler goes down with T (in addition to K + R) and hence more accurate samples can be obtained. In particular, the sampler now monotonically approaches the target distribution as the total number of proposed samples approaches infinity, T(K + R) + co, and the bias is of the order O ((イ( )
We note that the per-iteration computational cost of LAIR is comparable to running K R parallel chains of MWG, with the exception of: marginalising the missing variables from the likelihood, as in AC-MWG, which may often be cheap, and evaluating the denominator of the importance weights w(²) in eq. (6), which requires that each of the K + A proposed samples & must be evaluated with the densities of all K + R components in the mixture proposal in eq. (5), hence needing (K + R)² ovaluations. However, since the components of the proposal distribution in eq. (5) are typically all simple distributions, such as a diagonal Gaussians, the computational cost is often negligible for moderate number of proposals K + R. Moreover, the cost may be reduced by trading-off for higher-variance of the estimator, see footnote Finally, the computational cost of the final resampling in step 4 (accordingly, lines 10 to 12 in algorithm 2) is negligible since all the required quantities have already been computed in the past iterations.
4.2.1 Verification of LAIR on synthetic VAE
We now verify the proposed method, LAIR, on the synthetic VAE example in section 3 (see additional details in appendix C.1). The results demonstrated in fig. (see also additional figures in appendix D.1), where we have used K = 19 particles and R = replenishing components (corresponding to c = 0.05). We can see that the method mitigates the three main pitfalls: poor mixing (left), poor exploration (center), and is less sensible to poor initialisation (right). Moreover, in ablation studies performed in appendices D.2 and D.4 we further investigate the sensitivity of the method to choices of e K+H and find that the method performs well as long as 0 <<<1 1.
5 Evaluation
In sections 4.1 and 4.2 we have introduced our methods, AC-MWG and LAIR, for conditional sampling of VAFs which mitigate the potential pitfalls of Gibbs-like samplers (section 3) verified in sections 4.1.1
9

---

Published in Transactions on Machine Learning Research (11/2023)
Pitfall
Pitfall II
Pitfal III
0.8
True VAE
Pseudo-Gii Gihbs
MWG
LAIR
0.4
p(Zmies 2 [aabx)
2
Tmis
Tuiis
3
Figure 3: The proposed LAIR sampler (yellow) with K 19 particles and R = replenishing components 2D VAE sompling problems, same as in fig. 1. (The figure is best viewed in colour.) LAIR (yellow) samples the target distribution (bluc) more accurately than MWG (pink) and pscudo-Gible Gibbs (orange). MWG and pseudo-Gible were run for 50k iterations, and LAIR was run for 2.5k iterations to match the number of generative model evaluations.
and 4.2.1. As motivated in section 2.1, conditional sampling is fundamental tool for multiple imputation of missing data (Rubin, 1987; 1996), where the goal is to generate plausible values of the missing variables with correct uncertainty representation. We here evaluate the newly proposed methods for missing data imputation. We assume that we have pre-trained VAE model, trained on complete data, and aim to generate imputations of the missing variables at test time.
5.1 Mixture-of
-fature-f-GGaussians Gaussians MNIST
Evaluating the quality of imputations from data alone is a difficult task since the imputations represent guesses of unobserved values from an unknown conditional distribution (Abayomi al., 2008; van Buuren, 2018, Section 2.5). Hence, accurately evaluate the proposed methods, in this section first. fit mixture- of of-Gaussians (MoG) model to the MNIST data sct, which then as the ground truth to simulate semi-synthetic data set that is subsequently fitted by VAE model (see appendix C.2 for more details). Using an intermediate MoG model enables us to tractably sample the reference conditional distribution (which would otherwise be unknown) when evaluating the accuracy of the conditional VAE samples obtained using the proposed and existing methods.
In fig. demonstrate the performance of the methods on 10 sampling problems (see appendix D.3 for additional figures and metries). We mcasure the performance using the Fréchct inception distance (FID, Heusel et al., 2017), where for the inception features use the final layer outputs of the encoder network. The figures show that the proposed hods, AC-MWG (pink) and LAIR (yellow), significantly outperform the performance of the Gibbs-like samplers from sections 2.2 and 2.3 (blue and green). In appendix D.4 we further perform an ablation study for the proposed methods, where: we validate that both the mixture proposal in eq. (3) and the collapsed-Gibbs target eq. (4) are key the good performance AC-MWG; and we find that LAIR can perform well for number of values of of K as long as <c
The results for MWG (green) use iseudo-Gibbs warm-up, suggested by the authors Mattei & Frellsen (2018), to mitigate the effects of poor initialisation. We further investigated two different warm-up methods for MWG: an approxinate MAP initialisation using stochastic gradient ascent on the log-likelihood, and LAIR. Both schemes improved over the base MWG but we found that the initialisation using LAIR generally performed better (see fig. 12 in the appendix). MWG with LAIR initialisation is denoted in fig. 4 as MWG' (orange). We observe that with better initialisation the performance of MWG can be significantly improved, hence confirming the sensitivity of MWG poor initialisation as discussed in section 3. However, with few exceptions MWG' (orange) still generally performs worse than the proposed methods (pink and yellow),
10

---

Published in Transactions on Machine Learning Research (11/2023)
10a
10
FTD
109
10"
10"1
100
10"
10
10
10"2
10"22
10-1
10
FID
10
10-1
10-1
10-1
10
10"2
10
10-2
10-1
Pienddo Cibin
MWC
MWG
AC-MWG
AC
LAIR
Paeardo Cibiic
MWW
MWG
AC AC-AWIG
AC-MWG"
LAIR
MWG
MWG
AC-MWG
MW
LAIR
Psendo- Cibis
MWC
MWG
AC MC-MWG
AC-MWO'
LAIR
Pienrdo Cibine
MWG
MWG
AC-MWG
AC
LAIR
Figure 4: Fréchct inception distance (FID) between samples from the ground truth conditional p(z Fobs),
and samples from the imputation methods. Each panel in the figure corresponds to a different conditional sampling problem p( wobs). Each evaluation is repeated 20 times, and the box-plot represents the inter-quartile range, including the median, and the whiskers show the overall range of the results.
Gas
Power
Hepmass
Miniboone
x 10
+
3 10-1
x
Sinkhorn
10-1
3 10-2
2 30-1
6 10
10-2
x
2 10-2
+
RASE
3
2.0
4.5
1.8
LAIR -
Marginal
Pacullo Ceuulo-Gibls- Giblas
MWG'
AC-MWG'
Marginal
Pseudo Gibles
MWGY
AC-MW
LAIR
Marginal
Pseudo Gibbs
MWG'
AC-MWG'
LAIR
Marginal
Paeuul Palldo-Gibls Gibls-
MWG'
AC-MWG'
LAIR
Figure Sampling performance on four real-world UCT data sets. Top: Sinkhorn distance of the imputed data sets evaluated on 50k data-point subset of test data (except for Miniboone where the full test data set used). Bottom: Average RMSE of the imputations the whole test data set. In both rows imputations from the final iteration of each algorithm are used and uncertainty is shown over different runs.
hence suggesting that the poor performance of MWG can be in part explained by the poor mixing of the sampler as discussed in section 3, that is addressed by the proposed methods.
5.2 Real-world UCI data sets
We now evaluate the proposed methods on real-world data sets from the UCI repository (Dua Graff, 2017; Papamakarios al., 2017). We train VAE model with ResNet architecture on complete training data and evaluate the sampling accuracy of the existing and proposed methods incomplete test data with 50%
11

---

Published in Transactions on Machine Learning Research (11/2023)
Missing 1/4 quadrants
Missing 2/4 quadrants
Missing 3/4 quadrants
0.35
0.4
0.250
0.3
0.225
0.25
0.200
0.2
0.20
0.200
0.20
0.150
SSIM
0.175
0.125
0.15
0.150
0.100
0.125
0.10
Marginal
Peenro Gibbbs
MWG
MWG
AC-MWG'
LAIR-
Marginal
Pseudo Gibbss
MWG
MWG'
AC-MWG'
LAIR
Marginal
Pseude- Gilbss
MWG
MWG'
AC-MWG'
LAIR
Figure 6: Imputation accuracy on the binarised Omniglot test set with 1-3 randomly missing quadrants. The top and bottom rows show Fl and average SSIM scores (higher is belter for both metrics) respectively between the imputed and the ground truth values. In both rows imputations from the final iteration of each algorithm used and uncertainty is shown over different runs.
missingness (see appendix C.3 for more details). We also include simple baseline where imputations are sampled from the marginal distribution p(xmis) of the VAE. Moreover, in line with the observations from section 5.1 for MWG and AC-MWG we use LAIR initialisation as we have found it to considcrably improve the performance of both methods. We here assess the performance using two metrics: Sinkhorn distance (Cuturi, 2013) between the imputed and ground truth data sets (computed using geomloss package by Feydy et al., 2019), and average RMSE of the imputations (for additional metrics, see appendix D.5).
The results are shown in fig. 5. First, the figure shows that all methods outperform marginal imputations (blue), with exception of pseudo-Gibbs (green) Hepmass data, where the Sinkhorn distance is slightly higher than the baseline. Second, as before, pseudo-Gibbs (green) is typically improved-upon by MWG (orange). The only exception is the Gas data with the Sinkhorn distance as metric (first row, first column) where the performance shows high variability. Other metrics (second row, first column, and appendix D.5) do not display this behaviour. Third, we see that the proposed methods, AC-MWG (pink) and LAIR (yellow), show better or comparable performance the existing methods terms of Sinkhorn distance (lop row), and always improve on the existing methods in terms of the point-wise RMSE (bottom row). In summary, the results in this section match our findings from section 5.1, and hence further highlight the importance of mitigaling the pitsalls in section 3 when dealing with real-world tasks.
5.3 Omniglot data set
In this section we evaluate the methods for conditional sampling a VAE model trained on fully-observed binarised Omniglot data of handwritten characters (Lake al., 2015). For the VAE model we use convolu- tional ResNet. encoder and decoder networks with 50 latent dimensions (see appendix C.4 for more details). We then evaluate the existing and proposed methods for conditional imputation of test set images that miss 1, 2, and 3 random quadrants. Similar to the previous section, we include simple bascline where imputa- tions are sampled from the marginal distribution p(*mis) of the VAE. The accuracy of the imputations on the binarised Omniglot is assessed using F1 score (Mattei & Frellsen, 2018) and structural similarity index measure (SSIM, Wang et al., 2004) between the ground truth and imputed values.
The results are shown in fig. We first note that all conditional sampling methods perform better than marginal imputations (deep blue). Furthermore, we that the metrics for the existing methods imply
12

---

Published in Transactions on Machine Learning Research (11/2023)
the ranking pseudo-Gibbs (green) < MWG (orange) < MWG' (pink), before. Finally, observe that the proposed methods, AC-MWG (yellow) and LAIR (light blue), further improve the accuracy of the imputations over the existing methods.
6 Discussion
Conditional sampling is key challenge for downstream applications of VAEs and improcise or incfficient samplers can cause unreliable results. We have examined the potential pitfalls of using Gibbs-like samplers, such as MWG, to conditionally sample from unconditional VAE models. While the outlined pitfalls are related the well-known limitations of standard Gibbs sampler, we work out their significance in the context of VAEs. Pitfalls and outline two reasons for poor mixing of MWG: strong relationship between the latents and visibles æ; and lack of exploration when the variational encoder distribution is used proposal. Pitfall III highlights the importance of good initialisation for the performance of the sampler.
We introduced two samplers for conditional sampling of VAEs that address the pitfalls and show improved performance when compared to MWG and other baselines. The proposed methods, adaptive collapsed- Metropolis-within Gibbs (AC-MWG) and latent-adaptive importance resampling (LAIR), mitigate pitfall by marginalising the missing variables amis when (approximately) sampling the latents 2, and then sample the missing values miis ~ p(xmis Tobs: z). Therefore, in contrast to Gibbs sampling, the two methods can be seen as approximate ancestral sampling methods with asymptotic exactness guarantees. To mitigate pitfall II we have constructed proposal distributions from mixture composed of the variational encoder distribution and the prior, which balances exploitation and exploration. Finally, we have found that poor initialisation (pitfall IIII affects LAIR much less than the MCMC methods due to its ability to use information from multiple points in the latent space, and hence using LAIR to initialise MWG and AC-MWG can further improve their respective performances.
Depending on the task, computational budget, and accuracy requirements one may choose to use either AC-MWG or LAIR for conditional sampling of VAEs. For example, in tasks where the target distribution is changing between iterations, such learning VAE model from incomplete data (Simkus et al., 2023), LAIR could be more efficient than AC-MWG; this is because LAIR produces valid (although potentially biased) samples from the larget distribution at any iteration, while AC-MWG requires a "burn-in" " period until the sampler converges to the target distribution. On the other hand, on a strict computational budget AC-MWG might be preferred over LAIR: while the cost of AC-MWG is comparable to MWG (and hence also pseudo-Gibbs), each iteration of LAIR involves equivalent computations on K R particles and hence the computational cost and memory requirements is about K + R times the cost of MWG. Finally, the convergence properties of the two methods are distinct: AC-MWG converges asymptotically in number of iterations, whereas the convergence in LAIR additionally scales in the number of particles K + R and therefore parallelisation ion may be used to improve the speed of convergence at the cost of additional memory usage.
We have focused on conditional sampling of VAE models with podderate-dimensionn latent spaces. To this end, we have addressed the "exploration "exploration-exploitaaaaa" dilemma by constructing the proposal distribution from the prior and variational encoder distributions. But, what works well in moderate dimensions might not work well in high dimensions, direct consequence of the infamous "curse of dimensionality". This means that exploring the posterior by sampling the prior distribution might become impractical in higher dimensions. To scale the methods, alternative exploration strategies could be constructed by replacing the mixture proposal in eq. (3) with for example, a mixture composed of annealed versions of the variational encoder distribution. Moreover, since the proposed methods belong to the large and general families of adaptive MCMC (Haario al., 2001; Warnes, 2001; Roberts & Rosenthal, 2007; Holden et al., 2009; Liang et al., 2010) and adaptive importance sampling (AIS, Cappé et al., 2004; Bugallo ct al., 2017), work opens up additional opportunities to further improve the conditional sampling of VAEs.
13

---

Published in Transactions on Machine Learning Research (11/2023)
References
Kobi Abayomi, Andrew Gelman, and Mare Levy. Diagnostics for multivariate imputations. Journal of the Royal Statistical Society: Series C (Applied Statistics), 57(3)2273 291, 2008, ISSN 1467-9876. doi: (Cited on pg. 10)
0014446-97762200.000
David Barber. Bayesian Reasoning and Machine Learning. Cambridge University Press, 2017. ISBN 978-0-
511-80477-9. doi:
1804779. (Cited on pg. 1, 19)
Guillem Boquet, Jose Lopez Vicario, Autoni Morell, and Javier Serrano. Missing Data in Traffic Estimation: A Variational Autoencoder Imputation Method, In IEEE International Conference on Acoustics, Speech and Signal Processing (TCASSP), pp. 2882-2886, 2886, May 2019. doi: 10.1109/CCASSP.201.... 201...0.11... (Cited on pg. 2)
Monica F. Bugallo, Victor Elvira, Luca Martino, David Luengo, Joaquin Miguez, and Petar M. Djuric. Adaptive Importance Sampling: The past, the present, and the future. IEEE Signal Processing Magazine, 34(4):60- 79, July 2017. ISSN 1558-0792. doi: (0.100/MMPP2017.222
(Cited on pg. 13)
Oliver Cappé, Arnaud Guillin, Jean-Michel Marin, and Christian P. Robert. Population Monte Carlo. Journal of Computational and Graphical Statistics, 13(4):907 -29, December 2004. ISSN 1061-8600, doi: (0.1198/106186004XI22000 (Cited on pg. 9, 13)
Rewon Child. Very Deep VA Generalize Autoregressive Models and Can Outperform Them on Images. International Conference on Learning Representations (ICLR), March 2021. (Cited on pg. 1)
Nicolas Chopin and Omiros Papaspiliopoulos. An Introduction to Sequential Monte Carlo. Springer Series in Statistics. Springer, 2020. (Cited on pg. 7, 8, 22)
Chris Cremer, Xuechen Li, and David Duvenaud. Inference Suboptimality in Variational Autoencoders. In International Conference on Machine Learning (ICML), May 2018. (Cited on pg. 2)
Marco Cuturi. Sinkhorn Distances: Lightspeed Computation of Optimal Transport. In Advances in Neural
Information Processing Systems (NeurIPS), 2013. (Cited on pg. 12)
Randal Doue, Arnaud Guillin, Jean-Michel Marin, and Christian P. Robert. Convergence of Adaptive Mixtures of Importance Sampling Schemes. The Annals of Statistics, 35(1):420 448, 2007. ISSN 0090- 5364. (Cited on pg. 9)
Dheeru Dua and Casey Graff. UCI Machine Learning Repository, 2017. (Cited pg. 11, 24)
Victor Elvira and Luca Martino. Advances Importance Sampling, March 2022. (Cited on pg. 8)
Victor Elvira, Luca Martino, David Luengo, and Mónica F. Bugallo. Generalized Multiple Importance Sampling. Statistical Science, 34(1), February 2019. ISSN 0883-4237. doi: 10.1214/18 00.1214/18-STS63 (Cited on pg. 8)
Jean Feydy, Thibault Séjourné, François-Xavel Xavier Vialard, Shun-ichi Amari, Alain Trouvé, and Gabriel Peyré. Interpolating between Optimal Transport and MMD using Sinkhorn Divergences. In International Con- ference on Artificial Intelligence and Statistics (AISTATS), 2019. doi: 10.48550/ar 14455550022Xx1111100007777 (Cited
pg. 12)
Andrew Gelman and Donald B. Rubin. Inference from Tterative Simulation Using Multiple Sequences. Sta- tistical Science, 7(4):457-4P) 472, November 1992. ISSN 0883-4237, 2168-8745. doi: 10. 00.1214/ss/1177011133 (Cited pg. 3)
Stuart Geman and Donald Geman. Stochastic Relaxation, Gibbs Distributions, and the Bayesian Restora- tion of Images. IEEE Transactions on Paltern Analysis and Machine Intelligence, PAMI-6(6):721 741, November 1984. doi:
(Cited on pg. 2, 3)
Samuel J. Gershman and Noah D. Goodman. Amortized Inference in Probabilistic Reasoning. In Annual Meeling of the Cognitive Science Sociely, volume 36, 2014. (Cited pg. 2)
14

---

Published in Transactions on Machine Learning Research (11/2023)
Rafael Jonez-Bombarell;, Jennifer N. Wei, David Duvenaud, José Miguel Hernández-Lobato, Benjamín Sánchez-Lengee Lengeling, Dennis Sheberla, Jorge Aguilera-Iparraguine and Alán Aspuru-Guus Automatic Chemical Design Using a Data-Driven Continuous Representation of Molecules. ACS Central Science, 4(2):268-2) 276, February 2018. ISSN 2374-7943. doi: 10.1021/acscontee 7b00572. (Cited on pg. 1)
Timothy D. Hirzel, Ryan P. Adams,
lan Goodfellow, Yoshua Bengio, and Aaron Courville. Deep Learning. MIT Press, Cambridge, MA, USA, 2016. (Cited on pg. 1)
Yongtao Guan, Roland Fleißner, Paul Joyce, and Stephen M. Krone. Markov Chain Monte Carlo small worlds. Statistics and Computing, (6(2):193-203) June 2006. ISSN 1573-1375. doi: 10.1007/ s11222-006-6966- (Cited on pg. 5)
Heikki Haario, Eero Saksman, and Johanna Tamminen. An Adaptive Metropolis Algorithm. Bernoulli, 7 (2):223 242, 2001. ISSN 1350-7265. doi: 10.2307/331877 (Cited pg. 13)
Kuan Han, Haiguang Wen, Junxing Shi, Kun-Han Lu, Yizhen Zhang, Di Fu, and Zhongming Liu. Variational autoencoder: An unsupervised model for encoding and decoding fMRI activity in visual cortex. Newrolm- age, 198:125-136, September 2019. ISSN 1095-9572. doi:
201.0.0.....01.000..... (Cited on
pg. 1)
Wilfred Keith Hastings. Monte Carlo Sampling Methods Using Markov Chains and Their Applications. Biometrika, 57(1):97-109,, 1970. ISSN 0006-3444. doi: 10.2307 03307/2333440. (Cited on pg. 1, 3)
David Heckerman, David Maxwell Chickering, Christopher Meek, Robert Rounthwaite, and Carl Kadie. Dependency Networks for Inference, Collaborative Filtering, and Data Visualization. Journal of Machine Learning Research, 1(oct):49-75, 2000. (Cited on pg- 2)
Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. GANs Trained by a Two Time-Scale Update Rule Converge Local Nash Equilibrium. In Advonces in Neural Information Processing Systems (NeurIPS), 2017. (Cited on pg. 10)
Lars Holden. Convergence of Markov Chains in the Relative Supremum Norm. Journal of Applied Probability, 37(4):074-11083 2000. ISSN 0021-9002. (Cited pg. 22)
Lars Holden, Ragnar Hauge, and Marit Holden. Adaptive independent Eetropolis-Hasttnne The Annals of Applied Probability, 19( (0(1):395 413, February 2009. ISSN 1050-5164, 2168-8737 doi: 00.2114008AAP545 (Cited on pg. 6, 13, 18, 20, 22)
Diederik P. Kingma and Jimmy Lei Ba. Adam: A Method for Stochastic Optimization. In International Conference on Learning Representations (ICLR), December 2014. (Cited on pg. 24, 25)
Diederik P. Kingma and Max Welling. Auto-Encoding Variational Bayes. In International Conference on Learning Representations (ICLR), December 2013. (Cited on pg. 1)
Diederik P. Kingma, Danilo Jimenez Rezende, Shakir Mohamed, and Max Welling, Semi-SSupervised Learning with Deep Generative Models. Advances in Neural Information Processing Systems (NeurIPS), June 2014. (Cited on pg. 2)
Brenden M. Lake, Ruslan Salakhutdinov, and Joshua B. Tenenbaum. Human-level concept learning through probabilistic program induction. Science, 2015. doi: 00.122//scieceeaabbbe (Cited on pg. 12, 25)
Chongxuan Li, Jun Zhu, and Bo Zhang. Learning to Generate with Memory. In International Conference on Machine Learning (ICML), June 2016. (Cited on pg. 2)
Yingzhen Li, Richard E. Turner, and Qiang Liu. Approximate Inference with Amortised MCMC, May 2017.
(Cited pg. 2)
15

---

Published in Transactions on Machine Learning Research (11/2023)
Faming Liang, Chuanhai Liu, and Raymond Carroll. Advanced Markov Chain Monte Carlo Methods: Learn- ing from Past Samples. John Wiley & Sons, Incorporated, New York, 2010. ISBN 978-0-470-66973-00 (Cited on pg. 13, 20)
Luca Martino, Roberto Casarin, Fabrizio Leisen, and David Luengo. Adaptive independent sticky MCMC algorithms. EURASIP Journal Advances in Signal Processing, 2018(1):5, January 2018. ISSN 1687- 6180. doi: 10.1186/111. 10.1186/s43634-0111 037----3----000222-- (Cited on pg. 6)
Alexan-ee Mattei and Jes Frellsen. Leveraging the Exact Likelihood of Deep Latent Variable Models. In Advances in Neural Information Processing Systems (NewrIPS), February 2018. (Cited on pg. 2, 3, 4, 10, 12, 35)
Nicholas Metropolis, Arianna W. Rosenbluth, Marshall N. Rosenbluth, Augusta H. Teller, and Edward Teller. Equation of State Calculations by Fast Computing Machines. The Journal of Chemical Physics, 21(6):1087 1092, June 1953. ISSN 0021-9606. doi: (1663/11169914. (Cited on pg. 3)
Art B. Owen. Monte Carlo Theory, Methods and Ezamples. on pg. 8, 9, 23)
//c//////////////// 2013. (Cited
Topi Paananon, Juho Piironen, Paul-Christian Bürkner, and Aki Vehtari. Implicitly adaptive importance sampling. Statistics and Compuling, (2):116,, March 2021. ISSN 0960-3174, 1573-1375. doi: 10.1007/ s11222-000-0998222 (Cited on pg. 9, 23)
George Papamakarios, Theo Pavlakou, and Tain Murray. Masked Autoregressive Flow for Density Estimation. Advances in Newral Information Processing Systems (NeurIPS), 30, 2017. (Cited on pg. 11, 24)
Danilo Jimenez Rezende, Shakir Mohamed, and Daan Wierstra. Stochastic Backpropagation and Approxi- mate Inference. In International Conference on Machine Learning (ICML), Beijing, China, 2014. (Cited
on pg. 1, 2)
Danilo Jimenez Rezende, S. M. Ali Eslami, Shakir Mohamed, Peter Battaglia, Max Jaderberg, and Nicolas Heess. Unsupervised Learning of 3D Structure from Images. In Advances in Newral Information Processing Systems (NeurIPS), June 2018. doi: 1155500arXiv.10777
(Cited on pg. 2)
Christian P. Robert and George Casella. Monte Carlo Statistical Methods. Springer, 2004. ISBN 0-387- 21239-6. (Cited on pg. 8)
Gareth O. Roberts and Jeffrey S. Rosenthal. Coupling and Ergodicity of Adaptive Markov Chain Monte Carlo Algorithms. Journal of Applied Probability, 44(2):458 475, 2007. ISSN 0021-9002. (Cited on pg. 6,
13)
Geoffrey Roeder, Yuhuai Wu, and David K. Duvenaud. Sticking the Landing: Simple, Lower- Variance Gradient Estimators for Variational Inference. Advonces in Neural Information Processing Systems, 30, 2017. (Cited pg. 24, 25)
Donald B. Rubin. Multiple Imputation for Nonresponse in Surveys John Wiley Sons, New York, 1987. ISBN 0-471-08705-X. doi: 10.2307/3172772. (Cited on pg. 2, 10)
Donald B. Rubin. Multiple Imputation After 18+ Years. Journal of the American Statistical Association, 91(4344::473-485 489, 1996. ISSN 0162-1459. doi: 10.2307 10.2307/2291688 22916635. (Cited on pg. 2, 10)
Vaidotas Simkus, Benjamin Rhodes, and Michael U. Gutmann. Variational Gibbs Inference for Statistical Model Estimation from Incomplete Data. Journal of Machine Learning Research, 24(196):1-72, 2023. ISSN 1533-7928. (Cited on pg. 9, 13)
Stef van Buuren. Flexible Imputation of Missing Data. CRC Press LLC, 2 edition, 2018 ISBN 978-1-138- 58831-8. (Cited on pg. 10)
Aäron van den Oord, Oriol Vinyals, and Koray Kavukeuoglu. Neural Discrete Representation Learning. In Advances in Neural Information Processing Systems (NeurIPS), 2017. (Cited on pg. 1)
16

---

Published in Transactions on Machine Learning Research (11/2023)
David A. van Dyk and Xiyun Jino. Metropolis-Hastiine Within Partially Collapsed Gibbs Samplers. Journal of Compulational and Graphical Slatislics, 24(2):301-322 2015. ISSN 1061-8600. (Cited pg. 5)
David A. van Dyk and Taeyoung Park. Partially Collapsed Gibbs Samplers. Journal of the American Statistical Association, 103(482):790-796 -76, June 2008. ISSN 0162-1459. doi: 10.1198 111198//012215050000000000000000 (Cited pg. 5)
Zhou Wang, Alan Conrad Bovik, Hamid Rahim Sheikh, and Eero P. Simoncelli. Image quality assessment: From error visibility to structural similarity. IEEE Transactions on Image Processing, 13(4):600-612, April 2004. ISSN 1941-0042. doi: 2000//TT.2000.8186661. (Cited pg. 12)
Gregory R. Warnes. The Normal Kernel Coupler: An Adaptive Markov Chain Monte Carlo Method for Efficiently Sampling From Multi-Modal Distributions. Technical Report 39, University of Washington, March 2001. (Cited on pg. 13)
Greg C. G. Wei and Martin A. Tanner. A Monte Carlo Implementation of the EM Algorithm and the Poor Man's Data Augmentation Algorithms. Journal of the American Statistical Association, 85(411):699 704, September 1990. doi: 10. 00000//1122449.199.....
(Cited on pg. 9)
Minglian Zhang, Peter Hayes, and David Barber. Generalization Gap in Amortized Inference. In Workshop on Bayesian Deep Learning at Neural Information Processing Systems (NeurIPS), pp. 6, 2021. (Cited on pg. 2)
17

---

Published in Transactions on Machine Learning Research (11/2023)
A AC-MWG proofs
Informally, showing convergence of MCMC samplers generally boils down to answering two questions: (i) does the Markov chain (asymptotically) reach the unique stationary distribution, and (ii) does the sampler remain in the stationary distribution after reaching it. 10
First, we will focus on the latter question: does the AC-MWG sampler remain in the stationary distribution once it has been reached? Let p' denote the distribution after iterations, and T( Fmis) = p(z, Fiis Tobs) denote the target distribution. The following theorem formalises the answer to the question." 11
Theorem A.1. The limiting distribution of the AC-MWG sampler conditioned the history Hyis is invariant, that is (2"1 1-1 Hmmee) x===11 (m/))) implies p' H Hmmss) = x<2²,, x mmtse)
Proof- Let us denote w² Hmjis Hymis the variables made available in the history Hyis after iteration of the algorithm. Note that wt is a random variable since it depends on the accept/reject decision in lines 9 and 12 of algorithm 1. In the proof we will show that by construction of the algorithm w and the new state (z", miss) are independent, and that the statement in the theorem then follows.
Following algorithm 1 we now work out what are the new historical values of w at each itcration t. If a proposal & is rejected then line 12 of algorithm 1 corresponds to setting w = Ø. More generally, we allow adding to the history variables that depend on the rejected state but not on the current state z'-1, If proposal & is accepted then line 9 of algorithm 1 corresponds to sctting w to be the set of imputations that were generated using the previous value of = z For instance, if new proposals were rejected for the last r iterations, then 2f-1-r=z----y =...= = =21=1, and hence " depend on 2²-1, i.e. 1-11 1-r the variable w² will contain the set of imputations { 1r----- that were drawn from π (Imis 2²=1) in
- (11---+1 ..... 2 would all
miis f-1-r+1
mis ~ (2 11 Thus, in the case of proposal acceptance,
the past iterations. We define the conditional distribution of w² as n(w') [2) - 2 is 2='1 if a new proposal was accepted, or I if a proposal was rejected. This construction of the history ensures that the proposal distribution Te Tobe; @mis) in lines and 5 algorithm 1 is independent of the current z²-1, and hence is a key ingredient to the proof.
() where
We denote the transition kernel of AC-MWG as k(2", , w² H The kernel, which depends the history H takes the current state of =111 and produces the new state (=", afiis) and the new historical variable w². We further use f2- (@iis) to denote the probability of sampling a historical imputation ømis from the available history Hise in line 4 of algorithm 1. The kernel of AC-MWG is then defined as follows
k(2", jiints w"| |-=-= (1((111))
P (Rmiss)) / ( Bobs, Dmiis) Tmiss )8(2" ,))=))))r(( | 211
= < 2²)
Σ
Banis E Hiit
+ (z Bobes Bmis) [1 P=(2, x' #mia)) x1=","==1) (o(1o") |z) dz
= ( 2²
Σ
Prr (amis)) [6(2"1 Bobss Tinis (p((z", 2"=11"1===1 Brmin (w 2
Denis E Himi
/
+ +8(2" 2) 6c( Bobs: @mia) [1 pc(E, x² 2-1 (miia) r(w"" ()) dz
1he proofs in this section will consider a single observed data-point Tobe, and hence nearly all quantities would depend on it. To ease the notation we will therefore suppress the conditioning on Bobs in all quantities, except for the proposal distribution Ge in eq. (3) to keep it consistent with algorithm 1.
The theorem is analogous to Theorem 1 by Holden et al. (2009) but extend their proof to the component-wise setting of AC-MWG that involves additional sampling step æ ~ p(Emia Tobe, x'), and where the history is maintained Emis.
Note that the kernel does not depend on the current - since the new state only depends the new 2", L.e. at
73Tmis )
18

---

Published in Transactions on Machine Learning Research (11/2023)
The term in the parentheses corresponds the standard Metropolis Hastingg kernel (see e.g. Barber, 2017, Section 27.4.2) with the addition of w/ to denote the new variables to be appended to the history at iteration t.
Assuming that at iteration t t-1 1 the sampler already at the stationary distribution π (2 amm)) ) we now integrate the kernel with respect to the distribution of the current state (2²-1, mis obtain the marginal over 2', x miis: and w
(2 (r mits w Hemlte = / k==", a'miss w² 1²-1; : Homle (=(== mmmit dzt-1 da
Marginalising the wiii
= -
k(2² wiis w² 1 H ( dz
Inserting the definition the kernel & and pushing the integral w.rtt. zt-1 inside the sum over Emis
Σ (Tmis))
f de (z(2" Bobss @mmis)ee22 z'-1, (rii))((oo' (r(2)))))))) dzz
/
+ 8(25,2/-1)
(c(2) Bobss Fmiss) [1 Pe(2,2" 2²-1, :eins) a(w" d)d (2=m( (= dz
Marginalising the zt-1 in the second integral
T (2onis
Σ fx
f ($mlaa)
Benis Hinta
/ (2" Dobs: Emis (elis)pp(e.. 21=,, Emins (wo" 22' (1--))((2-1)) dz
/
(c(2 Dobs: Emis) [1 - Pr( 2²; a @miss] x(w² 2)m((²) dz
Expanding the second summand
Σ frr (amis)
Peniss Ae
Tc (2' l Bobss @mmis)p(2''s x' (miis)((u =²-"))((2"" dz
-f Tc(E Bobss Pmis)pp(2,4 # (w²² ())((()) de
+ +x(2)) Te( Robs: mT (w² dæ
Using detailed balance (z Bobaa @mis )rr (z, 2²; Tm )T (=*) q. (z' second summand above to obtain two identical integrals that cancel
Bobss Emiss pe , 2; (mis)
the
TT( 2²
Σ
(
Penis EHe
f Te (2" Fohss a @mixs)p(z' z²-1; bTw"" | ((-1)r(("-)) dz
- |Jlle" Babss amis (r( ; 2; amis) nT (w² ())()()))))))))))))))))))))))))) ())
+ = f (c(8 Bobss Bmis) (Tw" d
19

---

Published in Transactions on Machine Learning Research (11/2023)
Cancelling the integral terms and rearranging we obtain the marginal distribution
HH (Tmis) / a.(2 Bohss Tmiss (T(w" 2)) dz.
pl (z" mias w/ H = (
Σ
Importantly, the factorisation shows that (a 2²) and w² are independent, and hence
y'(" H(miz)) = / pr w² Homle) dz dar'ni
Σ ( / @.(2)
aiiss
Robs: Tr 8) dz.
Baiis E
Therefore it immediately follows that
wtt Hmmk'
p' (=", miis Hius
Hmid
U (w*})
nt (ar z²),
p'("" Hmme)
which validates that the algorithm remains in the stationary distribution once it has reached it.
Given that the sampler remains in the stationary distribution as shown in the above proof, we now show that the sampler can reach it. As discussed in section 4.1 the AC-MWG sampler corresponds to an ancestral sampler, which draws samples from p(z Tobb)) using non-Markovian adaptive Metsonolis-Hasttnns,, and then draws from p(zmis| Robs, 2) obtain joint samples (z, Fmiis) ~ p(z, Tmis Dobe). Therefore, to prove (hat the sampler reaches the stationary distribution (question (i) from the start of the section) we only need to show that the Metropolis Hastings sampler reaches p(z Tobs). Let T(2) =p(z Tobs) denote the target distribution, a E 1] function that depends on the historical sample a' fit (x'11) re-sampled from the history Hims in line 4 of algorithm 1 at ileration i, and X mis = (miis...., Fiiss which denotes all those Emis drawn up to iteration i, whose distribution we denote with (X We formalise the answer to question (i) in the following theorem. 13
Hastings,
Theorem A.2. If the likelihood of the model is bounded and the prior variational mixture proposal in eq. (3) uses an e >> 0, then there is a function a (TTis) E (0, that satisfies the strong Doeblin condition
(z x Cobs: Fmiss 2 a" (a for VZ and Va'
(7)
and the total variation distance is bounded from aboue
Ilp'(z)) - <<(>>\l S S (8 Iio a (øT (Fimee)
(8)
Hence the algorithm samples the target distribution within finâte number of iterations with probability arbilrarily close to 1.
Proof. The key observation for this proof is to note that, conditionally on the history H each iteration of the sampler corresponds to one ileration of (generalised) rejection sampler (see e.g. Liang et al., 2010, Section 3.1.1). Let us denote a' {0. Bernoulli random variable with probability distribution p(a' 2, ) probability s (2, ( of a rejection sampler. We obtain by lower-bounding the MH acceptance probability p² in eq. (4). We first rewrite the MH acceptance probability
that siguifies acceptance or rejection of a proposal 2 with a success
T())
(z'-1 Tobs t
2 21-1, a = min
mis
π(2--1)
(2(2 Bobss 25
T(2)
π
qe Babs,
ge (z
Bobsy of
min
Te (2 Lobss
Zt mis
n
Tt (x1-11)
1urur theorem here analogous Theorem 2 by Holden (2009) but extend the proof the where the proposal distribution sampled stochastically using the history.
20

---

Published in Transactions on Machine Learning Research (11/2023)
Lower-bounding the second term above to get a' (*
(=(Z
Tobs miss)
6e(2'-
Tobsr Fiiss)
min (c(2) Dobs T
a' (T mis
min min
x(2)
TT(!-1)
x aT2)
We finally obtain a lower- bouer-doudddd acceptance probability sf to the MH acceptance probability p²
( 21= ; a (#
( miis)) 15
( Tobs
3 mis)
We will now show that with a probability of at least at (Ttiis) the sampler can jump to the stationary distribution π²2) at any iteration t.
The conditional distribution of accepted samples of a rejection sampler is
p :iis: a' = (2
Tobs, mis )p(ss = 1 2, mmis))
o (c Bobs mis))p(a' = 1 2,, )iiss)
p(at ffiis)
Inserting p(a" =1 2, (mita)) = (2, mniss) we obtain
T(2)
p( zt mis: a² 1) x (z Tobs ai )a (
a c (amnis)r(2)) TT)
(z Tobs 2 mis)
p(a' = =1 Fmmis - f ge( Tobs, m )pla'
2, mis dž = a' (()) = a a'(#" miss)
at
Hence it follows that the accepted samples follow the target distribution
a' a' mis )T(
p Eimis a' =1 J a (+)
T(2).
The analogy between AC-MWG and rejection sampling allows to conclude that the conditional probability to jump to the stationary distribution at any iteration is (at least) a af(ffis) This conditional probability depends on the historical sample Emis ~ 1-1 (#mis) but is independent of the current distribution of z'-1
We can now show that the probability to be the stationary distribution within a finite number of iterations can be made arbitrarily close 1. Let be be the probability that the sampler does not jump the stationary distribution in iterations
6' 6' mis)
II (1 a" (#
where _ (# ... aliis), and let p' px" X denote the conditional distribution of 2 after iterations
p' (=" ( = T(2')( B (X)m))) (2" X X )o (X(
which can be seen as a mixture of the stationary distribution x--) with probability (1- ('( and non- stationary distribution (-) with probability 6' ( The marginal distribution at the t-th iteration is then
p'(z')= = 1 p' p'(2' X (X' ( dX mis
We now derive bound on the total variation distance
1p'((z)) <(2*)
14 Note that taking the min makes a² (Ft independent of the state 21-1g
18 Note that due p² E [0, I] also have that the Bernoulli sucoess probability sf [0. 1].
21

---

Published in Transactions on Machine Learning Research (11/2023)
= - (²(21 mi mis) dX mis T(²*)
dz
Inserting the definition of (z X and using linearity of expectation to take T(2') into the expectation over X mis
= -/1s (1(= mis)s) + vz X T(=')) p4s (mm)s) cl mis dz
Expanding 5 (1=)(1 10 ((( )is)) and cancelling terms
= - (-m(z') + (''' xmm)) 6y (X min)p (X dx mis dx
Applying Jensen's 's inequality to the (convex) norm function
-fl- + (2" Xmis)) dz'b't (x='6'(X (X miss) (x Xmin) dX
mis
Applying triangle inequality 11(x') T(2)) $ [z(=)) dz
=2
2 f 10
8 mis (ir (iss)) dx mis = 2E pi (
II (1 a" (min))
Hence, the algorithm converges almost everywhere the product goes to zero with + co. Therefore, if a" (a > 0 infinitely often then the sampler samples the target distribution л x(2) with probability arbitrarily close 1.
To complete the proof we now show that the strong Doeblin condition (Holden, 2000; Holden et al., 2009) in eq. (7) holds, which requires that there exists a² a > 0 for all ž and Emis- Informally, the condition requires that the proposal distribution has heavier tails than the target distribution. We rewrite the condition
in eq. (7) in its equivalent form as follows
(9)
ge 2 Cobs x mis
a' a mis)
Inserting the definition of (²)) = p( Woba)) and (=
Wobs a from eq. (3) to the left side we obtain
π
p Doba)
T Toss E Tnis)
(1 -c)q(Z Dobs . Tf + cp(2)
p(ž, Nobs)
p( p(Dobs) ((1 - ())(2)
Dobs t + cp(2))
p(
p(Dats)
(11 c)
+C)
p(z)
pl Wabs
2)) (1
9 Bobss shiis)
+0)
p( ((ob))))
p(2) (
Hence the ratio is bounded if E > and if the likelihood bounded, which we can safely assume since this is already necessary condition to well learn the model. Since the left hand side of eq. (9) is bounded it follows that af aatfme) > 0, which completes the proof.
B Background: Importance resampling
We can generate samples following eq. (1) by using importance resampling (IR, e.g., Chopin & Papaspiliopou-
los, 2020) to (approximately) sample p(z) Robs)) and then sampling p(Dmis Mlass 2) as in standard ancestral
22

---

Published in Transactions on Machine Learning Research (11/2023)
sampling. We start with the standard importance sampling formulation for approximating the marginal p(Zobn):
p(Dbb)) pl platobss 2) z) = a(2) P(a 2) dz = By(=) [w(z)]]
(10)
((z)
where q(=) is proposal distribution that is assumed easy to sample and evaluate, and w(2) p( /)/((2)
are the (unnormalisod)) importance weights, which are also computationally tractable.
The importance weight function wx-) can then be used to re-weigh the samples from the proposal distri-
bution (=)) to follow the model posterior p(= | Bobs). We denote w(x) ( [w] to be the
(self-) joomalleod importance weights, and show that samples from the proposal can be re-weighted to follow the target distribution
plaahar ((2)
w By(()) [w x
(2)) = (x) (()))(((z) = (xy()
(z)
Ex)(
Ōz(2)
= p(= [otaa)
(11)
p(Zoba)
where (-)+) is the Dirac delta distribution centred at point 8.
In practice, self-normalised importance resampling is generally implemented in four steps:
1. Draw M samples from proposal =1 ...., MM ((z).
Compute the (unnormalised)) importance weights ur(#m) plaahe..
m(#m) for all Vm E [1, M].
3. Self-normalise the weights w(*")) >. w(*')
w(="") for all Vm E M..
4. Resample n with replacement from the set 1ath JM using the normalised probabilities w("").
Self-normalised importance sampling consistent in the number M of proposed samples and hence samples p(= (ohss) exactly as M - do but has bias of the order of O(1MM) (Owen, 2013; Paananen al., 2021).
Samples mmis p(Emia| Toba) can then be obtained by sampling p(#mia| Zobs:
In standard importance sampling applications the proposal distribution ((z) is traditionally chosen heuris- tically using the domain knowledge of the target distribution. However, in the context of VA specifying a good proposal can difficult due to poor prior knowledge about the latent space of the model. Moreover, the efficiency of the sampler depends on the quality of the proposal distribution q(z) and a poor proposal distribution can cause weight degeneracy in the non-asymptotic regime (M < oo), where only a few of the proposed samples have non-zero weights, and hence poorly approximate the target distribution.
C Experiment details
In this appendix we provide additional details on the experiments.
C.1 Synthetic 2D VAE
To investigate and illustrate the pitfalls of MWG constructed simple synthetic VAE model that approx- imates mixture-of- mixtuure-f-Caaussianns data, see fig. 7. The visibles are 2-dimensional and parametrised with di- agonal Gaussian decoder p(æ| 2), the latents are 1-dimensional with uniform prior p(z) = Uniform(O, 1), and the variational proposal d(= x) is a Beta distribution amortised with a neural network. The low- dimensional example lets us compute, via numerical integration, and visualise the conditional distributions p(amis Roba), p(#mis: z Tobs), and p(z Tabs). As demonstrated in the two right-most panels of fig. 7 mixing in the joint space of the missing variable and the latent (Iv, 2) may be poor due to low probability valleys between the modes (third panel), but could be easier in the marginal space of (last panel).
23

---

Published in Transactions on Machine Learning Research (11/2023)
p(ar)
plato 21=
p(z 2;=11.0)
-1
Figure Left-to-right: the marginal distribution of the visibles p(æ) of the VAE; the posterior expected value of the latents 2, i.e. ppp( æ)) [a]; joint conditional distribution of zo and for an observed æı; conditional distribution of 2 for observed Z1.
For pscudo-Gibbs and MWG in figs. 1 to we perform a single run of each algorithm for 50k iterations, with both methods initialised at the same location using a sample drawn from the marginal distribution p(amis) of the VAE. Similarly, in fig. the proposed method AC-MWG performs a single run of the algorithm for 50k iterations with mixture coefficient e 0.01, initialised at the same location as pseudo Gibbs and MWG. Finally, in fig. 3 the proposed method LAIR performs single run of the algorithm for 2.5k iterations using 19 imputation particles (K 19) and 1 replenishing mixture component (R 1), the algorithm is initialised with K- = 19 samples from the marginal distribution pimmis) of the VAE.
C.2 Cuxture-of-Gausseee oa-aasss MNIST
We construct a mixture-of- Gassssians (MoG) ground truth model with 10 multivariate Gaussian components and uniform component probability x(c) 1 Each Gaussian component is fitted on all samples from the 10'
MNIST data set (downsampled to 14x14 4 and transformed with a logit transformation) with a particular label E [1, 10]. We then generated a semi-synthetic data set of 18k samples and fit a VAE model with a latent space dimensionality of 25. For the VAE, we have used a diagonal Gaussian decoder using ConvResNet architecture with 4 convolutional residual blocks of feature map depths of 128, 64, 32, and 32, and a dropout of 0.2. The prior distribution over the latents is standard normal distribution. The variational distribution is parametrised with a diagonal Gaussian encoder using ConvResNet architecture with 4 convolutional residual blocks of feature map depths 32, 64, 128, and 256, and dropout of 0.2. To optimise the VAE model we have used the treeiing-tee-andding gradients (Roeder et al., 2017) and fit the model using batch size of 200 for 6000 epochs using Adam optimiscr (Kingma & Ba, 2014) with a learning rate of 10-4.
For pseudo-Gibbs we ran 5 independent chains for 10k iterations each, and to stabilise the sampler, the imputations were clipped to the minimum and maximum values of the data set for each dimension multiplied by 2. For MWG we have initialised 5 independent chains by running pscudo-Gibbs for 120 iterations with clipping and then run the MWG sampler for 9880 iterations on each chain. For MWG' we have initialised 5 independent chains by running LAIR with K=4 and R-1 for 120 iterations for each chain, and then run the MWG sampler for 9880 iterations on each chain. For AC-MWG we have initialised independent chains from the marginal distribution of the VAE and then run AC-MWG with c = 0.05 for 10k iterations. For AC-MWG' we have initialised independent chains by running LATRR with K=4 and R=1 for 120 iterations for each chain, and then run the AC-MWG sampler for 9880 iterations on cach chain with 0.05. For LAIR we have initialised K particles from the marginal distribution of the VAE and then run the sampler with K 4 and R = 1 for 10k iterations.
C.3 UCI data sets
We fit VAEs on four data sets from the UCI repository (Dua & Graff, 2017) with the preprocessing of (Papamakarios et al., 2017). For all models, the variational and the generator (decoder) distributions were
24

---

Published in Transactions on Machine Learning Research (11/2023)
fitted to be in the diagonal Gaussian family. For the encoder and decoder networks of the VAEs we fit MLP neural networks with residual block architecture using Adam optimiser (Kingma & Ba, 2014) with learning rate of 10-3 for a total of 200k stochastic gradient ascent steps (except for Miniboone where 22k steps were used) using batch size of 512 (except for Miniboone where batch size of 1024 was used), while using 8 Monte Carlo samples in each iteration to approximate the variational ELBO and siicking-the-landiing gradients to reduce variance (Roeder et al., 2017). For Gas, Power, and Hepmass data the encoder and decoder networks used 2 residual blocks each with hidden dimensionality of 256, ReLU activation functions, and a latent space of 16. In addition, for Power data we add small Gaussian noise to each batch with a standard deviation of 0.001. For Miniboone data the encoder used 5 residual blocks with hidden dimensionality of 256 and decoder networks used 2 residual blocks with hidden dimensionality of 256, ReLU activation functions, a latent space of 32, and dropout of 0.5.
For pseudo- Gibbs we ran independent chains for 3k iterations each, and to stabilise the sampler on Gas and Hepmass data sets imputations were clipped to the minimum and maximum values of the data set for each dimension multiplied by 2. For MWG we have initialised 5 independent chains by running LAIR with K=4 and R=1 for 100 iterations for each chain, and then run the MWG sampler for 2900 iterations on each chain. For AC-MWG we have initialised 5 independent chains by running LAIR with K=4 and R=1 for 100 iterations for each chain, and then run the AC-MWG sampler for 2900 iterations on each chain with c = 0.3. For LAIR we have initialised K = 4 particles from the marginal distribution of the VAE and then run the sampler with K = 4 and R = 1 for 3k iterations. Each method evaluations were repeated with 5 different sceds, and the uncertainty reported in the figures reflects the uncertainty over different runs.
C.4 Handwritten character Omniglot data set
We fit a VAE on a statically binarised Omniglot data set (Lake et al., 2015) downsampled to 28 x 28 pixels. We have used a fixed standard Gaussian prior distribution over the latents p(z) with a dimensionality of 50, an encoder distribution q(= æ)) in the diagonal Gaussian family, and a decoder distribution p(æ: 2) in a Bernoulli family. For the encoder and decoder networks we have used convolution neural networks with ReLU activations, dropout probability of 0.2, and residual block architecture with 4 residual blocks in each networks. For the encoder the residual block hidden dimensionalities were 32, 64, 128, and 256, and for the decoder they were 128, 64, 32, and 32, We used Adam optimiser (Kingma & Ba, 2014) with a learning rate of 10-1 and a cosine annealing schedule, for a total of 3k stochastic gradient ascent steps using a batch size of 200, Moreover sticking-the- landing gradients were used to reduce encoder network gradient variance (Roeder et al., 2017).
For pseudo-Gibbs ran 5 independent chains for 5k iterations each. For MWG we have initialised 5 independent chains by running pscudo-Gibbs for 120 iterations, and then running the MWG sampler for 4880 iterations each chain. For MWG' we have initialised 5 independent chains by rumning LAIR with K=4 and R=1 for 120 iterations for each chain, and then run the MWG sampler for 4880 iterations on each chain. For AC-MWG' we have initialised 5 independent chains by running LAIR with K=4 and R-1 for 120 iterations for each chain, and then run AC-MWG for 4880 iterations on each chain with = 0.05. For LAIR we have initialised K = 4 particles from the marginal distribution of the VAE and then run the sampler with K- 4 and R- = 1 for 5k iterations. The above evaluations were repeated with 5 different seeds, and the uncertainty reported in the figures reflects the uncertainty over different runs.
D Additional figures
In this appendix we provide additional figures for the experiments in this paper.
D.1 Synthetic 2D VAE
To aid with the understanding of the pitfalls in section 3 and our remedies in section 4, we here include additional figures on the synthetic VAE model (sec details in appendix C.1). Specifically, in the top row of fig. 8 we plot the marginal distributions of the latents p(z zoba) that provide an additional perspective of the failure modes described in section 3: A method that is able to sample the joint distribution p(mmis; 2 Bobs)
25

---

Published in Transactions on Machine Learning Research (11/2023)
must also be able to effectively sample the marginal p( Tobs), and if it is able to do so, then the joint p(Tmis, (obss) and the marginal of the missing variables p( P(Emis! Rohs) are recovered via. ancestral sampling of eq. (1).
In the left-most column (pitfall 1) we can see that MWG fails to explore the unimodal posterior p(> zobe). As described in section 3 this is because the decoder distribution p( mmis ε) places little density/ /mass on the previous value of 2mis wiis which in turn gets such latent proposals å rejected. As a result, the MWG sampler remains "stuck" in small part of the (marginal) posterior. The middle column provides additional view of pitfall II. In particular, case is multi-modal. However, an encoder q(z Tobs, Fmiss) conditioned on specific completed data-point Bohs mis is unlikely to propose a latent value & that would reach one of the alternative modes. As result, the pseudo-Gibbs and MWG samplers never reach the alternative modes the posterior p(> Øobs), and remain stuck in a single mode. Finally, the right-most column reinforces the understanding of pitfall ITI. Specifically, we see that if MWG is initialised in a low-probabiiity location of p(z Tobs), may fail to reach the high-probability mode.
see that the posterior distribution p(z | Øobs) in this
The second and third rows of fig. 8 show the posterior approximations obtained using AC-MWG (section 4.1) and LAIR (section 4.2). As we can see, similar to the results in sections 4.1.1 and 4.2.1 the proposed methods are able to avoid the pitsalls of pseudo-Gibbs and MWG. The proposed methods remedy pitfaall by targeting the marginal p(= Tobs)) instead of the joint p(mmia: Tobs). Once approximate samples from the marginal p(z Wohs) are obtained then the methods use this approximation to perform ancestral sampling of the joint p( x Tobs). Moreover, the methods address pitfall II by using the prior-variational mixture proposals in eqs. (3) and (5), which enable exploration of the latent space. The remedy to pitfall III is related to the remedies for pitfalls I and II: the prior variational mixture proposal enables search of the latent space and targeting the marginal distribution p(> Tohs) allows the sampler to move from the poor initial location to better one by not conditioning on the previous imputation value miss = x f-1
26

---

Published in Transactions on Machine Learning Research (11/2023)
Pitfall 1 pdf
Pitfall pdf
Pitfall III pdf
10
15
10
20
10
True VAE p(z [obe)
Pseudo-Gible
0.6
MWG
p(amies Jobel
Batex](a]
0.2
-2
mmis
Tmis
Tims
Pitfall 1
Pitfall pdf
Piufall III pdf
pdf
10
True VAE p(= aabel
Pseudo-Gible
MWG
AC-MWG
p( dabel
Batexo]le]
-2
Tiis
Tmis
Tiis
Pitfall 1
Pitfall II plf
Pitfall III pdf
pdl
10
15
10
20
True VAE p(z aabe)
0.8
Pseudo-Gibbe
MWG
LAIR
0.4
p(amins [atw)
0.2
Enis
Fnis
Euiis
Figure Additional figures on the synthetic VAE, showing the marginals p(2 [obss) (The figure is best viewed in colour.) Top: showing only the true marginal p(= Tobs) in blue colour and the marginals of pseudo-Gibbs (orange) and MWG (pink). Middle: showing the marginal of AC-MWG (yellow). Bottom: showing the marginal of LAIR (yellow).
27

---

Published in Transactions on Machine Learning Research (11/2023)
D.2 Ablation study: Synthetic 2D VAE
In this section perform an ablation study of AC-MWG and LAIR that supplements the results in sec- tions 4.1.1 and 4.2.1.
Pirfall
Pitfall II
Pitfal III
4-
True VAE
0.8
AC-MWG
0.6
AC-MWG (e (e= 0)
pdf
AC-MWG
0.4
(MWG target)
p(Zmme 2 2oins)
-2
Tmis
Tmis
Tiis
Pitfall
Pitfal II
Pitfll III
True VAE
2.5
0.8
LAIR
2.0
0.6
LAIR 0.0)
pdf
LAIR 0.5)
0.4
LAIR 1.0)
10
0.5
p(Zmis Tabss)
T'mss
I Tinia
Tiis
in figs. to 3. (The figure is best viewed in colour.) Top: the same AC-MWG before with 0.01 (yellow), AC-MWG with = 0.0 (light blue) that does not "explore" the latent space via samples from the prior, and AC-MWG with = 0.01 but the Metropolis Hastiing target of p(z Tobs, Fmis)) (red), same as MWG. Bottom: the same LAIR as before with 19 and = (i.e. e = 0.05, in yellow), LAIR with K = 20 and R (i.e. = 0.0, in light blue) that does not "explore" the latent space using samples from the prior, LAIR with K 10 and R 10 (i.e. 0.5, in red), and LAIR with K 0 and R 20 (i.e. e 1.0, in purple), which corresponds to standard (non-adaptive) importance resampling using the prior distribution proposal.
Figure 9: Ablation studies on the 2D VAE sampling problems,
In the top of fig. 9 we show two ablation cases of AC-MWG. In the first case (light blue) set 0.0 in the variationl mixture proposal in eq. (3). Without the prior component the sampler fails to "explore" the latent space (see the middle panel in the figure, where the light blue and red curves overlap) due to insufficiently exploratory proposal distribution (i.c. pitfall II). In the second case (red) we change the target distribution of the Metropolis Hastings step from p(z Foba) in eq. (4) to p(z Tobes Tmis) used standard MWG, that is, using the acceptance probability in eq. (2). We observe that with the MH target changed to p(2 Boba, mmis) the sampler also fails to mix between nearby modes in the latent space in the left-most panel (i.e. pitfall 1), similar to MWG, which also affects the other two cases (middle and right panels). We therefore validate that the two modifications (the mixture proposal and the collapsed-Gibl MH target p(z )) introduced in section 4.1 are key components of the AC-MWG sampler.
R In the bottom row fig, we show three ablation cases of LAIR by varying the prior probability e = K+R in the mixture proposal oq. (5). The first case (light blue) corresponds to LAIR with = 0.0 (or R= 0) and performs sub-optinally (see the middle panel) due lack of exploration in the latent space (see pitfall II).
The second case (red) corresponds to LAIR with 0.5 (or R K 10) and performs similarly to
28

---

Published in Transactions on Machine Learning Research (11/2023)
base LAIR case (yellow). The third case (purple) is LAIR with 1.0 and corresponds to standard non-adaptive importance resampling with the prior distribution the proposal. The standard importance sampling (purple) performs equally-well because of the simplicity and low-dimensionality of the latent space, however as the latent space gets more complex and higher dimensional the adaptive LAIR sampler will perform better (see results appendix D.4).
D.3 Peeture-of-Gausiian MNIST
Figure 10 shows the conditional mean and standard deviation at each "missing" pixel of the image, conditional on the "observed" pixels surrounded by red border. Top-left shows the ground truth values, and the rest show values estimated from samples produced using the VAE and the (approximate) samplers. Furthermore, fig. 11 shows the absolute error in the conditional means (black better) and signed error on the standard deviations (blue is underestimated, red is overestimated, white is perfect). The figures show a complementary view of the results in section 5.1. Interestingly, we can that pseudo- Gibus and MWG can overestimate the variance at some pixels while at the same time underestimating it at other pixels. The proposed methods, AC-MWG and LAIR, are less affected by this issue
Figure 12 corresponds to fig. 4 the main text but we additionally show MWG with MAP initialisation using stochastic gradient ascent with random restarts (red). Furthermore, fig. 13 shows the experiment results using additional metries. The additional metrics mirror the results in the main text.
29

---

Published in Transactions on Machine Learning Research (11/2023)
The MoG MNIST
Pseudo-Gihe
Pr
MWG
MWG
6
AC-MWG
LAIR
Prufee
6
6
dratt
E
Gaussians MNIST. The top-left panel shows the ground-truth values, and the other panels show estimates from imputations generated by the evaluated samplers. The pixels surrounded by a red border are the observed values Lobs.
Figure 10: Conditional mean Va and slandard devialion az aat on the
30

---

Published in Transactions on Machine Learning Research (11/2023)
Pseido Gibbe
d
MWG
MWG
JCO
AC-MWG
LAIR
-B
d
Figure 11: The absolute error on the conditional mean Pr was and the signed error on the standard deviation aemi wats on the mixture-of Saatuureof-Gauusians Gaussians MNIST. We can clearly see that the proposed methods (bottom row) outperform the existing samplers.
31

---

Published in Transactions on Machine Learning Research (11/2023)
10a
10a
10
101
FTD
10P
10P
10-1
100
100
10
10"8
10
101
100
10
10-1
10-1
10
10-2
10-1
10-2
10-1
ParudoGibll Gibbs
MWWG
MWG+SCDALAP
MWG
AC-BIW
NC-.WG'
EAIR
Plendh Paaiee-Cilbis Cilbi
MWG
SIN++SDMMPP..
MWG'
AC MWG
AC MWG
EAT
Pseuce- Cibbs
MWG
SCDMMP
MWG
AC-WGG
AC-MWG/
LAIR
Pando-GGlbls
MWG+SGDMAP
MWG'
AC-
MWWW
LAIR
Poeude Cibid-Cibbis Gibbe
MWG
MWG SCDMAP
MWG
AC-MWG
MWC
EAIR
Figure 12: Same as fig. but with additional method included, MWG+SGDMAP (red), which initialises
MWG using stochastic gradient ascent on the likeliild (with restarts).
32

---

Published in Transactions on Machine Learning Research (11/2023)
JSD
10
10-1
10-1
10P
10-2
10
10-3
10-3
10-4
10
10
100
10P
F
10k
100
150
10-1
10-1
10-1
10-2
10
10"22
10-2
10
10-3
Panudl Gibbs
MWG
MWWG -SDDIAP
MWG
ACMWGGG
AC-MWC/
LAIR
Pannde- Cibbo
MWG
MRVG+SGDMAP
MWG
AC
MCW
LAIR
Panudo Cilbss
MWC
MWC SCDMAP
MWG
AG-WG
LAIR
Gible
SCDMAP
MWG'
AC-WGG
AC-MW
LAIR
Pseude- Gibbs
MWG
MWC SCDMIAP
MWWG
AC-WGGG
AC-MWC/
LAIR
(a) Jensen Shannon divergence (JSD) between the ground truth conditional p(z @obe), and estimator f( [obs))
N 1= p(z aobss miss), where aiis come from the impulation methods and N is the total number imputations.
103
16
FID
10a
102
102
10a
10a
10-1
10"
10a
10P
102
FED
10A
10
10a
10P
109
10a
Pocadh-Gibte
MWGG
MWG
AC-MG
AC-WGG'
LAIR
Paeashe Glhbe
MWG
MWG-SGDMAP
MWG'
AC MWG
&C LCW.G
EAIR
Pscade Clabe
MWC
WWG+SCDMAPP
MWG
ACMWGG
MCWMWG'
LAIR
Picaal-Giila
MWG.
SWGGSS
MWG
ACAMWG-
AC
EAIR
Pseadee Glbbe
MWG
MWG MWG.SGDMAA SCDMAP
MWG
AC-MW
MWW
EAIR
(b) Fréchet inception distance (FID) between samples from the ground truth conditional p(> dota), and samples
obtained from imputation methods. The inception model features used FID computation are the final layer
outputs of a classifier neural network.
Figure 13: Additional metrics on the miature-of- miiaure-of-Gaussianns MNIST. Each panel in the subfigures corresponds different conditional sampling problem p(æmis Zobs). Each evaluation is repeated 20 times, and the box-plot represents the inter-quartile range, including the median, and the whiskers show the overall range of the results.
33

---

Published in Transactions on Machine Learning Research (11/2023)
D.4 Ablation study: onature-of-Gausiiie MNIST
This section shows an ablation study of AC-MWG and LAIR on the mixt wriluure-o-Gaussiia MNIST data set that supplements the results in section 5.1.
10
101
10a
10-1
10-1
10
10
10-2
10-2
10-1
10-2
10a
2
10A
10P
100
10-1
10-1
10
10-2
10-1
10-2
10-1
AC-MWGG
ACMWG (e - 0.0)
AC-MWG (MIWG targes)
LAUt
LAIR (c= 0.0)
LAIR (c- 0.6)
LAIR (c- 1.0)
ACAWW
AC-WGG (e 0.)
AC-MWG (MWG (mpps))
LAIB
LAIR 0.0)
LAIR (.6)
LAIR 1.0)
AC-MWG
AC-WG 0.0)
AC-MWG (MWG (aege))
LAIRt
LAIR (e=0.00
LAIR (c- 1.0)
AC-MWG
AC-MWG & 0.00
AC-MWG (MWG (anes)
LAIR
LAIR (e= 0.00
LAIR (c= (06)
LAIR (c- 1.0)
AC-AWWG
AC-WG (E- 0.0)
AC-MWG (MIWG Darges)
LAUt
LAIR (e 0.0)
LAIR (e- 0.6)
LAIR (c- 1.0)
LAIL (c-
Figure 14: Ablation studies on the MoG MNIST sampling problems, same as in fig. 4. The FID score is computed using the final layer outputs of the encoder network as the inception features. Left part of each panel: AC-MWG (deep blue) is the same AC-MWG as in fig. 4 with = 0.05, AC-MWG with = 0.0 (green) corresponding to no prior component in the proposal distribution but the Metropolis Hastings target of p(z Robs mmis) (orange), same MWG. Right part of each ponel: LAIR (pink) the same LAIR as in fig. 4 with K 4 and R (i.e. 0.2), LAIR with K 5 and R 0 (i.e. 0.0, yellow), LAIR with K 2 and R = 3 (i.e. 0.6, light blue), and LAIR with K 0 and R = 5 (i.e. 1.0, red) corresponding to standard (non-adaptive) importance resampling with the prior distribution as the proposal.
eq. (3), and AC-MWG with e = 0.05
The left part of each panel in fig. 14 shows two ablation cases of AC-MWG. In the first case (green) we set = 0.0 in the varit-variational mixture proposal in eq. (3). As explained in pitfall II the sampler fails to explore the latent space and hence exhibits degraded performance compared to AC-MWG with >> 0 (deep bluc). In the second case (orange) we change the target distribution of the Mctropolis Metropolis-Hastings stop from p(z Tohs) in eq. (4) to p(z Tobs; Tmis) used in standard MWG, that is, using the acceptance probability in eq. (2). Similarly, we see that this ablation significantly reduces the performance of the sampler (orange versus deep blue). With this evaluation, similar to the results in appendix D.2, we validate that the proposed components (mixture proposal and galapsed-Gibbe MH target) key the performance of the AC-MWG method.
The right-hand part of each panel in fig. 14 shows three ablation cases of LAIR with varying prior probabilities c= -R+ E {0.0. 0.6, 1.0} (or equivalently, varying K and R) in the mixture proposal in eq. (5). The first case (yellow) is LAIR with 6 = 0.0 (i.e. R = 0), which corresponds to not using the prior distribution in the mixture proposal eq. (5), and exhibits a significantly downgraded performance over LAIR with 0.2 (or K = and R 1, pink). The second case (light blue) is LAIR with 0.6 and performs similarly to LAIR with 0.2 (pink), hence showing that the method is not highly sensitive to the choice of long as the edge cases (c = 0 and c= 1) are avoided. The third case (red) is LAIR with < 1.0 and corresponds to a standard non-adaptive importance resampling with the prior distribution as the proposal. As we see here, the non-adaptive importance resampling (red) performs sub-optimally and hence validates that the adaptation in LAIR is important for good performance of the method.
34

---

Published in Transactions on Machine Learning Research (11/2023)
D.5
UCI data sets
In fig. 15 we show additional metrics of the experiments in section 5.2. We also include MWG with pseudo- Gibbs initialisation (red) as originally proposed in Mattei & Frellsen (2018). The first two rows show energy-distanco MMD and Laplacian MMD between the imputed data sets and the ground truth data. Wo observe a similar behaviour to the results in the main text. The main exception is the Hepmass data where MWG' (orange) seems to be preferred. However, we note that part of the good performance of MWG' (orange) on Hepmass data due the use of LAIR initialisation, while using pseudo- Gibbs initialisation (red) performs similarly to LAIR (yellow). Moreover, the final row shows the average mean absolute error, and the proposed methods, AC-MWG (pink) and LATR (yellow), are preferred over the existing methods on all data sets.
Gas
Power
Hepmass
Miniboone
2 10-3
Evergy MMD
10-2
10-2
10-3
10-3
10
6 10
10-4
1x 10
10
3 10-1
Gas
Power
Hepmass
Miniboone
Laplacian MMD
10
x 10-5
3
10
10-4
10
10-5
10-3
10-6
10-3
x 10"
2.1 10
10a
5 10
x 10"
2 10
4.6 x 109
MAE
2 10"
1.9 10
44 10"
4 10"
188 x 10
4.2 109
109
10"
1.7 10
6 10-1
Marginal
Peendo Cibbe
MWG
MWG'
AC-MWG'
LAIR
Marginal
Psendo-Gible
MWG
MWG'
AC-MWG"
LAIR
Marginal
Peendo-Gii Gibbe
MWG
MWG'
AC-MWG'
LAIR
Marginal
Peendo-Gible
MWG
MWG'
AC-MWG'
LAIR
Figure 15: Additional metrics sampling performance on four real-world UCI data sets. Top: energy MMD. Middle: Laplacian MMD. Bottom: average MAE of the imputations. The divergences are evaluated a 50k data-point subset of test data (except for Miniboone where the full test data set was used), and the MAE is averaged over the full test data set. In all rows imputations from the final iteration of the corresponding algorithms are used and uncertainty shown over different runs.
35


Reviews:
[{"review_id": "Xczwg65fpH", "reviewer": "Reviewer_V822", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths**:\n- The paper is fairly well-written.\n- The analysis of the previous methods is reasonable.\n- The proposed methods seem to be sensible and to work well in practice.\n- The theoretical analysis and the experimental results support well the claims. \n\n**Weaknesses/Questions**:\n- I think that the “pitfalls” and the “solution” can be clarified a bit more. \n\nAs regards the pitfalls:\n\n1) If the sampled $\\tilde{z}^t$ is “far” from the $\\tilde{z}^{t-1}$ then the likelihood $p(x_{obs}, x_{mis}^{t-1} | \\tilde{z}^t)$ is going to be low and the sample will be rejected. The problem here is that decoders imply distributions with very low entropy? (Pitfall I)\n\n2) If the sampled $\\tilde{z}^t$ is “close” to the $\\tilde{z}^{t-1}$ then the likelihood $p(x_{obs}, x_{mis}^{t-1} | \\tilde{z}^t)$ is high and the sample is accepted. (Pitfall II)\n\nSo both pitfalls essentially limit the exploration of the sampler in the non-assymptotic regime?\n\nAs regards the solutions:\n\n1) The proposed proposal in both methods depends on some parameters. How sensitive are the methods with respect to the hyperparameters $\\varepsilon, R, K, T$?\n\n2) The likelihood $p(x_{obs}|\\tilde{z}^t)$ is computed instead of the full joint, but if the $\\tilde{z}^t$ is far from the actual code of the $x_{obs}$ the associated likelihood will be again low? Does the LAIR approach fix this by using multiple latent proposals in every step?\n\n- The 1-dimensional examples are interesting and informative, but a bit hard to understand. Could you provide examples for 2-dimensional settings? In addition, an example to show the proposal distribution for both methods (in 2-dim) would have been very interesting. There is already Fig. 6, which I think can be improved.\n\n- Perhaps an ablation study as regards the hyperparameters would have been beneficial to see how much they affect the performance.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The authors propose two methods for conditional sampling from a pre-trained VAE, where the conditional distribution implies that only a part of a test point is observed. The main drawback of previous approaches is that the sampling process can potentially be trapped in a mode and thus fail to explore the true distribution well. The proposed methods alleviate this issue by providing a better proposal distributions and a more suitable acceptance probability in an MCMC framework, and also, in an adaptive importance-sampling scheme. Theoretical results support the proposed samplers, while their efficiency is demonstrated in the experimental section.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Please check weaknesses. I have no additional requests for changes.", "broader_impact_concerns": "I think that a Broder Impact statement is not necessary as this is mainly a technical paper with no direct ethical implications.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "hwEuFYaQ5W", "reviewer": "Reviewer_5zjr", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "# Strengths\nThe quality of writing is very high\n- The pitfalls (Section 3) are clearly explained, and the accompanying figures are helpful illustrations\n- Key methodological developments (Algorithm 1, Algorithm 2) are nicely paired with motivating problems, and the paper adequately connects to previous works\n- Figures (Figure 1, Figure 2 etc) are legible, and accompanying captions clearly spell out the takeaways.\n\nThe paper's numerical experiments are likely easy to replicate independently (pending release of source code after successful submission)\n- Experimental setups are easy to understand\n- The authors promise to release code\n\nThe paper's claims are supported by theoretical/empirical evidence \n- Figure 1 and Figure 2 provide empirical evidence that the proposed algorithms can overcome the pitfalls encountered by existing methods \n- Appendices contain correct proofs of the infinite-time accuracy of Algorithms 1 and Algorithm 2\n\n# Weaknesses\nNone of note.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper proposes two sampling algorithms for the task of conditional sampling in variational autoencoders. The paper theoretically proves that the algorithms are perfectly accurate in the infinite-time limit. In finite-time, the paper numerically demonstrates that the proposed techniques alleviate pitfalls of existing ones, and gives meaningful inferences on various types of data (synthetic, UCI).", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "The following are discussion questions I think the paper would benefit from (although they are not necessary):\n- how to tune the number of particles K and the number of draws from the samples R in Equation 5? \n  - A possible answer to this is plotting the relationship between the Frechet inception distance and K and/or R.\n- are there conditions in which LAIR (Algorithm 2) is preferable to AC-MWG (Algorithm 1)? \n  - in section 3, the paper nicely outlines situations in which LAIR is better than baselines (Pseudo-Gibbs or Metropolis-within-Gibbs) and AC-MWG is better than baselines. \n  - what are situations in which AC-MWG is better than LAIR, in either statistical or computational sense?", "broader_impact_concerns": "None", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "Nuc7Au44dj", "reviewer": "Reviewer_x1jP", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n1. The paper focuses on a practically important problem of drawing conditional samples in VAEs.\n2. The paper is overall easy to read. I really like the structure: Section 3 introduces pitfalls of existing methods, and Section 4 fixes them with the proposed methods.\n3. The proposed methods are well-motivated and work well empirically.\n\nWeaknesses:\n1. Both methods seem quite complex, while at the same time being 'modular': one could easily imagine the same methods without, say, Collapsed Gibbs. This raises the question of how practically important each piece of the methods is. Another 'module' is adaptive mixture proposal distributions. I would be very keen to see ablations for the methods.\n2. All experiments in the paper have been run with latent space of 16..32 units. That's lower than 50 latent units in (Mattei & Frellsen, 2018), which makes me wonder about the scaling of the methods with respect to the latent space dimensionality. In particular, LAIR relies on importance resampling which means the bias would increase with higher dimensionality.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper proposes two novel methods for conditional sampling of VAEs: Adaptive collapsed Metropolis-within-Gibbs and Latent-adaptive importance resampling. These methods are motivated by three pitfalls of conditional samplers for VAEs (degenerate likelihoods causing poor mixing; encoder distributions not being wide enough; poor initialisation preventing mixing). Empirically, these methods significantly improve upon prior methods: pseudo-Gibbs and Metropolis-within-Gibbs.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "1. (Important) Ablation studies for the methods, see the weaknesses part above.\n2. (Less important, but would strengthen the paper). Exploration of how the methods behave with increasing latent space dimensionality.\n3. (Minor) Ideally Figures 1&2 should be made readable in black and white print.", "broader_impact_concerns": "None.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: Reviewers unanimously agreed that the paper's contribution is solid and should be published under TMLR.

The main concerns raised by the reviewers include (1) some unclear points in the discussed pitfalls, and (2) ablation studies for the components of the proposed approach. In reply, the authors added more illustrations and experiments in the updated manuscript, and the reviewers are satisfied with the revision. Therefore I suggest this paper can be accepted in its current revised form.
Decision: Accept (poster)