Prior Work
A variety of approaches to arbitrary conditioning have been previously proposed. ACE is an autoregressive, energy-based method that is the current state-of-the-art for arbitrary conditional likelihood estimation and imputation, although it can be computationally intensive for very high dimensional data [ 38 ] . ACFlow is a variant of normalizing flows that can give analytical arbitrary conditional likelihoods [ 24 ] . Several other methods, including Sum-Product Networks [ 32 , 6 ] , Neural Conditioner [ 2 ] , and Universal Marginalizer [ 10 ] , also have the ability to estimate conditional likelihoods.

Rezende et al. [ 34 ] were among the first to suggest that VAEs can be used for imputation. More recently, VAEAC was proposed as a VAE variant designed for arbitrary conditioning [ 18 ] .
Unlike Posterior Matching, VAEAC is not a general framework and cannot be used with typical pretrained VAEs.
EDDI is a VAE-based approach to active feature acquisition and relies on arbitrary conditioning [ 25 ] . The authors introduce a “Partial VAE” in order to perform the arbitrary conditioning, which, similarly to Posterior Matching, tries to model p(\mathbf{z}\mid\mathbf{x}_{o}) . Unlike Posterior Matching, they do this by maximizing a variational lower bound on p(\mathbf{x}_{o}) using a partial inference network q(\mathbf{z}\mid\mathbf{x}_{o}) (there is no standard VAE posterior q(\mathbf{z}\mid\mathbf{x}) in EDDI). Gong et al. [ 13 ] use a similar approach that is based on the Partial VAE of EDDI. The major drawback of these methods is that, unlike with Posterior Matching, q(\mathbf{z}\mid\mathbf{x}_{o}) must be reparameterizable in order to optimize the lower bound (the authors use a diagonal Gaussian). Thus, certain more expressive distributions (e.g., autoregressive) cannot be used. Additionally, these methods cannot be applied to existing VAEs.
The methods of Ipsen et al. [ 17 ] and Collier et al. [ 9 ] are also similar to EDDI, where the former optimizes an approximation of p(\mathbf{x}_{o},\mathbf{b}) and the latter optimizes a lower bound on p(\mathbf{x}_{o}\mid\mathbf{b}) . Ipsen et al. [ 17 ] also focuses on imputation for data that is missing “not at random”, a setting that is outside the focus of our work.

There are also several works that have considered learning to identify desirable regions in latent spaces. Engel et al. [ 11 ] start from a pretrained VAE, but then train a separate GAN [ 14 ] with special regularizers to do their conditioning. They only condition on binary vectors, \mathbf{y} , that correspond to a small number of predefined attributes, whereas we allow for conditioning on arbitrary subsets of continuous features \mathbf{x}_{o} (a more complicated conditioning space). Also, their resulting GAN does not make the likelihood q(\mathbf{z}\mid\mathbf{y}) available, whereas Posterior Matching directly (and flexibly) models q(\mathbf{z}\mid\mathbf{x}_{o}) , which may be useful for downstream tasks (e.g. Section 5.5 ) and likelihood evaluation (see Appendix). Furthermore, Posterior Matching trains directly through KL, without requiring an additional critic. Whang et al. [ 40 ] learn conditional distributions, but not arbitrary conditional distributions (a much harder problem). They also consider normalizing flow models, which are limited to invertible architectures with tractable Jacobian determinants and latent spaces that have the same dimensionality as the data (unlike VAEs). Cannella et al. [ 7 ] similarly do conditional sampling from a model of the joint distribution, but are also restricted to normalizing flow architectures and require a more expensive MCMC procedure for sampling.