Introduction
Variational Autoencoders (VAEs) [ 21 ] are a widely adopted class of generative model that have been successfully employed in numerous areas [ 4 , 15 , 26 , 33 , 16 ] . Much of their appeal stems from their ability to probabilistically represent complex data in terms of lower-dimensional latent codes.

Like most other generative models, VAEs are typically designed to model the joint data distribution, which communicates likelihoods for particular configurations of all features at once. This can be useful for some tasks, such as generating images, but the joint distribution is limited by its inability to explicitly convey the conditional dependencies between features. In many cases, conditional distributions, which provide the likelihood of an event given some known information, are more relevant and useful. Conditionals can be obtained in theory by marginalizing the joint distribution, but in practice, this is generally not analytically available and is expensive to approximate.

Easily assessing the conditional distribution over any subset of features is important for tasks where decisions and predictions must be made over a varied set of possible information.
For example, some medical applications may require reasoning over: the distribution of blood pressure given age and weight ; or the distribution of heart-rate and blood-oxygen level given age , blood pressure , and BMI ; etc.
For flexibility and scalability, it is desirable for a single model to provide all such conditionals at inference time.
More formally, this task is known as arbitrary conditioning , where the goal is to model the conditional density p(\mathbf{x}_{u}\mid\mathbf{x}_{o}) for any arbitrary subsets of unobserved features \mathbf{x}_{u} and observed features \mathbf{x}_{o} . In this work, we show, by way of a simple and general framework, that traditional VAEs can perform arbitrary conditioning, without modification to the VAE model itself.

Our approach, which we call Posterior Matching, is to model the distribution p(\mathbf{z}\mid\mathbf{x}_{o}) that is induced by some VAE, where \mathbf{z} is the latent code. In other words, we consider the distribution of latent codes given partially observed features. We do this by having a neural network output an approximate partially observed posterior q(\mathbf{z}\mid\mathbf{x}_{o}) . In order to train this network, we develop a straightforward maximum likelihood estimation objective and show that it is equivalent to maximizing p(\mathbf{x}_{u}\mid\mathbf{x}_{o}) , the quantity of interest. Unlike prior works that use VAEs for arbitrary conditioning, we do not make special assumptions or optimize custom variational lower bounds. Rather, training via Posterior Matching is simple, highly flexible, and without limiting assumptions on approximate posteriors (e.g., q(\mathbf{z}\mid\mathbf{x}_{o}) need not be reparameterized and can thus be highly expressive).

We conduct several experiments in which we apply Posterior Matching to various types of VAEs for a myriad of different tasks, including image inpainting, tabular arbitrary conditional density estimation, partially observed clustering, and active feature acquisition. We find that Posterior Matching leads to improvements over prior VAE-based methods across the range of tasks we consider.