Posterior Matching Beyond Arbitrary Conditioning
The concept of matching VAE posteriors is quite general and has other uses beyond the application of arbitrary conditioning. We consider one such example, which still has ties to arbitrary conditioning, in order to give a flavor for other potential uses.

A common application of arbitrary conditioning is active feature acquisition [ 13 , 23 , 25 ] , where informative features are sequentially acquired on an instance-by-instance basis. In the unsupervised case, the aim is to acquire as few features as possible while maximizing the ability to reconstruct the remaining unobserved features (see Figure 3 for example).

One approach to active feature acquisition is to greedily select the feature that will maximize the expected amount of information to be gained about the currently unobserved features [ 23 , 25 ] . For VAEs, Ma et al. [ 25 ] show that this is equivalent to selecting each feature according to

\operatorname*{argmax}_{i\in u}H(\mathbf{z}\mid\mathbf{x}_{o})-\mathbb{E}_{x_{i}\sim p(\cdot\mid\mathbf{x}_{o})}\Big{[}H(\mathbf{z}\mid\mathbf{x}_{o},x_{i})\Big{]}=\operatorname*{argmin}_{i\in u}\mathbb{E}_{x_{i}\sim p(\cdot\mid\mathbf{x}_{o})}\Big{[}H(\mathbf{z}\mid\mathbf{x}_{o},x_{i})\Big{]}

For certain families of posteriors, such as multivariate Gaussians, the entropies in Equation 6 can be analytically computed.
In practice, approximating the expectation in Equation 6 is done via entropies of the posteriors p^{(i)}(\mathbf{z}\mid\mathbf{x}_{o})\equiv\mathbb{E}_{x_{i}\sim p_{\theta,\phi}(\cdot\mid\mathbf{x}_{o})}\Big{[}q_{\theta}(\mathbf{z}\mid\mathbf{x}_{o},x_{i})\Big{]}, where samples from p_{\theta,\phi}(x_{i}\mid\mathbf{x}_{o}) are produced by first sampling \mathbf{z}\sim q_{\theta}(\cdot\mid\mathbf{x}_{o}) and then passing \mathbf{z} through the VAE’s decoder p_{\phi}(x_{i}\mid\mathbf{z}) (we call p^{(i)}(\mathbf{z}\mid\mathbf{x}_{o}) the “lookahead” posterior for feature i , since it is obtained by imagining what the posterior will look like after one acquisition into the future).
Hence, computing the resulting entropies requires one network evaluation per sample of x_{i} to encode z , for i\in u . Thus, if using k samples for each x_{i} ,
each greedy step will be \Omega(k\cdot|u|) , which may be prohibitive in high dimensions.

In analogous fashion to the Posterior Matching approach that has already been discussed, we can train a neural network to directly output the lookahead posteriors for all features at once. The Posterior Matching loss in this case is

\mathcal{L}_{PM-Lookahead}(\mathbf{x},o,u,\omega,\theta,\phi)=\sum_{i\in u}\mathbb{E}_{x_{i}\sim p_{\theta,\phi}(\cdot\mid\mathbf{x}_{o})}\Bigg{[}\mathbb{E}_{\mathbf{z}\sim q_{\theta}(\cdot\mid\mathbf{x}_{o},x_{i})}\Big{[}-\log q_{\omega}^{(i)}(\mathbf{z}\mid\mathbf{x}_{o})\Big{]}\Bigg{]}, (7)

where \omega is the parameters of the lookahead posterior network.
In practice, we train a single shared network with a final output layer that outputs the parameters of all q_{\omega}^{(i)}(\mathbf{z}\mid\mathbf{x}_{o}) .
Note that given the distributions q_{\omega}^{(i)}(\mathbf{z}\mid\mathbf{x}_{o}) for all i , computing the greedy acquisition choice consists of doing a forward evaluation of our network, then choosing the feature i\in u such that the entropy of q_{\omega}^{(i)}(\mathbf{z}\mid\mathbf{x}_{o}) is minimized. In other words, we may bypass the individual samples of x_{i} , and use a single shared network for a faster acquisition step. In this setting, we let q_{\omega}^{(i)}(\mathbf{z}\mid\mathbf{x}_{o}) be a multivariate Gaussian so that the entropy computation is trivial. See Appendix for a diagram of the entire process. This use of Posterior Matching leads to large improvements in the computational efficiency of greedy active feature acquisition (demonstrated empirically in Section 5.5 ).