RiteeshGrandhi/sciclaimeval-shared-task huggingface dataset:
D.4 Very Fast Greedy Feature Acquisition
In the feature acquisition experiments, we use relatively simple VAE models. For flattened MNIST, our encoder and decoder are MLPs with hidden layers of sizes 50, 100, and 200. We use this same architecture for EDDI as well. For the convolutional model, the encoder has four layers with the following (hidden units, kernel, stride) : (32, 3, 1), (32, 3, 2), (64, 3, 2), (64, 1, 1) . The decoder has the layers (64, 8, 1), (64, 5, 2), (32, 5, 1), (32, 5, 1), (1, 3, 1) . For both versions, the encoder outputs a diagonal Gaussian posterior, and the decoder outputs Bernoulli distributions. The latent dimension is 10 for all models.

For the partially observed posterior, we use a network with the same architecture as the fully observed encoder. Since we want to be able to compute the posterior entropy analytically when computing the information gains, we also use a Gaussian for the partially observed posterior. However, rather than letting it be diagonal, we parameterize the Gaussian with a lower triangular matrix L such that the covariance matrix is C=LL^{\top} . We do not stop gradients on samples from the VAE encoder when computing the Posterior Matching loss.

We first train the VAEs with Posterior Matching, before learning the lookahead posteriors. During training, we uniformly generate masks that set between 0% and 20% of the features as observed. Our models are trained for 200000 steps with a batch size of 128. We use the Adam [ 20 ] optimizer with an initial learning rate of 0.001 and an exponential decay schedule with a rate of 0.9 every 5000 steps.

After the VAE with Posterior Matching has been trained, we then freeze this model and train the lookahead posterior network. This network outputs one diagonal Gaussian for each feature. Given that the number of features can be relatively large and we are limited by the memory of the GPU, we randomly select a subset of these distributions to update at each training step. That is, we subsample the terms in the sum in Equation 7 . For the MLP model, we subsample 128 indices, and for the convolutional model we subsample 32 indices. We then estimate the expectation in Equation 7 over multiple samples from the already trained VAE model. We found using multiple samples to be important for getting good performance. For the MLP model, we use 64 samples, and for the convolutional model we use 16 samples. The MLP model is trained for 50000 steps with a batch size of 64 and the convolutional model is trained for 60000 steps with a batch size of 32. We again use the Adam optimizer with an initial learning rate of 0.001 and an exponential decay schedule with a rate of 0.9 every 5000 steps.
D.5 Autoregressive Posterior Details
As described in the main text, one of the advantages of Posterior Matching is the freedom to use highly expressive distributions for the partially observed posterior, as we do not require it to be reparameterizable. Here, we describe an autoregressive distribution that we use in some of our experiments. It is based on the proposal distributions used in Strauss and Oliva [ 38 ] and Nash and Durkan [ 28 ] , which were shown by Strauss and Oliva [ 38 ] to outperform prior state-of-the-art arbitrary conditioning methods for likelihoods and imputation, despite being very simple.

The distribution consists of an MLP with residual connections that outputs a mixture of Gaussians for each covariate. This network accepts partially observed inputs as well as a conditioning vector as input. In our case, the conditioning vector is the output of the partially observed posterior encoder. We then compute q(\mathbf{z}\mid\mathbf{x}_{o}) in an autoregressive fashion as

q(\mathbf{z}\mid\mathbf{x}_{o})=\prod_{i=1}^{D}q(z_{i}\mid\mathbf{x}_{o},\mathbf{z}_{<i}),

where D is the dimensionality of \mathbf{z} . Each q(z_{i}\mid\mathbf{x}_{o},\mathbf{z}_{<i}) term is obtained from a separate evaluation of the autoregressive distribution’s network, where the partially observed inputs change to reflect the appropriate \mathbf{z}_{<i} . The conditioning vector that represents \mathbf{x}_{o} remains constant for all of these evaluations though. Note that when computing likelihoods, these evaluations can be done efficiently in parallel. Sampling, however, requires an O(D) sequential procedure. However, we do not need to sample this distribution during training, and D is generally small anyway.

We chose this particular distribution for its combination of simplicity and good performance. However, we did not experiment extensively with other types of autoregressive distribution for the partially observed posterior. As previously noted, though, there is a large degree of flexibility in this choice.
Figure 1: A two-dimensional VAE latent space (left) that represents the distribution of codes of handwritten 3s, 5s, and 8s. Conditioning on the subset of pixels shown in the center should then result in a distribution over latent codes that only correspond to 3s and 8s, as shown on the right.; A two-dimensional VAE latent space (left) that represents the distribution of codes of handwritten 3s, 5s, and 8s. Conditioning on the subset of pixels shown in the center should then result in a distribution over latent codes that only correspond to 3s and 8s, as shown on the right.
Figure 2: Overview of the Posterior Matching framework. The partially observed encoder can be appended to any existing VAE to enable arbitrary conditioning. At inference time, we can decode samples from q_{\theta}(\mathbf{z}\mid\mathbf{x}_{o}) to perform imputation or compute p(\mathbf{x}_{u}\mid\mathbf{x}_{o}) .; Overview of the Posterior Matching framework. The partially observed encoder can be appended to any existing VAE to enable arbitrary conditioning. At inference time, we can decode samples from q_{\theta}(\mathbf{z}\mid\mathbf{x}_{o}) to perform imputation or compute p(\mathbf{x}_{u}\mid\mathbf{x}_{o}) .
Figure 3: Example acquisitions from our CNN model using the lookahead posteriors (see Section 5.5 ). Top row shows \mathbf{x}_{o} , and blue pixels are unobserved. Bottom row shows imputation of \mathbf{x}_{u} .; Example acquisitions from our CNN model using the lookahead posteriors (see Section 5.5 ). Top row shows \mathbf{x}_{o} , and blue pixels are unobserved. Bottom row shows imputation of \mathbf{x}_{u} .
Figure 4: UMAP [ 27 ] visualization of the latent space of a VAE trained on MNIST. Black dots represent samples from the distribution q_{\theta}(\mathbf{z}\mid\mathbf{x}_{o}) learned via Posterior Matching. Images of \mathbf{x}_{o} are to the right of their respective plot. Samples from each mode are decoded and shown.; UMAP [ 27 ] visualization of the latent space of a VAE trained on MNIST. Black dots represent samples from the distribution q_{\theta}(\mathbf{z}\mid\mathbf{x}_{o}) learned via Posterior Matching. Images of \mathbf{x}_{o} are to the right of their respective plot. Samples from each mode are decoded and shown.
(a) Omniglot; (b) MNIST; (c) CelebA; Example image inpaintings from VDVAE with Posterior Matching. First two columns on the left are \mathbf{x} and \mathbf{x}_{o} . Inpainting samples are on the right.
Figure 6: Partially observed clustering accuracy achieved by Posterior Matching with VaDE.; Partially observed clustering accuracy achieved by Posterior Matching with VaDE.
Figure 7: Average RMSE of reconstructions for greedy active feature acquisition.; Average RMSE of reconstructions for greedy active feature acquisition.
Figure 8: Overview of how Posterior Matching is used to learn “lookahead” posteriors for greedy active feature acquisition. The top path illustrates how samples are produced for use in the lookahead Posterior Matching loss. At inference time, we obtain the entropy of the distributions outputted by the rightmost networks. Taking the top path to obtain these entropies is the more expensive sampling-based approach. Using the learned Lookahead Encoder on the bottom path is the much faster approach that only requires a single network evaluation.; Overview of how Posterior Matching is used to learn “lookahead” posteriors for greedy active feature acquisition. The top path illustrates how samples are produced for use in the lookahead Posterior Matching loss. At inference time, we obtain the entropy of the distributions outputted by the rightmost networks. Taking the top path to obtain these entropies is the more expensive sampling-based approach. Using the learned Lookahead Encoder on the bottom path is the much faster approach that only requires a single network evaluation.
Figure 9: Demonstration of the zero-imputing approach to imputation with VAEs. In each image, the left column shows the inputs to a base VAE’s encoder, and the right column shows the corresponding outputs. We show the masked regions of the inputs on the bottom row in green for clarity, even though those values are actually zero when passed to the encoder.; Demonstration of the zero-imputing approach to imputation with VAEs. In each image, the left column shows the inputs to a base VAE’s encoder, and the right column shows the corresponding outputs. We show the masked regions of the inputs on the bottom row in green for clarity, even though those values are actually zero when passed to the encoder.
Figure 10: VDVAE Omniglot inpaintings.; VDVAE Omniglot inpaintings.
Figure 11: VDVAE MNIST inpaintings.; VDVAE MNIST inpaintings.
Figure 12: VDVAE CelebA inpaintings.; VDVAE CelebA inpaintings.
Figure 13: VQ-VAE Omniglot inpaintings.; VQ-VAE Omniglot inpaintings.
Figure 14: VQ-VAE MNIST inpaintings.; VQ-VAE MNIST inpaintings.
Figure 15: VQ-VAE CelebA inpaintings.; VQ-VAE CelebA inpaintings.
(a) Omniglot; (b) MNIST; (c) CelebA; VQ-VAE image samples from the joint distribution, a special case obtained by sampling q_{\theta}(\mathbf{z}\mid\mathbf{x}_{o}=\emptyset) . Note that the models were not explicitly trained to model the joint and never saw \mathbf{x}_{o}=\emptyset during training.
Peak signal-to-noise ratio (PSNR) and precision/recall scores [ 35 ] for image inpaintings. We report mean and standard deviation across five evaluations with different random masks. VAEAC and ACFlow results are taken from Li et al. [ 24 ] . Higher is better for all metrics.
Test normalized root-mean-square error (NRMSE) and arbitrary conditional log-likelihoods (LL) for UCI datasets. Lower is better for NRMSE, and higher is better for LL. Results for methods other than Posterior Matching are reproduced from Strauss and Oliva [ 38 ] . Mean and standard deviation are reported over 5 random observed masks for each instance.
Additional hyperparameters for UCI experiments. Hidden Units, Residual Blocks, and Layer Normalization refer to the VAE encoder/decoder networks.
Dataset-dependent hyperparameters for VQ-VAE.