Title: Epitomic Variational Autoencoders
Authors: Serena Yeung
Anitha Kannan
Yann Dauphin
Li Fei-Fei
Venue: ICLR 2017
Keywords: Unsupervised Learning

Abstract: In this paper, we propose epitomic variational autoencoder (eVAE), a probabilistic generative model of high dimensional data. eVAE is composed of a number of sparse variational autoencoders called `epitome' such that each epitome partially shares its encoder-decoder architecture with other epitomes in the composition. We show that the proposed model greatly overcomes the common problem in variational autoencoders (VAE) of model over-pruning. We substantiate that eVAE is efficient in using its model capacity and generalizes better than VAE, by presenting qualitative and quantitative results on MNIST and TFD datasets.

TL;DR: We introduce an extension of variational autoencoders that learns multiple shared latent subspaces to address the issue of model capacity underutilization.

Review discussion:
What prevents this algorithm from collapsing to use only 1 or a small number of epitomies? It seems like the proposed functional form should be just as capable of permanently turning off groups of units as a standard VAE. What makes it more robust?
In sec 2.1, you claim that C_vae trades off between data reconstruction and satisfying a factorial assumption. My interpretation would rather be that the second term pulls the approximate posterior to resemble the prior. I wouldn't imagine it was the factorial nature of the prior that was the issue ...
but rather that this term directly discourages q(z_i | x) from being a function of x. I wonder if you can expand on the factorial claim a bit more?
Last paragraph in sec 3.1 -- You claim that a standard VAE cannot deactivate a unit for a single training example. I believe this is untrue. The standard VAE learns a function of x for both the mean and variance, and that function is capable of setting the mean to 0 and the variance to 1 for any sample. Is there a reason to believe that a standard VAE lacks this flexibility?
"Overcoming over-pruning" section: What do you believe the mechanism is by which overfitting to the data leads to samples that look less like the data? This seems counterintuitive.
Why does having more units, but using only a subset of them, lead to sample degradation? It would seem that increasing the number of latent units beyond 2 should either help (if the units are used), or be irrelevant to performance (if the units are turned off).
Table 1:
Parzen window evaluation is appropriate for real valued data, and the other models you are comparing against should all be evaluating on real valued MNIST. You performed your experiment on *binarized* MNIST. Could you clarify the MNIST Parzen window experiment?
Note also that you are reporting a Parzen log likelihood which is *higher* than would be achieved by samples from a *perfect* model, which is concerning in terms of interpretation. (See [A note on the evaluation of generative models, L Theis, A Oord, M Bethge, 2015] for some reasons why Parzen window style evaluation is generally a bad idea.)
Your training algorithm maximizes a variational lower bound on the log likelihood. Why don't you report, and compare models using, that variational bound, as is commonly done for VAEs?
Thank you for the questions.  Our responses are as follows:
1.  What prevents this algorithm from collapsing to use only 1 or a small number of epitomies? It seems like the proposed functional form should be just as capable of permanently turning off groups of units as a standard VAE. What makes it more robust?
In our experiments, we have not found the algorithm to collapse to using a smaller number of epitomes. The use of stride forces epitomes to share parameters and provides the regularization that causes the epitomes to be used.
In addition, during training, examples are assigned to the epitome that best explains them (Alg. 1, ln. 3), encouraging specialization of epitomes.
Also, for each epoch, minibatches are constructed to not only have a random subset of training examples but also be balanced w.r.t. to epitome assignment (Alg. 1, ln. 4).  We will update the paper to clarify this point.
2.  In sec 2.1, you claim that C_vae trades off between data reconstruction and satisfying a factorial assumption. My interpretation would rather be that the second term pulls the approximate posterior to resemble the prior. I wouldn't imagine it was the factorial nature of the prior that was the issue ...
but rather that this term directly discourages q(z_i | x) from being a function of x. I wonder if you can expand on the factorial claim a bit more?
The exact posterior q(z_1, .. z_D|x) is highly structured but intractable.
So, mean field or fully factorized distribution, the product over all i of independent q(z_i|x), is used as a surrogate. This factorized distribution leads to a simpler form for the bound in which each q(z_i|x) can independently satisfy its own KL term, by resembling its prior.
As a result of the factorization, a subset of the z_i can therefore be used solely to satisfy the KL term and provide a big reduction in C_vae, while only the remaining z_i are used to model the data (and optimize the data reconstruction term). We can also see this in Figure 1.
3.
Last paragraph in sec 3.1 -- You claim that a standard VAE cannot deactivate a unit for a single training example. I believe this is untrue. The standard VAE learns a function of x for both the mean and variance, and that function is capable of setting the mean to 0 and the variance to 1 for any sample. Is there a reason to believe that a standard VAE lacks this flexibility?
The point we make here is different:  For C_vae to have zero contribution from the KL term of a particular z_d (in other words, that unit is deactivated), it has to have all the examples in the training set be deactivated (KL term of zero) for that unit. We contrast this with the epitomeVAE: the KL term of a unit is automatically zero for all examples except the ones for which the particular epitome(s) containing the unit is used. This means that only a small number of examples in the training set contributes a possible non-zero value to z_d's KL term in C_evae.  This added flexibility gives the model the freedom to use more total units without deactivating them, while optimizing the bound.
Also, see Figure 4 and its corresponding explanation.
4. "Overcoming over-pruning" section: What do you believe the mechanism is by which overfitting to the data leads to samples that look less like the data? This seems counterintuitive. Why does having more units, but using only a subset of them, lead to sample degradation? It would seem that increasing the number of latent units beyond 2 should either help (if the units are used), or be irrelevant to performance (if the units are turned off).
VAE is trained as an autoencoder with the goal to reconstruct the training set.
However during test time, the decoder or the generative model is used to generate samples.  This mismatch during training objective and later use as a generative model is what leads to this counterintuitive behavior.
When we overfit to the data, the manifold learned by the decoder will be tuned only to that dataset. This leaves a lot of “holes” in the manifold;
when we sample, the sampling procedure, starting from the prior over z, may lead to generating from these poorly modeled regions and hence  performance degradation. The severity of this effect is more pronounced as the dimensionality of z gets large. In epitomeVAE, we overcome this by making an important observation that is well-studied in (group) sparse-coding literature: the inherent dimensionality of a  given data point is small (and can be modeled by an epitome), while overall latent dimensionality is large.
5. Table 1: Parzen window evaluation is appropriate for real valued data, and the other models you are comparing against should all be evaluating on real valued MNIST. You performed your experiment on *binarized* MNIST. Could you clarify the MNIST Parzen window experiment?
We perform the Parzen window experiment on the output of the decoder before binarizing, so it is real-valued and comparable with the other models. We will clarify this in the paper.
6. Note also that you are reporting a Parzen log likelihood which is *higher* than would be achieved by samples from a *perfect* model, which is concerning in terms of interpretation. (See [A note on the evaluation of generative models, L Theis, A Oord, M Bethge, 2015] for some reasons why Parzen window style evaluation is generally a bad idea.)
As noted by (Theis et al, 2015) the Parzen log-likelihood does have some drawbacks, but they also highlight the shortcoming of competing approaches.  We have chosen to use the Parzen estimator because we can only evaluate the model from its samples, and in order to be able to compare to results in the literature.
7. Your training algorithm maximizes a variational lower bound on the log likelihood. Why don't you report, and compare models using, that variational bound, as is commonly done for VAEs?
A higher log likelihood does not translate to a better decoder useful for sampling (see argument before, and response to “Overcoming over-pruning” question). Our premise is that a better decoder can be learned by ensuring that the model can make use of its model capacity more efficiently.  Since our focus is on improving sample generation, we use the Parzen window evaluation which is a better measure of that ability.
This paper proposes an elegant solution to a very important problem in VAEs, namely that the model over-regularizes itself by killing off latent dimensions. People have used annealing of the KL term and “free bits” to hack around this issue but a better solution is needed.
The offered solution is to introduce sparsity for the latent representation: for every input only a few latent distributions will be activated but across the dataset many latents can still be learned.
What I didn’t understand is why the authors need the topology in this latent representation. Why not place a prior over arbitrary subsets of latents? That seems to increase the representational power a lot without compromising the solution to the problem you are trying to solve. Now the number of ways the latents can combine is no longer exponentially large, which seems a pity.
The first paragraph on p.7 is a mystery to me: “An effect of this …samples”. How can under-utilization of model capacity lead to overfitting?
The experiments are modest but sufficient.
This paper has an interesting idea that may resolve a fundamental issue of VAEs and thus deserves a place in this conference.
Thank you for your comments. With regards to the questions:
1. Why the topology is needed vs. a prior over arbitrary subsets of latent units.
The strided epitome topology allows the model to learn O(D) specialized subspaces, that when sampled at generation time can each produce good samples.
In contrast, when a sparsity prior is simply introduced over arbitrary subsets (e.g. with Bernoulli latent units to specify if corresponding z is on or off),  it can lead to poor generation results (which we confirmed empirically but did not report). The reason for this is as follows:  due to an exponential number of potential combinations of latent units 2^D, sampling a subset from the prior at generation time cannot be straightforwardly guaranteed to be a good configuration for a subconcept in the data, and often leads to uninterpretable samples.  If we want to use this approach, a potential solution, that we leave for future work, is to use an autoregressive model to model valid configurations of latent units; this adds complexity and loses the ordered grouping of units in the strided epitome topology, but could add increased flexibility of representation. We will clarify the intuitions in the paper.
2. Clarification of “An effect of this under-utilization of model capacity is that VAE overfits to the training data, leading to good reconstruction (examples are shown in Fig. 8) but poorly modeled posterior manifold due to over pruning, causing poor generation samples.”
What we mean by this is that VAE learns to model only regions of the posterior manifold near training samples, instead of generalizing to model the full posterior manifold well, an effect we would like to encourage through the KL term. VAE chooses to operate in a mode where the objective is minimized through a combination of active units that are sufficient for modeling / compression of the training samples (good reconstruction and low reconstruction error, at the cost of high KL for these units), and turning-off units to have very low (or zero) KL. This behavior is contrary to the aim of the KL term to uniformly encourage all units to be close to the prior, in order to model well the stochastic latent manifold and obtain good generation. We agree the sentence is confusing and will reword it in the paper.
The paper presents a version of a variational autoencoder that uses a discrete latent variable that masks the activation of the latent code, making only a subset (an "epitome") of the latent variables active for a given sample. The justification for this choice is that by letting different latent variables be active for different samples, the model is forced to use more of the latent code than a usual VAE.
While the problem of latent variable over pruning is important and has been highlighted in the literature before in the context of variational inference, the proposed solution doesn't seem to solve it beyond, for instance, a mixture of VAEs.
Indeed, a mixture of VAEs would have been a great baseline for the experiments in the paper, as it uses a categorical variable (the mixture component) along with multiple VAEs. The main difference between a mixture and an epitomic VAE is the sharing of parameters between the different "mixture components" in the epitomic VAE case.
The experimental section presents misleading results.
1. The log-likelihood of the proposed models is evaluated with Parzen window estimator. A significantly more accurate lower bound on likelihood that is available for the VAEs is not reported. In reviewer's experience continuous MNIST likelihood of upwards of 900 nats is easy to obtain with a modestly sized VAE.
2. The exposition changes between dealing with binary MNIST and continuous MNIST experiments. This is confusing, because these versions of the dataset present different challenges for modeling with likelihood-based models. Continuous MNIST is harder to model with high-capacity likelihood optimizing models, because the dataset lies in a proper subspace of the 784-dimensional space (some pixels are always or almost always equal to 0), and hence probability density can be arbitrarily large on this subspace. Models that try to maximize the likelihood often exploit this option of maximizing the likelihood by concentrating the probability around the subspace at the expense of actually modeling the data. The samples of a well-tuned VAE trained on binary MNIST (or a VAE trained on continuous MNIST to which noise has been appropriately added) tend to look much better than the ones presented in experimental results.
3. The claim that the VAE uses its capacity to "overfit" to the training data is not justified. No evidence is presented that the reconstruction likelihood on the training data is significantly higher than the reconstruction likelihood on the test data. It's misleading to use a technical term like "overfitting" to mean something else.
4. The use of dropout in dropout VAE is not specified: is dropout applied to the latent variables, or to the hidden layers of the encoder/decoder? The two options will exhibit very different behaviors.
5. MNIST eVAE samples and reconstructions look more like a more diverse version of 2d VAE samples/reconstructions - they are blurry, the model doesn't encode precise position of strokes. This is consistent with an interpretation of eVAE as a kind of mixture of smaller VAEs, rather than a higher-dimensional VAE. It is misleading to claim that it outperforms a high-dimensional VAE based on this evidence.
In reviewer's opinion the paper is not yet ready for publication. A stronger baseline VAE evaluated with evidence lower bound (or another reliable method) is essential for comparing the proposed eVAE to VAEs.
Thank you very much for your review and feedback. We have significantly updated the paper and added experiments providing additional insight. We believe these address your concerns and hope you will consider updating your review.
With respect to the comment that the main difference between a mixture and an epitomic VAE is a sharing of parameters, we agree this is the case, and the mixture model (mVAE) can be considered an ablated version of the full eVAE. We have added experiments for this ablated version in Tables 1 and 2. The advantage of the shared parameters is that each epitome can also benefit from general features learned across the training set, and we analyze this quantitatively across different models in Fig. 6 and Table 1. The effect is more pronounced as encoder / decoder capacity becomes smaller, and as data complexity increases (TFD vs. MNIST).
With respect to numbered comments:
1. We found that lower bound on likelihood was not a good measure of comparing generation ability (in fact, better lower bound often had worse generation samples), and that the Parzen estimator was better for evaluation on this task. Our findings on the lower bound measure for evaluating generation ability are consistent with Kingma and Welling 2014;
In particular, the Fig. 2 and Fig. 5 in their paper shows better lower bound but worse generation as the latent dimension is increased. We therefore use the Parzen estimator in our experiments.
However, taking into account your suggestion, we have also included Sec. 8.3 in the appendix reporting both lower bound and Parzen numbers, and analysis of the difference. We agree that evaluation metric for generation still lacks an ideal solution, and we hope that this additional section as well as additional qualitative samples will provide useful insight into the problem.
2. Thank you for pointing this out. We have changed and clarified the experimental results to be consistently on continuous MNIST.  Lower bound numbers in the appendix are reported on binarized MNIST to be consistent with the literature.
3. We agree the usage of the term “overfitting” is confusing.  Our intent was not to describe the standard technical definition of overfitting as you pointed out, but instead the phenomenon that VAE learns to model well only regions of the posterior manifold near training samples, instead of generalizing to model the full  manifold well. (See also the 2nd point in the response to AnonReviewer5 below, who brought up the same confusion.)  We have already rephrased this in an earlier revision of the paper.
4. Dropout is applied to the hidden layers of the encoder and decoder. We have now clarified in the paper.
5. The referenced eVAE samples (e.g. Fig. 5) are somewhat blurry because an epitome size of K=2 was used for all examples, in order to qualitatively illustrate the effect of increasing total latent dimension D from D=2 to D=20 under a fixed epitome size. As Fig. 6 quantitatively shows, epitome size K=2 is suboptimal, and these were not the best samples obtained overall. We have included a new Fig. 7 with samples from the eVAE obtaining the highest log-density.
This paper replaces the Gaussian prior often used in a VAE with a group sparse prior. They modify the approximate posterior function so that it also generates group sparse samples. The development of novel forms for the generative model and inference process in VAEs is an active and important area of research. I don't believe the specific choice of prior proposed in this paper is very well motivated however. I believe several of the conceptual claims are incorrect. The experimental results are unconvincing, and I suspect compare log likelihoods in bits against competing algorithms in nats.
Some more detailed comments:
In Table 1, the log likelihoods reported for competing techniques are all in nats. The reported log likelihood of cVAE using 10K samples is not only higher than the likelihood of true data samples, but is also higher than the log likelihood that can be achieved by fitting a 10K k-means mixture model to the data (eg as done in "A note on the evaluation of generative models"). It should nearly impossible to outperform a 10K k-means mixture on Parzen estimation, which makes me extremely skeptical of these eVAE results.
However, if you assume that the eVAE log likelihood is actually in bits, and multiply it by log 2 to convert to nats, then it corresponds to a totally believable log likelihood. Note that some Parzen window implementations report log likelihood in bits. Is this experiment comparing log likelihood in bits to competing log likelihoods in nats? (also, label units -- eg bits or nats -- in table)
It would be really, really, good to report and compare the variational lower bound on the log likelihood!!
Alternatively, if you are concerned your bound is loose, you can use AIS to get a more exact measure of the log likelihood.
Even if the Parzen window results are correct, Parzen estimates of log likelihood are extremely poor. They possess any drawback of log likelihood evaluation (which they approximate), and then have many additional drawbacks as well.
The MNIST sample quality does not appear to be visually competitive.
Also -- it appears that the images are of the probability of activation for each pixel, rather than actual samples from the model. Samples would be more accurate, but either way make sure to describe what is shown in the figure.
There are no experiments on non-toy datasets.
I am still concerned about most of the issues I raised in my questions below.
Briefly, some comments on the authors' response:
1. "minibatches are constructed to not only have a random subset of training examples but also be balanced w.r.t. to epitome assignment (Alg. 1, ln. 4)."
Nice! This makes me feel better about why all the epitomes will be used.
2. I don't think your response addresses why C_vae would trade off between data reconstruction and being factorial. The approximate posterior is factorial by construction -- there's nothing in C_vae that can make it more or less factorial.
3. "For C_vae to have zero contribution from the KL term of a particular z_d (in other words, that unit is deactivated), it has to have all the examples in the training set be deactivated (KL term of zero) for that unit"
This isn't true. A standard VAE can set the variance to 1 and the mean to 0 (KL term of 0) for some examples in the training set, and have non-zero KL for other training examples.
4. The VAE loss is trained on a lower bound on the log likelihood, though it does have a term that looks like reconstruction error. Naively, I would imagine that if it overfits, this would correspond to data samples becoming more likely under the generative model.
5/6. See Parzen concerns above. It's strange to train a binary model, and then treat it's probability of activation as a sample in a continuous space.
6. "we can only evaluate the model from its samples"
I don't believe this is true. You are training on a lower bound on the log likelihood, which immediately provides another method of quantitative evaluation.
Additionally, you could use techniques such as AIS to compute the exact log likelihood.
7. I don't believe Parzen window evaluation is a better measure of model quality, even in terms of sample generation, than log likelihood.
Thank you very much for your review and feedback. We have significantly updated the paper and added experiments providing additional insight. We believe these address your concerns and hope you will consider updating your review.
Re: nats vs. bits
Our experimental results are in nats, and we have clarified in the paper.
While k-means can be considered a strong baseline,  it is not a ceiling.  It is actually quite reasonable to outperform this, and other works have also done so, e.g. Adversarial Autoencoders. We note that even with very large VAE models we are able to outperform k-means on MNIST.
Re: reporting log likelihood
As we also responded to AnonReviewer3, we found that lower bound on likelihood was not a good measure of comparing generation ability (in fact, better lower bound often had worse generation samples), and that the Parzen estimator was better for evaluation of this task. Our findings on the lower bound measure for evaluating generation ability are consistent with Kingma and Welling 2014;
in particular, Fig. 2 and Fig. 5 in their paper shows better lower bound but worse generation as the latent dimension is increased. We therefore use the Parzen estimator in our experiments.
However, taking into account your suggestion, we have also included Sec. 8.3 in the appendix reporting both lower bound and Parzen numbers, and analysis of the difference. We agree that evaluation metric for generation still lacks an ideal solution, and we hope that this additional section as well as additional qualitative samples will provide useful insight into the problem.
Re: MNIST sample quality
The MNIST samples (Fig. 5) are somewhat blurry because an epitome size of K=2 was used for all examples, in order to qualitatively illustrate the effect of increasing total latent dimension D from D=2 to D=20 under a fixed epitome size. (See also response point #5 to AnonReviewer3 below, who brought up the same confusion).  This was not the highest-performing model. We agree this is confusing and have included a new Fig. 7 with samples from the eVAE obtaining the highest log-density.
With respect to numbered comments:
1. Thank you for your feedback.
2. You are correct that the approximate posterior is factorial by construction. The insight we are referring to, is that this factorial construction encourages some units to be used solely for optimizing the reconstruction term as much as possible, and other units to be used solely for optimizing the KLD term (by making units inactive) as much as possible. This split between how units are used is the trade-off we are referring to.
3. In the statement “For C_vae to have zero contribution from the KL term of a particular z_d”, we mean that it is zero when summed over the entire training set.  We agree with your statement on the standard VAE;
however, in the case you mention where KL is 0 for some examples and non-zero for others, it is still nonzero when summed over the training set and not the “deactivated” unit we are referring to.  Our message with the referenced sentence is that a deactivated unit (KL term of zero for all examples) is what is optimal to strongly minimize the KL term for that unit in C_vae. We have updated that paragraph in the paper to explain more clearly.
4. I believe the confusion here is our usage of the term “overfitting”, which we did not intend to mean overfitting in the standard sense. Our intent was not to describe the standard technical definition of overfitting as you pointed out, but instead the phenomenon that VAE learns to model well only regions of the posterior manifold near training samples, instead of generalizing to model the manifold induced by the prior p(z). Other reviewers also brought up this confusion, and please refer to the 2nd point in the response to AnonReviewer5 below for further clarification. We have already rephrased this in an earlier revision of the paper.
5. We have changed and clarified the experiment to be on continuous MNIST. Please see updated results.
6 / 7.  We found that in practice likelihood bound was not a good measure of generation ability, which is our objective, and that the Parzen estimator on samples was a better measure (as discussed above in the “Re: reporting log likelihood” section). We have added Section 8.3 in the appendix with further discussion of this, and comparison with likelihood bound numbers.
We thank the reviewers for their helpful comments and suggestions.  Based on them, we have significantly updated the paper to include:
1. Comparisons with mVAE (mixture VAE), an ablated version of eVAE that does not share parameters between epitomes
2. Section 4.2 analyzing the effect of epitome size on generative performance
3. Section 4.3 analyzing the effect of encoder / decoder architectures on over-pruning and generative performance
4. Fig. 7 with qualitative samples from best eVAE models
5.  Section 8.3 in the Appendix comparing the effectiveness of likelihood lower bound and Parzen log-density as a metric for generation ability, and reporting numbers for both
6. Updating and clarifying Parzen experiments to be on MNIST and lower bound experiments to be on binarized MNIST, consistent with literature
We believe these updates address the reviewers' comments as well as provide additional insight.

Meta-review:
role: PC
data: comment: This paper addresses issues faced when using VAEs and the pruning of latent variables. Improvements to the paper have been accounted for and improved the paper, but after considering the rebuttal and discussion, the reviewers still felt that more was needed, especially in terms of applicability across multiple different data sets. For this reason, the paper is unfortunately not yet ready for inclusion in this year's proceedings.

Decision: Reject