Title: On the Quantitative Analysis of Decoder-Based Generative Models
Authors: Yuhuai Wu
Yuri Burda
Ruslan Salakhutdinov
Roger Grosse
Venue: ICLR 2017
Keywords: Deep learning
Unsupervised Learning

Abstract: The past several years have seen remarkable progress in generative models which produce convincing samples of images and other modalities. A shared component of some popular models such as generative adversarial networks and generative moment matching networks, is a decoder network, a parametric deep neural net that defines a generative distribution. Unfortunately, it can be difficult to quantify the performance of these models because of the intractability of log-likelihood estimation, and inspecting samples can be misleading. We propose to use Annealed Importance Sampling for evaluating log-likelihoods for decoder-based models and validate its accuracy using bidirectional Monte Carlo. Using this technique, we analyze the performance of decoder-based models, the effectiveness of existing log-likelihood estimators, the degree of overfitting, and the degree to which these models miss important modes of the data distribution.

TL;DR: We propose to use Annealed Importance Sampling to evaluate decoder-based generative network, and investigate various properties of these models.

Review discussion:
Does "running the decoder on z" (page 9) refer to sampling from $p_\sigma(x | z)$  or deterministic reconstruction?
It refers to deterministic reconstruction, which can also be interpreted as the mean of the distribution $p_\sigma(x | z)$.
Regarding equations leading to Equation 5, does f_1(z) represents the unnormalized part of p_1(z)=p(z) (which is the prior p(z) in this case according to the text at the top of page 4)? I think f_1(z) = p(z) (or maybe Z_1=1 ?) is needed to obtain Equation 5.
For the Figure 1, have tried evaluating the images according to the image-quality measure (Salimans et al.)? If so, are the values similar between the three models?
When you report computation time, are the implementations of IWAE and KDE use GPU?
Also, in Figures 1, 2 and 3, what is LLD stands for?
1) Yes, f_t(z) is the unnormalized intermediate distributions. The initial distribution has f_1(z) = p_1(z) = p(z), with normalizing constant 1.
2) Have not tried evaluating the inception score. Note that the inception score is obtained by input the images into "Inception" model, which is usually trained on color-images (imagenet). I will try training a MNIST classifier to obtain a "MNIST" score if that's needed.
3) Yes. We report the best numbers for IWAE and KDE evaluation using GPUs and CPUs.
4) LLD stands for log-likelihood,  see the label on y-axis. Abbreviated because of space limit.
The paper describes a method to evaluate generative models such as VAE, GAN and GMMN. This is very much needed in our community where we still eyeball generated images to judge the quality of a model.
However, the technical increment over the NIPS 16 paper: “Measuring the reliability of MCMC inference with bidirectional Monte Carlo” is very small, or nonexistent (but please correct me if I am wrong!).  (Grosse et al). The relative contribution of this paper is the application of this method to generative models.
In section 2.3 the authors seem to make a mistake. They write E[p’(x)] <= p(x) but I think they mean: E[log p’(x)] <= log E[p’(x)] = log p(x).
Also,  for what value of x? If p(x) is normalized it can’t be true for all values of x. Anyways, I think there are typos here and there and the equations could be more precise.
On page 5 top of the page it is said that the AIS procedure can be initialized with q(z|x) instead of p(z).
However, it is unclear what value of x is then picked? Is it perhaps Ep(x)[q(z|x)] ?
I am confused with the use of the term overfitting (p8 bottom). Does a model A overfit relative to a another model B if the test accuracy of A is higher than that of B even though the gap between train and test accuracy is also higher for B than for A. I think not. Perhaps the last sentence on page 8 should say that VAE-50 underfits less than GMMN-50?
The experimental results are interesting in that it exposes the fact that GANs and GMMNs seem to have much lover test accuracy than VAE despite the fact that their samples look great.
Thank the reviewer for providing helpful comments.
Relative contribution small:
It’s encouraging that our writing is clear enough that the ideas seem to flow naturally.
But the approach is obvious only in retrospect. There has been a large amount of interest in the questions surrounding likelihoods of decoder-based models, including whether the models are missing modes or memorizing training examples. Our submission is the first to quantitatively analyze these issues using accurate log-likelihood estimates. Anecdotally, in our conversations with other researchers who cared about these questions and were familiar with the basic techniques, nobody suggested using a similar approach.
Here are some of our novel contributions: (1) formulating KDE as simple importance sampling, which shows that it gives a log-likelihood stochastic lower bound for a particular generative model (with Gaussian observations); this let us formulate the problem of estimating the log-likelihoods more accurately under this generative model; (2) using BDMC to validate the log-likelihood estimates in decoder-based models (a very different setting from where it was previously applied); (3) using the q network as the initial distribution, which makes the AIS estimator very efficient for VAEs, (4) using posterior samples as a way to diagnose missing modes.
Much progress in science comes about because someone designs and validates a new genetic technique and then uses it to measure phenomena that were previously inaccessible. This is exactly what we wanted to achieve in this work: we engineered and validated a much improved log-likelihood estimator (orders of magnitude more accurate than the previous approach!), and then used it to answer important questions about generative models which we previously didn’t have a way to answer.
Section 2.3: Thanks reviewer for pointing out the typo with missing “log”. We have corrected it. Note that the expectation is taken over the stochasticity of producing the estimate log p’(x). Mathematically, the inequality holds for every x (not just on average w.r.t. x).
q(z|x): The approximate posterior, i.e., the recognition network is used as the initialization distribution for AIS chains. The x refers to the given data to be evaluated.
overfitting:
What we mean by “overfitting” follows the standard usage in the field. A model overfits if it models idiosyncrasies in the training set, thereby achieving a training likelihood much higher than its generalization (test) likelihood. A model underfits if it fails to model the structure in the training set.
Despite the names, the two are not mutually exclusive, i.e. a given model can both overfit and underfit the data. (For instance, if a network somehow memorized 50% of the training examples and ignored the other 50%, it would both overfit and underfit, as reflected in poor likelihood on the training set coupled with a large gap between training and test.)
You are correct that it’s not generally meaningful to compare the “amount of overfitting” of two different models -- overfitting isn’t a quantity we can measure.
However, in our experiments, the size of the effect was large: the VAEs overfit substantially, whereas we saw no evidence of overfitting in the case of the GANs or GMMNs.
Summary:
This paper describes how to estimate log-likelihoods of currently popular decoder-based generative models using annealed importance sampling (AIS) and HMC. It validates the method using bidirectional Monte Carlo on the example of MNIST, and compares the performance of GANs and VAEs.
Review:
Although this seems like a fairly straight-forward application of AIS to me (correct me if I missed an important trick to make this work), I very much appreciate the educational value and empirical contributions of this paper. It should lead to clarity in debates around the density estimation performance of GANs, and should enable more people to use AIS.
Space permitting, it might be a good idea to try to expand the description of AIS. All the components of AIS are mentioned and a basic description of the algorithm is given, but the paper doesn’t explain well “why” the algorithm does what it does/why it works.
I was initially confused by the widely different numbers in Figure 2. On first glance my expectation was that this Figure is comparing GAN, GMMN and IWAE (because of the labeling at the bottom and because of the leading words in the caption’s descriptions). Perhaps mention in the caption that (a) and (b) use continuous MNIST and (c) uses discrete MNIST. “GMMN-50” should probably be “GMMN-10”.
Using reconstructions for evaluation of models may be a necessary but is not sufficient condition for a good model. Depending on the likelihood, a posterior sample might have very low density under the prior, for example. It would be great if the authors could point out and discuss the limitations of this test a bit more.
Minor:
Perhaps add a reference to MacKay’s density networks (MacKay, 1995) for decoder-based generative models.
In Section 2.2, the authors write “the prior over z can be drastically different than the true posterior p(z|x), especially in high dimension”. I think the flow of the paper could be improved here, especially for people less familiar with importance sampling/AIS. I don’t think the relevance of the posterior for importance sampling is clear at this point in the paper.
In Section 2.3 the authors claim that is often more “meaningful” to estimate p(x) in log-space because of underflow problems. “Meaningful” seems like the wrong word here. Perhaps revise to say that it’s more practical to estimate log p(x) because of underflow problems, or to say that it’s more meaningful to estimate log p(x) because of its connection to compression/surprise/entropy.
We thank the reviewer for recognizing its educational value.
“How AIS works?” -- We have edited the paper to include a short discussion on this.
“Figure 2”-- Thanks for pointing out the typo. We changed the caption as the reviewer suggests.
“Using reconstruction” -- We agree with the reviewer. Visualizing reconstruction is a way to show one particular property of the model: whether it misses any important modes of the data distribution. And we also agree that this is not a sufficient condition to be a good generative model.
“Density networks” -- We have added the citation.
“AIS/posterior”-- We edited the paper to show the relevance.
“Meaningful”-- We changed the phrasing.
# Review
This paper proposes a quantitative evaluation for decoder-based generative models that use Annealed Importance Sampling (AIS) to estimate log-likelihoods. Quantitative evaluations are indeed much needed since for some models, like Generative Adversarial Networks (GANs) and Generative Moment Matching Networks (GMMNs), qualitative evaluation of samples is still frequently used to assess their generative capability.
Even though, there exist quantitative evaluations like Kernel Density Estimation (KDE), the authors show how AIS is more accurate than KDE and how it can be used to perform fine-grained comparison between generative models (GAN, GMMs and Variational Autoencoders (VAE)).
The authors report empirical results comparing two different decoder architectures that were both trained, on the continuous MNIST dataset, using the VAE, GAN and GMMN objectives. They also trained an Importance Weighted Autoencoder (IWAE) on binarized MNIST and show that, in this case, the IWAE bound underestimates the true log-likelihoods by at least 1 nat (which is significant for this dataset) according to the AIS evaluation of the same model.
# Pros
Their evaluation framework is public and is definitely a nice contribution to the community.
This paper gives some insights about how GAN behaves from log-likelihood perspective. The authors disconfirm the commonly proposed hypothesis that GAN are memorizing training data. The authors also observed that GANs miss important modes of the data distribution.
# Cons/Questions
It is not clear for me why sometimes the experiments were done using different number of examples (100, 1000, 10000) coming from different sources (trainset, validset, testset or simulation/generated by the model).
For instance, in Table 2 why results were not reported using all 10,000 examples of the testing set?
It is not clear why in Figure 2c, AIS is slower than AIS+encoder? Is the number of intermediate distributions the same in both?
16 independent chains for AIS seems a bit low from what I saw in the literature (e.g. in [Salakhutdinov & Murray, 2008] or [Desjardins etal., 2011], they used 100 chains). Could it be that increasing the number of chains helps tighten the confidence interval reported in Table 2?
I would have like the authors to give their intuitions as to why GAN50 has a BDMC gap of 10 nats, i.e. 1 order of magnitude compared to the others?
# Minor comments
Table 1 is not referenced in the text and lacks description of what the different columns represent.
Figure 2(a), are the reported values represents the average log-likelihood of 100 (each or total?) training and validation examples of MNIST (as described in Section 5.3.2).
Figure 2(c), I'm guessing it is on binarized MNIST?
Also, why are there fewer points for AIS compared to IWAE and AIS+encoder?
Are the BDMC gaps mentioned in Section 5.3.1 the same as the ones reported in Table2 ?
Typo in caption of Figure 3: "(c) GMMN-10" but actually showing GMMN-50 according to the graph title and subcaption.
Thank the reviewer for providing helpful comments.
Different number of examples:
We ran our experiments on academic scale infrastructure, so we didn’t have the resources to run the maximum number of examples in all conditions. We will release an updated version where all experiments use the maximum number of test examples.
However, we reported confidence intervals for the results, and even with only 1000 examples for the training sets, the differences between models were still highly statistically significant.
AIS+encoder:
AIS works better when the initial distribution is a better match to the target (posterior) distribution. In those cases where an encoder net is available, it ought to provide a better initial distribution than the prior (which is what we used in the rest of the experiments).
Therefore, AIS+encoder can reach comparable accuracy in a much smaller number of steps.
# of AIS chains:
We have tested different number of independent chains. In our experiments we find that when the number of intermediate distribution goes beyond 1000, the evaluation does not vary much for different choices of number of chains. 16 chains are enough for evaluation.
Large gaps for GAN-50:
We speculate that the posterior distribution might be more complicated in the case of GAN-50.
However, the difference between GAN-50 and the other conditions is not necessarily surprising or interesting; there’s no reason that different models ought to have posterior distributions which are equally difficult to sample from. We suspect the fact that the other five models have very similar BDMC gaps is just a coincidence.
Minor comments:
We have edited table 1 and the reference in the paper
They are average value.
Adding another point for AIS will either make the plot too large in y-axis or x-axis.
Yes the BDMC gap refer to the values in Table 2.
Thanks for pointing out this typo.
A new version of the paper is updated. We revised the section 2.2 on the introduction to AIS, and several other places according to reviewers' comments.

Meta-review:
role: PC
data: comment: This paper describes a method to estimate likelihood scores for a range of models defined by a decoder.
 
 This work has some issues. The paper mainly applies existing ideas. As discussed on openreview, the isotropic Gaussian noise model used to create a model with a likelihood is questionable, and it's unclear how useful likelihoods are when models are obviously wrong. However, the results, lead to some interesting conclusions, and on balance this is a good paper.

Decision: Accept (Poster)