Bounding the Test Log-Likelihood of Generative Models
Authors: Yoshua Bengio, Li Yao, KyungHyun Cho
ICLR 2014

Abstract: Several interesting generative learning algorithms involve a complex probability distribution over many random variables, involving intractable normalization constants or latent variable normalization. Some of them may even not have an analytic expression for the unnormalized probability function and no tractable approximation. This makes it difficult to estimate the quality of these models, once they have been trained, or to monitor their quality (e.g. for early stopping) while training. A previously proposed method is based on constructing a non-parametric density estimator of the model's probability function from samples generated by the model. We revisit this idea, propose a more efficient estimator, and prove that it provides a lower bound on the true test log-likelihood, and an unbiased estimator as the number of generated samples goes to infinity, although one that incorporates the effect of poor mixing (making the estimated likelihood worse, i.e., more conservative).


Peer review:
[Review]:
In this paper, the authors propose a new way to estimate the probability of data under a probabilistic model from which sampling is hard but for which an efficient Markov chain procedure exists. They first present an asymptotically unbiased estimator, then a more efficient biased estimator.

The idea is undeniably interesting. Some of the most used generative models satisfy these constraints and being able to calculate the probability of data under these models is crucial to comparing them. However, the results presented in this paper are underwhelming. For models where AIS was usable (the DBN, the DBM and the RBM), the CSL results wildly differ from the AIS ones. Since the results on the small RBM (Table 2) give a clear advantage to AIS, I am inclined to believe these results more.

Another caveat, unfortunately extremely difficult to avoid, is that the effectiveness of these methods can only be empirically proven on tiny models where mixing problems do not occur. I really do not blame the authors for that but this really limits the potential impact of the method.

Experiment in Figure 1 is also very light to conclude on the effectiveness of Biased CSL. Binary MNIST is a very particular dataset and this experiment does not convince me that it is actually usable to compare models, especially of different types.

Conclusion: this paper does not prove the effectiveness of the proposed method. The propositions are not worth publication by themselves.

Other comments:
- CSL is only a lower bound on the true log probability of the data in expectation. This should be made clearer in the paper.
- The pseudo code should either be commented or removed entirely. As it is, it is only useful to people who already understood the algorithm.
- Could you give more details on the parameters for AIS? How many chains? How many intermediate temperatures? How does the computation time compare to CSL?