Bounding the Test Log-Likelihood of Generative Models
Authors: Yoshua Bengio, Li Yao, KyungHyun Cho
ICLR 2014

Abstract: Several interesting generative learning algorithms involve a complex probability distribution over many random variables, involving intractable normalization constants or latent variable normalization. Some of them may even not have an analytic expression for the unnormalized probability function and no tractable approximation. This makes it difficult to estimate the quality of these models, once they have been trained, or to monitor their quality (e.g. for early stopping) while training. A previously proposed method is based on constructing a non-parametric density estimator of the model's probability function from samples generated by the model. We revisit this idea, propose a more efficient estimator, and prove that it provides a lower bound on the true test log-likelihood, and an unbiased estimator as the number of generated samples goes to infinity, although one that incorporates the effect of poor mixing (making the estimated likelihood worse, i.e., more conservative).


Peer review:
[Review]:
Dear Reviewer (0661),

We thank you for the thorough and insightful comment. Allow us to respond to some of your comments below.

'it places the idea of used by Breuleux et al. in a rigorous framework'

We agree that the proposed method is closely related to that by Breuleux et al. However, we claim that it is an improvement over the method by Breuleux et al. in two ways:

(1) the CSL is more efficient because each sample of latent variables h can cover many x's.
(2) the CSL does not have any tuning parameter such as bandwidth.

'it exhibits very substantial divergence from AIS on larger models, as shown in Table 1.'

As you have pointed out earlier in your review, the proposed CSL estimator does not only evaluate the model itself but also the sampling procedure. When 'mixing' by MCMC sampling is fast (as in GSN), the CSL estimate tend to converge quickly, while in the opposite case (as in RBM and DBN), the convergence is slow. 

'the number of samples that's needed in order to accurate compute a log probability grows exponentially with the entropy of the distribution'

This is true, but one thing to be noted is that we are not aware of any alternative approach that does not suffer from this problem, when an approach such as AIS is not applicable.  Furthermore, we believe the fact that the CSL estimator uses the samples of latent variables h which are in a more abstract space than the raw input space, may make sampling-based methods such as the proposed CSL greatly reduce the curse of dimensionality.

Nonetheless, we agree that this is where future work lies, and we are indeed currently exploring different ways of exploiting the presence of a high-level (deep) representation to make the problem of likelihood estimation much easier and its convergence faster. Much more work is needed before these new ideas can be proven right and this paper should instead be judged in comparison with the past published work.