Bounding the Test Log-Likelihood of Generative Models
Authors: Yoshua Bengio, Li Yao, KyungHyun Cho
ICLR 2014

Abstract: Several interesting generative learning algorithms involve a complex probability distribution over many random variables, involving intractable normalization constants or latent variable normalization. Some of them may even not have an analytic expression for the unnormalized probability function and no tractable approximation. This makes it difficult to estimate the quality of these models, once they have been trained, or to monitor their quality (e.g. for early stopping) while training. A previously proposed method is based on constructing a non-parametric density estimator of the model's probability function from samples generated by the model. We revisit this idea, propose a more efficient estimator, and prove that it provides a lower bound on the true test log-likelihood, and an unbiased estimator as the number of generated samples goes to infinity, although one that incorporates the effect of poor mixing (making the estimated likelihood worse, i.e., more conservative).


Peer review:
[Review]:
This paper proposes an estimator for the log-likelihood of intractable models with latent variables. The approach is simple in that it has no free parameters, doesn’t require an explicit likelihood, and only needs samples from the model. The approach is most useful for model comparison since the estimate is conservative rather than optimistic.

I enjoyed reading this paper. The proposed method is quite novel and elegant and has the potential to be a useful tool for model comparison. One issue is that the estimator seems to require a large number of samples in order to converge, and this potentially exacerbated by increasing model size. As stated in the paper, this is likely to do with the convergence of MCMC within the model itself. One empirical test of this would be to compare the efficiency of the estimator with exact samples vs MCMC in e.g., a small RBM.

The biased CSL is also novel, but seems to be even more optimistic than AIS. The argument of the paper is based on the idea that we would prefer conservative estimates to optimistic estimates for model comparison. When would the authors expect the biased CSL method to be useful in practice? How many steps would be required before biased CSL matches AIS?

Minor thoughts and some found typos below.

1. In table 1 the AIS and CSL estimates are vastly different. One is optimistic and one is conservative - which one is closer to the truth? Is there a reason that GSN is so much better? Obviously the truth is impossible to determine, but it is clear that more samples are needed before the estimate converges.
2. The RBM used in table 2 is quite small, using only 5 hidden units. 20 hidden units is slightly larger but still tractable. It would be good to see how the efficiency of the estimator is affected by model size.
3. An RBM trained with PCD is thought to have much better likelihood than an RBM trained with CD. Is this reflected in CSL estimates?

formulat -> formulate (section 1)
collecte -> collect (section 2)
in -> in (Monte-Carlo estimator in section 4)
30 steps -> 300 steps (or the the legend in Figure 1 has a typo, section 6)
mode -> model (section 7)