Task: Language Identification
Input (Spanish, English): The paper describes a method to evaluate generative models such as VAE, GAN and GMMN. This is very much needed in our community where we still eyeball generated images to judge the quality of a model. However, the technical increment over the NIPS 16 paper: “Measuring the reliability of MCMC inference with bidirectional Monte Carlo” is very small, or nonexistent (but please correct me if I am wrong!).  (Grosse et al). The relative contribution of this paper is the application of this method to generative models.  In section 2.3 the authors seem to make a mistake. They write E[p’(x)] <= p(x) but I think they mean: E[log p’(x)] <= log E[p’(x)] = log p(x). Also,  for what value of x? If p(x) is normalized it can’t be true for all values of x. Anyways, I think there are typos here and there and the equations could be more precise. On page 5 top of the page it is said that the AIS procedure can be initialized with q(z|x) instead of p(z). However, it is unclear what value of x is then picked? Is it perhaps Ep(x)[q(z|x)] ? I am confused with the use of the term overfitting (p8 bottom). Does a model A overfit relative to a another model B if the test accuracy of A is higher than that of B even though the gap between train and test accuracy is also higher for B than for A. I think not. Perhaps the last sentence on page 8 should say that VAE-50 underfits less than GMMN-50? The experimental results are interesting in that it exposes the fact that GANs and GMMNs seem to have much lover test accuracy than VAE despite the fact that their samples look great.
Output (English): en