Venue: ICLR 2017 conference

Reviews:

final_score: rating: 5: Marginally below acceptance threshold
review_content: The paper proposes two approaches to boosting generative models, both based on likelihood ratio estimates. The approaches are evaluated on synthetic data, as well as on MNIST dataset for the tasks of generating samples and semi-supervised learning.
While the idea of boosting generative models and the proposed methods are interesting, the reviewer finds the experiments unconvincing for the following reasons.
1. The bagging baseline in section 3.1 seems to be just refitting a model to the same dataset, raising the probability to power alpha, and renormalizing. This makes it more peaked, but it's not clear why this is a good baseline. Please let me know if I misunderstood the procedure.
2. The sample generation experiment in section 3.2 uses a very slowly converging Markov chain, as can be seen in the similarity of plots c and f, d and g, e and h. It seems unlikely therefore that the resulting samples are from the stationary distribution. A qualitative evaluation using AIS seems to be necessary here.
3. In the same section the choices for alphas seem quite arbitrary - what happens when a more obvious choice of alpha_i=1 for all i is made?
4. It seems hard to infer anything from the semisupervised classification results reported: the baseline RBM seems to perform as well as the boosted models.

The work is mostly clearly written and (as far as the reviewer knows) original.
review_title: Interesting work, many weak points

final_score: rating: 5: Marginally below acceptance threshold
review_content: The authors propose two approaches to combine multiple weak generative models into a stronger one using principles from boosting. The approach is simple and elegant and basically creates an unnormalized product of experts model, where the individual experts are trained greedily to optimize the overall joint model. Unfortunately, this approach results in a joint model that has some undesirable properties: a unknown normalisation constant for the joint model and therefore an intractable log-likelihood on the test set; and it makes drawing exact samples from the joint model intractable. These problems can unfortunately not be fixed by using different base learners, but are a direct result of the product of experts formulation of boosting.
  
The experiments on 2 dimensional toy data illustrate that the proposed procedure works in principle and that the boosting formulation produces better results than individual weak learners and better results than e.g. bagging. But the experiments on MNIST are less convincing: Without an undisputable measure like e.g. log-likelihood it is hard to draw conclusions from the samples in Figure 2; and visually they look weak compared to even simple models like e.g. NADE.

I think the paper could be improved significantly by adding a quantitative analysis: investigating the effect of combining undirected (e.g. RBM), undirected (e.g. VAE) and autoregressive (e.g. NADE) models and by measuring the improvement over the number of base learners. But this would require a method to estimate the partition function Z or estimating some proxy.

review_title: Simple & elegant approach -- but weak results & a model with some undesirable properties.

final_score: rating: null
review_content: Could the authors provide a more detailed comparison to the work of Tu (2007) ? There seems to be a significant overlap with Sec. 2.2 (including Theorem 2 and its proof), which is not fairly represented in the related works section IMO. I believe more credit and transparency in this regard would benefit the paper, without adversely impacting novelty.

As for the experimental section, what is the added benefit of mixing various base-learners ? Even though the method allows for it, there is no clear rationale for doing so. Unfortunately, this seems to come at the expense of careful evaluation of each approach. As it stands, the experimental section does not convincingly answer the following questions: which approach of DiscGBM or GenGBM is better ? In which condition does boosting outperform the ML baselines ? Table 1 seems to imply there may a win in terms of training time, but this seems unconvincing without likelihoods estimates, convergence curves, etc. In particular, the samples obtained by the baseline models on MNIST seem poor compared to what can be found in the literature.

review_title: Request for clarifications

final_score: rating: 6: Marginally above acceptance threshold
review_content: This paper extends boosting to the task of learning generative models of data. The strong learner is obtained as a geometric average of “weak learners”, which can themselves be normalized (e.g. VAE) or un-normalized (e.g. RBMs) generative models (genBGM), or a classifier trained to discriminate between the strong learner at iteration T-1 and the true data distribution (discBGM). This latter method is closely related to Noise Contrastive Estimation, GANs, etc.

The approach benefits from strong theoretical guarantees, with strict conditions under which each boosting iteration is guaranteed to improve the log-likelihood. The downside of the method appears to be the lack of normalization constant for the resulting strong learner and the use of heuristics to weight each weak learner (which seems to matter in practice, from Sec. 3.2). The discriminative approach further suffers from an expensive training procedure: each round of boosting first requires generating a “training set” worth of samples from the previous strong learner, where samples are obtained via MCMC.

The experimental section is clearly the weak point of the paper. The method is evaluated on a synthetic dataset, and a single real-world dataset, MNIST: both for generation and as a feature extraction mechanism for classification. Of these, the synthetic experiments were the clearest in showcasing the method. On MNIST, the baseline models are much too weak for the results to be convincing. A modestly sized VAE can obtain 90 nats within hours on a single GPU, clearly an achievable goal. Furthermore, despite arguments to the contrary, I firmly believe that mixing base learners is an academic exercise, if only because of the burden of implementing K different models & training algorithms. This section fails to answer a more fundamental question: is it better to train a large VAE by maximizing the elbow, or e.g. train 10 iterations of boosting, using VAEs 1/10th the size of the baseline model ? Experimental details are also lacking, especially with respect to the sampling procedure used to draw samples from the BGM. The paper would also benefit from likelihood estimates obtained via AIS.

With regards to novelty and prior work, there is also a missing reference to “Self Supervised Boosting” by Welling et al [R1]. After a cursory read through, there seems to be strong similarities to the GenBGM approach which ought to be discussed.

Overall, I am on the fence. The idea of boosting generative models is intriguing, seems well motivated and has potential for impact. For this reason, and given the theoretical contributions, I am willing to overlook some of the issues highlighted above, and hope the authors can address some of them in time for the rebuttal.

[R1] https://papers.nips.cc/paper/2275-self-supervised-boosting.pdf

PROS:
Novel and intriguing idea
Strong theoretical guarantees

CONS:
Resulting boosted model is un-normalized
Discriminator based boosting is expensive, due to sampling via MCMC
Weak experimental section

review_title: A promising idea with strong theoretical contributions, but poor experimental validation

Meta-review: The idea of boosting has recently seen a revival, and the ideas presented here are stimulating. After discussion, the reviewers agreed that the latest updates and clarifications have improved the paper, but overall they still felt that the paper is not quite ready, especially in making the case for when taking this approach is desirable, which was the common thread of concern for everyone. For this reason, this paper is not yet ready for acceptance at this year's conference.
Decision: Reject