Title: BOOSTED GENERATIVE MODELS

Abstract: We propose a new approach for using boosting to create an ensemble of generative models, where models are trained in sequence to correct earlier mistakes. Our algorithm can leverage many existing base learners, including recent latent variable models. Further, our approach allows the ensemble to leverage discriminative models trained to distinguish real from synthetic data during sample generation. We show theoretical conditions under which incorporating a new model to the ensemble will improve the fit and empirically demonstrate the effectiveness of boosting on density estimation, sample generation, and unsupervised feature learning on real and synthetic datasets.

1. INTRODUCTION:

Many of the recent successful applications of machine learning in computer vision, speech recognition, and natural language processing are based on discriminative models. Learning generative models has proven to be much more difficult. Deep architectures, including latent variable models such as Boltzmann machines (Smolensky, 1986) , variational autoencoders (Kingma & Welling, 2014) , and generative adversarial networks (Goodfellow et al., 2014) , have recently shown great success. Despite significant progress, existing generative models cannot fit complex distributions with a sufficiently high degree of accuracy.

In this paper, we propose a technique for ensembling (imperfect) generative models to improve their overall performance. Our meta-algorithm is inspired by boosting, a powerful technique used in supervised learning to construct ensembles of weak classifiers (e.g., decision stumps or trees), which individually might not perform well on a given classification task. The boosting algorithm will attempt to learn a classifier to correct for the mistakes made and repeat this procedure recursively. Under some conditions on the weak classifiers' effectiveness, the boosting meta-algorithm can drive the (training) error to zero (Freund et al., 1999) . Boosting can also be thought as a feature learning algorithm, where at each round a new feature is learned by training a classifier on a re-weighted version of the original dataset. In practice, algorithms based on boosting (such as boosted trees) perform extremely well in machine learning competitions (Caruana & Niculescu-Mizil, 2006) .

We show that a similar procedure can be applied to generative models. Given an initial generative model that provides an imperfect fit to the data distribution, we construct a second model to correct for the error, and repeat recursively. The second model is also a generative one, which is trained on a re-weighted version of the original training set. Our meta algorithm is general and can construct ensembles of many existing generative models such as restricted Boltzmann machines and variational autoencoders. Surprisingly, our method can even leverage discriminative models, which have been shown to perform extremely well in practice (Krizhevsky et al., 2012; LeCun et al., 2015) . Specifically, we train a discriminator to distinguish true data samples from "fake" ones generated by the current model and provide a principled way to include this discriminator in the ensemble. We also provide conditions under which adding a model to the ensemble is guaranteed to improve the fit and recover the true data distribution under idealized conditions.

To evaluate our algorithmic framework, we learn several ensembles of weakly-trained generators and discriminators and test them on popular use cases of generative models: density estimation, sample generation, and unsupervised feature learning. We show how boosted generative models can outperform baseline models without any additional computation cost.

2. BOOSTING GENERATIVE MODELS:

Boosting is an ensembling technique for supervised learning, providing an algorithmic formalization of the hypothesis that a sequence of weak learners can create a single strong learner (Schapire & Freund, 2012) . In this section, we propose a framework that extends boosting to unsupervised settings for learning joint distributions using generative models. For ease of presentation, all distributions are w.r.t. any arbitrary x ∈ R d , unless otherwise specified.

Formally, we consider the following maximum likelihood estimation (MLE) setting. Given some data points X = {x i ∈ R d } m i=1 sampled i.i.d. from an unknown distribution with p.d.f. p, we provide a model class Q parametrizing the distributions that can be represented by the generative model and minimize the KL-divergence w.r.t. the true distribution,

EQUATION

In practice, we only observe samples from p and hence, maximize the log-likelihood of the observed data X. Selecting the model class for maximum likelihood learning is non-trivial; the maximum likelihood estimate w.r.t. a small class can be very far from the true distribution whereas a large class poses the risk of overfitting.

2.1. FACTORED LIKELIHOODS FOR UNSUPERVISED BOOSTING:

In unsupervised boosting, we factorize the joint distribution specified by a generative model as a geometric average of T + 1 intermediate model distributions {h t } T t=0 , each assigned an exponentiated weight α t ,

q T = T t=0 h αt t Z T

where the partition function Z T = T t=0 h αt t dx. Because a joint optimization over all the intermediate model distributions and weights is computationally prohibitive, we instead perform a greedy optimization at every round. The joint distribution of a boosted generative model (BGM) can be recursively expressed as,

qt = h αt t • qt-1 (2)

where qt is the unnormalized BGM distribution (at round t). The base model distribution h 0 is learned using the maximum likelihood principle. Given suitable weights, we now derive conditions on the intermediate model distributions that allow us to make "progress" in every round of boosting.

Theorem 1. Let δ t KL (h t , α t ) = D KL (p q t-1 ) -D KL (p q t ) denote the reduction in KL divergence at the t th round of boosting. Then, for all 0 ≤ α t ≤ 1, the following conditions hold,

1. Sufficient: If E p [log h t ] ≥ log E qt-1 [h t ], then δ t KL (h t , α t ) ≥ 0. 2. Necessary: If δ t KL (h t , α t ) ≥ 0, then E p [log h t ] ≥ E qt-1 [log h t ]. Proof. See Appendix A.1.1. Algorithm 1 DiscBGM(X = {x i } m i=1 , T rounds) Initialize D 0 (x i ) = 1/m for all i = 1, 2, . . . , m. Set (unnormalized) density estimate q0 = h 0 Train generative model h 0 to maximize E xi∼D0 [log h 0 (x i )] for t = 1, . . . , T do • Generate k negative samples from q t-1 • Train discriminative model d t to maximize E xi∼D0 [log d t ] + E xi∼qt-1 [log(1 -d t )]. • Set h t = γ • dt 1-dt where γ = k/m. • Choose α t . • Set (unnormalized) density estimate qt = h αt t • qt-1 . end for Estimate Z T = qT (x)dx. return q T = qT /Z T

The equality in the above conditions holds true for α t = 0 which corresponds to the trivial case where the intermediate model distribution in the current round is ignored in the BGM distribution in Eq. ( 2). For all other valid α t > 0, the non-degenerate versions of the sufficient inequality guarantees progress towards the true data distribution. Note that the intermediate models increase the overall capacity of the BGM at every round.

From the necessary condition, we see that a "good" intermediate density h t necessarily assigns a better-or-equal log-likelihood under the true desired distribution as opposed to the BGM distribution in the previous round, q t-1 . This condition suggests two learning objectives for intermediate models which we discuss next.

2.2. DISCRIMINATIVE APPROACH FOR BOOSTING GENERATIVE MODELS:

In the discriminative approach for boosting generative models, the intermediate model distribution is specified as the odds ratio of a binary classifier. Specifically, consider the following binary classification problem: we observe m samples drawn i.i.d. from the true data distribution p (w.l.o.g. assigned the label y = +1), and k samples drawn i.i.d. from the BGM distribution in the previous round q t-1 (assigned the label y = -1). The objective of the binary classifier is to learn a conditional distribution d t ∈ D t that maximizes the cross-entropy,

max dt∈Dt E x∼p [log d t ] + E x∼qt-1 [log(1 -d t )].

(3) Definition 1. If u t denotes the joint distribution over (x, y) at round t, then a binary classifier with density d t is Bayes optimal iff,

d t (x) = u t (y = +1 | x).

Theorem 2. If a binary classifier d t trained to optimize Eq. ( 3) is Bayes optimal, then the BGM distribution at the end of the round will immediately converge to the true data distribution if we set α t = 1 and

EQUATION

)

where γ = k/m.

Proof. See Appendix A.1.2.

In practice, a classifier with limited capacity trained on a finite dataset will not generally be Bayes optimal. The above theorem, however, suggests that a good classifier can provide a "direction of improvement". Additionally, if the intermediate model distribution h t obtained using Eq. ( 4) satisfies the conditions in Theorem 1, it is guaranteed to improve the BGM distribution.

Algorithm 2 GenBGM(X = {x i } m i=1 , T rounds) Initialize D 0 (x i ) = 1/m for all i = 1, 2, . . . , m. Train generative model h 0 to maximize E xi∼D0 [log h 0 (x i )].

Set (unnormalized) density estimate q0 = h 0 for t = 1, 2, . . . , T do • Update D t using Eq. ( 5).

• Train generative model

h t to maximize E xi∼Dt [log h t (x i )]. • Choose α t . • Set (unnormalized) density estimate qt = qt-1 • h αt t . end for Estimate Z T = qT (x)dx. return q T = qT /Z T .

The pseudocode for the corresponding boosting algorithm DiscBGM is given in Algorithm 1. At every round of boosting, we train a binary classifier to optimize the objective in Eq. ( 3). Note that the BGM distributions at the intermediate boosting rounds can be specified up to a normalization constant if samples from the previous BGM distribution are generated via MCMC sampling. If the partition function is required, it can be estimated using techniques such as Annealed Importance Sampling (Neal, 2001) . 1The weights 0 ≤ α t ≤ 1 can be interpreted as our confidence in the classifier density estimate. While in practice we use heuristic strategies for assigning weights to the intermediate models, the greedy optimum value of these weights at every round is a critical point for δ t KL (defined in Theorem 1). We consider a few special cases below.

• If d t is uninformative, i.e., d t ≡ 0.5, then δ t KL (h t , α t ) = 0 for all 0 ≤ α t ≤ 1. • If d t is Bayes optimal, then δ t KL attains a maxima at α t = 1. (Theorem 2). • For a completely adversarial classifier w.r.t. the Bayes optimality criteria, i.e., d t (x) = u(y = -1|x), we have the following result.

Corollary 1. If d t is completely adversarial, then δ t KL attains a maxima of zero at α t = 0.

Proof. See Appendix A.1.3.

2.3. GENERATIVE APPROACH FOR BOOSTING GENERATIVE MODELS:

In the greedy optimization framework of unsupervised boosting, we want to learn an intermediate model distribution at every round that maximizes δ t KL when factored as a product with the BGM distribution in the previous round. In the generative approach, the intermediate model specifies a ratio of densities and maximizes the log-likelihood of data sampled from a reweighted data distribution,

max ht E x∼Dt [log h t ]

where D t ∝ p q t-1 .

(5)

The pseudocode for the corresponding unsupervised boosting algorithm, GenBGM is is given in Algorithm 2. Starting with a uniform distribution over X, GenBGM learns an intermediate model at every round that maximizes the log-likelihood of data sampled from a reweighted data distribution.

2.4. GENERATIVE-DISCRIMINATIVE APPROACH FOR BOOSTING GENERATIVE MODELS:

Intermediate models need not be exclusively generators or discriminators as in Algorithm 1 and Algorithm 2; we can design a boosting algorithm that uses any ensemble of generators and discriminators as intermediate models. 

3. EMPIRICAL EVALUATION:

We evaluated the performance of BGMs as a general-purpose meta-algorithm for generative modeling applications on real and synthetic datasets for three tasks: density estimation, sample generation, and unsupervised feature learning for downstream semi-supervised classification.

3.1. DENSITY ESTIMATION:

A common pitfall with training generative models is model misspecification w.r.t. the true underlying data distribution. To illustrate how BGMs can effectively correct for model misspecification, we consider density estimation on synthetic data. The true data distribution in Figure 1 (a) is a equi-weighted mixture of four Gaussians centered symmetrically around the origin, each having an identity covariance matrix. We only observe 1,000 samples drawn i.i.d. from the data distribution (shown as black dots), and the task is to learn this distribution.

Experimental setup. As a baseline (misspecified) model, we fit a mixture of two Gaussians to the data shown in Figure 1 (b). We evaluate the following ensembling techniques.

1. Bagging. Bagging works just like GenBGM but without any reweighting at every round of ensembling. The intermediate generative models are mixtures of two Gaussians.

2. GenBGM (Algorithm 2). The intermediate models are mixtures of two Gaussians.

3. DiscBGM (Algorithm 1). The binary classifiers used to specify the intermediate models are support vector machines (SVMs) with a radial basis function kernel.

In all ensembles, equal weights are heuristically assigned to every model such that T i=1 α i = 1. For the bagging and GenBGM approaches, ensembling is stopped after T = 3 rounds when the addition of a new model does not result in any significant change in the density estimate. For the DiscBGM approach, ensembling is stopped after T = 15 rounds to prevent overfitting.

Results and discussion. The contour plots for the density estimated by the three approaches are shown in Figure 1 . While the bagging approach is not very effective, GenBGM and DiscBGM steer the initial misspecified distribution towards the true distribution. DiscBGM is more conservative in assigning density mass to outliers and requires more rounds of boosting as opposed to GenBGM.

3.2. SAMPLE GENERATION:

In this task, we generate samples from the learned BGM models and visually inspect their quality. We consider sample generation for the binarized MNIST handwritten digits dataset (LeCun et al., 2010) , which contains 50,000 train, 10,000 validation, and 10,000 test images of dimensions 28×28.

Experimental setup. Boosting is a particularly attractive framework for improving weak learners. For a baseline generative model, we consider the following two latent variable models. Variational Autoencoder (Kingma & Welling, 2014) . VAEs are directed models with continuous latent units where the posterior over the latent units is specified using a neural network. We use the evidence lower bound as a proxy for approximately evaluating the log-likelihood during learning.

Restricted Boltzmann Machine (Smolensky, 1986) . RBMs are undirected 2-layer models with discrete latent units such that the latent and visible layer form a fully-connected bipartite graph. In RBMs, log-likelihood can be tractably computed up to a normalization constant and learning is done using contrastive divergence (Hinton, 2002) .

We compare the baseline models with several BGMs. The starting distribution h 0 for a BGM is specified using a VAE or an RBM with the same model specification and learning procedure as the baseline models. However, when baseline models are used on their own (T = 0, i.e., no boosting) the training is run for 50 epochs. It is reduced to 10 epochs when the base model is used to specify The model architectures, learning procedure, and hyperparameters for the VAE, RBM, and CNN are described in Appendix A.2. The boosting sequences we consider are as follows.

1. T = 0: Baseline VAE, Baseline RBM.

2. T = 1: VAE → VAE, RBM → RBM, VAE → RBM, VAE → CNN, RBM → CNN. 3. T = 2: VAE → CNN → RBM.

The weights, α's, at every round are set to unity for all BGMs, with exceptions made in the cases of RBM → RBM (α 0 = 0.1, α 1 = 0.9) and VAE → RBM (α 0 = 0.3, α 1 = 0.7) where the tuned weights offered significant performance enhancements over the default setting. For the baseline models, we use the respective customary sampling technique, i.e., forward sampling for VAEs and blocked Gibbs sampling for RBMs. Samples from the BGMs are generated by running a Markov chain using the Metropolis-Hastings algorithm with a discrete, uniformly random proposal and the BGM distribution as the stationary distribution for the chain.

Results and discussion. The samples generated by the baseline models and BGM models are shown in Figure 2 . While BGMs significantly improve over baselines models in all cases, evaluating the relative performance of the intermediate models purely based on the samples is hard since these models have different architectures and parameter settings. Figure 3 : Semi-supervised classification using unsupervised feature learning. The boosted generative models are competitive and can also outperform baseline models with an appropriate sequence of intermediate models.

Learning in BGMs is also computationally efficient. We show the wall-clock time taken to learn these models in Table 1 . In many cases, BGMs generate significantly better looking samples with a lower training time compared to the baseline models. A key observation that emerges from the results is that discriminators are more computationally expensive since they require MCMC sampling from the previous generative model distribution as opposed to using intermediate generative models, which only require reweighting of the training set.

3.3. UNSUPERVISED FEATURE LEARNING:

Latent variable models are particularly attractive for unsupervised feature learning since they directly learn hidden representations that model interdependencies between the data variables. In this task, we evaluate the latent representations learned by BGMs for semi-supervised classification on the MNIST dataset consisting of 10 classes.

Setup. We consider the same baselines as before and compare against BGM sequences that have more than a single generator. For the BGM sequences, we concatenate the parameters specifying the posterior over the latent variables (conditioned on the observed variables) in the intermediate models to form a feature representation which we then feed into a transductive-SVM. For the transductive-SVM, we use a publicly available implemention (Joachims, 1999) with a linear kernel and all other parameters set to their default values. Due to computational constraints on the classification procedure, we experiment with subsets of the training dataset and perform semi-supervised classification on a class-balanced sampling of 1,000 and 2,000 training data points varying the number of labelled instances per class from 5 to 80. The procedure is repeated 10 times for statistical significance.

Results and discussion. The classification accuracies are show in Figure 3 . We observe that BGMs closely match, and in some cases outperform the representations learned by baseline models in spite of making fewer passes over the data. A likely explanation of this phenomena is due to the fact that the learning objective for intermediate models is aware of the BGM distribution at the previous round, and hence, is more computationally efficient in modeling specific regions of the underlying distribution that are not covered by the BGM distribution in the previous round.

While it is difficult to make general statements about the intermediate models (which are likely to be dataset specific), a surprising observation is that representations learned by the sequence VAE → CNN → RBM are weaker for classification purposes than the representations learned by a similar VAE → RBM sequence. The likely explanation for this observation is that having a generator later in the sequence offers diminishing advantage from the perspective of feature learning, assuming the previous intermediate models are making progress in modeling the underlying true distribution.

4. DISCUSSION AND RELATED WORK:

In this work, we revisited boosting, a meta-algorithmic framework developed in response to a seminal question posed by Kearns & Valiant (1994) : can a set of weak learners create a strong learner? For the supervised learning problem, boosting has offered interesting theoretical insights into the fundamental limits of learning and led to the development of practical algorithms that work well in practice (Schapire, 1990; Freund et al., 1999; Friedman, 2002; Caruana & Niculescu-Mizil, 2006) .

The algorithmic framework we propose in this work builds on the insights offered by prior work in boosting, yet is significantly different as the motivation is to learn generative models in unsupervised settings. In order to do so, we first defined an appropriate objective function for the generative model. We considered the standard log-loss because of its tight connections with the maximum likelihood principle. In the supervised setting, Lebanon & Lafferty (2002) have shown theoretical results connecting the log-loss for exponential families and the exponential loss minimized by Ad-aBoost (Freund et al., 1999) . Subsequently, we showed how we can greedily optimize a factored generative model as a sequence of intermediate models. Finally, we incorporated the boosting intuition to develop an algorithmic framework where the intermediate models are tightly coupled with the previous models in the sequence and yet can be efficiently learned in practice.

In the context of unsupervised learning, recent advancements in deep generative models have significantly improved our ability to model high-dimensional distributions. For example, highly expressive models such as pixel-RNNs (Oord et al., 2016) and ladder networks (Rasmus et al., 2015) exhibit state-of-the-art performance in generating natural images and semi-supervised learning respectively. The flexibility in choosing intermediate models in BGMs allows for potential integration of these models in our proposed framework.

There has also been a renewed interest in the use of density ratios to distinguish data samples from the model samples. This unsupervised-as-supervised learning approach was first proposed by Friedman et al. (2001) and forms the basis for using binary classifiers for specifying intermediate models in Algorithm 1. The approach has subsequently been applied elsewhere, including parameter estimation in unnormalized models (Gutmann & Hyvärinen, 2010) . Tu (2007)'s approach for generative modeling is closely related to Algorithm 1, but fails to account for imperfections in learning of discriminative models, and the ability to incorporate generative models alongside discriminative models. Hybrid generative-discriminative classifiers have been applied to supervised settings (Truyen et al., 2006; Grabner et al., 2007; Negri et al., 2008; Ratner et al., 2016) .

Recently, the unsupervised-as-supervised learning approach has been successfully applied for sample generation in generative adversarial networks (GAN) (Goodfellow et al., 2014) . GANs consist of a pair of generative-discriminative networks. While the discriminator maximizes the conditional entropy as in Eq. (3), the generator minimizes the same objective. For a parametric class of generative and discriminative networks, the stationary point is a saddle point i.e., a local minima for the generator and a local maxima for the discriminator. Accordingly, the GAN objective is not guaranteed to converge, and stable training of GANs is difficult in practice (Goodfellow, 2014) . Additionally, GANs do not explicitly represent the likelihood of the generative model limiting their applicability, and Parzen window estimates of the model's log-likelihood (Breuleux et al., 2011 ) can be misleading (Theis et al., 2016) .

Borrowing terminology from Mohamed & Lakshminarayanan (2016) , our work fits into the framework of prescribed probabilistic models that provide an explicit characterization of the loglikelihood and can still benefit from the unsupervised-as-supervised learning approach by incorporating intermediate models specified using discriminative approaches.

5. CONCLUSION:

We presented a general-purpose framework for boosting generative models by explicit factorization of the model likelihood as a product of simpler intermediate model distributions. These intermediate model distributions are learned greedily using discriminative or generative approaches, gradually increasing the overall model's capacity. We demonstrated the effectiveness of boosted generative models by designing ensembles of weakly trained variational autoencoders, restricted Boltzmann machines, and convolutional neural networks. Our ensembles improve upon baseline generative models on density estimation, sample generation, and unsupervised feature learning without incurring any significant computational overhead.

As a future work, we would like to apply our framework to more sophisticated models on complex datasets such as natural images. The optimal weighting for intermediate models also remains an open question to explore in future work. Finally, in the proposed framework, an intermediate model specified using a discriminator requires MCMC sampling from the BGM distribution at the previous round. This can be expensive, and future work could explore the design of more efficient strategies.

A.2 MODEL ARCHITECTURES AND PARAMETER SETTINGS:

The baseline VAE model consists of a deterministic hidden layer with 500 units between the visible layer and stochastic hidden layer with 50 latent variables. The inference network specifying the posterior also contains a single deterministic layer with 500 units. The prior over the latent variables is standard Gaussian, the hidden layer activations are tanh, and learning is done using Adam (Kingma & Ba, 2015) with a learning rate of 10 -3 and mini-batches of size 100.

The baseline RBM model consists of 250 hidden units trained for 50 epochs using Stochastic Gradient Descent with a learning rate of 5 × 10 -2 , mini-batches of size 100, and 15 steps of contrastive divergence.

The CNN contains two convolutional layers and a single full connected layer with 1024 units. Convolution layers have kernel size 5 × 5, and 32 and 64 output channels, respectively. We apply ReLUs and 2 × 2 max pooling after each convolution. The net is randomly initialized prior to training, and learning is done for 2 epochs using Adam (Kingma & Ba, 2015) with a learning rate of 10 -3 and mini-batches of size 100.

:

For many applications of generative models such as sample generation and feature learning, we can sidestep computing the partition function.

Figures and Tables:

Figure 1: Densities estimated using GenBGM (f-h) and DiscBGM (i-k) can correct for model misspecification (b) w.r.t. the true distribution (a) unlike densities estimated using other bagging-style ensembling methods (d-f).

Figure 2: Samples generated from the boosted generative models (c-h) demonstrate how boosting may be used to ensemble weak learners (a, b) into stronger models with samples that are much more visually reflective of the true data distribution.

If an intermediate distribution is required to be a generator, we train a generative model by appropriately reweighting our training set. If a discriminator odds ratio is used to specify an intermediate distribution, we set up the corresponding binary classification problem.In practice, we want BGMs to generalize to data points outside the training set X. Regularization in BGMs is imposed primarily in two ways. First, every intermediate model can be independently regularized by incorporating explicit terms in the learning objective, early stopping of training based on validation error, specialized techniques such as dropout, etc. Moreover, regularization in BGMs is also imposed by restricting the number of rounds of boosting. If the intermediate models are exseveral applications of pressive enough, then very few rounds of boosting are required. We now do an empirical study of BGMs for several applications of generative modeling.

Table content: <table/>

Training time (in seconds) of the baseline models and boosted generative models.

Table content: <table><tr><td/><td/><td/><td colspan="2">Model</td><td/><td/><td colspan="4">Train time Model</td><td>Train time</td></tr><tr><td/><td/><td/><td colspan="4">Baseline VAE</td><td/><td>267</td><td/><td>Baseline RBM</td><td>472</td></tr><tr><td/><td/><td/><td colspan="4">VAE → VAE VAE → CNN VAE → RBM</td><td/><td>140 285 213</td><td/><td>RBM → RBM RBM → CNN VAE → RBM → CNN</td><td>266 554 625</td></tr><tr><td>0</td><td>10</td><td>20</td><td>30</td><td>40</td><td>50</td><td>60</td><td>70</td><td>80</td><td>90</td><td>100</td></tr><tr><td/><td/><td/><td colspan="5">Number of labels per class</td><td/><td/><td/></tr></table>

Peer review (iclr2017):
The authors propose two approaches to combine multiple weak generative models into a stronger one using principles from boosting. The approach is simple and elegant and basically creates an unnormalized product of experts model, where the individual experts are trained greedily to optimize the overall joint model. Unfortunately, this approach results in a joint model that has some undesirable properties: a unknown normalisation constant for the joint model and therefore an intractable log-likelihood on the test set; and it makes drawing exact samples from the joint model intractable. These problems can unfortunately not be fixed by using different base learners, but are a direct result of the product of experts formulation of boosting. The experiments on 2 dimensional toy data illustrate that the proposed procedure works in principle and that the boosting formulation produces better results than individual weak learners and better results than e.g. bagging. But the experiments on MNIST are less convincing: Without an undisputable measure like e.g. log-likelihood it is hard to draw conclusions from the samples in Figure 2; and visually they look weak compared to even simple models like e.g. NADE. I think the paper could be improved significantly by adding a quantitative analysis: investigating the effect of combining undirected (e.g. RBM), undirected (e.g. VAE) and autoregressive (e.g. NADE) models and by measuring the improvement over the number of base learners. But this would require a method to estimate the partition function Z or estimating some proxy.