Title: LEARNING DEEP GENERATIVE MODELS WITH DISCRETE LATENT VARIABLES Anonymous

Abstract: There have been numerous recent advancements on learning deep generative models with latent variables thanks to the reparameterization trick that allows to train deep directed models effectively. However, since reparameterization trick only works on continuous variables, deep generative models with discrete latent variables still remain hard to train and perform considerably worse than their continuous counterparts. In this paper, we attempt to shrink this gap by introducing a new architecture and its learning procedure. We develop a hybrid generative model with binary latent variables that consists of an undirected graphical model and a deep neural network. We propose an efficient two-stage pretraining and training procedure that is crucial for learning these models. Experiments on binarized digits and images of natural scenes demonstrate that our model achieves close to the state-of-the-art performance in terms of density estimation and is capable of generating coherent images of natural scenes.

1. INTRODUCTION:

Building generative models that are capable of learning flexible distributions over high-dimensional sensory input, such as images of natural scenes, is one of the fundamental problems in unsupervised learning. Historically, many multi-layer generative models, including sigmoid belief networks (SBNs) (Neal, 1992) , deep belief networks (DBNs) (Hinton et al., 2006) , and deep Boltzmann machines (DBMs) (Salakhutdinov & Hinton, 2009) , contain multiple layers of binary stochastic variables. However, since the debut of variational autoencoder (VAE) (Kingma & Welling, 2013) and reparameterization trick, models with continuous variables have largely replaced previous discrete versions. Many improvements (Burda et al., 2015; Kingma et al., 2016; Chen et al., 2016; Gulrajani et al., 2016) along this direction have been pushing forward the state-of-the-art for years.

Comparing with continuous models, existing discrete models have two major disadvantages. First, models with continuous latent variables are easier to optimize due to the reparameterization trick. Second, every layer in models, including SBNs and DBNs, is stochastic. Such design pattern restricts the depth of these models because adding one layer can only provide small additional representation power while the extra stochasticity increases the optimization difficulty and thus out-weights the benefit.

In this paper we explore learning discrete latent variable models that perform equally well with its continuous counterparts. Specifically, we propose an architecture that resembles the DBN but uses deep deterministic neural networks for inference and generative networks. From the VAE perspective, this can also be seen as deep VAE with one set of binary latent variables and learnable restricted Boltzmann machine (RBM) prior (Hinton, 2002) . We develop a two-stage pretraining, training procedure for learning such models and show that they are necessary and effective. Finally, we demonstrate that our models can closely match the state-of-the-art continuous models on MNIST in terms of log-likelihood and are capable of generating coherent images of natural scenes.

2. BACKGROUND:

Although discrete models are largely replaced by continuous models in practice, there has been a surge in the interest of the learning algorithms that accommodate discrete latent variable models, such as sigmoid belief networks (SBNs) (Mnih & Gregor, 2014; Bornschein & Bengio, 2014; Mnih & Rezende, 2016) . In this section, we briefly review those methods that lay the foundation of our learning procedure.

To learn a generative model p(x) on a given dataset, we introduce latent variable z and decompose the objective as log p(x) = log z p(x, z). Posteriors samples from p(z|x) are required to efficiently estimate the exponential sum z p(x, z). However, when p(x, z) is parameterized by a deep neural network, exact posterior sampling is no longer possible. One way to overcome it is to simultaneously train an inference network q(z|x) that approximates the true posterior p(z|x). With samples from q distribution, we can train p by optimizing the variational lower bound:

EQUATION

Meanwhile, q(z|x) has to be optimized towards p(z|x) in order to keep the variational bound tight.

In the Wake-Sleep algorithm (Hinton et al., 1995; 2006) , the wake phase corresponds to maximizing the objective in Eq. 1 with respect to the parameter of p(x, z) using samples from q(z|x) given a datapoint x. In the sleep phase, a pair of samples z, x is drawn from the generative distribution p(x, z) and then q is trained to minimize the KL divergence KL(p(z|x) q(z|x)). This objective, however, is not theoretically sound as we should instead be minimizing its reverse: KL(q(z|x) p(z|x)).

Reweighted Wake-Sleep (RWS) (Bornschein & Bengio, 2014) brings two major improvements to the original Wake-Sleep algorithm. The first one is to reformulate the log-likelihood objective as an importance-weighted average and derive a tighter lower bound:

log p(x) = log E zi∼q(z|x) 1 K K i=1 p(x, z i ) q(z i |x) ≥ E zi∼q(z|x) log 1 K K i=1 p(x, z i ) q(z i |x) = L K . (2)

In the wake phase of RWS, parameters in p are learned by optimizing the new lower bound defined in Eq. 2 (Burda et al., 2015) . The second improvement is to add a wake phase for learning q. The wake phase for q can be viewed as minimizing the KL divergence KL(p(z|x) q(z|x)) for a given datapoint x data instead of x sample as in the sleep phase. The authors empirically show that the new wake phase works better than the sleep phase in the original Wake-Sleep and works even better when combined with the sleep phase.

Although RWS tightens the bound and works well in practice, it still does not optimize a well-defined objective for inference network q. Mnih & Rezende (2016) propose a new method named VIMCO, which solves this problem. In VIMCO, both p and q are optimized jointly against the lower bound in Eq. 2. However, the gradient w.r.t parameters in q will have high variance if we compute them naively. VIMCO algorithm utilizes the multiple samples to compose a baseline for each sample using the rest of samples (we refer readers to the original paper for more technical details). The author shows that VIMCO performs equally well as RWS when training SBNs on MNIST.

3. MODEL:

Let us consider a latent variable model p(x) = z p(x|z)p(z) with the distribution p(z) defined over the latent space. In addition, an inference network q(z|x) is used to approximate the intractable posterior distribution p(z|x). This fundamental formulation is shared by many deep generative models with latent variables, including deep belief networks (DBNs), and variational autoencoders (VAEs). Different realizations result in different architectures and corresponding learning algorithms.

In our model, the prior distribution p ϕ (z) is multivariate Bernoulli modeled by a restricted Boltzmann machine (RBM) with a parameter vector ϕ. The approximate posterior distribution q φ (z|x) is multivariate Bernoulli with its mean modeled by a deep neural network φ. The generative distribution p θ (x|z) is modeled by a deep neural network θ as well. Note that both networks are deterministic.

This model has several advantages. First, compared with VAEs, RBMs can handle both discrete and continuous latent variables. It also allows for a much richer family of latent distributions compared to simple factorized distributions as in vanilla VAEs. Although for VAEs, the posterior distribution is regularized towards a factorized Gaussian prior by minimizing KL divergence, in practice the KL divergence is never zero, especially when modeling complex datasets. Such discrepancy between the prior and learned posterior can often damage the generative quality. The RBM approach, however, instead of pulling the posterior to some pre-defined prior, learns to mimic the posterior. During generation process, prior samples drawn by running the Markov chain defined by the RBM can often lead to images with higher visual quality than those drawn from vanilla VAEs.

Second, compared with SBNs and DBNs, only communication between inference and generative networks uses stochastic binary states. In this case, the inference and generative networks become fully differentiable so that multiple layers can be jointly optimized by back-propagation. This is radically different from SBNs and DBNs where each inference layer is trained to approximate the posterior distribution of a specific generative layer. Our framework can greatly increase the model capacity by allowing more complicated transformation between high dimensional input space and latent space. In addition, networks can exploit modern network design techniques, including convolution, pooling, dropout (Srivastava et al., 2014) , or even ResNet (He et al., 2015) and DenseNet (Huang et al., 2016) , in a very easy and straightforward way. Therefore, similar to VAEs, models under this framework can be scaled to handle more complex datasets compared to traditional SBNs and DBNs.

3.1. PRETRAINING WITH AUTOENCODER:

Training a hybrid model is never a trivial task. Notably in our case, the encoder and decoder networks can be very deep and gradient cannot be propagated through stochastic states. In addition, RBMs are often more sensitive to training compared to feed-forward neural networks. Therefore, as in DBNs and DBMs, a clever pretraining algorithm that can help find a good weight initialization can be very beneficial.

To learn a good image prior, we jointly train parameters of the inference network φ and generative network θ as an autoencoder. To obtain a binary latent space, we use additive i.i.d uniform noise (Ballé et al., 2016) together with a modified hardtanh function to realize "soft-binarization".1 This method can be described as the following function:

EQUATION

and z = E(x) is the output of the encoder. This soft-binarization function will encourage the encoder to encode x into ([-∞, -1] ∪ [1, +∞]) |z| to maximize the information that follows through this bottleneck while allowing gradient descent methods to find such solution efficiently. To avoid overfitting, dropout can be applied after B function. The adoption of dropout in z space can also prevent the co-adaptation between latent codes, which makes it easier for RBMs to model.

In practice, we find that this pretraining procedure produces well binarized latent space on which RBMs can be successfully trained. Therefore, we can then pretrain the RBMs on z using contrastive divergence (Hinton, 2002) or persistent contrastive divergence (Tieleman, 2008) . After pretraining, we remove B and append a sigmoid function σ to the end of the encoder to convert it to the inference network, i.e.φ = σ • E. The decoder is then used to initialize the generator θ.

3.2. TRAINING:

Since our model shares the fundamental formulation with many of the existing variational based models, we can modify the state-of-the-art learning algorithms to train it. The specific algorithms we are interested in are the reweighted wake-sleep (RWS) (Bornschein & Bengio, 2014) and VIMCO (Mnih & Rezende, 2016) which give the best results on SBN models and can handle discrete latent variables.

As reviewed in the background section, both RWS and VIMCO are importance sampling based methods, that need to compute weights w i = p(x, z i )/q(x|z i ) for multiple samples z i given input x.

These weights are then normalized as wi = w i /( j w j ) to decide the contribution of each sample to the gradient estimator. In our model, the joint probability p(x, z) is intractable due to the partition Z function introduced by RBM. However, it can be substituted by its unnormalized counterpart:

EQUATION

as the coefficient Z will be canceled during the weight normalization step. The F(z) is the free energy assigned to z by RBM, which can be computed analytically.

The RBM is also trained using multiple samples as part of the generative module. In both RWS and VIMCO, the gradient for RBM with parameter ϕ is:

EQUATION

The second term in Eq. 5 is the intractable model dependent term which needs to be estimated using samples from the RBM. The samples are obtained by running a persistent Markov chain as in persistent contrastive divergence (Tieleman, 2008) .

There is one more modification for the sleep phase in RWS. The generative process now starts from a Markov chain defined by RBM instead of a direct draw from unconditional Bernoulli prior. This can be seen as the contrastive version of RWS. For completeness, we put the detail of Contrastive RWS and VIMCO in Appendix A.

3.3. EVALUATION:

Quantitative evaluation of deep generative models is very crucial to measure and compare different probabilistic models. Fortunately, we can refer to a rich set of existing techniques for quantitative evaluations. One way to evaluate our model is to decompose the lower bound L K in Eq. 1 as follows:

EQUATION

and estimate partition function Z with Annealed Importance Sampling (AIS) (Salakhutdinov & Murray, 2008) . This method is very efficient since we only need to estimate Z once no matter how large the K is. However, since AIS gives an unbiased estimate of Z, it on average tends to underestimate log Z since log Z = log E( Ẑ) ≥ E(log Ẑ) (Burda et al., 2014) .

Another method that yields conservative estimates is Reverse AIS Estimator (RAISE) (Burda et al., 2014) , which returns an unbiased estimate of p(z). However, RAISE can be quite time consuming since it needs to run an independent chain for every z. Therefore, we suggest to use AIS as main tool for evaluation during training and model comparison, since empirically AIS provides fairly accurate estimates, but also run RAISE as a safety check before reporting final results to avoid unrealistically high estimates of L K .

4. RELATED WORK:

In Figure 1 , we show the visualization of our model together with three closely related existing latent variable models, DBNs (Hinton et al., 2006) , DEMs (Ngiam et al., 2011) and VAEs (Kingma & Welling, 2013) .

The major difference between DBNs and our models is that every layer in the inference and generative networks in DBNs is stochastic. Such design drastically increases the difficulty in training and restrict the model from using modern deep convolutional architectures. Although convolution, combined with a sophisticated probabilistic pooling technique, has been applied to DBNs (Lee et al., 2009) , the resulted convolutional DBNs is still difficult to train. It is also unclear how more recent techniques like residual connections (He et al., 2015) can be adapted for them. Our models, on the other hand, can integrate these techniques easily and learn deeper networks effectively.

The deep energy models (DEMs) by Ngiam et al. (2011) are previous attempts to use multiple deterministic layers to build deep generative models. In their setting, only the top-most layer, which resembles the hidden layer in an RBM, is stochastic. There is no explicit generator in the model and sampling is carried out through Hamiltonian Monte Carlo (HMC). In practice, we find that HMC samplers are too sensitive to hyper-parameters, making them extremely hard to use for sampling from deep convolutional networks. The generator solution in our model is simpler, more robust, and more scalable.

VAEs (Kingma & Welling, 2013) are modern deep generative models that have shown impressive success in a wide variety of applications (Yang et al., 2017; Pu et al., 2016) . VAEs are directed graphical models that also consist of stochastic latent layers and deterministic deep neural networks. However, VAEs use factorized prior distributions, which can potentially limit the networks' modeling capacity by placing a strong restriction on the approximate posterior (Burda et al., 2015) . There have been several works trying to resolve this issue by deriving more flexible posteriors (Jimenez Rezende & Mohamed, 2015; Kingma et al., 2016) . The RBM in our model can represent more complex prior distributions by design, which can possibly lead to more powerful models.

PixelRNNs (van den Oord et al., 2016a) and GANs (Goodfellow et al., 2014) are two other popular generative models. PixelRNNs are fully observable models that use multiple layer of LSTMs to model images as a sequence of pixels. PixelRNN and its various variant PixelCNN (van den Oord et al., 2016b; Salimans et al., 2017; Gulrajani et al., 2016) exhibit excellent capacity on modeling local detail on images and are the state-of-the-art models in terms of density estimation. GANs simultaneously train a discriminator and a generator. The discriminator is trained to distinguish generated samples from real data while generator is trained to fool discriminator by generating realistic samples. GANs can generate visually appealing images but they are hard to evaluate quantitatively. Although several methods have been discussed recently (Theis et al., 2015; Wu et al., 2016) , quantitative evaluation of GANs still remains a challenging problem.

5. EXPERIMENTS:

We now describe our experimental results. Through a series of experiments we 1) quantitatively evaluate the importance of the pretraining step and compare the performance of our model trained with Contrastive RWS and VIMCO algorithms, 2) scale our model with ResNet (He et al., 2015) to approach the state-of-the-art result on MNIST, 3) scale our model to modeling images of natural scenes, and show that it performs comparable with its continuous counterparts in terms of density estimation, while being able to generate coherent samples. Please refer to Appendix C for details on the hyper-parameters and network architectures.

5.1. MNIST:

We run our first set of experiments on the statically binarized MNIST dataset (Salakhutdinov & Murray, 2008; Larochelle & Murray, 2011) . To model binary output, the generator θ computes the mean of the Bernoulli distribution p θ (x|z). We first train a simple model where both inference and generative networks are multi-layer perceptrons. The inference network contains five layers (784-200-200-100-100-200) and the generator contains the same layers in reverse order. We use ELU (Clevert et al., 2015) is normally larger since it is supposed to transmit only binary information. The RBM has 200 visible units z and 400 hidden units h. The model is first pretrained and then trained with Contrastive RWS or VIMCO using 50 samples per data point. The learning curves of the first 200 epochs for models trained with both methods are shown in Figure 2a .

To evaluate our model, we use the AIS method following Eq.6 in Section 3.3 with K = 5e4. Z is estimated by running 5000 AIS chains with 1e5 intermediate distributions. Table 1 shows performance of our fully connected model together with several previous works that use a similar network size.

From the table and the learning curves in Figure 2a , we can see that our model trained with VIMCO objective performs better compared to training with Contrastive RWS. The superiority of VIMCO over Contrastive RWS is consistent during our experiments with various network configurations. In addition, we need to carefully tune the learning rate for inference and generative network separately to make Contrastive RWS work well on our model, which may be caused by the fact that wake-sleep algorithms are not optimizing a well defined objective.

To emphasize the importance of the pretraining algorithm, we also train the model with VIMCO directly starting from random initialization and it performs significantly worse. This result shows that the pretraining stage is very effective and is also necessary to make our model work. We also evaluate the best model with RAISE method to make sure that the result is not over-optimistic. For RAISE, we use fewer samples (K = 5e3) due to its high computation cost. The RAISE result is shown in the parenthesis in Table 1 . The two estimators agree closely with each other, indicating that the results are accurate. Finally, we show samples from our model trained with VIMCO in Figure 2b . deep networks can indeed result in better performance in terms of density estimation. Notably, our model also outperforms IWAE (Burda et al., 2015) , which in principle can be seen as the continuous counterpart of our model without an RBM prior. To fully test the capacity of our framework, we train a deep convolutional model with ResNet (He et al., 2015) blocks. The result is shown in Table 2 . Our model surpasses the previous best models that use purely VAEs (Gregor et al., 2015; Kingma et al., 2016) and is only slightly behind the state-of-the-art models that use PixelRNN (van den Oord et al., 2016a) or VAEs combined with PixelCNNs (Chen et al., 2016; Gulrajani et al., 2016) . In principle, PixelCNN can also be integrated into our framework as decoder, but we will leave this for future work.

Comparing with other methods in:



5.2. CIFAR10:

CIFAR10 has been a challenging benchmark for generative modeling. To model real value pixel data, we set the generative distribution p θ (x|z) to be discretized logistic mixture following Salimans et al. (2017) . In the pretraining stage, the objective is to minimize the negative log-likelihood. The marginal distribution of the encoded z space and the reconstruction of test images are shown in Figure 5 in Appendix B. We note that the pretrained autoencoder preserves enough information while converting high dimensional real value data to binary. This transformation makes it possible apply simple models like RBM to challenging tasks such as modeling CIFAR10 images. For the network architecture, we use ResNets (He et al., 2015) for inference and generative networks. The latent space has 1024 dimensions and is modeled by RBM with 2048 hidden units. Similar to what we have discovered during the MNIST experiments, we find that VIMCO is more effective and robust to hyperparameters than Contrastive RWS. Therefore, the model is trained using VIMCO with 10 samples per data point.

For quantitative and qualitative comparisons under controlled variates, we train an IWAE (Burda et al., 2015) with roughly the same networks and the same amount of posterior samples per data point.

The quantitative results are shown in Table 3 and samples from both models are shown in Figure 3a and Figure 4a . Here, our model performs slightly worse than IWAE in terms of density estimation, but the samples from our model have much higher visual quality. Note that results from both models are far behind those from state-of-the-art models (Salimans et al., 2017) . To achieve significantly better results for VAE family models on CIFAR10, we often need to use more complicated networks with multiple sets of latent variables (Kingma et al., 2016) or use autoregressive decoders for output distribution (Chen et al., 2016) or both (Gulrajani et al., 2016) . In this work, however, we keep our models simple to focus on the learning procedure.

To facilitate visual comparison, we also reproduce samples from a popular GAN model (Arjovsky et al., 2017) in Figure 4b . Samples from our model look natural, coherent but blurry, while samples from WGAN look clear, detailed but distorted. We admit that with many advanced techniques (Salimans et al., 2016; Arora et al., 2017; Dai et al., 2017) , GANs still produce the highest quality images. However, our model has the advantage that it can be properly evaluated as a density model. Additionally, the flexibility of our framework could also accommodate potential future improvements.

5.3. IMAGETNET64:

We next use the 64 × 64 ImageNet (van den Oord et al., 2016a) to test the scalability of our model. Figure 3b , shows samples generated by our model. Although samples are far from being realistic 

6. CONCLUSION:

In this paper we presented a novel framework for constructing deep generative models with RBM priors and develop efficient learning algorithms to train such models. Our models can generate appealing images of natural scenes, even in the large-scale setting, and, more importantly, can be evaluated quantitatively. There are also several interesting directions for further extensions. For example, more expressive priors, such as those based on deep Boltzmann machines (Salakhutdinov & Hinton, 2009) , can be used in place of RBMs, while autoregressive (Gregor et al., 2013) 6: Normalize the weights wi = w i i w i 7: Sample {z -} from M CD/PCD chains and pass through generator θ to obtain {x -} 8: Wake update for generative network θ with gradient: K i=1 wi∇θ log p θ (x, zi) 9: Wake and sleep updates for inference network φ with gradient:

K i=1 wi∇φ log q φ (z (k) |x) + 1 M M j=1 ∇ φ log q φ (z - j |x - j )

10: Update RBM ϕ with gradient:

- K i=1 wi∇ϕFϕ(zi) + 1 M M j=1 ∇ϕFϕ(z - j )

A.2 VIMCO Algorithm 2 VIMCO for a single observation 1: Sample x from training distribution. 2: for i = 1 to K do 3: Sample zi from q φ (z|x) 4: end for 5: Compute unnormalized weights wi =

p * θ (x,z i ) q φ (z i |x) 6: Compute multi-sample variational bound: LK = log 1 K K i=1 wi 7: for i = 1 to K do 8:

Compute geometric mean of the rest of samples: w-i = j =i wj 1 K-1

9::

Compute the baseline learning signal: L-i = log 1 K w-i + j =i wj 10: end for 11: Normalize the weights wi = w i i w i 12: Sample {z -} from M CD/PCD chains and pass through generator θ to obtain {x -} 13: Update generative network θ with gradient: K i=1 wi∇θ log p θ (x, zi) 14: Update inference network φ with gradient: These are expected value of the output distribution without further sampling.

C EXPERIMENTAL SETUP:

In this section, we describe the training details and network configurations for experiments in Section 5. Code will be released as well.

The general training procedure is as follows. We first pretrain the inference and generative networks as autoencoder by maximizing log-likelihood on training data. Then we pretrain RBM with contrastive divergence starting from 1 step (CD1) and gradually increase to 25 steps (CD25). This training method has been previously used to produce the best RBM on MNIST dataset (Salakhutdinov & Murray, 2008) . We additionally train the RBM using persistent contrastive divergence with 25 steps (PCD25) or more. Finally, we train all three components jointly with Contrastive RWS or VIMCO. In Contrastive RWS and VIMCO, samples from RBM are drawn from a persistent chain. We use SGD with learning rate decay for learning RBMs and Adam or Adamax (Kingma & Ba, 2014) elsewhere.

We experiment with three activation functions ReLU, LeakyReLU and ELU (Clevert et al., 2015) , and find out that ELU performs slightly better. Inspired by (Kingma et al., 2016) , we use weight normalization (Salimans & Kingma, 2016) in deep ResNet models as we find that it works better than batch normalization for our model as well.

In MNIST experiments, the shallow fully connected model uses an inference network with five layers (784-200-200-100-100-200 ) and a generative network with the same layers in reversed order. The RBM has 200 visible units z and 400 hidden units h. For the deep ResNet model, the inference network uses three basic pre-activation (He et al., 2016) residual blocks with 25, 50, 50 feature maps. Each block uses kernel size 3 and is repeated twice with stride 2 and 1 respectively. After residual blocks, there is a fully connected layer with 200 neurons. The RBM has 200 visible units and 400 hidden units. The generative network uses the same blocks but with stride one. We upsample the feature map with nearest neighbour interpolation by a factor of 2 before feeding it into each block and shortcut to avoid checkerboard artifact (Odena et al., 2016) .

In CIFAR10 and ImageNet64 experiments, the output distribution p θ is a discretized mixture of 10 logistic distributions (Salimans et al., 2017) . The network for CIFAR10 uses 4 residual blocks with 64, 128, 192, 256 feature maps. Each block is repeated twice as in MNIST. There is no fully connected layer in this model and final feature map (256 × 2 × 2) is flattened to a 1024 dimensional vector. The RBM has 1024 visible units and 2048 hidden units. The network for ImageNet64 uses 5 residual blocks with 64, 128, 128, 256, 256 feature maps. Each block uses stride 2 and is only repeated once. The RBM is the same as the one in CIFAR10.

:

Although real-valued data can be modeled by a Gaussian-Bernoulli RBM, it is often much harder to train and not the focus of this paper.

Figures and Tables:

Figure 1: Comparison between (a) deep belief networks, (b) deep energy models, (c) variational autoencoders and (d) our models. Dashed boxes denote stochastic layers and solid boxes denote deterministic layers. Bidirectional arrows denote undireicted connections. For simplicity, recognition networks in VAE and our model are represented by a single upward arrow.

Figure 2: Left: Negative log-likelihood on MNIST test set during training with Contrastive RWS and VIMCO.Right: Samples generated by running Gibbs sampling in RBM for 1000 steps and passing generated z through generator θ, no further sampling in pixel space.

Figure 3: Samples generated by our model trained on CIFAR10 (left) and ImageNet64 (right).

Figure 5: Left: Marginal distribution of z in the encoded CIFAR10. Right: Reconstruction of test images.

Average test negative log-probabilities on MNIST.Burda et al. (2015). Numbers of our model are computed with the AIS method in Section 3.3 while number in parenthesis is computed with the RAISE method.

Table content: <table><tr><td>Model</td><td>NLL Test</td><td>Model</td><td>NLL Test</td></tr><tr><td>DBN [1]</td><td>84.55</td><td>DRAW [1]</td><td>80.97</td></tr><tr><td>AR-SBN/SBN (RWS) [2]</td><td>84.18</td><td>IAF VAE [2]</td><td>79.88</td></tr><tr><td>IWAE [3]</td><td>85.32</td><td>PixelRNN [3]</td><td>79.20</td></tr><tr><td colspan="2">Our Model (VIMCO, no pretrain) 121.65 Our Model (Contrastive RWS) 84.33</td><td colspan="2">VLAE [4] Gated PixelVAE [5] 78.96 79.03</td></tr><tr><td>Our Model (VIMCO)</td><td>83.69 (83.77)</td><td>Our ResNet Model</td><td>79.58 (79.64)</td></tr></table>

Average test negative log-

Table content: <table/>

Table 1, our model clearly outperforms previous models that use multiple stochastic layers with or without RBM. The improvement indicates that using continuous Average test negative log-probabilities on CIFAR10 in bits/dim. [1] Kingma et al. (2016), [2] Chen et al. (2016), [3]Salimans et al. (2017). Numbers of our model are computed with the AIS method.

Table content: <table><tr><td>Model</td><td colspan="2">NLL Train NLL Test</td></tr><tr><td>ResNet VAE with IAF [1]</td><td/><td>3.11</td></tr><tr><td>DenseNet VLAE [2]</td><td/><td>2.95</td></tr><tr><td>PixelCNN++ [3]</td><td/><td>2.92</td></tr><tr><td>IWAE</td><td>4.45</td><td>4.54</td></tr><tr><td>Our Model</td><td>4.73</td><td>4.84</td></tr></table>

or recurrent networks (van den Oord et al., 2016a) can be used for inference and generative networks.

Table content: <table><tr><td colspan="3">A DETAILS ON CONTRASTIVE REWEIGHTED WAKE-SLEEP AND VIMCO</td></tr><tr><td colspan="3">A.1 CONTRASTIVE REWEIGHTED WAKE-SLEEP</td></tr><tr><td colspan="3">Algorithm 1 Contrastive Reweighted Wake-Sleep for a single observation</td></tr><tr><td colspan="2">1: Sample x from training distribution.</td></tr><tr><td colspan="2">2: for i = 1 to K do</td></tr><tr><td>3:</td><td>Sample zi from q φ (z|x).</td></tr><tr><td colspan="2">4: end for 5: Compute unnormalized weights wi =</td><td>p  *  θ (x,z i ) q φ (z i |x)</td></tr></table>

Peer review (iclr2018):
 Summary of the paper: The paper proposes to augment a variational auto encoder (VAE) with an binary restricted Boltzmann machine (RBM) in the role of the prior of the generative model. To yield a good initialisation of the parameters of the RBM and the inference network a special pertaining procedure is introduced. The model produces competitive Likelihood results on MNIST and was further tested on CIFAR 10. Clarity and quality: 1. From the description of the pertaining procedure and the appendix B I got the impression that the inference network maps into [0,1] and not into {0,1}. Does it mean, you are not really considering binary latent variables (making the RBM model the values in [0,1] by its probability p(z|h))? 2. on page 2: RWS...."derive a tighter lower bound": Where does the "tighter" refer to? 3. "multivariate Bernoulli modeled by an RBM": Note, while in a multivariate Bernoulli the binary variables would be independent from each others, this is usually not the case for the visible variables of RBMs (only in the conditional distribution given the state of the hidden variables). 4. The notation could be improved, e.g.: -x_data and x_sample are not explained - M is not defined in equation 5. 5. "this training method has been previously used to produce the best results on MNIST" Note, that parallel tempering often leads to better results when training RBMs (see http://proceedings.mlr.press/v9/desjardins10a/desjardins10a.pdf) . Furthermore, centred RBMs are also get better results than vanilla RBMs (see: http://jmlr.org/papers/v17/14-237.html). Originality and significance: As already mentioned in a comment on open-review the current version of the paper misses to mention one very related work: "discrete variational auto encoders". Also "bidirectional Helmholtz machines" could be mentioned as generative model with discrete latent variables. The results for both should also be reported in Table 1 (discrete VAEs: 81,01, BiHMs: 84,3). From the motivation the advantages of the model did not become very clear to me. Main advantage seems to be the good likelihood result on MNIST (but likelihood does not improve compared to IWAE on CIFAR 10 for example). However, using an RBM as prior has the disadvantage that sampling from the generative model requires running a Markov chain now while having a solely directed generative model allows for fast sampling. Experiments show good likelihood results on MNIST. Best results are obtained when using a ResNet decoder. I wondered how much a standard VAE is improved by using such a powerful decoder. Reporting this, would allow to understand, how much is gained from using a RBM for learning the prior. Minor comments: page 1: "debut of variational auto encoder (VAE) and reparametrization trick" -> debut of variational auto encoders (VAE) and the reparametrization trick", page 2: "with respect to the parameter of p(x,z)" -> "with respect to the parameters of p(x,z)" "parameters in p" -> "parameters of p" "is multivariate Bernoulli" -> "is a multivariate Bernoulli" "we compute them" -> "we compute it" page 3: "help find a good" -> "help to find a good" page 7: "possible apply" -> "possible to apply"