Venue: ICLR 2018 conference

Reviews:

final_score: rating: 4: Ok but not good enough - rejection
review_content: Summary of the paper:
The paper proposes to augment a variational auto encoder (VAE) with an binary restricted Boltzmann machine (RBM) in the role of the prior of the generative model. To yield a good initialisation of the parameters of the RBM and the inference network a special pertaining procedure is introduced. The model produces competitive Likelihood results on MNIST and was further tested on CIFAR 10. 

Clarity and quality: 

1. From the description of the pertaining procedure and the appendix B I got the impression that the inference network maps into [0,1] and not into {0,1}.  Does it mean, you are not really considering binary latent variables (making the RBM model the values in [0,1] by its probability p(z|h))? 

2. on page 2:
RWS...."derive a tighter lower bound": Where does the "tighter" refer to? 

3. "multivariate Bernoulli modeled by an RBM": Note,  while in a multivariate Bernoulli the binary variables would be independent from each others, this is usually not the case for the visible variables of RBMs (only in the conditional distribution given the state of the hidden variables).

4. The notation could be improved, e.g.:
-x_data and x_sample are not explained
- M is not defined in equation 5. 

5. "this training method has been previously used to produce the best results on MNIST" Note, that parallel tempering often leads to better results when training RBMs (see http://proceedings.mlr.press/v9/desjardins10a/desjardins10a.pdf) . Furthermore, centred RBMs are also get better results than vanilla RBMs (see: http://jmlr.org/papers/v17/14-237.html).

Originality and significance:
As already mentioned in a comment on open-review the current version of the paper misses to mention one very related work: "discrete variational auto encoders". Also "bidirectional Helmholtz machines" could be mentioned as generative model with discrete latent variables.  The results for both should also be reported in Table 1 (discrete VAEs: 81,01, BiHMs: 84,3). 

From the motivation the advantages of the model did not become very clear to me. Main advantage seems to be the good likelihood result on MNIST (but likelihood does not improve compared to IWAE on CIFAR 10 for example). However, using an RBM as prior has the disadvantage that sampling from the generative model requires running a Markov chain now while having a solely directed generative model allows for fast sampling. 

Experiments show good likelihood results on MNIST. Best results are obtained when using a ResNet decoder. I wondered how much a standard VAE is improved by using such a powerful decoder. Reporting this, would allow to understand, how much is gained from using a RBM for learning the prior. 

Minor comments:
page 1:
"debut of variational auto encoder (VAE) and reparametrization trick" -> debut of variational auto encoders (VAE) and the reparametrization trick",
page 2:
"with respect to the parameter of  p(x,z)" -> "with respect to the parameters of  p(x,z)"
"parameters in p" -> "parameters of p" 
"is multivariate Bernoulli" ->  "is a multivariate Bernoulli"
"we compute them" -> "we compute it" 
page 3:
"help find a good" ->  "help to find a good"
page 7:
"possible apply" -> "possible to apply"
review_title: The paper proposes to augment a variational auto encoder (VAE) with an binary restricted Boltzmann machine (RBM) in the role of the prior of the generative model. Clarity could be improved in some aspects and the advantages of the proposed model compared to existing ones did not become totally clear to me.

final_score: rating: 5: Marginally below acceptance threshold
review_content: While I acknowledge that training generative models with binary latent variables is hard, I'm not sure this paper really makes valuable progress in this direction. The only results that seem promising are those on binarized MNIST, for the non-convolutional architecture, and this setting isn't particularly exciting. All other experiments seem to suggest that the proposed model/algorithm is behind the state of the art. Moreover, the proposed approach is fairly incremental, compared to existing work on RWS, VIMCO, etc.

So while this work seem to have been seriously and thoughtfully executed, I think it falls short of the ICLR acceptance bar.
review_title: OK, but fairly incremental and results underwhelming

final_score: rating: null
review_content: Interesting work!

Just to clarify: You pretrain the encoder/decoder pair and pretrain a RBM on their latent representation? And then during joint training (section 3.2) you block direct gradient flow (disable the soft-binarization) and use VIMCO? 

I’m not convinced I follow the second part of the argument in 3.0: Training SBNs or DBNs with e.g. Gumbel-softmax based methods would allow gradients to flow in a very similar fashion. Doesn’t the “gradient flow” depend on the training method / gradient estimator (in your case soft-binarization) rather than this models structure? 

From this perspective your model is directly comparable to Discrete VAEs, isn’t it? Where Discrete VAEs introduced a different reparam. based training method. 

Have you tried alternative pretraining methods? E.g. using Gumbel-softmax based instead eqn. (3), or VIMCO trained factorial top later SBN? Do we have any idea why joint training might be so hard?

The title seems very broad - large parts of the paper propose and evaluate a pretraining procedure for a specific two layer DBM architecture. 

I’m curious: What are typical log Z estimate for your models in table 1? 
review_title: Interesting work

final_score: rating: 4: Ok but not good enough - rejection
review_content: Interesting work, but I’m not convinced by the arguments nor by the experiments. Similar models have been trained before; it’s not clear that the proposed pretraining procedure is a practical step forwards. And quite some decisions seem ad-hoc and not principled. 

Nevertheless, interesting work for everyone interested in RBMs as priors for “binary VAEs”. 


review_title: Interesting work

Meta-review: The reviewers agreed that while this is a well-written paper, it is low on novelty and does not make a substantial enough contribution. They also pointed out that although the reported MNIST results are highly competitive, possibly due to the use of a powerful ResNet decoder, the CIFAR10/ImageNet results are underwhelming.
Decision: Reject