Venue: ICLR 2018 conference

Reviews:

final_score: rating: 6: Marginally above acceptance threshold
review_content: The paper presents a deep Poisson model where the last layer is the vector of word counts generated by a vector Poisson. This is parameterized by a matrix vector product, and the vector in this parameterizeation is itself generated by a vector Gamma with a matrix-vector parameterization. From there the vectors are all Gammas with matrix-vector parameterizations in a typical deep setup.

While the model is reasonable, the purpose was not clear to me. If only the last layer generates a document, then what use is the deep structure? For example, learning hierarchical topics as in Figure 4 doesn't seem so useful here since only the last layer matters. Also, since no input is being mapped to an output, what does going deeper mean? It doesn't look like any linear mapping is being learned from the input to output spaces, so ultimately the document itself is coming from a simple linear Poisson model just like LDA and other non-deep methods.

The experiments are otherwise thorough and convincing that quantitative performance is improved over previous attempts at the problem.
review_title: a deep Poisson model

final_score: rating: 6: Marginally above acceptance threshold
review_content: The authors develop a hybrid amortized variational inference MCMC inference 
framework for deep latent Dirichlet allocation. Their model consists of a stack of
 gamma factorization layers with a Poisson layer at the bottom. They amortize 
inference at the observation level using a Weibull approximation. The structure 
of the inference network mimics the MCMC sampler for this model. Finally they 
use MCMC to infer the parameters shared across data. A couple of questions:

1) How effective are the MCMC steps at mixing? It looks like this approach helps a 
bit with local optima?

2) The gamma distribution can be reparameterized via its rejection sampler 

@InProceedings{pmlr-v54-naesseth17a,
  title = 	 {{Reparameterization Gradients through Acceptance-Rejection Sampling Algorithms}},
  author = 	 {Christian Naesseth and Francisco Ruiz and Scott Linderman and David Blei},
  booktitle = 	 {Proceedings of the 20th International Conference on Artificial Intelligence and Statistics},
  pages = 	 {489--498},
  year = 	 {2017}
}

I think some of the motivation for the Weibull is weakened by this work. Maybe a 
comparison is in order?

3) Analytic KL divergence can be good or bad. It depends on the correlation between 
the gradients of the stochastic KL divergence and the stochastic log-likelihood

4) One of the original motivations for DLDA was that the augmentation scheme 
removed the need for most non-conjugate inference. However, this approach doesn't 
use that directly. Thus, it seems more similar to inference procedure in deep exponential 
families. Was the structure of the inference network proposed here crucial?

5) How much like a Weibull do you expect the posterior to be? This seems unclear.
review_title: official review

final_score: rating: 5: Marginally below acceptance threshold
review_content: The authors propose a hybrid Bayesian inference approach for deep topic models that integrates stochastic gradient MCMC for global parameters and Weibull-based multilayer variational autoencoders (VAEs) for local parameters. The decoding arm of the VAE consists of deep latent Dirichlet allocation, and an upward-downward structure for the encoder. Gamma distributions are approximated as Weibull distributions since the Kullback-Leibler divergence is known and samples can be efficiently drawn from a transformation of samples from a uniform distribution. 

The results in Table 1 are concerning for several reasons, i) the proposed approach underperfroms DLDA-Gibbs and DLDA-TLASGR. ii) The authors point to the scalability of the mini-batch-based algorithms, however, although more expensive, DLDA-Gibbs, is not prohibitive given results for Wikipedia are provided. iii) The proposed approach is certainly faster at test time, however, it is not clear to me in which settings such speed (compared to Gibbs) would be needed, provided the unsupervised nature of the task at hand. iv) It is not clear to me why there is no test-time difference between WAI and WHAI, considering that in the latter, global parameters are sampled via stochastic-gradient MCMC. One possible explanation being that during test time, the approach does not use samples from W but rather a summary of them, say posterior means, in which case, it defeats the purpose of sampling from global parameters, which may explain why WAI and WHAI perform about the same in the 3 datasets considered.

- \Phi is in a subset of R_+, in fact, columns of \Phi are in the P_0-dimensional simplex.
- \Phi should have K_1 columns not K.
- The first paragraph in Page 5 is very confusing because h is introduced before explicitly connecting it to k and \lambda. Also, if k = \lambda, why introduce different notations?
review_title: WHAI: WEIBULL HYBRID AUTOENCODING INFERENCE FOR DEEP TOPIC MODELING

Meta-review: The paper proposes a new approach for scalable training of deep topic models based on amortized inference for the local parameters and stochastic-gradient MCMC for the global ones.  The key aspect of the method involves using Weibull  distributions (instead of Gammas) to model the variational posteriors over the local parameters, enabling the use of the reparameterization trick. The resulting methods perform slightly worse that the Gibbs-sampling-based approaches but are much faster at test time. Amortized inference has already been applied to topic models, but the use of Weibull posteriors proposed here appears novel. However, there seems to be no clear advantage to using stochastic-gradient MCMC instead of vanilla SGD to infer the global parameters, so the value of this aspect of WHAI unclear.
Decision: Accept (Poster)