Venue: ICLR 2017 conference

Reviews:

final_score: rating: null
review_content: In some of the experiments, it seems like ProdLDA + neural variational inference (NVI) works best. Given that NVI is worse than collapsed Gibbs sampling for LDA, the experiments seem to confound improvements to model vs inference. Can you evaluate a difference inference scheme for ProdLDA? That'd help tease out the relative significance of the contributions. 
review_title: improvements to model (ProdLDA / LDA) vs improvements to inference (NVI / CGS)

final_score: rating: 6: Marginally above acceptance threshold
review_content: This is an interesting paper on a VAE framework for topic models. The main idea is to train a recognition model for the inference phase which, because of so called “amortized inference” can be much faster than normal inference where inference must be run iteratively for every document. Some comments:
Eqn 5: I find the notation p(theta(h)|alpha) awkward. Why not P(h|alpha) ?
The generative model seems agnostic to document length, meaning that the latent variables only generate probabilities over word space. However, the recognition model is happy to radically change the probabilities q(z|x) if the document length changes because the input to q changes. This seems undesirable. Maybe they should normalize the input to the recognition network?
The ProdLDA model might well be equivalent to exponential family PCA or some variant thereof: http://jmlr.csail.mit.edu/proceedings/papers/v9/li10b/li10b.pdf
Section 4.1: error in the equation. The last term should be 
Prod_i exp(delta*_r_i) * exp((1-delta)*s_i).
Last paragraph 4.1. The increment relative to NVDM seems small: approximating the Dirichlet with a Gaussian and high momentum training. While these aspects may be important in practice they are somewhat incremental.
I couldn’t find the size of the vocabularies of the datasets in the paper. Does this method work well for very high dimensional sparse document representations?
The comment on page 8 that the method is very sensitive to optimization tricks like very high momentum in ADAM and batch normalization is a bit worrying to me. 
In the end, it’s a useful paper to read, but it’s not going to be the highlight of the conference. The relative increment is somewhat small and seems to heavily rely optimization tricks.

review_title: VAE model for LDA. Interesting idea, but a incremental.  

final_score: rating: 7: Good paper, accept
review_content: This paper proposes the use of neural variational inference method for topic models. The paper shows a nice trick to approximate Dirichlet prior using softmax basis with a Gaussian and then the model is trained to maximize the variational lower bound. Also, the authors study a better way to alleviate the component collapsing issue, which has been problematic for continuous latent variables that follow Gaussian distribution. The results look promising and the experimental protocol sounds fine.

Minor comments:
Please add citation to [1] or [2] for neural variational inference, and [2] for VAE. 
A typo in “This approximation to the Dirichlet prior p(θ|α) is results in the distribution”, it should be “This approximation to the Dirichlet prior p(θ|α) results in the distribution”

In table 2, it is written that DMFVI was trained more than 24hrs but failed to deliver any result, but why not wait until the end and report the numbers?

In table 3, why are the perplexities of LDA-Collapsed Gibbs and NVDM are lower while the proposed models (ProdLDA) generates more coherent topics? What is your intuition on this?

How does the training speed (until the convergence) differs by using different learning-rate and momentum scheduling approaches shown as in figure 1?

It may be also interesting to add some more analysis on the latent variables z (component collapsing and etc., although your results indirectly show that the learning-rate and momentum scheduling trick removes this issue).

Overall, the paper clearly proposes its main idea, explain why it is good to use NVI, and its experimental results support the original claim. It explains well what are the challenges and demonstrate their solutions. 

[1] Minh et al., Neural Variational Inference and Learning in Belief Networks, ICML’14
[2] Rezende et al., Stochastic Backpropagation and Approximate Inference in Deep Generative Models, ICML’14
review_title: Nice paper to read

final_score: rating: 6: Marginally above acceptance threshold
review_content: The authors propose NVI for LDA variants. The authors compare NVI-LDA to standard inference schemes such as CGS and online SVI. The authors also evaluate NVI on a different model ProdLDA (not sure this model has been proposed before in the topic modeling literature though?)

In general, I like the direction of this paper and NVI looks promising for LDA. The experimental results however confound model vs inference which makes it hard to understand the significance of the results. Furthermore, the authors don't discuss hyper-parameter selection which is known to significantly impact performance of topic models. This makes it hard to understand when the proposed method can be expected to work. 

Can you maybe generate synthetic datasets with different Dirichlet distributions and assess when the proposed method recovers the true parameters?

Figure 1: Is this prior or posterior? The text talks about sparsity whereas the y-axis reads "log p(topic proportions)" which is a bit confusing.

Section 3.2: it is not clear what you mean by unimodal in softmax basis. Consider a Dirichlet on K-dimensional simplex with concentration parameter alpha/K where alpha<1 makes it multimodal. Isn't the softmax basis still multimodal?

None of the numbers include error bars. Are the results statistically significant?


Minor comments:

Last term in equation (3) is not "error"; reconstruction accuracy or negative reconstruction error perhaps?

The idea of using an inference network is much older, cf. Helmholtz machine. 

review_title: Promising direction, but the paper needs more work

final_score: rating: 5: Marginally below acceptance threshold
review_content: The comparison to NVDM looks unfair since the user introduces a couples tricks (Dirichlet prior, batch normalisation, high momentum training, etc.) which NVDM doesn't use. A more convincing experimental design is to explore the effect of each trick separately in neural variational inference. 

review_title: Comparison to NVDM looks unfair

Meta-review: The reviewers agree that the approach is interesting and the paper presents useful findings. They also raise enough questions and suggestions for improvements that I believe the paper will be much stronger after further revision, though these seem straightforward to address.
Decision: Accept (Poster)