Venue: ICLR 2017 conference

Reviews:

final_score: rating: 8: Top 50% of accepted papers, clear accept
review_content: This work combines a LDA-type topic model with a RNN and models this by having an additive effect on the predictive distribution via the topic parameters. A variational auto-encoder is used to infer the topic distribution for a given piece of text and the RNN is trained as a RNNLM. The last hidden state of the RNNLM and the topic parameters are then concatenated to use as a feature representation.

The paper is well written and easy to understand. Using the topic as an additive effect on the vocabulary allows for easy inference but intuitively I would expect the topic to affect the dynamics too, e.g. the state of the RNN. The results on using this model as a feature extractor for IMDB are quite strong. Is the RNN fine-tuned on the labelled IMDB data? However, the results for PTB are weaker. From the original paper, an ensemble of 2 LSTMs is able to match the topicRNN score. This method of jointly modelling topics and a language model seems effective and relatively easy to implement.

Finally, the IMDB result is no longer state of the art since this result appeared in May (Miyato et al., Adversarial Training Methods for Semi-Supervised Text Classification).

Some questions:
How important is the stop word modelling? What do the results look like if l_t = 0.5 for all t?

It seems surprising that the RNN was more effective than the LSTM. Was gradient clipping tried in the topicLSTM case? Do GRUs also fail to work?

It is also unfortunate that the model requires a stop-word list. Is the link in footnote 4 the one that is used in the experiments?

Does factoring out the topics in this way allow the RNN to scale better with more neurons? How reasonable does the topic distribution look for individual documents? How peaked do they tend to be? Can you show some examples of the inferred distribution? The topics look odd for IMDB with the top word of two of the topics being the same: 'campbell'. It would be interesting to compare these topics with those inferred by LDA on the same datasets.

Minor comments:
Below figure 2: GHz -> GB
\Gamma is not defined.
review_title: Nice work on feature extraction

final_score: rating: 7: Good paper, accept
review_content: This paper presents TopicRNN, a combination of LDA and RNN that augments traditional RNN with latent topics by having a switching variable that includes/excludes additive effects from latent topics when generating a word. 
Experiments on two tasks are performed: language modeling on PTB, and sentiment analysis on IMBD. 
The authors show that TopicRNN outperforms vanilla RNN on PTB and achieves SOTA result on IMDB.

Some questions and comments:
- In Table 2, how do you use LDA features for RNN (RNN LDA features)? 
- I would like to see results from LSTM included here, even though it is lower perplexity than TopicRNN. I think it's still useful to see how much adding latent topics close the gap between RNN and LSTM.
- The generated text in Table 3 are not meaningful to me. What is this supposed to highlight? Is this generated text for topic "trading"? What about the IMDB one?
- How scalable is the proposed method for large vocabulary size (>10K)?
- What is the accuracy on IMDB if the extracted features is used directly to perform classification? (instead of being passed to a neural network with one hidden state). I think this is a fairer comparison to BoW, LDA, and SVM methods presented as baselines. 
review_title: review

final_score: rating: null
review_content: - How long did it take to train TopicRNN on the PTB and IMBD datasets?
- How much worse are the TopicRNN results with LSTM cell compared to standard RNN cell? 
- Could you please elaborate a little bit more your speculation that "topic models are more effective than LSTM at capturing global semantic information; standard RNNs and topic models are really complementary in this case."? Is this based on your observations on the performance of baseline contextual LSTM models?
review_title: Prereview Questions

final_score: rating: 6: Marginally above acceptance threshold
review_content: This paper introduces a model that blends ideas from generative topic models with those from recurrent neural network language models. The authors evaluate the proposed approach on a document level classification benchmark as well as a language modeling benchmark and it seems to work well. There is also some analysis as to topics learned by the model and its ability to generate text. Overall the paper is clearly written and with the code promised by the authors others should be able to re-implement the approach. I have 2 potentially major questions I would ask the authors to address:

1 - LDA topic models make an exchangability (bag of words) assumption. The discussion of the generative story for TopicRNN should explicitly discuss whether this assumption is also made. On the surface it appears it is since y_t is sampled using only the document topic vector and h_t but we know that in practice h_t comes from a recurrent model that observes y_t-1. Not clear how this clean exposition of the generative model relates to what is actually done. In the Generating sequential text section it’s clear the topic model can’t generate words without using y_1 - t-1 but this seems inconsistent with the generative model specification. This needs to be shown in the paper and made clear to have a complete paper.


2 -  The topic model only allows for linear interactions of the topic vector theta. It seems like this might be required to keep the generative model tractable but seems like a very poor assumption. We would expect the topic representation to have rich interactions with a language model to create nonlinear adjustments to word probabilities for a document. Please add discussion as to why this modeling choice exists and if possible how future work could modify that assumption (or explain why it’s not such a bad assumption as one might imagine)




Figure 2 colors very difficult to distinguish. 
review_title: null

Meta-review: Though the have been attempts to incorporate both "topic-like" and "sequence-like" methods in the past (e.g, the work of Hanna Wallach, Amit Gruber and other), they were quite computationally expensive, especially when high-order ngrams are incorporated. This is a modern take on this challenge: using RNNs and the VAE / inference network framework. The results are quite convincing, and the paper is well written.
 
 Pros:
 -- clean and simple model
 -- sufficiently convincing experimentation
 
 Cons:
 -- other ways to model interaction between RNN and topic representation could be considered (see comments of R2 and R1)
Decision: Accept (Poster)