Title: TopicRNN: A Recurrent Neural Network with Long-Range Semantic Dependency
Authors: Adji B. Dieng
Chong Wang
Jianfeng Gao
John Paisley
Venue: ICLR 2017
Keywords: Natural language processing
Deep learning

Abstract: In this paper, we propose TopicRNN, a recurrent neural network (RNN)-based language model designed to directly capture the global semantic meaning relating words in a document via latent topics. Because of their sequential nature, RNNs are good at capturing the local structure of a word sequence – both semantic and syntactic – but might face difficulty remembering long-range dependencies. Intuitively, these long-range dependencies are of semantic nature. In contrast, latent topic models are able to capture the global underlying semantic structure of a document but do not account for word ordering. The proposed TopicRNN model integrates the merits of RNNs and latent topic models: it captures local (syntactic) dependencies using an RNN and global (semantic) dependencies using latent topics. Unlike previous work on contextual RNN language modeling, our model is learned end-to-end. Empirical results on word prediction show that TopicRNN outperforms existing contextual RNN baselines. In addition, TopicRNN can be used as an unsupervised feature extractor for documents. We do this for sentiment analysis on the IMDB movie review dataset and report an error rate of 6.28%. This is comparable to the state-of-the-art 5.91% resulting from a semi-supervised approach. Finally, TopicRNN also yields sensible topics, making it a useful alternative to document models such as latent Dirichlet allocation.


Review discussion:
- How long did it take to train TopicRNN on the PTB and IMBD datasets?
- How much worse are the TopicRNN results with LSTM cell compared to standard RNN cell?
- Could you please elaborate a little bit more your speculation that "topic models are more effective than LSTM at capturing global semantic information; standard RNNs and topic models are really complementary in this case."? Is this based on your observations on the performance of baseline contextual LSTM models?
Thank you for your comment. We added the answers to your first two questions in the main text and posted a revision.
Regarding your third question:
We are making this speculation given that TopicLSTM gave worse performance in terms of perplexity on PTB. This could be because LSTM already tries to model long-term dependencies and that might interact negatively with the topic bias. This might cause the model to get stuck in a bad local mode. With TopicRNN we only need a few steps (note this prevents vanishing/exploding gradients issues during optimization). The topic bias provided can be seen as some sort of "memory" for the RNN.
Thank you updating the paper.
So would it be fair to say that TopicRNN > RNN but TopicLSTM < LSTM? What about Topic RNN ---- LSTM?
Thanks for your questions. We think it is fair to say that TopicRNN is better than RNN in terms of perplexity.  And we also show in the paper that TopicRNN is better than many variants of LSTM in terms of sentiment classification results in Figure 4. As far as we know, this is the best result on this task. For TopicLSTM, we haven’t had a very definitive conclusion yet except that it is performing slightly worse than TopicRNN. We are working on a journal version of this paper to address these comparisons including TopicLSTM vs LSTM.
Given the fact that Mikolov et al. 2012 and Ghosh et al. (the LSTM version of Mikolov et al.) showed that adding contextual bias to an RNN (vanilla RNN or LSTM) yields better perplexity scores than not adding contextual bias, we hope we can find a better way to optimize TopicLSTM so that we can confirm this.
This work combines a LDA-type topic model with a RNN and models this by having an additive effect on the predictive distribution via the topic parameters. A variational auto-encoder is used to infer the topic distribution for a given piece of text and the RNN is trained as a RNNLM. The last hidden state of the RNNLM and the topic parameters are then concatenated to use as a feature representation.
The paper is well written and easy to understand. Using the topic as an additive effect on the vocabulary allows for easy inference but intuitively I would expect the topic to affect the dynamics too, e.g. the state of the RNN. The results on using this model as a feature extractor for IMDB are quite strong. Is the RNN fine-tuned on the labelled IMDB data?
However, the results for PTB are weaker. From the original paper, an ensemble of 2 LSTMs is able to match the topicRNN score. This method of jointly modelling topics and a language model seems effective and relatively easy to implement.
Finally, the IMDB result is no longer state of the art since this result appeared in May (Miyato et al., Adversarial Training Methods for Semi-Supervised Text Classification).
Some questions:
How important is the stop word modelling? What do the results look like if l_t = 0.5 for all t?
It seems surprising that the RNN was more effective than the LSTM. Was gradient clipping tried in the topicLSTM case? Do GRUs also fail to work?
It is also unfortunate that the model requires a stop-word list. Is the link in footnote 4 the one that is used in the experiments?
Does factoring out the topics in this way allow the RNN to scale better with more neurons? How reasonable does the topic distribution look for individual documents? How peaked do they tend to be? Can you show some examples of the inferred distribution? The topics look odd for IMDB with the top word of two of the topics being the same: 'campbell'. It would be interesting to compare these topics with those inferred by LDA on the same datasets.
Minor comments:
Below figure 2: GHz -> GB
\Gamma is not defined.
Thank you for your feedback and questions!
- We added an explanation for our rationale of passing the topic vector directly to the output layer at the bottom of page 4 of the revised paper.
- The RNN is trained using only the training data reviews. The labels were not used for training TopicRNN. The labels are used to train a simple classifier that uses the features extracted fromTopicRNN.
- We believe the results from PTB are very encouraging.
For example, with only 100 neurons we are able to achieve lower perplexity score with TopicGRU than a stack of 2 layers of LSTMS with 200 hidden units in each layer: 112.4 vs 115.9. See table 2.
- Thank you for mentioning the Miyato et al. reference. We added it to the paper. We were not aware of it by the time we submitted the paper.
However, note their approach is semi-supervised. They use the labels to train their network whereas we use an unsupervised approach. We added their SOTA score to table 4.
- Regarding the stop word modeling, we added a note on the bottom of page 4.
- TopicLSTM and TopicGRU actually perform better than TopicRNN when the number of hidden units is greater than 10. We corrected a bug on the computation of the ELBO and reported the new results on Table 2. Yes, we performed gradient clipping.
-The stop word list is the one we provided in the link. As we mention in the discussion section, we leave as future work the dynamic discovery of the stop words.
When the stop words are discovered dynamically a stop words list won’t be needed.
-We added inferred distributions from three different documents on page 6 in figure 2.  Thank you for suggesting this! As can be seen in that figure, some “topic” components are a lot higher than others for different documents.
Thanks for the updated paper and the results with the bug corrected. I am keeping my rating.
Thank you AnonReviewer1!
This paper presents TopicRNN, a combination of LDA and RNN that augments traditional RNN with latent topics by having a switching variable that includes/excludes additive effects from latent topics when generating a word.
Experiments on two tasks are performed: language modeling on PTB, and sentiment analysis on IMBD.
The authors show that TopicRNN outperforms vanilla RNN on PTB and achieves SOTA result on IMDB.
Some questions and comments:
- In Table 2, how do you use LDA features for RNN (RNN LDA features)?
- I would like to see results from LSTM included here, even though it is lower perplexity than TopicRNN. I think it's still useful to see how much adding latent topics close the gap between RNN and LSTM.
- The generated text in Table 3 are not meaningful to me. What is this supposed to highlight? Is this generated text for topic "trading"? What about the IMDB one?
- How scalable is the proposed method for large vocabulary size (>10K)?
- What is the accuracy on IMDB if the extracted features is used directly to perform classification? (instead of being passed to a neural network with one hidden state). I think this is a fairer comparison to BoW, LDA, and SVM methods presented as baselines.
Thanks for your questions and for suggesting we add results for TopicLSTM!
- In Table 2, the first two lines were results reported in Mikolov et al 2012. They run LDA separately and extract features for words using the topic matrix.
- We included results from TopicRNN, TopicLSTM, and TopicGRU.
Contrary to what we mentioned earlier, TopicLSTM and TopicGRU actually perform very well. We corrected a bug on the computation of the ELBO. The new results are consistent with the story and are reported in table 2. (We are also running experiments with TopicGRU/TopicLSTM on IMDB data.
However, it takes some time to finish. We will add them when they are available.)
- Each text is generated using one example input document. The input for IMDB was a negative review. That sentiment is reflected in the generated text. Note one can sample from the prior for the topic vector \theta and use that as bias on the trained model.
-TopicRNN is a language model. As reported at the bottom of page 5, the complexity is dominated by the computation of the softmax output layer as is the case for language models.
As such, all methods for dealing with the softmax layer are also applicable to TopicRNN. We reported the computation time in the experiments section to give an idea.
- We followed the procedure in Paragraph Vector. The main comparison here is against other unsupervised neural network based approaches (ex: Le and Mikolov 2014). Note it is also possible to train the classifier directly with TopicRNN.
However, we wanted to highlight TopicRNN as unsupervised feature extractor.
This paper introduces a model that blends ideas from generative topic models with those from recurrent neural network language models. The authors evaluate the proposed approach on a document level classification benchmark as well as a language modeling benchmark and it seems to work well. There is also some analysis as to topics learned by the model and its ability to generate text.
Overall the paper is clearly written and with the code promised by the authors others should be able to re-implement the approach. I have 2 potentially major questions I would ask the authors to address:
1 - LDA topic models make an exchangability (bag of words) assumption. The discussion of the generative story for TopicRNN should explicitly discuss whether this assumption is also made. On the surface it appears it is since y_t is sampled using only the document topic vector and h_t but we know that in practice h_t comes from a recurrent model that observes y_t-1. Not clear how this clean exposition of the generative model relates to what is actually done. In the Generating sequential text section it’s clear the topic model can’t generate words without using y_1 - t-1 but this seems inconsistent with the generative model specification. This needs to be shown in the paper and made clear to have a complete paper.
2 -  The topic model only allows for linear interactions of the topic vector theta. It seems like this might be required to keep the generative model tractable but seems like a very poor assumption. We would expect the topic representation to have rich interactions with a language model to create nonlinear adjustments to word probabilities for a document. Please add discussion as to why this modeling choice exists and if possible how future work could modify that assumption (or explain why it’s not such a bad assumption as one might imagine)
Figure 2 colors very difficult to distinguish.
Thanks for your questions.
1. We believe there was a misunderstanding of our proposed model.
Unlike LDA, TopicRNN is a sequential model (as expressed in the generative process in the middle of page 4) and as such does not make the exchangeability assumption.
The inference network that produces the topic vector \theta used as bias takes as input Xc which is a bag of words representation of the document. Xc excludes stop words just as done in topic modeling. This is where exchangeability is needed and maybe where the confusion is coming from.
But this is the inference network for \theta, not the actual generative model.
2. There are three main reasons behind our choice of using the topic vector as bias instead of passing it into the hidden states of the RNN.
a) First, this enables us to have a clear separation of the contributions of global semantics and those of local dynamics. The global semantics come from the topics which are meaningful when stop words are excluded.
However, these stop words are needed for the local dynamics of the language model. We hence achieve this separation of global vs local via a binary decision model for the stop words. It is unclear how to achieve this if we pass the topics to the hidden states of the RNN. This is because the hidden states of the RNN will account for all words (including stop words) whereas topics exclude stop words. Passing the topics through the hidden states of the RNN violates this.
b) Second, we show empirical evidence that our approach does better than previous ways of integrating topic models into RNNs. This modeling choice also allows discovering interpretable topics within a single model.
c) Finally, this modeling choice allows easier end-to-end training of the model. And we argue that although we do not have the topics directly going into the hidden states, they will affect the whole trained model, including the hidden states due to our end-to-end training approach (unlike the previous work that use pre-trained topic models).
We added this note on page 4 of the manuscript.
We thank the reviewers and the anonymous commenters for the helpful feedback and questions!
We first summarize the main idea of this paper below:
Neural network-based language models have achieved state of the art results on many NLP tasks. One difficult problem is to capture long-range dependencies as motivated in the introduction of this paper. We propose to solve this by integrating latent topics as context and jointly training these contextual features with the parameters of an RNN language model. We provide a natural way of doing this integration by modeling stop words that are excluded by topic models but needed for sequential language models. This is done via binary classification where the probability of being a stop word is dictated by the hidden layer of the RNN.
This modeling approach is possible when the contextual features as provided by the topics are passed directly to the softmax output layer of the RNN as additional bias. We illustrate the performance of this approach on two tasks and two datasets: word prediction on PTB and sentiment analysis on IMDB. We provide competitive perplexity scores on PTB showing more generalization capabilities (for example we only need a TopicGRU with 100 neurons to achieve a better perplexity than stacking 2 LSTMs with 200 neurons each ---112. 4 vs 115. 9). "This method of jointly modeling topics and a language model seems effective and relatively easy to implement. " quoted from AnonReviewer1.
We have revised the paper and added the following changes:
1- we added a line on the middle of page 7 to clarify even more how we compute the topic vector \theta using a sliding window for word prediction.
2- we added the test perplexity scores for TopicRNN, TopicLSTM, and TopicGRU as required by AnonReviewer3.
3- we added the inferred distributions from some documents as required by AnonReviewer1.
4- we added an explanation of why we passed the topics directly to the output layer at the bottom of page 4.
We answer each reviewer individually. See below.

Meta-review:
role: PC
data: comment: Though the have been attempts to incorporate both "topic-like" and "sequence-like" methods in the past (e.g, the work of Hanna Wallach, Amit Gruber and other), they were quite computationally expensive, especially when high-order ngrams are incorporated. This is a modern take on this challenge: using RNNs and the VAE / inference network framework. The results are quite convincing, and the paper is well written.
 
 Pros:
 -- clean and simple model
 -- sufficiently convincing experimentation
 
 Cons:
 -- other ways to model interaction between RNN and topic representation could be considered (see comments of R2 and R1)

Decision: Accept (Poster)