Recurrent Hierarchical Topic-Guided Neural Language Models

Abstract: To simultaneously capture syntax and semantics from a text corpus, we propose a new larger-context language model that extracts recurrent hierarchical semantic structure via a dynamic deep topic model to guide natural language generation. Moving beyond a conventional language model that ignores long-range word dependencies and sentence order, the proposed model captures not only intra-sentence word dependencies, but also temporal transitions between sentences and inter-sentence topic dependences. For inference, we develop a hybrid of stochastic-gradient MCMC and recurrent autoencoding variational Bayes. Experimental results on a variety of real-world text corpora demonstrate that the proposed model not only outperforms state-of-the-art larger-context language models, but also learns interpretable recurrent multilayer topics and generates diverse sentences and paragraphs that are syntactically correct and semantically coherent.

Reviews and author responses:

[Additional review]
This paper proposes a technique to incorporate document-level topic model information into language models. 

While the underlying idea is interesting, my biggest issue is with the misleading assertions at the very beginning of the paper. In the second paragraph of Section 1, the paper claims that RNN-based LMs often make independence assumptions between sentences, hence why they develop a topic modelling approach to model document-level information. Some issues with this claim:

1. Pretty much every LM paper that evaluates on language modelling benchmark (PTB, WT-103, Wikitext-2) uses LSTMs/Transformers incorporate cross-sentential, document-level information as context, through a very simple approach of just concatenating all the sentences and adding a unique token to mark sentence boundaries.

2. Prior work has shown that LSTMs/Transformers with cross-sentential context can, and in fact do, make use of information from previous sentences.

a. Evidence 1: Khandelwal et al. (2018) showed that LSTMs memorise word orders from the past ~50 tokens, and retain semantic information from the past ~200 tokens; both of which extend far beyond the length of an average sentence, suggesting that information from the previous sentences is used in the predictions of the current sentence.

b. Evidence 2: Language models that operate on single sentences typically do worse than language models that take into account cross-sentential context, e.g. the language model of Kim et al. (2019) that operates on single sentences gets ~90 ppl. on PTB test set, while LSTMs that condition on multiple sentences get a much better ~50-something ppl. on Mikolov PTB.

Crucially, these prior works defeat the paper’s motivation of why it claims to need topic models in the first place (i.e. to model cross-sentential context), while just concatenating multiple sentences as context would do, and in fact has been done many times.

2. Prior work (mostly in Transformer-land) has come up with ways to make use of very long-range context, from Transformer-XL to the more recent compressive Transformer (https://openreview.net/forum?id=SylKikSYDH) that can condition on entire books. While these are done for Transformers, in principle one can also apply similar techniques to LSTMs.

3. While Transformer-XL has the potential to make use of word orders in the preceding sentences, it seems that this paper’s approach cannot do that, since they only take the bag-of-words from the preceding sentences. It thus seems that their bag-of-word approach is less expressive, and hence less powerful, than the simpler alternative of concatenating sentences.

4. The perplexity results (Table 1) are not done on very standard datasets (no PTB evaluation for instance). It is thus hard to evaluate the strength of the baseline models. In the paper's defense, it seems that they were following the experimental setup of Wang et al. (2019), but the paper should elaborate more on the choice of evaluation datasets.

5. The inference part is not particularly self-contained. The paper simply refers the TLASGR-MCMC method (which is an important part to make inference scalable) to prior work (Cong et al., 2017; Zhang et al., 2018), yet does not explain (even briefly) how the approach works, and how it can be combined with their recurrent topic model formulation.  

6. Evaluation of the induced topic hierarchy (Figure 4) is only done through qualitative samples, and the paper does not really explain how to pick the samples (i.e. possible cherry-picking). I am not very familiar with the topic modelling literature, but it would be nice if the induced hierarchy can be evaluated quantitatively.

References:
1. Urvashi Khandelwal, He He, Peng Qi, and Dan Jurafsky. Sharp nearby, fuzzy far away. In Proc. of ACL 2018.
2. Yoon Kim, Alexander Rush, Lei Yu, Adhiguna Kuncoro, Chris Dyer, and Gabor Melis. Unsupervised recurrent neural network grammars. In Proc. of NAACL 2019.
3. Wenlin Wang, Zhe Gan, Hongteng Xu, Ruiyi Zhang, Guoyin Wang, Dinghan Shen, Changyou Chen, and Lawrence Carin. Topic-guided variational autoencoders for text generation. In Proc. of NAACL 2019.
4. Yulai Cong, Bo Chen, Hongwei Liu, and Mingyuan Zhou. Deep latent Dirichlet allocation with topic-layer-adaptive stochastic gradient Riemannian MCMC. In Proc. of ICML 2017
5. Hao Zhang, Bo Chen, Dandan Guo, and Mingyuan Zhou. WHAI: Weibull hybrid autoencoding inference for deep topic modeling. In Proc. of ICLR 2018

The model description is confusing and lots of statements are presented without appropriate or enough justification. For example, (1) in the last paragraph of page 2, they claimed that the language component is used in their model to capture syntactic information, which I do not feel comfortable to accept; (2) in the first paragraph of page 3, it says "we define d_j as the BoW vector summarizing only the preceding sentences", without further information, I have no idea what a BoW vector looks like or how it is constructed; (3) in the last paragraph of page 3, it says using Dirichlet priors to make "the latent representation more identifiable and interpretable, but also facilitates inference", which I really don't know what it means. There are a few more examples like these. 

More importantly, I think Eq. (5) is wrong, which makes me question their whole methodology. To be specific, in their definition, d_j refers to a summary of all the sentences other than s_j. That means, 
- for s_1, d_1 is defined on s_2, s_3, s_4, ..., s_J; and 
- for s_2, d_2 is defined on s_1, s_3, s_4, ..., s_J. 
In other words, there is a huge overlap between any two d_j and d_{j'}. Therefore, I am not sure the decomposition on the right hand side of equation 5 (particularly, the decomposition of p(d_j | ...) ) is valid. 

Although they have some interesting results and the lowest PPLx comparing to other models, I do not think this paper is ready to be accepted. 

We appreciate your comments and suggestions. Given the time constraint, we leave comprehensive comparisons between rGBN-RNN and Transformer based language models for future study. We have followed your suggestion to revise our claim to be: "the proposed model not only outperforms state-of-the-art larger-context RNN-based language models, but also..." 

Thank you for your comments and suggestions. We have revised our paper accordingly and highlighted the main changes in red.

Q1: It seems strange not to mention all of the recent high-profile work on LM-based pre-training, since my impression is that these models operate effectively with large multi-sentence contexts. Do models like BERT and GPT-2 fail to take into account inter-sentence relations, as the paper claims most LMs do? I would like to see more discussion of how this work fits with that. 

A1: Thanks for your suggestion and we have now added related discussion about the recent high-profile work on LM-based pre-training (e.g., in the third paragraph of Section 3.1). First, while Transformer based LMs have been shown to be powerful, they often have significantly more parameters and require significantly more computation to train than RNN based LMs. Second, we do not view rGBN-RNN and Transformer based LMs as directly comparable given their differences in sizes and structures; after carefully comparing rGBN-RNN with Transformer based LMs, we believe a promising future extension of rGBN-RNN is to replace its stacked RNN with a Transformer-based LM, i.e., constructing a rGBN guided Transformer (rGBN-Transformer). Note the AC had made related comments, to which we had provided detailed response; please see that response for more details. 

Q2: I don’t know that it makes sense to highlight as the contribution of this model that it can "simultaneously capture syntax and semantics". It’s not clear to me that other language models fail to capture semantics (keeping in mind that semantics applies within a sentence and not just at a global level) – rather, it seems that the strength of this model is in capturing semantic relations above the sentence level. If this is correct, that should be expressed more precisely.

A2: Thank you for your insightful comments. We have changed “semantics” to “global semantics” to more precisely express the idea that rGBN-RNN can capture semantic relations above the sentence level.

Q3:  It’s not clear to me what we learn from Figure 3. The claim is that "the color of the hidden states of the stacked RNN based language model at layer 1 changes quickly ... because lower layers are in charge of learning short-term dependencies", but looking at the higher layers I’m not seeing clear evidence of capturing of long-distance dependencies, or even clear capturing of syntactic constituents. The takeaways from that figure should be made clearer and should be sure to correspond to what we can actually confidently conclude from that analysis.

A3:  Thank you for your suggestion. We have carefully revised the first paragraph of Section 3.2 to explain in detail how to interpret Figure 3 (by comparing the segments exhibited in the L2-norm sequence of a hidden layer to the corresponding words in the sentence), which now more clearly supports our claim.

Thank you for your comments and suggestions. We have revised our paper accordingly and highlighted the main changes in red.

Q1: Though the paper is relatively well written, it would have been good to explain some points on architecture and inference. It would have been better to provide the rationale behind some architectural decisions like associating \theta^1 and g^1 as against g^3. Related to this, Figure 1 has a typo where \theta^2 is associated with g^3.

A1: Thank you for your insightful comments, which make us realize that the upward arrows in Fig. 1(b) and/or the upward red arrows in Fig. 1(c) can be flipped, leading to three additional architectural variations of the proposed rGBN-RNN. Since the current rGBN-RNN already clearly outperforms other RNN based models, given both the space and time constraints, we leave these three architectural variations of rGBN-RNN to future study. Also thank you for catching the typo! We have fixed it.

Q2: An explanation on combining all the latent representation in the RNN model used for language modeling will be helpful, though this is motivated by previous approaches.

A2: We have added two reasons to explain why combining all the latent representation to help language modeling. Please see the paragraph below Equation (4) for detail.

Q3: A proper explanation the TLASGR-MCMC approach for sampling from the posterior of rGBN parameters is missing in the main paper. It would be good to provide some details of this in the main paper.

A3: We have added more details on TLASGR-MCMC to Appendix C. For now, we have kept all the details of TLASGR-MCMC in Appendix C due to the space constraint (we find it difficult to move only one or two equations back to the main paper while maintaining the clarity of the technical details). We are open to your further suggestions on this.  

Q4: Experimental section compares the proposed approach against many SOTA approaches for the language modeling task. It would have been good to provide a quantitive evaluation of the topic modeling task also in addition to demonstrating them qualitatively. 

A4: We have not provided quantitative evaluation of the topic modeling task mainly for two reasons. First, as our paper is focused on improving an RNN-based language model with a deep dynamic topic model, adding that evaluation may distract it from the main purpose. Second, our topics learned by rGBN-RNN are hierarchical between different layers and recurrent at the same layer. Thus it is unclear whether existing quantitative measures (e.g., topic coherence), which are often designed to evaluate conventional single-layer topic model without temporal structure, would be appropriate to evaluate the quality of the rGBN topics that are both hierarchical and temporally linked.


Thank you for your positive feedback. We have revised our paper accordingly and highlighted the main changes in red.

Q: One suggestion is that the authors didn’t include computational analysis about the complexity and loads of the proposed method as compared with the baseline methods.

A: Thank you for your suggestion. We have revised our paper to include a comprehensive complexity analysis, as shown in Appendix E. Examining Table 1 and the newly added Table 4 in Appendix E, one can find that the proposed rGBN-RNN achieves better performance with fewer parameters than comparable baselines. We note that when rGBN-RNN is used as a language model after training (which means the inferred topics $\Phi$ are no longer needed), the number of parameters of the rGBN topic model component is dominated by that of the RNN language model component.


This paper presents rGBN-RNN, a model that integrates a hierarchical recurrent topic model with an RNN-based language model in order to incorporate global semantic information and improve capturing of inter-sentence relations. The proposed model improves in perplexity across the three tested datasets over state of the art models of comparable type, and follow-up analyses show strong performance in sentence and paragraph generation, as well as learning of sensible hierarchical topics.

Overall I think this is a clearly-written paper with a well-motivated and interesting model, strong results, and a good range of follow-up analyses. I think that it is a solid paper to accept for publication. 

Some areas for improvement:

It seems strange not to mention all of the recent high-profile work on LM-based pre-training, since my impression is that these models operate effectively with large multi-sentence contexts. Do models like BERT and GPT-2 fail to take into account inter-sentence relations, as the paper claims most LMs do? I would like to see more discussion of how this work fits with that.

I don't know that it makes sense to highlight as the contribution of this model that it can "simultaneously capture syntax and semantics". It's not clear to me that other language models fail to capture semantics (keeping in mind that semantics applies within a sentence and not just at a global level) -- rather, it seems that the strength of this model is in capturing semantic relations above the sentence level.  If this is correct, that should be expressed more precisely.

It's not clear to me what we learn from Figure 3. The claim is that "the color of the hidden states of the stacked RNN based language model at layer 1 changes quickly ... because lower layers are in charge of learning short-term dependencies", but looking at the higher layers I'm not seeing clear evidence of capturing of long-distance dependencies, or even clear capturing of syntactic constituents. The takeaways from that figure should be made clearer and should be sure to correspond to what we can actually confidently conclude from that analysis.



The paper  proposes deep recurrent topic model guided language modeling using a stacked RNN, and uses a novel variational recurrent inference network to learn the parameters.  The proposed model can capture the dependence across the sentences in language generation though the recurrent latent topics. Moreover, the deep rGBN architecture provides Gamma distributed topic topic weight vectors which can be associated with every layer of the stacked RNN generating the sentence. The parameters of both the hierarchical recurrent topic model and language model are learnt using a hybrid inference algorithm  combining  variational inference to estimate language model and inference network parameters and MCMC to infer rGBN parameters. The effectiveness of the proposed model on the language modeling task is demonstrated on 3 datasets using Perplexity and BLEU score. The paper also provides a visual representation of the topics and their temporal trajectories. 

The proposed model extends previous approaches on topic guided language modeling by using deep rGBN model. Though the novelty of the model is limited, learning and inference with the proposed model is non-trivial. Further, the paper show an improvement in performance on language modeling using the proposed approach over SOTA approaches, demonstrating the significance of the proposed approach.  

Though the paper is relatively well written, it would have been good to explain some points on architecture and inference. It would have been better to provide the rationale behind some architectural decisions like associating \theta^1 and g^1 as against g^3. Related to this, Figure 1 has a typo where \theta^2 is associated with g^3.  An explanation on combining all the  latent representation in the RNN model used for language modeling will be helpful, though this is motivated by previous approaches. A proper explanation the TLASGR-MCMC approach for sampling from the posterior of  rGBN parameters is missing in the main paper.  It would be good to provide some details of this in the main paper. 

Experimental section compares the proposed approach against many SOTA approaches for the language modeling task. It would have been good to provide a quantitive evaluation of the topic modeling task also  in addition to demonstrating them qualitatively. 

This paper presents a method for natural language generation, using a language model, informed by a topic model. 
The topic model is a hierarchical recurrent topic model that attempts to extract document-level word concurrence patterns and topic weight vectors for sentences. 
The language model is a stacked RNN model, aiming to capture word sequential dependencies. 

The proposed method is a combination of two existing methods, i.e. gamma-belief networks  and stacked RNN, where the stacked RNN is improved with the information from recurrent gamma belief network. 

Overall, this is a well written paper, clearly presented, with certain novelties. The method is well formulated mathematically and evaluated experimentally. The results look interesting especially for capturing the long-range dependencies, as shown by the  BLEU scores. One suggestion is that the authors didn't include computational analysis about the complexity and loads of the proposed method as compared with the baseline methods. 

Thank you for the clarification. In the abstract of the paper and elsewhere, it says that the proposed model "outperforms state-of-the-art larger-context language models". I would argue that state of the art larger context language models are based on Transformers, and without a comparison this claim has not been validated. Given that claims of state-of-the-art results seem to be a major selling point for the paper, I think these claims can either be toned down ("outperforms other RNN-based language models"), or an empirical comparison could be performed.

Dear Pankaj,

Thank you for suggesting both Larochelle & Lauly (2012) and your own publication in ICLR 2019.

As this paper is focusing on improving a language model with a deep dynamic topic model, we think it would be unnecessary, even distracting, to evaluate topic coherence in this paper. We note there are publications, such as [3] and [4], that have evaluated the topic coherence for the topics produced by gamma belief networks (GBNs); we encourage you to check these publications for details. If we submit a paper in the future that is focused on improving a topic model with the help of a language model, we will cite your paper and add comparison, if appropriate. 

[3] He Zhao, Lan Du, Wray Buntine, and Mingyuan Zhou. "Dirichlet belief networks for topic structure learning." In NeurIPS 2018.

[4] He Zhao, Lan Du, Wray Buntine, and Mingyuan Zhou. "Inter and intra topic structure learning with word embeddings." In International Conference on Machine Learning, pp. 5887-5896. 2018.


Thank you for bringing these two papers to our attention and suggesting Transformer-XL for comparison. While we are studying both papers carefully to see whether we can design additional experiments to provide meaningful comparisons, we have a number of clear reasons to explain why the proposed rGBN-RNN and Transformer-XL are not directly comparable:

1) Model size: For language modeling, the model size of Transformer-XL is one or two orders of magnitude larger than that of rGBN-RNN. For example, without considering the word embedding layers, Transformer-XL 12L and 24L have 41M and 277M parameters, respectively, while the proposed rGBN-RNN with 3 stochastic hidden layers has as few as 7.3M parameters. 

2) Model construction: rGBN-RNN guides stacked-RNN, a sentence-level language model, with rGBN, a deep dynamic topic model, to construct a larger-context language model, which clearly enhances the performance of stacked-RNN. Therefore, a promising extension is to replace stacked-RNN with Transformer (which essentially consists of stacked multi-head self-attention modules), i.e., constructing a rGBN guided Transformer (rGBN-Transformer). In other words, Transformer-XL and rGBN-Transformer would be comparable. However, rGBN-Transformer is beyond the scope of the current paper and will be our future work.

3) Interpretability: In comparison to Transformer-XL, rGBN-RNN is much more interpretable and one can clearly understand its underlying mechanism to capture short-range, middle-range, and longe-range word dependencies.  For example, rGBN-RNN has the ability to learn interpretable recurrent multilayer topics from the documents, as shown in Figure 4. Besides having the ability to generate sentences from random noises, we can also generate  sentences conditioning on a single topic of a certain layer, or a combination of topics from different layers.  

4) Larger-context language model: Transformer by itself is not a larger-context language model. While Transformer-XL improves Transformer to capture longer-range dependencies, it still does not respect the natural document boundary of the words. By contrast, rGBN-RNN does respect the word-sentence-document structure, using the deep dynamic topic model to guide the language model to  capture not only  the short-range local word dependencies, but also both the sequential dependencies between the document contexts of sentences, and the long-range document-level word dependencies. 

This paper looks quite interesting, but recent state-of-the-art results in language modeling and generation are largely based on Transformer-based models [1,2]. However, any comparison or even mention of these models seems to be conspicuously missing from this paper. I wonder: have the authors compared with any models? My suspicion is that these models are already able to capture topic to some extent, and may obviate the need for the methods proposed in this paper (but I would be happy to be proven wrong).

If not, but the authors would be interested in performing a comparison, I would suggest Transformer-XL [2], which as a model that is specifically designed to be able to capture long-distance context.

[1] Radford, Alec, et al. "Improving language understanding by generative pre-training." Preprint (2018).
[2] Dai, Zihang, et al. "Transformer-XL: Attentive language models beyond a fixed-length context." ACL (2019).

Missing references: 
[1] Hugo Larochelle and Stanislas Lauly. A neural autoregressive topic model. In NIPS 2012.
[2] Pankaj Gupta, Yatin Chaudhary, Florian Buettner, Hinrich Schütze. textTOvec: Deep Contextualized Neural Autoregressive Topic Models of Language with Distributed Compositional Prior. In ICLR 2019.

- include the neural network based topic models [1] in introduction  
- As mentioned in introduction section that traditional topic models ignore word ordering. The recent work [2] addresses the issue by introducing word ordering in topic models via composite modeling of a topic model and an LSTM based language model to deal with BoW issues in topic modeling. 
-  While this work focuses on improving LMs using topics, I would appreciate if you could also show quantitative results on topic modeling portion, such as topic coherence similar to Topic-RNN, TCNLM, etc.


Meta-review: This paper was a very difficult case. All three original reviewers of the paper had never published in the area, and all of them advocated for acceptance of the paper. I, on the other hand, am an expert in the area who has published many papers, and I thought that while the paper is well-written and experimental evaluation is not incorrect, the method was perhaps less relevant given current state-of-the-art models. In addition, the somewhat non-standard evaluation was perhaps causing this fact to be masked. I asked the original reviewers to consider my comments multiple times both during the rebuttal period and after, and unfortunately none of them replied.

Because of this, I elicited two additional reviews from people I knew were experts in the field. The reviews are below. I sent the PDF to the reviewers directly, and asked them to not look at the existing reviews (or my comments) when doing their review in order to make sure that they were making a fair assessment. 

Long story short, Reviewer 4 essentially agreed with my concerns and pointed out a few additional clarity issues. Reviewer 5 pointed out a number of clarity issues and was also concerned with the fact that d_j has access to all other sentences (including those following the current sentence). I know that at the end of Section 2 it is noted that at test time d_j only refers to previous sentences, but if so there is also a training-testing disconnect in model training, and it seems that this would hurt the model results.

Based on this, I have decided to favor the opinions of three experts (me and the two additional reviewers) over the opinions of the original three reviewers, and not recommend the paper for acceptance at this time. In order to improve the paper I would suggest the following (1) an acknowledgement of standard methods to incorporate context by processing sequences consisting of multiple sentences simultaneously, (2) a more thorough comparison with state-of-the-art models that consider cross-sentential context on standard datasets such as WikiText or PTB. I would encourage the authors to consider this as they revise their paper.

Finally, I would like to apologize to the authors that they did not get a chance to reply to the second set of reviews. As I noted above, I did try to make my best effort to encourage discussion during the rebuttal period.

Decision: reject