Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 1: Reject
review_content: [Additional review]
This paper proposes a technique to incorporate document-level topic model information into language models. 

While the underlying idea is interesting, my biggest issue is with the misleading assertions at the very beginning of the paper. In the second paragraph of Section 1, the paper claims that RNN-based LMs often make independence assumptions between sentences, hence why they develop a topic modelling approach to model document-level information. Some issues with this claim:

1. Pretty much every LM paper that evaluates on language modelling benchmark (PTB, WT-103, Wikitext-2) uses LSTMs/Transformers incorporate cross-sentential, document-level information as context, through a very simple approach of just concatenating all the sentences and adding a unique token to mark sentence boundaries.

2. Prior work has shown that LSTMs/Transformers with cross-sentential context can, and in fact do, make use of information from previous sentences.

a. Evidence 1: Khandelwal et al. (2018) showed that LSTMs memorise word orders from the past ~50 tokens, and retain semantic information from the past ~200 tokens; both of which extend far beyond the length of an average sentence, suggesting that information from the previous sentences is used in the predictions of the current sentence.

b. Evidence 2: Language models that operate on single sentences typically do worse than language models that take into account cross-sentential context, e.g. the language model of Kim et al. (2019) that operates on single sentences gets ~90 ppl. on PTB test set, while LSTMs that condition on multiple sentences get a much better ~50-something ppl. on Mikolov PTB.

Crucially, these prior works defeat the paper’s motivation of why it claims to need topic models in the first place (i.e. to model cross-sentential context), while just concatenating multiple sentences as context would do, and in fact has been done many times.

2. Prior work (mostly in Transformer-land) has come up with ways to make use of very long-range context, from Transformer-XL to the more recent compressive Transformer (https://openreview.net/forum?id=SylKikSYDH) that can condition on entire books. While these are done for Transformers, in principle one can also apply similar techniques to LSTMs.

3. While Transformer-XL has the potential to make use of word orders in the preceding sentences, it seems that this paper’s approach cannot do that, since they only take the bag-of-words from the preceding sentences. It thus seems that their bag-of-word approach is less expressive, and hence less powerful, than the simpler alternative of concatenating sentences.

4. The perplexity results (Table 1) are not done on very standard datasets (no PTB evaluation for instance). It is thus hard to evaluate the strength of the baseline models. In the paper's defense, it seems that they were following the experimental setup of Wang et al. (2019), but the paper should elaborate more on the choice of evaluation datasets.

5. The inference part is not particularly self-contained. The paper simply refers the TLASGR-MCMC method (which is an important part to make inference scalable) to prior work (Cong et al., 2017; Zhang et al., 2018), yet does not explain (even briefly) how the approach works, and how it can be combined with their recurrent topic model formulation.  

6. Evaluation of the induced topic hierarchy (Figure 4) is only done through qualitative samples, and the paper does not really explain how to pick the samples (i.e. possible cherry-picking). I am not very familiar with the topic modelling literature, but it would be nice if the induced hierarchy can be evaluated quantitatively.

References:
1. Urvashi Khandelwal, He He, Peng Qi, and Dan Jurafsky. Sharp nearby, fuzzy far away. In Proc. of ACL 2018.
2. Yoon Kim, Alexander Rush, Lei Yu, Adhiguna Kuncoro, Chris Dyer, and Gabor Melis. Unsupervised recurrent neural network grammars. In Proc. of NAACL 2019.
3. Wenlin Wang, Zhe Gan, Hongteng Xu, Ruiyi Zhang, Guoyin Wang, Dinghan Shen, Changyou Chen, and Lawrence Carin. Topic-guided variational autoencoders for text generation. In Proc. of NAACL 2019.
4. Yulai Cong, Bo Chen, Hongwei Liu, and Mingyuan Zhou. Deep latent Dirichlet allocation with topic-layer-adaptive stochastic gradient Riemannian MCMC. In Proc. of ICML 2017
5. Hao Zhang, Bo Chen, Dandan Guo, and Mingyuan Zhou. WHAI: Weibull hybrid autoencoding inference for deep topic modeling. In Proc. of ICLR 2018
review_title: Official Blind Review #4

final_score: rating: 1: Reject
review_content: The model description is confusing and lots of statements are presented without appropriate or enough justification. For example, (1) in the last paragraph of page 2, they claimed that the language component is used in their model to capture syntactic information, which I do not feel comfortable to accept; (2) in the first paragraph of page 3, it says "we define d_j as the BoW vector summarizing only the preceding sentences", without further information, I have no idea what a BoW vector looks like or how it is constructed; (3) in the last paragraph of page 3, it says using Dirichlet priors to make "the latent representation more identifiable and interpretable, but also facilitates inference", which I really don't know what it means. There are a few more examples like these. 

More importantly, I think Eq. (5) is wrong, which makes me question their whole methodology. To be specific, in their definition, d_j refers to a summary of all the sentences other than s_j. That means, 
- for s_1, d_1 is defined on s_2, s_3, s_4, ..., s_J; and 
- for s_2, d_2 is defined on s_1, s_3, s_4, ..., s_J. 
In other words, there is a huge overlap between any two d_j and d_{j'}. Therefore, I am not sure the decomposition on the right hand side of equation 5 (particularly, the decomposition of p(d_j | ...) ) is valid. 

Although they have some interesting results and the lowest PPLx comparing to other models, I do not think this paper is ready to be accepted. 
review_title: Official Blind Review #5

final_score: rating: 8: Accept
review_content: This paper presents a method for natural language generation, using a language model, informed by a topic model. 
The topic model is a hierarchical recurrent topic model that attempts to extract document-level word concurrence patterns and topic weight vectors for sentences. 
The language model is a stacked RNN model, aiming to capture word sequential dependencies. 

The proposed method is a combination of two existing methods, i.e. gamma-belief networks  and stacked RNN, where the stacked RNN is improved with the information from recurrent gamma belief network. 

Overall, this is a well written paper, clearly presented, with certain novelties. The method is well formulated mathematically and evaluated experimentally. The results look interesting especially for capturing the long-range dependencies, as shown by the  BLEU scores. One suggestion is that the authors didn't include computational analysis about the complexity and loads of the proposed method as compared with the baseline methods. 
review_title: Official Blind Review #1

final_score: rating: 8: Accept
review_content: 
The paper  proposes deep recurrent topic model guided language modeling using a stacked RNN, and uses a novel variational recurrent inference network to learn the parameters.  The proposed model can capture the dependence across the sentences in language generation though the recurrent latent topics. Moreover, the deep rGBN architecture provides Gamma distributed topic topic weight vectors which can be associated with every layer of the stacked RNN generating the sentence. The parameters of both the hierarchical recurrent topic model and language model are learnt using a hybrid inference algorithm  combining  variational inference to estimate language model and inference network parameters and MCMC to infer rGBN parameters. The effectiveness of the proposed model on the language modeling task is demonstrated on 3 datasets using Perplexity and BLEU score. The paper also provides a visual representation of the topics and their temporal trajectories. 

The proposed model extends previous approaches on topic guided language modeling by using deep rGBN model. Though the novelty of the model is limited, learning and inference with the proposed model is non-trivial. Further, the paper show an improvement in performance on language modeling using the proposed approach over SOTA approaches, demonstrating the significance of the proposed approach.  

Though the paper is relatively well written, it would have been good to explain some points on architecture and inference. It would have been better to provide the rationale behind some architectural decisions like associating \theta^1 and g^1 as against g^3. Related to this, Figure 1 has a typo where \theta^2 is associated with g^3.  An explanation on combining all the  latent representation in the RNN model used for language modeling will be helpful, though this is motivated by previous approaches. A proper explanation the TLASGR-MCMC approach for sampling from the posterior of  rGBN parameters is missing in the main paper.  It would be good to provide some details of this in the main paper. 

Experimental section compares the proposed approach against many SOTA approaches for the language modeling task. It would have been good to provide a quantitive evaluation of the topic modeling task also  in addition to demonstrating them qualitatively. 
review_title: Official Blind Review #3

final_score: rating: 8: Accept
review_content: This paper presents rGBN-RNN, a model that integrates a hierarchical recurrent topic model with an RNN-based language model in order to incorporate global semantic information and improve capturing of inter-sentence relations. The proposed model improves in perplexity across the three tested datasets over state of the art models of comparable type, and follow-up analyses show strong performance in sentence and paragraph generation, as well as learning of sensible hierarchical topics.

Overall I think this is a clearly-written paper with a well-motivated and interesting model, strong results, and a good range of follow-up analyses. I think that it is a solid paper to accept for publication. 

Some areas for improvement:

It seems strange not to mention all of the recent high-profile work on LM-based pre-training, since my impression is that these models operate effectively with large multi-sentence contexts. Do models like BERT and GPT-2 fail to take into account inter-sentence relations, as the paper claims most LMs do? I would like to see more discussion of how this work fits with that.

I don't know that it makes sense to highlight as the contribution of this model that it can "simultaneously capture syntax and semantics". It's not clear to me that other language models fail to capture semantics (keeping in mind that semantics applies within a sentence and not just at a global level) -- rather, it seems that the strength of this model is in capturing semantic relations above the sentence level.  If this is correct, that should be expressed more precisely.

It's not clear to me what we learn from Figure 3. The claim is that "the color of the hidden states of the stacked RNN based language model at layer 1 changes quickly ... because lower layers are in charge of learning short-term dependencies", but looking at the higher layers I'm not seeing clear evidence of capturing of long-distance dependencies, or even clear capturing of syntactic constituents. The takeaways from that figure should be made clearer and should be sure to correspond to what we can actually confidently conclude from that analysis.

review_title: Official Blind Review #2

Meta-review: This paper was a very difficult case. All three original reviewers of the paper had never published in the area, and all of them advocated for acceptance of the paper. I, on the other hand, am an expert in the area who has published many papers, and I thought that while the paper is well-written and experimental evaluation is not incorrect, the method was perhaps less relevant given current state-of-the-art models. In addition, the somewhat non-standard evaluation was perhaps causing this fact to be masked. I asked the original reviewers to consider my comments multiple times both during the rebuttal period and after, and unfortunately none of them replied.

Because of this, I elicited two additional reviews from people I knew were experts in the field. The reviews are below. I sent the PDF to the reviewers directly, and asked them to not look at the existing reviews (or my comments) when doing their review in order to make sure that they were making a fair assessment. 

Long story short, Reviewer 4 essentially agreed with my concerns and pointed out a few additional clarity issues. Reviewer 5 pointed out a number of clarity issues and was also concerned with the fact that d_j has access to all other sentences (including those following the current sentence). I know that at the end of Section 2 it is noted that at test time d_j only refers to previous sentences, but if so there is also a training-testing disconnect in model training, and it seems that this would hurt the model results.

Based on this, I have decided to favor the opinions of three experts (me and the two additional reviewers) over the opinions of the original three reviewers, and not recommend the paper for acceptance at this time. In order to improve the paper I would suggest the following (1) an acknowledgement of standard methods to incorporate context by processing sequences consisting of multiple sentences simultaneously, (2) a more thorough comparison with state-of-the-art models that consider cross-sentential context on standard datasets such as WikiText or PTB. I would encourage the authors to consider this as they revise their paper.

Finally, I would like to apologize to the authors that they did not get a chance to reply to the second set of reviews. As I noted above, I did try to make my best effort to encourage discussion during the rebuttal period.
Decision: Reject