Recurrent Hierarchical Topic-Guided Neural Language Models
Authors: Dandan Guo
Bo Chen
Ruiying Lu
Mingyuan Zhou
Venue:  (2020)
Keywords: Bayesian deep learning
recurrent gamma belief net
larger-context language model
variational inference
sentence generation
paragraph generation

Abstract: To simultaneously capture syntax and semantics from a text corpus, we propose a new larger-context language model that extracts recurrent hierarchical semantic structure via a dynamic deep topic model to guide natural language generation. Moving beyond a conventional language model that ignores long-range word dependencies and sentence order, the proposed model captures not only intra-sentence word dependencies, but also temporal transitions between sentences and inter-sentence topic dependences. For inference, we develop a hybrid of stochastic-gradient MCMC and recurrent autoencoding variational Bayes. Experimental results on a variety of real-world text corpora demonstrate that the proposed model not only outperforms state-of-the-art larger-context language models, but also learns interpretable recurrent multilayer topics and generates diverse sentences and paragraphs that are syntactically correct and semantically coherent.

Under review as a conference paper at ICLR 2020
RECURRENT HIERARCHICAL TOPIC-GUIDED NEURAL LANGUAGE MODELS
Anonymous authors Paper under double-blind review
ABSTRACT
To simultaneously capture syntax and global semantics from text corpus, pro- pose new larger-context recurrent neural network (RNN) based language model, which extracts recurrent hierarchical semantic structure via a dynamic deep topic model to guide natural language generation. Moving beyond conventional RNN based language model that ignores long-range word dependencies and sentence order, the proposed model captures not only intra-sentence word dependencies, but also temporal transitions between sentences and inter-sentence topic dependences. For inference, develop hybrid of stochastic-gratee gransiitt MCMC and recurrent autoencoding variational Bayes. Experimental results on a variety of real-world text corpora demonstrate that the proposed model not only outperforms of- the-art larger-context RNN-based language models, but also learns interpretable recurrent multilayer topies and generates diverse sentences and paragraphs that are syntactically correct and semantically coherent.
INTRODUCTION
Both topic and language models are widely used for text analysis. Topic models, such as latent Dirichlet allocation (LDA) (Blei et al. 2003) Griffiths & Steyvers 2004 Hoffman al. (013)) and its nonparametric Bayesian generalizations (Teh et [206]) Zhou & Carin 2013), are well suited extract document-level word concurrence patterns into latent topies from text corpus. Their modeling power has been further enhanced by introducing multilayer deep representation Srivasttaal al. 2013 Mnih Gregor 2014 Gan al., 20155 Zhou et al.) [016 Zhao al.) [0018] Zhang et al. 2018). While having semantically meaningful latent representation, they typically treat each document as a bag of words (BoW), ignoring word order (Griffiths et al. 2004, Wallach, 2006).
Language models have become key components of various natural language processing (NLP) tasks. such as text summarization (Rush al. 2015 Gehrmann et a 20188. speech recognition (Mikolov
al. 2010: Graves et al. 2013), machine translation (Sutskever et al., 2014 Cho et al. 20144), and
image captioning (Vinyals et al. 2015 Mao et al. 2015] Xu et al. 2015] Gan et al. 2017: Rennie
Gan
et al. 2017). The primary purpose of language model is to capture the distribution of word
sequence, commonly with a recurrent neural network (RNN) (Mikolov et al. 2011 Graves 2013
Transformer based neural network (Vaswani al., 2017 Dai al. 2019 Devlin et al. 2019
Radford et al.) 2018 2019). 9). In this paper, we focus on improving RNN-based language models that often have much fewer parameters and are easier to perform end-to-end training.
While RNN-based language models do not ignore word order, they often assume that the sentences of a document are independent each other. This simplifies the modeling task independently assigning probabilities to individual sentences, ignoring their orders and document context (Tian &
Cho 2016). Such language models may consequently fail capture the long-range dependencies
and global semantic meaning of a document (Dieng et al., 2017]] |aan et al. 2018). To relax the sentence independence assumption in language modeling, Tian (2016) propose larger-context language models that model the context of a sentence by representing its preceding sentences as either a single or a sequence of BoW vectors, which are then fed directly into the sentence modeling RNN. An alternative approach attracting significant recent interest is leveraging topic models to improve RNN-based language models. Mikolov & Zweig (2012) use pre-trained topic model features additional input to the RNN hidden states and/or output. Dieng al. (2017) 2001); Ahn et al. (2017) combine the predicted word distributions, given by both a topic model and a language model, under variational autoencoder (Kingma & Welling 2013). Lau et al. (2017) introduce attention based

---

Under review as a conference paper at ICLR 2020
convolutional neural network extract semantic topics, which are used to extend the RNN cell. Wang et al. (218)) learn the global semantic coherence of a document via a neural topic model and use the learned latent topics to build cexttur--of-epeee language model. [aang et al. (2019) further specify a Gaussian mixture model as the prior of the latent code in variational autoencoder, where each mixture component corresponds to topic.
While clearly improving the performance of the end task, these existing topic-guided methods still have clear limitations. For example, they only utilize shallow topic models with only single stochastic hidden layer in their data generation process. Note several neural topic models use deep neural networks to construct their variational encoders, but still use shallow generative models (decoders) (Miao et al.) 2017 Srivastava & Sutton, 2017). Another key limitation lies in ignoring the sentence order, as they treat each document as a bag of sentences. Thus once the topie weight vector learned from the document context is given, the task is often reduced to independently assigning probabilities to individual sentences (Lau et al. 2017,
Wang et al. 2018 2019).
In this paper, as depicted in Fig. [] we propose to use recurrent gamma belief network (rGBN) to guide a stacked RNN for language modeling. We refer to the model as IGBN-RNN, which integrates rGBN (Guo et al. 2018), a deep recurrent topic model, and stacked RNN (Graves) 2013 Chung et al. 2017) a neural language model, into a novel larger-context learns deep recurrent topic model, extracting document- level multi-layer word concurrence patterns and sequential topic weight vectors for sentences, and an expressive language model, capturing both short- and long-range word sequential dependencies. For inference, we equip rGBN-RNN (decoder) with a novel variational recurrent inference network (encoder), and train it end-to-end by maximizing the evidence lower bound (ELBO). Different from the stacked RNN based language model in Chung et al. (2017). which relies on three types of customized training operations (UPDATE, COPY, FLUSH) to extract multi-scale structures, the language model in TGBN-RNN learns such structures purely under the guidance of the temporally and hierarchically connected stochastic layers of rGBN. The effectiveness of rGBN-RNN as a new larger-context language model is demonstrated both quantitatively, with perplexity and BLEU scores, and qualitatively, with interpretable latent structures and randomly generated sentences and paragraphs. Notably, rGBN- ENN can generate a paragraph consisting of a sequence of semantically coherent sentences.
based language model. It simultaneously
2 RECURRENT HIERARCHICAL TOPIC-GUIDEI LANGUAGE MODEL
Denote a document of J sentences as D = S2.... SJ), where Sy = (gi.1.... BY.T,) consists Tf words from vocabulary of size V. Conventional statistical language models often only focus on the word sequence within sentence. Assuming that the sentences of document are independent to each other, they often define P(D) I2== P (S,) IT'= I p (y5.: Vy.<<)) p (V5,)). RNN based neural language models define the conditional probability of each word My.e given all the previous words yj.<t within the sentence S3. through the softmax function of a hidden state hj,, as
(ya V),<x)
p ( hj.r): hj.t f (h3 <t, 1))
(1)
where (() is non-linear function typically defined as an RNN cell, such as long short-term memory (LSTM) (Hochreiter & Schmidhuber 1997) and gated recurrent unit (GRU) (Cho al. 2014).
These RNN-based statistical language models are typically applied only at the word level, without exploiting the document context, and hence often fail to capture long-range dependencies. While
Dieng et al. (217)): ) Lau et al. (2017); Wang et al.) (2018) 2019) remedy the issue by guiding the language model with topic model, they still treat a document as bag of sentences, ignoring the order of sentences, and lack the ability to extract hierarchical and recurrent topic structures.
We introduce IGBN-RNN, as depicted in Fig, [a), as a new larger-context language model. consists of two key components: (i) a hierarchical recurrent topic model (rGBN), and (ii) a stacked RNN based language model. We use rGBN to capture both global semantics across documents and range inter-sentence dependencies within a document, and use the language model to learn the local syntactic relationships between the words within a sentence. Similar to Lau et al. (2017) Wang et al. (2018), we represent document
sequence of sentence-context pairs as where d, E ZY: summarizes the document excluding Sj, specifically (1S1, d ... ..... (S,, d Sy). into BoW count vector, with V, as the size of the vocabulary excluding (S,....
stop words. Note naive way is to treat each sentence as document, use a dynamic topic model
2

---

Under review as a conference paper at ICLR 2020
Language Model
Language Model
0
9
Language
Language Model
LSTM
RNN
RNNN
LSTM werd
RNN
RNNE
LSTM
word
RNN
RNNN.
1-1
Word embedding
d
1-1:T
(a)
(c)
(a) The generative model of three-hidden-layy iGBN-RNN, where the bottom part is the deep recurrent topic model (rGBN), document contexts of consecutive sentences used as observed data, and upper the language model. (b) Overview of the language model component, where input 2,,c denotes the rth word in jth sentence of document. j,t Mj,4-1, 1. ft,, is the hidden state of the stacked RNN at time step I, and 0j is the topic weight vector sentence at layer I. (c) The overall architecture of the proposed model, including the decoder (rGBN and language model) and encoder (variational recurrent inference), where the red arrows denote the inference latent topic weight vectors, black ones the data generation.
Figure 1:
(Blei & Lafferty 2006) to capture the temporal dependencies of latent topic-weight vectors, which is fed to the RNN to model the word sequence of the corresponding sentence. However, the sentences are often too short to be well modeled by a topic model. In our setting, as d, summarizes the document-leve' context of Sj. it is in general sufficiently long for topic modeling. Note during testing, we redefine dy as the BoW vector summarizing only the preceding sentences, i.e., S1.5-1, which will be further clarified when presenting experimental results.
2.1 HIERARCHICAL RECURRENT TOPIC MODEL
Shown in Fig. (a), to model the time-varying sentence-contex count vectors dy in document D, the generative process of the rGBN component, from the top to bottom hidden layers, is expressed as
0. ~ Gam (IT- J-11 TO) 0¹ Gam (Φ ( + II'O' J-1: T)) ......
0 Gam (x'0² + -I 11 TO). d, Pois (Ø'0)),
where 8¹ R denotes the gamma distributed topic weight vectors of sentence j at layer 4, II's R Ki KixKi the transition matrix of layer that captures cross-topic temporal dependencies, Φ R - xK, the loading matrix at layer I, Kj the number of topics of layer 1, and To R4 scaling and of Gam (M,70), where y 1x,- Finally, Dirichlet priors are placed on the columns of II' and क' i.e., т/. and di which not only makes the latent representation more identifiable and interpretable, but also facilitates inference. The count vector d, can be factorized into the product of $ and 0} under the Poisson likelihood. The shape parameters of 0' E RK can be factorized into the sum of 1 capturing inter-layer hierarchical dependence, and IT'e) 1 capturing intra-layer temporal dependence. rGBN not only captures the document-level word occurrence patterns inside the training text corpus, but also the sequential dependencies of the sentences inside a document. Note ignoring the recurrent structure, rGBN will reduce to the gamma belief network (GBN) of Zhou et al. (2016), which can be considered as muulti-stochastic-layee deep generalization LDA (Cong et al. 2017a) If igmoring its hierarchical structure (i.e., L = 1), rGBN reduces to Poisson gannnaggaaaaaaam dynamical systems (Schein ctt al. 2016). We refer to the IGBN-RNN without its recurrent structure as GBN-RNN, which no longer models sequential sentence dependencies; Appendix A for more details.
hyperparameter. At 1, 0 Gam (#/+10/+1 To) for I 1,...,

---

Under review as a conference paper at ICLR 2020
2.2 LANGUAGE MODEL
Different from conventional RNN-based language model, which predicts the next word only using the preceding words within the sentence, we integrate the hierarchical recurrent topic weight vectors 0 into the language model to predict the word sequence in the jth sentence. Our proposed language model is built upon the stacked RNN proposed in Grrvevs (2013); Chung ct al. (2017). but with the help of rGBN, it no longer requires specialized training heuristics to extract multi-scale structures. As shown in Fig. (b), to generate Uj.t, the 8 token of sentence j in a document, we construct the hidden states j,r of the language model, from the bottom to top layers, as
word (h 1- W. (3),)), LSTM word (h),-11. a,
LSTM
ifl=1. = 1, if L 1L21>1
(3)
where LSTM denotes the word-level LSTM at layer I. We E RV are word embeddings to be learned, and 2₁, Uj.t-1. Note a'S denotes the coupling vector, which combines the temporal topic weight vectors 0) and hidden output of the word-level LSTM h each time step t.
Following Lau et al. 2017), a gating unit similar to GRU (Cho et al. 2014 combines 6 of sentence j with its hidden state hi,e of word-level LSTM at layer 7 and time I as a², = g' (h) 0¹) We defer the details on g' to Appendix B Denote W. as a weight matrix with V rows and a as the concatenation of ji, across all layers; different from T the conditional probability of Uje becomes
p (yg.e Uj.4e, 0)) softmax (W 1:4)
(4)
There are two main reasons for combining all the latent representations a First, the latent representations exhibit different statistical properties at different stochastic layers of IGBN-RNN, and hence are combined together to enhance their representation power. Second, having "skip connections" from all hidden layers to the output one makes it casier to train the proposed network, reducing the number of processing steps between the bottom of the network and the top and hence mitigating the "vanishing gradient" problem (Graves) 2013).
I:J. for language modeling.
To sum up, depicted in Fig. (a). the topic weight vector 6) of sentence j quantifies the topic usage of its document context d, at layer I. It is further used as an additional feature the language model to guide the word generation inside sentence as shown in Fig. (b). It is clear that rGBN-RNN has two temporal structures: a deep recurrent topic model to extract the temporal topic weight vectors from the sequential document contexts, and language model estimate the probability of each sentence given its corresponding hierarchical topic weight vector. Characterizing the word-sentence- document hierarchy to incorporate both intra- and inter-sentence information, rGBN-RNN learns more coherent and interpretable topics and increases the generative power of the language model. Distinct from existing topic-guided guided language models, the temporally related hierarchical topics of rGBN exhibit different statistical properties across layers, which better guides language model to improve its language generation ability.
2.3 MODEL LIKELIHOOD AND INFERENCE
For rGBN-RNN, given (Φ. )T') the marginal likelihood of the sequence of sentence-context pairs (181, d1) .... ( d of document D is defined as
P ( I = /II'' (d, x'00)
[ (wy Lusiee, 9 [I' (0) le', mo)] do0.. (5)
1-1- where ef := 10+10/t11111 1 The inference task to learn the parameters of both the topic model and language model components. One naive solution is to alternate the training between these two components in each iteration: First, the topic model is trained using a sampling based iterative algorithm provided in Guo et al. (2018); Second, the language model is trained with maximum likelihood estimation under a standard cross-entropy loss. While this naive solution can utilize readily available inference algorithms for both rGBN and the language model, it may suffer from stability and convergence issues. Moreover, the need to perform a sampling based iterative algorithm for rGBN inside each iteration limits the scalability of the model for both training and testing.
4

---

Under review as a conference paper at ICLR 2020
Algorithm 1 Hybrid SG-MCMC and recurrent autocncoding variational inference for rGBN-RNN.
Set mini-batch size and the number of layer L Initialize encoder and neural language model parameter parameter 52, and topic model parameter (Φ', II'I HE= for iter 1, 2, ..... do
IT')
Randomly select a mini-batch of m documents consisting of J sentences to form a subset X (d:; $6,1-J }
Draw random noise { m,J.L
J ا-1يز=ي,= from uniform distribution;
Calculate V. (52, 4 IT X, 5.)) according to @ and update Q2 Sample f from 7 and X via R
to update (IT') [== and Φ " Ji= will described Appendix
end for
To this end, we introduce a variational recurrent inference network (encoder) to learn the latent temporal topic weight vectors 01:-. Denoting Q = II II {(0) dgy), , the ELBO of the log marginal likelihood shown in ( can be constructed as
L= Σラ Bq Iup (d, 4 +'0)) + E/ lup (u5, Ma.er. (..)..))))) 0/'0]] Ej Eg
In
2 ((
(6)
le).mo)
which unites both the terms that primarily responsible for training the recurrent hierarchical topic model component, and terms for training the neural language model component. Similar to Zhang al. (2018), we define q(0), dgg) Weibull(kk, (K), N3), random sample from which can be
obtained by transforming standard uniform variables e). as
03 (- In(1 et)) 1/6)
To capture the temporal dependencies between the topic weight vectors, both kjj and X from the bottom top layers, can be expressed
hz RNN! sent (hi-1: hz kg = Jh (h5g),
(8)
where h2 d,, het O, RNN! sent denotes the sentence-level recurrent encoder at layer / implemented with basic RNN cell, capturing the sequential relationship between sentences within document, hi denotes the hidden state of RNN sent and superscript in h s. denotes "sentence-
level RNN" used to distinguish the hidden state of language model ( Note both fk and fi are nonlinear functions mapping state h; to the parameters of 0), implemented with f(xx) We x + b)).
Rather than finding a point estimate of the global parameters (Φ' 1 of the rGBN, adopt hybrid inference algorithm by combining TLASGR-MCME described in Cong et al. (2017a); Zhang al. (2018) and our proposed recurrent variational inference network. In other words, the global parameters x II' | can be sampled with TLASGR-
while the parameters of the language model and variational recurrent inference network, denoted by Ω can updated via stochastic gradient descent (SGD) by maximizing the ELBO in 6 We describe hybrid variational/sampling inference for rGBN-RNN in Algorithm I and provide more details about sampling {Øb, 1 with TLASGR-MCMC in Appendix C
We defer the details on model complexity to Appendix
To sum up, as shown in Fig. [c). the proposed rGBN-RNN works with a recurrent variational autoencoder inference framework, which takes the document context of the jth sentence within document as input and learns hierarchical topic weight vectors 0):² that evolve sequentially with j. The learned topic vectors in different layer then used to reconstruct the document context input and as an additional feature for the language model to generate the jth sentence.
EXPERIMENTAL RESULTS
We consider three publicly available corpora, including APNEWS, IMDB, and BNC. The links, preprocessing steps, and summary statistics for them are deferred to Appendix D We consider recurrent variational inference network for rGBN-RNN to infer 8¹, as shown in Fig. Tc), whose number of hidden units in ( set the same the number of topics in the corresponding layer.
5

---

Under review as a conference paper at ICLR 2020
Following Lau et al. 2017). word embeddings are pre-trained 300-dimension word2vec Google News vectors lhttps://coe, www.gg/ccoom/cchh///////222222/
Dropout with a rate of 0.4 is used to the input of the -N at each layer, i.c., dj, or We [ in (3). The gradients are clipped if the norm of the parameter vector exceeds 5. We use the Adam optimizer (Kingma & Ba 2015) with learning rate 10-3. The length of input sentence is fixed to 30. We set the mini-batch size as 8, number of training epochs as 5, and scaling hyperparameter TO as 1. Code in TensorFlow is provided.
3.1 QUANTITATIVE COMPARISON
Perplexity: For fair comparison, we standard language model perplexity as the evaluation met- ric. by considering the following baselines: (i) a standard LSTM language model (Hochreiter & Schmidhuber 1997): (ii) LCLM (Tian & Cho 2016), a larger-context language model that incorpo- rates context from preceding sentences, which are treated as a bag of words; (iii) a standard LSTM language model incorporating the topic information of a separately trained LDA (LDA+LSTM); (iv) Topic-RNN (Dieng et al. 2017), hybrid model rescoring the prediction of the next word by incorporating the topic information through linear transformation; (v) TDLM (Lau al., 2017). joint learning framework which learns convolutional based topic model and a language model simultaneously. (vi) TCNLM (Wang et al., 2018), which extracts the global semantic coherence of a document via a neural topic model, with the probability of each learned latent topic further adopted to build a mixture-of-expert language model; (vii) TGVAE (Wang et al. 2019), combining a variational auto-encoder based neural sequence model with neural topic model; (vii) GBN-RNN, simplified IGBN-RNN that removes the recurrent structure of its rGBN component.
For rGBN-RNN, to ensure the information about the words in the jth sentence to be predicted is not leaking through the sequential document context vectors at the testing stage, the input d, in ( only
summarizes the preceding sentences Szj. For GBN-RNN, following TDLM (Lau al. 2017) and
TCNLM (Wang et al., 2018), all the sentences in a document, excluding the one being predicted, are used obtain the BoW document context. As shown in Table rGBN-RNN outperforms all baselines, and the trend of improvement continues as its number of layers increases, indicating the effectiveness of assimilating recurrent hierarchical topic information. IGBN-RNI consistently outperforms GBN-RNN, suggesting the benefits of exploiting the sequential dependencies of the centence-context for language modeling. Moreover, comparing Table Jand Table of Appendix F suggests rGBN-RNN, with its hierarchical and temporal topical guidance, achieves better performance with fewer parameters than comparable RNN-based baselines.
Note that for language modeling, there has been significant recent interest in replacing RNNs with Transformer (Vaswani et al. 2017), which consists of stacked multi-head self-attention modules, and its variants (Dai et al.] 2019 Devlin et al. 2019 Radford et al.) 2018 2019). While Transformer based language models have been shown to be powerful in various natural language processing tasks, they often have significantly more parameters, require much more training data, and take much longer to train than RNN-based language models. For example, Transformer- XL with 12L. and that with 24L (Dai et al., ).19) which improve Transformer capture longer-range dependencies, have 41M and 277M parameters, respectively, while the proposed rGBN-RNN with three stochastic hidden layers has as few as 7.3M parameters, as shown in Table 4 when used for language modeling. From a structural point-of-view, we consider the proposed IGBN-RNN as complementary to rather than competing with Transformer based language models, and consider replacing RNN with Transformer to construct rGBN guided Transformer as promising future extension.
BLEU: Following Wang et al.) (2019), we use test-BLEU to evaluate the quality of generated sentences with set of real test sentences as the reference, and Belf-BLEU to evaluate the diversity of the generated sentences (Zhu et al. 2018). Given the global parameters of the deep recurrent topic model (rGBN) and language model, we can generate the sentences by following the data generation process of rGBN-RNN: we first generate topic weight vectors 0f randomly and then downward
propagate it through the rGBN as in (2 to generate weight vectors with the hidden states of the language model in each layer depicted in (3), we generate corresponding sentence, where we start from zero hidden state at each layer in the language model, and sample words sequentially until the www.oof--setetneeeee symbol is generated. Comparisons of the BLEU scores between different methods are shown in Fig. 2 using the benchmark tool in Texygen (Zhu et al., (2188): We show below BLEU-3 and BLEU-4 for BNC and defer the analogous plots for IMDB and APNEWS to Appendix G and H Note we the validation dataset as the ground-truth.
By assimilating the random draw topic

---

Under review as a conference paper at ICLR 2020
Table I: Comparison of perplexity on three different datasets.
Perplexity
Model
LSTM Size
Topic Size
APNEWS
IMDB
BNC
LCLM (Tan & Cho
2016
600
54.18
67.78
96.50
900-900
50.63
67.86
87.77
LDA+LSTM
Lau
2017
600
100
55.52
69.64
96.50
900-900
100
50.75
87.77
TopicRNN
Dieng et
600
100
54.54
67.83
93.57
900-900
100
50.24
61.39
84.62
TDLM Lau et al
2017
900-900
100
48.97
59.04
81.83
TCNLM Wang et al.
2018
600
100
52.63
62.64
86.44
900-900
100
80.14
TGVAE Wang al
2019
600
50
48.73
57.11
87,86
600
64.13
72.14
102.89
basic-LSTM (Hochreiter Schmidhuber
1997
900-900
58.89
66.47
94.23
900 900 900
60.13
65.16
95.73
600
100
47.42
57.01
86.39
GBN RNN
600-512
100-80
44.64
55.42
82.95
600-512-256
100-80-50
44.35
54.53
80.25
600
81.94
IGBN-RNN
600-512
100-80
43.26
53.82
80.25
600-512-256
100-80-50
42.71
51.36
79.13
0.35
6LSTMM
0LSTM:
0.30
0.16
0.14
0.25
0.12
RIG
SIGENRAN-
SIGEN RAN-
0.20
6GIN-NNS2
0.10
RNN-J
GBN-RNN
RGNN-NNN
GEN
0.08
GBN-RNN
#TDLM
0.06
6TDLM
0.30
0:35
0.50
0.55 0.50
0.20
0.25
0.30
IaI-BEEU
tettBLEU
(a) BLEU-3
(b) BLEU-4
Figure 2: BLEU scores different methods for BNC. x-axis denotes lest-BLEU, y-axis self-BLEU, and better BLEU score would fall within the lower right comer.
g'oup (x)
hige
money
10
10
SEVVEE umolyys interrdone usicn abled
C7M
STOMD
meedh
[b]
ndiaa
vina
makro
hage
emounts
money
Figure 3: Visualization of Lz-norm of hidden states the language model of MTGN--NNNN shown in the top-row, and that of GBN-RNN, shown in the bottom row.
For all datasets, it is clear that iGBN-RNN yields both higher test-BLEU and lower self-BLEU scores than related methods do, indicating the stacked-RAl -NNN based language model IGBN-RNN generalizes well and does not suffer from mode collapse (i.e., low diversity).
3.2 QUALITATIVE ANALYSIS
Hierarchical structure of language model: In Fig. 3 we visualize the hierarchical multi-scale structures learned with the language model of IGBN-RNN and that of GBN- by visualizing the L2-norm of the hidden states in each layer, while reading sentence from the APNEWS validation set as "the service employee international union asked why ome group needs tax relief when making huge amounts of money?" As shown in Fig. Aa), in the bottom hidden layer (h1), the L2 norm sequence varies quickly from word to word, except within short phrases such "service employee", "international union." and "tax relief," suggesting layer hl is in charge of capturing short-term local dependencies. By contrast, in the top hidden layer (h3), the Lg norm sequence varies slowly and exhibits semantic/syntactic meaningful long segments, such as "service employee international union." "asked why eme group needs tax relief." "when is," and "making huge amounts of," suggesting that layer h3 in charge of capturing long-range dependencies. Therefore, the language model in
7

---

Under review as a conference paper at ICLR 2020
at budget lnommahera gor revenus princol community logisaatio
35 penerated sentence: the counci on group of
loaminng mundayy the assmmbly udded a propocalts balance the hurpptt medicad stan for raningme to 342 millon, Whotess yours to million...
democrati Looks proposed
public defferders, brandoon ballimore onder state and local officiss, have campaigned in channs defenntnng and speaking confident.
huture etste
spending association
62 budget gov. reverue vone conts mayor consercptive
57 generated sentance: nationali spedwaay vated to accept the proposal.
lwmmakers proposal lation credi: session meeting pumbling
prices inessnncoee treds otisa supply management
generated sentence:
san on thursday uod an officer
news refeose sising the two groupe will take more than 5 40,000 800 contacts
75 lewmalerr propocal log slation
62 Incgambting credzaveess meckeiilmmettmet
busget revence can goins
Bov. vote months consercation
treasury incentiees profits
celloteeforn fundraasing
lrwmakers would giv retirament payment.
the
75 generated sentence:
the gambling and vostng
62 generated sestence:
offieing x 22 million hond an S
generated
office north dalons has been
Ad penerated sentence:
the
mecoy demecratic
rfancenf thegame.
53 m ilionn budget with the
proseeds from Cacrow accours..
tire hall of year.
the rest
lecuse
40-57-42 peeerstedeenteen::
generated sentence:
the preposal would them pathwoy to cotzenship for the year before, , but they don't have a chance participataa the eiection..
the arkansa neas cenate has ourcussed 5 500 millon state bond a proposed medicald exparalio
shows the
other the repubriccnn
has been hit by number of
that rabed signiicant amounts of
shorncomings joddal burlget
sperding last yeur and unded with cuta from persitur gooer rmment churdoown.
proposal the past docade.
Topics and their temporal trajectories inferred by a three-hidden-layer rGBN-RNN from the APNEWS dataset, and the generated sentences under topic guidance (hest viewed in color). Top words of cach topic at layer 3, 2, 1 are shown in orange, yellow and blue boxes respectively, and each sentence is shown in a dotted line box labeled with the corresponding topic index. Sentences generated with a combination of topics in different layers at the bottom of the figure.
Figure 4:
GGBN-NNN can allow more specific information to transmit through lower layers, while allowing more general higher level information to transmit through higher layers. Our proposed model have the ability to learn hierarchical structure of the sequence, despite without designing the multiscale RNNs on purpose like Chenn et al. (2017). We also visualize the language model of GBN-RNN in Fig. bb:: with much less smoothly time-evolved deeper layers, GBN-RNN fails to utilize its stacked RNN structure as effectively as -GBN-RNN does. This suggests that the language model is much better trained in rGBN-RNN than in GBN-RNN for capturing long-range temporal dependencies, which helps explain why iGBN- exhibits clearly boosted BLEU scores in comparison to GBN-RNN.
Hierarchical topics: We present an example topic hierarchy inferred by a three-layer rGBN-RNN from APNEWS. In Fig. 4 we select a large- weighted topic at the top hidden layer and move down the network to include any lower-layer topics connected to their ancestors with sufficiently large weights. Horizontal arrows link temporally related topics at the same layer, while top-down arrows link hierarchically related topics across layers. For example, topic 48 of layer 3 on "budget, lawmakers, gov., revenue," is related not only in hierarchy to topic 57 on "lawmakers, pay, proposal, legislation" " and topic 60 of the lower layer on "budget, gov., revenue, vote, costs, mayor," but also in time to topic 35 of the same layer on "democratic, taxes, proposed, future, state." Highly interpretable hierarchical relationships between the topics different layers, and temporal relationships between the topics the same layer are captured by IGBN-RNN, and the topics are often quite specific semantically at the bottom layer while becoming increasingly more general when moving upwards.
Sentence generation under topic guidance: Given the learned rGBN-RNN, we can sample the sentences both conditioning on single topic of certain layer and on combination of the topics from different layers. Shown in the dotted- line boxes in Fig. most of the generated sentences conditioned on a single topic or combination of topics are highly related to the given topics in terms of their semantical meanings but not necessarily in key words, indicating the language model is successfully guided by the recurrent hierarchical topics. These observations suggest that rGBN-RNN has successfully captured syntax and global semantics simultancously for natural language generation.
Sentence/paragraphh generation conditioning on a paragraph: Given the GBN-RNN and rGBN- RNN learned on APNEWS, we further present the generated sentences conditioning as shown in Fig. 5 To randomly generate sentences, we encode the paragraph into a hierarchical latent representation and then feed it into the stacked-RNN, Besides, we can generate a paragraph with -NNN using its recurrent inference network to encode the paragraph into a dynamic hierarchical latent representation, which is fed into the language model to predict the word sequence
paragraph,
8

---

Under review as a conference paper at ICLR 2020
Document
the senate spansor (-.)), house commitien iten last week removed photo ids iscurd by publiic callegos and universities from the measure susan lyou, who said she agreed with the change. the house approved the bill 65-30 vote on monday evening. hut republican sem, bill lectoon in statement noted that the upper chamber overwhelm Ingly rejected eftorts take student ids out of the bill when passed 21-8 earlier this morth. ketrun said be would take the bill to conference commitlee if needed.
sponsored by republican
Generated Sentences
GBN-RNN if the house and senate agres, it will be the first time they 'Il have to seek their first meeting. the proposal would also give lawmakers with more money to protect public safety, he said..
Generated Sentences with rGBN-RNN
the proposal, which was introduced in the house on vote on wednesday, has already passed the senste floor to the house.
the city commission vated last week to approve the law, which would have allowed the council to approve the new bill.
Generated temporal Sentences with rCBN-RMN RNN (Paragraph)
senate president pro meazure version of the bill enact a senate committee to approve the emergency manager 's emergency license. the hause gave the bill to six weeks of testimany, but the vote now goes the full house for consideration. jaceson signed his paperwork wednesday with the lngisiature proposal would also give lawmakkers department of public safety says it has been selected for a special meeting for the state senate to investigate his proposed law a new state house committee has voted approve campsign also launched website at the university of california, irvine, which are studying the current proposal.
said the governar office has never resolved deadinn for
proposal
more money pratect public safety. he
"a spokesman
federal
let idaho wiiihh
national
ban private school systems public schools the
Examples of generated sentences and paragraph conditioned on document from APNEWS (green denotes novel words, blue the key words in document and generated sentences.)
Figure 5:
in each sentence of the input paragraph. It is clear that both the proposed GBN-RNN and IGBN-RNN can successfully capture the key textual information of the input paragraph, and generate diverse realistic sentences. Interestingly, the rGBN-RNN can generate semantically coherent paragraphs, incorporating contextual information both within and beyond the sentences. Note that with the topics that extract the document-level word cooccurrence patterns, our proposed models can generate semantically- muaningf words, which may not exist in the original document.
4 CONCLUSION
We propose recurrent gamma belief network (rGBN) guided neural language modeling framework. novel method to learn a language model and a deep recurrent topic model simultancously. For scalable inference, we develop hybrid SG-MCMC and recurrent autoencoding variational inference, allowing efficient end-to-end training. Experiments results conducted on real world corpora demonstrate that the proposed models outperform variety of wwwlow-toopi--mde--guueee and effectively generate the sentences from the designated multi-level topics or noise, while inferring interpretable hicrarchical latent topic structure of document and hierarchical multiscale structures of sequences. For future work, we plan to extend the proposed models to specific natural language processing tasks, such as machine translation, image paragraph captioning, and text summarization. Another promising extension is to replace the LaackedRNNN in rGBN-RNN with Transformer, i.c., constructing an rGBN guided Transformer as new larger-context neural language model.
neural language models,
REFERENCES
Sungjin Ahn, Heeyoul Choi, Tanel Parnamaa, and Yoshua Bengio. A neural knowledge language model. arXiv: Computation and Language, 2017.
D. M. Blei and D. Lafferty. Dynamic topic models. In ICML. 2006.
David M Blei, Andrew Ng, and Michael Jordan. Latent Dirichlet allocation. Journal of Machine Learning Research, 3:923-1022, 2003.
Kyunghyun Cho, Bart Van Merrienboer, Caglar Gulcehre, Fethi Bougares, and Yoshua Bengio. Learning phrase representations using RNN encoder-decoder for statistical machine translation. In Computer Science, 2014.
Junyoung Chung, Sungjin Ahn, and Yoshua Bengio. Hierarchical multiscale recurrent neural networks. In ICLR, 2017.
Yulai Cong, Bo Chen, Hongwei Liu, and Mingyuan Zhou. Deep latent Dirichlet allocation with topic-layer-adaptive stochastic gradient Riemannian MCMC. In ICMI., 2017a.

---

Under review as a conference paper at ICLR 2020
Yulai Cong, Bo Chen, and Mingyuan Zhou. Fast simulation of hyperplane-trumectt truncatte multivariate normal distributions. Bayesian Anal., 1244) 2017b.
BNC Consortium. The British National Corpus, version 3 (BNC XML Edition). http://wwe /ww
natcorp. ac .uk 2007.
Zihang Dai, Zhilin Yang, Yiming Yang, Jaime G Carbonell, Quoc V Le, and Ruslan Salakhutdinov. Transformer-xl: Attentive language models beyond a fixed-length context. ACL. 2019.
Jacob Devlin, Mingwei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. In north american chapter of the association for computational linguistics, pp. 4171-4188 2019.
Adji B Dieng, Chong Wang. Jianfeng Gao, and John Paisley. TopicRNN: A recurrent neural network with long-range semantic dependency. In ICLR, 2017.
Zhe Gan, Changyou Chen, Ricardo Henao, David Carlson, and Lawrence Carin. Scalable deep Poisson factor analysis for topic modeling. In ICML, pp. 1823-1832, 2015.
Zhe Gan, Chuang Gan, Xiaodong He, Yunchen Pu, Kenneth Tran, Jianfeng Gao, Lawrence Carin, and Li Deng. Semantic compositional networks for visual captioning. In CVPR, pp. 1141-1150, 2017.
Sebastian Gehrmann, Yuntian Deng, and Alexander Rush. Bottom-up abstractive summarization. In EMNLP, pp. 4098 4098-4109, 2018.
Mark A Girolami and Ben Calderhead. Riemann manifold Langevin and Hamiltonian Monte Carlo methods. Journal of The Royal Statisticul Society Series B-statistical Methodology, 1232)12 2011.
A. Graves, A. Mohamed, and G. Hinton. Speech recognition with deep recurrent neural networks. In ICASSP, pp. 6645-6649, 2013.
Alex Graves. Generating sequences with recurrent neural networks. arXiv: Neural and Evolutionary Computing, 2013.
T. L. Griffiths and M. Steyvers. Finding scientific topics. Proceedings of the National Academy of Sciences, 101:5228-5235, 2004.
Thomas L Griffiths, Mark Steyvers, David M Blei, and Joshua B Tenenbaum. Integrating topics and syntax. In NeurlPS, pp. 537-544, 2004.
Dandan Guo, Bo Chen, Hao Zhang, and Mingyuan Zhou. Deep Poisson gamma dynamical systems. In NeurlPS, pp. 8451-8461, 2018
Sepp Hochreiter and Jurgen Schmidhuber. Long short-term memory. Neural Computation, 9(8): 1735-1780, 1997.
Matthew D Hoffman, David M Blei, Chong Wang, and John Paisley. Stochastic variational inference. The Journal of Machine Learning Research, 1431)1133-1 2013.
Diederik Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In ICLR, 2015.
Diederik Kingma and Max Welling. Auto-encoding variational bayes. In ICLR, 2013.
D Klein and Christopher D Manning. Accurate unlexicalized parsing. In Meeting of the Association for Computational Linguistics. 2003.
Jey Han Lau, Timothy Baldwin, and Trevor Cohn. Topically driven neural language model. In meeting of the association for computational linguistics, pp. 355-365, 2017. 2017.
C. Li. C. Chen, D. Carlson, and L. Carin. Preconditioned stochastic gradient Langevin dynamics for deep neural networks. arXiv, 2015.
10

---

Under review as a conference paper at ICLR 2020
Y. Ma, Т. Chen, and E. Fox. A complete recipe for stochastic gradient MCMC. In NIPS, pp. 2899 1899-2907, 2015.
Andrew L. Maas, Raymond E. Daly, Peter T. Pham, Huang Dan, Andrew Y. Ng, and Christopher Potts. Learning Word Vectors for Sentiment Analysis. In Meeting of the Association for Computational Linguistics: Human Language Technologies, 2011.
Junhua Mao, Wei Xu, Yi Yang, Jiang Wang, Zhiheng Huang, and Alan L Yuille. Deep captioning with multimodal recurrent neural networks (m-RNN). In ICLR, 2015.
Yishu Miao, Edward Grefenstette, and Phil Blunsom. Discovering discrete latent topics with neural variational inference. In ICMI,, pp. 2410-2419,201
Tomas Mikolov and Geoffrey Zweig. Context dependent recurrent neural network language model. In SLT, pp. 234-239,22 2012.
Tomas Mikolov, Martin Karafiat, Lukas Burget, Jan Cernocky, and Sanjeev Khudanpur. Recurrent neural network based language model. In Interspeech, 2010.
Tomas Mikolov, Stefan Kombrink, Lukas Burget, Jan Cernocky, and Sanjeev Khudanpur. Extensions of recurrent neural network language model. In ICASSP, pp. 5528-5531, 2011.
Andriy Mnih and Karol Gregor. Neural variational inference and learning in belief networks. In ICML, pp. 1791-1799, 2014.
Sam Patterson and Yee Whye Teh. Stochastic gradient Riemannian Langevin dynamics on the probability simplex. In NIPS, pp. 3102-3110, 2013.
Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. Improving language under- standing by generative pre-training. 2018.
Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. Language models unsupervised multitask learners. 2019.
Steven Rennie, Etienne Marcheret, Youssef Mroueh, Jerret Ross, and Vaibhava Goel. Self-critical sequence training for image captioning. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 7008-7024, 2017.
Danilo Jimenez Rezende, Shakir Mohamed, and Daan Wierstra. Stochastic backpropagation and approximate inference in deep generative models. In ICML, pp. 1278-1286, 2014.
Alexander M Rush, Sumit Chopra, and Jason Weston. A neural attention model for abstractive sentence summarization. In EMNLP, pp. 379-389, 2015.
Aaron Schein, Hanna Wallach, and Mingyuan Zhou, Poisson-gammi gamma dynamical systems. Neural Information Processing Systems, 2016.
Akash Srivastava and Charles Sutton. Autoencoding variational inference for topic models. In ICLR, 2017.
Nitish Srivastava, Ruslan Salakhutdinov, and Geoffrey E Hinton. Modeling documents with deep Boltzmann machines. In Uncertainty in Artificial Intelligence, 2013.
Ilya Sutskever, Oriol Vinyals, and Quoc Le. Sequence to sequence learning with neural networks. In Advances in neural information processing systems, pp. 3104-311 3112, 2014.
Yee Whye Teh, Michael I Jordan, Matthew J Beal, and David M Blei. Hierarchical Dirichlet processes. Publications of the American Statistical Association, 101(4))) 1581, 2006.
Wang Tian and Kyunghyun Cho. Larger-context language modelling with recurrent neural network. In Meeting of the Association for Computational Linguistics, 2016.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Neural Information Processing Systems, pp. 6000-6010, 2017.
11

---

Under review as a conference paper at ICLR 2020
Oriol Vinyals, Alexander Toshev, Samy Bengio, and Dumitru Erhan. Show and tell: A neural image
caption generator. In CVPR, 2015.
Hanna M Wallach. Topic modeling: beyond bag-of-words. In ICML, pp. 977-984, 2006.
Wenlin Wang, Zhe Gan, Wenqi Wang, Dinghan Shen, Jiaji Huang, Wei Ping, Sanjeev Satheesh, and Lawrence Carin. Topic compositional neural language model. In A/STATS, pp. 356-365, 2018.
Wenlin Wang, Zhe Gan, Hongteng Xu, Ruiyi Zhang, Guoyin Wang, Dinghan Shen, Changyou Chen, and Lawrence Carin. sopic-guided variational autocncoders for text generation. In NAACL, 2019.
Max Welling and Yee Whye Teh. Bayesian learning via stochastic gradient Langevin dynamics. In ICML, pp. 681-688. 2011.
Kelvin Xu, Jimmy Ba, Ryan Kiros, Kyunghyun Cho, Aaron C Courville, Ruslan Salakhudinov, Rich Zemel, and Yoshua Bengio. Show, attend and tell: Neural image caption generation with visual attention. In ICML, pp. (048-2057, 2015.
Hao Zhang, Bo Chen, Dandan Guo, and Mingyuan Zhou. WHAL: Weibull hybrid autoencoding inference for deep topic modeling. ICLR, 2018.
He Zhao, Lan Du, Wray Buntine, and Mingyuan Zhou. Dirichlet belief networks for topic structure learning. In Neural Information Processing Systems, pp. 7955-7966, 2018.
Mingyuan Zhou and Lawrence Carin. Negative binomial process count and mixture modeling. IEEE Transactions on Pattern Analysis and Machine Intelligence, 37(2):307-320, 2015.
Mingyuan Zhou, Yulai Cong, and Bo Chen. Augmentable gamma belief networks. J. Mach. Learn Res., 17(163):1 2016.
Yaoming Zhu, Sidi Lu, Lei Zheng, Jiaxian Guo, Weinan Zhang, Jun Wang, and Yong Yu. Texygen: A benchmarking platform for text generation models. SIGIR, 2018.
12

---

Under review as a conference paper at ICLR 2020
A THE GBN-RNN
GBN-RNN: {Vi:r, d} denotes a sentence-context pair, where E ZY: represents the document-level context as a word frequency count vector, the uth element of which counts the number of times the wth word in the vocabulary appears in the document excluding sentence y1.r. The hierarchical model of L-hidden-layer GBN, from top to bottom, is expressed
0C Gam (r,c)))). ...... O' Gam ( 0 c(t)).....
0' Gam (x²0² .²), d~ Pois (D'0").
(9)
The stacked-RNN based language model described in is also used in GBN-RNN.
Statistical inference: To infer GBN- REN- we consider a hybrid of stochastic gradient MCMC
(Welling & Tch 2011 Patterson & Tch 2013 Li al. 2013 Ma et al. 22015 Cong et al. | (0)7a).
used for the GBN topics Øi, and auto-encoding variational inference Kingma & Welling 2013
Rezende et al. 2014), used for the parameters of both the inference network (encoder) and RNN
More specifically, GBN-RNN generalizes Weibull hybrid auto-encoding inference (WHAI) of Zhang al. (2018): it a deterministic-dowwwarrr wtwp:aaadd--tcaattt-uu---eeee
inference network encode the bag-of-words representation of d into the latent ropic-weight variables O' across all hidden layers, which are fed into not only GBN to reconstruct d. but also stacked RNN in language model, as shown in (3) to predict the word sequence in y1:r. The topics & can be sampled with topic-layer- adaptive stochastic gradient Riemannian (TLASGR) MCMC, whose details can be found in Cong et al. (2017a); Zhang et al. (201)). omitted here for brevity. Given the sampled topics & the joint marginal likelihood of (31.TT dj is defined as
L [o' gor dour. (10)
P (Vir,, (I / (d o'
ILe (v vha-1,0'h
For efficient inference, an inference network as Q II 8(0' d, 01+20(+1 is used provide ELBO of the log joint marginal likelihood as
L Eq
a(0' d d piol +1+10++1)
L(gET, d) = Eq Imp (d) a @'0') + Σ Inp (ve v1--1,
Σ
In
and the training is performed by maximizing (m d) [Ly:TT d)}; following Zhang et al. 2018).
we define q8') (d, 0'4 1 Weibull + $1+1g gl+1 X²). where both k'' and X are determin- istically transformed from d using neural networks. Distinct from usual variational auto-encoder whose inference network has a pure bottom-up structure, the inference network here has determistic- upward upward-stoachstic-e stwachstic-oowwweeeeee ladder structure (Zhang et al.) 2018).
B THE COUPLING VECTOR
Following Lau et al.
2017 7 , the a²,, g6 (h/ 8¹) can be implemented with gating unit similar to
a GRU (Cho et al 2014). describe as
f), (W. + U + bz)
r/c = (W'0¹ + UU bb,)
(12)
j.r tanh (W).0' -U, (r), @ oh/,)) + bb)) d/ (1 =1,0 2 n4e + 2/,0 @ oh/
C SGMCMC FOR RGBN-RNN
To allow for scalable inference, we apply the saric-layer-ddappae stochastic gradient Riemannian (TLASGR) MCMC algorithm described in Cong al. (2017a): Zhang et al. (2018), which can be used to sample constrained global parameters Cong et (2[17)) in a mini-batch based manner. It improves its sampling efficiency via the use of the Fisher information matrix (FIM)
13

---

Under review as a conference paper at ICLR 2020
Girolami & Calderhead (201T), with adaptive step-sizes for the latent factors and transition matrices of different layers. In this section, we discuss how to update the global parameters { II') T==1 of iGBN in detail and give complete one in Algorithm in I
Sample the auxiliary counts: This step is about the "backward" and "upward" pass. Let us denote
Z
(1.1) -k==1 Zkyky. Z 0, and "kj dej. where dj {dij, ... dej. dras) is shown
in
Working backward for j = J.... 1 and upward for I = 1,.... I., we draw
(Akagi Akko)) ~ Multi
(L.)) Tkj
(13)
ΣnI k==
Exi &aan kid
1+1 2kj
CRT
Alas Z'n.41. T0
Σ Enip
Ke+1
2+1
0kkes
0 ReL13
R Then O
(14)
1+1 will be influenced by the effects from Note that via the deep structure, the latent counts Økj
both time j- 1 at layer and time j at layer + With P1 == Ex Thke Oki-i and p2 ==
Emeni= 0khr,, K4415 we can sample the latent counts at layer and + b
(#k) 2kj
Multi (zit),pi) PI/(pr + p2))2,2////) + x22)
(15)
and then draw
0
Tkk K 1-1
(Zaagi.. 4kkis)
Multi
1+1.. Tkj
kkl
1.j 1
(16)
K1 =k==1
Tkk 0
h::j1
Kr Emz=1 Take
In rGBN, the prior and the likelihood of { |/ TLASGR MCMC sampling algorithm on both of them conditioned on the auxiliary counts.
,
is very similar with If= so we also apply the
Sample the hierarchical components { For Φi. the ktth column of the loading matrix d of layer 1, its sampling can be efficiently realized
(DA' +16) - +Ki (db)), En (o)) n+1 - (oh)
Pk
nnn (p)) [diag( Pk
+N
0,
[diag
(0h))
(17)
where P is calculated using the estimated FIM, Alyz- = Σ Alang Ale. = (Ai Ak. )r
and M.g. Eki -m Ah..- Al Ahiss comes from the augmented latent counts A in 13 v/ denote the prior of Φ%. and H2 denotes a simplex constraint.
Sample the transmission matrix {IT' For T/. the kth column of the transition matrix IT' of layer its sampling can be efficiently realized as
(TT)) n+1
(Th)
En M
- (p (o's (Tri)
n
2 [diag( Th)) in -0 (n (T/
N
(18)
M
where M calculated using the estimated FIM, kij- = ビ Zhe kjj 2/ = - 3 K.y. )r
and Z -t 3 Zhi comes from the augmented latent counts Z' in (T6) and [.12 denotes
a simplex constraint, and n'a denotes the prior of π(. more details about TLASGR-MCME for our proposed model can be found in Cong al. 2017a).
D DATASETS
We consider three publicly available corpora]] APNEWS is a collection of Associated Press news articles from 2009 to 2016, IMDB a set of movie reviews collected by Maas et al. (201 and BNC
hhtps://ihmmmco ent. http:://hhh.oo.ooccooooooooossst
14

---

Under review as a conference paper at ICLR 2020
is the written portion of the British National Corpus (Consortium 2007). Following the preprocessing steps in Lau et al. (2017), we tokenize words and sentences using Stanford CoreNLP (Klein & Manning, 2003),, lowercase all word tokens, and filter out word tokens that occur less than 10 times. For the topic model, we additionally exclude stopword and the top 0.1% most frequent words. All these corpora partitioned into training. validation, and testing sets, whose summary statistics provided in Table of the Appendix.
Table 2: Summary statistics for the datasets.
Vocubalry
Training
Validation
Testing
Dataset
LM
TM
Docs
Sents
Tokens
Does
Sents
Tokens
Does
Sents
Tokens
APNEWS
34231
32169
50K
0.8M
15M
2K
33K
0.6M
2K
32K
0.6M
IMDB
36009
34925
75K
LIM
20M
12.5K
0.18M
0.3M
12.5K
0.18M
0.3M
BNC
43703
41552
15K
IM
18M
1K
57K
1M
1K
66K
1M
E COMPLEXITY OF RGBN-RNN
The proposed iGBN-RNN consists of both language model and topic model components. For the
topic model component, there are the global parameters of rGBN (decoder), including (Ø', IT') [
in 2 and the parameters of the variational recurrent inference network (encoder), consisting of RNN
sent> fig. and f in ( The language model component is parameterized by LSTM in
and the coupling vectors o described in Appendix B We summarize in Table 3 the complexity of IGBN-RNN (ignoring all bias terms), where V denotes the vocabulary size of the language model, E the dimension of word embedding vectors, V. the size of the vocabulary of the topic model that
excludes stop words, H the number of hidden units of the word-level LSTM at layer / (stacked-RNN language model), HJ the number of hidden units of the sentence-level RNN at layer / (variational recurrent inference network), and K2 the number of topics at layer 1.
Table 4 further compares the number of parameters between various RNN-based language models. where we follow the convention to ignore the word embedding layers. Some models in Table [ are not included here, because we could not find sufficient information from their corresponding
papers or code to accurately calculate the number of model parameters. Note when used for language generation at the testing stage, -GBN-RNN no longer needs its topics (Φ'). J. whose parameters are
hence not counted. Note the number of parameters of the topic model component is often dominated by that of the language model component.
Table 3: Complexity of the three-layer rGBN-RNN.
Component
Langunge Model
Topic Model
Param
LSTM in
of
in 2
I'' in
RNN
8
1 in
in
Layeri
x
HPJ
O13 T
OVV x R.
O
Kul
OTV +
HD
O
OUR
Layer2
x1x (8)
O13 x [K2
O(K
K2
OiK
x Kaj
O( H. H3
OlK x H
Layer3
0 x (85
A 88
013 x (Ks
ME
O(Kg
Ks)
OlKs Ka)
R
O
OLK
Hi
Table 4: Comparison of the number of parameters of different models when used for language generation.
Model
LSTM Size
Topic Size
# LM Param
#TM Puram
AAll Param
TDLM Lau ct 2017
100
3.35M
0019M 0.01M
900-9000
100
13:38M
13.40M
600
2.16M
2.16M
basic-LSTM Hochreiter Schmidhuber
900-900
10.80M
900-900-900
17.68M
17.68M
600
100
3.40M
8O2M
342M
GBN-RNN
6001-512
100-80
6.50M
0.4M
6.54M
600-512-256
100-800-00
7,20M
0.05M
7.25M
600
100
3.40M
0.03
3,43
IGBN-RMN
600-512
100-80
6.50M
0.06M
6.56M
600-512-256
100-80-50
7.20M
0.07M
7.27M
We use Mallet's stopword list: hwwp://gmmikooomiiiiiaaaaiccccnr
15

---

Under review as a conference paper at ICLR 2020
F
MORE EXPERIMENTAL RESULTS ON IMDB AND BNC
13 Interesting actor hard poyed fun went producsion screen
hose that someone
make throug?
say actors and budget grectneis solandd movic.
penerated sentencee for studic
horror big oction auy minutas ending shows special enjoy
12 Eims kind wetee making lit found burgeet world
10 generated sentence:
the brosdway. shiging the acting. the utter lock of thrill refuses for you you man to make fan ofit.
place money whot video cossic hollywood
revelising horror reping
26 hurrer lig action shows unjoy left hope entertaining ill
policetells stage tamous shots coming leave american
place hollywoodd music song direction future iestttltteeess
hand
29 generated sentence: must say this very well constructed film.. and my nome, the musi, as also adventurn movie.
generated sentence:
26 generatedsentence::
generated sentence:
this moste horrible had
twing a christtn by one of my family...... i always enjoyeed watching the show a5 horror.
are consirced
mode overaal just rented
husband agrze to go to police scrdemy and sase the day.
and again!
12-1-3 sentence:
cducational sitcction for
13-10-20 29 generated sentinne: the story very Intoresting
13-10-23 generated sentence:
generated the acting homrible molodramatie.
dobotes to the
nothing this movic makes sonce the beginning and the script for this mavie ridculous.
and the movie well cratted cousle years and that S what it about.
attluddes eaisting
decidle.
Figure 6: Topics and their temporal trajectories inferred by a three-hiideen-lyer rGBN- RNN from the IMDB dataset, and the generated sentences under topic guidance (best viewed in color). Top words of each topic are shown orange, yellow and blue box, and each sentence is shown a dotted line box labeled with the corresponding topic index. Sentences generated with combination of topics in different layers are at the bottom of the figure.
socits members aducation forma appracca ouriic Aure qusationtre
the adkance the cducatioral
genersted swntense:
rovliindusryy erecrstary bus sharch agreements
surwey on ayweeeer highkgredd these prine pic and many other major
73 geverated sentence:
geeeertee sontensee good scample the poor the howe raing gowernere
78 assocation sodary Berolla chortten connibbloon onal
how we get to know what the reediu for the carlo to conader incons deringtoot our right to thaicel cructal catsome
edheatedieattonn wellers sortial young cuminu clas farma
included forowe setticn single unment percralmmeting
apprstes cuvetieraine cantoulasshhoss formal formal decitine toe ringpiing many
memterseduatiin wellers wallers oenrre misierrewen refore boneft
society sutootiiseernbbrtttt 5mm election voters professinoal viliaoo
ancecational Eeteriiration dota la dedisknn britsh, Fance
ganeerated sentence:
conditerig
whichdhe advir
generated sentence:
as the present dosrited therelormsa opportunts compe the soveriat eppeccite,
genecated sendance
the govermment was cand enterpp woreddin ih meeting with hancs
38 geneerted sentence
beer discted necent
the alamn aad entttted aulfacal poltearsto slow a lepil erceron
tam
15-87 generated sertanoc:
15-4-5 geserated senteace: the oocoaiational citatinn the mebotes, to righ to the imple ne caroratooolllo of the studen: at enitudes, and interett editingsecaa servicett
15-73-28 peseoted
the scholl was frund ima the property eechunges for fartor subuttene exange ooioonee dedection
second of the and
contented
creaped
cemonetrape that the practical
govermenn on the peril of the old janata alnce and ganuraly isen one- datt voveres whth eumpeon.
ststege the astodution is be
nepotintec by gareral sncretary tfeeeggeeeee..
pe
Topics and their temporal trajectories inferred by a three-hiddennllyer rGBN-RNN from the BNC dataset, and the generated sentences under topic guidance (hest viewed in color). Top words of each topic are shown in orange, yellow and blue box. and each sentence is shown in a dotted line box labeled with the corresponding topic index. Sentences generated with a combination of topics in different layers are at the bottom of the figure.
Figure 7:
16

---

Under review as a conference paper at ICLR 2020
G BLEU SCORES FOR IMDB
0.50
ISTM
0.275
0.250
0.40
0.35
GeGEN-RNN-I
BrGBN-RNN-1
0.30
AGDN-RNN2
GRV-RAN-I
RNN-2
0.150
GGEN-RNN-I
GHN--NNN-2
GFGIN-RNN-1
0.20
TOLM
CEN-RNN-3
Ground ruth
GEN-RNN-
0.100
0.125
GBN-RNNS @GGNN
Greand-Iruuh
0.55
0.60
0.
65
0,70 Lest BL--BL EU
0.75
0.80
0.85
0.3
0.4
0.5
0.5
0.7
lett-BEEU
Figure 8: BLEU scores of different methods for IMDB. x-axis denotes test lest-BLEU. and y axis Bff-BLEU. Left panel BLEU-3 and right BLEU--4 and better BI. BLEU score would fall within the lower right comer, where black point represents mean value and circles with different colors denote elliptical surface of prohability of BLEU in two-dimensional space.
H BLEU SCORES FOR APNEWS
LSTMS
LSTMS
0.5-
0.35
0.30
0.4
0.25
0.20
0.3
GBNN-NNN-
GBN RNN-1
GGBN-RNN-1
GBN-RNN-J
GUN RNN-2
GBIN RNN-3
GGENARNN.3
0.10
GBN-RNNS
Groand
(GBN-RNNA
TDLM
Ground-Truth
GBN-RNN3
0.44
0.46
0.50
0.52 0.54 0.56 0.58 0.60
0.200 0.225
0.250
0.275 0.300 0.325 0.350 BLEU
BeSt-BEEU
Figure 9: BLEU scores of different methods for APNEWS. x-axis denotes test-BLEU, and y-axis self-BLEU. Left panel is BLEU-3 and right is BLEU-4, and better BLEU score would fall within lower right corner, where black point represents mean value and circles with different colors denote the elliptical surface of probability of BLEU in a dimensionnl space.
17


Reviews:
[{"review_id": "rkxufFDxqS", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "\nThe paper  proposes deep recurrent topic model guided language modeling using a stacked RNN, and uses a novel variational recurrent inference network to learn the parameters.  The proposed model can capture the dependence across the sentences in language generation though the recurrent latent topics. Moreover, the deep rGBN architecture provides Gamma distributed topic topic weight vectors which can be associated with every layer of the stacked RNN generating the sentence. The parameters of both the hierarchical recurrent topic model and language model are learnt using a hybrid inference algorithm  combining  variational inference to estimate language model and inference network parameters and MCMC to infer rGBN parameters. The effectiveness of the proposed model on the language modeling task is demonstrated on 3 datasets using Perplexity and BLEU score. The paper also provides a visual representation of the topics and their temporal trajectories. \n\nThe proposed model extends previous approaches on topic guided language modeling by using deep rGBN model. Though the novelty of the model is limited, learning and inference with the proposed model is non-trivial. Further, the paper show an improvement in performance on language modeling using the proposed approach over SOTA approaches, demonstrating the significance of the proposed approach.  \n\nThough the paper is relatively well written, it would have been good to explain some points on architecture and inference. It would have been better to provide the rationale behind some architectural decisions like associating \\theta^1 and g^1 as against g^3. Related to this, Figure 1 has a typo where \\theta^2 is associated with g^3.  An explanation on combining all the  latent representation in the RNN model used for language modeling will be helpful, though this is motivated by previous approaches. A proper explanation the TLASGR-MCMC approach for sampling from the posterior of  rGBN parameters is missing in the main paper.  It would be good to provide some details of this in the main paper. \n\nExperimental section compares the proposed approach against many SOTA approaches for the language modeling task. It would have been good to provide a quantitive evaluation of the topic modeling task also  in addition to demonstrating them qualitatively. ", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "SJxPxG6G6H", "reviewer": "AnonReviewer5", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The model description is confusing and lots of statements are presented without appropriate or enough justification. For example, (1) in the last paragraph of page 2, they claimed that the language component is used in their model to capture syntactic information, which I do not feel comfortable to accept; (2) in the first paragraph of page 3, it says \"we define d_j as the BoW vector summarizing only the preceding sentences\", without further information, I have no idea what a BoW vector looks like or how it is constructed; (3) in the last paragraph of page 3, it says using Dirichlet priors to make \"the latent representation more identifiable and interpretable, but also facilitates inference\", which I really don't know what it means. There are a few more examples like these. \n\nMore importantly, I think Eq. (5) is wrong, which makes me question their whole methodology. To be specific, in their definition, d_j refers to a summary of all the sentences other than s_j. That means, \n- for s_1, d_1 is defined on s_2, s_3, s_4, ..., s_J; and \n- for s_2, d_2 is defined on s_1, s_3, s_4, ..., s_J. \nIn other words, there is a huge overlap between any two d_j and d_{j'}. Therefore, I am not sure the decomposition on the right hand side of equation 5 (particularly, the decomposition of p(d_j | ...) ) is valid. \n\nAlthough they have some interesting results and the lowest PPLx comparing to other models, I do not think this paper is ready to be accepted. ", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory."}, "extra_text": {}}, {"review_id": "HygW6Tz7ar", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "[Additional review]\nThis paper proposes a technique to incorporate document-level topic model information into language models. \n\nWhile the underlying idea is interesting, my biggest issue is with the misleading assertions at the very beginning of the paper. In the second paragraph of Section 1, the paper claims that RNN-based LMs often make independence assumptions between sentences, hence why they develop a topic modelling approach to model document-level information. Some issues with this claim:\n\n1. Pretty much every LM paper that evaluates on language modelling benchmark (PTB, WT-103, Wikitext-2) uses LSTMs/Transformers incorporate cross-sentential, document-level information as context, through a very simple approach of just concatenating all the sentences and adding a unique token to mark sentence boundaries.\n\n2. Prior work has shown that LSTMs/Transformers with cross-sentential context can, and in fact do, make use of information from previous sentences.\n\na. Evidence 1: Khandelwal et al. (2018) showed that LSTMs memorise word orders from the past ~50 tokens, and retain semantic information from the past ~200 tokens; both of which extend far beyond the length of an average sentence, suggesting that information from the previous sentences is used in the predictions of the current sentence.\n\nb. Evidence 2: Language models that operate on single sentences typically do worse than language models that take into account cross-sentential context, e.g. the language model of Kim et al. (2019) that operates on single sentences gets ~90 ppl. on PTB test set, while LSTMs that condition on multiple sentences get a much better ~50-something ppl. on Mikolov PTB.\n\nCrucially, these prior works defeat the paper’s motivation of why it claims to need topic models in the first place (i.e. to model cross-sentential context), while just concatenating multiple sentences as context would do, and in fact has been done many times.\n\n2. Prior work (mostly in Transformer-land) has come up with ways to make use of very long-range context, from Transformer-XL to the more recent compressive Transformer (https://openreview.net/forum?id=SylKikSYDH) that can condition on entire books. While these are done for Transformers, in principle one can also apply similar techniques to LSTMs.\n\n3. While Transformer-XL has the potential to make use of word orders in the preceding sentences, it seems that this paper’s approach cannot do that, since they only take the bag-of-words from the preceding sentences. It thus seems that their bag-of-word approach is less expressive, and hence less powerful, than the simpler alternative of concatenating sentences.\n\n4. The perplexity results (Table 1) are not done on very standard datasets (no PTB evaluation for instance). It is thus hard to evaluate the strength of the baseline models. In the paper's defense, it seems that they were following the experimental setup of Wang et al. (2019), but the paper should elaborate more on the choice of evaluation datasets.\n\n5. The inference part is not particularly self-contained. The paper simply refers the TLASGR-MCMC method (which is an important part to make inference scalable) to prior work (Cong et al., 2017; Zhang et al., 2018), yet does not explain (even briefly) how the approach works, and how it can be combined with their recurrent topic model formulation.  \n\n6. Evaluation of the induced topic hierarchy (Figure 4) is only done through qualitative samples, and the paper does not really explain how to pick the samples (i.e. possible cherry-picking). I am not very familiar with the topic modelling literature, but it would be nice if the induced hierarchy can be evaluated quantitatively.\n\nReferences:\n1. Urvashi Khandelwal, He He, Peng Qi, and Dan Jurafsky. Sharp nearby, fuzzy far away. In Proc. of ACL 2018.\n2. Yoon Kim, Alexander Rush, Lei Yu, Adhiguna Kuncoro, Chris Dyer, and Gabor Melis. Unsupervised recurrent neural network grammars. In Proc. of NAACL 2019.\n3. Wenlin Wang, Zhe Gan, Hongteng Xu, Ruiyi Zhang, Guoyin Wang, Dinghan Shen, Changyou Chen, and Lawrence Carin. Topic-guided variational autoencoders for text generation. In Proc. of NAACL 2019.\n4. Yulai Cong, Bo Chen, Hongwei Liu, and Mingyuan Zhou. Deep latent Dirichlet allocation with topic-layer-adaptive stochastic gradient Riemannian MCMC. In Proc. of ICML 2017\n5. Hao Zhang, Bo Chen, Dandan Guo, and Mingyuan Zhou. WHAI: Weibull hybrid autoencoding inference for deep topic modeling. In Proc. of ICLR 2018", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "H1xrSldCKr", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper presents rGBN-RNN, a model that integrates a hierarchical recurrent topic model with an RNN-based language model in order to incorporate global semantic information and improve capturing of inter-sentence relations. The proposed model improves in perplexity across the three tested datasets over state of the art models of comparable type, and follow-up analyses show strong performance in sentence and paragraph generation, as well as learning of sensible hierarchical topics.\n\nOverall I think this is a clearly-written paper with a well-motivated and interesting model, strong results, and a good range of follow-up analyses. I think that it is a solid paper to accept for publication. \n\nSome areas for improvement:\n\nIt seems strange not to mention all of the recent high-profile work on LM-based pre-training, since my impression is that these models operate effectively with large multi-sentence contexts. Do models like BERT and GPT-2 fail to take into account inter-sentence relations, as the paper claims most LMs do? I would like to see more discussion of how this work fits with that.\n\nI don't know that it makes sense to highlight as the contribution of this model that it can \"simultaneously capture syntax and semantics\". It's not clear to me that other language models fail to capture semantics (keeping in mind that semantics applies within a sentence and not just at a global level) -- rather, it seems that the strength of this model is in capturing semantic relations above the sentence level.  If this is correct, that should be expressed more precisely.\n\nIt's not clear to me what we learn from Figure 3. The claim is that \"the color of the hidden states of the stacked RNN based language model at layer 1 changes quickly ... because lower layers are in charge of learning short-term dependencies\", but looking at the higher layers I'm not seeing clear evidence of capturing of long-distance dependencies, or even clear capturing of syntactic constituents. The takeaways from that figure should be made clearer and should be sure to correspond to what we can actually confidently conclude from that analysis.\n", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I made a quick assessment of this paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I did not assess the derivations or theory."}, "extra_text": {}}, {"review_id": "BJxNohr45S", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper presents a method for natural language generation, using a language model, informed by a topic model. \nThe topic model is a hierarchical recurrent topic model that attempts to extract document-level word concurrence patterns and topic weight vectors for sentences. \nThe language model is a stacked RNN model, aiming to capture word sequential dependencies. \n\nThe proposed method is a combination of two existing methods, i.e. gamma-belief networks  and stacked RNN, where the stacked RNN is improved with the information from recurrent gamma belief network. \n\nOverall, this is a well written paper, clearly presented, with certain novelties. The method is well formulated mathematically and evaluated experimentally. The results look interesting especially for capturing the long-range dependencies, as shown by the  BLEU scores. One suggestion is that the authors didn't include computational analysis about the complexity and loads of the proposed method as compared with the baseline methods. ", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper was a very difficult case. All three original reviewers of the paper had never published in the area, and all of them advocated for acceptance of the paper. I, on the other hand, am an expert in the area who has published many papers, and I thought that while the paper is well-written and experimental evaluation is not incorrect, the method was perhaps less relevant given current state-of-the-art models. In addition, the somewhat non-standard evaluation was perhaps causing this fact to be masked. I asked the original reviewers to consider my comments multiple times both during the rebuttal period and after, and unfortunately none of them replied.

Because of this, I elicited two additional reviews from people I knew were experts in the field. The reviews are below. I sent the PDF to the reviewers directly, and asked them to not look at the existing reviews (or my comments) when doing their review in order to make sure that they were making a fair assessment. 

Long story short, Reviewer 4 essentially agreed with my concerns and pointed out a few additional clarity issues. Reviewer 5 pointed out a number of clarity issues and was also concerned with the fact that d_j has access to all other sentences (including those following the current sentence). I know that at the end of Section 2 it is noted that at test time d_j only refers to previous sentences, but if so there is also a training-testing disconnect in model training, and it seems that this would hurt the model results.

Based on this, I have decided to favor the opinions of three experts (me and the two additional reviewers) over the opinions of the original three reviewers, and not recommend the paper for acceptance at this time. In order to improve the paper I would suggest the following (1) an acknowledgement of standard methods to incorporate context by processing sequences consisting of multiple sentences simultaneously, (2) a more thorough comparison with state-of-the-art models that consider cross-sentential context on standard datasets such as WikiText or PTB. I would encourage the authors to consider this as they revise their paper.

Finally, I would like to apologize to the authors that they did not get a chance to reply to the second set of reviews. As I noted above, I did try to make my best effort to encourage discussion during the rebuttal period.
Decision: Reject