Recurrent Hierarchical Topic-Guided Neural Language Models

1 INTRODUCTION . Both topic and language models are widely used for text analysis . Topic models , such as latent Dirichlet allocation ( LDA ) ( Blei et al. , 2003 ; Griffiths & Steyvers , 2004 ; Hoffman et al. , 2013 ) and its nonparametric Bayesian generalizations ( Teh et al. , 2006 ; Zhou & Carin , 2015 ) , are well suited to extract document-level word concurrence patterns into latent topics from a text corpus . Their modeling power has been further enhanced by introducing multilayer deep representation ( Srivastava et al. , 2013 ; Mnih & Gregor , 2014 ; Gan et al. , 2015 ; Zhou et al. , 2016 ; Zhao et al. , 2018 ; Zhang et al. , 2018 ) . While having semantically meaningful latent representation , they typically treat each document as a bag of words ( BoW ) , ignoring word order ( Griffiths et al. , 2004 ; Wallach , 2006 ) . Language models have become key components of various natural language processing ( NLP ) tasks , such as text summarization ( Rush et al. , 2015 ; Gehrmann et al. , 2018 ) , speech recognition ( Mikolov et al. , 2010 ; Graves et al. , 2013 ) , machine translation ( Sutskever et al. , 2014 ; Cho et al. , 2014 ) , and image captioning ( Vinyals et al. , 2015 ; Mao et al. , 2015 ; Xu et al. , 2015 ; Gan et al. , 2017 ; Rennie et al. , 2017 ) . The primary purpose of a language model is to capture the distribution of a word sequence , commonly with a recurrent neural network ( RNN ) ( Mikolov et al. , 2011 ; Graves , 2013 ) or a Transformer based neural network ( Vaswani et al. , 2017 ; Dai et al. , 2019 ; Devlin et al. , 2019 ; Radford et al. , 2018 ; 2019 ) . In this paper , we focus on improving RNN-based language models that often have much fewer parameters and are easier to perform end-to-end training . While RNN-based language models do not ignore word order , they often assume that the sentences of a document are independent to each other . This simplifies the modeling task to independently assigning probabilities to individual sentences , ignoring their orders and document context ( Tian & Cho , 2016 ) . Such language models may consequently fail to capture the long-range dependencies and global semantic meaning of a document ( Dieng et al. , 2017 ; Wang et al. , 2018 ) . To relax the sentence independence assumption in language modeling , Tian & Cho ( 2016 ) propose larger-context language models that model the context of a sentence by representing its preceding sentences as either a single or a sequence of BoW vectors , which are then fed directly into the sentence modeling RNN . An alternative approach attracting significant recent interest is leveraging topic models to improve RNN-based language models . Mikolov & Zweig ( 2012 ) use pre-trained topic model features as an additional input to the RNN hidden states and/or output . Dieng et al . ( 2017 ) ; Ahn et al . ( 2017 ) combine the predicted word distributions , given by both a topic model and a language model , under variational autoencoder ( Kingma & Welling , 2013 ) . Lau et al . ( 2017 ) introduce an attention based convolutional neural network to extract semantic topics , which are used to extend the RNN cell . Wang et al . ( 2018 ) learn the global semantic coherence of a document via a neural topic model and use the learned latent topics to build a mixture-of-experts language model . Wang et al . ( 2019 ) further specify a Gaussian mixture model as the prior of the latent code in variational autoencoder , where each mixture component corresponds to a topic . While clearly improving the performance of the end task , these existing topic-guided methods still have clear limitations . For example , they only utilize shallow topic models with only a single stochastic hidden layer in their data generation process . Note several neural topic models use deep neural networks to construct their variational encoders , but still use shallow generative models ( decoders ) ( Miao et al. , 2017 ; Srivastava & Sutton , 2017 ) . Another key limitation lies in ignoring the sentence order , as they treat each document as a bag of sentences . Thus once the topic weight vector learned from the document context is given , the task is often reduced to independently assigning probabilities to individual sentences ( Lau et al. , 2017 ; Wang et al. , 2018 ; 2019 ) . In this paper , as depicted in Fig . 1 , we propose to use recurrent gamma belief network ( rGBN ) to guide a stacked RNN for language modeling . We refer to the model as rGBN-RNN , which integrates rGBN ( Guo et al. , 2018 ) , a deep recurrent topic model , and stacked RNN ( Graves , 2013 ; Chung et al. , 2017 ) , a neural language model , into a novel larger-context RNN-based language model . It simultaneously learns a deep recurrent topic model , extracting document-level multi-layer word concurrence patterns and sequential topic weight vectors for sentences , and an expressive language model , capturing both short- and long-range word sequential dependencies . For inference , we equip rGBN-RNN ( decoder ) with a novel variational recurrent inference network ( encoder ) , and train it end-to-end by maximizing the evidence lower bound ( ELBO ) . Different from the stacked RNN based language model in Chung et al . ( 2017 ) , which relies on three types of customized training operations ( UPDATE , COPY , FLUSH ) to extract multi-scale structures , the language model in rGBN-RNN learns such structures purely under the guidance of the temporally and hierarchically connected stochastic layers of rGBN . The effectiveness of rGBN-RNN as a new larger-context language model is demonstrated both quantitatively , with perplexity and BLEU scores , and qualitatively , with interpretable latent structures and randomly generated sentences and paragraphs . Notably , rGBN-RNN can generate a paragraph consisting of a sequence of semantically coherent sentences . 2 RECURRENT HIERARCHICAL TOPIC-GUIDED LANGUAGE MODEL . Denote a document of J sentences as D = ( S1 , S2 , . . . , SJ ) , where Sj = ( yj,1 , . . . , yj , Tj ) consists of Tj words from a vocabulary of size V . Conventional statistical language models often only focus on the word sequence within a sentence . Assuming that the sentences of a document are independent to each other , they often define P ( D ) ≈ ∏J j=1 P ( Sj ) = ∏J j=1 ∏Tj t=2 p ( yj , t | yj , < t ) p ( yj,1 ) . RNN based neural language models define the conditional probability of each word yj , t given all the previous words yj , < t within the sentence Sj , through the softmax function of a hidden state hj , t , as p ( yj , t | yj , < t ) = p ( yj , t |hj , t ) , hj , t = f ( hj , < t , yj , t−1 ) , ( 1 ) where f ( · ) is a non-linear function typically defined as an RNN cell , such as long short-term memory ( LSTM ) ( Hochreiter & Schmidhuber , 1997 ) and gated recurrent unit ( GRU ) ( Cho et al. , 2014 ) . These RNN-based statistical language models are typically applied only at the word level , without exploiting the document context , and hence often fail to capture long-range dependencies . While Dieng et al . ( 2017 ) ; Lau et al . ( 2017 ) ; Wang et al . ( 2018 ; 2019 ) remedy the issue by guiding the language model with a topic model , they still treat a document as a bag of sentences , ignoring the order of sentences , and lack the ability to extract hierarchical and recurrent topic structures . We introduce rGBN-RNN , as depicted in Fig . 1 ( a ) , as a new larger-context language model . It consists of two key components : ( i ) a hierarchical recurrent topic model ( rGBN ) , and ( ii ) a stacked RNN based language model . We use rGBN to capture both global semantics across documents and long-range inter-sentence dependencies within a document , and use the language model to learn the local syntactic relationships between the words within a sentence . Similar to Lau et al . ( 2017 ) ; Wang et al . ( 2018 ) , we represent a document as a sequence of sentence-context pairs as ( { S1 , d1 } , . . . , { SJ , dJ } ) , where dj ∈ ZVc+ summarizes the document excluding Sj , specifically ( S1 , ... , Sj−1 , Sj+1 , ... , SJ ) , into a BoW count vector , with Vc as the size of the vocabulary excluding stop words . Note a naive way is to treat each sentence as a document , use a dynamic topic model ( Blei & Lafferty , 2006 ) to capture the temporal dependencies of the latent topic-weight vectors , which is fed to the RNN to model the word sequence of the corresponding sentence . However , the sentences are often too short to be well modeled by a topic model . In our setting , as dj summarizes the document-level context of Sj , it is in general sufficiently long for topic modeling . Note during testing , we redefine dj as the BoW vector summarizing only the preceding sentences , i.e. , S1 : j−1 , which will be further clarified when presenting experimental results . 2.1 HIERARCHICAL RECURRENT TOPIC MODEL . Shown in Fig . 1 ( a ) , to model the time-varying sentence-context count vectors dj in document D , the generative process of the rGBN component , from the top to bottom hidden layers , is expressed as θLj ∼ Gam ( ΠLθLj−1 , τ0 ) , · · · , θlj ∼ Gam ( Φl+1θl+1j + Π lθlj−1 , τ0 ) , · · · , θ1j ∼ Gam ( Φ2θ2j + Π 1θ1j−1 , τ0 ) , dj ∼ Pois ( Φ1θ1j ) , ( 2 ) where θlj ∈ R Kl + denotes the gamma distributed topic weight vectors of sentence j at layer l , Πl ∈ RKl×Kl+ the transition matrix of layer l that captures cross-topic temporal dependencies , Φl ∈ RKl−1×Kl+ the loading matrix at layer l , Kl the number of topics of layer l , and τ0 ∈ R+ a scaling hyperparameter . At j = 1 , θl1 ∼ Gam ( Φl+1θl+11 , τ0 ) for l = 1 , . . . , L− 1 and θL1 ∼ Gam ( ν , τ0 ) , where ν = 1KL . Finally , Dirichlet priors are placed on the columns of Π l and Φl , i.e. , πlk and φ l k , which not only makes the latent representation more identifiable and interpretable , but also facilitates inference . The count vector dj can be factorized into the product of Φ1 and θ1j under the Poisson likelihood . The shape parameters of θlj ∈ R Kl + can be factorized into the sum of Φ l+1θl+1j , capturing inter-layer hierarchical dependence , and Πlθlj−1 , capturing intra-layer temporal dependence . rGBN not only captures the document-level word occurrence patterns inside the training text corpus , but also the sequential dependencies of the sentences inside a document . Note ignoring the recurrent structure , rGBN will reduce to the gamma belief network ( GBN ) of Zhou et al . ( 2016 ) , which can be considered as a multi-stochastic-layer deep generalization of LDA ( Cong et al. , 2017a ) . If ignoring its hierarchical structure ( i.e. , L = 1 ) , rGBN reduces to Poisson–gamma dynamical systems ( Schein et al. , 2016 ) . We refer to the rGBN-RNN without its recurrent structure as GBN-RNN , which no longer models sequential sentence dependencies ; see Appendix A for more details .

Summary: The paper  proposes deep recurrent topic model guided language modeling using a stacked RNN, and uses a novel variational recurrent inference network to learn the parameters.  The proposed model can capture the dependence across the sentences in language generation though the recurrent latent topics. Moreover, the deep rGBN architecture provides Gamma distributed topic topic weight vectors which can be associated with every layer of the stacked RNN generating the sentence. The parameters of both the hierarchical recurrent topic model and language model are learnt using a hybrid inference algorithm  combining  variational inference to estimate language model and inference network parameters and MCMC to infer rGBN parameters. The effectiveness of the proposed model on the language modeling task is demonstrated on 3 datasets using Perplexity and BLEU score. The paper also provides a visual representation of the topics and their temporal trajectories.