textTOvec: DEEP CONTEXTUALIZED NEURAL AUTOREGRESSIVE TOPIC MODELS OF LANGUAGE WITH DISTRIBUTED COMPOSITIONAL PRIOR

1 INTRODUCTION . Probabilistic topic models , such as LDA ( Blei et al. , 2003 ) , Replicated Softmax ( RSM ) ( Salakhutdinov & Hinton , 2009 ) and Document Neural Autoregressive Distribution Estimator ( DocNADE ) variants ( Larochelle & Lauly , 2012 ; Zheng et al. , 2016 ; Lauly et al. , 2017 ; Gupta et al. , 2019 ) are often used to extract topics from text collections , and predict the probabilities of each word in a given document belonging to each topic . Subsequently , they learn latent document representations that can be used to perform natural language processing ( NLP ) tasks such as information retrieval ( IR ) , document classification or summarization . However , such probabilistic topic models ignore word order and represent a given context as a bag of its words , thereby disregarding semantic information . To motivate our first task of extending probabilistic topic models to incorporate word order and language structure , assume that we conduct topic analysis on the following two sentences : Bear falls into market territory and Market falls into bear territory When estimating the probability of a word in a given context ( here : P ( “ bear ” |context ) ) , traditional topic models do not account for language structure since they ignore word order within the context and are based on “ bag-of-words ” ( BoWs ) only . In this particular setting , the two sentences have the same unigram statistics , but are about different topics . On deciding which topic generated the word “ bear ” in the second sentence , the preceding words “ market falls ” make it more likely that it was generated by a topic that assigns a high probability to words related to stock market trading , where “ bear territory ” is a colloquial expression in the domain . In addition , the language structure ( e.g. , syntax and semantics ) is also ignored . For instance , the word “ bear ” in the first sentence is a proper noun and subject while it is an object in the second . In practice , topic models also ignore functional words such as “ into ” , which may not be appropriate in some scenarios . Recently , Peters et al . ( 2018 ) have shown that a deep contextualized LSTM-based language model ( LSTM-LM ) is able to capture different language concepts in a layer-wise fashion , e.g. , the lowest layer captures language syntax and topmost layer captures semantics . However , in LSTM-LMs the probability of a word is a function of its sentence only and word occurrences are modeled in a fine granularity . Consequently , LSTM-LMs do not capture semantics at a document level . To this end , recent studies such as TDLM ( Lau et al. , 2017 ) , Topic-RNN ( Dieng et al. , 2016 ) and TCNLM ( Wang et al. , 2018 ) have integrated the merits of latent topic and neural language models ( LMs ) ; however , they have focused on improving LMs with global ( semantics ) dependencies using latent topics . Similarly , while bi-gram LDA based topic models ( Wallach , 2006 ; Wang et al. , 2007 ) and n-gram based topic learning ( Lauly et al. , 2017 ) can capture word order in short contexts , they are unable to capture long term dependencies and language concepts . In contrast , DocNADE variants ( Larochelle & Lauly , 2012 ; Gupta et al. , 2019 ) learns word occurrences across documents i.e. , coarse granularity ( in the sense that the topic assigned to a given word occurrence equally depends on all the other words appearing in the same document ) ; however since it is based on the BoW assumption all language structure is ignored . In language modeling , Mikolov et al . ( 2010 ) have shown that recurrent neural networks result in a significant reduction of perplexity over standard n-gram models . Contribution 1 : We introduce language structure into neural autoregressive topic models via a LSTM-LM , thereby accounting for word ordering ( or semantic regularities ) , language concepts and long-range dependencies . This allows for the accurate prediction of words , where the probability of each word is a function of global and local ( semantics ) contexts , modeled via DocNADE and LSTMLM , respectively . The proposed neural topic model is named as contextualized-Document Neural Autoregressive Distribution Estimator ( ctx-DocNADE ) and offers learning complementary semantics by combining joint word and latent topic learning in a unified neural autoregressive framework . For instance , Figure 1 ( left and middle ) shows the complementary topic and word semantics , based on TM and LM representations of the term “ fall ” . Observe that the topic captures the usage of “ fall ” in the context of stock market trading , attributed to the global ( semantic ) view . While this is a powerful approach for incorporating language structure and word order in particular for long texts and corpora with many documents , learning from contextual information remains challenging in settings with short texts and few documents , since ( 1 ) limited word co-occurrences or little context ( 2 ) significant word non-overlap in such short texts and ( 3 ) small training corpus of documents lead to little evidence for learning word co-occurrences . However , distributional word representations ( i.e . word embeddings ) ( Pennington et al. , 2014 ) have shown to capture both the semantic and syntactic relatedness in words and demonstrated impressive performance in NLP tasks . For example , assume that we conduct topic analysis over the two short text fragments : Deal with stock index falls and Brace for market share drops . Traditional topic models with “ BoW ” assumption will not be able to infer relatedness between word pairs such as ( falls , drops ) due to the lack of word-overlap and small context in the two phrases . However , in the distributed embedding space , the word pairs are semantically related as shown in Figure 1 ( left ) . Related work such as Sahami & Heilman ( 2006 ) employed web search results to improve the information in short texts and Petterson et al . ( 2010 ) introduced word similarity via thesauri and dictionaries into LDA . Das et al . ( 2015 ) and Nguyen et al . ( 2015 ) integrated word embeddings into LDA and Dirichlet Multinomial Mixture ( DMM ) ( Nigam et al. , 2000 ) models . Recently , Gupta et al . ( 2019 ) extends DocNADE by introducing pre-trained word embeddings in topic learning . However , they ignore the underlying language structure , e.g. , word ordering , syntax , etc . In addition , DocNADE and its extensions outperform LDA and RSM topic models in terms of perplexity and IR . Contribution 2 : We incorporate distributed compositional priors in DocNADE : we use pre-trained word embeddings via LSTM-LM to supplement the multinomial topic model ( i.e. , DocNADE ) in learning latent topic and textual representations on a smaller corpus and/or short texts . Knowing similarities in a distributed space and integrating this complementary information via a LSTM-LM , a topic representation is much more likely and coherent . Taken together , we combine the advantages of complementary learning and external knowledge , and couple topic- and language models with pre-trained word embeddings to model short and long text documents in a unified neural autoregressive framework , named as ctx-DocNADEe . Our approach learns better textual representations , which we quantify via generalizability ( e.g. , perplexity ) , interpretability ( e.g. , topic extraction and coherence ) and applicability ( e.g. , IR and classification ) . To illustrate our two contributions , we apply our modeling approaches to 7 long-text and 8 short-text datasets from diverse domains and demonstrate that our approach consistently outperforms state-ofthe-art generative topic models . Our learned representations , result in a gain of : ( 1 ) 4.6 % ( .790 vs .755 ) in topic coherence , ( 2 ) 6.5 % ( .615 vs .577 ) in precision at retrieval fraction 0.02 , and ( 3 ) 4.4 % ( .662 vs .634 ) in F1 for text classification , averaged over 6 long-text and 8 short-text datasets . When applied to short-text and long-text documents , our proposed modeling approaches generate contextualized topic vectors , which we name textTOvec . The code is available at https : //github.com/pgcool/textTOvec . 2 NEURAL AUTOREGRESSIVE TOPIC MODELS . Generative models are based on estimating the probability distribution of multidimensional data , implicitly requiring modeling complex dependencies . Restricted Boltzmann Machine ( RBM ) ( Hinton et al. , 2006 ) and its variants ( Larochelle & Bengio , 2008 ) are probabilistic undirected models of binary data . RSM ( Salakhutdinov & Hinton , 2009 ) and its variants ( Gupta et al. , 2018 ) are generalization of the RBM , that are used to model word counts . However , estimating the complex probability distribution of the underlying high-dimensional observations is intractable . To address this challenge , NADE ( Larochelle & Murray , 2011 ) decomposes the joint distribution of binary observations into autoregressive conditional distributions , each modeled using a feed-forward network . Unlike for RBM/RSM , this leads to tractable gradients of the data negative log-likelihood . 2.1 DOCUMENT NEURAL AUTOREGRESSIVE TOPIC MODEL ( DOCNADE ) . An extension of NADE and RSM , DocNADE ( Larochelle & Lauly , 2012 ) models collections of documents as orderless bags of words ( BoW approach ) , thereby disregarding any language structure . In other words , it is trained to learn word representations reflecting the underlying topics of the documents only , ignoring syntactical and semantic features as those encoded in word embeddings ( Bengio et al. , 2003 ; Mikolov et al. , 2013 ; Pennington et al. , 2014 ; Peters et al. , 2018 ) . DocNADE ( Lauly et al. , 2017 ) represents a document by transforming its BoWs into a sequence v = [ v1 , ... , vD ] of size D , where each element vi ∈ { 1 , 2 , ... , K } corresponds to a multinomial observation ( representing a word from a vocabulary of size K ) . Thus , vi is the index in the vocabulary of the ith word of the document v. DocNADE models the joint distribution p ( v ) of all words vi by decomposing it as p ( v ) = ∏D i=1 p ( vi|v < i ) , where each autoregressive conditional p ( vi|v < i ) for the word observation vi is computed using the preceding observations v < i ∈ { v1 , ... , vi−1 } in a feed-forward neural network for i ∈ { 1 , ... D } , hDNi ( v < i ) = g ( e+ ∑ k < i W : ,vk ) and p ( vi = w|v < i ) = exp ( bw +Uw , :h DN i ( v < i ) ) ∑ w′ exp ( bw′ +Uw′ , :h DN i ( v < i ) ) ( 1 ) where g ( · ) is an activation function , U ∈ RK×H is a weight matrix connecting hidden to output , e ∈ RH and b ∈ RK are bias vectors , W ∈ RH×K is a word representation matrix in which a column W : ,vi is a vector representation of the word vi in the vocabulary , and H is the number of hidden units ( topics ) . The log-likelihood of any document v of any arbitrary length is given by : LDN ( v ) = ∑D i=1 log p ( vi|v < i ) . Note that the past word observations v < i are orderless due to BoWs , and may not correspond to the words preceding the ith word in the document itself . Algorithm 1 Computation of log p ( v ) Input : A training document v Input : Word embedding matrix E Output : log p ( v ) 1 : a← e 2 : q ( v ) = 1 3 : for i from 1 to D do 4 : compute hi and p ( vi|v < i ) 5 : q ( v ) ← q ( v ) p ( vi|v < i ) 6 : a← a+W : ,vi 7 : log p ( v ) ← log q ( v )

Summary: The proposed method is not novel. For example, Lauly et al., 2017 have proposed a similar way of combining LM and DocNADE. This paper does not provide some motivations or theories behind such artificial combination (i.e., just linearly combine their hidden state) to explain why it works better than other alternatives (e.g., what about adding some linear layers before combining h_i^{DN} and h_i^{LM}).