Title: textTOvec: DEEP CONTEXTUALIZED NEURAL AUTOREGRESSIVE TOPIC MODELS OF LANGUAGE WITH DISTRIBUTED COMPOSITIONAL PRIOR

Abstract: We address two challenges of probabilistic topic modelling in order to better estimate the probability of a word in a given context, i.e., P (word|context) : (1) No language structure in context: Probabilistic topic models ignore word order by summarizing a given context as a "bag-of-word" and consequently the semantics of words in the context is lost. In this work, we incorporate language structure by combining a neural autoregressive topic model (TM) (e.g., DocNADE) with a LSTM based language model (LSTM-LM) in a single probabilistic framework. The LSTM-LM learns a vector-space representation of each word by accounting for word order in local collocation patterns, while the TM simultaneously learns a latent representation from the entire document. In addition, the LSTM-LM models complex characteristics of language (e.g., syntax and semantics), while the TM discovers the underlying thematic structure in a collection of documents. We unite two complementary paradigms of learning the meaning of word occurrences by combining a topic model and a language model in a unified probabilistic framework, named as ctx-DocNADE. (2) Limited context and/or smaller training corpus of documents: In settings with a small number of word occurrences (i.e., lack of context) in short text or data sparsity in a corpus of few documents, the application of TMs is challenging. We address this challenge by incorporating external knowledge into neural autoregressive topic models via a language modelling approach: we use word embeddings as input of a LSTM-LM with the aim to improve the word-topic mapping on a smaller and/or short-text corpus. The proposed Doc-NADE extension is named as ctx-DocNADEe. We present novel neural autoregressive topic model variants coupled with neural language models and embeddings priors that consistently outperform state-of-theart generative topic models in terms of generalization (perplexity), interpretability (topic coherence) and applicability (retrieval and classification) over 7 long-text and 8 short-text datasets from diverse domains.
Bear falls into market territory and Market falls into bear territory environment

1. INTRODUCTION:

Probabilistic topic models, such as LDA (Blei et al., 2003) , Replicated Softmax (RSM) (Salakhutdinov & Hinton, 2009) and Document Neural Autoregressive Distribution Estimator (DocNADE) variants (Larochelle & Lauly, 2012; Zheng et al., 2016; Lauly et al., 2017; Gupta et al., 2019) are often used to extract topics from text collections, and predict the probabilities of each word in a given document belonging to each topic. Subsequently, they learn latent document representations that can be used to perform natural language processing (NLP) tasks such as information retrieval (IR), document classification or summarization. However, such probabilistic topic models ignore word order and represent a given context as a bag of its words, thereby disregarding semantic information.

To motivate our first task of extending probabilistic topic models to incorporate word order and language structure, assume that we conduct topic analysis on the following two sentences: When estimating the probability of a word in a given context (here: P ("bear"|context)), traditional topic models do not account for language structure since they ignore word order within the context and are based on "bag-of-words" (BoWs) only. In this particular setting, the two sentences have the same unigram statistics, but are about different topics. On deciding which topic generated the word "bear" in the second sentence, the preceding words "market falls" make it more likely that it was generated by a topic that assigns a high probability to words related to stock market trading, where "bear territory" is a colloquial expression in the domain. In addition, the language structure (e.g., syntax and semantics) is also ignored. For instance, the word "bear" in the first sentence is a proper noun and subject while it is an object in the second. In practice, topic models also ignore functional words such as "into", which may not be appropriate in some scenarios.

Recently, Peters et al. (2018) have shown that a deep contextualized LSTM-based language model (LSTM-LM) is able to capture different language concepts in a layer-wise fashion, e.g., the lowest layer captures language syntax and topmost layer captures semantics. However, in LSTM-LMs the probability of a word is a function of its sentence only and word occurrences are modeled in a fine granularity. Consequently, LSTM-LMs do not capture semantics at a document level. To this end, recent studies such as TDLM (Lau et al., 2017) , Topic-RNN (Dieng et al., 2016) and TCNLM (Wang et al., 2018) have integrated the merits of latent topic and neural language models (LMs); however, they have focused on improving LMs with global (semantics) dependencies using latent topics.

Similarly, while bi-gram LDA based topic models (Wallach, 2006; Wang et al., 2007) and n-gram based topic learning (Lauly et al., 2017) can capture word order in short contexts, they are unable to capture long term dependencies and language concepts. In contrast, DocNADE variants (Larochelle & Lauly, 2012; Gupta et al., 2019 ) learns word occurrences across documents i.e., coarse granularity (in the sense that the topic assigned to a given word occurrence equally depends on all the other words appearing in the same document); however since it is based on the BoW assumption all language structure is ignored. In language modeling, Mikolov et al. (2010) have shown that recurrent neural networks result in a significant reduction of perplexity over standard n-gram models.

Contribution 1: We introduce language structure into neural autoregressive topic models via a LSTM-LM, thereby accounting for word ordering (or semantic regularities), language concepts and long-range dependencies. This allows for the accurate prediction of words, where the probability of each word is a function of global and local (semantics) contexts, modeled via DocNADE and LSTM-LM, respectively. The proposed neural topic model is named as contextualized-Document Neural Autoregressive Distribution Estimator (ctx-DocNADE) and offers learning complementary semantics by combining joint word and latent topic learning in a unified neural autoregressive framework. For instance, Figure 1 (left and middle) shows the complementary topic and word semantics, based on TM and LM representations of the term "fall". Observe that the topic captures the usage of "fall" in the context of stock market trading, attributed to the global (semantic) view.

While this is a powerful approach for incorporating language structure and word order in particular for long texts and corpora with many documents, learning from contextual information remains challenging in settings with short texts and few documents, since (1) limited word co-occurrences or little context (2) significant word non-overlap in such short texts and (3) small training corpus of documents lead to little evidence for learning word co-occurrences. However, distributional word representations (i.e. word embeddings) (Pennington et al., 2014) have shown to capture both the semantic and syntactic relatedness in words and demonstrated impressive performance in NLP tasks.

For example, assume that we conduct topic analysis over the two short text fragments: Deal with stock index falls and Brace for market share drops. Traditional topic models

v 1 Latent Topics v 2 v i-i v i ctx-DocNADEe v i+1 v D-1 v D W Coarse Granuality: Global View

Word occurrences across document ... ... with "BoW" assumption will not be able to infer relatedness between word pairs such as (falls, drops) due to the lack of word-overlap and small context in the two phrases. However, in the distributed embedding space, the word pairs are semantically related as shown in Figure 1 (left). Sahami & Heilman (2006) employed web search results to improve the information in short texts and Petterson et al. (2010) introduced word similarity via thesauri and dictionaries into LDA. Das et al. (2015) and Nguyen et al. (2015) integrated word embeddings into LDA and Dirichlet Multinomial Mixture (DMM) (Nigam et al., 2000) models. Recently, Gupta et al. (2019) extends DocNADE by introducing pre-trained word embeddings in topic learning. However, they ignore the underlying language structure, e.g., word ordering, syntax, etc. In addition, DocNADE and its extensions outperform LDA and RSM topic models in terms of perplexity and IR.

h i DN h i LM v i h i x i+1 x i x i-1 x 1 Fine Granuality: Local View Word occurences in collocation patterns x 2 LSTM-LM DocNADE (DN) = W or = W + E Embedding layer ... c i = x 1 , x 2 ,...,x i-1 v 1 DocNADE U v i-1 v i v D v 1 v 2 v D v i h 1 h 2 h i h D U W ... ...

Related work such as:

Contribution 2: We incorporate distributed compositional priors in DocNADE: we use pre-trained word embeddings via LSTM-LM to supplement the multinomial topic model (i.e., DocNADE) in learning latent topic and textual representations on a smaller corpus and/or short texts. Knowing similarities in a distributed space and integrating this complementary information via a LSTM-LM, a topic representation is much more likely and coherent.

Taken together, we combine the advantages of complementary learning and external knowledge, and couple topic-and language models with pre-trained word embeddings to model short and long text documents in a unified neural autoregressive framework, named as ctx-DocNADEe. Our approach learns better textual representations, which we quantify via generalizability (e.g., perplexity), interpretability (e.g., topic extraction and coherence) and applicability (e.g., IR and classification).

To illustrate our two contributions, we apply our modeling approaches to 7 long-text and 8 short-text datasets from diverse domains and demonstrate that our approach consistently outperforms state-ofthe-art generative topic models. Our learned representations, result in a gain of: (1) 4.6% (.790 vs .755) in topic coherence, (2) 6.5% (.615 vs .577) in precision at retrieval fraction 0.02, and (3) 4.4% (.662 vs .634) in F 1 for text classification, averaged over 6 long-text and 8 short-text datasets.

When applied to short-text and long-text documents, our proposed modeling approaches generate contextualized topic vectors, which we name textTOvec. The code is available at https: //github.com/pgcool/textTOvec.

2. NEURAL AUTOREGRESSIVE TOPIC MODELS:

Generative models are based on estimating the probability distribution of multidimensional data, implicitly requiring modeling complex dependencies. Restricted Boltzmann Machine (RBM) (Hinton et al., 2006) and its variants (Larochelle & Bengio, 2008) are probabilistic undirected models of binary data. RSM (Salakhutdinov & Hinton, 2009) and its variants (Gupta et al., 2018) are generalization of the RBM, that are used to model word counts. However, estimating the complex probability distribution of the underlying high-dimensional observations is intractable. To address this challenge, NADE (Larochelle & Murray, 2011) decomposes the joint distribution of binary observations into autoregressive conditional distributions, each modeled using a feed-forward network. Unlike for RBM/RSM, this leads to tractable gradients of the data negative log-likelihood.

2.1. DOCUMENT NEURAL AUTOREGRESSIVE TOPIC MODEL (DOCNADE):

An extension of NADE and RSM, DocNADE (Larochelle & Lauly, 2012 ) models collections of documents as orderless bags of words (BoW approach), thereby disregarding any language structure. In other words, it is trained to learn word representations reflecting the underlying topics of the documents only, ignoring syntactical and semantic features as those encoded in word embeddings (Bengio et al., 2003; Mikolov et al., 2013; Pennington et al., 2014; Peters et al., 2018) . DocNADE (Lauly et al., 2017) represents a document by transforming its BoWs into a sequence v = [v 1 , ..., v D ] of size D, where each element v i ∈ {1, 2, ..., K} corresponds to a multinomial observation (representing a word from a vocabulary of size K). Thus, v i is the index in the vocabulary of the ith word of the document v. DocNADE models the joint distribution p(v) of all words v i by decomposing it as p(v) = D i=1 p(v i |v <i ), where each autoregressive conditional p(v i |v <i ) for the word observation v i is computed using the preceding observations v <i ∈ {v 1 , ..., v i-1 } in a feed-forward neural network for i ∈ {1, ...D},

EQUATION

where g(•) is an activation function, U ∈ R K×H is a weight matrix connecting hidden to output, e ∈ R H and b ∈ R K are bias vectors, W ∈ R H×K is a word representation matrix in which a column W :,vi is a vector representation of the word v i in the vocabulary, and H is the number of hidden units (topics). The log-likelihood of any document v of any arbitrary length is given by:

L DN (v) = D i=1 log p(v i |v <i ).

Note that the past word observations v <i are orderless due to BoWs, and may not correspond to the words preceding the ith word in the document itself.

Algorithm 1 Computation of log p(v):

Input: A training document v Input: Word embedding matrix E Output: log p(v) 1: a ← e 2: q(v) = 1 3: for i from 1 to D do 4: compute hi and p(vi|v<i) 5: q(v) ← q(v)p(vi|v<i) 6: a ← a + W:,v i 7: log p(v) ← log q(v) model h i p(v i |v <i ) DocNADE h DN i ← g(a) equation 1 h i ← h DN i ctx-DocNADE h LM i ← LSTM(c i , embedding = W) equation 2 h i ← h DN i + λ h LM i ctx-DocNADEe h LM i ← LSTM(c i , embedding = W + E) equation 2 h i ← h DN i + λ h LM i

Table 1 : Computation of hi and p(vi|v<i) in DocNADE, ctx-DocNADE and ctx-DocNADEe models, correspondingly used in estimating log p(v) (Algorithm 1).

2.2. DEEP CONTEXTUALIZED DOCNADE WITH DISTRIBUTIONAL SEMANTICS:

We propose two extensions of the DocNADE model: (1) ctx-DocNADE: introducing language structure via LSTM-LM and (2) ctx-DocNADEe: incorporating external knowledge via pre-trained word embeddings E, to model short and long texts. The unified network(s) account for the ordering of words, syntactical and semantic structures in a language, long and short term dependencies, as well as external knowledge, thereby circumventing the major drawbacks of BoW-based representations.

Similar to DocNADE, ctx-DocNADE models each document v as a sequence of multinomial observations. Let [x 1 , x 2 , ..., x N ] be a sequence of N words in a given document, where x i is represented by an embedding vector of dimension, dim. Further, for each element v i ∈ v, let c i = [x 1 , x 2 , ..., x i-1 ] be the context (preceding words) of ith word in the document. Unlike in DocNADE, the conditional probability of the word v i in ctx-DocNADE (or ctx-DocNADEe) is a function of two hidden vectors: LSTM-based components of ctx-DocNADE, respectively:

h DN i (v <i ) and h LM i (c i ),

EQUATION

where h DN i (v <i ) is computed as in DocNADE (equation 1) and λ is the mixture weight of the LM component, which can be optimized during training (e.g., based on the validation set). The second term h LM i is a context-dependent representation and output of an LSTM layer at position i -1 over input sequence c i , trained to predict the next word v i . The LSTM offers history for the ith word via modeling temporal dependencies in the input sequence, c i . The conditional distribution for each word v i is estimated by equation 2, where the unified network of DocNADE and LM combines global and context-dependent representations. Our model is jointly optimized to maximize the pseudo log likelihood, log p(v) ≈ D i=1 log p(v i |v <i ) with stochastic gradient descent. See Larochelle & Lauly (2012) for more details on training from bag of word counts.

In the weight matrix W of DocNADE (Larochelle & Lauly, 2012) , each row vector W j,: encodes topic information for the jth hidden topic feature and each column vector W :,vi is a vector for the word v i . To obtain complementary semantics, we exploit this property and expose W to both global and local influences by sharing W in the DocNADE and LSTM-LM componenents. Thus, the embedding layer of LSTM-LM component represents the column vectors. ctx-DocNADE, in this realization of the unified network the embedding layer in the LSTM component is randomly initialized. This extends DocNADE by accounting for the ordering of words and language concepts via context-dependent representations for each word in the document. ctx-DocNADEe, the second version extends ctx-DocNADE with distributional priors, where the embedding layer in the LSTM component is initialized by the sum of a pre-trained embedding matrix E and the weight matrix W. Note that W is a model parameter; however E is a static prior.

Algorithm 1 and Table 1 show the log p(v) for a document v in three different settings: Doc-NADE, ctx-DocNADE and ctx-DocNADEe. In the DocNADE component, since the weights in the matrix W are tied, the linear activation a can be re-used in every hidden layer and computational complexity reduces to O(HD), where H is the size of each hidden layer. In every epoch, we run an LSTM over the sequence of words in the document and extract hidden vectors h LM i , corresponding to c i for every target word v i . Therefore, the computational complexity in ctx-DocNADE or ctx-DocNADEe is O(HD + N), where N is the total number of edges in the LSTM network (Hochreiter & Schmidhuber, 1997; Sak et al., 2014) . The trained models can be used to extract a textTOvec representation, i.e., h(

v * ) = h DN (v * ) + λ h LM (c * N +1 ) for the text v * of length D * , where h DN (v * ) = g(e + k≤D * W:,v k ) and h LM (c * N +1 ) = LSTM(c * N +1 , embedding = W or (W + E)).

ctx-DeepDNEe: DocNADE and LSTM can be extended to a deep, multiple hidden layer architecture by adding new hidden layers as in a regular deep feed-forward neural network, allowing for improved performance. In the deep version, the first hidden layer is computed in an analogous fashion to DocNADE variants (equation 1 or 2). Subsequent hidden layers are computed as:

h DN i,d (v <i ) = g(e d + W i,d • h i,d-1 (v <i )) or h LM i,d (c i ) = deepLST M (c i , depth = d) for d = 2, ...n,

where n is the total number of hidden layers (i.e., depth) in the deep feed-forward and LSTM networks. For d=1, the hidden vectors h DN i,1 and h LM i,1 correspond to equations 1 and 2. The conditional p(v i = w|v <i ) is computed using the last layer n, i.e., 236 .493 .480 .798 .587 .356 .754 .882 .543 .715 .513 .693 .638 .736 .588 814 .542 .685 glove(FV) . Table 3 : State-of-the-art comparison: IR (i.e, IR-precision at 0.02 fraction) and classification F 1 for short texts, where Avg: average over the row values, the bold and underline: the maximum for IR and F1, respectively.

h i,n = h DN i,n + λ h LM i,n . Model 20NSshort TREC6 R21578title Subjectivity Polarity TMNtitle TMN AGnewstitle Avg IR F 1 IR F 1 IR F 1 IR F 1 IR F 1 IR F 1 IR F 1 IR F 1 IR F 1 glove(RV) .

3. EVALUATION:

We apply our modeling approaches (in improving topic models, i.e, DocNADE using language concepts from LSTM-LM) to 8 short-text and 7 long-text datasets of varying size with single/multiclass labeled documents from public as well as industrial corpora. We present four quantitative measures in evaluating topic models: generalization (perplexity), topic coherence, text retrieval and categorization. See the appendices for the data description and example texts. Table 2 shows the data statistics, where 20NS and R21578 signify 20NewsGroups and Reuters21578, respectively.

Baselines: While, we evaluate our multi-fold contributions on four tasks: generalization (perplexity), topic coherence, text retrieval and categorization, we compare performance of our proposed models Wang et al., 2018) . .050 ----DocNADE(RV) .270 . Experimental Setup: DocNADE is often trained on a reduced vocabulary (RV) after pre-processing (e.g., ignoring functional words, etc.); however, we also investigate training it on full text/vocabulary (FV) (Table 2 ) and compute document representations to perform different evaluation tasks. The FV setting preserves the language structure, required by LSTM-LM, and allows a fair comparison of DocNADE+FV and ctx-DocNADE variants. We use the glove embedding of 200 dimensions. All the baselines and proposed models (ctx-DocNADE, ctx-DocNADEe and ctx-DeepDNEe) were run in the FV setting over 200 topics to quantify the quality of the learned representations. To better initialize the complementary learning in ctx-DocNADEs, we perform a pre-training for 10 epochs with λ set to 0. See the appendices for the experimental setup and hyperparameters for the following tasks, including the ablation over λ on validation set.

Model 20NSsmall Reuters8 20NS R21578 SiROBs AGnews Avg IR F 1 IR F 1 IR F 1 IR F 1 IR F 1 IR F 1 IR F 1 glove(RV) .

Data glove-DMM glove-LDA DocNADE ctx-DNE ctx-DNEe Data glove-DMM DocNADE ctx-DNE ctx-DNEe W10 W20 W10 W20 W10 W20 W10 W20 W10 W20 W10 W20 W10 W20 W10 W20 W10 W20 20NSshort .512 . 575 .616 .767 .669 .779 .682 .794 .696 .801 Subjectivity .538 .433 .613 .749 .629 .767 .634 .771 TREC6 .410 .475 .551 .736 .699 .818 .714 .810 .713 .809 AGnewstitle .584 .678 .731 .757 .739 .858 .746 .865 R21578title .364 .458 .478 .677 .701 .812 .713 .802 .723 .834 20NSsmall .578 .548 .508 .628 .546 .667 .565 .692 Polarity .637 .363 .375 .468 .610 .742 .611 .756 .650 .779 Reuters8 .372 .302 .583 .710 .584 .710 .592 .714 TMNtitle .633 .778 .651 .798 .712 .822 .716 .831 .735 .845 20NS .458 .374 .606 .729 .615 .746 .631 .759 TMN .705 .444 .550 .683 .642 .762 .639 .759 .709 .825 Avg (all) .527 . 452 .643 .755 .654 .772 .672 .790 Table 6 : Average coherence for short and long texts over 200 topics in FV setting, where DocNADE ↔ DNE

We run TDLM3 (Lau et al., 2017) for all the short-text datasets to evaluate the quality of representations learned in the spare data setting. For a fair comparison, we set 200 topics and hidden size, and initialize with the same pre-trained word embeddings (i.e., glove) as used in the ctx-DocNADEe.

3.1. GENERALIZATION: PERPLEXITY (PPL):

To evaluate the generative performance of the topic models, we estimate the log-probabilities for the test documents and compute the average held-out perplexity (P P L) per word as, P P L = exp -

1 z z t=1 1 |v t | log p(v t )

, where z and |v t | are the total number of documents and words in a document v t . For DocNADE, the log-probability log p(v t ) is computed using L DN (v); however, we ignore the mixture coefficient, i.e., λ=0 (equation 2) to compute the exact log-likelihood in ctx-DocNADE versions. The optimal λ is determined based on the validation set. Table 5 quantitatively shows the PPL scores, where the complementary learning with λ = 0.01 (optimal) in ctx-DocNADE achieves lower perplexity than the baseline DocNADE for both short and long texts, e.g., (822 vs 846) and (1358 vs 1375) on AGnewstitle and 20NS4 datasets, respectively in the FV setting.

3.2. INTERPRETABILITY: TOPIC COHERENCE:

We compute topic coherence (Chang et al., 2009; Newman et al., 2009; Gupta et al., 2018) to assess the meaningfulness of the underlying topics captured. We choose the coherence measure proposed by Röder et al. (2015) , which identifies context features for each topic word using a sliding window over the reference corpus. Higher scores imply more coherent topics.

We use the gensim module (radimrehurek.com/gensim/models/coherencemodel.html, coherence type = c v) to estimate coherence for each of the 200 topics (top 10 and 20 words). Table 6 shows average coherence over 200 topics, where the higher scores in ctx-DocNADE compared to DocNADE (.772 vs .755) suggest that the contextual information and language structure help in generating more coherent topics. The introduction of embeddings in ctx-DocNADEe boosts the topic coherence, leading to a gain of 4.6% (.790 vs .755) on average over 11 datasets. Note that the proposed models also outperform the baselines methods glove-DMM and glove-LDA. Qualitatively, Table 8 illustrates an example topic from the 20NSshort text dataset for DocNADE, ctx-DocNADE and ctx-DocNADEe, where the inclusion of embeddings results in a more coherent topic.

Additional Baslines: We further compare our proposed models to other approaches that combining topic and language models, such as TDLM (Lau et al., 2017 ), Topic-RNN (Dieng et al., 2016) and TCNLM (Wang et al., 2018) . However, the related studies focus on improving language models using topic models: in contrast, the focus of our work is on improving topic models for textual representations (short-text or long-text documents) by incorporating language concepts (e.g., word ordering, syntax, semantics, etc.) and external knowledge (e.g., word embeddings) via neural language models, as discussed in section 1.

To this end, we follow the experimental setup of the most recent work, TCNLM and quantitatively compare the performance of our models (i.e., ctx-DocNADE and ctx-DocNADEe) in terms of topic coherence (NPMI) on BNC dataset. Table 7 (left) shows NPMI scores of different models, where the results suggest that our contribution (i.e., ctx-DocNADE) of introducing language concepts into BoW topic model (i.e., DocNADE) improves topic coherence5 . The better performance for high val- The sliding window is one of the hyper-parameters for computing topic coherence (Röder et al., 2015; Wang et al., 2018) . A sliding window of 20 is used in TCNLM; in addition we also present results for a window of size 110. λ is the mixture weight of the LM component in the topic modeling process, and (s) and (l) indicate small and large model, respectively. The symbol '-' indicates no result, since word embeddings of 150 dimensions are not available from glove vectors. (Right): The top 5 words of seven learnt topics from our models and TCNLM. The asterisk (*) indicates our proposed models and (#) taken from TCNLM (Wang et al., 2018) . ues of λ illustrates the relevance of the LM component for topic coherence (DocNADE corresponds to λ=0). Similarly, the inclusion of word embeddings (i.e., ctx-DocNADEe) results in more coherent topics than the baseline DocNADE. Importantly, while ctx-DocNADEe is motivated by sparse data settings, the BNC dataset is neither a collection of short-text nor a corpus of few documents. Consequently, ctx-DocNADEe does not show improvements in topic coherence over ctx-DocNADE.

In Table 7 (right), we further qualitatively show the top 5 words of seven topics (topic name summarized by Wang et al. (2018) ) from TCNML and our models. Observe that ctx-DocNADE captures a topic expression that is a collection of only verbs in the past participle. Since the BNC dataset is unlabeled, we are here restricted to comparing model performance in terms of topic coherence only.

3.3. APPLICABILITY: TEXT RETRIEVAL AND CATEGORIZATION:

Text Retrieval: We perform a document retrieval task using the short-text and long-text documents with label information. We follow the experimental setup similar to Lauly et al. (2017) , where all test documents are treated as queries to retrieve a fraction of the closest documents in the original training set using cosine similarity measure between their textTOvec representations (section 2.2). To compute retrieval precision for each fraction (e.g., 0.0001, 0.005, 0.01, 0.02, 0.05, etc.), we average the number of retrieved training documents with the same label as the query. For multi-label datasets, we average the precision scores over multiple labels for each query. Since, Salakhutdinov & Hinton (2009) and Lauly et al. (2017) have shown that RSM and DocNADE strictly outperform LDA on this task, we solely compare DocNADE with our proposed extensions.

Table 3 and 4 show the retrieval precision scores for the short-text and long-text datasets, respectively at retrieval fraction 0.02. Observe that the introduction of both pre-trained embeddings and language/contextual information leads to improved performance on the IR task noticeably for short texts. We also investigate topic modeling without pre-processing and filtering certain words, i.e. the FV setting and find that the DocNADE(FV) or glove(FV) improves IR precision over the baseline RV setting. Therefore, we opt for the FV in the proposed extensions. On an average over the 8 shorttext and 6 long-text datasets, ctx-DocNADEe reports a gain of 7.1% (.630 vs .588) (Table 3 ) 6.0% (.601 vs .567) (Table 4 ), respectively in precision compared to DocNADE(RV). To further compare with TDLM, our proposed models (ctx-DocNADE and ctx-DocNADEe) outperform it by a notable margin for all the short-text datasets, i.e., a gain of 14.5% (.630 vs .550: ctx-DocNADEe vs TDLM) Figures (3a, 3b, 3c, 3d, 3e and 3f) illustrate the average precision for the retrieval task on 6 datasets. Observe that the ctx-DocNADEe outperforms DocNADE(RV) at all the fractions and demonstrates a gain of 6.5% (.615 vs .577) in precision at fraction 0.02, averaged over 14 datasets. Additionally, our proposed models outperform TDLM and ProdLDA6 (for 20NS) by noticeable margins.

Text Categorization::

We perform text categorization to measure the quality of our textTovec representations. We consider the same experimental setup as in the retrieval task and extract textTOvec of 200 dimension for each document, learned during the training of ctx-DocNADE variants. To perform text categorization, we employ a logistic regression classifier with L2 regularization. While, ctx-DocNADEe and ctx-DeepDNEe make use of glove embeddings, they are evaluated against the topic model baselines with embeddings. For the short texts (Table 3 ), the glove leads DocNADE in classification performance, suggesting a need for distributional priors in the topic model. Therefore, the ctx-DocNADEe reports a gain of 4.8% (.705 vs .673) and 3.6%(.618 vs .596) in F 1, compared to DocNADE(RV) on an average over the short (Table 3 ) and long (Table 4 ) texts, respectively. In result, a gain of 4.4% (.662 vs .634) overall.

In terms of classification accuracy on 20NS dataset, the scores are: DocNADE (0.734), ctx-DocNADE (0.744), ctx-DocNADEe (0.751), NTM (0.72) and SCHOLAR (0.71). While, our proposed models, i.e., ctx-DocNADE and ctx-DocNADEe outperform both NTM (results taken from Cao et al. (2015) , Figure 2 ) and SCHOLAR (results taken from Card et al. (2017) , Table 2 ), the DocNADE establishes itself as a strong neural topic model baseline.

3.4. INSPECTION OF LEARNED REPRESENTATIONS:

To further interpret the topic models, we analyze the meaningful semantics captured via topic extraction. Table 8 shows a topic extracted using 20NS dataset that could be interpreted as computers, which are (sub)categories in the data, confirming that meaningful topics are captured. Observe that Additionally, we show the quality of representations learned at different fractions (20%, 40%, 60%, 80%, 100%) of training set from TMNtitle data and use the same experimental setup for the IR and classification tasks, as in section 3.3. In Figure 4 , we quantify the quality of representations learned and demonstrate improvements due to the proposed models, i.e., ctx-DocNADE and ctx-DocNADEe over DocNADE at different fractions of the training data. Observe that the gains in both the tasks are large for smaller fractions of the datasets. For instance, one of the proposed models, i.e., ctx-DocNADEe (vs DocNADE) reports: (1) a precision (at 0.02 fraction) of 0.580 vs 0.444 at 20% and 0.595 vs 0.525 at 100% of the training set, and (2) an F1 of 0.711 vs 0.615 at 20% and 0.726 vs 0.688 at 100% of the training set. Therefore, the findings conform to our second contribution of improving topic models with word embeddings, especially in the sparse data setting.

3.5. CONCLUSION:

In this work, we have shown that accounting for language concepts such as word ordering, syntactic and semantic information in neural autoregressive topic models helps to better estimate the probability of a word in a given context. To this end, we have combined a neural autoregressive topic-(i.e., DocNADE) and a neural language (e.g., LSTM-LM) model in a single probabilistic framework with an aim to introduce language concepts in each of the autoregressive steps of the topic model. This facilitates learning a latent representation from the entire document whilst accounting for the local dynamics of the collocation patterns, encoded in the internal states of LSTM-LM. We further augment this complementary learning with external knowledge by introducing word embeddings. Our experimental results show that our proposed modeling approaches consistently outperform stateof-the-art generative topic models, quantified by generalization (perplexity), topic interpretability (coherence), and applicability (text retrieval and categorization) on 15 datasets.

Label: training Instructors shall have tertiary education and experience in the operation and maintenance of the equipment or sub-system of Plant. They shall be proficient in the use of the English language both written and oral. They shall be able to deliver instructions clearly and systematically. The curriculum vitae of the instructors shall be submitted for acceptance by the Engineer at least 8 weeks before the commencement of any training.

Label: maintenance The Contractor shall provide experienced staff for 24 hours per Day, 7 Days per week, throughout the Year, for call out to carry out On-call Maintenance for the Signalling System.

Label: cables Unless otherwise specified, this standard is applicable to all cables which include single and multi-core cables and wires, Local Area Network (LAN) cables and Fibre Optic (FO) cables.

Label: installation The Contractor shall provide and permanently install the asset labels onto all equipment supplied under this Contract. The Contractor shall liaise and co-ordinate with the Engineer for the format and the content of the labels. The Contractor shall submit the final format and size of the labels as well as the installation layout of the labels on the respective equipment, to the Engineer for acceptance.

Label: operations, interlocking It shall be possible to switch any station Interlocking capable of reversing the service into "Auto-Turnaround Operation". This facility once selected shall automatically route Trains into and out of these stations, independently of the ATS system. At stations where multiple platforms can be used to reverse the service it shall be possible to select one or both platforms for the service reversal. The SiROBs is our industrial corpus, extracted from industrial tender documents. The documents contain requirement specifications for an industrial project for example, railway metro construction.

There are 22 types of requirements i.e. class labels (multi-class), where a requirement is a paragraph or collection of paragraphs within a document. We name the requirement as Requirement Objects (ROBs). Some of the requirement types are project management, testing, legal, risk analysis, financial cost, technical requirement, etc. We need to classify the requirements in the tender documents and assign each ROB to a relevant department(s). See Table 12 for the hyperparameters in the document retrieval task.

B.3 EXPERIMENTAL SETUP FOR DOC2VEC MODEL:

We used gensim (https://github.com/RaRe-Technologies/gensim) to train Doc2Vec models for 12 datasets. Models were trained with distributed bag of words, for 1000 iterations using a window size of 5 and a vector size of 500.

B.4 CLASSIFICATION TASK:

We used the same split in training/development/test as for training the Doc2Vec models (also same split as in IR task) and trained a regularized logistic regression classifier on the inferred document vectors to predict class labels. In the case of multilabel datasets (R21578,R21578title, RCV1V2), we used a one-vs-all approach. Models were trained with a liblinear solver using L2 regularization and accuracy and macro-averaged F1 score were computed on the test set to quantify predictive power.

B.5 EXPERIMENTAL SETUP FOR GLOVE-DMM AND GLOVE-LDA MODELS:

We used LFTM (https://github.com/datquocnguyen/LFTM) to train glove-DMM and glove-LDA models. Models were trained for 200 iterations with 2000 initial iterations using 200 topics. For short texts we set the hyperparameter beta to 0.1, for long texts to 0.01; the mixture parameter lambda was set to 0.6 for all datasets. The setup for the classification task was the same as for doc2vec; classification was performed using relative topic proportions as input (i.e. we inferred the topic distribution of the training and test documents and used the relative distribution as input D ADDITIONAL BASELINES D.1 DOCNADE VS SCHOLAR PPL scores over 20 topics: DocNADE (752) and SCHOLAR (921), i.e., DocNADE outperforms SCHOLAR in terms of generalization.

Topic coherence (NPMI) using 20 topics: DocNADE (.18) and SCHOLAR (.35), i.e., SCHOLAR (Card et al., 2017) generates more coherence topics than DocNADE, though worse in PPL and text classification (see section 3.3) than DocNADE, ctx-DocNADE and ctx-DocNADEe.

IR tasks: Since, SCHOLAR (Card et al., 2017) without meta-data equates to ProdLDA and we have shown in section 3.3 that ProdLDA is worse on IR tasks than our proposed models, therefore one can infer the performance of SCHOLAR on IR task.

The experimental results above suggest that the DocNADE is better than SCHOLAR in generating good representations for downstream tasks such as information retrieval or classification, however falls behind SCHOLAR in interpretability. The investigation opens up an interesting direction for future research.

:

focuses on incorporating meta-data (author, date, etc.) into TMs; SCHOLAR w/o meta-data ≡ ProdLDA

though focused on improving language models using topic models, different to our motivation

https://github.com/jhlau/topically-driven-language-model

PPL scores in (RV/FV) settings: DocNADE (665/1375) outperforms ProdLDA (1168/2097) on 200 topics

NPMI over (50/200) topics learned on 20NS by: ProdLDA (.24/.19) and DocNADE (.15/.12) in RV setting

IR-precision scores at 0.02 retrieval fraction on the short-text datasets by ProdLDA: 20NSshort (.08), TREC6 (.24), R21578title (.31), Subjectivity (.63) and Polarity (.51). Therefore, the DocNADE, ctx-DocNADE and ctx-DocNADEe outperform ProdLDA in both the settings: data sparsity and sufficient co-occurrences.

Figures and Tables:

Figure 1: (left): A topic-word distribution due to global exposure, obtained from the matrix W as row-vector. (middle): Nearest neighbors in semantics space, represented by W in its column vectors. (right): BoW and cosine similarity illustration in distributed embedding space.

Figure 2: (left): DocNADE for the document v. (right): ctx-DocNADEe for the observable corresponding to vi ∈ v. Blue colored lines signify the connections that share parameters. The observations (double circle) for each word vi are multinomial, where vi is the index in the vocabulary of the ith word of the document. h DN i

ctx-DocNADE and ctx-DocNADEe with related baselines based on: (1) word representation: glove(Pennington et al., 2014), where a document is represented by summing the embedding vectors of it's words, (2) document representation: doc2vec (Le & Mikolov, 2014), (3) LDA based BoW TMs: ProdLDA(Srivastava & Sutton, 2017) and SCHOLAR 1(Card et al., 2017) (4) neural BoW TMs: DocNADE and NTM(Cao et al., 2015) and , (5) TMs, including pre-trained word embeddings: Gauss-LDA (GaussianLDA)(Das et al., 2015), and glove-DMM, glove-LDA(Nguyen et al., 2015). (6) jointly 2 trained topic and language models: TDLM (Lau et al., 2017), Topic-RNN(Dieng et al., 2016) and TCNLM (

Figure 3: Retrieval performance (IR-precision) on 6 datasets at different fractions

labels are not used during training. The class labels are only used to check if the retrieved documents have the same class label as the query document. To perform document retrieval, we use the same train/development/test split of documents discussed in data statistics (experimental section) for all the datasets during learning.

stemming from the DocNADE-based and Data statistics: Short/long texts and/or small/large corpora from diverse domains. Symbols-Avg:

Table content: <table><tr><td/><td colspan="2">short-text</td><td/><td/><td/><td colspan="2">long-text</td></tr><tr><td>Data</td><td colspan="2">Train Val Test |RV| |FV|</td><td colspan="2">L C Domain</td><td>Data</td><td colspan="2">Train Val Test |RV| |FV|</td><td>L</td><td>C Domain</td></tr><tr><td colspan="4">20NSshort 1.3k 0.1k 0.5k 1.4k 1.4k 13.5 20</td><td colspan="3">News 20NSsmall 0.4k 0.2k 0.2k</td><td>2k 4555 187.5 20</td><td>News</td></tr><tr><td colspan="2">TREC6 5.5k 0.5k 0.5k</td><td colspan="2">2k 2295 9.8 6</td><td>Q&amp;A</td><td colspan="2">Reuters8 5.0k 0.5k 2.2k</td><td>2k 7654 102 8</td><td>News</td></tr><tr><td colspan="2">R21578title  † 7.3k 0.5k 3.0k</td><td colspan="2">2k 2721 7.3 90</td><td>News</td><td colspan="2">20NS 7.9k 1.6k 5.2k</td><td>2k 33770 107.5 20</td><td>News</td></tr><tr><td colspan="2">Subjectivity 8.0k .05k 2.0k</td><td colspan="2">2k 7965 23.1 2</td><td>Senti</td><td colspan="2">R21578  † 7.3k 0.5k 3.0k</td><td>2k 11396 128 90</td><td>News</td></tr><tr><td colspan="2">Polarity 8.5k .05k 2.1k</td><td colspan="2">2k 7157 21.0 2</td><td>Senti</td><td colspan="3">BNC 15.0k 1.0k 1.0k 9.7k 41370 1189 -</td><td>News</td></tr><tr><td colspan="2">TMNtitle 22.8k 2.0k 7.8k</td><td colspan="2">2k 6240 4.9 7</td><td>News</td><td colspan="2">SiROBs  † 27.0k 1.0k 10.5k</td><td>3k 9113</td><td>39 22</td><td>Indus</td></tr><tr><td colspan="2">TMN 22.8k 2.0k 7.8k</td><td colspan="2">2k 12867 19 7</td><td>News</td><td colspan="2">AGNews 118k 2.0k 7.6k</td><td>5k 34071</td><td>38 4</td><td>News</td></tr><tr><td colspan="2">AGnewstitle 118k 2.0k 7.6k</td><td colspan="2">5k 17125 6.8 4</td><td>News</td><td/><td/></tr></table>

IR-precision at fraction 0.02 and classification F 1 for long texts

Table content: <table><tr><td/><td>Model</td><td>PPL</td><td/><td>Model</td><td>PPL</td></tr><tr><td>Subjectivity Reuters8</td><td colspan="2">DocNADE 980 ctx-DocNADE 968 ctx-DocNADEe 966 DocNADE 283 ctx-DocNADE 276 ctx-DocNADEe 272</td><td>AGnewstitle 20NS</td><td>DocNADE 846 ctx-DocNADE 822 ctx-DocNADEe 820 DocNADE 1375 ctx-DocNADE 1358 ctx-DocNADEe 1361</td></tr><tr><td>TMNtitle</td><td colspan="2">DocNADE 1437 ctx-DocNADE 1430 ctx-DocNADEe 1427</td><td>20NSshort</td><td>DocNADE 646 ctx-DocNADE 656 ctx-DocNADEe 648</td></tr></table>

Generalization: PPL

Table content: <table/>

(Left): Topic coherence (NMPI) scores of different models for 50, 100 and 150 topics on BNC dataset.

Table content: <table/>

DocNADEe extracts a more coherent topic due to embedding priors. To qualitatively inspect the contribution of word embeddings and textTOvec representations in topic models, we analyse the text retrieved for each query using the representations learned from DocNADE and ctx-DoocNADEe models. Table9illustrates the retrieval of the top 3 texts for an input query, selected from TMNtitle dataset, where #match is YES if the query and retrievals have the same class label. Observe that ctx-DocNADEe retrieves the top 3 texts, each with no unigram overlap with the query. Illustration of the top-3 retrievals for an input query

Table content: <table><tr><td/><td>0.6</td><td/><td/><td/><td/><td/><td/><td/><td>0.73</td><td/><td/></tr><tr><td>IR-precision at fraction 0.02</td><td>0.42 0.44 0.46 0.48 0.5 0.52 0.54 0.56 0.58</td><td>20%</td><td>40%</td><td>60%</td><td colspan="2">80% ctx-DocNADEe 100% ctx-DocNADE DocNADE(FV)</td><td/><td>macro-F1 score</td><td>0.65 0.66 0.67 0.68 0.69 0.7 0.71 0.72 0.61 0.64 0.63 0.62</td><td>20%</td><td>40%</td><td>60%</td><td>80% ctx-DocNADEe 100% ctx-DocNADE DocNADE(FV)</td></tr><tr><td/><td/><td/><td colspan="3">Fraction of training set</td><td/><td/><td/><td/><td/><td colspan="2">Fraction of training set</td></tr><tr><td/><td/><td/><td colspan="3">(a) Text Retrieval</td><td/><td/><td/><td/><td/><td colspan="2">(b) Text classification</td></tr><tr><td colspan="13">Figure 4: Evaluations at different fractions (20%, 40%, 60%, 80%, 100%) of the training set of TMNtitle</td></tr><tr><td colspan="3">the ctx-DocNADE vga, screen, computer, sell, color, powerbook,</td><td colspan="2">ctx-DocNADE computer, color, screen, offer, vga, card,</td><td>ctx-DocNADEe svga, graphics bar, macintosh, san, windows,</td><td>ctx-</td><td>-DocNADEe</td><td colspan="5">Query :: "emerging economies move ahead nuclear plans" #match #IR1 :: imf sign lifting japan yen YES #IR2 :: japan recovery takes hold debt downgrade looms YES #IR3 :: japan ministers confident treasuries move YES</td></tr><tr><td colspan="3">sold, cars, svga, offer .554</td><td colspan="3">terminal, forsale, utility, monitor, gov, vesa computer, processor .624 .667</td><td/><td>DocNADE</td><td colspan="5">#IR1 :: nuclear regulator back power plans #IR2 :: defiant iran plans big rise nuclear #IR3 :: japan banks billion nuclear operator sources</td><td>NO NO YES</td></tr><tr><td colspan="6">Table 8: A topic of 20NS dataset with coherence</td><td/><td/><td/><td/><td/><td/></tr></table>

SiROBs data: Example Documents (Requirement Objects) with their types (label).

Table content: <table><tr><td>Hyperparameter</td><td>Search Space</td></tr><tr><td>learning rate</td><td>[0.001]</td></tr><tr><td>hidden units</td><td>[200]</td></tr><tr><td>iterations</td><td>[2000]</td></tr><tr><td>activation function</td><td>sigmoid</td></tr><tr><td>λ</td><td>[1.0, 0.8, 0.5, 0.3, 0.1, 0.01, 0.001]</td></tr></table>

Hyperparameters in Generalization in the DocNADE and ctx-DocNADE variants for 200 topics

Table content: <table/>

Therefore, we analyze such documents to automate decision making, tender comparison, similar tender as well as ROB retrieval and assigning ROBs to a relevant department(s) to optimize/expedite tender analysis. See some examples of ROBs from SiROBs corpus in Table10. EXPERIMENTAL SETUP AND HYPERPARAMETERS FOR IR TASK We set the maximum number of training passes to 1000, topics to 200 and the learning rate to 0.001 with tanh hidden activation. For model selection, we used the validation set as the query set and used the average precision at 0.02 retrieved documents as the performance measure. Note that the

Table content: <table><tr><td>B EXPERIMENTAL SETUP B.1 EXPERIMENTAL SETUP AND HYPERPARAMETERS FOR GENERALIZATION TASK See Table 11 for hyperparameters used in generalization. Search Space retrieval fraction [0.02] learning rate [0.001] hidden units [200] activation function tanh iterations [2000] B.2 Hyperparameter λ [1.0, 0.8, 0.5, 0.3, 0.1, 0.01, 0.001]</td></tr></table>

Hyperparameters in the Document Retrieval task.

Table content: <table><tr><td>Dataset</td><td>Model</td><td/><td>λ</td><td/></tr><tr><td/><td/><td>1.0</td><td>0.1</td><td>0.01</td></tr><tr><td>20NSshort</td><td>ctx-DocNADE ctx-DocNADEe</td><td colspan="2">899.04 829.5 890.3 828.8</td><td>842.1 832.4</td></tr><tr><td>Subjectivity</td><td>ctx-DocNADE ctx-DocNADEe</td><td>982.8 977.1</td><td>977.8 975.0</td><td>966.5 964.2</td></tr><tr><td>TMNtitle</td><td colspan="4">ctx-DocNADE ctx-DocNADEe 1877.7 1480.2 1484.7 1898.1 1482.7 1487.1</td></tr><tr><td>AGnewstitle</td><td colspan="3">ctx-DocNADE ctx-DocNADEe 1279.2 853.3 1296.1 861.1</td><td>865 862.9</td></tr><tr><td>Reuters-8</td><td>ctx-DocNADE ctx-DocNADEe</td><td>336.1 323.3</td><td>313.2 312.0</td><td>311.9 310.2</td></tr><tr><td>20NS</td><td colspan="4">ctx-DocNADE ctx-DocNADEe 1247.1 1211.6 1206.1 1282.1 1209.3 1207.2</td></tr></table>

Perplexity scores for different λ in Generalization task: Ablation over validation set

Table content: <table/>

Peer review (iclr2019):
 DocNADE has great performance so this is a welcome bit of research extending it. There has been a huge amount of activity in combining topic models with (1) embeddings and (2) neural networks such as LSTMs and RNNs. I will say I have great sympathy for the poor author trying to do fair comparisons against start-of-the-art because the standards are moving quickly. In this case, some neural network papers I have seen are TopicRNNs by Dieng, Wang Gao and Paisley, and LLA by Zaheer, Ahmed and Smola. The latter is still a bag-of-words model and but places the LSTM over the sequence of topic proportions. The Gauss-LDA and glove-DMM work is fairly dated (in our fast-paced ML world) and their performance is known to be poor, as some papers in 2017 show. Now I know historically LDA has been fairly poor with IR tasks, but I would expect the recent supervised LDA methods, some also have word embeddings, to do better as well. So the discussion of related work and comparative experiments are poor. If you want to illustrated good improvememts gained using embeddings, it helps to try different proportions, say 20/40/60/80% of a data set and plot. Usually, you should see embeddings aid performance dramatically for smaller fractions of data sets. Hence, your results seem strange. Note the data sets are all fairly small, which makes me wonder about the computation time. Could you give some computational performance stats for a data set? In section 2.2 top of page 5, why is it "pseudo" log likelihood? Isn*t that formula exact? The paper has a relatively small part devoted to the model, and virtually nothing on the algorithm, although this is probably covered in earlier DocNADE papers. I*m assuming the model is trained by SGD on the log likelihood with all the parameters shoved in there in one go. Is that right? Would be nice to mention whatever it is. The use of four different kinds of evaluations (classification, IR, perplexity, etc.) is good. Note that the improvement over the earlier DocNADE is quite small but clearly significant, and improvement of adding embeddings seems even smaller, though seems better for short texts. I wonder if the method for including embeddings is much good! Not fully convinced. AFTER RESPONSE: Wow guys, what a great revision. Thanks so much.