Introduction
With the rapid growth of the internet, huge amounts of text data are generated in social networks, online shopping and news websites, etc. These data create demand for powerful and efficient text analysis techniques. Probabilistic topic models such as Latent Dirichlet Allocation (LDA) BIBREF0 are popular approaches for this task, by discovering latent topics from text collections. Many conventional topic models discover topics purely based on the word-occurrences, ignoring the meta information (a.k.a., side information) associated with the content. In contrast, when we humans read text it is natural to leverage meta information to improve our comprehension, which includes categories, authors, timestamps, the semantic meanings of the words, etc. Therefore, topic models capable of using meta information should yield improved modelling accuracy and topic quality.
In practice, various kinds of meta information are available at the document level and the word level in many corpora. At the document level, labels of documents can be used to guide topic learning so that more meaningful topics can be discovered. Moreover, it is highly likely that documents with common labels discuss similar topics, which could further result in similar topic distributions. For example, if we use authors as labels for scientific papers, the topics of the papers published by the same researcher can be closely related.
At the word level, different semantic/syntactic features are also accessible. For example, there are features regarding word relationships, such as synonyms obtained from WordNet BIBREF1 , word co-occurrence patterns obtained from a large corpus, and linked concepts from knowledge graphs. It is preferable that words having similar meaning but different morphological forms, like “dog” and “puppy”, are assigned to the same topic, even if they barely co-occur in the modelled corpus. Recently, word embeddings generated by GloVe BIBREF2 and word2vec BIBREF3 , have attracted a lot of attention in natural language processing and related fields. It has been shown that the word embeddings can capture both the semantic and syntactic features of words so that similar words are close to each other in the embedding space. It seems reasonable to expect that these word embedding will improve topic modelling BIBREF4 , BIBREF5 .
Conventional topic models can suffer from a large performance degradation over short texts (e.g., tweets and news headlines) because of insufficient word co-occurrence information. In such cases, meta information of documents and words can play an important role in analysing short texts by compensating the lost information in word co-occurrences. At the document level, for example, tweets are usually associated with hashtags, users, locations, and timestamps, which can be used to alleviate the data sparsity problem. At the word level, word semantic similarity and embeddings obtained or trained on large external corpus (e.g., Google News or Wikipedia) have been proven useful in learning meaningful topics from short texts BIBREF6 , BIBREF7 .
The benefit of using document and word meta information separately is shown in several models such as BIBREF8 , BIBREF9 , BIBREF5 . However, in existing models this is usually not efficient enough due to non-conjugacy and/or complex model structures. Moreover, only one kind of meta information (either at document level or at word level) is used in most existing models. In this paper, we propose MetaLDA, a topic model that can effectively and efficiently leverage arbitrary document and word meta information encoded in binary form. Specifically, the labels of a document in MetaLDA are incorporated in the prior of the per-document topic distributions. If two documents have similar labels, their topic distributions should be generated with similar Dirichlet priors. Analogously, at the word level, the features of a word are incorporated in the prior of the per-topic word distributions, which encourages words with similar features to have similar weights across topics. Therefore, both document and word meta information, if and when they are available, can be flexibly and simultaneously incorporated using MetaLDA. MetaLDA has the following key properties:
We conduct extensive experiments with several real datasets including regular and short texts in various domains. The experimental results demonstrate that MetaLDA achieves improved performance in terms of perplexity, topic coherence, and running time.
Related Work
In this section, we review three lines of related work: models with document meta information, models with word meta information, and models for short texts.
At the document level, Supervised LDA (sLDA) BIBREF10 models document labels by learning a generalised linear model with an appropriate link function and exponential family dispersion function. But the restriction for sLDA is that one document can only have one label. Labelled LDA (LLDA) BIBREF11 assumes that each label has a corresponding topic and a document is generated by a mixture of the topics. Although multiple labels are allowed, LLDA requires that the number of topics must equal to the number of labels, i.e., exactly one topic per label. As an extension to LLDA, Partially Labelled LDA (PLLDA) BIBREF9 relaxes this requirement by assigning multiple topics to a label. The Dirichlet Multinomial Regression (DMR) model BIBREF8 incorporates document labels on the prior of the topic distributions like our MetaLDA but with the logistic-normal transformation. As full conjugacy does not exist in DMR, a part of the inference has to be done by numerical optimisation, which is slow for large sets of labels and topics. Similarly, in the Hierarchical Dirichlet Scaling Process (HDSP) BIBREF12 , conjugacy is broken as well since the topic distributions have to be renormalised. BIBREF13 introduces a Poisson factorisation model with hierarchical document labels. But the techniques cannot be applied to regular topic models as the topic proportion vectors are also unnormalised.
Recently, there is growing interest in incorporating word features in topic models. For example, DF-LDA BIBREF14 incorporates word must-links and cannot-links using a Dirichlet forest prior in LDA; MRF-LDA BIBREF15 encodes word semantic similarity in LDA with a Markov random field; WF-LDA BIBREF16 extends LDA to model word features with the logistic-normal transform; LF-LDA BIBREF5 integrates word embeddings into LDA by replacing the topic-word Dirichlet multinomial component with a mixture of a Dirichlet multinomial component and a word embedding component; Instead of generating word types (tokens), Gaussian LDA (GLDA) BIBREF4 directly generates word embeddings with the Gaussian distribution. Despite the exciting applications of the above models, their inference is usually less efficient due to the non-conjugacy and/or complicated model structures.
Analysis of short text with topic models has been an active area with the development of social networks. Generally, there are two ways to deal with the sparsity problem in short texts, either using the intrinsic properties of short texts or leveraging meta information. For the first way, one popular approach is to aggregate short texts into pseudo-documents, for example, BIBREF17 introduces a model that aggregates tweets containing the same word; Recently, PTM BIBREF18 aggregates short texts into latent pseudo documents. Another approach is to assume one topic per short document, known as mixture of unigrams or Dirichlet Multinomial Mixture (DMM) such as BIBREF19 , BIBREF6 . For the second way, document meta information can be used to aggregate short texts, for example, BIBREF17 aggregates tweets by the corresponding authors and BIBREF20 shows that aggregating tweets by their hashtags yields superior performance over other aggregation methods. One closely related work to ours is the models that use word features for short texts. For example, BIBREF6 introduces an extension of GLDA on short texts which samples an indicator variable that chooses to generate either the type of a word or the embedding of a word and GPU-DMM BIBREF7 extends DMM with word semantic similarity obtained from embeddings for short texts. Although with improved performance there still exists challenges for existing models: (1) for aggregation-based models, it is usually hard to choose which meta information to use for aggregation; (2) the “single topic” assumption makes DMM models lose the flexibility to capture different topic ingredients of a document; and (3) the incorporation of meta information in the existing models is usually less efficient.
To our knowledge, the attempts that jointly leverage document and word meta information are relatively rare. For example, meta information can be incorporated by first-order logic in Logit-LDA BIBREF21 and score functions in SC-LDA BIBREF22 . However, the first-order logic and score functions need to be defined for different kinds of meta information and the definition can be infeasible for incorporating both document and word meta information simultaneously.
The MetaLDA Model
Given a corpus, LDA uses the same Dirichlet prior for all the per-document topic distributions and the same prior for all the per-topic word distributions BIBREF23 . While in MetaLDA, each document has a specific Dirichlet prior on its topic distribution, which is computed from the meta information of the document, and the parameters of the prior are estimated during training. Similarly, each topic has a specific Dirichlet prior computed from the word meta information. Here we elaborate our MetaLDA, in particular on how the meta information is incorporated. Hereafter, we will use labels as document meta information, unless otherwise stated.
Given a collection of INLINEFORM0 documents INLINEFORM1 , MetaLDA generates document INLINEFORM2 with a mixture of INLINEFORM3 topics and each topic INLINEFORM4 is a distribution over the vocabulary with INLINEFORM5 tokens, denoted by INLINEFORM6 . For document INLINEFORM7 with INLINEFORM8 words, to generate the INLINEFORM9 ( INLINEFORM10 ) word INLINEFORM11 , we first sample a topic INLINEFORM12 from the document's topic distribution INLINEFORM13 , and then sample INLINEFORM14 from INLINEFORM15 . Assume the labels of document INLINEFORM16 are encoded in a binary vector INLINEFORM17 where INLINEFORM18 is the total number of unique labels. INLINEFORM19 indicates label INLINEFORM20 is active in document INLINEFORM21 and vice versa. Similarly, the INLINEFORM22 features of token INLINEFORM23 are stored ∂in a binary vector INLINEFORM24 . Therefore, the document and word meta information associated with INLINEFORM25 are stored in the matrix INLINEFORM26 and INLINEFORM27 respectively. Although MetaLDA incorporates binary features, categorical features and real-valued features can be converted into binary values with proper transformations such as discretisation and binarisation.
Fig. FIGREF5 shows the graphical model of MetaLDA and the generative process is as following:
where INLINEFORM0 , INLINEFORM1 , INLINEFORM2 are the gamma distribution, the Dirichlet distribution, and the categorical distribution respectively. INLINEFORM3 , INLINEFORM4 , and INLINEFORM5 are the hyper-parameters.
To incorporate document labels, MetaLDA learns a specific Dirichlet prior over the topics for each document by using the label information. Specifically, the information of document INLINEFORM0 's labels is incorporated in INLINEFORM1 , the parameter of Dirichlet prior on INLINEFORM2 . As shown in Step UID12 , INLINEFORM3 is computed as a log linear combination of the labels INLINEFORM4 . Since INLINEFORM5 is binary, INLINEFORM6 is indeed the multiplication of INLINEFORM7 over all the active labels of document INLINEFORM8 , i.e., INLINEFORM9 . Drawn from the gamma distribution with mean 1, INLINEFORM10 controls the impact of label INLINEFORM11 on topic INLINEFORM12 . If label INLINEFORM13 has no or less impact on topic INLINEFORM14 , INLINEFORM15 is expected to be 1 or close to 1, and then INLINEFORM16 will have no or little influence on INLINEFORM17 and vice versa. The hyper-parameter INLINEFORM18 controls the variation of INLINEFORM19 . The incorporation of word features is analogous but in the parameter of the Dirichlet prior on the per-topic word distributions as shown in Step UID9 .
The intuition of our way of incorporating meta information is: At the document level, if two documents have more labels in common, their Dirichlet parameter INLINEFORM0 will be more similar, resulting in more similar topic distributions INLINEFORM1 ; At the word level, if two words have similar features, their INLINEFORM2 in topic INLINEFORM3 will be similar and then we can expect that their INLINEFORM4 could be more or less the same. Finally, the two words will have similar probabilities of showing up in topic INLINEFORM5 . In other words, if a topic “prefers” a certain word, we expect that it will also prefer other words with similar features to that word. Moreover, at both the document and the word level, different labels/features may have different impact on the topics ( INLINEFORM6 / INLINEFORM7 ), which is automatically learnt in MetaLDA.
Inference
Unlike most existing methods, our way of incorporating the meta information facilitates the derivation of an efficient Gibbs sampling algorithm. With two data augmentation techniques (i.e., the introduction of auxiliary variables), MetaLDA admits the local conjugacy and a close-form Gibbs sampling algorithm can be derived. Note that MetaLDA incorporates the meta information on the Dirichlet priors, so we can still use LDA's collapsed Gibbs sampling algorithm for the topic assignment INLINEFORM0 . Moreover, Step UID12 and UID9 show that one only needs to consider the non-zero entries of INLINEFORM1 and INLINEFORM2 in computing the full conditionals, which further reduces the inference complexity.
Similar to LDA, the complete model likelihood (i.e., joint distribution) of MetaLDA is: +rCl+x* k=1K v=1V k,vnk,v d=1D k=1K d,kmd,k where INLINEFORM0 , INLINEFORM1 , and INLINEFORM2 is the indicator function.
Sampling λ l,k \lambda _{l,k}:
To sample INLINEFORM0 , we first marginalise out INLINEFORM1 in the right part of Eq. ( SECREF4 ) with the Dirichlet multinomial conjugacy: +rCl+x* d=1D (d,)(d, + md,)Gamma ratio 1 k=1K (d,k + md,k)(d,k)Gamma ratio 2 where INLINEFORM2 , INLINEFORM3 , and INLINEFORM4 is the gamma function. Gamma ratio 1 in Eq. ( SECREF17 ) can be augmented with a set of Beta random variables INLINEFORM5 as: +rCl+x* (d,)(d, + md,)Gamma ratio 1 qd qdd,-1 (1-qd)md,-1 where for each document INLINEFORM6 , INLINEFORM7 . Given a set of INLINEFORM8 for all the documents, Gamma ratio 1 can be approximated by the product of INLINEFORM9 , i.e., INLINEFORM10 .
Gamma ratio 2 in Eq. ( SECREF17 ) is the Pochhammer symbol for a rising factorial, which can be augmented with an auxiliary variable INLINEFORM0 BIBREF24 , BIBREF25 , BIBREF26 , BIBREF27 as follows: +rCl+x* (d,k + md,k)(d,k)Gamma ratio 2 = td,k=0md,k Smd,ktd,k d,ktd,k where INLINEFORM1 indicates an unsigned Stirling number of the first kind. Gamma ratio 2 is a normalising constant for the probability of the number of tables in the Chinese Restaurant Process (CRP) BIBREF28 , INLINEFORM2 can be sampled by a CRP with INLINEFORM3 as the concentration and INLINEFORM4 as the number of customers: +rCl+x* td,k = i=1md,k Bern(d,kd,k+i) where INLINEFORM5 samples from the Bernoulli distribution. The complexity of sampling INLINEFORM6 by Eq. ( SECREF17 ) is INLINEFORM7 . For large INLINEFORM8 , as the standard deviation of INLINEFORM9 is INLINEFORM10 BIBREF28 , one can sample INLINEFORM11 in a small window around the current value in complexity INLINEFORM12 .
By ignoring the terms unrelated to INLINEFORM0 , the augmentation of Eq. ( SECREF17 ) can be simplified to a single term INLINEFORM1 . With auxiliary variables now introduced, we simplify Eq. ( SECREF17 ) to: +rCl+x* d=1D k=1K qdd,k d,ktd,k
Replacing INLINEFORM0 with INLINEFORM1 , we can get: +rCl+x* d=1D k=1K e- d,k 1qd l=1Ldoc k=1K l,kd=1D fd,l td,k
Recall that all the document labels are binary and INLINEFORM0 is involved in computing INLINEFORM1 iff INLINEFORM2 . Extracting all the terms related to INLINEFORM3 in Eq. ( SECREF17 ), we get the marginal posterior of INLINEFORM4 : +rCl+x* e- l,k d=1:fd,l=1D 1qd d,kl,k l,k d=1D fd,l td,k where INLINEFORM5 is the value of INLINEFORM6 with INLINEFORM7 removed when INLINEFORM8 . With the data augmentation techniques, the posterior is transformed into a form that is conjugate to the gamma prior of INLINEFORM9 . Therefore, it is straightforward to yield the following sampling strategy for INLINEFORM10 : +rCl+x* l,k Ga( ', 1/”)
' = 0 + d=1: fd,l = 1D td,k
” = 1/0 - d=1:fd,l=1D d,kl,k qd
We can compute and cache the value of INLINEFORM0 first. After INLINEFORM1 is sampled, INLINEFORM2 can be updated by: +rCl+x* d,k d,k 'l,kl,k 1 d D : fd,l = 1 where INLINEFORM3 is the newly-sampled value of INLINEFORM4 .
To sample/compute Eqs. ( SECREF17 )-( SECREF17 ), one only iterates over the documents where label INLINEFORM0 is active (i.e., INLINEFORM1 ). Thus, the sampling for all INLINEFORM2 takes INLINEFORM3 where INLINEFORM4 is the average number of documents where a label is active (i.e., the column-wise sparsity of INLINEFORM5 ). It is usually that INLINEFORM6 because if a label exists in nearly all the documents, it provides little discriminative information. This demonstrates how the sparsity of document meta information is leveraged. Moreover, sampling all the tables INLINEFORM7 takes INLINEFORM8 ( INLINEFORM9 is the total number of words in INLINEFORM10 ) which can be accelerated with the window sampling technique explained above.
Sampling δ l ' ,k \delta _{l^{\prime },k}:
Since the derivation of sampling INLINEFORM0 is analogous to INLINEFORM1 , we directly give the sampling formulas: +rCl+x* l',k Ga( ', 1/”)
' = 0 + v=1: gv,l' = 1V t'k,v
” = 1/0 - q'k v=1:gv,l'=1V k,vl',k where the two auxiliary variables can be sampled by: INLINEFORM0 and INLINEFORM1 . Similarly, sampling all INLINEFORM2 takes INLINEFORM3 where INLINEFORM4 is the average number of tokens where a feature is active (i.e., the column-wise sparsity of INLINEFORM5 and usually INLINEFORM6 ) and sampling all the tables INLINEFORM7 takes INLINEFORM8 .
Sampling topic z d,i z_{d,i}:
Given INLINEFORM0 and INLINEFORM1 , the collapsed Gibbs sampling of a new topic for a word INLINEFORM2 in MetaLDA is: +rCl+x* Przd,i = k (d,k + md,k) k,v + nk,vk, + nk, which is exactly the same to LDA.
Experiments
In this section, we evaluate the proposed MetaLDA against several recent advances that also incorporate meta information on 6 real datasets including both regular and short texts. The goal of the experimental work is to evaluate the effectiveness and efficiency of MetaLDA's incorporation of document and word meta information both separately and jointly compared with other methods. We report the performance in terms of perplexity, topic coherence, and running time per iteration.
Datasets
In the experiments, three regular text datasets and three short text datasets were used:
Reuters is widely used corpus extracted from the Reuters-21578 dataset where documents without any labels are removed. There are 11,367 documents and 120 labels. Each document is associated with multiple labels. The vocabulary size is 8,817 and the average document length is 73.
20NG, 20 Newsgroup, a widely used dataset consists of 18,846 news articles with 20 categories. The vocabulary size is 22,636 and the average document length is 108.
NYT, New York Times is extracted from the documents in the category “Top/News/Health” in the New York Times Annotated Corpus. There are 52,521 documents and 545 unique labels. Each document is with multiple labels. The vocabulary contains 21,421 tokens and there are 442 words in a document on average.
WS, Web Snippet, used in BIBREF7 , contains 12,237 web search snippets and each snippet belongs to one of 8 categories. The vocabulary contains 10,052 tokens and there are 15 words in one snippet on average.
TMN, Tag My News, used in BIBREF5 , consists of 32,597 English RSS news snippets from Tag My News. With a title and a short description, each snippet belongs to one of 7 categories. There are 13,370 tokens in the vocabulary and the average length of a snippet is 18.
AN, ABC News, is a collection of 12,495 short news descriptions and each one is in multiple of 194 categories. There are 4,255 tokens in the vocabulary and the average length of a description is 13.
All the datasets were tokenised by Mallet and we removed the words that exist in less than 5 documents and more than 95% documents.
Meta Information Settings
Document labels and word features. At the document level, the labels associated with documents in each dataset were used as the meta information. At the word level, we used a set of 100-dimensional binarised word embeddings as word features[fn-pre-process], which were obtained from the 50-dimensional GloVe word embeddings pre-trained on Wikipedia. To binarise word embeddings, we first adopted the following method similar to BIBREF29 : +rCl+x* g'v,j = {ll 1, if g”v,j > Mean+(g”v)
-1, if g”v,j < Mean-(g”v)
0, otherwise . where INLINEFORM0 is the original embedding vector for word INLINEFORM1 , INLINEFORM2 is the binarised value for INLINEFORM3 element of INLINEFORM4 , and INLINEFORM5 and INLINEFORM6 are the average value of all the positive elements and negative elements respectively. The insight is that we only consider features with strong opinions (i.e., large positive or negative value) on each dimension. To transform INLINEFORM7 to the final INLINEFORM8 , we use two binary bits to encode one dimension of INLINEFORM9 : the first bit is on if INLINEFORM10 and the second is on if INLINEFORM11 . Besides, MetaLDA can work with other word features such as semantic similarity as well.
Default feature. Besides the labels/features associated with the datasets, a default label/feature for each document/word is introduced in MetaLDA, which is always equal to 1. The default can be interpreted as the bias term in INLINEFORM0 / INLINEFORM1 , which captures the information unrelated to the labels/features. While there are no document labels or word features, with the default, MetaLDA is equivalent in model to asymmetric-asymmetric LDA of BIBREF23 .
Compared Models and Parameter Settings
We evaluate the performance of the following models:
MetaLDA and its variants: the proposed model and its variants. Here we use MetaLDA to indicate the model considering both document labels and word features. Several variants of MetaLDA with document labels and word features separately were also studied, which are shown in Table TABREF32 . These variants differ in the method of estimating INLINEFORM0 and INLINEFORM1 . All the models listed in Table TABREF32 were implemented on top of Mallet. The hyper-parameters INLINEFORM2 and INLINEFORM3 were set to INLINEFORM4 .
LDA BIBREF0 : the baseline model. The Mallet implementation of SparseLDA BIBREF30 is used.
LLDA, Labelled LDA BIBREF11 and PLLDA, Partially Labelled LDA BIBREF9 : two models that make use of multiple document labels. The original implementation is used.
DMR, LDA with Dirichlet Multinomial Regression BIBREF8 : a model that can use multiple document labels. The Mallet implementation of DMR based on SparseLDA was used. Following Mallet, we set the mean of INLINEFORM0 to 0.0 and set the variances of INLINEFORM1 for the default label and the document labels to 100.0 and 1.0 respectively.
WF-LDA, Word Feature LDA BIBREF16 : a model with word features. We implemented it on top of Mallet and used the default settings in Mallet for the optimisation.
LF-LDA, Latent Feature LDA BIBREF5 : a model that incorporates word embeddings. The original implementation was used. Following the paper, we used 1500 and 500 MCMC iterations for initialisation and sampling respectively and set INLINEFORM0 to 0.6, and used the original 50-dimensional GloVe word embeddings as word features.
GPU-DMM, Generalized Pólya Urn DMM BIBREF7 : a model that incorporates word semantic similarity. The original implementation was used. The word similarity was generated from the distances of the word embeddings. Following the paper, we set the hyper-parameters INLINEFORM0 and INLINEFORM1 to 0.1 and 0.7 respectively, and the symmetric document Dirichlet prior to INLINEFORM2 .
PTM, Pseudo document based Topic Model BIBREF18 : a model for short text analysis. The original implementation was used. Following the paper, we set the number of pseudo documents to 1000 and INLINEFORM0 to 0.1.
All the models, except where noted, the symmetric parameters of the document and the topic Dirichlet priors were set to 0.1 and 0.01 respectively, and 2000 MCMC iterations are used to train the models.
Perplexity Evaluation
Perplexity is a measure that is widely used BIBREF23 to evaluate the modelling accuracy of topic models. The lower the score, the higher the modelling accuracy. To compute perplexity, we randomly selected some documents in a dataset as the training set and the remaining as the test set. We first trained a topic model on the training set to get the word distributions of each topic INLINEFORM0 ( INLINEFORM1 ). Each test document INLINEFORM2 was split into two halves containing every first and every second words respectively. We then fixed the topics and trained the models on the first half to get the topic proportions ( INLINEFORM3 ) of test document INLINEFORM4 and compute perplexity for predicting the second half. In regard to MetaLDA, we fixed the matrices INLINEFORM5 and INLINEFORM6 output from the training procedure. On the first half of test document INLINEFORM7 , we computed the Dirichlet prior INLINEFORM8 with INLINEFORM9 and the labels INLINEFORM10 of test document INLINEFORM11 (See Step UID12 ), and then point-estimated INLINEFORM12 . We ran all the models 5 times with different random number seeds and report the average scores and the standard deviations.
In testing, we may encounter words that never occur in the training documents (a.k.a., unseen words or out-of-vocabulary words). There are two strategies for handling unseen words for calculating perplexity on test documents: ignoring them or keeping them in computing the perplexity. Here we investigate both strategies:
In this experiment, the perplexity is computed only on the words that appear in the training vocabulary. Here we used 80% documents in each dataset as the training set and the remaining 20% as the test set.
Tables TABREF48 and TABREF49 show: the average perplexity scores with standard deviations for all the models. Note that: (1) The scores on AN with 150 and 200 topics are not reported due to overfitting observed in all the compared models. (2) Given the size of NYT, the scores of 200 and 500 topics are reported. (3) The number of latent topics in LLDA must equal to the number of document labels. (4) For PLLDA, we varied the number of topics per label from 5 to 50 (2 and 5 topics on NYT). The number of topics in PPLDA is the product of the numbers of labels and topics per label.
The results show that MetaLDA outperformed all the competitors in terms of perplexity on nearly all the datasets, showing the benefit of using both document and word meta information. Specifically, we have the following remarks:
By looking at the models using only the document-level meta information, we can see the significant improvement of these models over LDA, which indicates that document labels can play an important role in guiding topic modelling. Although the performance of the two variants of MetaLDA with document labels and DMR is comparable, our models runs much faster than DMR, which will be studied later in Section SECREF67 .
It is interesting that PLLDA with 50 topics for each label has better perplexity than MetaLDA with 200 topics in the 20NG dataset. With the 20 unique labels, the actual number of topics in PLLDA is 1000. However, if 10 topics for each label in PLLDA are used, which is equivalent to 200 topics in MetaLDA, PLLDA is outperformed by MetaLDA significantly.
At the word level, MetaLDA-def-wf performed the best among the models with word features only. Moreover, our model has obvious advantage in running speed (see Table TABREF66 ). Furthermore, comparing MetaLDA-def-wf with MetaLDA-def-def and MetaLDA-0.1-wf with LDA, we can see using the word features indeed improved perplexity.
The scores show that the improvement gained by MetaLDA over LDA on the short text datasets is larger than that on the regular text datasets. This is as expected because meta information serves as complementary information in MetaLDA and can have more significant impact when the data is sparser.
It can be observed that models usually gained improved perplexity, if INLINEFORM0 is sampled/optimised, in line with BIBREF23 .
On the AN dataset, there is no statistically significant difference between MetaLDA and DMR. On NYT, a similar trend is observed: the improvement in the models with the document labels over LDA is obvious but not in the models with the word features. Given the number of the document labels (194 of AN and 545 of NYT), it is possible that the document labels already offer enough information and the word embeddings have little contribution in the two datasets.
To test the hypothesis that the incorporation of meta information in MetaLDA can significantly improve the modelling accuracy in the cases where the corpus is sparse, we varied the proportion of documents used in training from 20% to 80% and used the remaining for testing. It is natural that when the proportion is small, the number of unseen words in testing documents will be large. Instead of simply excluding the unseen words in the previous experiments, here we compute the perplexity with unseen words for LDA, DMR, WF-LDA and the proposed MetaLDA. For perplexity calculation, INLINEFORM0 for each topic INLINEFORM1 and each token INLINEFORM2 in the test documents is needed. If INLINEFORM3 occurs in the training documents, INLINEFORM4 can be directly obtained. While if INLINEFORM5 is unseen, INLINEFORM6 can be estimated by the prior: INLINEFORM7 . For LDA and DMR which do not use word features, INLINEFORM8 ; For WF-LDA and MetaLDA which are with word features, INLINEFORM9 is computed with the features of the unseen token. Following Step UID9 , for MetaLDA, INLINEFORM10 .
Figure FIGREF51 shows the perplexity scores on Reuters, 20NG, TMN and WS with 200, 200, 100 and 50 topics respectively. MetaLDA outperformed the other models significantly with a lower proportion of training documents and relatively higher proportion of unseen words. The gap between MetaLDA and the other three models increases while the training proportion decreases. It indicates that the meta information helps MetaLDA to achieve better modelling accuracy on predicting unseen words.
Topic Coherence Evaluation
We further evaluate the semantic coherence of the words in a topic learnt by LDA, PTM, DMR, LF-LDA, WF-LDA, GPU-DMM and MetaLDA. Here we use the Normalised Pointwise Mutual Information (NPMI) BIBREF31 , BIBREF32 to calculate topic coherence score for topic INLINEFORM0 with top INLINEFORM1 words: INLINEFORM2 , where INLINEFORM3 is the probability of word INLINEFORM4 , and INLINEFORM5 is the joint probability of words INLINEFORM6 and INLINEFORM7 that co-occur together within a sliding window. Those probabilities were computed on an external large corpus, i.e., a 5.48GB Wikipedia dump in our experiments. The NPMI score of each topic in the experiments is calculated with top 10 words ( INLINEFORM8 ) by the Palmetto package. Again, we report the average scores and the standard deviations over 5 random runs.
It is known that conventional topic models directly applied to short texts suffer from low quality topics, caused by the insufficient word co-occurrence information. Here we study whether or not the meta information helps MetaLDA improve topic quality, compared with other topic models that can also handle short texts. Table TABREF65 shows the NPMI scores on the three short text datasets. Higher scores indicate better topic coherence. All the models were trained with 100 topics. Besides the NPMI scores averaged over all the 100 topics, we also show the scores averaged over top 20 topics with highest NPMI, where “rubbish” topics are eliminated, following BIBREF22 . It is clear that MetaLDA performed significantly better than all the other models in WS and AN dataset in terms of NPMI, which indicates that MetaLDA can discover more meaningful topics with the document and word meta information. We would like to point out that on the TMN dataset, even though the average score of MetaLDA is still the best, the score of MetaLDA has overlapping with the others' in the standard deviation, which indicates the difference is not statistically significant.
Running Time
In this section, we empirically study the efficiency of the models in term of per-iteration running time. The implementation details of our MetaLDA are as follows: (1) The SparseLDA framework BIBREF30 reduces the complexity of LDA to be sub-linear by breaking the conditional of LDA into three “buckets”, where the “smoothing only” bucket is cached for all the documents and the “document only” bucket is cached for all the tokens in a document. We adopted a similar strategy when implementing MetaLDA. When only the document meta information is used, the Dirichlet parameters INLINEFORM0 for different documents in MetaLDA are different and asymmetric. Therefore, the “smoothing only” bucket has to be computed for each document, but we can cache it for all the tokens, which still gives us a considerable reduction in computing complexity. However, when the word meta information is used, the SparseLDA framework no longer works in MetaLDA as the INLINEFORM1 parameters for each topic and each token are different. (2) By adapting the DistributedLDA framework BIBREF33 , our MetaLDA implementation runs in parallel with multiple threads, which makes MetaLDA able to handle larger document collections. The parallel implementation was used on the NYT dataset.
The per-iteration running time of all the models is shown in Table TABREF66 . Note that: (1) On the Reuters and WS datasets, all the models ran with a single thread on a desktop PC with a 3.40GHz CPU and 16GB RAM. (2) Due to the size of NYT, we report the running time for the models that are able to run in parallel. All the parallelised models ran with 10 threads on a cluster with a 14-core 2.6GHz CPU and 128GB RAM. (3) All the models were implemented in JAVA. (4) As the models with meta information add extra complexity to LDA, the per-iteration running time of LDA can be treated as the lower bound.
At the document level, both MetaLDA-df-0.01 and DMR use priors to incorporate the document meta information and both of them were implemented in the SparseLDA framework. However, our variant is about 6 to 8 times faster than DMR on the Reuters dataset and more than 10 times faster on the WS dataset. Moreover, it can be seen that the larger the number of topics, the faster our variant is over DMR. At the word level, similar patterns can be observed: our MetaLDA-0.1-wf ran significantly faster than WF-LDA and LF-LDA especially when more topics are used (20-30 times faster on WS). It is not surprising that GPU-DMM has comparable running speed with our variant, because only one topic is allowed for each document in GPU-DMM. With both document and word meta information, MetaLDA still ran several times faster than DMR, LF-LDA, and WF-LDA. On NYT with the parallel settings, MetaLDA maintains its efficiency advantage as well.
Conclusion
In this paper, we have presented a topic modelling framework named MetaLDA that can efficiently incorporate document and word meta information. This gains a significant improvement over others in terms of perplexity and topic quality. With two data augmentation techniques, MetaLDA enjoys full local conjugacy, allowing efficient Gibbs sampling, demonstrated by superiority in the per-iteration running time. Furthermore, without losing generality, MetaLDA can work with both regular texts and short texts. The improvement of MetaLDA over other models that also use meta information is more remarkable, particularly when the word-occurrence information is insufficient. As MetaLDA takes a particular approach for incorporating meta information on topic models, it is possible to apply the same approach to other Bayesian probabilistic models, where Dirichlet priors are used. Moreover, it would be interesting to extend our method to use real-valued meta information directly, which is the subject of future work.
Acknowledgement
Lan Du was partially supported by Chinese NSFC project under grant number 61402312. Gang Liu was partially supported by Chinese PostDoc Fund under grant number LBH-Q15031.

Question: Which real world datasets do they experiment on?
Answer: Reuters, 20 Newsgroup, New York Times, Web Snippet, Tag My News, ABC News

Reuters, 20NG, 20 Newsgroup, NYT, New York Times, WS, Web Snippet, TMN, Tag My News, AN, ABC News

Reuters-21578 dataset, 20 Newsgroup, New York Times, Tag My News, ABC News, Web Snippet

Question: Which other models that incorporate meta information do they compare against?
Answer: LLDA, PLLDA, DMR, WF-LDA, LF-LDA, GPU-DMM

LDA BIBREF0, LLDA, Labelled LDA BIBREF11 and PLLDA, Partially Labelled LDA BIBREF9, DMR, LDA with Dirichlet Multinomial Regression BIBREF8, WF-LDA, Word Feature LDA BIBREF16, LF-LDA, Latent Feature LDA BIBREF5, GPU-DMM, Generalized Pólya Urn DMM BIBREF7, PTM, Pseudo document based Topic Model BIBREF18

Question: How do they measure topic quality?
Answer: Normalised Pointwise Mutual Information

NPMI scores

Question: Which data augmentation techniques do they use?
Answer: augmented with a set of Beta random variables, auxiliary variable

the introduction of auxiliary variables