Venue: NAACL 2016

# Online Multilingual Topic Models with Multi-Level Hyperpriors

Kriste Krstovski\({}^{,}\lx@sectionsign\), David A. Smith\({}^{\ddagger}\) and Michael J. Kurtz \({}^{\dagger}\)

\({}^{\dagger}\)Harvard-Smithsonian Center for Astrophysics, Cambridge, MA

\({}^{\lx@sectionsign}\)College of Information and Computer Sciences, University of Massachusetts Amherst, Amherst, MA

\({}^{\ddagger}\)College of Computer and Information Science, Northeastern University, Boston, MA

kkrstovski@cfa.harvard.edu, dasmith@ccs.neu.edu, kurtz@cfa.harvard.edu

###### Abstract

For topic models, such as LDA, that use a bag-of-words assumption, it becomes especially important to break the corpus into appropriately-sized "documents". Since the models are estimated solely from the term cooccurrences, extensive documents such as books or long journal articles lead to diffuse statistics, and short documents such as forum posts or product reviews can lead to sparsity. This paper describes practical inference procedures for hierarchical models that smooth topic estimates for smaller sections with hyperpriors over larger documents. Importantly for large collections, these online variational Bayes inference methods perform a single pass over a corpus and achieve better perplexity than "flat" topic models on monolingual and multilingual data. Furthermore, on the task of detecting document translation pairs in large multilingual collections, polylingual topic models (PLTM) with multi-level hyperpriors (mlnPLTM) achieve significantly better performance than existing online PLTM models while retaining computational efficiency.

## 1 Introduction

Bag of words models simplify the representation of documents by discarding grammatical information and simply relying on document-level word cooccurrence statistics. Topic models, such as latent Dirichlet allocation (LDA) [4], use this representation. A major drawback of the bag of words representation, especially in collections of large documents, is that the word co-occurrence statistics are computed on a document level and as such they do not capture the effect of words co-occurring close to each other versus words co-occurring further apart.

One alternative approach to longer documents that has received attention in the past has been to directly model local--i.e., Markov--dependencies among tokens. For example, the topical n-gram model (TNG) introduced by Wang et al. wang2007models' (2007) models unigram and n-gram phrases as mixture of topics based on the nearby word context. More recently, jameel2013model proposed an LDA extension that uses word sequence information to generate topic distribution over n-grams and performs topic segmentation using segment and paragraph information. While these and many other approaches offer a better and more realistic modeling of word sequences, they don't model topical variations across document sections either in mono- or multilingual collections.

In this paper, we focus on hierarchical models for improving topic models of long documents. In the past, document-topic based hierarchical prior structures have been explored for LDA. For example, wallach2009multitask showed that Gibbs sampling implementation of asymmetric Dirichlet priors provide better modeling of documents, across the whole collection, compared to the original LDA approach. More recently, Kim2013 introduced tiLDA, a topic model of monolingual document collections with nested hierarchies. In order to achieve reasonable performance over large document collections with deep hierarchies, tiLDA utilizes parallel variational Bayes (VB) inference. While VB is known to converge faster than Gibbs sampling, and paral

lel implementations are even faster, they, as with Gibbs sampling, still require multiple iterations over the whole collection besides the overhead of parallelizing the model parameters. Furthermore these approaches focus on monolingual collections.

We propose an online VB inference approach for topic models that captures the document specific effect of local and long range word co-occurrence by modeling individual document sections using multilevel Dirichlet prior structure. The proposed models assign Dirichlet priors to individual document sections that are coupled by a document level hierarchical Dirichlet prior which facilitates explicit modeling of the variation in topics across documents in mono- and multilingual collections. This in turn streamlines the use of topic models in collections of large documents where there is a predetermined section structure. Our contribution is twofold: (1) we present an online VB inference approach for topic models with multi-level Dirchlet prior structure and more importantly (2) introduce a polylingual topic model (PLTM) with multi-level hyperpriors (mlhPLTM) which is capable of efficiently modeling topical variations across document sections in large multilingual collections.

## 2 Efficient Multi-level Hyperpriors

The original LDA model and its multilingual variant, PLTM, use symmetric Dirichlet priors over the document-topic distributions \(\theta_{d}\) and topic-word distributions \(\varphi_{k}\) which means that the concentration parameter \(\alpha\) of the Dirichlet distribution is fixed and that the base measure \(u\) across all topics is uniform. Symmetric Dirichlet priors assume that all documents in the collection are drawn from the same family of distributions. This assumption is not suitable for collections of documents that cover a diverse set of topics. In the past this issue has been addressed with asymmetric priors where the base measures are non-uniform. One way to assign asymmetric priors to individual documents is to treat the base measures vector \(u\) as a hidden variable and assign a symmetric Dirichlet prior to it which creates a hierarchical Dirichlet prior structure over all document-topic distributions in the collection. This approach was used by Wallach et al. [38]. Unlike Wallach et al. [38], who use a single document-topic distribution \(\theta_{d}\), we introduce section-topic distributions \(\theta_{s}\). The existing symmetric Dirichlet prior over \(\theta_{d}\) creates a hierarchical Dirichlet prior over \(\theta_{s}\) (\(\theta=\theta_{d},\theta_{s_{1}},\theta_{s_{2}},...,\theta_{s_{S}}\)):

\[p(\theta|\alpha_{d}u,\alpha_{s})\propto p(\theta_{d}|\alpha_{ d}u)\prod_{s}p(\theta_{s}|\alpha_{s}\theta_{d}) \tag{1}\]

In this setting the most widely used approach for estimating \(\theta_{d}\) is Minka's [37] fixed-point iteration approach which is also used in Kim et al. [40]. Instead we use a more efficient approach for estimating the Dirichlet-multinomial hyperparameters by approximating the digamma differences in Minka's approach which was showcased in Wallach [38] to be more efficient. Figure 1 shows the graphical model representation (left) of our model, which we refer to as multi-level hyperpriors LDA (mlhkDA), along with the free variational parameters for approximating the posteriors (right).

### Inference using Online VB

Due to its ease of implementation, the most widely used approach for inferring LDA posterior distributions is Gibbs sampling Griffiths and Steyvers [56]. For example, this approach was used by Wallach et al. [38] and was originally used for PLTM. On the other hand the VB approach Blei et al. [9] offers more efficient computation but as in the case of Gibbs sampling requires iterating over the whole collection multiple times (e.g. Kim et al. [40]). More recently Hoffman et al. [37] introduced online LDA (oLDA) that relies on online stochastic optimization and requires a single pass over the whole collection. The same approach was also extended to PLTM (oPLTM) Krstovski and Smith [40]. In our work we also utilize online VB to implement multi-level hyperprior (mlh) structure in LDA and PLTM. Similar to batch VB, in online

Figure 1: mlhLDA: Graphical representation (left); Free variational parameters for the online VB approximation (right).

 

VB locally optimal values of the free variational parameters \(\gamma\) and \(\phi\), which are used to approximate the posterior \(\theta\) and \(z\), are computed in the E step of the algorithm but on a batch \(b\) of documents \(d_{i}\) (rather than the whole collection \(D\) as in the case of batch VB) while holding the topic-word variational parameter \(\lambda\) fixed. In the M step, \(\lambda\) is updated using stochastic gradient algorithm by first computing the optimal values of \(\tilde{\lambda}\) using the batch optimal values of \(\phi^{b}\): \(\tilde{\lambda}_{kw}=\eta+\frac{D}{|b|}\sum_{i=1}^{|b|}n_{d_{i}w}\phi_{wk}^{ d_{i}}\). This value is then combined with value of \(\lambda\) computed on the previous batch through weighted average:

\[\lambda_{kw}^{b}\leftarrow(1-\rho_{b})\,\lambda_{kw}^{b-1}+\rho_{b}\tilde{ \lambda}_{kw} \tag{2}\]

When computing the section-topic variational parameters we follow the proof of the lower bound which was derived by Kim et al. (2013). This lower bound, which is looser than the original VB Evidence Lower Bound (ELBO), allows for the batch VB approach to be used with asymmetric priors. More specifically, given the document-topic variational parameter \(\gamma_{dk}\) in the E step of our online VB approach the update for the section-topic variational parameter \(\gamma_{sk}\) becomes:

\[\gamma_{sk}=\alpha_{s}(\frac{\gamma_{dk}}{\sum_{k}\gamma_{dk}})+\sum_{w}n_{w}^ {s}~{}\phi_{wk}^{s} \tag{3}\]

## 3 Online PLTM with multi-level Dirichlet Priors

Given an aligned multilingual document tuple, PLTM assumes that: (1) there exists a single tuple-specific distribution across topics and (2) sets of language specific topic-word distributions. Each word is generated from a language- and topic-specific multinomial distribution \(\varphi_{k}^{l}\) as selected by the topic assignment variable \(z_{n}^{l}\):

\[w_{n}^{l}\sim p\left(w_{n}^{l}~{}|~{}z_{n}^{l},\varphi_{k}^{l}\right) \tag{4}\]

We extend this model by introducing sections specific topic distributions \(\theta_{s}\) across the different languages in the tuple which are coupled by the tuple specific document-topic distribution \(\theta_{d}\).

Given a collection of document tuples \(d\) where each tuple contains \(l\) documents that are translations of each other in different languages, mlhPLTM assumes the following generative process. For each language \(l\) in the collection the model first generates a set of \(k\in\{1,2,...,K\}\) topic-word distributions, \(\varphi_{k}^{l}\) which are drawn from a Dirichlet prior with language specific hyperparameter \(\beta^{l}\): \(\varphi_{k}^{l}\sim Dirichlet(\beta^{l})\). For each document \(d^{l}\) with \(s_{d}\) sections in tuple \(d\), mlhPLTM then assumes the following generative process:

* Choose \(\theta_{d}\sim Dir.(\alpha_{d})\)
* For each section \(s_{d}\) in document tuple \(d\):
* Choose \(\theta_{s}\sim Dir.(\alpha_{s}\theta_{d})\)
* For each language \(l\) in section \(s\):
* For each word \(w\) in section \(s_{d}^{l}\):
* Choose a topic \(z\sim Multi.(\theta_{s}^{l})\)
* Choose a word \(w\sim Multi.(\varphi_{z}^{l})\)

Figure 2 shows the graphical representation of mlhPLTM. The free variational parameters for the online VB approximation of the posteriors are shown in Figure 3.

## 4 Modeling Sections in Scientific Articles

We explore the ability of mlhLDA to model variations across document sections found in scientific articles using a collection of journal articles from the Astrophysics Data System (ADS) Kurtz et al. (2000). Our collection consists of 130k training articles (888,346 sections) and a held-out set of 8,078 articles (54,502 sections). Figure 4 shows an example mhlhLDA representation of an ApJ article with 100 topics. Shown on the top is the inferred topic representation of the whole document (\(\theta_{d}\)) which, in the mlhLDA model, serves as a prior for the section-topic distributions (\(\theta_{s}\)). Shown on the bottom are ex

Figure 3: mlhPLTM: Graphical representation of the free variational parameters for the online VB approximation.

Figure 2: mlhPLTM: Graphical model representation.

 

amples of 2 article sections (out of 7), their inferred topic distributions along with the top 10 words for each of the top 3 section topics.

The left side of Figure 5 shows the held-out perplexity comparison between oLDA and mlhLDA across 13 different topic configurations. For this set of experiments we used the above training set of 130k articles and the set of 8,078 held-out articles. From these comparisons we clearly see the advantage of using the multi-level Dirchlet prior structure. Another way of evaluating topic models is through an extrinsic evaluation task which was not available for this collection. In the case of oLDA, article sections were treated as individual documents. In the original oLDA1 implementation the per document concentration parameter \(\alpha_{d}\) was set to \(\frac{1}{K}\) which we also use in our case for both the symmetric \(\theta_{d}\) and asymmetric \(\theta_{s}\) (same goes for PLTM and mlhPLTM). Since in our case we perform relative comparison between oLDA and mlhkDA we weren't concerned with experimenting with different concentration parameters but we rather used the default one implemented in oLDA.

Footnote 1: [http://www.cs.princeton.edu/~mdhoffma](http://www.cs.princeton.edu/~mdhoffma)

With a random subset of 10k training and 1k held-out articles we compared the performance of oLDA and mlhLDA with the original batch VB2 implementation of Blei et al. (2003). Unlike the implementations of oLDA and mlhLDA which are written in Python the original VB algorithm is written in C and requires multiple iterations over the whole collection. The right side of Figure 5 shows the speed (in natural log scale) vs. perplexity comparison across the three models.

Footnote 2: [http://www.cs.princeton.edu/~blei/lda-c](http://www.cs.princeton.edu/~blei/lda-c)

## 5 Modeling and Retrieving Speeches in Europarl Sessions

We compared the modeling performance of oPLTM and mlhPLTM on a subset of the English-Spanish Europarl collection (Koehn, 2005). The subset consists of \(\sim\)64k training pairs of English-Spanish speeches that are translations of each other which originate from 374 sessions of the European Parliament (Europarl) and a test set of \(\sim\)14k speech translation pairs from 112 sessions. With oPLTM we modeled individual speech pairs while with mlhPLTM we utilized the session hierarchy and modeled pairs of speeches as document sections. Comparisons were performed intrinsically (using perplexity) and extrinsically on a cross-language information retrieval (CLIR) task. This task, along with the Europarl subset, have been previously defined by Mimno et al. (2009) and used across other publications (Platt et al., 2010; Krstovski and Smith, 2013). Given a query English speech, the CLIR task is to retrieve its Spanish translation equivalent. It involves performing comparison across topic representations of all Spanish speeches using Jensen-Shannon divergence and sorting the results. Models are evaluated using precision at rank one (P@1). Figure 6 shows the CLIR task performance comparisons results using 13 different topic configurations. We performed comparisons across three different settings of the concentration parameters \(\alpha_{d}\) and \(\alpha_{s}\) (\(\alpha_{d}\)=\(\alpha_{s}\)=\(\frac{1}{K}\), 0.4 and 1.0).

Figure 4: mhlLDA representation of the ApJ article “Inferred Colors of the Gamma-Ray Detected Blazers”.

Figure 5: oLDA vs. mlhLDA: perplexity comparison (left); speed vs. perplexity comparisons with batch VB (right).

 

Across the different concentration parameter values and across the 13 different topic configurations we observe that the performance of oPLTM fluctuates as we increase the numbers of topics. On the other hand, across the three different concentration parameter settings, mlhPLTM performance is very steady and tends to increase with the number of topics. Across the different topic configurations both models provide the best performance with \(\alpha_{d}=\alpha_{s}=0.4\). Setting the concentration parameters to \(\frac{1}{K}\) gives the overall worst performance.

In our initial experiments we unintentionally reordered our set of training Europarl sessions based on two digit years which was different from the experimental setup in [29] and [57] where the order of the presentation data (Europarl speeches) was chronological. This emphasized the fact that in online VB, order of presentation of documents plays an important role especially in the training step where the model learns the per topic-word distributions. Figure 7 shows the performance comparison results between oPLTM and mlhPLTM when documents in the training and test steps are ordered numerically. In our initial experimental setup concentration parameters where set to \(\alpha_{d}=\alpha_{s}=\frac{1}{K}\). To the left is the perplexity comparison between the two models. The CLIR task performance comparisons results are shown on the right. Unordered mlnPLTM achieves high P@1 after 2,000 topics. While it takes much longer in terms of the number of topics unordered mlhPLTM ultimately achieves similar performance results as ordered mlhPLTM.

## 6 Conclusion

We presented online topic models with multi-level Dirichlet prior structure that provide better modeling of topical variations across document sections in mono- and multilingual collections. We showed that documents with rich sub-document level structure could be modeled with higher likelihood compared to regular online LDA and PLTM models while offering the same efficiency. Furthermore on the task of retrieving document translations we showed that mlhPLTM achieves significantly better retrieval results compared to online PLTM.

## Acknowledgments

This work was supported in part by the Harvard-Smithsonian CfA predoctoral fellowship, in part by the Center for Intelligent Information Retrieval and in part by NSF grant #IIS-0910884. Any opinions, findings and conclusions or recommendations expressed in this material are those of the authors and do not necessarily reflect those of the sponsor.

## References

* Blei et al. (2003) David M. Blei, Andrew Y. Ng, and Michael I. Jordan. 2003. Latent Dirichlet allocation. _JMLR_, 3:993-1022.
* Griffiths and Steyvers (2004) T. L. Griffiths and M. Steyvers. 2004. Finding scientific topics. _Proceedings of the National Academy of Sciences_, 101(Suppl. 1):5228-5235.
* Hoffman et al. (2010) Matthew Hoffman, David Blei, and Francis Bach. 2010. Online learning for latent Dirichlet allocation. In _NIPS_, pages 856-864.
* Jameel and Lam (2013) Shoaib Jameel and Wai Lam. 2013. An unsupervised topic segmentation model incorporating word order. In _SIGIR_, pages 203-212.
* Krizhevsky et al. (2012)

Figure 6: oPLTM vs. mlhPLTM: Performance comparison on the CLIR task using chronological ordering of sessions across different hyperparameter settings, \(\alpha_{d}\)=\(\alpha_{s}\)=\(\frac{1}{K^{\prime}}\), 0.4 and 1.0 .

Figure 7: oPLTM vs. mlhPLTM: perplexity comparison (left); performance comparison on the CLIR task (right). Documents were presented out of chronological order and thus performance is lower, especially for oPLTM.

 

* [Kim et al.2013] Do-Kyum Kim, Geoffrey Voelker, and Lawrence K. Saul. 2013. A variational approximation for topic modeling of hierarchical corpora. In _ICML_, pages 55-63.
* [Koehn2005] Philipp Koehn. 2005. Europarl: A parallel corpus for statistical machine translation. In _MT Summit_, pages 79-86.
* [Krstovski and Smith2013] Kriste Krstovski and David A. Smith. 2013. Online polylingual topic models for fast document translation detection. In _WMT_, pages 252-261.
* [Kurtz et al.2000] Michael J. Kurtz, Guenther Eichhorn, Alberto Accomazzi, Carolyn S. Grant, Stephen S. Murray, and Joyce M. Watson. 2000. The nasa astrophysics data system: Overview. _Astronomy and Astrophysics Supplement Series_, 143:41-59.
* [Mimno et al.2009] David Mimno, Hanna Wallach, Jason Naradowsky, David A. Smith, and Andrew McCallum. 2009. Polylingual topic models. In _EMNLP_, pages 880-889.
* [Minka2000] Thomas P. Minka. 2000. Estimating a dirichlet distribution. Technical report, MIT.
* [Platt et al.2010] John Platt, Kristina Toutanova, and Wen tau Yih. 2010. Translingual document representations from discriminative projections. In _EMNLP_, pages 251-261.
* [Wallach et al.2009] Hanna M. Wallach, David Mimno, and Andrew McCallum. 2009. Rethinking LDA: Why priors matter. In _NIPS_, pages 1973-1981.
* [Wallach2008] Hanna M. Wallach. 2008. _Structured Topic Models for Language_. Ph.D. thesis, University of Cambridge.
* [Wang et al.2007] Xuerui Wang, Andrew McCallum, and Xing Wei. 2007. Topical n-grams: Phrase and topic discovery, with an application to information retrieval. In _ICDM_, pages 697-702.