Venue: NAACL 2019

# Scalable Collapsed Inference for High-Dimensional Topic Models

Rashidul Islam and James Foulds

Department of Information Systems

University of Maryland, Baltimore County

{islam.rashidul,jfoulds}@umbc.edu

###### Abstract

The bigger the corpus, the more topics it can potentially support. To truly make full use of massive text corpora, a topic model inference algorithm must therefore scale efficiently in 1) documents and 2) topics, while 3) achieving accurate inference. Previous methods have achieved two out of three of these criteria simultaneously, but never all three at once. In this paper, we develop an online inference algorithm for topic models which leverages _stochasticity_ to scale well in the number of documents, _sparsity_ to scale well in the number of topics, and which operates in the _colapest representation_ of the topic model for improved accuracy and run-time performance. We use a Monte Carlo inner loop in the online setting to approximate the collapsed variational Bayes updates in a sparse and efficient way, which we accomplish via the Metropolis-Hastings Walker method. We showcase our algorithm on LDA and the recently proposed mixed membership skip-gram topic model. Our method requires only amortized \(O(k_{d})\) computation per word token instead of \(O(K)\) operations, where the number of topics occurring for a particular document \(k_{d}\ll\) the total number of topics in the corpus \(K\), to converge to a high-quality solution.

## 1 Introduction

Topic models are powerful tools for analyzing today's massive, constantly expanding digital text information by representing high-dimensional data in a low-dimensional subspace. We can recover the main themes of a corpus by using topic models such as latent Dirichlet allocation (LDA) to organize, understand, search, and explore the documents Blei et al. (2003).

Traditional LDA inference techniques such as variational Bayes and collapsed Gibbs sampling do not readily scale to corpora containing millions of documents. To scale up inference, the main approaches are distributed algorithms Newman et al. (2008) and stochastic algorithms Hoffman et al. (2010, 2013). Stochastic algorithms, such as stochastic variational inference (SVI), operate in an online fashion, and hence do not need to see all of the documents before updating the topics, so they can be applied to corpora of any size, without expensive distributed hardware Hoffman et al. (2010). The "collapsed" representation of topic models is also frequently important, as it leads to faster convergence, efficient updates, and lower variance in estimation Griffiths and Steyvers (2004). The stochastic collapsed variational Bayesian inference (SCVB0) algorithm, proposed by Foulds et al. (2013), combines the benefits of stochastic and collapsed inference.

Larger corpora typically support more topics, which brings the additional efficiency challenge of training a larger model Mimno et al. (2012). This challenge has been addressed by exploiting sparsity to perform updates in time sublinear in the number of topics. A sparse variant of the SVI algorithm for LDA, SSVI, proposed by Mimno et al. (2012), is scalable to large numbers of topics, but does not fully exploit the collapsed representation of LDA, which is important for faster convergence and improved inference accuracy, due to a better variational bound Teh et al. (2007). The Metropolis Hastings Walker (MHW) method Li et al. (2014) scales well in the number of topics, and uses a collapsed inference algorithm, but it operates in the batch setting, so it is not scalable to large corpora. LightLDA Yuan et al. (2015)

is a distributed approach to the MHW method which adopts a data-and-model-parallel strategy to maximize memory and CPU efficiency. However, it is not an online approach, and furthermore requires multiple expensive computer clusters to converge faster. Tensor methods are another approach to speeding up topic models [10, 2], which theoretically guarantee the recovery of the true parameters by overcoming the problem of local optima. These techniques use the method of moments instead of maximum likelihood estimation or Bayesian inference, which leads to lower data efficiency, and sometimes unreliable performance.

In this work, we propose a highly efficient and scalable inference algorithm for topic models. We develop an online algorithm which leverages _stochasticity_ to scale well in the number of documents, _sparsity_ to scale well in the number of topics, and which operates in the _collapsed_ representation of topic models. We thereby combine the individual benefits of SVI, SSVI, SCVB0, and MHW into a single algorithm. Our approach is to develop a sparse version of SCVB0. Inspired by SSVI, we use a Monte Carlo inner loop to approximate the SCVB0 variational distribution updates in a sparse and efficient way, which we accomplish via MHW method. To show the generality of our algorithm, we explore the benefits of our inference method for LDA and another recently proposed topic model, MMSGTM, with experiments on both small and large-scale datasets.

## 2 Background

To build the foundation for our proposed method, in this section we provide the necessary background on LDA and MMSGTM topic models and their associated inference algorithms. This is followed by a description of the MHW sampler for reducing topic model sampling complexity.

### Latent Dirichlet Allocation and SCVB0

Probabilistic topic models such as LDA [35] use latent variables to encode co-occurrence patterns between words in text corpora and other bag-of-words represented data. In LDA, we assume that the \(D\) documents in a corpus are each from mixture distributions of \(K\) individual topics \(\phi_{k}\), \(k\in\{1,...,K\}\), each of which are discrete distributions over words. For a document \(j\) of length \(N_{j}\), the local (document-level) variables \(\theta_{j}\) are a distribution over topics drawn from a Dirichlet prior with parameters \(\alpha_{k}\) and for each token, global variables (corpus-level) \(\phi_{k}\) are drawn from a Dirichlet prior with parameters \(\beta_{w}\). Due to conjugacy, we can marginalize out topics \(\mathbf{\Theta}\) and distributions over topics \(\mathbf{\Phi}\), and perform inference only on the topic assignments \(\mathbf{Z}\) in the collapsed representation of LDA [8]. For scalable and accurate inference, Foulds et al. [49] proposed a stochastic collapsed variational inference algorithm, SCVB0. The SCVB0 approach computes a variational discrete distribution \(\gamma_{ij}\) over the \(K\) topic assignment probabilities for each word \(i\) in each document \(j\), but does not maintain the \(\gamma\) variables that increase the memory requirement of original batch CVB0 algorithm [10]. SCVB0 iteratively updates each \(\gamma_{ij}\) using

\[\gamma_{ijk}:\propto\frac{N^{\Phi}_{w_{ij}k}+\beta_{w_{ij}}}{N^{E}_{k}+\sum_{ w}\beta_{w}}(N^{\Theta}_{jk}+\alpha_{k}) \tag{1}\]

for each topic \(k\), with \(j^{th}\) document's \(i^{th}\) word \(w_{ij}\). The \(\mathbf{N^{Z}}\), \(\mathbf{N^{\Theta}}\), and \(\mathbf{N^{\Phi}}\) are referred to as CVB0 statistics, where \(\mathbf{N^{Z}}\) is the vector of expected number of words assigned to each topic, each entry \(j\), \(k\) of matrix \(\mathbf{N^{\Theta}}\), and each entry \(w\), \(k\) of matrix \(\mathbf{N^{\Phi}}\) are the expected number of times document \(j\), and word \(w\) are assigned to topic \(k\), respectively, across the corpus. To do stochastic updates of these variables, one sequence of stepizes \(\rho^{\Phi}\) for \(\mathbf{N^{\Phi}}\) and \(\mathbf{N^{Z}}\) and another sequence \(\rho^{\Theta}\) for \(\mathbf{N^{\Theta}}\) are maintained. The update of \(\mathbf{N^{\Theta}_{jj}}\) for every token \(i\) of document \(j\) with an online average of the current value and its expected value is

\[\mathbf{N^{\Theta}_{ji}}:=(1-\rho^{\Theta}_{t})\mathbf{N^{\Theta}_{jj}}+\rho^{\Theta}_ {t}C_{j}\gamma_{ij} \tag{2}\]

where \(C_{j}\) is the document length. In practice, it is too expensive to update \(\mathbf{N^{\Phi}}\) after every token. This leads to the use of minibatch updates with the average of the \(M\) per-token estimates of the form \(\mathbf{Y_{ij}}\), which is a \(W\times K\) matrix with the \(w_{ij}\)th row being \(\gamma_{ij}\) and with zeros in the other entries:

\[\mathbf{N^{\Phi}} :=(1-\rho^{\Phi}_{t})\mathbf{N^{\Phi}}+\rho^{\Phi}_{t}\mathbf{\hat{N}^{ \Phi}}\] (3) \[\mathbf{N^{Z}} :=(1-\rho^{\Phi}_{t})\mathbf{N^{Z}}+\rho^{\Phi}_{t}\mathbf{\hat{N}^{Z}}~{},\] (4) 

where \(\mathbf{\hat{N}^{\Phi}}=\frac{C}{|M|}\sum_{ij\in M}\mathbf{Y_{ij}}\), \(\mathbf{\hat{N}^{Z}}=\frac{C}{|M|}\sum_{ij\in M}\gamma_{ij}\), and \(C\) is the number of words in the corpus. The SCVB0 algorithm outperforms stochastic VB [14] on large corpora by converging faster and often to a better solution [29]. However, the SCVB0 algorithm does not leverage sparsity, and hence requires \(O(K)\) operations per word token.

### MMSG Topic Model

To show the generality of our approach to topic models other than LDA, we will also apply our method to a recent model called the Mixed Membership Skip-gram Topic Model (MMSGTM) [28], which combines ideas from topic models and word embeddings (cf. also [35, 6]). MMSGTM's generative model for words and their surrounding context is:

* For each word \(w_{i}\) in the corpus
* Sample a topic \(z_{i}\sim\text{Discrete}(\theta_{w_{i}})\)
* For each word \(w_{c}\in context(i)\)
* Sample a context word \(w_{c}\sim\text{Discrete}(\phi_{z_{i}})\) .

The inferred model can then be used to train embeddings for topics and words, although we do not consider this here. The MMSGTM admits a collapsed Gibbs sampler (CGS) which efficiently resolves the cluster assignments. With Dirichlet priors on the parameters, the CGS update is

\[p(z_{i}=k|.)\propto(N_{w_{i}k}^{(\Phi)-i}+\alpha_{k})\times\\ \prod_{c=1}^{|context(i)|}\frac{N_{w_{c}k}^{(\Phi)-i}+\beta_{ w_{c}}+N_{w_{c}}^{(i,c)}}{N_{k}^{(Z)-i}+\sum_{w}\beta_{w}+c-1}~{},\] (5) where \[\alpha\] and \[\beta\] are parameter vectors for Dirichlet priors over the topic and word distributions, \[\mathbf{N_{w_{i}}^{\Phi}}\] and \[\mathbf{N_{w_{c}}^{\Phi}}\] are input and output word-topic counts (excluding the current word), \[N^{Z}\] is the total topic counts in output word-topic counts, and \[N_{w_{c}}^{(i,c)}\] is the number of occurrences of word \[w_{c}\] before the \[c^{th}\] word in the \[i^{th}\] context. MMSGTM exploits the MHW algorithm, which scales sub-linearly in K, but not in the number of training documents.

### Metropolis-Hastings-Walker Sampler

The MHW method [7], which is a key component of our approach, uses a data structure called an alias table which allows sampling from a discrete distribution in amortized \(O(1)\) time. Assuming initial probabilities \(p_{0},p_{1},...,p_{l-1}\) of a distribution over \(l\) outcomes and average of probabilities \(a=\frac{1}{l}\), the alias table \(A\) can be formed as follows [10]:

* Initialize: for \(i\) from \(0\) to \(l-1\)
* \(A_{alias}[i]=i\) and \(A_{prob}=(i+1)a\)
* Do the following steps \(n-1\) times
* Find smallest \(p_{i}\) and largest \(p_{j}\)
* Set \(A_{alias}[i]=j\) and \(A_{prob}=i\times a+p_{i}\)
* \(p_{j}:=p_{j}-(a-p_{i})\) and \(p_{i}:=a~{}\) .

Then, to sample from \(p\) using the alias table:

* Roll \(l\)-sided fair die to choose element \(i\) of \(A\)
* If \(Rand(1)<A_{prob}[i]\) return \(i\), else return \(A_{alias}[i]\).

Li et al. LiEtAl 

number of documents but also in the number of topics, while gaining the benefits of collapsed inference. We refer to our method as the sparse stochastic collapsed variational Bayesian inference (SparseSCVB0) algorithm.

In SparseSCVB0, we obtain sparsity by substituting sparse Monte Carlo approximations for the original SCVB0 variational distributions \(\gamma_{ij}^{(pseudo)}\). The justification for this procedure, also used by [37], is that the expected value of an average over one-hot samples from a distribution is equal to that distribution:

\[E_{s_{i}\sim p(\mathbf{s})}\Big{[}\frac{\sum_{i=1}^{S}\delta_{k}( s_{i})}{S}\Big{]}=\frac{1}{S}\sum_{i=1}^{S}E_{s_{i}\sim p(\mathbf{s})}\Big{[} \delta_{k}(s_{i})\Big{]}\] \[=\frac{1}{S}\sum_{i=1}^{S}\sum_{k^{\prime}}\delta_{k}(s_{i}=k^{ \prime})p(s_{i}=k^{\prime})=p(s=k)~{}.\]

Thus, the overall procedure is still a valid stochastic optimization algorithm. We approximate the inner loop sampler in time sublinear in \(K\), constructing \(\gamma^{(pseudo)}\) by generating \(S\) samples from \(\gamma_{ij}\) using the MHW method. To describe the general form of our algorithm, we introduce SparseSCVB0 statistics: local (e.g. document-level) expected counts \(\mathbf{N^{L}}\), global (corpus-level) expected counts \(\mathbf{N^{G}}\), and total expected topic counts \(\mathbf{N^{Z}}\). We approximate local sufficient statistics \(\mathbf{N^{L}}\) for each token \(i\) in document \(j\) via:

\[N_{jk}^{L}\approx C_{j}E_{\gamma_{ij}^{(pseudo)}}\Big{[}\frac{\sum_{s\in S} \delta_{z_{ij}^{s}=k}}{S}\big{]}~{}.\]

Since \(\gamma^{(pseudo)}\) is sparse, we can efficiently update these statistics using only its non-zero entries. This approach allows us to learn high-dimensional topic models efficiently on very large corpora. Before updating global parameters in a similar fashion, it may also be beneficial to perform a small number of burn-in passes to learn the local parameters \(\mathbf{N^{L}}\)[25]. For large-scale datasets (e.g. Wikipedia), SparseSCVB0 operates in a "mini-epoch" approach where we process a large subset of the corpus (e.g. \(5,000\) documents) several (e.g. \(3\)) times, before discarding and processing the next subset, and so on. This allows a "warm start" of \(\mathbf{N^{L}}\) in repeating iterations, with a small memory overhead. Pseudo-code of SparseSCVB0 for a mini-epoch is provided in Algorithm 1, which we discuss more in the next two sections, including model-specific aspects.

\begin{table}
\begin{tabular}{|l|c|l|l|l|l|l|l|} \hline \multicolumn{4}{|c|}{SparseSCVB0} & \multicolumn{4}{|c|}{Original SCVB0} & \multicolumn{4}{|c|}{SVI} \\ \hline units & neurons & support & data & matrix & eeg & word & cells & circuit \\ unit & model & kernel & clustering & pca & time & words & cell & current \\ net & synaptic & margin & cluster & linear & data & character & cortex & circuits \\ hidden & input & function & clusters & principal & brain & recognition & response & figure \\ output & neuron & vector & algorithm & eigenvectors & activity & characters & firing & input \\ neural & response & svm & model & eigenvalues & signal & trained & cortical & analog \\ networks & cell & machines & problem & eigenvalue & analysis & input & inhibitory & filter \\ \hline \end{tabular}
\end{table}
Table 1: Randomly selected example topics, while models trained on the NIPS corpus for \(K=500\).

 



## 4 SparseSCVB0 for LDA

To deploy SparseSCVB0 for LDA, we use the M-H proposal distribution from Li et al. (2014) which involves drawing samples exactly from document-specific sparse terms or approximately using cached samples from the alias table:

\[q(k):=\frac{P_{L}}{P_{L}+Q_{G}}p_{L}(k)+\frac{Q_{G}}{P_{L}+Q_{G}} q_{G}(k) \tag{8}\]

where \(p_{L}(k)\) and \(q_{G}(k)\) represent the sparse and dense part, respectively from Equation 6 and \(P_{L}=\sum_{k}p_{L}(k)\), \(Q_{G}=\sum_{k}q_{G}(k)\). When \(\frac{P_{L}}{P_{L}+Q_{G}}>RandUnif(1)\), we sample from the sparse part in \(O(k_{d})\) time depending on only the non-zero entries of \(\mathbf{N^{L}_{j}}\) (analogous to \(\mathbf{N^{G}_{j}}\)), as \(\mathbf{N^{L}_{j}}\) is sparse in the LDA setting. Otherwise, we sample from the dense part in amortized \(O(1)\) time using the alias method. Unfortunately, due to the stochastic update, any entry of \(\mathbf{N^{L}_{j}}\) never becomes exactly zero, however it may maintain a very small value. To address this, we apply a sparsification heuristic, where we threshold \(\mathbf{N^{L}_{j}}\) after burn-in passes for each document iteration. We parameterize the threshold as \(\tau\rho^{L}_{t_{Cj}}C_{j}\); where \(C_{j}\) is the length of the \(j\) document, \(\rho^{L}_{t_{Cj}}\) is the step size for the last token of this document, and constant \(0<\tau\leq 1\) controls the sparsity. In our preliminary experiments, we found that, somewhat counter-intuitively, the Monte Carlo and sparsification approximations actually improve convergence in early iterations. We believe that this is because they help SCVB0 escape the initial high-entropy regime, during which convergence of variational algorithms is poor Salakhutdinov et al. (2003). This property makes the benefit of annealing insignificant, so we do not use simulated annealing for LDA inference, fixing \(T_{j}=1\).

An additional optimization of SparseSCVB0 for LDA inference can be performed by "clumping" Teh et al. (2007); Foulds et al. (2013), where one update of the local parameters is performed for each distinct word type in each document. This is performed by fixing the variational distribution, and scaling the update by number of copies of the distinct word type in the document. If we observe the distinct word type \(w_{aj}\), which occurs \(m_{aj}\) times in document \(j\), the update is

\[\mathbf{N^{L}_{j}} :=(1-\rho^{L}_{t})^{m_{aj}}\mathbf{N^{L}_{j}}\] \[+(1-(1-\rho^{L}_{t})^{m_{aj}})C_{j}\gamma^{(pseudo)}_{aj}~{}. \tag{9}\]

## 5 SparseSCVB0 for MMSGTM

The main contribution of our approach for the MMSGTM algorithm is to scale this algorithm in number of documents with online inference, as MMSGTM already scales sublinearly in \(K\) using MHW. Foulds et al. (2018) use an MHW proposal which approximates the CGS update, interpreted as a product of experts Hinton (2002) in which each word in the context is an "expert" which weighs in multiplicatively on the update, with a mixture of experts. In the proposal, they draw a

\begin{table}
\begin{tabular}{|c|c|c|c|c|c|c|c|c|} \hline \multicolumn{3}{|c|}{SparseSCVB0} & \multicolumn{3}{|c|}{Original SCVB0} & \multicolumn{3}{|c|}{SVI} \\ \hline infection & human & liver & decreased & dna & medication & mutations & hormone & intensive \\ infected infections & humans & hvc & decrease & replication & patients & mutation & invasive & icu \\ infectious & chimpanzee & hbv & increased & chromatin & medications & mutated & androgen & delirium \\ host & macaque & hepatic & increase & origins & prescribed & crosses & hormones & occurrences \\ infectious & monkey & hepatitis & decreases & ssdma & patient & missense & testosterone & sedation \\ antiviral & primates & hepatocytes & decreasing & primase & prescription & mut & chr & haloperidol \\ inoculation & chimpanzees & cirrhosis & dramatically & xenopus & pharmacy & hpf & hormonal & psychotropic \\ \hline  & \multicolumn{1}{c}{} & \multicolumn{1}{c}{} & \multicolumn{1}{c}{} & \multicolumn{1}{c}{} & \multicolumn{1}{c}{} & \multicolumn{1}{c}{} & \multicolumn{1}{c}{} \\ \end{tabular}
\end{table}
Table 2: Randomly selected example topics, while models trained on the PubMed corpus with \(K=1,000\).

\begin{table}
\begin{tabular}{|l|l|l|l|l|l|l|l|l|} \hline \multicolumn{3}{|c|}{SparseSCVB0} & \multicolumn{3}{|c|}{Original SCVB0} & \multicolumn{3}{|c|}{SVI} \\ \hline band & town & blood & army & data & gold & president & japanese & engine \\ music & city & treatment & war & system & silver & political & china & test \\ released & road & disease & division & software & diamond & court & chinese & center \\ show & south & patients & battle & computer & golden & senate & japan & small \\ song & river & medical & forces & systems & bronze & constitution & people & gas \\ live & village & health & corps & version & diamonds & politics & countries & engines \\ records & school & effects & infantry & bit & pit & elected &asia & base \\ \hline  & \multicolumn{1}{c}{} & \multicolumn{1}{c}{} & \multicolumn{1}{c}{} & \multicolumn{1}{c}{} & \multicolumn{1}{c}{} & \multicolumn{1}{c}{} & \multicolumn{1}{c}{} & \multicolumn{1}{c}{} \\ \end{tabular}
\end{table}
Table 3: Randomly selected example topics, while models trained on the Wikipedia corpus for \(K=1,000\).



context word \(w_{c}\) uniformly from the context of the current word, \(w_{c}\sim Uniform(|context(w_{i})|)\), and then sample a word based on the chosen context word's contribution to the update:

\[q(k):\propto\frac{N^{G}_{wc}+\beta_{w_{c}}}{N^{Z}_{k}+\sum_{w}\beta_{w}} \tag{10}\]

where \(\mathbf{N^{G}_{we}}\) is analogous to the output context word-topic counts \(\mathbf{N^{\Phi}_{we}}\) of original MMSGTM model. The proposal samples via the alias method in amortized \(O(1)\) time, instead of \(O(k_{d})\) time, since it does not involve the sparse term. We use this proposal to approximate the CVB0 update for the model, which is a deterministic version of Equation 5, neglecting to exclude the current assignment of \(z_{i}\). We update \(\mathbf{N^{G}_{w_{i}}}\) (analogous to \(\mathbf{N^{\Phi}_{we}}\)) for each current word \(w_{i}\) locally, but update \(\mathbf{N^{G}_{w_{e}}}\) and \(\mathbf{N^{z}}\) via minibatch counts \(\hat{\mathbf{N}^{G}_{we}}\) and \(\hat{\mathbf{N}^{Z}}\), respectively, for each context word \(w_{c}\) of current word \(w_{i}\). Unlike for LDA, \(C_{j}\) in the local updates represents the total number of input word \(j\) in the corpus. As we draw multiple output words from each topic assignment, the effective temperature of the MMSGTM model is much lower than for standard LDA which may cause problems with mixing and leads it to get stuck in the initial regime. Following Foulds et al. (2018), we perform simulated annealing which varies the M-H acceptance ratio to improve mixing. We parameterize the temperature schedule as \(T_{j}=T_{0}+\lambda\kappa^{\frac{\mu j}{D}}\), where \(T_{0}\) is the target final temperature, \(\kappa\leq 1\), constant \(\mu\) controls the amount of temperature reduction after each document iteration \(j\) and \(\lambda\) controls the initial temperature.

## 6 Experiments

In this section we study the performance of our SparseSCVB0 1 algorithm, on small as well as large corpora to validate the proposed method for topic models such as LDA and MMSGTM, and to compare with other state-of-the-art algorithms.

Footnote 1: Code implementing SparseSCVB0 can be found at [https://github.com/dr97531/SparseSCVB0](https://github.com/dr97531/SparseSCVB0).

### Experimental Environment and Datasets

We compared SparseSCVB0 to SCVB0 and SVI. For a fair comparison, we implemented all of them in the fast high-level language Julia V0.6.2 Bezanson et al. (2017). We conducted all experiments on a computer with 64GB memory and an Intel Xeon E5-2623 V4 processor with 2.60 GHz clock rate, 8\(\times\)256KB L2 Cache and 10MB L3 Cache. As we only use one single thread for sampling across all experiments, only one CPU core is active throughout the experiment with only 256KB available L2 Cache.2

Footnote 2: Since SSVI relies on multiple complex implementation details, we were unable to develop a fair implementation, nor were we able to obtain source code for a previous implementation. We expect that its accuracy would be similar to SVI, with a speed-up at or below that bestowed by a MHW-based inner loop. Mimno et al. (2012) apply it with only 200 topics for most of their experiments, and at most 1000 topics.

We used _NIPS_, _Reuters-150_, _PubMed Central_, and _Wikipedia_ as representative very small, small, medium, and large-scale datasets, respectively. The _NIPS_ corpus has \(1740\) scientific articles from years 1987-1999 with \(2.3\)M tokens, due to Sam Roweis. The newswire corpus _Reuters-150_ contains \(15,500\) articles with dictionary size of \(8,350\) words. _PubMed Central_ has \(320M\) tokens across \(165,000\) scientific articles and a vocabulary size of around \(38,500\) words. The _Wikipedia_ corpus contained \(4.6\) million arti

Figure 1: Comparison of runtime per iteration for LDA in terms of: (a) number of topics \(K\) and (b) number of iterations when \(K=10,000\). Original SCVB0 is linear in \(K\), while SparseSCVB0 is sublinear in \(K\).

 

encyclopedia. We used the dictionary of \(7,700\) words which was extracted by Hoffman et al. [13]. There were \(811\)M tokens in the corpus.

### Performance for LDA

We implement SparseSCVB0, original SCVB0 and SVI algorithms using the clumping optimization [26] technique. In all LDA experiments, each algorithm was trained using mini-batches of size \(20\) for the NIPS corpus and \(100\) for other corpora. For PubMed and Wikipedia, we chose mini-epoch subsets of size \(5,000\) documents and processed for \(5\) passes. We used a step-size schedule of \(\frac{scale}{(\eta+t)^{\kappa}}\) as in original SCVB0 for global parameters, where \(t\) is the document iteration with \(scale=100.0\), \(\eta=1000.0\) and \(\kappa=0.9\). For document-level parameters, we used the \(scale=1.0\), \(\eta=10.0\) and \(\kappa=0.9\), with \(t\) referring here to the word iteration of the current document. In case of PubMed corpus, we found out that original SCVB0 and SVI tend to stuck in the initial regime for document-level step-size parameter \(\eta=1.0\) which we later fixed by setting \(\eta=10.0\), while SparseSCVB0 didn't suffer from this problem due to the extra randomness from the sparse sampled updates. Finally, we choose hyper-parameters \(\alpha=0.1\) and \(\beta=0.01\) and burn-in pass of \(5\) for each document in all LDA experiments. For SparseSCVB0, we used sample size \(S=5\) to approximate \(\gamma^{(pseudo)}\) and \(\tau=1/K\) for the sparsification heuristic on local parameters.

To study the acceleration benefits of our approach, we evaluated the runtime performance per

Figure 2: Per-topic coherence for LDA when \(K=1,000\) on (a) NIPS, (b) PubMed, and (c) Wikipedia. Spars-eSCVB0 completely outperforms other models for large-scale corpus.

\begin{table}
\begin{tabular}{l} \hline network data communication information communicate connection \\ \hline prison prisoners prisoner imprisoned jail escaped detained guards \\ \hline dog dogs shepherd hounds bred coat scent insitet eating companion \\ \hline song sing singing sings gang songs recorded meloty tune \\ \hline votes vote cast elections voted candidate parties majority election \\ \hline wind winds blowing speed blowy direction high low blown chill \\ \hline hour hours noon time daylight minutes midnight morning seconds \\ \hline \end{tabular}
\end{table}
Table 5: Randomly selected topics from a \(10,000\)-topic model trained using SparseSCVB0 on Wikipedia 

iteration on the number topics and the number of iterations. In Figure 1(a), SparseSCVB0 is compared to original SCVB0 in terms of the average runtime per document iteration as a function of the number topics. We see that original SCVB0 requires average linear runtime due to \(O(K)\) operations to compute collapsed variational distribution, while the average runtime for SparseSCVB0 grows sublinearly in \(K\), due to \(O(k_{d})\) operations instead of \(O(K)\) operations. SparseSCVB0 starts with approximately \(O(K)\) operations in its initial stage of iterations, but it starts getting a benefit from sparsification heuristic after burning in, as shown in Figure 1(b) for \(K=10,000\).

To evaluate the performance in terms of learned topic quality, we start by comparing all of the algorithms in qualitative experiments (see Table 1, Table 2, and Table 3) where we show randomly selected example topics, while all the models were trained on the NIPS, PubMed, and Wikipedia corpus for \(K=500\), \(K=1,000\), and \(1,000\), respectively. To get a quantitative insight we evaluated the topics using the per topic coherence metric, which measures the semantic quality of a topic based on the \(W\) most probable words for the topics [46], thereby approximating the user viewing experience. In Figure 2, we see that SparseSCVB0 generates better quality topics with higher coherence scores than the other two models for \(K=1000\) with \(W=10\) after running all the models on each corpus for the same amount of time. The coherence performance of SparseSCVB0 increases substantially in the case of the large-scale corpus (Figure 2(c)), since it gets the opportunity to use its runtime advantage and process more documents than the other algorithms.

To investigate model convergence, we measured the held-out log-probability versus wall-clock time for all the algorithms. For each experiment we held-out a set of documents (\(150\) documents for NIPS, \(3500\) documents for Reuters, and \(1000\) documents for all other corpora) as test data

Figure 3: Comparison of average log-likelihood \(vs\). Time for LDA on (a) NIPS, (b) PubMed, and (c) Wikipedia.

 

and trained the model on the rest of the corpus. Then, we split each test document in half, estimated local parameters on first half and finally computed the log-likelihood of the remaining half of the document. Figure 3 shows the comparison of average log-likelihood versus wall-clock time for all four corpora. In terms of log-likelihood, SparseSCVB0 provides an approximately similar result to original SCVB0 for the small corpus, but it converged to a better solution than others in the case of large corpora like Wikipedia (see Figure 3(c)), likely due to its processing a larger number of documents.

SparseSCVB0 enables the large-scale computation needed to learn high-dimensional topic models that could not feasibly be trained using previous methods due to their runtime complexity in the number of documents and/or topics. We show randomly selected topics from the LDA model with \(K=10,000\) in Table 5. This big topic model was trained for \(36\) hours using SparseSCVB0 on Wikipedia. We performed a dense initialization, running original SCVB0 for the first \(5\) hours, which was found to help avoid local optima.

### Performance for MMSGTM

We also conducted experiments to evaluate the performance of SparseSCVB0 for MMSGTM and compare with original SCVB0. In all MMSGTM experiments, we kept the same step size schedule for global parameters as \(scale=1.0\), \(\eta=5.0\) and \(\kappa=0.9\), but for local parameter updates we maintain a separate step-size schedule of \(\frac{scale}{(\eta+t)^{\kappa}}\) for each input word, with \(t\) referring to the number of times we processed this input word, while \(\eta\) and \(\kappa\) values remained the same. For simulated annealing of SparseSCVB0, we used \(T_{0}=0.00001\), \(\kappa=0.9\), \(\mu=5\) and \(\lambda=|context|\) with a \(context\) size of \(5\). We kept the same number of document burn-in passes as we did for the LDA experiments.

In Figure 4, we show the runtime improvement of SparseSCVB0 over original SCVB0 for MMSGTM in a similar experiment to the one for LDA. For MMSGTM, SparseSCVB0 substantially outperforms original SCVB0 by processing each document in amortized \(O(1)\) time. We provide qualitative results in the case of MMSGTM model by showing several top words in the top \(2\) topics for an input word using original SCVB0 and SparseSCVB0 in Table 4 for \(K=500\). As for LDA, SparseSCVB0 allows us to generate topics with higher coherence scores compared to the original SCVB0 after running for the same amount of time (Figure 5) on both small and large corpora.

## 7 Conclusions

This paper introduced SparseSCVB0, a sparse version of the SCVB0 inference algorithm which performs fast, scalable high-dimensional topic model inference. SparseSCVB0 leverages stochasticity to scale well in both the corpus size and in the number of topics. It operates in the collapsed representation of topic models which leads to fast convergence while providing an improved variational bound. We show that SparseSCVB0 reduces the operational complexity for the variational Bayes update of online topic models from \(O(K)\) to \(O(k_{d})\) time for LDA and amortized \(O(1)\) time for MMSGTM. We evaluated and compared the performance of our approach with state-of-the-art models such as original SCVB0 and SVI to demonstrate that SparseSCVB0 converges much more efficiently, while maintaining high quality topics with a better per-topic coherence score.

Figure 4: Comparison of runtime per iteration for MMSGTM in terms of: (a) number of topics \(K\) and (b) number of iterations when \(K=5,000\). SparseSCVB0 runs in amortized \(O(1)\) time, while original SCVB0 is linear in \(K\).

Figure 5: Per-topic coherence for MMSGTM, \(K=500\), on (a) NIPS and (b) Wikipedia. SparseSCVB0 has higher coherence scores for both the small and large corpora.

 



## References

* Anandkumar et al. (2014) Animashree Anandkumar, Rong Ge, Daniel Hsu, Sham M Kakade, and Matus Telgarsky. 2014. Tensor decompositions for learning latent variable models. _The Journal of Machine Learning Research_, 15(1):2773-2832.
* Arora et al. (2012) Sanjeev Arora, Rong Ge, and Ankur Moitra. 2012. Learning topic models-going beyond SVD. In _Foundations of Computer Science (FOCS), 2012 IEEE 53rd Annual Symposium on_, pages 1-10. IEEE.
* Asuncion et al. (2009) Arthur Asuncion, Max Welling, Padhraic Smyth, and Yee Whye Teh. 2009. On smoothing and inference for topic models. In _Proceedings of the 25th Conference on Uncertainty in Artificial Intelligence_, pages 27-34. AUAI Press.
* Bezanson et al. (2017) Jeff Bezanson, Alan Edelman, Stefan Karpinski, and Viral B Shah. 2017. Julia: A fresh approach to numerical computing. _SIAM review_, 59(1):65-98.
* Blei et al. (2003) David M Blei, Andrew Y Ng, and Michael I Jordan. 2003. Latent Dirichlet allocation. _Journal of Machine Learning Research_, 3(Jan):993-1022.
* Das et al. (2015) Rajarshi Das, Manzil Zaheer, and Chris Dyer. 2015. Gaussian LDA for topic models with word embeddings. In _Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and The 7th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)_, volume 1, pages 795-804.
* Foulds (2018) J. R. Foulds. 2018. Mixed membership word embeddings for computational social science. _Proceedings of the 21st International Conference on Artificial Intelligence and Statistics (AISTATS)_.
* Foulds et al. (2013) James Foulds, Levi Boyles, Christopher DuBois, Padhraci Smyth, and Max Welling. 2013. Stochastic collapsed variational Bayesian inference for latent Dirichlet allocation. In _Proceedings of the 19th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining_, pages 446-454. ACM.
* Griffiths and Steyvers (2004) Thomas L Griffiths and Mark Steyvers. 2004. Finding scientific topics. _Proceedings of the National Academy of Sciences_, 101(suppl 1):5228-5235.
* Hinton (2002) Geoffrey E Hinton. 2002. Training products of experts by minimizing contrastive divergence. _Neural Computation_, 14(8):1771-1800.
* Hoffman et al. (2010) Matthew Hoffman, Francis R Bach, and David M Blei. 2010. Online learning for latent Dirichlet allocation. In _Advances in Neural Information Processing Systems_, pages 856-864.
* Hoffman et al. (2013) Matthew D Hoffman, David M Blei, Chong Wang, and John Paisley. 2013. Stochastic variational inference. _The Journal of Machine Learning Research_, 14(1):1303-1347.
* Li et al. (2014) Aaron Q Li, Amr Ahmed, Sujith Ravi, and Alexander J Smola. 2014. Reducing the sampling complexity of topic models. In _Proceedings of the 20th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining_, pages 891-900. ACM.
* Liu et al. (2015) Yang Liu, Zhiyuan Liu, Tat-Seng Chua, and Maosong Sun. 2015. Topical word embeddings. In _AAAI_, pages 2418-2424.
* Marsaglia et al. (2004) George Marsaglia, Wai Wan Tsang, Jingbo Wang, et al. 2004. Fast generation of discrete random variables. _Journal of Statistical Software_, 11(3):1-11.
* Mimno et al. (2012) David Mimno, Matt Hoffman, and David Blei. 2012. Sparse stochastic inference for latent Dirichlet allocation. _Proceedings of the 29th International Conference on Machine Learning_.
* Mimno et al. (2011) David Mimno, Hanna M Wallach, Edmund Talley, Miriam Leenders, and Andrew McCallum. 2011. Optimizing semantic coherence in topic models. In _Proceedings of the Conference on Empirical Methods in Natural Language Processing_, pages 262-272. Association for Computational Linguistics.
* Newman et al. (2008) David Newman, Padhraic Smyth, Max Welling, and Arthur U Asuncion. 2008. Distributed inference for latent Dirichlet allocation. In _Advances in Neural Information Processing Systems_, pages 1081-1088.
* Salakhutdinov et al. (2003) Ruslan Salakhutdinov, Sam T Roweis, and Zoubin Ghahramani. 2003. Optimization with EM and expectation-conjugate-gradient. In _Proceedings of the 20th International Conference on Machine Learning (ICML-03)_, pages 672-679.
* Teh et al. (2007) Yee W Teh, David Newman, and Max Welling. 2007. A collapsed variational Bayesian inference algorithm for latent Dirichlet allocation. In _Advances in Neural Information Processing Systems_, pages 1353-1360.
* Yuan et al. (2015) Jinhui Yuan, Fei Gao, Qirong Ho, Wei Dai, Jinliang Wei, Xun Zheng, Eric Po Xing, Tie-Yan Liu, and Wei-Ying Ma. 2015. LightLDA: Big topic models on modest computer clusters. In _Proceedings of the 24th International Conference on World Wide Web_, pages 1351-1361. International World Wide Web Conferences Steering Committee