Year: 2022

Abstract:
A topic model is often formulated as a generative model that explains how each word of a document is generated given a set of topics and document-specific topic proportions. It is focused on capturing the word co-occurrences in a document and hence often suffers from poor performance in analyzing short documents. In addition, its parameter estimation often relies on approximate posterior inference that is either not scalable or suffers from large approximation error. This paper introduces a new topic-modeling framework where each document is viewed as a set of word embedding vectors and each topic is modeled as an embedding vector in the same embedding space. Embedding the words and topics in the same vector space, we define a method to measure the semantic difference between the embedding vectors of the words of a document and these of the topics, and optimize the topic embeddings to minimize the expected difference over all documents. Experiments on text analysis demonstrate that the proposed method, which is amenable to mini-batch stochastic gradient descent based optimization and hence scalable to big corpora, provides competitive performance in discovering more coherent and diverse topics and extracting better document representations. The code is available at https://github.com/BoChenGroup/WeTe.

heading: 1 INTRODUCTION
text: For text analysis, topic models are widely used to extract a set of latent topics from a corpus (a collection of documents). The extracted topics, revealing common word co-occurrence patterns within a document, often correspond to semantically meaningful concepts in the training corpus. Bayesian probabilistic topic models (BPTMs), such as latent Dirichlet allocation (LDA) (Blei et al., 2003; Griffiths & Steyvers, 2004) and its nonparametric Bayesian generalizations (Teh et al., 2006; Zhou et al., 2012), have been the most popular ones. A BPTM is often formulated as a generative model that explains how each word of a document is generated given a set of topics and document-specific topic proportions. Bayesian inference of a BPTM is usually based on Gibbs sampling or variational inference (VI), which can be less scalable for big corpora and need to be customized accordingly.
With the recent development in auto-encoding VI, originated from variational autoencoders (VAEs) (Kingma & Welling, 2014; Rezende et al., 2014), deep neural networks have been successfully used to develop neural topic models (NTMs) (Miao et al., 2016; Srivastava & Sutton, 2017; Burkhardt & Kramer, 2019; Zhang et al., 2018; Dieng et al., 2020; Zhao et al., 2021). The key advantage of NTMs is that approximate posterior inference can be carried out easily via a forward pass of the encoder network, without the need for expensive iterative inference scheme per test observation as in both Gibbs sampling and conventional VI. Hence, NTMs enjoy better flexibility and scalability than BPTMs. However, the reparameterization trick in VAEs cannot be directly applied to the Dirichlet (Burkhardt & Kramer, 2019) or gamma distributions (Zhang et al., 2018), which are usually used as the prior and conditional posterior of latent topics and topic proportions, so approximations have to be used, potentially putting additional complexity or approximation errors.
∗Equal contribution.
To address the above shortcomings, we in this paper propose a novel topic modeling framework in an intuitive and effective manner of enjoying several appealing properties over previously developed BPTMs and NTMs. Like other TMs, we also focus on learning the global topics shared across the corpus and the document-specific topic proportions, which are the two key outputs of a topic model. Without building an explicit generative process, we formulate the learning of topic model (e.g., optimizing the likelihood) as the process of minimizing the distance between each observed document j and its corresponding trainable distribution. More specifically, the former (document j) can be regarded as as an empirical discrete distribution Pj , which has an uniform measure over all the words within this document. To construct the latter (trainable distribution), we can represent Pj with K shared topics and its K-dimensional document-specific topic proportion, defined as Qj , where we view shared topics as K elements and topic proportion as the probability measure in Qj . It is very reasonable since the k-th element in topic proportion measures the weight of topic k for a document, and the document can be represented perfectly using the learned topic proportion and topics from a desired TM. Recalling that each topic and word are usually live in the V -dimensional (vocabulary size) space in TMs, it might be difficult to directly optimize the distance between Pj and Qj over V -dimensional space. Motivated by Dieng et al. (2020), we further assume that both topics and words live in a H-dimensional embedding space, much smaller than vocabulary space. By abuse of notation, we still use Pj over the word embeddings and Qj over the topic embeddings as two representations for document j. Below, we turn towards pushing the document-specific to-be-learned distribution Qj to be as close as possible to the empirical distribution Pj .
To this end, we develop a probabilistic bidirectional transport based method to measure the semantic difference between the two discrete distributions in an embedding space. By minimizing the expected difference between two Pj and Qj over all documents, we can learn the topic and word embeddings directly. Importantly, we naturally leverage semantic distances between topics and words in an embedding space to construct the point-to-point cost of moving between them, where the cost becomes a function of topic embeddings. Notably, we consider linking the word embeddings in Pj and topic embeddings in Qj in a bidirectional view. That is, given a word embedding drawn from Pj , it is more likely to be linked to a topic embedding that both is closer to it in the embedding space and exhibits a larger proportion in Qj ; vice versa. Our proposed framework has several key properties: 1) By bypassing the generative process, our proposed framework avoids the burden of developing complex sampling schemes or approximations for the posterior of BPTMs or NTMs. 2) The design of our proposed model complies with the principles of TMs, whose each learned topic describes an interpretable semantic concept. More interestingly, our model is flexible to learn word embeddings from scratch or use/finetune pretrained word embeddings. When pretrained word embeddings are used, our model naturally alleviates the issue of insufficient word co-occurrence information in short texts as discussed by prior work (Dieng et al., 2020; Zhao et al., 2017; 2021), which is one of the key drawbacks of many BPTMs and NTMs. 3) Conventional TMs usually enforce a simplex constraint on the topics over a fixed vocabulary, which hinders their applications in the case where the vocabulary varies. In our method, we view a document as a mixture of a set of words, which facilitates the deployment of the model when the size of the vocabulary varies.
Finally, we have conducted comprehensive experiments on a wide variety of datasets in the comparison with advanced BPTMs and NTMs, which show that our model can achieve the state-of-the-art performance as well as applealing interpretability.

heading: 2 BACKGROUND
text: Topic Models: TMs usually represent each document in a corpus as a bag-of-words (BoW) count vector x ∈ RV+ , where xv represents the occurrences of word v in the vocabulary of size V . A TM aims to discover K topics in the corpus, each of which describes a specific semantic concept. A topic is or can be normalized into a distribution over the words in the vocabulary, named word distribution, φk ∈ ΣV , where ΣV is a V − 1 dimensional simplex and φvk indicates the weight or relevance of word v under this topic k. Each document comes from a mixture of topics, associated with a specific mixture proportion, which can be captured by a distribution over K topics, named topic proportion, θ ∈ ΣK , where θk indicates the weight of topic k for a document. As the most fundamental and popular series of TMs, BPTMs (Blei et al., 2003; Zhou et al., 2012; Hoffman et al., 2010) generate the document x with latent variables (i.e., topics {φk}Kk=1 and topic proportion θ) sampled from pre-specified prior distributions (e.g., Gamma or Dirichlet). Like
other Bayesian models, the learning process of a BPTM relies on Bayesian inference, such as variational inference or Gibbs sampling. Recently, NTMs (Miao et al., 2016; Srivastava & Sutton, 2017; Burkhardt & Kramer, 2019; Zhang et al., 2018; Dieng et al., 2020; Zhao et al., 2021) have attracted significant research interests in topic modeling. Most existing NTMs can be regarded as extensions of BPTM like LDA within the VAEs framework (Zhao et al., 2021). In general, NTMs consist of an encoder network that maps the (normalized) BoW input x to its topic proportion θ, and a decoder network that generates x conditioned on the topics {φk}Kk=1 and proportion θ. Despite their appealing flexibility and scalability, due to the unusable reparameterization trick in original VAEs for the Dirichlet or gamma distributions, NTMs have to develop complex sampling schemes or approximations, leading to potentially large approximation errors or learning complexity.
Compare Two Discrete Distributions: This paper aims to quantify the difference between two discrete distributions (word embeddings and topic embeddings), whose supports are points in the same embedding space. Specifically, let p and q be two discrete probability measures on the arbitrary space X ⊆ RH , formulated as p = ∑n i=1 uiδxi and q = ∑m j=1 vjδyj , where u = [ui] ∈ Σn and v = [vj ] ∈ Σm denote two distributions of the discrete states. To measure the distance between p and q, a non-trivial way is to use optimal transport (OT) (Peyré & Cuturi, 2019), which defines the distance as an optimization problem as
OT (p, q) := min T∈Π(u,v)
Tr ( T>C ) , (1)
where C ∈ Rn×m≥0 is the transport cost matrix with Cij = c (xi, yj), T ∈ R n×m >0 a doubly stochastic transport matrix such that Π(u,v) = { T | T1Dv = u,T>1Du = v } , Tij the transport probability between xi and yj , and Tr(·) the matrix trace. Since the transport plan is imposed on the constraint of T ∈ Π(u,v), it has to be computed via constrained optimizations, such as the iterative Sinkhorn algorithm when an additional entropy regularization term is added (Cuturi, 2013).
The recently introduced conditional transport (CT) framework (Zheng & Zhou, 2020) can be used to measure the difference between two discrete distributions, which, like OT distance, does not require the distributions to share the same support. CT considers the transport plan in a bidirectional view, which consists of a forward transport plan as Tp→q and backward transport plan Tp←q. Therefore, the transport cost between two empirical distributions in CT can be expressed as
CT (p, q) := min Tp→q,Tq→p
Tr [ (Tp→q)>C + (Tq→p)>C ] . (2)
CT specifies Tp→qij =ui vje −dψ(yj,xi)∑m
j′=1 vj′e −dψ(xi,yj′ )
and Tp←qij =vj uie −dψ(yjxi)∑n
i′=1 ui′e −dψ(xi′ ,yj)
and hence
Tp→qij 1Dv = u and (T p←q ij ) T1Du = v but in general T p←q ij 1Dv 6= u and (T p→q ij ) T1Du 6= v. This provides a simpler way to measure the difference between p and q. Here dψ (x, y) = dψ (y, x) parameterized by ψ is defined to measure the difference between two vectors. This flexibility of CT potentially facilitates an easier integration with deep neural networks with a lower complexity and better scalability. These properties can be helpful to us in the development of a new topic modeling framework based on transportation between distributions, especially for neural topic models.

heading: 3 LEARNING MIXTURE OF TOPIC EMBEDDINGS
text: Now we will describe the details of the proposed model. Since it represents a mixture of Word Embeddings as a mixture of Topic Embeddings, we refer to it as WeTe. Specifically, consider a corpus of J documents, where the vocabulary contains V distinct terms. Unlike in other TMs, where a document is represented as a BoW count vector x ∈ RV+ as shown in Section 2, we denote each document as a set of its words, defined as Dj = [wji], where wji ∈ {1, . . . , V } means the i-th word in the j-th document with i ∈ [1, Nj ], andNj is the length of the j-th document. Assume E ∈ RH×V as the word embedding matrix whose columns contain the embedding representations of the terms in the vocabulary. By projecting each word into the corresponding word-embedding space, we thus represent each document as an empirical distribution Pj on the word embedding space as follows
Pj = Nj∑ i=1 1 Nj δwji , wji ∈ RH . (3)
Similar to other TMs, we aim to learn K topics from the corpus. However, instead of representing a topic as a distribution over the terms in the vocabulary, we use an embedding vector for each topic, αk ∈ RH . Here topic embedding αk is a distributed representation of the k-th topic in the same semantic space of the word embeddings. Collectively, we form a document-specific empirical topic distribution Qj (on the embedding space), defined as
Qj = K∑ k=1 θ̃jkδαk , αk ∈ RH . (4)
Here θ̃j,1:K denotes the normalized topic proportions of document j, i.e., θ̃j := θj/ ∑K k=1 θjk. We focus on learning the topic distribution Qj that is close to distribution Pj . Exploiting the CT loss defined in Eq. (2), we introduce WeTe as a novel topic model for text analysis. For document j, we propose to minimize the expected difference between the word embeddings from Pj and topic embeddings from Qj in terms of its topic proportion and topic embeddings. For all the documents in the corpus, we can minimize the average CT loss
min α,Θ
1
J J∑ j=1 [CT(Pj , Qj)]. (5)
As a bidirectional transport, CT(·) consists of a doc-to-topic CT that transports the word embeddings to topic embeddings, and a topic-to-doc CT that reverses the transport direction. We define a conditional distribution specifying how likely a given topic embedding φk will be transported to word embedding αji in document j as
πNj (wji |αk) = Pj(wji)e −d(wji,αk)∑Nj i′=1 Pj(wji′)e −d(wji′ ,αk) = e−d(wji,αk)∑Nj i′=1 e −d(wji′ ,αk) , wji ∈ {wj1, . . . ,wjNj}, (6) where d(wi,αk) = d(αk,wi) indicates the semantic distance between the two vectors. Intuitively, if αk and wji have a small semantic distance, the π(wi |αk) would have a high probability. This construction makes it easier to transport αk to a word that is closer to it in the embedding space. For document j with Nj words {wj1, . . . ,wjNj}, the topic-to-doc CT cost can be expressed as
LQj→Pj = Eαk∼QjEwji∼πNj (· |αk)[c(wji,αk)] = K∑ k=1 θ̃jk Nj∑ i=1 c(wji,αk)π(wji |αk), (7)
where c(wji,αk) = c(αk,wji) ≥ 0 denotes the point-to-point cost of transporting between word embeddingwji and topic embedding αk, and θ̃k,j can be considered as the weight of transport cost between all words in document j and topic embedding k from a geometric viewpoint. Similar to but different from Eq. (7), we introduce the doc-to-topic CT, whose transport cost is defined as
LPj→Qj = Ewji∼PjEαk∼πK(· |wji)[c(wji,αk)] = Nj∑ i=1 1 Nj K∑ k=1 c(wji,αk)π(αk |wji), (8)
where 1Nj denotes the weight of transport cost between wordwji in document j and topic embeddings from a geometric viewpoint. In contrast to Eq. (6), we define the conditional transport probability from word embedding wji in document j to a topic embedding φk with
π(αk |wji) = Qj(αk)e −d(wji,αk)∑K k′=1Qj(αk′)e −d(wji′ ,αk′ ) = e−d(wji,αk)θ̃jk∑K k′=1 e −d(wji,αk′ )θ̃jk′ , (9)
where θ̃jk = Qj(αk) can be interpreted as the prior of global topic embedding αk in document j.
We have not specified the form of c(wji,αk) and d(wji,αk). A naive definition of the transport cost or semantic distance between two points is some distance between their raw feature vectors. In our framework, we specify the following construction of cost function:
c(wji,αk) = e −wTjiαk . (10)
Here the cost function is defined for two reasons: the inner product is the commonly-used way to measure the difference between two embedding vectors, and the cost needs to be positive. For semantic distance, we directly take the inner product of the word embedding wji and the topic embedding αk, i.e., d(wji,αk) = −wTjiαk, although other choices are possible.

heading: 3.1 REVISITING OUR PROPOSED MODEL FROM TOPIC MODELS
text: Generally speaking, conditioned on an observed document, traditional TMs (Blei et al., 2003; Zhou et al., 2012; Srivastava & Sutton, 2017) often decompose the distribution of the document’s words into two learnable factors: the distribution of words conditioned on a certain topic, and the distribution of topics conditioned on the document. Here, we establish the connection between our model and traditional TMs. Recall that d(wi,αk) indicates the semantic distance between topic k and word i in the embedding space. For arbitrary αk and wi, the more similar they are, the smaller underlying distance they have. Following this viewpoint, we assume φk ∈ RV+ as the distribution-over-words representation of topic k and treat its element as
φvk := e−d(vv,αk)∑V
v′=1 e −d(vv′ ,αk)
, (11)
where vv ∈ RH denotes the embedding of the v-th word in the vocabulary. Therefore, the column vector φk weights the importance of the words in the corresponding topic k. With this form, our proposed model assigns a probability to a word in topic k by measuring the agreement between the word’s and topic’s embeddings. Conditioned on Φ = [φk], the flexibility of CT enables multiple ways to learn or define the topic proportions of documents, i.e., Θ detailed in Section 3.2. With CT’s ability for modeling geometric structures, our model avoids developing the prior/posterior distributions and the associated sampling schemes, which are usually nontrivial in traditional TMs.

heading: 3.2 LEARNING TOPIC EMBEDDINGS AND TOPIC PROPORTIONS
text: Given the corpus of J documents, we wish to learn the topic embedding matrix α and topic proportions of documents Θ. Based on the doc-to-topic and topic-to-doc CT losses and the definitions of c and d in Eq. (6-10) and ∑K k=1 θ̃jk = 1 , we can rewrite the CT loss in Eq. 5 as
1
J J∑ j=1 CT(Pj , Qj) = 1 J J∑ j=1  K∑ k=1 θ̃jk∑Nj i′=1 e wT ji′αk 1
Nj
+  Nj∑ i=1 1 Nj∑K k′=1 e wTjiαk′ θ̃jk′  , (12) whose detailed derivation is shown in Appendix A. The two terms in the bracket exhibit appealing symmetry properties between the normalized topic proportion θ̃j and word prior 1Nj . To minimize the first term, for a given document whose topic proportion has a non-negligible activation at the k-th topic, the inferred k-th topic needs to be close to at least one word (in the embedding space) of that document. Similarly each word in document j needs to find at least a single non-negligibly-weighted topic that is sufficiently close to it. In other words, the learned topics are expected to have a good coverage of the word embedding space occupied by the corpus by optimizing those two terms.
Like other TMs, the latent representation of the document is a distribution over K topics: θ̃j ∈ ΣK , each element of which denotes the proportion of one topic in this document. Previous work shows that the data likelihood can be helpful to regularize the optimization of the a transport based loss (Frogner et al., 2015; Zhao et al., 2021). To amortize the computation of θj and provide additional regularization, we introduce a regularized CT loss as
min α,W
1
J J∑ j=1 Eθj∼qW (· |xj) [CT(Pj , Qj)− log p(xj ;α,θj)] , (13)
where qW (θj |xj) is a deterministic or stochastic encoder, parameterized by W , p(xj ; Φ,θj) = Poisson(xj ; ∑K k=1 φkθjk) (φk is defined as Eq. (11)) is the likelihood used in Poisson factor analysis (Zhou et al., 2012), and is a trade-off hyperparameter between the CT loss and log-likelihood. Here, we encode θ with the Weibull distribution: qW (θj |xj) = Weibull(fW (xj), gW (xj)), where f and g are two related neural networks parameterized byW . Similar to previous work (Zhang et al., 2018; Duan et al., 2021), we choose Weibull mainly because it resembles the gamma distribution and is reparameterizable, as drawing m ∼ Weibull(k, λ) is equivalent to mapping m = f̂( ) := λ(−log(1 − ))1/k, ∼ Uniform(0, 1). Different from previous work, here qW (θj |xj) does not play the role of a variational inference network that aims to approximate the posterior distribution given the likelihood and a prior. Instead, it is encouraged to strike a balance between minimizing the
CT cost, between the document representation in the word embedding space and that in the topic embedding space, and minimizing the negative log-likelihood of Poisson factor analysis, with the document representation shared between both components of the loss.
The loss of Eq. (13) is differentiable in terms of α and W, which can be optimized jointly in one training iteration. The training algorithm is outlined in Appendix B. Benefiting from the encoding network, after training the model, we can obtain θj by mapping the new input xj with the learned encoder W, avoiding the hundreds iterations in MCMC or VI to collect posterior samples for local variables. The algorithm for WeTe can either use pretrained word embeddings, e.g., GloVe (Pennington et al., 2014), or learn them from scratch. Practically speaking, using pretrained word embeddings enables more efficient learning for reducing the parameter space, and has been proved beneficial for short documents for leveraging the rich semantic information in pretrained word embeddings. In our experiments, WeTe by default uses the GloVe word embedding.

heading: 4 RELATED WORK
text: Models with Word Embeddings: Word embeddings have been widely used as complementary information to improve topic models. Skipgram-based models (Shi et al., 2017; Moody, 2016; Park & Lee, 2020) jointly skip-gram word embeddings and the latent topic distributions under the Skipgram Negative-Sampling objective. Those models incorporate the topical context into the central words to generate its surrounding words, which share similar idea with the topic-to-doc transport in WeTe that views the topic vocter as the central words, and words within a document as the surrounding words. Besides, WeTe forces the inferred topics are close to at least one word embedding vector of a given document by the doc-to-topic transport, which is not considered in those models. For BPTMs, word embeddings are usually incorporated into the generative process of word counts (Petterson et al., 2010; Nguyen et al., 2015; Li et al., 2016; Zhao et al., 2017; Keya et al., 2019). Benefiting from the flexibility of NTMs, word embeddings can be either incorporated as part of the encoder input, such as in Card et al. (2018), or used in the generative process of words, such as in Dieng et al. (2020). Because these models construct the explicit generative processes from the latent topics to documents and belong to the extensions of BPTMs or NTMs, they may still face these previously mentioned difficulties in TMs. Our method naturally incorporates word embeddings into the distances between topics and words with the bidirectional transport framework, which is different from previous ones.
Models by Minimizing Distances of Distributions: Yurochkin et al. (2019) adopt the OT distance to compare two documents’ similarity between their topic distributions extracted from a pretrained LDA, but their focus is not to learn a topic model. Nan et al. (2019) extend the framework of Wasserstein AutoEncoders (WAEs) (Tolstikhin et al., 2018) to minimize the Wasserstein distance between the fake data generated with topics and real data, which can be interpreted as an OT variant to NTMs based on VAE. In addition, Xu et al. (2018) introduce Distilled Wasserstein Learning (DWL), where an observed document is approximated with the weighted Wasserstein barycentres of all the topic-word distributions and the weights are viewed as the topic proportion of that document. The Optimal Transport based LDA (OTLDA) of Huynh et al. (2020) is proposed to minimize the regularized optimal transport distance between document distribution xj and topic distribution about M in the vocabulary space. Also, Neural Sinkhorn Topic Model (NSTM) of Zhao et al. (2020) is proposed to learn the topic proportion θ from the encoder to be as close the normalized BoW vector x̃j . Compared with NSTM, by representing a document as a mixture of word embeddings and a mixture of topic embeddings, our model directly minimizes the CT cost between them in the same embedding space. Moreover, NSTM needs to feed the pretrained word embeddings to construct the cost matrix in Sinkhorn algorithm, while our WeTe can learn word and topic embeddings jointly from scratch. Finally, our model avoids Sinkhorn iterations during each iteration at the training stage.

heading: 5 EXPERIMENTS

heading: 5.1 EXPERIMENTAL SETTINGS
text: Datasets: To demonstrate the robustness of our WeTe in terms of learning topics and document representation, we conduct the experiments on six widely-used textual data, including regular and short documents, varying in scales. The datasets include 20 News Group (20NG), DBpedia (DP) (Lehmann et al., 2015), Web Snippets (WS) (Xuan et al., 2008), Tag My News (TMN) (Vitale et al., 2012), Reuters extracted from the Reuters-21578 dataset, and Reuters Corpus Volume 2 (RCV2) (Lewis et al., 2004), where WS, DP and TMN are short documents. The statistics and detailed descriptions of the datasets are shown in Appendix C.
Evaluation metrics: Following Dieng et al. (2020) and Zhao et al. (2020), we use Topic Coherence (TC) and Topic Diversity (TD) to evaluate the quality of the learned topics. TC measures the average Normalized Pointwise Mutual Information (NPMI) over the top 10 words of each topic, and a higher score indicates more interpretable topics. TD denotes the percentage of unique words in the top 25 words of the selected topics. To comprehensively evaluate topic quality, we choose the topics with the highest NPMI and report the average score over those selected topics, where we vary the proportion of the selected topics from 10% to 100%. A good TM also provides good document representation, we thus calculate Normalized Mutual Information (NMI) (Schütze et al., 2008) and purity on WS, RCV2, DP and 20NG on clustering tasks, where we use the 6 super-categories as 20NG’s ground truth and denote it as 20NG(6). We split all datasets according to their default training/testing division, and train a model on the training documents. Given the trained model, we collect the topic proportion θ on the testing documents and apply the K-Means algorithm on it, where the purity and NMI of the K-Means clusters are measured. Similar to Zhao et al. (2020), we set the number of clusters of K-Means as N = 52 for RCV2 and N = 20 for other datasets. For all the metrics, higher values mean better performance. Baseline methods and their settings: We compare the performance of our proposed model with the following baselines: 1) traditional BPTMs, including LDA (Blei et al., 2003), a well-known topic model (here we use its collapsed Gibbs sampling extension (Griffiths & Steyvers, 2004)) and Poisson Factor Analysis (PFA) (Zhou et al., 2012), a hierarchical Bayesian topic model under the Poisson likelihood; 2) VAEs based NTMs, such as Dirichlet VAE (DVAE) (Burkhardt & Kramer, 2019) and Embedded Topic Model (ETM), a generative model that marries traditional topic models with word embeddings; 3) OT based NTM, Neural Sinkhorn Topic model (NSTM) (Zhao et al., 2020), which learns the topic proportions by directly minimizing the OT distance to a document’s word distribution; 4) TMs designed for short texts, including Pseudo-document-based Topic Model (PTM) (Zuo et al., 2016) and Word Embedding Informed Focused Topic Model (WEI-FTM) (Zhao et al., 2017). In summary, ETM, NSTM, WEI-FTM, and our WeTe are the ones with pretrained word embeddings. For all baselines, we use their official default parameters with best reported settings.
Quantitative results: For all models, we run the algorithms in comparison five times by modifying only the random seeds and report the mean and standard deviation (as error bars). We first examine the quality of the topics discovered by WeTe. Fig. 1(a) shows the results of TC and TD on three corpora (more result can be found in Appendix D.), varying in scales. Due to limited space, we only choose PFA and WEI-FTM as representatives of their respective methods. Since the Gibbs sampling based methods (e.g., PFA, WEI-FTM) require walking through all documents in each iteration, it is not scalable to big data like RCV2. WEI-FTM only works on short texts. There are several observations drawn from different aspects. For the short texts (WS), WeTe has comparable performance with NSTM, and is much better than WEI-FTM, which are designed specifically for short texts. This observation confirms that our WeTe is effective and efficient in learning coherent and diverse topics from the short texts with pretrained word embeddings, without designing the specialized architecture. In addition, for the regular and large datasets (20NG, RCV2), our proposed WeTe significantly outperforms the others in TC while achieving
higher TD. Although some TMs (NSTM, ETM, WEI-FTM) utilize the pretrained word embeddings, it is demonstrated that how to assimilate them into topic model is the key factor. Thus we provide a reference for future studies along the line of combining word embeddings and topic model. Compared with WeTe, WeTe(F) and WeTe(N) need to learn word and topic embeddings from the current corpus, whose size usually less than 1M, resulting in sub-optimal topics discovering. From Fig. 1(a), we can further find that those two variants achieve comparable result with other NTMs for top-20% topics, which means the proposed model has the ability to discover interpretable topics only from the given corpus without loading pretrained word embeddings. Fig. 1(b) denotes topic quality of our WeTe and its variants with different corpus scalar. it shows that WeTe(F) and WeTe(N) reach a performance close to that of WeTe as the scalar of the corpus becomes large, suggesting that the proposed model has the potential to learn meaningful word embeddings on large datasets.
The clustering Purity and NMI for various methods are shown in Table 1. Overall, the proposed model is superior to its competitors on all datasets. Compared with NSTM, which learns topic proportions θ by minimizing OT distance to a document’s word distribution, WeTe employs a probabilistic bidirectional transport method to learn θ and topic embeddings jointly, resulting in more distinguishable document representations. Besides, with the ability to finetune/learn word embeddings from the current corpus, WeTe(F) and WeTe(N) can better infer the topic proportion θ, and hence give better performance in terms of document clustering. Those encouraging results show that not only the proposed model can discover the topics with high quality, but also learn good document representations for downstream clustering task on both short and regular documents. It
thus indicates the benefit of minimizing the semantic distance between mixture of word embeddings and mixture of topic embeddings.
hyperparameter sensitivity We fix the hyperparameter = 1.0, which control the weight of the Poisson likelihood in Eq. 13 in the previous experiments for fair comparison. Here we report the result of WeTe on 20NG with different in Fig. 2, where topic quality (TQ) is calculated as TQ = TC ∗ TD. We also report two variant of WeTe, one trained using only CT cost (WeTe(CT)), and the other using only likelihood (WeTe(TM)). We can see that 1), can be fine-tuned to balance betweeen document representation and topic quality. By carefully fine-tuning for each dataset, one can achieve even better performance than those reported in our experiments; 2), the CT cost leads to high topic quality, and the likelihood has benefits for the representation of documents. By combining these two objectives together, WeTe can produce better performance than using only either of them.
Qualitative analysis: Fig. 3(a) visualizes the learned topic embeddings. we present the top-9 topics with the highest NPMI learned by our proposed model on 20NG. For each topic, we select its top-6 words according to φk, and then feed their embeddings together into the t-SNE tool (Van der Maaten & Hinton, 2008). We can observe that the topics are highly interpretable in the word embedding space, where each topic is close to semantically related words. Besides, those words under the same topic are closer together, and words under different topics are far apart. We can also see that the related topics are also closer in the embedding space, such as topic #2 about “people, children” and topic #5 about “school, student.” Fig. 3(b) gives an overview of all word embeddings and learned topic embeddings. We find that the topic embeddings (red stars) are distributed evenly in the word embedding space, each of which plays the role of a cluster center surrounded by semantically related words. Those interesting results illustrate our motivation that we can use the mixtures of topic embeddings to represent mixtures of word embeddings based on the CT cost between them. Given the query desktop, Fig. 3(c) compares three most related topics learned from NSTM and WeTe, where WeTe tends to discover more diverse topics than NSTM. We attribute this to the introduction of topic-to-doc cost, which enforces the topic to transport to all words that are semantically related to it with some probability. More qualitative analysis on topics are provided in the Appendix.

heading: 6 CONCLUSION
text: We introduce WeTe, a new topic modeling framework where each document is viewed as a bag of word embedding vectors and each topic is modeled as an embedding vector in the shared word embedding space. WeTe views the learning of a topic model as the process of minimizing the expected difference between those two sets over all documents. To this end, we develop a bidirectional transport based method to learn the topic embeddings as well as topic proportions for documents efficiently, which avoids several challenges of existing TMs. Extensive experiments show that the proposed model outperforms competitive methods for both mining high quality topics and deriving better document representation tasks. Thanks to the introduction of the pretrained word embeddings, WeTe achieves superior performance on short and regular texts. Moreover, the proposed model reduces the need to pre-define the size of the vocabulary, which makes WeTe more flexible in practical tasks.

heading: 7 ACKNOWLEDGEMENTS
text: Bo Chen acknowledges the support of NSFC (U21B2006 and 61771361), Shaanxi Youth Innovation Team Project, the 111 Project (No. B18039) and the Program for Oversea Talent by Chinese Central Government.

heading: A DERIVATION OF EQUATION 12
text: The total CT loss can be written as:
L = 1
J J∑ j=1 [LQj→Pj + LPj→Qj ]
= 1
J J∑ j=1 [ K∑ k=1 θ̃jk Nj∑ i=1 c(wji,αk)π(wji |αk) + Nj∑ i=1 1 Nj K∑ k=1 c(wji,αk)π(αk |wji)]
Where
πNj (wji |αk) = e−d(wji,αk)∑Nj i′=1 e −d(wji′ ,αk)
and
π(αk |wji) = e−d(wji,αk)θ̃jk∑K
k′=1 e −d(wji,αk′ )θ̃jk′
Recall the definition of c(wji,αk) in Equation 10 of the main paper, and d(wji,αk) = −wTjiαk. With the fact that ∑K k=1 θ̃jk = 1, then we can rewrite the total CT loss L as:
L = 1
J J∑ j=1 [ K∑ k=1 θ̃jk Nj∑ i=1 e−w T jiαk ew T jiαk∑Nj i′=1 e wTjiαk + Nj∑ i=1 1 Nj K∑ k=1 e−w T jiαk ew T jiαk θ̃jk∑K k′=1 e wTjiαk)θ̃jk′ ]
= 1
J J∑ j=1  K∑ k=1 θ̃jk∑Nj i′=1 e wT ji′αk 1
Nj
+  Nj∑ i=1 1 Nj∑K k′=1 e wTjiαk′ θ̃jk′  which, to the best of our knowledge, does not resemble any existing topic modeling loss functions. The two terms in the bracket have a very intriguing relationship, where in the fraction formula θ̃jk
and 1Nj swap their locations and ∑ k and ∑ i also swap their locations. To minimize the first term,
we will need to ensure the denominator ∑Nj i′=1 e wT ji′αk 1 Nj to be sufficiently large whenever θ̃jk is non-negligible, which can be achieved only if the inner products of the words in document j and topic k aggregate to a sufficiently large value whenever θ̃jk > 0 (i.e., each inferred topic embedding vector needs to be close to at least one word embedding vector of a given document when that topic has a non-negligible proportion in that document). To minimize the second term, we will need to ensure the denominator ∑K k′=1 e
wTjiαk′ θ̃jk′ to be large for every single word, which for word i can be achieved only if there exists at least one topic that has a large inner product with word i (i.e., each word can find at least a single non-negligibly-weighted topic that is sufficiently close to it, in other words, the inferred topics need to have a good coverage of the word embedding space occupied by the corpus).

heading: B TRAINING ALGORITHM
text: The training algorithm of our WeTe is shown in Algorithm 1.
Algorithm 1 Training algorithm for our proposed model. Input: training documents, pretrained word embeddings E, topic number K, hyperparameter . Initialize: topic embeddings α, encoder parameters W. for iter = 1,2,3,... do
Sample a batch of J input documents and represent them as the empirical distributions {Pj}Jj=1 and form the document-specific empirical topic distribution {Qj}Jj=1; With the cost function in Equation 10 and transport probabilities in Equation 9 and Equation 6, compute the CT loss with Equation 12 as the first term of Equation 13; Compute the topic M with Equation 11 and the topic proportions {θj} with input xj , denoted as q(θj |xj) = Weibull(fW (xj), gW (xj)); compute the second term of Equation 13; Update α and W according to Equation 13;
end for
C DATASETS
Table C. 1: Statistics of the datasets
Number of docs Vocabulary size(V) average length Number of labels
20NG 18,864 22,636 108 6 DP 449,665 9,835 22 14 WS 12,337 10,052 15 8
TMN 32,597 13,368 18 7 Reuters 11,367 8,817 74 N/A RCV2 804,414 7,282 75 52
Our experiments are conducted on six widely-used benchmark text datasets, varying in scales and document lengths, including 20 News Group (20NG), DBpedia (DP) (Lehmann et al., 2015), Web Snippets (WS) (Xuan et al., 2008), Tag My News (TMN) (Vitale et al., 2012), Reuters extracted from the Reuters-21578 dataset, and Reuters Corpus Volume 2 (RCV2) (Lewis et al., 2004), where WS, DP and TMN are short documents. To demonstrate the scalability of the proposed model for document clustering task, we pre-processed multi-label RCV2 dataset following previous works (Nguyen et al., 2015), in where documents in test dataset with single label at second level topics are left. We load the pretrained word embedding from GloVe1 (Pennington et al., 2014).
• 20NG2: 20 Newsgroups consists of newsgroups post including 18,846 articles. We remove stop words and words with document frequency less than 100 times. We also ig-
1https://nlp.stanford.edu/projects/glove/ 2http://qwone.com/ jason/20Newsgroups
nore documents that contain only one word from the corpus. We only use the 6 supercategories as 20NG’s ground truth and denote it as 20NG(6) in the clustering task, as there are confusing overlaps in its official 20 categories, e.g., comp.sys.ibm.pc.hardware and comp.sys.mac.hardware.
• DP3: DBpedia is a crowd-sourced dataset extracted from Wikipedia pages. We follow the pre-processing process in Zhang et al. (2015), where the fields we used for this dataset contain title and abstract of each Wikipedia article.
• WS: Web Snippets, used in Li et al. (2016) and Zhao et al. (2020), contains 12,237 web search snippets with 8 categories. There are 10,052 tokens in the vocabulary and the average length of a snippet is 15.
• TMN4: Tag My News, consists of 32,597 RSS news snippets from Tag My News with 7 categories. Each snippet contains a title and a short description, and the average length of a snippet is 18.
• Reuters5 is widely used corpus extracted from the Reuters-21578 dataset. We only use it on topic quality task, and there are 11,367 documents with 8,817 tokens in vocabulary.
• RCV26: Reuters Corpus Volume 2, used in Zhao et al. (2020), consists of 804,414 documents, whose vocabulary size is 7282 and average length is 75.
A summary of dataset statistics is shown in Table C. 1.

heading: D ADDITIONAL TOPIC QUALITY RESULT
text: In Fig. D. 1, we report topic coherence (TC) and topic diversity (TD) for varied methods on TMN and Reuters dataset, which confirms that our proposed model outperforms the others in high quality topic discovering.
When the topic number becomes insufficient, the topic distribution p(mk|k) often resembles the corpus distribution p(w), where high frequency words become the top terms related to most topics. We want topics learned from WeTe to be specific (e.g., not overly general). Topic Specificity (TS) is defined by the average KL divergence from each topic’s distribution to the corpus distribution:
TS = 1
K K∑ k=1 KL(p(mk|k)||p(w))
Jointly with topic diversity and topic coherence, we report topic specificity (TS) of various methods on six datasets at Table. D. 1. it can be found that the proposed model is superior to its competitors on all datasets, which indicates that WeTe produces more useful and specific topics than other NTMs.
In Table D. 2, D. 3, and D. 4, we show the top-10 words of the selected topics learned from WeTe and its two variants on 20NG, TMN, and RCV2, respectively. We note that the proposed model can not only learn meaningful topics from the pretrained word embeddings, but also learn word and topic embeddings jointly from scratch, discovering equally meaningful topics.

heading: E THE LEARNED WORD EMBEDDINGS
text: WeTe(N) provides a new method to learn word embeddings from scratch. Recall the topic-to-doc CT cost for a special document j in WeTe:
Cj = c(wji,φk) e−d(wji,φk)∑Nj i′=1 e −d(wji′ ,φk) , wji ∈ {wj1, . . . ,wjNj}
This transport cost mirrors the likelihood in skip-gram model. Such skip-gram models use the central word to predict the surrounding words. In contrast, our WeTe uses the topic embedding vectors φ as
3https://en.wikipedia.org/wiki/Main_Page 4http://acube.di.unipi.it/tmn-dataset/ 5https://kdd.ics.uci.edu/databases/reuters21578/reuters21578.html 6https://trec.nist.gov/data/reuters/reuters.html
Figure D. 1: The first row and second row show topic coherence (TC) and topic diversity (TD) for varied methods on TMN and Reuters dataset. In each subfigure, the horizontal axis indicates the proportion of selected topics according to their NPMIs. For both TC and TD, higher is better. Where, WeTe(F) and WeTe(N) denote that we finetune the word embeddings or learn it from scratch, respectively.
Table D. 1: Topic specificity (TS) of various methods on web(WS), 20NG, DP, RCV2, TMN and Twitter datasets, higher is better.
Method WS 20NG DP RCV2 TMN Twitter LDA 3.84 4.67 5.42 7.08 3.89 3.95
DVAE 2.50 3.12 4.04 5.45 2.86 1.73 NSTM 1.49 1.97 4.47 6.24 1.07 2.27
WeTe 4.51 5.71 5.58 7.94 4.16 4.43 WeTe(F) 4.48 5.74 5.74 7.42 4.07 4.36 WeTe(N) 4.01 5.42 5.38 6.98 3.89 4.13
Table D. 2: Topics learned from WeTe, WeTe(F) and WeTe(N) on 20NG dataset, where top-10 words for each topic are visualized.
Method Top words
WeTe space nasa orbit spacecraft mars shuttle launch flight rocket solar
window image display color screen graphics output motif mode format game team hockey nhl play teams players win player league season
WeTe(F) space satellite launch nasa shuttle mission research lunar earth technology
window problem card monitor mouse video windows driver memory screen team hockey game players season league play goal year teams
WeTe(N) space launch satellite nasa shuttle earth lunar first mission system
window display application server mit screen problem use get program year game team players baseball runs games last season league
the central words, and generates the document words, rather than a window of surrounding words. In other words, skip-gram models can be viewed as a special variant of WeTe with the window size c = Nj . To evaluate the word embeddings learned from WeTe(N), given a query word, we visualize top-8 words that are most closest to it. We compare WeTe(N) with GloVe at Table. E. 1.
Table D. 3: Topics learned from WeTe, WeTe(F) and WeTe(N) on TMN dataset, where top-10 words for each topic are visualized.
Method Top words
WeTe million billion company buy group share amp firm bid sell
wedding idol royal william prince singer star kate rock taylor team season sports league teams soccer field manchester briefing club
WeTe(F) million billion deal group company firm offer buy shares sell
star stars movie fans idol hollywood box fan film super players nfl coach draft teams football basketball player nba lockout
WeTe(N) million company video deal online internet apple google mobile media
show star theater book idol royal dies space wedding music coach nfl players team state season sports national tournament basketball
Table D. 4: Topics learned from WeTe, WeTe(F) and WeTe(N) on RCV2 dataset, where top-10 words for each topic are visualized.
Method Top words
WeTe million total billion asset worth sale cash debt cost payout
oil gas fuel barrel palm petroleum gulf shell bpd cubic olein network internet custom access microsoft web design tv broadcast media
WeTe(F) sale sold bought sell retail buy chain auction supermarket shop discount
oil barrel nymex brent gas petroleum fuel gallon wti gulf system network personnel microsoft inform chief internet web unit custom
WeTe(N) percent billion year million market rate month economic growth dollar oil gas barrel brent fuel output sulphur petroleum nymex diesel gallon
network channel radio tv media station broadcast film video disney
Compared to glove, the word embedding we learned tends to be more semantically diverse. For example, "download", "modem" for "pc", and "goal", "win" for "game". We attribute this to the document level context.
Table E. 1: Comparison of the most relevant words for the query words on RCV2 dataset.
Query word Method Top words
pc GloVe desktop computer software macintosh computers pentium pcs microsoft xpWeTe(N) pc desktop macintosh pcs microsoft internet os download mac modem
game GloVe game games season play match player league team scored playoffsWeTe(N) game season play match team playoff bowl goal win coach world GloVe world cup international olympic european championships event europeWeTe(N) world cup international european event asian asia women nation team school GloVe school college university schools students education elementary graduateWeTe(N) school high student campus district church program degree taught harvard

heading: F COMPLEXITY ANALYSIS
text: As a neural topic model, WeTe has a comparable complexity to other neural topic models. In detail, for a mini-batch of documents with batch-size B, NB denotes the total words in the mini-batch. We summary the time and space complexity at Table. F. 1. where, CT denotes conditional transport and TM means the topic model, we here ignore the 3-layer neural encoder, due to it is shared with other neural topic models. V is the vocabulary size, K is the number of topics and d is the embedding size. We can see that CT obtains linear complexity in both time and space with respect to the vocabulary and the total number of words in the mini-batch.
We also compare WeTe with other three NTMs on large RCV2 (V=13735,N=804,414) with a large topic setting (K=500). All the methods run on an Nvidia RTX 2080-Ti GPU with batch size of 500. The normalized training loss is shown in Fig. F. 1, where the direct comparability between losses is not available due to the different designs. It demonstrates that the proposed model has
Table F. 1: Time and space complexity analysis of WeTe. CT denotes the conditional transport part, and TM denotes the topic model part, respectively. We ignore the 3-layer encoder because it is shared with all neural topic models.
CT TM
Time complexity (2V+4NB)K 2VKd2 + BVKK2 + 4VB Space complexity (V+K)d + (2V+4NB)K (V+B)K+VB
Figure F. 1: Training loss on RCV2 over batches (a) and seconds (b).
acceptable learning speed compared with other NTMs. Fig. F. 1(a) shows that WeTe requires fewer iterations compared to DVAE and ETM. And Fig. F. 1(b) demonstrates that our WeTe has similar time consumption to DVAE. Although ETM and NSTM have faster training speed, their performance on both topic quality and clustering task is incomparable to ours. In other words, WeTe balances the performance and speed well.

Review summaries:
The manuscript proposes a combined topic modeling and word embedding method.  The idea is to define word and topic embeddings in the same space, represent documents based on both of them, and use a transport-based method to train these two representations to be similar to each other.  The final objective function combines a "conditional transport" objective with a log-likelihood term, and is optimized via Adam.  Experiments find that the method achieves good topic coherence scores and document clustering metrics compared to baselines. 

Jointly learning latent topic representations with word embeddings has caught some attention in the past. The reason why researchers choose this route is to exploit the complementary advantage of both these models, for instance, to improve the performance of the model on short texts. In this paper, the authors study a new topic model where topics and words are encoded in the same embedding space. Usually, in the traditional topic modelling literature, the posterior inference is realised using algorithms such as Gibbs sampling or collapsed Gibbs sampling; however, in this paper, since the authors exploit complementary knowledge between the two models, the end-to-end model is jointly trained the document-specific discrete latent topic distribution is made to appear as close as possible to the discrete distribution over words in the document. A probabilistic bidirectional transport model is developed that measures the difference between the discrete distributions in the embedding space. The authors also discuss several key advantages of the model.  Experiments are then followed with topic diversity and coherence measures as evaluation metrics which are widely used in the literature. Qualitatively, the authors show some example results from their dataset in the form of a t-SNE plot.

This paper presents a new topic modeling framework called WeTe: each document in text corpus is represented as a bag of word embeddings vectors, and each topic is modeled as an embedding vector in the shared word embedding space. WeTe minimizes the expected difference between those two sets over all documents. A bidirectional transport-based method is proposed to learn the topic embeddings as well as topic proportions for documents efficiently. Extensive experiments on news and web pages show that the proposed model outperforms competitive methods for both deriving high-quality latent topics and generating better document representations for clustering tasks.