Representing Mixtures of Word Embeddings with Mixtures of Topic Embeddings
Authors: dongsheng wang
Dan dan Guo
He Zhao
Huangjie Zheng
Korawat Tanwisuth
Bo Chen
Mingyuan Zhou
Venue: ICLR 2022 Poster (2022)
Keywords: topic model
text mining
distribution matching

Abstract: A topic model is often formulated as a generative model that explains how each word of a document is generated given a set of topics and document-specific topic proportions.  It is focused on capturing the word co-occurrences in a document and hence often suffers from poor performance in analyzing short documents. In addition, its parameter estimation often relies on approximate posterior inference that is either not scalable or suffering from large approximation error. This paper introduces a new topic-modeling framework where each document is viewed as a set of word embedding vectors and each topic is modeled as an embedding vector in the same embedding space. Embedding the words and topics in the same vector space, we define a method to measure the semantic difference between the embedding vectors of the words of a document and these of the topics, and optimize the topic embeddings to minimize the expected difference over all documents. Experiments on text analysis demonstrate that the proposed method, which is amenable to mini-batch stochastic gradient descent based optimization and hence scalable to big corpora, provides competitive performance in discovering more coherent and diverse topics and extracting better document representations. 

Published as a conference paper at ICLR 2022
REPRESENTING MIXTURES OF WORD EMBEDDINGS WITH MIXTURES OF TOPIC EMBEDDINGS
Dongsheng Wang'; Dandan Guo²; He Zhao', Huangjie Zheng',
Korawat Tanwisuth', Bo Chen', Mingyuan Zhou' 1 "iiiian University "hh Chinese University of Hong Kong, Shenzhen MMonash University "The University of Texas at Austin
curhan ethan. zhao zww.@moassh..ccomoooooooooooo edu, huangjie, zhenguutexas edu korawat tanwi suth@utexas edu, bchen i 1 xi di an. edu. cn mingyuan, zhouêmcccombs utexas.edu
wds@stu. lis@@stuxidian. xidian. edu. en, guodandang cuhk edu. en
ABSTRACT
A topic model is often formulated as generative model that explains how each word of document is generated given a set of topics and document-specine topic proportions. It is focused capturing the word co-occurrences in document and hence often suffers from poor performance in analyzing short documents. addition, its parameter estimation often relies on approximate posterior inference that is either not scalable or suffering from large approximation error. This paper introduces new oopicmodeelin framework where each document viewed a set of word embedding vectors and each topic is modeled as an embedding vector in the same embedding space. Embedding the words and topics in the same vector space, we define method to measure the semantic difference between the embedding vectors of the words of a document and these of the topics, and optimize the topic embeddings minimize the expected difference over all documents. Experiments on text analysis demonstrate that the proposed method, which is amenable to mini-batch stochastic gradient descent based optimization and hence scalable to big corpora, provides competitive performance in discovering more coherent and diverse topics and extracting better document representations.
INTRODUCTION
For text analysis, topic models widely used to extract of latent topics from corpus (a collection of documents). The extracted topics, revealing common word co-occurrence patterns within document, often correspond to semantically meaningful concepts the training corpus. Bayesian probabilistic topic models (BPTMs), such as latent Dirichlet allocation (LDA) (Blei et al. 2003
Griffiths Steyvers 2004) and its nonparametrie Bayesian generalizations (Thh al. 2006 Zhou
et al. 2012), have been the most popular ones. A BPTM is often formulated as generative model that explains how each word of a document is generated given set of topics and document- speciffc topic proportions. Bayesian inference of BPTM is usually based on Gibbs sampling variational inference (VI), which can be less scalable for big corpora and need to be customized accordingly.
With the recent development in auto-encoding VI, originated from variational autoencoders (VAFs) (Kingma & Welling 2014 Rezende et al., 2014), deep neural networks have been successfully used to develop neural topic models (NTMs) (Miao al. 2016, Srivastava & Sutton/ 2017 Burkhardt & Kramer 2019, Zhang et al. 2018 Dieng et al. 2020 Zhao et al. 2021). The key advantage of be carried out easily via forward pass of the encoder network, without the need for expensive iterative inference scheme per test observation as in both Gibbs sampling and conventional VI. Hence, NTMs enjoy better flexibility and scalability than BPTMs. However, the reparameterization trick in VAEs cannot be directly applied to the Dirichlet (Burkhardt Kramer 2019) gamma distributions (Zhang et al. 2018). which are usually used as
NTMs is that approximate posterior inference
"quual contribution.

---

Published as a conference paper at ICLR 2022
the prior and conditional posterior of latent topies and topie proportions, so approximations have to be used, potentially introducing additional complexity or approximation errors.
To address the above shortcomings, we propose a novel topic modeling framework in intuitive and effective manner of enjoying several appealing properties over previously developed BPTMs and NTMs. Like other TMs, we also focus on learning the global topics shared across the corpus and the document-specinne topic proportions, which are the two key outputs of a topic model. Without building an explicit generative process, we formulate the learning of topic model (e.g.. optimizing the likelihood) as the process of minimizing the distance between each observed document, j and its corresponding trainable distribution. More specifically, the former (document j) can be regarded as as an empirical discrete distribution P, which has an uniform measure over all the words within this document. To construct the latter (trainable distribution), we can represent Pj with K shared topics and its K --ddimnnionnal document-specifie topic proportion, defined as Q where we view shared topics as K elements and topie proportion as the probability measure in 2j- It is very reasonable since the k-th element in topic proportion measures the weight of topic k for document, and the document can be represented perfectly using the learned topic proportion and topics from a desired TM. Recalling that each topic and word usually reside in the V V-diimensioal (vocabulary size) space in TMs, it might be difficult to directly optimize the distance between P3 and Q5 over V-dimensional space. Motivated by Dieng et al. (2020), we further assume that both topics and words live in H-dimensional embedding space, much smaller than the vocabulary space. With a slight abuse of notation, we still use Pj over the word embeddings and Q, over the topic embeddings as two representations for document j. Below, we turn towards pushing the spcciiet-speccffic to-be-learned distribution 2, to be as close as possible to the empirical distribution P'j.
To this end, we develop a probabilistic bidirectional rransprt-basee method to measure the semantic difference between the two discrete distributions in an embedding space. By minimizing the expected difference between P3 and 22 over all documents, we can learn the topic and word embeddings directly. Importantly, we naturally leverage semantic distances between topics and words in an embedding space to construct the point-to-point cost of moving between them, where the cost becomes a function of topic embeddings. Notably, we consider linking the word embeddings in P; and topic embeddings in Q2 in a bidirectional view. That is, given word embedding drawn from Pj. it is more likely to be linked to a topic embedding that both is closer to it in the embedding space and exhibits larger proportion in Qj: vice versa. Our proposed framework has several key properties: 1) By bypassing the generative process, our proposed framework avoids the burden of developing complex sampling schemes or approximations for the posterior of BPTMs NTMs. 2) The design of our proposed model complies with the principles of TMs, whose each learned topic describes an interpretable semantic concept. More interestingly, our model is flexible to learn word embeddings from scratch or use/finetunc pretrained word embeddings. When pretrained word embeddings are used, our model naturally alleviates the issue of insufficient word co-occurrence information in short texts as discussed by prior work (Dieng et al.) 2020; (Zhao et al.) 2017a] 2021), which is one of the key drawbacks of many BPTMs and NTMs. 3) Conventional TMs usually enforce a simplex constraint on the topics over a fixed vocabulary, which hinders their applications in the case where the vocabulary varies. In our method, we view document as a mixture of word embedding vectors, which facilitates the deployment of the model when the size of the vocabulary varies. Finally, we have conducted comprehensive experiments on a wide variety of datasets in the comparison with advanced BPTMs and NTMs, which show that model can achieve the ofate-of-the-ar performance as well as applealing interpretability. The code is available at
2 BACKGROUND
Topic Models: TMs usually represent each document in a corpus as bag-of-words (BoW) count vector æ E RVY where x, represents the occurrences of word in the vocabulary of size V. A TM aims to discover K topics in the corpus, each of which describes a specific semantic concept. A topic is or can be normalized into distribution over the words in the vocabulary, named word distribution, Φe E Σv, where Σy is a V - 1 dimensional simplex and Oek indicates the weight or relevance of word e under topic k. Each document comes from a mixture of topics, associated with specific mixture proportion, which can be captured by a distribution over K topics, named topic proportion, 0 E Ex. where 0g indicates the weight of topic for document.
2

---

Published as a conference paper at ICLR 2022
As the most fundamental and popular series of TMs, BPTMs (Blei et al. 2003 Zhou al. 2012 Hoffman et al. 2010 generate the document a with latent variables (i.c., topics i O 14 k= and topic
proportion 0) sampled from pre-specified prior distributions (e.g., gamma and Dirichlet). Like other Bayesian models, the learning process of a BPTM relies on Bayesian inference, such as variational inference and Gibbs sampling. Recently, NTMS (Zhang al.; 2018 Burkhardt & Kramer 2019
Dieng et al. 2020; Wang et al. [2020; Duan et al.] [021b) have attracted significant research interests in topic modeling. Most existing NTMs can be regarded as extensions of BPTM like LDA within
the VAEs framework (Zhao et al., (021). In general, NTMs consist of an encoder network that maps the (normalized) BoW imput x its topic proportion 0, and a decoder network that generates conditioned on the topics {Φx j k=1 and proportion 0. Despite their appealing flexibility and scalability, due to the unusable reparameterization trick in original VAEs for the Dirichlet or gamma distributions, NTMs have to develop complex sampling schemes or approximations, leading to potentially large approximation errors or learning complexity.
Compare Two Discrete Distributions:
This paper aims to quantify the difference between two discrete distributions (word embeddings and topic embeddings), whose supports are points in the same embedding space. Specifically, let and q be two discrete probability measures the arbitrary space X C " formulated as = Σ;= uzdz, and q = Σj= dy0y,, where u = [u,] E En and
= [uy] E Em denote two distributions of the discrete states. To measure the distance between and g, a non-trivial way is to use optimal transport (OT) (Peyré & Cuturi 2019) defined as
(T q) == minren(w...) Tr (TC)),
(1)
where C' Bax" is the transport cost matrix with Cip c (3 U5), T E TR""" doubly stochastic transport matrix such that II(u, v) = {T Tlp, = u, T "p. = =). . Tı the transport probability between xz and yj. and Tr(-) the matrix trace. Since the transport plan imposed on the constraint of E [u,, v). it has to be computed via constrained optimizations, such as the iterative Sinkhorn algorithm when an additional entropy regularization term is added (Cuturi 2013).
The recently introduced conditional transport (CT) framework (Zheng & Zhou 2020 Tanwisuth et al. 202T) can be used to quantify the difference between two discrete distributions, which, like OT, does not require the distributions to share the same support. CT considers the transport plan in bidirectional view, which consists of a forward transport plan as TP and backward transport plan TP+-q Therefore, the transport cost between two empirical distributions in CT can be expressed
min CT(p,9) T--9,
Tr [(T ")T
T C (TT-P) C].
(2)
T =1; Eye (xglign Σm ujen dgte and TW -リo u.c (vgrs)
Ei= uue (F and hence Te-$11,, = u and (T 1 = v, but in general T 1p. * u and (TP In, # v. Here dg. (x, dee (v.x) x)) parameterized by to measures the difference between two vectors. This flexibility of CT potentially facilitates an easier integration with deep neural networks with a lower complexity and better scalability. These properties can be helpful to us in the development of new Complexitymmman
CT specifies
1j
topic modeling framework based on transportation between distributions, especially for NTMs.
LEARNING MIXTURE OF TOPIC EMBEDDINGS
Now we will describe the details of the proposed model. Since it represents mixture of Word Embeddings as a mixture of Topic Embeddings, we refer to it as We'Te. Specifically, consider a corpus of J documents, where the vocabulary contains V distinct terms. Unlike in other TMs, where a document is represented as a BoW count vector æ E RV as shown in Section 2
we denote each document as a set of its words, defined as Dj- [W]e]. where Wys (l...., V} means the i-th word in the j-th document with i E [1, N2), and N, is the length of the j-th document. Denote E TR HVV as the word embedding matrix whose columns contain the embedding representations of the terms in the vocabulary. By projecting each word into the corresponding word-embedding space, we thus represent each document as an empirical distribution P, on the word embedding space as follows
Pj
= 1= N. Swyy W. E R.
ΣE
Similar to other TMs, we aim to learn K topics from the corpus. However, instead of representing a topic as a distribution over the terms in the vocabulary, we use embedding vector for each topic,
3

---

Published as a conference paper at ICLR 2022
Oy IR" Here topic embedding ag is distributed representation of the k-tth topic in the same semantic space of the word embeddings. Collectively, we form a document-speciffe empirical topic distribution 2j (on the embedding space), defined as
Q2 - -x= Byxdas, OG E R...
(4)
Here f5.I.K denotes the normalized topic proportions of document i.c., : 0/ / 8jk- We focus on learning the topic distribution Q2 that is close to distribution Py. Exploiting the CT loss defined in Eq. (2) we introduce We'Te as novel topic model for text analysis. For document j. we propose to minimize the expected difference between the word embeddings from Pj and topic embeddings from Q3 in terms of its topic proportion and topie embeddings. For all the documents in the corpus, we can minimize the average CT loss
min J E/=,(T((P,, Q,)] a.
As a bidirectional transport, CT(-) consists of a doc-to-topic CT that transports the word embeddings to topic embeddings, and topie-to-doe CT that reverses the transport direction. We define conditional distribution specifying how likely given topic embedding Φx will be transported to word embedding Orji in document j as
( NN, (wy5 (xx) ΣxL, P, (w 4(m E de go Wye E (urf1..... WyN,), (6)
where d(w,, 04)) d(ax, w;) indicates the semantic distance between the two vectors. Intuitively. would have a high probability. This
if ag and W have small semantic distance, the π r(w; (x)) construction makes it easier to transport ag to word that is closer it the embedding space. For document with N, words (w,l...., WyN, J. the topic-to-doc CT cost can be expressed as
LQ,-P; = lax Qu, (1 (.1a))) [c| [c(,,, an)] = >
k=l 0,R Ei c(Wji, (U)TT(W oa)), (7)
where dwji, .4)) c(crk, wyj) 2 denotes the point-to-point cost of transporting between word embedding w and topic embedding Ors, and 0R., can be considered as the weight of transport cost between all words in document and topic embedding k. Similar to but different from Eq. ( introduce the doc-to-topic CT, whose transport cost is defined as
Lp,-Q; = En Ear (-1 wyy) (o( ax)) Σ N; E c(wje, on) wyys)
(8)
where N,
1 denotes the weight of transport cost between word wys in document j and topic embeddings.
In contrast to Eq. (6). we define the conditional transport probability from word embedding wy: in document j to a topie embedding Φb with
(a(a d(w)..a))
π (ax w,))
(9)
D (a(a)) A6 (w)
where 83ز- (a(c can be interpreted as the prior weight of topic embedding ag in document j. We have not specified the form of c(w,,, ac) and d(wst, a4). A naive definition of the transport cost or semantic distance between two points is some distance between their raw feature vectors. In our framework, we specify the following construction of cost function:
c(w ax)) = = w..aa
(10)
Here the cost function defined for two reasons: the inner product the commonly-use way to measure the difference between two embedding vectors, and the cost needs to be positive. For semantic distance, we directly take the inner product of the word embedding wye and the topic embedding ag, i.e., d(ur,j, ona) af although other choices are possible.
3.1 REVISITING OUR PROPOSED MODEL FROM TOPIC MODELS
Generally speaking, conditioned observed document, traditional TMs (Blei et al. 2003 Zhou ctt al.) 2012 Srivastava & Sutton 2017) often decompose the distribution of the document's 's words into learnable factors: the distribution of words conditioned certain topic, and the distribution
4

---

Published as a conference paper at ICLR 2022
of topics conditioned on the document. Here, establish the connection between model and traditional TMs. Recall that d(w,; (x)) indicates the semantic distance between topic k and word i in the embedding space. For arbitrary a and w;, the more similar they are, the smaller underlying distance they have. Following this viewpoint, we assume Φ E TRy as the distributiionoove--wrrks representation of topic It and treat its elements
Duk
(11)
where vy IR# denotes the embedding of the e-th term in the vocabulary. Therefore, the column vector Φh weights the word importance in the corresponding topic k. With this form, our proposed model assigns a probability to a word in topic & by measuring the agreement between the word's 's and topic's embeddings. Conditioned on o = [ the flexibility of CT enables multiple ways to learn or define the topic proportions of documents, i.e., O detailed in Section 3.2 With CT's ability for modeling geometric structures, our model avoids developing the prior/posterior distributions and the associated sampling schemes, which are usually nontrivial in traditional TMs (Zhou et al. 2016).
3.2 LEARNING TOPIC EMBEDDINGS AND TOPIC PROPORTIONS
Given the corpus of J documents, we wish to learn the topic embedding matrix and topic propor- tions of documents Q Based on the doc-to-topic and topic-to-doe CT losses and the definitions of e and d in Eq. 66 10 and Ek= 8,t
1, we can rewrite the CT loss in Eq. as
J C 1=1
K
CT(P,
(12)
OO Ny
=== av 0,
2=1
whose detailed derivation is shown in Appendix The two terms in the bracket exhibit appealing symmetry properties between the normalized topic proportion 0; and word prior N To minimize the first term, for a given document whose topic proportion has a non- non-negligible activation at the k-th topic, the inferred k-th topic needs to be close to at least one word (in the embedding space) of that document. Similarly each word in document j needs to find at least a single noo-negiigibly-weighee topic that is sufficiently close to it. In other words, the learned topics are expected to have good coverage of the word embedding space occupied by the corpus by optimizing those two terms.
Like other TMs, the latent representation of the document is a distribution over K topics: 0, E Σ each element of which denotes the proportion of one topic in this document. Previous work shows that the data likelihood can be helpful to regularize the optimization of the a transport based loss (Frogner et al., [0015] Zhao al. 2021). ). To amortize the computation of 0, and provide additional regularization, we introduce a regularized CT loss as
mina, w 1 Σ Ee (((((()))))))))) [CT(P,, Q,) clogp(æ;;;, 0,)]1
(13)
where qw (0, x3)) is deterministic or stochastic encoder, parameterized by W. p(z,; 2 0,)
Poisson( v; Σx ()(0))) (#e is defined as in Eq. (1b) is the likelihood used in Poisson factor anal- ysis (Zhou et al. 2012). and E is a trade-off hyperparameter between the CT loss and likeliodl Here, we encode with the Weibull distribution: qw (0, x,) = Weibull (zz), gw (xz)). where and g are two related neural networks parameterized by W. Similar to previous work ZZhng jtt al. 2018 Guo et al., 2020 Duan et al. 2021a), we choose Weibull mainly because it resembles the gamma distribution and is reparameterizable,
drawing m Weibull A) is equivalent to mapping m f(e) == X X(-log(1 c)) e Uniform(0. 1). Different from previous work, here a (0;
does not play the role of variational inference network that aims to approximate the posterior distribution given the likelihood and a prior. Instead, it is encouraged to strike a balance between minimizing the CT cost, between the document representation in the word embedding space and that in the topic embedding space, and minimizing the negative log- likeliodd of Poisson factor analysis, with the document representation shared between both components of the loss.
The loss of Eq. (13 is differentiable in terms of a and W. which can be optimized jointly in one training iteration. The training algorithm is outlined in Appendix B Benefiting from the encoding network, after training the model, we can obtain 0, by mapping the new input x, with the learned encoder W. avoiding the hundreds iterations in MCMC or VI collect posterior samples for
5

---

Published as a conference paper at ICLR 2022
local variables. The algorithm for We're can either use pretrained word embeddings, e.g., Ve (Pennington et al.] (014), or learn them from scratch. Practically speaking, using pretrained word embeddings enables more efficient learning for reducing the parameter space, and has been proved beneficial for short documents for leveraging the rich semantic information in pretrained word embeddings. In experiments, We'Te by default uses the GloVe word embeddings.
4 RELATED WORK
Models with Word Embeddings: Word embeddings have been widely used as complementary information to improve topic models. Skipgram-based models (Shi et al. 2017] Moody 2016 Paar Lee, 2020) jointly model the skip-gram word embeddings and latent topic distributions under the Skipgram Regative-Sampling objective. Those models incorporate the topical context into the central words to generate its surrounding words, which share similar idea with the topic-to-doc transport in We'Te that views the topic vectors as the central words, and words within document as the surrounding words. Besides, We'ee forces the inferred topics to be close to at least one word embedding vector of a given document by the doc-to-topic transport, which is not considered in those models. For BPTMs, word embeddings are usually incorporated into the generative process of word counts (Petterson et al. 2010; Nguyen et al., 2015 Li et al. 2016 Zhho al. 2017a b Keya et al., 2019). Benefiting from the flexibility of NTMs, word embeddings can be either incorporated as part of the encoder input, such as Card et al. (2018), or used in the generative process of words, such as in Dieng et al. (2020). Because these models construct the explicit generative processes from the latent topics to documents and belong to the extensions of BPTMs or NTMs, they may still face these previously mentioned difficulties in TMs. Our method naturally incorporates word embeddings into the distances between topics and words under the bidirectional transport framework, which is different from previous ones.
Models by Minimizing Distances of Distributions: Yurochkin et al. (2019) adopt OT to compare two documents' similarity between their topic distributions extracted from pretrained LDA, but their focus is not to learn a topic model. [Nan et al. (201)) extend the framework of Wasserstein autoEncoders (Tolstikhin et al., 2018) to minimize the Wasserstein distance between the fake data generated with topics and real data, which can be interpreted as an OT variant to NTMs based on VAE. In addition, Xu et al. (2018) introduce distilled Wasserstein learning, where an observed document is approximated with the weighted Wasserstein barycentres of all the topic-word distributions and the weights are viewed as the topic proportion of that document. The Optimal Transport based LDA (OTLDA) of Huynh et al. (2020) is proposed to minimize the regularized optimal transport distance between document distribution æ, and topic distribution about M in the vocabulary space. Also, Neural Sinkhorn Topic Model (NSTM) of Zhao et al. (2020) is proposed to directly minimize the OT distance between the topic proportion @ output from the encoder and the normalized BoW vector ø. Compared with NSTM, by representing document as mixture of word embeddings and a mixture of topic embeddings, our model directly minimizes the CT cost between them the same embedding space. Moreover, NSTM needs to feed the pretrained word embeddings to construct the cost matrix in Sinkhorn algorithm, while our WeTe can learn word and topic embeddings jointly from scratch. Finally, our model avoids the use of Sinkhorn iterations within each training iteration.
5 EXPERIMENTS
5.1 EXPERIMENTAL SETTINGS
Datasets: To demonstrate the robustness of our WeTe in terms of learning topics and document
representation, we conduct the experiments on six widely-used textual data, including regular and short documents, varying in scales. The datasets include 20 News Group (20NG), DBpedia (DP) (Lehmann et al. 2015). Web Snippets (WS) (Phan et al., 2008), Tag My News (TMN) (Vitale et al. 2012), Reuters extracted from the 25uters-21578 dataset. and Reuters Corpus Volume 2 (RCV2)
(Lewis et al., (004)), where WS, DP, and TMN consist of short documents. The statistics and detailed descriptions of the datasets are provided in Appendix C
Evaluation metrics: Following Dieng et al.) (2020) and Zhao et al. 2020). we use Topic Coherence (TC) and Topic Diversity (TD) evaluate the quality of the learned topics. TC measures the average Normalized Pointwise Mutual Information (NPMI) over the top 10 words of each topic, and a higher score indicates more interpretable topics. TD denotes the percentage of unique words in the top 25

---

Published as a conference paper at ICLR 2022
WS(TC)
20NG(TC)
RCV2(TC)
0.2
0.2
0.2
WeTe
0.15
0.15
WeTe(N)
0.1
0.1
NETM
0.05
PFA
WCLFTM
0.06
00.05
0.1
WS(TD)
20NO(TD)
RCV2ITD)
TO (x.1x000)
0.2
0.15
0.6
0.7
0.5
0.5
Wate Techi
10% 50%
100% 10%
50%
100%
60% 100%
10
11
(og(N)
12
13
(a)
(b)
Figure 1: (a) The first and second rows show topic coherence (TC) and topic diversity (TD), respectively, for different methods five datasets. In each subfigure. the horizontal axis indicates the proportion of selected topies according to their NPMIs. For hoth TC and TD, higher is better. (b) topic quality (TQ TC TD) tendency of WeTe and its variants as the corpus size grows. Where, WeTe(FF and WeTe(N) denote that we finetune the word embeddings and learn it from scratch, respectively.
words of the selected topics. To comprehensively evaluate topic quality, we choose the topics with the highest NPMI and report the average score over those selected topics, where we vary the proportion of the selected topics from 10% to 100%. Besides topic quality, we calculate Normalized Mutual Information (NMI) (Schütze et al. 2008) and purity on WS, RCV2, DP, and 20NG on clustering as 20NG's ground truth and denote it as 20NG(6). We
tasks, where we use the 6 super-categorie split all datasets according to their default training/testing division, and train model on the training documents. Given the trained model, we collect the topic proportion @ on the testing documents and apply the K-Means algorithm on it, where the purity and NMI of the K-Means clusters are measured. Similar to Zhao et al. (2020), we set the number of clusters of K-Means as N = 52 for RCV2 and N 20 for the other datasets. For all the metries, higher values mean better performance.
Baseline methods and their settings: We compare the performance of our proposed model with the following baselines: 1) traditional BPTMs, including LDA (Blei et al. 2003), a well-known topic model (here we use its collapsed Gibbs sampling extension (Griffiths & Steyvers 2004)) and Poisson Factor Analysis (PEA) (Zhou et al., 2012). a hierarchical Bayesian topic model under the Poisson likelihood: 2) VAEs based NTMs, such as Dirichlet VAE (DVAE) (Burkhardt & Kramer 2019) and Embedded Topic Model (ETM) (Dieng et al.) 2020), a generative model that marries traditional topic models with word embeddings; 3) OT based NTM, Neural Sinkhorn Topic model (NSTM) (Zhao et al., 22020). which learns the topic proportions by directly minimizing the OT distance to a document's s word distribution; 4) TMs designed for short texts, including becudo-document-baee Topic Model (PTM) (Zuo et al. (016) and Word Embedding Informed Focused Topic Model (WEI- FTM) (Zhao et al. (017a). In summary, ETM, NSTM, WEI-FTM, and our WeTe are the ones with pretrained word embeddings. For all baselines, we use their official default parameters with best reported settings.
Settings for our proposed model: Besides the default We're which loads the pretrained word embeddings from GloVe, we propose two variants of WeTe. The first variant initializes word embeddings from the Gaussian distribution (000 0.02) and learn word and topic embeddings jointly from the given dataset. The second variant loads the GloVe embeddings and fine-tunes them with other parameters. We denote those two variants WeTeeNN and WeTe((F), respectively. We set the number of topics as K = 100. For our encoder, we employ a neural network stacked with a 3-layer V-256-100 100 fulle-comnected layer (V is the vocabulary size), followed by a softplus layer. We set the trade-off hyperparameter as e = 1.0 and batch size as 200. We use the Adam optimizer (Kingma & Ba 2015) with learning rate 0.001. All experiments are performed on an Nvidia RTX 2080-Ti GPU and implemented with PyTorch
5.2 RESULTS
Quantitative results: For all models, we run the algorithms in comparison five times by modifying only the random seeds and report the mean and standard deviation (as error bars). We first examine
7

---

Published as a conference paper at ICLR 2022
Table 1: Comparison of K-Means clustering purity (km-Purity) and NMI (km- Ain-MM) various methods. We as 2UNG's ground truth and denote it as 20NG(6). The best and second best scores of
use the 6 each dataset are highlighted in boldface and with an underline. respectively.
Method
km-Purity %)
km-NMI(S)
WS
RCV2
DP
20NG(6)
WS
RCV2
DP
2UNG(6)
LDA-Gibbs
46.4.10.6
52.411..4
60.8 10.5
59.2.10.6
25.1
38.2.10.5
54.7 10.3 55.4+00.5
32.4 10,4
PFA
55.7+004
64.6 +0.7
61.2+006
31.1+0.3
32.7 +1.1
PTM
331
56.3 +17
7.9+1.4
45.2 11.5 59.7+11.
WEI-FTM
14.6+1.5
65.3 +2.4
32.4=1.55
DVAE
2..6+1.5
52.621.2
67.2 +11
64.6
+0.8
31.3::099
50.8 #0.6
29.8 =0.6
ETM
32.9+2.3
50.2=0.6
63.1 +1.
62.6
12.3=2.3
30.3+1.0
53.2 +0.7
29.3 11.5
NSTM
42.1100.6
53.811.0
20.2 1007
62.611.2
17.4 10.6
36.810.3
6.6310.11
31.1 112
WeTe
59.010.1
59.2
75.8 1.0.8
67.3 10.6
34.5=10.1
40.3::044
62.5
35.0 0.4
WeTe(N) WeTo(F) 60.8:10.2
59.7.10.1
58.510.3
74.1 13.3
70,2 11.0
34.1
41.2.00.1
60.111.1
34.3 10.8
62.910.5
77.1 110
68.5 10.2
34.9-0.4
42.8.1033
63.7.10.4
36.3 -02
the quality of the topics discovered by WeTe. Fig. aa) shows the results of TC and TD on three corpora (more results can be found in Appendix [D. varying in scales. Due to limited space, we only choose PFA and WEI-FTM as representatives of their respective methods. Since the Gibbs sampling based methods (e.g.. PFA, WEI-FTM) require walking through all documents in each iteration, it is not scalable big data like RCV2. WEI-FTM only works on short texts. There are several observations drawn from different aspects. For the short texts (WS), We'Te has comparable performance with NSTM, and is much better than WEI-FTM, which designed specifically for short texts. This observation confirms that our WeTe is effective and efficient in learning coherent and diverse topics from the short texts with pretrained word embeddings, without designing specialized architectures. In addition, for the regular and large datasets (20NG, RCV2), our proposed WeTe significantly outperforms the others in TC while achieving higher TD. TD. Although some TMs (NSTM, ETM. WEI-FTM) utilize the pretrained word embeddings, it is demonstrated that how to assimilate them into topic model is the key factor. Thus we provide a reference for future studies along the line of combining word embeddings and topic models. Compared with We'T,, We'Te(F) and WeTee(N) need to learn word and topic embeddings from the current corpus, whose size is usually less than IM, resulting sub-optimal topics discovering. From Fig. aa), we can further find that those two variants achieve comparable result with other NTMs for top-20% topics, which means the proposed model has the ability to discover interpretable topics only from the given corpus without loading pretrained word embeddings. Fig. bb) denotes topic quality of our We'Te and its variants with different corpus scalar. shows that WeTe(FF) and WeTe(NN reach a performance close to that of We're as the scalar of the corpus becomes large, suggesting that the proposed model alone has the potential to learn meaningful word embeddings by itself on a large dataset.
The clustering Purity and NMI for various methods are shown in Table T Overall, the proposed model is superior to its competitors all datasets. Compared with NSTM, which learns topic proportions by minimizing the OT distance to document's word distribution, We'Te employs a probabilistic bidirectional transport method to learn & and topic embeddings jointly, resulting in more distinguishable document represen- lations. Besides, with the ability to finetune/learn word em- beddings from the current corpus, WeTe(F) and WeTe(N) can better infer the topic proportion 6, and hence give better per- formance in terms of document clustering. Those encouraging results show that not only the proposed model can discover the topics with high quality, but also learn good document repre- sentations for downstream clustering task on both short and regular documents. It thus indicates the benefit of minimizing the semantic distance between mixture of word embeddings and mixture of topic embeddings.
Km-Purity(%%
68
66
84
WeTe|TMM WoTe(CT)
62
60
0.2
0.4
0.6
0.8
TQ(x100)
6.6
6.2
0.4
Figure 2: Parameter sensitivity of
We're on 20NG dataset, KMeans cluster- ing purity (Km- Km-Purity) and Topic Qual- ity (TQ).
Hyperparameter sensitivity We fix the hyperparameter 1.0, which controls the weight of the Poisson likelihood in Eq.
13 in the previous experiments for fair comparison. Here
8

---

Published as a conference paper at ICLR 2022
summary
pharges
(b)
grimiaal
pase
prticles
Jopic_ 4
3
artice
JEWS
pour.
Jan
jaige
published
book
mars
uitversity pducation
merrico son
200 c_1
solleye pchool
3
Stteer
using used
prb
Schools
zudents
children
Jopic
system
Space
phuttle
peaple
jopk_ 8
fhurch
Hibe
computer
prive
pars
Dike
NSTM oumpate: hanbo sofironc sur,, interfass, uas,, web, compuis, sarver, suftiiuee intenet, flle
dettep
handware
PTpic 0 phristin
windows
Joom
door
13r
josic & pring
Windowy
driver
gysve,, corppeeer drive, chip, meed,
walls
privers
Wele
fils wob, gropries fike, server dats fox
FIB, color, imrge, blcck, images, ware, light
(c)
Figure 3: (a): L-SNE visualisation of selected topics and their top-6 words in the shared word embedding space. Different colors distinguish different topics: (b): Panoramic view of all words and learned topics; (c): Comparison of cherry-picked top-3 NSTM and WeTe topics on 20NG related to desktop keyword. In (a) and (b), stars and dots represent topie embeddings and word embeddings, respectively.
report the result of We're on 20NG with different in Fig. 2 where topic quality (TQ) is calculated as TQ = TC * TD. We also report two variants of We'Te, where one is trained using only the CT cost (WeTe(CT)), while the other using only likelihood (WeTe(TM)). We can see that 1), can be fine-tuned to balance betweeen document representation and topic quality. By carefully fine-tuning e for each dataset, one can achieve even better performance than those reported in our experiments. 2). The CT cost leads to high topic quality, and the likelihood has benefits for the representation of documents. By combining these two objectives together, WeTe can produce better performance than using only either of them.
Fig. Bl a) visualizes the learned topic embeddings. we present the top-9 topics with the highest NPMI learned by our proposed model on 20NG. For each topic, we select its top-6 words according to dk. and then feed their embeddings together into the t-SNE tool (Van der Maaten & Hinton 2008). . We can observe that the topics are highly interpretable in the word embedding space, where each topic is close to semantically related words. Besides, those words under the same topic are closer together, and words under different topics are far apart. We can also that the related topics are also closer in the embedding space, such as topic #2 and topic #5. Fig. (bb) gives overview of all word embeddings and learned topic embeddings. We find that the topic embeddings (red stars) are distributed evenly in the word embedding space, each of which plays the role of a concept center surrounded by semantically related words. Those interesting results illustrate our motivation that we can use the mixtures of topic embeddings to represent mixtures of word embeddings based on the CT cost between them. Given the query desktop. Fig. 3je) c) compares three most related topics learned from NSTM and We'T,, where We're tends to discover more diverse topics than NSTM. We attribute this to the introduction topic-to-doe cost, which enforces the topic to transport to all words that are semantically related to it with some probability. More qualitative analysis on topics are provided in Appendix
Qualitative analysis:
CONCLUSION
We introduce WeTe, a new topic modeling framework where each document is viewed as a bag of word-embedding vectors and each topic is modeled an embedding vector in the shared word- embedding space. We'e views the learning of a topic model as the process minimizing the expected CT cost between those two sets over all documents. which avoids several challenges of existing TMs. Extensive experiments show that the proposed model outperforms competitive methods for both mining high quality topics and deriving better document representation. Thanks to the introduction of the pretrained word embeddings, We'ee achieves superior performance on short and regular texts. Moreover, the proposed model reduces the need to pre-define the size of the vocabulary, which makes WeTe more flexible in practical tasks.

---

Published as a conference paper at ICLR 2022
7 ACKNOWLEDGEMENTS
H. Zheng, K. Tanwisuth, and M. Zhou acknowledge the support of NSF IIS-181269) and a gift fund from ByteDance Inc. Chen acknowledges the support of NSFC (U21 and 61771361), Shaanxi Youth Innovation Team Project, the 111 Project (No. B18039) and the Program for Oversea Talent by Chinese Central Government.
REFERENCES
David M. Blei, Andrew Y. Ng, and Michael I. Jordan. Latent Dirichlet allocation. J. Mach. Learn. Res., 3:993-1022, 2003. o http:////m...a.o.//////////////a
Sophie Burkhardt and Stefan Kramer. Decoupling sparsity and smoothness in the Dirichlet variational autoencoder topic model. Journal of Machine Learning Research, 20(131):1-27, 2019.
Dallas Card, Chenhao Tan, and Noah A. Smith. Neural models for documents with metadata. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics, ACI. 2018, Melbourne, Australia, July 15-20, 2018, Volume I: Long Papers, pp. 2031-2040. Association for Computational Linguistics, 2018.
Marco Cuturi. Sinkhorn distances: Lightspeed computation of optimal transport. Advances in neural information processing systems, 26:2292-2300 2013.
Adji B Dieng, Francisco JR Ruiz, and David M Blei. Topic modeling in embedding spaces. Transac- tions of the Association for Computational Linguistics, 8:439-453, 2020.
Zhibin Duan, Dongsheng Wang, Bo Chen, Chaojic Wang, Wenchao Chen, Yewen Li, Jie Ren, and Mingyuan Zhou. Sawtooth factorial topic embeddings guided gamma belief network. In International Conference Machine Learning, pp.
PMLR, 2021a.
Zhibin Duan, Yi Xu, Bo Chen, Chaojie Wang, Mingyuan Zhou, et al. Topicnet: Semantic graph- guided topic discovery. Advances in Neural Information Processing Systems, 34, 2021b.
Charlie Frogner, Chiyuan Zhang, Hossein Mobahi, Mauricio Araya-Polo, and Tomaso A. Poggio. Learning with a Wasserstein loss. In Advances in Neural Information Processing Systems 28: Annual Conference on Neural Information Processing Systems 2015, December 7-12, 2015, Montreal, Quebec, Canada, pp. 1053-20611, 2015.
Thomas L Griffiths and Mark Steyvers. Finding scientific topics. Proceedings of the National academy of Sciences, 101(s 1):5228-522 5235, 2004.
Dandan Guo, Bo Chen, Ruiying Lu, and Mingyuan Zhou. Recurrent hierarchical RNN for language generation. In International Conference on Machine Learning, pp. 3810-3821. PMLR, 2020.
Matthew Hoffman, Francis Bach, and David Blei. Online learning for latent Dirichlet allocation. In advances in neural information processing systems, pp. 856-864. Citeseer, 2010.
Viet Huynh, He Zhao, and Dinh Phung. OTLDA: A geometry-aware optimal transport approach for topic modeling. In Advances in Neural Information Processing Systems, 2020.
Kamrun Naher Keya, Yannis Papanikolaou, and James R. Foulds. Neural embedding allocation: Distributed representations of topic models. CoRR, abs/1909 2019. URL BtLLhttp://axxi
cxq/aba/1909.0o 04702
Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In 3rd International Conference on Learning Representations, 2015.
Diederik P. Kingma and Max Welling. Auto-encoding variational Bayes. In 2nd International Conference on Learning Representations, ICLR 2014, Banff, AB, Canada, April 14-16, 2014, Conference Track Proceedings, 2014.
10

---

Published as a conference paper at ICLR 2022
Jens Lehmann, Robert Isele, Max Jakob, Anja Jentzsch, Dimitris Kontokostas, Pablo N Mendes, Sebastian Hellmann, Mohamed Morsey, Patrick Van Kleef, Sören Auer, et al. Dbpedia-a large- scale, multilingual knowledge base extracted from wikipedia. Semantic web, 6(2):167-195, 2015.
D. D. Lewis, Y. Yang, T. G. Rose, and F. Li. Revl: A new benchmark collection for text categorization research. Journal of Machine Learning Research, 5:361-397, 2004.
Chenliang Li, Haoran Wang, Zhiqian Zhang, Aixin Sun, and Zongyang Ma. Topic modeling for short texts with auxiliary word embeddings. In Proceedings of the 39th International ACM SIGIR conference on Research and Development Information Retrieval, pp- 165-174, 2016.
Yishu Miao, Lei Yu, and Phil Blunsom. Neural variational inference for text processing. In International conference on machine learning, pp. 1727-1736. PMLR, 2016.
Christopher E. Moody, Mixing Dirichlet topic models and word embeddings to make Ida2vec. CoRR, org/
000s/1605,0009, 2016. URL
02019
Feng Nan, Ran Ding, Ramesh Nallapati, and Bing Xiang. Topic modeling with Wasserstein au- toencoders. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pp. 6345-6381, 2019.
Dat Quoc Nguyen, Richard Billingsley, Lan Du, and Mark Johnson. Improving topic models with latent feature word representations. Transactions of the Association for Computational Linguistics, 3:299-311 313, 2015.
Heesoo Park and Jongwuk Lee. Decoupled word embeddings using latent topics. In Proceedings of the 35th Annual ACM Symposium Applied Computing, pp. 875-882, 2020.
Jeffrey Pennington, Richard Socher, and Christopher D Manning. GloVe: Global vectors for word representation. In Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP), pp. 1532-1543, 2014.
James Petterson, Alexander J Smola, Tibério S Caetano, Wray L Buntine, Shravan M Narayanamurthy. et al. Word features for latent Dirichlet allocation. In NIPS, pp. 1921-1929, 2010.
Gabriel Peyré and Marco Cuturi. Computational optimal transport. Found. Trends Mach. Learn., 11(5-6):355-677, 2019. doi: 10.1561/2200000073. URL 2200000073
1561/
Xuan-Hieu Phan, Le-Minh Nguyen, and Susumu Horiguchi. Learning classify short and sparse text & web with hidden topics from large-scale data collections. In Proceedings of the 17th international conference World Wide Web, pp. 91-100, 2008
Danilo Jimenez Rezende, Shakir Mohamed, and Daan Wierstra. Stochastic backpropagation and approximate inference in deep generative models. International conference on machine learning, pp. 1278-1286. PMLR, 2014.
Hinrich Schütze, Christopher D Manning, and Prabhakar Raghavan. Introduction to information retrieval, volume 39. Cambridge University Press Cambridge, 2008
Bei Shi, Wai Lam, Shoaib Jameel, Steven Schockaert, and Kwun Ping Lai. Jointly learning word embeddings and latent topics. In Noriko Kando, Tetsuya Sakai, Hideo Joho, Hang Li, Arjen P. de Vries, and Ryen W. White (eds.), Proceedings of the 40th International ACM SIGIR Conference on Research and Development in Information Retrieval, Shinjuku, Tokyo, Japan, August 7-11, 2017, pp. 375-384 ACM, 2017, doi: 0411145/307713630088866 URL httts::/doi.. DO..11453307110
30 080806
Akash Srivastava and Charles Sutton. Autoencoding variational inference for topic models. In 5th International Conference Learning Representations, 2017.
Korawat Tanwisuth, XINJIE FAN, Huangjie Zheng, Shujian Zhang, Hao Zhang, Bo Chen, and Mingyuan Zhou. A prototype-oriented framework for unsupervised domain adaptation. In A. Beygelzimer, Y. Dauphin, P. Liang, and J. Wortman Vaughan (eds.), Advances in Neural Information Processing Systems, 2021. URL https://openzevit net/ ffo.umidddd
yH2VrkpiCK6
11

---

Published as a conference paper at ICLR 2022
Yee Whye Teh, Michael I Jordan, Matthew J Beal, and David Blei. Hierarchical Dirichlet processes. Publications of the American Statistical Association, 101(476):1566 1581, 2006.
Ilya O. Tolstikhin, Olivier Bousquet, Sylvain Gelly, and Bernhard Schölkopf. Wasserstein auto- encoders. In 6th International Conference on Learning Representations, 2018.
Laurens Van der Maaten and Geoffrey Hinton. Visualizing data using t-SNE. Journal of machine learning research, 9(11), 2008.
Daniele Vitale, Paolo Ferragina, and Ugo Scaiella. Classification of short texts by deploying topical annotations. In European Conference Information Retrieval, pp. 376-387. Springer, 2012.
Chaojie Wang, Hao Zhang, Bo Chen, Dongsheng Wang, Zhengjue Wang, and Mingyuan Zhou. Deep relational topic modeling via graph Poisson gamma belief network. Advances in Neural Information Processing Systems, 33:488-500, 2020.
Hongteng Xu, Wenlin Wang, Wei Liu, and Lawrence Carin. Distilled Wasserstein learning for word embedding and topic modeling. In Advances in Neural Information Processing Systems 31: Annual Conference on Neural Information Processing Systems 2018, NeurIPS 2018, December 3-8, 2018, Montréal, Canada, pp. 1723-1732, 2018
Mikhail Yurochkin, Sebastian Claici, Edward Chien, Farzaneh Mirzazadeh, and Justin M. Solomon. Hierarchical optimal transport for document representation. In Advances in Neural Information Processing Systems, pp. 1599-1609, 2019.
Hao Zhang, Bo Chen, Dandan Guo, and Mingyuan Zhou. WHAI Weibull hybrid autoencoding inference for deep topic modeling. In 6th International Conference Learning Representations, 2018.
Xiang Zhang, Junbo Zhao, and Yann LeCun. Character-level convolutional networks for text classification, arXiv preprint arXiv: 150900002222 2015.
He Zhao, Lan Du, and Wray Buntine. A word embeddings informed focused topic model. In Asian conference on machine learning, pp. 423 438, 2017a.
He Zhao, Lan Du, Wray Buntine, and Gang Liu. MetaLDA: A topic model that efficiently incorporates meta information. In 2017 IEEE International Conference Data Mining (ICDM), pp. 635-644, 2017b.
He Zhao, D) Phung, V Huynh, Le, and W Buntine. Neural topic model via optimal transport, 2020.
He Zhao, Dinh Phung, Viet Huynh, Yuan Jin, Lan Du, and Wray Buntine. Topic modelling meets deep neural networks: survey. In The 30th International Joint Conference on Artificial Intelligence (IJCAI), pp. 4713-4720 2021.
Huangjie Zheng and Mingyuan Zhou. Comparing probability distributions with conditional transport. arXiv preprint arXiv:2012.14100 2020.
Mingyuan Zhou, Lauren Hannah, David Dunson, and Lawrence Carin. Beta-negative binomial process and Poisson factor analysis. In Artificial Intelligence and Statistics, pp. 1462-1471. PMLR, 2012.
Mingyuan Zhou, Yulai Cong, and Bo Chen. Augmentable gamma belief networks. The Journal of Machine Learning Research, 17(1)5656-5699 2016.
Yuan Zuo, Junjie Wu, Hui Zhang, Hao Lin, Fei Wang, Ke Xu, and Hui Xiong. Topic modeling of short texts: A pseedd-dooooe document view. In Proceedings of the 22nd ACM SIGKDD international conference on knowledge discovery and data mining, pp. 2105-21 2016.
12

---

Published as a conference paper at ICLR 2022
A DERIVATION OF EQUATION 12
The total CT cost can be written as:
J
L = Σ +P,
+ L9,
J
N
N₂
K cwwa On (T [);)),
jk
c W az ( (c))
j=1 k=1
1=1
k=
where
TN, (wzj (xx)) = e-d(wsc:
No Σ -v=11 e des
and
a wji)
ΣK= === C add(wa.a))8)
Recall the definition of c(W O2k) in Equation 10 of the main paper, and d(wss: ax) -wo ork.
With the fact that Σ
K
0
5k
1, we can rewrite the total CT cost L
K
N
e e00,,0
N,
K
e u "0
L
w.
a
w
OEk
jk
2k'=1 e w au)) 8,
1=1
P=
:=1
k=1
ト
jk
K ==
o 0g
x==
which, to the best of our knowledge, does not resemble any existing topic modeling loss functions. The two terms in the bracket have very intriguing relationship, where in the fraction formula 65R and Nj swap their locations and Σn and E. also swap their locations. To minimize the first term, we will need to ensure the denominator E "3 N, to be sufficiently large whenever 85x is non-negligible, which can be achieved only if the inner products of the words in document j and topic h aggregate to sufficiently large value whenever 8z+ > 0 (i.c., each inferred topic embedding vector needs be close to at least one word embedding vector of a given document when that topic has non-negligible proportion in that document). To minimize the second term, we will need to ensure the denominator Σ e 0jk: be large for every single word, which for word can be achieved only if there exists at least one topic that has large inner product with word (i.c., each word can find at least single non-negligibly-we weighted topie that is sufficiently close to it, other words, the inferred topics need to have good coverage of the word embedding space occupied by the corpus).
B TRAINING ALGORITHM
The training procedure of the proposed WeTe is shown in Algorithm I
C DATASETS
Our experiments are conducted on six widely-used benchmark text datasets, varying in seales and document lengths, including 20 News Group (20NG), DBpedia (DP) (Lehmann al. 2015). Web Snippets (WS) (Phan et al.) 2008), Tag My News (TMN) (Vitale et al. 2012), Reuters extracted from the Seuters-21575 dataset, and Reuters Corpus Volume (RCV2) (Lewis et al. 2004), where Ws. DP, and TMN consist of short documents. To demonstrate the scalability of the proposed model for document clustering, For multi-label RCV2 dataset, we left documents in test dataset with single label at second level, resulting in 52 categories, We load the pretrained word embeddings from Glo (Pennington et al. 2014).
13

---

Published as a conference paper at ICLR 2022
Algorithm 1 Training algorithm for our proposed model.
Input: training documents, pretrained word embeddings E, topic number K, hyperparameter c. Initialize: topic embeddings a, encoder parameters W. for iter = 1.2.3.... do
Sample a batch of imput documents and represent them as the empirical distributions {Po
and form the document-specifee empirical topic distribution [Qy1)n1
With the cost function Equation I0 and transport probabilities Equation 9 and Equation o
compute the CT loss with Equation 12 as the first term of Equation T3:
Compute the topic M with Equation and the topic proportions (0, with input a ; denoted
q(O, xz)) Weibull (w(x)) (2z). [ty]]; compute the second term of Equation 13
Update a and W according to Equation 13
end for
Table C. 1: Statistics of the datasets
Number of docs
Vocabulary size(V)
average length
Number of labels
20NG
18,864
22,636 9,835
108
6
DP
449,665
22
14
WS
12,337
10,052
15
8
TMN
32,597
13,368
18
7
Reuters
11,367
8,817
74
N/A
RCV2
804,414
7,282
75
52
20NG 20 Newsgroups consists of 18,846 articles. We remove stop words and words with document frequency less than 100 times. We also ignore documents that contain only one word from the corpus. We only use the 6 super-categories denote as 20NG(6) in the clustering task, as there are confusing overlaps its official 20 sam.p.syymcc..kee mac. hardware.
20NG's 's ground truth and
categories, e.g., comp.sys... sbs. pc. hardware and comp.sys.
D DBpedia is crowd- dataset extracted from Wikipedia pages. We follow the pre-processing process in Zhang et al. (2013). where the fields that we use for this dataset contain title and abstract of each Wikipedia article.
WS: Web Snippets, used in Li et al. (2016) and Zhao et al. (2020), contains 12,237 web search snippets with 8 categories. There are 10,052 tokens vocabulary and the average length of a snippet is 15.
TMN
Tag My News consists of 32,597 RSS news snippets from Tag My News with 7 categories. Each snippet contains a title and short description, and the average length of snippet is 18.
Reuter A widely used corpus extracted from the Reuters-21578 dataset. We only use it on topic quality task, and there are 11,367 documents with 8,817 tokens vocabulary.
RCV Reuters Corpus Volume used Zhao al. 2020), consists of 804,414 docu- ments, whose vocabulary size is 7282 and average length is 75.
summary of dataset statistics shown Table C.
D ADDITIONAL TOPIC QUALITY RESULT
In Fig. D. we report topic coherence (TC) and topic diversity (TD) for varied methods on TMN and Reuters dataset, which confirms that our proposed model outperforms the others in high quality topic discovering.
http:///woneeccmm/ http:///w....ommmoooo////////nno
Page
http://aubee.di.uiii.iiiie dataset/
htt::///ddc...uuccdddaaassssssss html
14

---

Published as a conference paper at ICLR 2022
When the topic number becomes insufficient, the topic distribution p(m= k) often resembles the corpus distribution p(w), where high frequency words become the top terms related to most topics. We want topics learned from WeTe to be specific (e.g., not overly general). Topic Specificity (TS) is defined by the average KI. divergence from each topic's distribution to the corpus distribution:
1 K
TS
KLpp(m | pb)l((w))
Jointly with topic diversity and topic coherence, we report topic specificity (TS) of various methods six datasets in Table D. it be found that the proposed model superior its competitors all datasets, indicating that We'Te produces more useful and specific topics than other NTMs.
In Tables D. D. 3 and D. 4 show the top-10 words of the selected topies learned from We're and its two variants on 20NG, TMN, and RCV2, respectively. We note that the proposed model can only learn meaningful topics from the pretrained word embeddings, but also learn word and topic embeddings jointly from scratch, discovering equally meaningful topics.
TMN(TC)
Reuters(TC)
WeTe
WeTe(F))
WeTe(N)
NSTM
0.2
0.15
0.1
0.15
PFA
WEI-FTM
0.05
-0.05
10%
50%
100%
10%
50%
100%
TMN(TD)
Reuters(TD)
0.9
0.8
0.8
0.8
100%
100%
Figure D. 1: The first and second rows show topic coherence (TC) and topic diversity (TD), respectively. for different methods on the TMN and Reuters datasets. In each subfigure, the horizontal axis indicates the proportion of selected topics according their NPMIs. For both TC and TD, higher hetter. WeTe(F) and WeTc(N) denote that we finetune the word embeddings and learn from scratch, respectively.
Table D. 1:
Topic specificity (TS) of various methods on web 20NG, DP, RCV2, TMN, and Twitter datasets, higher is better.
Method LDA
WS
20NG
DP
RCV2
TMN
Twitter 3.95
3.84
4.67
5.42
7.08
3.89
DVAE
2.50
3.12
4.04
5.45
2.86
1.73
NSTM
1.49
1.97
4.47
6.24
1.07
2.27
WeTe
4.51
5.71
5.58
7.94
4.16
4.43
WeTe(F)
4.48
5.74
5.74
7.42
4.07
4.36
WeTe(N)
4.01
5.42
5.38
6.98
3.89
4.13
15

---

Published as a conference paper at ICLR 2022
Table D. 2:
Topics learned from WeTe, WeTe(F), and WeTe(N) on the 20NG dataset, where the top-10 10 words for each topic are visualized.
Method
Top words
space nasa orbit spacecraft mars shuttle launch flight rocket solar window image display color screen graphics output motif mode format game team hockey nhl play teams players win player league season space satellite launch nasa shuttle mission research lunar earth technology window problem card monitor mouse video windows driver memory screen
WeTe
WeTe(F)
team hockey game players season league play goal year teams space launch satellite nasa shuttle earth lunar first mission system window display application server mit screen problem get program year game team players baseball runs games last season league
WeTe(N)
Table D. 3: Topics learned from WeTe, WeTe(F), and WeTe(N) on the TMN dataset, where the top-10 words for each topic visualized.
Method
Top words
million billion company buy group share amp firm bid sell wedding idol royal william prince singer star kate rock taylor team season sports league teams soccer field manchester briefing club million billion deal group company firm offer buy shares sell star stars movie fans idol hollywood box fan film super players nfl coach draft teams football basketball player nba lockout million company video deal online internet apple google mobile media show star theater book idol royal dies space wedding music coach nfl players team state season sports national tournament basketball
WeTe
WeTe(F)
WeTe(N)
Table D. 4: Topics learned from WeTe, WeTe(F), and WeTe(N) on the RCV2 dataset, where the top- words for each topic are visualized.
Method
Top words
million total billion asset worth sale cash debt cost payout oil gas fuel barrel palm petroleum gulf shell bpd cubic olein network internet custom access microsoft web design tv broadcast media sale sold bought sell retail buy chain auction supermarket shop discount oil barrel nymex brent gas petroleum fuel gallon wti gulf system network personnel microsoft inform chief internet web unit custom percent billion year million market rate month economic growth dollar oil gas barrel brent fuel output sulphur petroleum nymex diesel gallon network channel radio tv media station broadcast film video disney
WeTe
WeTe(F)
WeTe(N)
E THE LEARNED WORD EMBEDDINGS
WeTe(N) provides a new method to learn word embeddings from scratch. Recall the topic-to-doc CT cost for special document j in WeTe:
c c-d(wy
C = cc ②b)
E (0)) Wis E { [W WyN,
This transport cost mirrors the likelihood in skip-gram model. Such skip-gram models use the central word to predict the surrounding words. By contrast, our WeTe uses the topic embedding vectors & as the central words and generates the document words, rather than a window of surrounding words. In other words, kkip-gram models can be viewed c= = N₁. To evaluate the word embeddings learned from WeTe(N), given a query word, we visualize top-8 words that are most closest to it. We compare WeTe(NN with GloVe in Table
special variant of WeTe with the window size
16

---

Published as a conference paper at ICLR 2022
Compared to GloVe, the word embedding learned by WeTe(NN tends to be more semantically diverse. For example, "download", "modem" for "pe", and "goal", "win" for "game". We attribute this to the of the document-lever context.
Table E.
Comparison of the most relevant words for the query words on RCV2 dataset.
Query word
Method
Top words
GloVe
pe desktop computer software macintosh computers pentium pes microsoft xp pe desktop macintosh pes microsoft internet os download mac modem game games season play match player league team scored playoff's game scason play match team playoff bowl goal win coach
pe
(cTTe(N)
GloVe
game
WeTe(N)
GloVe
world cup international olympic curopcan championships event curope world cup international curopean event asian asia women nation tcam school college university schools students education elementary graduate school high student campus district church program degree taught harvard
world
WeTe(N)
GloVe
school
WcTe(N)
F COMPLEXITY ANALYSIS
Table F. I: Time and space complexity analysis of WeTe. CT and TM denote the conditional transport part and topic model part, respectively. We ignore the 3-layer encoder because is shared with all neural topic models.
CT
TM
Time complexity Space complexity
(2V+4N (V+K)d (2V+4N
2VKd + BVKK2 + +4VB (V+B)K+VB
As neural topic model, WeTe has comparable complexity to other neural topic models. In detail, for mini-bated of documents with batch-size B, NB denotes the total words in the mini-bar batih We summary the time and space complexity Table I where CT denotes conditional transport and TM means the topic model, we here ignore the 3-layer neural encoder, due to it shared with other neural topic models. V is the vocabulary size, K is the number of topics and d is the embedding size. We can see that CT obtains linear complexity in both time and space with respect to the vocabulary and the total number of words in the mini-batch.
(a)
1.00
Normalized Loss
WeTe
0.75
NSTM
ETM
0.50
DVAE
0.25
0.00
2000
6000
8000
Batch(n)
(b)
We also compare WeTe with other three NTMs on large RCV2 -==13755,N80444411 with a large topic setting (K=500). All the methods evaluated with batch size of 500. The normalized training loss is shown Fig. F. where the direct comparability between losses is not available due to the different designs. It demonstrates that the proposed model has acceptable learning speed compared with other NTMs. Fig. F. a[a) shows that We're requires fewer iterations compared to DVAE and ETM. And Fig- F. T[b) demonstrates that our WeTe has similar time consumption to DVAE. Although ETM and NSTM have faster training speed, their performance both topic quality and clustering task incomparable to ours. In other words, WeTe balances the performance and speed well.
1.00
Normalized Loss
Nvidia RTX 2080-Ti GPU
0.75
0.50
0.25
0.00
500
1000
1500
2000
2500
Time(s)
Figure F. 1: Training loss on RCV2
over batches (a) and seconds (b).
17


Reviews:
[{"review_id": "lSjv4BGk4w8", "reviewer": "Reviewer_eWQp", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strong points:\n\n- While the combination of topic models and word embeddings has received substantial study, it remains an interesting problem.  An optimal transport (or conditional transport) approach such as the one proposed is a promising direction.\n\n- The ability to initialize with pre-trained word embeddings and fine-tune the model is valuable (though this paper is not the first to do this).\n\n- Experimental results versus a range of baseline methods are encouraging.\n\n- The qualitative results on embeddings of words and topics were nice to see.\n\nWeak points:\n\n- My biggest complaint with this paper is that the necessity and motivation for the proposed method is unclear.  While combining topic models and word embeddings is a worthy problem, and transport methods are an interesting potential solution, this paper is far from the first to study the intersection of topic models and word embeddings.  This is already a crowded space which includes the cited papers, such as the ETM and neural topic models, and several others as well.  The present manuscript doesn't make a compelling argument that the proposed method resolves urgent problems in this area.  Furthermore, the proposed method is stated then studied without a principled justification. It reads like the authors wanted to try this method, so they did, rather than that there was a particular reason or mathematical justification for why the method is designed the way that it is.\n\n- The manuscript was quite hard to follow at times, partly because the lack of clearly stated justification for the proposed approach leads to substantial effort to understand what exactly the authors are doing.  The use of non-standard notation in some instances was a contributing factor in this issue.\n\n- Transformer-based language models such as BERT have become the state of the art for representation learning in natural language processing in recent times.  This work should ideally compare with BERT, etc, or justify why it does not do so.  It also needs to discuss the merits of the proposed approach versus transformer-based language models.\n\n- Since the objective function (Equation 13) combines the conditional transport objective with a likelihood objective, both an approach using conditional transport alone and an approach using likelihood optimization alone should be compared to as baselines.\n\n- Separately encoding transport costs c(w, \\phi) and distances d(w, \\phi), which are \"related but also with differences\" (pg 4), seems inelegant. \n\n- If speed advantages are claimed (cf. pg 6, \"more efficient learning\"), this should be demonstrated with rigorous experiments.\n\n- Hyperparameter tuning could have been done more systematically for the experiments, rather than using fixed values for most experiments, followed by a brief sensitivity test.\n\n- The training algorithm could have been explained more clearly, and in the paper itself instead of in the appendix.\n\n\nAdditional feedback / minor suggestions:\n\nPg 6, Equation 6, it looks like there is an issue in the last step, which doesn't clearly follow from the previous step.  I think there is a missing term on the top and bottom here corresponding to P_j(W_{ji}).  Compare to Equation 9, which has \\theta terms on the top and bottom of the last equation in the line.\n\nPg. 10 (references) - \"Latent dirichlet allocation\" needs a capital \"D.\"\n\nPg. 12 (Appendix), \"the training algorithm of our WeTe is shown in Algorithm 1\" needs a capital T.\n\nA couple of highly relevant missing references:\nK. Keya, Y. Papanikolaou, and J. R. Foulds. Neural embedding allocation: Distributed representations of topic models. ArXiv preprint arXiv:1909.04702 [cs.CL], 2019.\n\nJ. R. Foulds. Mixed Membership Word Embeddings for Computational Social Science. Proceedings of the 21st International Conference on Artificial Intelligence and Statistics (AISTATS), 2018.", "summary_of_the_paper": "The manuscript proposes a combined topic modeling and word embedding method.  The idea is to define word and topic embeddings in the same space, represent documents based on both of them, and use a transport-based method to train these two representations to be similar to each other.  The final objective function combines a \"conditional transport\" objective with a log-likelihood term, and is optimized via Adam.  Experiments find that the method achieves good topic coherence scores and document clustering metrics compared to baselines.\n", "summary_of_the_review": "While the proposed method has potential and there are some positive aspects of the work, the manuscript is not ready for publication due to deficiencies in several aspects, including: motivation of the need for the general approach, justification of the specific choices in the approach, other clarity issues, relationship to BERT and transformer-based language models, and certain aspects of the experiments.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "R0fXa9RFXO", "reviewer": "Reviewer_2jw3", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n1. The authors develop a novel model to jointly model latent topics and word embeddings. While the model is novel, there are already existing works that have jointly modelled latent topics and word embeddings exploiting their complementary advantages. The authors also present the derivation of the model and the key equations that demonstrate how the model works.\n2. The authors have also tried to intertwine their model with the experiments. As a result, the authors have conducted some experiments to demonstrate that the model improves upon some existing methods.\n\nThere are several ways in which this paper can be improved:\n1. The authors can strengthen their experiments by experimenting against some downstream applications such as document classification. Unsure of why the authors did not cite this work that has used document classification as one of the experiments (https://dl.acm.org/doi/10.1145/3077136.3080806).\n2. Relating to the work cited in (1) above, it would be nice to have some cooperation on how the proposed work differs from the work that is already published by Bei et al. There are some follow-up works too since this work was published and the authors are requested to have a go at them too. Where Bei's work would be crucial is mainly in comparing against the run-time performance of the models where Bei's work is a much simpler version of a model that jointly learns latent topics and embeddings. Besides that, the effectiveness of the model could be compared as a result.\n3. Correcting some grammar issues, e.g., facilitates instead of facilities, latter instead of later; authors can use the PDF search feature to find such words.\n4. The authors could consider theoretically comparing their embeddings with those static embeddings obtained using Skipgram or GloVe. Or, do we expect that the embeddings that the authors learn using their model are different in property from traditional static embeddings such as GloVe or Skipgram.", "summary_of_the_paper": "Jointly learning latent topic representations with word embeddings has caught some attention in the past. The reason why researchers choose this route is to exploit the complementary advantage of both these models, for instance, to improve the performance of the model on short texts. In this paper, the authors study a new topic model where topics and words are encoded in the same embedding space. Usually, in the traditional topic modelling literature, the posterior inference is realised using algorithms such as Gibbs sampling or collapsed Gibbs sampling; however, in this paper, since the authors exploit complementary knowledge between the two models, the end-to-end model is jointly trained the document-specific discrete latent topic distribution is made to appear as close as possible to the discrete distribution over words in the document. A probabilistic bidirectional transport model is developed that measures the difference between the discrete distributions in the embedding space. The authors also discuss several key advantages of the model.\n\nExperiments are then followed with topic diversity and coherence measures as evaluation metrics which are widely used in the literature. Qualitatively, the authors show some example results from their dataset in the form of a t-SNE plot.", "summary_of_the_review": "While there are strengths in terms of the model novelty, there is some weakness too. The paper can be further improved from various angles such as strengthening the experiments section by comprising with some closely relevant works including conducting some downstream application analysis such as document classification performance.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "Not applicable"}, "extra_text": {}}, {"review_id": "GQoPOvkDpvw", "reviewer": "Reviewer_Bvas", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n1. A diversity of datasets including web pages and news articles are used in the evaluation, and experiments show the superiority of the proposed topic modeling method. \n2. The algorithm is reasonable and has the advantage that it can learn word embeddings and topic embeddings from scratch. \n3. The paper is well written and easy to follow. The authors provide concrete figures and tables to show the results of experiments. \n\nWeaknesses:\n1. Given a few recent works on learning dense word vectors jointly with latent document-level mixtures of topic vectors, the novelty of this paper is not significant. For example, there's a missing reference:  Moody, Christopher E. \"Mixing dirichlet topic models and word embeddings to make lda2vec.\" arXiv preprint arXiv:1605.02019 (2016). \n\n2. The experiment results shown in this paper do not include t-test results such as p-values. We cannot tell how significant the performance of the proposed algorithm is, compared to other state-of-the-art methods. \n\n3. To show the interpretability of the resulting topics, it'd be great if the authors can provide qualitative analysis such as in Chang, Jonathan, et al. \"Reading tea leaves: How humans interpret topic models.\" Advances in neural information processing systems. 2009.\n\n4. For comparison of time/space complexities, it'd be great if the authors can provide a detailed complexity analysis. \n\n\n", "summary_of_the_paper": "This paper presents a new topic modeling framework called WeTe: each document in text corpus is represented as a bag of word embeddings vectors, and each topic is modeled as an embedding vector in the shared word embedding space. WeTe minimizes the expected difference between those two sets over all documents. A bidirectional transport-based method is proposed to learn the topic embeddings as well as topic proportions for documents efficiently. Extensive experiments on news and web pages show that the proposed model outperforms competitive methods for both deriving high-quality latent topics and generating better document representations for clustering tasks. ", "summary_of_the_review": "The idea of learning latent topics for text documents using bag-of-words embeddings is not quite new given that a few previous papers describe similar methods. However, it's still empirically useful to introduce concrete optimization methods to learn such a model. This paper presents extensive experiments on news and web pages to show the superiority of the proposed method, compared to other state-of-the-art methods. \n\n", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "BaibbZD8HzD", "reviewer": "Reviewer_W5Cw", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strength of the paper : the point of view is relatively original, since it provides a new framework for topic modelling. It relies on a version of bidirectional optimal transport which is quite origineal and embedd words and topics in the same space. The relevance of the model is assess by numerical experiments comparing in terms of perplexity this approach with others ones more classical as vanilla LDA, OTLDA...\n\nWeaknesses : the idea of mixing OT and LDA is not so new, even if bidirectional OT has not been used yet. In addition, this version of bidirectional OT is just a symmetrisation, which is not so original", "summary_of_the_paper": "The aim of the paper is to revisit topic modeling through the lens of word embeddings and using a bidirectional version optimal transport. TIn this paper each document is considered as a set of embeddings words, each topic is seen as a mixture of embedding living in the same space as words embeddings\nThe aim is to minimize the distance of the topic distribution and the words distribution for each documents", "summary_of_the_review": "The paper mixes OT and LDA in a rather original way. Nevertheless, this idea is not new and has been already proposed in other papers", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This is a borderline paper on the well researched theme of Topic models.
The strongest point of the paper is that it proposes a new topic modelling framework where both word and topic embeddings live in the same space.
 It then appeals to optimal transport theory to do the necessary training using SGD. However, this is not the first paper to examine
Topic models and Optimal Transport theory. Several papers[1,2,3] in the recent past have started investigating this line of research.  In the rebuttal phase, the author(s) justify the choice of state of the art methodologies and also discuss the key conceptual difference between 
existing literature and the submitted one. The major difference seems to 
they approach the problem differently leading to better quality topics(as measured by several metrics) and computational efficiency---existing state of the art requires more complicated iterations whereas proposed approach works with SGD. 
The manuscript, if accepted, needs to be updated considerably to reflect some of these aspects. 




[1] Hongteng Xu, Wenlin Wang, Wei Liu, and Lawrence Carin. Distilled wasserstein learning for wordembedding and topic modeling. In NeurIPS2018.

[2] Viet Huynh, He Zhao, and Dinh Phung. OTLDA: A geometry-aware optimal transport approach fortopic modeling. In NeurIPS2020.


[3] He Zhao, D Phung, V Huynh, T Le, and W Buntine. Neural topic model via optimal transport, 2020
Decision: Accept (Poster)