Neural Topic Model via Optimal Transport
Authors: He Zhao
Dinh Phung
Viet Huynh
Trung Le
Wray Buntine
Venue: ICLR 2021 Spotlight (2021)
Keywords: topic modelling
optimal transport
document analysis

Abstract: Recently, Neural Topic Models (NTMs) inspired by variational autoencoders have obtained increasingly research interest due to their promising results on text analysis. However, it is usually hard for existing NTMs to achieve good document representation and coherent/diverse topics at the same time. Moreover, they often degrade their performance severely on short documents. The requirement of reparameterisation could also comprise their training quality and model flexibility. To address these shortcomings, we present a new neural topic model via the theory of optimal transport (OT). Specifically, we propose to learn the topic distribution of a document by directly minimising its OT distance to the document's word distributions. Importantly, the cost matrix of the OT distance models the weights between topics and words, which is constructed by the distances between topics and words in an embedding space. Our proposed model can be trained efficiently with a differentiable loss. Extensive experiments show that our framework significantly outperforms the state-of-the-art NTMs on discovering more coherent and diverse topics and deriving better document representations for both regular and short texts.

Published as a conference paper at ICLR 2021
NEURAL TOPIC MODEL VIA OPTIMAL TRANSPORT
He Zhao, Dinh Phung, Viet Huynh, Trung Le, Wray Buntine Department of Data Science and Artificial Intelligence, Faculty of Information Technology
Monash University, Australia {ethan. zhao, dinh. phung, viet. aheethhunh, trunglm, wray seay.bun www.buuiinnn inn) @monash. edu
ABSTRACT
Recently, Neural Topic Models (NTMs) inspired by variational autoencoders have obtained increasingly research interest due to their promising results text analysis. However, it is usually hard for existing NTMs to achieve good document representation and coherent/diverse topics at the same time. Moreover. they often degrade their performance severely on short documents. The requirement of reparameterisation could also comprise their training quality and model flexibility. To address these shortcomings, we present new neural topic model via the theory optimal transport (OT). Specifically, we propose to learn the topic distribution of document by directly minimising its OT distance to the document's 's word distributions. Importantly, the cost matrix of the OT distance models the weights between topics and words, which is constructed by the distances between topics and words in an embedding space. Our proposed model can be trained efficiently with differentiable loss. Extensive experiments show that framework significantly outperforms the state-of-the-ard NTMs on discovering more coherent and diverse topics and deriving better document representations for both regular and short texts.
INTRODUCTION
As an unsupervised approach, topic modelling has enjoyed great success in automatic text analysis. In general, topic model aims to discover set of latent topics from a collection of documents, each of which describes an interpretable semantic concept. Topic models like Latent Dirichlet Allocation (LDA) [Blii et al. 2003) and its hicrarchical/Bayesiin extensions, e.g. in Blei et al. (2010); Paisley et al. 20155] Gan et al. (2015); ::; Zhou et al. (2016) have achieved impressive performance for docu- ment analysis. Recently the developments of Variational AutoEncoders (VAEs) and Autoencoding Variational Inference (AVI) (Kingma & Welling 2013) Rezende et al.) 2014) have facilitated the proposal of Neural Topic Models (NTMs) such as in Miao et al. (2016); SSriaastvaa & Sutton (2017); Krishnan et al. (2018); >); BBurkhartt & Kramer (2019). Inspired by VAE, many NTMs use encoder that takes the Bag-of- Words (BoW) representation of a document as input and approximates the posterior distribution of the latent topics. The posterior samples are further input into decoder to reconstruct the BoW representation. Compared with conventional topic models, NTMs usually enjoy better flexibility and scalability, which are important for the applications on large-scale data.
Despite the promising performance and recent popularity, there are several shortcomings for existing NTMs, which could hinder their usefulness and further extensions. i) The training and inference processes of NTMs are typically complex due to the prior and posterior constructions of latent topics. To encourage topic sparsity and smoothness, Dirichlet (Burkhardt & Kramer| 2019) or gamma (Zhang et al. 2018) distributions are usually used as the prior and posterior of topics, but reparameterisstion inapplicable to them, thus, complex sampling schemes or approximations have to be used, which could limit the model flcxibility. ii) A desideratum of topic model is to generate better topical representations of documents with more coherent and diverse topics; but for many existing NTMs, is hard to achieve good document representation and coherent/diverse topics at the same time. This is because the objective of NTMs is to achieve lower reconstruction error, which usually means topics are less coherent and diverse, as observed and analysed in Srivastava & Sutton (017): Burkhardt & Kramer (2019). iii) It is well well-known that topic models degrade their performance severely on short documents such tweets, news headlines and product reviews, each individual document contains insufficient word co-occurrence information. This issue can be exacerbated for NTMs because of the use of the encoder and decoder networks, which are usually more vulnerable to data sparsity.

---

Published as a conference paper at ICLR 2021
To address the above shortcomings for NTMs, we in this paper propose neural topic model, which is built upon a novel Optimal Transport (OT) framework derived from a new view of topic modelling. For a document, we consider its content to be encoded by two representations: the observed representation, a, a distribution over all the words in the vocabulary and the latent representation, 2, a distribution over all the topics. a can be obtained by normalising document's word count vector while needs to be learned by model. For a document collection, the vocabulary size (i.e., the number of unique words) can be very large but one individual document usually consists of a tiny subset of the words. Therefore, I is a sparse and low-level representation of the semantic information of a document. As the number of topies is much smaller than the vocabulary size, z the relatively dense and high-level representation of the same content. Therefore, the learning of a topic model can be viewed as the process of learning the distribution z to as close to the distribution a as possible. Accordingly, is crucial to investigate how to measure the distance between two distributions with different supports (i.e., words to æ and topics to z). As optimal transport is a powerful tool for measuring the distance travelled in transporting the mass in one distribution to match another given specific cost function, and recent development on computational OT (e.g.. in Cuturi (2013); Frogner et al. (2015); (Seguy) et al.] (2018); |Peyé et al. (2199) has shown the promising feasibility to efficiently compute OT for large-scale problems, it is natural for us to develop a new NTM based on the minimisation of OT.
Specifically, our model leverages an encoder that outputs topic distribution z of a document by taking its word count vector as input like standard NTMss but we minimise the OT distance hetween æ and z, which are two discrete distributions on the support of words and topics, respectively. Notably, the cost function of the OT distance specifies the weights between topics and words, which we define as the distance in an embedding space. To represent their semantics, all the topics and words are embedded in this space. By leveraging the pretrained word embeddings, the cost function is then a function of topic embeddings, which will be learned jointly with the encoder. With the advanced properties of OT on modelling geometric structures on spaces of probability distributions, our model able to achieve a better balance between obtaining good document representation and generating coherent/diverse topics. In addition, model eases the burden of designing complex sampling schemes for the posterior of NTMs. More interestingly, our model is a natural way of incorporating pretrained word embeddings, which have been demonstrated to alleviate the issue of insufficient
word co-occurrence information in short texts (Zhao et al.) [017]] Dieng et al. 2020). With extensive
experiments, model can be shown enjoy the state-of state-of-the-ar art performance in terms of both topic quality and document representations for both regular and short texts.
2
BACKGROUND
In this section, we recap the essential background of neural topic models and optimal transport.
2.1 NEURAL TOPIC MODELS
Most of existing NTMs can be viewed as the extensions of the framework of VAEs where the latent variables can be interpreted as topics. Suppose the document collection to be analysed has V unique words (i.e., vocabulary size). Each document consists of a word count vector denoted as æ NV and a latent distribution over K topics: E K An NTM assumes that z for document is generated from a prior distribution p(z) and æ is generated by the conditional distribution (x(x(2)) that is modelled by decoder é. The model's goal is to infer the topic distribution given the word counts, i.e. to calculate the posterior p(z(ær), which is approximated by the variational distribution qo (2 (z) modelled by an encoder f. Similar to VAEs, the training objective of NTMs is the maximisation of the Evidence Lower BOund (ELBO):
max (x)a)) [log Pa (z|=)] - KL [2)()) II p(z))) 0.6
(1)
The first term above is the expected log-likelihood or reconstruction error. As æ is a count-valued vec- tor, it is usually assumed to be generated from the multinomial distribution: Ps (xx [z)=z)== :- Mu))) where ((x) is a probability vector output from the decoder. Therefore, the expected likeliieddooo is proportional to x" log dxx). The second term is the Kullback-Leible (KL) divergence that regu- larises qo (2) ax) to be close to its prior p(z). To interpret topics with words, d(x) is usually constructed by langle-layer network (Srivastava & Sutton 2017): Φ(2) == softmax W z), where W T V
2

---

Published as a conference paper at ICLR 2021
indicates the weights between topics and words. Different NTMs may vary in the prior and the posterior of 2, for example, the model in MMao et all (2017) applies Gaussian distributions for them and Srivastava & Sutton (217)); Burrkhrrtt & Kramer (2019) show that Dirichlet is a better choice. However, reparameterisation cannot be directly applied to a Dirichlet, so various approximations and sampling schemes have been proposed.
2.2 OPTIMAL TRANSPORT
OT distances have been widely used for the comparison of probabilities. Here we limit our discussion OT for discrete distributions, although applies for continuous distributions well. Specifically, let us consider two probability vectors r E A and E A=s where A D denotes D - simplex. The OT distance between the two probability vectors can be defined as:
dm(r,c) == min PeU(r.c)
(P.M),
(2)
where (.,) denotes the Frobenius dot-product; M E 12 1Doox is the cost matrix/function of the transport; E DRD..DD De is the transport matrix/plan; U(r,c) c) denotes the transport polytope of and which is the polyhedral set of D, Dr matrices: c(r,c)= == [P E 1o0xD.IP1n 1D, c}; and 1n is the D dimensional vector of ones. Intuitively, if we consider two discrete random variables X ~ Categorical(() and Y ~ Categorical c), the transport matrix is a joint probability of (X, F), p(X Y Pij and (r, c) is the set of all the joint probabilities. The above optimal transport distance can be computed by finding the optimal transport matrix P also noteworthy that the Wasserstein distance can be viewed as a specific case of the OT distances.
As directly optimising Eq. 2 can be time-consuming for large-scale problems, regularised optimal transport distance with an entropic constraint is introduced in Cuturi (2013), named the Sinkhorn distance:
( c)
(P.M), (r.,(
min
(3)
where U. (r, == (P ()PP) 2 h(r) + h(c) a), h(-) is the entropy function, and a E [0, o.). To compute the Sinkhorn distance, a Lagrange multiplier is introduced for the entropy constraint minimise Eq. 33, resulting in the Sinkhorn algorithm, widely-used for discrete OT problems.
3 PROPOSED MODEL
Now we introduce the details of our proposed model. Specifically, we present each document as a distribution over V words, 0 E V obtained by normalising x: ă := x e/S where S = >V the length of a document. Also, each document is associated with a distribution over K topics: E A each entry of which indicates the proportion of one topic in this document. Like other NTMs, we leverage an encoder to generate z from ē: softmax (((mmaxx((()) ((())). Notably, is implemented with neural network with dropout layers for adding randomness. As 2 and z are two distributions with different supports for the same document, to learn the encoder, we propose to minimise the following OT distance to push z towards .:
min (w.
Here M E VxX is the cost matrix, where m.k indicates the semantic distance between topic k and word v. Therefore, each column of M captures the importance of the words in the corresponding topic. In addition to the encoder, M is variable that needs to be learned in our model. However, learning the cost function is reported to be conitaivial task (Cuturi & Avis 2014 Sun et al. 2020). To address this problem, we specify the following construction of M:
m 1 cos(ev.gk)
(5)
where cos(..)) E [-1, 1] is the cosine similarity; g R& and e, E R are the embeddings of topic A and word v, respectively.
'To be precise, an OT distance hecomes "distance metric" in mathematics only if the cost function M is induced from distance metrie. We call it "OT distance" assist the readahility of paper.
3

---

Published as a conference paper at ICLR 2021
The embeddings expected to capture the semantic information of the topics and words. Instead of learning the word embeddings, we propose to feed them with pretrained word embeddings such word2vec (Mikolov et al., 2013) and GloVe (Pennington et al. 2014). This not only reduces the parameter space to make the learning of M more stable but also enables us to leverage the rich semantic information in pretrained word embeddings, which is beneficial for short documents. Here the cosine distance instead of others is used for two reasons: it is the most commonly-used distance metric for word embeddings and the cost matrix M is positive thus the similarity metric requires to be upper- bounded. As cosine similarity falls in the range of [-1, 1], we have M E [0, 2|V*
For easy presentation, we denote G E R fg K and E E F as the collection of the embeddings of all topics and words, respectively. Now we can rewrite Eq. 44 as:
min (
(6)
0,G
Although the mechanisms are totally different, both M of our model and W in NTMs (See Sec- tion 2.1 capture the relations between topics and words (M is distance while W is similarity). Here M is the cost function of our OT loss while W is the weights in the decoder of NTMs. Different from other NTMs based on VAEs, our model does not explicitly has decoder to project back to the word space to reconstruct æ, as the OT distance facilitates us to compute the distance between and ø directly. To further understand our model, we can actually project to the space of æ by "virtually" defining a decoder: ((z) == softmax (2 M)z). With the notation of o(z), we show the following theorem to reveal the relationships between other NTMs and ours, whose proof is shown in Section A the appendix.
Theorem 1. When V 2 and M E [0, 2 we have:
dm 2) - -= =o'og 6(2).
With Theorem we have:
Lemma 1. Maximising the expected multinomial log- ligeliiel of NTMs is equivalent to minimising the upper bound of the OT distance in our model.
Frogner et al. (013)) propose to minimise the OT distance between the predicted and true label
distributions for classification tasks. It is reported in the paper that combining the OT loss with the conventional cross- coos-eetropyy loss gives better performance on using either of them. As the expected multinomial log-likelihood easier to learn and can helpful to guide the optimisation the OT distance, empirically inspired by Frogger et al. (2015) and theoretically motivated by Theorem [ we propose the following joint loss for our model that combines the OT distance with the expected log-likelihood:
max (a log ((2) dm dM (ū, z)) G.G
(8)
If we compare the above loss with the ELBO of Eq. (T. it can be observed that similar to the KL divergence of NTMs, OT distance can be viewed regularisation term to the expected log- likelihood (#" log Ø(z) == x log dz))). Compared with other NTMs, our model cases the burden of developing the prior/posterior distributions and the associated sampling schemes. Moreover, with OT's ability to better modelling geometric structures, our model is able to achieve better performance in terms of both document representation and topic quality. In addition, the cost function of the OT distance provides a natural way of incorporating pretrained word embeddings, which boosts our model's 'ss performance on short documents.
Finally, we replace the OT distance with the Sinkhorn distance (Cuturi 2013 which leads to the final loss function:
max (cx log ((z) dM B.G
where softmax (f())): M is parameterised by G; Φ(z) softmax M)z); and ã the word count vector and its normalisation, respectively; e is the hyperparameter that controls the weight of the expected likelihood; is the hyperparameter for the Sinkhorn distance.
To compute the Sinkhorn distance, we leverage the Sinkhorn algorithm (Cutuni 2013). Accordingly, name our model Neural Sinkhorn Topic Model (NSTM), whose training algorithm is shown in
4

---

Published as a conference paper at ICLR 2021
input :Input documents, Pretrained word embeddings E, Topic number K, output: G
Randomly initialise 0 and G;
while Not converged do
Sample batch of B input documents X; Column-wisely normalise to get X
Compute M with G and E by Eq. (5)
Compute softmax (B(X)) Compute the first term of Eq. (
Sinkhorn iterations
1 ones K, B) D2 ones(V, (ees(V,B)// B)/V; H -M/a,
while 1 changes or any other relevant stopping criterion do
L = X @ 1/
v Z O 1/ ( 2);
end
Compute the second term of Eg. Compute the gradients of Eq. in terms of 0, G;
9 dM sum V "H 1)
Update 0, G with the gradients; end
Training algorithm for NSTM. X NVX and Z E R consists of the word count vectors and topic distributions for all the documents, respectively; @ is the element-wise multiplication.
Algorithm 1:
Algorithm I is noteworthy that the Sinkhorn iterations can be implemented with the tensors of Tensorhlow/PyTrcc (Patrini et al., 2020). Therefore, the loss of Eq. (9) is differentiable in terms of 0 and G, which can be optimised jointly in one training iteration. After training the model, we can infer z by conducting a forward passs of the encoder 8 with the input ă. In practice, æ can be normalised by other methods e.g., softmax one TF-IDF the input data of the encoder.
RELATED WORKS
We first consider NTMs (e.g. in Miao et (2016); Srivastava Sutton (017 Krishnan al. (2018): Card et al. (2018): Burkhardt & Kramer (2019 Dieng et al. (2020) reviewed in Section 2.1 as the closest line of related works to ours. For detailed survey of NTMs, we refer to Zhao et al. 22221. Connections and comparisons between our model and NTMs have been discussed in Section 3 addition, word embeddings have been recently widely-used as complementary metadata for topic models, especially for modelling short texts. For Bayesian probabilistic topic models, word embeddings are usually incorporated into the generative process of word counts, such as in Petterson al. (2010); Nguyen al. (2055) Li al, (2016); /hhao et al. (007)). Due to the flexibility of NTMs, word embeddings can be incorporated as part of the encoder input, such as in Card et al.) (2018) or they can be used in the generative process of words such as Dieng
(2020). Our novelty with NSTM is that word embeddings are naturally incorporated in the cost function of the OT distance.
To knowledge, the works that connect topic modelling with OT are still very limited. In Yurochkin et al. (019) authors proposed to compare two documents' similarity with the OT distance between their topic distributions extracted from pretrained LDA, but the aim is not to learn a topic model. Another recent work related to ours is Wasserstein LDA (WLDA) (Nan et al. 2019). which adapts the framework of Wasserstein AutoEncoders (WAEs) (Tolstikhin et al.) [218)) The key difference from ours is that WLDA minimises the Wasserstein distance between the fake data generated with topics and real data, which can viewed as an OT variant to VAE-NTMs. However, our NSTM directly minimises the OT distance between and æ. where there are explicit generative processes from topics to data. Other two related works are Distilled Wasserstein Learning (DWL) (Xu et al. 2018) and Optimal Transport LDA (OTLDA) (Huynh et al.] (220), which adapt the idea of Wasserstein barycentres and Wasserstein Dictionary Learning (Rolet et al. 2016 Schmitz et al. 2018). There fundamental differences of ours from DWL. and OTLDA in terms of the relations between
5

---

Published as a conference paper at ICLR 2021
Table Statistics of the datasets
Number of does
Vocabulary size (V)
Total number of words
Number of labels
20NG
18,846
22,636
2,037.671
20
WS
12,337
10,052
192,483
8
TMN
32,597
13,368
592,973
Reuters
11,367
8,817
836,397
N/A
RCV2
804,414
7,282
60,209,009
N/A
documents, topics, and words. Specifically, in DWL and OTLDA, documents and topics locate in space of words (i.e., both are distributions over words) and can be approximated with the weighted Wasserstein barycentres all the topic-word distributions, where the weights can be interpreted the topic proportions of the document, i.e., z. However, in NSTM, document locates in both the topic space and the word space and topics and words are embedded in the embedding space. These differences lead to different views of topic modelling and different frameworks well. Moreover, DWL mainly focuses on learning word embeddings and representations for International Classification of Diseases (ICD) codes, while NSTM aims be a general method of topic modelling. Finally, DWL and OTLDA are not neural network models while ours is.
5
EXPERIMENTS
We conduct extensive experiments several benchmark text datasets to evaluate the performance of NSTM against the state-of-the-ar neural topic models.
5.1 EXPERIMENTAL SETTINGS
Our experiments are conducted on five widely-used benchmark text datasets, varying in different sizes, including 20 News Groups (20NG Web Snippets (WS) (Phan et al. (2008). Tag My News (TMN) (Vitale et al. 2012 Reuters extracted from the Reuters-21578 datase Reuters Corpus Volume 2 (RCV2) (Lewis al. 2004 The statistics of the datasets in the experiments are shown in [ In particular, WS and TMN are short documents; 20NG, WS, and TMN are associated with document label
Datasets:
We report Topic Coherence (TC) and Topic Diversity (TD) as performance metrics for topic quality. TC measures the semantic coherence in the most significant words (top words) of a topic, given a reference corpus. We apply the widely-used Normalized Pointwise Mutual Information (NPMI) (Aletras & Stevenson 2013 Lau et al. 2014)) computed over the top 10 words of each topic, by the Palmetto package (Röder et al.) 2015 As not all the discovered topics are interpretable (Yang et al. 2015 Zhao et al. 2018), to comprehensively evaluate the topic quality, we choose the topics with the highest NPMI and report the average score over those selected topics. We vary the proportion of the selected topics from 10% to 100%, where 10% indicates the top 10% topics with the highest NPMI are selected and 100% means all the topics are used. TD, as its name implies, measures how diverse the discovered topics are. We define topic diversity to be the percentage of unique words in the top 25 words (Dieng et al. 2020) of the selected topics, similar in TC. TD close to 0 indicates redundant topics; TD close to indicates more varied topics. As doc-topic distributions can be viewed as unsupervised document representations,
Evaluation metrics:
evaluate the quality of such representations, we perform document clustering tasks and report the purity and Normalized Mutual Information (NMI) (Manning al. 20088) on 20NG, WS, and TMN, where the document labels are considered. With the default training/testing splits of the datasets, we train a model on the training documents and infer the topic distributions z on the testing documents. Given z, we
html
We do not consider the labels Reuters and RCV2 as there are multiple labels for one document.
http::/palmettoo.aassse

---

Published as a conference paper at ICLR 2021
(a) 20NG
(b) WS
(c) TMN
(d) Reuters
(e) RCV2
65
36
Figure The first row shows the TC scores for all the datasets and the second row shows the corresponding TD scores. In each subfigure, the horizontal axis indicates the proportion of selected topics according to their NPMIs.
(a) 20NG
(h) WS
(c) TMN
(a) Over batches
VSTM
0.56
2X01
4030
1001
0530
10900
2230
BVM
2200
344
21300
Frod.04
2003
4030
8001
8000
16000
(b) Over seconds
M5TM
611
40-
2430
DY4E
2330
1000
Figure 2: The first row shows the km- Am-Purity scores and the second row shows the corresponding km-NMI scores. In each subfigure, the horizontal axis indicates the number of KMeans clusters.
Figure 3: Training loss.
adopt two strategies to perform the document clustering task: i) Following Nguyen et al. (2015). we use the most significant topic of a testing document as its clustering assignment to compute purity and NMI (denoted by top-Purity and top-NMI); ii) We apply the KMeans algorithm (over all the topics) of the testing documents and report the purity and NMI of the KMeans clusters (denoted by km-Purity and km-NMT). For the first strategy, the number of clusters equals to the number of topics while for the second one, we vary the number of clusters of KMeans in the range of {20, 40, 60, S0, 100). Note that our goal is not achieve the state-of-the-e the--arrt document clustering results but compare document representations of topic models. For all the metrics, higher values indicate better performance.
Baseline methods and their settings: We compare with the sate-of-the--an NTMs, including: LDA with Products of Experts (ProdLDA) (Srivastava & Sutton (2017). which replaces the mixture model in LDA with a product of experts and uses the AVI for training: Dirichlet VAE (DVAE) (Burkhardr & Kramer 2019), which is a neural topie model imposing the Dirichlet prior/posterior on 2. We use variant of DVAE with rejection sampling VI, which is reported to perform the best; Embedding Topic Model (ETM) (Dieng et al. 2020), which a topic model that incorporates word embeddings and is learned by AVI; Wasserstein LDA (WLDA) (Nan al. 2019), which is WAE-based topic model. For all the above baselines, we use their official code with the best reported settings.
Settings for NSTM: NSTM is implemented on TensorFlow. For the encoder 0, to keep simplicity, we use a fully-connected neural network with one hidden layer of 200 units and ReLU as the activation function, followed by dropout layer (rate=0.75) and batch norm layer, same to the
7

---

Published as a conference paper at ICLR 2021
Table 2: top-Purity and top-NMI for document clustering. The best and second scores of each dataset are highlighted in boldface and with an underline, respectively.
top-Purity
top-NMI
20NG
WS
TMN
20NG
WS
TMN
LDA 0.398=0.013
0.446=0.022
0.470=:0.008
0.320++0.010
0.185++00013
0.125
Dr
0.417=0.004 0.203=0.023
0.40510.157
0.321+100004
0.06611.016
0.091
DVAE 0.281=1.0066 0.284=00.005
0,477=0,012
0.187++0.005
0.059410001
0.113:10.004
ETM
0.003 0.205
0.55610.022
0.005110.005
0.003+100033
0.328
WLDA 0.117=00001 0.239=0.003
0,260=0.002
0.060++0.001
0.026+#0.01
0.009++00001
NSTM
0.477=+0001
0.451=0.009
0.637=0.010
0.415++0.012
0.201+00004
0.334+0.004
settings of Burkhardt & Kramer (201)). For the Sinkhorn algorithm, following Cuturi 2013). the maximum number of iterations is 1,000 and the stop tolerance is 0.005 In all the experiments, we fix = 20 and = 0.07. We further vary the two hyperparameters to study our model's sensitivity to them in Figure [.I of the appendix. Finetuning the parameters specifically to a dataset may give better results. The optimisation of NSTM done by Adam (Kingma Ba 2015) with learning rate 0.001 and batch size 200 for maximally 50 iterations. For NSTM and ETM, the 50-dimensional (i.e.. == 50, see Eq. (5) Glo word embeddings (Pennington et al. 2014) pre-trained on Wikipedia" are used. We use the number of topics K = 100 in most cases and set K 500 RCV2 to test our model's s scalability.
5.2 RESULTS
Quantitative results: We run all the models in comparison five times with differentt random seeds and report the mean and standard deviation (as error bars). We show the results of TC and TD in Figure I and top-Purity/NMI in Table and km-Purity/NMI in Figure 2 following remarks about the results: i) Our proposed NSTM outperforms the others significantly in terms of topic coherence while obtaining high topic diversity on all the datasets. Although others may have higher TD than ours in one dataset or two, they usually cannot achieve a high TC at the same time. ii) In terms of document clustering, our model performs the best general with significant gap over other NTMs, except the case where ours is the second for the KMeans clustering on 20NG. This demonstrates that NSTM is not only able to discover interpretable topics with better quality but also learn good document representations for clustering. It also shows that with the OT distance, our model can achieve better balance among the comprehensive metrics of topic modelling. iii) For all the evaluation metrics, model is consistently the best on the short documents including WS and TMN, This demonstrates the effectiveness of our way of incorporating pretrained word embeddings, which shows model's potential on short text topic modelling. Although ETM also uses pretrained word embeddings, its performance is incomparable to ours.
respectively. We have the
Scalability:
NSTM has comparable scalability with other NTMs and is able to scale on large datasets with a large number of topics. To demonstrate the scalability, we run NSTM, DVAF, ProdLDA (as these three are implemented in TensorFlow, while ETM is in PyTorch,, and WLDA is in MXNet) on RCV2 with K 500. The three models run on a Titan RTX GPU with batch size 1,000. Figure 3 shows the training losses, which demonstrate that NSTM has similar learning speed to ProdLDA, better than DVAE. The TC and TD scores of this experiment are shown in Section of the appendix, where it can be observed that with 500 topics, our model shows similar performance advantage over others.
Qualitative analysis: As topics in our model are embedded in the same space as pretrained word embeddings, they share similar geometric properties. Figure shows a qualitative analysis. For the t-SNE (Maaten & Hinton| 2008) visualisation, we select the top 50 topics with the highest NPMI learned by run of NSTM on RCV2 with K = 100 and feed their (50 dimensional) embeddings into the t-SNE method. We also show the top five words and the topic number (I to 50) of each topic.
"The Sinkhorn algorithm usually reaches the stop tolerance in less than 50 iterations in NSTM httpss:/n/l wt..sttd.ffddooooooooooo wtt.s/oor..eedd//rrrr/etts/////a
8

---

Published as a conference paper at ICLR 2021
mer parli fiv
"ansar"
'ceer'
"ippder in sapil
perk now High diy earth
boybeas
sussen aan man pira anlly
nas vill sesianvense fornig
horch
baachbrrry
"pppel
hupple's
Torwer
'cmp'
57 fall ritoo..
lare repert femoart Caranar
interns year percent cie lngal spprel thould sought
"raccsoff"
Tphone"
"pad"
Nartort
'aagrr
Toplc de
Top ielated worch
16 dibmn callin narnet
P. Teiphonggeaie dite
"upple: tage 44t
"lean
bloak"
"eaing
slek tall percing puah)
brutt
"ylure
de avop lew drmwnd the ful oudput arount
It: add tros lood moat sager
rike sunn mote
Hi: yne finsl runch seci wn Pattle onsigh secondfrruth Loc war macheesssiigg tiedie huantartt
'saciard"
Sony
Tax Miviii 'narphi of mond
c6 your disa sesa ose tall 27 mikeed poririic studyrmun
pirsupple
Figure 4: Left: t-SNE visualisation of topic embeddings on RCV2. One red dot represents a topic. The top 5 words and the topic number (1 50) of each topic are also shown. Right: interactions between word and topic embeddings.
observe that although the words of the topics are different, the semantic similarity between the topics captured by the embeddings is highly interpretable. In addition, we take the GloVe embeddings of the polysemantic word "apple" and find the closest 10 related words among the 0.4 million words of the Glo vocabulary according to their cosine similarity. It can be seen that by default "apple" refers to the Apple company more Glo Either adding the embeddings of topic that describes the concept of "food" or subtracting the embeddings of topic 46 that describes the concept of "tech companies" reveals the fruit semantic for the word "apple". More qualitative analysis on topics are provided in Section of the appendix.
6 CONCLUSION
In this paper, we presented a novel neural topic model based on optimal transport, where a document endowed with two representations: the word distribution, and the topic distribution, z. An OT distance is leveraged to compare the semantic distance between the two distributions, whose cost function is defined according to the cosine similarities between topics and words in the embedding space. z is obtained from encoder that takes ã as input and is trained by minimising the OT distance between and ã. With pretrained word embeddings, topic embeddings are learned by the same minimisation of the OT distance in terms of the cost function. Our model has shown appealing properties that are able to overcome several shortcomings of existing neural topic models. extensive experiments have been conducted, showing that our model achieves Mfte-of-theear performance on both discovering quality topics and deriving useful document representations for both regular and short texts.
ACKNOWLEDGMENT
Trung Le was supported by AOARD grant 19-1-4000 Wray Buntine was supported by the Australian Research Council under award DP190100017.
REFERENCES
Nikolaos Aletras and Mark Stevenson. Evaluating topic coherence using distributional semantics. In International Conference on Computational Semantics, pp. 13-22, 2013.
David M Blei, Andrew Y Ng, and Michael JJoddan Latent Dirichlet allocation. JMLR, 3:923-1022, 2003.
David M Blei, Thomas I. Griffiths, and Michael I Jordan. The nested Chinese restaurant process and Bayesian nonparametric inference of topic hierarchies. Journal of the ACM, 57(2).7, 2010.
Sophie Burkhardt and Stefan Kramer. Decoupling sparsity and smoothness in the Dirichlet variational autoencoder topic model. JMLR, 20 2019.

---

Published as a conference paper at ICLR 2021
Dallas Card, Chenhao Tan, and Noah A Smith. Neural models for documents with metadata. In ACL, pp. 1031-20400 2018.
Marco Cuturi. Sinkhorn distances: Lightspeed computation of optimal transport. In NIP,, pp. 2292-2300, 2013.
Marco Cuturi and David Avis. Ground metric learning. JMLR, 15(1):533-564 2014.
Adji B Dieng, Francisco JR Ruiz, and David M Blei. Topic modeling in embedding spaces. TACL, 8: 439-455 2020.
Charlie Frogner, Chiyuan Zhang. Hossein Mohahi, Mauricio Araya, and Tomaso A Poggio. Learning with a Wasserstein loss. In NIPS, pp. 1053-2061, 2015.
Zhe Gan, R. Henao, D. Carlson, and Lawrence Carin. Learning deep sigmoid belief networks with data augmentation. In AISTATS, pp. 268-276, 2015.
Viet Huynh, He Zhao, and Dinh Phung. OTLDA: A geometry-awar optimal transport approach for topic modeling. NeurlPS, 2020.
Diederik Kingma and Jimmy Ba. Adam: method for stochastic optimization. In ICLR. 2015.
Diederik Kingma and Max Welling, Auto-encoding variational Bayes. ICLR, 2013.
Rahul Krishnan, Dawen Liang, and Matthew Hoffman. On the challenges of learning with inference networks on sparse, high-dimensional data. In AISTATS, pp. 441, 2018
John D Lafferty and David M Blei. Correlated topic models. In NIPS, pp. 147-154, 2006.
Jey Han Lau, David Newman, and Timothy Baldwin. Machine reading tea leaves: Automatically evaluating topic coherence and topic model quality. In EACL, pp. 530-539, 2014.
David D Lewis, Yiming Yang, Tony G Rose, and Fan Li. RCVI: A new benchmark collection for text categorization research. JMIR, 5(Apr):361-377, 2004.
Chenliang Li, Haoran Wang, Zhiqian Zhang, Aixin Sun, and Zongyang Ma. Topic modeling for short texts with auxiliary word embeddings. SIGIR, pp. 165-174, 2016.
Laurens van der Maaten and Geoffrey Hinton. Visualizing data using t-SNE, JMLR, 9(Nov): 2579-2605, 2008
Christopher D Manning, Prabhakar Raghavan, Hinrich Schütze, et al. Introduction to Information Retrieval. Cambridge University Press, Cambridge, 2008
Yishu Miao, Lei Yu, and Phil Blunsom. Neural variational inference for text processing. In ICML, pp. 1727-1736, 2016.
Yishu Miao, Edward Grefenstette, and Phil Blunsom. Discovering discrete latent topics with neural variational inference. In ICML, pp. 2017.
Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. Efficient estimation of word representa- tions in vector space. ICLR, 2013.
Feng Nan, Ran Ding, Ramesh Nallapati, and Bing Xiang. Topic modeling with Wasserstein autoen- coders. In ACL, pp. -381, 2019.
Dat Quoc Nguyen, Richard Billingsley, Lan Du, and Mark Johnson. Improving topic models with latent feature word representations. TACI, 3:299-313, 2015.
John Paisley, Chong Wang, David M Blei, and Michael I Jordan. Nested hierarchical Dirichlet processes. TPAMI, 3762):56-270, 2015.
Giorgio Patrini, Rianne van den Berg, Patrick Forre, Marcello Carioni, Samarth Bhargav, Max Welling, Tim Genewein, and Frank Nielsen. Sinkhorn autoencoders. In UAI, pp. 733-743, 2020.
10

---

Published as a conference paper at ICLR 2021
Jeffrey Pennington, Richard Socher, and Christopher Manning. GloVe: Global vectors for word representation. In EMNLP, pp. 1532-1543, 2014.
James Petterson, Wray Buntine, Shravan M Narayanamurthy, Tibério S Caetano, and Alex Smola. Word features for latent Dirichlet allocation. In NIPS. pp. 1921-1929, 2010.
Gabriel Peyré, Mareo Cuturi, et al. Computational optimal transport. Foundations and Trends() in Machine Learning, 11(5-6)335-600) 607, 2019.
Xuan-Hine Hieu Phan, Le-Minh Nguyen, and Susumu Horiguchi. Learning classify short and sparse text web with hidden topics from large-scale data collections. WWW, pp. 91-100, 2008.
Danilo Jimenez Rezende, Shakir Mohamed, and Daan Wierstra. Stochastic backpropagation and approximate inference in deep generative models. In ICML., pp. 1278-1286, 2014.
Michael Röder, Andreas Both, and Alexander Hinneburg. Exploring the space of topic coherence measures. In WSDM, pp. 399-408, 2015.
Antoine Rolet, Marco Cuturi, and Gabriel Peyré. Fast dictionary learning with a smoothed Wasserstein loss. In AISTATS, pp. 630-638, 2016.
Morgan Schmitz, Matthieu Heitz, Nicolas Bonneel, Fred Ngole, David Coeurjolly, Marco Cuturi, Gabriel Peyré, and Jean-Luc Starck. Wasserstein dictionary learning: Optimal transport-based unsupervised nonlinear dictionary learning. SIAM Journal on Imaging Sciences, 11(1):6436678, 2018.
Vivien Seguy, Bharath Bhushan Damodaran, Remi Flamary, Nicolas Courty, Antoine Rolet, and Mathieu Blondel. Large scale optimal transport and mapping estimation. ICLR, 2018.
Akash Srivastava and Charles Sutton. Autoencoding variational inference for topic models. ICLR, 2017.
Haodong Sun, Haomin Zhou, Hongyuan Zha, and Xiaojing Ye. Learning cost functions for optimal transport. arXiv preprint arXiv:2002.09650, 2020.
Ilya Tolstikhin, Olivier Bousquet, Sylvain Gelly, and Bernhard Schoelkopf. Wasserstein auto-encoders. ICLR, 2018.
Daniele Vitale, Paolo Ferragina, and Ugo Scaiella. Classification of short texts by deploying topical annotations. In ECIR, pp. 376-387, 2012.
Hongteng Xu, Wenlin Wang, Wei Liu, and Lawrence Carin. Distilled Wasserstein learning for word embedding and topic modeling. In NeurlPS, pp. 1716-1725,
2018. 2018.
Yi Yang, Doug Downey, and Jordan Boyd-Graber Efficient methods for incorporating knowledge into topic models. In EMNLP, pp. 308-317, 2015.
Mikhail Yurochkin, Sebastian Claici, Edward Chien, Farzaneh Mirzazadeh, and Justin M Solomon. Hierarchical optimal transport for document representation. In NeurIPS, pp. 1599-1609, 2019.
Hao Zhang, Bo Chen, Dandan Guo, and Mingyuan Zhou. WHAI: Weibull hybrid autoencoding inference for deep topic modeling. 2018.
He Zhao, Lan Du and Wray Buntine. A word embeddings informed focused topic model. In ACML, pp. 423-438, 2017.
He Zhao, Lan Du, Wray Buntine, and Mingyuan Zhou. Dirichlet belief networks for topic structure learning. NeurlPS, pp. 7966-7977, 2018
He Zhao, Dinh Phung, Viet Huynh, Yuan Jin, Lan Du, and Wray Buntine. Topic modelling meets deep neural networks: A survey. arXiv preprint 103::2203.00499, 2021.
Mingyuan Zhou, Yulai Cong, and Bo Chen. Augmentable gamma belief networks. JMLR, 17(163): 1-44, 2016.
11

---

Published as a conference paper at ICLR 2021
A PROOF OF THEOREM
Proof. Before showing the proof, we introduce the following notations: We denote k E (1..... K) and E {1, .V) as the indexes; The 5th (s (1,... .) token of the document picks word in the vocabulary, denoted by wa E [I, J the normaliser in the softmax function of d(z) is denoted as á so:
V
Cisn (2
Eis FATEck
=
e
==1
1=1
With these notations, we first have the following equation for the multinomial log- likeliihohoo::
8
log (())
8 am1
8
K
2k
(2
mw k)
log a
8
==1
k=1
8
K
= 2 - log
2k Mhukk
(A.1)
8
Recall that in Eq. (I) of the main paper, the transport matrix P is one of the joint distributions of T and z. We introduce the conditional distribution of z given æ as Q, where g(v, k) indicates the probability of assigning token of word topic k:
Given that P satisfies P E U 2) and = Z, k). Q must satisfy U²(2,z) = {Q RV VV* K 200
EV ( R) 2k With Q, we rewrite the OT distance
V.K
(ā, = min
Σ
Log(
k)mbk
QEU' (..
u 1,k=1
K
s
1 min
Σ>
Σ
(wa, k)m
5 QEU'(E,s)
If we let q(v,k) 2k. meaning that all the tokens of document to the topics according to the document's doc-topic distribution, then Q satisfies U" 2), which leads to:
K
$
dM x) S
(A.2)
5
k=1sm1
Together with Eq. (A.1). the definition of b. and the fact that m.rk I 2. we have:
K Σ
T log (z) = loog 3
5
sm
k=1
< - llg Σ
ΣK- Fumms
1=1
-loog V - 2) (ūe, z)
< dd (0, 2),
(A.3)
where the last equation holds if log V > 2, i.e., V 2 8.
12

---

Published as a conference paper at ICLR 2021
(a) TC
(b) TD
(c) cop-Purity
(d) top-NMI
(e) km-Purity
(f) km-NMI
Figure B.1: Parameter sensitivity of NSTM on 20News. The first and second show the performance with different values of and a respectively. In the first row, fix 20 and vary while in the second row, we fix e = 0.07 and vary a.
(a) TC
(b) TD
0.8
NSTM
ProdLDA DVAE
0.7
0.2
0.6
0.1
0.5
0.4
00.1
0.3
-0.2
0.2
20%
40%
60%
80%
100%
20%
40%
60%
80%
100%
Figure C.I: TC and TD RCV2 with 500 topics.
B PARAMETER SENSITIVITY
In the previous experiments, we fix the values of and a, which control the weight of the multinomial likelihood in Eq (9) and the weight of the entropic regularisation in the Sinkhorn distance, respectively, Here we report the performance of NSTM on 20NG (blue lines) under different settings of the two hyperparameters in Figure B.1 Moreover, we propose two variants of NSTM. The first removes the Sinkhorn distance in the training loss of Eq. (9) (i.e., only the expected log-likelihood term left) and its performance is shown as the red lines. The second variant removes the the expected log-likelihood term in the training loss of Eq. (9) (i.e., only Sinkhorn distance left) and its performance shown as yellow lines.
C TC AND TD ON RCV2 WITH 500 TOPICS
The results are shown in Figure C.I
D AVERAGE SINKHORN DISTANCE WITH VARIED NUMBER OF TOPICS
In Figure D.I show the average Sinkhorn distance with varied number of topics on 20NG. WS. TMN, and Reuters. It can be observed that when K increases, there is a clear trend that dm(z, x) decreases.
13

---

Published as a conference paper at ICLR 2021
(a) 20NG
(b) WS
(c) TMN
(d) Reuters
32
110
0.5
34
44
18
15
10
Figure D.1: Sinkhorn distance with varied K. Vertical axis: the average Sinkhorn distance over all the training documents, i.e., mean dm (z, Φ). Horizontal axis: the number of topics, i.e., K
ann leura bu bara jane
37: sereen featurs digital instance allows
company rogres fneerran
24: miller davis walker mike jonnson
schnelcer waller smithh cos johrooon suilin surner
25: tesm second this play
34: league tonoto tonott
chingo phoen dallas
17: al alimnhommedd abdul ahmust ahmed
2: chang in lee chung chan
44. pupla ras mac knisin sings
user harclurre computer mobi coftuaee
compusee hardware software standard ual
dog wall featured scrics stary star grost
18. company softwore iōm wirdlos computer
5i gas fuet water produce energy 41: space warth time ssample spell
file copy user soflware
42: home moves opened york day
nample
1o: otie vinginta traas camlina michi gan
21: nehmond melboume nottingham sminngarm cologne hamburg berkin sboocholm murach
legs information provide higher increase
Figure F.1: t-SNE visualisation of topic embeddings on 20NG.
E MORE TOPIC EMBEDDING VISUALISATIONS
show the visualisations of 20NG, WS, TMN, and Reuters, respectively. We note that the topic embeddings in general present much better clustering structures of topies in the semantic space. Such topic correlations can only be detected by specialised topic models (e.g..in Lafferty & Blei (2006); Blei et al. (2010); Zhou al. (2016)). Instead, the correlations of topics our model are implicitly captured by the semantic embeddings.
In Figure
14

---

Published as a conference paper at ICLR 2021
telentiion broadestt pbe
31: cillnsmm tn moze her worrer flicter berger
magaziee nowsg opers
1G: graduate indeograii ate seaching mealtyy acidemic Biclogy bienedica scincce sciecces reurcicenn encee
1S fllie liner diseete parmreters wettor
3: mathemoriial compuration comp innirnatinall mentodo engyleeyeen emoiitiill 32: tor minslogo prorundat on atbrodation language apclling
fustball
toumament firs ctampions manch muns 4G bbs game games glants
63 paymene cosn pay tax income
47: market pi expecuaions price rats
solec meloomme be duane carsit
53. web coline internet users user
11: rorthem soutsam regior wastem susters
velidl 27: airuuft mery larse flot
germany
yang la tho ping wong
11: man quatecrth nasa discoarry g
51: rediaal rursing medions pischutric dinii dhorden diabetes disase sancer
18: interfaus spen sounce for faricionally into foces weds based 26 vizzopiccessssssmooo processo tooll pos procesiors
decsss lucrua discass vins flu
Figure E.2: t-SNE visualisation of topic embeddings on WS.
ootholic relilous orthooos
calebratnng
sense brrnp tramendous passion expereence queston reason explain impact
12: lince thoulder eltow ankie thigh
28: cancer patients diabetas cheess complaztins
47: aed wild bins unmels ekeshant
12: decose flu vins discses spidomic
5: mar here mets yariower dodgers
campbell
10: ger mans cacch oustrian austria german
I: mesicn pers ends chile colombaa
moriners variewss
& carlos me uss gonaler radrguez
14: mother daughter wrte
23: chelsea strkkr Iverpool defender in midhaiderr
31: griduste school school college teaching student
7% bus cartrata trsi ponsentgrr whhidle oboard ship flihtt plane landing toyota missan honda metors
21: masoun vignn carcira clso maryland
10: phoems dalla seatte shrladsichia denver
XV: nurtin sigeets rango uutanene dustmum
pay paying
X: thelland malayca indoneds bangladenh indis
11: chan yang wos pong lu
oohn chtgrup secorilec
Figure E.3: t-SNE visualisation of topic embeddings on TMN.
32: iser quaner petsciioeespeott parto prefit quattresmmizg comings roverue rat comolatiee
garmany sunch stodhoree flantfurt marke fizz werver ont-scheeli hein:
codes antiei jusu on pn di paeein a nes
gowermmnt samee uny lased chicago bused taranto basdd invermmet houdi company
22: comganyssto dury fim
13: wel lngginn triecane noveace mebooure parts phoneic tat delbdee ceccnnse denver
LP: ejnodaary petrolmum compony venture corporatice eveer cilr se retai retsllr
10: canoinn oho
corn suger bearamei juice
14: ship drips bost uacsel
42: eszarta price demand
xi prine prine rose diopped frll
22: shane shanes expecta
45: charges ullaged charged change
38: proveiim requirment regilatio latora moce provilen
calinc profit quarter year mose
empodo mar fadttahar shand
30: sampple Inatunce in acdition
18 plane besed company busieess agreed
45: subiturially weater encervaieed monginallyy cro
aroughen booe
4: elction parts deroccas ut canditate wate 26: recting mooneet not tocsdoy wodistsdp masnwti trade china thursday tunsdes
Figure E.4: t-SNE visualisation of topic emheddings on Reuters.
15


Reviews:
[{"review_id": "hm5yZhRcIRL", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes a new variant of neural topic model leveraging optimal\ntransport in order to incorporate information from pre-trained word vectors.\nSpecifically, the authors replaced the KL-divergence reguralization term with\nan optimal transport between topic distribution and empirical word distribution\nin each text.\nExperimental evaluation yields generally better topic coherence and high\nprecision of K-means clustering on the induced topic distributions.\n\nBasically this paper is interesting, but still leaves some questions.\n\n- First of all, evaluations are based only on high NPMI topics (section 5.2).\nTherefore, it is trivial that the induced topic-word distribution of these\ntopics are good. In my experience, original vanilla LDA yields mostly\ninterpretable topic-word distributions; however, even for the high-NPMI topics,\neach topic in Figure 4 is somewhat noisy, and unshown topics might be worse.\nTherefore, I would like to know the comparison between the proposed method and\nthe original LDA too.\n\n- The paper first says that \"good document representation and coherent/diverse\ntopics\" is difficult. Then why not including perplexity evaluation in the \nexperiments? K-means results are only auxiliary evaluation of the former, thus\nthe reader would like to know whether the proposed model could yield better \nperplexity on documents or not.\n\n- It seems that the choice of dimensionality and the number of topics seems\ntoo low and arbitrary. Why only the 50-dimensional word vectors are used?\nIs there any difference over the competitors when that dimensionality is\nchanged?\nAlso, I could not know why only the results with K=100 is shown in main text.\nAppendix E shows that K=500 consistently yields better results; why are they\nnot included?\n\n- Finally, I cannot understand what kind of optimization w.r.t M is conducted\nin this paper. To make the paper as self-contained as possible, I strongly\nrecommend to show what kind of optimization is actually done.\n\nThe proposed OT regularization seem to work better, but I cannot see why the\nbaseline of word-vector based topic models like (Dieng+ 2020) is inferior.\nIs there any intuition or explanation over these trivial competitors?\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "bdTINidyy5g", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary: The paper proposes a neural topic model which log-likelihood is regularized by Sinkhorn distance, instead of following Variational AutoEncoder (VAE) approach. The proposed model is hence cannot be interpreted as a probabilistic generative model. Still, with respect to metrics such as Topic Coherence and Topic Diversity which don't require probabilistic interpretation of topic model, the proposed model performs very well across five standard benchmark datasets for topic modeling.\n\nQuality and Clarity: At the very high level, the use of Optimal Transport distance for topic modeling is sensible. Neither word counts nor topic distributions have clear ordering of indices, and Optimal Transport distance is usually well-suited for such histogram-like features. However, authors do not provide much discussion on why Optimal Transport distance is particularly well-suited for topic modeling. I can see that compared to the expected log-likelihood, the Optimal Transport distance $d_M(\\tilde{x},z)$ is a more optimistic estimate, as the topic distribution for each word could be individually assigned as long as marginal distribution of topics is equal to $z$. Maybe this is related to some sort of variational approximation which disentangles the usual constraint that topic distribution of all words in the same document are exactly the same. Still, I had hard time understanding why this is favorable for topic modeling, or finding better interpretations of Optimal Transport distance on topic modeling. At the least, some qualitative analysis of how the hyperparameter $\\varepsilon$, which controls the tradeoff between expected likelihood and Sinkhorn distance, affects the characteristics of topic models would be nice. Appendix D. shows how evaluation metrics change as $\\varepsilon$ changes, but not much interpretation of results is provided.\n\nIt's good to see how a well-established technique can be applied to topic modeling and show encouraging empirical results, but the lack of justifications on the modeling approach would make it difficult for the further development and adoption of the method, especially because the proposed method is not a probabilistic generative model anymore. I would encourage others to do a deeper qualitative analysis on the effect of Sinkhorn distance as a regularizer.\n\nOriginality and Significance: The establishment of the connection between Optimal Transport and Topic Modeling would potentially lead to active follow-up research, as both are well-established areas of research.\n\n* Pros:\n  * good empirical results\n  * connects two well-established topics of research\n* Cons:\n  * Justifications of the modeling approach is lacking", "extra_scores": {}, "extra_text": {}}, {"review_id": "R-ToaUWH0Mx", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper proposes a neural topic model derived from the perspective of optimal transport (OT). Topic embeddings are learned as part of the training process and is used to construct the cost matrix of the transport.  The cost function based on the OT distance is further improved by combining with the cross-entropy loss and by using the Sinkhorn distance to replace the OT distance.\n\nThe paper is well written.  The proposed method is first explained from the perspective from the optimal transport and then developed by considering the cross-entropy loss and the Sinkhorn distance.  It also explains how the model incorporates the word embedding and introduce the topic embedding to simply the cost matrix M of the transport.  The proposed method is sound.\n\nThe experiments included recent neural topic models for comparison.  The chosen test data sets include also some with short text. In general, the proposed method has been show to perform better than other methods in terms of topic coherence and topic diversity.  The experiment section also some quality analysis on the topic discovered by the proposed model.  The experimental results are convincing.\n\nThe paper explains quite clearly the relationship between the proposed method and related methods. In particular, the paper seems to have adequately credited the sources of ideas during the development of the proposed model.  The novelty of the proposed method appears to be the use of optimal transport for developing neural topic model and the construction of the cost matrix M based on the cosine similarity between word embedding and topic embedding.  The novelty is sufficient.\n\nAlthough the clarity of the proposed method is good, the rationale for using OT distances for comparing probabilities may deserve more explanation.  The paper gives an intuition for the transport matrix P in section 2.2.  It may be better if it can also give more explanation on the role of M and what the meaning of the distance between two probability vector is.  References may also be given to other works that use OT distances for comparing probabilities.\n\nThe experiment includes recent neural topic models but does not include other recent topic models not based on neural networks. It would be interesting to see how the proposed method performs compared to those models.\n\nI cannot find any indication that source code will be released.  It is suggested to do so for reproducibility and for the use of practitioners.\n\nMinor comment:\n\n- Add the data set name to the caption of Figure 4.", "extra_scores": {}, "extra_text": {}}, {"review_id": "6KWNmcCvExx", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper builds off existing neural topic model work that's based on variational autoencoders and specifically introduces a novel loss function based on optimal transport (more specifically, the Sinkhorn distance). The paper demonstrates impressive experimental results, where the proposed method Neural Sinkhorn Topic Model (NSTM) outperforms (or is competitive with) a number of SOTA baselines, with a dramatic performance gain in the setting where the text documents are very short.\n\nOverall, I enjoyed reading this paper and found the proposed method well-described albeit somewhat ad hoc, although the authors do justify certain aspects of the method (e.g., Lemma 1). The \"weight\" of this paper seems to really be carried by the experimental results though, which I found to be quite compelling.\n\nStrengths:\n- well-explained method that provides a neat usage of optimal transport in topic models\n- Theorem 1/Lemma 1 provide a nice theoretical comparison of the proposed method vs other neural topic models\n- strong experimental performance especially in the setting of short documents, which is very relevant in practice\n\nWeaknesses:\n- the proposed method comes off as a bit ad hoc\n- while the authors do reference Card et al's paper, I think a comparison with Card et al's Scholar framework would be helpful to get a sense of how the l1 regularization in Scholar/SAGE handles short text documents compared to NSTM\n- there are small typos/grammar glitches here and there - please proofread carefully\n\nIn the conclusion, the authors point out correlated topic models and dynamic topic models as variants worth exploring next. Supervised versions of topic models I think also are worth exploring.", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The reviewers unanimously agreed that this is an interesting paper that belongs at ICLR. The use of optimal transport in neural topic models is novel and the paper is well-written.

A common theme among the reviewers was that they would like to see more intuition and justification. I suggest you bear this in mind while editing the final version of the paper. I also believe that R3 brings up valid points about evaluating perplexity -- I don't think the lack of perplexity results are a reason to reject the paper, but I believe they can be calculated here (see eg the reference R3 provided) and they would give a clearer view of the model's performance.

Decision: Accept (Spotlight)