FASTopic: Pretrained Transformer is a Fast, Adaptive, Stable, and Transferable Topic Model
Authors: Xiaobao Wu
Thong Thanh Nguyen
Delvin Ce Zhang
William Yang Wang
Anh Tuan Luu
Venue: NeurIPS 2024 poster (2024)
Keywords: topic model
topic modeling
document embedding
Primary area: natural_language_processing

Abstract: Topic models have been evolving rapidly over the years, from conventional to recent neural models. However, existing topic models generally struggle with either effectiveness, efficiency, or stability, highly impeding their practical applications. In this paper, we propose FASTopic, a fast, adaptive, stable, and transferable topic model. FASTopic follows a new paradigm: Dual Semantic-relation Reconstruction (DSR). Instead of previous conventional, VAE-based, or clustering-based methods, DSR directly models the semantic relations among document embeddings from a pretrained Transformer and learnable topic and word embeddings. By reconstructing through these semantic relations, DSR discovers latent topics. This brings about a neat and efficient topic modeling framework. We further propose a novel Embedding Transport Plan (ETP) method. Rather than early straightforward approaches, ETP explicitly regularizes the semantic relations as optimal transport plans. This addresses the relation bias issue and thus leads to effective topic modeling. Extensive experiments on benchmark datasets demonstrate that our FASTopic shows superior effectiveness, efficiency, adaptivity, stability, and transferability, compared to state-of-the-art baselines across various scenarios.

FASTopic: Pretrained Transformer is a Fast, Adaptive, Stable, and Transferable Topic Model
Xiaobao Wul Thong Nguyen² Delvin Ce Zhang?
Anh Tuan Luu National University of Singapore University of California, Santa Barbara
William Yang Wang'
'Nanyang Technological University
The Pennsylvania State University
xiaobao0028e ntu. edu.sg sg e09981470u. nus. edu delvin. wwwangggmaill.oooommo com anhtuan. luuentu. eee.ntu.edu.... edu.
william@cs ucsb. edu
Abstract
Topic models have been evolving rapidly over the years, from conventional to re- cent neural models. However, existing topic models generally struggle with either effectiveness, efficiency, or stability, highly impeding their practical applications. In this paper, propose FASTopic, fast, adaptive, stable, and transferable topic model. FASTopic follows a new paradigm: Dual Semantic-relation Reconstruction (DSR). Instead of previous conventional, VAE-based, or clustering-basee meth- ods, DSR directly models the semantic relations among document embeddings from pretrained Transformer and learnable topic and word embeddings. By reconstructing through these semantic relations, DSR discovers latent topics. This brings about neat and efficient topic modeling framework. We further propose a novel Embedding Transport Plan (ETP) method. Rather than early straightforward approaches, ETP explicitly regularizes the semantic relations optimal transport plans. This addresses the relation bias issue and thus leads to effective topic model- ing. Extensive experiments on benchmark datasets demonstrate that our FASTopic shows superior effectiveness, efficiency, adaptivity, stability, and transferability, compared to state-of-the-an baselines across various scenarios.
PASTOpiC
LDA-Mallet
250
NMF
BERTopic
Running Speed Rank
BERTopic
CombinedTM
CM
Running time
200
FASTopIC
150
Miner
200
1 Mal-Mallet
ECRTM
50
GiNepic
ProGiNn
10k
size
Overall Performance Rank
(a)
(b)
Figure 1: (a): Running speed rank and overall performance rank on the experiments with 6 benchmark datasets, including topic quality, doc-topic distribution quality, downstream tasks, and transferability. (b): Running time under the WoS dataset with varying sizes. See complete results in Figure
"Corresponding authors.
'We release our code at httpps//ggthuu.ooooooo//o/////tp org/ /ttp//eccc//ftsppcccc////
package at htttp://pppt
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

(c)
Decoder
Pretrained Transformer
(a)
E2
doc ambwitinng
(b)
Ww
top enieedding
word ambedding
samando
(ransport plan
samantic (ransport
TF-IDF
kop words lopic
Figure 2: Illustration of topic modeling paradigms. (a): VAB-based topie modeling with encoder and decoder [91 65 73.. (b): Clustering- -based topic modeling by clustering document embeddings [2 24). (c): Dual Semantic-relation Reconstruction (DSR), modeling doc-topic distributions as the semantic relations between document and topic embeddings (A), and modeling topic-word distributions the semantic relations between topic (A) and word embeddings (+). Here we model
these relations as the transport plans to alleviate the relation bias issue.
Introduction
Due to the unsupervised fashion and interpretability, topic models have derived broad spectrum of applications 12 14], such as content recommendation [39] 79. generation [17
88] and trend
analysis [18 36 Early conventional topic models follow probabilistic graphical models [T0 g
58]. But they rely on laborious model-specifee derivations and fall short on large-scale data (75) Due this. recent neural topic models have attracted more
non-negative matrix factorization
35
attention 99 751, including based
42
43 61
and clustering- based [59] 89 24
However, existing neural topic models lack either efficiency, effectiveness, stability. First, VAE- based topic models, while effective, are limited by their low efficiency. They follow the VAE framework [3 and often incorporate extra modules like graph neural networks [87] D external knowledge (66 80 resulting in complicated modeling structures. Owing to this, they suffer from intensive time complexity, e.g., consuming hours to process dataset of 10k documents 65 Second, clustering- topic models [89 224] excel in efficiency as they require no training, but they sacrifice effectiveness. They tend to yield repetitive topics other than desired diverse ones or infer inaccurate topic distributions of documents [73 What is worse, these neural topic models suffer from low performance stability. They are extremely sensitive hyperparameters, especially when applied to various scenarios concerning data domains, vocabulary sizes, and document length [27]. In consequence, these challenges hinder the applications of topic modeling practice.
To tackle these challenges, we in this paper propose a Fast, Adaptive, Stable, and Transferable topic model (FASTopii). Different from existing conventional, VAE-based, clustering -based approaches, we introduce a new paradigm for topic modeling: Dual Semantic-relation Reconstruction (DSR) as illustrated in Figure 2 Instead of complicated neural networks, DSR only considers three parameters: document embeddings from a pretrained Transformer, and topic and word embeddings. DSR models the dual semantic relations between (1) document and topic embeddings. and (2) topic and word embeddings, and interprets them as distributions for topic modeling. By reconstruction with these relations, DSR discovers latent topics in neat and efficient framework, avoiding the complicated structures in prior studies. To model these relations, we further propose the novel Embedding Transport Plan (ETP). Rather than simple parameterized softmax 21). ETP regularizes these relations as the optimal transport plans between document, topic, and word embeddings. This mitigates the relation bias issue and produces distinct topies and accurate topic distributions, enabling effective topic modeling. Following the DSR paradigm with ETP, FASTopic provides a solid solution to the challenges of current topic models. As reported in Figure D FASTopic shows both superior efficiency and effectiveness compared to state-of-the-a baselines. Additionally FASTopic shows high

---

transferability, robust adaptivity and stability across various scenarios, delivering better performance without hyperparameter tuning. We conclude the main contributions of this paper as follows:
We propose a novel topic model with a new dual semantic-relation reconstruction paradigm that models semantic relations among document, topic, and word embeddings, bringing about neat and efficient topic modeling framework.
We further propose novel embedding transport plan method that regularizes the semantic relations as optimal transport plans, which avoids the relation bias issue and leads to effective topic modeling.
We conduct extensive experiments and demonstrate that our model shows high effectiveness, efficiency, adaptivity, stability, and transferability compared to state-of-the-ar baselines.
2 Related Work
Conventional Topic Models These models have two types. The first type is probabilistic topic models [25] 7 9 [], e.g. LDA [10], using probabilistic graphical models with topics latent
variables and inferred by Gibbs sampling [62] or Variational Inference [I]] The second type uses
non-negative matrix factorization 22 588 These models have been extended to several scenarios
like short texts 82 67], multilingual 45 and dynamic topic modeling 9 64]. But they require
model-specific derivations for parameter inference and cannot well handle large-scale datasets.
VAE-based Neural Topic Models These models follow the Variational AutoEncoder [VAF, 31
55
framework and directly use gradient backpropagation optimize parameters [42
43
61
99
83
[7 (8 76, 777 (74). Although some work [91, 65, 86] like ECRTM 773 also uses optimal transport, we highlight that our method differs from them in that: (i) While they still follow the traditional complicated VAE framework, our FASTopic leverages the neat and efficient dual cemantic-relation reconstruction paradigm; (ii) While they use optimal transport only as alternative distance measures or regularization for VAE, our FASTopic leverages the novel embedding transport plan to model semantic relations. These differences not only bring about faster running speed, but also lead to higher topic modeling performance.
85 884 81
Clustering- Neural Topic Models They cluster pretrained word embeddings via clustering algorithms like KMeans to yield topics [59 2 89), but mostly cannot infer topic distributions of documents. BERTopic (24) clusters the document embeddings and approximates the topic distributions by comparing documents to each document cluster. Different from simple clustering in these studies, we focus on explicitly modeling the complex relations among the embeddings of documents, topics, and words, which enhances topic modeling performance.
Some recent studies leverage large language models and describe topics as conceptual descrip- tions [53], rather than the word distributions in LDA [10]. They can reach higher interpretability, (i) They require more resources. They need input each document as prompts to LLMs. This is time-consuming and computationally intensive, especially when handling large-scale datasets. (ii) They cannot produce precise distributions for topics and documents, which limits their applications in downstream tasks.
but we emphasize their two limitations:
3 Methodology: FASTopic
In this section, we first recall the problem setting of topic modeling. Then we propose the new paradigm Dual Semantic-relation Reconstruction (DSR) and the novel Embedding Transport Plan (ETP) method. Finally we introduce our new FASTopic.
3.1 Problem Setting and Notations
Consider a collection
x....,,(MM,) (N) with N documents and vocabulary size V. Topic modeling
targets discover K latent topies from the collection. Following LDA [1O], Topic#k defined as a distribution over all words, i.e., topic-word distribution, denoted as B E We have B= (B....., )ER V as the toric-word distribution matrix of all topics. Topic modeling also infers the topic distributions of document (what topics a document contains), i.e., doc-topic distribution.
VxK
We denote the doc-topic distribution of (4) (()) =AA. with Ax probability simplex.
3

---

0.12
0.12
0.08
Reletton weggt
0.0
Refntor
3.00
3.0
3.02
0.02
0.00
0.00
(a) Parameterized Softmax
(b) ETP
(c) Parameterized Softmax
(d) ETP
Figure (a. b): Relation weights of topics to documents. (c, d): t-SNE visualization 63 of
document ), and topic (A) embeddings under 50 topics (K=50), While most topic embeddings gather together in Parameterized Softmax (a,c) as it causes biased relations, ETP (b,d) separates all topic embeddings with regularized relations, avoiding the bias issue.
3.2 Dual Semantic-relation Reconstruction
In this section, we propose a new, neat, and efficient paradigm for topic modeling, Dual Semantic- relation Reconstruction (DSR). Figure illustrates the differences between DSR and previous VAE-based and clustering-based methods.
Parameterizing Documents, Topics, and Words topics, and words embeddings. Specifically, we embed documents into H-dimensional se- mantic space via a pretrained Transformer Sdee, e-g., BERT [16] or Sentence-BERT (54). Let D-(d,..-- d JER#*N
At the beginning, we parameterize documents,
H N denote all document embeddings, where d.= refers to the embed- ding of i-th document. Then randomly project all topies and words into the same semantic space as K topic embeddings .x EE) f and V word embeddings W-(w
W Hx
Notably we do use pretrained word embeddings like word2vec [44] GloVe (51) because they may not belong to the same semantic space as document embeddings, which hinders correctly measuring their distance.
Reconstruction through Dual Semantic Relations Then we model the dual semantic relations between the embeddings of (1) documents and topics, and (2) topics and words. We interpret these relations doc-topic distributions and topic-word distributions respectively. To be specific, we model
g( the probability of Topictke given i-th document as the semantic relation between embeddings d and t.. Similarly, model 3jk-- the probability of j-th word given Topic#k the semantic relation between embeddings t and w,. We detail how to model them later in Sec. 3.3 We learn these relations through reconstruction. As shown in Figure where transport the semanties from (()) each topic through (( and then from each topic to each word through B. Hence we formulate the objective for DSR as the reconstruction error:
reconstruct document x(!
(3)
LDSR Σ (1) (oo (3g()(())
where we transform ( into the Bag-of- Words following previous studies
By minimiz- ing this objective, we expect to push each topic embedding close to the embeddings of its semantically related documents and words; therefore we can learn informative semantic relations, i.e., meaningful topic-word distributions (3 for latent topics and doc-topic distributions (()) for documents.
The above DSR presents a neat and efficient topic modeling paradigm. Previous VAE-based methods incorporate complicated modeling structures with diverse objectives 80 [. Different from them, DSR solely includes the objective Eq. (T) that only involves the embeddings of documents, topics, and words. This sufficiently simplifies the topie modeling procedure and hence facilitates efficiency. Moreover, while prior clustering-basse methods (24) depend on indirect approximations, DSR explicitly models topic-word distributions and doc-topic distributions, resulting higher effectiveness (See experiments in Sec.

---

3.3 Embedding Transport Plan
In this section, we analyze how to model the semantic relations for topic modeling, and then propose a new solution Embedding Transport Plan (ETP).
How to Model Semantic Relations? We emphasize this a non-trivial problem. A common
answer the widely-used parameterized softmax function [28] 337 21]:
Σ Sk- exp(- - 12/TT) ByB = exp | w/1l2/T) E
exp(-Ild, tx112/T)
exp(- E
wy 112/T)
where we measure the relation between embeddings as their Fuclidean distance with hyperparameter Unfortunately, this straightforward way is ineffective, because it incurs the relation bias issue: most relations are minor as quantitatively illustrated in Figure Sa In consequence, most topic embeddings fail cover informative and distinct semantics but gather together in the space as shown in Figures anland7 7a| This issue leads to repetitive topics and less accurate doc-topic distributions (See ablation studies in Sec. 4.7p. Someone may guess this issue is because of the large topic number. We note that the relation bias issue still happens even under small number of topics (See experimental supports in Table 9). Owing these, we need alternatives to model the semantic relations.
Transport Plan from Documents to Topics Motivated by the above analysis, we propose the new Embedding Transport Plan (ETP) to address the relation bias issue with effective regularization.
To regularize relations, we model them as the transport plan of a specifically defined optimal transport problem. detail, we define two discrete measures "y and fi over document and topic embeddings: 7/1 > N dd, and - 1= Skor,, where 8- denotes the Dirac unit mass on z. We set the weight of each document embedding 1/N and the weight of each topic embedding Sk. where s= .........x sx) is a weight vector summing to 1. This later produces normalized doc-topic distributions. With these two, we formulate their entropic regularized optimal transport problem as
K
IN N
arg min TEX
Tuk Tat (log mx-1), s.t. TlK= and π 1I=S. (3)
The first term is the original optimal transport problem, and the second term is the entropic regular- ization with hyperparameter E1 to make this problem tractable [13 (22.. This equation aims to find a transport plan that minimizes the total cost of transporting the weights of document embeddings to topic embeddings under the two conditions (15), where 1 denotes a K K-ddimensional column vector of ones. Here Ti refers to the transport weight from d; tg. The transport cost between them is measured as Fuclidean distance - - 1x with C() as the transport cost matrix.
We can alleviate the relation bias issue with transport plan π as the semantic relations. Eq. 33) constraints π by two conditions. We set s=softmax (sg). where So is a learnable variable uniformly initialized as R L. The uniform initialization and softmax function prevent excessively biased s (28). Therefore as illustrated Figures 3b and [nd this avoids biased π as constrained by s, which mitigates the relation bias issue (See experiment results in Sec. 4.7). Besides, this approach flexibly captures the varying weight each topic within the document collection. since some topics may appear more frequently in the collection while others less, which aligns with the reality (See more interpretations Appendix[E)
Transport Plan from Topies to Words We further employ the transport plan between topic and word embeddings. We define two discrete measures over topic and word embeddings: 7/2 EK & dr and Ey=Σ/= uydw,. Here we specify the weight of each topic embedding as 1/K and the weight of each word embedding as uj where u is a weight vector and its sum is 1. This to produce normalized sopic-word distributions later. Following Eq. (3), we write the entropic regularized optimal transport problem between the two measures as
K
(22) Φi tedh, Φky 1), s.t. x
1k and & T Ix=u. (4)
arg min Lor P2; E2) *ERXXV
5

---

Table Topic quality results of Cv (topic coherence) and TD (topic diversity). The best is in bold. $ denotes the gain of FASTopic is statistically significant at 0.05 level.
20NG
NYT
WoS
NeurIPS
ACL
Wikitext-103
Model
Gv
TD
Cv
TD
Cy
TD
Gv
TD
Cv
TD
Cy
TD
LDA Mallell
R0,371
10.763
10,362
70.747
f0.352
F0,866
70.366
$0.573
F0.357
70,570
$0.407
10,607
NMF
10.372
10.473
10,378
10.403
10.388
10.430
10.375
10.292
10.362
10.300
10.411
10.429
BERTopic
10.406
10.654
10.413
10.679
10:437
T0.649
10.379
10.472
T0.399
10.462
10.432
10,608
ComhinedTT
10.375
10.523
10.375
10.617
10.399
10.563
10.400
$0.463
F0.359
10.424
1p.384
10.652
GINopic
R0.369
$0.862
F0.381
70.701
f0.362
F0,576
90.392
90.452
F0,370
90.537
$0.385
$0.699
ProGBN
10.378
10.587
10.381
10.371
10.363
10.521
10.406
10.617
HyperMiner
R0,371
10.613
10,375
10.573
10.356
T0.645
10,373
10.751
T0,360
70.754
t0.333
10.245
BCRTM
0.431
10.964
0,433
0.988
10.438
1.000
10.415
$0.98
10.409
10.930
10.425
10.958
FASTopic
0.426
0.983
0.437
0.999
0.457
1,000
0.422
0.998
0.420
0.998
0.439
0.992
Here Φ is the transport plan between topic embeddings and word embeddings, and Φxy denotes the weight to be transported from topic embedding t2 to word embedding wj. We measure the transport cost as Euclidean distance ( Up is learnable variable uniformly initialized as + lv. We model the semantic relations between topics and words with Φ to mitigate the relation bias issue.
w21 Similar to the above, we set u u=softmax(1)) where
Objective for ETP
With the above transport plans as semantic relations, we write ( the doc-topic
distribution of document (1) as
NT,, where π sinkhorn (7),((7))))))))
(5)
We employ Sinkhorn's 's algorithm 660 15 to compute the approximated solution x² of the optimal transport problem in Eq. (3). As proved in early studies [57, 22 23), π becomes a differentiable variable parameterized by transport cost matrix C(1), which thus enables gradient backpropagation. See algorithm details in Appendix C Here we rescale π; by N to output normalized ( (the sum of each row in π² is 1/N as previously constrained). In the same way, we model the topic-word distribution matrix 3 as
B KΦ", where o² (ornhhor
(6)
We rescale Ø² by K to produce normalized | (the sum of each row in o² is 1/K as previously constrained). We formulate the objective for ETP by minimizing the total transport cost weighted by these approximated transport plans as
N
K
LETP
(2) kj
okj
k=1j=1
This objective refines the embeddings by the regularized semantic relations, i.e., the approximated transport plans under constraints.
3.4 Objective for FASTopic
Combining Eq. ① and Eq. (7). formulate the overall objective for FASTopic
min T.W.s.u
LETP + LDSR-
(8)
In short, LETP refines topic and word embeddings with the regularized semantic relations; LDSR Iearns these relations by reconstruction. To reduce the number of hyperparameters, we set the weights of these two objectives as equal by default. See the training algorithm of FASTopic in Appendix C
This objective is extremely simple compared to previous work with complex encoders and decoders following VAE 42 61 75). It only optimizes four parameters: topic and word embeddings T, W and their weights and u. We freeze document embeddings D, because they have been pretrained already, and we may encounter over-fitting problems if we fine-tune them on a relatively small dataset. Due to this simple objective, our FASTopic enjoys super fast training. Previous models like ECRTM

---

Table 2: Text clustering results of Purity and NMI. The best is in bold.
Table 3:
Running time (in seconds) on different datasets. The best bold.
20NG
NYT
Wo8
Model
20NG
NYT
WoS
NeurIPS
ACL
Wikitext-103
Model
Purity
Puricy
Purity
LDA-Mallet
58.6
70.0
50.2
702.6
974.0
2083,0
LDA-Mallet
f0.514
-0.451
10.627
10.353
Tax438
10.329
74.8
268.6
939.0
NMF
40.180
-0.168
10.485
10.215
10.540
40.228
BERTopic
34.2
35.2
35.0
55.6
66.8
114.2
BERTopic
F0.451
-0.423
10.517
F0,319
f0.656
+0.340
CombinedTM
53.4
31.8
45.2
67.2
93.0
237.4
C
T0.459
10.435
10.500
T0.325
10.655
T0.345
GINopic
417.8
334.2
309.2
905.8
664.8
1878.2
GINopic
T0.385
10.278
T0.584
10.280
Ta.32
10.301
ProGBN
4..387
-0.370
10,571
10.293
10.598
10.324
ProGBN
337.0
765.8
831.0
675.0
864.2
2180.2
HyperMiner
10.405
T0.579
T0.315
10.549
HyperMiner
233.0
233.6
250,3
184.4
265.8
813.6
FCRTM
0,550
0.534
10.615
10.357
10,650
10.549
ECRTM
FAS Topic
0.577
0.525
0.662
0.369
0.672
0.365
FASTopic
ZING
NYT
WeS
Medel
Acc
F1
EDA-MMille
10.472
10,599
10.535
NMJ
10,528
T0.317
10.625
10.414
10.536
10.418
BERTopgic
10,299
10.285
+0.550
10,361
10.514
-0.415
ComtinedTM
10.448
FQ.426
50.650
f0,378
F0.562
50.434
GiNopic
to.057
Tau122
10.314
10,099
To,330
10.110
PaCtINN
10.178
40.504
10.250
50.373
Hypesfiner
10.276
10.262
10.50
10,361
To.512
10.367
2.90
2000
EORTM
10.481
10.460
10. 10.008
10.504
10.551
10.581
Lh
ECTheic
2.2 7W/Ree
PASTopic
0.593
0.754
Figure 4: (Left)) Text classification results of Accuracy (Acc) and F1. (Right): Transferability results. We use topic models trained on Wikitext- 103 to infer the doc-topic distributions of other datasets. The best is in bold.
1 also solve the optimal transport problem, but their objectives involve complicated encoders and decoders inherent from VAE, which slows them down.
Moreover, FASTopic needs much fewer hyperparameters. It mainly has hyperparameters for Sinkhorn's s algorithm E and E2 in Eq. (3) and { VAE-based models, like CombinedTM (5). ECRTM 773,, and GINopic (1. require hyperparameters set their encoders, decoders (dimensions, number of layers, and dropout), and prior distributions (Gaussian or Dirichlet). BERTopic needs hyperparameters to set its clustering and dimension reduction modules, like the number of neighbors and components of UMAP; the min cluster size, min samples, metrics of HDBSCAN.
3.5 Inferring Doe- Popic distributions for New Documents
Finally we discuss how to infer doc-topic distributions for new documents. Considering new document x' and its document embedding f'= we may directly follow the learning process in Sec. 3.3 and infer its doc-topic distribution 0' by computing the transport plan between d' and learned topic embeddings T. Unfortunately, this way is unworkable. It transports the weight one document to all topics; hence for any x'. the transport plan invariably becomes the learned topic weights s. Such trivial results are certainly unreasonable. To this end, we compute as
exp(-I d' 1'113/7)
8%
Pk EF=1 Pk
where Ph
Σ 1app--ltx - d.113/T)
with 7 as a temperature hyperparameter. Here we model the relation between d' and t as the Euclidean distance and regularize it by the total relations between t, and all training documents to approximate the learned topic weight in Sec. 3.3 Then we compute 8/ by normalizing over all topics. Since topic and word embeddings have been refined after training (Scc.3.4) we can infer accurate doc-topic distributions for new documents in this way. See empirical results in Sec. 4.2 and 4.3
4 Experiment
In this section we conduct comprehensive experiments to demonstrate that our FASTopic is fast, adaptive, stable, and transferable.
7

---

Table 4: Topic quality results of Cv (topic coherence) and TD (topic diversity) under different topic numbers (K). The best is in bold.
K=75
K=100
K=125
K=150
K=175
K=200
Model
Cv
TD
Cv
TD
Cy
TD
Cv
TD
Cv
TD
Cy
TD
LDA-Malles
40,360
$0.847
+0.361
40.828
40.363
$0.806
$0.364
$0.786
$0.369
-0.781
$0.371
+0.755
NMF
R0,393
10:415
10,390
10.382
10.392
10.357
$0.390
f0.332
10,391
10.307
10.393
10,298
BERTopic
10.444
10.638
10.450
10.607
0.458
10,566
0.457
10.529
0.457
10.543
0.455
10.541
ComhinedTM
40.388
40.482
+0.385
40.473
10.388
F0.455
10.387
$0.428
$0.390
+0.425
$p.389
40.443
GINopic
50.369
$0.485
$0,372
40.445
10.369
F0.432
10.372
$0.445
$0.377
70.446
$0.375
10.447
ProGBN
40.380
F0.732
$0.380
90.673
$0.380
F0.599
f0.379
$0.556
F0,374
90.472
$p.375
$0.444
HyperMiner
70,356
50,586
+0,350
10.595
10.351
10.568
10.352
$0.547
f0.349
70,518
$0.355
+0,490
BCRTM
0.482
10.974
0.477
10.951
0.461
10.958
0.458
10.957
10,448
0,953
10.437
0.941
FASTople
0.465
0.998
0,464
0.993
0.460
0,984
0.459
0.972
0.458
0.959
0.456
0.949
Table 5: Document clustering results of Purity and NMI under different topic numbers (K). The best is in bold.
K=75
K=100
K=125
K=150
K=175
K=200
Model
Purity
Purity
NMI
Purity
Purity
NMI
Purity
NMI
Purity
NMI
Malleelee
R0.661
10.333
F0,651
40.322
$0.660
F0.325
$0.675
$p.332
F0.684
90.332
$0.676
10.327
NMF
10,570
10.245
10.582
10.256
10.593
T0,263
10.606
t0.272
10.604
10,270
10.613
10,273
BERTopic
10.672
10.333
0.696
10.340
0.704
10.338
0.713
10.338
10.720
10.343
10.722
10.347
CombinedTM
Rg.672
40.343
40,683
40.347
$0.695
+0.348
+0.705
$p.352
+0.710
$0.355
40.709
+0.350
GINopic
10.644
10.308
10.654
10.311
10.662
10.319
10.672
tp.318
$0.682
10.323
tp.687
10.327
ProGBN
=0.624
10.313
+0.641
40.315
10.627
F0.303
$0.637
$0.302
#0.664
F0.320
$0.663
#0.316
HyperMiner
10.641
10,339
10,655
10.342
10.661
10,331
10.664
t0.330
F0.654
70,320
f0.666
10,324
BORTM
10.648
10.334
10.668
10.343
10.680
10,343
10.697
10.356
10,700
10,355
10,701
0.361
FASTopic
0.364
0.703
0.368
0.710
0.365
0.721
0.368
0.738
0.372
4.1 Experiment Setup
Datasets We adopt six benchmark datasets for experiments: (i) 20NG [20Newsgroup,
33 is one
of the most commonly-used datasets, covering news articles with 20 labels. (ii) NYT includes news articles from the New York Times with 12 categories. (iii) WoS [Web Of Science, 32 contains published papers from the Web of Science website with 7 categories. (iv) NeurlPS is a dataset with papers published at the NeurlPS conference from 1987 to 2017. (v) ACL 66 contains research
articles from the ACL anthology from 1970 to 2015. (vi) Wikitext-103 41 includes Wikipedia articles. See more dataset details in Appendix B
Evaluation Metrics Although topic modeling evaluation is still open problem (26). we follow 91 73 and evaluate the topic quality and doc-topic distribution quality. For topic quality, we consider: (i) Topic Coherence measures the coherence between top words of discovered topics. We employ the widely-used coherence metric Cv. which has been shown to
mainstream studies [9
outperform earlier NPMI, UCI, and UMass 46
34
56) . We use widely-used large Wikipedia
article collection as the external reference corpus to compute Cv. (ii) Topic Diversity means the differences between discovered topics. We measure this with the Topic Diversity (TD) metric [19].
which calculates the proportion of unique words in the topics. In terms of doc-topic distribution quality, we conduct document clustering, evaluated by Purity and NMI (38 following Zhao et 91]
We do not evaluate the perplexity since our method does not follow the VAE framework 442 43
and
the perplexity is incomparable across topic models as evidenced by early studies 90 75].
Baseline Models We consider the following basclines in three paradigms. For conventional topic models, we adopt (i) LDA-Mallet [40]. a prominent method competitive to some neural models (6): (ii) NMF, using non-negative matrix factorization. For clustering- based topic models, we have (iii) BERTopic (24. clustering document embeddings and discovering topics by TF-IDF For VAE-based neural topic models, we include (iv) CombinedTMM [ combining contextual features and BoW as inputs; (v) GINopic 0. following CombinedT M but using graph isomorphism networks; (vi) HyperMiner [80), using hyperbolic embeddings to model topics; (vii) ProGBN 20 progressively generating documents of different levels with graph decoders; (viii) ECRTM 73
8

---

state-of-the-ar method by regularizing embeddings with optimal transport. We fine-tune the hyperparameters of these baselines under different datasets and topic numbers.
4.2 Effectiveness: Topic Quality and Doc-Topic Distribution Quality
We demonstrate the superior effectiveness of FASTopic compared to state-of-the-ar baselines. Table] presents the topic quality results of topic coherence (Cv) and topic diversity (TD). We that our FASTopic commonly surpasses all baselines with the highest performance across all datasets. Moreover, reports the doc-topic distribution quality results concerning the Purity and NMI of document clustering. We observe that FASTopic reaches top performance as well. These results manifest that FASTopic produces high-quality topies and doc-topic distributions, showing better effectiveness. This also verifies the capability of our new DSR paradigm. See Appendix H for the examples of discovered topics.
4.3 Effectiveness: Text Classification as Downstream Task
We consider text classification as a downstream task evaluate topic models in an extrinsic manner. Following Wu et al. [73], we train SVM classifiers with the inferred doc-topic distributions as document features and then predict the class of each testing document. We measure this performance by Accuracy (Acc) and F1. Figure
4 reports that our FASTopic consistently outperforms baselines. We note that the improvements of FASTopic are statistically significant at 0.01 level. These results demonstrate that FASTopic can benefit more downstream classification tasks.
4.4 Efficiency: Running Speed
We show the exceptionally fast rumning speed of FASTopic. Table reports the running time of each model on each dataset. The running time indicates the duration from the completion of data loading to the finish of training. We that our FASTopic consistently emerges the fastest by a large margin, statistically significant at 0.01 level. FASTopic completes running within minute, while the longest takes 30 minutes. We notice that -DA-Mallet has increasing running time on the datasets with longer documents. For instance, it escalates from 50 seconds on 20NG to 2000 seconds on Wikitext- 103. In contrast, FASTopic maintains its rapid performance regardless of document length. Figure lb also evidences the fast speed of FASTopic in terms of varying dataset sizes. This is because FASTpoic adopts our neat and efficient DSR paradigm, which relieves from the complicated modeling structures in previous studies. See more running time analysis in Appendix G
4.5 Transferability
We verify the high transferability of FASTopic. In detail, we train a topic model Wikitext- 103, a general dataset with diverse topics, and then use it to infer the doc-topic distributions of documents in other datasets 20NG, NYT, and WoS. Following the previous setting, use these doc-topic distributions as features to train SVM classifiers for text classification. This measures the transferability of topic model from data domain to another. 4ssows that the transferability of FASTopic significantly outperforms basclines. The reason lies in that previous methods often rely on the Bag-of-Words Words [80 20 73 Differently, FASTopic leverages richer representations, the pretrained document embeddings, and learns the doc-topic distributions through the effective ETP method, bringing about higher transferability.
4.6 Adaptivity and Stability
We demonstrate the adaptivity and stability of FASTopic across various scenarios using the WoS dataset. First, Tables and 3jmmmmmaaariiz the performance under different topic numbers (K from 75 to 200). We observe that FA generally remains top performance across these variations. Second, Tables 13 and 14 report the results under varying dataset sizes (N from 15k to 40k). These results show that FAS ToSTopic mostly reaches the best results. As aforementioned in Figure Tb FASTopic also has the fastest running speed. Third, we experiment with different vocabulary sizes (V from 20k to 50k) Tables [5] and T6 Similarly, our FASTopic exhibits stable and high performance. We note that FASTopic uses the same hyperparameters in all these experiments (See Appendix D). The above

---

Table Ablation study. w/o ETP means using parameterized softmax (Eq. relations. See also Table Soleeforr for results on other datasets.
to model semantic
20NG
NYT
WoS
Model
Cv
TD
Purity
NMI
Cv
Purity
NMI
Cy
Purity
NMI
w/o ETP
10,368
10.391
10.401
10.452
10.363
10.338
10,553
10.294
10.395
10,522
10.653
10.350
FASTopic
0.426
0.983
0.577
0.525
0.437
0.999
0.662
0.369
0.457
1.000
0.672
0.365
results together highlight that our FASTopic can smoothly adapt to various scenarios with stable performance. This is vital advantage of our model for practical applications.
4.7 Ablation Study
validate the necessity our Embedding Transport Plan (ETP) method with ablation studies. Table 6 shows that using parameterized softmax rather than ETP (w/o ETP) to model semantic relations incurs degraded performance, concerning both topic and doc-topic distribution quality (See also the results other datasets in Table 8). For instance, the Cy and TD decrease from 0.426, 0.983 to 0.368, 0.391; the Purity and NMI decrease from 0.577, 0.525 to 0.401, 0.452, indicating low-quality repetitive topics and less accurate doc-topic distributions. We observe similar results even with only 10 topics (K=10) in Table g This is because our ETP properly regularizes semantic relations, which addresses the relation bias issue. These results manifest the necessity of our ETP to reach effective topic modeling.
5
Model Usage
We have released our FASTopic Python package at PyPI Users easily install FASTopic through pip. 5 code example to use FASTopic on a dataset. After preprocessing the given dataset, it discovers top words and infers doc-topic distributions. With these simple APIs, users can smoothly handle their data for their various purposes. See our GitHub for more tutorials and documentation of FASTopic.
pip from fastopic impert FASTopic
fastopic
from topmost. preproccsinn inport Proprocesring
# Preprocess the dataset.
docs "doc "doc 2", preprocessing -
model FASTopic toun__opics=50, preprocessing)
topic_top_rords doc_topic_dist model. fit trt_ffrrnooonndddc
Figure 5: A code example of using FASTopic. Install FASTopic via pip and use its APIs to handle dataset.
Conclusion
In this paper, we propose FASTopic, fast, adaptive, stable, and transferable topic model. Rather than traditional VAE-based or clustering-based approaches, FASTopic employs the new dual semantic- relation reconstruction paradigm to model latent topics with semantic relations and uses the new transport plan relation method to tackle the relation bias issue. Comprehensive experiments demon- strate the significantly superior performance of FASTopic in terms of effectiveness, efficiency, adaptivity, stability, and transferability. These advantages manifest the strong capability of FASTopic practice, which benefits a wide range of worl-world applications.
10

---

Acknowledgements
This research/project is supported by the National Research Foundation, Singapore under its Al Singapore Programme (AISG Award No: AISG2-TC--202-0005,
References
[1] Suman Adhya and Debarshi Kumar Sanyal. Ginopic: Topic modeling with graph isomorphism network. arXiv preprint arXiv:2404.02115, 2024.
[2] Dimo Angelov. Top22ee: Distributed representations of topics.
arXiv preprint
arX1v:2008.09470, 2020.
[3] Sanjeev Arora, Rong Ge, and Ankur Moitra. Learning topic models-going going beyond svd. In 2012 IEEE 53rd annual symposium foundations of computer science, pages 1-10. IEEE, 2012.
[4] Sanjeev Arora, Rong Ge, Yonatan Halpem, David Mimno, Ankur Moitra, David Sontag, Yichen Wu, and Michael Zhu. A practical algorithm for topic modeling with provable guarantees, In International conference on machine learning, pages 280-288. PMLR, 2013.
[5] Federico Bianchi, Silvia Terragni, and Dirk Hovy. Pre-training is a hot topic: Contextualized document embeddings improve topic coherence. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the I Ith International Joint Conference on Natural Language Processing (Volume 2: Short Papers), pages 756-766, 2021. URL
atttp:///axiiy www./org//oo.. wwt/pot////pp///ot..t/pppppttttp 2000./2000..0000 03974
[6] Steven Bird, Robert Dale, Bonnie Dorr, Bryan R Gibson, Mark Thomas Joseph, Min-Yer Kan, Dongwon Lee, Brett Powley, Dragomir R Radev, Yee Fan Tan, al. The acl anthology reference corpus: A reference dataset for bibliographic research in computational linguistics. In LREC, 2008. URL httt/t////////// http://wwwwwwaddiaaaaaa.a academia. a. /tt///////////////99333//////
[7] David Blei and John Lafferty. Correlated topic models. Advances in neural information pro- cessing systems, 18:147, 2006. URL https:///set htttppp://wwww.. cs. cmu edu/ http////////a//aaeeerr.r.
www/pub/ ctm pdf
[8] David M Blei. Probabilistic topic models. Communications of the ACM, 55(4):77-84 2012. URL httth::t////// academia edu/ httpa///w..aaaaaaaaaae//////aaaa
[9] David M Blei and John D Lafferty. Dynamic topic models. In Proceedings of the 23rd international conference on Machine learning, pages 113-120, 2006. URL https:tt::///1 acm
erg/doi/abas aatooi/abs//ii 10. 1 143844. 1143859
[10] David M Blei, Andrew Y Ng, and Michael Jordan. Latent dirichlet allocation. Journal of Machine Learning Research, 3(Jan)993-1022 2003. URL. htttps:///wstt jmml jmlr org/ hinerr//voolumm///eelll////////
[11] David M Blei, Alp Kucukelbir, and Jon McAuliffe. Variational inference: review for statisticians. Journal of the American statistical Association, 112(518):859 877, 2017. URL
https://arxiv> http//df/6001.0000 00670
[12] Jordan L Boyd-Graber, Yuening Hu, David Mimno, et al. Applications of topic models. volume 11. now Publishers Incorporated, 2017. URL https: /www www.ublisherrs...omoooo.. com/
article/Detaal
INR-030 030
[13] Guillermo Canas and Lorenzo Rosasco. Learning probability measures with respect to optimal transport metrics. F. Pereira, C.J. Burges, L. Bottou, and K.Q. Wein- berger, editors, Advances in Neural Information Processing Systems, volume 25. Curran Associates, Inc., 2012. URL htttps://proceedinnitt neur httprrrppsccpppppe/2///////////e .pdf
[14] Rob Churchill and Lisa Singh. The evolution of topic modeling. ACM Computing Surveys, 54 1105):1-35, 2022. URL https::: //dl acm. www.aaa....ooooo.o.oo.o wwt/////////////i///////o 1145/ /505//00070

---

[15] Marco Cuturi. Sinkhorn distances: Lightspeed computation of optimal transport. In C.J. Burges, L. Bottou, M. Welling, Z. Ghahramani, and K.Q. Weinberger, ed- itors, Advances in Neural Information Processing Systems, volume 26. Curran As- sociates, Inc., 2013. URL. hattps://proceeddngg newrips.eripp.... catsseec/paaae /aper/2013/fille a 42200377baa2/11557226666666686
httpe//s...pppppe////////11ee1 .pdf
[16] Jacob Devlin, Ming-Was Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv: 1180040005 2018. httptttss//axii..cooo wttp:///...rra///////0 04805.
[17] Adji B. Dieng, Chong Wang. Jianfeng Gao, and John Paisley. TopicRNN: A recurrent neural network with long-range semantic dependency. International Conference on Learning Representations, 2017. URL httpss: /eb.r//reeeeetooomooooooooeoee.
[18] Adji B Dieng, Francisco JR Ruiz, and David M Blei. The dynamic embedded topic model. arXiv preprint arXiv:190..05545, 2019. URL hatts:://axiiii
[19] Adji B Dieng, Francisco JR Ruiz, and David M Blei. Topic modeling in embedding spaces. Transactions of the Association for Computational Lingnistics, 8:439-453, 2020. URL https 00325/96463
//direct.m mit. www/tacl/lc/eee//ae//e htt.aa//aaaaacc///o/////taa 10. 1162/
[20] Zhibin Duan, Xinyang Liu, Yudi Su, Yishi Xu, Bo Chen, and Mingyuan Zhou. Bayesian progressive deep topic model with knowledge informed textual data coarsening process. In International Conference on Machine Learning, pages 8731-8744 PMLR, 2023. URL https
/ //proceedings.. mlr. (rtes//v222/uann/// hww//ddunnnnnnncccec pdf
[21] Maziar Moradi Fard, Thibaut Thonet, and Eric Gaussier. Deep k-means: Jointly clustering with k-means and learning representations. Pattern Recognition Letters, 138:185-192, 2020.
[22] Aude Genevay, Gabriel Peyré, and Marco Cuturi. Learning generative models with sinkhorn divergences. In International Conference on Artificial Intelligence and Statistics, pages 1608- 1617. PMLR, 2018.
[23] Aude Genevay, Gabriel Dulac-Arnold, and Jean-Philippe Vert. Differentiable deep clustering with cluster size constraints. CoRR, abs/1910.09036, 2019. /ttp:ttt://arxi..oo /bs/
1910. 09036
[24] Maarten Grootendorst. Bertopic: Neural topic modeling with a class-based tf-idf procedure. arXiv preprint 20X1::2003.05794 2022. URL.ttss:://axxx orgg awwpabbs....////////////// 05794
[25] Thomas Hofmann. Probabilistic latent semantic analysis. In Proceedings of the Fifteenth con- ference on Uncertainty in artificial intelligence, pages 289-296. Morgan Kaufmann Publishers
Inc., 1999.
[26] Alexander Hoyle, Pranav Goel, Andrew Hian-Cheong, Denis Peskov, Jordan Lee Boyd-Graber, and Philip Resnik. Is automated topic model evaluation broken? the incoherence of coherence. In A. Beygelzimer, Y. Dauphin, P. Liang, and J. Wortman Vaughan, editors, Advances in Neural Information Processing Systems, 2021. httppttss// TjdHCnPqoo
[27] Alexander Miserlis Hoyle, Rupak Sarkar, Pranav Goel, and Philip Resnik. Are neural topic models broken? In Findings of the Association for Computational Linguistics: EMNLP 2022, pages 5321-5344, 2022. URL https: ht/p/axxvvvvrr.. wttp /2100 16162
[28] Eric Jang, Shixiang Gu, and Ben Poole. Categorical reparamettriization with gumbel-softmax. 01144. pdf
arXiv preprini arXiv: 1.01 144, 2016. /tR//tttps//arriv.orrr/////
[29] Hannah Kim, Jaegul Choo, Jingu Kim, Chandan K Reddy, and Haesun Park. Simultaneous discovery of common and discriminative topics via joint nonnegative matrix factorization. In Proceedings of the 21th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pages 567-576, 2015. https:::ts: //dl. acm. wwwo/oii////oooooooooo 100 1..00.1455 1145/ 2783258 2783338
12

---

[30] Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In Yoshua Bengio and Yann LeCun, editors, 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings, 2015. URL httpp///rxiv. org/ abs/ 1412. 6980
[31] Diederik P Kingma and Max Welling, Auto-encoding variational bayes. In The International Conference on Learning Representations (ICLR), 2014. URL https: //axxiv org/ abs/
1312. 6114
[32] Kamran Kowsari, Donald E Brown, Mojtaba Heidarysafa, Kiana Jafari Meimandi,, Matthew Gerber, and Laura E Barnes. Hdltex: Hierarchical deep learning for text classification. In Machine Learning and Applications (ICMLA), 2017 16th IEEE International Conference on. IEEE, 2017.
[33] Ken Lang- Newsweeder: Learning filter netnews. In Proceedings of the Twelfth International Conference on Machine Learning, pages 331-339, 1995.
[34] Jey Han Lau, David Newman, and Timothy Baldwin. Machine reading tea leaves: Automatically evaluating topic coherence and topic model quality. In Proceedings of the 14th Conference of the European Chapter of the Association for Computational Linguistics, pages 530-539, 2014.
URL URL.tttps https:tt////. http::://aaaataataaa
org/ E14- 1056. pdf
[35] Daniel Lee and H Sebastian Seung. Algorithms for non-negative matrix fac- torization. Advances in neural information processing systems. 13, 2000. https: https::/pproceedingg wwwpiiinnnnnnuuuu.e.e.eeeer www.rips..cppppp..e wwwps//ppppee///////// F50111225444000001337777e6600000 haw.aa/eeeaaeaaaaaaooeoettttatad Abstract.
URL
[36] Yue Li, Pratheeksha Nair, Zhi Wen, Imane Chafi, Anya Okhmatovskaia, Guido Powell, Yannan Shen, and David Buckeridge. Global surveillance of covid- 19 by mining news media using a multi-source dynamic embedded topic model. In Proceedings of the 1 11th ACM International Conference on Bioinformatics, Computational Biology and Health Informatics, pages 1-14, 2020. URL https: / //ii -
mehi-wenn, wen www./eeeee//eeaaeeeeeeeeet htaff/covi--aee acmbcb. pdff
[37] Maddison, Mnih, and Teh. The conerete distribution: continuous relaxation of discrete random variables. In Proceedings of the international conference on learning Representations. International Conference on Learning Representations, 2017. URL htaps://arxiv... /df11111.0777
[38] Christopher D Manning, Prabhakar Raghavan, and Hinrich Schütze. Introduction to Information Retrieval. Cambridge University Press, New York, NY. USA, 2008. ISBN 0521865719, 9780521865715. URL hattps://wwt cis. uni www.uuuncceeenccmmmmmm htt////s//eaccch//////tttttt
19web. pdf
[39] Julian McAuley and Jure Leskovec. Hidden factors and hidden topics: understanding rating dimensions with review text. In Proceedings of the 7th ACM conference on Recommender systems, pages 165-172, 2013.
[40] Andrew Kachites McCallum. Mallet: A machine learning for languagetoolkit. UMass, 2002.
URL http::////m: wmass. edu
[41] Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. Pointer sentinel mixture models. arXiv preprint arXiv:1609 07843, 2016. URL https:/ /ww. cmu, edu/ af cs/
usr/lafferty wttpt///wwww/// /ww////w///////// pub/ ctm. ct
[42] Yishu Miao, Lei Yu, and Phil Blunsom. Neural variational inference for text processing.
In International conference on machine learning, pages 1727-1736, 2016. URL https
//proceedings.nt mlr arapoeedmnn.......rrrrs//////eee ppess/v48/miaole html
[43] Yishu Miao, Edward Grefenstette, and Phil Blunsom. Discovering discrete latent topics with neural variational inference. In Proceedings of the 34th International Conference on Machine Learning-Volen Volume 70, pages 2410-2419. JMLR, org, 2017. http:httt::/prroeedingg. mlr press/v70/miaet 17a. html html
13

---

[44] Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S Corrado, and Jeff Dean. Distributed repre- sentations of words and phrases and their compositionality. Advances neural information processing systems, 26, 2013.
[45] David Mimno, Hanna Wallach, Jason Naradowsky, David A Smith, and Andrew McCallum. Polylingual topic models. In Proceedings the 2009 conference on empirical methods in natural language processing, pages
Singapore, August 2009. Association for Computational org/ 1092
Linguistics. URL
[46] David Newman, Jey Han Lau, Karl Grieser, and Timothy Baldwin. Automatic evaluation of topic coherence. In Human Language Technologies: The 2010 Annual Conference of the North American Chapter of the Association for Computational Linguisties, pages 100- 108. Association for Computational Linguistics, 2010. ISBN 1932432655. URL https
//aclanthology... wwwpchhooooy.ooggg/e/oo wrg//100111111211
[47] Thong Thanh Nguyen, Xiaobao Wu, Xinshuai Dong, Cong-Duy T Nguyen, See-Kiong Ng. and Anh Tuan Luu. Topic modeling as multi-objecctvv optimization with setwise contrastive learning. The Twelfth International Conference on Learning Representations, 2024. URL
https:://ppnrrevee..o waet//orrumid-didooa.......
[48] OpenAl. Chatgpt: Optimizing language models for dialogue. OpenAl, 2022. //chat.oeeaa.. com/
URL https
[49] Fengjun Pan, Xiaobao Wu, Zongrui Li, and Anh Tuan Luu. Are Ilms good zero-shot fallacy classifiers? arXiv preprint arXiv:2410.1500 2024.
[50] Liangming Pan, Xiaobao Wu, Xinyuan Lu, Anh Tuan Luu, William Yang Wang, Min-Yer Kan, and Preslav Nakov. Fact-checking complex claims with program-guided reasoning. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 6981-7004, 2023.
[51] Jeffrey Pennington, Richard Socher, and Christopher D Manning. Glove: Global vectors for word representation. In Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP), pages 1532-1543, 2014. UtL./attss//aclaathholoo org/ D14-1162. pdf
[52] Gabriel Peyré, Marco Cuturi, et al. Computational optimal transport: With applications to data science. Foundations and Trendy@ in Machine Learning, 11(5-6):35-6677 2019.
[53] Chau Minh Pham, Alexander Hoyle, Simeng Sun, and Mohit Iyyer. Topicgpt: A prompt- based topic modeling framework. arXiv preprint 1.X142231..0144,, 2024. URL https
abs 2tt.. 01449
[54] Nils Reimers and Iryna Gurevych. Sentence-bert: Sentence embeddings using siamese bert- networks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Lan- guage Processing and the 9th International Joint Conference Natural Language Processing (EMNLP-IJCNLP), pages 3982-3992 2019. URL paatps://arxiz perxiv.orggi wwt.:/tgg///ooeeeee. /908 10084
[55] Danilo Jimenez Rezende. Shakir Mohamed, and Daan Wierstra. Stochastic backpropagation and approximate inference in deep generative models. In Proceedings ofthe 31th International Conference on Machine Learning, 2014. URL httts::/ htttp:://proceedings. mlr. www.s/veesss rezende14.n 4tmll
[56] Michael Röder, Andreas Both, and Alexander Hinneburg. Exploring the space of topic coherence measures. In Proceedings of the eighth ACM international conference on Web search and data mining, pages 399-408. ACM, 2015, https::a:... ://p:://dd... acm. org/ /wt///////////////// 111111....1111 1145/ 2684822. 2685324
[57] Tim Salimans, Han Zhang, Alec Radford, and Dimitris Metaxas. Improving gans using optimal transport. arXiv preprint arXiv:1803.05573, 2018.
14

---

[58] Tian Shi, Kyeongpil Kang, Jaegul Choo, and Chandan K Reddy. Short-text topic modeling via non-negative matrix factorization enriched with local word-context correlations. In Proceedings of the 2018 World Wide Web Conference, pages 1 1105-1114. International World Wide Web Conferences Steering Committee, 2018. URL https: //d1 acm. httgg/do/abb////eee 100bs/100.0 1145/
3178876.3186009
[59] Suzanna Sia, Ayush Dalmia, and Sabrina J. Mielke. Tired of topic models? clusters of pretrained word embeddings make for fast and good topics too! In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 1728- 1736, Online, November 2020. Association for Computational Linguistics. doi: 10. 10.18653/v1/ 3020.emmlp-1i 135. URL https:::/ http:::://aclantoloogyy org 2020. emnlp-main. 135
[60] Richard Sinkhorn. A relationship between arbitrary positive matrices and doubly stochastic matrices. The annals of mathematical statistics, 35(2):876-879, 1964.
[61] Akash Srivastava and Charles Sutton. Autoencoding variational inference for topic models. In 5th International Conference Learning Representations, ICLR 2017, Toulon, France, April 24-26, 2017, Conference Track Proceedings. OpenReview.net, 2017. URL https:// openreview wwtt/oorummidd-yyoottta
[62] Mark Steyvers and Tom Griffiths. Probabilistic topic models. Handbook of latent semantic analysis, 427(4))424 440, 2007. URL https: wtwpap///a....aammma....mo www.dmiaa.daeeeeee.am.. wwt.aa/ddddddddddddeeeoo.a /tt.///owlo.//1111/////11 s 5bmp@q4bog444ggiip...td
[63] Laurens van der Maaten and Geoffrey Hinton. Visualizing data using t-SNE. Journal of ma- chine learning research, 9NNo))25579 2008. URL URLhttps: httpp::/wwas jmlr . org/papers)
pdf
[64] Chong Wang. David Blei, and David Heckerman. Continuous time dynamic topic models. In Proceedings of the Twenty-Fourth Conference on Uncertainty in Artificial Intelligence, pages 579-586 2008. URL httts:///asxiiii www.aawww.....v.ooooeeeeeee ort/ /ppf////20 11266.32688688
[65] Dongsheng Wang, Dandan Guo, He Zhao, Huangjie Zhang, Korawat Tanwisuth, Bo Chen, and Mingyuan Zhou Representing mixtures of word embeddings with mixtures of topic embeddings. In International Conference on Learning Representations, 2022. URL https //openreview wet.:eooouonddd////uutu YYMTTT
[66] Dongsheng Wang. Xu, Miaoge Li, Zhibin Duan, Chaojie Wang, Bo Chen, Mingyuan Zhou, et al. Knowledge-aware bayesian deep topic model. Advances in Neural Information Processing Systems, 35:14331 2022. URL https: http:///pppcceeennssoo.ors.e.po.
pdf
[67] Xiaobao Wu and Chunping Li. Short Text Topic Modeling with Flexible Word Patterns. In International Joint Conference Neural Networks, 2019. httpsst/t//eeeeeeplltt ieee
[68] Xiaobao Wu, Chunping Li, Yan Zhu, and Yishu Miao. Learning Multilingual Topies with Neural Variational Inference. In International Conference on Natural Language Processing and Chinese Computing, 2020. URL /tttts:///pnntee springer. com/ chapteer/10.1007/ 96]
[69] Xiaobao Wu, Chunping Li, Yan Zhu, and Yishu Miao. Short text topic modeling with topic distribution quantization and negative sampling decoder. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 1772-1782, Online, November 2020. URL https: htapp:::///lllhooogog...
202g2020..0. emnlp mwin. 138. 138.pdf
[70] Xiaobao WU, Chunping Li, and Yishu Miao. Discovering topics in causal intervention. In Findings of the Association for Computational Linguisticy: ACL-IJCNLP 2021, pages 175-185 Online, August 2021. Association for Computational Linguistics. doi: 10. 18653/v 1/2221 findings-acl.115 URL URI.hatpss//cclaatttlooo.ooo org/220 findings-acl
corpora with
15
15

---

[71] Xiaobao Wu, Anh Tuan Luu, and Xinshuai Dong. Mitigating data sparsity for short text topic modeling by topic-semantic contrastive learning. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 2748-2760, Abu Dhabi, United Arab Emirates, December 2022. Association for Computational Linguistics. URL https
//aclanthology- org/ 2022. emnlp main. 176
[72] Xiaobao Wu, Xinshuai Dong, Thong Nguyen, Chaoqun Liu, Liang-Ming Pan, and Anh Tuan Luu. Infoctm: mutual information maximization perspective of cross-lingual topic modeling. In Proceedings of the AAAI Conference Artificial Intelligence, volume 37, pages 13763- 13771, 2023. URL httph:ttt///aatee http//xxivv.rr.geee abs/ 2abbs/2334..0 23304.33544
[73] Xiaobao Wu, Xinshuai Dong, Thong Nguyen, and Anh Tuan Luu. Effective neural topic modeling with embedding clustering regularization. In International Conference Machine Learning PMLR, 2023. URL /ttpl::t///////t /wwxiv..ivvvv.///////a.//aaaa. org/pdf htpppd/3000 /200 04217
[74] Xiaobao Wu, Xinshuai Dong, Liangming Pan, Thong Nguyen, and Anh Tuan Luu. Modeling dynamic topics in chain- fashion by evolution-tracking contrastive learning and unassociated word exclusion. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar, editors, Findings of the Association for Computational Linguistics ACL 2024, pages 3088-3105, Bangkok, Thailand and virtual meeting, August 2024. Association for Computational Linguistics. URL nttps://aclanthologg
hwggg/204..20 findings--cl..n.. 183
[75] Xiaobao Wu, Thong Nguyen, and Anh Tuan Luu. A survey on neural topic models: Methods, applications, and challenges. Artificial Intelligence Review, 2024. URL httpps://oii org/ 10. 10.1007/810462 023 10661-7
[76] Xiaobao Wu, Fengjun Pan, and Anh Tuan Luu. Towards the TopMost: A topic modeling system toolkit. In Yixin Cao, Yang Feng, and Deyi Xiong, editors, Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume System Demonstrations) pages 31-41, Bangkok, Thailand, August 2024. Association for Computational Linguistics. 202/ 2024 ael-demos...
URL.
[77] Xiaobao Wu, Fengjun Pan, Thong Nguyen, Yichao Feng, Chaoqun Liu, Cong-Duy Nguyen, and Anh Tuan Luu. On the affinity, rationality, and diversity of hierarchical topic modeling. In Proceedings of the AAAI Conference on Artificial Intelligence, 2024. URL htttp:://arx:: org/pdf/ 2401 14113. pdf
[78] Xiaobao Wu, Liangming Pan, William Yang Wang, and Anh Tuan Luu. Updating lan- guage models with unstructured facts: Towards practical knowledge editing. arXiv preprins 20X 2024.
[79] Qianqian Xie, Yutao Zhu, Jimin Huang, Pan Du, and Jian-Yun Nie. Graph neural collaborative topic model for citation recommendation. ACM Transactions on Information Systems (TOIS), 40(3):1-30, 2021. URL https://d/ acm wwt./ooo/////o.o abs 1145/3473973
[80] Yishi Xu, Dongsheng Wang, Bo Chen, Ruiying Lu, Zhibin Duan, and Mingyuan Zhou. Hyper- miner:
Topic taxonomy mining with hyperbolic embedding. In S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh, editors, Advances in Neural Information Processing Systems, volume 35, pages 31557-31570. Curran Associates, Inc., 2022. URL nttps://arxiv.. org/ abs///10/2222.... 10625
[81] Yishi Xu, Jianqiao Sun, Yudi Su, Xinyang Liu, Zhibin Duan, Bo Chen, and Mingyuan Zhou. Context-guided embedding adaptation for effective topic modeling in low-resource regimes. In Thirty- Conference on Neural Information Processing Systems, 2023. URL https //openreview. net/
[82] Xiaohui Yan, Jiafeng Guo, Yanyan Lan, and Xueqi Cheng. A biterm topic model for short texts. In Proceedings of the 22nd international conference on World Wide Web, pages 1445 1456. ACM, 2013. https:: acm. org/ /tt/dd//////s/// 10. 10. /144/2488383332. 24888144
[83] Ce Zhang and Hady W Lauw. Topic modeling on document networks with adjacent-encoder. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pages 6737-6745, 2020.
16

---

[84] Delvin Ce Zhang and Hady Lauw. Dynamic topic models for temporal document networks.
In International Conference Machine Learning, pages 26281 -26292. PMLR, 2022. URL httpps//prooeeedingg wttpeeeeedinns.m....reeeeeeee
mlr hhtp://...ees//////aaaaaaaaaaaaa pdf
[85] Delvin Ce Zhang and Hady Lauw. Meta-complementing the semantics of short texts in neural topic models. Advances in Neural Information Processing Systems, 35:29498 29511, 2022.
[86] Delvin Ce Zhang and Hady W Lauw. Topic modeling on document networks with dirichlet optimal transport barycenter. IEEE Transactions Knowledge and Data Engineering, 2024.
[87] Delvin Ce Zhang, Rex Ying, and Hady W Lauw. Hyperbolic graph topic modeling network with continuously updated topic tree, In Proceedings of the 29th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, pages 3206-3216, 2023.
[88] Yuxiang Zhang, Tao Jiang, Tianyu Yang, Xiaoli Li, and Suge Wang. Htkg: Deep keyphrase generation with neural hierarchical topic guidance. In Proceedings of the 45th International ACM SIGIR Conference on Research and Development in Information Retrieval, pages 1044 1054, 2022. URL httthtttt://pee //ersonall ntu. edu. sg/ xtxxxx1l/i////a httpa////paalllaaaiooo///l///../ pdf
[89] Zihan Zhang, Meng Fang, Ling Chen, and Mohammad Reza Namazi-Rad Is neural topic modelling better than clustering? an empirical study on clustering with contextual embeddings for topics. In NAACL. 2022-20222 Conference of the North American Chopter of the Association for Computational Linguistics: Human Language Technologies, Proceedings of the Conference, page 3886, 2022. URL https:///axxiiii: orgoorgg.ooo.o. /ptp:/dd////// 2204. 09874
[90]
He Zhao, Dinh Phung, Viet Huynh, Yuan Jin, Lan Du, and Wray Buntine. Topic modelling meets deep neural networks: A survey. In Zhi-Hua Zhou, editor, Proceedings of the Thirtieth Interna- tional Joint Conference on Artificial Intelligence, IJCAI-21, pages 4713-4720. International Joint Conferences on Artificial Intelligence Organization, 2021. doi: 20244963/jcaai.222//888 URL lttpss//:/// org/ 10. 24963/ 24963/1jcai /021/638 Survey Track.
[91] He Zhao, Dinh Phung, Viet Huynh, Trung Le, and Wray Buntine. Neural topic model via optimal transport. In International Conference Learning Representations, 2021. URL
17

---

LDA-Mallet
2500
NMF
BERTopic
CombinedTM
2000
GINopic
Running time
ProGBN
1500
HyperMiner
ECRTM
1000
FASTopic
500
10k
15k
20k
25k
30k
35k
40k
Dataset size
Figure 6: Running time under WoS with different data sizes. See also zoomed-in view in Figure lb
(a) Parameterized Softmax
(b) ETP
Figure 7: t-SNE visualization of topic (A) and word embeddings under 50 topics (K=50).
Table 7: Dataset statistics.
Dataset
#docs
Average Length
Vocabulary size
#labels
20NG
18,846
110.5
5,000
20
NYT
9,172
175.4
10,000
12
WoS
10,000
110.0
10,000
7
NeurlPS
7,237
2,085.9
10,000
ACL
10,560
2,023.0
10,000
Wikitext-103
28,532
1,355.4
10,000
A
Limitations
Our method achieves promising performance, but we mention that one limitation is the max input length of pretrained document embedding models. This may hamper the performance on extremely long documents. However, we show that our method can well handle long documents like academic papers in Sec. and this issue can be well resolved by newer and stronger pretrained models like large language models 48 50 78, 49
18

---

Algorithm Training algorithm for FASTopic.
Input: document collection {x(2), x(2), "...,, x(N), pretrained document embedding model fase:
Output: model parameters T, W, 8, u;
// Function Embedding Transport Plan (ETP)
2: function ETP(C, e, L2)
3: Sinkhorn's s algorithm;
4: M = exp(-C/e):
5: b - 12
6: while not converged and not reach max iterations do Lt
7:
a T. Mh
b
a
8: end while
9: return Miag(a)M diag(b);
end function
11: Initialize
R 1:
Initialize Ug + +lv;
13: Initialize d; 5 Sdoe (X:), for i= (1, // Initialize D
Randomly initialize T and W:
for to Ylepach do softmax
16:
u softmax Uo)
18:
// Embedding transport between documents and topics;
c(1) |d, 1 Transport cost matrix between documents topics
20:
π = ETP(C s, S1, K)) Transport plan between documents and topics
21:
Embedding transport between topics and words;
c = || w,ll?: // Transport cost matrix betveen topics and words
22:
23:
Φ (2) (2) u, 62, K.V.); // Transport plan between topics and words
24:
3 KΦ
(()) = NT; for i=
25:
26:
Compute Eq. (8)
27:
Update T, W, S. u with gradient step;
28:
end for
Table Ablation study. w/o ETP means using parameterized softmax (Eq. relations.
to model semantic
NeurIPS
ACL
Wikitext-103
Model
Cv
TD
Cy
TD
Cv
TD
w/o
10.380
102216
10.373
10.164
10.359
10.335
FASTopic
0.422
0.998
0.420
0.998
0.439
0.992
B Preprocessing Datasets
We follow the dataset preprocessing steps of TopMost [76] (1) tokenize documents and convert to lowercase; (2) remove punctuation; (3) remove tokens that include numbers; (4) remove tokens less than characters; (5) remove stopwords.
Table reports the statistics of preprocessed datasets.
19

---

Table 9: Ablation study under K=10, w/o ETP means using parameterized softmax (Eq. 2
to
model semantic relations.
20NG
NeurlI'S
Wikitext-103
Model
Cv
TD
Purity
NMI
Cv
TD
Purity
NMI
Cv
TD
Purty
NM1
CV
TD
Cy
TD
Cy.
TD
w/o BTP 10.367 10,571 10,294 10.408
10.371
10.620 10.512 10,263
10,107
10.537 "0.603 10.381
10,379 10,512
10.364
10,412
10.416 10,660
FASTopic
0/422
0414
Table 10: Topic quality results with different document embedding models.
20NG
NYT
WoS
NeurIPS
ACI.
Wikitext-103
Doe Embedding Model
Cv
Cy
TD
Cv
Cy-
TD
Cy
TD
Cr
all ---pett-baee-g
0.424
0.966
0.999
0.464
0415
0.906
0.426
0.996
0.455
111-diccllobbera-v
0.517
0.976
0.266
1.000
0.460
1.000
0.420
0.998
0.413
0.999
0.435
0.997
211-Mia11M-L.6-22
0.412
0.981
0.477
0.999
0.452
1.000
0.420
0.996
0.413
0.997
0.439
0.992
Table 11: Document clustering results with different document embedding models.
20NG
NYT
WoS
Embedding Model
Purity
NMI
Purity
NMI
Purity
NMI
al1 mpmet-peettee
base-va
0.611
0.550
0.669
0.381
0.680
0.368
all aa1-distilroberta-te
0.581
0.523
0.671
0.371
0.679
0.370
all-MiniLN-L6-Ve
0.570
0.522
0.662
0.369
0.669
0.362
0.0035
Arcrge requurr of wores
0.0030
Word Prequono
3.006
0.0015
3.004
0.00 0.0010
0.0005
0.000
0.015
6.020
0.025
0.030
0.035
0.00015
0.00020
0.00025
0.00030
0.00335
Learned topic weight
Learned word weight
(a)
(b)
Figure 8: (a): Learned topic weights and the average frequency of the top words in topics. (b): Leamnd word weights and the word frequency.
C Training Algorithm for FASTopic
Algorithm shows the training algorithm of FASTopic. ETP uses Sinkhorn's algorithm [60 15 to compute approximated transport plan " and Φ² It is iterative and fast, especially suited to the execution of GPU 23
D Model Implementation
See our code for implementation details.
We conduct our experiments with A6000 GPU. We use al1-MiniLM-16-v2 in Sentence- Transformers F to obtain the pretrained document embeddings, it is fast and also offers good quality. See results with other embedding models in Appendix We set the maximum number of
httpe///inggglgaacc.coossseeeeee
20

---

BERTopic
FASTopic
Step 1:
Load doc embeddings
7.10
Step 2:
Reduce dimenionality
23.13
Step 1: Load doc embeddings
7.10
Step 3:
Cluster doe embeddings
0.21
Step Training
5.85
Step 4:
Compute word weights
1.97
Sum
12.95
Sum
32.42
(b)
(a)
Table 12: Running time breakdowns (in seconds) of BERTopic and our FASTopic on the NYT dataset.
iterations 1,000 and the stop tolerance 0.005 for the Sinkhorn's 's algorithm 15 We set 1/3 and 52 as 1/2. We set T as 1.0 in Eq. (9). We optimize the model parameters through Adam [30] with as 0.002. We highlight that we the above same hyperparameters for all reported experiments to demonstrate the stability of our FASTopic.
200 epochs and learning
E Interpreting Learned Topic and Word Weights
As aforementioned, our FASTopic learns s, the weight of each topic within the document collection and u, the weight of each word within all topics. Figure 8 plots the relationship between the topic/word weights and the word frequency the collection. Generally, topies with higher weights include top words of higher frequency, and words with higher weights also exhibit higher frequency. These observations confirm the validity of learned topic and word weights.
F Influence of Document Embedding Model
We investigate the influence of document embedding models. Apart from the mentioned all-MiniLM-L6-v2. we also experiment with all all-mppett-aas--22 and all all-distilroberta- document embedding models in Tables T0 and We notice that the performance is overall stable across these models. Moreover, the performance grows with all-mpnet-base- and all-distilroberta- especially document clustering. This is because these two produce document embeddings of relatively higher quality [54].
G Running Time Breakdown
To precisely compare BERTopic and FASTopic, we break down their running time. Table 12 shows that they both load document embeddings, but BERTopic takes more steps. BERTopic has to reduce embedding dimensionality, cluster embeddings, and compute word weights; contrast, our FASTopic enjoys faster training. This is because FASTopic employs Sinkhorn's algorithm to solve the optimal transport, which is quite fast as proven by previous studies [15] 22). Moreover, objective is simple and straightforward as it only optimizes four parameters: topic and word embeddings and their weight vectors (Eq. (8).
21

---

Table 13: Topic quality results of coherence (Cv) and diversity (TD) under different dataset sizes (N) of WoS. The best is in bold.
N=15k
N=20k
N=25k
Necilkk
N-w35k
A=ilk
Model
CV
TD
CV
TD
CV
TD
CV
TD
CV
TD
CV
TD
L.M-Malllrr
-0.862
10.351
-0.352
10.357
10.858
-0.354
10.860
-0.500
10,849
NMF
103396
10.413
10.385
10.439
T0,393
10.444
10.396
T0.444
-0.389
1o.442
10.394
TQ.446
BEKiopic
T0,441
10.438
T0.625
CombinedTMM
10.390
R0.562
10.390
+0.589
+0.381
F0.584
F0.382
10.611
+0.385
10.607
-0.388
10.615
GlNopic
10.366
-0.607
10:367
-0.367
10.660
10.362
+0.600
10.678
l0.719
PoutN
10.375
10.842
10.370
10.856
10.378
10.881
10.378
10.866
70,372
10.867
10.373
10882
HyperMiner
10.356
10436
10.353
T0.065
10,353
10.611
10.357
T0.719
10,352
10.725
70.359
106699
ECRIM
0.480
1,000
10,442
1,000
10.442
1.000
10,447
1,000
70.445
1.000
0.452
1.000
FAS Tupic
0.451
0.999
0.456
1,000
0.458
1,000
0,458
1,000
0.454
1,000
0,460
1,000
Table 14: Document clustering results of Purity and NMI under different dataset sizes (N) of WoS. The best is bold.
N=15k
N=20k
N=15k
N=30k
N=15k
N=s0k
Model
Puriy
NMI
Purny
NM4
Puriy
NMI
Purty
NM1
Purity
NMI
Purty
NMI
LDA-Mallet
10.556
+0.331
10.637
-0.323
-0,638
10.319
10.638
-0.318
-0.639
10.317
-0.527
10.314
10.228
10.309
10.531
10.204
Ta.196
BERTopic
10.642
41.322
10.645
40.316
-0.635
11.314
40.641
+0.310
-0.641
10.308
-0.438
11,304
CombinedTMM
CINopic
10.634
10.301
10,630
10.254
10.633
10.294
10.631
10.289
10.637
10:290
70.632
T0.284
PoitRN
10.611
10.376
10.618
10.378
10.615
10:318
10.623
50.608
10.317
HyperMinnr
40.643
+0.357
10.639
10.352
+0.632
10.349
40.640
+0.339
-0.622
10.325
-0.826
Fg.336
10.640
10.355
10,649
10.350
10.644
'0.646
PASTopic
0.679
0.369
0.672
0.364
0.672
0.363
0.671
0.354
0.669
0.353
0.670
0.351
Table 15: Topic quality results of Cv (topic coherence) and TD (topic diversity) under different vocabulary sizes (V) of WoS. The best is in bold.
V=20k
V=30k
V=4ik
V=50k
Model
CV
TD
CV
TD
CVV
TD
CV
TD
LDA-Mallet
10.351
'0,855
10,350
10.843
10.354
10.833
10.356
10.849
NMF
t0.397
-0.431
10,394
10.415
10,392
10.439
10,396
10.449
BERTopic
F0.463
-0.676
10.443
F0.675
F0.384
10.680
$0.400
70.672
CombinedTM
10.399
10.392
10.725
10.397
GINopic
t0.371
*0.623
10.370
10.718
10.368
10.713
10,376
10.725
ProGBN
90.370
-0,767
10.370
F0.766
10,386
10.634
10,381
10.614
HyperMiner
Fo.356
-0.661
40,352
F0.646
F0.354
$0.637
$0.357
$0,583
ECRTM
0.467
-0.931
40.409
F0.952
F0.386
$0.895
$0.358
$0,853
EASTopic
0.470
1,000
0.467
0.979
0.464
0.934
0.460
0.917
Table 16: Document clustering results of Purity and NMI under different vocabulary sizes (V) of WoS. The best is in bold.
V=20k
V=30k
V=40k
V=50k
Model
Parity
NMI
Purity
NMI
Parity
NMI
Purity
NMI
LDA-Mallet
10.637
-0.332
10.640
10.334
10.636
10.326
10.644
10.331
NMF
10.548
'0.229
10.545
10.233
10.560
10.240
10.552
10,235
BERTopic
10.660
'0.338
10.655
T0.333
70.644
10.352
10.652
CombinedJT
F0.656
*0.34
10.667
10.319
F06661
10.346
10.663
10.347
GINopic
10.629
-0.302
10.624
10.303
106634
10.303
10.636
10.304
HyperMiner
10.639
0.363
10.629
0.366
T0663
10.360
10.630
10.356
ProGBN
706620
*0.338
10.625
f0.335
*0.641
10,341
10.653
10,351
ECRTM
F0.649
-0.341
10.658
F0.346
F0.662
40.340
F0.654
$0.342
EASTopic
0.673
0.368
0.676
0.369
0.676
0.373
0.679
0.376
22

---

H Lists of Discovered Topics
Here we list all the discovered topics of different models. Compared prior CombinedTM and BERTopic, FASTopic anchors topics with more specific and relevant words. Specifically in the NeurlPS dataset, the topics of CombinedTM and BERTopic usually contain high-frequency words like "algorithm", "data", "model", "learning", "neural", and "network". Admittedly they are related to some extent, but they are too general to anchor topic semanties [3] 4 For instance, Topic#30 of BERTopic is about chip manufacturing, but contains general words like "neural", and "weight". In contrast, Topic#28 of FASTopic includes more specific ones like "silicon", "transistor", "emos", and "neuromorphic" In the Wikitext-103 dataset, Topic#7 of BERTopic is about species, but has general words "known", "long", "large", "small", and "white". Differently Topic#2 FASTopic covers specific words "breeding", "prey", "subspecies" and "predators". See quantitative results of discovered topics Sec. 4.2
23

---

TombinedTM (NeurIPS dataset)
#1: gradient dual convex convergence regularized accelerated preximal descent convexity smooth smoothness following generalized optimization
mirror
#2: clustering laplacian matrix spectral eigenvectors clusters dimensional means matrices kemel graphs correpanding analysis reduction
manifold
#3: behind contradicts proxy expectations negatives sam problematic exponcatiated providing dumitru french equals addressing defines yuval
#4: network output part inpur graduate units activation rumelhart activations meclelland perturbation devices backpropagation
#5: semantic text language set object example relations examples context caption sonse target amnotations word label
#6: policy value reinforcemmett state mddss uptimal trajectury actions pondp policies mdpp iteration reward action horizon
#7: lopic document modeling model latent Ida collapsed chinese lopics prior specilic stick corpus distribution word
#8: image object shape friston images segmentation objects patches videos bounding using features adversarial part
#9: admit negatives sir exponentiated completes addressing yuval dumimu procy french upon virginia parallelize unfolding prints neurons alexander neuron plasticity cortical inhibitory rule mediated inputs firing patterns synapses synaptic input dendritic
#11: model attention Istm sequence models mns ran istms modeling arxiv memory topic long sequential different
loss complexity regret arm hest ranking setting armed learning algorithms confidence bounds upper boosting bound
#13: exponentiated node variables given tree sampling pomdp probability factor variable finite gibbs new number state
#14: fire fig stimulus stable increasing head strongth plasticity behind timing dog contrast trains spikes cells
#15: gaussian prior process likelihood latent posterior gamma covariance standard observations providing gas mean processes likelihoodss
#16; clustering clusters number items rundom size cluster data algorithms acm algorithm users item meuns maximum
#17: drop measures empirical let positive learnaility theorem generalization following boosting delined since uniform
#18: support adaboost space covering initiation inner olkopl margin lends vapnik mistakes notation risk conventions taylor
#19: attention temponl spatial hop similar visual level morrison second video frame encnuraged predict spatiotemporal
#20: units hubhard oulput net morgan stack rumelhart symbolic science internal weight weights epochs letters connectionist
#21: log complexity thearem case algorithm lemma pages regret bounds active upper let bound algorithms lnsses
#22: variables causal message nade edges variable marginal marginals propuattion inference chain sum degree graphical nodes
#23: classification distance positive metric framework label multi codes learning semi tasks accuracy kernel test
#24: pointed promoting problematic stack home place accumulated unknowns fibers gracpel thresholded locomation aliasing hamilton tseng
#25: patch color scenes prototypes shape object patches seene voxels tracking image objects pixels audio
#26: regret algorithms strategies games mdps player armed subsequenly algorithm confidence equilibrium best online known mdp
unfolding output morgan inductively jacobs biases thesis realizes smoother iearning problematic forces species protein shortcoming
#28: observer sources speech souree audio sur carbonell gardner signals pitch uapublished location middle perceptual harmenic
learning generative training use output samples costamizadeh objective adversurial deep arxiv drapout hatch networks work
#30: qin problematic admit expectations pooled providing uddressing yuval french unalysed drop carlos crucially regressors collectively
set node clustering tree sets clusters meuns search algorithms graph points given cost number solution
M32: stimulus fig coding noise responses correlation natural sensory estimated decoding information population agency neurons tricky
#33: functions upper theorem bounds case map define polynomial influence show lemma set following max defined
#34; motor left field movements location motion perceptual bottom right turned locations cog bei hamans bar
#35: samples log posterior variational gaussian likelihood latent bayesian approximate test elbo peior inducing data monte
#36: norm lasso solution problem matrix following matrices low sparse constraint rank sparsity methods global selection
#37: providing proxy addreasing admit sam problematic qin contradicts carlos virginia french webpage exponentiated analysed dumitru
#38: sample statistical catimation distributions statistics estimator lasso dimensional log copula condition estimators theorem families estimating
#39; intelligent morgan expectations complete unification host damping interfacing required thesis ingredients sam drop triple agency
N4D: master transfer results datasel image images classification imagenet plain representation ranging deep generator convolutions based
#41: making behuvior plusticity decision observer estimated change sensory time effect trials term trains fit trial
#42: behind guide providing appended problematic proxy addressing admit gin negatives yuval orabona french carlos intelligent
#43: network size networks pooling layer convolation layers accuracy weights without gradient sequence deep batch neural
#44; agents agent actions expert reinforeement qin task goal decisions based rewards skills world observation
#45; matrix power tensor columns completion rank matrices sparsity via decompasition singular iteration high iterations analysis
mediated guide damping vivo intellieent problematic addressing cerebellum axon crucially mostafa behind dumitru unfolding pull
#47: guide æbí database distance classifier sets classifiers roe faces classification validation vectors performed cuclidean cascade
#48: loss norm statistical convex following theorem convexity functions exeess term conditions ferrari mirror observes condition
#49; kxt cells inhibitory selectivity cortical qin neurosci cones cell har effects christoph address sam bottom
machine algorithms prohlem pull functions objective learning loss sei optimizzaiion solution syan constraints boosting methods
24

---

BERTopic (NeurIPS dataset)
#1: learning model data set algorithm using lunction one training network two ligure time number models
#2: kereel learning active algorithm kernels svm training examples fanction theorem data bound margin
#3: convex convergence gradient optimization al algorihhm stnchustic descent algorithms loss regret online problems problem rate method
#4: spike neurons
synaptic firing spikes neural time model spiking activity population stimalus input information
#5: policy state reward ageat action learning reinforcement value function game agents actions games optimal algorithm
#6: speech rocognition speaker training neural network recurrent sequence input model output word networks hidden
#7: visual motion eye cells saliency orientation model spatial image rocoptive ncurons figure response cell stimulus
#8: sparse lasso peu matrix norm algorithm recuvery problem data principal spursity log rank analysis solution
#9: graph graphical variables inference map graphs node algorithm models free nodes edge set problem log
gaussian posterior model process bayesian models time covariance likelihood prior distribution function mean #:l: graph manifold metric distance points data emhodding nearest learning graphs neighhor space dimensional kernel local
network networks neural units ingut weights hidden ourpot function leamning layer weight one nules unit
#13: deep layer networks training convolutional layers network gan image learning neural anxyy images generative cnn
topic word words topics Ida document model documents models language dirichlet latent data distribution corpus
#15: model memory decision stimulus reward response trial figure stimali learning task trials models two time
#16: brain oog subjoct fouri subjects functional data comnectivity voxels bei spatial voxel time activity motor
#17: image images objoct features visual model shot classes question training class feature learning set objects
#18: regret arm bundit arms bandits ucb algorithm bound reward armed problem log action setting exploration
#19: matrix rank tensor
completion low entries matrices algorithm decomposition problem factorization tensors theorem recovery
#20: policy function state optimization value belief pomdp optimal uncertainty algorithm pardpp search mdp reward problem
#21: variational mixture inference posterior log models data dirichlet distribution gradient bayesian parameters likelihond model components
#22: clustering clusters cluster spectral means algorithm cut data points graph matrix prohlem partition set mumher
423: video motion pose frames frame model tracking image temporal human body object using flow figure
#24; task tasks learning domain targel multi transfer source data adapiation training domains feature multitask peoblem
#25: ranking item rank users query ratings model pairwise collaborative top algorithm rankings matrix
#26: label labels unlabeled supervised labeled classification data learning semi class loss set examples multiclass problem
#27: control controller motor trajectory model movement forward system learning network robot time inverse movements
#28: unditory frequency sound sounds cochlear signal neurons model time stimuli localization resporee stimulus system response
nodes influence network node social networks community communities model edges graph link edge algoritim
#30: chip circuit analog vollage neuron visi synapse input circuits current gate digital output w weight neural
image images resolution denoising reflectance depth blur pixel noise color pixels scene shading deconvolution scale #32: density kernel test sample estimation mmd tests estimator statistic characteristic distribution null kernels two samples
model clinical risk data models cancer decision process event individual #34; causal variables graph data model treatment observational interventions models discovery discrimination causes directed set effects
patient survival patients
#35: ica separation source sources signals blind signal independent mixing matrix components algorithm component speoch mixtures
#36: protein gene genes proteins expression prediction sequence sequences model data species amino structure features binding
sampling hmce monte carlo hamiltonian sample distribution mome samples chain scan sme proposal stochastic dynamics
#38: privacy private differentially differential mechanism data algorithm log theorem party utility let protocol distribution queries
#39; sparse coding basis sparsity coeflicients image prior dictionary wavelet overcompiete model reconstruction data slab images
#40; face facial images faces recognition image human features cun feature pea verification subjects classification
#41: submedular functions function algorithm greedy set appaxximation maximization monatone submodularity solution algorithms problem
streaming representatives
#42: price market revenue prices auctions regret bid reserve sirategic profit maker markets algorithm optimall trading
#43: workers worker crowdsourcing labels voting tasks task label anootators altematives crowd majority true items model
#44; music musical harmonic notos note rules rule pitch song beat tags time system structure signal
#45; hashing hash codes binary bits hamming hit similarity precision search data functions retrieval query
#46: choice preferences utility preference model user decision rankings options users set choices models item option
memory capacity associative memories stered patterns hoplield network pattern recall storage netwerks sdm number retrieval
#48: treas tree decision forests split leaf bode forest training dags random nodes beeiman feature splits
routing traffic call aniial channel load zate time policy service mobile calls control loss
#50: relational entities relations link entity relation links model tensor factorization relationships models data learning rank
25

---

EASTopic (NeurIPS dataset)
#1: wahba steinwart sriperumbudur sollich rasch soboley christmann integrable heteroscedastic blanchard Jukumizu unregularized qui
mediarmid functionals
#2: graphs vertex vertices edges graph trees tree erige mode nodes message directed loopy lifted treewidth
43: ranking
items item documents topics topic Ida document social web rankings crowdsourcing vates
#4: robot controller controllers learning alkeson backup bradtke satinder planner kacibling doina bellemare antonoglou samas pendulum
#5: deep convolutional layers onn layer ran encoder Istm trained recureett drepout architecture arxiv archittcctres bengio
#6: crammer halfspaces pereeptron multilabel koby disagreement beygelzimer classi holdout mistake gentile aggressive littlestone queried
claudio
#7: net units network activation networks unit nets cupacity back hidden connection multilayer pullirms backpropagation oulput
#8: dauphin desjardins ganguli wojeiech gulcehre razvan rgen gloret barham theans pascanu abudi dahl urjossky zaremba
#9: cognitive automaton verhal pollack teach hillsdale recalled psychological cards cognition psychology infants teaching chess episodic
monte hyperparameters salimans titsias ranginath gans elho langevin vae radford hamiltonian revende
#11: waibel frasconi widrow squashing holmdel giles jaidly zipser retraining hahdanau freeze microstracture retrained ronan denver
trajectory dynamics trjectories control dynamical dynamic state states transition transitions sequences system kalman temponi forward
#13: price regret adversary round market revenue prices suctions hedge bid markets profit ads multiarmed reserve
#14: receptive Ign striate geniculate retinotopic topography moeshon afferents ocro ventral ocular parictal orientation hubel lond
#15: stimulus stimali cortex colls neurons activity sensory brain responses ncuron cell response cortical neuroscience trial
#16: classifier clasiliers svm unlabeled margin boosting labeled classification label labels supervised classes class examples syms
#17: manifold kernels kernel dimensionality eigenvectors laplacian eigenvalues metric euclidean embedding principul pea langent isomap
cigenfunctions
#18: mika smo kundola holloway scholkopf fukunaga quinlan misclassifications tsang grandvalet wripper zien mirepositoryhtml
#19: causal model structure models group acroes data individual modeling features specific different three used experiment
#20: amino acid acids conserved jojic szeliski mol registration isometric proteins analyzers tomasi molecular shading tissue
#21: clustering clusters cluster hashing hash Ish linkage anomaly clusterings dissimilarities ugglomerative kmeans indyk luxhurg churikar
#22: estimators estimator regression multivaritte covariance donsity estimating variance additive gaussion parametric asymptotic estimation bias
estimates
#23: recovery tensor rank sparsity entries sparse completion lasso singular columns matrices norm matrix factorization decompasition
#24: scene object image images pixels pixel segmentation pose vision lace eypr patches color objects video
baar printed strokes eigenfaces karayev photographs dowasampling zip satheesh shelhamer resized sermanet cafle bruna
#26: ancestral propositional kemp dumais wallach grammars predicate jurafsky syntax darwiche noun predicates grammar parser naacl
reaction events triggering survival diffusion mice reactions longitudinal regulatory durations progression species event #28: chip circuit unalog vollage circuits visi silicon cmas transstorr transistors chips voltages neuromorhic axon fahricated
winther bumtine opper paisley andriy digamma moitra niranjan andrieu knowles cumulant kappen doucet harber minka
M30: alp boyan ghavamzadeh puterman nord dimitri ortner lille lazaric optimism esaba szepesari tsitsiklis restart pulyak
#31: bounds minimax sup risk incquality privacy corollary lemma bounded bound theorem proof inf holds upper
#32: bebert ramanan urtasun articulated schiele volumetric triggs lampert occlusions indoor hoiem animation mathicu saenko metz
posterior gibbs inference bayesian priors mixture dirichlet likelihood latent variational sampler meme marginal probabilistic poisson
#34: policy reward agent actions policies reinforvement agents mdp games player action exploration planning rewards
#35; nongaussian diagnostics candela snelson ard visualisation warped imputation reversible duvenaud vague aie dbus carlin neil
#36: bifurcation chaes basins chaotic attracters oscillates escillate basin attraction landscapes intercomnection intercennection sompolinsky tak
salesman
M37: speech speaker hrmm autio acoustic phoneme speakers phone phometic utterances utterance spoken female voice recognizer
#38: spikes spiking spike synaptic firing symppses membrame trains posisymptiic connectivity stdp calcium hippocampal epsps potentiation
#39: language word semantic words sentence embeddings category captions phrase mikolov caption eaptioning answering sentences
#40: eye motion velocity saliency gaze fixation saccades saccadic optic itti observers photoreceptor salience distractors photorcceptors
#41: caruana expertise disorders atlas diagnosis diagnostic pet mitchell prototype classifications discriminating niculescu clues exercise
#42: rip doncho dantzig isomerry candes incoherent obozinski johnstone emmanuel baraniuk negahban fazel parrilo caramanis vershynin
darken soviet finness beating santosh morrison purdue schraudolph submission splines penrose sherman placements hassibi
#44: descent proximal sgd primal dual coordinate streugly convex convergence smooth gradient accelerated svig kxt nesterov
#45: brownian elliptical bivariate dunson breakdown climate covariates forecasting cconometric forecast semiparametric comnometries forecasts observational cdr
submudular approximation solution algorithms algorithm problem functions optimization function problems approximate objective
constraints linear greedy
#47: learner active queries online decision hypothesis query rule strategy mechanism adaptive cost every concept making
#48: sketching threads mahoney drineas woodruff parallelizing parallelize preconditioning epus cores thread tak speedups paralleiization richt
#49; auditory eeg sources ica signals sounds meg cochlear khz hearing microphone ear acoust acoustical
#50: naor mossel lov iyer combinatorics submodularity mesheny asz karp bilmes talwar feige nemhauser karger wolsey
26

---

TM (Wikitext-103 dataset)
#1: general war confederate washington fort massachusetts york grant ammy kentucky william convention men states united
#2: storm september october upgraded southwest day northestward strengthened convection briefly tropical hermuda island southeast disciparing
#3: secrend lap place drivers driver third ahead fourth position edwards time stage stops classification caution
#4: also music released one film disacy million animation time show made year first vidco new
#5: ride station closed location ownership bistoric train rail restaurant parking services building norwegian stations underground
#6: vidco song top number chart music hot dancers performance carey madonna week dancing performed love
N7: city island
river area pupulation many region san coast spanish settlements century
#8: queen prince king made years royal charles england london william death later family henzy father
#9: title match
championship world won lag defeated ring wrestling face joe cage referce raw
#10: court courts ri right amendment criminal clause requires cent public person singapore law case without dollar
stone century house period william dates wall chapel tower henry buildings gothic england site
#12: species years populations found cause large known prey conservation water feed muscle individuals many humting
first one made year world years three new two team won also season second
#14: heads residential ends junction southeast redesignated county designated portion splits route briefly woods divided intorehange
#15: constantinopls chronicle bungary byzantine prince sourees empire count emporor conflict kingdom conquest rebellion brother commanded
#16: brigade army division unit forces general infantry battalion casualtics hill men artillery corps flank commanded
#17: york year time said family guy lelevision years new life later american show award obuma
#18: game player games gumeplay playstation version guitar soundtrack hero features mode original multiplayer ign mario
#19; stroke beat rowing olympic cox progressed alead oxford excluding finishing loss referred athlete push gold
#20: listing musician tragic albums drums personnel notes vocal liner garde drummer instumental dan sounds soundtrack
#21: life jesus work god women wrote philosophy moral spirital ideas scientific book argues religious views
#22: party national members leadership leader state govermment conservatives college assembly elections minister liheral support prime
#23: season pitched games played year basehall basketball team leugue giants ncan home signed record named
education dogree board profossor enrolled mayor serve attended deputy worked founder associate massachusetts district assombly
#25: unk known many also used dynasty period speakers horses century ancient greek word found form
#26: episode nielsen scully tom andy broadcast jenna tracy reviews scene files simpsons
watched funny
#27: battle two british three lire flect men made day line ships island left warships position
music harrison band bands songs composers work musician musicians album progressive beatles musical recorded
#29: storm flooded damage island precipitation surge downed toraado water homes damaged along tropical near coast
story characters character anime quest game manga final original protagonists novels voiced player volumes protagonist
M31: tons armor steam knots adriatic hattery armored turrets displaced laid aft monitor bailers consisted stemn
alhum
record songs released copies studio group chart release recording records
M33: discovery ignore commemorating nomenclature outlined indirectly planet divide uddressing garde eamest discovered masses shifl judged
#34: air pilots aircraft wing squadron aviation fying fighter training service flew nuclear squadrons flight pilot
#35: club made arsenal played minute striker substitute half united liverpool side scored england cricket first
#36; city stadium club college located company students sports foothall school include campus hosted largest schools
#37: soviet moscow nazi german government finnish line polish regime germany country jews electric hitler
#38: film movie bond role reviews grossed scenes films critics production grossing india rotten picture director
#39: game bowl first yards time quarter also kick season line offense possession tech alabama virginia
#40: mushroom bearing shape gill growing flesh attached color taste orange diocese fungus epithet cap fine
#41: line river along bridge railway miles freeway junction county purt rail raal interchange built mile
#42: bat chart lewis charts tempo digital latin vocal dancing forty singles billboard debuted backgroumd charted
#43: nielsen bart adrian simpson arives broadcast reference peter burns wallace watching olds fox realizes pearson
#44; stable properties known used material hydrogen curve protein experiment form applications processes process formula space
#45; flect turrets british seas ships port tons ship baltic class convoy armor fire aft lasted
#46: ignore judged commemorating monopoly divide garde outlindd enhanced addressing constantly obituary unprecedented carnest exploit
busy
#47: editions magazine artist book critic photographs published art paint publisher stories publishers notes publication read
#48: species conservation brown populations mature feed slightly typical tree legs individuals tail iuen eggs
#49; relationship ben neighbours episode soup storylines character any rachel mitchell davies finale episodes gel producers
flooeed preparations downed corvection upgraded warning downgraded homeless affected storm eye meteorologccal originated decreused ushore
27

---

BERTopic (Wikitexf-103 dataset)
#1: unk also one first new two time film song later game war three album city
#2: alhum song hand music songs chart number released video single rock tour recording track madonna
#3: episode show series season episodes homer character television viewers simpsons scully said scene hart doctor
#4: ship ships squadron fleet aireraft guns air navy british japanese furce two hattle gun
#5: scason team game games first league runs innings played caroer sccond scored test won record
#6: highway route road state county north frecway creck interehange
intersection along avenue
#7: species unk genus found birds known shark fomales males long large brown small white common
#8: storm tropical burricame winds cyclone mph damage depression rainfall wind lyphoon systemi landfall september flooding
#9: lilm films bond role character unk best production disney actor story also movie million director
division army infantry german brigade baitle forces troops batialion
corps north
regiment
#:l: game player games players gameplay fantasy released characters character nintendo final version series development release
king scotland henry england edward century scottish ryyal unk william english
son queen hishop
#13: castle bridge town century huilt house river building canal area local south road centre west
stories painting novel unk comics published poem fiction works story issue magazine
#15: election govemor party stare president campaign government republican senate kentucky democratic new house political elected
#16 race lap car stage cambridge oxford racing drivers team lead second won raees riders points
#17: unk anime manga series also one language released greek character first english century story used
#18: polish unk soviet poland creutia russian war amy emperor byzamine military empire forces hungary government
#19: unk temple government city chinese singapore dynasty china state emperor also count india arab minister
#20; club league cup season football goal match scored team arsenal goals stadium lirst final win
#21: station railway line trains train ride london services passenger roller class service stations opened milways
#22: music opera orchestm composer musical works symphony unk piano work gilbert sullivan first theatre performance
423: nuclear unk atomic laboratory compounds used element metal project physies hydrogen chemical energy reseurch university
#24; star planet earth sun planets stars orbil jupiter solar mass magnitude surface system dwarf moon
#25: match championship wrestling tag event team defeated ring title champion world angle triple feud
#26: church god unk century congregation christian charches pope religious catholic one christ also bailding moral
#27: airport norwegian station norway unk party trains tunnel swedish service services started built
#28: discase protein cells unk cell symptoms risk blood dna treatment acid cause virus cases infection
flows #30: school students university college campus student education georgia schools research program academic faculty tech building
river park volcanic water creek area
valley canyon mountain
court chicago city state park indiana states county supreme huilding district united river illinois
#32: company restaurant chicken food king unk product beer products wine menu new restaurants chain ingredients
spanish texas mexican san city govermment spain houston mexica bay juan plaza puerto political
#34; trek enterprise episode star space crew upollo series mission spacecraft kirk nasa first season earth
#35: coins flag coin dollar silver design struck cent gold picces cagle dollars statue reverse flags
#36: oil darwin bank evolution natural species unk plants billion energy investment organisms evolutionary selection animals
horses breed
breeds sheep breeding dogs dog bred used registered century riding white unk theater
#38: botel building library theatre mall center square feet floor
#39: police said murder case evidence fire murders trial found death court told prison government people
#40: apple windows software system data console unk playstation microsoft ninteado user users bardware device released
#41: formula function theory space numbers number matrix frequency unk example used mechanical consiant mathematical linear
#42: flight airline airlines aireraft air boeing airport crash flights passengers accident international aviation plane
943: radio network stations television station broadcusting digital programming mutual spun broadcasi pararmount news channel cable
#44: podro brazil brazilian emperor govermment argentina naval navy ships chile portuguese portugal cabinet
#45; resident evil god game war playstation lcon umbrella relcased ii player series character claire games
#46; adriatic croatin traffic toll route port section interchange river
kilometres rest construction basin
#47: football women team fifa national cup peru country world toumament players association teams competition played
spider man poter parker film amazing character soay comic hanyy webb jane comics mary miles
#49; children show street television producers workshop educational research curriculum viewers production lesser clues goals productions harry potter book film books ron series million magic children philosopher novel phoenix stone relcased
28

---

EASTopic Wikiteext1133 dataset)
#1: kentucky virginis masachusetts ohio indiana confederate lincoln illinois pennsylvania congressman sherman missouri jefferson democrat
congressional
#2: species birds males hreediing fermales animals bird fish breed prey eggs breeds nest subspecies predators
#3: ottnn movement many life unk social among others age popular children years hecome considered
#4: railway creck bridge lake river park road county valley street miles construction opened canal line
#5: ships ship floct guns admiral tons torpedo navy hms inch noval gun deek cruiser knots
#6: students university college school campus chicago student education program rescarch schools business company arts building
47: viewers jenna viewership storylines sturyline dwight ratings soup tracy jim fringe timeslol linale
#8: homer simpsoes burt scully lisa liles lox dana households springlield burns leslie nielsen sired marmers
#9: goddess deity dialect ingredients chicken wine lolklore sheep cooking gods rituals silk pig bread
#10: vampire villains antagonist creature kills villain escapes backstory jake demons monsters kill demon stan flees
century church temple population centre india chinese roman scotland ancient period site region local
#12: tower floor walls storey courtyard architects roof architectural constituency brick excavations architect carved castle mance
basehall basketball pitcher rookie freshman pitching oventme nean expn awsists tigers leagues bockey suphomore traded
#14: wrestling liverpool match raw ring gealkeepper striker ham referee footballer matches champion championship pinned
#15: planet plancts volcanic jupiter magnitude orbital goological orbit mincrals goology observatory cluster melting, platcau dioxide
#16: cricket innings test olympic australia matches match bowling won race stage event win competition
#17: flotilla adriatic dockyard emeemates amidships masts gunners austro keel sms conning brondside bombarded lowed cruising
#18: mario computer soltware gamer eurogamer graphical puzzles informer consoles microsolt user arcade puzzle sonic interactive
#19: show comedy guest relationship audience interview really shows girl breadeast friends television think commented sex
#20: torrential thunderstrmm inundated intensify intensifying outflow outages downgraded periphery saffir thunderstorms shelters disorganized
intensificution currents
#21: grossing screenplay theaters grossed filmmakers chert screenwriter cinematographer tomatoes picture cinematography cinema paramount
movies rotten
#22: trains train airport passengers passenger stations cars airline ride roller freight destinations railways airlines runway
#23: drummer bassist vocalist guitars vinyl riff nme riff's guitarist demos headlining unreleased keyboards labels dylan
#24: elergy bishops coclesiastical protestant cardinal theological priests monks diocese catholics theology teachings papal mamuscripts
catholicism
narrator feminist novelist autobiographicaal reprinted prose poetic poets essays illustrations imagination anthology reader essay realism
#26: freeway interchange intersection highway terminus avenue passes crosses continues lane turns alignment heads interstate intersects
mathematics curriculum economics professor thesis undergradunte lectures psychology lecture mathematical ethics nohel harvard physics
journalism
aircraft flight air flying fighter wing mission squadron pilot operations landing bomber pilots
#29; disease risk treatment blood cases symptoms cell cells peotein diagnosis clinical infection patients medical brain
#30: confluence flanking flourished stratogically advocating headquartered undermine formulated overron strife annexed affiliation landowners
aristocracy bordered
#31: cadet howe sergeant scout scouts citation volunteered badge decorations bravery discharged instructor scouting trenches rifle
#32: cylinder engine specifications prototype machine capabiity weight mechanical barrel variants configuration manufactured mamufacture
muzzle wheel
#33: polish flag soviet poland russia russian creatia nationalist dutch republic jews countries serbian isracli socialist
#34; emperor reign king empire roman byzantine momarchy throne casiles archhishop castle ruler monarch revolt persian
M35: film films episode character
script actor movie series episodes filming production director
#36: league foottall goals goal cup couch yard stadium yards scored club season teams players leam
#37: overthrow communists coup bin partisan marched embassy pact peaceful factions faction exiled massacre negotiate rebel
#38: mary married sir william london wife thomas queen henry edward lord george died elizabeth charles
#39: lap cambridge oxford riders races rowing seconds cycling olympics rider athiletes drivers caution lengths
#40: infantry battalion brigade regiment troops army corps artillery forces soldiers division battle command attack wounded
novel books stories author works
art poem opera literary text poems published poetry
#42: album chart band song songs billboard albums guitar recording lyrics pop vocals singles madonna music
tropicll hurricane cyclone storm winds rainfall depression flooding intensity mph landfall wind damage typhoon
#44; energy earth mass formula surface carbon gas type hydrogen chemical data temperature systems example process
prison murder police prosecution trial jury investigation munders guilly convicted alberta conviction sentence crime testified
M4Б; piano orchestra compuser dancers conductor violin pianist symphony singers lenor orchestral composers composition duet concert
#47: gameplay fantasy nintendo playstation game player anime manga soundtrack xbox characters games players dragon mode
#48: lyrically certifications sirplay pitchfork synth synthesizers remix remixes remixed catchy rapper vibe listings downloads slant
#49; cap fruit fungus spores phylogenetic taxonomic morphological clade hairs basal mushroom microscopic stem morphology epithet
#50: election party law govermeent political court president minister senate republican democratic constituion rights committee elected
29

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: The main claims in the abstract and introduction reflect the contributions of our proposed new topic model, FASTopic.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
We discuss the limitations in Appendix A
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
30

---

Justification:
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: We describe our model detail Sec. 3 and appendix D We also upload our code with submission.
Guidelines:
The answer NA means that the paper does include experiments.
If the paper includes experiments, No answer this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily the architecture clearly and fully.
model architecture, the paper should describe
(c) If the contribution is new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility, In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
31

---

Answer: [Yes]
Justification:
We upload our code with our submission.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.co cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (c.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https guwdes/ddeeeeee Cooeeeoodeesssseeeeee //nips.com htc/://bbb///gdddd///eeeeeueisss ssiniiiicyy for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Does the paper specify all the training and test details (c.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary understand the results?
Question:
Answer: [Yes]
Justification: We include experimental settings and details in Sec. 4.1 and appendices B D
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification:
We report statistical significance tests in Sec.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the enror bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
32

---

It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification:
We report experiments compute resources Appendix D
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics http://awrrr.cccuuuu.uccctcccaaa
Answer: [Yes]
Justification:
We conform with the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
We discuss societal impacts in Sec.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no socictal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
33

---

The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of MI..
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Lieenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: We cite the original sources of code, data, and models in Sec. 4.1
Guidelines:
The answer NA means that the paper does not existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
34

---

If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification:
Guidelines:
The answer NA means that the paper does not release new assets,
Researchers should communicate the details of the dataset/code/moodl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Ethies, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
Guidelines:
The answer NA means that the paper does involve crowdsourcing research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
35


Reviews:
[{"review_id": "H5ihwrKfw2", "reviewer": "Reviewer_FTng", "summary": "The author found that existing methods (VAE-based or clustering) suffer from low efficiency, poor quality of topic words, and instability. To address these issues, this paper proposes a novel topic modeling paradigm called Dual Semantic-Relation Reconstruction (DSR) for efficient modeling of semantic relations among three types of embeddings: document, topic, and word embeddings. The author attributes the low quality and instability of previous methods to relation bias issues, leading to repetitive topics and inaccurate document-topic distributions. To tackle this, the paper introduces the Embedding Transport Plan (ETP) to regularize relations among the three embeddings. Combined, DSR and ETP form the proposed topic model, FASTopic, which is evaluated on six benchmark datasets, showing encouraging performance.", "questions": "1. In Line 76, the author thinks that methods like Topicgpt, which uses LLM to describe topics, deviate from the original LDA setting. Can the author provide a more detailed explanation in this section?\n2. In Line 132, it is mentioned that some studies think that repetitive topics and less accurate doc-topic distributions are due to a large number of topics (K being set too high). The paper, however, attributes these issues mainly to relation bias. Does a large number of topics also affect FASTopic in the same way?", "limitations": "1. The proposed method relies on embedding transport semantic relations and may be limited by the max input length of pre-trained document embedding models.\n2. See Weakness 3.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The paper is clearly presented and easy to follow, with an explanatory and easy-to-understand mathematical formulation.\n2. The proposed DSR objective is effective, significantly reducing training time compared to VAE-based methods such as ECRTM and CombinedTM.\n3. The experimental evaluation is extensive, showing that FASTopic consistently outperforms multiple strong baselines on topic coherence and topic diversity, while also demonstrating advantages in terms of running speed and transferability.\n4. The paper demonstrates the model's robustness under multiple numbers of topics (K=75, 100, ..., 200).", "weaknesses": "1. Tables 6, 8, and 9 present the ablation study of FASTopic using ETP and parameterized softmax. ETP, compared to the ECR used in the previous VAE-based method ECRTM, adds semantic relations between topics and documents. The paper lacks proof the dual transmission effectiveness of ETP by not comparing it with ECR (which only includes semantic relations between topics and words) and another case that only includes semantic relations between documents and topics.\n2. I am concerned that while the DSR training method improves efficiency compared to VAE-based methods, it may make it difficult to build meaningful relations between topic embeddings and document embeddings for some long-text corpora due to its singular objective.\n3. It would be nice to showcase a comparison of topic words or transferability between FASTopic and ECRTM to demonstrate that FASTopic offers improvements in multiple dimensions, not just speed.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your feedback! We're happy that you appreciate our clear writing, effective method, and extensive experiments.  \nWe sincerely hope our responses can address your concerns and improve your rating.\n\n\n__Q1: difference between ETP and the ECR of ECRTM (Wu_2023)__\n\nWe clarify that __the ECR of ECRTM is NOT an alternative to our ETP for ablation study.__  \nThe ECR (embedding clustering regularization) of ECRTM only regularizes topic and word embeddings and does __NOT__ model the semantic relations between them as topic-word distributions. ECRTM follows the traditional VAE to model topic-word and doc-topic distributions for topic modeling.  \nInstead of VAE, our method uses ETP to model the semantic relations as topic-word and doc-topic distributions for topic modeling.\nThus, the previous ECR does not model the semantic relations as our ETP, and it is NOT an alternative to our ETP for ablation study.  \nBesides, only the semantic relations between documents and topics or only between topics and words cannot do topic modeling, because we need both of them for reconstruction in Eq(1).\n\nThank you for the question. We've added more clarifications in the related work.\n\n\n\n__Q2: performance on long-text corpora__\n\nWe clarify that __we have experimented on long-context corpora, NeurIPS, ACL, and Wikitext-103.__  \nAs reported in Appendix_B Table_7, they are long academic publications or Wikipedia articles, ranging from __1k to 2k words__. We have shown that our model works well on these long-context corpora in Section_4.2, 4.3, and 4.4.\n\n\n__Q3: showcase comparisons to ECRTM__\n\nWe clarify that __we have showcased the comparisons to ECRTM on topic words and doc-topic distributions (Section_4.2), running speed (Section_4.4), transferability ( Section_4.5), and adaptivity (Section_4.6).__\n\nHere we illustrate some cases of topic words.\n\n|ECRTM|\n|:----|\n|#14: boeing software microsoft customer trains airline hardware updates airlines consoles|\n|#50: typhoon carrier boeing aircraft airlines carriers airline philippines pilots japanese|\n\n|FASTopic|\n|:----|\n|#18: mario computer software gamer eurogamer graphical puzzles informer consoles microsoft|\n|#22: trains train airport passengers passenger stations cars airline ride roller|\n|#43: tropical hurricane cyclone storm winds rainfall depression flooding intensity mph|\n\n\nWe see ECRTM mixes the topics of airlines, software, and typhoon.\nIn contrast, FASTopic produces more separated topics about them respectively.\n\n\n__Q4: Differences between TopicGPT and LDA__\n\nWe explain that __their main difference lies in how they define topics and understand documents.__  \nLDA defines a topic as a word distribution, for example, a distribution [0.1, 0.2, 0.1 ...] over words [farmer, products, agricultural, ...].\nDifferently, TopicGPT defines a topic as a natural language description, like *Agriculture: Discusses policies relating to agricultural practices and products...*.  \nBesides, LDA infers distributions over topics to understand documents, while TopicGPT classifies a document with topics as label space by prompting.\nTopicGPT reaches higher interpretability, but cannot give precise distributions for downstream tasks.\n\nThank you for the question. We have updated the paper with more explanations.\n\n\n\n__Q5: does a large number of topics affect FASTopic?__\n\nWe clarify that __we've reported in Table 4 and 5 the results under large numbers of topics (K=75, 100, ... 200).__ Our model remains high-performance with large numbers of topics, especially on topic diversity.\n\nWe clarify that in Line_132, we intend to show using parameterized softmax leads to the relation bias issue even with a small number of topics, as supported by Table_9. This motivates us to propose the new ETP method.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "YBehfruYOW", "reviewer": "Reviewer_Py7Z", "summary": "This paper proposes a fast, adaptive, stable, and transferable topic model, FASTopic. Instead of using the VAE or clustering method, it incorporates a new model structure named Dual Semantic-relation Reconstruction (DSR). DSR learns topics by directly optimizing the semantic relations among topics, documents, and words. The semantic relations are further regularized by an Embedding Transport Plan (ETP) method as an optimal transport problem. Experiments demonstrate the effectiveness of the proposed model.", "questions": "See weaknesses.", "limitations": "Yes", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The paper is well-written and easy to follow\n2. The proposed DSR framework is simple and neat.\n3. The experiments and ablation studies demonstrate the effectiveness and efficiency of the proposed FASTopic method.", "weaknesses": "1. [76,64] should be compared as baselines in the experiments since they both incorporate optimal transport objectives in their model as FASTopic does.\n2.  As far as I know, the time complexity of solving the optimal transmission problem is very high. Is there any technique used in FASTopic to efficiently derive a solution? Furthermore, it would be great to theoretically analyze the time complexity of the current FASTopic framework.\n3. Is it fair to compare FASTopic, which takes document embeddings as input extracted by sentence-BERT, with other baselines in the experiments?\n4. As LLM has shown great performances in many NLP tasks, I believe it is necessary to include a discussion about the reasons and advantages of developing a topic model that is not LLM-based.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your helpful feedback!  We're glad that you appreciate our well-written paper, neat method, and extensive experiments.  \nWe sincerely hope our responses can address your concerns and improve your rating.\n\n\n__Q1: comparison to earlier NSTM (2022) and WeTe (2023)__\n\nThank you for your comment. We explain that __we've included the newest baselines__: ProGBN (ICML 2023), ECRTM (ICML 2023), and GINopic (NAACL 2024).  \nHere we additionally report the results of NSTM and WeTe:\n\n|Model|20NG|||NYT|||WoS|||NeurIPS|||ACL|||Wikitext-103||\n|:----|:----|:----|:----|:----|:----|:----|:----|:----|:----|:----|:----|:----|:----|:----|:----|:----|:----|\n||$C_V$|TD||$C_V$|TD||$C_V$|TD||$C_V$|TD||$C_V$|TD||$C_V$|TD|\n|NSTM|0.395|0.427||0.374|0.803||0.432|0.832||0.412|0.487||0.393|0.455||0.398|0.892|\n|WeTe|0.383|0.949||0.401|0.947||0.425|0.989||0.388|0.908||0.370|0.920||0.376|0.752|\n|__FASTopic__|__0.426__|__0.983__||__0.437__|__0.999__||__0.457__|__1.000__||__0.422__|__0.998__||__0.420__|__0.998__||__0.439__   |__0.992__|\n\n|Model|20NG| | |NYT| | |WoS| |\n|:----|:----|:----|:----|:----|:----|:----|:----|:----|\n| |Purity|NMI| |Purity|NMI| |Purity|NMI|\n|NSTM|0.354|0.356| |0.447|0.229| |0.476|0.262|\n|WeTe|0.268|0.304| |0.526|0.279| |0.555|0.349|\n|__FASTopic__|__0.577__|__0.525__| |__0.662__|__0.369__| |__0.672__|__0.365__|\n\nWe see our model can surpass them too.\n\n\n__Q2: why solving the optimal transport is so fast__\n\nWe explain that __we employ the fast Sinkhorn's algorithm to solve the optimal transport and our objective is much simpler.__  \nAs mentioned in Section_3.3 and Appendix_D, the Sinkhorn's algorithm is fast and suited to the execution of GPU (Cuturi_2013, Peyré_2019, Genevay_2019).\nBesides, as indicated in Section_3.4, __our objective is straightforward without complicated networks. It only invovles four parameters: topic and word embeddings and their weight vectors (Eq(8)).__\nPrevious models, WeTe and ECRTM, also solve the optimal transport, but __their objectives rely on complicated encoders and decoders, which slows them down.__  \nThank you for the question. We've added more explanations in the paper.\n\n\n\n__Q3: is the comparison fair due to document embeddings?__\n\nWe clarify that __our comparisons are fair since we've included the baselines using the same document embeddings.__  \nBERTopic uses document embeddings for clustering; CombinedTM uses document embeddings as input features.\nThese baselines use exactly the same document embeddings as our method for fair comparisons.\nSome other models originally cannot incorporate document embeddings but use pretrained word embeddings, like ProGBN, HyperMiner, and ECRTM.\n\n\n\n\n__Q4: discuss LLM-based topic models__\n\nLLM-based topic models are very promising such as TopicGPT, but we explain __they currently have two main limitations__:\n1. __LLM-based topic models require more resources__. They need to input each document as prompts to LLMs. This is time-consuming and computationally intensive, especially when handling large-scale datasets.\n2. __LLM-based topic models cannot produce precise distributions for topics and documents__. They can only use natural languages to describe topics and documents.\n\nAs shown in the paper, our model has a fast running speed (Figure_1) and also provides precise distributions (Section_3.3).  \nThank you for the question. We have added more discussions in the related work.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "qHGdNjheVg", "reviewer": "Reviewer_ceW2", "summary": "This paper introduces a fast, adaptive, stable, and transferable (FAST) topic modeling paradigm by using dual semantic-relation reconstruction (DSR) to model topic-document and topic-word relations. It enhances topic modeling by incorporating a embedding transport plan (ETP) method to address relation biases. Experimental results show FASTopic outperforms existing baselines.", "questions": "1. In Figure~2, is it intentional to design the variation in line thickness ($\\pi_{11}$, $\\pi_{12}$ and $\\pi_{13}$) ?\n2. Could the authors provide more detailed motivation and experimental validation for the choice of using pretrained document embedding model? How if train embedding model together?\n3. What are the performance metrics (both training and inference) when FASTopic is run on a CPU environment?\n4. Although it is claimed that FASTopic performs well without extensive hyperparameter tuning, the paper does not discuss experiments varying hyperparameters. Could you elaborate on how sensitive FASTopic is to changes in its hyperparameters?", "limitations": "No potential negative societal impacts.", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "1. This work introduces FASTopic by integrating the DSR paradigm, which is straightforward and provides a fresh aspect on handling semantic relations in topic modeling. Authors further propose ETP method to avoid relation bias issue.\n2. Comprehensive experiments show FASTopic's superiority in effectiveness, efficiency, and adaptability, stability, and transferability.\n3. The paper is well-written, and the code is available.", "weaknesses": "1. The method's performance may heavily rely on the document embedding model, which makes it harder to interpret the model results. It need to futher discuss that how specific changes in embeddings affect topic modeling. \n2. The misplacement of tables and figures, such as Table~1, could potentially distract readers but do not detract significantly from the content's quality. But it does not matter.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your comments! We're glad that you believe our paper is well-written, our experiments are comprehensive, and our model is straightforward and fresh.  \nWe hope our responses can address your concerns and improve your rating.\n\n\n__Q1: how document embeddings affect the method__\n\nThank you for the question. We clarify that __we have experimented with different document embeddings in Appendix_F (Table_10 and 11).__  \nGenerally, higher-quality document embeddings bring about better topic modeling performance.  \nWe explain that __relying on pretrained document embeddings has become prevalent in topic modeling now.__ This is because we can easily access plentiful high-quality document embeddings, such as Sentence-Transformers which provides document embeddings for 50+ languages.\n\n\n__Q2: why use pretrained document embeddings and what if train them together__\n\nWe explain that __pretrained document embeddings contain abundant features, and using them is a common practice for topic modeling__, such as BERTopic, CombinedTM, and CTMneg. We don't need to retrain an embedding model, which saves us lots of effort.  \nAs mentioned in Section_3.4 Line_183, training the document embedding model together may lead to over-fitting problems, because our target datasets are regularly smaller than their pretraining datasets.\nThis also greatly increases the training time and computational cost. Due to these reasons, the above early studies don't train document embeddings either.\n\n\n\n__Q3: running speed performance on CPU__\n\nHere we report the running time (seconds) of BERTopic and our model, including training (Train) and inference (Infer):\n\n|Model|20NG| | |NYT| | |WoS| | |NeurIPS| | |ACL| | |Wikitext-103| |\n|:----|:----|:----|:----|:----|:----|:----|:----|:----|:----|:----|:----|:----|:----|:----|:----|:----|:----|\n| |Train|Infer| |Train|Infer| |Train|Infer| |Train|Infer| |Train|Infer| |Train|Infer|\n|BERTopic|107.06|12.05| |120.70|9.55| |128.16|6.51| |208.23|89.89| |252.29|126.93| |592.62|228.08|\n|__FASTopic__|95.93|0.02| |112.96|0.02| |121.88|0.01| |177.58|0.01| |209.75|0.01| |552.70|0.04|\n\n\nWe see that our model is slightly faster than BERTopic on training and has much shorter inference time.\nThis is because BERTopic has to extract and compare the n-grams in documents for inference, while our model directly uses the fast matrix calculations in Eq(9) for inference.  \nThe other performance (like topic quality) is the same on CPU, since the training process remains unchanged.\n\nThank you for the question. We've added these results to the paper.\n\n\n__Q4: hyperparameter sensitivity of the model__\n\nHere we report the results by varying the main hyperparameters $\\varepsilon_{1}$ and $\\varepsilon_{2}$ in Eq(5, 6):\n\n|$\\varepsilon_{1}$|$C_V$|TD|Purity|NMI|\n|:----|:----|:----|:----|:----|\n|0.1|0.469|0.997|0.659|0.363|\n|0.2|0.470|1.000|0.682|0.364|\n|0.333|0.457|1.000|0.672|0.365|\n|0.5|0.432|1.000|0.681|0.372|\n\n|$\\varepsilon_{2}$|$C_V$|TD|Purity|NMI|\n|:----|:----|:----|:----|:----|\n|0.1|0.424|1.000|0.669|0.353|\n|0.2|0.448|1.000|0.655|0.360|\n|0.333|0.435|1.000|0.677|0.368|\n|0.5|0.457|1.000|0.672|0.365|\n\n\nWe see that the performance remains generally stable.\nThank you for the question. We have added these results to the paper.\n\n\n__Q5: the line thickness in Figure 2__\n\nWe vary the line thickness to indicate the values of semantic relations.  \nThank you for the comment. We have added this to the Figure 2 caption.\n\n\n__Q6: the positions of some tables__\n\nThank you for the suggestion. We have updated their positions in the paper.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "kQ6wlB0Uqe", "reviewer": "Reviewer_YWcC", "summary": "The paper proposes FASTopic, a topic model using pre-trained document embeddings and embedding transport between documents and topics as well as words and topics. The minimized objective function is then a combination of the DSR and ETP which is optimized via finding topic, and word embeddings.", "questions": "- Are the created word embeddings semantically meaningful? Would clustering the word embeddings e.g. [7] give meaningfull topics? If yes, this is an additional benefit which I would suggest to include in the paper.\n\n\n\n[7] Sia, S., Dalmia, A., & Mielke, S. J. (2020). Tired of topic models? clusters of pretrained word embeddings make for fast and good topics too!. arXiv preprint arXiv:2004.14914.", "limitations": "Not needed", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- Interesting idea which is well documented\n- Initial experimental results show potential \n- potential benefit of getting interpretable word-embeddings while training topic model", "weaknesses": "- Overall, I quite like the paper and the idea. However, I have some reservations, primarily due to the aggressive tone of the presentation, starting from the abstract. The paper repeatedly criticizes comparable methods, while presenting its own method as the ultimate and superior topic model. Such claims need robust support through experimental evidence. Although the experiments provide some validation, the tone of the paper creates a discrepancy, making the experimental results appear insufficiently convincing.\n\n  Overall, more results are needed to back up the strong claims made in the paper.\n\n  Below are some suggestions that, in my opinion, would support the claims more effectively. If these suggestions are unjustified, I would appreciate an explanation.\n\n\n  **Minor:** I think the high-resolution plots are causing some trouble. I cannot really scroll through the paper smoothly.\n\n\n1. **Include further models** -> ETM [1], ProdLDA [2], CEDC [3], CTMneg [4]. While I wouldn't suspect ETM/ProdLDA to be better than any other of the models, the simple models CTMneg and CEDC have shown to outperform BERTopic.\n\n2. **Incorporate other evaluation metrics**: since evaluation is very difficult [5]. E.g. use some presented in [3] or [6].\n\n3. **How do you measure training time for all of the models?**\n   - 3.1) Are all steps, including the document encodings, part of the taken training time?\n   - 3.2) Do you use the same encoding model for all comparison models (where applicable)?\n   - 3.3) Given that FASTopic outperforms BERTopic in terms of speed, I wonder for how many epochs you are training your model, and what it is that takes so long in BERTopic? The document encoding steps are the same for both models and while dimensionality reduction takes some time using UMAP, I have a hard time believing that it is slower than training FASTopic for a reasonable amount of epochs.\n\n4. **How did you choose the number of topics for BERTopic?** Did you use Kmeans instead of HDBSCAN or hierarchicaly reduced the number of topics?\n\n5. **How many parameters does FASTopic have compared to the other neural models?**\n\n\n[1] Dieng, A. B., Ruiz, F. J., & Blei, D. M. (2020). Topic modeling in embedding spaces. Transactions of the Association for Computational Linguistics, 8, 439-453.\n\n[2] Srivastava, A., & Sutton, C. (2017). Autoencoding variational inference for topic models. arXiv preprint arXiv:1703.01488.\n\n[3] Adhya, S., Lahiri, A., Sanyal, D. K., & Das, P. P. (2023). Improving contextualized topic models with negative sampling. arXiv preprint arXiv:2303.14951.\n\n[4] Thielmann, A., Reuter, A., Seifert, Q., Bergherr, E., & Säfken, B. (2024). Topics in the haystack: Enhancing topic quality through corpus expansion. Computational Linguistics, 1-37.\n\n[5] Hoyle, A., Goel, P., Hian-Cheong, A., Peskov, D., Boyd-Graber, J., & Resnik, P. (2021). Is automated topic model evaluation broken? the incoherence of coherence. Advances in neural information processing systems, 34, 2018-2033.\n\n[6] Stammbach, D., Zouhar, V., Hoyle, A., Sachan, M., & Ash, E. (2023). Revisiting automated topic model evaluation with large language models. arXiv preprint arXiv:2305.12152.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your insightful reviews! We appreciate you find our work interesting and results potential.  \nWe hope our responses can address your concerns and improve your rating.\n\n__Q1: comparison to more models__\n\nWe emphasize __we have included the newest baselines__: HyperMiner(NeurIPS2022), ProGBN(ICML2023), ECRTM(ICML2023), GINopic(NAACL2024).  \nHere we report the comparisons to CTMneg and CEDC:\n\n|Model|20NG||NYT||WoS||NeurIPS||ACL||Wikitext-103||\n|:-|:-|:-|:-|:-|:-|:-|:-|:-|:-|:-|:-|:-|\n||$C_V$|TD|$C_V$|TD|$C_V$|TD|$C_V$|TD|$C_V$|TD|$C_V$|TD|\n|CTMneg|0.378|0.615|0.381|0.633|0.377|0.571|0.402|0.526|0.393|0.388|0.389|0.708|\n|CEDC|0.413|0.375|0.382|0.456|0.443|0.566|0.343|0.212|0.306|0.267|0.400|0.471|\n|__FASTopic__|__0.426__|__0.983__|__0.437__|__0.999__|__0.457__|__1.000__|__0.422__|__0.998__|__0.420__|__0.998__|__0.439__|__0.992__|\n\n\nWe see although CTMneg is better than CombinedTM, FASTopic still outperforms them.  \nThank you for mentioning these important work. We have updated the paper and cited them.\n\n__Q2: include more evaluation metrics__\n\nHere we additionally report the results of Word Embedding Coherence (WEC) and Inversed Rank-Biased Overlap (IRBO) from Adhya_2023.\nWEC measures coherence with pretrained word embeddings, and IRBO measures diversity.\n\n|Model|20NG| |NYT| |WoS| |NeurIPS| |ACL| |Wikitext-103| |\n|:-|:-|:-|:-|:-|:-|:-|:-|:-|:-|:-|:-|:-|\n||WEC|IRBO|WEC|IRBO|WEC|IRBO|WEC|IRBO|WEC|IRBO|WEC|IRBO|\n|LDA-Mallet|0.034|0.993|0.054|0.996|0.070|0.997|0.041|0.989|0.052|0.984|0.068|0.990|\n|NMF|0.040|0.987|0.028|0.978|0.081|0.983|0.045|0.971|0.045|0.970|0.069|0.976|\n|BERTopic|0.043|0.990|0.056|0.992|0.109|0.990|0.049|0.975|0.058|0.978|0.077|0.984|\n|CombinedTM|0.038|0.986|0.029|0.993|0.084|0.990|0.036|0.986|0.036|0.972|0.050|0.992|\n|GINopic|0.038|0.996|0.049|0.991|0.076|0.990|0.047|0.984|0.037|0.987|0.053|0.991|\n|ProGBN|0.044|0.967|0.052|0.961|0.103|0.985|0.042|0.886|0.054|0.964|0.070|0.940|\n|HyperMiner|0.038|0.988|0.052|0.990|0.077|0.990|0.047|0.997|0.045|0.997|0.065|0.988|\n|ECRTM|0.042|__1.000__|0.040|0.998|0.066|__1.000__|0.051|__1.000__|0.054|__1.000__|0.088|__1.000__|\n|__FASTopic__|__0.054__|__1.000__|__0.066__|__1.000__|__0.130__|__1.000__|__0.053__|__1.000__|__0.060__|__1.000__|__0.093__|__1.000__|\n\nThe above shows our model also performs better on these metrics.\nThank you for the suggestion. We have added these results to the paper.\n\n__Q3: the number of topics for BERTopic__\n\nWe explain __we set the number of topics of BERTopic to be the same as the other models for fair comparisons.__  \nEach model produces 50 topics in Table_1. Otherwise, it'll be unfair if BERTopic reduces the number of topics to 10.\n\n__Q4: how is the training time measured?__\n\n__We measure the training time from when the dataset is loaded until the training is finished.__\nThe training time includes loading document embeddings.\n\nWe clarify __we use the same document embeddings for each model when applicable for fair comparisons.__\nAs mentioned in Section_D Line_574, we use `all-MiniLM-L6-v2` for document embeddings by default.\n\n__Q5: why FASTopic is faster than BERTopic__\n\nAs mentioned in Appendix_D Line_578, we train FASTopic with 200 epochs.  \nWe break down the training time (seconds) as\n\n|BERTopic||\n|:-|:-|\n|Step 1: Load doc embeddings|7.10|\n|Step 2: Reduce dimensionality|23.13|\n|Step 3: Cluster doc embeddings|0.21|\n|Step 4: Compute word weights|1.97|\n|__Sum__|32.41|\n\n|FASTopic||\n|:-|:-|\n|Step 1: Load doc embeddings|7.10|\n|Step 2: Training|5.85|\n|__Sum__|12.95|\n\nWe see BERTopic has to reduce embedding dimensionality, cluster embeddings, and compute word weights.  \nIn contrast, our model enjoys faster training.\nThis is because it employs Sinkhorn's algorithm to solve optimal transport, which is quite fast as proven by previous studies (Cuturi_2013, Genevay_2019).\nMoreover, __its objective is simple and straightforward, optimizing only four parameters__: topic and word embeddings and their weight vectors (Eq(8)). This avoids the complicated encoders and decoders of VAE-based methods.  \nPrevious ECRTM also uses optimal transport, but its objective based on the complicated encoder and decoder slows it down.\n\nThank you for the question. We've added more explanations to the paper.\n\n__Q6: the number of hyperparameters in FASTopic__\n\nWe summarize the hyperparameters as follows:\n\n1. __FASTopic__ has hyperparameters for Sinkhorn's algorithm $\\varepsilon_{1}$ and $\\varepsilon_{2}$ in Eq(5,6) and the temperature in Eq(9).\n2. VAE-based models (CombinedTM, ProGBN, HyperMiner, ECRTM, GINopic) need more hyperparameters to set their encoders, decoders (dimensions, number of layers, and dropout), and prior distributions (Gaussian or Dirichlet).\nSome need more hyperparameters, like the hierarchy of ProGBN and HyperMiner, the regularization weight of ECRTM, and the graph neural networks of GINopic.\n1. BERTopic has hyperparameters to set its clustering and dimension reduction modules, like the number of neighbors and components of UMAP; the min cluster size, min samples, metrics of HDBSCAN.\n\n\n__We explain the fewer hyperparameters of our model result from its extremely simplified structure.__  \nWe don't use the traditional VAE with complicated encoders and decoders or the complex dimension reduction and clustering modules as BERTopic.\nOur model only includes three kinds of embeddings and the embedding transport plans between them.\n\nThank you for the comment. We've added more discussions to the paper.\n\n__Q7: cluster the word embeddings__\n\nWe clarify __our method already works as clustering the word embeddings.__  \nFigure_2(c) illustrates we can view our ETP as a clustering process: topic embeddings as cluster centers, word embeddings as samples, and the semantic relations between them as cluster assignments.\nWe refine these assignments during learning through objective Eq(8).\n\n__Q8: take longer time to load high-resolution figures__\n\nThank you for your comment. We've updated the paper with compressed figures.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: Summary
============
This paper parameterizes the creation of topic embeddings from semantic relations over word embeddings and semantic relations mapping fixed document embeddings to topic embeddings.

Justification
============
The method is fast and does well on an impressive, comprehensive set of *automated* metrics.  I like the approach, and the reviewers do a good job of outlining the strength of the approach.

Where I think the paper still falls short is:
* The paper does not compare against any conventional topic model approaches, which can be quite fast.  E.g., while Arora et al. is cited, they do not compare against this approach.  Stochastic variational inference could also be compared against, and it's unclear if they compare against Yao et al's fast Gibbs Sampling.
* There are now human evaluations.  While Hoyle et al. is cited, they do not follow the advice of the paper to vet automatic topic evaluations with human evaluations, particularly for neural models.
* The exposition could be clearer; I would have liked to have seen a little more intuition for describing the semantic relations (e.g., with a toy example of generating the representations).

Ultimately, I think this could be a *great* paper with another round of revision, but it isn't quite there yet.
Decision: Accept (poster)