FASTopic: Pretrained Transformer is a Fast, Adaptive, Stable, and Transferable Topic Model

Authors: xiaobao  wu
thong  nguyen
delvin ce zhang
william yang wang
anh tuan luu
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-14 14:37:57

Abstract: Topic models have been evolving rapidly over the years, from conventional to recent neural models. However, existing topic models generally struggle with either effectiveness, efficiency, or stability, highly impeding their practical applications. In this paper, we propose FASTopic, a fast, adaptive, stable, and transferable topic model. FASTopic follows a new paradigm: Dual Semantic-relation Reconstruction (DSR). Instead of previous conventional, VAE-based, or clustering-based methods, DSR directly models the semantic relations among document embeddings from a pretrained Transformer and learnable topic and word embeddings. By reconstructing through these semantic relations, DSR discovers latent topics. This brings about a neat and efficient topic modeling framework. We further propose a novel Embedding Transport Plan (ETP) method. Rather than early straightforward approaches, ETP explicitly regularizes the semantic relations as optimal transport plans. This addresses the relation bias issue and thus leads to effective topic modeling. Extensive experiments on benchmark datasets demonstrate that our FASTopic shows superior effectiveness, efficiency, adaptivity, stability, and transferability, compared to state-of-the-art baselines across various scenarios.

Main paper: 
 introduction: 
 Due to the unsupervised fashion and interpretability, topic models have derived a broad spectrum of applications [12, 14] , such as content recommendation [39, 79] , generation [17, 88] , and trend analysis [18, 36] . Early conventional topic models follow probabilistic graphical models [10, 8] or non-negative matrix factorization [35, 58] . But they rely on laborious model-specific derivations and fall short on large-scale data [75] . Due to this, recent neural topic models have attracted more attention [90, 75] , including VAE-based [42, 43, 61] and clustering-based [59, 89, 24] .However, existing neural topic models lack either efficiency, effectiveness, or stability. First, VAEbased topic models, while effective, are limited by their low efficiency. They follow the VAE framework [31] and often incorporate extra modules like graph neural networks [87, 1] or external knowledge [66, 80] , resulting in complicated modeling structures. Owing to this, they suffer from intensive time complexity, e.g., consuming hours to process a dataset of 10k documents [65] . Second, clustering-based topic models [89, 24] excel in efficiency as they require no training, but they sacrifice effectiveness. They tend to yield repetitive topics other than desired diverse ones or infer inaccurate topic distributions of documents [73, 1] . What is worse, these neural topic models suffer from low performance stability. They are extremely sensitive to hyperparameters, especially when applied to various scenarios concerning data domains, vocabulary sizes, and document length [27] . In consequence, these challenges hinder the applications of topic modeling in practice.To tackle these challenges, we in this paper propose a Fast, Adaptive, Stable, and Transferable topic model (FASTopic) . Different from existing conventional, VAE-based, or clustering-based approaches, we introduce a new paradigm for topic modeling: Dual Semantic-relation Reconstruction (DSR) as illustrated in Figure 2 . Instead of complicated neural networks, DSR only considers three parameters: document embeddings from a pretrained Transformer, and topic and word embeddings. DSR models the dual semantic relations between (1) document and topic embeddings, and (2) topic and word embeddings, and interprets them as distributions for topic modeling. By reconstruction with these relations, DSR discovers latent topics in a neat and efficient framework, avoiding the complicated structures in prior studies. To model these relations, we further propose the novel Embedding Transport Plan (ETP). Rather than simple parameterized softmax [37, 21] , ETP regularizes these relations as the optimal transport plans between document, topic, and word embeddings. This mitigates the relation bias issue and produces distinct topics and accurate topic distributions, enabling effective topic modeling. Following the DSR paradigm with ETP, FASTopic provides a solid solution to the challenges of current topic models. As reported in Figure 1 , FASTopic shows both superior efficiency and effectiveness compared to state-of-the-art baselines. Additionally FASTopic shows high Some recent studies leverage large language models and describe topics as conceptual descriptions [53] , rather than the word distributions in LDA [10] . They can reach higher interpretability, but we emphasize their two limitations: (i) They require more resources. They need to input each document as prompts to LLMs. This is time-consuming and computationally intensive, especially when handling large-scale datasets. (ii) They cannot produce precise distributions for topics and documents, which limits their applications in downstream tasks.3 Methodology: FASTopicIn this section, we first recall the problem setting of topic modeling. Then we propose the new paradigm Dual Semantic-relation Reconstruction (DSR) and the novel Embedding Transport Plan (ETP) method. Finally we introduce our new FASTopic. 
problem setting and notations: 
 Consider a collection {x (1) , . . . , x (N ) } with N documents and vocabulary size V . Topic modeling targets to discover K latent topics from the collection. Following LDA [10] , Topic#k is defined as a distribution over all words, i.e., topic-word distribution, denoted as β k ∈R V . We have β = (β 1 , . . . , β K )∈R V ×K as the topic-word distribution matrix of all topics. Topic modeling also infers the topic distributions of a document (what topics a document contains), i.e., doc-topic distribution. We denote the doc-topic distribution of x (i) as θ (i) ∈∆ K , with ∆ K as a probability simplex. [63] of document (■), and topic (▲) embeddings under 50 topics (K=50). While most topic embeddings gather together in Parameterized Softmax (a,c) as it causes biased relations, ETP (b,d) separates all topic embeddings with regularized relations, avoiding the bias issue.In this section, we propose a new, neat, and efficient paradigm for topic modeling, Dual Semanticrelation Reconstruction (DSR). Figure 2 illustrates the differences between DSR and previous VAE-based and clustering-based methods.Parameterizing Documents, Topics, and Words At the beginning, we parameterize documents, topics, and words as embeddings. Specifically, we embed documents into an H-dimensional semantic space via a pretrained Transformer f doc , e.g., BERT [16] or Sentence-BERT [54] . Let D=(d 1 , . . . , d N )∈R H×N denote all document embeddings, where d i =f doc (x (i) ) refers to the embedding of i-th document. Then we randomly project all topics and words into the same semantic space as K topic embeddings T=(t 1 , . . . , t K )∈R H×K and V word embeddings W=(w 1 , . . . , w V )∈R H×V . Notably we do not use pretrained word embeddings like word2vec [44] or GloVe [51] , because they may not belong to the same semantic space as document embeddings, which hinders correctly measuring their distance.Reconstruction through Dual Semantic Relations Then we model the dual semantic relations between the embeddings of (1) documents and topics, and (2) topics and words. We interpret these relations as doc-topic distributions and topic-word distributions respectively. To be specific, we model θ(i)k , the probability of Topic#k given i-th document as the semantic relation between embeddings d i and t k . Similarly, we model β jk , the probability of j-th word given Topic#k as the semantic relation between embeddings t k and w j . We detail how to model them later in Sec. 3.3. We learn these relations through reconstruction. As shown in Figure 2 , we reconstruct document x (i) as βθ (i) , where we transport the semantics from x (i) to each topic through θ (i) and then from each topic to each word through β. Hence we formulate the objective for DSR as the reconstruction error:EQUATIONwhere we transform x (i) into the Bag-of-Words following previous studies [42, 43, 61] . By minimizing this objective, we expect to push each topic embedding close to the embeddings of its semantically related documents and words; therefore we can learn informative semantic relations, i.e., meaningful topic-word distributions β for latent topics and doc-topic distributions θ (i) for documents.The above DSR presents a neat and efficient topic modeling paradigm. Previous VAE-based methods incorporate complicated modeling structures with diverse objectives [5, 80, 73, 1] . Different from them, DSR solely includes the objective Eq. ( 1) that only involves the embeddings of documents, topics, and words. This sufficiently simplifies the topic modeling procedure and hence facilitates efficiency. Moreover, while prior clustering-based methods [24] depend on indirect approximations, DSR explicitly models topic-word distributions and doc-topic distributions, resulting in higher effectiveness (See experiments in Sec. 4).In this section, we analyze how to model the semantic relations for topic modeling, and then propose a new solution Embedding Transport Plan (ETP).How to Model Semantic Relations? We emphasize this is a non-trivial problem. A common answer is the widely-used parameterized softmax function [28, 37, 21] :EQUATIONwhere we measure the relation between embeddings as their Euclidean distance with hyperparameter τ . Unfortunately, this straightforward way is ineffective, because it incurs the relation bias issue: most relations are minor as quantitatively illustrated in Figure 3a . In consequence, most topic embeddings fail to cover informative and distinct semantics but gather together in the space as shown in Figures 3c and 7a . This issue leads to repetitive topics and less accurate doc-topic distributions (See ablation studies in Sec. 4.7). Someone may guess this issue is because of the large topic number. We note that the relation bias issue still happens even under a small number of topics (See experimental supports in Table 9 ). Owing to these, we need alternatives to model the semantic relations.Transport Plan from Documents to Topics Motivated by the above analysis, we propose the new Embedding Transport Plan (ETP) to address the relation bias issue with effective regularization.To regularize relations, we model them as the transport plan of a specifically defined optimal transport problem. In detail, we define two discrete measures γ 1 and ρ 1 over document and topic embeddings:γ 1 = N i=1 1 N δ di and ρ 1 = K k=1 s k δ t k, where δ x denotes the Dirac unit mass on x. We set the weight of each document embedding as 1/N and the weight of each topic embedding as s k , where s=(s 1 , . . . , s K ) is a weight vector summing to 1. This later produces normalized doc-topic distributions. With these two, we formulate their entropic regularized optimal transport problem as arg minEQUATION)The first term is the original optimal transport problem, and the second term is the entropic regularization with hyperparameter ε 1 to make this problem tractable [13, 52] . This equation aims to find a transport plan π that minimizes the total cost of transporting the weights of document embeddings to topic embeddings under the two conditions [15] , where 1 K denotes a K-dimensional column vector of ones. Here π ik refers to the transport weight from d i to t k . The transport cost between them is measured as Euclidean distance C(1) 1) as the transport cost matrix. We can alleviate the relation bias issue with transport plan π as the semantic relations. Eq. (3) constraints π by two conditions. We set s = softmax(s 0 ), where s 0 is a learnable variable uniformly initialized as 1 K 1 K . The uniform initialization and softmax function prevent excessively biased s [28] . Therefore as illustrated in Figures 3b and 3d , this avoids biased π as it is constrained by s, which mitigates the relation bias issue (See experiment results in Sec. 4.7). Besides, this approach flexibly captures the varying weight of each topic within the document collection, since some topics may appear more frequently in the collection while others less, which aligns with the reality (See more interpretations in Appendix E).ik =∥d i -t k ∥ 2 , with C (Transport Plan from Topics to Words We further employ the transport plan between topic and word embeddings. We define two discrete measures over topic and word embeddings:γ 2 = K k=1 1 K δ t k and ρ 2 = V j=1 u j δ wj .Here we specify the weight of each topic embedding as 1/K and the weight of each word embedding as u j , where u is a weight vector and its sum is 1. This is to produce normalized topic-word distributions later. Following Eq. (3), we write the entropic regularized optimal transport problem between the two measures as arg min Here ϕ is the transport plan between topic embeddings and word embeddings, and ϕ kj denotes the weight to be transported from topic embedding t k to word embedding w j . We measure the transport cost as Euclidean distance Cϕ∈R K×V + L OT (γ 2 , ρ 2 ; ε 2 )= K k=1 V j=1 C (2) kj ϕ kj +εϕ kj (log ϕ kj -1), s.t. ϕ1 K = 1 K K and ϕ ⊤ 1 K =u. (4)C V TD C V TD C V TD C V TD C V TD C V TD LDA-Mallet ‡ 0.(2)kj =∥t k -w j ∥ 2. Similar to the above, we set u = softmax(u 0 ), where u 0 is a learnable variable uniformly initialized as 1 V 1 V . We model the semantic relations between topics and words with ϕ to mitigate the relation bias issue.Objective for ETP With the above transport plans as semantic relations, we write θ (i) the doc-topic distribution of document x (i) asEQUATIONWe employ Sinkhorn's algorithm [60, 15, 52] to compute the approximated solution π * of the optimal transport problem in Eq. ( 3). As proved in early studies [57, 22, 23] , π * becomes a differentiable variable parameterized by transport cost matrix C (1) , which thus enables gradient backpropagation. See algorithm details in Appendix C. Here we rescale π * i by N to output normalized θ (i) (the sum of each row in π * is 1/N as previously constrained). In the same way, we model the topic-word distribution matrix β asEQUATIONWe rescale ϕ * by K to produce normalized β (the sum of each row in ϕ * is 1/K as previously constrained). We formulate the objective for ETP by minimizing the total transport cost weighted by these approximated transport plans asEQUATIONEQUATIONThis objective refines the embeddings by the regularized semantic relations, i.e., the approximated transport plans under constraints.Combining Eq. ( 1) and Eq. ( 7), we formulate the overall objective for FASTopic as min T,W,s,uEQUATIONIn short, L ETP refines topic and word embeddings with the regularized semantic relations; L DSR learns these relations by reconstruction. To reduce the number of hyperparameters, we set the weights of these two objectives as equal by default. See the training algorithm of FASTopic in Appendix C.This objective is extremely simple compared to previous work with complex encoders and decoders following VAE [42, 61, 75] . It only optimizes four parameters: topic and word embeddings T, W, and their weights s and u. We freeze document embeddings D, because they have been pretrained already, and we may encounter over-fitting problems if we fine-tune them on a relatively small dataset. Due to this simple objective, our FASTopic enjoys super fast training. Previous models like ECRTM We use topic models trained on Wikitext-103 to infer the doc-topic distributions of other datasets. The best is in bold.[73] also solve the optimal transport problem, but their objectives involve complicated encoders and decoders inherent from VAE, which slows them down.Moreover, FASTopic needs much fewer hyperparameters. It mainly has hyperparameters for Sinkhorn's algorithm ε 1 and ε 2 in Eq. ( 3) and (4). VAE-based models, like CombinedTM [5] , ECRTM [73] , and GINopic [1] , require hyperparameters to set their encoders, decoders (dimensions, number of layers, and dropout), and prior distributions (Gaussian or Dirichlet). BERTopic needs hyperparameters to set its clustering and dimension reduction modules, like the number of neighbors and components of UMAP; the min cluster size, min samples, metrics of HDBSCAN.Finally we discuss how to infer doc-topic distributions for new documents. Considering a new document x ′ and its document embedding d ′ =f doc (x ′ ), we may directly follow the learning process in Sec. 3.3 and infer its doc-topic distribution θ ′ by computing the transport plan between d ′ and learned topic embeddings T. Unfortunately, this way is unworkable. It transports the weight of one document to all topics; hence for any x ′ , the transport plan invariably becomes the learned topic weights s. Such trivial results are certainly unreasonable. To this end, we compute θ ′ asEQUATIONwith τ as a temperature hyperparameter.  
experiment: 
 In this section we conduct comprehensive experiments to demonstrate that our FASTopic is fast, adaptive, stable, and transferable. Evaluation Metrics Although topic modeling evaluation is still an open problem [26] , we follow mainstream studies [19, 91, 73] and evaluate the topic quality and doc-topic distribution quality.Model K=75 K=100 K=125 K=150 K=175 K=200 C V TD C V TD C V TD C V TD C V TD C V TD LDA-Mallet ‡ 0.For topic quality, we consider: (i) Topic Coherence measures the coherence between top words of discovered topics. We employ the widely-used coherence metric C V , which has been shown to outperform earlier NPMI, UCI, and UMass [46, 34, 56] . We use a widely-used large Wikipedia article collection as the external reference corpus to compute C V . (ii) Topic Diversity means the differences between discovered topics. We measure this with the Topic Diversity (TD) metric [19] , which calculates the proportion of unique words in the topics. In terms of doc-topic distribution quality, we conduct document clustering, evaluated by Purity and NMI [38] following Zhao et al. [91] . We do not evaluate the perplexity since our method does not follow the VAE framework [42, 43] , and the perplexity is incomparable across topic models as evidenced by early studies [90, 75] .Baseline Models We consider the following baselines in three paradigms. For conventional topic models, we adopt (i) LDA-Mallet [40] , a prominent method competitive to some neural models [26] ; (ii) NMF, using non-negative matrix factorization. For clustering-based topic models, we have (iii) BERTopic [24] , clustering document embeddings and discovering topics by TF-IDF.For VAE-based neural topic models, we include (iv) CombinedTM [5] , combining contextual features and BoW as inputs; (v) GINopic [1] , following CombinedTM but using graph isomorphism networks; (vi) HyperMiner [80] , using hyperbolic embeddings to model topics; (vii) ProGBN [20] , progressively generating documents of different levels with graph decoders; (viii) ECRTM [73] ,a state-of-the-art method by regularizing embeddings with optimal transport. We fine-tune the hyperparameters of these baselines under different datasets and topic numbers.We demonstrate the superior effectiveness of FASTopic compared to state-of-the-art baselines. Table 1 presents the topic quality results of topic coherence (C V ) and topic diversity (TD). We see that our FASTopic commonly surpasses all baselines with the highest performance across all datasets. Moreover, Table 2 reports the doc-topic distribution quality results concerning the Purity and NMI of document clustering. We observe that FASTopic reaches top performance as well. These results manifest that FASTopic produces high-quality topics and doc-topic distributions, showing better effectiveness. This also verifies the capability of our new DSR paradigm. See Appendix H for the examples of discovered topics.We consider text classification as a downstream task to evaluate topic models in an extrinsic manner. Following Wu et al. [73] , we train SVM classifiers with the inferred doc-topic distributions as document features and then predict the class of each testing document. We measure this performance by Accuracy (Acc) and F1. Figure 4 reports that our FASTopic consistently outperforms baselines. We note that the improvements of FASTopic are statistically significant at 0.01 level. These results demonstrate that FASTopic can benefit more downstream classification tasks.We show the exceptionally fast running speed of FASTopic. Table 3 reports the running time of each model on each dataset. The running time indicates the duration from the completion of data loading to the finish of training. We see that our FASTopic consistently emerges as the fastest one by a large margin, statistically significant at 0.01 level. FASTopic completes running within 1 minute, while the longest takes 30 minutes. We notice that LDA-Mallet has increasing running time on the datasets with longer documents. For instance, it escalates from 50 seconds on 20NG to 2000 seconds on Wikitext-103. In contrast, FASTopic maintains its rapid performance regardless of document length. Figure 1b also evidences the fast speed of FASTopic in terms of varying dataset sizes. This is because FASTpoic adopts our neat and efficient DSR paradigm, which relieves from the complicated modeling structures in previous studies. See more running time analysis in Appendix G.We verify the high transferability of FASTopic. In detail, we train a topic model on Wikitext-103, a general dataset with diverse topics, and then use it to infer the doc-topic distributions of documents in other datasets 20NG, NYT, and WoS. Following the previous setting, we use these doc-topic distributions as features to train SVM classifiers for text classification. This measures the transferability of a topic model from one data domain to another. Figure 4 shows that the transferability of FASTopic significantly outperforms baselines. The reason lies in that previous methods often rely on the Bag-of-Words [80, 20, 73] . Differently, FASTopic leverages richer representations, the pretrained document embeddings, and learns the doc-topic distributions through the effective ETP method, bringing about higher transferability.We demonstrate the adaptivity and stability of FASTopic across various scenarios using the WoS dataset. First, Tables 4 and 5 summarize the performance under different topic numbers (K from 75 to 200). We observe that FASTopic generally remains top performance across these variations. Second, Tables 13 and 14 report the results under varying dataset sizes (N from 15k to 40k). These results show that FASTopic mostly reaches the best results. As aforementioned in Figure 1b , FASTopic also has the fastest running speed. Third, we experiment with different vocabulary sizes (V from 20k to 50k) in Tables 15 and 16 . Similarly, our FASTopic exhibits stable and high performance. We note that FASTopic uses the same hyperparameters in all these experiments (See Appendix D). The above results together highlight that our FASTopic can smoothly adapt to various scenarios with stable performance. This is a vital advantage of our model for practical applications.We validate the necessity of our Embedding Transport Plan (ETP) method with ablation studies. Table 6 shows that using parameterized softmax rather than ETP (w/o ETP) to model semantic relations incurs degraded performance, concerning both topic and doc-topic distribution quality (See also the results on other datasets in Table 8 ). For instance, the C V and TD decrease from 0.426, 0.983 to 0.368, 0.391; the Purity and NMI decrease from 0.577, 0.525 to 0.401, 0.452, indicating low-quality repetitive topics and less accurate doc-topic distributions. We observe similar results even with only 10 topics (K=10) in Table 9 . This is because our ETP properly regularizes semantic relations, which addresses the relation bias issue. These results manifest the necessity of our ETP to reach effective topic modeling. 
model usage: 
 We have released our FASTopic as a Python package at PyPI2 . Users can easily install FASTopic through pip.  
conclusion: 
 In  
experiment statistical significance: 
 Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?Answer: [Yes]Justification: We report statistical significance tests in Sec. 4.Guidelines:• The answer NA means that the paper does not include experiments.• The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper. • The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions). • The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.) • The assumptions made should be given (e.g., Normally distributed errors).• It should be clear whether the error bar is the standard deviation or the standard error of the mean. Guidelines:• The answer NA means that the paper does not include experiments.• The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage. • The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute. • The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper). 
code of ethics: 
 Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines? 
broader impacts: 
 Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?Answer: [Yes] Justification: We discuss societal impacts in Sec. 1.Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.• The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?Answer: [NA]Justification:Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?Answer: [Yes] Justification: We cite the original sources of code, data, and models in Sec. 4.1.Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided. • If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. • For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.• If this information is not available online, the authors are encouraged to reach out to the asset's creators. • We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution. • For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review. 
: 
 , modeling doc-topic distributions as the semantic relations between document (■) and topic embeddings (▲), and modeling topic-word distributions as the semantic relations between topic (▲) and word embeddings (•). Here we model these relations as the transport plans to alleviate the relation bias issue. 
 
 Appendix: Our method achieves promising performance, but we mention that one limitation is the max input length of pretrained document embedding models. This may hamper the performance on extremely long documents. However, we show that our method can well handle long documents like academic papers in Sec. 4, and this issue can be well resolved by newer and stronger pretrained models like large language models [48, 50, 78, 49] .Training algorithm for FASTopic.Input: document collection {x (1) , x (2) , . . . , x (N ) }, pretrained document embedding model f doc ; Output: model parameters T, W, s, u; M = exp(-C/ε); 5: b ← 1 L2 ; 6:while not converged and not reach max iterations do 7:a ← 1 L1 1 L 1 Mb , b ← v M ⊤ a ; 8:end while9:return diag(a)M diag(b); 10: end function π * = ETP(C (1) , s, ε 1 , N , K); // Transport plan between documents and topics11: Initialize s 0 ← 1 K 1 K ; 12: Initialize u 0 ← 1 V 1 V ; 13: Initialize d i ← f doc (x i ), for i = {1,// Embedding transport between topics and words; ϕ * = ETP(C (2) , u, ε 2 , K, V ,); // Transport plan between topics and words 24:β = Kϕ * 25: θ (i) = N π * i , for i = {1, 2, . . . , N } 26:Compute Eq. ( 8);27:Update T, W, s, u with a gradient step; 28: end for EQUATIONAlgorithm 1 shows the training algorithm of FASTopic. ETP uses Sinkhorn's algorithm [60, 15] to compute approximated transport plan π * and ϕ * . It is iterative and fast, especially suited to the execution of GPU [23] .See our code for implementation details.We conduct our experiments with A6000 GPU. We use all-MiniLM-L6-v2 in Sentence-Transformers 5 to obtain the pretrained document embeddings, as it is fast and also offers good quality. See results with other embedding models in Appendix F. We set the maximum number of iterations as 1,000 and the stop tolerance as 0.005 for the Sinkhorn's algorithm [15] . We set ε 1 as 1/3 and ε 2 as 1/2. We set τ as 1.0 in Eq. ( 9). We optimize the model parameters through Adam [30] with 200 epochs and learning rate as 0.002. We highlight that we use the above same hyperparameters for all reported experiments to demonstrate the stability of our FASTopic.As aforementioned, our FASTopic learns s, the weight of each topic within the document collection and u, the weight of each word within all topics. Figure 8 plots the relationship between the topic/word weights and the word frequency in the collection. Generally, topics with higher weights include top words of higher frequency, and words with higher weights also exhibit higher frequency. These observations confirm the validity of learned topic and word weights.We investigate the influence of document embedding models. Apart from the mentioned all-MiniLM-L6-v2, we also experiment with all-mpnet-base-v2 and all-distilroberta-v1 as document embedding models in Tables 10 and 11 . We notice that the performance is overall stable across these models. Moreover, the performance grows with all-mpnet-base-v2 and all-distilroberta-v1, especially on document clustering. This is because these two produce document embeddings of relatively higher quality [54] .To precisely compare BERTopic and FASTopic, we break down their running time. Table 12 shows that they both load document embeddings, but BERTopic takes more steps. BERTopic has to reduce embedding dimensionality, cluster embeddings, and compute word weights; in contrast, our FASTopic enjoys faster training. This is because FASTopic employs Sinkhorn's algorithm to solve the optimal transport, which is quite fast as proven by previous studies [15, 22] . Moreover, its objective is simple and straightforward as it only optimizes four parameters: topic and word embeddings and their weight vectors (Eq. ( 8). Here we list all the discovered topics of different models. Compared to prior CombinedTM and BERTopic, FASTopic anchors topics with more specific and relevant words. Specifically in the NeurIPS dataset, the topics of CombinedTM and BERTopic usually contain high-frequency words like "algorithm", "data", "model", "learning", "neural", and "network". Admittedly they are related to some extent, but they are too general to anchor topic semantics [3, 4] . For instance, Topic#30 of BERTopic is about chip manufacturing, but contains general words like "neural", and "weight". In contrast, Topic#28 of FASTopic includes more specific ones like "silicon", "transistor", "cmos", and "neuromorphic". In the Wikitext-103 dataset, Topic#7 of BERTopic is about species, but has general words "known", "long", "large", "small", and "white". Differently Topic#2 FASTopic covers specific words "breeding", "prey", "subspecies", and "predators". See quantitative results of discovered topics in Sec. 4.2.# Guidelines:• The answer NA means that the paper does not include experiments.• The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them. • The full details can be provided either with the code, in appendix, or as supplemental material.Justification: We conform with the NeurIPS Code of Ethics.Guidelines:• The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.• If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics. • The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).https://pypi.org/project/fastopic/.https://github.com/bobxwu/FASTopichttps://github.com/bobxwu/topmosthttps://huggingface.co/sentence-transformers 


Decision: Summary
============
This paper parameterizes the creation of topic embeddings from semantic relations over word embeddings and semantic relations mapping fixed document embeddings to topic embeddings.

Justification
============
The method is fast and does well on an impressive, comprehensive set of *automated* metrics.  I like the approach, and the reviewers do a good job of outlining the strength of the approach.

Where I think the paper still falls short is:
* The paper does not compare against any conventional topic model approaches, which can be quite fast.  E.g., while Arora et al. is cited, they do not compare against this approach.  Stochastic variational inference could also be compared against, and it's unclear if they compare against Yao et al's fast Gibbs Sampling.
* There are now human evaluations.  While Hoyle et al. is cited, they do not follow the advice of the paper to vet automatic topic evaluations with human evaluations, particularly for neural models.
* The exposition could be clearer; I would have liked to have seen a little more intuition for describing the semantic relations (e.g., with a toy example of generating the representations).

Ultimately, I think this could be a *great* paper with another round of revision, but it isn't quite there yet.

Reviews: review: paper_summary: paper_summary: The author found that existing methods (VAE-based or clustering) suffer from low efficiency, poor quality of topic words, and instability. To address these issues, this paper proposes a novel topic modeling paradigm called Dual Semantic-Relation Reconstruction (DSR) for efficient modeling of semantic relations among three types of embeddings: document, topic, and word embeddings. The author attributes the low quality and instability of previous methods to relation bias issues, leading to repetitive topics and inaccurate document-topic distributions. To tackle this, the paper introduces the Embedding Transport Plan (ETP) to regularize relations among the three embeddings. Combined, DSR and ETP form the proposed topic model, FASTopic, which is evaluated on six benchmark datasets, showing encouraging performance.
strength_weakness: strength_weakness: 1. Tables 6, 8, and 9 present the ablation study of FASTopic using ETP and parameterized softmax. ETP, compared to the ECR used in the previous VAE-based method ECRTM, adds semantic relations between topics and documents. The paper lacks proof the dual transmission effectiveness of ETP by not comparing it with ECR (which only includes semantic relations between topics and words) and another case that only includes semantic relations between documents and topics.
2. I am concerned that while the DSR training method improves efficiency compared to VAE-based methods, it may make it difficult to build meaningful relations between topic embeddings and document embeddings for some long-text corpora due to its singular objective.
3. It would be nice to showcase a comparison of topic words or transferability between FASTopic and ECRTM to demonstrate that FASTopic offers improvements in multiple dimensions, not just speed.
questions: questions: 1. In Line 76, the author thinks that methods like Topicgpt, which uses LLM to describe topics, deviate from the original LDA setting. Can the author provide a more detailed explanation in this section?
2. In Line 132, it is mentioned that some studies think that repetitive topics and less accurate doc-topic distributions are due to a large number of topics (K being set too high). The paper, however, attributes these issues mainly to relation bias. Does a large number of topics also affect FASTopic in the same way?
limitations: limitations: 1. The proposed method relies on embedding transport semantic relations and may be limited by the max input length of pre-trained document embedding models.
2. See Weakness 3.
paper_summary: paper_summary: This paper proposes a fast, adaptive, stable, and transferable topic model, FASTopic. Instead of using the VAE or clustering method, it incorporates a new model structure named Dual Semantic-relation Reconstruction (DSR). DSR learns topics by directly optimizing the semantic relations among topics, documents, and words. The semantic relations are further regularized by an Embedding Transport Plan (ETP) method as an optimal transport problem. Experiments demonstrate the effectiveness of the proposed model.
strength_weakness: strength_weakness: 1. [76,64] should be compared as baselines in the experiments since they both incorporate optimal transport objectives in their model as FASTopic does.
2.  As far as I know, the time complexity of solving the optimal transmission problem is very high. Is there any technique used in FASTopic to efficiently derive a solution? Furthermore, it would be great to theoretically analyze the time complexity of the current FASTopic framework.
3. Is it fair to compare FASTopic, which takes document embeddings as input extracted by sentence-BERT, with other baselines in the experiments?
4. As LLM has shown great performances in many NLP tasks, I believe it is necessary to include a discussion about the reasons and advantages of developing a topic model that is not LLM-based.
questions: questions: See weaknesses.
limitations: limitations: Yes
paper_summary: paper_summary: This paper introduces a fast, adaptive, stable, and transferable (FAST) topic modeling paradigm by using dual semantic-relation reconstruction (DSR) to model topic-document and topic-word relations. It enhances topic modeling by incorporating a embedding transport plan (ETP) method to address relation biases. Experimental results show FASTopic outperforms existing baselines.
strength_weakness: strength_weakness: 1. The method's performance may heavily rely on the document embedding model, which makes it harder to interpret the model results. It need to futher discuss that how specific changes in embeddings affect topic modeling. 
2. The misplacement of tables and figures, such as Table~1, could potentially distract readers but do not detract significantly from the content's quality. But it does not matter.
questions: questions: 1. In Figure~2, is it intentional to design the variation in line thickness ($\pi_{11}$, $\pi_{12}$ and $\pi_{13}$) ?
2. Could the authors provide more detailed motivation and experimental validation for the choice of using pretrained document embedding model? How if train embedding model together?
3. What are the performance metrics (both training and inference) when FASTopic is run on a CPU environment?
4. Although it is claimed that FASTopic performs well without extensive hyperparameter tuning, the paper does not discuss experiments varying hyperparameters. Could you elaborate on how sensitive FASTopic is to changes in its hyperparameters?
limitations: limitations: No potential negative societal impacts.
paper_summary: paper_summary: The paper proposes FASTopic, a topic model using pre-trained document embeddings and embedding transport between documents and topics as well as words and topics. The minimized objective function is then a combination of the DSR and ETP which is optimized via finding topic, and word embeddings.
strength_weakness: strength_weakness: - Overall, I quite like the paper and the idea. However, I have some reservations, primarily due to the aggressive tone of the presentation, starting from the abstract. The paper repeatedly criticizes comparable methods, while presenting its own method as the ultimate and superior topic model. Such claims need robust support through experimental evidence. Although the experiments provide some validation, the tone of the paper creates a discrepancy, making the experimental results appear insufficiently convincing.

  Overall, more results are needed to back up the strong claims made in the paper.

  Below are some suggestions that, in my opinion, would support the claims more effectively. If these suggestions are unjustified, I would appreciate an explanation.


  **Minor:** I think the high-resolution plots are causing some trouble. I cannot really scroll through the paper smoothly.


1. **Include further models** -> ETM [1], ProdLDA [2], CEDC [3], CTMneg [4]. While I wouldn't suspect ETM/ProdLDA to be better than any other of the models, the simple models CTMneg and CEDC have shown to outperform BERTopic.

2. **Incorporate other evaluation metrics**: since evaluation is very difficult [5]. E.g. use some presented in [3] or [6].

3. **How do you measure training time for all of the models?**
   - 3.1) Are all steps, including the document encodings, part of the taken training time?
   - 3.2) Do you use the same encoding model for all comparison models (where applicable)?
   - 3.3) Given that FASTopic outperforms BERTopic in terms of speed, I wonder for how many epochs you are training your model, and what it is that takes so long in BERTopic? The document encoding steps are the same for both models and while dimensionality reduction takes some time using UMAP, I have a hard time believing that it is slower than training FASTopic for a reasonable amount of epochs.

4. **How did you choose the number of topics for BERTopic?** Did you use Kmeans instead of HDBSCAN or hierarchicaly reduced the number of topics?

5. **How many parameters does FASTopic have compared to the other neural models?**


[1] Dieng, A. B., Ruiz, F. J., & Blei, D. M. (2020). Topic modeling in embedding spaces. Transactions of the Association for Computational Linguistics, 8, 439-453.

[2] Srivastava, A., & Sutton, C. (2017). Autoencoding variational inference for topic models. arXiv preprint arXiv:1703.01488.

[3] Adhya, S., Lahiri, A., Sanyal, D. K., & Das, P. P. (2023). Improving contextualized topic models with negative sampling. arXiv preprint arXiv:2303.14951.

[4] Thielmann, A., Reuter, A., Seifert, Q., Bergherr, E., & Säfken, B. (2024). Topics in the haystack: Enhancing topic quality through corpus expansion. Computational Linguistics, 1-37.

[5] Hoyle, A., Goel, P., Hian-Cheong, A., Peskov, D., Boyd-Graber, J., & Resnik, P. (2021). Is automated topic model evaluation broken? the incoherence of coherence. Advances in neural information processing systems, 34, 2018-2033.

[6] Stammbach, D., Zouhar, V., Hoyle, A., Sachan, M., & Ash, E. (2023). Revisiting automated topic model evaluation with large language models. arXiv preprint arXiv:2305.12152.
questions: questions: - Are the created word embeddings semantically meaningful? Would clustering the word embeddings e.g. [7] give meaningfull topics? If yes, this is an additional benefit which I would suggest to include in the paper.



[7] Sia, S., Dalmia, A., & Mielke, S. J. (2020). Tired of topic models? clusters of pretrained word embeddings make for fast and good topics too!. arXiv preprint arXiv:2004.14914.
limitations: limitations: Not needed

Comments: title: Looking forward to your further feedback!
Response to Reviewer ceW2
Response to Reviewer Py7Z
Response to Reviewer YWcC
Rebuttal Acknowledgment
comment: Dear Reviewer FTng,

Thank you sincerely for your previous helpful reviews!  
We mention that we've submitted our responses, including __clarifications on our differences from ECRTM and our performance on long-text corpora and large numbers of topics.__  
We hope our responses can address your concerns. We are looking forward to your insightful feedback!

Thank you.

Best,  
Authors
Dear Reviewer ceW2,

Thank you for your reply! We appreciate your recommendation!

Thank you.
Dear Reviewer Py7Z,

Thank you for your reply! We're glad that our responses have addressed your concerns.

Thank you.
Your response provides a detailed explanation! I will keep my score.
Dear authors,
Thanks for your responses, and they do address my concerns. I have raised my score from 5 to 6.
Dear Reviewer YWcC,

Thank you for your reply! We're glad that our responses have addressed your concerns.

Thank you.
Dear authors,

thank you for your answers.
All of my questions have been adequately answered, or have been already answered within the original manuscript. Thank you for bringing that to my attention.


I have adapted my score accordingly
5 -> 7