HiCOT: Improving Neural Topic Models via Optimal Transport and Contrastive Learning
Authors: Hoang Tran Vuong, Tue Le, Tu Vu, Tung Nguyen, Linh Ngo Van, Sang Dinh, Thien Huu Nguyen
Accepted at: acl25

HiCOT: Improving Neural Topic Models via Optimal Transport and Contrastive Learning

Hoang Tran Vuong1*, Tue Le1*, Tu Vu2*, Tung Nguyen1*, Linh Van Ngo1†, Sang Dinh1, Thien Huu Nguyen3
1Hanoi University of Science and Technology (HUST), Vietnam
2Bytedance Inc
3University of Oregon, USA

Abstract

Recent advances in neural topic models (NTMs) have improved topic quality but still face challenges: weak document-topic alignment, high inference costs due to large pretrained language models (PLMs), and limited modeling of hierarchical topic structures. To address these issues, we introduce HiCOT (Hierarchical Clustering and Contrastive Learning with Optimal Transport for Neural Topic Modeling), a novel framework that enhances topic coherence and efficiency. HiCOT integrates Optimal Transport to refine document-topic relationships using compact PLM-based embeddings, captures semantic structure of the documents. Additionally, it employs hierarchical clustering combine with contrastive learning to disentangle topic-word and topic-topic relationships, ensuring clearer structure and better coherence. Experimental results on multiple benchmark datasets demonstrate HiCOT’s superior effectiveness over existing NTMs in topic coherence, topic performance, representation quality, and computational efficiency.

1 Introduction

Topic modeling has been recognized as a fundamental task in natural language processing (NLP), aims to uncover latent topic structures within a corpus, while simultaneously providing document topic distributions (Hofmann, 1999; Griffiths et al., 2003; Srivastava and Sutton, 2017; Wu et al., 2024a). In recent years, neural topic models (NTMs) (Zhao et al., 2021; Xu et al., 2022; Wu et al., 2024d; Nguyen et al., 2025a,b) have emerged as a promising alternative, leveraging deep neural networks to enhance flexibility, improve the quality of discovered topics, help to overcome limitations about inefficient parameter inference of traditional topic models.

Most NTMs (Dieng et al., 2020; Wu et al., 2023b; Pham et al., 2024b) are built upon the Variational Autoencoders (VAEs) (Kingma and Welling, 2013) framework, where an inference encoder generates document-topic distributions, and a generative decoder reconstructs the original texts. Beyond VAEs, other architectures such as using dual semantic relation reconstruction paradigm (Wu et al., 2024b), or through the development of variations of TF-IDF (Grootendorst, 2022), have been proposed to improve topic coherence and interpretability. In addition to architectural innovations, various advanced techniques have been introduced to enhance topic modeling performance. One widely explored approach involves integrating contextual information, such as word embeddings (Pennington et al., 2014), or sentence embeddings (Reimers and Gurevych, 2019a), to provide more semantically meaningful topic distributions. Another methods based on Optimal Transport also have been employed to model relationships between documents, topics, and words more effectively (Zhao et al., 2022; Wu et al., 2023b; Xu et al., 2023), while some integrating pretrained language models (PLMs) (Devlin, 2018; Reimers and Gurevych, 2019a) to capture complex linguistic structures (Han et al., 2023; Pham et al., 2024b).

Despite these advancements, NTMs still face several challenges. Recent models exhibit limitations in effectively capturing document-topic relationships, lacking information and showing weak alignment between document representations and topic proportions. While some approaches leverage PLMs to enhance topic modeling (Wu et al., 2023a; Han et al., 2023), the process of extracting embeddings from large PLMs significantly increases inference costs, making them less practical for situations requiring low inference times. It is entirely feasible to replace PLM-based embeddings with alternative methods that still achieve good performance, while significantly reducing computational overhead. To
address the lack of document-topic relationships, we propose an approach that effectively utilizes compact PLMs with a simple neural network. Our method incorporates with Optimal Transport to refine relationships between documents and topics, enhancing the model’s ability to learn more structured and semantic topic distributions through pre-trained document embeddings.

Another limitation of NTMs is their tendency to overlook hierarchical topic structure, limiting their ability to model semantic dependencies at different levels of abstraction. To tackle this issue, we introduce a hierarchical clustering framework combined with contrastive learning, explicitly disentangling relationships between topics and words, as well as between topics themselves. Our approach regularizes semantic relations among topic and word embeddings, ensuring clear separation between topic clusters while maintaining meaningful relationships within each cluster. This leads to enhanced topic interpretability, coherence, and representation quality. In this work, we propose HiCOT (Hierarchical Clustering and Contrastive Learning with Optimal Transport for Neural Topic Modeling), a novel framework that effectively addresses these challenges. We summarize the contributions of our study as follows:

• We introduce an efficient topic modeling approach leveraging compact PLMs and Optimal Transport to enhance document-topic alignment, capture semantic relationships between documents while ensuring computational efficiency.

• We develop a novel framework that integrates hierarchical clustering with contrastive learning to strengthen topic-word associations and topic-level relationships, enhancing interpretability and coherent topic representations.

• We conduct comprehensive experiments on multiple benchmark datasets, demonstrating that our approach effectively improves overall topic modeling performance compared to existing NTMs.

2 Background

Consider a collection of Bag-of-Words (BoW) representations, denoted as \( \mathbf{X} = \{ \mathbf{x}_d \}_{d=1}^D \), corresponding to \( D \) documents and defined over a vocabulary of \( V \) distinct words. Topic modeling aims to uncover \( K \) latent topics within \( \mathbf{X} \). Each topic \( k \) is associated with a topic-word distribution \( \beta_k \in \mathbb{R}^{V \times 1} \), forming a topic-word distribution matrix of \( K \) desired topics \( \beta = (\beta_1, \ldots, \beta_K) \in \mathbb{R}^{V \times K} \). Let \( L \) be the word embedding dimension, we define the word embedding matrix as \( \mathbf{W} = (\mathbf{w}_1, \ldots, \mathbf{w}_V) \in \mathbb{R}^{V \times L} \) and the topic embedding matrix as \( \mathbf{T} = (\mathbf{t}_1, \ldots, \mathbf{t}_K) \in \mathbb{R}^{K \times L} \). Recent advances in neural topic modeling (Wu et al., 2023b; Pham et al., 2024b; Wu et al., 2024c) have shifted away from factorizing \( \beta \) as the product of word embeddings \( W \) and topic embeddings \( T \). Instead, \( \beta \) is expressed as:

\[
\beta_{ij} = \frac{\exp \left( - \| \mathbf{w}_i - \mathbf{t}_j \|_2^2 / \tau \right)}{\sum_{j'=1}^K \exp \left( - \| \mathbf{w}_i - \mathbf{t}_{j'} \|_2^2 / \tau \right)},
\]

where \( \tau \) is a temperature hyperparameter. Another objective of neural topic models is the inference of topic proportions for each document \( x_i \), represented as \( \theta_i \in \mathbb{R}^K \) which characterizes the distribution of topics within the document. In VAE-based topic models, topic proportion \( \theta \) is inferred through a latent variable \( z \) that follows a logistic-normal prior distribution \( p(z) = \mathcal{N}(z|\mu_0, \Sigma_0) \). Given a document \( x_i \), its Bag-of-Words (BoW) representation is passed through an inference network to compute the parameters of a Gaussian posterior distribution, where mean and diagonal covariance matrix are given by \( \mu = h_\mu(x_d, \gamma) \) and \( \Sigma = \text{diag}(h_\Sigma(x_d, \gamma)) \), respectively, with \( \gamma \) denoting the parameters of inference network. Using the reparameterization trick (Kingma and Welling, 2013), the latent variable \( z \) is subsequently sampled from the posterior distribution \( q(z|x_i) = \mathcal{N}(z|\mu, \Sigma) \). Finally, the BoW representation is reconstructed through \( \beta \) and \( \theta \), where the generative process follows a multinomial distribution \( \hat{\mathbf{x}}_{\text{BoW}} \sim \text{Multi}(\text{softmax}(\beta \theta)) \). The loss function for topic modeling is composed of two components: a reconstruction term and a regularization term, as detailed below:

\[
\mathcal{L}_{\text{TM}} = \frac{1}{D} \sum_{i=1}^D \left[ - (\mathbf{x}_i \text{BoW})^\top \log(\text{softmax}(\beta \theta_i)) \right. \\
\left. + \text{KL}(q(z|x_i) \| p(z)) \right].
\]

Unlike VAE-based models, FASTopic (Wu et al., 2024c) adopts a different inference mechanism. FASTopic utilizes Optimal Transport to directly compute document-topic and topic-word distributions. Specifically, it defines cost matrices \( C^{(1)} \in \)
\( \mathbb{R}^{D \times K} \) and \( C^{(2)} \in \mathbb{R}^{K \times V} \) based on Euclidean distances between document embeddings, topic representations, and word embeddings. The resulting optimal transport plans \( \psi^* \) and \( \phi^* \) yield document-topic and topic-word distributions \( \theta = D \psi^* \) and \( \beta = K \phi^* \), with the overall loss:

\[
\mathcal{L} = - \frac{1}{D} \sum_{i=1}^{D} (x_i, \mathrm{BoW})^\top \log(\beta \theta_i) \\
+ \sum_{i,k} C^{(1)}_{ik} \psi^*_{ik} + \sum_{k,j} C^{(2)}_{kj} \phi^*_{kj}
\]

(3)

Then, given a new document \( x' \), FASTopic performs inference by mapping it to an embedding \( d' \) and estimates its topic distribution \( \theta' \) as follows:

\[
\theta'_k = \frac{p_k}{\sum_{k'=1}^K p_{k'}}, \quad p_k = \frac{\exp(-\| t_k - d' \|_2^2 / \tau)}{\sum_{i=1}^D \exp(-\| t_k - d_i \|_2^2 / \tau)}
\]

(4)

where \( \tau \) is a temperature hyperparameter. Despite its efficiency, FASTopic is limited by inconsistencies between training and testing, and its dependence on the choice of pretrained language model.

3 Proposed Method

In this section, we present a detailed analysis of our proposed, which leverages compact PLMs and Optimal Transport to improve document-topic alignment. Furthermore, we introduce a novel framework that incorporates hierarchical clustering and contrastive learning to reinforce topic-word associations and topic-level relationships.

3.1 Efficient and Consistent Topic Modeling with MLP and Optimal Transport

Existing topic models, such as FASTopic (Wu et al., 2024b), suffer from inconsistencies between training and inference, where inference relies on a pretrained language model, leading to significant computational overhead. To address this, we employ a unified VAE architecture and replace the reliance on large Transformer-based encoders (e.g., BERT (Devlin, 2018), Sentence-BERT (Reimers and Gurevych, 2019b)) with a lightweight alternative, that ensures consistency between training and inference while enabling efficient inference without reducing representation quality.

Specifically, we initialize document embeddings using Doc2Vec (Le and Mikolov, 2014), where \( \mathbf{E} = \{ \mathbf{e}_1, \mathbf{e}_2, \ldots, \mathbf{e}_D \} \in \mathbb{R}^{D \times M} \) denotes the document embedding matrix, with \( M \) is the embedding dimension. Then we use an MLP network to project document embeddings into the topic embedding space: \( \phi_E(\mathbf{E}) \), with learnable weights \( W_{\phi_E} \in \mathbb{R}^{M \times L} \). This projection is designed to preserve the relationships of document embeddings generated from the pretrained language model, ensuring that topic embeddings can capture highly meaningful semantic structures. To achieve this, we model the relationship between document and topic embeddings through optimal transport, where the transportation cost between a document \( i \) and a topic \( k \) is defined as: \( C^{(ik)}_{DT} = \| \phi_E(\mathbf{e}_i) - \mathbf{t}_k \|_2^2 \). The objective is then to minimize the weighted transport distance between outputs of the MLP network and topic embeddings, formulated as follows:

\[
\mathcal{L}_{DT} = \sum_{i=1}^D \sum_{k=1}^K C^{(ik)}_{DT} \pi_{ik}^*
\]
where \( \pi^* = \arg \min_{\pi \in \mathbb{R}^{D \times K}} \langle C_{DT}, \pi \rangle - \nu H(\pi) \)
s.t. \( \pi 1_K = \frac{1}{D} 1_D, \pi^\top 1_D = \frac{1}{K} 1_K \)

(5)

By optimizing this objective, the distance between projected document embeddings and topic embeddings is minimized. This results in a topic structure where the topic embeddings inherit the good semantic structure of document embeddings from the pretrained model. This alignment enhances the interpretability and coherence of the topic embeddings. Our approach still achieves computational efficiency while maintaining topic modeling performance, making it well-suited for scaling to larger datasets.

3.2 Topic Regularization via Optimal Transport for Semantic Relationships

We propose a novel topic regularization approach leveraging Optimal Transport (Peyré and Cuturi, 2020). The core idea is to align topic distributions with semantic relationships by utilizing a transport plan that integrates both topic proportions and semantic similarities between documents, utilizing the strengths of PLMs. This ensures that topic distributions are preserved while also capturing the underlying meaningful structure of the document collection. Let \( D \) be the number of documents, we define \( C \in \mathbb{R}^{D \times D} \) as the cost matrix in Euclidean space for topic proportions
\{ \theta_1, \theta_2, \ldots, \theta_D \}, where \( c_{ij} = \| \theta_i - \theta_j \|_2^2 \). In parallel, we define a matrix \( P \) to capture the semantic similarity between documents. The elements of \( P \) are computed using cosine similarity between document embeddings:

\[
p_{ij} = \frac{\langle e_i, e_j \rangle}{\|e_i\|_2 \cdot \|e_j\|_2}
\]

Since document embeddings \( E \) are initialized using Doc2Vec, they encode high-level semantic information by capturing both contextual and distributional similarities between documents. This allows matrix \( P \) to effectively capture the semantic structure of the document corpus. To ensure the alignment between topic-based relationships and semantic structures, we incorporate a regularization term that forces the transport plan \( Q \) to approximate the semantic relationships captured in matrix \( P \). This is formulated by introducing an additional KL-divergence constraint into the following optimization problem:

\[
\begin{aligned}
& \min_{Q \in \mathbb{R}^{D \times D}} \langle Q, C \rangle - \lambda_1 H(Q) + \lambda_2 \mathrm{KL}(Q||P) \\
& \quad \text{s.t. } Q \mathbf{1}_D = Q^\top \mathbf{1}_D = \frac{1}{D} \mathbf{1}_D
\end{aligned}
\]

where \( \lambda_1, \lambda_2 > 0 \), \( \mathbf{1}_D \) is \( D \)-dimensional vector with all elements equal to 1, and \( H(Q) = -(Q, \log Q - 1) \) represents the Shannon entropy of \( Q \) (Cuturi, 2013a). The transport plan \( Q \) uncovers the topic relationships by considering topic proportions, effectively capturing information is transferred between topics based on their inter-relations. By solving the problem in Equation 7, we can also minimize \( \mathrm{KL}(Q||P) \). Adding a KL-divergence constraint forces \( Q \) to learn the semantic relationships between documents while maintaining the consistency of topic proportions. The motivation behind using Optimal Transport is its ability to model the transfer of information between topics while preserving the mass (i.e., topic proportions). Similar to the transportation of mass, our approach ensures that topic proportions remain consistent while enabling information transfer between topics. This not only improves the coherence of the topic model but also enhances the understanding of semantic within the document.

3.3 Contrastive Learning for Topic Clustering

In this paper, we introduce an novel regularization method for topic modeling, integrating hierarchical clustering with contrastive learning. By leveraging hierarchical clustering, our method automatically identifies topic clusters, enabling us to impose contrastive losses that enhance both intra-cluster and inter-cluster relationships. Contrastive learning is employed between topic and word embeddings in a same cluster, as well as among topic embeddings between clusters. This solution enhances the semantic relationships between topic and word embeddings, as well as at the topic level, improving both coherence and quality of topic representations. Specifically, we use hierarchical agglomerative clustering (HAC) (Murtagh and Contreras, 2012) to divide topic embeddings into clusters. Assume that after performing hierarchical clustering, we partition \( K \) topics into \( G \) clusters. For each cluster \( i \), we denote its set of topic embeddings as: \( \mathcal{T}_i = \{ t_1, t_2, \ldots, t_{N_i} \} \), where \( N_i \) is the number of topics within cluster \( i \), ensuring that \( \sum_{i=1}^G N_i = K \).

3.3.1 Contrastive Loss between Topic Embeddings in a Cluster

Once the topics are clustered, we apply contrastive learning to the topic embeddings within the same cluster. To achieve this, we first compute the cosine similarity between each word embedding and all topic embeddings. A word is then assigned to the topic with the highest similarity. Within each cluster, we consider only topics that contain at least one assigned word embedding. Formally, consider topic \( i \) in cluster \( j \) with a set of word embeddings \( Z_{ij} = \{ w_1, w_2, \ldots, w_{M_{ij}} \} \), where \( M_{ij} \) denotes the number of word embeddings assigned to topic \( i \) in cluster \( j \). The total number of assigned words across all topics and clusters satisfies the constraint:

\[
\sum_{j=1}^G \sum_{i=1}^{N_j} M_{ij} = V, \text{ where } V \text{ is the vocabulary size.}
\]

We then define the contrastive components based on Triplet Loss (Schroff et al., 2015), as follows:

• Anchor \( a_{ij} \): The average embedding of all words in topic \( i \) within cluster \( j \), computed as:

\[
a_{ij} = \frac{1}{M_{ij}} \sum_{k=1}^{M_{ij}} w_k
\]

• Positive sample \( p_{ij} \): A randomly selected word from the same topic, i.e., \( p_{ij} \sim Z_{ij} \)

• Negative sample \( n_{ij} \): A randomly selected
word from another topic within the same cluster, i.e.,
\[
n_{ij} \sim Z_j \setminus Z_{ij}
\]
where \( Z_j = \bigcup_{i=1}^{N_j} Z_{ij} \) is the set of all word embeddings assigned to topics in cluster \( j \).

The objective is to pull words within the same topic closer in the embedding space, while simultaneously pushing words from different topics apart, improving the topic coherence within each cluster. To achieve this, we minimize the following contrastive loss:

\[
\mathcal{L}_{CLT} = \sum_{j=1}^G \sum_{i=1}^{N_j} \max \left( d(a_{ij}, p_{ij}) - \frac{1}{k} \sum_{s=1}^k d(a_{ij}, n_{ij s}) + m, 0 \right)
\]

where \( k \) denotes the number of negative samples, \( d(x, y) \) represents a distance function, and \( m \) is a margin ensuring that the anchor-positive distance is smaller than the anchor-negative distance.

3.3.2 Contrastive Loss between Clusters
Beyond intra-cluster contrastive learning, we extend contrastive framework to the clustering level. We also define the following components based on Triplet Loss (Schroff et al., 2015), as follows:

• Anchor \( a_i \): The mean embedding of all topics within a given cluster \( i \), computed as:
\[
a_i = \frac{1}{N_i} \sum_{j=1}^{N_i} t_j
\]
• Positive sample \( p_i \): A randomly selected topic embedding from the same cluster, i.e., \( p_i \sim \mathcal{T}_i \)
• Negative sample \( n_i \): A randomly selected topic embedding from a different cluster, i.e.,
\[
n_i \sim \mathbf{T} \setminus \mathcal{T}_i
\]
To promote tighter topic coherence within each cluster and enhance the separation between clusters, we minimize the following contrastive loss:

\[
\mathcal{L}_{CLC} = \sum_{i=1}^G \max \left( d(a_i, p_i) - \frac{1}{k} \sum_{s=1}^k d(a_i, n_{i s}) + m, 0 \right)
\]

Here, \( d(x, y) \) denotes the chosen distance metric, \( k \) is the number of negative samples, and \( m \) serves as a margin parameter. This loss encourages topics within the same cluster to have similar representations, while ensuring that topics from different clusters are distinct in the embedding space. As a result, it improves the separation between clusters while maintaining meaningful relationships within each cluster.

3.4 Overall objective function
Building on the approach of (Wu et al., 2023b), we integrate the Embedding Clustering Regularization (ECR) regularizer to minimize the weighted distance between word and topic embeddings, formulated as:

\[
\mathcal{L}_{ECR} = \sum_{j=1}^V \sum_{k=1}^K ||\mathbf{w}_j - \mathbf{t}_k||^2 \psi_{jk}^*
\]

Let \( C_{ECR} \) denote the transport cost between word and topic embeddings measured by Euclidean distance. The optimal transport plan \( \psi^* \) is computed through Sinkhorn’s algorithm (Cuturi, 2013b):

\[
\begin{align*}
\min_{\psi \in \mathbb{R}^{V \times K}} & \langle \psi, C_{ECR} \rangle - \epsilon H(\psi) \\
\text{s.t. } & \psi \mathbf{1}_K = \frac{1}{V} \mathbf{1}_V, \; \psi^\top \mathbf{1}_V = \frac{1}{K} \mathbf{1}_K
\end{align*}
\]

We now formalize our training process as a two-stage approach to ensure effective topic modeling and clustering. In **Stage 1**, the model undergoes initial training to capture the semantic structure of the data and refine the learned embeddings before clustering is introduced. During this phase, we update transport plan \( Q \) following Equation 7, and then optimize the following objective function:

\[
\mathcal{L}_{stage1} = \mathcal{L}_{TM} + \lambda_{DT} \mathcal{L}_{DT} + \lambda_{ECR} \mathcal{L}_{ECR}
\]

After the model has learned meaningful representations, we proceed to **Stage 2**, where hierarchical clustering is applied. Clustering is first initialized at a predefined epoch, and subsequently refined at fixed intervals. During this stage, the model still updates transport plan \( Q \) according to the iterative process of Equation 7, and optimizes additional contrastive losses alongside the topic modeling objective from **Stage 1**:
\[
\mathcal{L}_{\text{stage2}} = \lambda_{\text{CLC}} \mathcal{L}_{\text{CLC}} + \lambda_{\text{CLT}} \mathcal{L}_{\text{CLT}} + \mathcal{L}_{\text{TM}}
+ \lambda_{\text{DT}} \mathcal{L}_{\text{DT}} + \lambda_{\text{ECR}} \mathcal{L}_{\text{ECR}}
\]
(17)
where \( \lambda_{\text{CLC}}, \lambda_{\text{CLT}}, \lambda_{\text{DT}}, \lambda_{\text{ECR}} \) are weight hyperparameters. This two-stage approach ensures that clustering is performed only after the model has sufficiently learned the meaningful representations of the data, resulting in more coherent and well-separated topic clusters. The full algorithm are described in Appendix A.

4 Experiments

4.1 Settings

Datasets. Our experiments employ five prominent datasets, covering a diverse range of domains. We utilize three widely recognized benchmarks in topic modeling: **20 News Groups (20NG)** (Lang, 1995), a standard dataset for topic modeling; **AG-News** (Zhang et al., 2015) which comprises news articles from over 2000 sources; and IMDB (Maas et al., 2011), a collection of movie reviews. Additionally, we examine two short-text datasets: **SearchSnippets** (Phan et al., 2008), which contains over 12,000 web search snippets across 8 domains; and **GoogleNews** (Yin and Wang, 2016), featuring more than 10000 news article titles categorized into 152 topics. This dataset collection enables a comprehensive analysis across different text types and domains. The preprocessing steps and statistics of all datasets are detailed in Appendix G.2.

Evaluation metrics. We evaluate our model using the framework proposed by (Wu et al., 2023b), focusing on both topic quality and document-topic distributions. Topic quality is assessed using coherence and diversity metrics. Coherence is quantified by \( C_V 15 \), which measures the semantic consistency of the top 15 words within each topic and has been shown to strongly correlate with human interpretability (Röder et al., 2015). Coherence scores are computed using a modified version of the Wikipedia corpus\footnote{https://github.com/dice-group/Palmetto/} as a reference source. For topic diversity, we use TD15, which computes the proportion of unique words among the top 15 topic words. To evaluate document-topic distributions, we employ Normalized Mutual Information (NMI) and Purity (Manning et al., 2008), following the approach in (Wang et al., 2022), where each document is assigned to its most probable topic.

Baseline models. We consider several advanced topic modeling frameworks, including **ETM** (Dieng et al., 2020), which incorporates word embeddings; **NTM + CL** (Nguyen and Luu, 2021), which leverages contrastive learning to model relationships between similar and dissimilar documents; **ECRTM** (Wu et al., 2023b), which improves topic coherence via clustering regularization; **FASTopic** (Wu et al., 2024b), which models document-word-topic relationships using Optimal Transport; and **NeuroMax** (Pham et al., 2024b; Nguyen et al., 2025d), which refines topic distributions through pretrained embeddings and mutual information maximization.

4.2 Topic and Doc-Topic Distribution Quality

We conduct experiments to assess both topic quality and the effectiveness of document-topic distributions across five benchmark datasets: 20NG, AG-News, IMDB, SearchSnippets, and GoogleNews. Tables 1 and 2 present the evaluation results for models trained with 50 and 100 topics, respectively. Our approach consistently surpasses baseline models, improving overall topic quality. Furthermore, it significantly enhances the quality of document-topic distributions, as reflected in higher Purity and NMI scores, which indicate better clustering performance and greater distinguishability of topic groups. In addition to improving document-topic distributions, HiCOT also strengthens topic coherence across most datasets, as demonstrated by its superior \( C_V \) score compared to other models.

4.3 Contrastive Learning Strategies

We evaluated the impact of contrastive learning strategies within our topic modeling framework, focusing on sampling strategies and contrastive learning method. For sampling strategies, we examine the selection process for positive and negative samples relative to the anchor. Beyond the default of random selection for both positive and negative samples ("Random / Random"), we evaluated two alternative approaches:

• Hard negatives: Selecting negative samples that exhibit the smallest embedding space distance to the anchor.

• Hard positives: Selecting samples at an intermediate distance, intended to increase learning difficulty while maintaining relevance.

We performed these experiments on 20NG and GoogleNews datasets, setting number of topics
<table>
  <tr>
    <th rowspan="2">50 Topics</th>
    <th colspan="4">20NG</th>
    <th colspan="4">AGNews</th>
    <th colspan="4">IMDB</th>
  </tr>
  <tr>
    <th>C<sub>V</sub></th>
    <th>Purity</th>
    <th>NMI</th>
    <th>TD</th>
    <th>C<sub>V</sub></th>
    <th>Purity</th>
    <th>NMI</th>
    <th>TD</th>
    <th>C<sub>V</sub></th>
    <th>Purity</th>
    <th>NMI</th>
    <th>TD</th>
  </tr>
  <tr>
    <td>ETM ‡</td>
    <td>0.375</td>
    <td>0.347</td>
    <td>0.319</td>
    <td>0.704</td>
    <td>0.364</td>
    <td>0.679</td>
    <td>0.224</td>
    <td>0.819</td>
    <td>0.346</td>
    <td>0.660</td>
    <td>0.038</td>
    <td>0.557</td>
  </tr>
  <tr>
    <td>NTM + CL</td>
    <td>0.437</td>
    <td>0.582</td>
    <td>0.491</td>
    <td>0.802</td>
    <td>0.440</td>
    <td>0.322</td>
    <td>0.100</td>
    <td>0.441</td>
    <td>0.396</td>
    <td>0.657</td>
    <td>0.044</td>
    <td>0.617</td>
  </tr>
  <tr>
    <td>ECRTM ‡</td>
    <td>0.431</td>
    <td>0.560</td>
    <td>0.524</td>
    <td>0.964</td>
    <td>0.466</td>
    <td>0.802</td>
    <td>0.367</td>
    <td>0.961</td>
    <td>0.393</td>
    <td>0.694</td>
    <td>0.058</td>
    <td>0.974</td>
  </tr>
  <tr>
    <td>FASTopic</td>
    <td>0.427</td>
    <td>0.583</td>
    <td>0.528</td>
    <td>0.980</td>
    <td>0.379</td>
    <td>0.831</td>
    <td>0.352</td>
    <td>0.960</td>
    <td>0.371</td>
    <td>0.683</td>
    <td>0.055</td>
    <td>0.969</td>
  </tr>
  <tr>
    <td>NeuroMax ‡</td>
    <td>0.435</td>
    <td>0.623</td>
    <td>0.570</td>
    <td>0.912</td>
    <td>0.385</td>
    <td>0.804</td>
    <td>0.410</td>
    <td>0.952</td>
    <td>0.402</td>
    <td>0.709</td>
    <td>0.061</td>
    <td>0.936</td>
  </tr>
  <tr>
    <td>HiCOT</td>
    <td><b>0.451</b></td>
    <td><b>0.626</b></td>
    <td><b>0.583</b></td>
    <td>0.852</td>
    <td><b>0.446</b></td>
    <td><b>0.857</b></td>
    <td><b>0.412</b></td>
    <td><b>0.992</b></td>
    <td><b>0.404</b></td>
    <td><b>0.737</b></td>
    <td><b>0.082</b></td>
    <td><b>0.837</b></td>
  </tr>
  <tr>
    <th rowspan="2">100 Topics</th>
    <th colspan="4">20NG</th>
    <th colspan="4">AGNews</th>
    <th colspan="4">IMDB</th>
  </tr>
  <tr>
    <th>C<sub>V</sub></th>
    <th>Purity</th>
    <th>NMI</th>
    <th>TD</th>
    <th>C<sub>V</sub></th>
    <th>Purity</th>
    <th>NMI</th>
    <th>TD</th>
    <th>C<sub>V</sub></th>
    <th>Purity</th>
    <th>NMI</th>
    <th>TD</th>
  </tr>
  <tr>
    <td>ETM ‡</td>
    <td>0.369</td>
    <td>0.394</td>
    <td>0.339</td>
    <td>0.573</td>
    <td>0.371</td>
    <td>0.674</td>
    <td>0.204</td>
    <td>0.773</td>
    <td>0.341</td>
    <td>0.648</td>
    <td>0.037</td>
    <td>0.371</td>
  </tr>
  <tr>
    <td>NTM + CL</td>
    <td>0.420</td>
    <td>0.626</td>
    <td>0.490</td>
    <td>0.624</td>
    <td>0.415</td>
    <td>0.280</td>
    <td>0.050</td>
    <td>0.277</td>
    <td>0.382</td>
    <td>0.705</td>
    <td>0.044</td>
    <td>0.492</td>
  </tr>
  <tr>
    <td>ECRTM ‡</td>
    <td>0.405</td>
    <td>0.555</td>
    <td>0.494</td>
    <td>0.904</td>
    <td>0.416</td>
    <td>0.812</td>
    <td>0.428</td>
    <td>0.981</td>
    <td>0.373</td>
    <td>0.694</td>
    <td>0.049</td>
    <td>0.887</td>
  </tr>
  <tr>
    <td>FASTopic</td>
    <td>0.400</td>
    <td>0.622</td>
    <td>0.522</td>
    <td>0.861</td>
    <td>0.385</td>
    <td>0.833</td>
    <td>0.330</td>
    <td>0.912</td>
    <td>0.369</td>
    <td>0.680</td>
    <td>0.048</td>
    <td>0.886</td>
  </tr>
  <tr>
    <td>NeuroMax ‡</td>
    <td>0.412</td>
    <td>0.602</td>
    <td>0.516</td>
    <td>0.913</td>
    <td>0.406</td>
    <td>0.828</td>
    <td>0.389</td>
    <td>0.957</td>
    <td>0.381</td>
    <td>0.706</td>
    <td>0.059</td>
    <td>0.870</td>
  </tr>
  <tr>
    <td>HiCOT</td>
    <td><b>0.424</b></td>
    <td><b>0.652</b></td>
    <td><b>0.568</b></td>
    <td>0.741</td>
    <td><b>0.435</b></td>
    <td><b>0.862</b></td>
    <td><b>0.388</b></td>
    <td><b>0.960</b></td>
    <td><b>0.388</b></td>
    <td><b>0.739</b></td>
    <td><b>0.071</b></td>
    <td><b>0.733</b></td>
  </tr>
</table>

Table 1: Evaluation results on standard datasets, assessed using C<sub>V</sub>, TD, Purity and NMI under \( K = 50 \) topics and \( K = 100 \) topics. The highest-performing results are marked in bold, while the second-best values are underlined. Results reported in (Pham et al., 2024b).

\( K = 50 \) in Table 3. The results indicate that no single strategy consistently outperforms others across all metrics. "Random / Random" achieving highest scores in both cases for topic coherence (C<sub>V</sub>). However, incorporating "Hard" sampling introduces trade-offs: "Hard Negatives" enhance TD on GoogleNews but slightly reduce C<sub>V</sub>, "Hard Positives" improve Purity and NMI on 20NG ("Hard / Random") yet show different impacts on GoogleNews. These findings suggest that optimal contrastive sampling configuration is not fixed but depends on dataset characteristics. While "Random / Random" offers a reliable default, the results strongly motivate future research into developing more sampling techniques tailored to specific dataset properties.

For contrastive learning methods, alongside Triplet Loss employed in our main experiments, we evaluated performance using two contrastive losses: InfoNCE (Oord et al., 2018) and Circle Loss (Sun et al., 2020). To ensure a fair comparison, these experiments utilized same sampling strategy as applied with Triplet Loss. The results, presented in Table 4, demonstrate that all three loss functions achieve comparable performance across both datasets and all metrics. While minor fluctuations exist (e.g., slight advantages Purity and NMI for Circle Loss on AGNews, Triplet Loss on SearchSnippets), the differences are consistently small. These results highlight the robustness of our contrastive learning framework, suggesting that its effectiveness primarily from contrastive paradigm rather than the choice of loss function.

4.4 Robustness of HiCOT across embeddings

While Doc2Vec captures less semantic structure than SentenceTransformer, using a lightweight model like Doc2Vec instead of advanced pretrained language models (PLMs) demonstrates HiCOT’s superior performance without relying heavily on extensive pretrained knowledge. As shown in Tables 1 and 2, HiCOT with Doc2Vec surpasses PLM-based models such as FASTopic and NeuroMax. This highlights HiCOT’s robustness, which relies primarily on its architecture rather than the representational power of the pretrained language model. To further investigate this, we evaluated HiCOT with Doc2Vec and SentenceTransformer (all-MiniLM-L6-v2 (Reimers, 2019)) embeddings on 20NG dataset with \( K = 50 \) and
<table>
  <tr>
    <th rowspan="2"> </th>
    <th colspan="6">K = 50</th>
    <th colspan="6">K = 100</th>
  </tr>
  <tr>
    <th colspan="3">SearchSnippets</th>
    <th colspan="3">GoogleNews</th>
    <th colspan="3">SearchSnippets</th>
    <th colspan="3">GoogleNews</th>
  </tr>
  <tr>
    <th></th>
    <th>Cv</th>
    <th>Purity</th>
    <th>NMI</th>
    <th>TD</th>
    <th>Cv</th>
    <th>Purity</th>
    <th>NMI</th>
    <th>TD</th>
    <th>Cv</th>
    <th>Purity</th>
    <th>NMI</th>
    <th>TD</th>
  </tr>
  <tr>
    <td>ETM ‡</td>
    <td>0.397</td>
    <td>0.688</td>
    <td>0.389</td>
    <td>0.594</td>
    <td>0.402</td>
    <td>0.366</td>
    <td>0.560</td>
    <td>0.916</td>
    <td>0.389</td>
    <td>0.691</td>
    <td>0.365</td>
    <td>0.448</td>
    <td>0.398</td>
    <td>0.554</td>
    <td>0.713</td>
    <td>0.677</td>
  </tr>
  <tr>
    <td>NTM + CL</td>
    <td>0.403</td>
    <td>0.215</td>
    <td>0.030</td>
    <td>0.532</td>
    <td>0.433</td>
    <td>0.041</td>
    <td>0.005</td>
    <td>0.301</td>
    <td>0.406</td>
    <td>0.217</td>
    <td>0.020</td>
    <td>0.394</td>
    <td>0.432</td>
    <td>0.039</td>
    <td>0.005</td>
    <td>0.367</td>
  </tr>
  <tr>
    <td>ECRTM ‡</td>
    <td>0.450</td>
    <td>0.711</td>
    <td>0.419</td>
    <td>0.998</td>
    <td>0.441</td>
    <td>0.396</td>
    <td>0.615</td>
    <td>0.987</td>
    <td>0.432</td>
    <td>0.789</td>
    <td>0.443</td>
    <td>0.966</td>
    <td>0.418</td>
    <td>0.342</td>
    <td>0.491</td>
    <td>0.991</td>
  </tr>
  <tr>
    <td>FASTopic</td>
    <td>0.356</td>
    <td>0.793</td>
    <td>0.497</td>
    <td>0.519</td>
    <td>0.401</td>
    <td>0.252</td>
    <td>0.570</td>
    <td>0.235</td>
    <td>0.350</td>
    <td>0.801</td>
    <td>0.466</td>
    <td>0.463</td>
    <td>0.366</td>
    <td>0.237</td>
    <td>0.459</td>
    <td>0.100</td>
  </tr>
  <tr>
    <td>NeuroMax</td>
    <td>0.427</td>
    <td>0.743</td>
    <td>0.427</td>
    <td>0.920</td>
    <td>0.409</td>
    <td>0.359</td>
    <td>0.590</td>
    <td>1.000</td>
    <td>0.439</td>
    <td>0.854</td>
    <td>0.472</td>
    <td>0.960</td>
    <td>0.427</td>
    <td>0.664</td>
    <td>0.834</td>
    <td>0.915</td>
  </tr>
  <tr>
    <td>HiCOT</td>
    <td><b>0.460</b></td>
    <td><b>0.818</b></td>
    <td><b>0.478</b></td>
    <td><b>1.000</b></td>
    <td><b>0.454</b></td>
    <td><b>0.465</b></td>
    <td><b>0.657</b></td>
    <td><b>0.920</b></td>
    <td><b>0.449</b></td>
    <td><b>0.857</b></td>
    <td><b>0.480</b></td>
    <td><b>0.940</b></td>
    <td><b>0.470</b></td>
    <td><b>0.763</b></td>
    <td><b>0.864</b></td>
    <td><b>0.802</b></td>
  </tr>
</table>

Table 2: Evaluation results on short datasets, assessed using Cv, TD, Purity and NMI under \( K = 50 \) topics and \( K = 100 \) topics. The highest-performing results are marked in bold, while the second-best values are underlined. Results reported in (Nguyen et al., 2025a).

<table>
  <tr>
    <th colspan="6">20NG</th>
  </tr>
  <tr>
    <th>Positive</th>
    <th>Negative</th>
    <th>Cv</th>
    <th>Purity</th>
    <th>NMI</th>
    <th>TD</th>
  </tr>
  <tr>
    <td>Random</td>
    <td>Random</td>
    <td>0.451</td>
    <td>0.626</td>
    <td>0.583</td>
    <td>0.852</td>
  </tr>
  <tr>
    <td>Hard</td>
    <td>Random</td>
    <td>0.434</td>
    <td>0.641</td>
    <td>0.584</td>
    <td>0.856</td>
  </tr>
  <tr>
    <td>Random</td>
    <td>Hard</td>
    <td>0.438</td>
    <td>0.620</td>
    <td>0.575</td>
    <td>0.901</td>
  </tr>
  <tr>
    <td>Hard</td>
    <td>Hard</td>
    <td>0.445</td>
    <td>0.638</td>
    <td>0.582</td>
    <td>0.843</td>
  </tr>
</table>

<table>
  <tr>
    <th colspan="6">GoogleNews</th>
  </tr>
  <tr>
    <th>Positive</th>
    <th>Negative</th>
    <th>Cv</th>
    <th>Purity</th>
    <th>NMI</th>
    <th>TD</th>
  </tr>
  <tr>
    <td>Random</td>
    <td>Random</td>
    <td>0.454</td>
    <td>0.465</td>
    <td>0.657</td>
    <td>0.920</td>
  </tr>
  <tr>
    <td>Hard</td>
    <td>Random</td>
    <td>0.446</td>
    <td>0.458</td>
    <td>0.663</td>
    <td>0.987</td>
  </tr>
  <tr>
    <td>Random</td>
    <td>Hard</td>
    <td>0.439</td>
    <td>0.456</td>
    <td>0.661</td>
    <td>0.995</td>
  </tr>
  <tr>
    <td>Hard</td>
    <td>Hard</td>
    <td>0.443</td>
    <td>0.462</td>
    <td>0.667</td>
    <td>0.989</td>
  </tr>
</table>

Table 3: Performance of contrastive sampling strategies on 20NG and GoogleNews datasets under \( K = 50 \) topics.

<table>
  <tr>
    <th>Dataset</th>
    <th>CL Method</th>
    <th>Cv</th>
    <th>Purity</th>
    <th>NMI</th>
    <th>TD</th>
  </tr>
  <tr>
    <td rowspan="3">AGNews</td>
    <td>InfoNCE</td>
    <td>0.442</td>
    <td>0.855</td>
    <td>0.412</td>
    <td>1.000</td>
  </tr>
  <tr>
    <td>Circle Loss</td>
    <td>0.442</td>
    <td>0.858</td>
    <td>0.416</td>
    <td>0.997</td>
  </tr>
  <tr>
    <td>Triplet Loss</td>
    <td>0.446</td>
    <td>0.857</td>
    <td>0.412</td>
    <td>0.992</td>
  </tr>
  <tr>
    <td rowspan="3">SearchSnippets</td>
    <td>InfoNCE</td>
    <td>0.464</td>
    <td>0.814</td>
    <td>0.476</td>
    <td>1.000</td>
  </tr>
  <tr>
    <td>Circle Loss</td>
    <td>0.464</td>
    <td>0.809</td>
    <td>0.476</td>
    <td>1.000</td>
  </tr>
  <tr>
    <td>Triplet Loss</td>
    <td>0.460</td>
    <td>0.818</td>
    <td>0.478</td>
    <td>1.000</td>
  </tr>
</table>

Table 4: Comparison of contrastive loss functions on AGNews and SearchSnippets datasets with 50 topics.

<table>
  <tr>
    <th colspan="5">20NG – 50 topics</th>
  </tr>
  <tr>
    <th>Method</th>
    <th>Cv</th>
    <th>Purity</th>
    <th>NMI</th>
    <th>TD</th>
  </tr>
  <tr>
    <td>HiCOT + Doc2Vec</td>
    <td>0.451</td>
    <td>0.626</td>
    <td>0.583</td>
    <td>0.852</td>
  </tr>
  <tr>
    <td>HiCOT + SBERT</td>
    <td>0.438</td>
    <td>0.649</td>
    <td>0.585</td>
    <td>0.868</td>
  </tr>
</table>

<table>
  <tr>
    <th colspan="5">20NG – 100 topics</th>
  </tr>
  <tr>
    <th>Method</th>
    <th>Cv</th>
    <th>Purity</th>
    <th>NMI</th>
    <th>TD</th>
  </tr>
  <tr>
    <td>HiCOT + Doc2Vec</td>
    <td>0.424</td>
    <td>0.652</td>
    <td>0.568</td>
    <td>0.741</td>
  </tr>
  <tr>
    <td>HiCOT + SBERT</td>
    <td>0.416</td>
    <td>0.632</td>
    <td>0.556</td>
    <td>0.779</td>
  </tr>
</table>

Table 5: HiCOT performance on 20NG with two document embeddings at \( K=50 \) and \( K=100 \) topics.

\( K = 100 \) topics. Table 5 show improvements on several evaluation metrics, indicating the potential benefits of integrating stronger encoders within the HiCOT framework in the future works.

4.5 Inference time

To evaluate the inference of our proposed method, HiCOT, we conducted experiments to assess the time required for inferring on the whole dataset against state-of-the-art (SOTA) methods, namely FASTopic and NeuroMax, as well as traditional methods, including ETM and ECRTM. As detailed in Table 6, HiCOT achieves significantly faster inference times than FASTopic while also demonstrating superior topic quality, as discussed in Section 4.2. Although traditional methods exhibit faster inference times, their topic quality metrics, as presented in Tables 1 and 2, are substantially lower compared to both the SOTA baselines and HiCOT. These findings underscore the effectiveness of our approach in mitigating the high inference costs typically associated with pretrained language models.

4.6 Visualization of Embedding Space

We visualize the learned topic and word embeddings using t-SNE (van der Maaten and Hinton, 2008) on the GoogleNews dataset with 50 topics. As shown in Figure 1, HiCOT effectively maintains
<table>
  <tr>
    <th>Dataset</th>
    <th>NeuroMax</th>
    <th>FASTopic</th>
    <th>HiCOT</th>
    <th>ETM</th>
    <th>ECRTM</th>
  </tr>
  <tr>
    <td>20NG</td>
    <td>2.98</td>
    <td>9.41</td>
    <td>4.01</td>
    <td>0.13</td>
    <td>1.25</td>
  </tr>
  <tr>
    <td>AGNews</td>
    <td>2.53</td>
    <td>4.87</td>
    <td>3.43</td>
    <td>0.11</td>
    <td>0.94</td>
  </tr>
  <tr>
    <td>IMDB</td>
    <td>5.92</td>
    <td>19.01</td>
    <td>9.31</td>
    <td>0.37</td>
    <td>2.34</td>
  </tr>
  <tr>
    <td>SearchSnippets</td>
    <td>3.17</td>
    <td>6.58</td>
    <td>4.47</td>
    <td>0.15</td>
    <td>1.17</td>
  </tr>
  <tr>
    <td>GoogleNews</td>
    <td>2.90</td>
    <td>5.91</td>
    <td>4.15</td>
    <td>0.12</td>
    <td>1.19</td>
  </tr>
</table>

Table 6: Inference time of various topic modeling methods across different datasets with 50 topics. All experiments are conducted on a NVIDIA RTX 3090 GPU.

![t-SNE visualization of word embeddings (●) and topic embeddings (▲) for GoogleNews dataset under 50 topics. Four subfigures labeled (a) ETM, (b) NeuroMax, (c) FASTopic, (d) HiCOT.](page_184_682_1082_312.png)

Figure 1: t-SNE visualization of word embeddings (●) and topic embeddings (▲) for GoogleNews dataset under 50 topics.

the structure of topic embeddings and prevents their collapse, whereas state-of-the-art baselines often suffer from embedding collapse or fail to capture clear word-topic relationships. This highlights the effectiveness of our method in clustering semantically coherent words. Additional visualizations for other datasets are provided in Appendix K.

4.7 Ablation study

We present an ablation study on the AGNews and Search Snippets datasets with 50 topics, to evaluate the contribution of each model component. Specifically, we define DT-SimpleNet as the model using \( \mathcal{L}_{TM} \), \( \mathcal{L}_{DT} \) and \( \mathcal{L}_{ECR} \). We then systematically remove the contrastive loss, include \( \mathcal{L}_{CLT} \) and \( \mathcal{L}_{CLC} \), as well as the update of transport plan \( Q \), subsequently evaluating their impact on model performance. Table 7 presents the results obtained. For document-topic distribution quality, evaluated by NMI and Purity, adding our components significantly improves performance, with DT-SimNet also exhibiting superior improvement. The model that integrates all components achieves the highest performance. Regarding topic quality, including coherence and diversity, the model remains competitive. In some datasets, our approach yields higher scores, while in others, the baseline is better.

<table>
  <tr>
    <th rowspan="2"></th>
    <th colspan="4">AGNews</th>
    <th colspan="4">SearchSnippets</th>
  </tr>
  <tr>
    <th>Cv.</th>
    <th>Purity</th>
    <th>NMI</th>
    <th>TD</th>
    <th>Cv.</th>
    <th>Purity</th>
    <th>NMI</th>
    <th>TD</th>
  </tr>
  <tr>
    <td>ECRTM</td>
    <td><b>0.466</b></td>
    <td>0.802</td>
    <td>0.367</td>
    <td>0.961</td>
    <td>0.450</td>
    <td>0.711</td>
    <td>0.419</td>
    <td>0.998</td>
  </tr>
  <tr>
    <td>DT-SimNet</td>
    <td>0.450</td>
    <td>0.848</td>
    <td>0.404</td>
    <td><b>0.992</b></td>
    <td>0.454</td>
    <td>0.800</td>
    <td>0.454</td>
    <td>1.000</td>
  </tr>
  <tr>
    <td>+ \( \mathcal{L}_{CLT} \)</td>
    <td>0.448</td>
    <td>0.855</td>
    <td>0.409</td>
    <td><b>0.992</b></td>
    <td>0.460</td>
    <td>0.815</td>
    <td>0.471</td>
    <td>1.000</td>
  </tr>
  <tr>
    <td>+ \( \mathcal{L}_{CLC} \)</td>
    <td>0.452</td>
    <td>0.852</td>
    <td>0.405</td>
    <td>0.987</td>
    <td>0.458</td>
    <td>0.815</td>
    <td>0.471</td>
    <td>1.000</td>
  </tr>
  <tr>
    <td>+ update \( Q \)</td>
    <td><u>0.452</u></td>
    <td>0.853</td>
    <td>0.408</td>
    <td>0.984</td>
    <td><u>0.463</u></td>
    <td>0.814</td>
    <td>0.471</td>
    <td>1.000</td>
  </tr>
  <tr>
    <td>HiCOT</td>
    <td>0.446</td>
    <td><b>0.857</b></td>
    <td><b>0.412</b></td>
    <td><b>0.992</b></td>
    <td>0.460</td>
    <td><b>0.818</b></td>
    <td><b>0.478</b></td>
    <td><b>1.000</b></td>
  </tr>
</table>

Table 7: Ablation study on AGNews and SearchSnippets datasets. The best and second-best results are highlighted in bold and underlined, respectively.

5 Conclusion

In conclusion, we introduce HiCOT, a novel neural topic modeling framework that integrates hierarchical clustering, contrastive learning, and optimal transport. By leveraging compact PLMs and optimal transport, HiCOT enhances document-topic alignment and captures semantic relationships between documents. Additionally, contrastive learning strengthens both topic-word associations and inter-topic interactions, resulting in more coherent topic representations. Comprehensive experiments on benchmark datasets demonstrate that HiCOT achieves superior performance in generating high-quality topics and document-topic distributions, providing a robust and scalable solution for neural topic modeling.

Limitations

While HiCOT has significantly improved topic quality, there are still some limitations. The selection of intervals for clustering updates needs to be carefully adjusted to ensure that model has sufficient time to learn stable semantic representations before applying clustering. However, excessively long intervals may slow convergence. An adaptive scheduling mechanism could further optimize this process. Additionally, the choice of positive and negative samples for contrastive learning remains an open research direction. These limitations suggest potential future studies for further improving the model.
Ethical Considerations

We comply with the ACL Code of Ethics and all relevant license terms. Our research in topic modeling is designed to enhance the field. When applied responsibly, it carries no significant societal risks.

Acknowledgments

This research has been supported by the NSF grant # 2239570. This research is also supported in part by the Office of the Director of National Intelligence (ODNI), Intelligence Advanced Research Projects Activity (IARPA), via the HIATUS Program contract 2022-22072200003. The views and conclusions contained herein are those of the authors and should not be interpreted as necessarily representing the official policies, either expressed or implied, of ODNI, IARPA, or the U.S. Government.

References

Dimo Angelov. 2020. Top2vec: Distributed representations of topics. arXiv preprint arXiv:2008.09470.

Dimo Angelov and Diana Inkpen. 2024. Topic modeling: Contextual token embeddings are all you need. In Findings of the Association for Computational Linguistics: EMNLP 2024, pages 13528–13539, Miami, Florida, USA. Association for Computational Linguistics.

Tran Xuan Bach, Nguyen Duc Anh, Ngo Van Linh, and Khoat Than. 2021. Dynamic transformation of prior knowledge into bayesian models for data streams. IEEE Transactions on Knowledge and Data Engineering, 35(4):3742–3750.

Federico Bianchi, Silvia Terragni, and Dirk Hovy. 2021. Pre-training is a hot topic: Contextualized document embeddings improve topic coherence. In ACL-IJCNLP (Volume 2: Short Papers), pages 759–766. Association for Computational Linguistics.

David M Blei, Andrew Y Ng, and Michael I Jordan. 2003. Latent dirichlet allocation. Journal of Machine Learning Research, 3:993–1022.

Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020. Language models are few-shot learners. In Advances in Neural Information Processing Systems, 33, pages 1877–1901.

Ricardo JGB Campello, Davoud Moulavi, and Jörg Sander. 2013. Density-based clustering based on hierarchical density estimates. In Pacific-Asia conference on knowledge discovery and data mining, pages 160–172. Springer.

Marco Cuturi. 2013a. Sinkhorn distances: Lightspeed computation of optimal transport. In Advances in Neural Information Processing Systems, 26. Curran Associates, Inc.

Marco Cuturi. 2013b. Sinkhorn distances: Lightspeed computation of optimal transportation distances. Preprint, arXiv:1306.0895.

Jacob Devlin. 2018. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805.

Adji B Dieng, Francisco JR Ruiz, and David M Blei. 2020. Topic modeling in embedding spaces. Transactions of the Association for Computational Linguistics, 8:439–453.

Anh Nguyen Duc, Ngo Van Linh, Anh Nguyen Kim, and Khoat Than. 2017. Keeping priors in streaming bayesian learning. In Advances in Knowledge Discovery and Data Mining, pages 247–258, Cham. Springer International Publishing.

Thomas Griffiths, Michael Jordan, Joshua Tenenbaum, and David Blei. 2003. Hierarchical topic models and the nested chinese restaurant process. In Advances in Neural Information Processing Systems, 16.

Maarten Grootendorst. 2022. Bertopic: Neural topic modeling with a class-based tf-idf procedure. CoRR, abs/2203.05794.

Cuong Ha, Van-Dang Tran, Linh Ngo Van, and Khoat Than. 2019. Eliminating overfitting of probabilistic topic models on short and noisy text: The role of dropout. International Journal of Approximate Reasoning, 112:85–104.

Sungwon Han, Mingi Shin, Sungkyu Park, Changwook Jung, and Meeyoung Cha. 2023. Unified neural topic model via contrastive learning and term weighting. In Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics, pages 1802–1817. Association for Computational Linguistics.

Thomas Hofmann. 1999. Probabilistic latent semantic indexing. In Proceedings of the 22nd annual international ACM SIGIR conference on Research and development in information retrieval, pages 50–57.

Diederik P. Kingma and Jimmy Ba. 2015. Adam: A method for stochastic optimization. In 3rd International Conference on Learning Representations, ICLR 2015.

Diederik P Kingma and Max Welling. 2013. Auto-encoding variational bayes. In 2nd International Conference on Learning Representations, ICLR 2014.

Ken Lang. 1995. Newsweeder: Learning to filter net-news. In Machine Learning Proceedings 1995, pages 331–339, San Francisco (CA). Morgan Kaufmann.
Quoc V. Le and Tomas Mikolov. 2014. Distributed representations of sentences and documents. Preprint, arXiv:1405.4053.

Raymond Li, Felipe Gonzalez-Pizarro, Linzi Xing, Gabriel Murray, and Giuseppe Carenini. 2023. Diversity-aware coherence loss for improving neural topic models. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 1710–1722, Toronto, Canada. Association for Computational Linguistics.

Andrew L. Maas, Raymond E. Daly, Peter T. Pham, Dan Huang, Andrew Y. Ng, and Christopher Potts. 2011. Learning word vectors for sentiment analysis. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies, pages 142–150, Portland, Oregon, USA. Association for Computational Linguistics.

J MacQueen. 1967. Some methods for classification and analysis of multivariate observations. In Proceedings of 5-th Berkeley Symposium on Mathematical Statistics and Probability/University of California Press.

Khai Mai, Sang Mai, Anh Nguyen, Ngo Van Linh, and Khoat Than. 2016. Enabling hierarchical dirichlet processes to work better for short texts at large scale. In Advances in Knowledge Discovery and Data Mining: 20th Pacific-Asia Conference, PAKDD 2016, Auckland, New Zealand, April 19-22, 2016, Proceedings, Part II 20, pages 431–442. Springer.

Christopher D. Manning, Prabhakar Raghavan, and Hinrich Schütze. 2008. Introduction to Information Retrieval. Cambridge University Press.

Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. 2013. Efficient estimation of word representations in vector space. In 1st International Conference on Learning Representations, ICLR 2013.

Fionn Murtagh and Pedro Contreras. 2012. Algorithms for hierarchical clustering: an overview. Wiley Interdisciplinary Reviews: Data Mining and Knowledge Discovery, 2(1):86–97.

Duc Anh Nguyen, Kim Anh Nguyen, Canh Hao Nguyen, Khoat Than, et al. 2021. Boosting prior knowledge in streaming variational bayes. Neurocomputing, 424:143–159.

Ha Nguyen, Hoang Pham, Son Nguyen, Ngo Van Linh, and Khoat Than. 2022a. Adaptive infinite dropout for noisy and sparse data streams. Machine Learning, 111(8):3025–3060.

Quang Duc Nguyen, Tung Nguyen, Duc Anh Nguyen, Linh Ngo Van, Sang Dinh, and Thien Huu Nguyen. 2025a. Glocom: A short text neural topic model via global clustering context. In Proceedings of the Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies.

Thong Nguyen and Anh Tuan Luu. 2021. Contrastive learning for neural topic model. Preprint, arXiv:2110.12764.

Tung Nguyen, Tue Le, Hoang Tran Vuong, Quang Duc Nguyen, Duc Anh Nguyen, Linh Ngo Van, Sang Dinh, and Thien Huu Nguyen. 2025b. Sharpness-aware minimization for topic models with high-quality document representations. In Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), pages 4507–4524.

Tung Nguyen, Trung Mai, Nam Nguyen, Linh Ngo Van, and Khoat Than. 2022b. Balancing stability and plasticity when learning topic models from short and noisy text streams. Neurocomputing, 505:30–43.

Tung Nguyen, Tung Pham, Linh Ngo Van, Ha-Bang Ban, and Khoat Than. 2025c. Out-of-vocabulary handling and topic quality control strategies in streaming topic models. Neurocomputing, 614:128757.

Tung Nguyen, Linh Ngo Van, Anh Nguyen Duc, and Sang Dinh Viet. 2025d. A framework for neural topic modeling with mutual information and group regularization. Neurocomputing, page 130420.

Van-Son Nguyen, Duc-Tung Nguyen, Linh Ngo Van, and Khoat Than. 2019. Infinite dropout for training bayesian models from data streams. In 2019 IEEE International Conference on Big Data (Big Data), pages 125–134.

Aaron van den Oord, Yazhe Li, and Oriol Vinyals. 2018. Representation learning with contrastive predictive coding. arXiv preprint arXiv:1807.03748.

Jeffrey Pennington, Richard Socher, and Christopher D Manning. 2014. Glove: Global vectors for word representation. In Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP), pages 1532–1543.

Gabriel Peyré and Marco Cuturi. 2018. Computational optimal transport. Found. Trends Mach. Learn., 11(5-6):355–607.

Gabriel Peyré and Marco Cuturi. 2020. Computational optimal transport. Preprint, arXiv:1803.00567.

Chau Minh Pham, Alexander Hoyle, Simeng Sun, and Mohit Iyyer. 2024a. Topicpt: A prompt-based topic modeling framework. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), pages 2956–2984.

Duy-Tung Pham, Thien Trang Nguyen Vu, Tung Nguyen, Linh Ngo Van, Duc Anh Nguyen, and Thien Huu Nguyen. 2024b. Neuromax: Enhancing neural topic modeling via maximizing mutual information and group topic regularization. In Findings of the Association for Computational Linguistics: EMNLP 2024.
Xuan-Hieu Phan, Le-Minh Nguyen, and Susumu Horiguchi. 2008. Learning to classify short and sparse text & web with hidden topics from large-scale data collections. In Proceedings of the 17th international conference on World Wide Web, pages 91–100.

Jipeng Qiang, Qian Zhenyu, Yun Li, Yunhao Yuan, and Xindong Wu. 2022. Short text topic modeling techniques, applications, and performance: A survey. IEEE Trans. Knowl. Data Eng., pages 1427–1445.

N Reimers. 2019. Sentence-bert: Sentence embeddings using siamese bert-networks. arXiv preprint arXiv:1908.10084.

Nils Reimers and Iryna Gurevych. 2019a. Sentence-bert: Sentence embeddings using siamese bert-networks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 3982–3992. Association for Computational Linguistics.

Nils Reimers and Iryna Gurevych. 2019b. Sentence-bert: Sentence embeddings using siamese bert-networks. Preprint, arXiv:1908.10084.

Michael Röder, Andreas Both, and Alexander Hinneburg. 2015. Exploring the space of topic coherence measures. In Proceedings of the Eighth ACM International Conference on Web Search and Data Mining, page 399–408. Association for Computing Machinery.

Florian Schroff, Dmitry Kalenichenko, and James Philbin. 2015. Facenet: A unified embedding for face recognition and clustering. In 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), page 815–823. IEEE.

Suzanna Sia, Ayush Dalmia, and Sabrina J Mielke. 2020. Tired of topic models? clusters of pretrained word embeddings make for fast and good topics too! In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 1728–1736. Association for Computational Linguistics.

Akash Srivastava and Charles Sutton. 2017. Autoencoding variational inference for topic models. In International Conference on Learning Representations.

Dominik Stanmbach, Vilém Zouhar, Alexander Hoyle, Mrinmaya Sachan, and Elliott Ash. 2023. Revisiting automated topic model evaluation with large language models. arXiv preprint arXiv:2305.12152.

Yifan Sun, Changmao Cheng, Yuhan Zhang, Chi Zhang, Liang Zheng, Zhongdao Wang, and Yichen Wei. 2020. Circle loss: A unified perspective of pair similarity optimization. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 6398–6407.

Anh Phan Tuan, Bach Tran, Thien Huu Nguyen, Linh Ngo Van, and Khoat Than. 2020. Bag of biterms modeling for short texts. Knowledge and Information Systems, 62(10):4055–4090.

Laurens van der Maaten and Geoffrey Hinton. 2008. Visualizing data using t-sne. Journal of Machine Learning Research, 9(86):2579–2605.

Ngo Van Linh, Tran Xuan Bach, and Khoat Than. 2022. A graph convolutional topic model for short and noisy text streams. Neurocomputing, 468:345–359.

Dongsheng Wang, Dandan Guo, He Zhao, Huangjie Zheng, Korawat Tanwisuth, Bo Chen, and Mingyuan Zhou. 2022. Representing mixtures of word embeddings with mixtures of topic embeddings. In The Tenth International Conference on Learning Representations, ICLR 2022.

H. Wang, N. Prakash, N. Hoang, M. Hee, U. Naseem, and R. Lee. 2023. Prompting large language models for topic modeling. In 2023 IEEE International Conference on Big Data (BigData), pages 1236–1241.

Xiaobao Wu, Xinshuai Dong, Thong Nguyen, Chaoqun Liu, Liang-Ming Pan, and Anh Tuan Luu. 2023a. Infoctm: A mutual information maximization perspective of cross-lingual topic modeling. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 37, pages 13763–13771.

Xiaobao Wu, Xinshuai Dong, Thong Thanh Nguyen, and Anh Tuan Luu. 2023b. Effective neural topic modeling with embedding clustering regularization. In International Conference on Machine Learning, pages 37335–37357.

Xiaobao Wu, Thong Nguyen, and Anh Tuan Luu. 2024a. A survey on neural topic models: Methods, applications, and challenges. Artificial Intelligence Review, 57(2):1–30.

Xiaobao Wu, Thong Nguyen, Delvin Ce Zhang, William Yang Wang, and Anh Tuan Luu. 2024b. Fastopic: A fast, adaptive, stable, and transferable topic modeling paradigm. In Advances in Neural Information Processing Systems.

Xiaobao Wu, Thong Nguyen, Delvin Ce Zhang, William Yang Wang, and Anh Tuan Luu. 2024c. Fastopic: A fast, adaptive, stable, and transferable topic modeling paradigm. arXiv preprint arXiv:2405.17978.

Xiaobao Wu, Fengjun Pan, Thong Nguyen, Yichao Feng, Chaoqun Liu, Cong-Duy Nguyen, and Anh Tuan Luu. 2024d. On the affinity, rationality, and diversity of hierarchical topic modeling. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pages 19261–19269.

Weijie Xu, Xiaoyu Jiang, Srinivasan Sengamedu Hanumantha Rao, Francis Iannacci, and Jinjin Zhao. 2023. vontss: vmt based semi-supervised neural topic modeling with optimal transport. In Findings of the Association for Computational Linguistics: ACL 2023,
page 4433–4457. Association for Computational Linguistics.

Yishi Xu, Dongsheng Wang, Bo Chen, Ruiling Lu, Zhibin Duan, and Mingyuan Zhou. 2022. Hyperminer: Topic taxonomy mining with hyperbolic embedding. Preprint, arXiv:2210.10625.

Jianhua Yin and Jianyong Wang. 2016. A model-based approach for text clustering with outlier detection. In 2016 IEEE 32nd International Conference on Data Engineering (ICDE), pages 625–636.

Xiang Zhang, Junbo Zhao, and Yann LeCun. 2015. Character-level convolutional networks for text classification. In Advances in Neural Information Processing Systems, 28. Curran Associates, Inc.

Zihan Zhang, Meng Fang, Ling Chen, and Mohammad Reza Namazi Rad. 2022. Is neural topic modelling better than clustering? an empirical study on clustering with contextual embeddings for topics. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 3886–3893. Association for Computational Linguistics.

He Zhao, Dinh Phung, Viet Huynh, Trung Le, and Wray Buntine. 2021. Neural topic model via optimal transport. In 9th International Conference on Learning Representations, ICLR 2021.

He Zhao, Dinh Phung, Viet Huynh, Trung Le, and Wray Buntine. 2022. Neural topic model via optimal transport. Preprint, arXiv:2008.13537.
Algorithm 1 Learning HiCOT

Input: Document collection \( X = \{ x_d \}_{d=1}^D \), number of topics \( K \), pretrained word embedding \( W_{\text{pretrained}} \), pretrained document embedding model \( f_{\text{doc}} \), total number of training epochs \( N \), threshold epoch for clustering \( M \), clustering update interval \( I \).
Output: Model parameters \( \delta \), word embedding \( W \), topic embedding \( T \).
Initialize \( W = W_{\text{pretrained}} \)
Random initialize \( T \)
Initialize document embedding \( E_i \) with \( e_i = f_{\text{doc}}(x_i) \) for \( i = \{1, 2, \cdots, D\} \)
for \( t = 1, 2, \ldots, N \) do
    for each minibatch \( B \) do
        if \( t < M \) then
            // Stage 1
            Update \( Q \) following the Equation 7
            Update \( \pi \) as the solution of problem 5 by Sinkhorn’s algorithm
            Update \( \psi \) using Sinkhorn algorithm to sovle problem 14
            Calculate \( \mathcal{L} = \mathcal{L}_{TM} + \lambda_{DT}\mathcal{L}_{DT} + \lambda_{ECR}\mathcal{L}_{ECR} \)
            Update \( W, T, E, \delta \) with a gradient step based on the loss \( \mathcal{L} \).
        else
            // Stage 2
            if \( t = M \) then
                Use hierarchical clustering algorithm to perform clustering
            end if
            Update \( Q \) according to Equation 7
            Update \( \pi, \psi \) using Sinkhorn algorithm to sovle problem 5 and 14, respectively
            Calculate \( \mathcal{L}_{CL} = \lambda_{CLT}\mathcal{L}_{CLT} + \lambda_{CLC}\mathcal{L}_{CLC} \)
            Calculate \( \mathcal{L} = \mathcal{L}_{CL} + \mathcal{L}_{TM} + \lambda_{DT}\mathcal{L}_{DT} + \lambda_{ECR}\mathcal{L}_{ECR} \)
            Update \( W, T, E, \delta \) with a gradient step based on the loss \( \mathcal{L} \).
            if \( t \% I = 0 \) then
                Use hierarchical clustering algorithm to perform re-clustering
            end if
        end if
    end for
end for

A Algorithm

The detailed training algorithm for HiCOT is presented in Algorithm A.

B Related work

Topic Models and Neural Topic Models. The goal of topic modeling is to find hidden topics in a corpus of documents. Traditionally, this has been tackled with graphical probabilistic models (Hofmann, 1999; Blei et al., 2003), with extensions for specialized settings such as short texts (Tuan et al., 2020; Ha et al., 2019; Nguyen et al., 2022a; Mai et al., 2016) and streaming data (Duc et al., 2017; Nguyen et al., 2019, 2022b, 2025c, 2021). More recently, neural models have gained prominence for their superior generalization and performance (Wu et al., 2024a; Srivastava and Sutton, 2017; Dieng et al., 2020; Wu et al., 2023b; Pham et al., 2024b). Most neural topic models are based on the Variational Autoencoder (VAE) framework (Kingma and Welling, 2013), where an encoder infers a document’s topic distribution and a decoder reconstructs the text using a topic-word distribution. Recent improvements include integrating pre-trained language model (PLM) or word embeddings (Bach et al., 2021; Van Linh et al., 2022; Reimers, 2019; Brown et al., 2020) into the encoder (Wu et al., 2023a; Han et al., 2023). Other approaches directly cluster document representations to form topics (Grootendorst, 2022; Sia et al., 2020; Zhang et al., 2022), though they often lack clear document-specific topic proportions. Additionally, large language models have been used to generate conceptual topic descriptions (Wang et al., 2023; Pham et al., 2024a), but these methods typically struggle with providing detailed word distributions. Notably, Wu et al. (2024b) propose an Optimal Transport-based framework (Peyré and Cuturi, 2018) that effectively captures the semantic relationships among documents, topics, and word embeddings.

Clustering methods. Clustering plays a key role in unsupervised learning by grouping data based on similarities. Traditional algorithms like KMeans (MacQueen, 1967), Hierarchical Agglomerative Clustering (HAC) (Murtagh and Contreras, 2012), and HDBSCAN (Campello et al., 2013) are widely used. In topic modeling, these methods are applied to cluster documents (based on topic distributions
<table>
  <tr>
    <th colspan="2">20NG</th>
    <th>C_V</th>
    <th>Purity</th>
    <th>NMI</th>
    <th>TD</th>
  </tr>
  <tr>
    <th>Model</th>
    <th>Distance metric</th>
    <th></th>
    <th></th>
    <th></th>
    <th></th>
  </tr>
  <tr>
    <td rowspan="2">HiCOT + HAC</td>
    <td>Euclidean</td>
    <td>0.451</td>
    <td>0.626</td>
    <td>0.583</td>
    <td>0.852</td>
  </tr>
  <tr>
    <td>Cosine</td>
    <td>0.447</td>
    <td>0.624</td>
    <td>0.580</td>
    <td>0.851</td>
  </tr>
  <tr>
    <td rowspan="2">HiCOT + HDBSCAN</td>
    <td>Euclidean</td>
    <td>0.449</td>
    <td>0.628</td>
    <td>0.580</td>
    <td>0.863</td>
  </tr>
  <tr>
    <td>Cosine</td>
    <td>0.445</td>
    <td>0.630</td>
    <td>0.585</td>
    <td>0.856</td>
  </tr>
  <tr>
    <th colspan="2">GoogleNews</th>
    <th>C_V</th>
    <th>Purity</th>
    <th>NMI</th>
    <th>TD</th>
  </tr>
  <tr>
    <th>Model</th>
    <th>Distance metric</th>
    <th></th>
    <th></th>
    <th></th>
    <th></th>
  </tr>
  <tr>
    <td rowspan="2">HiCOT + HAC</td>
    <td>Euclidean</td>
    <td>0.454</td>
    <td>0.465</td>
    <td>0.657</td>
    <td>0.920</td>
  </tr>
  <tr>
    <td>Cosine</td>
    <td>0.457</td>
    <td>0.466</td>
    <td>0.655</td>
    <td>0.923</td>
  </tr>
  <tr>
    <td rowspan="2">HiCOT + HDBSCAN</td>
    <td>Euclidean</td>
    <td>0.458</td>
    <td>0.464</td>
    <td>0.653</td>
    <td>0.921</td>
  </tr>
  <tr>
    <td>Cosine</td>
    <td>0.458</td>
    <td>0.464</td>
    <td>0.654</td>
    <td>0.921</td>
  </tr>
</table>

Table 8: Impact of Hierarchical Clustering Methods on the 20NG and GoogleNews Datasets with 50 Topics, Using Euclidean and Cosine Distance Metrics for Contrastive Learning.

from LDA (Blei et al., 2003) or neural models) (Wu et al., 2024a), words (using embeddings like Word2Vec (Mikolov et al., 2013) or BERT (Devlin, 2018), or directly for topic formation (clustering document embeddings, e.g., Top2Vec (Angelov, 2020), BERTopic (Grootendorst, 2022)). Recent advances integrate clustering with neural approaches, showing superior performance, particularly when clustering embeddings.

C  Impact of Clustering Algorithm and Distance Metric on Contrastive Learning

To assess the impact of hierarchical clustering algorithms, we conduct experiments with HAC (Murtagh and Contreras, 2012) and HDBSCAN (Campello et al., 2013). In contrastive learning, we also evaluate different distance metrics, using same metric for \( \mathcal{L}_{CLT} \) and \( \mathcal{L}_{CLC} \) for simplicity. Our experiment considers two metrics: Euclidean and Cosine, on the 20NG and GoogleNews datasets, with \( K = 50 \) topics. As demonstrated in Table 8, the selection of hierarchical clustering algorithms and distance metrics within contrastive learning framework has minimal impact on model performance. This robustness highlights that the effectiveness of our approach mainly comes from its architectural and algorithmic innovations, rather than dependence on choosing clustering algorithms or distance metrics.

D  Comparison with Other Neural Topic Models

D.1  Comparison with Clustering-based Neural Topic Models

<table>
  <tr>
    <th rowspan="2">50 Topics</th>
    <th colspan="4">20NG</th>
    <th colspan="4">AGNews</th>
    <th colspan="4">IMDB</th>
  </tr>
  <tr>
    <th>C_V</th>
    <th>Purity</th>
    <th>NMI</th>
    <th>TD</th>
    <th>C_V</th>
    <th>Purity</th>
    <th>NMI</th>
    <th>TD</th>
    <th>C_V</th>
    <th>Purity</th>
    <th>NMI</th>
    <th>TD</th>
  </tr>
  <tr>
    <td>BERTopic</td>
    <td>0.382</td>
    <td>0.376</td>
    <td>0.448</td>
    <td>0.680</td>
    <td>0.390</td>
    <td>0.687</td>
    <td>0.340</td>
    <td>0.735</td>
    <td>0.341</td>
    <td>0.677</td>
    <td><b>0.086</b></td>
    <td>0.505</td>
  </tr>
  <tr>
    <td>C-Top2Vec</td>
    <td>0.408</td>
    <td>0.564</td>
    <td>0.496</td>
    <td>0.577</td>
    <td>0.371</td>
    <td>0.842</td>
    <td>0.356</td>
    <td>0.417</td>
    <td>0.403</td>
    <td>0.686</td>
    <td>0.049</td>
    <td>0.149</td>
  </tr>
  <tr>
    <td>HiCOT</td>
    <td><b>0.451</b></td>
    <td><b>0.626</b></td>
    <td><b>0.583</b></td>
    <td><b>0.852</b></td>
    <td><b>0.446</b></td>
    <td><b>0.857</b></td>
    <td><b>0.412</b></td>
    <td><b>0.992</b></td>
    <td><b>0.404</b></td>
    <td><b>0.737</b></td>
    <td><b>0.082</b></td>
    <td><b>0.837</b></td>
  </tr>
</table>

Table 9: Evaluation results on standard datasets, assessed using C_V, TD, Purity and NMI under \( K = 50 \) topics. The highest-performing results are marked in bold, while the second-best values are underlined.

We conducted experiments on three standard datasets with 50 topics, comparing two clustering-based models: BERTopic (Grootendorst, 2022), which clusters document embeddings and discovers topics using
TF-IDF, and C-Top2Vec (Angelov and Inkpen, 2024), which leverages contextual token embeddings for multi-vector document representations. Our experimental results (see Table 9) demonstrate that HiCOT consistently achieves superior performance across all datasets. Specifically, HiCOT attains the highest scores across all metrics. This indicates that the topics generated by HiCOT not only enhance the quality of document-topic distributions but are also semantically coherent and sufficiently diverse, effectively mitigating topic collapse.

In contrast, although C-Top2Vec achieves the second-highest \( C_V \) score, it suffers from extremely low TD values. These low TD scores suggest that the model’s topics lack the necessary diversity to capture a wide range of themes. Consequently, this leads to significant topic redundancy, reducing the model’s ability to generate well-separated and meaningful topics. As illustrated in Table 20, this highlights the fundamental limitations of C-Top2Vec in maintaining both topic quality and diversity. These findings further underscore HiCOT’s effectiveness as a robust solution for clustering-based neural topic modeling

D.2 Comparison with Auxiliary Loss-based Models

To further evaluate the effectiveness of our framework, we compare it against a recent approach that incorporates auxiliary training objectives to enhance topic coherence. Specifically, we consider the method proposed by Li et al. (2023), which introduces a novel auxiliary loss function designed to improve the performance of neural topic models.

We conducted experiments by integrating this auxiliary loss two baseline models: ETM and ECRTM (denoted as ETM + Aux and ECRTM + Aux, respectively). We then compare these models with their HiCOT-enhanced counterparts (ETM + HiCOT and ECRTM + HiCOT) on the 20NG dataset with \( K = 50 \) topics. The results are presented in Table 10. Our findings show that models augmented with HiCOT consistently outperform those using the auxiliary loss from Li et al. (2023) across all metrics. These results highlight the strength of HiCOT in enhancing topic modeling performance.

<table>
  <tr>
    <th rowspan="2">Model</th>
    <th colspan="4">ETM</th>
    <th colspan="4">ECRTM</th>
  </tr>
  <tr>
    <th>\( C_V \)</th>
    <th>Purity</th>
    <th>NMI</th>
    <th>TD</th>
    <th>\( C_V \)</th>
    <th>Purity</th>
    <th>NMI</th>
    <th>TD</th>
  </tr>
  <tr>
    <td>Baseline</td>
    <td>0.375</td>
    <td>0.347</td>
    <td>0.319</td>
    <td>0.704</td>
    <td>0.431</td>
    <td>0.560</td>
    <td>0.524</td>
    <td><b>0.964</b></td>
  </tr>
  <tr>
    <td>+ Auxiliary Loss</td>
    <td>0.374</td>
    <td>0.420</td>
    <td>0.394</td>
    <td>0.681</td>
    <td>0.427</td>
    <td>0.593</td>
    <td>0.547</td>
    <td>0.922</td>
  </tr>
  <tr>
    <td>+ HiCOT (ours)</td>
    <td><b>0.444</b></td>
    <td><b>0.581</b></td>
    <td><b>0.493</b></td>
    <td><b>0.756</b></td>
    <td><b>0.451</b></td>
    <td><b>0.626</b></td>
    <td><b>0.583</b></td>
    <td>0.852</td>
  </tr>
</table>

Table 10: Comparison of ETM and ECRTM variants on the 20NG dataset with 50 topics. The highest-performing results are marked in bold.

D.3 Comparison with the CombinedTM VAE Baseline

We evaluated the performance of our proposed HiCOT against CombinedTM baseline (Bianchi et al., 2021) on three standard benchmark datasets: 20NG, AGNews, and IMDB. Experiments were conducted with topic numbers set to 50 and 100. As shown in Table 11, HiCOT consistently outperforms CombinedTM across all datasets and metrics. These results demonstrate the effectiveness of our approach relative to CombinedTM (Bianchi et al., 2021), highlighting its effectiveness within the class of VAE-based topic models.

D.4 Comparison with Short-Text Topic Models

We have conducted comparative experiments with GloCOM (Nguyen et al., 2025a) on the SearchSnippets and GoogleNews datasets (with \( K = 50 \) and \( K = 100 \) topics). It is pertinent to note that GloCOM is specifically designed for short-text topic modeling, leverages aggregation techniques tailored for this data type. As shown in Table 12, our proposed method, without necessitating specialized short-text strategies, achieves good performance in terms of overall topic quality on the SearchSnippets dataset compared to GloCOM. This result underscores the effectiveness of our approach.

Furthermore, our method has the potential to be integrated with techniques used in GloCOM. A potential direction for future work is incorporating clustering mechanisms, or designing objective functions based
<table>
  <tr>
    <th rowspan="2">Topics</th>
    <th rowspan="2">Model</th>
    <th colspan="4">20NG</th>
    <th colspan="4">AGNews</th>
    <th colspan="4">IMDB</th>
  </tr>
  <tr>
    <th>C_V</th>
    <th>Purity</th>
    <th>NMI</th>
    <th>TD</th>
    <th>C_V</th>
    <th>Purity</th>
    <th>NMI</th>
    <th>TD</th>
    <th>C_V</th>
    <th>Purity</th>
    <th>NMI</th>
    <th>TD</th>
  </tr>
  <tr>
    <td rowspan="2">50</td>
    <td>CTM</td>
    <td>0.426</td>
    <td>0.565</td>
    <td>0.465</td>
    <td>0.792</td>
    <td><b>0.460</b></td>
    <td>0.801</td>
    <td>0.324</td>
    <td>0.950</td>
    <td>0.395</td>
    <td>0.687</td>
    <td>0.051</td>
    <td>0.709</td>
  </tr>
  <tr>
    <td>HiCOT</td>
    <td><b>0.451</b></td>
    <td><b>0.626</b></td>
    <td><b>0.583</b></td>
    <td><b>0.852</b></td>
    <td>0.446</td>
    <td><b>0.857</b></td>
    <td><b>0.412</b></td>
    <td><b>0.992</b></td>
    <td><b>0.404</b></td>
    <td><b>0.737</b></td>
    <td><b>0.082</b></td>
    <td><b>0.837</b></td>
  </tr>
  <tr>
    <td rowspan="2">100</td>
    <td>CTM</td>
    <td><b>0.424</b></td>
    <td>0.591</td>
    <td>0.465</td>
    <td>0.718</td>
    <td>0.415</td>
    <td>0.827</td>
    <td>0.344</td>
    <td>0.648</td>
    <td>0.376</td>
    <td>0.686</td>
    <td>0.042</td>
    <td>0.476</td>
  </tr>
  <tr>
    <td>HiCOT</td>
    <td><b>0.424</b></td>
    <td><b>0.652</b></td>
    <td><b>0.568</b></td>
    <td><b>0.741</b></td>
    <td><b>0.435</b></td>
    <td><b>0.861</b></td>
    <td><b>0.388</b></td>
    <td><b>0.960</b></td>
    <td><b>0.388</b></td>
    <td><b>0.739</b></td>
    <td><b>0.071</b></td>
    <td><b>0.733</b></td>
  </tr>
</table>

Table 11: Performance comparison of HiCOT and CombinedTM on 20NG, AGNews, and IMDB datasets for 50 and 100 topics, using C_V, Purity, NMI and TD. The highest-performing results are marked in bold.

on Optimal Transport (OT) and contrastive learning. These enhancements have the potential to further improve our approach, and we intend to explore them in future research.

<table>
  <tr>
    <th rowspan="2">Topics</th>
    <th rowspan="2">Model</th>
    <th colspan="4">SearchSnippets</th>
    <th colspan="4">GoogleNews</th>
  </tr>
  <tr>
    <th>C_V</th>
    <th>Purity</th>
    <th>NMI</th>
    <th>TD</th>
    <th>C_V</th>
    <th>Purity</th>
    <th>NMI</th>
    <th>TD</th>
  </tr>
  <tr>
    <td rowspan="2">50</td>
    <td>GloCOM</td>
    <td>0.453</td>
    <td>0.806</td>
    <td>0.502</td>
    <td>0.956</td>
    <td>0.475</td>
    <td>0.586</td>
    <td>0.817</td>
    <td>0.999</td>
  </tr>
  <tr>
    <td>HiCOT</td>
    <td>0.460</td>
    <td>0.818</td>
    <td>0.478</td>
    <td>1.000</td>
    <td>0.454</td>
    <td>0.465</td>
    <td>0.657</td>
    <td>0.920</td>
  </tr>
  <tr>
    <td rowspan="2">100</td>
    <td>GloCOM</td>
    <td>0.443</td>
    <td>0.822</td>
    <td>0.501</td>
    <td>0.920</td>
    <td>0.450</td>
    <td>0.761</td>
    <td>0.900</td>
    <td>0.944</td>
  </tr>
  <tr>
    <td>HiCOT</td>
    <td>0.449</td>
    <td>0.857</td>
    <td>0.480</td>
    <td>0.940</td>
    <td>0.470</td>
    <td>0.763</td>
    <td>0.864</td>
    <td>0.802</td>
  </tr>
</table>

Table 12: Comparison between GloCOM and HiCOT on SearchSnippets and GoogleNews datasets.

E Text Classification

To assess extrinsic performance, we conduct text classification experiments as downstream tasks following (Wu et al., 2023b). Specifically, we utilize document-topic distributions generated by topic models as document features and train SVMs to predict class of each document. For evaluation, we employ three standard datasets with 50 topics. Performance is evaluated using Accuracy (Acc) and F1 score. As shown in Figure 2, HiCOT achieves performance comparable to recent state-of-the-art models, such as NeuroMax and FASTopic, while being better than other baseline methods across three datasets. Moreover, it notably enhances topic quality, as discussed in Section 4.2. These results highlight the effectiveness of HiCOT in downstream classification tasks.

F Topic Model Evaluation with Large Language Models

Following (Stammbach et al., 2023), we utilize two LLM-based evaluation tasks to assess topic model quality: Rating Task and Intruder Detection Task. In the rating task, an LLM assigns a coherence score to topic words on a scale from 1 to 3 ("1" = not very related, "2" = moderately related, "3" = very related), reflecting the semantic relatedness of the top 10 words per topic. With intruder detection task, we randomly select five words from the top 10 topic words and introduce an intruder word from a different topic (not among the top 50 words of the current topic), challenging the LLM to identify the outlier. In all experiments, we use the gemini-2.0-flash model.

For the rating task, Table 13 presents the mean of LLM ratings over three evaluations across 50 topics for each model and dataset. The results indicate that HiCOT achieves the highest average scores, underscoring HiCOT’s superior topic coherence. For the intruder detection task, Figure 3 illustrates the distribution of detection accuracies across three test series per topic for 50 topics, with accuracies averaged over these series for each model. HiCOT exhibits the highest median accuracy, suggesting that our method generates more distinct and coherent topics.
Figure 2: Comparison of text classification performance for HiCOT and baseline models across three standard datasets with 50 topics.

Figure 3: Distribution of average intruder detection accuracies across three test series per topic for 50 topics on the AGNews dataset.
<table>
  <tr>
    <th>Dataset</th>
    <th>ETM</th>
    <th>ECRTM</th>
    <th>NeuroMax</th>
    <th>FASTopic</th>
    <th>HiCOT</th>
  </tr>
  <tr>
    <td>20NG</td>
    <td>2.01</td>
    <td>1.96</td>
    <td>1.99</td>
    <td>1.01</td>
    <td>2.04</td>
  </tr>
  <tr>
    <td>SearchSnippets</td>
    <td>1.78</td>
    <td>1.45</td>
    <td>1.80</td>
    <td>1.83</td>
    <td>1.88</td>
  </tr>
  <tr>
    <td>GoogleNews</td>
    <td>1.37</td>
    <td>1.36</td>
    <td>1.67</td>
    <td>1.20</td>
    <td>1.74</td>
  </tr>
</table>

Table 13: Average scores for 50 topics across datasets and models, based on LLM ratings over three evaluation runs per topic.

<table>
  <tr>
    <th>Dataset</th>
    <th># of texts</th>
    <th>average text length</th>
    <th># of labels</th>
    <th>vocab size</th>
  </tr>
  <tr>
    <td>20NG</td>
    <td>18846</td>
    <td>110.5</td>
    <td>20</td>
    <td>5000</td>
  </tr>
  <tr>
    <td>AGNews</td>
    <td>12500</td>
    <td>20.1</td>
    <td>4</td>
    <td>5000</td>
  </tr>
  <tr>
    <td>IMDB</td>
    <td>50000</td>
    <td>95.0</td>
    <td>2</td>
    <td>5000</td>
  </tr>
  <tr>
    <td>SearchSnippets</td>
    <td>12294</td>
    <td>14.4</td>
    <td>8</td>
    <td>4618</td>
  </tr>
  <tr>
    <td>GoogleNews</td>
    <td>11019</td>
    <td>5.8</td>
    <td>152</td>
    <td>3473</td>
  </tr>
</table>

Table 14: Dataset statistics after preprocessing.

G Experiment Details

G.1 Implementation Details.

All experiments are performed on a system featuring a GeForce RTX 3090 GPU (24GB RAM), using PyTorch 2.1.0 with CUDA 12.1 in a Python 3.12 environment. The model is trained for 400 epochs with a batch size of 128, using the Adam optimizer (Kingma and Ba, 2015) with a learning rate of 0.002. The weight hyperparameters are selected from the following ranges:

• \( \lambda_{ECR} \in [10, 30, 40, 50, 100, 200] \)

• \( \lambda_{DT} \in [0.5, 0.7, 1, 2, 5, 10] \)

• \( \lambda_{CLT}, \lambda_{CLC} \in [0.5, 1, 2, 5, 10] \)

The clustering process begins at a specified threshold epoch M and is periodically updated at fixed intervals of I. These parameters are searched in ranges as follows:

• Threshold epoch \( M \in [50, 70, 100, 150, 200, 250] \)

• Clustering update interval \( I \in [30, 50, 70, 100, 150, 200] \)

G.2 Dataset Statistics

Our evaluation was conducted on several widely used benchmark datasets: three standard corpora—20 News Groups (20NG) (Lang, 1995), AGNews (Zhang et al., 2015), and IMDB (Maas et al., 2011)—along with two informal datasets: SearchSnippets (Phan et al., 2008), containing short, noisy text snippets, and GoogleNews (Yin and Wang, 2016), composed of brief article headlines.

For standard datasets, we followed the preprocessing pipeline outlined in (Wu et al., 2023b) to construct bag-of-words representations. For short-text corpora, we utilized preprocessed versions provided by STTM library\(^2\) (Qiang et al., 2022). Further refinement steps were applied, including the removal of words appearing fewer than 3 times and the exclusion of documents containing fewer than 2 tokens. All preprocessing was performed using the TopMost framework\(^3\). The final dataset statistics after processing are summarized in Table 14.

\(^2\)https://github.com/qiang2100/STTM
\(^3\)https://github.com/bobxwu/topmost
H Influence of coefficient

This sensitivity was part of our hyperparameter tuning process, as documented in the Appendix G.1. To provide the concrete details requested, we now present the specific results from this analysis. Table 15 illustrate the impact of varying each coefficient, showing results on a representative dataset 20NG (for \( \lambda_{DT} \)), IMDB (for \( \lambda_{CLT} \)), GoogleNews (for \( \lambda_{CLC} \)), and AGNews (for \( \lambda_{ECR} \)) respectively (all \( K = 50 \) topics). For example, on 20NG, setting \( \lambda_{DT} = 1 \) yields a good performance, while performance slightly changes with other values. Similarly, for IMDB, \( \lambda_{CLT} = 1 \) achieves strong results, with performance degrading at significantly lower or higher values. This detailed analysis demonstrates the model’s sensitivity profile to these hyperparameters and provides clear empirical justification for the values selected and reported in our main experiments.

<table>
  <tr>
    <th rowspan="2">λ<sub>DT</sub></th>
    <th colspan="4">20NG</th>
    <th colspan="4">AGNews</th>
  </tr>
  <tr>
    <th>C<sub>V</sub></th>
    <th>Purity</th>
    <th>NMI</th>
    <th>TD</th>
    <th>λ<sub>ECR</sub></th>
    <th>C<sub>V</sub></th>
    <th>Purity</th>
    <th>NMI</th>
    <th>TD</th>
  </tr>
  <tr>
    <td>0.5</td>
    <td>0.449</td>
    <td>0.621</td>
    <td>0.568</td>
    <td>0.771</td>
    <td>10</td>
    <td>0.469</td>
    <td>0.816</td>
    <td>0.336</td>
    <td>0.617</td>
  </tr>
  <tr>
    <td>0.7</td>
    <td>0.447</td>
    <td>0.636</td>
    <td>0.579</td>
    <td>0.775</td>
    <td>30</td>
    <td>0.470</td>
    <td>0.832</td>
    <td>0.344</td>
    <td>0.912</td>
  </tr>
  <tr>
    <td>1</td>
    <td>0.451</td>
    <td>0.626</td>
    <td>0.583</td>
    <td>0.852</td>
    <td>40</td>
    <td>0.478</td>
    <td>0.849</td>
    <td>0.375</td>
    <td>0.776</td>
  </tr>
  <tr>
    <td>2</td>
    <td>0.451</td>
    <td>0.644</td>
    <td>0.587</td>
    <td>0.768</td>
    <td>50</td>
    <td>0.446</td>
    <td>0.857</td>
    <td>0.412</td>
    <td>0.992</td>
  </tr>
  <tr>
    <td>5</td>
    <td>0.438</td>
    <td>0.649</td>
    <td>0.585</td>
    <td>0.868</td>
    <td>100</td>
    <td>0.436</td>
    <td>0.754</td>
    <td>0.282</td>
    <td>0.987</td>
  </tr>
  <tr>
    <td>10</td>
    <td>0.437</td>
    <td>0.635</td>
    <td>0.581</td>
    <td>0.849</td>
    <td>200</td>
    <td>0.416</td>
    <td>0.675</td>
    <td>0.218</td>
    <td>0.968</td>
  </tr>
</table>

<table>
  <tr>
    <th rowspan="2">λ<sub>CLT</sub></th>
    <th colspan="4">IMDB</th>
    <th colspan="4">GoogleNews</th>
  </tr>
  <tr>
    <th>C<sub>V</sub></th>
    <th>Purity</th>
    <th>NMI</th>
    <th>TD</th>
    <th>λ<sub>CLC</sub></th>
    <th>C<sub>V</sub></th>
    <th>Purity</th>
    <th>NMI</th>
    <th>TD</th>
  </tr>
  <tr>
    <td>0.5</td>
    <td>0.389</td>
    <td>0.685</td>
    <td>0.067</td>
    <td>0.973</td>
    <td>0.5</td>
    <td>0.445</td>
    <td>0.456</td>
    <td>0.663</td>
    <td>0.992</td>
  </tr>
  <tr>
    <td>1</td>
    <td>0.404</td>
    <td>0.737</td>
    <td>0.082</td>
    <td>0.837</td>
    <td>1</td>
    <td>0.454</td>
    <td>0.465</td>
    <td>0.657</td>
    <td>0.920</td>
  </tr>
  <tr>
    <td>2</td>
    <td>0.390</td>
    <td>0.679</td>
    <td>0.065</td>
    <td>0.973</td>
    <td>2</td>
    <td>0.437</td>
    <td>0.457</td>
    <td>0.662</td>
    <td>0.987</td>
  </tr>
  <tr>
    <td>5</td>
    <td>0.397</td>
    <td>0.682</td>
    <td>0.055</td>
    <td>0.979</td>
    <td>5</td>
    <td>0.431</td>
    <td>0.455</td>
    <td>0.660</td>
    <td>0.981</td>
  </tr>
  <tr>
    <td>10</td>
    <td>0.393</td>
    <td>0.686</td>
    <td>0.055</td>
    <td>0.984</td>
    <td>10</td>
    <td>0.431</td>
    <td>0.465</td>
    <td>0.664</td>
    <td>0.987</td>
  </tr>
</table>

Table 15: Evaluation metrics across datasets with different hyperparameters.

I Training Time

HiCOT’s integration of multiple techniques is crucial for enhancing topic quality and representation learning, but it also increases computational cost compared to less complex models. This design choice reflects a deliberate trade-off: the model prioritizes improvements in topic quality and enhanced representation learning, which inherently require a increase in computational complexity. However, we emphasize that this additional cost is well justified, particularly in light of the significant improvements observed in evaluation metrics and model’s efficient inference capabilities. We measured average training time per epoch (in seconds) on two datasets: 20NG and AGNews, with 50 topics, as shown in Table 16. Despite this increase, HiCOT’s training time remains reasonable, suggesting that the performance benefits of HiCOT come at an acceptable computational cost.

<table>
  <tr>
    <th>Datasets</th>
    <th>ETM</th>
    <th>NTM+CL</th>
    <th>ECRTM</th>
    <th>FASTopic</th>
    <th>NeuroMax</th>
    <th>HiCOT</th>
  </tr>
  <tr>
    <td>20NG</td>
    <td>0.263</td>
    <td>1.104</td>
    <td>1.332</td>
    <td>2.161</td>
    <td>2.823</td>
    <td>3.033</td>
  </tr>
  <tr>
    <td>AGNews</td>
    <td>0.236</td>
    <td>0.826</td>
    <td>1.132</td>
    <td>1.911</td>
    <td>1.989</td>
    <td>2.512</td>
  </tr>
</table>

Table 16: Training time per epoch (seconds) for HiCOT and baselines on 20NG and AGNews with 50 topics.
J  Statistical Significance Test

We conducted statistical paired t-tests comparing our HiCOT against FASTopic and NeuroMax on the AGNews dataset with 50 topics. Table 17 presents the p-values for TD, NMI, Purity, and \( C_V \) metrics. The result show that our improvement is statistically significant with all p-value < 0.05, validating our reported improvements.

<table>
  <tr>
    <th>Metric</th>
    <th>HiCOT vs. FASTopic</th>
    <th>HiCOT vs. NeuroMax</th>
  </tr>
  <tr>
    <td>TD</td>
    <td>0.0123</td>
    <td>0.0087</td>
  </tr>
  <tr>
    <td>NMI</td>
    <td>0.0356</td>
    <td>0.0194</td>
  </tr>
  <tr>
    <td>Purity</td>
    <td>0.0412</td>
    <td>0.0335</td>
  </tr>
  <tr>
    <td>\( C_V \)</td>
    <td>0.0278</td>
    <td>0.0449</td>
  </tr>
</table>

Table 17: p-values from paired t-tests comparing HiCOT to FASTopic and NeuroMax on AGNews dataset with 50 topics.

We evaluated HiCOT across multiple random initializations to assess its stability. Due to space constraints, standard deviations for results in 4.2 are omitted from the main paper. Here, Table 18 and 19 present the means and standard deviations for \( C_V \), Purity, NMI, and TD metrics, corresponding to Table 1 and 2 in the main paper. These results show HiCOT’s consistent and strong performance, highlighting its robustness across initializations. Baseline results are reported in NeuroMax (Pham et al., 2024b) and GloCOM (Nguyen et al., 2025a).

K  Expanded Visualization of Embedding Space

We extend the visualization of learned topic and word embeddings using t-SNE (van der Maaten and Hinton, 2008) to additional datasets, including SearchSnippets and GoogleNews, each with 50 topics. Figures 4 and 5 further demonstrate that HiCOT preserves topic structures and mitigates embedding collapse.

L  Examples of Topics

Below are the discovered topics of HiCOT with the 20NG and IMDB datasets under 50 topics, as described in Tables 21 and 22.
(a) ETM        (b) ECRTM        (c) NeuroMax

(d) FASTopic   (e) HiCOT

Figure 4: t-SNE visualization of word embeddings (●) and topic embeddings (▲) for SearchSnippets dataset with 50 topics.

(a) ETM        (b) ECRTM        (c) NeuroMax

(d) FASTopic   (e) HiCOT

Figure 5: t-SNE visualization of word embeddings (●) and topic embeddings (▲) for GoogleNews dataset with 50 topics.
<table>
  <tr>
    <th rowspan="2">50 Topics</th>
    <th colspan="4">20NG</th>
    <th colspan="4">AGNews</th>
    <th colspan="4">IMDB</th>
  </tr>
  <tr>
    <th>C_v</th><th>Purity</th><th>NMI</th><th>TD</th>
    <th>C_v</th><th>Purity</th><th>NMI</th><th>TD</th>
    <th>C_v</th><th>Purity</th><th>NMI</th><th>TD</th>
  </tr>
  <tr>
    <td>ETM</td><td>0.375</td><td>0.347</td><td>0.319</td><td>0.704</td>
    <td>0.364</td><td>0.679</td><td>0.224</td><td>0.819</td>
    <td>0.346</td><td>0.660</td><td>0.038</td><td>0.557</td>
  </tr>
  <tr>
    <td>NTM + CL</td><td>0.437\pm0.005</td><td>0.582\pm0.002</td><td>0.491\pm0.007</td><td>0.802\pm0.002</td>
    <td>0.440\pm0.002</td><td>0.322\pm0.005</td><td>0.100\pm0.002</td><td>0.441\pm0.001</td>
    <td>0.396\pm0.003</td><td>0.657\pm0.006</td><td>0.044\pm0.001</td><td>0.617\pm0.006</td>
  </tr>
  <tr>
    <td>ECRTM</td><td>0.431</td><td>0.560</td><td>0.524</td><td>0.964</td>
    <td>0.466</td><td>0.802</td><td>0.367</td><td>0.961</td>
    <td>0.393</td><td>0.694</td><td>0.058</td><td>0.974</td>
  </tr>
  <tr>
    <td>FASTopic</td><td>0.427\pm0.005</td><td>0.583\pm0.010</td><td>0.528\pm0.001</td><td>0.980\pm0.011</td>
    <td>0.379\pm0.002</td><td>0.831\pm0.007</td><td>0.352\pm0.003</td><td>0.960\pm0.015</td>
    <td>0.371\pm0.002</td><td>0.683\pm0.007</td><td>0.055\pm0.002</td><td>0.969\pm0.015</td>
  </tr>
  <tr>
    <td>NeuroMax</td><td>0.435</td><td>0.623</td><td>0.570</td><td>0.912</td>
    <td>0.385</td><td>0.804</td><td>0.410</td><td>0.952</td>
    <td>0.402</td><td>0.709</td><td>0.061</td><td>0.936</td>
  </tr>
  <tr>
    <td>HiCOT</td><td>0.451\pm0.012</td><td>0.626\pm0.006</td><td>0.583\pm0.009</td><td>0.852\pm0.007</td>
    <td>0.446\pm0.008</td><td>0.857\pm0.003</td><td>0.412\pm0.001</td><td>0.992\pm0.004</td>
    <td>0.404\pm0.003</td><td>0.737\pm0.009</td><td>0.082\pm0.004</td><td>0.837\pm0.006</td>
  </tr>
  <tr>
    <th rowspan="2">100 Topics</th>
    <th colspan="4">20NG</th>
    <th colspan="4">AGNews</th>
    <th colspan="4">IMDB</th>
  </tr>
  <tr>
    <th>C_v</th><th>Purity</th><th>NMI</th><th>TD</th>
    <th>C_v</th><th>Purity</th><th>NMI</th><th>TD</th>
    <th>C_v</th><th>Purity</th><th>NMI</th><th>TD</th>
  </tr>
  <tr>
    <td>ETM</td><td>0.369</td><td>0.394</td><td>0.339</td><td>0.573</td>
    <td>0.371</td><td>0.674</td><td>0.204</td><td>0.773</td>
    <td>0.341</td><td>0.648</td><td>0.037</td><td>0.371</td>
  </tr>
  <tr>
    <td>NTM + CL</td><td>0.420\pm0.003</td><td>0.626\pm0.013</td><td>0.490\pm0.008</td><td>0.624\pm0.004</td>
    <td>0.415\pm0.005</td><td>0.280\pm0.003</td><td>0.050\pm0.002</td><td>0.277\pm0.002</td>
    <td>0.382\pm0.003</td><td>0.705\pm0.012</td><td>0.044\pm0.009</td><td>0.492\pm0.005</td>
  </tr>
  <tr>
    <td>ECRTM</td><td>0.405</td><td>0.555</td><td>0.494</td><td>0.904</td>
    <td>0.416</td><td>0.812</td><td>0.428</td><td>0.981</td>
    <td>0.373</td><td>0.694</td><td>0.049</td><td>0.887</td>
  </tr>
  <tr>
    <td>FASTopic</td><td>0.400\pm0.002</td><td>0.622\pm0.001</td><td>0.522\pm0.008</td><td>0.861\pm0.014</td>
    <td>0.385\pm0.009</td><td>0.833\pm0.015</td><td>0.330\pm0.004</td><td>0.912\pm0.008</td>
    <td>0.369\pm0.003</td><td>0.680\pm0.007</td><td>0.048\pm0.004</td><td>0.886\pm0.008</td>
  </tr>
  <tr>
    <td>NeuroMax</td><td>0.412</td><td>0.602</td><td>0.516</td><td>0.913</td>
    <td>0.406</td><td>0.828</td><td>0.389</td><td>0.957</td>
    <td>0.381</td><td>0.706</td><td>0.059</td><td>0.870</td>
  </tr>
  <tr>
    <td>HiCOT</td><td>0.424\pm0.004</td><td>0.652\pm0.007</td><td>0.568\pm0.006</td><td>0.741\pm0.002</td>
    <td>0.435\pm0.008</td><td>0.862\pm0.009</td><td>0.388\pm0.005</td><td>0.960\pm0.008</td>
    <td>0.388\pm0.008</td><td>0.739\pm0.009</td><td>0.071\pm0.002</td><td>0.733\pm0.005</td>
  </tr>
</table>

Table 18: Evaluation of topic models on 20NG, AGNews, and IMDB datasets with 50 and 100 topics. Each cell reports mean \pm standard deviation.
<table>
  <tr>
    <th rowspan="2">50 Topics</th>
    <th colspan="4">SearchSnippets</th>
    <th colspan="4">GoogleNews</th>
  </tr>
  <tr>
    <th>C_v</th>
    <th>Purity</th>
    <th>NMI</th>
    <th>TD</th>
    <th>C_v</th>
    <th>Purity</th>
    <th>NMI</th>
    <th>TD</th>
  </tr>
  <tr>
    <td>ETM</td>
    <td>0.397</td>
    <td>0.688</td>
    <td>0.389</td>
    <td>0.594</td>
    <td>0.402</td>
    <td>0.366</td>
    <td>0.560</td>
    <td>0.916</td>
  </tr>
  <tr>
    <td>NTM + CL</td>
    <td>0.403\(_{±0.002}\)</td>
    <td>0.215\(_{±0.008}\)</td>
    <td>0.030\(_{±0.006}\)</td>
    <td>0.532\(_{±0.011}\)</td>
    <td>0.433\(_{±0.009}\)</td>
    <td>0.041\(_{±0.003}\)</td>
    <td>0.005\(_{±0.001}\)</td>
    <td>0.301\(_{±0.008}\)</td>
  </tr>
  <tr>
    <td>ECRTM</td>
    <td>0.450</td>
    <td>0.711</td>
    <td>0.419</td>
    <td>0.998</td>
    <td>0.441</td>
    <td>0.396</td>
    <td>0.615</td>
    <td>0.987</td>
  </tr>
  <tr>
    <td>FASTopic</td>
    <td>0.356\(_{±0.021}\)</td>
    <td>0.793\(_{±0.006}\)</td>
    <td>0.497\(_{±0.009}\)</td>
    <td>0.519\(_{±0.030}\)</td>
    <td>0.401\(_{±0.015}\)</td>
    <td>0.252\(_{±0.016}\)</td>
    <td>0.570\(_{±0.031}\)</td>
    <td>0.235\(_{±0.014}\)</td>
  </tr>
  <tr>
    <td>NeuroMax</td>
    <td>0.427\(_{±0.012}\)</td>
    <td>0.743\(_{±0.020}\)</td>
    <td>0.427\(_{±0.006}\)</td>
    <td>0.920\(_{±0.028}\)</td>
    <td>0.409\(_{±0.017}\)</td>
    <td>0.359\(_{±0.011}\)</td>
    <td>0.590\(_{±0.026}\)</td>
    <td>1.000\(_{±0.005}\)</td>
  </tr>
  <tr>
    <td>HiCOT</td>
    <td>0.460\(_{±0.013}\)</td>
    <td>0.818\(_{±0.027}\)</td>
    <td>0.478\(_{±0.018}\)</td>
    <td>1.000\(_{±0.020}\)</td>
    <td>0.454\(_{±0.012}\)</td>
    <td>0.465\(_{±0.021}\)</td>
    <td>0.657\(_{±0.010}\)</td>
    <td>0.920\(_{±0.006}\)</td>
  </tr>
  <tr>
    <th rowspan="2">100 Topics</th>
    <th colspan="4">SearchSnippets</th>
    <th colspan="4">GoogleNews</th>
  </tr>
  <tr>
    <th>C_v</th>
    <th>Purity</th>
    <th>NMI</th>
    <th>TD</th>
    <th>C_v</th>
    <th>Purity</th>
    <th>NMI</th>
    <th>TD</th>
  </tr>
  <tr>
    <td>ETM</td>
    <td>0.389</td>
    <td>0.691</td>
    <td>0.365</td>
    <td>0.448</td>
    <td>0.398</td>
    <td>0.554</td>
    <td>0.713</td>
    <td>0.677</td>
  </tr>
  <tr>
    <td>NTM + CL</td>
    <td>0.406\(_{±0.011}\)</td>
    <td>0.217\(_{±0.010}\)</td>
    <td>0.020\(_{±0.001}\)</td>
    <td>0.394\(_{±0.004}\)</td>
    <td>0.432\(_{±0.009}\)</td>
    <td>0.039\(_{±0.002}\)</td>
    <td>0.005\(_{±0.001}\)</td>
    <td>0.367\(_{±0.004}\)</td>
  </tr>
  <tr>
    <td>ECRTM</td>
    <td>0.432</td>
    <td>0.789</td>
    <td>0.443</td>
    <td>0.966</td>
    <td>0.418</td>
    <td>0.342</td>
    <td>0.491</td>
    <td>0.991</td>
  </tr>
  <tr>
    <td>FASTopic</td>
    <td>0.350\(_{±0.003}\)</td>
    <td>0.801\(_{±0.009}\)</td>
    <td>0.466\(_{±0.011}\)</td>
    <td>0.463\(_{±0.017}\)</td>
    <td>0.366\(_{±0.005}\)</td>
    <td>0.237\(_{±0.012}\)</td>
    <td>0.459\(_{±0.016}\)</td>
    <td>0.100\(_{±0.003}\)</td>
  </tr>
  <tr>
    <td>NeuroMax</td>
    <td>0.439\(_{±0.007}\)</td>
    <td>0.854\(_{±0.018}\)</td>
    <td>0.472\(_{±0.009}\)</td>
    <td>0.960\(_{±0.014}\)</td>
    <td>0.427\(_{±0.013}\)</td>
    <td>0.664\(_{±0.008}\)</td>
    <td>0.834\(_{±0.016}\)</td>
    <td>0.915\(_{±0.022}\)</td>
  </tr>
  <tr>
    <td>HiCOT</td>
    <td>0.449\(_{±0.003}\)</td>
    <td>0.857\(_{±0.008}\)</td>
    <td>0.480\(_{±0.011}\)</td>
    <td>0.940\(_{±0.010}\)</td>
    <td>0.470\(_{±0.008}\)</td>
    <td>0.763\(_{±0.005}\)</td>
    <td>0.864\(_{±0.012}\)</td>
    <td>0.802\(_{±0.009}\)</td>
  </tr>
</table>

Table 19: Evaluation of topic models on SearchSnippets and GoogleNews datasets with 50 and 100 topics. Each cell reports mean ± standard deviation.
<table>
  <tr>
    <th>Topic</th>
    <th>Top 10 phrases</th>
  </tr>
  <tr>
    <td><b>Topic #7</b></td>
    <td>automotive concepts, subject automotive, experience riding, new motorcycles, eisa bus, <u>dealer service</u>, road suite, motorcycle tip, corporation lines, buying motorcycle</td>
  </tr>
  <tr>
    <td><b>Topic #17</b></td>
    <td>automotive concepts, subject automotive, dealer service, auto dealers, reducing dealer, performance cars, car buying, reduce dealer, dealer profit, corporation lines</td>
  </tr>
  <tr>
    <td><b>Topic #20</b></td>
    <td>automotive concepts, subject automotive, full auto, garage ama, road suite, drives feature, trucks read, road independence, experience riding, auto dealers</td>
  </tr>
  <tr>
    <td><b>Topic #16</b></td>
    <td>list biblical, follow teachings, <u>online bible</u>, gospel accounts, spiritual needs, daily verse, positive belief, teachings james, requires faith, spiritual world</td>
  </tr>
  <tr>
    <td><b>Topic #24</b></td>
    <td>list biblical, <u>online bible</u>, follow teachings, reading bible, spiritual world, gospel accounts, spiritual needs, daily verse, peoples spiritual, accept messiah</td>
  </tr>
  <tr>
    <td><b>Topic #2</b></td>
    <td>scripture arrogant, follow teachings, displaying ignorance, necessarily arrogant, agree definition, strong atheism, requires faith, reason believe, weak atheism, positive belief</td>
  </tr>
  <tr>
    <td><b>Topic #4</b></td>
    <td>strong atheism, introduction atheism, <u>weak atheism</u>, atheists organization, positive belief, reason believe, caused atheism, weak atheist, requires faith, share beliefs</td>
  </tr>
  <tr>
    <td><b>Topic #3</b></td>
    <td>keywords encryption, <u>importance encryption</u>, encryption management, <u>encryption product</u>, encryption providing, encryption differ, encryption method, strong encryption, approach encryption, cover encryption</td>
  </tr>
  <tr>
    <td><b>Topic #32</b></td>
    <td>importance encryption, <u>encryption product</u>, keywords encryption, <u>encryption providing</u>, encryption management, commercial encryption, nsa classified, trust nsa, encryption wiretap, encryption devices</td>
  </tr>
  <tr>
    <td><b>Topic #5</b></td>
    <td>organization portal, organization express, inc lines, organization netcom, corp internet, organization telecom, reply organization, corporation lines, <u>organization ncr</u>, organization polytechnic</td>
  </tr>
  <tr>
    <td><b>Topic #14</b></td>
    <td>corporation lines, <u>inc lines</u>, organization express, organization portal, organization netcom, corp internet, <u>organization sdpa</u>, forsale organization, corp distribution, incorporated lines</td>
  </tr>
  <tr>
    <td><b>Topic #22</b></td>
    <td>reply organization, organization portal, <u>content communications</u>, computers social, organization netcom, proposed newsgroup, <u>organization express</u>, corp internet, organization sdpa, group newsreader</td>
  </tr>
  <tr>
    <td><b>Topic #25</b></td>
    <td>reply organization, <u>organization ncr</u>, associated press, organization nyx, organization bell, organization bnr, <u>organization portal</u>, <u>content communications</u>, washington post, inc newsreader</td>
  </tr>
  <tr>
    <td><b>Topic #38</b></td>
    <td>federal agents, agencies messages, organization ncr, wiretap drug, organization nyx, secret service, organization california, <u>organization bell</u>, agencies tools, determine agencies</td>
  </tr>
  <tr>
    <td><b>Topic #41</b></td>
    <td>keywords frequently, join organized, beyond column, join reform, organization nyx, research cpr, organization ncr, keywords jpl, word processing, organization california</td>
  </tr>
  <tr>
    <td><b>Topic #49</b></td>
    <td>organization ncr, organization optilink, initiative congress, <u>organization nyx</u>, public spending, <u>organization sdpa</u>, policy research, administration saying, public interest, policy members</td>
  </tr>
</table>

Table 20: Top 10 phrases of some topics from 50 topics from 20NG by C-Top2Vec. Repeated phrases are underlined. With the topic diversity value of 0.577, these topics semantically collapse towards each other with many repetitive phrases.
HiCOT with 20NG (K = 50)

Topic #1 : pitt marriage derek bonds morgan professor richardson rush timothy spending
Topic #2 : sahak anatolia turkish turkey armenians greeks armenian greek proceeded inhabitants
Topic #3 : escrow encryption encrypted phones chip enforcement rsa privacy tapped algorithm
Topic #4 : battery intellect beast tank discharge gear cad prince tanks cartridge
Topic #5 : nhl standings oilers espn playoff bruins penguins champs hockey traded
Topic #6 : simms centris lciii slots quadra vga powerbook vlb ram mac
Topic #7 : die oil anymore everybody minority lot sell going think weekend
Topic #8 : font fonts xterm terminal screen workstation salvation width christ patrick
Topic #9 : widget window sunos cursor display server bitmap client resource int
Topic #10 : braves pitcher pitching partners bps votes serial characters hitter analog
Topic #11 : jpeg widget gif graphics formats pub format sunos mac color
Topic #12 : insurance taxes tax printer radius car health billion servicing scanner
Topic #13 : int output null max file stream entry tiff byte files
Topic #14 : privacy rsa pub encryption anonymous ftp cancer pgp requests networks
Topic #15 : honda motorcycles rear levine ama tire bmw dog gardner wheels
Topic #16 : cds manuals sale warranty rider disks buyer amp shipping excellent
Topic #17 : armenians armenian soldiers tragedy neighbors president crowd turkey burned secretary
Topic #18 : firearms handgun gun weapons arms crime possession constitutional violent firearm
Topic #19 : suck duo games clipper melbourne sony alex game yamaha brad
Topic #20 : god theists atheists bible christianity gods beliefs morality rosenau believers
Topic #21 : gordon racism luis netherlands postscript science genocide yugoslavia ruling terrorism
Topic #22 : windows dos diamond swap logo mouse microsoft driver novell drivers
Topic #23 : henry rutgers pens nasa troy temple fax arbor museum Calif
Topic #24 : optilink yankees won switzerland toyota italy stadium angels suspension greece
Topic #25 : min reagan det portal que micro danny livni benedikt howard
Topic #26 : bike bikes son duke killed lock motorcycle hudson murder suicide
Topic #27 : church captain sexual homosexual christians churches militia orientation orthodox arrogance
Topic #28 : theodore satan gary kim sutherland keith serdar joy icon bryan
Topic #29 : mormons nsa christian koresh alien nra classified consent dick incidents
Topic #30 : pitchers batting pitches players mets defensive phillies morris scored season
Topic #31 : gaza jake arabs israel arab palestine civilians inhabitants jerusalem soldiers
Topic #32 : msg shameful skepticism foods sensitivity yeast diagnosed chinese patients symptoms
Topic #33 : atf bds survivors reno fbi murders gun weapons firearms blast
Topic #34 : helmet resurrection philadelphia moon detroit heaven toronto oakland scripture icons
Topic #35 : bmp motif rec hawks leigher beyer bbs ottawa petaluma ext
Topic #36 : male nyx locks sox beer taste don men compression percent
Topic #37 : max morality solntzevpsgicom schneider cookamunga tourist cliff objective islam atheists
Topic #38 : voltage amp circuit circuits wire wires wiring detector detectors cable
Topic #39 : scsi ide controller drives bios jumper drive vlb floppy boot
Topic #40 : jesus sins romans unto god messiah trinity bible gods sin
Topic #41 : devils rangers islanders radar champions andre roger capitals stanley playoffs
Topic #42 : revelation jewish catholics feelings jimmy faith clh utter occupied label
Topic #43 : spacecraft zoology satellite jpl missions payload launch orbital mission atmosphere
Topic #44 : cubs bnr eisa tin helsinki finland mathew newsreader isa beleive
Topic #45 : blues accelerator shaped boots priest space mask interrupt span image
Topic #46 : xxdate nuntius useragent xxmessage csutexasedu gerald olchowy lib gmt apr
Topic #47 : covington georgia advance malcolm sale photography rgb monitor irq michael
Topic #48 : catholic iran pat mary austria islamic elizabeth armenia chen stevens
Topic #49 : baseball europeans lebanon water bosnians homeland occupation peace borders israelis
Topic #50 : cramer clayton modem ticket abortion gay modems homosexuality tickets intergraph

Table 21: Top 10 related words of 50 topics from 20NG. Some repeated words are bold and underlined. The topic diversity value of 0.852 in the HiCOT model, though lower than FASTopic 0.980, remains high enough to maintain a diverse range of topics. While some topic-words overlap - such as "soldiers" appearing in both Topic 17 and Topic 31 - this does not result in topic collapse. Instead, the two topics retain distinct focuses: Turkey - Armenia war and Israel - Palestine war.
HiCOT with IMDB (K = 50)

Topic #1 : scary horror gore vampire terrible boring dracula bad vampires crappy
Topic #2 : novel jane timothy adaptation emma shakespeare novels book charlotte versions
Topic #3 : preview nicole hanks hbo premiere comedians bette stale screens edgy
Topic #4 : bergman diamond thrill realised masterful deranged preposterous hokey wrenching tasteless
Topic #5 : worst awful waste avoid terrible poorly wasted worse laughable amateurish
Topic #6 : pun unlikeable book races sue hack replace disney females childish
Topic #7 : season seasons episodes episode show abc series sitcom hooked network
Topic #8 : mute unlikable unreal formulaic sarah max coupled jean melodramatic matrix
Topic #9 : channels differently existent quotes dan sports dialogues respectively rank interaction
Topic #10 : bollywood arnold batman cliche eastwood manipulative puppets tends werewolf showcase
Topic #11: drew nancy freeman romantic morgan jim comedic chemistry bruce definitely
Topic #12: stinks stunned stranded ridiculously crashes lately french heat france crash
Topic #13: ingredients ish caliber dance tap losers dancing phil priceless valley
Topic #14: amanda williams encourage filler sticking blast ryan bus tad complaints
Topic #15: godfather spoiled correctly desired ward neo fooled guessing viewings wizard
Topic #16: match stan jerry eddie baseball hardy bugs wrestling tag football
Topic #17: superman planet robot trek alien batman fiction space technology science
Topic #18: powell arthur hudson oscar dorothy chaplin broadway fonda henry moore
Topic #19: gems australian sean alex inspire aim monty savage derek scottish
Topic #20: society sexuality images desire cultural china nature catholic louis religious
Topic #21: waste worst crap terrible awful wasting please wasted worse avoid
Topic #22: amused murphy suspend jokes dave thumbs mail shouting freaks smiling
Topic #23: horror eerie haunted gothic carpenter creepy pitt murders caine curtis
Topic #24: twins widescreen lift remake kong models portrayals ford grand hitchcock
Topic #25: angela martha claude accomplish classical davis brazil titanic credited exposure
Topic #26: eddie stan hardy lucy taxi heist pearl lifeless monkey garden
Topic #27: sinatra broadway musicals musical powell sing kelly mgm dancers dance
Topic #28: chan jackie martial arts dragon kung ninja kong hong chinese
Topic #29: spielberg burton profanity entitled roberts talked racist ross barry biased
Topic #30: jesus christ bible documentary religious christian religion church catholic beliefs
Topic #31: pointless development uninteresting dimensional boring fails disjointed sex depth gay
Topic #32: marie ann mom charlie married marry mothers daughters finds dad
Topic #33: germans propaganda hitler nazis jews war soviet nazi documentary germany
Topic #34: stupid sucks crap gonna dumb guy laughing lame kid wanna
Topic #35: dinosaur dinosaurs shark alien superman seagal scientist bullets rubber dragon
Topic #36: andrews noir sheriff stewart eastwood police trail harry fbi clint
Topic #37: apocalypse khan april sometime vietnam nerd ken camp tacky war
Topic #38: santa kills lisa sheriff killed bat killing mom killer nurse
Topic #39: streep beautifully meryl deeply brilliantly fonda captures emotionally emotional passion
Topic #40: zombie zombies vampires gore slasher vampire gory werewolf nudity metal
Topic #41: stanley billed les predator welles murray quinn glenn ned sums
Topic #42: sheen hats preferred greatness alert turkey bush dropping lasting shoulders
Topic #43: fifth net subplots mathew aimed cheating robin remained refers exquisite
Topic #44: walken junior preachy karate bang foul severely buff favour acid
Topic #45: lion disney santa animals adults christmas bugs bears copy king
Topic #46: freddy hoot annie lip nails eva hilarity chuckle hitler rude
Topic #47: glover indian stuart continually abused jonathan baker roller fields nyc
Topic #48: eighites sixties seventies stumbled gary taylor wig irish korean comparing
Topic #49: documentary art cinema silent images artists documentaries color contemporary visual
Topic #50: parent schools juvenile teacher teaching posters sons overacting beverly adult

Table 22: Top 10 related words of 50 topics from IMDB. Some repeated words are bold and underlined. The topic diversity value of 0.837 in the HiCOT model, although lower than ECRTM 0.974, is still sufficiently high to preserve a broad range of topics. While some topic-words overlap - such as "book" appearing in both Topic 2 and Topic 6 - this does not result in topic collapse. Instead, the two topics retain distinct focuses: "classic literature and novel adaptations" and "criticism and controversy in books".

Reviewer paper summaries: This paper propose using smaller pretrained embeddings (Glove for word, Doc2Vec for documents) and propose a two-stage training pipeline where they first use optimal transport to align document with topic embeddings by minimizing a cost function that measures distance between the projected document vectors and topic vectors. They also add a contrastive loss that pull together words in the same cluster to obtain well-formed clusters.

This paper introduces Hierarchical Clustering and Contrastive Learning with Optimal Transport for Neural Topic Modeling to address weak document-topic alignment and high inference costs due to large pre-trained language models. This method also employs hierarchical clustering to combine with contrastive learning.
Summary of strengths: - More efficient inference (from smaller embeddings) and consistent gains in coherence and doc-topic clustering metrics (Purity, NMI).
- The approach is novel in the sense that it both refines topics through OT (better coherence) and reinforcing separations among distinct topic groups (more diverse and distinctive topics).
- Paper is well-written.

1. The baseline methods are comprehensive as it includes SOTA method such as FASTopic.
2. Topic modeling with Hierarchical structures is very useful in practice for visualization and deep dive.
Summary of weaknesses: - Lack of comparsion/discussion with other objectives [1, 2], other VAE-based models [3], or even clustering based methods [4]
- No statistical significance tests on the results, the authors also didn't mention the number of random seeds used, which is concerning
- Limited ablations of contrastive sampling techniques 


[1] GloCOM: A Short Text Neural Topic Model via Global Clustering Context
[2] Diversity-Aware Coherence Loss for Improving Neural Topic Models
[3] Pre-training is a Hot Topic: Contextualized Document Embeddings Improve Topic Coherence
[4] BERTopic: Neural topic modeling with a class-based TF-IDF procedure

1. Doc2Vec does not capture semantic structure as well as SentenceTransformer. The limitation for SentenceTransformer is its performance on longer doc and Doc2Vec may perform even worse than that due to its limitation to capture semantic structure.
2. This paper mostly leverages unsupervised method, which is not convincing for me. To justify LLM performance, It may be useful to include some human evaluation[1] or LLM evaluation[2]. 
3. Inference time experiment may need more explanations. Are you talking about inferencing one doc or the whole datasets? Why not compare with more traditional method such as ETM on inference since the method is more similar?
4. Ablation study can be more concrete by including how different lambda coefficient (E.1) influence the result. 

[1] Reading Tea Leaves: How Humans Interpret Topic Models
[2] LLM Reading Tea Leaves: Automatically Evaluating Topic Models with Large Language Models
Comments, suggestions and typos: There is limited discussion of the benefits of  hierarchical topic structure and how they can be effectively evaluated (since this is one the contribution of this method). 
This paper could also benefits from a figure to illustrate their two-stage approach.

How to solve Optimal Transport is standard and it contains too much details in section 3.2 about how to solve it. Why not putting it to appendix?
Why does the author choose 15 words instead of 20 words, 10 words in evaluation? Will it hugely influence the result?
Overall assessment scores: 3.5

3.5
Metareview score: 3.5
Metareview: To address the weak document-topic alignment issue and high inference costs, this paper introduces a neural topic model HiCOT (Hierarchical Clustering and Contrastive Learning with Optimal Transport).
It combines several techniques, hierarchical clustering, contrastive learning, and optimal transport.
The optimal transport refines document-topic distributions, and the hierarchical clustering with contrastive learning disentangles topic-word relationships.
The experimental results demonstrate that the proposed model performs well, regarding topic coherence, topic diversity, representation quality, and inference time.
Summary of reasons to publish: - The paper incorporates several techniques to improve topic modeling performance, and these are justified with ablation studies.
- The included baseline methods are comprehensive with several newest models.
- The experiments are sufficient with multiple benchmark datasets.
- The proposed neural topic model performs well across the datasets.
- The proposed model has more efficient inference.