Combating Label Sparsity in Short Text Topic Modeling via Nearest Neighbor Augmentation (2024)

Abstract: Extracting semantic topics from short texts presents a significant challenge in the field of data mining. While efforts have been made to mitigate data sparsity issue, the limited length of short documents also results in the absence of semantically relevant words, causing biased evidence lower bound and incomplete labels for likelihood maximization. We refer to this issue as the label sparsity problem. To combat this problem, we propose kNNTM, a neural short text topic model that incorporates a k-Nearest-Neighbor-based label completion algorithm by augmenting the reconstruction label with k nearest documents to complement these relevant but unobserved words. Furthermore, seeking a precise reflection of distances between documents, we propose a fused multiview distances metric that takes both local word similarities and global topic semantics into consideration. Extensive experiments on multiple public short-text datasets show that kNNTM model outperforms the state-of-the-art baseline models and can derive both high-quality topics and document representations.

heading: 1 Introduction
text: Depiste the success of topic models in numerous NLP tasks (Boyd-Graber et al., 2017) for uncovering the underlying semantic concepts (Blei et al., 2003), traditional topic models often suffer from poor performances when applied to short text contents, e.g., social media posts and news
*Corresponding Author
headlines (Yan et al., 2013). This deficiency can be attributed to the lack of word co-occurrence information due to the limited length for a single short document, known as the data sparsity problem (Murshed et al., 2022).
Many topic models have been developed to overcome the data sparsity issue. The Dirichlet Multinomial Mixture (DMM) model (Yin and Wang, 2014; Li et al., 2016, 2017) constraints that each short text is generated by a single topic. Biterm Topic Model (Yan et al., 2013; Cheng et al., 2014) utilizes the rich corpus-level word co-occurrence patterns for inferring topics. And some works (Mehrotra et al., 2013; Quan et al., 2015; Zuo et al., 2016) aggregate semantically similar texts into long pseudodocuments. Recently, with the developments of neural topic models (NTMs) (Srivastava and Sutton, 2017), there are also attempts to mitigate the data sparsity issue by utilizing biterm graph (Zhu et al., 2018) and topic quantization techniques (Wu et al., 2020, 2022).
Though these above works have achieved good performances and mitigated the data sparsity issue to some extent, there are still problems that they ignore. Under the variational autoencoding framework, current mainstream NTMs are optimized by the maximum likelihood objective, which is achieved by maximizing the evidence lower bound (ELBO). However, the limited length of short texts results that only a few words get described in a document, while many other semantically relevant words remain unobserved (Zhang and Lauw, 2022).
13762
As shown in the motivating example in Figure 1, the document in the center talks about CPU chips and contains words like ’chip’ and ’intel’. However, many semantically related words like ’core’, ’processor’, and ’cpu’ remain uncovered. Such ’incomplete’ short documents will lead to a biased evidence lower bound as the possibilities of those unobserved but relevant words are completely ignored, resulting in a sub-optima optimization of the maximum likelihood. To be more specific, the absence of these relevant words leads to an incomplete target for the reconstruction objective during variational autoencoding, which makes the probabilities of the absent words get inappropriately suppressed and results in biased training signals. This problem, different from the data sparsity problem of the input data, is referred to as the label sparsity problem in this paper.
Inspired by the above observation, we propose to explicitly augment the reconstruction target in short text NTM with semantically related words to provide unbiased training signals. One direct approach to derive these words is to leverage the similarities between pre-trained word embeddings. However, word embeddings trained with general corpora may not capture the word co-occurrence patterns from a specific domain. Moreover, simply relying on word similarities ignores the context information on the document level. In this paper, we propose kNNTM, a short text topic modeling framework, which incorporates a k-Nearest-Neighborbased label completion algorithm by aggregating k documents semantically closest to the target document to augment its reconstruction label. As illustrated in Figure 1, documents with semantically relevant words are retrieved by k-nearest neighbor searching, and the label distribution gets augmented by complementing the probabilities of originally unobserved words. The kNN approach is shown to be effective in multiple fields for in-
formation supplementation and data completion, like machine translation (Khandelwal et al., 2021), healthcare prediction (Zhang et al., 2021), and computer vision (Yu et al., 2021). In our scenario of short text topic modeling, aggregating k nearest documents for label completion helps to make full use of the word co-occurrence information and document relations from the original dataset.
However, one remaining challenge of the kNNbased label completion is to seek a proper distance metric that could precisely reflect the similarities between short documents with scarce context. A good metric should reflect both the word-level similarities and the global semantic resemblance. Therefore, we propose a multi-view distance metric by fusing the distances from the input space and the semantic space to leverage both local and global similarities information. The distance metric in the input space depicts the local word similarity, which is defined with the optimal transport distance between bag-of-words distributions, with cost functions built upon word similarities from both general corpora and the specific dataset. And the metric in the hidden space reflects the global semantic resemblance, which is defined through the lens of topic semantics with the similarities between document-topic distributions. With the fused multiview distance metric, we can take various factors into account when evaluating the distances between documents, and provide a reasonable distance metric for the kNN algorithm.
Our contributions are summarized as follows: • We identify the label sparsity problem in short
text neural topic modeling, and propose a novel topic modeling framework, kNNTM, to combat this issue by a kNN-based label completion algorithm with similar document aggregation. • We propose a fused multi-view distance metric that takes both global and local semantic similarities into consideration to support the kNN label completion algorithm. • Extensive experiments are conducted on three short text datasets, and both quantitative and qualitative results demonstrate that kNNTM outperforms state-of-the-art baselines, and could generate high-quality topics and meaningful document representations.

heading: 2 Related Works
text: Neural Topic Modeling With the recent developments of neural variational inference (Kingma and
Welling, 2014; Rezende et al., 2014), many neural topic models (NTMs) are proposed for higher scalability and easier inference. NVDM (Miao et al., 2016) and ProdLDA (Srivastava and Sutton, 2017) are two representative works, which leverage Gaussian and logistic normal distribution as approximations of the Dirichlet prior. And many subsequent NTMs have been investigated. Some focus on improving the encoder network, e.g., recurrent networks (Rezaee and Ferraro, 2020), graph neural networks (Yang et al., 2020; Xie et al., 2021). Some works aim for a better approximation of the Dirichlet prior, e.g., Wasserstein autoencoders (Nan et al., 2019), reject sampling (Burkhardt and Kramer, 2019), and Weibull distribution (Zhang et al., 2018). Some works attempt to find new training paradigms, e.g., adversarial training (Wang et al., 2019, 2020; Hu et al., 2020), and optimal transport (Zhao et al., 2021; Wang et al., 2022). The most recent work of NTM to the best of our knowledge is ECRTM (Wu et al., 2023), which incorporates an embedding clustering regularization on the topic and word embeddings. Despite their success in modeling topics on normal long texts, these works still suffer from the sparsity issue of short texts.
Topic Models for Short Text Conventional short text topic models can be mainly categorized into three classes. The Dirichlet Multinomial Mixture (DMM) models (Yin and Wang, 2014; Li et al., 2016, 2017) assume that each short text is generated by a single topic, thus reducing the complexity for topic inference. Biterm Topic Model (BTM) (Yan et al., 2013; Cheng et al., 2014) utilizes the rich corpus-level word co-occurrence patterns and splits the entire dataset into numerous biterms. Self-aggregation models (Mehrotra et al., 2013; Quan et al., 2015; Zuo et al., 2016) tend to aggregate semantically similar short text into long pseudo-documents to infer topics.
Another line of research focuses on neural topic modeling for short texts. GraphBTM (Zhu et al., 2018) generalizes the BTM model and performs variation autoencoding on biterm graph from randomly-sampled mini-corpus. NQTM (Wu et al., 2020) shares similar insights with DMM and quantizes document-topic distributions to obtain peakier distributions. And the TSCTM model (Wu et al., 2022) further improves upon NQTM by introducing a contrastive loss on quantized distributions. MCTM (Zhang and Lauw, 2022) focuses on short
texts in the variable-length corpus and learns a semantics predictor based on long documents within the corpus. However, current neural topic models for short text mainly focus on mitigating the data sparsity problem from the input side, yet ignore the insufficient training signals for the reconstruction objective, namely the label sparsity issue, brought by the limited length of a single document.

heading: 3 Methodology

heading: 3.1 Problem Formulation
text: Consider a corpus D with ND documents, where each document d contains Nd words {xd1 , . . . , xdNd} belonging to a vocabulary of size V . The target is to discover K topics from the corpus. Each topic is defined as a distribution βk ∈ ∆V over the words in the vocabulary, namely the topic-word distribution. Meanwhile, for each input document, the model should also infer a distribution over the topics, i.e., the document-topic distribution, denoted as θ ∈ ∆K .

heading: 3.2 Model Architecture
text: We choose the Quantization Topic Model (QTM) (Wu et al., 2020, 2022), as the basic model architecture for kNNTM. QTM is a VAE-based neural topic model that quantizes topic representations for peakier topic distributions. Here we briefly introduce the model architecture. For more detailed implementations, please refer to the original paper (Wu et al., 2020).

heading: 3.2.1 Text encoder
text: The text encoder takes document d in the form of bag-of-words as the input xd, and produces corresponding hidden topic representation hd ∈ RK , and the topic representation is further normalized into a probability simplex to obtain the documenttopic distribution θd ∈ ∆K by a softmax function θd = softmax ( hd ) .

heading: 3.2.2 Topic Quantization
text: The document-topic distributions is further quantized to alleviate the data sparsity problem, The quantized distribution is defined as
θdq = ek, where k = argminj ∥∥∥θd − ej ∥∥∥ 2 , (1) where e = (e1, e2, ..., eK) ∈ RK×K are K preset quantization prototypes. These prototypes are initialized as different one-hot vectors and get optimized during training.

heading: 3.2.3 Decoder and Objective Function
text: The decoder network consists of topic-word distributions β, and tries to reconstruct the observed texts with the quantized distributions Let xd denote the bag-of-words form of a document d, then the reconstruction objective for topic models is
LRECON ( xd ) = −xd⊤ log ( softmax ( βθdq )) .
(2) Besides the reconstruction loss, the QTM leverages a regularizer constraining the distances between original and quantized distributions, LREG(θd) = ∥∥∥sg(θd)− θdq ∥∥∥ 2 + λ ∥∥∥sg(θdq )− θd ∥∥∥ 2 , (3) where sg(·) is a stop gradient operation, and λ is set to 0.1 following (Wu et al., 2020, 2022). The final objective function for the model is
LTM(xd) = LRECON(xd) + LREG(θd) (4)
3.3 kNN-based Label Completion The core difference between kNNTM and other neural topic models is the kNN-based label completion module. As discussed in the Introduction and Figure 1, short text modeling faces the label sparsity problem. The limited length of short texts makes some semantically related words unobserved in the short document. The probabilities of these words are ignored and lead to a biased evidence lower bound during optimization. When optimizing the neural topic model with LTM, the probabilities of the observed words get encouraged, while the probabilities of those unobserved but relevant words get discouraged as the predicted probability vector get normalized by the softmax function, leading to biased training signal and suboptimal optimization. Therefore, a label completion module is needed to derive the hidden relevant
words and construct an unbiased label for the reconstruction objective.
Motivated by the above thoughts, we propose the kNN-based label completion algorithm. Given a document xd, we find its nk nearest neighbors with a distance metric, dist(·, ·) (we leave the design of the metric to the next section). The set of nk nearest neighbors is denoted asNxd , and the reconstruction label is augmented with a coefficient α as
x̃d = xd + α ∗ 1 nk
∑
xj∈Nxd xj . (5)

heading: 3.4 Fused Multi-View Distance Metric
text: To perform an effective kNN algorithm, a reliable distance metric is required to precisely measure the similarities between short documents. Here we propose a fused multi-view distance, which fuses distances from both the input BoW space and hidden topical semantic space and takes information from both local word-level relations and global topic-level similarities.

heading: 3.4.1 Distance from the Input Space
text: As the original form of the input document, the bagof-word vectors naturally contain the information for evaluating the distances between documents. However, directly comparing the bag-of-word vectors would result in a bad distance metric, as the hidden semantic relations between words are not explored. Documents with different but highly relevant word sets would be considered dissimilar without considering the hidden relation between words. Therefore, we propose to evaluate the distance between two bag-of-words vectors with a word semantic-based optimal transport (OT) distance. Firstly we introduce the OT distance between two probability vectors a ∈ ∆Da and b ∈ ∆Db , which
is defined as:
distOTM (a, b) := min γ∈U(a,b) ⟨γ,M⟩, (6)
where ⟨·, ·⟩ denotes the Frobenius dot-product, U(a, b) denotes the transport polytope of a and b, U(a, b) := {γ ∈ RDa×Db+ | γ1 = a, γ⊤1 = b}, and M ∈ RDa×Db≥0 is a cost matrix indicating the transportation cost between probability vectors.
Therefore, with an appropriate cost matrix M ∈ RV×V≥0 depicting the semantic similarities between words, the OT distances will become a suitable distance metric for gauging the distance between two bag-of-words vectors, namely distOTM (x̂
di , x̂dj ), where x̂d is the normalized bag-of-words vector.
We propose two perspectives to build cost matrix M. The first aspect is to leverage the word embeddings pre-trained with general corpora. The cosine similarities between pre-trained word embeddings have been proven to be highly effective in reflecting the semantic similarities between words. The cost matrix is built as
Mcosi,j = (1.0− scos (wi, wj)) /2, (7)
where scos(·) is the cosine similarity function, and wi is the i-th word in the vocabulary.
Though effective, word embeddings pre-trained with general corpora may not be able to capture the co-occurrence patterns in a specific domain. Hence, we propose another way to build the cost matrix M with the word co-occurrences of the current corpus. Concretely, the cost matrix is built as
Mcooi,j = 1.0− (p(wi|wj) + p(wj |wi)) /2, (8)
where p(wi|wj) is the conditional probability of word wi given wj , and is calculated as p(wi|wj) = df(wi, wj)/df(wj), where df(wi, wj) is the frequency that the word wi and wj co-occur.
Finally, to leverage both the rich information from general corpora and specific patterns from the current dataset, we fuse the two OT distances to formulate the distance from input space as:
distBoW(xdi , xdj ) = ρ ∗ distOTMcos(x̂di , x̂dj ) + (1.0− ρ) ∗ distOTMcoo(x̂di , x̂dj ). (9)
where ρ is a fusing hyper-parameter.

heading: 3.4.2 Distance from the Semantic Space
text: Besides the bag-of-words vector, the topic distribution for each document is also an effective semantic representation of documents. The metric
defined on the bag-of-words vectors mainly utilizes local word-level similarities, whereas the metric defined on the topic distributions can depict semantic distances between documents from a global view, since the topics reveal the hidden semantic structures of the entire corpus. Incorporating the metric defined on topic distributions would encourage topically similar documents to fall into the neighborhood of the target document.
However, one important problem is that the document-topic distributions keep evolving during training and cannot be pre-computed beforehand. It is also impractical to go through the entire dataset to compute the distributions whenever the nearest neighbors are needed due to excessively high time costs. Therefore, we maintain a memory bank {m1, ...mND}, to store the most recent documenttopic distributions. Entries within the memory bank get updated every time the document-topic distribution are computed during training,
md = θ d t , (10)
where θdt is the document-topic distribution for document d computed in the t-th iteration during training. And the distance from the topical semantic space is defined as
distTopic(xdi , xdj ) = ∥θdi −mdj∥2. (11)

heading: 3.4.3 Multi-View Distance Fusion
text: To get a balanced distance metric considering both similarities in the BoW and the topic spaces, we fuse the above two distances with a hyperparameter η as
distFuse(xdi , xdj ) = η ∗ distTopic(xdi , xdj ) + (1.0− η) ∗ distBoW(xdi , xdj ). (12)

heading: 3.5 Training Procedure and Objective
text: To stabilize the training process, we adopt a twophase training strategy. We first pre-train the topic model with the objective in Eq.4 without label augmentation for P epochs to get more accurate document-topic distributions for distance calculation. After pre-training, we use the augmented label from Eq.5 for the reconstruction loss,
LkNN(xd) = −x̃d ⊤ log ( softmax(βθdq ) ) . (13)
and the final training objective is
LkNNTM(xd) = LkNN(xd) + LREG(θd). (14) The overall structure of kNNTM is shown in Fig.2, and we provide the detailed training procedure in the Algorithm 1 in Appendix A.

heading: 4 Experiment Settings

heading: 4.1 Datasets
text: In the experiments, we use three public benchmark short text datasets: 1) GoogleNews with titles of over 10,000 news articles categorized into 152 clusters, 2) Snippet consisting of over 10,000 web search results across 8 domains, 3) StackOverflow with 20,000 question titles from 20 different tags.
We utilized the aforementioned datasets provided by the STTM library1 (Qiang et al., 2020). Additionally, we further filter out words with a frequency below 3 and documents with a length less than 2. Please refer to Appendix B.1 for the detailed statistics for each dataset after preprocessing.

heading: 4.2 Baseline Methods
text: We compare our model with the following state-ofthe-art baselines: prodLDA (Srivastava and Sutton, 2017), a prominent work of NTM with black-box neural variational inference; WLDA (Nan et al., 2019), a NTM with the Wasserstein autoencoder framework; ECRTM (Wu et al., 2023), a NTM with an topic embedding clustering regularization, which is the state-of-the-art NTM for normal long documents; NQTM (Wu et al., 2020), a neural short text topic model with topic distribution quantization and negative sampling; MCTM (Zhang and Lauw, 2022), a NTM that predicts missing semantics for short documents based on other long documents in variable-length corpora; TSCTM (Wu et al., 2022), a state-of-the-art short text neural topic model based on NQTM with a contrastive objective on quantized distributions.

heading: 4.3 Implementation Datails
text: We follow the settings for hyper-parameters shared with (Wu et al., 2022), including learning rate, batch size, epoch number, etc. And for hyperparameters exclusive to our method, we conduct grid search to determine the optimal values. Please refer to Appendix B.3 for detailed settings.

heading: 5 Experimental Results
text: We evaluate the topic models from two perspectives: topic-word distribution and document-topic distribution. For the former, we assess the quality of topics based on coherence and diversity. Regarding the latter, we utilize the performances from the clustering task as previous studies (Zhao et al.,
1https://github.com/qiang2100/STTM
2021; Wang et al., 2022). To verify models’ effectiveness under different topic numbers, following previous work (Wu et al., 2022), we conducted experiments under 50 and 100 topics, respectively.

heading: 5.1 Topic Quality
text: Metric Following previous work in topic modeling (Dieng et al., 2020; Wu et al., 2022), we evaluate the quality of learned topics from two perspectives, Topic Coherence and Topic Diversity.
For topic coherence, we adopt a widely-used coherence metric, CV (Röder et al., 2015), which is shown to be better than other coherence metrics like UMASS (Mimno et al., 2011) and NPMI (Aletras and Stevenson, 2013) and have been adopted by many works in short text topic modeling (Wu et al., 2020; Wang et al., 2021; Wu et al., 2022). We use the well-adopted library Palmetto2 to compute CV with Wikipedia texts as the reference corpus. For topic diversity, we employ the Topic Uniqueness (TU ) for evaluation (Nan et al., 2019; Dieng et al., 2020). which is defined as the proportion of unique words among all the topical words.
Moreover, as pointed out in (Wu et al., 2020), there exists a trade-off relation between the coherence and diversity metrics. Higher TU scores tend to cause lower CV scores and vice versa. To provide a more comprehensive metric for topic quality, following previous work (Dieng et al., 2020), we adopt the Topic Quality (TQ) metric as the product of the topic diversity and coherence score,
TQ = CV ∗ TU. (15) We take the top 15 words with the highest probabilities of each topic for the aforementioned metrics following (Wu et al., 2022).
Results The results are shown in Table 1. From the results, we could find that our kNNTM model outperforms or achieves compatible performances with the start-of-the-art baselines, which proves the existence of the label sparsity problem and the effectiveness of our solution. We can find the kNNTM achieves high TU scores under many settings, which indicates that labels augmented by multiple documents bring more diverse information for topic optimization. Furthermore, in terms of the comprehensive evaluation metric, Topic Quality (TQ), kNNTM outperforms the baseline models in four distinct settings and attains comparable results in the remaining two scenarios. This underscores
2https://github.com/dice-group/Palmetto
the capability of our model to effectively strike a balance between topic coherence and diversity, resulting in the extraction of high-quality topics that exhibit both coherence and diversity.
It is worth noting that MCTM model achieves the highest CV scores under almost every setting, whereas its TU scores are notably diminished. This indicates that a set of coherent words frequently repeats across MCTM’s topics. Therefore, in spite of some coherent topics being discovered, many of those are repetitive and uninformative, hence making its TU and TQ scores hardly comparable with other methods.

heading: 5.2 Text Clustering
text: Metric To evaluate the quality of document-topic distributions, we leverage the short text clustering task following (Wang et al., 2022; Wu et al., 2022), and report the Purity and Normalized Mutual Information (NMI) (Schütze et al., 2008), where document labels are used during evaluation. Specifically, to compute Purity and NMI, following previous works (Zhao et al., 2021; Wu et al., 2022), we directly take the most significant topic as the cluster
assignment for each document, and the metrics are denoted as top-Purity and top-NMI. Moreover, we also calculate the results with the cluster assignments from K-Means algorithm, which could be found in Appendix D.
Results We report the results of text clustering Table 1. From the results, we can find out that the kNNTM model consistently outperforms all baseline models under almost every setting. The performances indicate that kNNTM can infer highquality document-topic distributions which accurately reflect the semantics of documents.

heading: 5.3 Ablation Studies
text: To analyze the effects of different modules of kNNTM, we compare kNNTM with its following
0 10 20 30 40 50 60 nk
0.300 0.305 0.310 0.315 0.320 0.325 0.330 0.335 0.340 0.345 C V & T Q
CV TU TQ
0 10 20 30 40 50 60 nk
0.74 0.75 0.76 0.77 0.78 0.79 0.80 0.81 0.82 Pu rit y top-Purity top-NMI0.94 0.95 0.96 0.97 0.98 0.99 TU 0.37 0.38 0.39 0.40 0.41 0.42 0.43
0.44
N M
I
Varying nk on Snippet
Figure 3: Sensitivity analysis on neighbor number nk.
0 0.01 0.1 1 10 0.300 0.305 0.310 0.315 0.320 0.325 0.330 0.335 0.340 0.345 C V & T Q CV TU TQ
0 0.01 0.1 1 10 0.74 0.75 0.76 0.77 0.78 0.79 0.80 0.81 0.82 Pu rit y top-Purity top-NMI0.94 0.95 0.96 0.97 0.98 0.99 TU 0.37 0.38 0.39 0.40 0.41 0.42 0.43
0.44
N M
I
Varying on Snippet
Figure 4: Sensitivity analysis on the coefficient α.
variants: 1) kNNTM-w/o-kNN: kNNTM without kNN label completion module, degenerating to the basic QTM model. 2) kNNTM-w/o-Topic: kNNTM with a distance metric without considering distances from the hidden topic space. 3) kNNTMw/o-BoW: kNNTM with a distance metric without considering the distances from the input BoW space. 4) kNNTM-w/-sim: kNNTM without kNN algorithm, and complementing labels directly with words based on pre-trained word embeddings.
The ablation studies are conducted on the Snippet dataset under 100 topics. The results are reported in Table 2, and the standard deviations are shown in Appendix Table 7 due to space limit. The effectiveness of the kNN label completion module is proved by the improvement from kNNTMw/o-kNN to the original kNNTM. The decreases in kNNTM-w/o-Topic and kNNTM-w/o-BoW indicate the importance of both views in the fused distance metric. Moreover, while CV sightly increases on kNNTM-w/-sim, the decreases of the performances on TU , TQ, and text clustering metrics indicate the kNN-based completion method would lead to higher topic quality and better topic distributions as it can better utilize the word patterns from current dataset.

heading: 5.4 Sensitivity Analysis
text: We conduct sensitivity analysis on two important hyper-parameters of kNNTM, the number of aggregated neighbors nk, and the coefficient α balancing the original label and the augmented label.
As shown in Figure 3, as nk gradually increases, the coherence score CV and the overall topic quality TQ increase initially and then decline when nk gets too large. A similar phenomenon can be found
on those metrics about text clustering, but the TU score keeps increasing. The reason might be that increasing nk will bring more diverse documents during label completion and further increase the diversity of learned topics. But more diverse documents will introduce more noisy words that are unrelated to the current document, and finally degrades the overall performance of the topic model.
For the coefficient α, a similar phenomenon could be found in Figure 4. As α increases, the training objective of the model puts more attention on the retrieved neighbor documents, whereas it can dominate the probabilities of the original documents once α gets too large, which can also degrade the model performance.

heading: 5.5 Topic Visualization
text: For qualitative evaluations of topics, we show the examples of topic words yielded by different baselines and our kNNTM model on the Snippet dataset in Table 3. We can observe that baseline models with lower TU , such as WLDA and MCTM, generate some repetitive topics with repeated words, such as "wikipedia", "encyclopedia", "physics", "physicist" and "astrophysics". Such repetitive sets of coherent words will lead to abnormally high results on CV scores, making it unfair to compare with other methods. However, we can see that kNNTM only generates a single coherent topic for each corresponding topic and its topic quality is apparently higher.

heading: 6 Conclusions
text: In this paper, we identify the label sparsity problem in short text topic modeling, resulting from the inherently limited document length. Subsequently, we design an novel neural short text topic modeling framework dubbed kNNTM, which mitigates the label sparsity problem with a kNN label completion module that aggregates semantically similar documents to augment the reconstruction labels. To effectively find similar documents, a fused multiview distance metric is proposed considering both local word similarities and global document semantics. Extensive experiments show that kNNTM outperforms the baselines and can generate highquality topic and document representations.
Limitations
One limitation of the proposed kNNTM model is the time complexity of computing the OT distances.
The original OT distance metric is known to have high complexity, and computing distances between each pair of documents also increases the time overhead. The inherent small text lengths and the accelerated algorithms for OT metric can help alleviate this issue, and the computations of the distances between different text pairs can be easily parallelized, but when dealing with excessively large datasets, kNNTM still faces high time cost. For future work, we hope to design a sampling strategy for kNNTM , aiming to restrict the nearest neighbor searching to a limited number of candidate documents instead of the entire dataset, and thus lowering the time cost of our model.
Ethics Statement
We comply with the ACL Code of Ethics. Our method is proposed to enhance short text topic modelling, and we believe our model would not cause significant social risks if applied appropriately.

heading: Acknowledgments
text: This work is supported by the National Natural Science Foundation of China (No.U23A20468).

heading: A kNNTM Algorithm Framework
text: Algorithm 1 The training procedure of kNNTM framework.
1: Input: the input corpus D, topic number K, pre-training epoch number P , total epoch number T , the number of nearest neighbors nk, hyperparameters α, η, ρ. 2: Output: K topic-word distributions βk, ND document-topic distribution θd 3: for epoch from 1 to T do 4: for a random batch of B documents do 5: Lbatch ← 0; 6: for each document d in the batch do 7: compute the topic distribution θd; 8: if epoch ≤ P then 9: Lbatch ← Lbatch + LTM by Eq.4; 10: else 11: get augmented reconstruction label x̃d by Eq.5 with distance in Eq.12; 12: Lbatch ← Lbatch + LkNNTM by Eq.14; 13: end if 14: update md with θd by Eq.10; 15: end for 16: update model parameters with∇Lbatch 17: end for 18: end for

heading: B Experimental Details
text: B.1 Dataset Statistics We conduct our experiments on the following public short text datasets:
GoogleNews: The GoogleNews dataset is from the Google News site and includes the titles of over 10,000 news articles categorized into 152 clusters.
Snippet: The Snippet dataset consists of over 10,000 search results from web across 8 different domains, obtained using predefined phrases.
StackOverflow: The StackOverflow dataset is sourced from the challenge data released by Kaggle. The dataset we use is a subset of 20,000 question titles randomly selected from 20 different tags from the original dataset.
We present the detailed statistics of the above three datasets after preprocessing in Table 4.
B.2 Baselines Here we provide brief introductions to the baseline methods compared in this paper.
prodLDA: prodLDA (Srivastava and Sutton, 2017) is a prominent work in neural topic models. It employs black-box neural variational inference and approximates the Dirichlet prior via a logistic normal distribution.
WLDA: WLDA (Nan et al., 2019) utilizes the Wasserstein autoencoder framework for neural topic modeling and directly enforces the Dirichlet prior through Maximum Mean Discrepancy.
ECRTM: To the best of our knowledge, ECRTM (Wu et al., 2023) is the current stateof-the-art neural topic model for normal long documents. It incorporates an embedding clustering regularization that encourages word embeddings to cluster around topic embeddings.
NQTM: NQTM (Wu et al., 2020) proposes learning peakier topic distributions and discovering better topics through topic distribution quantization and negative sampling.
MCTM: MCTM (Zhang and Lauw, 2022) focuses on variable-length corpora and utilizes metalearning to train a missing semantics predictor for short documents based on other long documents.
TSCTM: TSCTM (Wu et al., 2022) is a stateof-the-art short text neural topic model. It builds upon NQTM and introduces a contrastive objective on quantized distributions.
B.3 Implementation Details Regarding the training environment, our method is implemented using PyTorch 1.12.1 with Python 3.9.16, and the experiments are conducted on four GeForce RTX 2080Ti GPUs. Regarding the model architecture, the encoder network consists of a 3- layer MLP, and we set the hidden layer’s dimension to 128. Training is performed using the Adam optimizer (Kingma and Ba, 2014) with a learning rate of 0.002. We use a batch size (B) of 200, 20 pre-training epochs (P ), and a total of 200 epochs (T ). For other hyper-parameters, please refer to Table 5. We use grid search to determine the value of the above hyperparameters. And for all baselines, we follow the hyperparameter settings reported in their original papers. Additionally, we employ 300-dimensional GloVe embeddings as pretrained word embeddings for all the methods that require word embeddings.

heading: C Metrics for Topic-Word Distribution
text: C.1 Topic Coherence We use CV as the topic coherence metric in our experiments. For a topic z with T words {x1, x2, ..., xT }, the definition of CV is
CV (z) = 1
T
T∑
i=1
scos (vNPMI (xi) ,vNPMI (x1:T )) ,
(C.1) where scos(·) is the cosine similarity between two vectors, and the vNPMI vector is defined as
vNPMI (xi) = {NPMI (xi, xj)}j=1,...,T
vNPMI (x1:T ) =
{ T∑
i=1
NPMI (xi, xj)
}
j=1,...,T
.
(C.2) And the NPMI indicates the Normalized Pointwise Mutual Information between words and is calculated as
NPMI (xi, xj) = log
p(xi,xj)+ϵ p(xi)p(xj)
− log (p (xi, xj) + ϵ) , (C.3)
where p(xi, xj) is the co-occurrence probability within a reference corpus.
C.2 Topic Diversity We use Topic Uniqueness (TU ) as the metric for topic diversity. The TU of K topics {z1, z2, ..., zK} could be calculated as:
TU = 1
K
K∑
i=1
1
T
T∑
j=1
1
cnt ( xzij ) , (C.4)
where cnt(xi) indicates number of times that the word xi appears in all topics.

heading: D More Results on Text Clustering
text: Metrics To evaluated the models on text clustering method, besides top-Purity and top-NMI, we also apply the K-Means algorithm to assign clusters to different documents. We set cluster number set as the topic number K and apply the KMeans algorithm on all the document-topic distribution vectors. The metrics are denoted as km-Purity and km-NMI.
Results We show the results in Table 6. From the results, we could draw the same conclusions as in section 5.2. Our kNNTM model outperforms all baseline models under almost every setting, and achieves compatible results in a few scenarios. That indicates kNNTM possesses the ability
to obtain high quality document-topic distributions and derive the hidden semantics for each document.

heading: E More Results of Ablation Studies
text: Due to space limit, the standard deviations of the results in the Ablation Studies section are not reported in the main paper. Here we provide the main and the standard deviation of the results in Table 7, corresponding to Table 2.

heading: Future Work
text: For future work, we hope to design a sampling strategy for kNNTM , aiming to restrict the nearest neighbor searching to a limited number of candidate documents instead of the entire dataset, and thus lowering the time cost of our model