MIST: Mutual Information Maximization for Short Text Clustering (2024)

Abstract: Short text clustering poses substantial challenges due to the limited amount of information provided by each text sample. Previous efforts based on dense representations are still inadequate as texts are not sufficiently segregated in the embedding space before clustering. Even though the state-of-the-art method utilizes contrastive learning to boost performance, the process of summarizing all local tokens to form a sequence representation for the whole text includes noise that may obscure limited key information. We propose Mutual Information Maximization Framework for Short Text Clustering (MIST), which overcomes the information drown-out by including a mechanism to maximize the mutual information between representations on both sequence and token levels. Experimental results across eight standard short text datasets show that MIST outperforms the state-of-the-art method in terms of Accuracy or Normalized Mutual Information in most cases.1

heading: 1 Introduction
text: Text clustering is a vital task for a wide range of downstream applications. It aims to partition texts into groups of similar categories in an unsupervised manner. The growth of social media, discussion forums, and news aggregator websites has led to a large number of short-length texts being produced daily. Therefore, clustering these short texts has become important for many real-world applications, ranging from recommendation to text retrieval (Yohannes and Assabie, 2021).
In short texts, the most informative words and phrases of the text content usually appear only once. This exacerbates the sparsity problem, posing an additional hurdle for clustering short texts. Traditional methods, such as BoW and TF-IDF, provide relatively sparse representation vectors with
∗Equal Supervision. 1The code is available at https://github.com/
c4n/clustering_mist.
limited descriptive power. Hence, they perform poorly when clustered using a standard distancebased clustering algorithm (Hadifar et al., 2019).
To address this problem, most recent methods (Xu et al., 2017; Hadifar et al., 2019; Yin et al., 2021) utilize deep neural networks to map highdimensional data into meaningful dense representations in a lower-dimensional space and adopt a multi-stage scheme in which the clustering process is performed after learning feature representations. However, the clustering performance of these methods remains unsatisfactory, as texts still have a lot of overlap among categories in the latent space before clustering (Zhang et al., 2021).
Alternatively, an end-to-end clustering scheme (Zhang et al., 2021; Xie et al., 2016) simultaneously optimizes representation learning and clustering objectives. To achieve desirable outcomes, Zhang et al. (2021) propose a method that employs contrastive representation learning, which has been successful in self-supervised learning, to help spread out overlapping categories and produce effective short text representations.
As shown in Zhang et al. (2021), improving representation is crucial for enhancing the clustering performance. Nevertheless, the contrastive learning method used in Zhang et al. (2021) only considers sequence-level embeddings that are formed by averaging all local tokens in each text instance, including uninformative noise. This could generate a representation in which limited yet informative terms used to describe the text content may be obscured by noise, potentially affecting the clustering performance. We consider the preservation of limited information in such a low signal-to-noise environment as a vital feature for short text clustering. Addressing this gap will result in sequence representations that are more semantically representative and robust to noisy tokens in short texts.
In this paper, we introduce the Mutual Information Maximization Framework for Short
11309
Text Clustering (MIST), a new multi-stage approach. We aim to improve representation learning stage for short text clustering using two contrastive learning objectives operating at the sequence and token levels. In particular, we apply the concept of mutual information (MI) maximization to facilitate us in comparing the semantic similarity between representations across the two hierarchical levels.
The crux of our method lies in integrating the sequence-level and token-level MI maximization objectives concurrently for the following purposes.
1. Learning Distinct Text Representation: The first learning objective maximizes MI between each positive pair at the sequence level; 2. Informative Token Preservation: The second objective is designed to enforce each text representation at the sequence level to extract local information shared across all its individual tokens by directly maximizing MI between them. This way, we mitigate the obscurityby-noise problem and preserve limited key information in a weak signal environment.
The growth in the size of short text sequences may exacerbate a poor signal-to-noise ratio. To deal with short text samples with various signal-tonoise ratios, we additionally propose an adaptive weighting function that dynamically determines an appropriate ratio between the two objectives based on the length of text inputs. To our knowledge, the method of combining two MI maximization objectives logically is presented for the first time. Note that the representations at different levels have a direct implication on one another, and the sequence representations are subsequently used in the clustering stage by applying the k-means algorithm.
We conduct extensive experimental studies over the eight standard benchmarks. MIST improves the clustering performance in terms of Accuracy and Normalized Mutual Information in most cases compared to the current state-of-the-art while using an identical configuration across all datasets. This demonstrates the generalizability of our method.
Our main contributions are outlined as follows: (1) We propose a novel representation learning technique for short text clustering through the integration of sequence-level and token-level MI maximization objectives. (2) To balance the two objectives, we introduce an adaptive weighting function. (3) Our ablation study provides a further demonstration of how different prioritization of the two MI objectives impacts the clustering per-
formance across datasets of various text lengths; as text length increases, the preservation of limited local information becomes more significant.

heading: 2 Related Work
text: Short Text Clustering. There are several strategies to overcome the sparsity of short text representations. Some recent deep clustering methods either perform by breaking down the clustering framework into multiple stages, i.e., the clustering process is performed after learning feature representations, or by jointly optimizing both the representation learning and clustering.
Xu et al. (2015, 2017) propose a multi-stage clustering method, named STCC. For each dataset, word embeddings are initially pretrained using Word2Vec (Mikolov et al., 2013) on a large indomain corpus. A convolutional neural network is then employed to learn non-biased representations. Self-Train (Hadifar et al., 2019) utilizes Smooth Inverse Frequency (Arora et al., 2017) to enhance the pretrained Word2Vec embeddings. During training, it enriches discriminative features by jointly tuning an autoencoder with soft clustering assignments derived from a clustering method. The assignments are used as supervision to update the weights of the encoder network. SCA-AE (Yin et al., 2021) also adopts a cluster assignment constraint into the embedding space of the autoencoder for clusteringfriendly feature learning. For the aforementioned methods, they apply the k-means algorithm on the learned representations to obtain the final clusters.
The current state-of-the-art, SCCL (Zhang et al., 2021), is a deep joint clustering method optimized in an end-to-end fashion. It utilizes contrastive representation learning to encourage greater separation between overlapped categories in the original data space. By jointly optimizing a contrastive learning loss and a clustering objective, SCCL yields cutting-edge results. In addition, other contrastive learning methods have also been experimented on short text clustering, such as using entities for contrastive learning to provide supervision signals for their related sentences (Nishikawa et al., 2022), and using virtual augmentation for contrastive learning to circumvent the discrete nature of language (Zhang et al., 2022).
An alternative approach is to adopt clustering enhancement concepts, which aim to improve the quality of the initial clustering. HAC-SD (Rakib et al., 2020) proposes an iterative classification
method that applies outlier removal to detect and remove outliers. The authors found that the performance of hierarchical clustering combined with iterative classification outperforms other settings. To boost the clustering quality further, Pugachev and Burtsev (2021) exploit deep sentence representations (Cer et al., 2018) and make modifications to the iterative classification in Rakib et al. (2020).
Moreover, RSTC (Zheng et al., 2023) presents a new technique for short text clustering. It comprises two modules: a pseudo-label generation module and a robust representation learning module. The former module generates pseudo-labels, which are robust against the imbalance in data, as the supervision for the latter. The combination of class-wise and instance-wise contrastive learning is also utilized in the learning module to further improve robustness against the noise in data.
Self-Supervised Learning. Self-supervision has gained popularity and become a common technique in unsupervised representation learning for a variety of downstream purposes (Chen et al., 2020; He et al., 2020; Caron et al., 2020; Grill et al., 2020).
Learning meaningful representations by estimating and maximizing MI is one of the prominent contrastive learning strategies. Its effectiveness has been demonstrated in both vision (Hjelm et al., 2019; Bachman et al., 2019; Sordoni et al., 2021) and text domains (Kong et al., 2020; Caron et al., 2020; Giorgi et al., 2021). Deep Infomax (DIM) (Hjelm et al., 2019) introduces global and local MI maximization objectives for learning image representations. The authors find success in optimizing the local MI maximization objective by maximiz-
ing MI between local features and global features. However, the global and local MI objectives are implemented separately according to the end task. Inspired by local Deep InfoMax, Zhang et al. (2020) learns sentence representations by maximizing the MI between the sentence-level representation and its CNN-based n-gram contextual dependencies.
On the contrary, our method integrates two MI maximization strategies concurrently to learn textual representations for various short text characteristics. We also introduce a generalized adaptive weighting function for effectively balancing both MI maximization objectives.

heading: 3 Proposed Method: MIST
text: We propose a short text clustering framework consisting of two stages. First, we train a model using feature representation learning objectives as illustrated in Figure 1. Second, we apply the kmeans algorithm on the trained representations at inference time to obtain the final clusters. This investigation focuses on improving the first stage.
The main idea of our solution lies in the learning objective function L that takes into account an MI objective LMI and an unsupervised clustering objective LCluster, which is used to enforce the encoder to capture categorical structure and provide a suitable representation space for clustering task. L = βLMI + ηLCluster, (1) where β and η represent the trade-off between LMI and LCluster. In our experiments, we set β to 1 and η to 2 to provide more weight to LCluster.
In Section 3.1, we describe our main contribution, the MI maximization learning procedure, in-
cluding (1) sequence-level and token-level MI maximization objectives, and (2) an adaptive weighting function that is also incorporated to balance them. Section 3.2 presents the auxiliary clustering objective utilized in the representation learning stage.

heading: 3.1 Representation Learning with MI Maximization
text: Short texts are challenging to cluster due to the weak signal caused by noise. In the context of this study, short texts are recognized as those that are short in length and typically contain informal fragmental non-sentence structures, e.g., tweets and news snippets. One strategy to improve the clustering performance is to adopt contrastive learning to construct an embedding space that minimizes local invariance for each positive pair. However, a standard contrastive learning with PLM, which is performed by contrasting between sequence representations (global features), may allow noise to drown out sparse but informative local-token embeddings (local features) when these tokens are mean-pooled to form a sequence representation. Consequently, optimizing solely contrastive learning at the sequence level is insufficient for learning representations in a weak signal environment.

heading: 3.1.1 Hierarchical MI Objective
text: In contrast to previous works on MI maximization learning, which utilized each MI objective separately, we incorporate the learning of both sequence and token representations into a single objective. This strategy offers two advantages: (1) it mitigates the problem of information drown-out by allowing individual tokens to participate in the MI maximization process; (2) it supports weight adjustment between these two MI levels to handle short text inputs with various signal-to-noise ratios. Sequence-Token MI Maximization. According to Tian et al. (2020), contrastive learning is equivalent to maximizing the lower bound of MI between a sequence representation and its augmented version (positive). Intuitively, it reflects how much more precisely we can determine the representation given a positive compared to when we are unaware of the positive (Bachman et al., 2019). This principle enables us to incorporate an additional mechanism beyond the sequence-level contrastive learning.
We build our framework based on the MI maximization concept through the integration of two MI maximization objectives. In this way, our model can effectively learn distinct short text representations using the sequence-level MI objective while
simultaneously preserving local information using an additional objective. Specifically, the token-level MI objective helps alleviate the information obscurity from noise by maximizing the MI between each local token and its sequence representation.
As a result, the overall learning objective LMI consists of two components: (1) sequence-level MI maximization Iseq, and (2) token-level MI maximization Itok, operating concurrently in a sequencetoken hierarchy as shown in Figure 1.
LMI = −(1− λ)Iseq − λItok, (2) where λ corresponds to the balancing weight for Iseq and Itok objectives, which is defined in Eq.3. Adaptive Weighting Function. According to our analysis, short text inputs vary in length across different datasets, ranging from fragmental sequences of 6 words to 28 words on average. Regarding signal-to-noise, larger sequences tend to contain a greater proportion of noise that does not provide useful semantics for the clustering step, whereas important informative terms usually still appear once. This exacerbates the information drown-out problem due to a poor signal-to-noise ratio.
While other short text clustering techniques treat all text samples in the same fashion, we argue that different-length short texts should be handled differently. We propose a MI maximization strategy adaptable to text length so that our method can efficiently deal with short text instances containing varying signal-to-noise ratios, without the need for a hyperparameter search for any particular dataset. Since larger sequences require more effort to preserve limited crucial information, we place more weight on the Itok objective by encouraging λ to be larger as the total number of tokens in the text grows. Thus, our generalized adaptive weighting function (Eq.3) is introduced to assign the weight for λ depending on the average number of tokens in text samples for each minibatch of size N :
λ = max ( 0, ⌊ 0.1
N
N∑
i=1
li
⌉ − 1 ) × 0.1, (3)
where li denotes the number of tokens in a text xi and it is directly proportional to the text length.
In the representation learning stage, we randomly sample a minibatch Xo = xo1, ..., x o N of N original texts with empirical probability distribution P. Then, we generate an augmented version for each text to obtain an augmented batch Xa = xa1, ..., x a N , where X
o and Xa are of identical size. The encoder fθ, a pretrained language
model (PLM) network, encodes an input text x into a sequence of contextualized token embeddings with length l, fθ(x) := {f (i)θ (x) ∈ Rd}li=1, where i is the token index and d is the number of dimension. These token representations are then mean pooled m(fθ(x)) to generate a sequence representation denoted as g(x) = m(fθ(x)) ∈ Rd.

heading: 3.1.2 Computing the Sequence-level MI.
text: This learning objective, Iseq, aims to learn distinct text representations through the maximization of MI between the original sample and its augmented version at the sequence level. By treating each original text g(xo) and its augmentation g(xa) as positive pairs, the Iseq objective is defined as:
Iseq = 1 N ( ∑
x∈X ÎJSD(g(xo); g(xa))) (4)
We adopt a Jensen-Shannon estimator (Nowozin et al., 2016; Hjelm et al., 2019) to estimate a lower bound of MI, ÎJSDθ :
ÎJSDθ (g(xo); g(xa)) := EP [−sp(−g(xo) · g(xa)).] − EP×P̃ [sp(g(xo) · g(x̃a))] ,
(5)
where x̃a is a negative augmented textual input sampled from distribution P̃ = P, and sp(z) = log(1 + ez) is the softplus function.

heading: 3.1.3 Computing the Token-level MI
text: In contrast to Zhang et al. (2020), we constrain a sequence representation containing high MI with each token to preserve limited local information in the short text— by maximizing MI between the sequence representation and all of its token representations directly— instead of its local contextual n-gram embeddings. In particular, we attempt to maximize the average MI between a sequence representation and all its token representations while minimizing MI with the tokens of other texts. We define Itok for each minibatch as
Itok = 1 2N ( ∑
xo∈Xo
lxo∑
i=1
ÎJSD(g(xo); f (i)θ (xo)))
+ ∑
xa∈Xa
lxa∑
i=1
ÎJSD(g(xa); f (i)θ (xa))). (6)
An estimated MI for each sequence g(x) and token representations f (i)θ (x) is calculated as follows:
ÎJSDθ (g(x); f (i) θ (x)) :=
EP[−sp(−g(x) · f (i)θ (x))] − EP×P̃[sp(g(x) · f (i) θ (x̃))],
(7)
where x̃ is a different text on the minibatch.

heading: 3.2 Clustering with KL Divergence
text: In addition to the MI objective, we employ LCluster during the training stage to encourage the coalescence of samples that are most likely to belong to the same cluster. We follow the clustering method proposed by Xie et al. (2016), which is also used by Zhang et al. (2021). This method involves computing soft cluster assignments and formulating the clustering objective using KL divergence.
For the first step, we follow Xie et al. (2016) using the Student’s t-distribution Q to compute a soft cluster assignment for each text instance xj ∈ X and the centroid µk where µk ∈ {1, ...,K} for the dataset with K-clusters. Specifically, we compute the probability qjk of assigning a text xj to a cluster µk as follows.
qjk = (1 + ∥g(xj)− µk∥22 /α)− α+1 2
∑K k′=1(1 + ∥g(xj)− µk′∥22 /α)− α+1 2 (8)
The α symbol represents the degree of freedom of the distribution, and we set α to 1. Following Zhang et al. (2021), each centroid µk is approximated by the linear clustering head cθ.
The second step is calculating an auxiliary target distribution P and using it to assist in refining clusters’ centroids. The main idea is to give more importance to text samples with high clustering confidence. The probability pjk ∈ P is defined as
pjk = q2jk/ ∑ j′ qj′k∑
k′(q 2 jk′/ ∑ j′ qj′k′) . (9)
To match the soft cluster assignments to the target distribution, the KL-divergence between probability distributions P and Q is computed as follows.
ℓCj = KL [pj ||qj ] = K∑
k=1
pjk log pjk qjk
(10)
We then formulate it as a clustering objective for each minibatch of size N as
LCluster = N∑
j=1
ℓCj /N. (11)

heading: 4 Experimental Setup
text: Datasets. Following previous works (Rakib et al., 2020; Zhang et al., 2021; Pugachev and Burtsev, 2021; Zheng et al., 2023), we conduct experiments and evaluate the performance of MIST on the eight standard short text clustering datasets. The descriptions and statistics of the datasets are provided in Appendix A.1
Implementation. We implement our model in PyTorch (Paszke et al., 2017) and use the paraphrase-mpnet-base-v2 in Sentence Transformers library (Reimers and Gurevych, 2019) as the encoder, with a linear clustering head following Zhang et al. (2021). The encoder is trained for 1,200 iterations for all datasets and we use Adam optimizer with a batch size of 256. The learning rate of the encoder and the clustering head are set to 6e−6 and 6e−5, respectively. We follow Xu et al. (2017) and Hadifar et al. (2019) by randomly selecting 10% of data as the validation set. Furthermore, we follow Zhang et al. (2021) by not performing any preprocessing operations on all eight datasets. Although some of the existing works preprocess the texts by removing symbols, stop words, and punctuation, or converting them to lowercase.
In the training stage, the original and augmented texts are taken into consideration as inputs for the MI objective LMI, since we found that they are more effective than employing two augmented pairs. We follow Zhang et al. (2021) by utilizing Contextual Augmenter (Kobayashi, 2018; Ma, 2019) to generate augmented samples for each text instance, as it was demonstrated to produce the
best outcomes in their study. To assess clustering performance, we use the same standard metrics—Accuracy (ACC) and Normalized Mutual Information (NMI)— as used in all competitive methods 2. The results are averaged over five trials.

heading: 5 Experimental Results
text: We extensively compare the performance of MIST with state-of-the-art methods including STCC (Xu et al., 2017), Self-Train (Hadifar et al., 2019), HACSD (Rakib et al., 2020), SCA-AE (Yin et al., 2021), SCCL (Zhang et al., 2021), and RSTC (Zheng et al., 2023). The abovementioned methods are described in Section 2. Two reproduced versions of SCCL are also included for more comprehensive comparisons. In addition, this section provides ablation studies on our proposed method in various settings to understand the impact of each component.

heading: 5.1 Main Results
text: As shown in Table 1, MIST achieves state-of-theart results in terms of Accuracy and NMI for most cases on the eight benchmark datasets. In contrast,
2The Accuracy is calculated via the Hungarian algorithm, and NMI measures the information shared between the ground truth assignments and the predicted assignments.
HAC-SD and Self-Train attain the best results in only two cases, whereas SCCL and RSTC produce the best outcome in only one case. Note that, the performances of MIST are collected using the identical setting and training iteration across all datasets to demonstrate generalizability. As a result, the need for a specific configuration for each dataset is avoided, enabling a reduction in model overhead.
For datasets with a small number of clusters in the upper section of the table, MIST shows superior performances on AgNews for both metrics and StackOverflow in terms of NMI. Notably, there are two datasets that MIST is outperformed by competitors for both ACC and NMI, i.e., Biomedical and SearchSnippets. For Biomedical, Hadifar et al. (2019) dominate all competitive methods. They achieve the best results using a large in-domain biomedical corpus to train word embeddings, whereas the dataset used to pretrain our encoder and other recent methods is general domain.
For SearchSnippets, we observe that most of the text samples are collections of keywords and terminologies rather than coherent sentence structures. Moreover, SearchSnippets samples are mediumlength fragmental sequences; as a result, the tokenlevel MI maximization objective is more emphasized due to the length of the texts. These two factors exert a direct impact on the token-level MI maximization objective while it is being executed in the learning stage. Since the token vectors are contextualized representations, forcing the model to learn from incoherent contextual signals can be detrimental to the overall sequence representations, which are subsequently used in the clustering stage. This can be more problematic when the same keywords appear in multiple clusters.
As demonstrated in the lower section of the table, MIST obtains the best outcomes on most of the datasets containing a large number of clusters. However, due to the fine-grained categorization of these datasets, short texts in different clusters may share similar content or keywords, hence inducing ambiguity. This ambiguity in textual data and ground truths can lead to erroneous predictions. Moreover, another cause of inaccuracy is when the text content in one cluster is a subtopic of another. For GoogleNews-T, which only contains news headlines that are relatively short with few keywords, it presents a challenge for clustering these extremely short texts into a large number of clusters. In terms of Accuracy, our method
achieves a result comparable to that of Rakib et al. (2020) on GoogleNews-T. We conjecture that the hierarchical clustering and the outlier removal algorithms employed in their method can better deal with the hierarchical nature of data in this scenario. However, MIST outperforms Rakib et al. (2020) in terms of NMI on this dataset.
Although GoogleNews-S and GoogleNews-TS share the same challenges as GoogleNews-T, clustering texts in both datasets is more accurate due to the benefit of additional context and information provided in the texts themselves. As GoogleNewsS contains snippets of news, and GoogleNews-TS includes both titles and snippets of news. Consequently, MIST achieves superior clustering performances on both datasets for both matrices.
Furthermore, we thoroughly compare MIST with SCCL, as this current state-of-the-art model also utilizes the advantage of contrastive representation learning and aims to improve the effectiveness of representations for short text clustering, which is similar to our contributions, by reproducing SCCL in two versions for a fair comparison: (1) an end-toend (original) version and (2) a multi-stage version. For the latter version, we add the clustering stage by applying k-means algorithm on top of SCCL representations to get the final clusters, referred to as SCCL-Multi. In particular, the architecture of SCCL-Multi is analogous to our framework, except for the representation learning technique. Moreover, both reimplemented versions of SCCL use the same PLM backbone and augmentation setting as our proposed model in this study.
The comparative results show that MIST outperforms SCCL for both versions in most cases. More specifically, the superior performances of MIST compared to SCCL-Multi demonstrate that our proposed representation learning procedure improves short text representations more effectively than the standard contrastive learning objective used in the SCCL framework for short text clustering task. MIST also consistently surpasses both reimplemented versions of SCCL in other settings, including settings indicated in their publication in most cases, as shown in Appendix A.6.

heading: 5.2 Ablation Study
text: To better understand the effect of the various model modifications on the clustering performance and the analysis versus text lengths, we conducted additional experiments by varying the trade-off between components in our training procedure.

heading: 5.2.1 The Impact of Sequence- and Token-MI Maximization Objectives
text: This experiment studies the impact of the ratio between two MI maximization objectives on the clustering performance and the importance of incorporating both objectives in our representation learning procedure. We report and analyze the performance of our model using four different values of λ. Particularly, λ denotes the weight of the token-level MI maximization objectives Itok, and 1-λ represents the weight of the sequence-level MI maximization objectives Iseq. We consider the following settings: (1) MIST-seq: our model with a sequence-only MI maximization objective (λ = 0), (2) MIST-tok: our model with a token-only MI maximization objective (λ = 1), (3) MIST-equal: our model with both objectives are given an equivalent weight (λ = 0.5), and (4) MIST: our proposed version, i.e., our model with the value of λ determined by the adaptive weighting function defined in Eq.3, varying according to input text length.
As shown in Figure 2, MIST with the value of λ set by Eq.3 yields the best performances in terms of Accuracy for most datasets and shows performance gains compared to other settings. We also discovered that NMI tends to follow the same trend as Accuracy, as presented in Appendix A.2. This demonstrates that the length of short texts has a great impact on determining the appropriate ratio between the two MI maximization objectives, i.e. the optimal ratio varies by text inputs. By utilizing the proposed adaptive weighting function, MIST can perform effectively across various datasets.
For datasets containing medium or large fragmental sequences, such as GoogleNews-TS, the value of λ calculated by Eq.3 is greater than 0 in this scenario. The proposed version of MIST yields the best outcomes. Remarkably, MIST-equal and MIST-tok always outperform MIST-seq in this case.
This shows that only the sequence-level objective is inadequate when dealing with lengthy short texts, as larger fragments usually have a poor signal-tonoise ratio. However, this issue can be mitigated by performing the token-level MI maximization during the learning representation stage.
For small fragment datasets, such as Tweet, text samples are relatively short and contain less signalto-noise problem. In this scenario, the weight λ calculated by Eq.3 is equal to 0, i.e., MIST is identical to MIST-seq, which outperforms all other settings. MIST-tok and MIST-equal may encourage the encoder to learn text representations by placing emphasis on keywords that could also appear in multiple clusters, causing ambiguity and error in clustering. Hence, the token-level MI objective provides advantages when used in a suitable condition.
In addition, we investigate the situation in which the MI objective is removed (β = 0), MIST-noMI. The ablation results show significant drops in the performance on all datasets. This implies that the MI objective is essential for performance gain.

heading: 5.2.2 The Impact of the Clustering Objective
text: As shown in Figure 2, the clustering performance drops drastically when we remove the clustering objective (η = 0) during learning representations, MIST-noClstr. This demonstrates that the categorical structure imposed by jointly optimizing the
clustering objective with the MI objective is a crucial component in boosting clustering performance. Furthermore, we observe that as the weight of the clustering objective (η) increases, the performances continuously improve until η reaches its saturation point at 2. In Figure 3, the average Accuracy and NMI for all eight datasets improve as the clustering weight is steadily increased until it reaches 2.

heading: 6 Conclusion
text: We propose a novel multi-stage short text clustering framework that mainly focuses on improving the representation learning stage. Our adaptive learning approach integrates two MI maximization objectives operating at the sequence and token levels to produce effective representations. This mechanism allows us to simultaneously learn distinct text representations while preserving limited information in a weak signal environment. In addition, we introduce a generalized adaptive weighting function that considers the length of text inputs to determine an optimal ratio between the two MI maximization objectives during the learning stage.
MIST outperforms competitive methods in most cases in terms of Accuracy and NMI across eight benchmark datasets. This demonstrates that utilizing the MI maximization strategy for learning representation in a constrained environment could potentially be a promising tactic. Further study would be worthwhile since it might enhance the quality of textual representations for other tasks.

heading: Limitations
text: This section discusses the limitations of our proposed framework. Firstly, the encoder of our model is pretrained using general domain data. Hence, the performance of MIST drops when it runs on short texts in a specific domain, such as Biomedical. Furthermore, short text inputs containing only of keywords or incoherent text sequences hinder the performance of our representation learning method. In particular, when dealing with lengthy texts that lack coherence, optimizing both token-level MI and sequence-level MI maximization forces a sequence representation to resemble each individual token embedding. The token-level MI maximization objective provides no further improvement in this case. This issue is exacerbated when some terms are shared across clusters. This constraint should be taken into account in future research.
Another limitation involving the general operation of contrastive learning is the choice of augmen-
tation technique, which directly affects the clustering performance. Notably, the best augmentation strategy is still a subject of discussion and needs more exploration. An exploration of data augmentations in Zhang et al. (2021) and our own experiments on various augmentation settings show that the choice of augmentation and the configuration parameters both affect the clustering performance. Additionally, even if the augmenter and the parameters used to generate augmented texts are exactly the same, there is a possibility that the outcomes from the two trials may vary, adding a variance to the performance results.

heading: Acknowledgements
text: The authors would like to express our deepest appreciation to Narin Kunaseth for his help with coding. Additionally, we are extremely grateful to Patsorn Sangkloy for her invaluable suggestions and feedback on the presentation aspect of this work.

heading: A Appendices

heading: A.1 Datasets
text: Following previous works (Rakib et al., 2020; Zhang et al., 2021; Pugachev and Burtsev, 2021; Zheng et al., 2023), we conduct experiments and assess the performance of our model on eight English benchmark datasets for short text clustering. Table 2 presents the important statistics of all datasets.
• AgNews: a subset of the English news titles dataset (Zhang and LeCun, 2015) in 4 different topics, with 2,000 samples chosen randomly from each topic by Rakib et al. (2020).
• SearchSnippets: a dataset consisting of 12,340 web search snippets from 8 different categories (Phan et al., 2008).
• Biomedical: 20,000 paper titles, from 20 different Medical Subject Headings (MeSH), randomly selected by Xu et al. (2017) from the PubMed data distributed by BioASQ3.
• StackOverflow: challenge data published on Kaggle and randomly chosen by Xu et al. (2017), comprising 20,000 questions from Stack Overflow related to 20 distinct tags.
• Tweet: a dataset comprising 2,472 tweets with 89 groups (Yin and Wang, 2016).
• GoogleNews: GoogleNews-TS is a collection of titles and text snippets from 11,109 news articles covering 152 events (Yin and Wang, 2016). Only titles and snippet of each news article were extracted to produce GoogleNewsT and GoogleNews-S, respectively.

heading: A.2 The Effects of Sequence- and Token-MI Maximization Objectives on NMI
text: Figure 4 shows the impact of the different ratios between the two MI maximization objectives on the clustering performance in terms of NMI across eight short text datasets. It follows the same trend as Accuracy as discussed in Section 5.2.1. MIST with our proposed generalized adaptive weighting function obtains the best clustering performance in terms of NMI for most datasets.

heading: A.3 Positive Pairs in Contrastive Learning
text: It is a common practice in contrastive learning frameworks to only consider augmented texts as inputs, excluding original samples. However, we adopt a different input scheme. We discovered that feeding both original and augmented samples into our representation learning framework (as shown in Figure 1) yields better clustering results than exclusively taking two augmented texts as an input pair. One plausible reason is that when augmented texts are generated, the augmenter replaces some keywords in the original texts with new words. Short texts inherently have few keywords; hence, the absence of crucial words required for text categorization impacts the clustering performance.

heading: A.4 The Analysis of the Clustering Objective
text: As discussed in Section 5.2.2, the clustering performance is substantially affected by the weight of the clustering objective. Table 3 presents the performance of MIST across eight datasets in three situations, i.e., the coefficient of the clustering objective, η, in Eq.1 is assigned to 0, 1, and 2. The optimal results for the majority in terms of ACC and NMI are produced when η is set to 2.

heading: A.5 Exploration of Data Augmentations
text: According to Zhang et al. (2021), which has studied the impacts of data augmentation in extensive details. The Contextual Augmenter has shown that it substantially outperforms other augmenters in their study. They hypothesized that since both the Contextual Augmenter and their encoder use the pretrained transformers as the backbones, this allows the Contextual Augmenter to produce augmentation texts that are more informative and beneficial to their framework. We also adopted a pretrained transformer as the encoder in our framework and we observed that the experimental results followed
the same trend as Zhang et al. (2021). We thus employ this augmenter in our experiments.
In this section, we investigate the impact of the Contextual Augmenter in different configurations: masked language models and word substitution ratios. As shown in Table 4, we found that MIST using augmented texts generated from BERT model with 20% substitution rate yields the best overall performance, we then use this setting in our framework. Moreover, MIST with augmented texts produced by other masked language models with a 20% substitution rate also yields outcomes close to those of BERT with the same substitution rate.

heading: A.6 SCCL Reimplementation
text: To thoroughly compare the performance of our proposed representation learning strategy against the standard contrastive learning method in SCCL (Zhang et al., 2021), we reproduced SCCL in (1) an end-to-end version (SCCL) and (2) a multiple-stage version (SCCL-Multi). For the latter version, we add the clustering stage by applying the k-means algorithm on top of SCCL representations to make their pipeline identical to our framework except for the representation learning strategy.
To be more specific, in this study, we report the experimental results of both reimplemented versions of SCCL using the backbone specified in the experimental setup of their publication. Moreover, SCCL considers the Contextual Augmenter with three configurations by setting the word substitution ratio of each text instance to 10%, 20%, and 30%. However, their study does not identify which setting produces the best outcomes. Therefore, we evaluate both reproduced versions of SCCL using three masked language models: BERT-base, RoBERTa, and DistilBERT, with the aforementioned word substitution ratios for augmented pair generation to cover various scenarios and for comprehensive comparison.
Table 5 reports the clustering performance of SCCL in both reproduced versions and in all configurations mentioned previously. The reported performances show that despite the reproduced SCCL employing the configuration specified in their reference paper, their outcomes are still inferior to MIST in most cases. More specifically, the proposed version of MIST with the setup described in Section 4 outperforms SCCL and SCCL-Multi with the best parameter settings in most cases.
The fact that MIST produces better clustering performance than SCCL-Multi in this study emphasizes that our proposed representation learning technique improves short text representations more effectively than the standard contrastive learning objective in the SCCL framework for short text clustering task. This demonstrates the success and efficiency of our proposed learning method even when compared with SCCL in various settings. Note that we collected the experimental results of reimplemented versions of SCCL from the best iteration for each dataset throughout 3000 iterations instead of using a stopping criterion, which is not indicated in their publication.
Interestingly, the percentage of word replacement and the choice of masked language models for augmented text generation directly impact the clustering performance. Moreover, the best setting for these two parameters varies across different datasets. However, the performances of our proposed method presented in Table 1 are reported by using only a single setting for all datasets.

heading: A.7 Discussion on Training Times
text: In this section, we discuss training times and computational resource requirements of our proposed method. From our experimental studies, we have found variation in the training times for different short text datasets based on two main factors: the number of clusters (NCluster) and the average num-
ber of words in each document (NWord). Depending on the specific dataset, MIST requires 0.63 to 1.40 GPU hours on a Tesla V100 32G GPU to complete the framework. The Googlenews-TS dataset takes the longest training time, while the Tweet dataset takes the least.
The auxiliary clustering objective employed in our representation learning stage involves the computation of the probability of assigning each text to each cluster and the computation of the KL divergence between those clustering assignments and the target distribution in each iteration, which is time-consuming. This is exacerbated when the model runs on datasets contaning a large number of clusters. For example, GoogleNews-S with 152 clusters takes 1.27 times longer in terms of running time than AgNews with 4 clusters, where both datasets have a similar average number of words in each document. Therefore, our model works slowly in this scenario.
The number of words in each document is also an important factor affecting the running time of MIST. The number of words in a sentence is proportional to the total number of tokens, which directly determines the number of computations to perform the token-level MI maximization objective of the sentence. For instance, Biomedical requires 1.21 times more running time than StackOverflow, despite both datasets containing exactly the same number of clusters and documents.
Moreover, we study the training times of SCCL in the same environment as MIST: the computational resource, the backbone, and the augmentation setting. As there are no stopping criteria or collection conditions defined clearly in their publication, we then examine the running times of SCCL in three scenarios for comprehensive comparisons.
We first consider training SCCL using 1200 iterations, which is the same number of training iterations as the proposed MIST model. SCCL requires 0.83 to 1.43 GPU hours, depending on the training dataset and its number of clusters. SCCL shows the resemblance trend and range of running times to our method across standard short text datasets.
Secondly, we measure the training time of SCCL at the iteration that provides the best result within the maximum iteration, i.e., 3000, specified in their code. We discovered that the best iteration varies according to the input dataset. For example, AgNews and Tweet require more than 2400 and 2800 iterations to achieve optimal outcomes, respectively. Both demand longer training times compared to the proposed MIST with the setting described in Section 4, about 1.92 and 4.12 times, respectively. On the other hand, some datasets require less than 1200 iterations for training the model to achieve the best results before gradually decreasing when the number of iterations is increased, such as SearchSnippets and GoogleNewsTS, which require around 600 training iterations.
However, the practical training for observation and collecting the best results corresponds to the maximum iteration—3000 iterations. Therefore, the third comparison is assessed based on the running time of SCCL at iteration 3000. Depending on the input dataset, training SCCL with the maximum iteration requires 2.55 to 3.28 GPU hours, which is a significant amount of time more than the MIST model with the proposed setting in total.

heading: Future Work
text: This constraint should be taken into account in future research. Another limitation involving the general operation of contrastive learning is the choice of augmen-
tation technique, which directly affects the clustering performance