STSPL-SSC: Semi-Supervised Few-Shot Short Text Clustering with Semantic text similarity Optimized Pseudo-Labels (2024)

Abstract: This study introduces the Semantic Textual Similarity Pseudo-Label Semi-Supervised Clustering (STSPL-SSC) framework. The STSPL-SSC framework is designed to tackle the prevalent issue of scarce labeled data by combining a Semantic Textual Similarity Pseudo-Label Generation process with a Robust Contrastive Learning module. The process begins with employing k-means clustering on embeddings for initial pseudo-Label allocation. Then we use a Semantic Text Similarityenhanced module to supervise the secondary clustering of pseudo-labels using labeled data to better align with the real clustering centers. Subsequently, an Adaptive Optimal Transport (AOT) approach fine-tunes the pseudo-labels. Finally, a Robust Contrastive Learning module is employed to foster the learning of classification and instance-level distinctions, aiding clusters to better separate. Experiments conducted on multiple real-world datasets demonstrate that with just one label per class, clustering performance can be significantly improved, outperforming state-of-the-art models with an increase of 1-6% in both accuracy and normalized mutual information, approaching the results of fully-labeled classification.

heading: 1 Introduction
text: With Large Language Models (LLM) and Pretrained Language Models (PLM) advancing rapidly, downstream tasks are increasing, demanding larger datasets, especially in early-stage businesses or specialized domains. These settings often lack labeled data, hindering traditional algorithms. Obtaining task-specific labels is time-consuming and costly, leading researchers to explore unsupervised text clustering. However, such methods require prior knowledge of clustering categories and suffer from uncontrollable clustering centers. Semi-supervised
*Corresponding author
learning under small samples offers a promising solution.
Few-Shot Learning (FSL) (Wang et al., 2020) efficiently categorizes data into meaningful categories with minimal labeled examples. Unlike traditional learning methods that rely on large volumes of labeled data to train models. This is particularly valuable in scenarios where labeled data is scarce or costly to obtain, but unlabeled data is abundant.
Pseudo-labeling generates artificial labels for unlabeled data, aiding training in few-shot learning scenarios (Cascante-Bonilla et al., 2021). This approach leverages the model’s own predictions to assign labels to unlabeled instances, effectively using the model’s current understanding to augment its training data. In few-shot learning, where labeled examples are minimal, pseudo-label can significantly enhance the learning process by providing a larger, albeit synthetically labeled, dataset. This method allows for iterative refinement of the model’s performance, as the pseudo-label data helps bridge the gap between the scarcity of labeled examples and the abundance of unlabeled data. It is particularly valuable in few-shot learning as it circumvents the limitation of having only a few labeled examples, enabling models to learn more complex patterns and improve generalization capabilities.
In this study, we introduce the Semantic text similarity-enhanced Pseudo-Label Enhanced Clustering (STSPL-SSC) framework, a novel semisupervised learning approach aimed at overcoming the limitations posed by scarce labeled data across various domains. Unlike traditional methods, STSPL-SSC integrates Semantic text similarityenhanced Pseudo-Label Generation with Robust Contrastive Learning to refine the clustering process effectively. The framework begins by applying k-means clustering on embeddings to generate initial pseudo-labels for each data point. A subse-
12174
quent refinement process, guided by the Semantic text similarity between authentically labeled and pseudo-label data, improves the pseudo-labels’ accuracy. This is achieved by employing secondary clustering that not only enhances the clustering effectiveness but also adjusts the pseudo-label to align more closely with the actual clustering centers through Adaptive Optimal Transport (AOT).This is achieved through secondary clustering, which not only improves clustering effectiveness but also adjusts the pseudo-label to align more closely with the actual clustering centers using Adaptive Optimal Transport (AOT). Additionally, STSPL-SSC incorporates a Robust Contrastive Learning module that generates augmentation pairs, facilitating the learning of both categorical and instance-level distinctions. This innovative method significantly bolsters the framework’s robustness against imbalanced and noisy datasets, ensuring more reliable clustering outcomes. Through extensive experiments conducted on eight short text clustering datasets, STSPL-SSC demonstrates superior performance, highlighting its effectiveness in semisupervised learning for short text clustering.

heading: 2 Related work

heading: 2.1 Semi-Supervised Few-Shot
text: In the the few-shot scenario, semi-supervised learning is a good solution. Recent research efforts have explored the application of semi-supervised learning to address the few-shot problem: (Hadifar et al., 2019) leveraged an effective Self-Training (ST) method within the realm of semi-supervised learning. Similarly, (Xu et al., 2023) employed LLMs to synthesize data and utilized Self-Training to learn features from the synthesized data. They utilized assignments from a clustering algorithm as supervision to update the weights of the encoder network.
In our research, we opted for real data to ensure minimal errors stemming from external factors. Following the paradigm of Self-Training, we optimize the overall training process using Semantic text similarity.

heading: 2.2 Pseudo-Label
text: Pseudo-label generates predicted labels for unlabeled data, enhancing learning performance with limited annotated data. However, the accuracy of pseudo-labels directly impacts the model’s generalization ability, as inaccurate pseudo-labels may
lead to performance degradation. There are several common practices: one method (Wang et al., 2021; Tsai et al., 2022) is based on a self-training strategy, where the basic model is first trained on labeled data and then the model is retrained on unlabeled data and labeled with high-confidence pseudo-labels. Another approach (Sohn et al., 2020) combines the idea of coherence learning, which employs unlabeled data to enhance model robustness under data perturbation. Building on these approaches (Yang et al., 2023)develops previous pseudo-labeling research using prototype learning, which enhances text representations by clustering them using prototypes for low-density separation, and mitigating unbalanced class bias through prototype-guided pseudo-labeling.
In our study, we utilize semantic similarity enhancement and Adaptive Optimal Transport (AOT) to optimize the generation of pseudo-labels, ensuring that the obtained pseudo-labels closely resemble the labeled data.

heading: 2.3 Baseline Articles
text: Our methodology references and improves upon the methods in these two articles. (Zhang et al., 2021) proposed the Supporting Clustering with Contrastive Learning (SCCL) framework, which improves clustering effectiveness by combining self-supervised instance contrastive learning and unsupervised clustering loss. The SCCL model employs pre-trained Sentence Transformer as an encoder and optimizes clustering loss and contrastive loss through end-to-end training. (Zheng et al., 2023) introduced Robust Short Text Clustering (RSTC), which addresses data imbalance and noise issues by introducing Self-Adaptive Optimal Transport (SAOT) and contrastive learning. Our methodology builds upon and enhances the techniques introduced in these two papers. By leveraging semantic similarity enhancement between labeled data and pseudo-labels, we obtain more informative features, thereby improving the effectiveness of subsequent AOT and contrastive learning. Experimental results also validate the feasibility of our approach.

heading: 3 Method

heading: 3.1 Semantic Textual Similarity Pseudo-Label Semi-Supervised Clustering (STSPL-SSC)
text: The STSPL-SSC framework introduces an innovative approach to address the challenge of limited
labeled datasets in various domains, a common obstacle in semi-supervised learning requiring extensive expert tuning (Ren et al., 2018). This framework combines Semantic text similarity enhanced Pseudo-Label Generation with Robust Contrastive Learning, as illustrated in Figure 1. Initially, it employs k-means (MacQueen et al., 1967) clustering on embeddings to assign each data point a pseudo-label P. This is followed by a refinement step where secondary clustering enhances clustering effectiveness, guided by the Semantic text similarity between authentically labeled data Ld, and pseudo-label P. This yields improved pseudo-labels S, while tracking the variances between P and S. To enhance further clustering accuracy, Adaptive Optimal Transport (AOT) is utilized to adjust pseudolabels Q, closer to the true clustering centers of Ld. Finally, the framework introduces a Robust Contrastive Learning module. This module generates augmentation pairs for each data point, creating augmented batches that facilitate the contrastive learning of categorical and instance-level distinctions. This method improves robustness against imbalanced and noisy data, leading to more stable clustering results.

heading: 3.2 Semantic Textual Similarity Pseudo-Label (STSPL)
text: The Semantic text similarity-enhanced PseudoLabel Generation module, a cornerstone of the STSPL-SSC framework, aims to address the limitations observed in deep joint clustering methods such as those proposed by (Xie et al., 2016; Hadifar et al., 2019; Zhang et al., 2021; Zheng et al.,
2023). Despite their popularity, these methods face challenges primarily due to the lack of supervisory information, which hampers the learning of discriminative representations, and their susceptibility to degenerate solutions, especially in severely imbalanced datasets (Hu et al., 2021; Yang et al., 2017; Ji et al., 2019).
Our module incorporates labeled data Ld during the generation of pseudo-labels P, mirroring a semi-supervised process but eliminating the need for continuous expert optimization of labels. By utilizing Ld as a supervisory signal, we compute the cosine similarity between the embeddings of Ld and the pseudo-label S to gauge their Semantic text similarity. This similarity assessment helps identify the deviation of clustering centers from Ld, thereby enhancing pseudo-label generation and the adaptive optimal transport (AOT) process.
This module unfolds in three primary steps as depicted in Figure 1: Step 1 involves clustering assignment where initial pseudo-labels are assigned. In Step 2, a semi-supervised Semantic text similarity enhancement process leverages the labeled data Ld to refine the pseudo-label S, enhancing their accuracy and relevance. Finally, Step 3 applies AOT to adjust the clustering centers closer to Ld, further refining the pseudo-labels. This approach addresses the challenges of label scarcity and clustering center deviation.

heading: 3.2.1 Clustering assignment
text: The objective of the clustering assignment is to categorize samples with a null label through an initial unsupervised clustering, aiming to derive predic-
tive values for the original texts. To accomplish this, we employ the BGE-M3 model (Xiao et al., 2023) as the encoding network Φ, which is pivotal due to the crucial role of Semantic text similarity enhancement. The effectiveness of utilizing an advanced pre-trained model for word embeddings is confirmed by our experiments. We innovatively combine semantic similarity into the optimization and clustering of pseudo-labels to obtain better clustering results.
The encoding process can be formalized as E = Φ(X) ∈ RN×D1 , where X denotes the original text, E the encoded representation, N the batch size, and D1 the dimensionality of the representation.
Subsequently, a fully connected layer, serving as the clustering network Gp, is utilized to predict the clustering assignment probabilities: Gp(E) = P ∈ RN×C , where C represents the number of categories. It is essential to highlight that within this module, both the encoding network Φ and the clustering network Gp are kept constant.

heading: 3.2.2 Semi-supervised Semantic text similarity enhancement
text: The aim of semi-supervised Semantic text similarity enhancement is to enhance the clustering assignment outcomes from Step 1. By discerning the extent of deviation from the labeled data, this process attempts to draw cluster centers nearer to the labeled data, hence mitigating the challenges posed by unknown category distributions. Securing more reliable pseudo-labels is a significant concern in such scenarios. Common semi-supervised methods combine supervised learning with unsupervised learning in deep neural networks (Rasmus et al., 2015), or use self-training (ST) (Artetxe et al., 2018; Cai and Lapata, 2019; Gera et al., 2022) approaches typically focus on using student-teacher models to assign pseudo-labels to the unlabelled data, thereby improving accuracy. we compute the cosine similarity between the embeddings of Ld and the pseudo-label P to gauge their Semantic text similarityto get the new pseudo-label S.
The reason for choosing semantic text similarity lies in its similarity to clustering principles, involving computation of vector differences. It is capable of deeply exploring the distances between the pseudo-labels P assigned post-clustering and each labeled data Ld. P will undergo cosine similarity calculation with each Ld to obtain the most similar one, recording the new label as the pseudo-label S.
The formula is expressed as follows:
S = argmaxP
( P · Ld
∥P∥2||Ld||2
) (1)

heading: 3.2.3 Adaptive Optimal Transport (AOT) Method
text: We refer to the AOT method as outlined in RSTC. The Adaptive Optimal Transport (AOT) method is designed to optimize pseudo-label generation by solving a discrete optimal transport (OT) problem. This process involves several key components and parameters as described below. The AOT optimization problem is formulated as:
min π,b
⟨π,M⟩+ ϵ1H(π) + ϵ2(Ψ(b))T 1 (2)
subject to the constraints π1 = a, πT 1 = b, π ≥ 0, and bT 1 = 1, where the objective function aims to minimize the transportation cost between the probability mass of samples and classes, adjusted by entropy regularization and a penalty function related to class distribution.
After obtaining π, pseudo-labels can be generated via an argmax operation as follows:
Qij =    1, if j = argmax j′ πij′ ,
0, otherwise. (3)
This operation ensures that each sample is assigned to the class with the highest probability, resulting in a one-hot encoded pseudo-label matrix Q.
Hyperparameters Description:
• ϵ1 and ϵ2 are balance hyper-parameters that regulate the impact of entropy regularization and the penalty function, respectively, allowing for a flexible adjustment to accommodate various data characteristics.
• Ψ(b) = − log b − log(1 − b) is the penalty function that addresses the distribution of classes by penalizing extreme values of b, thereby encouraging a more uniform distribution of class assignments and preventing clustering degeneracy.
• H(π) = ⟨π, log π − 1⟩ represents the entropy regularization term, which smoothens the transport plan by discouraging overly sparse solutions, thus facilitating a more robust pseudo-label generation process.
• a = 1N 1 signifies the uniform distribution of samples, ensuring that each sample contributes equally to the transport process.
• b indicates the initially unknown class distribution.

heading: 3.3 Robust Contrastive Learning module
text: In the Robust Contrastive Learning module, we employ instance augmentation techniques to expand the set of examples and introduce noise to the model, thereby improving its robustness. This is inspired by a body of research that underscores the utility of text augmentation in enhancing model resilience across various settings, as discussed by (Wenzel et al., 2022). Further inspiration comes from (Chen et al., 2020; Zhang et al., 2021; Dong et al., 2022), and the RSTC framework (Zheng et al., 2023), which suggests that post-pseudo-label clustering can exploit instance-level contrastive learning with augmented positive and negative samples to facilitate cluster consolidation and separation.
For implementation, contextual augmenters (Kobayashi, 2018; Ma, 2019) generate two augmented versions of the original text, termed X(1) and X(2). Considering the entire framework utilizes BGE-M3 for embedding analysis, the same method for generating word embeddings is adopted here. This yields augmented representations E(1) and E(2), denoted as E(1) ∈ RN×D1 , E(2) ∈ RN×D1 , where N is the batch size and D1 is the dimensionality of the embeddings. These are followed by k-means clustering to obtain predicted values P (1) and P (2), expressed as P (1) ∈ RN×C , P (2) ∈ RN×C , where C is the number of clusters. A fully connected layer, serving as the projection network Gz , maps these representations to a new space, facilitating the application of instance-level contrastive loss. The projected representations Z(1) and Z(2) are thus Z(1) ∈ RN×D2 , Z(2) ∈ RN×D2 , with D2 representing the dimensionality of the new space.
In category-level contrastive learning, we aim for the consistency of cluster predictions between augmentations deemed as positive pairs. Two augmentations from the same original text are treated as a positive pair, with a contrastive task defined on these pairs. The pseudo-label Q serve as the target for the augmented texts, with the Ld acting as the ultimate target. The discrepancy between Q and
Ld, represented as α, is calculated as:
α = Q− S N
(4)
This discrepancy α plays a positive role in the computation of the category-level contrastive loss, which is defined subsequently.
LC = α× 1
N
( ∥Q− logP (1)∥+ ∥Q− logP (2)∥ )
(5) Instance-level contrastive learning seeks to enhance the consistency between the projection representations of positive augmentation pairs while maximizing the distance between those of negative pairs. For a batch of 2N augmented texts, their projection representations are Z = [Z(1), Z(2)]T . In this batch, for any positive pair (two augmented texts derived from the same original text), the remaining 2(N − 1) augmented texts are treated as negative samples. The loss function for a positive pair (i, j), where i and j come from the same original text and the rest are considered negatives, is defined as:
ℓ(i, j) = − log exp(sim(Zi, Zj)/τ)∑2N k=1 1{k ̸=i} exp(sim(Zi, Zk)/τ)
(6) Within this framework, sim(Zi, Zj) denotes the cosine similarity computed between Zi and Zj , and τ is the temperature parameter. The instance-level contrastive loss calculates the loss for all positive pairs within a batch, including both (i, j) and (j, i):
LI = 1
2N
N∑
i=1
(ℓ(i, 2i) + ℓ(2i, i)) (7)
By integrating pseudo-supervised category-level contrastive learning with instance-level contrastive learning, we are able to derive robust representations that can accurately distinguish between different clusters.

heading: 3.4 Overall Framework
text: The total loss function of the STSPL-SSC model is formulated through a combination of pseudosupervised class-level contrastive loss and instancelevel contrastive loss. Specifically, the overall loss expression of STSPL-SSC is given by:
LTotal = LC + λI · LI , (8)
where λI represents a hyperparameter that balances the two types of losses. Adopting this strategy
enhances the STSPL-SSC model’s ability to handle dataset imbalances and boosts its robustness against data noise. The model consists of two mutually reinforcing modules that form a closed loop, facilitating optimization towards labeled data. As iterations progress, representation learning becomes more robust, and clustering predictions become more accurate, thanks to the more reliable pseudolabels obtained during the iterative process.
The specific operational procedure is as follows: Initially, we use the k-means clustering algorithm to initialize the embedding, obtaining P , which are then compared with the labeled data Ld to enhance Semantic text similarity, generating pseudo-labels Q. Under the guidance of these pseudo-labels, the model is trained in batches to learn robust representations. Throughout the training process, we dynamically update the Q values using the logarithmic distribution method proposed by (YM. et al., 2020). Finally, by examining the column indices corresponding to the maximum values in each row of the P matrix, we obtain the clustering assignments. Training is terminated when the changes in clustering assignments between two consecutive updates of P are less than a predefined threshold δ, or when the maximum number of iterations is reached. The threshold δ represents the baseline rate of change for the pseudo-labels Q; if this baseline is reached, the optimization improvement is minimal. If the maximum number of iterations is reached without achieving the threshold δ, it indicates that the model may be overfitting, with cluster centers unable to approach the labeled data Ld. Thus, the Adaptive Optimal Transport (AOT) continuously alters the pseudo-labels Q, indicating that no amount of training will result in optimization. This design allows the STSPL-SSC model to self-improve during iterations, optimizing representation and clustering prediction accuracy, thereby achieving higher data processing effectiveness and robustness.

heading: 4 Experiments
text: In this section, we conduct experiments on realworld datasets to emulate the environment encountered in actual work settings. Through these experiments, significant improvements were observed across all datasets, with accuracy (ACC) enhancement rates between 1-7% and Normalized Mutual Information (NMI) enhancement rates also between 1-8%, compared to state-of-the-art short text
clustering methods. This illustrates that under the same word embedding model, our Semantic text similarity-enhanced pseudo-label generation module can successfully augment performance, and we have experimentally determined ideal hyperparameters.

heading: 4.1 Datasets
text: Detailed experiments were performed on eight real-world datasets: AgNews, StackOverflow, Biomedical, SearchSnippets, GoogleNews-TS, GoogleNews-T, GoogleNews-S, and Tweet. Among these, AgNews, StackOverflow, and Biomedical are balanced datasets; SearchSnippets is a mildly imbalanced dataset, while GoogleNews, GoogleNews-T, GoogleNews-S, and Tweet are severely imbalanced datasets. Following (Zhang et al., 2021), raw data was used as input to demonstrate our training process’s robustness to noise, ensuring a fair comparison. Additional details about the datasets are provided in Appendix A.1.

heading: 4.2 Experimental Setup
text: Our models are implemented in PyTorch 2.0 (Paszke et al., 2019) and trained using the Adam optimizer (Kingma and Ba, 2017).We experimentally selected labeled data number, λI , ϵ1, ϵ2. More details can be found in Appendix A.2. Following previous studies (Xu et al., 2017; Hadifar et al., 2019; Rakib et al., 2020; Zhang et al., 2021; Zheng et al., 2023), since our method mainly addresses the problem of scarce real data, the number of clusters is set to the actual number of classes, and the evaluation metrics are Accuracy (ACC) and Normalized Mutual Information (NMI). The exact definitions of these metrics are elaborated in Appendix A.3. All experiments are repeated five times and the average results are reported.

heading: 4.3 Baselines
text: Here is the translation in the style of an academic paper: We compare our proposed method with the following short text clustering techniques and semisupervised classification methods. SCCL (Zhang et al., 2021) surpasses these approaches by utilizing SBERT (Reimers and Gurevych, 2019) as its backbone and introducing instance-level contrastive learning to support clustering. Moreover, SCCL adopts the deep joint clustering objective proposed by (Xie et al., 2016), obtaining the final clustering assignment via k-means. RSTC
(Zheng et al., 2023) extends SCCL by incorporating a pseudo-label generation module using the SAOT solution and combining it with SCCL’s contrastive learning module to enhance robustness against noise. Multi-MCCR (Zhou et al., 2023) is a semi-supervised classification method that consists of multiple models with the same structure and the C-BiKL loss strategy. The C-BiKL loss strategy is proposed to minimize the combination of bidirectional weights. BGE-M3 (Xiao et al., 2023) is a novel self-knowledge distillation approach designed to improve the performance of single retrieval mode. It is the first embedding model that supports all three retrieval methods.

heading: 4.4 Clustering Performance
text: The comparative results on eight datasets are shown in Table 1. From the analysis, we identify several key findings: SCCL achieved improved results by introducing instance-level contrastive learning to mitigate the noise issue but is prone to degenerate solutions and poor application of k-means. RSTC, using SBERT for word embeddings, outperformed previous methods; however, the cluster centers obtained through k-means do not necessarily reflect the labeled data and require iterative refinement, especially for scattered datasets. Multi-MCCR and BGE-M3 simulate the semi-supervised effect by limiting the number of training labels, and their performance is positively correlated with the number of training labels. Table 1 shows the ACC when the number of labels is 100. As they are semi-supervised methods, there is no evaluation of NMI. More experiments are in Appendix A.5. STSPL-SSC surpasses all baselines, demonstrating the effectiveness of enhancing semantic text similarity with labeled data to achieve better clustering performance.

heading: 4.5 In-depth analysis

heading: 4.5.1 Ablation Study
text: To explore the effects of Semantic text similarity and different word embeddings on STSPL-SSC’s performance, we compared STSPL-SSC against variants including STSPL-SSC-SS and STSPLSSC-B. STSPL-SSC-SS utilizes SBERT for word embedding generation, keeping the Semantic text similarity-enhanced pseudo-label generation and Robust Contrastive Learning modules intact. Conversely, STSPL-SSC-B, employing BGE-M3 for word embeddings, excludes the Semantic text similarity enhancement, losing the guidance of labeled data Ld in the pseudo-label Q and loss, maintaining the Robust Contrastive Learning module. It can be observed that both semantic similarity enhancement and the replacement of pre-trained word embedding models have played a significant role. The semantic similarity enhancement module has a notable effect on severely imbalanced models. The reason is that with an increase in the number of dataset categories, semantic similarity enhancement can prevent clustering degradation, thereby improving clustering performance.

heading: 4.5.2 Visualization
text: To further demonstrate the effectiveness of the key Semantic text similarity-enhanced module, we employed t-SNE (van der Maaten and Hinton, 2008) for visualizing the representations derived from RSTC, STSPL-SSC-SS, STSPL-SSC-B, and STSPL-SSC. The visualization results on the StackOverflow dataset are depicted in Figures 2(a)-(d). It is evident that: STSPL-SSC achieves the most optimal text representations, characterized by smaller intra-cluster distances and larger inter-cluster distances, with only a minimal number of points misclassified. The underlying reasons for these observations are consistent with the findings analyzed in the ablation study.

heading: 4.5.3 Effect of hyper-parameter
text: We investigate the impact of hyperparameters on model performance, including the number of labeled data, ϵ1, ϵ2, and λI . Given that the core component is the Semantic Text Similarity-enhanced module, we primarily discuss the influence of the number of labeled data; details on the remaining hyperparameters can be found in Appendix A.4. In datasets where the number of labeled data is sufficient, we experiment with varying the number of labeled data to {1, 2, 5, 10}, observing negligible
impact on balanced datasets with fewer categories, such as AG News and Stack Overflow. However, in the case of SearchSnippets, an increase in labeled data paradoxically led to a decrease in performance, potentially due to the emergence of uncertainty in cluster centroids as the number of labeled data grows, resulting in a deterioration of performance. Based on our experiments, we ultimately opt for number of labeled data = 1.

heading: 5 Conclusion
text: This paper presents a robust semi-supervised short text clustering model that includes a Semantic text similarity-enhanced Pseudo Label Generation module and a Robust Contrastive Learning module. Utilizing a semi-supervised approach for generating pseudo labels with labeled data for supervision, our innovation significantly enhances clustering performance by employing few-shot learning to bolster Semantic text similarity, achieving near fully-
supervised clustering effectiveness with just one correct label. This greatly increases the usability of unlabeled data for meaningful clustering, reducing costs and providing potential solutions for the lack of training data in downstream tasks of LLM and PLM transfer. Our method demonstrates state-ofthe-art performance across eight datasets.

heading: 6 Limitations
text: While the model requires only a minimal number of samples, it still necessitates determining the number of sample categories. Performance degradation can occur when categories have inherently minimal differences, making it challenging for contrastive learning to facilitate cluster separation, potentially leading to data points clustering at the inter-cluster boundaries. Future efforts will focus on overcoming issues of excessive similarity to enhance cluster separation.

heading: A Experiments
text: A.1 Datasets We conduct extensive experiments on eight popularly used real-world datasets to assess the effectiveness and generality of our approach. The details of each dataset are as follows:
• AgNews (Rakib et al., 2020): A subset of AG’s news corpus collected by (Zhang et al., 2015), consisting of 8,000 news titles across four topic categories.
• StackOverflow (Xu et al., 2017): Comprises 20,000 question titles with 20 different tags, randomly selected from the challenge data published on Kaggle.com.
• Biomedical (Xu et al., 2017): Consists of 20,000 paper titles from 20 different topics, selected from the challenge data published on BioASQ’s official website.
• SearchSnippets (Phan et al., 2008): Contains 12,340 snippets from eight different classes, selected from the results of web search transactions.
• GoogleNews (Yin and Wang, 2016): The titles and snippets of 11,109 news articles about 152 events, divided into three datasets: the full dataset is GoogleNewsTS, while GoogleNews-T only contains titles and GoogleNews-S only includes snippets.
• Tweet (Yin and Wang, 2016): Consists of 2,472 tweets related to 89 queries, with the original data from the 2011 and 2012 microblog track at the Text Retrieval Conference.
A.2 Experiment Settings In our experiments, we chose the bge-base-en-v1.5 model (Xiao et al., 2023) from the Sentence Transformer (Reimers and Gurevych, 2019) library for text encoding, with the maximum input length set to 32. The learning rate was set to 5 × 10−6 for optimizing the encoding network, and 5×10−4 for optimizing the projection network and clustering network. The dimensions of the text representations and projection representations were set to D1 = 768 and D2 = 128, respectively. The batch size was set to N = 200. The temperature parameter was set to τ = 1, and the threshold δ was set
to 0.01. For all other baselines, including SCCL (under MIT-0 license) and RSTC, we used the code released by their respective authors.
A.3 Evaluation Metrics
We employ two prevalent metrics for evaluating text clustering outcomes: accuracy (ACC) and normalized mutual information (NMI), as adopted by prior research (Xu et al., 2017; Hadifar et al., 2019; Zhang et al., 2021; Zheng et al., 2023).
ACC is given by:
ACC =
∑N i=1 1{yi = map(ŷi)}
N , (9)
where yi and ŷi denote the ground truth and the predicted label for the text xi, respectively.
NMI is given by:
NMI(Y, Ŷ ) = I(Y, Ŷ )√ H(Y )H(Ŷ ) , (10)
where Y and Ŷ represent the vectors of ground truth and predicted labels, I denotes the mutual information, and H denotes the entropy.
A.4 Hyperparametric effect We investigate the impact of hyperparameters on model performance, including ϵ1, ϵ2, and λI . We begin by examining the effects of ϵ1 and ϵ2, varying them within the sets {0.05, 0.1, 0.2, 0.5} and
{0, 0.001, 0.01, 0.1, 1}, respectively. The results are reported in Figures 4(a) and 4(b). Figure 4(a) illustrates that the accuracy is insensitive to ϵ1. Figure 4(b) highlights the importance of choosing appropriate hyperparameters for datasets with different levels of imbalance, especially for the severely imbalanced GoogleNews-T dataset. Empirically, we select ϵ1 = 0.1 and ϵ2 = 0.1 for balanced datasets, ϵ2 = 0.01 for mildly imbalanced datasets, and ϵ2 = 0.001 for severely imbalanced datasets. Subsequently, we explore the influence of λI by varying it within the set {0, 1, 5, 10, 20, 50, 100}. The results on three datasets are shown in Figure 4(c) and 4(d). It is observed that performance improves with an increase in λI , then remains relatively stable after λI reaches 1, and finally decreases when λI becomes too large. We conclude that when λI is too small, it fails to fully leverage the capabilities of instance-level contrastive learning. Conversely, when λI is too large, it suppresses the ability of category-level contrastive learning, thereby diminishing clustering performance. Based on experience, we select λI = 10 for all datasets.
A.5 Comparison with Semi-supervised Methods
In the experiments of Multi-MCCR and BGE-M3 (Figure 5 and Figure 6), referring to the experimental setup of the more advanced semi-supervised method Multi-MCCR, training is performed at the number of labeled data (1, 10, 25, 50, 100), and the training epochs are all 100. The dashed lines represent the training results of the two methods under different label amounts, while the solid lines represent the results of our method. The hyperparameters of Multi-MCCR are set according to the original paper: Hyperp: C = 4, dropout = 0.2, α = 6, Epochs = 8, Maxlength = 128, Batchsize = 12. Under conditions similar to our experiments, our method performs well on severely imbalanced datasets. Due to the large number of categories, if the specific number of categories is uncertain and only partially labeled data is available, it will lead to label imbalance, and the classification situation will be very unsatisfactory. Our method avoids this problem.
A.6 Computational Budget The training environment we used is the GeForce RTX 4090 GPU, with each dataset taking approximately 15-30 minutes to run.

heading: Future Work
text: Future efforts will focus on overcoming issues of excessive similarity to enhance cluster separation