Robust Representation Learning with Reliable Pseudo-labels Generation via Self-Adaptive Optimal Transport for Short Text Clustering
Year: 2023

Abstract:
Short text clustering is challenging since it takes imbalanced and noisy data as inputs. Existing approaches cannot solve this problem well, since (1) they are prone to obtain degenerate solutions especially on heavy imbalanced datasets, and (2) they are vulnerable to noises. To tackle the above issues, we propose a Robust Short Text Clustering (RSTC) model to improve robustness against imbalanced and noisy data. RSTC includes two modules, i.e., pseudo-label generation module and robust representation learning module. The former generates pseudo-labels to provide supervision for the later, which contributes to more robust representations and correctly separated clusters. To provide robustness against the imbalance in data, we propose self-adaptive optimal transport in the pseudo-label generation module. To improve robustness against the noise in data, we further introduce both class-wise and instance-wise contrastive learning in the robust representation learning module. Our empirical studies on eight short text clustering datasets demonstrate that RSTC significantly outperforms the state-of-the-art models. The code is available at: https://github.com/hmllmh/RSTC.

heading: 1 Introduction
text: Text clustering, one of the most fundamental tasks in text mining, aims to group text instances into clusters in an unsupervised manner. It has been proven to be beneficial in many applications, such as, recommendation system (Liu et al., 2021, 2022a,b), opinion mining (Stieglitz et al., 2018), stance detection (Li et al., 2022), etc. With the advent of digital era, more and more people enjoy sharing and discovering various of contents on the web, where short text is an import form of information carrier. Therefore, it is helpful to utilize short text clustering for mining valuable insights on the web.
∗Corresponding author.
However, short text clustering is not a trivial task. On the one hand, short text has many categories and the category distributions are diversifying, where the heavy imbalanced data is common. The heavy imbalanced data is prone to lead to degenerate solutions where the tail clusters (i.e., the clusters with a small proportion of instances) disappear. Specifically, the recent deep joint clustering methods for short text clustering, (Hadifar et al., 2019) and (Zhang et al., 2021), adopt the clustering objective proposed in (Xie et al., 2016), which may obtain a trivial solution where all the text instances fall into the same cluster (Yang et al., 2017; Ji et al., 2019). (Zhang et al., 2021) introduces instancewise contrastive learning to train discriminative representations, which avoids the trivial solution to some extent. However, (Zhang et al., 2021) still tends to generate degenerate solutions, especially on the heavy imbalanced datasets.
On the other hand, short text is typically characterized by noises, which may lead to meaningless or vague representations and thus hurt clustering accuracy and stability. Existing short text clustering methods cope with the noise problem in three ways, i.e., (1) text preprocessing, (2) outliers postprocessing, and (3) model robustness. Specifically, earlier methods (Xu et al., 2017; Hadifar et al., 2019) apply preprocessing procedures on the text (HaCohen-Kerner et al., 2020) for reducing the negative impact of noises. The recent method (Rakib et al., 2020) proposes to postprocess outliers by repeatedly reassigning outliers to clusters for enhancing the clustering performance. However, both preprocessing and postprocessing methods do not provide model robustness against the noise in data. The more recently short text clustering method SCCL (Zhang et al., 2021) proposes to utilize the instance-wise contrastive learning to support clustering, which is useful for dealing with the noises in the perspective of model robustness. However, the learned representations of SCCL lack discrim-
10493
inability due to the lack of supervision information, causing insufficiently robust representations.
In summary, there are two main challenges, i.e., CH1: How to provide model robustness to the imbalance in data, and avoid the clustering degeneracy? CH2: How to improve model robustness against the noise in data, and enhance the clustering performance?
To address the aforementioned issues, in this paper, we propose RSTC, an end-to-end model for short text clustering. In order to improve model robustness to the imbalance in data (solving CH1) and the noise in data (solving CH2), we utilize two modules in RSTC, i.e., pseudo-label generation module and robust representation learning module. The pseudo-label generation module generates pseudo-labels for the original texts. The robust representation learning module uses the generated pseudo-labels as supervision to facilitate intra-cluster compactness and inter-cluster separability, thus attaining more robust representations and more correctly separated clusters. The better cluster predictions in turn can be conductive to generate more reliable pseudo-labels. The iterative training process forms a virtuous circle, that is, the learned representations and cluster predictions will constantly boost each other, as more reliable pseudo-labels are discovered during iterations.
The key idea to solve CH1 is to enforce a constraint on pseudo-labels, i.e., the distribution of the generated pseudo-labels should match the estimated class distribution. The estimated class distribution is dynamically updated and expected to get closer to the ground truth progressively. Meanwhile, the estimated class distribution are encouraged to be a uniform distribution for avoiding clustering degeneracy. We formalize the idea as a new paradigm of optimal transport (Peyré et al., 2019) and the optimization objective can be tractably solved by the Sinkhorn-Knopp (Cuturi, 2013) style algorithm, which needs only a few computational overheads. For addressing CH2, we further introduce class-wise contrastive learning and instancewise contrastive learning in the robust representation learning module. The class-wise contrastive learning aims to use the pseudo-labels as supervision for achieving smaller intra-cluster distance and larger inter-cluster distance. While the instancewise contrastive learning tends to disperse the representations of different instances apart for the separation of overlapped clusters. These two modules
cooperate with each other to provide better short text clustering performance.
We summarize our main contributions as follows: (1) We propose an end-to-end model, i.e., RSTC, for short text clustering, the key idea is to discover the pseudo-labels to provide supervision for robust representation learning, hence enhancing the clustering performance. (2) To our best knowledge, we are the first to propose self-adaptive optimal transport for discovering the pseudo-label information, which provides robustness against the imbalance in data. (3) We propose the combination of class-wise contrastive learning and instance-wise contrastive learning for robustness against the noise in data. (4) We conduct extensive experiments on eight short text clustering datasets and the results demonstrate the superiority of RSTC.

heading: 2 Related Work

heading: 2.1 Short Text Clustering
text: Short text clustering is not trivial due to imbalanced and noisy data. The existing short text clustering methods can be divided into tree kinds: (1) traditional methods, (2) deep learning methods, and (3) deep joint clustering methods. The traditional methods (Scott and Matwin, 1998; Salton and McGill, 1983) often obtain very sparse representations that lack discriminations. The deep learning method (Xu et al., 2017) leverages pre-trained word embeddings (Mikolov et al., 2013) and deep neural network to enrich the representations. However, the learned representations may not appropriate for clustering. The deep joint clustering methods Hadifar et al. (2019); Zhang et al. (2021) integrate clustering with deep representation learning to learn the representations that are appropriate for clustering. Moreover, Zhang et al. (2021) utilizes the pre-trained SBERT (Reimers and Gurevych, 2019) and contrastive learning to learn discriminative representations, which is conductive to deal with the noises. However, the adopted clustering objectives are prone to obtain degenerate solutions (Yang et al., 2017; Ji et al., 2019), especially on heavy imbalance data.
Among the above methods, only Zhang et al. (2021) provides model robustness to the noise in data. However, its robustness is still insufficient due to the lack of supervision information. Besides, Zhang et al. (2021) cannot deal with various imbalanced data due to the degeneracy problem. As a contrast, in this work, we adopt pseudo-label
technology to provide reliable supervision to learn robust representations for coping with imbalanced and noisy data.

heading: 2.2 Pseudo-labels for Unsupervised Learning
text: Pseudo-labels can be helpful to learn more discriminative representations in unsupervised learning (Hu et al., 2021). Caron et al. (2018) shows that k-means clustering can be utilized to generate pseudo-labels for learning visual representations. However, it does not have a unified, well-defined objective to optimize (i.e., there are two objectives: k-means loss minimization and cross-entropy loss minimization), which means that it is difficult to characterize its convergence properties. Asano et al. (2020) proposes SeLa to optimize the same objective (i.e., cross-entropy loss minimization) for both pseudo-label generation and representation learning, which can guarantee its convergence. Besides, SeLa transforms pseudo-label generation problem into an optimal transport problem. Caron et al. (2020) proposes SwAV which combines SeLa with contrastive learning to learn visual representations in an online fashion. However, both SeLa and SwAV add the constraint that the distribution of generated pseudo-labels should match the uniform distribution, to avoid clustering degeneracy. With the constraint, it is hard for them to cope with imbalanced data. As a contrast, in this work, we propose self-adaptive optimal transport to simultaneously estimate the real class distribution and generate pseudo-labels. Our method enforce the distribution of the generated pseudo-labels to match the estimated class distribution, and thus can avoid clustering degeneracy and adapt to various imbalanced data.

heading: 3 Methodology

heading: 3.1 An Overview of RSTC
text: The goal of RSTC is to discover and utilize the pseudo-labels to provide supervision for robust representation learning. RSTC consists of pseudolabel generation module and robust representation learning module, as illustrated in Fig.1. The pseudo-label generation module aims to generate reliable pseudo-labels for the robust representation learning module. To achieve this aim, we first obtain cluster predictions by the cluster assignment step, then we excavate pseudo-label information from the predictions by the self-adaptive optimal transport (SAOT) step. The robust representation
learning module aims to use the generated pseudolabels as supervision to train robust representations. To achieve this goal, we introduce class-wise and instance-wise contrastive learning. In this way, RSTC can provide robustness to imbalanced and noisy data, thus enhancing the clustering performance.

heading: 3.2 Pseudo-label Generation Module
text: We first introduce the pseudo-label generation module. Although the deep joint clustering methods (Xie et al., 2016; Hadifar et al., 2019; Zhang et al., 2021) are popular these days, their clustering performance is limited due to the following reasons. Firstly, lacking supervision information prevents the deep joint clustering methods from learning more discriminative representations (Hu et al., 2021). Secondly, they are prone to obtain degenerate solutions (Yang et al., 2017; Ji et al., 2019), especially on heavy imbalanced datasets. Therefore, to provide reliable supervision information for various imbalanced data, we propose SAOT in the pseudo-label generation module to generate pseudo-labels for the robust representation learning module. The overview of pseudo-label generation module is shown in Fig.1(a), which mainly has two steps: Step 1: cluster assignment, and Step 2: SAOT.
Step 1: cluster assignment. Cluster assignment aims to obtain cluster predictions of the original texts. Specifically, we adopt SBERT (Reimers and Gurevych, 2019) as the encoding network Φ to encode the original text X as Φ(X) = E ∈ RN×D1 where N denotes batch size and D1 is the dimension of the representations. We utilize the fully connected layers as the clustering network Gp to predict the cluster assignment probability (predictions), i.e., Gp(E) = P ∈ RN×C , where C is the category number. The encoding network and the clustering network are fixed in this module.
Step 2: SAOT. SAOT aims to exploit the cluster predictions to discover reliable pseudo-label. Asano et al. (2020) extends standard cross-entropy minimization to an optimal transport (OT) problem to generate pseudo-labels for learning image representations. This OT problem can be regarded as seeking the solution of transporting the sample distribution to the class distribution. However, the class distribution is unknown. Although Asano et al. (2020) sets it to a uniform distribution to avoid degenerate solutions, the mismatched class
distribution will lead to unreliable pseudo-labels. Therefore, it is essential to estimate real class distribution for addressing this issue. The recent research (Wang et al., 2022) studies the class distribution estimation, but it tends to cause clustering degeneracy on heavy imbalanced data, which we will further discuss in Appendix A. Hence, to discover reliable pseudo-labels on various imbalanced data, we propose SAOT.
We will provide the details of SAOT below. We expect to minimize the cross entropy loss to generate the pseudo-labels by solving a discrete OT problem. Specifically, we denote the pseudo-labels as Q ∈ RN×C . Let π = 1NQ be the transport matrix between samples and classes, M = − logP be the cost matrix to move probability mass from samples to classes. The reason that we use 1N between π and Q is the transport matrix should be a joint probability (Cuturi, 2013), i.e., the sun of all values in the π should be 1, while the sum of each raw in Q is 1. We have, Q∗ = argmin
Q ⟨Q,− logP ⟩ =
Nargmin π ⟨π,M⟩. Thus, the OT problem is as follows:
min π ⟨π,M⟩+ ϵH(π) s.t. π1 = a,πT1 = b,π ≥ 0, (1)
where ϵ is a balance hyper parameter, H(π) = ⟨π, logπ−1⟩ is the entropy regularization (Cuturi, 2013), a = 1N 1 is the sample distribution, and b is an unknown class distribution. To avoid clustering degeneracy and obtain reliable transport matrix with randomly initialized b, we introduce a penalty function about b to the OT objective and update b
during the process of solving the transport matrix. We formulate the SAOT optimization problem as:
min π,b ⟨π,M⟩+ ϵ1H(π) + ϵ2(Ψ(b))T1 s.t. π1 = a,πT1 = b,π ≥ 0, bT1 = 1, (2)
where ϵ1 and ϵ2 are balance hyper-parameters, Ψ(b) = − log b−log(1−b) is the penalty function about b. The penalty function not only limits bj (a value of b) ranges from 0 to 1, but also avoids clustering degeneracy by encouraging b to be a uniform distribution. The encouragement is achieved by increasing the punishment for bj that is close to 0 or 1. Besides, the level of the encouragement can be adjusted by ϵ2. Specifically, there are two critical terms in Equation (2) for exploring b, i.e., (1) the cost matrix M and (2) the penalty function Ψ(b), and we use ϵ2 to balance these two terms. For balanced data, both M and Ψ(b) encourage b to be a uniform distribution. For imbalanced data, M encourages the head clusters (i.e., the clusters with a large proportion of instances) to have larger bj and the tail clusters (i.e., the clusters with a small proportion of instances) to have smaller bj . When bj of a tail cluster approaches 0, this tail cluster tends to disappear (clustering degeneracy). Whereas Ψ(b) still encourages b to be a uniform distribution for avoiding the degeneracy. With a decent trade-off parameter ϵ2, SAOT can explore appropriate b and obtain reliable π for various imbalanced data. We provide the optimization details in Appendix B. After obtaining π, we can get pseudo-labels by
argmax operation, i.e,
Qij =    1, if j = argmax j′ πij′
0, otherwise. (3)
It should be noted that, for convenience, we let π = 1NQ before. However, π is essentially a join probability matrix and πij can be decimals, while each row of Q is a one-hot vector.
Through the steps of cluster assignment and selfadaptive optimal transport, we can generate reliable pseudo-labels on various imbalanced data for the robust representation learning module.

heading: 3.3 Robust Representation Learning module
text: We then introduce the robust representation learning module. To begin with, motivated by (Wenzel et al., 2022), we propose to adopt instance augmentations to improve the model robustness against various noises. Furthermore, inspired by (Chen et al., 2020), (Zhang et al., 2021) and (Dong et al., 2022), we adopt both class-wise and instance-wise contrastive learning to utilize the pseudo-labels and the augmented instance pairs for robust representation learning, as shown in Fig.1(b). The class-wise contrastive learning uses pseudo-labels as the supervision to pull the representations from the same cluster together and push away different clusters. While the instance-wise contrastive learning disperses different instances apart, which is supposed to separate the overlapped clusters.
Next, we provide the details of the robust representation learning module. We utilize contextual augmenter (Kobayashi, 2018; Ma, 2019) to generate augmented pairs of the original texts as X(1) and X(2). Like the cluster assignment step in the pseudo-labels generation module, we can obtain the representations of augmented pairs X(1) and X(2) as E(1) ∈ RN×D1 and E(2) ∈ RN×D1 , respectively. We can obtain the predictions of them as P (1) ∈ RN×C and P (2) ∈ RN×C , respectively. We use the fully connected layers as the projecting network Gz to map the representations to the space where instance-wise contrastive loss is applied, i.e., Gz(E(1)) = Z(1) ∈ RN×D2 and Gz(E
(2)) = Z(2) ∈ RN×D2 , where D2 is the dimension of the projected representations. The encoding network and the clustering network share weights with the pseudo-label generation module.
The class-wise contrastive learning enforces consistency between cluster predictions of positive
pairs. Specifically, the two augmentations from the same original text are regarded as a positive pair and the contrastive task is defined on pairs of augmented texts. Moreover, the pseudo-label of an original text is considered as the target of corresponding two augmented texts. We use the augmented texts with the targets as supervised data for cross-entropy minimization to achieve the consistency. The class-wise contrastive loss is defined as below:
LC = 1 N ⟨Q,− logP (1)⟩+ 1 N ⟨Q,− logP (2)⟩.
(4) The instance-wise contrastive learning enforces consistency between projected representations of positive pairs while maximizing the distance between negative pairs. Specifically, for a batch, there are 2N augmented texts, their projected representations are Z = [Z(1),Z(2)]T , given a positive pair with two texts which are augmented from the same original text, the other 2(N − 1) augmented texts are treated as negative samples. The loss for a positive pair (i, j) is defined as:
l(i, j) = − log exp(sim(Zi,Zj)/τ)∑2N k=1 1k ̸=iexp(sim(Zi,Zk)/τ) ,
(5) where sim(u,v) denotes cosine similarity between u and v, τ denotes the temperature parameter, and 1 is an indicator. The instance-wise contrastive loss is computed across all positive pairs in a batch, including both (i, j) and (j, i). That is,
LI = 1
2N
N∑
i=1
(l(i, 2i) + l(2i, i)). (6)
By combining the pseudo-supervised class-wise contrastive learning and the instance-wise contrastive learning, we can obtain robust representations and correctly separated clusters.

heading: 3.3.1 Putting Together
text: The total loss of RSTC could be obtained by combining the pseudo-supervised class-wise contrastive loss and the instance-wise contrastive loss. That is, the loss of RSTC is given as:
L = LC + λILI , (7)
where λI is a hyper-parameter to balance the two losses. By doing this, RSTC not only provides robustness to the imbalance in data, but also improve robustness against the noise in data.
The whole model with two modules forms a closed loop and self evolution, which indicates that the learned representations (more robust) and cluster predictions (more accurate) elevate each other progressively, as more reliable pseudo-labels are discovered during the iterations. Specifically, we firstly initialize the pseudo-labels Q by performing k-means on text representations. Next, we train the robust representation learning module by batch with the supervision of pseudo-labels. Meanwhile, we update Q throughout the whole training process in a logarithmic distribution, following (Asano et al., 2020). Finally, we can obtain the cluster assignments by the column index of the largest entry in each row of P . The training stops if the change of cluster assignments between two consecutive updates for P is less than a threshold δ or the maximum number of iterations is reached.

heading: 4 Experiment
text: In this section, we conduct experiments on several real-world datasets to answer the following questions: (1) RQ1: How does our approach perform compared with the state-of-the-art short text clustering methods? (2) RQ2: How do the SAOT, and the two contrastive losses contribute to the performance improvement? (3) RQ3: How does the performance of RSTC vary with different values of the hyper-parameters?

heading: 4.1 Datasets
text: We conduct extensive experiments on eight popularly used real-world datasets, i.e., AgNews, StackOverflow, Biomedical, SearchSnippets, GoogleNews-TS, GoogleNews-T, GoogleNewsS and Tweet. Among them, AgNews, StackOverflow and Biomedical are balanced datasets, SearchSnippets is a light imbalanced dataset, GoogleNews, GoogleNews-T, GoogleNews-S and Tweet are heavy imbalanced datasets. Following (Zhang et al., 2021), we take unpreprocessed data as input to demonstrate that our model is robust to noise, for a fair comparison. More details about the datasets are shown in Appendix C.1.

heading: 4.2 Experiment Settings
text: We build our model with PyTorch (Paszke et al., 2019) and train it using the Adam optimizer (Kingma and Ba, 2015). We study the effect of hyper-parameters ϵ1 and ϵ2 on SAOT by varying them in {0.05, 0.1, 0.2, 0.5} and
{0, 0.001, 0.01, 0.1, 1}, respectively. Besides, we study the effect of the hyper-parameter λI by varying it in {0, 1, 5, 10, 20, 50, 100}. The more details are provided in Appendix C.2. Following previous work (Xu et al., 2017; Hadifar et al., 2019; Rakib et al., 2020; Zhang et al., 2021), we set the cluster numbers to the ground-truth category numbers, and we adopt Accuracy (ACC) and Normalized Mutual Information (NMI) to evaluate different approaches. The specific definitions of the evaluation methods are shown in Appendix C.3. For all the experiments, we repeat five times and report the average results.

heading: 4.3 Baselines
text: We compare our proposed approach with the following short text clustering methods. BOW (Scott and Matwin, 1998) & TF-IDF (Salton and McGill, 1983) applies k-means on the TF-IDF representations and BOW representations respectively. STC2LPI (Xu et al., 2017) first uses word2vec to train word embeddings on the in-domain corpus, and then uses a convolutional neural network to obtain the text representations where k-means is applied for clustering. Self-Train (Hadifar et al., 2019) follows (Xie et al., 2016) uses an autoencoder to get the representations, and finetunes the encoding network with the same clustering objective. The difference are that it uses the word embeddings provided by (Xu et al., 2017) with SIF (Arora et al., 2017) to enhance the pretrained word embeddings, and obtains the final cluster assignments via k-means. K-means_IC (Rakib et al., 2020) first applies k-means on the TF-IDF representations and then enhances clustering by the iterative classification algorithm. SCCL (Zhang et al., 2021) is the more recent short text clustering model which utilizes SBERT (Reimers and Gurevych, 2019) as the backbone and introduces instance-wise contrastive learning to support clustering. Besides, SCCL uses the clustering objective proposed in (Xie et al., 2016) for deep joint clustering and obtains the final cluster assignments by k-means.

heading: 4.4 Clustering Performance (RQ1)
text: Results and discussion The comparison results on eight datasets are shown in Table 1. SBERT(kmeans) denotes the pre-trained SBERT model with k-means clustering, which is the initial state of our RSTC.
From the results, we can find that: (1) Only adopting traditional text representations (BOW and
TF-IDF) cannot obtain satisfying results due to the data sparsity problem. (2) Deep learning methods (STC2-LPI and Self-Train) outperform traditional ones, indicating that the application of pre-trained word embeddings and deep neural network alleviates the sparsity problem. (3) SCCL obtains better results by introducing instance-wise contrastive learning to cope with the noises problem. However, the clustering objective of SCCL is prone to obtain degenerate solutions (Yang et al., 2017; Ji et al., 2019). Besides, it is suboptimal for extra application of k-means (Van Gansbeke et al., 2020). The degeneracy gives the representation learning wrong guidance, which degrades the final k-means clustering performance. (4) RSTC outperforms all baselines, which proves that the robust representation learning with pseudo-supervised class-wise contrastive learning and instance-wise contrastive learning can significantly improve the clustering performance. To better show the clustering degeneracy problem, we visualize how the number of predicted clusters are changing over iterations on SCCL and RSTC in Appendix C.4. From the results, we verify that SCCL has relatively serious clustering degeneracy problem while RSTC solves it. The visualization results illustrate the validity of our model.

heading: 4.5 In-depth Analysis (RQ2 and RQ3)
text: Ablation (RQ2) To study how each component of RSTC contribute to the final performance, we compare RSTC with its several variants, including RSTC-OT, RSTC-C and RSTC-I. RSTCOT adopts both the pseudo-supervised class-wise contrastive learning and instance-wise contrastive learning, while the pseudo-labels are generated by traditional OT with fixed random class distribution. RSTC-C only adopts the pseudo-supervised class-wise contrastive learning, the pseudo-labels are generated by SAOT. RSTC-I only adopts the instance-wise contrastive learning while the clustering results are obtained by k-means.
The comparison results are shown in Table 1. From it, we can observe that they all cannot achieve satisfactory results due to their limitations. Specifically, (1) RSTC-OT will be guided by the mismatched distribution constraint to generate unreliable pseudo-labels. (2) RSTC-C is good at aggregating instances, but it has difficulties to address the situation when different categories are overlapped with each other in the representation space at the beginning of the learning progress, which may lead to a false division. (3) RSTCI is good at dispersing different instances, but it has limited ability to aggregate instances which
may lead to the unclear boundaries between clusters. (4) RSTC achieves the best performance with the combination of pseudo-supervised class-wise contrastive learning and instance-wise contrastive learning while the pseudo-labels are generated by SAOT. Overall, the above ablation study demonstrates that our proposed SAOT and robust representation learning are effective in solving the short text clustering problem.
Visualization To further show the functions and the limitations of the pseudo-supervised class-wise contrastive learning and the instance-wise contrastive learning, we visualize the representations using t-SNE (Van der Maaten and Hinton, 2008) for SBERT (initial state), RSTC-C, RSTC-I and RSTC. The results on SearchSnippets are shown in Fig.2(a)-(d). From it, we can see that: (1) SBERT (initial state) has no boundaries between classes, and the points from different clusters have significant overlap. (2) Although RSTC-C groups the representations to exact eight clusters, a large proportion of points are clustered by mistake. (3) RSTC-I disperses the overlapped categories to some extent, but the points are not clustered. (4) With the combination of RSTC-C and RSTC-I, RSTC obtains best text representations with small intra-cluster distance, large inter-cluster distance while only a small amount of points are clustered wrongly. The reasons for these phenomenons are the same as previous results analyzed in Ablation.
Effect of hyper-parameter (RQ3) We now study the effects of hyper-parameters on model
performance, including ϵ1, ϵ2 and λI . We first study the effects of ϵ1 and ϵ2 by varying them in {0.05, 0.1, 0.2, 0.5} and {0, 0.001, 0.01, 0.1, 1}, respectively. The results are reported in Fig. 3(a) and Fig. 3(b). Fig. 3(a) shows that the accuracy are not sensitive to ϵ1. Fig. 3(b) shows that choosing the proper hyper-parameters for different imbalance levels of datasets is important, especially on the heavy imbalanced dataset GoogleNews-T. Empirically, we choose ϵ1 = 0.1 on all datasets, ϵ2 = 0.1 on the balanced datasets, ϵ2 = 0.01 on the light imbalanced datasets, and ϵ2 = 0.001 on the heavy imbalanced datasets. Then we perform experiments by varying λI in {0, 1, 5, 10, 20, 50, 100}. The results on three datasets are shown in Fig. 4. From them, we can see that the performance improves when λI increases, then keeps a relatively stable level after λI reaches 1 and finally decreases when λI becomes too large. We can conclude that when λI is too small, the ability of instance-wise contrastive learning cannot be fully exploited. When λI is too large, the ability of class-wise contrastive learning will be suppressed, which also reduces the clustering performance. Empirically, we choose λI = 10 for all datasets.

heading: 5 Conclusion
text: In this paper, we propose a robust short text clustering (RSTC) model, which includes pseudo-label generation module and robust representation learning module. The former generates pseudo-labels as the supervision for the latter. We innovatively propose SAOT in the pseudo-label generation mod-
ule to provide robustness against the imbalance in data. We further propose to combine classwise contrastive learning with instance-wise contrastive learning in the robust representation learning module to provide robustness against the noise in data. Extensive experiments conducted on eight real-world datasets demonstrate the superior performance of our proposed RSTC.

heading: 6 Limitations
text: Like existing short text clustering methods, we assume the real cluster number is known. In the future, we would like to explore a short text clustering method with an unknown number of clusters. Moreover, the time complexity of self-adaptive optimal transport is O(n2), we are going to seek a new computation to reduce the complexity.

heading: Acknowledgements
text: This work was supported in part by the Leading Expert of “Ten Thousands Talent Program” of Zhejiang Province (No.2021R52001) and National Natural Science Foundation of China (No.72192823).

heading: A Different Class Distribution Estimation Methods
text: We have tried three class distribution estimation methods, including: (1) M1(ours): The method proposed in our paper with the penalty function Ψ(b = − log(b)− log(1− b)), and b can be updated during the process of solving the OT problem. (2) M2: The method proposed in (Wang et al., 2022) holds no assumption on b, and b can be updated by the model predicted results with moving-average mechanism, that is, b = µb̂+(1−µ)γ, where µ is the moving-average parameter, b̂ is the last updated b and γj = 1N ∑N i=1 1(j = argmaxPi). (3) M3: This method replaces the penalty function in our method with the common entropy regularization Ψ(b) = KL(b ∥ b̂), where b̂ is the last updated b, and the current b can be updated the same way our method does. Note that, the parameters of M2 are following (Wang et al., 2022), the parameters of M3 are the same as M1(ours).
For comprehensive comparison, we conduct the experiments on one imbalanced dataset GoogleNews-T and one balanced dataset StackOverflow with randomly initialized b for visualizing how the accuracy and the number of predicted clusters are changing over iterations. Moreover, except the update of b, everything else about the experiments is the same for three methods. The results are shown in Fig.5(a)-(d). From them, we can find that: (1) For the imbalanced dataset, M1(ours) achieves the best accuracy and converges to the real category number, while other methods have clustering degeneracy problem. (2) For the balanced dataset, M2 achieves best accuracy more quickly while M1(ours) catches up in the end, and all methods obtain real category number. Although M3 can obtain good accuracy on the imbalanced dataset, it has the worst accuracy on the balanced dataset. In addition, although M2 achieves good accuracy on the balanced dataset, it has the worst accuracy on the imbalanced dataset. Only M1(ours) achieves
fairly good performance on both datasets, which indicates that our method are robust to various imbalance levels of datasets. The experiments prove the effectiveness of our class distribution estimation method.

heading: B SAOT
text: As mentioned in Section 3.2, the SAOT problem is formulated as:
min π,b ⟨π,M⟩+ ϵ1H(π) + ϵ2(Ψ(b))T1,
s.t. π1 = a,πT1 = b,π ≥ 0, bT1 = 1. (8)
where ϵ1 and ϵ2 are balance hyper-parameters, Ψ(b) = − log b − log(1 − b) is the penalty function about b. We adopt the Lagrangian multiplier algorithm to optimize the problem:
min π,b ⟨π,M⟩+ ϵ1H(π) + ϵ2(Ψ(b))T1
− fT (π1− a)− gT (πT1− b)− h(bT1− 1), (9) where f , g, and h are all Lagrangian multipliers. Taking the differentiation of Equation (9) on the variable π, we can obtain:
πij = exp( fi + gj −Mij
ϵ1 ) > 0. (10)
We first fix b, due to the fact that π1 = a and πT1 = b, we can get:
exp ( fi ϵ1 ) = ai∑C j exp( gj−Mij ϵ1 ) , (11)
exp ( gj ϵ1 ) = bj∑N i exp( fi−Mij ϵ1 ) . (12)
Then we fix f and g, and update b by:
min b
[ ϵ2(Ψ(b)) T1+ gTb− h(bT1− 1) ] . (13)
Taking the differentiation of Equation (13) on the variable b, we can obtain:
(gj − h)b2j − ((gj − h) + 2ϵ2)bj + ϵ2 = 0. (14)
It is easy to get the discriminant of Equation (14) ∆j = (gj − h)2 + 4ϵ22 > 0,
bj(h) = (gj − h+ 2ϵ2)±
√ ∆j
2(gj − h) . (15)
Note that,
bj(h) = ((gj − h) + 2ϵ2) +
√ ∆j
2(gj − h) ≥ 1. (16)
Thus, we choose the following bj(h):
bj(h) = ((gj − h) + 2ϵ2)−
√ ∆j
2(gj − h) . (17)
Taking Equation (17) back to the original constraint bT1 = 1, the formula is defined as below:
(b(h))T1− 1 = 0, (18)
where h is the root of Equation (18), and we can use Newton’s method to work out it. Specifically, we first define that f(h) = (b(h))T1 − 1, then h can be updated by:
h← h− f(h) f ′(h) , (19)
where the iteration number is set to 10. Then we can obtain b by Equation (17). In short, through iteratively updating Equation (11), (12), (19), and (17), we can obtain the transport matrix π on Equation (10). We show the iteration optimization scheme of SAOT in Algorithm 1.
Algorithm 1 The optimization scheme of SAOT Input: The cost distance matrix: M . Output: The transport matrix: π. Procedure:
1: Initialize f and g randomly; 2: Initialize b randomly and perform normaliza-
tion so that bT1 = 1; 3: Initialize h = 1. 4: for i = 1 to T do 5: Update f in Equation (11); 6: Update g in Equation (12); 7: Update b in Equation (17) with the constraint bT1 = 1. 8: end for 9: Calculate π in Equation (10).

heading: C Experiment
text: C.1 Datasets
We conduct extensive experiments on eight popularly used real-world datasets. The details of each dataset are as follows.
AgNews (Rakib et al., 2020) is a subset of AG’s news corpus collected by (Zhang et al., 2015) which consists of 8,000 news titles in 4 topic categories. StackOverflow (Xu et al., 2017) consists of 20,000 question titles associated with 20 different tags, which is randomly selected from the challenge data published in Kaggle.com1. Biomedical (Xu et al., 2017) is composed of 20,000 paper titles from 20 different topics and it is selected from the challenge data published in BioASQ’s official website2. SearchSnippets (Phan et al., 2008) contains 12,340 snippets from 8 different classes, which is selected from the results of web search transaction. GoogleNews (Yin and Wang, 2016) consists of the titles and snippets of 11,109 news articles about 152 events (Yin and Wang, 2014) which is divided into three datasets: the full dataset is GoogleNewsTS, while GoogleNews-T only contains titles and GoogleNews-S only has snippets. Tweet (Yin and Wang, 2016) consists of 2,472 tweets related to 89 queries and the original data is from the 2011 and 2012 microblog track at the Text REtrieval Conference3. The detailed statistics of these datasets are shown in Table 2.
1https://www.kaggle.com/c/predict-closed-questions-onstack-overflow/
2http://participants-area.bioasq.org/ 3https://trec.nist.gov/data/microblog.html
C.2 Experiment Settings
We choose distilbert-base-nli-stsb-mean-tokens in Sentence Transformer library (Reimers and Gurevych, 2019) to encode the text, and the maximum input length is set to 32. The learning rate is set to 5×10−6 for optimizing the encoding network, and 5 × 10−4 for optimizing both the projecting network and clustering network. The dimensions of the text representations and the projected representations are set to D1 = 768 and D2 = 128, respectively. The batch size is set to N = 200. The temperature parameter is set to τ = 1. The threshold δ is set to 0.01. The datasets specific tuning is avoided as much as possible. For BOW and TF-IDF, we achieved the code with scikit-learn (Pedregosa et al., 2011). For all the other baselines, i.e., STC2-LPI4, Self-Train 5, K-means_IC6, and SCCL7 (MIT-0 license), we used their released code. Besides, we substitute the accuracy evaluation code of K-means_IC with the evaluation method described in our paper.
In addition, as STC2-LPI and Self-Train use the word embeddings pre-trained with in-domain corpus, and there are only three datasets’ pre-trained word embeddings provided, therefore we do not report the results of other five datasets for them.
C.3 Evaluation Metrics
We report two widely used evaluation metrics of text clustering, i.e., accuracy (ACC) and normalized mutual information (NMI), following (Xu et al., 2017; Hadifar et al., 2019; Zhang et al., 2021).
4https://github.com/jacoxu/STC2 5https://github.com/hadifar/stc_clustering 6https://github.com/rashadulrakib/short-text-clustering-
enhancement 7https://github.com/amazon-science/sccl
Accuracy is defined as:
ACC =
∑N i=1 1yi=map(ŷi)
N , (20)
where yi and ŷi are the ground truth label and the predicted label for a given text xi respectively, map() maps each predicted label to the corresponding target label by Hungarian algorithm(Papadimitriou and Steiglitz, 1998). Normalized mutual information is defined as:
NMI(Y, Ŷ ) = I(Y, Ŷ )√ H(Y )H(Ŷ ) , (21)
where Y and Ŷ are the ground truth labels and the predicted labels respectively, I() is the mutual information, and H() is the entropy.
C.4 Visualization To better show the clustering degeneracy problem, we visualize how the number of predicted clusters (we call it clusters later) are changing over iterations on SCCL and RSTC. The results are shown in Fig.6. From it, we verify that SCCL has relatively serious clustering degeneracy problem while RSTC solves it to some extent. Specifically, the clusters of SCCL is much less than the real category number. Moreover, the degeneracy has a negative effect on the final k-means clustering performance because it makes the representations getting worse. Whereas the clusters of RSTC almost convergent to real category number, which assures the high accuracy of RSTC. The visualization results illustrate the validity of our model.
C.5 Computational Budget The number of parameters in our model is 68M. Our training for each dataset takes about 10-30 minutes, using a GeForce RTX 3090 GPU.
ACL 2023 Responsible NLP Checklist

heading: A For every submission:
text: 3 A1. Did you describe the limitations of your work?
Section 6
A2. Did you discuss any potential risks of your work? Not applicable. Left blank.
3 A3. Do the abstract and introduction summarize the paper’s main claims? Abstract and Section 1
7 A4. Have you used AI writing assistants when working on this paper? Left blank.
B 3 Did you use or create scientific artifacts? Left blank.
3 B1. Did you cite the creators of artifacts you used? Section 4.3, Appendix C.1 and Appendix C.2
3 B2. Did you discuss the license or terms for use and / or distribution of any artifacts? Appendix C.2
7 B3. Did you discuss if your use of existing artifact(s) was consistent with their intended use, provided that it was specified? For the artifacts you create, do you specify intended use and whether that is compatible with the original access conditions (in particular, derivatives of data accessed for research purposes should not be used outside of research contexts)? We use the datasets the same way as existing work.
7 B4. Did you discuss the steps taken to check whether the data that was collected / used contains any information that names or uniquely identifies individual people or offensive content, and the steps taken to protect / anonymize it? The datasets we use only have the text instances and their category IDs.
3 B5. Did you provide documentation of the artifacts, e.g., coverage of domains, languages, and linguistic phenomena, demographic groups represented, etc.? Appendix C.1
3 B6. Did you report relevant statistics like the number of examples, details of train / test / dev splits, etc. for the data that you used / created? Even for commonly-used benchmark datasets, include the number of examples in train / validation / test splits, as these provide necessary context for a reader to understand experimental results. For example, small differences in accuracy on large test sets may be significant, while on small test sets they may not be. Appendix C.1
C 3 Did you run computational experiments? Left blank.
3 C1. Did you report the number of parameters in the models used, the total computational budget (e.g., GPU hours), and computing infrastructure used? Appendix C.5
The Responsible NLP Checklist used at ACL 2023 is adopted from NAACL 2022, with the addition of a question on AI writing assistance.
3 C2. Did you discuss the experimental setup, including hyperparameter search and best-found hyperparameter values? Section 4.2, Section 4.5, and Appendix C.2
3 C3. Did you report descriptive statistics about your results (e.g., error bars around results, summary statistics from sets of experiments), and is it transparent whether you are reporting the max, mean, etc. or just a single run? Section 4.2 and Section 4.4
3 C4. If you used existing packages (e.g., for preprocessing, for normalization, or for evaluation), did you report the implementation, model, and parameter settings used (e.g., NLTK, Spacy, ROUGE, etc.)? Appendix C.2
D 7 Did you use human annotators (e.g., crowdworkers) or research with human participants? Left blank.
D1. Did you report the full text of instructions given to participants, including e.g., screenshots, disclaimers of any risks to participants or annotators, etc.? No response.
D2. Did you report information about how you recruited (e.g., crowdsourcing platform, students) and paid participants, and discuss if such payment is adequate given the participants’ demographic (e.g., country of residence)? No response.
D3. Did you discuss whether and how consent was obtained from people whose data you’re using/curating? For example, if you collected data via crowdsourcing, did your instructions to crowdworkers explain how the data would be used? No response.
D4. Was the data collection protocol approved (or determined exempt) by an ethics review board? No response.
D5. Did you report the basic demographic and geographic characteristics of the annotator population that is the source of the data? No response.

heading: Future Work
text: In the future, we would like to explore a short text clustering method with an unknown number of clusters. Moreover, the time complexity of self-adaptive optimal transport is O(n2), we are going to seek a new computation to reduce the complexity