Clustering-Aware Negative Sampling for Unsupervised Sentence Representation
Year: 2023

Abstract:
Contrastive learning has been widely studied in sentence representation learning. However, earlier works mainly focus on the construction of positive examples, while in-batch samples are often simply treated as negative examples. This approach overlooks the importance of selecting appropriate negative examples, potentially leading to a scarcity of hard negatives and the inclusion of false negatives. To address these issues, we propose ClusterNS (Clusteringaware Negative Sampling), a novel method that incorporates cluster information into contrastive learning for unsupervised sentence representation learning. We apply a modified Kmeans clustering algorithm to supply hard negatives and recognize in-batch false negatives during training, aiming to solve the two issues in one unified framework. Experiments on semantic textual similarity (STS) tasks demonstrate that our proposed ClusterNS compares favorably with baselines in unsupervised sentence representation learning. Our code has been made publicly available.1

heading: 1 Introduction
text: Learning sentence representation is one of the fundamental tasks in natural language processing and has been widely studied (Kiros et al., 2015; Hill et al., 2016; Cer et al., 2018; Reimers and Gurevych, 2019). Reimers and Gurevych (2019) show that sentence embeddings produced by BERT (Devlin et al., 2019) are even worse than GloVe embeddings (Pennington et al., 2014), attracting more research on sentence representation with pretrained language models (PLMs) (Devlin et al., 2019; Liu et al., 2019; Radford et al., 2019). Li et al. (2020a) and Ethayarajh (2019) further find out that PLM embeddings suffer from anisotropy, motivating more researchers to study this issue (Su et al., 2021; Gao et al., 2021). Besides, Gao et al.
∗Corresponding authors 1https://github.com/djz233/ClusterNS
(2021) show that contrastive learning (CL) is able to bring significant improvement to sentence representation. As pointed out by Wang and Isola (2020), contrastive learning improves the uniformity and alignment of embeddings, thus mitigating the anisotropy issue.
Most previous works of constrastive learning concentrate on the construction of positive examples (Kim et al., 2021; Giorgi et al., 2021; Wu et al., 2020; Yan et al., 2021; Gao et al., 2021; Wu et al., 2022) and simply treat all other in-batch samples as negatives, which is sub-optimal. We show an example in Figure 1. In this work, we view sentences having higher similarity with the anchor sample as hard negatives, which means they are difficult to distinguish from positive samples. When all the negatives are sampled uniformly, the impact of hard negatives is ignored. In addition, various negative samples share different similarity values with the anchor sample and some may be incorrectly labeled (i.e., false negatives) and pushed away in the semantic space.
Recently quite a few researchers have demonstrated that hard negatives are important for contrastive learning (Zhang et al., 2022a; Kalantidis et al., 2020; Xuan et al., 2020). However, it is not
8713
trivial to obtain enough hard negatives through sampling in the unsupervised learning setting. Admittedly, they can be obtained through retrieval (Wang et al., 2022b) or fine-grained data augmentation (Wang et al., 2022a), but the processes are usually time-consuming. Incorrectly pushing away false negatives in the semantic space is another problem in unsupervised learning scenarios, because all negatives are treated equally. In fact, in-batch negatives are quite diverse in terms of similarity values with the anchor samples. Therefore, false negatives do exist in the batches and auxiliary models may be required to identify them (Zhou et al., 2022). In sum, we view these issues as the major obstacles to further improve the performance of contrastive learning in unsupervised scenarios.
Since the issues mentioned above have a close connection with similarity, reasonable differentiation of negatives based on similarity is the key. In the meanwhile, clustering is a natural and simple way of grouping samples into various clusters without supervision. Therefore, in this paper, we propose a new negative sampling method called ClusterNS for unsupervised sentence embedding learning, which combines clustering with contrastive learning. Specifically, for each mini-batch during training, we cluster them with the K-means algorithm (Hartigan and Wong, 1979), and for each sample, we select its nearest neighboring centroid (cluster center) as the hard negative. Then we treat other sentences belonging to the same cluster as false negatives. Instead of directly taking them as positive samples, we use the Bidirectional Margin Loss to constrain them. Since continuously updating sentence embeddings and the large size of the training dataset pose efficiency challenges for the clustering, we modify the K-means clustering to make it more suitable for training unsupervised sentence representation.
Overall, our proposed negative sampling approach is simple and easy to be plugged into existing methods, boosting the performance. For example, we improve SimCSE and PromptBERT in RoBERTabase by 1.41/0.59, and in BERTlarge by 0.78/0.88 respectively. The main contributions of this paper are summarized as follows:
• We propose a novel method for unsupervised sentence representation learning, leveraging clustering to solve hard negative and false negative problems in one unified framework. • We modify K-means clustering for unsuper-
vised sentence representation, making it more efficient and achieve better results. • Experiments on STS tasks demonstrate our evident improvement to baselines and we reach 79.74 for RoBERTabase, the best result with this model.

heading: 2 Related Works

heading: 2.1 Contrastive Learning
text: Contrastive learning is a widely-used method in sentence representation learning. Early works focus on positive examples, and have raised various kinds of effective data augmentations (Giorgi et al., 2021; Wu et al., 2020; Yan et al., 2021; Gao et al., 2021). Following these works, Wu et al. (2022) improve positive construction based on Gao et al. (2021). Zhou et al. (2022) improve the uniformity of negative. Besides, Zhang et al. (2022b) modify the objective function and Chuang et al. (2022) introduce Replaced Token Detection task (Clark et al., 2020), reaching higher performance.

heading: 2.2 Negative Sampling
text: In-batch negative sampling is a common strategy in unsupervised contrastive learning, which may have limitations as we mentioned above. To fix these issues, Zhang et al. (2022a) and Kalantidis et al. (2020) synthesize hard negatives by mixing positives with in-batch negatives. Wang et al. (2022a) utilize dependency parsing to create the negation of original sentences as soft negatives. Following Jiang et al. (2022) who use different prompt templates as positive, Zeng et al. (2022) derive negatives from the negation of the templates. The two methods create negatives with fixed templates and rules, thus may introduce bias. Chuang et al. (2020) design a debiased contrastive objective that corrects the false negatives without true labels. Zhou et al. (2022) use a trained model to distinguish false negatives, which results in addition module comparing with our method.

heading: 2.3 Neural Clustering
text: Clustering methods have been extended to deep learning and used for unsupervised representation learning (Xie et al., 2016; Yang et al., 2017; Caron et al., 2018; Li et al., 2020b; Zhang et al., 2021b). Prototypical Network (Snell et al., 2017), a variety of clustering, is widely used in few-shot learning (Cui et al., 2022; Ding et al., 2020; Gao et al., 2019). Several works have combined clustering
with contrastive learning (Li et al., 2020b; Caron et al., 2020; Zhang et al., 2021b; Wang et al., 2021). Among them, Li et al. (2020b) argue that clustering encodes high-level semantics, which can augment instance-wise contrastive learning.

heading: 3 Methods

heading: 3.1 Preliminaries
text: Our clustering-based negative sampling method for unsupervised sentence representation can be easily integrated with contrastive learning approaches like SimCSE (Gao et al., 2021) or PromptBERT (Jiang et al., 2022). An illustration of ClusterNS and the original contrastive learning framework is shown in Figure 2. For a sentence xi in one mini-batch {xi}Ni=1 (N samples in each mini-batch), SimCSE uses Dropout (Srivastava et al., 2014) and PromptBERT uses different prompt-based templates to obtain its positive example x+i . Then they treat the other samples in the mini-batch as “default” negatives and apply the InfoNCE loss (Oord et al., 2018) in Eq. (1), where τ is the temperature coefficient.
Lcl = − log esim(xi,x
+ i )/τ
N∑ j=1 e sim(xi,x+j )/τ
(1)

heading: 3.2 Boosting Negative Sampling
text: Our main contribution of this work is to improve the negative sampling method with clustering. To be more specific, we combine clustering with contrastive learning in the training process, recognizing false negatives in the mini-batch and providing additional hard negatives based on the clustering result. The clustering procedure will be introduced
in Section 3.3 in detail and for the moment, we assume the samples in each mini-batch have been properly clustered.
Supposing that there are K centroids c = {ci}Ki=1 after clustering, standing for K clusters C = {Ci}Ki=1. For a sample xi in the mini-batch, we sort the clusters C = [Ci1,Ci2,...,CiK] and centroids c = [ci1,ci2,...,ciK] by their cosine similarity cos(xi, cij) with xi. In this case, ci1 and ciK are the nearest and farthest centroids to xi, respectively. Therefore, xi is the most similar to ci1 and belongs to cluster Ci1. We define the set x∗i = {x∗ij} count(Ci1) j=1 , whose elements belong to Ci1, the same cluster as xi.
Hard Negatives Zhang et al. (2022a) show that hard negatives bring stronger gradient signals, which are helpful for further training. The critical question is how to discover or even produce such negatives. In our method, the introduced centroids c can be viewed as hard negative candidates. We get rough groups in mini-batch after clustering and sorting by similarity. For the sample xi, we pick the centroid ci2 as its hard negative. The reason is that ci2 gets the highest similarity with xi among all the centroids (except for ci1 which xi belongs to) while having a different cluster.
In this way, all the samples have proper centroids as their hard negatives, and the training objective Lcl is as follows:
Lcl =− log esim(xi,x
+ i )/τ
N∑ j=1
( e sim(xi,x+j )/τ + μe sim(xi,x−j )/τ )
(2)
where x−j is the hard negative corresponding to xj , μ is the weight of the hard negative. Note that ci1 is more similar to xi compared with ci2, which is another candidate for the hard negative. We have compared the different choices in the ablation study described in Section 4.4.
False Negatives For sample xi, we aim to 1) recognize the false negatives in the mini-batch and 2) prevent them from being pushed away incorrectly in the semantic space. For the former, we treat elements in x∗i as false negatives, since they belong to the same cluster and share higher similarity with xi. For the latter, it is unreliable to directly use them as positives, since the labels are missing under the unsupervised setting. However, the different similarity between the anchor sample and others can be summarized intuitively as the following Eq. (3):
cos(xi, x − i ) ≤ cos(xi, x∗ij) ≤ cos(xi, x+i ) (3) where x∗ij ∈ x∗i . False negatives have higher similarity with the anchor than normal negatives while lower similarity than the positives. Inspired by Wang et al. (2022a), we introduce the bidirectional margin loss (BML) to model the similarity between the false negative candidates and the anchor:
Δxi = cos(xi, x ∗ i )− cos(xi, x+i ) (4)
Lbml = ReLU(Δxi+α)+ReLU(−Δxi−β) (5) BML loss aims to limit cos(xi, x∗i ) in an appropriate range by limiting Δxi in the interval [−β, −α]. Accordingly, we find the potential false negatives in the mini-batch and treat them differently. Combining Eq. (2) and Eq. (5), we obtain the final training objective function as follows:
L = Lcl + λLbml (6) where λ is a hyperparameter.

heading: 3.3 In-Batch Clustering
text: K-means clustering is the base method we use, while we need to overcome computational challenges during the training process. It is very inefficient to cluster the large training corpus. However, we need to do clustering frequently due to the continuously updating embeddings. Therefore, we design the training process with clustering in Algorithm 1. Briefly speaking, we use cosine similarity as the distance metric, cluster the mini-batch and update the centroids with momentum at each step.
Algorithm 1 Training with Clustering. Input: Model parameters: θ; Training dataset: D;
Total update steps: T ; Warm-up steps: S
1: for t = 1 to T do 2: Get the sentence embeddings {xi}Ni=1 for each mini-batch 3: if t == S then 4: Initialize centroids c with mini-batch samples heuristically 5: end if 6: if t > S then 7: Update centroids c with {xi}Ni=1 8: Provide centroids as hard negatives {x−i }Ni=1 9: Calculate Lbml for false negatives
10: end if 11: Calculate Lcl 12: Loss backward and optimize θ 13: end for
Centroids Initialization We show the initialization in line 3–5 in Algorithm 1. The clustering is not performed at the beginning, since high initial similarity of embeddings harms the performance. Instead, we start clustering a few steps after the training starts, being similar to the warm-up process. When initializing, as line 4 shows, we select K samples as initial centroids heuristically: each centroid to be selected should be the least similar to last centroid.
Clustering and Updating We now describe line 7 in detail. First, we assign each sample into the cluster whose centroid have the highest cosine similarity with the sample. After clustering finishes, we calculate a new centroid embedding for each cluster by averaging embeddings of all samples in the cluster with Eq. (7), and then update the centroid in the momentum style with Eq. (8):
x̃i = 1
Ni
∑
xj∈Ci xj (7)
ci = (1− γ)ci + γx̃i (8) where γ is the momentum hyperparameter and Ni indicates the number of elements in cluster Ci. Finally, based on the clustering results, we calculate the loss and optimize the model step by step (in line 9–12).
The method can be integrated with other contrastive learning models, maintaining high efficiency.

heading: 4 Experiments

heading: 4.1 Evaluation Setup
text: Our experiments are conducted on 7 semantic textual similarity (STS) tasks (Agirre et al., 2012, 2013, 2014, 2015, 2016; Cer et al., 2017; Marelli et al., 2014) and the models are evaluated with the SentEval Toolkit (Conneau and Kiela, 2018). We take the Spearman’s correlation coefficient as the metric and follow the Gao et al. (2021)’s aggregation method of results.

heading: 4.2 Implementation Details
text: Our code is implemented in Pytorch and Huggingface Transformers. The experiments are run on a single 32G Nvidia Tesla V100 GPU or four 24G Nvidia RTX3090 GPUs. Our models are based on SimCSE (Gao et al., 2021) and PromptBERT
(Jiang et al., 2022), and named as Non-Prompt ClusterNS and Prompt-based ClusterNS, respectively. We use BERT (Devlin et al., 2019) and RoBERTa (Liu et al., 2019) as pre-trained language models for evaluation, with training for 1 epoch and evaluating each 125 steps on the STS-B development set. We also apply early stopping to avoid overfitting. Hyperparameter settings and more training details are listed in Appendix A.

heading: 4.3 Main Results
text: We present the experiment results in Table 1. We compare with four types of models totally, 1) vanilla embeddings of Glove, BERT and RoBERTa models, we report their results provided by Gao et al. (2021). 2) Baseline models: BERT-flow (Li et al., 2020a), BERT-whitening (Su et al., 2021), SimCSE (Gao et al., 2021) and Prompt-
BERT (Jiang et al., 2022). 3) SimCSE-based models: MixCSE (Zhang et al., 2022a), DCLR (Zhou et al., 2022) and ESimCSE (Wu et al., 2022). 4) PromptBERT-based models: ConPVP (Zeng et al., 2022) and SNCSE (Wang et al., 2022a). We compare SimCSE-based models with Non-Prompt ClusterNS, and PromptBERT-based models with Prompt-based ClusterNS, respectively. In this way, identical representation of sentence embeddings is guaranteed for a fair comparison.
Our conclusions are as follows, comparing with two baseline models, SimCSE and PromptBERT, all ClusterNS models achieve higher performance, indicating their effectiveness and the importance of negative sampling. For non-prompt models, ClusterNS surpasses MixCSE and DCLR in BERTlarge, and for prompt-based models, ClusterNS also surpasses ConPVP and SNCSE in BERTlarge and RoBERTabase. All these models improve negative samples through sampling or construction, demonstrating our models’ strong competitiveness. At last, Prompt-based ClusterNS achieves the stateof-the-art performance of 79.74, which is the best result for models with RoBERTabase.

heading: 4.4 Ablation Study
text: Our proposed method focuses on two issues, producing hard negatives and processing false negatives. To verify the contributions, we conduct the ablation studies by removing each of the two components on test sets of the STS tasks, with Non-prompt BERT and RoBERTa models. As we mentioned in Section 3.2, we also replace hard negatives with the most similar centroids to verify our choice of hard negatives (named repl. harder negative), and replace both centroids for hard and false negatives with random clusters to verify our choice of cluster centroids (named repl. random clusters). The results are in Table 2.
We observe from Table 2 that removing either component or replacing any part of models lead to
inferior performance: 1) Providing hard negatives yields more improvement, since we create high similarity sample leveraging clustering. 2) Processing false negatives solely (without hard negatives) even further harm the performance, indicating that providing virtual hard negatives is much easier than distinguishing real false negatives. 3) Replacing hard negative with most similar centroids also degrades the performance. Since they belong to the identical cluster, the candidate hard negatives could be actually positive samples. And 4) random clusters are also worse, indicating that the selection of clusters does matter. We discuss more hyperparameter settings in Appendix E.

heading: 5 Analysis
text: To obtain more insights about how clustering helps the training process, we visualize the variation of diverse sentence pairs similarity during training after clustering initialization in the Non-Prompt ClusterNS-RoBERTabase model, and analyze the results in detail.

heading: 5.1 In-Batch Similarity
text: We visualize the average similarity of positive, inbatch negative and hard negative sentence pairs in Figure 3. We observe that similarity of in-batch negative drops rapidly as training progresses, indicating that in-batch negatives are difficult to provide gradient signal. The hard negatives provided by our method maintain higher similarity, which properly handles the issue. Also notice that the similarity of hard negatives is still much smaller than positive pairs, which avoids confusing the model.

heading: 5.2 Clustering Similarity
text: Furthermore, we also visualize the similarity related to clustering. In Figure 4, we show the average similarity of sample-nearest centroid pairs,
sample-hard negative pairs (second nearest centroids, same as hard negative sentence pairs in Figure 3), inter-centroid pairs, intra-cluster member pairs (same as false negative pairs) and in-batch negative pairs. First, similarity of inter-centroid pairs decreases during training, demonstrating that clusters representing diverse semantics slowly scatter. Second, false negative pairs get much higher similarity than in-batch negatives, which indicates the importance of recognizing them and the necessity of treating them differently. At last, samplenearest centroid pairs and sample-hard negative pairs maintain high similarity, demonstrating the stability of clustering during the training process.
To answer the question what is a good hard negative, we experiment with different similarity levels. We define a symbol σ, the average similarity threshold of in-batch sentence pairs when the centroids initialize. Since the similarity of hard negative pairs depends on σ, we adjust the similarity level with various σ settings.
We show the results in Figure 5 and Table 3. As we set the threshold σ smaller, clustering begins later and hard negatives gets larger similarity (with the anchor sample), meaning that starting clustering too early leads to less optimal hard negative candidates. The best performance is achieved at σ = 0.4, the middle similarity level, verifying the finding in our ablation study, i.e., hard negatives are not the most similar samples.
In Figure 4, similarity of false negative pairs is much smaller comparing with positive pairs, which shows the distinction between positive and false negative samples. False negatives are usually regarded as positive samples in supervised learning, while it is difficult to recognize precisely in the unsupervised setting. We argue that false negatives
retrieved by our methods share similar topics with the anchors, leading to higher similarity than “normal” negatives and lower similarity than positives. We use Eq. (5) to constrain false negatives based on this hypothesis. We also do case studies and experiments to approve it in Appendix C.
To verify our choice of the BML loss, we implement experiments on different processing strategies of false negatives. We compare BML loss with two common strategies: use false negatives as positives and mask all the false negatives. Results in Table 4 demonstrate the superiority of the BML loss.

heading: 6 Clustering Evaluation
text: We also evaluate the quality of sentence embedding through clustering. We first use the DBpedia dataset (Brümmer et al., 2016), an ontology classification dataset extracted from Wikipedia and consists of 14 classes in total. We implement K-means clustering (K=14) on the sentence embeddings of
DBpedia, and take the adjusted mutual information (AMI) score as the evaluation metric following Li et al. (2020b). The results in Table 5 show that both sentence embedding models improve the AMI score, indicating that the cluster performance is positively correlated with the quality of sentence embeddings. ClusterNS achieves a higher AMI score than SimCSE, verifying its effectiveness.
Furthermore, we follow Zhang et al. (2021a) to conduct a more comprehensive evaluation of the short text clustering on 8 datasets 2, including AgNews (AG) (Zhang and LeCun, 2015), Biomedical (Bio) (Xu et al., 2017), SearchSnippets (SS) (Phan et al., 2008), StackOverflow (SO) (Xu et al., 2017), GoogleNews (G-T, G-S, G-TS) (Yin and Wang, 2016) and Tweet (Yin and Wang, 2016). We perform K-means clustering on the sentence embeddings and take the clustering accuracy as the evaluation metric. Results are shown in Table 6. Our ClusterNS models achieve higher performance than SimCSE in both two models, with an overall improvement of 4.34 in BERTbase. Both main experiments and two clustering evaluations show the improvement of our method to the baseline, and verify the effectiveness of improved negative sampling. More details about evaluation metrics are shown in Appendix D.

heading: 7 Alignment and Uniformity
text: To investigate how ClusterNS improves the sentence embedding, we conduct further analyses on two widely used metrics in contrastive learning proposed by Wang and Isola (2020), alignment and uniformity. Alignment measures the expected distance between the embeddings of positive pairs:
Lalign E (x,x+)∼ppos ‖f(x)− f(x+)‖2 (9)
And uniformity measures the expected distance between the embeddings of all sentence pairs:
Luniform log E (x,y)∼pdata
e−2‖f(x)−f(y)‖ 2 (10)
2https://github.com/rashadulrakib/short-text-clusteringenhancement
Both metrics are better when the numbers are lower. We use the STS-B dataset to calculate the alignment and uniformity, and consider the sentence pairs with score higher than 4 as positive pairs. We show the alignment and uniformity of different models in Figure 6, along with the average STS test results. We observe that ClusterNS strikes a balance between alignment and uniformity, improving the weaker metric at the expense of the stronger one to reach a better balance. For the nonprompt models, SimCSE has great uniformity but weaker alignment compared to vanilla BERT and RoBERTa. ClusterNS optimizes the alignment. On the other hand, Prompt-based ClusterNS optimizes the uniformity since PromptRoBERTa performs the opposite of SimCSE. Besides, RoBERTa may suffer server anisotropy than BERT, meaning that sentence embeddings are squeezed in a more crowded part of the semantic space. Therefore, RoBERTa and PromptRoBERTa-untuned have extreme low value of alignment, but poor uniformity.

heading: 8 Conclusion
text: In this paper, we propose ClusterNS, a novel approach that focuses on improving the negative sampling for contrastive learning in unsupervised sentence representation learning. We integrate clustering into the training process and use the clustering results to generate additional hard negatives and identify false negatives for each sample. We also use a bidirectional margin loss to constrain the false negatives. Our experiments on STS tasks show improvements over baseline models and demonstrate the effectiveness of ClusterNS. Through this work,
we demonstrate that it is valuable to pay more attention to negative sampling when applying contrastive learning for sentence representation.

heading: Acknowledgements
text: We appreciate the anonymous reviewers for their valuable comments. We thank Zhaoyang Wang for his support. This work was supported by the National Natural Science Foundation of China (No. 62176270), the Guangdong Basic and Applied Basic Research Foundation (No. 2023A1515012832), and the Program for Guangdong Introducing Innovative and Entrepreneurial Teams (No. 2017ZT07X355).
Limitations
Our work has two limitations. First, we update the cluster centroids at each step during training, which requires a large mini-batch to maintain clustering accuracy and consumes more GPU memory. Second, our method still may not identify false negatives accurately, as we use the training model for coarse-grained clustering rather than a well-trained model. We leave the improvement of memory consumption and further improving false negative discrimination for the future.
Ethics Statement
All datasets used in our work are from public sources, which do not consist private information. We strictly followed the data usage policy. Any research based on our work must sign an ethical statement and ensure that they do not infer user privacy from it.

heading: A Training Details
text: We do grid search for the hyperparameters and list the searching space below.
• Total batch size [256, 512] • Learning rate [1e-5, 3e-5, 5e-5] • Hard negative weight μ [1.0] • Number of cluster K [96, 128, 256] • Momentum γ [1e-3, 5e-4, 1e-4] • Similarity threshold σ [0.2 0.3 0.4 0.5 0.6] • Weight of Lbml [1e-2, 1e-3, 1e-4, 1e-5] • Upper Bound of Lbml α [0, 0.05, 0.1, 0.15,
0.2, 0.25] • Lower Bound of Lbml β [0.3, 0.4, 0.5, 0.6]
Our method has two main improvements on hard negative and false negative, respectively. We apply both improvements to most of the models except one of them. We list the information in detail in Table 7. More hyperparameter experiments are discussed in Appendix E.

heading: B Transfer Tasks
text: Following previous works, we also evaluate our models on seven transfer tasks: MR (Pang and Lee, 2005), CR (Hu and Liu, 2004), SUBJ (Pang and Lee, 2004), MPQA (Wiebe et al., 2005), SST-2
(Socher et al., 2013), TREC (Voorhees and Tice, 2000) and MRPC (Dolan and Brockett, 2005). We evaluate with Non-Prompt ClusterNS models, and use the default configurations in SentEval Toolkit. Results are showed in Table 8. Most of our models achieve higher performance than SimCSE and the auxiliary MLM task also works for our methods.

heading: C False Negative Details
text: We show the case study in Table 10. As we mentioned in Section 5, our method is able to cluster sentences with similar topics such as religion and music, demonstrating that clustering captures higher-level semantics. However, intra-cluster sentences do not necessarily carry the same meaning and thus they are not suitable to be used as positives directly.
We also show the variation tendency of the false negative rate in Figure 7, which is equivalent to the sample percentage of clusters having more than two elements (i.e., the intra-cluster members are the false negatives of each other). We observe that the false negative rate maintains a high percentage in the whole training process, which verifies the necessity to specific handling the false negatives.

heading: D Clustering Evaluation Details
text: We use adjusted mutual information (AMI) score or clustering accuracy to evaluate clustering performance. AMI score measures the agreement between ground truth labels and clustering results. Two identical label assignments get the AMI score of 1, and two random label assignments are expected to get AMI score of 0. Clustering accuracy measures the clustering agreement with accuracy metric, which need to map clustering results to
ground truth labels with Hungary algorithm in advance.

heading: E Supplement Experiments
text: E.1 Batch Size and Cluster Number
We use large batch sizes and the cluster number K for our models in the main experiments. To show the necessity, we implement the quantitative analysis to compare with small batch sizes and cluster numbers, and show the results in Figure 8 and Figure 9. Both experiments of small batch sizes and cluster numbers perform worse. We attribute the performance degeneration to three factors: 1) Contrastive learning requires large batch sizes in general; 2) Smaller cluster numbers lead to more coarse-grained clusters, weakening the clustering performance; And 3) small batch sizes further restrain the number of clusters.
E.2 Centroids Initialization
We initialize the cluster centroids locally as mentioned in Section 3.3. While some other works adopt global initialization (Li et al., 2020b), they take the embeddings of whole dataset to initialize
the centroids. We compare the two strategies by implementing the global initialization version of ClusterNS (named global ClusterNS). We show the test results in Table 9, and the variation of clustering similarity in Figure 10. Overall, global ClusterNS does not improve the performance. We obverse that inter-centroid pairs have extreme high similarity, meaning that clusters do not scatter, and the similarity of hard negative pairs is very low, which means hard negatives are not able to provide strong gradient signal.
ACL 2023 Responsible NLP Checklist

heading: A For every submission:

heading: 3 A1. Did you describe the limitations of your work?
text: After the Conclusion section.

heading: 3 A2. Did you discuss any potential risks of your work?
text: We discuss them in Limitation.

heading: 3 A3. Do the abstract and introduction summarize the paper’s main claims?
text: Abstract and Introduction (Section 1).
7 A4. Have you used AI writing assistants when working on this paper? Left blank.
B 3 Did you use or create scientific artifacts? We describe them in Section 4.1 (Evaluation Setup) and Section 4.2 (Implementation Details).

heading: 3 B1. Did you cite the creators of artifacts you used?
text: We describe them in Section 4.1 (Evaluation Setup) and Section 4.2 (Implementation Details).
B2. Did you discuss the license or terms for use and / or distribution of any artifacts? Not applicable. Left blank.
7 B3. Did you discuss if your use of existing artifact(s) was consistent with their intended use, provided that it was specified? For the artifacts you create, do you specify intended use and whether that is compatible with the original access conditions (in particular, derivatives of data accessed for research purposes should not be used outside of research contexts)? We follow the same processing as previous works, and the datasets and code we used are compatible with the original conditions.
B4. Did you discuss the steps taken to check whether the data that was collected / used contains any information that names or uniquely identifies individual people or offensive content, and the steps taken to protect / anonymize it? Not applicable. Left blank.
B5. Did you provide documentation of the artifacts, e.g., coverage of domains, languages, and linguistic phenomena, demographic groups represented, etc.? Not applicable. Left blank.
7 B6. Did you report relevant statistics like the number of examples, details of train / test / dev splits, etc. for the data that you used / created? Even for commonly-used benchmark datasets, include the number of examples in train / validation / test splits, as these provide necessary context for a reader to understand experimental results. For example, small differences in accuracy on large test sets may be significant, while on small test sets they may not be. All the datasets we used are public and have standard data splits. We follow the same processing as previous works, which also do not mention the relevant statistics.
C 3 Did you run computational experiments? Experiments (Section 4).
7 C1. Did you report the number of parameters in the models used, the total computational budget (e.g., GPU hours), and computing infrastructure used? The models (BERT, RoBERTa) we employ in the paper are well-known.
The Responsible NLP Checklist used at ACL 2023 is adopted from NAACL 2022, with the addition of a question on AI writing assistance.

heading: 3 C2. Did you discuss the experimental setup, including hyperparameter search and best-found
text: hyperparameter values? We discuss the experimental setup in Evaluation Setup (Section 4.1) and Implementation Details (Section 4.2) and discuss the hyperparameter in Training Details (Appendix A) and Hyperparameters Choice
7 C3. Did you report descriptive statistics about your results (e.g., error bars around results, summary statistics from sets of experiments), and is it transparent whether you are reporting the max, mean, etc. or just a single run? We report the main results in a single run and repeat 5 random seeds experiments for our models later. Our standard deviation is about 0.1 0.2 and improvement is also significant statistically.

heading: 3 C4. If you used existing packages (e.g., for preprocessing, for normalization, or for evaluation), did
text: you report the implementation, model, and parameter settings used (e.g., NLTK, Spacy, ROUGE, etc.)? We discuss it in Implementation Details (Section 4.2)
D 7 Did you use human annotators (e.g., crowdworkers) or research with human participants? Left blank.
D1. Did you report the full text of instructions given to participants, including e.g., screenshots, disclaimers of any risks to participants or annotators, etc.? No response.
D2. Did you report information about how you recruited (e.g., crowdsourcing platform, students) and paid participants, and discuss if such payment is adequate given the participants’ demographic (e.g., country of residence)? No response.
D3. Did you discuss whether and how consent was obtained from people whose data you’re using/curating? For example, if you collected data via crowdsourcing, did your instructions to crowdworkers explain how the data would be used? No response.
D4. Was the data collection protocol approved (or determined exempt) by an ethics review board? No response.
D5. Did you report the basic demographic and geographic characteristics of the annotator population that is the source of the data? No response.

heading: Future Work
text: We leave the improvement of memory consumption and further improving false negative discrimination for the future