RankCSE : Unsupervised Sentence Representation Learning via Learning to Rank
Year: 2023

Abstract:
Unsupervised sentence representation learning is one of the fundamental problems in natural language processing with various downstream applications. Recently, contrastive learning has been widely adopted which derives highquality sentence representations by pulling similar semantics closer and pushing dissimilar ones away. However, these methods fail to capture the fine-grained ranking information among the sentences, where each sentence is only treated as either positive or negative. In many real-world scenarios, one needs to distinguish and rank the sentences based on their similarities to a query sentence, e.g., very relevant, moderate relevant, less relevant, irrelevant, etc. In this paper, we propose a novel approach, RankCSE, for unsupervised sentence representation learning, which incorporates ranking consistency and ranking distillation with contrastive learning into a unified framework. In particular, we learn semantically discriminative sentence representations by simultaneously ensuring ranking consistency between two representations with different dropout masks, and distilling listwise ranking knowledge from the teacher. An extensive set of experiments are conducted on both semantic textual similarity (STS) and transfer (TR) tasks. Experimental results demonstrate the superior performance of our approach over several state-of-the-art baselines.

heading: 1 Introduction
text: Sentence representation learning refers to the task of encoding sentences into fixed-dimensional em-
∗ Work done during internship at Meituan. † Corresponding authors: Dongyan Zhao
(zhaody@pku.edu.cn) and Rui Yan (ruiyan@ruc.edu.cn).
beddings. The sentence embeddings can be leveraged in various applications, including information retrieval (Le and Mikolov, 2014), text clustering (Ma et al., 2016) and semantic textual similarity comparison (Agirre et al., 2012). With the recent success of pre-trained language models (PLMs), such as BERT/RoBERTa (Devlin et al., 2019; Liu et al., 2019), a straightforward way to generate sentence representations is to directly use the [CLS] token embedding or the average token embeddings from the last layer of PLMs (Reimers and Gurevych, 2019). However, several studies (Ethayarajh, 2019; Li et al., 2020) have found that the native sentence representations derived by PLMs
13785
occupy a narrow cone in the vector space, and thus severely limits their representation capabilities, which is known as the anisotropy problem.
Supervised methods like SBERT (Reimers and Gurevych, 2019) usually generate better sentence representations, but require fine-tuning on a large amount of labeled data. Recent unsupervised models (Carlsson et al., 2021; Zhang et al., 2021; Giorgi et al., 2021) adopt contrastive learning framework without any labels, which pulls similar semantics closer and pushes dissimilar ones away. These methods usually design different augmentation algorithms for generating positive examples, such as back-translation (Zhang et al., 2021), dropout (Gao et al., 2021) and token shuffling or cutoff (Yan et al., 2021). In-batch negatives are further combined with the positives. Despite achieving promising results, they treat positives/negatives equally without capturing the fine-grained semantic ranking information, resulting in less effective sentence representations which fail to distinguish between very similar and less similar sentences. For example, Table 1 shows two examples of a query sentence and several target sentences from semantic textual similarity datasets. It is clear that the similarity scores produced by the contrastive learning method SimCSE are not optimized, where the sentence rankings are not preserved in the learned representations. On the other hand, our RankCSE generates effective sentence representations with consistent rankings to the ground-truth labels. Figure 1 further shows the advantage of RankCSE in terms of two ranking metrics. The fine-grained ranking information is crucial in various real-world
applications including search and recommendation. The ability to differentiate between subtle distinctions in sentence meaning can help these systems provide more relevant and accurate results, leading to a better user experience. Therefore, it is an important problem to learn ranking preserving sentence representations from unsupervised data.
To obtain semantically discriminative sentence representations, we propose a novel approach, RankCSE, which incorporates ranking consistency and ranking distillation with contrastive learning into a unified framework. Specifically, our model ensures ranking consistency between two representations with different dropout masks and minimizes the Jensen-Shannon (JS) divergence as the learning objective. In the meanwhile, our model also distills listwise ranking knowledge from the teacher model to the learned sentence representations. In our work, we explore two listwise ranking methods, ListNet (Cao et al., 2007) and ListMLE (Xia et al., 2008), and utilize the pre-trained SimCSE (Gao et al., 2021) models with coarse-grained semantic ranking information as the teachers to provide pseudo ranking labels. Our RankCSE is able to generalize fine-grained ranking information from the weak ranking knowledge learned by SimCSE. We conduct an extensive set of experiments on semantic textual similarity (STS) and transfer (TR) tasks. Experimental results show that RankCSE outperforms the existing state-of-the-art baselines.

heading: 2 Related Work
text: Unsupervised Sentence Representation Learning Early works typically augment the idea of word2vec (Mikolov et al., 2013) to learn sentence representations, including Skip-Thought (Kiros et al., 2015), FastSent (Hill et al., 2016) and QuickThought (Logeswaran and Lee, 2018). With the great success of PLMs, various attempts focus on generating sentence representations by leveraging the embedding of [CLS] token or applying mean pooling on the last layer of BERT (Reimers and Gurevych, 2019). However, Ethayarajh (2019) identifies the anisotropy problem in language representations, which means the native learned embeddings from PLMs occupy a narrow cone in the vector space. BERT-flow (Li et al., 2020) and BERTwhitening (Su et al., 2021) propose to resolve the anisotropy problem through post-processing.
Recently, contrastive learning has been adopted to learn sentence representations by designing dif-
ferent augmentation methods (Zhang et al., 2020; Carlsson et al., 2021; Giorgi et al., 2021; Yan et al., 2021; Kim et al., 2021; Gao et al., 2021). A typical example SimCSE uses dropout as a data augmentation strategy and is also the foundation of many following works. ArcCSE (Zhang et al., 2022) enhances the pairwise discriminative power and models the entailment relation among triplet sentences. DCLR (Zhou et al., 2022) alleviates the influence of improper negatives. DiffCSE (Chuang et al., 2022) introduces equivariant contrastive learning to SimCSE. PCL (Wu et al., 2022a) proposes contrastive representation learning with diverse augmentation strategies for an inherent anti-bias ability. InfoCSE (Wu et al., 2022b) learns sentence representations with the ability to reconstruct the original sentence fragments. Generative learning techniques (Wang et al., 2021; Wu and Zhao, 2022) have also been proposed to enhance the linguistic interpretability of sentence representations. Although achieving promising results, these methods fail to capture the fine-grained ranking knowledge among the sentences.
Learning to Rank Given a query example, learning to rank aims to rank a list of examples according to their similarities with the query. Learning to rank methods can be divided into three categories: pointwise (Li et al., 2007), pairwise (Burges et al., 2005, 2006) and listwise (Cao et al., 2007; Xia et al., 2008; Volkovs and Zemel, 2009; Pobrotyn and Bialobrzeski, 2021). Pointwise methods optimize the similarity between the query and each example, while pairwise approaches learn to cor-
rectly model the preference between two examples. Listwise methods directly evaluate the ranking of a list of examples based on the ground truth. In our framework, we leverage listwise ranking objectives for learning effective sentence representations, which have shown better performance compared to pointwise and pairwise methods.

heading: 3 Preliminary
text: We provide some conceptual explanations and definitions in learning to rank.
Top One Probability Given the scores of all objects S = {si}ni=1, the top one probability of an object is the probability of its being ranked at top-1: s̃i =
esi/τ∑n j=1 e sj/τ where τ is a temperature hyperpa-
rameter, usually utilized to smooth the distribution. We simply denote the formulation for calculating the top one distribution based on the scores S as: S̃τ = σ(S/τ).
Permutation Probability Let π = {π(i)}ni=1 denote a permutation of the object indexes, which represents that the π(i)-th sample is ranked i-th. The probability of a specific permutation π is given as: P (π|S, τ) = ∏ni=1 e sπ(i)/τ
∑n j=i e sπ(j)/τ .

heading: 4 Methodology

heading: 4.1 Problem Formulation
text: Our goal is to learn sentence representations such that semantic similar sentences stay close while dissimilar ones should be far away in an unsupervised manner. Specifically, We aim to find an optimal
function f that maps a sentence s ∈ ps to a ddimensional vector f(s) ∈ pe ⊆ Rd, where ps and pe denote the distributions of sentences and sentence representations, respectively. Supposing s1 and s2 are more semantic similar than s1 and s3 (s1, s2, s3 ∈ ps), a good mapping function f should satisfy that the distance between f(s1) and f(s2) is smaller than that between f(s1) and f(s3), i.e., d(f(s1), f(s2)) < d(f(s1), f(s3)), where d is the distance metric such as Euclidean distance and cosine distance. In this way, the similarities among the sentences are preserved in the learned sentence representations.
The general idea of RankCSE is to learn semantically discriminative sentence representations by capturing the ranking information among the sentences. As shown in Figure 2, our model consists of three components: (1) contrastive learning objective (Section 4.2); (2) ranking consistency loss which ensures ranking consistency between two representations with different dropout masks (Section 4.3); (3) ranking distillation loss which distills listwise ranking knowledge from the teacher (Section 4.4).

heading: 4.2 Contrastive Learning
text: Contrastive learning aims to learn effective representations by pulling similar semantics closer and pushing away dissimilar ones. SimCSE (Gao et al., 2021) creates positive examples by applying different dropout masks and takes a cross-entropy object with in-batch negatives (Chen et al., 2017). More specifically, for any sentence xi in a min-batch, we send it to the encoder f(·) twice and obtain two representations with different dropout masks f(xi), f(xi)
′. SimCSE use the InfoNCE loss (van den Oord et al., 2018) as the training objective:
LinfoNCE = − N∑
i=1
log eϕ(f(xi),f(xi) ′)/τ1 ∑N
j=1 e ϕ(f(xi),f(xj)′)/τ1
, (1)
where N is the batch size, τ1 is a temperature hyperparameter and ϕ(f(xi), f(xj)′) = f(xi)⊤f(xj)′ ∥f(xi)∥·∥f(xj)′∥ is the cosine similarity used in this work. Essentially, the contrastive learning objective is equivalent to maximizing the top one probability of the positive sample.
Although contrastive learning is effective in separating positive sentences with negative ones, it ignores the continuity modeling of the similarity. In other words, it is not effective in distinguishing highly similar sentences with moderate similar ones. To address this issue, we propose to
directly model the ranking information among the sentences, which could enhance the discrimination of semantic similarity in the learned sentence representations.

heading: 4.3 Ranking Consistency
text: The main drawback of contrastive learning is that the distinction between the in-batch negatives is not modeled, resulting in less effective sentence representations in capturing the fine-grained sentence similarity. Therefore, instead of treating the negatives equivalently, we propose to explicitly model the ranking information within the sentences by ensuring the ranking consistency between the two similarity sets (circled by the solid and dashed curves respectively in the right part of Figure 2).
Concretely, by taking a close look at the contrastive modeling in Section 4.2, there are two sets of sentence representations, f(xi) and f(xi)′, derived from different dropout masks. For each sentence xi, two lists of similarities with other sentences can be naturally obtained from the two representations, i.e., S(xi) = {ϕ(f(xi), f(xj)′)}Nj=1 and S(xi)′ = {ϕ(f(xi)′, f(xj))}Nj=1. We then enforce the ranking consistency between these two similarity lists in our modeling. Intuitively, all corresponding elements in S(xi) and S(xi)′ should have the same ranking positions.
Given two similarity lists S(xi) and S(xi)′, we can obtain their top one probability distributions S̃τ1(xi) = σ(S(xi)/τ1), S̃τ1(xi)
′ = σ(S(xi)′/τ1). The ranking consistency can be ensured by minimizing the Jensen-Shannon (JS) divergence between the two top one probability distributions:
Lconsistency = N∑
i=1
JS(Pi||Qi)
= 1
2
N∑
i=1
(KL(Pi||Pi +Qi 2 ) + KL(Qi||Pi +Qi 2 ))
= 1
2
N∑
i=1
(Pi log( 2Pi
Pi +Qi ) +Qi log( 2Qi Pi +Qi )),
(2)
where Pi and Qi represents S̃τ1(xi) and S̃τ1(xi) ′ respectively. The reason we choose JS divergence instead of Kullback-Leibler (KL) divergence is that the two distributions are symmetric rather than one side being the ground truth.

heading: 4.4 Ranking Distillation
text: Contrastive learning based methods like SimCSE learn effective sentence representations with coarsegrained semantic ranking information (shown in
Appendix F and G), which have demonstrated their effectiveness in various downstream tasks. Orthogonal to ranking consistency, we further introduce ranking distillation by distilling the ranking knowledge from pre-trained teacher models into our learned sentence representations, to generalize effective ranking information from the weak ranking knowledge learned in the teachers. More specifically, for each sentence in a min-batch, we obtain the similarity score list from the teacher model, which is then served as pseudo ranking labels in the ranking distillation. The intuitive idea is to transfer the ranking knowledge from the teacher to the student as guidance for learning ranking preserved sentence representations. In the ranking distillation, ListNet (Cao et al., 2007) and ListMLE (Xia et al., 2008) methods are utilized. Formally they are defined as:
Lrank = N∑
i=1
rank(S(xi), S T(xi)), (3)
where S(xi) and ST (xi) are the similarity score lists obtained from the student and the teacher, respectively, rank(·, ·) is the listwise method. ListNet The original ListNet minimizes the cross entropy between the permutation probability distribution and the ground truth as the training objective. However, the computations will be intractable when the number of examples n is large, since the number of permutations is n!. To reduce the computation complexity, the top one probability distribution is usually adopted as a substitute:
LListNet = − N∑
i=1
σ(ST (xi)/τ3) · log(σ(S(xi)/τ2)), (4)
where τ2 and τ3 are temperature hyperparameters.1
ListMLE Different from ListNet, ListMLE aims to maximize the likelihood of the ground truth permutation πTi which represents the sorted indexes of the similarity scores calculated by the teacher model. The objective of ListMLE is defined as:
LListMLE = − N∑
i=1
logP (πTi |S(xi), τ2). (5)
In this work, we propose to use a multi-teacher from which more listwise ranking knowledge can
1In practice, we exclude the score of the positive pair from the list to calculate the top one distribution used in Eq.(4), to enhance the ranking information of negatives, because the score of the positive pair occupies most in the full top one distribution calculated by the teacher SimCSE.
be transferred and preserved. In our experiments, we utilize the weighted average similarity scores of two teachers as pseudo ranking labels: ST (xi) = αST1 (xi) + (1 − α)ST2 (xi) where α is a hyperparameter to balance the weight of the teachers.
The contrastive learning loss LinfoNCE pushes apart the representations of different sentences to maximize the representation space, while the ranking consistency loss Lconsistency and the ranking distillation loss Lrank pull similar negatives closer, thus capturing fine-grained semantic ranking information. Combining the above three loss functions, we can obtain the overall objective:
Lfinal = LinfoNCE + βLconsistency + γLrank, (6)
where β and γ are hyperparameters to balance different losses.

heading: 5 Experiment

heading: 5.1 Setup
text: We evaluate our approach on two sentence related tasks, Semantic Textual Similarity (STS) and Transfer (TR). The SentEval toolkit (Conneau and Kiela, 2018) is used in our experiments. For STS tasks, we evaluate on seven datasets: STS12-16 (Agirre et al., 2012, 2013, 2014, 2015, 2016), STS Benchmark (Cer et al., 2017) and SICK-Relatedness (Marelli et al., 2014). These datasets contain pairs of sentences with similarity score labels from 0 to 5. Following SimCSE, we directly compute the cosine similarity between the sentence representations which means all the STS experiments are fully unsupervised, and report the Spearman’s correlation. For TR tasks, we evaluate on seven datasets with the default configurations from SentEval: MR (Pang and Lee, 2005), CR (Hu and Liu, 2004), SUBJ (Pang and Lee, 2004), MPQA (Wiebe et al., 2005), SST-2 (Socher et al., 2013), TREC (Voorhees and Tice, 2000) and MRPC (Dolan and Brockett, 2005). We use a logistic regression classifier trained on top of the frozen sentence representations, and report the classification accuracy.
For fair comparison, we use the same 106 randomly sampled sentences from English Wikipedia provided by SimCSE. Following previous works, we start from pre-trained checkpoints of BERT (Devlin et al., 2019) and RoBERTa (Liu et al., 2019), and utilize the embedding corresponding to [CLS] token as the representation of the input sentence. First we train SimCSE models including four variants: SimCSE-BERTbase, SimCSE-
BERTlarge, SimCSE-RoBERTabase and SimCSERoBERTalarge. We utilize the first two as a multiteacher for RankCSE-BERTbase and RankCSEBERTlarge, while the last two for RankCSERoBERTabase and RankCSE-RoBERTalarge. We evaluate our model every 125 training steps on the dev set of STS-B and keep the best checkpoint for the evaluation on test sets of all STS and TR tasks. More training details can be found in Appendix A.
We compare RankCSE with several unsupervised sentence representation learning methods, including average GloVe embeddings (Pennington et al., 2014), USE (Cer et al., 2018) and Skipthought (Kiros et al., 2015), average BERT embeddings from the last layer, post-processing methods such as BERT-flow (Li et al., 2020) and BERTwhitening (Su et al., 2021), and contrastive learning methods such as IS-BERT (Zhang et al., 2020) and ConSERT (Yan et al., 2021). We also include recent strong unsupervised sentence representation baselines, including SimCSE (Gao et al., 2021),
DCLR (Zhou et al., 2022), ArcCSE (Zhang et al., 2022), DiffCSE (Chuang et al., 2022), PaSER (Wu and Zhao, 2022) and PCL (Wu et al., 2022a). Since RankCSE and the teacher model SimCSE are using the same unsupervised training data, the comparison between RankCSE and baselines is fair.

heading: 5.2 Main Results
text: Results on STS Tasks As shown in Table 2, it is clear that RankCSE significantly outperforms the previous methods on all PLMs, which demonstrates the effectiveness of our approach. For example, compared with SimCSE, RankCSE has brought noticeable improvements: 4.11% on BERTbase, 2.19% on BERTlarge, 3.24% on RoBERTabase and 1.57% on RoBERTalarge. RankCSE-BERTbase even outperforms SimCSE-BERTlarge by nearly 2%. Compared with the previous state-of-theart methods, RankCSE still achieves consistent improvements, which validates that RankCSE is able to obtain more semantically discriminative
representations by incorporating ranking consistency and ranking distillation. We also observe that the performances of RankCSElistNet and RankCSElistMLE are very consistent across all datasets, which demonstrates the effectiveness of both listwise ranking methods.
Results on TR Tasks It can be seen in Table 3 that RankCSE achieves the best performance among all the compared baselines on all PLMs.
Note that for DiffCSE, we obtain the results from the publicly available code and checkpoints, because DiffCSE uses different dev sets to find the best hyperparameters for TR tasks than other baselines. More detailed explanation and comprehensive comparison are provided in Appendix B. Another observation is that the performance of the RankCSElistNet is slightly better than that of the RankCSElistMLE. Our hypothesis is that the inaccurate pseudo ranking labels introduce more errors in the calculation of the permutation probability than the top one probability. Nevertheless, both listwise methods achieve better results than the baselines, which is consistent with the results in Table 2.

heading: 5.3 Analysis and Discussion
text: Ablation Study To investigate the impact of different losses in our approach, we conduct a set of ablation studies by removing LinfoNCE, Lconsistency and Lrank from Eq.(6). The average results on STS and TR tasks are reported in Table 4. There are several observations from the results. First, when Lrank is removed, the performance significantly drops in both STS and TR tasks, which indicates the effectiveness of Lrank in our modeling. Second, it is also clear that without LinfoNCE or Lconsistency, the model performance also decreases, especially on TR tasks. Thirdly, it is worth mentioning that RankCSE with only Lrank can also outperform the teachers on STS tasks. The reason is that RankCSE is able to preserve ranking knowledge from multiple teachers, and generalize fine-grained ranking information from multiple coarse-grained representations. Fourthly, since Lconsistency does not explicitly distinguish the positives from negatives, RankCSE with only Lconsistency will preserve inaccurate rankings leading to significant performance drop. Finally, RankCSE with all components achieves the best performance on both STS and TR tasks.
Comparisons of Different Teachers We conduct experiments to explore the impact of dif-
ferent teachers on the performance of RankCSE. As shown in Table 5, RankCSE outperforms the teacher model which indicates that incorporating ranking consistency and ranking distillation leads to more semantically discriminative sentence representations. Comparing the performance of RankCSE using different teachers, we observe that better teacher leads to better RankCSE, which is consistent with our expectation since accurate ranking labels yield more effective ranking knowledge transfer. Another observation is that the performance of RankCSE with a multi-teacher is better than that with a single teacher, which verifies that RankCSE is able to preserve listwise ranking knowledge from more than one teacher. It is also interesting to see that using DiffCSE-BERTbase and SimCSE-BERTlarge as multi-teacher leads to even higher performance than the results in Table 2. We plan to conduct more investigation along this direction to explore the upper bound of improvements.
Effect of Hyperparameters To study the effect of temperature hyperparameters, we conduct experiments by setting different τ2 and τ3. As shown in Figure 3a, we find that large discrepancy between τ2 and τ3 leads to significant drop in the performance of RankCSEListNet. The best temperature setting for RankCSEListNet is τ2 : τ3 =
2 : 1. The performance of RankCSEListMLE has similar trends based on different PLMs, as shown in Figure 3b. For both RankCSEListNet and RankCSEListMLE, the temperature should be set moderate.
Robustness of RankCSE We conduct 5 runs of model training with the hyperparameter settings which can be referred to Appendix A with different random seeds, and then calculate the mean and standard deviation values. The results provided in Table 6 demonstrate both the superior performance and the robustness of our model. It can also be seen that RankCSElistMLE achieves similar performance but more stable results compared with RankCSElistNet.
Alignment and Uniformity Following previous works (Wang and Isola, 2020), we use alignment and uniformity to measure the quality of representation space. Alignment measures the distance between similar instances, while uniformity measures how well the representations are uniformly distributed (detailed in Appendix H). For both measures, the smaller value indicates the better result. We plot the distribution of ℓalign-ℓuniform for different models using BERTbase which are measured on the STS-B dev set. As shown in Figure 4, RankCSE effectively improves both alignment and uniformity compared with average BERT embeddings, while SimCSE and DiffCSE only improve uniformity and alignment respectively. Since RankCSE pulls similar negatives closer during incorporating ranking consistency and ranking distillation, RankCSE has smaller alignment and bigger uniformity than SimCSE. We consider that RankCSE achieves a better trade-off than SimCSE. When compared with DiffCSE, RankCSE has smaller uniformity whereas similar alignment. We can also observe that RankCSE outperforms PCL on both metrics.

heading: 6 Conclusion
text: In this work, we propose RankCSE, an unsupervised approach to learn more semantically discriminative sentence representations. The core idea of RankCSE is incorporating ranking consistency and ranking distillation with contrastive learning into a unified framework. When simultaneously ensuring ranking consistency and distilling listwise ranking knowledge from the teacher, RankCSE can learn how to make fine-grained distinctions in semantics,
leading to more semantically discriminative sentence representations. Experimental results on STS and TR tasks demonstrate that RankCSE outperforms previous state-of-the-art methods. We also conduct thorough ablation study and analysis to demonstrate the effectiveness of each component and justify the inner workings of our approach. We leave what is the upper bound of improvements of the teacher for future work.
Limitations
In this section, we discuss the limitations of our work as follows. First, despite achieving promising results, our model needs to calculate pseudo ranking labels of the teacher which requires additional training time per epoch than the teacher. The training efficiency of RankCSE and SimCSE can be seen in Appendix D. Second, we directly use SimCSEbase and SimCSElarge as a multi-teacher in our implementation and experiments. However, how to choose the best combination of the teacher models is worth further exploration. It could help researchers to better understand the upper bound of improvements. We plan to investigate more along this direction in the future.

heading: Acknowledgements
text: This work is supported by Ministry of Science and Technology Key R&D Program (2030 Artificial Intelligence) (No. 2020AAA0106600) and National Natural Science Foundation of China (NSFC Grant No. 62122089). We sincerely thank all reviewers for their valuable comments and suggestions, which are crucial for improving our work. We would also like to acknowledge Angela Li for her contributions in creating the figures used in this work.

heading: A Training Details
text: We implement all experiments with the deep learning framework PyTorch on a single NVIDIA Tesla A100 GPU (40GB memory). We carry out grid-search of learning rate ∈ {2e-5, 3e-5} and temperatures τ2, τ3 ∈ {0.0125, 0.025, 0.05}, while setting batch size to 128, temperature τ1 to 0.05, α to 1/3, β to 1, γ to 1 and the rate of linear scheduling warm-up to 0.05 for all the experiments. We train our models for 4 epochs, and evaluate the model every 125 steps on the dev set of STS-B and keep the best checkpoint for the final evaluation on test sets of all STS and TR tasks. The hyperparameter settings we adopt are shown in Table 9. Following SimCSE, we utilize the embedding corresponding to [CLS] token as the representation of the input sentence. We utilize SimCSE-BERTbase and SimCSE-BERTlarge as a multi-teacher for RankCSE-BERTbase and RankCSE-BERTlarge, while SimCSERoBERTabase and SimCSE-RoBERTalarge as a multi-teacher for RankCSE-RoBERTabase and RankCSE-RoBERTalarge.

heading: B DiffCSE Settings for Transfer Tasks
text: DiffCSE uses different dev sets to find the best hyperparameters for the two tasks (STS-B dev set for STS tasks, dev sets of 7 TR tasks for TR tasks), while other methods only use the STS-B dev set for both tasks, which is not fair. Therefore we obtain the results in Table 3 from its publicly available code and checkpoints for STS tasks2 instead of directly importing the results from its original paper. For a more comprehensive comparison with DiffCSE on TR tasks, we also use dev sets of 7 TR tasks to find the best hyperparameters and checkpoints. As shown in Table 10, RankCSE still outperforms DiffCSE in this setting.

heading: C Data Statistics
text: The complete listings of train/dev/test stats of STS and TR datasets can be found in Table 7 and 8, respectively. Note that for STS tasks, we only use test sets for the final evaluation and dev set of STSB to find best hyperparameters and checkpoints. The train sets of all STS datasets are not used in our experiments. For TR tasks, we follow the default settings of SentEval toolkit (Conneau and Kiela, 2018) to use 10-fold evaluation for all TR datasets
2https://github.com/voidism/DiffCSE
except SST. We can directly use the already split datasets to evaluate on SST.

heading: D Training Efficiency
text: We compare the training efficiency of SimCSE and RankCSE , which are tested on a single NVIDIA Tesla A100 GPU (40GB memory). We set batch size to 128 for both SimCSE and RankCSE, and training epoch to their original settings (1 for SimCSE, 4 for RankCSE). RankCSE utilizes SimCSEbase and SimCSElarge as a multi-teacher to provide pseudo ranking labels. As shown in Table 11, RankCSEbase and RankCSElarge can be trained within 2 hours and 3.7 hours respectively. Since RankCSE needs to calculate pseudo ranking labels of the teacher, it requires additional training time per epoch than SimCSE.

heading: E Cosine Similarity Distribution
text: We demonstrate the distribution of cosine similarities for sentence pairs of STS-B dev set in Figure 5. We can observe that cosine similarity distributions from all models are consistent with human ratings. However, the cosine similarities of RankCSE are slightly higher than that of SimCSE under the same human rating, as RankCSE pulls similar negatives closer during incorporating ranking consistency and ranking distillation, and shows lower variance. Compared with DiffCSE, RankCSE shows a more scattered distribution. This observation further validates that RankCSE can achieve a better alignmentuniformity balance.

heading: F Case Study
text: We present another two examples of a query sentence and several target sentences from the STS datasets, with their similarity scores and rankings in Table 12. It is obvious that the similarity scores produced by RankCSE are more effective than SimCSE, with consistent rankings to the ground-truth labels. It further demonstrates that SimCSE only captures coarse-grained semantic ranking information via contrastive learning, while RankCSE can capture fine-grained semantic ranking information. For example, SimCSE can distinguish between similar and dissimilar sentences, however, it can not distinguish between very similar and less similar sentences as RankCSE.

heading: G Ranking Tasks
text: We build the ranking task based on each STS dataset to verify that RankCSE can capture finegrained semantic ranking information. For one sentence xi, if there are more than three sentence pairs (xi, x j i ) containing xi with similarity score label yji in the dataset, we view {xi, x j i , y j i }kj=1(k > 3)
as a sample of the ranking task, as shown in Table 12. We adopt KCC (Kendall’s correlation coefficient (Abdi, 2007)) and NDCG (normalized discounted cumulative gain (Järvelin and Kekäläinen, 2002)) as evaluation metrics for ranking tasks, and demonstrate the results in Table 13. RankCSE outperforms SimCSE and DiffCSE on both KCC and NDCG, which validates that RankCSE can capture fine-grained semantic ranking information by incorporating ranking consistency and ranking distillation. Another observation is that SimCSE and DiffCSE also achieve moderate results, which shows they can distinguish coarse-grained semantic differences via contrastive learning.

heading: H Alignment and Uniformity
text: Wang and Isola (2020) use two properties related to contrastive learning, alignment and uniformity, to measure the quality of representation space. Alignment calculates expected distance between normalized representations of positive pairs ppos:
ℓalign ≜ E (x,x+)∼ppos ∥f(x)− f(x+)∥2, (7)
while uniformity measures how well the normalized representations are uniformly distributed:
ℓuniform ≜ log E x,y i.i.d.∼ pdata e−2∥f(x)−f(y)∥ 2 , (8)
where pdata denotes the distribution of sentence pairs. Smaller alignment means positive instances have been pulled closer, while smaller uniformity means random instances scatter on the hypersphere. These two measures are smaller the better, and well aligned with the object of contrastive learning.
ACL 2023 Responsible NLP Checklist

heading: A For every submission:

heading: 3 A1. Did you describe the limitations of your work?
text: Limitations

heading: 3 A2. Did you discuss any potential risks of your work?
text: Sections 1 and 6

heading: 3 A3. Do the abstract and introduction summarize the paper’s main claims?
text: Abstract and Section 1
7 A4. Have you used AI writing assistants when working on this paper? Left blank.
B 3 Did you use or create scientific artifacts? Section 5

heading: 3 B1. Did you cite the creators of artifacts you used?
text: Section 5
7 B2. Did you discuss the license or terms for use and / or distribution of any artifacts? We were unable to find the license for the dataset we used.

heading: 3 B3. Did you discuss if your use of existing artifact(s) was consistent with their intended use, provided
text: that it was specified? For the artifacts you create, do you specify intended use and whether that is compatible with the original access conditions (in particular, derivatives of data accessed for research purposes should not be used outside of research contexts)? Section 5
7 B4. Did you discuss the steps taken to check whether the data that was collected / used contains any information that names or uniquely identifies individual people or offensive content, and the steps taken to protect / anonymize it? The datasets we use are the commonly-used benchmarks.

heading: 3 B5. Did you provide documentation of the artifacts, e.g., coverage of domains, languages, and
text: linguistic phenomena, demographic groups represented, etc.? Section 5
3 B6. Did you report relevant statistics like the number of examples, details of train / test / dev splits, etc. for the data that you used / created? Even for commonly-used benchmark datasets, include the number of examples in train / validation / test splits, as these provide necessary context for a reader to understand experimental results. For example, small differences in accuracy on large test sets may be significant, while on small test sets they may not be. Appendix D
C 3 Did you run computational experiments?

heading: Appendix E

heading: 3 C1. Did you report the number of parameters in the models used, the total computational budget
text: (e.g., GPU hours), and computing infrastructure used? Section 5 and Appendix E
The Responsible NLP Checklist used at ACL 2023 is adopted from NAACL 2022, with the addition of a question on AI writing assistance.

heading: 3 C2. Did you discuss the experimental setup, including hyperparameter search and best-found
text: hyperparameter values? Section 5 and Appendix A

heading: 3 C3. Did you report descriptive statistics about your results (e.g., error bars around results, summary
text: statistics from sets of experiments), and is it transparent whether you are reporting the max, mean, etc. or just a single run? Section 5

heading: 3 C4. If you used existing packages (e.g., for preprocessing, for normalization, or for evaluation), did
text: you report the implementation, model, and parameter settings used (e.g., NLTK, Spacy, ROUGE, etc.)? Section 5
D 7 Did you use human annotators (e.g., crowdworkers) or research with human participants? Left blank.
D1. Did you report the full text of instructions given to participants, including e.g., screenshots, disclaimers of any risks to participants or annotators, etc.? No response.
D2. Did you report information about how you recruited (e.g., crowdsourcing platform, students) and paid participants, and discuss if such payment is adequate given the participants’ demographic (e.g., country of residence)? No response.
D3. Did you discuss whether and how consent was obtained from people whose data you’re using/curating? For example, if you collected data via crowdsourcing, did your instructions to crowdworkers explain how the data would be used? No response.
D4. Was the data collection protocol approved (or determined exempt) by an ethics review board? No response.
D5. Did you report the basic demographic and geographic characteristics of the annotator population that is the source of the data? No response.

heading: Future Work
text: We leave what is the upper bound of improvements of the teacher for future work. Limitations
In this section, we discuss the limitations of our work as follows. First, despite achieving promising results, our model needs to calculate pseudo ranking labels of the teacher which requires additional training time per epoch than the teacher. The training efficiency of RankCSE and SimCSE can be seen in Appendix D. Second, we directly use SimCSEbase and SimCSElarge as a multi-teacher in our implementation and experiments. However, how to choose the best combination of the teacher models is worth further exploration. It could help researchers to better understand the upper bound of improvements. We plan to investigate more along this direction in the future