SimCSE: Simple Contrastive Learning of Sentence Embeddings

Accepted to EMNLP 2021. The code and pre-trained models are available at https://github.com/princeton-nlp/simcse

section_content: Learning universal sentence embeddings is a fundamental problem in natural language processing and has been studied extensively in the literature (Kiros et al., 2015; Hill et al., 2016; Conneau et al., 2017; Logeswaran and Lee, 2018; Cer et al., 2018; Reimers and Gurevych, 2019, inter alia). 

In this work, we advance state-of-the-art sentence embedding methods and demonstrate that a contrastive objective can be extremely effective when coupled with pre-trained language models such as BERT (Devlin et al., 2019) or RoBERTa (Liu et al., 2019). We present SimCSE, a simple contrastive sentence embedding framework, which can produce superior sentence embeddings, from either unlabeled or labeled data. 

Our unsupervised SimCSE simply predicts the input sentence itself with only dropout (Srivastava et al., 2014) used as noise (Figure 1(a) ). In other words, we pass the same sentence to the pre-trained encoder twice: by applying the standard dropout twice, we can obtain two different embeddings as "positive pairs". Then we take other sentences in the same mini-batch as "negatives", and the model predicts the positive one among negatives. Although it may appear strikingly simple, this approach outperforms training objectives such as predicting next sentences (Logeswaran and Lee, 2018) and discrete data augmentation (e. g., word deletion and replacement) by a large margin, and even matches previous supervised methods. Through careful analysis, we find that dropout acts as minimal "data augmentation" of hidden representations, while removing it leads to a representation collapse. 

Our supervised SimCSE builds upon the recent success of using natural language inference (NLI) datasets for sentence embeddings (Conneau et al., 2017; Reimers and Gurevych, 2019 ) and incorporates annotated sentence pairs in contrastive learning (Figure 1(b) ). Unlike previous work that casts it as a 3-way classification task (entailment, neutral and contradiction), we leverage the fact that entailment pairs can be naturally used as positive instances. We also find that adding corresponding contradiction pairs as hard negatives further improves performance. This simple use of NLI datasets achieves a substantial improvement compared to prior methods using the same datasets. We also compare to other labeled sentence-pair

The pets are sitting on a couch.
section_name: Introduction

section_content: There are animals outdoors. 

There is a man. 

The man wears a business suit. 

A kid is skateboarding. 

A kit is inside the house. 

Two dogs are running. 

A man surfing on the sea. 

A kid is on a skateboard. datasets and find that NLI datasets are especially effective for learning sentence embeddings. 

To better understand the strong performance of SimCSE, we borrow the analysis tool from Wang and Isola (2020), which takes alignment between semantically-related positive pairs and uniformity of the whole representation space to measure the quality of learned embeddings. Through empirical analysis, we find that our unsupervised Sim-CSE essentially improves uniformity while avoiding degenerated alignment via dropout noise, thus improving the expressiveness of the representations. The same analysis shows that the NLI training signal can further improve alignment between positive pairs and produce better sentence embeddings. We also draw a connection to the recent findings that pre-trained word embeddings suffer from anisotropy (Ethayarajh, 2019; Li et al., 2020) and prove that-through a spectrum perspective-the contrastive learning objective "flattens" the singular value distribution of the sentence embedding space, hence improving uniformity. 

We conduct a comprehensive evaluation of Sim-CSE on seven standard semantic textual similarity (STS) tasks (Agirre et al., 2012 (Agirre et al.,, 2013 (Agirre et al.,, 2014 (Agirre et al.,, 2015 (Agirre et al.,, 2016;; Cer et al., 2017; Marelli et al., 2014) and seven transfer tasks (Conneau and Kiela, 2018). On the STS tasks, our unsupervised and supervised models achieve a 76. 3% and 81. 6% averaged Spearman's correlation respectively using BERT base, a 4. 2% and 2. 2% improvement compared to previous best results. We also achieve competitive performance on the transfer tasks. Finally, we identify an incoherent evaluation issue in the literature and consolidate results of different settings for future work in evaluation of sentence embeddings.
section_name: Different hidden dropout masks in two forward passes

section_content: Contrastive learning aims to learn effective representation by pulling semantically close neighbors together and pushing apart non-neighbors (Hadsell et al., 2006). It assumes a set of paired examples

, where x i and x + i are semantically related. We follow the contrastive framework in Chen et al. (2020) and take a cross-entropy objective with in-batch negatives (Chen et al., 2017; Henderson et al., 2017) : let h i and h + i denote the representations of x i and x + i, the training objective for (x i, x + i ) with a mini-batch of N pairs is:

where τ is a temperature hyperparameter and sim(h 1, h 2 ) is the cosine similarity

In this work, we encode input sentences using a pre-trained language model such as BERT (Devlin et al., 2019) or RoBERTa (Liu et al., 2019) : h = f θ (x), and then fine-tune all the parameters using the contrastive learning objective (Eq. 1). 

Positive instances. One critical question in contrastive learning is how to construct (x i, x + i ) pairs. In visual representations, an effective solution is to take two random transformations of the same image (e. g., cropping, flipping, distortion and rotation) as x i and x + i (Dosovitskiy et al., 2014). A similar approach has been recently adopted in language representations (Wu et al., 2020; Meng et al., 2021) by applying augmentation techniques such as word deletion, reordering, and substitution. However, data augmentation in NLP is inherently difficult because of its discrete nature. As we will see in §3, simply using standard dropout on intermediate representations outperforms these discrete operators. 

In NLP, a similar contrastive learning objective has been explored in different contexts (Henderson et al., 2017; Gillick et al., 2019; Karpukhin et al., 2020). In these cases, (x i, x + i ) are collected from supervised datasets such as question-passage pairs. Because of the distinct nature of x i and x + i, these approaches always use a dual-encoder framework, i. e., using two independent encoders f θ 1 and f θ 2 for x i and x + i. For sentence embeddings, Logeswaran and Lee (2018) also use contrastive learning with a dual-encoder approach, by forming current sentence and next sentence as (x i, x + i ). Alignment and uniformity. Recently, Wang and Isola (2020) identify two key properties related to contrastive learning-alignment and uniformityand propose to use them to measure the quality of representations. Given a distribution of positive pairs p pos, alignment calculates expected distance between embeddings of the paired instances (assuming representations are already normalized):

(2) On the other hand, uniformity measures how well the embeddings are uniformly distributed:

where p data denotes the data distribution. These two metrics are well aligned with the objective of contrastive learning: positive instances should stay close and embeddings for random instances should scatter on the hypersphere. In the following sections, we will also use the two metrics to justify the inner workings of our approaches.
section_name: Background: Contrastive Learning

section_content: The idea of unsupervised SimCSE is extremely simple: we take a collection of sentences {x i } m i=1 and use x + i = x i. The key ingredient to get this to work with identical positive pairs is through the use of independently sampled dropout masks for x i and x + i. In standard training of Transformers (Vaswani et al., 2017), there are dropout masks placed on fully-connected layers as well as attention probabilities (default p = 0. 1). We denote h z i = f θ (x i, z) where z is a random mask for dropout. We simply feed the same input to the encoder twice and get

for a mini-batch of N sentences. Note that z is just the standard dropout mask in Transformers and we do not add any additional dropout. 

Dropout noise as data augmentation. We view it as a minimal form of data augmentation: the positive pair takes exactly the same sentence, and their embeddings only differ in dropout masks. 

We compare this approach to other training objectives on the STS-B development set (Cer et al., 2017) h = f θ (g(x), z) and g is a (random) discrete operator on x. We note that even deleting one word would hurt performance and none of the discrete augmentations outperforms dropout noise. 

We also compare this self-prediction training objective to the next-sentence objective used in Logeswaran and Lee (2018), taking either one encoder or two independent encoders. As shown in Table 2, we find that SimCSE performs much better than the next-sentence objectives (82. 5 vs 67. 4 on STS-B) and using one encoder instead of two makes a significant difference in our approach. 

Why does it work? To further understand the role of dropout noise in unsupervised SimCSE, we try out different dropout rates in Table 3 and observe that all the variants underperform the default dropout probability p = 0. 1 from Transformers. We find two extreme cases particularly interesting: "no dropout" (p = 0) and "fixed 0. 1" (using default dropout p = 0. 1 but the same dropout masks for the pair). In both cases, the resulting embeddings for the pair are exactly the same, and it leads to a dramatic performance degradation. We take the checkpoints of these models every 10 steps during training and visualize the alignment and uniformity metrics 3 in Figure 2, along with a simple data augmentation model "delete one word". As clearly shown, starting from pre-trained checkpoints, all models greatly improve uniformity. However, the alignment of the two special variants also degrades drastically, while our unsupervised SimCSE keeps a steady alignment, thanks to the use of dropout noise. It also demonstrates that starting from a pretrained checkpoint is crucial, for it provides good initial alignment. At last, "delete one word" improves the alignment yet achieves a smaller gain on the uniformity metric, and eventually underperforms unsupervised SimCSE. 3 We take STS-B pairs with a score higher than 4 as ppos and all STS-B sentences as p data.
section_name: Unsupervised SimCSE

section_content: We have demonstrated that adding dropout noise is able to keep a good alignment for positive pairs (x, x + ) ∼ p pos. In this section, we study whether we can leverage supervised datasets to provide better training signals for improving alignment of our approach. Prior work (Conneau et al., 2017; Reimers and Gurevych, 2019) has demonstrated that supervised natural language inference (NLI) datasets (Bowman et al., 2015; Williams et al., 2018) are effective for learning sentence embeddings, by predicting whether the relationship between two sentences is entailment, neutral or contradiction. In our contrastive learning framework, we instead directly take (x i, x + i ) pairs from supervised datasets and use them to optimize Eq. 1. 

Choices of labeled data. We first explore which supervised datasets are especially suitable for constructing positive pairs (x i, x + i ). We experiment with a number of datasets with sentence-pair examples, including 1) QQP 4 : Quora question pairs; 2) Flickr30k (Young et al., 2014) : each image is annotated with 5 human-written captions and we consider any two captions of the same image as a positive pair; 3) ParaNMT (Wieting and Gimpel, 2018) : a large-scale back-translation paraphrase dataset 5 ; and finally 4) NLI datasets: SNLI (Bowman et al., 2015) and MNLI (Williams et al., 2018). 

We train the contrastive learning model (Eq. 1) with different datasets and compare the results in Table 4. For a fair comparison, we also run experiments with the same # of training pairs. Among all the options, using entailment pairs from the NLI (SNLI + MNLI) datasets performs the best. We think this is reasonable, as the NLI datasets consist of high-quality and crowd-sourced pairs. Also, human annotators are expected to write the hypotheses manually based on the premises and two sentences tend to have less lexical overlap. For instance, we find that the lexical overlap (F1 measured between two bags of words) for the entailment pairs (SNLI + MNLI) is 39%, while they are 60% and 55% for QQP and ParaNMT. 

Contradiction as hard negatives. Finally, we further take the advantage of the NLI datasets by using its contradiction pairs as hard negatives 6. In NLI datasets, given one premise, annotators are required to manually write one sentence that is absolutely true (entailment), one that might be true (neutral), and one that is definitely false (contradiction). Therefore, for each premise and its entailment hypothesis, there is an accompanying contradiction hypothesis 7 (see Figure 1 for an example). 

Formally, we extend

, where x i is the premise, x + i and x - i are entailment and contradiction hypotheses. The training objective i is then defined by (N is mini-batch size):

-log e sim(h i,h

. 

(5) As shown in Table 4, adding hard negatives can further improve performance (84. 9 → 86. 2) and this is our final supervised SimCSE. We also tried to add the ANLI dataset (Nie et al., 2020) or combine it with our unsupervised SimCSE approach, but didn't find a meaningful improvement. We also considered a dual encoder framework in supervised SimCSE and it hurt performance (86. 2 → 84. 2).
section_name: Supervised SimCSE

section_content: Recent work identifies an anisotropy problem in language representations (Ethayarajh, 2019; Li et al., 2020), i. e., the learned embeddings occupy a narrow cone in the vector space, which severely limits their expressiveness. Gao demonstrate that language models trained with tied input/output embeddings lead to anisotropic word embeddings, and this is further observed by Ethayarajh (2019) in pre-trained contextual representations. Wang et al. (2020) show that singular values of the word embedding matrix in a language model decay drastically: except for a few dominating singular values, all others are close to zero. 

A simple way to alleviate the problem is postprocessing, either to eliminate the dominant principal components (Arora et al., 2017; Mu and Viswanath, 2018), or to map embeddings to an isotropic distribution (Li et al., 2020; Su et al., 2021). Another common solution is to add regularization during training (Gao et al., 2019; Wang et al., 2020). In this work, we show that-both theoretically and empirically-the contrastive objective can also alleviate the anisotropy problem. 

The anisotropy problem is naturally connected to uniformity (Wang and Isola, 2020), both highlighting that embeddings should be evenly distributed in the space. Intuitively, optimizing the contrastive learning objective can improve uniformity (or ease the anisotropy problem), as the objective pushes negative instances apart. Here, we take a singular spectrum perspective-which is a common practice in analyzing word embeddings (Mu and Viswanath, 2018; Gao et al., 2019; Wang et al., 2020), and show that the contrastive objective can "flatten" the singular value distribution of sentence embeddings and make the representations more isotropic. 

Following Wang and Isola (2020), the asymptotics of the contrastive learning objective (Eq. 1) can be expressed by the following equation when the number of negative instances approaches infinity (assuming f (x) is normalized):

where the first term keeps positive instances similar and the second pushes negative pairs apart. When p data is uniform over finite samples {x i } m i=1, with h i = f (x i ), we can derive the following formula from the second term with Jensen's inequality:

Let W be the sentence embedding matrix corresponding to {x i } m i=1, i. e., the i-th row of W is h i. Optimizing the second term in Eq. 6 essentially minimizes an upper bound of the summation of all elements in WW, i. e., Sum(WW ) = m i=1 m j=1 h i h j. Since we normalize h i, all elements on the diagonal of WW are 1 and then tr(WW ) (the sum of all eigenvalues) is a constant. According to Merikoski (1984), if all elements in WW are positive, which is the case in most times according to Figure G. 1, then Sum(WW ) is an upper bound for the largest eigenvalue of WW. When minimizing the second term in Eq. 6, we reduce the top eigenvalue of WW and inherently "flatten" the singular spectrum of the embedding space. Therefore, contrastive learning is expected to alleviate the representation degeneration problem and improve uniformity of sentence embeddings. 

Compared to post-processing methods in Li et al. (2020) ; Su et al. (2021), which only aim to encourage isotropic representations, contrastive learning also optimizes for aligning positive pairs by the first term in Eq. 6, which is the key to the success of SimCSE. A quantitative analysis is given in §7.
section_name: Connection to Anisotropy

section_name: Experiment

section_content: We conduct our experiments on 7 semantic textual similarity (STS) tasks. Note that all our STS experiments are fully unsupervised and no STS training sets are used. Even for supervised SimCSE, we simply mean that we take extra labeled datasets for training, following previous work (Conneau et al., 2017). We also evaluate 7 transfer learning tasks and provide detailed results in Appendix E. We share a similar sentiment with Reimers and Gurevych (2019) that the main goal of sentence embeddings is to cluster semantically similar sentences and hence take STS as the main result. 

Semantic textual similarity tasks. We evaluate on 7 STS tasks: STS 2012-2016 (Agirre et al., 2012 (Agirre et al.,, 2013 (Agirre et al.,, 2014 (Agirre et al.,, 2015 (Agirre et al.,, 2016)), STS Benchmark (Cer et al., 2017) and SICK-Relatedness (Marelli et al., 2014). When comparing to previous work, we identify invalid comparison patterns in published papers in the evaluation settings, including (a) whether to use an additional regressor, (b) Spearman's vs Pearson's correlation, and (c) how the results are aggregated (Table B. 1). We discuss the detailed differences in Appendix B and choose to follow the setting of Reimers and Gurevych (2019) in our evaluation (no additional regressor, Spearman's correlation, and "all" aggregation). We also report our replicated study of previous work as well as our results evaluated in a different setting in Table B. 2 and Table B. 3. We call for unifying the setting in evaluating sentence embeddings for future research. 

Training details. We start from pre-trained checkpoints of BERT (Devlin et al., 2019) (uncased) or RoBERTa (Liu et al., 2019) (cased) and take the [CLS] representation as the sentence embedding 9 (see §6. 3 for comparison between different pooling methods). We train unsupervised SimCSE on 10 6 randomly sampled sentences from English Wikipedia, and train supervised SimCSE on the combination of MNLI and SNLI datasets (314k). 

More training details can be found in Appendix A. Table 5 : Sentence embedding performance on STS tasks (Spearman's correlation, "all" setting). We highlight the highest numbers among models with the same pre-trained encoder. ♣: results from Reimers and Gurevych (2019); ♥: results from Zhang et al. (2020) ; all other results are reproduced or reevaluated by ourselves. For BERT-flow (Li et al., 2020) and whitening (Su et al., 2021), we only report the "NLI" setting (see Table C. 1).
section_name: Evaluation Setup

section_content: We compare unsupervised and supervised Sim-CSE to previous state-of-the-art sentence embedding methods on STS tasks. Unsupervised baselines include average GloVe embeddings (Pennington et al., 2014), average BERT or RoBERTa embeddings 10, and post-processing methods such as BERT-flow (Li et al., 2020) and BERTwhitening (Su et al., 2021). We also compare to several recent methods using a contrastive objective, including 1) IS-BERT (Zhang et al., 2020), which maximizes the agreement between global and local features; 2) DeCLUTR (Giorgi et al., 2021), which takes different spans from the same document as positive pairs; 3) CT (Carlsson et al., 2021), which aligns embeddings of the same sentence from two different encoders. 11 Other supervised 10 Following Su et al. (2021), we take the average of the first and the last layers, which is better than only taking the last. 11 We do not compare to CLEAR (Wu et al., 2020), because they use their own version of pre-trained models, and the numbers appear to be much lower. Also note that CT is a concurrent work to ours. methods include InferSent (Conneau et al., 2017), Universal Sentence Encoder (Cer et al., 2018), and SBERT/SRoBERTa (Reimers and Gurevych, 2019) with post-processing methods (BERT-flow, whitening, and CT). We provide more details of these baselines in Appendix C. 

Table 5 shows the evaluation results on 7 STS tasks. SimCSE can substantially improve results on all the datasets with or without extra NLI supervision, greatly outperforming the previous stateof-the-art models. Specifically, our unsupervised SimCSE-BERT base improves the previous best averaged Spearman's correlation from 72. 05% to 76. 25%, even comparable to supervised baselines. When using NLI datasets, SimCSE-BERT base further pushes the state-of-the-art results to 81. 57%. The gains are more pronounced on RoBERTa encoders, and our supervised SimCSE achieves 83. 76% with RoBERTa large. 

In Appendix E, we show that SimCSE also achieves on par or better transfer task performance compared to existing work, and an auxiliary MLM objective can further boost performance.
section_name: Main Results

section_content: Unsup. Sup. 

[
section_name: Pooler

section_content: We investigate the impact of different pooling methods and hard negatives. All reported results in this section are based on the STS-B development set. We provide more ablation studies (normalization, temperature, and MLM objectives) in Appendix D. 

Pooling methods. Reimers and Gurevych (2019) ; Li et al. (2020) show that taking the average embeddings of pre-trained models (especially from both the first and last layers) leads to better performance than [CLS]. -log e sim(h i,h

where 1 j i ∈ {0, 1} is an indicator that equals 1 if and only if i = j. We train SimCSE with different values of α and evaluate the trained models on the development set of STS-B. We also consider taking neutral hypotheses as hard negatives. As shown in Table 7, α = 1 performs the best, and neutral hypotheses do not bring further gains.
section_name: Ablation Studies

section_content: In this section, we conduct further analyses to understand the inner workings of SimCSE. 

Uniformity and alignment. Figure 3 shows uniformity and alignment of different sentence embedding models along with their averaged STS results. In general, models which have both better alignment and uniformity achieve better performance, confirming the findings in Wang and Isola (2020). We also observe that (1) though pre-trained embeddings have good alignment, their uniformity is poor (i. e., the embeddings are highly anisotropic);

(2) post-processing methods like BERT-flow and BERT-whitening greatly improve uniformity but also suffer a degeneration in alignment; (3) unsupervised SimCSE effectively improves uniformity of pre-trained embeddings whereas keeping a good alignment; (4) incorporating supervised data in SimCSE further amends alignment. In Appendix F, we further show that SimCSE can effectively flatten singular value distribution of pre-trained embeddings. In Appendix G, we demonstrate that SimCSE provides more distinguishable cosine similarities between different sentence pairs. Qualitative comparison. We conduct a smallscale retrieval experiment using SBERT base and SimCSE-BERT base. We use 150k captions from Flickr30k dataset and take any random sentence as query to retrieve similar sentences (based on cosine similarity). As several examples shown in Table 8, the retrieved sentences by SimCSE have a higher quality compared to those retrieved by SBERT.
section_name: Analysis

section_content: Early work in sentence embeddings builds upon the distributional hypothesis by predicting surrounding sentences of a given one (Kiros et al., 2015; Hill SBERT base Supervised SimCSE-BERT base Query: A man riding a small boat in a harbor. 

#1 A group of men traveling over the ocean in a small boat. A man on a moored blue and white boat. #2 Two men sit on the bow of a colorful boat. 

A man is riding in a boat on the water. #3 A man wearing a life jacket is in a small boat on a lake. A man in a blue boat on the water. 

Query: A dog runs on the green grass near a wooden fence. 

#1 A dog runs on the green grass near a grove of trees. 

The dog by the fence is running on the grass. #2 A brown and white dog runs through the green grass. 

Dog running through grass in fenced area. #3 The dogs run in the green field. 

A dog runs on the green grass near a grove of trees. 2. et al., 2016; Logeswaran and Lee, 2018). Pagliardini et al. (2018) show that simply augmenting the idea of word2vec (Mikolov et al., 2013) with n-gram embeddings leads to strong results. Several recent (and concurrent) approaches adopt contrastive objectives (Zhang et al., 2020; Giorgi et al., 2021; Wu et al., 2020; Meng et al., 2021; Carlsson et al., 2021; Kim et al., 2021; Yan et al., 2021) by taking different views-from data augmentation or different copies of models-of the same sentence or document. Compared to these work, SimCSE uses the simplest idea by taking different outputs of the same sentence from standard dropout, and performs the best on STS tasks. Supervised sentence embeddings are promised to have stronger performance compared to unsupervised counterparts. Conneau et al. (2017) propose to fine-tune a Siamese model on NLI datasets, which is further extended to other encoders or pre-trained models (Cer et al., 2018; Reimers and Gurevych, 2019). Furthermore, Wieting and Gimpel (2018) ; Wieting et al. (2020) demonstrate that bilingual and back-translation corpora provide useful supervision for learning semantic similarity. Another line of work focuses on regularizing embeddings (Li et al., 2020; Su et al., 2021; Huang et al., 2021) to alleviate the representation degeneration problem (as discussed in §5), and yields substantial improvement over pre-trained language models.
section_name: Related Work

section_content: In this work, we propose SimCSE, a simple contrastive learning framework, which greatly improves state-of-the-art sentence embeddings on semantic textual similarity tasks. We present an unsupervised approach which predicts input sentence itself with dropout noise and a supervised approach utilizing NLI datasets. We further justify the inner workings of our approach by analyzing alignment and uniformity of SimCSE along with other baseline models. We believe that our contrastive objective, especially the unsupervised one, may have a broader application in NLP. It provides a new perspective on data augmentation with text input, and can be extended to other continuous representations and integrated in language model pre-training. and Gurevych (2019). Since the "all" setting fuses data from different topics together, it makes the evaluation closer to real-world scenarios, and unless specified, we take the "all" setting. We list evaluation settings for a number of previous work in Table B. 1. Some of the settings are reported by the paper and some of them are inferred by comparing the results and checking their code. As we can see, the evaluation protocols are very incoherent across different papers. We call for unifying the setting in evaluating sentence embeddings for future research. We will also release our evaluation code for better reproducibility. Since previous work uses different evaluation protocols from ours, we further evaluate our models in these settings to make a direct comparison to the published numbers. We evaluate SimCSE with "wmean" and Spearman's correlation to directly compare to Li et al. (2020) and Su et al. (2021)
section_name: Conclusion

section_content: We elaborate on how we obtain different baselines for comparison in our experiments:

• For average GloVe embedding (Pennington et al., 2014), InferSent (Conneau et al., 2017) and Universal Sentence Encoder (Cer et al., 2018), we directly report the results from Reimers and Gurevych (2019), since our evaluation setting is the same as theirs. 

• For BERT (Devlin et al., 2019) and RoBERTa (Liu et al., 2019), we download the pretrained model weights from HuggingFace's Transformers 13, and evaluate the models with our own scripts. 

• For SBERT and SRoBERTa (Reimers and Gurevych, 2019), we reuse the results from the original paper. For results not reported by Reimers and Gurevych (2019), such as the performance of SRoBERTa on transfer tasks, we download the model weights from Sen-tenceTransformers 14 and evaluate them. • For DeCLUTR (Giorgi et al., 2021) and contrastive tension (Carlsson et al., 2021), we reevaluate their checkpoints in our setting. 

• For BERT-flow (Li et al., 2020), since their original numbers take a different setting, we retrain their models using their code 15, and evaluate the models using our own script. 

• For BERT-whitening (Su et al., 2021), we implemented our own version of whitening script following the same pooling method in Su et al. (2021), i. e. first-last average pooling. Our implementation can reproduce the results from the original paper (see Table B. 2). 

For both BERT-flow and BERT-whitening, they have two variants of postprocessing: one takes the NLI data ("NLI") and one directly learns the embedding distribution on the target sets ("target"). We find that in our evaluation setting, "target" is generally worse than "NLI" (Table C. 1), so we only report the NLI variant in the main results.
section_name: C Baseline Models

section_content: Normalization and temperature. We train Sim-CSE using both dot product and cosine similarity with different temperatures and evaluate them on the STS-B development set. As shown in Table D. 1, with a carefully tuned temperature τ = 0. 05, cosine similarity is better than dot product.
section_name: D Ablation Studies

section_content: We evaluate our models on the following transfer tasks: MR (Pang and Lee, 2005), CR (Hu and Liu, 2004), SUBJ (Pang and Lee, 2004), MPQA (Wiebe et al., 2005), SST-2 (Socher et al., 2013), TREC (Voorhees and Tice, 2000) and MRPC (Dolan and Brockett, 2005). A logistic regression classifier is trained on top of (frozen) sentence embeddings produced by different methods. 

We follow default configurations from SentEval 16. Table E. 1 shows the evaluation results on transfer tasks. We find that supervised SimCSE performs on par or better than previous approaches, although the trend of unsupervised models remains unclear. We find that adding this MLM term consistently improves performance on transfer tasks, confirming our intuition that sentence-level objective may not directly benefit transfer tasks. We also experiment with post-processing methods (BERT- (2020). We highlight the highest numbers among models with the same pre-trained encoder. MLM: adding MLM as an auxiliary task with λ = 0. 1. 

flow/whitening) and find that they both hurt performance compared to their base models, showing that good uniformity of representations does not lead to better embeddings for transfer learning. As we argued earlier, we think that transfer tasks are not a major goal for sentence embeddings, and thus we take the STS results for main comparison.
section_name: E Transfer Tasks

section_name: F Distribution of Singular Values

section_content: To directly show the strengths of our approaches on STS tasks, we illustrate the cosine similarity distributions of STS-B pairs with different groups of Compared to all the baseline models, both unsupervised and supervised SimCSE better distinguish sentence pairs with different levels of similarities, thus leading to a better performance on STS tasks. In addition, we observe that SimCSE generally shows a more scattered distribution than BERT or SBERT, but also preserves a lower variance on semantically similar sentence pairs compared to whitened distribution. This observation further validates that SimCSE can achieve a better alignment-uniformity balance.
section_name: G Cosine-similarity Distribution

section_name: Acknowledgements

section_content: We thank Tao Lei, Jason Lee, Zhengyan Zhang, Jinhyuk Lee, Alexander Wettig, Zexuan Zhong, and the members of the Princeton NLP group for helpful discussion and valuable feedback. This research is supported by a Graduate Fellowship at Princeton University and a gift award from Apple.
section_name: Acknowledgements

section_name: A Training Details

section_content: We implement SimCSE with transformers package (Wolf et al., 2020). For supervised Sim-CSE, we train our models for 3 epochs, evaluate the model every 250 training steps on the development set of STS-B and keep the best checkpoint for the final evaluation on test sets. We do the same for the unsupervised SimCSE, except that we train the model for one epoch. We carry out grid-search of batch size ∈ {64, 128, 256, 512} and learning rate ∈ {1e-5, 3e-5, 5e-5} on STS-B development set and adopt the hyperparameter settings in Table A. 1. We find that SimCSE is not sensitive to batch sizes as long as tuning the learning rates accordingly, which contradicts the finding that contrastive learning requires large batch sizes (Chen et al., 2020). It is probably due to that all SimCSE models start from pre-trained checkpoints, which already provide us a good set of initial parameters. For both unsupervised and supervised SimCSE, we take the [CLS] representation with an MLP layer on top of it as the sentence representation. Specially, for unsupervised SimCSE, we discard the MLP layer and only use the [CLS] output during test, since we find that it leads to better performance (ablation study in §6. 3). 

Finally, we introduce one more optional variant which adds a masked language modeling (MLM) objective (Devlin et al., 2019) as an auxiliary loss to Eq. 1: + λ • mlm (λ is a hyperparameter). This helps SimCSE avoid catastrophic forgetting of token-level knowledge. As we will show in Table D. 2, we find that adding this term can help improve performance on transfer tasks (not on sentence-level STS tasks).
section_name: A Training Details

section_content: We elaborate the differences in STS evaluation settings in previous work in terms of (a) whether to use additional regressors; (b) frozen sentence embeddings for STS-B and SICK-R, and train the regressor on the training sets of the two tasks, while most sentence representation papers take the raw embeddings and evaluate in an unsupervised way. In our experiments, we do not apply any additional regressors and directly take cosine similarities for all STS tasks. Metrics. Both Pearson's and Spearman's correlation coefficients are used in the literature. Reimers et al. (2016) argue that Spearman correlation, which measures the rankings instead of the actual scores, better suits the need of evaluating sentence embeddings. For all of our experiments, we report Spearman's rank correlation. 

Aggregation methods. Given that each year's STS challenge contains several subsets, there are different choices to gather results from them: one way is to concatenate all the topics and report the overall Spearman's correlation (denoted as "all"), and the other is to calculate results for different subsets separately and average them (denoted as "mean" if it is simple average or "wmean" if weighted by the subset sizes). However, most papers do not claim the method they take, making it challenging for a fair comparison. We take some of the most recent work: SBERT (Reimers and Gurevych, 2019), BERT-flow (Li et al., 2020) and BERT-whitening (Su et al., 2021) 12 as an example: In Table B. 2, we compare our reproduced results to reported results of SBERT and BERT-whitening, and find that Reimers and Gurevych (2019) take the "all" setting but Li et al. (2020) ; Su et al. (2021) take the "wmean" setting, even though Li et al. (2020) claim that they take the same setting as Reimers
section_name: B Different Settings for STS Evaluation

section_content: BERTbase-whitening 0-1 1-2 2-3 3-4

1. 00 0. 75 0. 50 0. 25 0. 00 0. 25 0. 50 0. 75 1. 00
section_name: 4-5

section_content: Unsupervised SimCSE-BERTbase 0-1 1-2 2-3 3-4

1. 00 0. 75 0. 50 0. 25 0. 00 0. 25 0. 50 0. 75 1. 00
section_name: 4-5

section_content: SBERTbase 0-1 1-2 2-3 3-4

1. 00 0. 75 0. 50 0. 25 0. 00 0. 25 0. 50 0. 75 1. 00
section_name: 4-5

section_content: SBERTbase-whitening 0-1 1-2 2-3 3-4

1. 00 0. 75 0. 50 0. 25 0. 00 0. 25 0. 50 0. 75 1. 00
section_name: 4-5

section_content: Supervised SimCSE-BERTbase Figure G. 1: Density plots of cosine similarities between sentence pairs in STS-B. Pairs are divided into 5 groups based on ground truth ratings (higher means more similar) along the y-axis, and x-axis is the cosine similarity.
section_name: 4-5