SimCSE++: Improving Contrastive Learning for Sentence Embeddings from Two Perspectives
Authors: Jiahao Xu
Wei Shao
Lihui Chen
Lemao Liu
Venue: EMNLP 2023 Main (2023)
Keywords: sentence embedding
contrastive learning
dimention-wise contrastive learning
TL;DR: This paper improve contrastive learning for sentence embeddings by handling dropout noise in negatives and dimension-wise contrastive learning regulation.

Abstract: This paper improves contrastive learning for sentence embeddings from two perspectives: handling dropout noise and addressing feature corruption. Specifically, for the first perspective, we identify that the dropout noise from negative pairs affects the model's performance. Therefore, we propose a simple yet effective method to deal with such type of noise. Secondly, we pinpoint the rank bottleneck of current solutions to feature corruption and propose a dimension-wise contrastive learning objective to address this issue. Both proposed methods are generic and can be applied to any contrastive learning based models for sentence embeddings. Experimental results on standard benchmarks demonstrate that combining both proposed methods leads to a gain of 1.8 points compared to the strong baseline SimCSE configured with BERT base. Furthermore, applying the proposed method to DiffCSE, another strong contrastive learning based baseline, results in a gain of 1.4 points.

SimCSE++: Improving Contrastive Learning for Sentence Embeddings from Two Perspectives
Jiahao Xu' Wei Shao² Lihui Chen' Lemao Liu
Nanyang Technological University, CCity Univeristy of Hong Kong, Tenneett AI Lab "jiahao004@e. ntu. edu.sg Setccen@ntu.edu... edu.sg
2 we Tweishao4-c@my cityu..euuuu .du. hk
iredmonddiu@tenccet.
Abstract
et al. (2021) applies the whitening process to flatten uniform distribution of representations.
This paper improves contrastive learning for sentence embeddings from two perspectives: handling dropout noise and addressing feature corruption. Specifically, for the first perspec- tive, we identify that dropout noise from negative pairs affects the model's performance. Therefore, we propose a simple yet effective method deal with such type of noise. Sec- ondly, we pinpoint the rank bottleneck of cur- rent solutions to feature corruption and pro- pose dimension-wise contrastive learning ob- jective to address this issue. Both proposed methods are generic and can be applied to any contrastive learning based models for sentence emheddings. Experimental results on standard benchmarks demonstrate that combining both proposed methods leads to gain of 1.8 points compared to the strong baseline SimCSE con- figured with BERT base. Furthermore, apply- ing the proposed method to DifCCSE, another strong contrastive learning based baseline, re- sults in gain of 1.4 points.
More recently, remarkable advancements have been achieved by contrastive learning (CL) sen- tence embeddings (Gao et al., 2021), which clev- erly makes use of dropout randomness (Bouthillier et al., 2015) to construct positive pairs in an unsu- pervised way. Since then, many notable variants have been proposed under the contrastive learn- ing framework to intensify performance by con- structing hard contrastive pairs (Giorgi et al., 2021; Kim al., 2021; Yan et al., 2021; Wu et al., 2022: Zhang al., 2022b; Chuang al., 2020), intro- ducing other CL--based objectives (Zhang et al., 2021, 2022c; Chuang et al., 2022; Zhang et al., 2022a; Tan et al., 2022; Xu et al., 2023) or utilizing more sophisticated similarity metries (Zhang et al.. 2022c).
this paper, we improve the sentence embed- ding models from two perspectives: dropout noise and feature corruption. Specifically, first, we empir- ically study the effects of dropout randomness on positive pairs and negative pairs in the CL-based objective. We find that modest dropout noise in the positive pairs beneficial the model per- formance whereas dropout noise in negative pairs is harmful. We provide an explanation from the principle of noise contrastive estimation (Gutmann and Hyvärinen, 2012) and the role of dropout in constructing positive pairs. Based on these find- ings, we propose simple yet effective strategy, off- dropout, which turns off the dropout randomness in negative pairs to further improve the performance.
1 Introduction
Sentence representation, which transforms sen- tence semantic information from discrete language space into dense vectors, is one of the most funda- mental tasks in natural language processing, as it serves the central role for wide range of down- stream applications (c.g., information retrieval, se- mantic comparison, question answering, and lan- guage translation). Sentence representation has been constantly evolving (Pennington et al., 2014; Zhang et al., 2020; Carlsson et al., 2021), and achieves even stronger performance when utilizing pre-trained language models (PLM) (Devlin et al., 2019 Delobelle et al., 2020). Moreover, on top of PLMs, number of post-processing strategies achieve even better performance. For example, Li al. (2020) employs flow- model and Su
Second, we revisit the issue of feature corrup- tion on the sentence embedding and empirically study the well-known solution recently proposed by Zbontar et al. (2021); Klein and Nabi (2022) to this problem. Surprisingly, we find that this solution does not improve performance under the contrastive learning framework for sentence embed- dings. We further analyze this finding and identify
"The source code is available at Jiahao004/SincSE-llus palppplluuuuuu plus.
com/

---

the reason behind it as the rank bottleneck issue in the mini-batch embedding matrix. To tackle this issue, we propose a simple dimension-wise contrastive learning (DCL) to break down the bot- tleneck, which eventually enhances the baseline performance.
Zhang et al. (2020) utilize all the output token rep- resentations, Yan et al. (2021) enhance dropout augmentation, Giorgi et al. (2021) employ context sentences and Kim et al. (2021) contrast each layer representation within PLMs. Some studies aim to counter the PLMs bias towards sentence represen- tations: Carlsson et al. (2021) employ heteroge- neous model structure, Zhou et al. (2022) filter out noise from negatives. Others introduce more effec- tive contrastive learning framework: Chuang et al. (2022) introduce ELECTRA (Clark et al., 2020) with equivariant contrastive learning (Dangovski et al., 2021), Zhang et al. (2022c) utilize ArcFace (Deng et al.. 2019) framework.
As result, by combining the proposed off- dropout and DCL, we have advanced the SimCSE baseline by 1.9 points. Furthermore, our repro- duced results have shown that we advanced the current state-of-the of-the-rrt model, DiffCSE (Chuang et al., 2022). by 1.4 points.
In general, our contribution is three-fold:
1. We, for the first time, point out that dropout noise from negative pairs has side effect model performance and propose an off- sampling strategy to alleviate this side effect.
All the previous studies on sentence embed- dings have concentrated on developing more in- tricate frameworks based on the SimCSE frame- work. These advancements include creating more efficient training samples, introducing advanced metrics, and incorporating additional training tasks. In contrast to these existing studies, research aims enhance the contrastive learning frame- work itself. Specifically, we address two issues: the problem of dropout noise in the representation and the feature corruption caused by the correlation between different dimensions of the representation.
2. We identify the rank bottleneck in the current solution to the feature corruption problem and propose a novel dimension-win wise CL objective to avoid the bottleneck.
3. Experimental results on standard benchmarks for sentence embeddings show that the combi- nation of our proposed methods outperforms strong baselines by margin and achieves new state-of-the--ar.
2.2 Contrastive Learning and NCE
The importance of contrastive learning has long been recognized. In NLP research fields, con- trastive learning is introduced into sentence rep- resentations (Giorgi et al., 2021; Wu ct al., 2020), text classification (Fang et al., 2020), information extraction (Qin et al., 2021), machine translations (Pan et al., 2021), question answering (Karpukhin et al., 2020) etc.
2 Related Work
2.1
Sentence Representation
Early studies for sentence representations leverage the word2vec (Mikolov et al.) ideas. Semantic information can be captured by predicting a sen- tence from its surrounding sentences (Kiros et al., 2015; Hill et al., 2016; Logeswaran and Lee, 2018) 30. Pagliardini et al. (2018) 8) aggregates the n-gram em- beddings using a pooling strategy, which achieves strong result. With the development of large-scale pre-trained language models (Devlin et al., 2019; Liu et al.. 2020), sentence representation methods begin to utilize PLMs strong language representa- tion ability. For example, Reimers and Gurevych (2019) employs siamese network with PLMs for supervised sentence representation, while Li et al. (2020) and Su et al. (2021) apply post-processing top of PLM's representations.
The concept of contrastive learning is based on Noise Contrastive Estimation (NCE) (Gutmann and Hyvärinen, 2010), which involves maximizing the probability of target signals by comparing them with randomly sampled noise. While NCE uses nonlinear logistic regression to distinguish between observed data and artificially generated noise us- ing the log-density function, contrastive learning utilizes InfoNCE (Oord et al., 2018) objectives to discriminate between positive similarities and simi- larities among negative samples within the batch.
Recent studies on sentence embeddings are based on the strong baseline SimCSE (Gao al.,
The previous research on NCE and contrastive learning primarily concentrates on the noise arising from the sampling of negative examples. How- ever, this study investigates the noise originating
2021). Under the SimCSE framework, several stud- ies focus on constructing hard contrastive pairs:

---

from dropout randomness and examines the impact of dropout randomness on sentence embeddings, considering both negative and positive examples.
SimCSE (Gao et al., 2021) aims maximize the agreement between positive pairs (2) .2?) and min- imize the in-batch negatives ( Rj) using
the InfoNCE objective (Oord et al., 2018):
2.3 Feature Corruption Issue
es(2)
Feature corruption is a non-trivial problem in rep- resentation learning, where each dimension of the model shares high similarities with others. This is- sue hinders the expressive capacity to convey com- plex information effectively, the diversity of each dimension value is constrained by such correlation. Several studies (Li et al., 2020; Su et al., 2021) have attempted to address this issue by achieving a more independent embedding space through post- processing. However, as demonstrated in Wang et al. (2022), these post-processing methods primar- ily enhance performance for sentence pairs with low similarity and fail to improve performance for pairs with high similarity.
inoo log
(2)
( + ΣN وززز (R))
Here, s(-.)) -) is the similarity measure between two inputs (i.e., cos ros_sim(-;-)/T where T the tem- perature). In Equation (2), (2t, 2z) is a negative pair, and the dropout random variable & is used as an augmentation function for positive pairs, i.e., (2) 2-)
3.1 Dropout Noise in Negative Estimation
Empirical study dropout noise In PLMs such BERT, it is shown that dropout plays important role in training because of the regular- ization effect. CL-based sentence embeddings, the training objective Eq. (2) involves 2 x N BERT structures, and thus the role of dropout in Eq. (2) might be more complex. This motivates us to study the effect of dropout.
Recently, Zbontar al. (2021) proposed Bar- lowTwins as solution for such a issue in images. Inspired by the redundancy-reducttn
principle of neuroscientist H. Barlow, BarlowTwins minimizes redundancy between different dimensions, natu- rally reducing similarity across each dimension. Unlike post-processing methods, this approach addresses the problem in an end-to-end manner. Furthermore, direct application of BarlowTwins on sentence embeddings (Klein and Nabi, 2022) achieves comparable performance SimCSE.
As presented in Eq. (1), dropout is determined by the random variable and thus 2! (or =?) E [1) contains some noise due the ran- dom variable ç. To study the effect of dropout noise, we respectively add more noise (+Noise) or reduce some noise (-Noise) to z (or 2?) and then study their final performance.
In contrast to previous research that simply ap- plies the BarlowTwing objective to the SimCSE framework, our study investigates the rank bottle- neck issue of BarlowTwins in the context of sen- tence representation. We tackle this issue and im- prove the model's performance accordingly.
Specifically, to introduce more noise to 2! (or 2f we add small Gaussian noise as follows:
2 = [xy; (,) +
2, fax; (?) + g2
3 Improving Dropout Noise in CL
Where g² and 9² are Gaussian with the mean 0 and variance 0.1. On the other hand, according the Central Limit Theorem (Fischer), the K sample average converges to its expectation with 1/K of the original variance ! Therefore, to reduce the noise from 2! (or 2²), we could simply use the following mean sampling:
SimCSE framework plays a central role in recent sentence embedding strategies. It is simple con- trastive learning framework that learns by identify- ing positive pairs among in-batch negatives. Specif- ically, for a given sentence 24, let f(-) denotes a pre-trained language model, and it is used to gener- ate two views (2/ z²?) of the identical sentences a;
1 K flan
via different dropout patterns:
Σ
2 K
7 [(x
k=1
27 - f(x; {?)
(1)
2 K
2-
1
Σ
K
fai
22
where < and & denote two samples from the dropout random variable & (Srivastava et al., 2014).
KK = 10 in this paper.

---

Method
STS12
STS13
STS14
STS15
STS16
STS-B
SICK-R
Avg.
SimCSE
68.40
82.41
74.38
80.91
78.56
76.85
72.23
76.2.
+Noisc
68.59
82.57
73.26
80.79
76.72
75.56
69.24
75.25
Pos.
-Noise
55.22
71.78
60.55
70.79
72.95
64.58
63.00
65.55
+Noisc
65.13
82.45
71.69
79.67
77.73
75.03
70.04
74.53
Neg.
-Noise
70.25
83.73
75.61
82.25
78.77
77.58
71.26
77.06
Table I: Performance on STS benchmark with adding/reducing noise in positive and negative pairs.
where t/ and 2.2. are independently sampled from the dropout variable &, and thus 24" contains less noise than 2/-
large, as theoretically analyzed in Gutmann and Hyvärinen (2012). In this sense, reducing the noise in embeddings is achieved by mean pooling from multiple embeddings which implicitly increases the sample size with respect to the random variable ç and potentially leads to improved performance, i.e.. replacing 2! and 2 by 2, and 2 involving K samples (in both positive pairs and negative pairs within Eq. (2)) through mean sampling may obtain better performance.
Experimental results and findings Since Eq. (2) contains positive pair (z) and negative pair (2), we individually conduct experiments to estimate the impact of the noise in positive and negative pairs. Respectively, SimCSE+Pos+Noosee is achieved by replacing the positíve pair s(2], 2/) by s(2,"", ,2,,²) in Eq. (2), and Sim- NoE+Neg+Noise is achieved by replacing the negative pair 22) by s(z) 1,t "2)")) in Eq. (2).
However, enlarging the sample size affects pos- itive and negative pairs differently. As shown in Table 1, reducing noise in positive pairs through mean sampling results in unsatisfactory perfor- mance, while it improves performance in negative pairs. The main reason is that, under the SimCSE framework, the positive pairs require diversity as informative pairs for contrastive learning, which is reduced by mean sampling. Otherwise, the train- ing signal in Eq. (2) may become trivial if there is no diversity between 2! and 2 for positive pair, because sz),, =47 > s(2) 23) when 2 2? and i * j. In summary, diversity is crucial for posi- tive pairs, while minimizing noise is beneficial for negative pairs to achieve better performance.
Similary, SimCSE+Pos-Nooi applies s(2)", 2f
the replacement of positive pair s(2) 2/) and
SinCSF+Neg-NVe NNise uses s(2 2)))) to replace negative pair s(2) 22))
Table 1 shows that increasing the noise level for both positive and negative embeddings may de- generate the performance while reducing the noise level for negative embeddings is helpful for model performance. In summary, we can obtain the fol- lowing findings: I) having modest noise in positive pairs is necessary make CL successful and re- ducing noise in positive pairs is harmful the performance; 2) the model performance is related to the noise level of negative pairs: more noise de- grades the performance while less noise improves the performance.
3.2 Our Solution: Off-Dropout Sampling
Mean sampling significantly reduces the variance and yields better performance. However, K times average sampling requires time complexity over- head of O(KN).
Theoretical Explanation Contrastive learning compares the similarity of positive examples with negative ones. This idea is based on Noise Con- trastive Estimation (NCE) (Gutmann and Hyväri- nen, 2010), where the positive similarity score is the target signal that NCE tries to maximize, while the negative similarity score is the corresponding noise signal.
To address this overhead, we propose off- dropout sampling, which turns off the dropout when sampling negative example representations. Off-dropout sampling produces representations with zero variance. At a high level, off-dropout sampling is empirically equivalent to the mean of infinite times resampling, demonstrated by Hin- ton et al. (2012), which is also known as weight scaling inference rule (Goodfellow et al., 2016). Therefore, off off-dropout sampling provides unbiased estimation of representation with zero variance, and
The InfoNCE loss in Eq. (2) follows Noise Con- trastive Estimation (NCE) (Gutmann and Hyväri- nen, 2010). shows that the model converges faster and performs better when the sample size

---

the sampling overhead is equal to that of default random sampling. Consequently, the InfoNCE ob- jective for off-dropout sampling is:
Where D is the total number of dimensions (D=768 for base model), c, d are dimension in- dices, and = = are corresponding dimension values of the representation of the i-th sentence from a mini-batch of size N. However, such an objective does not yield gains over SimCSE when applied to sentence embeddings in STS tasks (Klein and Nabi, 2022).
Infe- log ( + Zj=L.j (((
(3)
where szi; 2z) represents the similarity between negative pairs, and 2i, 2j represents the represen- tations sampled without dropout. m a trade-off factor between positive and negative examples.
4.2 Rank Bottleneck for BarlowTwing
Barlow Twins aims to achieve orthogonalization of all dimensions in the representation by maxi- mizing the diagonal elements of the correlation matrix, denoted as C (Cod). while minimizing the non-diagonal elements. In linear algebra, a parametrized matrix can be optimized to become an orthogonal matrix if there exists a parameter that ensures the matrix is of full rank. However, both theoretically and empirically, we observe that C is far from being a full-rank matrix, meaning its rank is close to D.
It should be noticed that reducing the noise in negatives is very different from hyperparameter tun- ing: In principle, we investigate the sample size and thereby justify if the current sentence embedding methods satisfy the large sample size requirement from the NCE principle; In practice, tuning the dropout rate changes the distribution of dropout patterns, which violates the principle of controlling variables. Therefore, our strategy to reduce the noise in negatives is fundamentally different from parameter tuning in both principle and practice.
From theoretical standpoint, if the denominator of Cd remains constant for any e and C can be expressed as the product of matrix with dimen- sions D x N and another matrix with dimensions N x D. In this case, we can demonstrate that the rank of C is at most min(N, N, D). However, in the conventional settings of SimCSE, N is 64 and D is 768.- Consequently, the rank of C' is at most N, where N < D for any parameter.
4 Mitigating Feature Corruption
Feature Corruption Issue Feature corruption (Chen and He, 2021) illustrates the issue that each dimension of the output representation has high similarity with the other dimensions. Such correla- tion between dimensions reduces the model's 's rep- resentation capability and undermines downstream performance (Zbontar et al., 2021; Klein and Nabi, 2022).
From an empirical perspective, we randomly sample a batch of 64 sentences and compute the rank of their cross-correlation matrix. We observe that the rank of the SimCSE correlation matrix is 64. Consequently, it is impossible to optimize rank 64 matrix become rank 768 identity ma- trix using the BarlowTwing objective. The rank of the correlation matrix poses a bottleneck for the Barlow Twins objective, making it difficult to opti- mize to become a full-rank matrix. Therefore, there is a rank bottleneck issue when optimizing the Barlow Twins objective. This might explain why Barlow Twins does not perform well when applied on top of SimCSE, as demonstrated in Table 2.
4.1 Existing Solution
Zbontar et al. (2021) proposes BarlowTwin as an additive regulation to tackle such an issue, which is dimension decorrelation objective. Bar- lwwTwins tackles feature corruption by minimiz- ing the redundancy between each dimension and aims to produce -indeeeeen represen- tations. Formally, given a cross correlcooreattion matrix C E R D its objective is:
{BT = E( Cor)2 ABT
ΣΣ
C2
dipe
Empirical Justification of the Rank Bottleneck To verify the rank bottleneck hypothesis, one can adjust the batch size reduce the total number of representation dimensions. However, increasing
(4)
Cal
'Feature comrptiin issue is also known as "fea- turefrepresentation degeneration/collage problem, which originates from contrastive learning rescarch in computer vi- sion.
"Note that the batch size of 64 is the default setting in Sim- CSE, which achieved the hest performance in both SimCSE original paper and our preliminary experiments.

---

Objectives
STS12
STS13
STS14
STS15
STS16
STS-B
SICK-R
Avg.
SimCSF-BERT base
68.40
82.41
74.38
80.91
78.56
76.85
72.23
76.25
+Barlowwwin (D=768)
50.59
70.07
58.48
68.98
68.23
64.94
67.07
64.05
+ +100 artificial 2
66.19
77.62
67.67
75.17
73.34
72.37
67.26
71.37
+300 artificial
69.29
78.80
69.03
77.47
75.27
74.18
70.88
73.56
+ +704 artificial
70.52
81.86
73.72
81.17
76.95
77.21
71.10
76.08
Table 2: SimCSE performance with Twins additive objectives. We pad each mini-batch (batch size 64) embedding matrix with a group of artificial representations sampled from standard Gaussian distribution.
the batch size will alter the number of in-batch negatives, while reducing the representation di- mensions will exacerbate the dimension bottleneck problem. Both methods will modify the default settings of SimCSE and consequently affect its per- formance.
The term 8(21,) 2.g)) calculates the cross- dimension similarity between the e-th and d-th dimensions. We use dot product with batch normalization to measure similarity:
s s(2) 2. -=)== Σ
To address this, we conduct straightforward ex- periment without altering the SimCSE framework settings. We maintain the original SimCSE tings but introduce M artificial embeddings each mini-batch embedding matrix when calculating the BarlowTwins loss value. Thus, contrastive learning at the data level is still performed on N batch size embeddings, while dimension-ue wise decorrelation is applied to the padded embedding matrix of size N + M. Consequently, increase the rank of the correlation matrix by M without modifying SimCSE.
1,0 TDLL
24.0 Ze
Tsc
Here. 2:c - Ei 3,,0 02. N Ei -
The DCL objective represents dimension wise contrastive learning. It improves upon the Bar- lowTwins objective in several ways: 1) Intuitively, Eq. is relative optimization that can be more eas- ily optimized compared the absolute regression objective (Gutmann and Hyvärinen, 2012); 2) This relative optimization avoids the rank bottleneck is- sue by only requiring the dimension to relatively more "self-similar" compared to other dimensions. instead of necessitating full-rank identity matrix as the only optimal solution.
We employ this approach to train the model, and the results are presented in Table 2. The table il- lustrates that the performance of the BarlowTwin objective improves as the number of padding arti- ficial embeddings increases. By introducing these artificial embeddings, we successfully overcome the rank bottleneck issue of the correlation matrix.
By combining both proposed strategies with trade-off factor the final objective function for improving contrastive learning for sentence embed- dings as follows:
4.3
Our Solution: Dimension-Win Contrastive Learning
L = loffinfo + ACDCL
(6)
Previous experiments have confirmed the existence of the rank bottleneck issue in the Barlow Twins ob- jective and have addressed this problem by padding artificial embeddings. However, optimizing param- eters with large number of artificial embeddings reduces training efficiency. Therefore, we propose a Dimension- wise Contrastive Learning (DCL) ob- jective that naturally avoids the rank bottleneck issue. The DCL objective is defined as follows:
Experiments
5.1 Setups
Baselines We compare with several sentence representation methods on STS tasks, which in- cludes GloVe embeddings (Pennington et al,,,, 2014), Skip-thought (Kiros et al., 2015), BERT embed- dings with pooling aggregation (Devlin et al., 2019). BERT-FlOW(L et al., 2020), and BERT- Whitening(Su et al., 2021).
We also compare with several recently proposed (5) contrastive learning based sentence representation method, for instance, ISBERT (Zhang et al., 2020),
Σ log c=1
PDC..
Σb=

---

Method
STS12
STS13
STS14
STS15
STS16
STS-B
SICK-R
Avg.
GloVe(avg.)
55.14
70.66
59.73
68.25
63.66
58.02
53.76
61.32
BERTbase (first-last avg.)
39.70
59.38
49.67
66.03
66.19
53.87
62.06
56.70
baRe fflow
58.40
67.10
60.85
75.16
71.22
68.66
64.47
66.55
BERT base -whitening
57.83
66.90
60.90
75.08
71.31
68.24
63.73
66.28
IS-BERTEase base
56.77
69.24
61.21
75.23
70.16
69.21
64.25
66.58
CT-BERThase
61.63
76.80
68.47
77.50
76.48
74.31
69.19
72.05
ConSERT base
64.64
78.49
69.07
79.72
75.95
73.97
67.31
72.74
SCD-BERT base
66.94
78.03
69.89
78.73
76.23
76.30
73.18
74.19
SimCSE-BERY
68.40
82.41
74.38
80.91
78.56
76.85
72.23
76.25
"SimCSE++-BERT base
73.66
82.36
75.86
83.09
79.76
79.71
71.91
78.05
*4Off-Info
69.39
82.42
75.91
82.92
78.82
78.86
71.62
77.13
*+DCL
70.15
83.46
74.91
81.95
79.83
79.39
72.14
77.40
ConSERT large
70.69
82.96
74.13
82.78
76.66
77.53
70.37
76.45
MANCSE-BERTT Large
70.88
84.16
76.43
84.50
79.76
79.26
73.88
78.41
MUnCSEE+-BERTT
72.37
85.37
78.68
84.69
79.57
80.37
74.05
79.30
SBERT base
70.97
76.53
73.19
79.09
74.30
77.03
72.91
74.89
SimCSE-SBERT
69.41
80.76
74.37
82.61
77.64
79.92
76.62
77.33
72.92
83,45
77.19
83.46
79.38
81.54
76.79
79.25
SBERT large
72.27
78.46
74.90
80.99
76.25
79.23
73.75
76.55
SimCSE-SBERTT.a.
76.16
83.77
77.27
84.33
79.73
81.67
77.25
80.03
*SimCSE++SSEETTDDee
76.66
84.76
78.53
84.37
79.66
82.37
78.09
80.63
Table 3: Main results on Semantic Textual Similarity benchmark dataset performance (Spearman correlation, "all" setting). Our proposed methods are marked with "*", The highest numbers among models with the same pre- trained encoder are highlighted. Off-dropout, DCL sampling and their combination -otterffooooor the baseline SimCSE with < 0.005.
CT-BERT (Carlsson et al., 2021), ConSERT (Yan al., 2021), together with the current mainstream SimCSE (Gao et al., 2021) and SOTA DifICSE (Chuang et al., 2022).
checkpoint the final evaluation on test sets.
Implement Details We conduct the experiments using pre-trained checkpoints from BERT base and BERTharge (Devlin et al., 2019) with Huggingface Transformer (Wolf et al., 2020) framework. Be- sides, to illustrate the compatibility of SBERT, fol- lowing Zhang al. (2022c) settings, we also em- ploy SBERT (Reimers and Gurevych, 2019) NLI (Conneau al., 2017; Reimers and Gurevych, 2019) 9) variant checkpoints for experiments.
Dataset We use the default one million randomly sampled sentences from English Wikipedia for un- supervised training, as previous studies (Gao et al.. 2021; Chuang et al., 2022; Zhang et al., 2022c; Wu et al., 2022) are all conducted this corpus. We do conduct any data selection or sampling strategy during the training.
We evaluate our model on sentence semantic textual similarity (STS) tasks, which in- cludes STS tasks 2012-2016 (Agirre et al., 2012), STS Benchmark (Cer et al., 2017), and SICK- Relatedness (Marelli et al., 2014). We follow Sim- CSE (Gao et al., 2021) settings of MLP layers, and employ MLP on top of [CLS] token representa- tion for training while removing MLP for evalua- tion. We evaluate the model for every 125 updating steps based on the STS-B development set, without any gradient accumulation. And evaluate the best
Evaluation
During the training, the contrastive temperature is the same as SimCSE to be 0.05. And the trading-off ratio m is set as 0.9. For DCL, we set temperature TDCL as 5 and loss coefficient X as 0.1. We train the model for epoch with learning rate 35-5 for base model and -6 for the large model with the same batch size 64 and sequence length 32. The model is optimized by Adam (Kingma and Ba, 2014) optimizer with de- fault settings without gradient accumulation.

---

Objectives
STS12
STS13
STS14
STS15
STS16
STS-B
SICK-R
Avg.
BERT (first-last avg.)
39.69
59.37
49.67
66.03
66.19
53.88
62.06
56.70
#whitening(wiki)
45.64
64.38
56.57
70.35
68.64
60.32
63.42
61.33
DDCLowikimmrr
52.96
73.85
62.80
72.12
71.25
68.15
68.36
67.07
+flow (NLI)
58.40
67.10
60.85
75.16
71.22
68.66
64.47
66.55
+whitening(NLI)
57.83
66.90
60.90
75.08
71.31
68.24
63.73
66.28
+DCL(NLI, TDCL=0.05)
59.25
74.04
63.91
75.85
72.46
70.67
67.05
69.03
SimCSF-BERT base
68.40
82.41
74.38
80.91
78.56
76.85
72.23
76.25
+whitening
59.27
76.68
67.22
74.86
73.85
68.43
69.22
69.93
+flow
61.33
78.54
69.87
77.47
76.02
71.73
70.70
72.24
DifCCSE (Chuang et al., 2022)
72.28
84.43
76.47
83,90
80.54
80.59
71.23
78.49
DifCCSE (reproduced)'
69.41
82.47
74.52
82.82
80.06
79.39
72.05
77.25
*+
72.68
83.40
76.76
83.66
80.57
80.94
72.82
78.69
Table 4: Block DCL compared with post-processing methods. NLI is used without labels; Block 2: Post- processing methods on top of SimCSE lead to unsatisfying performance; Block 3: SimCSE+ is robust to the non-SimCSE framework. I: Using officially released source code, and our method improves its performance with p 0.005.
5.2 Main Results
Y
0.5
0.9
1.2
STS-B dev
68.55
81.61
83.77
79.11
79.99
71.49
The evaluation results are shown in Table SimCSE++ outperforms the previous approaches. Compared with its basclines, our methods advance the average Spearman correlation coefficient from 76.25% to 78.05%, and its large variant raises the average Spearman correlation score further to 79.30%. SimCSE+ also improves the SBERT variants' performances. Compared with SimCSE, SimCSE++ achieves 79.25% and 80.63% on base and large variants, which shows an even stronger representation ability.
X
0.02
0.05
0.1
0.2
0.5
1 77.79
STS-B
82.60
83.25
83.77
82.36
80.79
5
10
20
50
STS-B dev 82.16
82,43
83.77
83.56
83.37
81.76
Table 5: Searching for weight term DCL objective
weight and DCL temperature TDCL on STS-B devel- opment
Model
Training Time
SimCSE-BERT
1h 50 min
We also explore the contribution of DCL objec- tive and off-dropout sampling in Table 3. It shows that the off-dropout sampling strategy alone is able improve the sentence semantic representation 77.13% Spearman correlation score, and DCL objective with normal dropout augmented negative contrastive term achieves 77.40%.
*SimCSE++-BERT
1h 59 min
Table 6: epoch training time for SimCSE and our pro- posed SimCSE++
duct the qualitative comparison on sentence re- trieval tasks in Appendix B to further illustrate our improvement.
5.3 Ablation Study
We investigate the effect of the hyperparame- ters on the whole system the STS-B devel- opment set of BERT base model in Table We search m the range (0.5, 1,9, The optimal value is 0.9. We search the ag- gregation weight À for DCL within the range (0.02, 0.05, 0.1, 1}. and the optimum value is 0.1. We carry out the DCL temperature search in ranging in {1, 5, 10, 20, 50), and opti- mal DCL temperature is 5.
Comparing with post-processing We compare the single DCL objective with widely applied post-processing methods (i.e. whitening and flow model). Table 4 shows that the DCL objective out- performs all the post-processing methods.
Robustness to other framework In Table 4, we introduce our method into the DiffCSE framework using officially released source code with our pro- posed methods. As result, we further advance DifICSE baseline by 1.4 points based on our repro- duced results.
Following Gao et al. (2021), 1), we also plot lalign" funifiom joint plot at Appendix A. Further, we con-

---

Runtime Efficiency We compare the training time between SimCSE and our proposed Sim- CSB++. shows that, the off-dropout sampling, and DCL do not introduce noticeable running time overhead compared to the SimCSE baseline. Moreover, we observe that both SimCSE and our proposed SimCSE+ converge to their optimum within the first 5k training steps, which is around 30 minutes of training. Consequently, the overhead of our modification negligible.
The First Joint Conference on Lexical and Compu- tational Semantics Volume 1: Proceedings of the main conference and the shared task, and Volume 2: Proceedings of the Sixth International Workshop on Semantic Evaluation (SemEval 2012), pages 385- 393, Montréal, Canada. Association for Computa- tional Linguistics.
Xavier Bouthillier, Kishore Konda, Pascal Vincent, and Roland Memisevic. 2015. Dropout as data augmen- tation. arXiv preprint arXiv:1506.08777
Fredrik Carlsson, Amaru Cuba Gyllensten, Evan- gelia Gogoulou, Erik Ylipää Hellquist, and Magnus Sahlgren. 2021. Semantic re-tuning with contrastive tension. In International Conference on Learning Representations.
6 Conclusion
In this paper, we improve CL-based sentence em- beddings in dropout noise and feature corruption. The main findings are: 1) having modest dropout noise is successful for positive pairs and reduc- ing dropout noise from positive pairs is harmful whereas reducing dropout noise from negative pairs is beneficial; 2) the well-known solution to fea- ture corruption does not lead to gains sentence embedding due the rank bottleneck issue. Ac- cordingly, we propose off-dropout to eliminate the dropout randomness from negative pairs and dimension-wise CL objective to break the bottle- neck to alleviate feature corruption, both of which outperform strong baselines by a margin.
Daniel Cer, Mona Diab, Eneko Agirre, Tñigo Lopez- Gazpio, and Lucia Specia. 2017. SemEval-2017 task I: Semantic textual similarity multilingual and crosslingual focused evaluation. In Proceedings of the 11th International Workshop on Semantic Evaluation (SemEval-2017) pages 1-14, Vancouver, Canada. Association for Computational Linguisties.
Xinlei Chen and Kaiming He. 2021. Exploring simple siamese representation learning. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 15750-15758.
Ching-Yao Chuang, Joshua Robinson, Yen-Chen Lin, Antonio Torralba, and Stefanic Jegelka. 2020. Debi- ased contrastive learning. Advances in neural infor- mation processing systems, 33:8765-8775.
Ethical Considerations
This study focuses on the representation of sen- tences, the objective of which is to achieve better performance on general domain sentence similarity tasks. Therefore, the training corpus and bench- mark datasets open source and do not contain any personally sensitive information; And we em- ploy widely applied pre-trained language models with commonly used contrastive learning strategies, thereby having no impact on the political, social, or natural environment.
Yung-Sung Chuang, Rumen Dangovski, Hongyin Luo, Yang Zhang, Shiyu Chang, Marin Soljacic, Shang- Wen Li, Scott Yih, Yoon Kim, and James Glass. 2022. DifiCSE: Difference- contrastive learn- ing for sentence embeddings. In Proceedings of the 2022 Conference of the North American Chap- ter of the Association for Computational Linguis- tics: Human Language Technologies, pages 4207- 4218, Seattle, United States. Association for Com- putational Linguistics.
Kevin Clark, Minh-Thang Luong. Quoe V Le, and Christopher D Manning. 2020. Electra: Pre-training text encoders as discriminators rather than genera- tors.
Limitations
The limitations consist of two aspects: for dropout noise, novel sampling strategy for positive pairs is left unexplored; for DCL, it could be improved by applying more advanced data-win wise contrastive learning strategies.
Alexis Conneau, Douwe Kiela, Holger Schwenk, Loic Barrault, and Antoine Bordes. 2017. Supervised learning of universal sentence representations from natural language inference data. In Proceedings of the 2017 Conference on Empirical Methods in Nat- waal Language Processing, pages 670-680, Copen- hagen, Denmark. Association for Computational Linguistics.
References
Rumen Dangovski, Li Jing, Charlotte Loh, Seung- wook Han, Akash Srivastava, Brian Cheung, Pulkit Agrawal, and Marin Soljačić. 2021. Equivariant con- trastive learning. arXiv preprint arXiv:2111.00899.
Eneko Agirre, Daniel Cer, Mona Diab, and Aitor Gonzalez-Agirre, 2012. SemEval-2012 task pilot on semantic textual similarity. In #SEM 2012:

---

Pieter Delobelle, Thomas Winters, and Bettina Berendt. 2020. RobBERT: a Dutch RoBERTa--bee based Lan- guage Model. In Findings of the Association for Computational Linguistics: EMNLP 2020, pages 3255-3265, Online. Association for Computational Linguistics.
Felix Hill, Kyunghyun Cho, and Anna Korhonen. 2016. Learning distributed representations of sen- tences from unlabelled data. In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Hu- mun Language Technologies, pages 1367-1377. San Diego, California. Association for Computational Linguistics.
Jiankang Deng. Jia Guo, Niannan Xue, and Stefanos Zafeiriou. 2019. Arcface: Additive angular margin Proceedings of
loss for deep face recognition. the IEEE/CVF conference on computer vision and pallern recognition, pages 4690-4699.
Geoffrey E. Hinton, Nitish Srivastava, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhut- dinov. 2012. Improving neural networks by preventing co-adaptation of feature detectors. CoRR, 2012100705800
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and
Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language under- standing. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume (Long and Short Papers), pages 4171-4186, Minneapolis, Minnesota, Associ- ation for Computational Linguistics.
Vladimir Karpukhin, Barlas Oguz, Sewon Min, Patrick Lewis, Ledell Wu, Sergey Edunov, Danqi Chen, and Wen-tau Yih. 2020. Dense passage retrieval for open-domain question answering. In Proceedings of the 2020 Conference on Empirical Methods in Nat- ural Language Processing (EMNLP), pages 6769- 6781, Online. Association for Computational Lin- guistics.
Hongchao Fang, Sicheng Wang. Meng Zhou, Jiayuan Ding, and Pengtao Xie. 2020. Cert: Contrastive self-supervised learning for language understanding. arXiv preprint arXiv:2005.1276 12766.
Taeuk Kim, Kang Min Yoo, and Sang- goo Lee. 2021. Self-guided contrastive learning for BERT sentence representations. In Proceedings of the 59th Annual Meeting of the Association for Computational Lin- guistics and the 11th International Joins Conference Natural Language Processing (Volume 1: Long Papers). pages 2528-2540, Online. Association for Computational Linguistics.
Hans Fischer. A history of the central limit theo- rem: from classical to modern probability theory. Springer.
Tianyu Gao, Xingcheng Yao, and Danqi Chen. 2021. SimCSE: Simple contrastive learning of sentence embeddings. In Proceedings of 2021 Conference on Empirical Methods in Natural Language Process- ing, pages 6894-6910, Online and Punta Cana, Do- minican Republic. Association for Computational Linguistics.
Diederik P Kingma and Jimmy Ba. 2014. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980
Ryan Kiros, Yukun Zhu, Russ Salakhutdinov, Richard Zemel, Raquel Urtasun, Antonio Torralba, and Sanja Fidler. 2015. Skip-thought vectors. In Advances in Neural Information Processing Systems, volume 28. Curran Associates, Inc.
John Giorgi. Osvald Nitski, Bo Wang, and Gary Bader. 2021. DeCLUTR: Deep contrastive learning for Proceed-
unsupervised textual representations. ings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th Interna- tional Joint Conference on Natural Language Pro- cessing (Volume I: Long Papers), pages 879-895, Online. Association for Computational Linguistics.
Tassilo Klein and Moin Nabi. 2022. SCD: Self- contrastive decorrelation of sentence embeddings. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 394-400, Dublin, Ireland. Association for Computational Linguistics.
Tan Goodfellow, Yoshua Bengio, and Aaron Courville. 2016. Deep Learning. MIT Press. http://www. deeplearningbook.oo org.
Michael Gutmann and Aapo Hyvärinen. 2010. Noise- contrastive estimation: A new estimation principle for unnormalized statistical models. In Proceedings of the Thirteenth International Conference on Artifi- cial Intelligence and Statistics, volume 9 of Proceed- ings Machine Learning Research, pages 297-304, Chia Laguna Resort, Sardinia, Italy, PMLR.
Bohan Li, Hao Zhou, Junxian He, Mingxuan Wang. Yiming Yang, and Lei Li. 2020. On the sentence embeddings from pre-trained language models. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 9119-9130, Online. Association for Computa- tional Linguistics.
Michael U Gutmann and Aapo Hyvärinen. 2012. Noise-contrastive estimation of unnormalized sta- tistical models, with applications to natural image statisties. Journal of machine learning research, 13(2).
Yinhan Liu, Myle Ott. Naman Goyal, Jingfei Du, Man- dar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2020. Ro/bert robustly optimized [bert] pretraining approach.

---

Lajanugen Logeswaran and Honglak Lee. 2018. An ef- ficient framework for learning sentence representa- tions. In International Conference on Learning Rep- resentations.
3982-3992, Hong Kong, China. Association for Computational Linguistics.
Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. 2014. Dropout: a simple way to prevent neural networks from overfitting. The journal of machine learning research, (51199199--19955
Marco Marelli, Stefano Menini, Marco Baroni, Luisa Bentivogli, Raffaella Bernardi, and Roberto Zampar- elli. 2014, A SICK cure for the evaluation of compo- sitional distributional semantic models. In Proceed- ings of the Ninth International Conference on Lan- guage Resources and Evaluation (LREC"14), pages 216-223, Reykjavik, Iceland. European Language Resources Association (ELRA).
Jianlin Su, Jiarun Cao, Weijie Liu, and Yangyiwen Ou. 2021. Whitening sentence representations for bet- semanties and faster retrieval. arXiv preprint arXiv:2103. 15316.
Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. Efficient estimation of word representations in vector space.
Haochen Tan, Wei Shao, Han Wu, Ke Yang, and Linqi Song. 2022. A sentence is worth 128 pseudo to- kens: semantic-aware contrastive learning frame- work for sentence embeddings. In Findings of the Association for Computational Linguisticx: ACI. 2022, pages 246-256 Dublin, Ireland. Association for Computational Linguistics.
Aaron van den Oord, Yazhe Li, and Oriol Vinyals. 2018. Representation learning with contrastive pre- dictive coding. arXiv preprint arXiv:1807.03744
Matteo Pagliardini, Prakhar Gupta, and Martin Jaggi. 2018. Unsupervised learning of sentence embed- dings using compositional n- n-gram features. In Pro- ceedings of the 2018 Conference of the North Amer- ican Chapter of the Association for Computational Linguistics: Human Language Technologies, Vol- une 1 (Long Papers), pages 528-540, New Orleans, Louisiana. Association for Computational Linguis- tics.
Bin Wang, C.-C. Jay Kuo, and Haizhou Li. 2022. Just rank: Rethinking evaluation with word and sentence similarities. In Proceedings of the 60th Annual Meeting of the Association for Computational Lin- guistics (Volume I: Long Papers). pages 6060-6077, Dublin, Ireland. Association for Computational Lin- guistics.
Tongzhou Wang and Phillip Isola. 2020. Under- standing contrastive representation learning through alignment and uniformity on the hypersphere. In Proceedings of the 37th International Conference on Machine Learning, volume 119 of Proceedings of Machine Learning Research, pages 9929-9939. PMLR.
Xiao Pan, Mingxuan Wang, Liwei Wu, and Lei Li. 2021. Contrastive learning for many-to toay--to-maa many mul- tilingual neural machine translation. In Proceed- ings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th Internu- tional Joint Conference on Natural Language Pro- cessing (Volume 1: Long Papers), pages 244-258, Online. Association for Computational Linguistics.
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pier- ric Cistac, Tim Rault, Rémi Louf, Morgan Funtow- icz, Joe Davison. Sam Shleifer. Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander M. Rush. 2020. Transformers: State-of-the-an natural language pro- cessing, In Proceedings of the 2020 Conference Empirical Methods in Natural Language Processing: System Demonstrations, pages 38 45. Online. Asso- ciation for Computational Linguistics.
Jeffrey Pennington, Richard Socher, and Christopher Manning. 2014. GloVe: Global vectors for word representation. In Proceedings of the 2014 Confer- ence on Empirical Methods in Natural Language Processing (EMNLP), pages 5532-1543, Doha, Qatar, Association for Computational Linguistics.
Yujia Qin, Yankai Lin, Ryuichi Takanobu, Zhiyuan Liu, Peng Li, Heng Ji, Minlie Huang, Maosong Sun, and Jie Zhou. 2021. ERICA: Improving entity and re- lation understanding for pre-trained language mod- els via contrastive learning. In Proceedings of the 59th Annual Meeting of the Association for Compu- tational Linguistics and the 1 11th International Joint Conference on Natural Language Processing (Vol- ume 1: Long Papers), pages 3350-3363, Online. As- sociation for Computational Linguistics.
Qiyu WI, Chongyang Tao, Tao Shen, Can Xu, Xi- ubo Geng, and Daxin Jiang. 2022. Pel: Peer- contrastive learning with diverse augmentations for unsupervised sentence embeddings. arXiv preprint arXiv:2201.12093.
Zhuofeng Wu, Sinong Wang, Jiatao Gu. Madian Khabsa, Fei Sun, and Hao Ma. 2020. Clear: Con- trastive learning for sentence representation. arXiv preprint arXiv:2012.15466.
Nils Reimers and Iryna Gurevych. 2019. Sentence- BERT: Sentence embeddings using Stamese BERT- networks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natu- ral Language Processing (EMNLP-IJCNLP),, pages
Jiahao Xu, Wei Shao, Lihui Chen, and Lemao Liu. 2023. DistilICSE: Distilled contrastive learning for sentence embeddings. In Proceedings of the 2023

---

Conference on Empirical Methods Natural Lan- guage Processing. Association for Computational Linguistics.
Linguistics (Volume Long Papers), pages 4892- 4903.
Kun Zhou, Beichen Zhang, Wayne Xin Zhao, and Ji- Rong Wen. 2022. Debiased contrastive learning of unsupervised sentence representations. In Proceed- ings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume I: Long Pa- pers), pages 6120 6130.
Yuanmeng Yan, Rumei Li, Sirui Wang, Fuzheng Zhang, Wei Wu, and Weiran Xu. 2021. ConSERT: con- trastive framework for self-supervised sentence rep- resentation transfer. In Proceedings of the 59th An- nual Meeting of the Association for Computational Linguistics and the 11th International Joint Confer- ence Natural Language Processing (Volume 1: Long Papers), pages 5065-5075, Online. Associa- tion for Computational Linguistics.
Peter Young, Alice Lai, Micah Hodosh, and Julia Hock- enmaier. 2014. From image descriptions to visual denotations: New similarity metrics for semantic in- ference over event descriptions. Transactions of the Association for Computational Linguistics. 2:67-78.
Jure Zbontar, Jing, Ishan Misra, Yann LeCun, and Stéphane Deny. 2021. Barlow twins: Self- supervised learning via redundancy reduction. In In- ternational Conference on Machine Learning, pages PMLR. Miaoran Zhang, Marius Mosbach, David Adelani, Michael Hedderich. and Dietrich Klakow. 2022a. MCSE: Multimodal contrastive learning of sentence embeddings. In Proceedings of the 2022 Confer- ence of the North American Chapter of the As- sociation for Computational Linguistics: Human Language Technologies, pages 5959-5969, Seattle, United States. Association for Computational Lin- guistics.
Yan Zhang, Ruidan He, Zuozhu Liu, Lidong Bing, and Haizhou Li. 2021. Bootstrapped unsupervised tence representation learning. Proceedings of the 59th Annual Meeting of the Association for Compu- lational Linguistics and the 11th International Joint Conference on Natural Language Processing (Vol- ume 1: Long Papers), pages 5168-5180.
Yan Zhang, Ruidan Hc. Zuozhu Liu, Kwan Hui Lim. and Lidong Bing. 2020. unsupervised sentence embedding method by mutual information maxi- mization. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Process- ing (EMNLP), pages 1601-1610, Online. Associa- tion for Computational Linguistics.
Yanzhao Zhang, Richong Zhang, Samuel Mensah, Xudong Liu, and Yongyi Mao. 2022b. Unsuper- vised sentence representation via contrastive learn- ing with mixing negatives. In Proceedings of the AAAI Conference on Artificial Intelligence, vol- ume 36, pages 11730-11733
Yuhao Zhang, Hongji Zhu, Yongliang Wang, Nan Xu. Xiaobo Li, and Binqiang Zhao. 2022c. A con- trastive framework for learning sentence represen- tations from pairwise and triple-wise perspective angular space. Proceedings of the 60th An- nual Meeting of the Association for Computational

---

Unsup-SimCSE
SimCSE++
Queryl: An animal biting persons finger.
#1
A dog is biting twig.
A dog bites somcone 's finger.
#2
dog bites someone finger.
A dog is biting a twig.
#3
Small black dog biting on a person 's finger.
Small black dog biting on a person 's finger.
#4
dog biting mop.
The dog biting stick.
#5
dog biting man rear
dog biting at own rear.
Query2: A man plays the violin.
#1
woman plays the violin.
man playing the violin.
#2
man plays the violin on stage.
A man plays the violin stage.
#3
man playing the violin.
A woman plays the violin.
#4
A musician playing his violin.
A sitting man playing the violin.
#5
A man plays violin while smiling.
A musician playing his violin.
Table Retrieved top-5 examples by SimCSE and SimCSE++ from Flickr30k (150k sentences).
A Alignment and Uniformity
As illustrated by Wang and Isola (2020), models have both good alignment and uniformity and usu- ally achieve better performance. Aligmment measure for representation consistency of the same input instance:
0.5
lalign
1lf(a')-((tt
(7)
(
3.2
Since we adopt dropout as augmentation, i.e. x xt and the only difference is the dropout pattern cl and 2², And Ppas indicate the positive pairs sampled from positive datasets. Uniformity is measure for representation distribution on repre- sentation space, which is defined by:
Figure 1: Lalign funiform plot of base model.
semantic information for Query @ I, and preserves the correct gender information on third person pro- noun in for Query@2.
Luniform
log
E
(8)
F.y
Panta
Where Pdata denotes whole data distribution, and x and are instance randomly sampled from dataset. Fig shows the fulifo- joint plot, following Gao et al. (2021).
B Qualitative Comparison
We also conduct the retrieval comparison between SimCSE+ and its baseline SimCSE on Flickr30k dataset (Young et al., 2014). We use 150k cap- tions from Flickr30k for images and take any ran- dom sentence as query retrieve similar sentences (based on cosine similarity measure). Examples is shown in Table 7. We find that the retrieved sen- tences from SimCSE+ of higher quality com- pared to those retrieved from SimCSE: SimCSE++ retrieves better top-1 sentences with most similar


Reviews:
[{"review_id": "3VSBq7FvdX", "reviewer": "Reviewer_3KpX", "summary": "", "questions": "Refer to the weaknesses section.\n\n", "limitations": "", "rating": 4, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "### R1: runtime efficiency \n**[Negligible overhead]**: According to our experiments, the proposed methods, i.e. off-dropout sampling, and DCL do not introduce noticeable running time overhead compared to the SimCSE baseline. The table below shows the training time for 1 epoch iteration on the Wiki1m corpus.\n\n| **Model** | **Training Time** |\n|---|---|\n| SimCSE | 1h 50 min |\n| SimCSE++ | 1h 59 min |\n\n**[More data will not achieve better results]** Moreover, under unsupervised settings, both SimCSE and our proposed SimCSE++ converge to the optimum within the first 5k training steps, which is around 30 minutes of training. Since SimCSE is sensitive to batch size (batch size 64 gives the best performance), as reported in its original paper and our experimental observations, increasing more data will not contribute to model performance.\n\n### R2:  limited size of the training and model sizes\n**[Our method also improves the large model baseline performance]** As illustrated in the previous studies, more data will not achieve better results due to the unsupervised learning nature [1]. Indeed, increasing the model scale contributes to the performance, as demonstrated in our experimental results. In particular, when applying the proposed method on BERT-large, its performance still increases with the model scale (78.05% for SimCSE++-BERT-base to 79.30% for SimCSE++-BERT-large), which illustrates the contribution of our method is orthogonal to the contribution of the model scale. Therefore, we think that our method is effective when continuously enlarging the model size.\n\n### R3: The clarity of presentation\nThanks for the feedback, we will revise the equations for clarity.\n\n### Reference\n[1] Tianyu Gao, Xingcheng Yao, and Danqi Chen. 2021. SimCSE: Simple Contrastive Learning of Sentence Embeddings. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 6894–6910, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "A1. The paper shed light on the potential issues with a common sentence embedding representation algorithm. This itself will allow future work to improve the framework or design better algorithms. Also, the investigation done by the paper provide valuable insights into the inner working of such algorithms.\n\nA2. Although I'm more excited about the empirical investigation part of the paper, the proposed solutions achieve improve upon the original SimCSE, moreover the these solutions seem to be general and can be applied as an auxiliary objectives on top other algorithms.\n\nA3. The evaluation setup includes many baselines which helps to put the results in the context.", "reasons_to_reject": "I didn't find any major weakness with this paper. But the following could be used to improve the work. *though this is not my main area of expertise and I'm not very familiar with all the work in this line.*\n\nR1. The paper does not provide any runtime efficiency metrics for SimCSE++. It seems there are many reduction operation happening in the DCL objective. It's not clear if training with DCL objective is slower or not. If it's slower then one could argue that the time spend on computing DCL objective can be instead used to train on more data/iteration which can further improve the performance. \n\nR1. One limitation of the evaluation setup could be the limited size of the training and model sizes. It's convincible that with enough compute and data the issue tackled by the paper be mitigated or not be contributing to much.\n\nR2. The clarity of presentation in the paper can be improved. The mathematical notation is hard to follow especially in lines 262 and 468.", "questions_to_authors": "Refer to the weaknesses section.\n\n", "paper_topic_and_main_contributions": "This work first investigates the potential weaknesses in the common practices used in the self-supervised sentence embedding framework, SimCSE. First, they point to the dropout used in the contrastive loss. Through empirical investigation, the paper finds the noise to be useful mostly for positive pairs. Secondly, the paper addresses the \"Corruption Issue\" where the components of the representations become highly correlated. Typically, BarlowTwins objective is added to mitigate this issue. However, the paper finds an intrinsic issue (rank bottleneck) with the this method. For both of these problems, the paper proposes simple solutions that is shown to be effective in improving the SOTA. The training is done a subset of wikipedia sentences (typical in this field) and the evaluation is done on sentence similarity tasks (STS).", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "Line 254 second $z_i^{1,+}$ -> $z_i^{2,+}$\n\nLine 267: $\\langle z_i^1, z_i^1 \\rangle$ -> $\\langle z_i^1, z_i^2 \\rangle$"}, "extra_text": {}}, {"review_id": "5jVJKq06xe", "reviewer": "Reviewer_Amxx", "summary": "", "questions": "Is _feature corruption_ the commonly used name for the issue?  This name of the issue is not mentioned by Zbontar et al. (2021).\n\nWill you release the source code upon acceptance?", "limitations": "", "rating": 4, "confidence": 5, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "### R1:  Imprecise uses of terminology.\nThanks for the valuable feedback, we will polish the sentences and revise the terminology and equations for clarity, and we will release the source code once the paper is accepted.\n\n### Q1: Is feature corruption the commonly used name for the issue? \n\nFeature corruption, also known as *“feature/representation degeneration/collapse”*, is one of the fundamental problems in contrastive learning. Such a question originates from contrastive learning in images [1]. Since the studies for feature corruption are many and mainly in images, here, we only cite the paper that directly tries to address such an issue. We will add the footnote here to explain the feature corruption phenomenon for better understanding.\n\n### Q2: release the source code\nYes, we will release the source code with checkpoints and add the link to the content of the paper when it is published.\n\n### Reference\n[1] Chen, Xinlei, and Kaiming He. \"Exploring simple siamese representation learning.\" Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2021.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "I would like to extend my congratulations to the authors for successfully identifying two commonly overlooked aspects of SimCSE and for providing compelling solutions to address these issues. The authors have effectively presented the two main problems of SimCSE along with empirical evidence to support their arguments. Moreover, the suggested solutions are meticulously explained in great detail. Undoubtedly, the findings of this study will greatly benefit the research community.", "reasons_to_reject": "Unfortunately, this paper is not well written, as there are numerous typos and grammatical errors throughout. Additionally, there is an unfinished sentence present (caption of table 1). Consequently, reading and comprehending the paper becomes quite challenging.\n\nMoreover, there are also numerous imprecise uses of terminology. These misuses create confusion while reading and can hinder future work.\n- [241] Eq. (2) involves 2xN _BERT structures_.  ---> BERT representations\n- [249] we respectively add more noise (+Noise) or reduce some noise (-Noise) from z...\n-> we respectively add **more** noise (+Noise) or **less** noise (-Noise) **to** z...\n- Is _feature corruption_ the commonly used name for the issue?  This name of the issue is not mentioned by Zbontar et al. (2021).\n\nThe authors also did not mention whether the source code will be released publicly.  This will make the reproduction process much more difficult.", "questions_to_authors": "Is _feature corruption_ the commonly used name for the issue?  This name of the issue is not mentioned by Zbontar et al. (2021).\n\nWill you release the source code upon acceptance?", "paper_topic_and_main_contributions": "This paper identifies two issues with SimCSE and proposes two solutions.\n\nFirst, the application of dropout has not been thoroughly investigated. SimCSE only investigated the use of standard dropout as noise and only examined the influence of dropout rate. This paper further investigates the effect of dropout noise level by adding different amounts of noise to the positive and negative samples. They find that while adding some level of noise to the positive samples is beneficial, adding dropout noise to negative samples is purely harmful to performance. Therefore, they propose using off-sample for dropout noise. They avoid adding noise to the negative samples and maintain the standard dropout noise for positive samples.\n\nSecond, the authors argue that SimCSE has an issue with feature corruption. They describe the issue as the high similarities in dimensions of a model's representation. They propose addressing this issue by applying the recently proposed Barlow Twins method.\n\nBy combining these two methods, they are able to significantly increase the performance of SimCSE. The two methods are also compatible with newer SimCSE-based methods, such as DiffCSE.", "excitement": 4, "reproducibility": 2, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "This paper has several grammatical mistakes.  Here are some common ones:\n- Typically a comma is placed after an in-sentence equation, and the following clause explaining the equation should start with the lower case.  Take equation (3) as an example:\n$$ \\ell = ... (3), $$ **w**here $s$...\n- A space should be added between a bracket and its preceding word:\nJust to name a few occurrences:\n\n[503] as previous studies(Gao et al., 2021; ...\n\n[510] 2021-2016(Agirre et al., 2012)\n\n- Please double check capitalisation.  The first letter of each sentence should be capitalised.\n\n[286] ... performance. 2) **t**he model...\n\n[264] variable $\\xi$. and thus... -> the period should be a comma.\n\nOne-off typos:\n- [266] missing . in ... Eq (2) ...\n- [259] extra . after footnote mark 1.\n- [083] Need a space after the bracket in \"...learning (DCL)to break\"\n- [265+] Missing period in the sentence in footnote 1.\n\n\nPlease proof read the paper more carefully upon acceptance.  The aforementioned typos and mistakes are by no means exhaustive."}}, {"review_id": "mZanr9TnY2", "reviewer": "Reviewer_RiEf", "summary": "", "questions": "Can you make a comparison with the text embedding APIs provided by OpenAI?( https://platform.openai.com/docs/api-reference/embeddings)", "limitations": "", "rating": 4, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "### R1: Only experimenting on 1 million corpus and BERT-base/large architecture; the scalability of the proposed method \n**[1M wiki corpus is the default data setting typical in this field]** The wiki 1 million data used in this paper originated from SimCSE paper, and many proposed methods will utilize this corpus for experiments. Here we directly inherit this corpus for a fair comparison with previous methods.\n\n**[SBERT]** Besides BERT structure, we also introduce SBERT structure in the experiment sections in the paper, which follows previous studies [1].\n\n**[Scalability]** In principle, our proposed method is scalable to data scale (batch size) and model size (BERT-base to BERT-large), and our experiment shows that performance increases with the model size. Therefore, our method is generic to scalability to LLMs. However, fine-tuning LLMs is difficult but could be tackled by some specific technique, which remains as future work.\n\n### Q1:  comparison with the text embedding APIs provided by OpenAI\n**[On par performance to LLMs but low training cost and inference overhead]** Obtaining the sentence embeddings from large language models, i.e. Chat-gpt and GPT-4 is impossible since Open-AI does not provide such an API for model logits. \nHowever, they provide the embedding API from “text-embedding” series model and report the corresponding performance (please refer to the link: https://openai.com/blog/new-and-improved-embedding-model). Their average performance for STS tasks is around 80.5 on STS12-16 testing set, and such a performance is comparable to our large model.\nMoreover, compared to them, our method is of significantly low training cost. Besides, since our model size is much smaller than LLMs, we have a significantly shorter inference overhead.\n\n### Reference\n[1] Yuhao Zhang, Hongji Zhu, Yongliang Wang, Nan Xu, Xiaobo Li, and Binqiang Zhao. 2022. A Contrastive Framework for Learning Sentence Representations from Pairwise and Triple-wise Perspective in Angular Space. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 4892–4903, Dublin, Ireland. Association for Computational Linguistics.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. This work propose two important issues which have long existed in the Contrastive Learning-based representation but ignored by previous work. \n2. The proposed two strategies are simple yet effective, leading to improvement on several benchmark datasets;\n3. The authors provide in-detail analysis for their methods which are insightful and inspirational;\n", "reasons_to_reject": "Only experimenting on 1 million randomly sampled sentences and BERT-base/large architecture. In the context of large-language model, the scalability of the proposed method are not mentioned.", "questions_to_authors": "Can you make a comparison with the text embedding APIs provided by OpenAI?( https://platform.openai.com/docs/api-reference/embeddings)", "paper_topic_and_main_contributions": "This paper propose to improve the SimCSE method from two perspectives, namely handling dropout noise and addressing feature corruption. They propose two strategies, namely off-dropout and dimension-wise contrastive learning, to alleviate the dropout noise and feature corruption. \nExperiment results on several benchmark datasets verify the effectiveness of this method. \nThis work also provides in-detail analysis for the proposed strategies.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This is a strong paper on improving sentence representations. The authors start with "standard" contrastive learning and present two improvements. The paper presents an intuitive rationale and solid empirical results.

Ideally the paper would have been more polished at submission time. I don't see any reason to believe that the authors will have issues fixing the wording suggestions raised by the reviewers (and thank you to the reviewers for the thorough reviews!).

The paper would benefit from an analysis of why the method works. Solid empirical results are certainly a strength of the paper, but I recommend the authors explore when the method works better than the "standard" contrastive learning. This analysis will be limited by the choice of task (only sentence similarity), but I think it would be interesting. Does their method perhaps work better when syntactic structure is different? Is it better at subject replacements?
Decision: Accept-Main