OssCSE: Overcoming Surface Structure Bias in Contrastive Learning for Unsupervised Sentence Embedding
Authors: Zhan Shi
Guoyin Wang
Ke Bai
Jiwei Li
Xiang Li
qingjun cui
Belinda Zeng
Trishul Chilimbi
Xiaodan Zhu
Venue: EMNLP 2023 Main (2023)
Keywords: Unsupervised Sentence Embedding
Contrastive Learning

Abstract: Contrastive learning has been demonstrated effective in unsupervised sentence representation learning. Given one sentence, positive pairs are obtained by passing the sentence to the encoder twice using the different dropout masks, and negative pairs are obtained by taking another sentence in the same mini-batch. However, the method suffers from the surface structure bias, i.e., sentences with similar surface structures will be regarded as close in semantics while sentences with dissimilar surface structures will be viewed as distinct in semantics. This leads to the result that paraphrasing a sentence that is dissimilar in surface structure will receive a lower semantic similarity score than inserting a negative word into the sentence. In this paper, we first verify the bias by collecting a sentence transformation testset. Then we systematically probe the existing models by proposing novel splits based on benchmark datasets in accordance with semantic and surface structure similarity. We tackle the bias in two aspects: balancing the learning target by augmenting with data that counters the bias, and meanwhile preserving word semantics by leveraging recall loss to prevent catastrophic forgetting. We evaluate our model on standard semantic textual similarity (STS) tasks using different pre-trained backbones and achieve state-of-the-art averaged performance across the STS benchmarks. Particularly, our models that are fine-tuned with $RoBERTa_{base}$ and $RoBERTa_{large}$ achieve significantly better performance on most benchmark datasets.

OssCSE:
Overcoming Surface Structure Bias in Contrastive
Learning for Unsupervised Sentence Embedding
Zhan Shi
Guoyin Wang
Bytedance zhan. shi 1@bytedance.com com
Bytedance
guoyin. wwwg@byyedancee.commm
Ke Bai
Jiwei Li Zhejiang University
Xiang Li
Qingjun Cui
Duke University
Amazon
Amazon
Belinda Zeng
Trishul Chilimbi
Xiaodan Zhu Queen's University
Amazon
Amazon
representing the semantics, and the surface struc- ture being the actual sentence see. The deep fective in unsupervised sentence representation learning. Given one sentence, positive pairs are obtained by passing the sentence the en- coder twice using the different dropout masks, and negative pairs are obtained by taking an- other sentence in the same mini-batch. How- ever, the method suffers from the surface struc- ture bias, i.e., sentences with similar surface structures will be regarded as close in semantics while sentences with dissimilar surface struc- tures will viewed as distinct semantics. This leads the result that paraphrasing sen- tence that is dissimilar in surface structure will receive lower semantic similarity score than inserting negative word into the sentence. In this paper, we first verify the bias by collect- ing a sentence transformation testset. Then we systematically probe the existing models by proposing novel splits based on benchmark datasets in accordance with semantic and sur- face structure similarity. We tackle the bias in two aspects: balancing the learning target by augmenting with data that counters the bias, and meanwhile preserving word semantics by leveraging recall loss to prevent catastrophic forgetting. We evaluate our model on standard semantic textual similarity (STS) tasks using different pre-trained backbones and achieve state-of-te tie-theeatt art averaged performance across the STS benchmarks. Particularly, our mod- els that are fine-tuned with RoBERTatase and RoBERTa achieve significantly better per- formance on most benchmark datasets.
Abstract
Contrastive learning has been demonstrated ef-
structure of a linguistic expression is construct seeking to unify several related surface structures. particular deep structure can be represented by multiple surface structures, e.g., "I purchased some beautiful clothes" and "Some beautiful clothes were bought by me" are two surface structures that can be unified by the same deep structure.
Recent studies (Gao et al., 2021; Zhang et al., 2021a.b) show that contrastive learning schemes (Chopra et al., 2005) that are fine-tuned on deep Transformer-based language models pre- trained large generic corpora, e.g., BERT (De- vlin et al., 2018) and RoBERTa (Liu al., 2019), have significantly benefited the sentence represen- tation learning. The idea of contrastive learning is that positive and negative pairs are constructed given a mini- batch of sentences in training. Specif- ically, for unsupervised sentence representation learning, the positive pair is composed of two iden- tical sentences and the negative pair is constructed by one sentence with another sentence in the mini- batch. However, this method for forming positive and negative pairs may lead an inherent bias that the sentences with almost, not completely, iden- tical surface structures' have the same semantic meaning while the sentences with dissimilar sur- face structures have distinct semantic meanings, leading to the result that semantic meanings are consistently biased towards with surface structure similarities. The bias presents two intermediate questions: (1) To what extent do current models suffer the effects of bias? (2) How to boost the model performance by overcoming the bias?
1 Introduction
Deep and surface structures (Chomsky, 2009, 2014; Aarts et al., 2018; Xiao et al., 2023) for linguistic expression, e.g., sentence, are often used as sim- ple binary opposite terms, with the deep structure
To answer the first question, we first propose validate whether current models can correctly rank
Work done while interning at Amazon
The inputs are not identical with different dropout masks.

---

learning to preserve the word semantics from the pre-trained models.
DS Sim
Low
High
SS Sim
Low
Cont.
Oppn.
Our methods significantly outperform the baselines, achieving the state-of-the-an av- eraged performance across the benchmark datasets under the standard metric. We pro- vide detailed analyses on how the bias is me- diated.
High
Oppn.
Cont.
Table I: Two settings (Cont. and Oppn.) by Surface Structure Similarity (SS Sim) and Deep Structure Simi- larity (DS Sim) for a pair of sentences. Cont. and Oppn. stand for consistence and opposition, respectively.
Related Work
a few sentence transformations. Furthermore, to systematically evaluate how the bias effect exist- ing models, we split the existing datasets following the Consistency (Cont.) and Opposition (Oppn.) settings by surface structure and deep structure sim- ilarity in Table 1, and then perform evaluations on the two settings accordingly. We find that models with unsupervised training will assign higher scores to negations compared to paraphrases and there is a noticeable difference in performance between Cont. and Oppn. splits.
Unsupervised sentence representation learning has been widely studied. The early study attempts to leverage sentence internal structure (Socher et al., 201 I; Hill et al., 2016; Le and Mikolov, 2014), and sentence context (Kiros et al., 2015) to learn the representation. Pagliardini et al. (2017) proposes Sent2Vec, a simple unsupervised method that ob- tains sentence embeddings by combining word vec- tors with n-gram embeddings. Several efforts have been made to obtain sentence representations using the embedding of some special token or averagely pooled last layer representations, due to the great success of pre-trained language models (Devlin et al., 2018; Liu et al., 2019). However, Ethayarajh (2019) identified the anisotropy problem that the native embeddings from PLMs are concentrated in a small cone in the vector space. To address this issue, techniques such BERT-flow (Li et al.. 2020) and BERT-whitening (Su et al., 2021) have been proposed as post-processing methods. Recently, unsupervised sentence embeddings have utilized contrastive learning schemes to fur- ther boost the performance by different data augmentation methods, such as dropout (Gao et al., 2021; Yan et al., 2021), noise-based neg- atives (Zhou et al., 2022; Zhang et al., 2022a), deletion-reeorderig--uubtttii substitutonnniinii (Wu et al.. 2020), adversarial attack (Yan et al., 2021), span sam- pling (Giorgi et al., 2021), entity augmenta- tions (Nishikawa al., 2022) and BERT layer sampling (Kim et al., 2021). In addition to the above research that create pairs of positive and neg- ative representations in different ways, there have been research focusing on improved learning objec- tives, Zhang et al. (2021b) proposed bootstrapped loss with moving average of the online network. Zhang al. (2022b) proposed contrastive loss in angular space. Chuang et al. (2022) added an additional cross entropy loss based on the differ- ence between the original and the transformed sen- tence. Jiang et al. (2022) leveraged the technology
To answer the second question, since the ma- jor bottleneck of current models is their poor per- formance on the Oppn. datasets, we apply two strategies: (1) automatic sentence-level data aug- mentations in accordance with the Oppn. setting with max margin loss applied on the augmented data; (2) regularization loss to prevent catas- trophic forgetting in token-level semantics which is critical to constitute sentence meanings. In this work, we use match error rate (MER) (Morris et al.. 2004) to measure the surface structure similarity, which is based on the edit distance (Levenshtein et al., 1966). Edit distance is often used for quan- titatively capturing the linguistic characteristics of paraphrase pairs (Liu al., 2022; Androutsopoulos and Malakasiotis, 2010; Magnolini. 2014).
In short, we make the following contributions:
We investigate the surface structure bias in contrastive learning for the unsupervised sen- tence representation and systematically evalu- ate the bias by constructing datasets following the two designated settings on surface struc- ture and deep structure similarity.
We overcome the bias by leveraging data aug- mentation according the Oppn. setting, and then use the max margin loss to incorporate these data in the contrastive learning frame- work. We also use an additional regularization loss to reduce the catastrophic forgetting in

---

07
0.
0.
0.8
08
0.7
0.3
0.6
02
0.3
0.6
238
Unier_Parpptrus
Sup- Paraptrase
Undup, Nugall Sup, Nagall
UnSSup_
UrSup, Negation
())
(b)
Figure Semantic Similarity Score (Horizontal Axis) and Match Error Rate (Vertical Axis) all constructed examples. Sub-figure (a) compares the paraphrases by UnSup and Sup models; Sub-figure (b) compares the negations by UnSup and Sup models; Sub-figure (c) compares paraphrases and negations by the UnSup model.
larity values between the original and transformed sentence calculated by both unsupervised and su- pervised models, and an MER value that measures surface structure similarity.
66
55
The MER value (range from ~ 1) indicates the proportion of words that were incorrectly deleted, substituted and inserted: the lower the value, the higher the surface structure similarity.
62
Negeton
Urrssp BSu
MER S+D+I 5+0+1+C
Figure 2: Averaged Semantic Similarity Scores of sen- tence transformations based on all examples.
where S. D, and C are the number substitu- tions, deletions, insertions and correct words, re- spectively.
of template denoising in the contrastive loss. Al- though achieving promising results, these methods didn't 't consider reducing the surface structure bias for contrastive learning in their data augmentation methods modified learning objectives.
Example 1 Sentence Transformations of "Bryan Cranston will return as Walter White for breaking bad spin off, report claims."
(i) Paraphrase: It has been reported that Bryan Cranston will reprise his role Walter White in a spin-off of Breaking Bad. (UnSup: 0.72, Sup: 0.92,
3 Probing
3.1 Sentence Transformations
MER: 0.73)
To specifically exhibit the bias, we write down seven differently transformed sentences as shown in Example 1 considering six types of sentence transformations: (1) Paraphrasing sentence with dissimilar surface forms; (2) Contradicting a sen- tence by making minor changes; (3) Inserting few tokens while keeping the meaning unchanged; (4) Deleting few tokens while keeping the mean- ing unchanged: (5) Substituting few tokens while keeping the meaning unchanged; (6) Changing to another random sentence.
(û) Negation: Bryan Cranston will not return as Walter White for Breaking Bad spin off. report claims. (UnSup: 0.96, Sup: 0.75, MER: 0.06) (iii) Deletion: Bryan will return as Walter White for Breaking Bad spin off, report claims. (UnSup: 0.97, Sup: 0.95, MER: 0.07)
(iv) Insertion: Bryan Cranston will return as Wal- ter White for breaking bad spin off. a latest report claims. (UnSup: 0.95, Sup: 0.98, MER: 0.12) (v) Substitution: Bryan Cranston will come back as Walter White for Breaking Bad spin off. report claims. (UnSup: 0.97, Sup: 0.98, MER: 0.13) (vi) Random: Digital era threatens future of drive- ins. Uessp: 0.40, Sup: 0.08, MER: 1.0)
We probe the officially released unsupervised and supervised models with Bert-base-uncased as the backbone from SimCSE (Gao et al., 2021)².
Each sentence has three associated values: simi-
Negation vs. Paraphrase First of all, Example shows that unsupervised-leamiine based (UnSup)

---

Datasets
Cont.
Oppn.
Median human scores
Median MER scores
STS2012-MSRvid
406
344
2.4
0,444
STe2013-headline
495
255
2.6
0.625
STS2013-OnWN
286
275
2.4
0.556
S7S50144-cfftfomu
251
199
2.6
0.4375
STS2014- headlines
483
267
3
0.6
STS2014-images
461
289
3.2
STS201 17S2005-answers-Sii answers-students
494
256
0.75
S-eadins
517
233
2.6
0.625
STS201 -immagss
548
202
2.5
0.538
1922016-nsswrr-nsseer
134
120
STS2016-
160
89
0.625
11$2006-plagiarisie
153
77
0.737
STS201 6-sosti--osi
180
64
3
0,416
11$2006-guestioon-uusttoo
75
134
2
STSB-train
3532
2217
3
0.536
STSB-test
855
524
2.8
0,516
Table 2: Statistics of the constructed Cont. and Oppn. datasets and splitting criteria.
Cont. (UnSup) Cont. (Sup) Oppn. (UnSup) Oppn. (Sup)
Weighted Average
0.8574
0.8891
0.4982
0.6152
Table 3: Weighted Average Performance on Cont. and Oppn. datasets by UnSup and Sup models.
models will mistakenly assign higher scores to Negation (0.96) compared to Paraphrase (0.72),
Wikipedia, news headlines, and image descriptions. We first use automatic methods and then mamually filter these sentences to obtain the final six trans- formations for each sentence. For paraphrases, we use ChatGPT's (Ouyang et al., 2022) open API by using the template of "paraphrases of <sentence>" as the input, and among the multiple outputs, we select the sentence that is the most dissimilar from the original sentence surface structure. For nega- tions, we collect them by inserting negative words, e.g., not, neither and barely, or replacing words with their antonyms to the original sentences. For the insertions and substitutions, we use Bert-base- uncased the mask language model to replace insert around 5% to 10% of the total tokens. For deletion, randomly remove word in the sen- tence. We simply take another sentence in these sentences for the random transformation.
indicating that
wew.riiiedd-llldinggegee based (Sup)
models are biased towards surface structure simi- larity (MER of the negation is 0.06, which is much lower than the paraphrase: 0.73), while supervised- bearning-based models correctly rank the scores with Negation (0.75) and Paraphrase (0.92). One reason for this that the Sup model is trained on a natural language inference corpus that uses seman- tic contradicted sentences to build negative pairs.
Other Transformations Secondly, Example also shows that both UnSup and Sup models are able to effectively handle other types of transformations (assigning high similarity scores to sentences with low MER, and low similarity scores to sentences with high MER.), such as Insertion, Deletion, Sub- stitution, and Random. According to Table 1, the above-mentioned four transformations fall under Cont. setting, where surface structure similarity aligns with deep structure similarity. However, Negation and Paraphrase fall under Oppn. setting, where surface structure similarity contradicts deep structure similarity.
Figure further demonstrates that (1) The Un- Sup model mistakenly assigns higher scores to negations than paraphrases.
(2) The Sup model can correctly assign higher scores to paraphrases than the UnSup model. (3) The Sup model can correctly assign lower scores to the negations than UnSup model. Mcanwhile, Figure 2 shows that (1) UnSup and Sup models would give similar average scores in terms of the substitutions, insertions and deletions. (2) Random sentences would be given significantly lower semantic scores. As a result, the major bottleneck between Sup and UnSup models
3.2 Sentence Transformation Collection
To validate our findings on larger scale, gather total of 135 examples and each exam- ple shares the same transformations as Example 1. These sentences are from multiple sources, such as

---

Coanstro Water off. chime.
N
has less reperted that Beyan Cuaniic will reptiso lis role Walter White opt Brascing Bad
Couraro
Wilter Whin
A
nsub)
Acomp
(key
Had spin off, chime
PRPS My
NN dog
VHZ. likes
VBG eating
NN sausage
This was pubiided pages
Figure 4: dependency parsing tree example
lower than that of Cont. (0.8574). On the other hand, the major improvement from the Sup model in comparison to UnSup model is related to Oppn. dataset splits, which achieves increase of around 12% from 0.4982 to 0.6152 (Refer Appendix C for detailed results on all splits).
M
Contrastive Loss
Max Marginal Loss
Figure 3: An overview of the augmentation method. 8i, s 8,, are the original sentence, its paraphrase, its negation, and another sentence in the mini-batch. These sentences are used for different losses.
Method
is mainly related to transformations that fit into the Oppn. setting.
Unsupervised Contrastive Learning Given a dataset of paired sentences {s,, s. } where 8; and sft are semantically similar and regarded positive pair. The main idea behind unsupervised contrastive learning is utilize identical sentences to construct positive pairs. i.e., si Sj. and ran- dom two sentences in mini-batch as negative pairs. We feed s; to the encoder twice by applying dif- ferent dropout masks in each forward pass and ob- taining two sentence embeddings h4 ho as shown in Figure 3. With h2 and h for each sentence in mini-batch with batch size N, the contrastive learning objective is formulated as below:
3.3
Dataset Splits
As another step to evaluate the bias, we design new dataset splits based on two settings as illustrated by Table 1. The construction of Cont. and Oppn. dataset splits will help to prohe how much the ex- isting models are impacted by the bias.
More specifically, since the human annotation scores are only consistent on each dataset, we split the existing datasets following Cont. and Oppn. set- tings by surface structure similarity (MER: 1)
and semantic similarity (human annotations: 0~ 5)
on the basis of single dataset as shown Table 2. Note that all datasets in Table are from the stan- dard benchmark datasets (STS 2012~2016 (Agirre et al., 2012, 2013, 2014, 2015, 2016) and STS- B (Cer et al., 2017)). The table shows the number of Cont. and Oppn. data samples as well the split criteria (median human annotation and MER scores?), i.e., the sentence pairs that are above the median human annotation score and below the me- dian MER score, or below the median human an- notation score and above the median MER score would be categorized the Cont. setting, other- wise Oppn. setting.
sim((h
4. - -log
(2)
Ej=1 sim(la.
where 7 is temperature hyper- rrper-parameter and sim is the cosine similarity metric. Instead of using the original sentence as input and the last hidden layer of <CLS> token sen- tence representations. Following (Jiang al.. 2022), given any sentence 8, different prompts constitute positive pair. Sentence a: This sentence "s" means <mask> Sentence This sentence "s" means <mask> We use the last hidden layer of the <mask> token as the sentence representation. With the prompt, the performance would be higher with lower variance. Data augmentation Since the major bottleneck is the poor performance the Oppn. dataset splits as the current UnSup models fail correctly rank the negation and paraphrase transformations, we apply data augmentation based on these two sen- tence transformations and then leverage max mar- gin loss between negations and paraphrases so that
two slightly
We perform probing each above dataset by the standard spearman's correlation relationship between the model computed scores and human an- notated scores. As shown Table 3, there a gap in performance between Cont. and Oppn. splits for the UnSup model, and the weighted average per- formance of Oppn. splits (0.4982) is significantly
We only consider that datasct whose median human anno- tation score is between 2 and 3.5 to remove biased datascts

---

60
07
80
85
45
Berl-Baae
BarsLarge
RoBERT R-loo
RuBERTaeerg
Batt-Baoe
BerkLurge
RuéERTeebae
RuSEHTa-wop
Buseline
Boseline
Figure 5: The comparison of averaged performance Cont. (left) and Oppn. (right) dataset splits.
negations and paraphrases can be ranked properly. Following (Gao et al., 2021), we use the same one million sentences from English Wikipedia for un- supervised learning.
performance degradation on Oppo. dataset splits after 3,000 training steps (0.3 epoch). It might be due to the catastrophic forgetting of word mantics and Oppn. splits are more relied word semantics rather than surface structure. However, the fine-tuning process with pre-trained language models prone to catastrophic forgetting, and it will cause significant drop in performance on the Oppn. dataset splits particularly. We use the learning objective from (Chen et al., 2020), which proposes pretraining simulation mechanism to recall the knowledge from pretraining tasks Ds without data.
Step Generating a semantically contradicted ex- ample sentence s by only inserting negative words to the original sentence Ss, e.g., My dog likes eating sausage + My dog does not like eating sausage. Given the sentence, we use dependency parsing to find the position of the word that is tagged as root shown in Figure 4. Then insert the negative word based the pos tag of the root word, e.g., if its pos tag is VBZ, then we change "likes" "does not like".
-looo = ~0, (?)
2
Step 2: Given sentence 8,, we aim to generate a sentence se that is similar in semantics but dis- tinctive in surface structure. We use back transla- tion models on German-English" to obtain such candidate sen- tences. To filter the desired back-translation sen- tences that satisfy the requirement of surface struc- ture similarity, we then select the examples of which their MER meets the requirement (refer to Appendix A for details)
where ? is a and 8* the initial parameters of the pre-trained models. (Refer to Appendix for background of the recall loss). Overall, the loss function for our method is as below:
for this recall loss
English-
f= 1. I. Ale
Experiments
Hence, we propose the max margin loss based on the h. and he representations from sf and s/:
5.1 Experiment Setup
We perform experiments with backbones of RoBERTa Tapases RoBERTatorge (Liu et al., 2019),
In = max((im((h bT) sim(hs, Af) 0) max(maimm))))) hf) sim(h, k3) 8, 0)
BERT base. and BERTtarge (Devlin al., 2018).
We use semantic textual similarity tasks: STS 2012-2016 (Agirre et al., 2012, 2013, 2014, 2015,
Although h. is semantically contradictory to the original sentence, it is close to the original one in terms of semantic relatedness. Hence, design the above two-way max marginal loss, which can guide the model learn the following objective: a < sim(h,, h?) sim(hs, hT) < where a. are margins.
2016), STS Benchmark (Cer et al., 2017) and SICKRelatedness (Marelli et al., 2014). STS12- STS16 datasets do not have train or development sets, and thus we evaluate the models on the de- velopment set of STS-B to search for better set- tings of the hyper our method on the 7 transfer learning bench- mark tasks (MR (Pang and Lee, 2005), CR (Hu and Liu, 2004), SUBJ (Pang and Lee, 2004), MPQA (Wiebe et al., 2005), SST-2 (Socher et al.,
We also evalu-
Catastrophic forgetting As shown in Figure 6, we also observe that the training will suffer from
https://huuginnffcc..co/cccaoooo wn-nu_wm11--oo-nn www.:m/m--ddooo..mt19999900 wma 9-en-de

---

Method
STS12
STSI3
STS14
STS15
STSI6
STS-B
SICK-R
Avg.
RRRSSERRRRRTaaa labase
70.16
81.77
73.24
81.36
80.65
80.22
68.56
76.57
DifICSERROERK RoBERTa
70.05
83.43
75.49
82.81
82.12
82.38
71.19
78.21
DCLR-RoBERTaas
70.01
83.08
75.09
83.66
81.06
81.86
70.33
77.87
DSimCSF-RoBERTIIE Rabase
69.90
82.50
74.68
83.19
80.30
80,99
70.54
77.44
Prompt-RoBERTia..o
73.94
84.74
77.28
84.99
81.74
81.88
69.50
79.15
PCI.-RORERTayees
71.13
82.38
75.40
83.07
81.98
81.63
69.72
77.90)
(Ours)-OssCSE RoBERTabse
72.28
85.27
79.51
84.77
82.32
83.55
75.54
80.46
68.40
82.41
74.38
80.91
78.56
76.85
72.23
76.25
ArcCSE-BER BERT
72.08
84.27
76.25
82.32
79.54
79.92
72.39
78.11
DifICSE-BER SERSE-BERbbb...
72.28
84.43
76.47
83,90
80.54
80.59
71.23
78.49
DCLR-BER
70.81
83.73
75.11
82.56
78.44
78.31
71.59
77.22
ESimCSE-BERTI
73.40
83.27
77.25
82.66
78.81
80.17
72.30
78.27
Prompt-BERI buaec
71.56
84.58
76.98
84.47
80.60
81.60
69.87
78.54
InfOCSE-BERT.......
70.53
84.59
76,40
85.10
81.95
82.00
71.37
78.85
PCI.BERR base
72.84
83.81
76.52
83.06
79.32
80.01
73.38
78.42
(Ours)-OssCSE BERT have
71.78
84.40
77.71
83.95
79.92
80.57
75.25
79.08
72.86
83.99
75.62
84.77
81.80
81.98
71.26
78.90
RELL-RoBERTaaarae
Tatarge
73.09
84.57
76.13
85.15
81.99
82.35
71.8
79.30
RSimCSE-RoBBE
73.2
84.93
76.88
84.86
81,21
82.79
72.27
79.40
OssSS RoBERTaturge
74.56
85.54
79.88
86.64
82.10
84.42
78.65
81.68
SimCSE MAMCSE-BERTT.o.
70.88
84,16
76.43
84,50
79.76
79.26
73.88
78.41
AreCSE-BER
iurge
73.17
86.19
77.90
84.97
79.43
80.45
73.50
79.37
OCLR-BERT(car. iarge
71.87
84.83
77.37
84.70
79.81
79.55
74.19
78.90
ESimCSF--BEE BAAnnSSE-BRTTurree Turgg
73.21
85.37
77.73
84.30
78.92
80.73
74,89
79.31
InfoCSE-BERI BioCSEE-BERTrrrgg (arge
71.89
86.17
77.72
86.20
81.29
83,16
74.84
80.18
(Ours)-OsCCSF BERT(arge targe
72.64
86,36
79.16
85.04
80.80
82.61
76.65
80.47
Table The performance comparison of our methods with different backbones on sentence similarity tasks. The
performance is based on the default random seed. The baseline results are from their published papers.
Method
MR
CR
SUBJ
MPQA
SST
TREC
MRPC
Avg.
SimCSE-RoBBER RoBERTa
81.04
87.74
93.28
86.94
86.60
84.60
73.68
84.84
SimCSE-RoBB RoBERTasase Prompt
81.01
86.85
93.45
88.42
84.90
87.10
74.36
85.16
(OsSS)-OsSCSE RoBERTasase
82.88
88.13
93.53
90.54
87.26
88.80
77.97
87.02
Table 5: The performance comparison of our methods with RoBERTa-base on the transfer tasks.
2013), TREC (Voorhees and Tice, 2000) and MRPC (Dolan and Brockett, 2005)). The Sentlval toolkit is used for evaluation, and the Spearman correlation coefficient is used as the standard eval- uation metric. We also perform experiments on the dataset splits constructed by the Cont. and Oppn. settings in Table 2 to validate whether surface struc- ture bias has been alleviated. Refer to Appendix A for implementation details.
Sentence Similarity Tasks We show the sults of STS tasks in Table 4. OssCSE fine- tuned with different pre-trained models can sig- nificantly outperform all the baselines, partic- ularly with RoBERTaMee and RoBERTatorge EXSCSE-RRBERTaa.y raises the averaged Spear- man's correlation from 76.57% to 80.46% com- pared with SimCSE-RRBERRTanee and OssCSE- RoBERTatarge increases the averaged Spearman's correlation from 78.90% 81.68% compared with
5.2 Results and Analyses
Meanwhile, for BERT buse and BERT larges our method can also improve upon SimCSE-BERT..... and SimCSE- BERT significantly, with an improvement 2.8% and 2.5% respectively. The reason why our approach achieves greater improvement on RoBERTa than BERT backbones is that RoBERTa has been demonstrated have learned better word representations(LLu al., 2019), and thus mak- ing the recall loss more effective on the RoBERTa
Compared Baselines We mainly choose SimCSE for comparison, since we build our method based on it and shares the same setting in our approach. We also compare our results with several most recent strong work below: PromptBert (Jiang al., 2022), ArcCSE (Zhang al., 2022b), Dif- ICSE (Chuang et al., 2022), InfoCSE (Wu al., 2022b), ESimCSE (Wu et al., 2021). DCLR (Zhou et al., 2022), and PCL (Wu et al., 2022a).

---

045
04
136
0M
1.00
1902
15000
50000
Setting
Setting a (12)
Setting
Setting (12)
Figure 6: Training checkpoints on Cont. and Oppn. splits of STSB-dev, and w/o the recall loss (12).
01.5
50
Bose
Piote!
Promer&
Picrpt&&
Phorpt
Base
Pionel
W Pronnpt &
1 Promor
Vt Phompr
Figure 7: Ablation analysis of averaged performance on the seven sentence similarity datasets with RoBERTa-base (left) and RoBERTa-large (right) as backbones, where max denotes max margin loss.
backbones.
margin loss is removed, the average performance drops from 80.46% to around 79.2%. This result shows the importance have all designated loss items that exist together in the learning objective. We also find that recall loss can help stabilize the learning process because of its effectiveness in pre- venting catastrophic forgetting. Figure shows find that the performance the Oppn. split of STSB-de becomes stable when recall loss is used. Quantitative Analysis further take the original sentence in Example 1, and write down eight para- phrases and negations (Refer to Appendix B for the specific sentence transformations). When compar- ing OssCSE-BERT base with SimCSE-BERT base. find that the averaged score of eight paraphrases increases from 0.79 0.88 and the averaged score of eight negations decreases from 0.93 to 0.87.
Transfer Tasks We show the results of trans- fer tasks Table with RoBERTatee as back- bone. Compared with SinCSE-RoBERTawsee our approach can improve the averaged score from 84.84% to 87.02%. We find that using recall loss alone can greatly enhance performance on tasks involving transfer learning. (85.06% to 87.12%), confirming that catastrophic forgetting prevention objectives can also benefit transfer tasks.
Cont. and Oppn. We show the results of Cont. and Oppn. dataset splits in Figure 5. The figure indicates that there are performance boost with all the four backbones. We find that the improve- ment Oppn. is more significant, specifically when using the backbones of RoBERTa and RoBERTajarge achieving an averaged improve- ment of around 8%. The improvement on Cont. dataset splits relatively smaller, with an averaged increase of around 2.5% on the all backbones.
Conclusion
We investigate the surface structure bias in con- trastive learning for the unsupervised sentence em- bedding and systematically probe the bias by con- structing datasets following the two designated set- tings on the surface and deep structure similarity. We overcome the bias by data augmentation meth- ods and then use the max margin loss to incorporate these data the contrastive learning framework. We also use a recall loss to reduce catastrophic
Ablation Analysis Figure 7 shows the ablation of averaged performance with the backbones of RoBERTabase and RoBERTatarge- In OssCSE, both the recall loss and the max margin loss are cial because they consistently improve our method. Take RoBERTamee for instance, the averaged per- formance decreases from 80.46% to around 79% if the recall loss is removed. Similarly, if the max

---

forgetting in unsupervised learning to preserve the word semanties in the pre-trained models. The results significantly outperform the baselines and achieve state-of-the-an results on averaged perfor- mance with different pre-trained backbones.
on Semantic Evaluation (SemEval 2012). pages 385- 393.
Eneko Agirre, Daniel Cer, Mona Diah, Aitor Gonzalez- Agirre, and Weiwei Guo. 2013. *SEM 2013 shared task: Semantic textual similarity. Conference on Lexical and Computational Semantics (MSEM), Volume 1: Proceedings of the Main Confer- ence and the Shared Task: Semantic Textual Similar- ity, pages 32-43, Atlanta, Georgia, USA. Association for Computational Linguistics.
Second Joint
7 Limitations
First, we cannot guarantee the quality of the back- translation results as the augmentation target of paraphrase, There might be error propagation in the forward and backward translation process. We are considering using strong paraphrase models in the future. Second, we have not considered other minor modification methods that would be consid- ered significant to semantic meanings rather than negations. Missing these parts may weaken our model generalization ability such that it may not be applicable in specific domains.
Ion Androutsopoulos and Prodromos Malakasiotis. 2010. survey of paraphrasing and textual entail- ment methods. Journal of Artificial Intelligence Re- search, 187.
Daniel Cer. Mona Diab, Eneko Agirre, Inigo Lopez- Gazpio, and Lucia Specia. 2017. Semeval-2017 task I: Semantic textual similarity-muliiinuaal and cross-lingual focused evaluation. arXiv preprint 1rXiv:7708.0000
Sanyuan Chen, Yutai Hou, Yiming Cui, Wanxiang Che, Ting Liu, and Xiangzhan Yu. 2020. Recall and learn: Fine-tuning deep pretrained language models with less forgetting. arxiv preprint arXiv:2004. 2ee::0004.1..... 12651.
References
B Aarts, Chalker, and Weiner. 2018. The oxford dic- tionary of english grammar (2014). URL hitps://en. oxforddictionaries. com/deffnitioo/leeennggeeeen Ac- cessed.
Noam Chomsky. 2009. Syntactic structures. In Syntac- fic Structures. De Gruyter Mouton.
Noam Chomsky. 2014, Aspects of the Theory of Syntax, volume 11. MIT press.
Eneko Agirre, Carmen Banea, Claire Cardie, Daniel Cer. Mona Diab. Aitor Moizalez-Agirre, Weiwei Guo, Inigo Lopez-Gazpio, Montse Maritxalar, Rada Mi- halcea, et al. 2015. Semeval-2015 task 2: Semantic textual similarity, english, spanish and pilot on inter- pretability. In Proceedings of the 9th international workshop on semantic evaluation (SemEval 2015). pages 252-263.
Sumit Chopra, Raia Hadsell, and Yann LeCun. 2005. Learning a similarity metric discriminatively. with application to face verification. In 2005 IEEE Com- puter Society Conference on Computer Vision and Pattern Recognition (CVPR'0S, 05), volume I, pages 539 546. IEEE
Yung-Sung Chuang. Rumen Dangovski, Hongyin Luo. Yang Zhang, Shiyu Chang, Marin Soljačić, Shang- Wen Li, Wen-tau Yih, Yoon Kim, and James Glass. 2022. Diffese: rifference-basedd contrastive learning for sentence embeddings. arXiv preprint arXiv:2204.1022
Eneko Agirre, Carmen Banea, Claire Cardie, Daniel M Cer, Mona T Diab, Aitor Goazzlez-Agirre Weiwei Guo, Rada Mihalcea, German Rigau, and Janyce Wiebe, 2014. Semeval-2014 task 10: Multilingual semantic textual similarity. In SemEval COLING, pages 81-91.
Jacob Devlin, Ming-W Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. Bert: Pre-training of deep bidirectional transformers for language understand- ing, arXiv preprint arXiv.1810.04805.
Eneko Agirre. Carmen Banea, Daniel Cer, Mona Diab, Aitor Gonzalez Agirre, Rada Mihalcea, German Rigau Claramunt, and Janyce Wiebe. 2016. Semeval- 2016 task 1: Semantic textual similarity. monolin- gual and cross-lingual evaluation. In SemEval-2016. 10th International Workshop on Semantic Evalua- tion; 2016 Jun 16-17; San Diego, CA. Stroudsburg (PA): ACL; 2016. p. 497-511. ACL (Association for Computational Linguistics).
Bill Dolan and Chris Brockett 2005. Automati- cally constructing a corpus of sentential paraphrases. In Third International Workshop on Paraphrasing (IWP2005).
Kawin Ethayarajh. 2019. How contextual are contex- tualized word representations? comparing the ge- ometry of bert, elmo, and gpt-2 embeddings. arXiv preprint arXiv: 1909.00512.
Eneko Agirre, Daniel Cer, Mona Diab, and Aitor Gonzalez- Agirre. 2012. Semeval-2012 task 6: A pilot on semantic textual similarity, In * SEM 2012: The First Joint Conference on Lexical and Compu- tational Semantics- Volume I: Proceedings of the main conference and the shared lask, and Volume 2: Proceedings of the Sixth International Workshop
Tianyu Gao, Xingcheng Yao, and Danqi Chen. 2021. Simese: Simple contrastive learning of sentence em- beddings. arXiv preprint

---

John Giorgi, Osvald Nitski, Bo Wang, and Gary Bader. 2021. Declutr: Deep contrastive learning for unsuper- vised textual representations. In Proceedings of the 59th Annual Meeting of the Association for Compu- lational Linguistics and the 11th International Joint Conference on Natural Language Processing (Vol- ume 1: Long Papers), pages 879-895.
of the Ninth International Conference on Language Resources and Evaluation (LREC)14) 14), pages 216- 223.
Andrew Cameron Morris, Viktoria Maier, and Phil Green. 2004, From wer and ril to mer and wil: im- proved evaluation measures for connected speech recognition. In Eighth International Conference on Spoken Language Processing.
Felix Hill, Kyunghyun Cho. and Anna Korhonen. 2016. Learning distributed representations of sentences from unlabelled data. arXiv preprint arxivv/60..03483.
Sosuke Nishikawa, Ryokan Ri, Ikuya Yamada, Yoshi- masa Tsuruoka, and Isao Echizen. 2022. Ease: Entity-aware contrastive learning of sentence embed- ding. arXiv preprint arXiv:2205.042e0
Minqing Hu and Bing Liu. 2004. Mining and summa- rizing customer reviews. In Proceedings of the tenth ACM SIGKDD international conference Knowl- edge discovery and data mining. pages 168-177.
Long Ouyang, Jeff Wu, Xu Jiang. Diogo Almeida, Car- roll I. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. 2022. Training language models follow in- structions with human feedback. arXiv preprint arXiv:200.002155.
Ting Jiang, Shaohan Huang, Zihan Zhang, Deqing Wang, Fuzhen Zhuang, Furu Wei, Haizhen Huang, Liangji Zhang, and Qi Zhang. 2022. Prompthert: Improving bert sentence embeddings with prompts. arXiv preprint 22X1::2001.03337
Matteo Pagliardini, Prakhar Gupta, and Martin Jaggi. 2017. Unsupervised learning of sentence embed- dings using compositional n-gram features. arXiv preprim arXiv:
Taeuk Kim, Kang Min Yoo, and Sang-gou goo Lee. 2021. Self-guided contrastive learning for bert sentence representations. In ACL/JCNLP (1).
Ryan Kiros, Yukum Zhu, Russ R Salakhutdinov. Richard Zemel, Raquel Urtasun, Antonio Torralba, and Sanja Fidler. 2015. Skip-thought vectors. Advances in neural information processing systems, 28.
Bo Pang and Lillian Lec. 2004. A sentimental education: Sentiment analysis using subjectivity summarization based on minimum cuts. arXiv preprint cx/0409058.
Pang and Lillian Lee. 2005. Seeing stars: Exploiting class relationships for sentiment categorization with respect rating scales. arXiv preprint cs/0506075.
Quoe Le and Tomas Mikolov. 2014. Distributed repre- sentations of sentences and documents. In Interna- tional conference on machine learning, pages 1188- 1196. PMLR.
Richard Socher, Eric Huang. Jeffrey Pennin, Christo- pher D Manning, and Andrew Ng. 2011. Dynamic pooling and unfolding recursive autoencoders for paraphrase detection. Advances in neural informa- tion processing systems, 24.
Vladimir I Levenshtein et al. 1966. Binary codes capa- ble of correcting deletions, insertions, and reversals. Soviel physics doklady, volume 10, pages 707-710. Soviet Union.
Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D Manning, Andrew Y Ng, and Christopher Potts. 2013. Recursive deep models for semantic compositionality over a sentiment treebank. In Proceedings of the 2013 conference on empiri- cal methods in natural language processing, pages 163 1631-1642.
Bohan Li, Hao Zhou, Jumxian Hc, Mingxuan Wang. Yiming Yang, and Lei Li. 2020. On the sentence beddings from pre-trained language models. arXiv preprint 2xXx:2011.008666
Timothy Liu et al. 2022. Towards better characterization of paraphrases. In Proceedings of the 60th Annual Meeting of the Association for Computational Lin- guistics (Volume 1: Long Papers). pages
Jianlin Su, Jiarun Cao, Weijie Liu, and Yangyiwen Ou. 2021. Whiteming sentence representations for bet- ter semantics and faster retrieval. arXiv preprint arXiv:2103.15316
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Man- dar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized bert pretraining ap- proach. arXiv preprint arXiv: 7907.1 1692.
Ellen M Voorhees and Dawn M Tice. 2000. Building question answering test collection. In Proceedings of the 23rd annual international ACM SIGIR confer- ence on Research and development in information retrieval, pages 200-201
Simone Magnolini. 2014. A survey on paraphrase recog- nition. DWAI@ A/* IA, 1334:33 41.
Marco Marelli, Stefano Menini, Marco Baroni, Luisa Bentivogli, Raffaella Bernardi, and Roberto Zampar- elli. 2014. sick cure for the evaluation of composi- tional distributional semantic models. In Proceedings
Janyce Wiebe, Theresa Wilson, and Claire Cardie. 2005. Annotating expressions of opinions and emotions in language. Language resources and evaluation. 3952):165-210.

---

Qiyu Wu, Chongyang Tao, Tao Shen, Can Xu, Xi- ubo Geng, and Daxin Jiang. 2022a. Pel: Peer- contrastive learning with diverse augmentations for unsupervised sentence embeddings. arXiv preprint arXiv:2201.12092 120933
Xing Wu. Chaochen Gao, Zijia Lin, Jizhong Han, Zhongyuan Wang. and Songlin Hu. 2022b. Infocse:
Information--agrreaaedd contrastive learning of sen- tence embeddings. arXiv preprint arXiv:2210.06432.
Xing Wu, Chaochen Gao, Liangjun Zang. Jizhong Han, Zhongyuan Wang, and Songlin Hu. 2021. Esimese: Enhanced sample building method for contrastive learning of unsupervised sentence embedding. arXiv preprint aiv: 109.04380.
Zhuofeng Wu, Sinong Wang. Jiatao Gu, Madian Khabsa, Fei Sun, and Hao Ma. 2020. Clear: Contrastive learning for sentence representation. arXiv preprint arXiv:2012.15466.
Chenghao Xiao, Yang Long, and Noura Al Moubayed. 2023. On isotropy, contextualization and learning dy- namics of contrastive based sentence representation learning. In Findings of the Association for Compu- tational Linguistics: ACI. 2023, pages 12266-12283.
Yuanmeng Yan, Rumei Li, Sirui Wang, Fuzheng Zhang, Wei Wu, and Weiran Xu. 2021. Consert: A con- trastive framework for self supervieed sentence repre- sentation transfer. arXiv preprint arXiv:2105. 17741.
Dejiao Zhang, Shang-Wert Wen Li, Wei Xiao, Henghui Zhu, Ramesh Nallapati, Andrew O Arnold, and Bing Xi- ang. 2021a. Pairwise supervised contrastive leamn- ing of sentence representations. arXiv preprint arX552200..05424.
Yan Zhang, Ruidan He, Zuozhu Liu, Lidong Bing, and Haizhou Li. 2021b. Bootstrapped unsupervised sen- tence representation learning.
Proceedings of the 59th Annual Meeting of the Association for Compu- tational Linguistics and the I 1th International Joint Conference on Natural Language Processing (Vol- une 1: Long Papers), pages 5168-5180.
Yanzhao Zhang, Richong Zhang, Samuel Mensah, Xudong Liu, and Yongyi Mao. 2022a. Unsupervised sentence representation via contrastive learning with mixing negatives. Proceedings of the AAAI Con- ference on Artificial Intelligence, volume 36, pages 11738
Yuhao Zhang, Hongji Zhu, Yongliang Wang, Nan Xu. Xiaobo Li, and Binqiang Zhao. 2022b. A contrastive framework for learning sentence representations from pairwise and triple-wise perspective in angular space. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 4892-4903.
Kun Zhou, Beichen Zhang, Wayne Xin Zhao, and Ji- Rong Wen. 2022. Debiased contrastive learning of unsupervised sentence representations. arXiv preprint arXXv:2205.0065:

---

A Implementation Details
(4) bryan cranston cannot return walter white for breaking bad spin off. report claims
To make fair comparison, we use exactly the same settings with SimCSE except descriptions be- low. We train the UnSup models for two epoch. We carry out the batch size of 64 for all back- bones, and learning rate of le-5 for BERT base and RoBERTaesee and 5e-6 for RoBERTa and BERTiarge- We validate evey 125 steps on STS-B development set and adopt the following hyper- parameter settings and data augmentation crite- ria: (1) We use the following hyper-parameters for the two-way max margin loss: a 0.05, 8 = 0.2, À = 1e 3 and recall loss item (7 = 2e - 3), and we find our method achieves the best perfor- mance on the STSB-dev set using the above param- eters through grid search; (2) We select the back translation results of which the MER are within 0.15 to 0.6 as the criteria of the paraphrase trans- formations. We don't use the sentences that have even higher MER values because these sentences may have been corrupted by the translation errors.
(5) bryan cransion impossible to return as walter white for breaking bad spin off. report claims (6) bryan cranston never return as walter white for breaking bad spin off report claims
(7) bryan cranston return walter white for
breaking bad spin off report faport claims
(8) bryan cranston less likely return walter white for breaking bad spin off. report claims
C Performance on Cont. and Oppn. splits
All results of the Cont, and Oppn. dataset splits are shown in Table 6.
D Recall Loss
log(0)Ds)
(0 H'O''('''X00 0*)
(0 O' + Hyrior (0'))(0 0*)
B Example
N Σ Fi(0, -07)²
2
Example Sentence Transformations of "bryan cranston will return as walter white for breaking bad spin off, report claims."
NF > 2
(0: -0)
1
Σ
(0; 0,)²
(i) Paraphrase:
(1) report says bryan cransion will be back to act walter white as breaking bad sequel.
(2) in breaking bad sequel, the actor of walter white, bryan cranston, will return.
(3) breaking bad will still use bryan cranston as the actor of walter white in its spin off.
(4) walter white will continue to be acted by bryan cranston in its spin off
(5) the actor of walter white, i.e., bryan cranston, in breaking bad spin off will keep unchanged
(6) bryan cranston is reported to return for the breaking bad sequel as the actor of walter white (7) the role of walter white will continue be acted by bryan cranston, according to the report (8) the report says the breaking bad spin off will still have bryan cranston played same role, walter white
(ii) Negation:
(1) bryan cranston will not return walter white for breaking bad spin off', report claims
(2) bryan cranston will return as walter white for
breaking bad spin off, fake report claims
(3) bryan cranston unlikely to return as walter
white for breaking bad spin off, report claims

---

Datasets
Cont. (UnSup)
Cont. (Sup)
Oppn. (UnSup)
Oppn. (Sup)
STS2012-MSRRvid
0.8848
0.9388
0.7306
0.8862
STS2013-headline
0.8688
0.881
0.4487
0.5281
3-S2013-ONWN
0.8584
0.8788
0.7816
0.814
ST52014-deft-forum
0.7269
0.7942
0.355
0.3279
STS2014- headlines
0.8578
0.8898
0.4066
0.364
STS2014-images
0,8723
0.9108
0.3638
0.645
0.8394
0.8264
0.322
0.3992
STS2015-hcadlines
0.8932
0.9081
0.402
0,4602
STS2015-images
0.8926
0.9446
0.5648
0.8224
STS2006--asweer-aaswerr
0.7646
0.841
0.5011
0.607
ST52016-headlines
0.8887
0.8793
0.3836
0.4075
ST52016-plagiarism
0.9098
0.9137
0.5058
0.5425
11$2016-postediüng
0.8982
0.9064
0.0719
0,0835
STS201 (o--uestionnquuseteeeee
0.7187
0.7517
0.6347
0.6821
STSB-train
0.8577
0.8928
0.4966
0.6293
STSB-test
0.8354
0.8762
0.5415
0.7271
Weighted average
0.8574
0.8891
0.4982
0.6152
Table Performance on Cont. and Oppn, dataset splits when probed by unsupervised and supervised models.


Reviews:
[{"review_id": "9qQPZIuxqa", "reviewer": "Reviewer_nVw5", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "1. About the contributions: Thank you for your comment. I agree with you that the study of a linguistic expression's surface structure and deep structure has long been proposed. However, since the unsupervised method based on contrastive learning has been proposed, this problem has not been revisited and verified whether it has been solved (as mentioned by the Reviewer w2Dv). This paper investigates the surface structure bias in contrastive learning for the unsupervised sentence representation and systematically evaluates the bias by constructing datasets following the two designated settings on surface structure and deep structure similarity, and then we propose a solution by leveraging augmented data with max margin loss and a catastrophic forgetting recall loss. Extensive experiments and analyses have shown that we significantly outperformed the baselines and demonstrated the effectiveness of our experiments.\n\n2. About the improvement: Thank you for your comment. I assume our methods can continue to benefit from higher-quality synthetic examples thanks to the recent advancement of LLM models. We will leave it to future work.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- The paper presents an interesting unsupervised sentence embedding method based on contrastive learning. Notably, the authors first address the surface structure bias in unsupervised sentence embedding. They then employ a max margin loss to ensure proper ranking between negations and their paraphrased counterparts, and finally, they incorporate a recall loss to preserve knowledge from pretraining and minimize catastrophic forgetting.\n\n- The empirical study is comprehensive, spanning various experiments from STS to SICK-R. The methodology's effectiveness has been tested on several architectures, including BERT base, BERT large, RoBERTa base, and RoBERTa large.\n\n- The paper is well-written, with a clear delineation of its position relative to previous research. The emphasis on benchmarking against traditional baselines and surpassing them is commendable and will likely resonate with the broader community.\n", "reasons_to_reject": "- Conceptually the proposed idea is similar to existing techniques and it could be viewed as incremental. \n- The observed performance enhancements are somewhat modest, suggesting room for further refinement in the future.", "questions_to_authors": "", "paper_topic_and_main_contributions": "Previous contrastive learning methods suffers surface structure bias problem, often mistaking them for semantic similarity. As a result, paraphrased sentences with different surface structures might be deemed less semantically similar than sentences with merely a negative word insertion. This paper delves into this surface structure bias in unsupervised sentence embedding. By creating datasets that differentiate between surface and deep structure similarities, the authors present a two-fold solution: augmenting the data to counteract the bias with the max margin loss and utilizing recall loss to retain word semantics, thereby avoiding extreme forgetting.", "excitement": 4, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "QojMX11QVW", "reviewer": "Reviewer_ApYm", "summary": "", "questions": "1. See reasons to reject.\n\n2. In the ablation analysis, it seems that the \"base\" means original simcse baselines, and the authors ablate prompt, recall loss and the max margin loss components. I personally think the data that was generated certainly contributes a lot to the performance gain as well (particularly the back-translation part, because this is essentially positive pairs generated by other supervised models), and should be emphasized, instead of attributing the gain to max margin loss. Could the authors further justify this?\n\n3. Do unsupervised methods other than dropout-based methods have the same gap with supervised methods on negations and random?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "1. Why use PromptBert?\nThank you for your question. As highlighted by Reviewer w2Dv, random seeds can influence the experimental outcomes in this unsupervised learning task, particularly with the vanilla SimCSE. However, PromptBert offers a more consistent training process and yields superior, more consistent results by leveraging a prompt combined with the “[mask]” token for sentence representation, building upon the foundation of SimCSE. This approach has been adopted in several recent sentence embedding studies [1] [2]. Our experiments, based on PromptBert, show significantly less variance compared to that on SimCSE.\n[1] Yeon et al. Ranking-Enhanced Unsupervised Sentence Representation Learning. In Proceedings of ACL 2023\n[2] Zeng, Jiali, et al. \"Contrastive Learning with Prompt-derived Virtual Semantic Prototypes for Unsupervised Sentence Embedding.\" Findings of the Association for Computational Linguistics: EMNLP 2022. \n\n2. How does the augmented data contribute to the performance?\nThank you for your question. According to the augmentation method from Appendix A, there are around 590,000 augmented samples (out of 1 million total samples) that have both paraphrase and negation transformations. Inspired by your suggestions, we add three ablation results by using 50,000, 100,000 and 300,000 on Roberta base. \nHere are the Avg performances of the seven standard benchmarks. \n50,000 -> 79.78; 100,000 -> 80.21; 300,000 -> 80.49; 590,000 -> 80.46 (This work). \nWe find that using 100,000 augmented samples can basically achieve similar performance (80.21) to our reported result (80.46), and using 300,000 (80.49) is even better than our reported result (80.46). We will include this experiments in revision.\n\n3. Do other unsupervised methods also suffer from the bias?\nThank you for your inspiring question. From the perspective of current LLMs, pretrained models, either BERT or GPT-like, require a supervised fine-tuning period to learn the sentence-level meaning, maybe also called “instruction following ability”.  I believe we can significantly reduce the number of augmented samples by using stronger backbones and high quality samples. We will leave it for future work.\n\n4. Missing References: Thank you for your advice. It is really close related to this work and will be included in revision.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The paper is well-written. The authors are very familiar with related work. The motivations are clear, and certainly not incremental to the field. Identifying the problems first then propose methods accordingly is a good way to approach the gap.", "reasons_to_reject": "The paper uses many techniques, partly making the framework not too targeted towards addressing its central claim (that surface structure bias contributes a lot to the performance degrade of unsupervised methods). For instance, using PromptBERT instead of [cls] pooling seems to come out of nowhere (also, have you tried mean pooling)? Other than these, I don't see many reasons to reject.", "questions_to_authors": "1. See reasons to reject.\n\n2. In the ablation analysis, it seems that the \"base\" means original simcse baselines, and the authors ablate prompt, recall loss and the max margin loss components. I personally think the data that was generated certainly contributes a lot to the performance gain as well (particularly the back-translation part, because this is essentially positive pairs generated by other supervised models), and should be emphasized, instead of attributing the gain to max margin loss. Could the authors further justify this?\n\n3. Do unsupervised methods other than dropout-based methods have the same gap with supervised methods on negations and random?", "paper_topic_and_main_contributions": "The paper identifies two classes of text (negations and random) that contribute the most to the performance gap between unsupervised contrastive learning models and their supervised counterparts. The authors attribute the problems to that unsupervised methods learn surface-level structures, and propose methods accordingly to bridge the gap, providing good unsupervised performance.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Missing_References": "The extremely high intra-sentence similarity phenomenon brought by supervised contrastive learning [1] might be relavant to the phenomenon found in the paper (might not). Using dropout-based unsupervised methods, there's no lexical differences to provide such signals for the models to learn to rely on important words (and thus provide high intra-sentence similarity), such that the models learn to rely on \"surface-level structure\". This gap might particularly contribute to the \"random\" part.\n\n[1] On Isotropy, Contextualization and Learning Dynamics of Contrastive-based Sentence Representation Learning. In Findings of the Association for Computational Linguistics: ACL 2023."}}, {"review_id": "nbtV3clUjk", "reviewer": "Reviewer_u5Vf", "summary": "", "questions": "(A) Why do we need to restrict the learning objective in Eq (3) with margins?\n\n(B) Have you tried to add the synthetic positive and negative pairs into the original contrastive learning objective, instead of creating a new objective of max margin loss?", "limitations": "", "rating": 3, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "1. Why use PromptBert?\nThank you for your question. As highlighted by Reviewer w2Dv, random seeds can influence the experimental outcomes in this unsupervised learning task, particularly with the vanilla SimCSE. However, PromptBert offers a more consistent training process and yields superior, more consistent results by leveraging a prompt combined with the “[mask]” token for sentence representation, building upon the foundation of SimCSE. This approach has been adopted in several recent sentence embedding studies [1] [2]. Our experiments, based on PromptBert, show significantly less variance compared to that on SimCSE.\n[1] Yeon et al. Ranking-Enhanced Unsupervised Sentence Representation Learning. In Proceedings of ACL 2023\n[2] Zeng, Jiali, et al. \"Contrastive Learning with Prompt-derived Virtual Semantic Prototypes for Unsupervised Sentence Embedding.\" Findings of the Association for Computational Linguistics: EMNLP 2022. \n\n2. Ablation analysis on vanilla SimCSE (BERT-base): Thank you for your advice, here are the results:\n\n        STS12 | STS13 | STS14 | STS15 | STS16 | STS-B | SICK-R | Avg. \n\nSimCSE                  68.40 | 82.41 | 74.38 | 80.91 | 78.56 | 76.85 | 72.23 | 76.25\n\nSimCSE w Max             72.77 | 83.92 | 76.84 | 82.57 | 79.53 | 78.67 | 73.76 | 78.29\n\nSimCSE w Rec             69.20 | 82.19 | 74.16 | 82.96 | 80.60 | 79.50 | 72.86 | 77.35\n\nSimCSE w Max & Rec  70.61 | 85.48 | 76.39 | 84.59 | 79.91 | 80.12 | 74.99 | 78.87\n\nThe performance gain from leveraging max margin loss and recall loss based on vanilla SimCSE is similar to using PromptBERT as the backbone.\n\n3. Directly adding the augmented samples in SimCSE:\nThank you for your question. We tried directly utilizing these sentences as positive and negative samples in order to fit them into contrastive learning objectives. Our experiments indicate a substantial decrease in average performance from 80.46 to 75.18 when utilizing Roberta-base as the backbone. The contrastive loss is particularly sensitive to the quality of generated paraphrases. This is consistent with the findings presented in Table 1 and 4 of the SimCSE paper, where performance decreases when applying augmented or even labeled (such as image caption datasets) data in contrastive loss. \n\n4. Margins of Max Margin Loss: \nThank you for your question. Given this sensitivity of data quality, we have set alpha (0.05), which can guarantee the correct ranking between paraphrases and negations. Furthermore, a large alpha could increase the possibility of poorly generated paraphrases dominating the training.\nSecondly, it is not appropriate to view negation sentences as hard negatives. These sentences are semantically contradictory, yet still highly semantically related to the original sentence (usually a single-word change). As such, we have set a value for beta (0.2) to constrain these negations to also share a relatively high level of sentence similarity.\n\n5. Typos, structures, and grammar errors: Thank you for your comment, I will improve the readability of the paper and correct the typos and grammar errors in revision.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The authors provide rigorous and extensive experiments to address the problem of surface structure bias, showing its importance to unsupervised sentence embedding approaches. In addition, compared with several baselines, the proposed method seems to be effective, especially the enhancement on RoBERTa series models.", "reasons_to_reject": "The overall method in this paper uses the proposed method with the prompt technique, which is not related to the issue of structure bias. The prompt technique is supposed to serve as an amplifier. This causes two problems:\n\n(A) Many baselines did not use prompts, but the authors's approach did. The authors compare their approach with the baseline with prompt (SimCSE-RoBERTa_{base}+Prompt) in Table 5 on the NLU tasks, but they did not compare the proposed method with the setting in Table 4 on the similarity tasks, which are the main tasks in this paper.\n\n(B) It's unclear if the prompt technique plays a more important role over the proposed method. An ablation study may be required, but all the settings on top of \"Base\" in Figure 7 include prompts.", "questions_to_authors": "(A) Why do we need to restrict the learning objective in Eq (3) with margins?\n\n(B) Have you tried to add the synthetic positive and negative pairs into the original contrastive learning objective, instead of creating a new objective of max margin loss?", "paper_topic_and_main_contributions": "This paper addresses the issue of the positive and negative pairs formation for learning sentence representations with unsupervised contrastive learning. The problem is that two sentences different in structures (they call surface structures) may end up sharing the same semantic meanings, which they call similar deep structures. The authors tried to alleviate this issue with a novel objective for learning augmented data and a regularization technique for the pre-trained model.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "(A) The paper is hard to follow. For example, when Cont. and Oppn. first show up in L080, I expect to see clear definitions of Cont. and Oppn. with examples in the following lines or in Table 1, but only short indications in the caption of Table 1. And Figure 6. is mentioned at the left column of Page 6 but not show up until the last page.\n\n(B) L392 typo: Oppo. -> Oppn.\n\n(C) The legend of Figure 6. can be improved to match the caption."}}, {"review_id": "Eha8uHAhzk", "reviewer": "Reviewer_w2Dv", "summary": "", "questions": "Although there is no problem with the general logic of the paper in finding surface structure bias and proposing a method to solve it, I think that one of the points made by the author during the discussion has not been verified:\n\nQuestion1. The authors wrote \"This method for forming positive and negative pairs may lead to the surface structure bias\" in the paper. However, BERT-Flow shows that this problem already exists in the original BERT. However, the probing experiments in Section 3 of the paper do not report results from the original BERT model, so I don't think it can be said that the method of SimCSE for constructing positive and negative examples introduces this problem, probably because SimCSE doesn't (fully) address this problem that originally existed in the BERT-based Model.", "limitations": "", "rating": 4, "confidence": 5, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "1. Ablation analysis on vanilla SimCSE: Thank you for your advice, here are the results on BERT-base:\n\n        STS12 | STS13 | STS14 | STS15 | STS16 | STS-B | SICK-R | Avg. \n\nSimCSE                  68.40 | 82.41 | 74.38 | 80.91 | 78.56 | 76.85 | 72.23 | 76.25\n\nOurs w Max             72.77 | 83.92 | 76.84 | 82.57 | 79.53 | 78.67 | 73.76 | 78.29\n\nOurs w Rec             69.20 | 82.19 | 74.16 | 82.96 | 80.60 | 79.50 | 72.86 | 77.35\n\nOurs w Max & Rec  70.61 | 85.48 | 76.39 | 84.59 | 79.91 | 80.12 | 74.99 | 78.87\n\nThe performance gain from leveraging max margin loss and recall loss based on vanilla SimCSE is similar to using PromptBERT as the backbone.\n\n\n2. Number of augmented samples: Thank you for your question. I agree with you that augmented data is important for the final performance. According to the augmentation method from Appendix A, there are around 590,000 augmented samples (overall number of training examples is 1 million) that have both paraphrase and negation transformations. Inspired by your suggestions, we add three ablation results by using 50,000, 100,000 and 300,000 on Roberta base. Here are the Avg performances of the seven standard benchmarks.\n50,000 -> 79.78;\n100,000 -> 80.21;\n300,000 -> 80.49;\n590,000 -> 80.46;\nWe find that using 100,000 augmented samples can basically achieve similar performance (80.21) to our reported result (80.46), and using 300,000 (80.49) is even better than our reported result (80.46).\n\n3. Random seeds effect: Thank you for your suggestions, I agree with you on the random seed effect on the final performance, and all our reported results on based on the default seed (42). One reason why we choose PromptBERT as part of the overall method is that its variance in the results is smaller than vanilla SimCSE. We rerun our methods on Roberta-base experiments by setting 5 random seeds, and we find the Avg. performance of Roberta-base ranges from [80.36, 80.55]. We will report the variance in the final results when revision.\n\n4. Hyperparameter search and code release: Thank you for your constructive comments. The hyperparameters are revealed in Appendix A. We will report the hyperparameters selection experimental results when revision. The code and the trained model links have been provided in the supplemental materials. We will release our trained model and code once the paper is accepted.\n\n5. Bias in the original BERT: I agree with you that the original BERT might already have such bias according to the BERT-flow paper, and we will re-verify this in the revision. From the perspective of the recent LLMs, slightly supervised fine-tuning after the unsupervised pretraining stage is always necessary to learn the sentence-level meaning, also known as \"instruction following ability\". We will consider that as a possible way to enhance this work in the future.\n\n6. Missing References: Thank you for pointing this out, we will include the RankCSE paper in the revision.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1、The problem of highly correlated semantic similarity and edit distance that exists in BERT was found in BERT-Flow[1] a long time ago, which is called \"surface structure bias\" in this paper. However, after the unsupervised method based on contrastive learning has been proposed, this problem has not been revisited and verified whether it has been solved or not, and I think this work is valuable.\n\n2、 The logic of this paper is perfect,  it fully reveals to us the surface structure bias problem existing in SimCSE, and the experiments do prove that the method does effectively alleviate the bias problem.\n\n[1] On the Sentence Embeddings from Pre-trained Language Models. EMNLP2020", "reasons_to_reject": "The experimental part still has some flaws:\n\n1. The authors used SimCSE as the research object in all of the previous exploratory experiments. However, the \"prompt trick\" in PromptBERT becomes the fixed part to be added to the Experiment part.  I think the ablation study should include at least three experiments, \"w Max\", \"w Recall\" and \"w Recall & Max\", to illustrate the effectiveness of the purposed method directly on SimCSE.\n2.  According to my personal experience, random seeds have some influence on the experimental results. Referring to the previous work [1,2,3], reporting the average results of experiments with 3 or more random seeds is more convincing.\n3. The paper does not state how many new sentences were involved in the training process on top of the 1M sentences. It is important to know whether the improvement of this method is simply from the introduction of more data compared to those methods that only perform hard negative sample mining or new loss functions.\n4. Insufficient exploration of hyperparameters, neither the search range nor the degree of influence of hyperparameters is described in the paper.\n\nBesides, the authors did not commit to open-source code with the constructed probe dataset, which makes the results potentially difficult to reproduce.\n\nI will be glad to see the author's response!\n\n\n[1] Unsupervised Sentence Representation via Contrastive Learning with Mixing Negatives. AAAI2022\n\n[2] PCL: Peer-Contrastive Learning with Diverse Augmentations for Unsupervised Sentence Embeddings. EMNLP2022\n\n[3] On The Inadequacy of Optimizing Alignment and Uniformity in Contrastive Learning of Sentence Representations. ICLR2023", "questions_to_authors": "Although there is no problem with the general logic of the paper in finding surface structure bias and proposing a method to solve it, I think that one of the points made by the author during the discussion has not been verified:\n\nQuestion1. The authors wrote \"This method for forming positive and negative pairs may lead to the surface structure bias\" in the paper. However, BERT-Flow shows that this problem already exists in the original BERT. However, the probing experiments in Section 3 of the paper do not report results from the original BERT model, so I don't think it can be said that the method of SimCSE for constructing positive and negative examples introduces this problem, probably because SimCSE doesn't (fully) address this problem that originally existed in the BERT-based Model.", "paper_topic_and_main_contributions": "This paper finds and discusses the surface structure bias problem in SimCSE, an unsupervised sentence representation learning baseline method. \n\nThe solution in the paper is to obtain the \"Paraphrase\" and \"Negation\" forms of some sentences by data augmentation methods, then two triplet losses are used to keep a gap between similarity(original sentences, paraphrase form) and the similarity(original sentences, negation form). In addition, a regression loss is used to solve the catastrophic forgetting problem during training.", "excitement": 4, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "None"}, "extra_text": {"Missing_References": "Missing the latest SOTA[1], which was first uploaded and submitted to OpenReview last September, which cannot be considered contemporaneous work. The STS results in this paper are almost equivalent to the latest SOTA.\n\n[1] RankCSE: Unsupervised Sentence Representations Learning via Learning to Rank"}}]

Author rebuttal:

Meta-review: This paper addresses the surface structure bias issue in unsupervised sentence embedding models, particularly in contrastive learning methods. The authors propose a two-fold solution: first, they augment the data to counteract the bias, and second, they use a recall loss to preserve word semantics and avoid forgetting. This approach aims to improve the similarity assessment between sentences with different surface structures and sentences with semantic differences.
After author rebuttal, the reviewers updated their original views. In general, they agree that the paper is well-written, it's clearly motivated, and the empirical comparisons are comprehensive and sound. The reasons to reject are minors and the authors provided detailed comments that can added to the camera-ready version to enhance the paper.
Decision: Accept-Main