Improving Contrastive Learning of Sentence Embeddings with Focal InfoNCE
Authors: Pengyue Hou
Xingyu Li
Venue: EMNLP 2023 Findings (2023)
Keywords: Contrastive Learning
Sentence Textual Similarity
Sentence Embdding
Negative Sample Reweighing

Abstract: The recent success of SimCSE has greatly advanced state-of-the-art sentence representations. However, the original formulation of SimCSE does not fully exploit the potential of hard negative samples in contrastive learning. This study introduces an unsupervised contrastive learning framework that combines SimCSE with hard negative mining, aiming to enhance the quality of sentence embeddings. The proposed focal-InfoNCE function introduces self-paced modulation terms in the contrastive objective, downweighting the loss associated with easy negatives and encouraging the model focusing on hard negatives. Experimentation on various STS benchmarks shows that our method improves sentence embeddings in terms of Spearman's correlation and representation alignment and uniformity.

Improving Contrastive Learning of Sentence Embeddings with Focal-InfoNCE
Pengyue Hou
Xingyu Li University of Alberta
University of Alberta
pengyue@ualberta.
xingyu@ualberta.co
Abstract
proaches often face challenges in hard sample min- ing. Specifically, the original training paradigm of Sunsupervised-SimmSS proposes to use contra- diction sentences as "negatives". But such imple- mentation only guarantees that the contradiction sentences are "true negatives" but not necessarily hard. With a large number of easy negative samples, the contribution of hard negatives is thus prone being overwhelmed.
The recent success SimCSE has greatly advanced state-of-the-art sentence representa- tions. However, the original formulation of SimCSE does not fully exploit the potential of hard negative samples in contrastive learn- ing. This study introduces an unsupervised contrastive learning framework that combines SimCSE with hard negative mining, aiming to enhance the quality of sentence embeddings, The proposed InCo--I function intro- duces self- saced modulation terms in the con- trastive objective, downweighting the loss asso- ciated with easy negatives and encouraging the model focusing on hard negatives. Experimen- tation on various STS benchmarks shows that method improves sentence embeddings in terms of Spearman's 's correlation and represen- tation alignment and uniformity. Our code is available at: https:/ /gi thub. com/puerrrr/ FoCaC-InfoNCE.
To address this issue, we propose novel loss function. namely Focal-InfoNCE, in the paradigm of unsupervised SimCES for sentence embedding. Inspired by the focal loss (Lin et al., 2017), the pro- posed Focal-InfoNCE. loss assigns higher weights to the harder negative samples model training and reduces the influence of easy negatives ac- cordingly. By doing so, focal Info-NCE encour- ages the model to focus more on challenging pairs, forcing it to learn more discriminative sentence representations. In addition, to adapt the dropout strategy for positive pair construction in SimCSE, further incorporate a positive modulation term in the contrastive objective, which reweights the positive pairs in model optimization. We conduct extensive experiments on various STS benchmark datasets (Agirre et al., 2012, 2013, 2014, 2015, 2016; Cer et al., 2017; Marelli et al., 2014) to eval- uate the effectiveness of Focal Info-NCE. Our re- sults demonstrate that Focal Info-NCE significantly improves the quality of sentence embeddings and outperforms unsupervised-SimCS by average of 1.64%, 0.82%, 1.51%, and 0.75% Spearan's correlation on BERT-base, BaRT- RoBERTa- base, and RoBERTa-large, respectively.
1 Introduction
Unsupervised learning of sentence embeddings has been extensively explored natural language pro- cessing (NLP) (Cer et al., 2018; Giorgi et al., 2020; Yan et al., 2021), aiming to generate meaningful representations of sentences without the need for labeled data. Among various approaches, Sim- CSE (Gao et al., 2021) achieves state-of-the-ant performance in learning high-quality sentence em- beddings through contrastive learning. Due to its simplicity and effectiveness, various efforts have been made to improve the contrastive learning of sentence embeddings from different aspects, in- cluding alleviating false negative pairs (Wu et al., 2021a; Zhou et al., 2022) and incorporating more informative data augmentations (Wu et al., 2021b; Chuang al., 2022).
2 Related Work
2.1 Unsupervised SimCSE
Leveraging hagd-neeative samples in contrastive learning of significance (Schroff al., 2015;
SimCSE (Gao al., 2021) provides an unsuper- vised contrastive learning solution to SOTA perfor- mance in sentence embedding. Following previ-
Oh Song et al., 2016; Robinson et al., 2020). Nev-
ertheless, unsupervised contrastive learning ap-

---

ous work (Chen al.. 2020), it optimizes pre- trained model with the cross-entropy objective us- ing in-batch negatives. Formally, given mini- batch of N sentences, { let h2 be the sen- tence representation of 2y with the pre-trained lan- guage model such as BERT (Devlin et al., 2018) RoBERTa (Liu et al., 2019). SimCSE's training objective, InfoNCE, can be formulated as
3 Focal-InfoNCE for Sentence Embedding
This study follows the unsupervised SimCSE framework for sentence embedding. Instead of taking the InfoNCE loss in Eq. (1), we introduce self-paced reweighting objective function, Focal- InfoNCE, to up-weight hard negative samples in contrastive learning. Specifically, for each sentence 2i Focal-infoNce formulated as (1)
cemm(hsh 3/T
4 -log Ej=1 esim(baahg
S /T
1; -log Σ e (sh2
(2)
where T a temperature hyperparameter and sim(h, h2) represents the cosine similarity be- tween sentence pairs (z), Note, h is the rep- resentation of an augmented version of z, which constitutes the positive pair of Tz. For notation simplification, we will use e and std to represent the similarities between positive pairs and negative pairs in this paper.
mm)/T + (0////7
where m a hardness- aware hyperparameter that offers flexibility in adjusting the re- re-weighting strat-
egy. Within a mini-batch of sentences, the final loss function,
Di=1 4. can be derived as
L --oogg -log
Unsupervised SimCSE uses model's built-in dropout as the minimal "data augmentation" and passes the same sentence to the same encoder twice to obtain two sentence embeddings as positive pairs. Any two sentences within mini-batch form neg- ative pair. It should be noted that in contrastive learning, model optimizaiton with hard negative samples helps learn better representations. But SimCSE doesn't distinguish hard negatives and easy ones. We show in this work that incorporating hard negative sample mining in SimCSE boosts the quality of sentence embedding.
+f eth (s +m//T + e (8))9/T
(3)
Analysis of Focal-InfoNCE: Compare with In- foNCE in Eq. (1), InfoNCE introduces self- paced modulation terms on Sp and 8n, proportional to the similarity quantification. Let's first focus on the modulation term, s m, on negative pairs, Prior arts have shown that pre-trained language models usually suffer from anisotropy in sentence embedding (Wang and Isola, 2020). Finetuning the pretrained models with contrastive learning on negative samples, especially hard negative samples, improves uniformity of representations, mitigating the anisotropy issue. In SimCSE, si. quantifies the similarity between negatives 2; and æz.. If s/ is large, H; and zj hard negatives for current model. Improving the model with such hard nega- tive pairs encourage representation's uniformity. To this end, we propose to upweight the corresponding term s4. /r by modulation factor 5 + m. The partial derivative Infor-Inf with respect to s is
2.2 Sample Re-weighting in Machine Learning
Re-weighting is a simple yet effective strategy for addressing biases in machine learning. It down- weights the loss from majority classes and obtains balanced learning solution for minority groups. Re- Re- weighting is also common technique for hard example mining in deep metric learning (Schroff al., 2015) and contrastive learning (Chen al., 2020; Khosla et al., 2020). Recently, self-paced re- re-weighting is explored in various tasks, such as object detection(Lin et al., 2017), person re- identification(Sun et al., 2020), and adversarial training (Hou et al., 2023). It re-weights the loss of each sample adaptively according to model's opti- mization status and encourages model focus on learning hard cases. To the best of our knowledge, this study constitutes the first attempt to incorporate self self-paced re- ee-ieighting strategy in unsupervised sentence embedding.
ØL as
2 e di (st
tm)/7
(sh + m),
(4)
T
Zi
if1
(st2 mm)/ + dB//// Ac- where Z4 = lift
cording to Eq. (4), comparing easy negatives, hard negative samples that associates with higher similarity score 5 contribute more the loss function. This implies that a model optimized with the proposed negative samples. Our experiments also show that
focuses more on hard-

---

Focal-InfoNCE improves uniformity in sentence embeddings.
BERT
RoBERTa
base
large
base
large
batch size
64
64
512
512
To uncover the insight of the modulation term ep on positive cases, let's revisit SimCSE. In Sim- CSE, the positive pair is formed by dropout with random masking. Thus low similarity score Sp in- dicates semantic information loss introduced by dropout. Since such a low similarity is not at- tributed to model's representation capability, we should mitigate its effect on model optimization. Hence, Focal-inforNCE. assigns small weight to the dissimilary positive pair. The partial derivative with respect to st. is
learning rate
30-5
1e-5
le-5
3e-5
Table I: Experimental setting for our main results.
4.1 Comparison to Prior Arts
Table shows the performance of the differ- ent models with and without the Focal-InfoNce loss. In general, we observe improvements Spearman's 's correlation scores when incorporating the proposed Focal-InfoNCE, For example. with SimCSE-BERT base: the average score increases from 75.68 to 77.32 when using
(///)//T
ØL
1)sh
(5)
0s
Z4
4.2 Alignment and Uniformity
Alignment and uniformity are two key properties to measuring the quality of contrastive representa- tions (Gao et al., 2021). By specifically focusing on challenging negative samples, focal-InfoNCE encourages the model to pay closer attention negative instances that are difficult to distinguish from positive pairs. In Table. 3, we incorporate the proposed Focal-InfoNCE into different contrastive learning for sentence embeddings and show im- provements in both alignment and uniformity of the resulting representations.
which suggests that positive pairs with lower simi- larity scores in SimCSE contributes less to model optimization. We show in the experiments that Focal-InfoNCE improves the allignment of sen- tense embeddings as well.
Due the modulation terms on both positive and negative samples, Focal-InfoNCE, reduces the chances of the model getting stuck in sub-optimal solutions dominated by easy pairs. We show in our experiment that the proposed Focal-InfoNCE can easily fit into most contrastive training frameworks for sentence embeddings.
4.3 Ablation Studies on Hyperparameters
We conducted ablation studies to analyze two key factors in Focal-InfoNCE temperature 7 and the hardness hyperparameter
4 Experiments
We evaluate Focal-InfoNCe on semantic sim- ilarity tasks: STS 12-16 (Agirre et al., 2012, 2013, 2014, 2015, 2016), STS Benchmark (Cer et al., 2017) and SiCK-Relatedness (Marelli et al., 2014). Spearman's correlation between predicted and ground truth scores is used the numerical performance metric. Our implementation closely follows unsupervised-Simee SimCCSE (Gao al., 2021). Briefly, starting with pre-trained models BERT base. BERTiarge (Devlin et al., 2018) and RoBERTasase. and RoBERTajarge (Liu et al., 2019), we take the [dls) embeddings as the sentence representations and fine-tune the models with 106 randomly sam- pled English Wikipedia sentences. The hyperpa- rameter for the four models (0.7, 0.7, 0.5, 0.5) respectively and we use m 0.3 in this exper- iment. To make fair comparison, we adopted the same batch size and learning rate as unsupervised- SimCSE, shown in Table
Temperature is a myper-paramenn used in the InfoNCE loss function that scales the logits before computing the probabilities. (Wang and Liu, 2021) show that the temperature plays a key role in con- trolling the strength of penalties on hard negative samples. In this ablation, we set the temperature as 0.03, 0.05, 0.07, and 0.1, explore the effect of different temperature values on the model's 's perfor- mance and report the results in Table
The hardness hyper-parameter m controls the rescaling of the negative samples in the contrastive loss function. Figure. visualizes the rescaling effects of m. Specifically, our Focal-InfoNCE: loss regards negative pairs with cosine similarity larger than (1-m) hard negative examples and vice versa. The loss is then up-weighted or down- weighted proportionally. Table. demonstrates that our method is not sensitive to m and the optimal set- ting can usually be found between 0.2 and 0.3. One

---

Model
STS12
STS13
STS14
STS15
STS16
STS-B
SICK-R
Avg.
GloVe embeddings (avg.)
55.14
70.66
59.73
68.25
63.66
58.02
53,76
61.32
oe avg.)
39.70
59.38
49.67
66.03
66.19
53.87
62.06
56.70
b flow*
58.40
67.10
60.85
75.16
71.22
68.66
64.47
66.55
BHHRT whilening"
57.83
66.90
60.90
75.08
71.31
68.24
63.73
66.28
SimCSE-BERTE hear
67.33
82.45
72.30
80.76
79,04
76.38
71.58
75.68
+ Focal-InfoNCE
68.50
83.70
19.00
12.71
79.43
78.85 10
72.99 10
77.33 1000
SimCSE- CRICSE-BERTTng
70.31
84,73
75.52
83,06
78,90
78.20
74.56
77.90
+ Focul-InfoNCE
7..86
84.50
75.81+0.49
84.56+n.02
79.13+a.77
80.97
72.74+0.5
RoBIERTaas==ffrtt-tlaa avg.""
40,88
58.74
49.07
65.63
61,48
58.55
61.63
56.57
RoBERTaaes. MaaeeiKTaes--wwiiinnnn whiteming"
26.99
63.24
31.23
71.36
68.99
61.36
62.91
61.73
DDcCLLUTR-RoBLRE
52.41
75.19
65.52
77.12
78.63
72.41
68.62
69.99
SSinCEE-RoBElR RoBERTato
67.54
81.39
72.82
8[61
80.29
80.00
68.86
76.07
+ Ioocal-InfoNCH
70.84.0.0
82.5220.45
23.99
83.02
82.26
81.1619.0
77.60.0.5
"SiniCSE-
71.54
83.11
75.04
84.20
80.54
"1.32
69.99
77.99
+ Iofall-nfoNNC
71.45_110
83.24
74.48
85.101u.3
82.54
82.24
72.56.0.
78.801004
Table 2: Sentence embedding performance on STS tasks with Spearman's 's correlation. Gurevych, 2019); #: results from (Gao al., 2021).
results from (Reimers and
Model
ALGN
UNIF
SpCorr 1
BERT baar
RoBERTam
SiMCSE
0.190
-2,400
76,38
T1
STS-B
SICK-R
STS-B
SICK-R
Focal InfoNCE
0.134
-1,906
79.31
0.1
77.23
71.30
80.33
68.86
DifCCSE
0.081
-1.T93
79.53
0.2
79.37
71.70
81.42
69.60
+Focal InfoNCE
0.070
-1.335
80.09
0.3
79.31
72.72
81.72
69.52
DCLR
0.231
-2.790
78.31
0.4
78.99
71.75
80.07
64,21
+Focal InfoNCE
0.164
-2.366
79.30
Table Effects hyperparameter m on STS Bench- mark (Cer et al., 2017) with Spearman's correlation.
Table 3: Representation alignment (ALGN), unifor- mity (UNIF), and Spearman's 's correlation (SpCort) with BERT base the STS Benchmark (Cer et al., 2017).
Ns
1.4
m-03
BERT
RoBERTa
m=0.1
base
large
base
large
1.2
0.03
74.6 66
76.96
80.62
77.55
0.05
78.98
79.11
81.72
82,46
0.07
79.31
80.43
81.00
82.03
0.8
0.1
77.33
73.23
79.99
81.09
0.6
Table 4: Effects of temperature т on STS Bench- mark (Cer al., 2017) with Spearman's correlation.
0.9
possible explanation for this could be the simplic- ity of the data augmentation employed by SimCSE, resulting a consistent positive similarity score of approximately 0.85 (Wu et al., 2021a).
Figure Plot re-scaled negative pairs vs. their orig- inal value with different choices of m. Orange: up- weighted negative examples; green: down-weighted negative examples
5 Discussion
5.1 Qualititave Analysis
underlying tones and attitudes are completely dif- ferent. The proposed Focal InfoNCE effectively guides the model's attention by putting more penal- ties toward these cases and encourages the model to learn better sentence embedding separate them.
We conducted a qualitative analysis as follows. We sampled a subset of hard negative pairs from the STS-B training data and analyzed their characteris- tics. We noticed that these hard negative pairs often involved sentences with commain-specifie terms, or sentences with high degree of syntactic similar- very
5.2 Statistical Significance Analysis
ity. For example, sentences like "This unusual request." and think it's fine to ask this question." have a notable cosine similarity of 0.64 by the pre-trained BERT base model, though their
To validate the statistical significance of perfor- mance increase, we conducted three more sets experiments, each with different random seed, and

---

reported the mean and standard deviation (std) in Table 2. Based on the results, we calculated paired- ttests with a standard significance level of 0.05, All p-values over the various STS tasks with different base models are smaller than 0.05, which indicate that our focal-InfoNCE improves the performance significantly in the statistical sense.
References
Encko Agirre, Carmen Banea, Claire Cardie, Daniel Cer. Mona Diab, Aitor Gonzalez-Agirre, Weiwei Guo, Inigo Lopez-Gazpio, Montse Maritxalar, Rada Mi- halcea, et al. 2015. Semeval-2015 task 2: Semantic textual similarity, english, spanish and pilot on inter- pretability. In Proceedings of the 9th international workshop on semantic evaluation (SemEval 2015). pages 252-263.
5.3 Compatibility with Existing Methods
Encko Agirre, Carmen Banea, Claire Cardie, Daniel Cer, Mona Diab, Aitor Gonzalez-Agiree, Weiwei Guo. Rada Mihalcea, German Rigau, and Janyce Wiebe, 2014. Semeval-2014 14 task 10: Multilingual semantic textual similarity. In Proceedings of the 8th interna- tional workshop on semantic evaluation (SemEval 2014), pages 81-91.
As reported in Table 3, focal-InfoNCE improves DiffCSE in terms of representation alignment, uniformity, and Spearman's correlation. In con- trast, attempts to integrate Promp al., 2022) did not yield any improvements over the baseline models. We hy- pothesize that this outcome could be attributed to the different treatments of noise in positive pairs. In SimCSE and DifiCSE(Chhang et al., 2022), pos- itive pairs are generated with the same template but different dropout noise. Our focal- downweights the false positive pairs introduced by the random masking mechanism. But in Prompt- BERT where different positive templates are used for contrastive learning, the template biases have been incorporated in the contrastive loss. Conse- quently, downweighting false positive pairs with focal-InfoNCE might be unnecessary on Prompt- BERT.
with
Eneko Agirre, Carmen Banea, Daniel Cer, Mona Diab, Aitor Gonzalez Agirre, Rada Mihalcea, German Rigau Claramunt, and Janyce Wiebe. 2016. Semeval- 2016 task I: Semantic textual similarity, monolin- gual and cross-lingual evaluation. In SemEval-2016. 10th International Workshop on Semantic Evalua- tion; 2016 Jun 16-17; Son Diego, CA. Stroudsburg ACL: 2016. p. 497-511. ACL (Association for Computational Linguistics).
Eneko Agirre, Daniel Cer, Mona Diab, and Aitor Gonzalez-Agirre 2012. Semeval-2012 task 6: A pilot on semantic textual similarity. In SEM 2012: The First Joins Conference Lexical and Compu- tational Semantics Volume I: Proceedings of the main conference and the shared task, and Volume 2: Proceedings of the Sixth International Workshop on Semantic Evaluation (SemEval 2012), pages 385-
393.
6 Conclusions
Eneko Agirre, Daniel Cer, Mona Diab, Aitor Gonzalez- Agirre, and Weiwei Guo. 2013. sem 2013 shared task: Semantic textual similarity. In Second joint conference on lexical and computational semantics (* SEM), volume I: proceedings of the Main confer- ence and the shared task: semantic textual similarity, pages 32-43.
This paper introduced a novel unsupervised contrastive learning objective function, Focal- InfoNCE, to enhance the quality of sentence em- beddings. By combining SimCSE with self-paced hard negative re-weighting, model optimization was benefited from hard negatives. Extensive ex- periments shows the effectiveness of the proposed method various STS benchmarks.
Daniel Cer, Mona Diab, Eneko Agirre, Inigo Lopez- Gazpio, and Lucia Specia. 2017. Semeval-2017 task 1: Semantic textual similarity and cross- linsullingual focused evaluation. arXiv preprint 1788v:1708.00055.
7 Limitations
Daniel Cer, Yinfei Yang, Sheng-yi Kong, Nan Hua, Nicole Limtiaco, Rhomni St John, Noah Constant, Mario Guajardd-Cesspedes, Steve Yuan, Chris Tar, et al. 2018. Universal sentence encoder for english. In Proceedings of the 2018 conference empiri- cal methods in natural language processing: system demonstrations, pages 169-174.
The effectiveness of the Focal-InfoNCC depends on the quality of pre-trained models. When a pre- trained model leads to bad representations, the sim- ilarity scores may mislead model finetuing. In addi-
tion, the positive re- weighting strategy in this study is quite simple. We believe that more sophisticated mechanisms to address semantic information loss in positive pairs would further improve the perfor- mance.
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. 2020. A simple framework for contrastive learning of visual representations. In In- ternational conference on machine learning, pages 1597-1607. PMLR

---

Yung Yung-Sung Chuang, Rumen Dangovski, Hongyin Luo. Yang Zhang, Shiyu Chang, Marin Soljačíć, Shang- Wen Li, Wen-tau Yih, Yoon Kim, and James Glass. 2022. Diffcse: Difference-based contrastive learning for sentence embeddings. arXiv preprint arXiv:2204. 10298..
Joshua Robinson, Ching Yao Chuang, Suvrit Sra. and Stefanie Jegelka. 2020. Contrastive learn- ing with hard negative samples. arXiv preprint arXiv:2010.04592.
Florian Schroff. Dmitry Kalenichenko, and James Philbin. 2015. Facenet: A unified embedding for face recognition and clustering. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 815-823.
Jacob Devlin, Ming-Wei Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. Bert: Pre-training of deep bidirectional transformers for language understand- ing. arviv preprint arXiv:1810.04800
Yifan Sun, Changmao Cheng, Yuhan Zhang, Chi Zhang, Liang Zheng. Zhongdao Wang, and Yichen Wei. 2020. Circle loss: unified perspective of pair simi- larity optimization. In Proceedings of the IEEF/CVF conference computer vision and pattern recogni- tion, pages 6398 3998-6407.
Tianyu Gao, Xingcheng Yao, and Danqi Chen. 2021. Simese: Simple contrastive learning of sentence em- beddings. arXiv preprint arXiv:2104.0882
John Giorgi, Osvald Nitski, Bo Wang, and Gary Bader. 2020. Declutr: Deep contrastive learning for un- supervised textual representations. arXiv preprint arXiv:2006.03659)
Feng Wang and Huaping Liu. 2021. Understanding the behaviour of contrastive loss. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pages 2495-2504.
Pengyue Hou, Jie Han, and Xingyu Li. 2023. Im- proving adversarial robustness with self self-paced hard- class pair reweighting. In Proceedings of the Thirty- Seventh AAAI Conference on Artificial Intelligence.
Tongzhou Wang and Phillip Isola. 2020. Understanding contrastive representation learning through alignment and uniformity the hypersphere. In International Conference on Machine Learning, pages 9929-9939. PMLR.
Ting Jiang, Jian Jiao, Shaohan Huang, Zihan Zhang, Deqing Wang, Fuzhen Zhuang, Furu Wei, Haizhen Huang, Denvy Deng, Qi Zhang. 2022. Prompt- bert: Improving bert sentence embeddings with prompts. arXiv preprint 2tti::2000.04337.
Xing Wu. Chaochen Gao, Yipeng Su. Jizhong Han, Zhongyuan Wang, and Songlin Hu. 2021a. Smoothed contrastive learning for unsupervised sen- tence embedding. arXiv preprint arXiv:2109.0433
Prannay Khosla, Piotr Teterwak, Chen Wang, Aaron Sarna, Yonglong Tian, Phillip Isola, Aaron Maschinot, Ce Liu, and Dilip Krishnan. 2020. Su- pervised contrastive learning. Advances in Neural Information Processing Systems, 33: 16611-133....
Xing Wu, Chaochen Gao. Liangjun Zang. Jizhong Han, Zhongyuan Wang. and Songlin Hu. 202 Esimese: Enhanced sample building method for contrastive learning of unsupervised sentence embedding. arXiv preprint arXiv:2109.0m 093000
Tsung- Yi Lin. Priya Goyal, Ross Girshick, Kaiming He, and Piotr Dollár. 2017. Focal loss for dense object detection. In Proceedings of the IEEE international conference on computer vision, pages 2980-2988
Yuanmeng Yan, Rumei Li, Sirui Wang, Fuzheng Zhang, Wei Wu, and Weiran Xu. 2021. Consert: A con- trastive framework for self-supervised sentence repre- sentation transfer. arXiv preprint arXiv: 11741.
Yinhan Liu, Myle Oit, Naman Goyal, Jingfei Du, Man- dar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: robustly optimized bert pretraining ap- proach. arXiv preprint arXiv: 197.
Kun Zhou, Beichen Zhang, Wayne Xin Zhao, and Ji- Rong Wen. 2022. Debiased contrastive learning of unsupervised sentence representations. arXiv preprint arXiv:2205.00656.
Marco Marelli, Stefano Menini, Marco Baroni, Luisa Bentivogli, Raffaella Bernardi, Roberto Zamparelli, et al. 2014. A sick cure for the evaluation of com- positional distributional semantic models. In Lrec, pages 216-223. Reykjavik.
Hyun Oh Song. Yu Xiang. Stefanie Jegelka, and Silvio Savarese. 2016. Deep metric learning via lifted struc- tured feature embedding. In Proceedings of the IEEE conference on computer vision and pattern recogni- tion, pages 4012.
Nils Reimers and Iryna Gurevych, 2019. Sentence-bert: Sentence embeddings using siamese bert-networks. arXiv preprini 19X8.:1908. 10084.


Reviews:
[{"review_id": "xJ32mTxXJU", "reviewer": "Reviewer_qweB", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 2, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate the reviewer's feedback on our work. Our point-to-point response is as follows.\n\n**Q1. Improvement is modest.**\n\nThank you for recognizing our effort. This study's significance extends beyond the noted 1%-3% numerical enhancement. Firstly, as shown in Table 2 of our manuscript, the suggested loss is compatible with other contrastive algorithms for sentence embedding, such as DiffCSE [1], signifying a wider impact of our approach. Secondly, sentence embedding is considered a fundamental task in NLP. Even modest advancement in sentence embeddings can lead to substantial gains in downstream applications like text classification, similarity retrieval, and information retrieval. We believe that discussing the potential subsequent effects of these improvements in the revision will underline the contributions of this work.\n\n**Q2. The method relies on the quality of the pretrained model.**\n\nWe completely agree with this assessment. The objective of our study is to present an innovative approach that serves as a supplement to existing techniques and maximizes the potential of pretrained models through contrastive learning refinement. \n\n**Q3. More explanation and discussion on the results would be nice.**\n\nThanks for this comment. We will add in-depth discussion of our experimental results in the revision. In addition, as our response to Reviewer 15wZ, a qualitative analysis will also be included.\n\n\n[1] Chuang et al., 2022: DiffCSE: Difference-based Contrastive Learning for Sentence Embeddings", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- The paper is on important problem: improving a popular model such as SimCSE.\n- The method is intuitive. Simple loss modification led to improved results.\n- Generally consistent improvement compared to the baseline", "reasons_to_reject": "- Improvement is modest.\n- The method relies on the quality of the pretrained model.\n- More explanation and discussion on the results would be nice.", "questions_to_authors": "", "paper_topic_and_main_contributions": "Main contributions: improvement to the sentence embeddings of SimCSE\n- Makes the model to focus more on hard negatives\n- Results indicate better alignment and uniformity than SimCSE", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "2OcMy6cHMX", "reviewer": "Reviewer_15wZ", "summary": "", "questions": "Did you do any qualitative analysis? If so, did it offer any insights? E.g. what types of negative pairs Focal InfoNCE deems hard\n\nDid you perform any statistical significance analysis?", "limitations": "", "rating": 4, "confidence": 2, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We would like to thank the reviewer for the positive and constructive comments. Here are our point-to-point responses to the comments.\n\n**Q1. Did you do any qualitative analysis? If so, did it offer any insights? For example, what types of negative pairs does Focal InfoNCE deem hard?**\n\nThis is a very good question. To improve the quality of this study, we conducted a qualitative analysis as follows. We sampled a subset of hard negative pairs from the STS-B training data and analyzed their characteristics. We noticed that these hard negative pairs often involved sentences with domain-specific terms, or sentences with a high degree of syntactic similarity. For example, sentences like \"This is a very unusual request.\" and \"I think it's fine to ask this question.\" have a notable cosine similarity of 0.64 by the pre-trained BERT base model, though their underlying tones and attitudes are completely different. The proposed Focal InfoNCE effectively guides the model's attention by putting more penalties toward these cases and encourages the model to learn a better sentence embedding to separate them. We plan to include a dedicated subsection in our revision to discuss these qualitative findings.\n\n**Q2. Did you perform any statistical significance analysis?**\n\nThanks for bringing up this valuable question. To validate the significance of performance increase, we conducted three more sets of experiments, each with different random seed, and reported the mean and standard deviation (std) in the following table. Based on the results, we calculated paired-t tests with a standard significance level of 0.05. All p-values over the various STS tasks with different base models are smaller than 0.05, which indicate that our focal-InfoNCE improves the performance significantly in the statistical sense.\n\n| Model  | STS12 |STS13|STS14|STS15|STS16|STS-B|SICK-R|Avg.|\n| ------------- |:-------------:|:-------------:|:-------------:|:-------------:|:-------------:|:-------------:|:-------------:|:-------------:|\n|  BERT_base + Focal-InfoNCE| 68.50±0.08| 83.70±0.50 |75.09±0.15 |82.71±0.23| 79.43±0.40| 78.85±0.37| 72.99±0.10| 77.33±0.08  |\n| BERT_large + Focal-InfoNCE|71.86±1.55| 84.50±0.20| 75.81±0.49| 84.56±0.62| 79.13±0.97| 80.97±1.00| 72.74±0.45| 78.51±0.28|\n |RoBERTa_base + Focal-InfoNCE|70.84±0.59| 82.52±0.48 |73.99±0.85| 83.02±0.99 |82.26±0.45| 81.16±0.50| 69.43±0.77| 77.60±0.26|\n| RoBERTa_large + Focal-InfoNCE |71.45±1.10| 83.24±0.88| 74.48±1.07| 85.10±0.35| 82.54±1.06| 82.24±0.67 |72.56±0.96 |78.80±0.04|\n\n**Q3: Typos and presentation improvement.**\n\nThank you. We will correct the typos and the misplacement of bold highlights in Table 3, and use more distinguishable colors for Figure 1 in our revised manuscript.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- Unsupervised approach without the need for external semantic resources\n- Can be applied to a variety of contrastive learning systems\n- Well reasoned\n- Good quantitative analysis on both system performance and hyperparameter tuning", "reasons_to_reject": "- Lack of qualitative analysis gives readers little insight into how the authors' claim play out in a real world dataset and what the context their approach may or may not be suited toward", "questions_to_authors": "Did you do any qualitative analysis? If so, did it offer any insights? E.g. what types of negative pairs Focal InfoNCE deems hard\n\nDid you perform any statistical significance analysis?", "paper_topic_and_main_contributions": "This paper introduces Focal InfoNCE, a novel loss function for unsupervised contrastive learning which places an emphasis on hard negative examples. The proposed function self-modulates, decreasing the weight of negative pairs with low similarities (i.e. easy negatives) and thus causing the model to focus on negative pairs with high similarity (i.e. hard negatives). The proposed function can be used with a variety of unsupervised contrastive learning systems. The authors evaluate their proposed systems' performance against a series of baselines including SimCSE and show increased performance on a series of semantic siimilarity task.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "Paper contains 4 instances of \"unsupervised SimCSE\" and 4 instances of \"unsupervised-SimCSE\". I couldn't tell any differences from the usage so I suspect you'll want to chose one and use it consistently\n\nS1P3 line 053 - \"SimCES\" -> SimCSE\n\nS1P3 line 073 - \"Spearan's\" -> Spearman's\n\nTable 3 - I'm confused by the bolding for UNIF. SimCSE and DCLR have their values bolded despite being higher than +Focal InfoNCE (incorrectly implying that higher is better) while DiffCSE is correctly bolded due to its lower score\n\nFigure 1 - I don't have a solution but the chosen colors don't read well on the graph. E.g. the blue N/A line on the graph looks gray due to the other colored regions overlaying on it. Similarly, the red line appears more orange"}}, {"review_id": "wdkKQ4co8a", "reviewer": "Reviewer_Crvb", "summary": "", "questions": "- Was there statistical analysis conducted to confirm the significance of performance increase?\n- Will the code be publicly available in the future?\n- Is the method compatible with other contrastive-learning-based algorithms such as PromptBERT (Jiang et al., 2022), DiffCSE (Chuang et al., 2022)?", "limitations": "", "rating": 4, "confidence": 5, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We would like to thank the reviewer for the positive and constructive comments. Here are our point-to-point responses to the comments.\n\n**Q1. Was there statistical analysis conducted to confirm the significance of performance increase?**\n\nAs pointed out by the reviewer on the stability of the STS task, we also encountered this challenge during our study and have had discussions with the authors of SimCSE. Aware of this issue, we have carefully maintained a consistent experimental environment with the default hyperparameters in huggingface and ran the experiments multiple times to ensure robustness and consistency of our results. To further validate the significance of performance increase, we conducted three more sets of experiments, each with different random seed, and reported the mean and standard deviation (std) in the following table. As shown in the table, the results are consistent with our results reported in the manuscript. In addition, based on the results, we calculated paired-t tests with a standard significance level of 0.05. We observed that all p-values over the various STS tasks with different base models are smaller than 0.05, which indicate that our focal-InfoNCE improves the performance significantly in the statistical sense.\n\n| Model  | STS12 |STS13|STS14|STS15|STS16|STS-B|SICK-R|Avg.|\n| ------------- |:-------------:|:-------------:|:-------------:|:-------------:|:-------------:|:-------------:|:-------------:|:-------------:|\n|  BERT_base + Focal-InfoNCE| 68.50±0.08| 83.70±0.50 |75.09±0.15 |82.71±0.23| 79.43±0.40| 78.85±0.37| 72.99±0.10| 77.33±0.08  |\n| BERT_large + Focal-InfoNCE|71.86±1.55| 84.50±0.20| 75.81±0.49| 84.56±0.62| 79.13±0.97| 80.97±1.00| 72.74±0.45| 78.51±0.28|\n |RoBERTa_base + Focal-InfoNCE|70.84±0.59| 82.52±0.48 |73.99±0.85| 83.02±0.99 |82.26±0.45| 81.16±0.50| 69.43±0.77| 77.60±0.26|\n| RoBERTa_large + Focal-InfoNCE |71.45±1.10| 83.24±0.88| 74.48±1.07| 85.10±0.35| 82.54±1.06| 82.24±0.67 |72.56±0.96 |78.80±0.04|\n\n\n**Q2. Will the code be publicly available in the future?**\n\nOur code and the pre-trained models have now been released at https://github.com/puerrrr/Focal-InfoNCE.\n\n**Q3: Is the method compatible with other contrastive-learning-based algorithms such as PromptBERT (Jiang et al., 2022), DiffCSE (Chuang et al., 2022)?**\n\nWe appreciate your insightful question. In our experiments, we integrated our proposed focal-InfoNCE framework with DiffCSE and evaluated its performance on the STS Benchmark. As reported in Table 3 of our manuscript, focal-InfoNCE improves DiffCSE in terms of representation alignment, uniformity, and Spearman's correlation. We include the results here for ease of review.\n\n| Model | alignment  | uniformity |  Spearman's correlation | \n|--------|:--------:|:--------:|:--------:|\n|  DiffCSE   | 0.081  | -1.195 | 79.53|  \n|  DiffCSE + focal-InfoNCE  | 0.070 |  -1.335|  80.09 | \n\nIn contrast, our attempts to integrate focal-InfoNCE with PromptBERT did not yield any improvements over the baseline models. We hypothesize that this outcome could be attributed to the different treatments of noise in positive pairs. In SimCSE and DiffCSE, positive pairs are generated with the same template but different dropout noise. Our focal-InfoNCE downweights the false positive pairs introduced by the random masking mechanism. But in PromptBERT where different positive templates are used for contrastive learning, the template biases have been incorporated in the contrastive loss. Consequently, downweighting false positive pairs with focal-InfoNCE might be unnecessary on PromptBERT.\n\nIn our revised manuscript, we will provide more discussion about the potential compatibility of our method with existing contrastive learning approaches, including PromptBERT and DiffCSE. We will provide insights into how our method's focal-InfoNCE framework could be potentially extended to work with different algorithms.\n\n**Q4: References missing and presentation.**\n\nR4: Thanks for these comments. We will revise the manuscript accordingly and proofread it.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The novel use of focal loss is a simple but effective addition to the successful SimCSE algorithm.  The algorithm is compatible with all previous SimCSE settings.", "reasons_to_reject": "It appears that the authors only reported the best performance of their model and did not conduct any statistical analysis of the performance benefit. The performance of the STS task is prone to instability.\n\nAnother major issue with the paper is that it suggests SimCSE is the state-of-the-art. However, more recent works, such as PromptBERT (Jiang et al., 2022) and DiffCSE (Chuang et al., 2022), have surpassed SimCSE's performance. These works achieved much higher STS performance than SimCSE using the unsupervised setting, and even higher than the performance of Focal-InfoNCE. However, it seems that Focal-InfoNCE may be compatible with newer systems. It would be interesting to see if Focal-InfoNCE can further improve performance.\n\nFinally, the authors did not mention whether the implementation will be publicly available.  If not, it will be hard to reproduce the authors results.", "questions_to_authors": "- Was there statistical analysis conducted to confirm the significance of performance increase?\n- Will the code be publicly available in the future?\n- Is the method compatible with other contrastive-learning-based algorithms such as PromptBERT (Jiang et al., 2022), DiffCSE (Chuang et al., 2022)?", "paper_topic_and_main_contributions": "Inspired by focal loss, this paper proposes further improvement of SimCSE by introducing a novel Focal-InfoNCE loss. By incorporating this Focal-InfoNCE term into the original contrastive learning objective, the model effectively enhances SimCSE's performance across various STS benchmarks in the unsupervised setting.", "excitement": 4, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "- [279] finetuing -> finetuning\n- The limitation of the paper should be its own section, not a paragraph of the conclusion. ", "Missing_References": "Chuang et al., 2022: DiffCSE: Difference-based Contrastive Learning for Sentence Embeddings\n\nJiang et al., 2022: PromptBERT: Improving BERT Sentence Embeddings with Prompts"}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper gives a focused contribution to the field of contrastive learning of sentence embeddings. By emphasizing hard negative samples during the contrastive learning process, the method improves on a contrastive learning approach called SimCSE.

The reviewers raised some questions regarding qualitative and quantitative analysis of the results, which the authors have satisfactorily addressed in their rebuttal.
Decision: Accept-Findings