RobustEmbed: Robust Sentence Embeddings Using Self-Supervised Contrastive Pre-Training
Authors: Javad Rafiei Asl
Eduardo Blanco
Daniel Takabi
Venue: EMNLP 2023 Findings (2023)
Keywords: Trustworthy Machine Learning
Text Representation
Adversarial Attacks
Self-Supervised Contrastive Learning
TL;DR: We introduce a self-supervised framework for generating robust sentence embeddings that can withstand different types of adversarial attacks.

Abstract: Pre-trained language models (PLMs) have demonstrated their exceptional performance across a wide range of natural language processing tasks. The utilization of PLM-based sentence embeddings enables the generation of contextual representations that capture rich semantic information. However, despite their success with unseen samples, current PLM-based representations suffer from poor robustness in adversarial scenarios. In this paper, we propose RobustEmbed, a self-supervised sentence embedding framework that enhances both generalization and robustness in various text representation tasks and against diverse adversarial attacks. By generating high-risk adversarial perturbations to promote higher invariance in the embedding space and leveraging the perturbation within a novel contrastive objective approach, RobustEmbed effectively learns high-quality sentence embeddings. Our extensive experiments validate the superiority of RobustEmbed over previous state-of-the-art self-supervised representations in adversarial settings, while also showcasing relative improvements in seven semantic textual similarity (STS) tasks and six transfer tasks. Specifically, our framework achieves a significant reduction in attack success rate from 75.51% to 39.62% for the BERTAttack attack technique, along with enhancements of 1.20% and 0.40% in STS tasks and transfer tasks, respectively.

RobustEmbed: Robust Sentence Embeddings Using Self-Supervised Contrastive Pre-Training
Javad Rafiei Asl,' Eduardo Blanco, Daniel Takabi
'Georgia State University, ?University of Arizona, 'Old Dominion University jasll @ @student.gsu.ede eduardoblanco @ arizona.edu, takabi @ddu.edu
Abstract
2019), which effectively capture the semantic rep- resentation of the input text. This representation learning facilitates feature generation for classifica- tion tasks and enhances large-scale semantic search (Neelakantan et al., 2022).
Pre-trained language models (PLMs) have demonstrated exceptional performance across a wide range of natural language processing tasks. The utilization of PLM-based sentence embeddings enables the generation contex- tual representations that capture rich semantic information. However, despite their success with unseen samples, current PLM-based rep- resentations suffer from poor robustness in ad- versarial settings. In this paper, we propose RobustEmbed, self-supervised sentence em- bedding framework that enhances both gener- alization and robustness in various text rep- resentation tasks and against diverse set of adversarial attacks. By generating high-risk adversarial perturbations to promote higher in- variance in the embedding space and leverag- ing the perturbation within a novel contrastive objective approach, RobustEmbed effectively learns high-quality sentence embeddings. Our extensive experiments validate the superior- ity of RobustEmbed over the state-of-the-an self-supervised representations in adversarial settings, while also showcasing relative im- provements in seven semantic textual similarity (STS) tasks and six transfer tasks. Specifically, our framework achieves significant reduction in attack success rate from 75.51% to 39.62% for the BERTAuack attack technique, along with enhancements of 1.20% and 0.40% STS tasks and transfer tasks, respectively.
The assessment of PLM-based sentence repre- sentation relies on two crucial characteristics: gen- eralization and robustness. While considerable research efforts have been dedicated to develop- ing universal sentence embeddings using PLMs (Reimers and Gurevych, 2019; Zhang al., 2020; Ni al., 2022; Neelakantan al., 2022; Wang et al., 2023; Bölücü et al., 2023), it is worth not- ing that despite their promising performance across various downstream classification tasks (Sun et al.. 2019; Gao et al., 2021), demonstrating proficiency in generalization, these representations exhibit limi- tations in terms of robustness in adversarial settings and are vulnerable diverse adversarial attacks (Nie al., 2020; Wang al., 2021). Existing re- search (Garg and Ramakrishnan, 2020; Wu et al.. 2023; Hauser et al., 2023) highlights the poor ro- bustness of these representations, such as BERT- based representations, which can be deceived by replacing few words in the input sentence.
In this paper, we propose RobustEmbed, robust sentence embedding framework that takes both of these essential characteristies into account. The core concept involves introducing a small adversar- ial perturbation the input text and employing the contrastive objective (Chen et al., 2020) to learn high-quality sentence embeddings. RobustEmbed perturbs the embedding space rather than the raw text, which exhibits positive correlation with gen- eralization and promotes higher invariance. Our framework utilizes the original embedding along with the perturbed embedding "positive pairs," while other sentence embeddings in the same mini- batch serve as "negatives." The contrastive objec- tive identifies the positive pairs among the nega- tives. By incorporating norm-bounded adversarial
1 Introduction
Recent research has demonstrated the state-of-the- art performance of Pre-trained Language Models (PLMs) in learning contextual word embeddings (Devlin et al., 2019), leading to improved gener- alization in various Natural Language Processing (NLP) tasks (Yang al., 2019; He al., 2021: Ding al., 2023). The focus of PLMs has extended to acquiring universal sentence embeddings, such as Universal Sentence Encoder (USE) (Cer et al., 2018) and Sentence- Centence-BERT (Reimers and Gurevych,

---

perturbation and contrastive objectives, our method enhances the robustness of similar sentences and disperses sentences with different semantics. This straightforward and efficient approach yields supe- rior sentence embeddings in terms of both general- ization and robustness benchmarks.
2 Related Work
The early work in text representations focused on applying the distributional hypothesis to pre- diet words based on their context (Mikolov et al.. 2013b,a). There are extensive studies on learn- ing universal sentence embeddings using su- pervised and unsupervisse approaches, such Doc2vec (Le and Mikolov, 2014), SkipTThought (Zhu al., 2015), Universal Sentence Encoder (Cer et al., 2018), and Sentence-BERI (Reimers and Gurevych, 2019). More recently, self-supervised approaches have emerged, employing contrastive objectives to learn effective and robust text repre- sentations: SimCSE (Gao et al., 2021) introduced a minimal augmentation strategy predict the in- put sentence by applying two different dropout masks. The ConSERT model (Yan al., 2021) utilized four distinct data augmentation techniques to generate diverse views for the purpose of exe- cuting a contrastive objective: adversarial attacks, token shuffling, cut-off, and dropout. Qiu et al. (2021) introduced two adversarial training methods, CARL and RAR, to strengthen the ML model's 's defense against gradient-based adversarial attacks. CARL aims acquire resilient representation at the sentence level, whereas RAR focuses on enhancing the robustness of individual word repre- sentations. Rima et al. (2022) proposed adversar- ial training with contrastive learning for training natural language processing models. It involves applying linear perturbations to input embeddings and leveraging contrastive learning to minimize the distance between original and perturbed represen- tations. Pan et al. (2022) presents straightfor- ward approach for regularizing transformer-base encoders during the fine-tuning step. The model achieves noise-invariant representations by gener- ating adversarial examples perturbing word embed- dings and leveraging contrastive learning.
We conduct extensive experiments on a wide range of text representation and NLP tasks to ver- ify the effectiveness of RobustEmbed including semantic textual similarity (STS) tasks (Conneau and Kiela, 2018), transfer tasks (Conneau and Kiela, 2018), and TextAttack (Morris al., 2020). Two first series of experiments evaluate the qual- ity of sentence embeddings on semantic similarity and natural language understanding tasks, while the last series assess the robustness of the frame- work against state-of-the-art adversarial attacks. RobustEmbed demonstrates significant improve- ments robustness, reducing the attack success rate from 75.51% 39.62% for the BERTAttack at- tack technique and achieving similar improvements against other adversarial attacks. Additionally, the framework achieves performance improvements of 1.20% and 0.40% on STS tasks and NLP transfer tasks, respectively, when employing the BERT encoder.
Contributions. Our main contributions in this paper summarized as follows:
We introduce RobustEmbed, novel self- supervised framework for sentence embed- dings that generates robust representations capable of withstanding various adversarial attacks. Existing sentence embeddings are susceptible to such attacks, highlighting a vul- nerability in their security. RobustEmbed fills this gap by employing high-risk perturbations within novel contrastive learning approach.
In comparison to several existing contrastive ad- versarial learning approaches in the text represen- tation area (Yan et al., 2021; Meng et al., 2022; Qiu et al., 2021; Li et al., 2023; Rima et al., 2022: Pan et al., 2022), our framework stands out by gen- erating more efficient high-risk iterative perturba- tions in the embedding space. Furthermore, our framework leverages more powerful contrastive objective approach, leading high-quality text representations that demonstrate enhanced general- ization and robustness properties. Empirical results substantiate the superiority of our approach across
We conduct extensive experiments to demon- strate the efficacy of RobustEmbed across various text representation tasks and against state-of-the-ee adversarial attacks. Empiri- cal results confirm the high efficiency of our framework in terms of both generalization and robustness benchmarks.
facilitate further research in this important area, our source code is available in the Ro- bustEmbed Repository

---

various generalization and robustness benchmarks.
3.2 Contrastive Learning Based
Representation
3
Background
The objective of contrastive learning is to acquire effective low-dimensional representations by bring- ing semantically similar samples closer and push- ing dissimilar ones further apart (Hadsell et al., 2006). Self-supervised contrastive learning has demonstrated promising results in data represen- tation across domains such machine vision (Chen et al., 2020), natural language processing (Gao al., 2021; Neelakantan et al., 2022), and speech recognition (Lodagala al., 2023). Our framework adopts the contrastive learning con- cept proposed by Chen et al. (2020) to generate high-quality representations. Let {( ) de- note a set of N positive pairs, where 2; and 2;² are semantically correlated and (zi, 2,') represents the corresponding embedding vectors for the pos- itive pair (z,, x.)). We define z'ss positive set [=]00)] = a,. while the negative set {x; the of other positive pairs. Then, the contrastive training objective can be defined as follows:
In this section, present overview of the recent progress in adversarial perturbation generation and self-supervised contrastive learning.
3.1 Adversarial Perturbation Generation
Adversarial perturbation involves adding mali- ciously crafted perturbations benign data, which can deceive Machine Learning (ML) models, in- cluding deep learning methods (Goodfellow et al., 2015). These perturbations are designed to be im- perceptible to humans but can cause the model to make incorrect predictions (Metzen et al., 2017). Adversarial training, which involves incorporating adversarial perturbations during the model training process, has been shown to enhance the model's robustness against adversarial attacks (Madry et al., 2018; Shafahi et al., 2020; Xu et al., 2020; Wang et al., 2019b). While various perturbation genera- tion techniques have contributed to machine vision (Chakraborty et al., 2021), the progress of these techniques in the NLP domain has been at slower pace due the discrete nature of text (Jin al., 2020). In recent years, instead of directly applying adversarial perturbations to raw text, a few studies have focused on perturbing the embedding space (Wang et al., 2019;; Dong et al., 2021). However, these methods still face challenges in terms of gen- eralization, as they may not be applicable to any ML model and NLP tasks. Utilized within our framework,
Lcon,olris (xt J (o)
(2)
Exapory exp(sim((z (x:0=)) 1)/T) - logg Exegon paty papty exp(sim(zs, (z. (xg00, x((a)) 1)/7)
where T denotes temperature hyperparameter and sim(u, v) Tv is the cosine similarity between two representation vectors. The standard objective function only contains a single sample in the positive set. The total loss is computed over all positive pairs within a mini-batch.
|
generalized approach for gener- ating high-risk adversarial perturbations involves applying small noise ö within norm ball the embedding space, aiming to maximize the adver- sarial loss:
4
The Proposed Adversarial Self-supervised Contrastive Learning
We introduce RobustEmbed, simple yet effec- tive approach for generating universal text repre- sentations through adversarial training of self- supervised contrastive learning model. Given a PLM fa(-) as the encoder and a large unsupervised dataset D, RobustEmbed aims to pre-train fo(() on D to enhance the efficiency of sentence embed- dings across diverse NLP tasks (improved general- ization) and increase resilience against various ad- versarial attacks (enhanced robustness). Algorithm 1 demonstrates our framework's approach to gener- ating a norm-bounded perturbation using an itera- tive process, confusing the fo(.) model by treating the perturbed embeddings as different instances.
arg max L(fe + +),,).
(1)
S||sc
where f-)-) denotes an ML model parameterized with X as the sub-word embeddings, and is the truth label. Various gradient-basee based algorithms have been proposed to address this optimization prob- lem. We employ practical combination of the Fast Gradient Sign Method (FGSM) (Goodfellow al., 2015) and the Projected Gradient Descent (PGD) technique (Madry et al., 2018) to generate adversarial perturbations that represent worst-case examples.

---

Contrastive with benign and perturbed representations
PLM encoder Layer
PLM Encoder
High-risk perturbatioo
genorating positive pair forming
perturbed embeddingg
perturbed e embeddings
Iterative Perturbation Generator
berign emoedding.
benign embedding.
Embedding
Embeddings
embedding space
PLM Embedding Layer
Sentences mini batch
Sentence,
Sentence,
Figure The general architecture RobustEmmed framework. In contrastive learning step, a blue arrow indicate gathering positive pairs together, and red arrow refers to keeping distance among negative pairs
Our framework then employs a contrastive learn- ing approach to maximize the similarity between the embedding of an input instance and the adver- sarial embedding of its positive pair. Moreover, Figure provides overview of our RobustEm- bed framework, which aims to achieve adversarial robustness in representations. The framework in- volves an iterative collaboration between the per- turbation generator and the fe(.) model to generate high-risk perturbations for adversarial contrastive learning during the final training step. The subse- quent sections delve into the main components of our framework and provide detailed analysis of the training objective.
Algorithm 1: RobustEmbed Algorithm
Input:
Epoch number E, PLM Encoder fo. dataset of raw sentences
{=x) ) perturbation 6, dropout masks m71 and m2, perturbation bound c, step sizes a and 3, learning rate yy. perturbation modulator X, regularization parameter "/s perturbation generation iterators and T', contrastive learning objective
Leon. (eq. 2)
Output: Robust Sentence Representation for epoch = I,.... E do
for minibatch
D do 5² ~ ~NO,, r²I)
X fm.embedding mm)
X + = fg.embedding(B, m2)
for = 1. max( K, T) do
9(5") VaLom, 6' (X+))
if then
1 /1(("'/)))( 8
4.1 Perturbation Generation
end
if f S T then
As the primary step, RobustEmbed aims to generate small perturbations that fool the ML model, lead- ing incorrect predictions, while remaining nearly imperceptible humans. The framework uses an approuch based on combination of the PGD and FGSM algorithms to generate a perturbation that maximizes the self-supervised contrastive loss, fa- cilitating discrimination between various instances. RobustEmbed employs multiple iterations of this combination, specifically T-step FGSM and K-step PGD, to meticulously reinforce invariance within the embedding space, ultimately resulting in en- hanced generalization and robustness.
8 s++
II 10 (8' [sign(g(6")))
From
end
end
8, 18 (1 2)O Tosm
CRobut 6 (x, X +z
Lrceal LRoraeBBbed, o GLon (X+1) 0 0 V
end
end
applying the standard dropout twice, different embeddings of x+) are obtained "positive pairs" (Gao et al., 2021). The framework takes the following steps to update the perturbation sep- arately for the PGD and FGSM in iteration
In particular, considering the PLM-based en- coder f-) and an input sentence x, RobustEmbed passes the sentence to the fo(.) model twice: by

---

and + respectively:
Crota = Embbed, # + "Com, plande (2tme)))
(8)
pgd = 1 (8 +og(6*)//lg(0")) Ilp), (3)
where ad represents the adversarial perturbation of the input sample in the embedding space, and y denotes regularization parameter. The first part of the total contrastive loss (eq. 8) aims opti- mize the similarity between the input sample æ, its positive pair, and its adversarial perturbation. while the second part serves to regularize the loss by encouraging the convergence of the adversarial perturbation and the positive pair of z.
8t+1 fgsm
=1181115c(0 + (sign(g(0")),
(4)
where g(8") = VoCcon X (X+ with
n =t or k the gradient of the contrastive learn- ing loss with respect to 6. The perturbation is
generated by the Loo norm-ball around the input embedding with radius 6, and IT projects the per- turbation onto the c-ball. Further, a and ,3 are the step sizes of the attacks and sign(.) returns the sign of the vector. Essentially, T-step FGSM and K- step PGD are mathematically equivalent when P is either 2 co. Their primary distinctions lie in the number of iterations (i.e.. T and K) and the step size of the attack (i.e., a and /9) used to mod- ify the input perturbation, ultimately generating a unique high-level perturbation. The final perturba- tion can be obtained through the combination of T-step FGSM and K-step PGD:
Evaluation and Experimental Results
This section presents comprehensive set of exper- iments aimed at validating the effectiveness of our proposed framework in terms of generalization and robustness metrics, In the first two series of experi- ments, we investigate the performance of our frame- work on seven semantic textual similarity (STS) tasks and six transfer tasks within the SentEval framework assess the generalization capability of our framework in generating efficient sentence embeddings. In the final series of experiments, we measure the resilience of the embeddings against five state-of-the-ar adversarial attacks to assess the robustness capability of our framework in generat- ing robust text representation. Appendices A and B provide training details and ablation studics that illustrate the effects of hyperparameter tuning.
final >ō pgd + (1 2 Jgsm
(5)
where 0 S. modulates the relative signifi-
cance of each separate perturbation in the genera- tion of the final perturbation.
4.2 Robust Contrastive Learning
To achieve robust representation through self- supervised contrastive learning, adversarial learn- ing objective, which follows min-max formula- tion minimize the maximum risk for any pertur- bation 8 (Madry et al., 2018), could be defined as follows:
5.1 Semantic Textual Similarity (STS) Tasks
We evaluate our framework on set of seven mantic textual similarity (STS) tasks, which in- clude STS 2012-2016 (Agirre et al., 2012, 2013, 2014, 2015, 2016). STS Benchmark (Cer et al., 2017), and SICK-Relatedness (Marelli et al., 2014), In our experiments, we solely utilize fixed sen- tence embeddings without any training datasets or regressors. To benchmark formance, we compare it against various unsuper- vised sentence embedding approaches, including: 1) baseline methods such as GloVe (Pennington et al., 2014) and average BERT or RoBERTa em- beddings; 2) cost-processing methods like BERT- flow (Li et al., 2020a) and BERT-whitening (Su et al., 2021); and 3) of-te-of-the-rr methods such as SimCSE (Gao et al., 2021), ConSERT (Yan al., 2021), USCAL (Miao et al., 2021), and ATCL (Rima et al., 2022). We validate the findings of the SimCSE, ConSERT, and USCAL frameworks
arg ming ((s)>Dlmaxjan feon,o(X 6.6 +X+ D)), (6)
where X + 5 is the adversarial embedding gen- erated by the iterative gradient- gradient-based perturbation generation (eq. 5). Our framework utilizes ad- versarial examples generated in the embedding space, rather than using the original raw text, result- ing in an ultimate pre-trained model that is robust against m-way instance-wise adversarial attacks. The framework employs the contrastive learning objective to maximize the similarity between clean examples and their adversarial perturbation by in- corporating the adversarial example as the addi- tional element in the positive set:
framework's 's per-
L Stobust Fimbed, 0 = Leon (2r, {atos, padv)), (7)
Ihttps://gitiee http://tw..../eeeeeeeerrrrr//aae

---

Model
STS12
STS13
STS14
STS15
STS16
STS-B
SICK-R
Avg.
GloVe embeddings (avg.)
55.14
70.66
59.73
68.25
63.66
58.02
53.76
61.32
BERTbas base (first-last avg.)
39.70
59.38
49.67
66.03
66.19
53.87
62.06
56.70
BERT BeRThusefloww
58.40
67.10
60.85
75.16
71.22
68.66
64.47
66.55
BERT base- whitening
57.83
66.90
60.90
75.08
71.31
68.24
63.73
66.28
ConSERT-BERT base
64.56
78.55
69.16
79.74
76.00
73.91
67.35
72.75
ATCL-BERT huse
67.14
80.86
71.73
79.50
76.72
79.31
70.49
75.11
SimCSE-BERT
68.66
81.73
72.04
80.53
78.09
79.94
71.42
76.06
USCAL-BERT
69.30
80.85
72.19
81.04
77.52
81.28
71.98
76.31
#RobustEmbedd-BERTT.. base
70.52
82.13
73.56
82.38
77.72
82.97
73.24
77.51
RoBERTubue-
46.99
63.24
57.23
71.36
68.99
61.36
62.91
61.73
ConSERT-RoBERTTaane
66.90
79.31
70.33
80.57
77.95
81.42
68.16
74.95
SimCSSE-RoBERRaaan
68.75
80.81
71.19
81.79
79.35
82.62
69.56
76.30
USCAL-RoBERTihha ahase
69.28
81.15
72.81
81.47
80.55
83.34
70.94
77.08
69.71
81.77
73.34
81.98
79.74
83.70
71.10
77.33
USCAL-RoBERTa,e #RobustEmbedd-RoBERRTI
68.70
81.84
74.26
82.52
80.01
83.14
76.30
78.11
68.92
81,53
74.35
82.91
79.98
83.93
76.93
78.36
Table 1: Semantic Similarity performance STS tasks (Spearman's correlation, "all" setting) for sentence embedding models. We emphasize the pop-performing numbers among models that share the same pre-trained encoder. V: results from (Reimers and Gurevych, 2019); & results from (Gao al., 2021); I); All remaining results have been reproduced and reevaluated by our team. The symbol shows our framework.
by reproducing their results using our own config- uration for BERT and RoBERTa encoders. The results presented Table demonstrate the supe- rior performance of our RobustEmbed framework compared to various sentence embedding meth- ods across most of the semantic textual similarity tasks. Our framework achieves the highest aver- aged Spearman's correlation among state-of- the-art approaches. Specifically, when using the BERT en- coder, our framework outperforms the second-best embedding method, USCAL, by a margin of 1.20%. Additionally, RobustEmbed achieves the highest score in the majority of individual STS tasks (6 out of 7) compared to other embedding methods and performs comparably to the SimCSE method on the STS16 task. For the RoBERTa encoder, both the base version and the large version, RobustEim- bed outperforms the state-of-the-an embeddings in five out of seven STS tasks and achieves the highest averaged Spearman's correlation.
with detailed information provided in Appendix E, We adhere the standard methodology described in Conneau and Kiela (2018) and train a logistic regression classifier on top of the fixed sentence embeddings for our experimental procedure. We replicated the SimCSE, ConSERT, and USCAL frameworks using our configuration for both BERT and RoBERTa encoders. The results presented in Table indicate that our framework demonstrates superior performance in terms average accuracy when compared to other sentence embedding meth- ods. Specifically, when utilizing the BERT encoder, our framework outperforms the second-best embed- ding method by margin of 0.40%. Moreover, Ro- bustEmbed achieves the highest score in four out of six text classification tasks. The similar inter- pretation of the BERT encoder also maintained for the RoBERTa encoder, including both the basc version and the large version.
5.3 Adversarial Attacks
5.2 Transfer Tasks
In this section, we evaluate the robustness of our sentence embedding framework against various adversarial attacks, comparing it with two state- of-the- the-art sentence embedding models: SimSCE (Gao et al., 2021) and USCAL (Miao et al., 2021). Our evaluation involves fine-tuning BERT-based PLM using different embedding approaches on seven text classification and natural language in- ference tasks, namely MRPC (Dolan and Brockett,
This experiment leverages transfer tasks to evaluate the performance of our framework, RobustEmbed, diverse text classification tasks, including sen- timent analysis and paraphrase identification. Our assessment encompasses six transfer tasks: CR (Hu and Liu, 2004), SUBJ (Pang and Lee, 2004), MPQA (Wiebe et al., 2005), SST2 (Socher et al., 2013), and MRPC (Dolan and Brockett, 2005),

---

Model
MR
CR
SUBJ
MPQA
SST2
MRPC
Avg.
GloVe embeddings (avg.)
77.25
78.30
91.17
87.85
80.18
72.87
81.27
Skip-thought
76.50
80.10
93.60
87.10
82.00
73.00
82.05
BERT- [CLS] embedding
78.68
84.85
94.21
88.23
84.13
71.13
83.54
ConSERT-BERT base
79.52
87.05
94.32
88.47
85.46
72.54
84.56
SimCSE-BERT buse
81.29
86.94
94.72
89.49
86.70
75.13
85.71
USCAL-BERT base
81.54
87.12
95.24
89.34
85.71
75.84
85.80
+RobustEmbed-BEEit base
81.94
87.45
95.04
89.88
86.47
76.40
86.20
SimCSE-RoBERTTAg abase
81.15
87.15
92.38
86.79
86.24
75.49
84.87
USCAL-RoBERTian base
82.15
87.22
92.76
87.74
84.39
76.20
85.08
+RobustEmbed- Robectt-RRBeReaaeee
81.49
87.54
93.37
87.95
84.63
76.62
85.27
USCAL-RoBERTaa,e
82.84
87.97
93.12
88.48
86.28
76.41
85.85
*RobustEmbed ToobustEmbodRRoBBBKTinnngg
82.38
88.27
93.91
88.79
86.01
77.11
86.08
Table 2: Results of transfer tasks for different sentence emhedding models. + results from (Reimers and Gurevych, 2019); V: results from (Zhang et al., 2020); We emphasize the top-performing numbers among models that share the same pre-trained encoder. All remaining results have been reproduced and reevaluated by our team. The * symbol shows our framework.
Adversarial Attack
Model
IMDB
MR
SST2
YELP
MRPC
SNLI
MNLI-Mismatched
Avg.
SimCSE-BERTTh..
75.32
65.53
71.49
79.67
80.07
72.65
68.54
72.61
TextFooler
USCAL-BERTh...
61.94
48.71
55.38
62.30
60.18
54.82
53.74
56.72
RobustEmbed-BERTT...
40.55
32.69
36.17
44.25
38.88
37.61
35.63
37.97
SimCSSEBBETT
52.21
42.04
49.67
56.19
56.73
45.39
40.16
48.91
TextBugger
USCAL-BERT
39.16
27.37
31.90
41.25
37.86
30.79
25.45
33.40
RobustEmbed-BERR
23.70
18.03
20.24
28.58
20.89
19.07
16.33
20.98
SimCSE-BERI
64.41
55.73
60.48
67.54
68.15
56.09
$2.58
60.71
PWWS
USCAL-BERT,
51.95
40.67
45.29
52.30
46.86
50.92
39.37
46.77
RobustEmbed-BERT.
33.63
28.15
30.56
29.94
25.51
27.16
28.49
29.06
SimCSF-BERII base
73.50
61.83
68.27
75.15
77.84
69.06
65,43
70.15
BAE
USCAL-BERT, base
58.57
46.19
51.72
50.90
$1.16
$3.77
BabustEmbed-BERII
37.35
29.82
32,08
41.66
36.45
34.17
31,98
34,79
SimCSF-BER hase
78.42
66.94
73.59
80.87
82,16
74.35
72.22
75.51
BERTAttack
USCAL-BERT.. base
63.23
51.08
57.73
63.96
63.05
55.41
55.86
58.62
RobustEmbed-Be BERteedddd-RTTTt base
45.15
39.08
37.24
39.62
Table Attack success rates of various adversarial attacks applied three sentence embeddings (SimCSF-BERTT, USCAL-BERT, and RobustEmbed-BBRRT across five text classification and two natural language inference tasks.
2005), YELP (Zhang et al., 2015), IMDb (Maas et al., 2011), Movie Reviews (MR) (Pang and Lee, 2005a), SST2 (Socher et al., 2013), Standford NLI (SNLI) (Bowman et al., 2015), and Multi-NLI (MNLI) (Williams et al., 2018). Detailed infor- mation regarding these tasks can be found in Ap- pendix E. To assess the robustness of the fine-tuned models, we perform adversarial attacks using the TextAttack framework (Morris et al., 2020) to in- vestigate the impact of five efficient adversarial attack techniques: TextBugger (Li et al., 2019), PWWS (Ren et al., 2019), TextFooler (Jin et al., 2020), BAE (Garg and Ramakrishnan, 2020), and
BERTAtaakk (Li al., 2020b). acquire more comprehensive insight into the functionality these attacks, we provide more details in Appendix F. It should be noted that adaptive attacks cannot generate adversarial attacks using the main algo- rithm of framework, as it operates exclusively in the embedding space while the input instances of sentence embeddings are raw text. To ensure statistical validity, each experiment was conducted five times, each time using 1000 adversarial attack samples; the reported results shown in this section are the average results of five iterations.
Table 3 presents the attack success rates of five

---

adversarial attack techniques on three sentence em- beddings, including our framework. Our embed- ding framework consistently outperforms the other two embedding methods, demonstrating signifi- cantly lower attack success rates across all text classification and natural language inference tasks.
600
500
Query Number
400
Consequently, RobustEimbed achieves the lowest average attack success rate against all adversar- ial attack techniques. These findings validate the robustness of our embedding framework and high- light the vulnerabilities of the two saate-of-the-eeee sentence embeddings to various adversarial attacks.
300
100
10
20
30
40
50 60
70
80
Accuracy Decrease
Figure 2 depicts the average number of queries required and the resulting accuracy reduction for a set of 1000 attacks on two fine-tuned sentence embeddings. Green data points represent attacks on the RobustEmbed framework, while red points represent attacks on the USCAL approach (Miao et al., 2021). Connected pairs of points are asso- ciated with specific attack techniques. Ideally, robust sentence embedding should be situated in the top-left region of the diagram, indicating that the attack technique necessitates larger number of queries to deceive the target model while causing minimal performance degradation. The figure illus- trates that, for each attack, RobustEmbed exhibits greater stability compared to the USCAL method. In other words, a larger number of queries is re- quired for RobustEmbed, resulting in lower accu- racy reduction (i.e., better performance) compared USCAL. This observation holds true for all ap- plied adversarial attacks, indicating the robustness our framework.
Figure 2: Average number of queries and the result- ing accuracy reduction for a set of 1000 attacks on two fine-tuned sentence emheddings. Green points repre- sent attacks the RobustEmbed framework, while red points represent attacks on the USCAL approach.
ial dataset.
Table 4 illustrates the attack success rates of five different adversarial attack techniques (namely TextFooler, TextBugger, PWWS, BAE, and BERTAttack) applied to three sentence em- beddings, including our framework. These evalua- tions are carried out for two specific AdvSTS tasks, namely AdvSTS-B and AdvSICK-R. Notably, our embedding framework consistently outperforms the other two embedding methods, showing sig- nificantly lower attack success rates across both AdvSTS tasks and all employed adversarial attack techniques.
In conclusion, the extensive experiments con- ducted and the results presented in Tables 1, 2, 3, and 4, well Figure provide strong evidence of the exceptional performance of RobustEmbed in various text representation and classification tasks, as well as its resilience against various adversarial attacks and tasks. These findings support the notion that our framework possesses remarkable general- ization and robustness capabilities, underscoring its potential efficient and versatile approach for generating high-quality sentence embeddings.
5.4 Robust Embeddings
We introduce new task called Adversarial Se- mantic Textual Similarity (AdvSTS) to evaluate the resilience of sentence embeddings within our representation framework. AdvSTS uses an effi- cient adversarial approach, such as TextFooler, to manipulate pair of input sentences encourages the target model to produce a regression score that deviates as much as possible from the true score (the ground truth label). Consequently, we create an adversarial STS dataset by convert- ing all benign instances from the original dataset into adversarial examples. Similar the STS task, AdvSTS employs Pearson's correlation metric to assess the correlation between the predicted simi- larity scores generated by the target model and the human-aman annotated similarity scores for the adversar-
way that
5.5 Distribution of Sentence Embeddings
We followed the methodology proposed by Wang and Isola (2020) employ two critical evaluation metrics, termed alignment and uniformity, to assess the quality of our representations. the context of positive pairs represented by the distribution Pross alignment calculates the anticipated distance

---

&uiiform plot of models based on Adversarial Attack
Model
AdySTS-B
AdvSICK-R
Avg.
SimCSE-BERT basc
21.07
24.17
22.62
TextFooler
USCAL-BERT basc
16.52
18.71
17.62
RobustEmbed-BERI
7.48
8.95
8.22
base
SinCSE-BERT base
27.49
28.34
27.91
TextBugger
USCAL-BERT basc
21.52
24.88
23.20
RobustEmbed-BEINg BERT
base
11.76
13.01
12.39
SimCSF-BERT
24.15
26.82
25.49
base
PWWS
USCAL-BERT
21.28
23.65
22.47
base
RobustEmbed-BBil BERT
13.56
14.44
14.00
buse
SinCSE-BERT
26.92
28.81
27.86
base
BAE
USCAL-BERT base
22.92
25.48
24.20
RobustEmbed-BEnl BERT
11.13
12.82
11.98
base
SimCSE-BERG base
31.60
32.85
32.23
BERTAttack
USCAL-BERT base
26.02
28.51
27.26
RobustEmbed-BERi buse
12.99
13.18
13.09
Table Attack success rates of five adversarial attack techniques applied to three sentence embeddings (SimCSE, USCAL, and RobustEmbed) across two Adversarial STS (AdvSTS) tasks (i.e. AdvSTS-B and AdvSICK-R).
between the embeddings of paired instances:
fatign
E
1f(x)) 1 (9)
SiMCSE
0.225
(I, Jreppos
3.7
Uniformity quantifies how uniformly the embed- dings are distributed within the representation space:
Alignment
0.375
USCAL
0.325
Euniform log
E
f(wll
(10)
333
ATCL
Paaa
0.755
RotustSentEmbed
42
0,200
where Pdota represents the data distribution. The underlying principle of these metrics is that posi- tive instances should remain closely grouped, while embeddings for random instances should be spread across the hypersphere. Figure 3 illustrates the uniformity and alignment of various sentence em- bedding models, where lower values correspond to improved performance. comparison to alter- native representations, RobustEmbed achieves a similar level of uniformity (-2.293 vs. -2.305) but demonstrates superior alignment (0.058 vs. 0.073). This highlights the greater efficiency of our frame- work in optimizing the representation space in two distinct directions.
1.9
Uniformity
Figure BERT
lalign
performance in wide range of text representation and NLP tasks. Current sentence embeddings are vulnerable adversarial attacks. RobustEmbed fills this gap by leveraging high-risk adversarial perturbations within a novel contrastive objective approach. We demonstrated the effectiveness of our framework through extensive experiments on semantic textual similarity and transfer learning tasks. Furthermore. Empirical findings substanti- the robustness of RobustEmbed against diverse adversarial attacks. As future work, we aim explore the use of hard negative examples in the supervised setting to further enhance the efficiency of text representations.
6 Conclusion and Future Work
this paper, we proposed RobustEmbed,, self- supervised sentence embedding framework that sig- nificantly enhances robustness against various ad- versarial attacks while achieving state-of -fftth--rt

---

Limitations
Eneko Agirre. Daniel Cer, Mona Diab, and Aitor Nonzalez-Aginee 2012. SemEval-2012 task 6: A pilot on semantic textual similarity. In *SEM 2012: The First Joint Conference on Lexical and Compu- tational Semantics- Volume 1: Proceedings of the main conference and the shared task, and Volume 2: Proceedings of the Sixth International Workshop on Semantic Evaluation (SemEval 2012), pages 385- 393, Montréal, Canada. Association for Computa- tional Linguistics.
Despite the ingenuity of our methodology and its impressive performance, our framework does have some potential limitations:
Our framework is primarily designed and op- timized for descriptive models, such as BERT, which excel in understanding and representing language, as well as related tasks like text clas- sification. However, it may not be directly ap- plicable to generative models like GPT, which prioritize generating coherent and contextu- ally relevant text. Therefore, there may be limitations in applying our methodology to en- hance the generalization and robustness char- acteristies of generative pre-trained models.
Eneko Agirre, Daniel Cer, Mona Diab, Aitor Gonzalez- Agirre, and Weiwei Guo. 2013. *SEM 2013 shared task: Semantic textual similarity. In Second Joint Conference Lexical and Computational Semantics (*SEM), Volume 1: Proceedings of the Main Confer- ence and the Shared Task: Semantic Textual Similar- ity, pages 32-43, Atlanta, Georgia, USA. Association for Computational Linguistics.
Necva Bölücci Burcu Can, and Harun Artuner. 2023. A siamese neural network for learning semantically- informed sentence embeddings. Expert Systems with Applications, 214:119105
Our framework requires significant GPU re- sources for pre-training large-scale pre-trained models like RoBERTataree- Due to limitations in GPU availability, we had to utilize smaller batch sizes during pre-training While larger batch sizes (e.g., 256 or 512) generally lead to improved performance metrics, our exper- iments had to compromise and use smaller batch sizes generate sentence embeddings efficiently given the GPU resource constraints.
Samuel R. Bowman, Gabor Angeli, Christopher Potts, and Christopher D. Manning. 2015. A large anno- tated corpus for learning natural language inference. In Proceedings of the 2015 Conference Empiri- cal Methods in Natural Language Processing, pages Lisbon, Portugal. Association for Compu- tational Linguistics.
Daniel Cer, Mona Diab, Eneko Agirre, lñigo Lopez- Gazpio, and Lucia Specia. 2017. SemEval-2017 task I: Semantic textual similarity multilingual and crosslingual focused evaluation. In Proceedings of the 11hh International Workshop Semantic Evaluation (SemEval-2017 pages 1-14, Vancouver, Canada. Association for Computational Linguistics.
References
Eneko Agirre, Carmen Banea, Claire Cardie, Daniel Cer. Mona Diab, Aitor -ooiraez-Agirre, Weiwei Guo, Inigo Lopez-Gazpio, Montse Maritxalar, Rada Mi- halcea, et al. 2015. Semeval-2015 task 2: Semantic textual similarity, english, spanish and pilot on inter- pretability, In Proceedings of the 9th international workshop on semantic evaluation (SemEval 2015). pages 252-263.
Daniel Cer, Yinfei Yang, Sheng-yi Kong, Nan Hua, Nicole Limtiaco, Rhomni St. John, Noah Constant, Mario Suajardo-Cespedeer Steve Yuan, Chris Tar, Brian Strope, and Ray Kurzweil. 2018. Universal sentence encoder for English. In Proceedings of the 2018 Conference on Empirical Methods in Nat- ural Language Processing: System Demonstrations. pages 169-174, Brussels, Belgium. Association for Computational Linguistics.
Eneko Agirre. Carmen Banea, Claire Cardie, Daniel Cer. Mona Diab, Aitor Gonzalez-Agirre, Weiwei Guo, Rada Mihalcca, German Rigau, and Janyce Wiebe. 2014. SemEval-2014 task 10: Multilingual semantic textual similarity. In Proceedings of the 8th Interna- tional Workshop on Semantic Evaluation (SemEval 2014), pages 81-91, Dublin, Ireland. Association for Computational Linguistics.
Anirban Chakraborty, Manaar Alam, Vishal Dey, Anu- pam Chattopadhyay, and Dehdeep Mukhopadhyay. 2021. A survey on adversarial attacks and de- fences, CAAI Transactions on Intelligence Technol- ogy, 6(1):25-45
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. 2020. A simple framework for contrastive learning of visual representations. In In- ternational conference on machine learning, pages 1597-1607, PMLR.
Eneko Agirre, Carmen Banea, Daniel Cer, Mona Diab, Aitor Gonzalez-Agirre, Rada Mihalcea, German Rigau, and Janyce Wiebe. 2016. SemEval-201e task 1: Semantic textual similarity, monolingual and cross-lingual evaluation. In Proceedings of the 10th International Workshop Semantic Evaluation (SemEval-2016 pages 497-511, San Diego. Califor- nia. Association for Computational Linguistics.
Alexis Conneau and Douwe Kiela. 2018. SentEval: An evaluation toolkit for universal sentence representa- tions. In Proceedings of the Eleventh International

---

Conference on Language Resources and Evaluation (LREC 2018), Miyazaki, Japan. Buropean Language Resources Association (ELRA).
Pengcheng He, Xiaodong Liu, Jianfeng Gao, and
Weizhu Chen. 2021. Deberta: Decoding-enhanced bert with disentangled attention. In International Conference on Learning Representations.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language under- standing. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguisties: Human Language Tech- nologies, Volume I (Long and Short Papers), pages 4171-4186, Minneapolis, Minnesota. Association for Computational Linguistics.
Minqing Hu and Bing Liu. 2004, Mining and summa- rizing customer reviews. In Proceedings of the tenth ACM SIGKDD international conference on Knowl- edge discovery and data mining, pages 168-177.
Haoming Jiang, Pengcheng He, Weizhu Chen, Xi- aodong Liu, Jianfeng Gao, and Tuo Zhao. 2020. SMART: Robust and efficient fine-tuning for pre- trained natural language models through principled regularized optimization. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 2177-2190, Online. Association for Computational Linguistics.
Ning Ding, Yujia Qin, Guang Yang, Fuchao Wei, Zonghan Yang, Yusheng Su, Shengding Hu, Yulin Chen, Chi-Min Chan, Weize Chen, et al. 2023. garameter-efficien fine-tuning of large-scale pre- trained language models. Nature Machine Intelli- gence, 5(3):220-235.
Di Jin, Zhijing Jin, Jocy Tianyi Zhou, and Peter Szolovits. 2020. Is bert really robust? strong base- line for natural language attack text classification and entailment. In Proceedings of the AAAI con- ference on artificial intelligence, volume 34, pages 8018 8025.
William B. Dolan and Chris Brockett. 2005. Automati- cally constructing a corpus of sentential paraphrases. In Proceedings of the Third International Workshop on Paraphrasing (IWP2005).
Xinshuai Dong, Anh Tuan Luu, Rongrong Ji, and Hong Liu. 2021. Towards robustness against natural lan- guage word substitutions. In International Confer- ence on Learning Representations.
Quoe Le and Tomas Mikolov. 2014. Distributed repre- sentations of sentences and documents. In Interna- rional conference on machine learning, pages 1188- 1196. PMLR
Tianyu Gao, Xingeheng Yao, and Danqi Chen. 2021. SimCSE: Simple contrastive learning of sentence em- beddings. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Process- ing, pages 6894-6910, Online and Punta Cana, Do- minican Republic. Association for Computational Linguisties.
Bohan Li, Hao Zhou, Junxian He, Mingxuan Wang, Yiming Yang, and Lei Li. 2020a. On the sentence embeddings from pre-trained language models. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 9119-9133 9130, Online. Association for Computa- tional Linguistics.
Siddhant Garg and Goutham Ramakrishnan. 2020.
BAE: BERT-based adversarial examples for text clas- sification. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 6174-6181, Online. Association for Computational Linguistics.
Jinfeng Li. Shouling Ji, Tianyu Du, Bo Li, and Ting Wang. 2019. TextBugger: Generating adversarial text against real-world applications. In Proceedings 2019 Network and Distributed System Security Sym- posium. Internet Society.
lan J. Goodfellow, Jonathon Shlens, and Christian Szegedy. 2015. Explaining and harnessing adver- sarial examples. In 3rd International Conference on Learning Representations, ICLR 2015, San Diego CA, USA, May 7-9, 2015, Conference Track Proceed- ings.
Linyang Li, Ruotian Ma, Qipeng Guo, Xiangyang Xue, and Xipeng Qiu. 2020b. BERT-ATTACK: Adversar- ial attack against BERT using BERT. In Proceed- ings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)), pages 6193 6202, Online. Association for Computational Linguistics.
Raia Hadsell, Sumit Chopra, and Yann LeCun. 2006. Dimensionality reduction by learning an invariant mapping. In 2006 IEEE Computer Society Confer- ence on Computer Vision and Pattern Recognition (CVPR"06) 06), volume 2, pages
Zhuorong Li. Daiwei Yu, Minghui Wu, Canghong Jin, and Hongchuan Yu. 2023. Adversarial su- pervised contrastive learning. Machine Learning. 1126))2105-2
IEEE
Jens Hauser, Zhao Meng, Damian Pascual, and Roger Wattenhofer. 2023, Bert robust! case against word substitution-based adversarial attacks. In ICASSP 2023-2023 IEEE International Confer- ence on Acoustics, Speech and Signal Processing (ICASSP), pages 1-5. IEEE.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Man- dar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized bert pretraining ap- proach.

---

Vasista Sai Lodagala, Sreyan Ghosh, and S Umesh. 2023. Coc-wav2vee 2.0: Clustering aided cross con- trastive self-supervised learning of speech represen- tations. In 2022 IEEE Spoken Language Technology Workshop (SLT), pages 1-8. IEEE
adversarial training in nlp. In Proceedings of the 2020 Conference on Empirical Methods in Natu- ral Language Processing: System Demonstrations, pages 119-126.
Arvind Neelakantan, Tao Xu, Raul Puri, Alee Rad- ford, Jesse Michael Han. Jerry Tworek, Qiming Yuan, Nikolas Tezak, Jong Wook Kim, Chris Hallacy, Johannes Heidecke, Pranav Shyam, Boris Power, Tyna Eloundou Nekoul, Girish Sastry, Gretchen Krueger. David Schnurr, Felipe Petroski Such, Kenny Hsu, Madeleine Thompson, Tabarak Khan, Toki Sherbakov, Joanne Jang, Peter Welinder, and Lilian Weng. 2022. Text and code embeddings by con- trastive pre-training.
Andrew L. Maas. Raymond E. Daly, Peter T. Pham, Dan Huang, Andrew Y. Ng. and Christopher Potts. 2011. Learning word vectors for sentiment analysis. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies. pages 142-150, Portland, Oregon, USA. Association for Computational Lin- guistics.
Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. 2018. Towards deep learning models resistant to adversarial attacks. In 6th International Conference on Learning Representations, ICLR 2018, Vancouver, BC, Canada, April 30 May 3, 2018, Conference Track Proceed- ings. OpenReview.net.
Jianmo Ni, Gustavo Hernandez Abrego, Noah Constant, Ji Ma, Keith Hall, Daniel Cer, and Yinfei Yang. 2022. Sentence-t5: Scalable sentence encoders from pre- trained text-to-text models. In Findings of the As- sociation for Computational Linguistics: ACI. 2022, pages 1864-1874 Dublin, Ireland, Association for Computational Linguistics.
Marco Marelli, Stefano Menini, Marco Baroni, Luisa Bentivogli, Raffaella Bernardi, and Roberto Zampar- elli. 2014. sick cure for the evaluation of composi- tional distributional semantic models. In Proceedings of the Ninth International Conference on Language Resources and Evaluation (LREC'14) 14). pages 216- 223.
Yixin Nie, Adina Williams, Emily Dinan, Mohit Bansal, Jason Weston, and Douwe Kiela. 2020. Adversarial NLI: A new benchmark for natural language under- standing. In Proceedings of the 58th Anmual Meet- ing of the Association for Computational Linguistics. pages 4885-4901, Online, Association for Computa- tional Linguistics.
Zhao Meng. Yihan Dong, Mrinmaya Sachan, and Roger Wattenhofer. 2022. Self-supervised contrastive learn- ing with adversarial perturbations for defending word substiqution-bast based attacks. In Findings of the Associ- ation for Computational Linguistics: NAACL 2022, pages 87-101, Seattle, United States. Association for Computational Linguistics.
Lin Pan, Chung-Wei Hang, Avirup Sil, and Saloni Pot- dar. 2022. Improved text classification via contrastive adversarial training. In Proceedings of the AAAI Con- ference on Artificial Intelligence, volume 36, pages 11130-17 11138.
Bo Pang and Lillian Lee. 2004. A sentimental education: Sentiment analysis using subjectivity summarization based minimum cuts. In Annual Meeting of the Association for Computational Linguistics.
Jan Hendrik Metzen, Tim Genewein, Volker Fischer, and Bastian Bischoff. 2017. On detecting adversar- ial perturbations. In International Conference on Learning Representations.
Bo Pang and Lillian Lee. 2005a. Seeing stars: Exploit- ing class relationships for sentiment categorization with respect to rating scales. In Proceedings of the 43rd Annual Meeting of the Association for Compu- tational Linguistics (ACL '05), pages 115-124, Ann Arbor, Michigan. Association for Computational Lin- guistics.
Deshui Miao, Jiaqi Zhang, Wenbo Xie, Jian Song, Xin Li, Lijuan Jia, and Ning Guo. 2021. Simple con- trastive representation adversarial learning for nip tasks.
Tomás Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. 2013a. Efficient estimation of word representa- tions in vector space. In 1st International Conference on Learning Representations, ICLR 2013, Scottsdale, Arizona, USA, May 2-4, 2013, Workshop Track Pro- ceedings.
Bo Pang and Lillian Lee. 2005b. Seeing stars: Exploit- ing class relationships for sentiment categorization with respect to rating scales. In Proceedings of the 43rd Annual Meeting of the Association for Compu- lational Linguistics (ACL '05), pages 1 115-124, Ann Arbor, Michigan. Association for Computational Lin- guistics.
Tomas Mikolov, Ila Sutskever, Kai Chen, Greg S Cor- rado, and Jeff Dean. 2013b. Distributed representa- tions of words and phrases and their compositionality. Advances in neural information processing systems, 26.
Jeffrey Pennington, Richard Socher, and Christopher Manning. 2014. GloVe: Global vectors for word representation. In Proceedings of the 2014 Confer- ence on Empirical Methods in Natural Language Pro- cessing (EMNLP), pages 1532-1543, Doha, Qatar. Association for Computational Linguistics.
John Morris, Eli Lifland, Jin Yong Yoo, Jake Grigsby, Di Jin and Yanjun Qi. 2020. Textattack: A frame- work for adversarial attacks, data augmentation, and

---

Yao Qiu, Jinchao Zhang, and Jie Zhou, 2021. Improv- ing gradient- based adversarial training for text clas- sification by contrastive learning and auto-encoder. In Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021. pages 1698-1707. Association for Computational Linguistics.
Tongzhou Wang and Phillip Isola. 2020. Understanding contrastive representation learning through alignment and uniformity on the hypersphere. In International Conference on Machine Learning, pages 9929-9939. PMLR.
Yisen Wang, Xingjun Ma, James Bailey, Jinfeng Yi, Bowen Zhou, and Quanquan Gu. 2019b. On the convergence and robustness of adversarial training, In Proceedings of the 36th International Conference on Machine Learning, volume 97 of Proceedings of Machine Learning Research, pages 6586-6595. PMLR.
Nils Reimers and Iryna Gurevych. 2019. Sentence-bert: Sentence embeddings using siamese bert-networks. In Conference on Empirical Methods in Natural Lan- guage Processing.
Shuhuai Ren, Yihe Deng, Kun He, and Wanxiang Che. 2019. Generating natural language adversarial exam- ples through probability weighted word saliency. In Proceedings of the 57th Annual Meeting of the Asso- ciation for Computational Linguistics, pages 1085- 1097, Florence, Italy. Association for Computational Linguistics.
Janyce Wiebe, Theresa Wilson, and Claire Cardie. 2005. Annotating expressions of opinions and emotions in language. Language resources and evaluation, 39(2):165-215
Adina Williams, Nikita Nangia, and Samuel Bowman. 2018. A broad-coverage challenge corpus for sen- tence understanding through inference. In Proceed- ings of the 2018 Conference of the North American Chapter of the Association for Computational Lin- guisties: Human Language Technologies, Volume (Long Papers), pages 11 12-11222 New Orleans, Louisiana. Association for Computational Linguis- ties.
Daniela N. Rima, DongNyeong Hco, and Heeyoul Choi. 2022. Adversarial training with contrastive learning alp. Computer Speech Language. Submitted.
Ali Shafahi, Mahyar Najibi, Zheng Xu, John Dickerson, Larry S Davis, and Tom Goldstein. 2020. Universal adversarial training. In Proceedings of the AAAI Con- ference Artificial Intelligence, volume 34, pages 5636-3 5643.
Chen Wu, Ruqing Zhang, Jiafeng Guo, Maarten De Ri- jke, Yixing Fan, and Xueqi Cheng. 2023. Prada: Practical box adversarial attacks against neural ranking models. ACM Transactions on Information Systems, 41(4):1-27.
Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D. Manning, Andrew Ng, and Christopher Potts. 2013. Recursive deep models for semantic compositionality over sentiment treebank. In Proceedings of the 2013 Conference on Empiri- cal Methods in Natural Language Processing, pages 163 1644-, Seattle, Washington, USA. Association for Computational Linguistics.
Han Xu, Yao Ma, Hao- Liu, Debayan Deb, Hui Liu, Ji-Liang Tang, and Anil Jain. 2020. Adversarial attacks and defenses in images, graphs and text: A review. International Journal of Automation and Computing. 17(2):151-778
Jianlin Su, Jiarun Cao, Weijie Liu, and Yangyiwen Ou. 2021. Whitening sentence representations for better semantics and faster retrieval. CoRR, abs/2103.1 153
Yuanmeng Yan, Rumei Li, Sirui Wang. Fuzheng Zhang, Wei Wu, and Weiran Xu. 2021. COnSERT: A con- trastive framework for self-supervised sentence repre- sentation transfer. In Proceedings of the 59th Annual Meeting of the Association for Computational Lin- guistics and the 11th International Joins Conference on Natural Language Processing (Volume 1: Long Papers), pages 5065-5075, Online, Association for Computational Linguistics.
Chi Sun, Xipeng Qiu, Yige Xu, and Xuanjing Huang. 2019. How to fine-tune bert for text classification? In China national conference on Chinese computational linguistics. pages 194-206. Springer.
Dilin Wang, Chengyue Gong, and Qiang Liu. 2019a. Improving neural language modeling via adversarial training. In International Conference on Machine Learning, pages 6555-6565, PMLR.
Zhilin Yang. Zihang Dai, Yiming Yang. Jaime Car- bonell, Russ Salakhutdinov, and Quoc V Le. 2019. Xlnet: Generalized autoregressive pretraining for lan- guage understanding, Advances in neural informa- tion processing systems, 32.
Dong Wang, Ning Ding, Piji Li, and Hai-Tao Zheng. 2021. Cline: Contrastive learning with semantic neg- ative examples for natural language understanding. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 1 1th International Joint Conference on Natural Lan- guage Processing, pages 2332-2342, Association for Computational Linguistics.
Peter Young, Alice Lai, Micah Hodosh, and Julia Hock- enmaier. 2014. From image descriptions visual denotations: New similarity metrics for semantic in- ference over event descriptions. Transactions of the Association for Computational Linguistics, 2:67-78.
Qian Wang, Weiqi Zhang, Tianyi Lei, Yu Cao, Dezhong Peng. and Xu Wang. 2023. Clsep: Contrastive learn- ing of sentence embedding with prompt. Knowledge- Baxed Systems, 266:1 10381
Xiang Zhang, Junbo Zhao, and Yann LeCun. 20 2015. Character-level convolutional networks for text classi- fication. Advances in neural information processing systems, 28:649-657.

---

Yan Zhang, Ruidan He. Zuozhu Liu, Kwan Hui Lim, and Lidong Bing. 2020. An unsupervised sentence embedding method by mutual information maximiza- tion. In Proceedings of the 2020 Conference Em- pirical Methods in Natural Language Processing, pages 1601-1610. Association for Computational Linguistics.
Bell 1o:31
Betle-3, 1e.4)
Be(lo-4, tes]
Bettes, 16-6)
a=[le:2 1e3]-
71.3
71.7
71.1
70.2
The
a-|le-3,
74.1
74.5
73.3
70.6
71
Chen Zhu, Yu Cheng, Zhe Gan, Siqi Sun, Tom Gold- stein, and Jingjing Liu. 2020. Freelh: Enhanced ad- versarial training for natural language understanding. International Conference on Learning Representa- Lions.
0-11c-4, 1c51-
77.4
STS
1o.61- 77.4
77.5
75.3
73.8
Yukun Zhu, Ryan Kiros, Rich Zemel, Ruslan Salakhut- dinov, Raquel Urtasun, Antonio Torralba, and Sanja Fidler. 2015. Aligning books and movies: Towards story-like visual explanations by watching movies and reading books. In Proceedings of the IEEE in- ternational conference on computer vision, pages 19-27.
Figure 4: The impact of step sizes in perturbation gener- ation on the average performance of STS tasks.
B.1 Step Sizes in Perturbation Generation
A Training Details
As depicted in Algorithm 1, the RobustEmbed framework incorporates two step sizes, denoted and B, perform iterative updates during the PGD and FGSM perturbation generation processes, respectively. Figure 4 illustrates the collaborative effect of varying ranges for these two step sizes in generating high-risk perturbations, which sig- nificant for achieving efficient contrastive learning objective. The results indicate greater improve- ment when /3 is adjusted in a lower range while a is placed in upper range. Specifically, better per- formance is observed when and are assigned ranges of [le-4, 1c-6] and [le-2, le-4J, respectively. Therefore, we utilize le-5 and B = 1e-3 for our experiments as achieves the best results among the different arrangements.
our experimental setup, we initialize our sen- tence encoder, denoted as fo. using the check- points obtained from BERT (Devlin et al., 2019) and RoBERTa (Liu et al., 2019). For sentence em- bedding, RobustEmbed utilizes the representation of the [CLS] token as the starting point and incor- porates a pooler layer top of the [CLS] represen- tations to facilitate contrastive learning objectives. The training process of RobustEmbed involves epochs, with model evaluation conducted every 250 training steps. The best checkpoint, determined by the highest average STS (Semantic Textual Similar- ity) score, is selected for final evaluation. To train the model, we utilize a dataset consisting of 10 ran- domly sampled sentences from English Wikipedia, provided by the SimCSE framework (Gao et al., 2021). The average training time for RobustEmbed is 2-4 hours. As our framework is initialized with pre-trained checkpoints, it exhibits robustness that is not sensitive to batch sizes, thus enabling us to employ batch sizes of either 64 or 128. In terms of transfer tasks, we determine the best hyperparame- ters based on the averaged score obtained from the development sets of six transfer tasks.
B.2 Step Numbers in Perturbation Generation
RobustEmbed applies T-step FGSM and K-step PGD iterations to obtain high-risk adversarial per- turbations for the contrastive learning objective. To simplify the analysis of perturbation generation it- erations, we set K = T. Figure demonstrates the impact of different step numbers (N = K or T) on effectiveness. We observe gradual improvement as N increases from 9; however, beyond N=9, the improvement becomes negligible. Moreover, a higher leads to longer running-time and unfair resource allocation. Hence, we select N=5 for our experiments.
B Ablation Studies
In this section, analyze the influence of four key hyperparameters in our approach the overall performance. We utilize BERT base as the encoder and evaluate the hyperparameters using the devel- opment set of STS tasks.

---

ing its effectiveness in generating more powerful perturbations. Therefore, adopt this setting the configuration of our framework.
RobustEmbed Accuracy on STS
77.0
76.5
X
Correlation
76.0
76.36
75.5
0.25
76.91
0.5
77.51
75.0
0.75
77.04
74.5
N-PGD + N-FGSM
76.48
5
11 N Value
13
15
17
19
Table 6: The impact of the modulation factor the average performance different Semantic Textual Sim- ilarity (STS) tasks generating the final perturbation.
Figure 5: The effect of the step number (denoted as N K or T) in the T-step FGSM and K-step PGD methods different Semantic
on the averaged correlation Textual Similarity (STS) tasks.
C Adversarial Training Comparison
To compare our framework with other standard adversarial training methods, we fine-tuned our pre-trained model using similar adversarial train- ing approach as the one employed during the pre- training phase. Subsequently, we compared the fine-tuned model with three standard adversarial training methods, namely PGD, FreeLLB (Zhu al., 2020), and SMART (Jiang et al., 2020), after the fine-tuning step, and presented the experimental results in table As shown, our framework outper- forms the three other adversarial training methods, achieving the highest average accuracy for STS and transfer tasks and the lowest average attack success rate under TextFooler, TextBugger, and BERTAt- tack attacks.
B.3 Norm Constraint
To ensure the imperceptibilitty of the generated ad- versarial examples, the magnitude of the pertur- bation vector, denoted as 6, is controlled Ro- bustEmbed. Three commonly used norm functions, namely L1, L2. and Lx.. are employed to restrict the magnitude of ô to small values. Table 5 presents the averaged Spearman's correlation of these norm functions across different Semantic Textual Simi- larity tasks. The Lox norm demonstrates superior correlation compared the other two norms, thus it is selected as the norm function for our experi- mental evaluation.
Norm
Correlation
D Contrastive Learning Loss
L
77.51
L2
76.82
The first part of the total contrastive loss (Equa- tion 8) optimizes the similarity between the input instance and its positive pair (z:"0), along with the similarity between and its adversarial pertur- bation (z.dd)). Although it indirectly brings pos and zddv closer, our observations show that regu- larizing the main objective function (Equation 7) through direct contrastive learning between 2os and adv (the second part of Equation 8) helps us achieve improved clean accuracy and robustness. Table 8 illustrates the effect of different values of the regularization parameter (r) on the final perfor- mance of framework. As can be seen, when 1/128, the framework achieves the highest average accuracy for STS and transfer tasks and the lowest average attack success rate under the TextFooler attack. We employ = 1/128 for all other experiments.
L1
75.28
Table 5: The influence of the norm constraint on pertur- bation generation on the average performance of various Semantic Textual Similarity (STS) tasks.
B.4 Modulation Factor
RobustEmbed incorporates a modulation factor, de- noted as 0 < A < 1, to adjust the relative sig- nificance of each separate perturbation (PGD and FGSM) in the formation of the final perturbation. The performance efficiency of various values for this modulation factor on semantic textual simi- larity tasks presented in Table The results indicate that À = 0.5 achieves the highest averaged correlation among the tested magnitudes, indicat-

---

Model
STS
Transfer
TextFooler
TextBugger
BERTAttack
PGD
76.37
79.15
50.33
31.05
49.72
FreeLB
81.91
86.03
48.70
27.11
47.83
SMART
82.65
87.34
45.46
26.08
47.39
RobustEmbed
85.79
89.86
37.12
20.43
39.25
Table 7: Performance Comparison of Adversarial Training Methods
7
STS
Transfer
TextFooler
(IMDb) Review dataset (Maas et al., 2011) con- tains 25,000 training and 25,000 testing highly polar samples, with negative and positive classes corresponding to review scores of <4 and 27 out of 10, respectively. Rotten Tomatoes Movie Re- views (MR) (Pang and Lee, 2005a) is a sentence- level dataset consisting of 8,530 training and 1,066 testing highly polar samples, where negative and positive classes are assigned based on calibration among different critics. SNLI (Bowman al., 2015) (MNLI (Williams et al., 2018)) is three- class dataset comprising 550,152 (392,702) train- ing and 10,000 (19,643) testing human- written sen- tence pairs English. Each set of three pairs SNLI (MNLI) is created using a different image caption from the Flicker30K dataset (Young et al., 2014) (ten sources of text), with the premise sen- tence serving the first sentence in each set. The hypothesis sentence of the first, second, and third pair is generated to be in entailment (category 1), contradiction (category 2), and neutral (category 3) with the respective premise sentence. While SNLI uses premise sentences from a relatively ho- mogeneous image caption dataset, MNLI covers a broader range of text styles. The MNLI testing sam- ple pairs are divided into two categories: "Matched" and "Mismatched," where MNLI-Matched pairs share similar context and resemblance the train- ing pairs compared to MNLL-Mismatched pairs.
1/64
76.46
85.93
44.37
1/128
77.51
86.20
37.97
1/256
77.06
85.87
40.32
1/512
75.84
84.66
42.58
Table 8: Effect of Regularization Parameter (1) on our
Framework Performance
E Text Classification Tasks
This section presents additional information on the text classification tasks used assess the gener- alization and robustness capabilities of our frame- work in comparison to various sentence embedding methods. The MR (Movie Reviews) dataset (Pang and Lee, 2005b) consists sentence-level samples with sentiment polarity, comprising 8,530 training and 1,066 testing highly polar instances. The CR dataset (Hu and Liu, 2004) is customer review dataset collected in three steps: extracting prod- uets with customer comments, identifying opin- ion sentences, and labeling each sentence as posi- tive or negative. The SUBJ dataset (Pang and Lee, 2004) contains 5,000 subjective and 5,000 objec- tive sentences from movie reviews, labeled based on subjectivity status and polarity. The MPQA dataset (Wiebe et al., 2005) includes annotated doc- uments from diverse news sources, categorizing opinion states such beliefs, emotions, sentiments, and speculations. The SST2 dataset (Socher al., 2013) is a sentence-level dataset with 8,544 training and 2,210 testing highly polar samples, extracted from movie reviews and classified as negative or positive. The MRPC dataset (Dolan and Brock- ett, 2005) contains 5,801 sentence pairs from news articles, labeled by human annotators to indicate semantic equivalence relationships. The YELP Po- larity Review (YELP) dataset (Zhang et al., 2015) consists of document-level samples, with 560,000 training and 38,000 testing highly polar instances classified as negative (1- and 2-star) or positive (4- and 5-star) reviews. The Internet Movie Database
F Adversarial Attack Methods
This section presents additional details on the di- verse adversarial attack techniques employed assess the robustness our sentence embedding framework. The TextBugger method (Li et al., 2019) identifies important words using the Jaco- bian matrix of the target model and selects op- timal perturbation from five types of generated perturbations. The PWWS method (Ren al., 2019) utilizes synonym-swap technique based on combination of word saliency scores and max- imum word-swap effectiveness. TextFooler (Jin et al., 2020) identifies important words, gathers

---

synonyms, and replaces each important word with the most semantically similar and grammatically correct synonym. The BAE method (Garg and Ra- makrishnan, 2020) employs four adversarial attack strategies involving word replacement or/and word insertion operations, where a portion of the text is masked and BERT MLM is used to generate substi- tutions. The BERTAttack method (Li et al., 2020b) consists of two steps: (a) searching for vulnera- ble words/sub-words and (b) using BERT MLM to generate eemantic-preservig substitutes for the vulnerable tokens.


Reviews:
[{"review_id": "kJwGeUuGjt", "reviewer": "Reviewer_a5gT", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "### Reasons To Reject (Question 1)\n\nQuestion: If I did not miss something, I am a bit confused on how the authors frame PGD and FGSM in the paper. Formally FGSM is a one-step attack without constraint, but in the paper FGSM is basically equivalent to PGD with P = 2. The authors should better clarify on the name conventions.\n\n#### Response:\n\nWe build upon the FGSM by introducing a perturbation constraint and updating the perturbation iteratively to create the Multi-step FGSM approach (T-FGSM). Essentially, T-step FGSM and K-step PGD are mathematically equivalent when P is either 2 or $\\infty$. Their primary distinctions lie in the number of iterations (i.e. T and K) and the step size of the attack (i.e. $\\alpha$ and $\\beta$) used to modify the input perturbation, ultimately generating a unique high-level perturbation. Since both K-PGD and T-FGSM are employed within the same embedding context, our framework combines these consistent perturbations in a straightforward yet efficient manner to generate the final perturbation for the contrastive learning objective. We appreciate your observation and will address this in greater detail in the final version of the paper.\n\n--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------\n--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------\n\n### Reasons To Reject (Question 2)\nQuestion: Lack of more intuitive understanding of the experimental results. The authors should include more evaluations on the sentence embeddings. For example, measuring the uniformity and alignment of the sentence embeddings by evaluating the average cosine or Euclidean similarities between the sentence embeddings.\n\n#### Response:\n\nIn response to the reviewer's concern, we followed the approach outlined by Wang and Isola [1] to utilize two crucial metrics, alignment and uniformity, for assessing the quality of our representation space. The fundamental concept underpinning these metrics is that positive instances should remain closely situated, while embeddings for random instances should be dispersed across the hypersphere. The subsequent table presents the experimental results of alignment and uniformity metrics for our framework, RobustEmbed, along with two other state-of-the-art embeddings, namely, DiffCSE [2] and SimCSE [3], on the STS-B test set. Smaller values indicate better performance. In comparison to the other two representations, RobustEmbed achieves a similar level of uniformity (-2.295 vs. -2.305) but exhibits superior alignment (0.071 vs. 0.091). This demonstrates that our framework is more efficient in optimizing the representation space in two distinct directions.\n\n|Model               |Alignment  | Uniformity |STS             |\n|---------------------|---------------|---------------|-----------------|\n|SimCSE           |0.226         |**-2.305**   |76.31   \t    |\n|DiffCSE            |0.091         |-1.637        |76.18           |\n|RobustEmbed  |**0.071**   |**-2.295**   |  **77.51**   |\n\n\n### References\n\n1. Wang, T., & Isola, P. (2020, November). Understanding contrastive representation learning through alignment and uniformity on the hypersphere. In _International Conference on Machine Learning_ (pp. 9929-9939). PMLR.\n2. Tianyu Gao, Xingcheng Yao, and Danqi Chen. (2021). SimCSE: Simple Contrastive Learning of Sentence Embeddings. In _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing_ (pp. 6894–6910). Association for Computational Linguistics.\n3. Yung-Sung Chuang, Rumen Dangovski, Hongyin Luo, Yang Zhang, Shiyu Chang, Marin Soljacic, Shang-Wen Li, Scott Yih, Yoon Kim, and James Glass. (2022). DiffCSE: Difference-based Contrastive Learning for Sentence Embeddings. In _Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ (pp.  4207–4218). Association for Computational Linguistics.\n\n--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------\n--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------\n\n### Reasons To Reject (Question 3)\nQuestion: Not clear to me why the convergence loss in Equation 8 is important, since the x^adv and x^pos are already being encouraged to similar in the first term. The authors should either provide more explanations or include an ablation study.\n\n#### Response:\n\nThe first part of the total contrastive loss (Equation 8) optimizes the similarity between the input instance _x_ and its positive pair ($x^{pos}$), along with the similarity between _x_ and its adversarial perturbation ($x^{adv}$). Although it indirectly brings $x^{pos}$ and $x^{adv}$ closer, our observations show that regularizing the main objective function (Equation 7) through direct contrastive learning between $x^{pos}$ and $x^{adv}$ (the second part of Equation 8) helps us achieve improved clean accuracy and robustness. The following table illustrates the effect of different values of the regularization parameter ($\\gamma$) on the final performance of our framework. As can be seen, when $\\gamma=1/128$, the framework achieves the highest average accuracy for STS and transfer tasks and the lowest average attack success rate under the TextFooler attack. We employ $\\gamma=1/128$ for all other experiments.\n\n|$\\gamma$ |STS          |Transfer   |TextFooler  |\n|---------------|--------------|--------------|---------------|\n|1/64           |76.46        |85.93       |44.37         |\n|1/128         |**77.51**  |**86.20**  |**37.97**   |\n|1/256         |77.06        |85.87       |40.32         |\n|1/512         |75.84        |84.66       |42.58         |\n\n-------------------------------------------------------------------------------------------------------------------------------------------------------\n-------------------------------------------------------------------------------------------------------------------------------------------------------\n\n### Missing References:\nThe idea of creating hard positives for sentence embeddings is not new. The authors might want to involve more discussions of this idea in the related work section.[1]\n\n[1]. Self-Supervised Contrastive Learning with Adversarial Perturbations for Defending Word Substitution-based Attacks. In Findings of NAACL 2022.\n\n#### Response:\n\nThank you for noting the missing reference. It is pertinent, and we will include it. Unlike the referenced paper, our framework utilizes high-risk perturbations in a novel self-supervised contrastive learning approach to enhance the target model's resilience against a range of adversarial attacks. The target model is not restricted to the BERT model, and our framework is compatible with other discriminative Pre-trained Language Models (PLMs).", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1.\tThe paper is very well written and easy to follow.\n2.\tThe proposed method is very straightforward and simple. It basically aims to maximize the similarity between the two dropout obtained sentence embeddings and another perturbed sentence embeddings.\n3.\tExperimental results are thorough, showing gains on both sentence similarity/sentence classification and robustness evaluation tasks.\n", "reasons_to_reject": "1.\tIf I did not miss something, I am a bit confused on how the authors frame PGD and FGSM in the paper. Formally FGSM is a one-step attack without constraint, but in the paper FGSM is basically equivalent to PGD with P = 2. The authors should better clarify on the name conventions.\n2.\tLack of more intuitive understanding of the experimental results. The authors should include more evaluations on the sentence embeddings. For example, measuring the uniformity and alignment of the sentence embeddings by evaluating the average cosine or Euclidean similarities between the sentence embeddings.\n3.\tNot clear to me why the convergence loss in Equation 8 is important, since the x^adv and x^pos are already being encouraged to similar in the first term. The authors should either provide more explanations or include an ablation study.\n", "questions_to_authors": "", "paper_topic_and_main_contributions": "This paper proposes a new contrastive learning method for learning robust sentence embeddings. Specifically, the method first generates two different versions of embeddings of a sentence by applying dropout twice with a backbone encoder. \n\nA perturbation is then added to one of two embeddings to maximize the contrastive learning loss, and thus creates a hard positive for the contrastive learning objective. To generate the perturbation, the authors leverage PGD and FGSM, which move the sentence embedding towards the direction of increasing the contrastive learning objective. PGD and FGSM are performed for K and T steps, respectively. The final perturbation is a balanced combination of the PGD perturbation and the FGSM perturbation. \n\nThe learning objective has two terms, with the first one aiming to maximize the similarity between the original sentence embedding, the dropout obtained sentence embedding, and the perturbed sentence embedding. The second term aims to converge the sentence embeddings of the perturbed sentence embedding and the dropout obtained sentence embedding.\n\nExperimental results show the proposed method 1) learns better sentence embeddings for downstream tasks, including sentence similarity and sentence classification tasks, and 2) learns more robust sentence embeddings against several adversarial attacks.\n", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Missing_References": "1.\tThe idea of creating hard positives for sentence embeddings is not new. The authors might want to involve more discussions of this idea in the related work section.[1] \n\n[1]. Self-Supervised Contrastive Learning with Adversarial Perturbations for Defending Word Substitution-based Attacks. In Findings of NAACL 2022.\n"}}, {"review_id": "JN6XRCAMtC", "reviewer": "Reviewer_ZN7y", "summary": "", "questions": "Please see Point 4 in Weaknesses.", "limitations": "", "rating": 2, "confidence": 5, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "### Reasons To Reject (Question 1)\n\nQuestion: The improvements on the semantic textual similarity and transfer tasks seem to be marginal.\n\n#### Response:\n\nRecent research has introduced sentence embeddings based on Pre-trained Language Models (PLMs), demonstrating promising performance across a range of semantic textual similarity and various downstream classification tasks. However, these embeddings exhibit vulnerability in adversarial environments, especially when subjected to external attacks. The core concept behind our proposed framework is to effectively address this vulnerability by generating robust sentence embeddings that are resilient in adversarial settings, particularly against a variety of adversarial attacks, while still maintaining proficiency in generalization, as evidenced by their promising performance across diverse downstream tasks.\n\nThe extensive experiments we have conducted demonstrate the exceptional performance of the framework in defending against state-of-the-art adversarial attacks within adversarial environments, in addition to showcasing improvements in semantic textual similarity and transfer tasks. The collective experimental results from Tables 1 to 3 substantiate the framework's capacity to successfully achieve its primary objectives while also enhancing performance in various semantic textual similarity and downstream classification tasks.\n\n-----------------------------------------------------------------------------------------------------------\n-----------------------------------------------------------------------------------------------------------\n\n### Reasons To Reject (Question 2)\n\nQuestion: Strong SOTA methods on the unsupervised embedding methods are not included. Another main baseline is USCAL, which was published in 2021. More recent baselines are necessary. \n \n#### Response:\n\nWe have already compared our framework with four recent baselines, including the USCAL baseline: SimCSE (2021) [1], ConSERT (2021) [2], USCAL (2021) [3], and ATCL (2022) [4]. The framework outperforms the USCAL embeddings in all conducted experiments by achieving higher average accuracy for STS and transfer tasks and a lower average attack success rate in the adversarial setting (Tables 1-3 in our submitted manuscript). To address the concern raised by the reviewer, we compared our framework with two other recent baselines, namely ESimCSE [5] and DiffCSE [6], and presented the experimental results in the following table. All experimental results were reproduced and reevaluated using our own configuration. As shown, the framework outperforms the two other embedding baselines by achieving the highest average accuracy for STS and transfer tasks and the lowest average attack success rate under TextFooler, TextBugger, and BERTAttack attacks. We will expand our experiments by incorporating additional adversarial attacks and also providing task-specific results for all of the STS and transfer tasks utilized in our work.\n\n|Model                         |STS       |Transfer   |TextFooler  |TextBugger |BERTAttack |\n|-----------------------------|------------|-------------|---------------|----------------|-----------------|\n|ESimCSE-BERT        |76.29\t   |85.90       |65.19         |41.77          |64.19           |\n|DiffCSE-BERT      \t   |76.40     |85.74       |54.91         |36.72          |52.33           |\n|RobustEmbed-BERT |**77.51**|**86.20**  |**37.97**   |**20.98**    |**39.62**      |\n\n### References\n1. Tianyu Gao, Xingcheng Yao, and Danqi Chen. (2021). SimCSE: Simple Contrastive Learning of Sentence Embeddings. In _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing_ (pp. 6894–6910). Association for Computational Linguistics.\n2.  Yuanmeng Yan, Rumei Li, Sirui Wang, Fuzheng Zhang, Wei Wu, and Weiran Xu. (2021). ConSERT: A Contrastive Framework for Self-Supervised Sentence Representation Transfer. In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (pp 5065–5075). Association for Computational Linguistics.\n3. Deshui Miao, Jiaqi Zhang, Wenbo Xie, Jian Song, Xin Li, Lijuan Jia, and Ning Guo. (2021).  Simple contrastive representation adversarial learning for nlp tasks.\n4. Daniela N. Rima, DongNyeong Heo, and Heeyoul Choi. Adversarial training with contrastive learning in nlp.  Computer Speech & Language. Submitted.\n5.  Xing Wu, Chaochen Gao, Liangjun Zang, Jizhong Han, Zhongyuan Wang, and Songlin Hu. (2022). ESimCSE: Enhanced Sample Building Method for Contrastive Learning of Unsupervised Sentence Embedding. In _Proceedings of the 29th International Conference on Computational Linguistics_ (pp. 3898–3907). International Committee on Computational Linguistics.\n6. Yung-Sung Chuang, Rumen Dangovski, Hongyin Luo, Yang Zhang, Shiyu Chang, Marin Soljacic, Shang-Wen Li, Scott Yih, Yoon Kim, and James Glass. (2022). DiffCSE: Difference-based Contrastive Learning for Sentence Embeddings. In _Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ (pp. 4207–4218). Association for Computational Linguistics.\n-----------------------------------------------------------------------------------------------------------\n-----------------------------------------------------------------------------------------------------------\n\n### Reasons To Reject (Question 3)\n\nQuestion: The novelty is somewhat limited. The adversarial methods used in this paper including FGSM and PGD are well-known and general methods for embedding perturbation. If there are specific designs which are important to sentence embeddings, the proposed method can be more novel. For example, since embeddings are continuous and not all the embeddings in the continuous space are meaningful, how to ensure the adversarial perturbations are semantically meaningful can be worth thinking about.\n\n#### Response:\n\nIn recent research, various adversarial techniques, including Fast Gradient Sign Method (FGSM), Fast Gradient Method (FGM), Projected Gradient Descent (PGD), and Multi-step PGD (K-PGD), have been employed to generate adversarial perturbations in continuous embedding spaces. These techniques, although not originally designed for sentence embeddings, have been adapted for consistency within this space. Our framework intelligently expands upon the FGSM by introducing a Multi-step FGSM approach (T-FGSM) to acquire the perturbation that represents the worst-case scenario. As both K-PGD and T-FGSM are applied within the same embedding context, they can be combined into a novel practical perturbation generator. Consequently, the RobustEmbed framework employs a parameter $\\lambda$ to balance the relative significance of each individual perturbation in generating the final perturbation (Equation 5).    \n\nWe employ two key metrics [1] to assess the semantic meaningfulness of our sentence embedding framework, namely, alignment and uniformity. The metrics rely on the idea that similar semantic instances should be close together, while embeddings for random instances should be spread widely across the hypersphere. In our experimental results, which are shown in the following table, we evaluate the alignment and uniformity metrics for our framework using various perturbation generation techniques during training. Smaller metric values indicate better performance. Notably, in comparison to the other two perturbation generation methods, the combination of K-PGD and T-FGSM (K-PGD + T-FGSM) demonstrates superior efficiency in generating semantically meaningful perturbations within the embedding space.\n\n|Perturbation                   |Alignment  |Uniformity  |STS           |\n|--------------------------------|---------------|---------------|---------------|\n|_FGSM_                        |0.258         |  -1.005      |74.95         |\n|_PGD_                          |0.172          |-1.653        |76.16         |\n|_K-PGD_ + _T-FGSM_ |**0.071**    |**-2.295**   |  **77.51** |\n\n\n### References\n1. Wang, T., & Isola, P. (2020, November). Understanding contrastive representation learning through alignment and uniformity on the hypersphere. In _International Conference on Machine Learning_ (pp. 9929-9939). PMLR.\n\n------------------------------------------------------------------------------------------------------------------------------------------------------------------------------\n------------------------------------------------------------------------------------------------------------------------------------------------------------------------------\n\n### Reasons To Reject (Question 4)\n\nQuestion: Another major problem lies in the evaluation. The current experiment suggests the robustness of the proposed method using text classification tasks which are originally designed for detecting the robustness or BERT. But does robustness in classification mean robustness in sentence embeddings? Or does robustness in BERT mean robustness in sentence embeddings? The evaluation set are not strongly related to sentence embeddings. For example, two paraphrase sentences “this movie is not bad at all” vs. “This film is actually quite good.”, their semantics are almost the same, so the sentence embeddings should be very close. But for classification performances, different models might give different results. Similarly, when the classification results for two sentences are the same do not mean that their sentence embeddings are close to each other. To solve these problems, a new evaluation benchmark is necessary, which is specially designed for direct evaluations of sentence embeddings.\n\n#### Response: \n\nSemantic Textual Similarity (STS) is an NLP task that measures the similarity between text segments, usually sentences, by converting them into numerical vectors using sentence embeddings. This allows for comparing sentences based on their underlying semantic content to determine how closely their meanings match in a continuous vector space. To address the concern raised by the reviewer, we introduced a new task called Adversarial Semantic Textual Similarity (AdvSTS) to evaluate the robustness of sentence embeddings in our representation framework. AdvSTS employs an efficient adversarial technique, like TextFooler, to manipulate an input sentence pair in a way that misleads the target model into generating a regression score that maximally deviates from the actual score (truth label). Consequently, we create an adversarial STS dataset by transforming all benign instances from the original dataset into adversarial examples. Similar to the STS task, AdvSTS uses Pearson's correlation metric to assess the correlation between the predicted similarity scores produced by the target model and the human-annotated similarity scores for the adversarial dataset.\n\nThe table below presents the success rates of three adversarial attack techniques (TextFooler, TextBugger, and BERTAttack) on three sentence embeddings, including our framework, for two AdvSTS tasks (AdvSTS-B and AdvSICK-R). Our embedding framework outperforms the other two methods, demonstrating significantly lower attack success rates across both AdvSTS tasks and all adversarial attack techniques. We will expand our experiments by incorporating additional adversarial attacks and potentially more embedding methods to provide a comprehensive assessment of the robustness in sentence embeddings.\n\n\n|Adversarial Attack    |Model       |AdvSTS-B   |AdvSICK-R    |Avg.        |\n|---------------------------|--------------|-----------------|------------------|--------------|\n|TextFooler         |SimCSE-BERT             |21.07          |24.17          |22.62        |\n |TextFooler        |USCAL-BERT               |16.52          |18.71          |71.61        |\n|TextFooler         |RobustEmbed-BERT   | **7.48**      | **8.95**     | **8.22**    |\n|TextBugger        |SimCSE-BERT            |27.49          |28.34          |27.91        |\n|TextBugger        |USCAL-BERT              |21.52          | 24.88         | 23.2         |\n|TextBugger        |RobustEmbed-BERT  | **11.76**    |**13.01**    |**12.39**   |\n|BERTAttack      |SimCSE-BERT            |31.60           |32.85          |32.23        |\n|BERTAttack      |USCAL-BERT              |26.02           | 28.51         |27.26        |\n|BERTAttack      |RobustEmbed-BERT  |**12.99**      |**13.18**    |**13.09**   |\n\n------------------------------------------------------------------------------------------------------------------------------------------------------------------------------\n------------------------------------------------------------------------------------------------------------------------------------------------------------------------------\n\n### Missing References:\nRecent SOTA methods on self supervised sentence embeddings should be cited.\n\n#### Response:\n\nThank you for pointing out the missing references. We will include the following state-of-the-art (SOTA) self-supervised sentence embeddings in our paper:\n\n### References\n\n1.  Xing Wu, Chaochen Gao, Liangjun Zang, Jizhong Han, Zhongyuan Wang, and Songlin Hu. (2022). ESimCSE: Enhanced Sample Building Method for Contrastive Learning of Unsupervised Sentence Embedding. In _Proceedings of the 29th International Conference on Computational Linguistics_ (pp. 3898–3907). International Committee on Computational Linguistics.\n2. Yung-Sung Chuang, Rumen Dangovski, Hongyin Luo, Yang Zhang, Shiyu Chang, Marin Soljacic, Shang-Wen Li, Scott Yih, Yoon Kim, and James Glass. (2022). DiffCSE: Difference-based Contrastive Learning for Sentence Embeddings. In _Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ (pp. 4207–4218). Association for Computational Linguistics.\n\n------------------------------------------------------------------------------------------------------------------------------------------------------------------------------\n------------------------------------------------------------------------------------------------------------------------------------------------------------------------------\n\n### Typos Grammar Style And Presentation Improvements:\nThere are multiple errors for citation format. For example, “SimCSE Gao et al. (2021) introduced”, “Pan et al. Pan et al. (2022)”. Try to use cite or citet (or newcite) in the correct way.\n\n#### Response:\n\nThank you for bringing the citation errors to our attention. We will rectify these errors in the final version of the paper.\n\n------------------------------------------------------------------------------------------------------------------------------------------------------------------------------\n------------------------------------------------------------------------------------------------------------------------------------------------------------------------------\n\n### Typos Grammar Style And Presentation Improvements:\nSection 3.1 might not be necessary.\n\nResponse:\n\nWe will remove this section in the final version of the paper.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The overall idea is easy to follow and the related implementation will be released.\n2. On the classification tasks, the proposed method significantly outperforms the baselines.", "reasons_to_reject": "1. The improvements on the semantic textual similarity and transfer tasks seem to be marginal.\n\n\n2. Strong SOTA methods on the unsupervised embedding methods are not included. Another main baseline is USCAL, which was published in 2021. More recent baselines are necessary.\n\n\n3. The novelty is somewhat limited. The adversarial methods used in this paper including FGSM and PGD are well-known and general methods for embedding perturbation. If there are specific designs which are important to sentence embeddings, the proposed method can be more novel. For example, since embeddings are continuous and not all the embeddings in the continuous space are meaningful, how to ensure the adversarial perturbations are semantically meaningful can be worth thinking about.\n\n\n4. Another major problem lies in the evaluation. The current experiment suggests the robustness of the proposed method using text classification tasks which are originally designed for detecting the robustness or BERT. But does robustness in classification mean robustness in sentence embeddings? Or does robustness in BERT mean robustness in sentence embeddings? The evaluation set are not strongly related to sentence embeddings. For example, two paraphrase sentences “this movie is not bad at all” vs. “This film is actually quite good.”, their semantics are almost the same, so the sentence embeddings should be very close. But for classification performances, different models might give different results. Similarly, when the classification results for two sentences are the same do not mean that their sentence embeddings are close to each other. To solve these problems, a new evaluation benchmark is necessary, which is specially designed for direct evaluations of sentence embeddings.", "questions_to_authors": "Please see Point 4 in Weaknesses.", "paper_topic_and_main_contributions": "This paper works on robust sentence embeddings with self-supervised learning methods. The authors propose a robust sentence embedding method called RobustEmbed, which uses adversarial perturbations to strengthen sentence embeddings. The proposed method is evaluated on three settings, including semantic textual similarity tasks, six transfer tasks, and text classification tasks. Experimental results show that RobustEmbed can improve over investigated baselines.", "excitement": 2, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "Recent SOTA methods on self supervised sentence embeddings should be cited."}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "1. There are multiple errors for citation format. For example, “SimCSE Gao et al. (2021) introduced”, “Pan et al. Pan et al. (2022)”. Try to use cite or citet (or newcite) in the correct way.\n\n2. Section 3.1 might not be necessary."}}, {"review_id": "4mKju1oXyD", "reviewer": "Reviewer_hADL", "summary": "", "questions": "NA", "limitations": "", "rating": 3, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "### Reasons To Reject (Question 1)\n\nQuestion: Experiments on adversarial robustness with 100 examples are not convincing, even though the author ran them five times. The common practice is to use at least 1000 examples.\n\n#### Response:\n\nTo address the concern raised by the reviewer, we conducted adversarial robustness experiments with 1,000 samples per experiment. To ensure statistical validity, each experiment was repeated five times. The table below presents the success rates of three adversarial attack techniques (TextFooler, TextBugger, and BERTAttack) on three sentence embeddings, including our framework, for three downstream tasks (IMDB, MRPC, and SNLI). Our embedding framework outperforms the other three methods, demonstrating remarkably lower attack success rates across three classification tasks and all adversarial attack techniques. We plan to expand our experiments by incorporating additional adversarial attacks and more downstream classification tasks to provide a comprehensive assessment of the robustness in sentence embeddings.\n\n|Adversarial Attack|Model        \t         |IMDB        |MRPC     |SNLI         |\n|------------------------|-----------------------------|--------------|-------------|--------------|\n|TextFooler    \t      |SimCSE-BERT  \t  |72.50       |75.82       |69.42       |\n|TextFooler    \t      |USCAL-BERT   \t  |59.92       |59.43       |53.17       |\n|TextFooler    \t      |RobustEmbed-BERT | **38.13** | **37.09** | **37.25**|\n|TextBugger           |SimCSE-BERT  \t  |50.73       |54.91       |46.26       |\n|TextBugger           |USCAL-BERT   \t  |38.14       | 37.05      | 31.71      |\n|TextBugger           |RobustEmbed-BERT | **22.52** |**21.62**  |**20.16** |\n|BERTAttack          |SimCSE-BERT  \t  |74.66       |79.26       |75.52       |\n|BERTAttack          |USCAL-BERT   \t  |61.52       | 60.87\t    |54.03       |\n|BERTAttack          |RobustEmbed-BERT |**41.73**  |**40.65** |**41.49**  |\n\n-----------------------------------------------------------------------------------------------------------\n-----------------------------------------------------------------------------------------------------------\n\n### Reasons To Reject (Question 2)\n\nQuestion: There is a lack of comparison with other basic adversarial training methods, such as basic PGD or FreeLB.\n\n#### Response:\n\nFreeLB employs supervised adversarial training during fine-tuning, while our framework utilizes self-supervised adversarial training during pre-training. Due to this fundamental difference, a direct fair comparison between these methods is not feasible. Our framework is therefore only comparable with unsupervised/self-supervised adversarial training methods at the pre-training stage. In this context, we have conducted comparative assessments with two state-of-the-art methods, ConSERT [1] and USCAL [2], as detailed in Tables 1-3 of our submitted manuscript. To address the concern raised by the reviewer, we compared our framework with three unsupervised adversarial training methods, namely PGD, Multi-step PGD [3], and ALUM [4], and presented the experimental results in the following table.  As shown, the framework outperforms the three other baselines by achieving the highest average accuracy for STS and transfer tasks and the lowest average attack success rate under TextFooler, TextBugger, and BERTAttack attacks. We plan to expand our experiments by incorporating additional adversarial attacks and also providing task-specific results for all of the STS and transfer tasks utilized in our research work.\n\n|Model                          |STS          |Transfer    |TextFooler |TextBugger  |BERTAttack   |\n|------------------------------|--------------|--------------|---------------|----------------|-------------------|\n|PGD-BERT                   |64.24       |76.15  \t|51.36         |30.45           |49.17             |\n|Multi-step PGD-BERT  |66.91       |80.81       |47.14         |28.09           |46.96             |\n|ALUM-BERT       \t      |68.25       |84.37  \t|46.70         |27.38   \t|46.38             |\n|RobustEmbed-BERT    |**77.51** |**86.20**  |**37.97**   |**20.98**      |**39.62**       |\n\n\n### References\n1. Yuanmeng Yan, Rumei Li, Sirui Wang, Fuzheng Zhang, Wei Wu, and Weiran Xu. (2021). ConSERT: A Contrastive Framework for Self-Supervised Sentence Representation Transfer. In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (pp 5065–5075). Association for Computational Linguistics.\n2. Deshui Miao, Jiaqi Zhang, Wenbo Xie, Jian Song, Xin Li, Lijuan Jia, and Ning Guo. (2021).  Simple contrastive representation adversarial learning for nlp tasks.\n3. Chen Zhu, Yu Cheng, Zhe Gan, Siqi Sun, Tom Goldstein, and Jingjing Liu. (2020). FreeLB: Enhanced Adversarial Training for Language Understanding. In Proceedings of the International Conference on Learning Representations (ICLR).\n4. Liu, X., Cheng, H., He, P., Chen, W., Wang, Y., Poon, H., & Gao, J. (2020). Adversarial training for large neural language models. _arXiv preprint arXiv:2004.08994.\n\n-----------------------------------------------------------------------------------------------------------\n-----------------------------------------------------------------------------------------------------------\n\n### Reasons To Reject (Question 3)\n\nQuestion: As a critical point, the reviewer remains unconvinced that solely using unsupervised contrastive loss in perturbation construction can lead to adversarial robustness.\n\n#### Response:\n\nIt is noteworthy that unsupervised contrastive loss encourages the model to learn more discriminative features for input data. It encourages the model to push embeddings of similar instances closer together in the feature space while pushing embeddings of dissimilar instances apart. Furthermore, our framework employs an efficient combination of FGSM and PGD techniques iteratively to generate the worst-case perturbation in the embedding space. Subsequently, this perturbation is employed to optimize the similarity between the benign input sample and its corresponding adversarial perturbation in a contrastive manner. Consequently, unsupervised contrastive loss trains the model to align input samples and their worst-case perturbed embeddings within a meaningful continuous space. This strategic alignment effectively encompasses a wide range of adversarial examples in the raw text space. This enhanced discriminative ability makes it more challenging for adversaries to discover perturbations capable of deceiving the model.\n\nOur empirical findings in Table 3 demonstrate the robustness superiority of this idea, employed in our framework, against five challenging adversarial attacks when compared to two state-of-the-art sentence embedding methods.\n\n-----------------------------------------------------------------------------------------------------------\n-----------------------------------------------------------------------------------------------------------\n\n### Missing References:\nShi, Jiahui, Linjing Li, and Daniel Zeng. \"ASCL: Adversarial supervised contrastive learning for defense against word substitution attacks.\" Neurocomputing 510 (2022): 59-68.\n\n\n#### Response:\n\nThank you for noting the missing reference. It is pertinent, and we will include it. Unlike the referenced paper, our framework utilizes high-risk perturbations in a self-supervised contrastive learning approach to enhance the target model's resilience against a range of adversarial attacks.\n\n-----------------------------------------------------------------------------------------------------------\n-----------------------------------------------------------------------------------------------------------\n\n### Reproducibility: 2 \nWould be hard pressed to reproduce the results. The contribution depends on data that are simply not available outside the author's institution or consortium; not enough details are provided.\n\n#### Response: \n\nOur source code and necessary data for training the framework are publicly available in the following GitHub project: (https://github.com/jasl1/RobustEmbed). To train the framework, we utilize a dataset consisting of $10^6$ randomly sampled sentences from the English Wikipedia, as provided by the SimCSE framework [1]. Users can easily reproduce the experimental results reported in the paper by following the instructions available in the README file of the GitHub project. Detailed information about the training configuration is also provided in the \"Training Details\" (Appendix A) section of our manuscript.\n\n\n### References\n1. Tianyu Gao, Xingcheng Yao, and Danqi Chen. (2021). SimCSE: Simple Contrastive Learning of Sentence Embeddings. In _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing_ (pp. 6894–6910). Association for Computational Linguistics.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "* This work tackles the valuable problem of adversarial robustness.\n\n* The structure of the paper is clean.", "reasons_to_reject": "* Experiments on adversarial robustness with 100 examples are not convincing, even though the author ran them five times. The common practice is to use at least 1000 examples.\n\n* There is a lack of comparison with other basic adversarial training methods, such as basic PGD or FreeLB.\n\n* As a critical point, the reviewer remains unconvinced that solely using unsupervised contrastive loss in perturbation construction can lead to adversarial robustness.", "questions_to_authors": "NA", "paper_topic_and_main_contributions": "This paper introduces contrastive learning to enhance sentence embeddings of PLMs for improved adversarial robustness. Experiment results show significant robustness improvements on STS and transfer tasks.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "Shi, Jiahui, Linjing Li, and Daniel Zeng. \"ASCL: Adversarial supervised contrastive learning for defense against word substitution attacks.\" Neurocomputing 510 (2022): 59-68.", "Typos_Grammar_Style_and_Presentation_Improvements": "NA"}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper introduces RobustEmbed, a self-supervised method for learning robust sentence embeddings. The method applies dropout and adversarial perturbations to generate hard positives for contrastive learning. The method optimizes a loss function that encourages similarity between the original, dropout, and perturbed embeddings, while diverging the dropout and perturbed embeddings. The method shows improvement over baselines on various tasks as well, such as semantic textual similarity, transfer learning, and text classification. The paper is well written with a defined structure and provides high resilience against various adversarial attacks being comparable to other embeddings in representation and classification tasks at the same time.
Decision: Accept-Findings