Contrastive Learning of Sentence Embeddings from Scratch
Authors: Junlei Zhang
Zhenzhong Lan
Junxian He
Venue: EMNLP 2023 Main (2023)
Keywords: contrastive learning
sentence embeddings

Abstract: Contrastive learning has been the dominant approach to train state-of-the-art sentence embeddings. Previous studies have typically learned sentence embeddings either through the use of human-annotated natural language inference (NLI) data or via large-scale unlabeled sentences in an unsupervised manner. However, even in the case of unlabeled data, their acquisition presents challenges in certain domains due to various reasons. due to copyright restrictions, data distribution issues, and messy formats, among other factors. To address these issues, we present SynCSE, a contrastive learning framework that trains sentence embeddings with synthetic data. Specifically, we explore utilizing large language models to synthesize the required data samples for contrastive learning, including (1) producing positive and negative annotations given unlabeled sentences SynCSE-partial, and (2) generating sentences along with their corresponding annotations from scratch SynCSE-scratch. Notably, SynCSE-scratch constitutes the first contrastive learning method to learn sentence embeddings from scratch without manually collecting any data sample. Experimental results on sentence similarity and reranking tasks indicate that both SynCSE-partial and SynCSE-scratch greatly outperform unsupervised baselines, and SynCSE-partial even achieves comparable performance to the supervised models in most settings.

Contrastive Learning of Sentence Embeddings from Scratch
Junlei Zhang 1.2
Zhenzhong Lan² Junxian Hel
'Zhejiang University Schooo of Engineering, Westlake University
The Hong Kong University of Science and Technology (zhangjunlei, Lan
edu. cn, junxianh@cse att..a.iee......iiiiiiieeee .utt.kk hk
Abstract
I want collect some sentences from social medial platforms.
Contrastive learning has been the dominant ap- proach to train state-of-the-art sentence emhed- dings. Previous studies have typically learned sentence embeddings either through the use of human-annotated natural language inference (NLJ) data via large-scale unlabeled sen- tences in an unsupervised manner. However, even the case !junlabeled data, their acqui- sition presents challenges in certain domains due to various reasons. To address these is- sues, we present SynCSE, a contrastive learn- ing framework that trains sentence embeddings with synthesized data. Specifically, we explore utilizing large language models synthesize the required data samples for contrastive learn- ing, including (1) producing positive and neg- ative annotations given unlabeled sentences (SynCSE-partiue and (2) generating sentences along with their corresponding annotations from scratch (SynCSE-scraacc) Experimen- tal results on sentence similarity and rerank- ing tasks indicate that both SynCSE-partia and SynCSE-scratch greatly outperform unsu- pervised baselines, and SynCSE-partia even achieves comparable performance to the super- vised models in most settings.
Prompt
Devise ten distinct and diverse senterices thalymay appear in pieces of content shared on social media platforms, covering a range of subjects (education, food, technology, history, architecture, etc.). sentences elementary structures akin "Birds fly the sky." to more sophisticated ones Aim for low degree of lexical overlap
present mix complexity levels, from
this colarful sunset at the beach today O
Exploring city stumbled upon architecture!
Hard negative prompt
Positive prompt
sew sunset et the beach today.
spent the entire day indoors
My city exploration led me to beautifuu building.
The architecture in the city was disappointing unaitractive.
Figure I: An overview of the data synthesis process of SynCSE-scratch. We specify a desired domain and genre, and framework will generate diverse unla- beled data for that domain along with their positive and negative annotations.
1 Introduction
et al., 2022: Linkonchotiwat et al., 2022; Wu et al., 2022a; Wang et al., 2022c; He et al., 2023).
The objective of sentence representation learning is to derive sentence embeddings that can bene- fit a wide range of downstream tasks, including reranking (Lee et al., 2021; Barker et al., 2021). natural language understanding (Cer et al., 2018), and retrieval (Misra et al., 2016; Thakur et al., 2021; Wang et al., 2022a). Methods built contrastive learning, such SimCSE (Gao al., 2021) and PromCSE (Jiang et al.. 2022b), have dominated the field due to their competitive performance (Zeng
Contrastive learning trains sentence representa- tions through distinguishing positive samples from negative ones. In this framework, the quality of these positive and negative annotations plays crit- ical role. Supervised approaches typically gather these annotations from labeled natural language in- ference (NLI) datasets (Jiang et al., 2022a; Limkon- chotiwat et al., 2022) however, such sources are generally unavailable for most settings, and manu- ally creating them is cost- post-proiiiiive. As result, unsupervised methods that solely rely on unlabeled sentences attract significantly more attention re-
"Work done during Junlei's visit HKUST.
"Corresponding author. Coode and the synthesized datascts are available at

---

cently (Gao et al., 2021; Zhou et al., 2022; Wu al.. 2022a) they mostly develop methods to automatically obtain positive and negative samples to facilitate contrastive learning. A representative example is SimCSF (Gao et al., 2021), which lever- ages perturbed hidden states as the positive samples and in-batch sentences negatives to perform con- trastive learning. To differentiate between in-batch negatives and the annotated negatives, the latter often termed "hard negatives", which have proven to be significantly advantageous in enhancing sen- tence embeddings (Wang et al., 2022b.c).
supervised counterpart on STS, marking the first instance of an unsupervised method matching su- pervised results on this benchmark. We release our synthesized datasets to facilitate further research to learn better sentence embeddings.
2 SynCSE
2.1
Background
We base our approach on the formulation of Sim- CSE (Gao et al., 2021), which is one of the most common and effective contrastive learning frame- works to learn sentence embeddings. Formally, we denote the unlabeled sentence itive sample as 2;- Let h; and h denote the representations of x; and 2,t respectively, then the unsupervised SimCSE loss is defined as:
Despite considerable advances in recent years, the performance of these unsupervised methods still falls short when compared to their supervised counterparts. Moreover, the unavailability of large- scale unlabeled data for the targeted domain often poses additional limitations to these approaches. To overcome these challenges, we introduce SynCSE, an unsupervised contrastive framework that trains sentence embeddings with synthesized data. Con- cretely, we propose to prompt large language mod- els (LLMs) such as ChatGPT (OpenAI, 2022) to synthesize the samples needed for contrastive learn- ing. This is inspired by recent successes of prompt- ing large language models (LLMs) to perform various tasks (Chung et al., 2022; Ouyang al., 2022; OpenAI, 2023), especially the superior per- formance of LLMs over corke- on text an- notation (Gilardi et al., 2023). We investigate two variants of SynCSE in this work that correspond to two practical scenarios: (1) SynCSE-partial where large-scale unlabeled sentences are available and LLMs are prompted to produce positive and hard negative annotations, and (2) SymCSE-scratee -cratcch where large- scale unlabeled sentences are not avail- able, prompting LLMs to generate sentences and their corresponding annotations from scratch. The latter represents a particularly challenging yet prac- tical scenario where we aim to learn sentence em- beddings without any data samples.
and its pos-
- -log
(1)
M ===1
sim(hs,h 1/7
where M denotes the mini-batch's size, T is a tem- perature hyperparameter, and sim stands for a similarity function. Unsupervised SimCSE passes the same Ij twice to the encoder to form (h,, h, pairs due to random dropout, and other sentences within the same mini-batch are considered as neg- ative samples shown in Eq. 1. Supervised SimCSE further extends (z,, x. with hard neg- ative samples 2; to constitute the triplet datasets (x;, at. a i==1 and define the supervised loss:
simm(t,
-log
2==1 wim( D/T + simi(h, 1/T)
(2)
In supervised SimCSE, the (I,, 2. triplets are typically from annotated NI. datasets, where 2; is the premise, xrf and x; are the entailment and con- tradiction hypotheses. Supervised SimCSE signifi- cantly outperforms the unsupervised one due to the enhanced quality of positive and hard negative sam- ples. However, such annotated data are typically un- available in most settings, and manually annotating triplets (I;, "F. 2x. can be resource-intensive, ren- dering unsupervised approaches the most promis- ing choices in practice. In this work, we focus the supervised loss in Eq. 2, but synthesize (xez x,.) given x; or even generate (xi, a 2,. triplets from scratch, aiming to approach the perfor- mance of supervised models with an unsupervised method. We describe our data synthesis process next.
We conduct comprehensive experiments on the standard Semantic Textual Similarity (STS) bench- mark, along with four reranking tasks and four do- main adaptation tasks. Our results demonstrate that both SynCSE-partial and SynCSE-scratch substan- tially outperform the unsupervised baselines in all cases for example. SynCSEE-partial and SynCSE- scratch exceed the unsupervised SimCSE baseline by 5.37 and 4.18 absolute points respectively on STS. Particularly, SynCSE-pan partial often equals its

---

Hard negative prompts pools
Revise the provided sentence by swapping, changing, contradicting some details in order to express different meaning, while maintaining the general context and structure.
Promptl:
Prompi2:
Generate slightly modified version of the provided sentence to express an opposing or alternate meaning by changing one or two specific elements, while maintaining the overall context and sentence structure.
Prompt3:
Transform the input sentence by adjusting, altering, or contradicting its original meaning to create logical and sensible output sentence with a diffcrent meaning from the input sentence.
Generate sentence that conveys altering, contrasting opposite idea to the given input sentence, while ensuring the new sentence is logical, realistic, and grounded in common sense.
Prompt4:
Table I: Hard negative prompts pools. During the generation of hard negative samples, hard negative prompt is randomly sampled each time.
positive/hard negative labels with prompt/example pools.
[S-shot examplesi..
2.3 SynCSE-partial
Please paraphrase the input sentence, providing an alternative expression with the same meaning. The input is: instructions minutely.
Synthesizing positive and hard negative exam- ples: We prompt ChatGPT in a few-shot set- ting to annotate positive and hard negative sam- ples given sentence 24, illustrative example is shown in Figure 2. The structure of the prompts for generating positive and hard negative examples remains the same; the only difference lies in the prompts. our implementation with the ChatGPT model, we have designed a few-shot prompt in a multi-tury chat format.
number will carry your
What is your generated sentence?
One person from our group execute your instructions great attention to detail.
Figure 2: Few-shot examples of generating positive examples of the input sentence. We adopt 5-shot for generation.
Example and prompt pools: A significant chal- lenge in creating synthetic datasets lies in en- hancing the datasct's diversity. Ye et al. (2022b) suggested that merely increasing the size of the synthetic dataset might not lead to better perfor- mance, with one reason being the lack of diversity. Datasets labeled by groups of annotators rally help mitigate this problem due to the vari- ance in understanding and interpretation of prompts among different annotators. This variance results in diverse outputs, even for the same input. For example, Williams et al. (2018) utilized 387 anno- tators to create the MultiNLI dataset. Even with the same prompt, these annotators provided varied outputs due to their individual understanding of the prompt and their unique world knowledge, Icading a more diverse dataset. In attempt to mimic this variation among different annotators, em- ploy example pools and prompt pools. Specifically, we designed four types of positive/hard negative prompts (an example of hard negative prompts are
2.2 Data Synthesis from ChatGPT
We propose to prompt ChatGPT (OpenAI, 2022) to synthesize the required data in contrastive learning, inspired by recent successes of prompting LLMs fulfill multiple tasks (Chung al., 2022: Ope- nAI, 2023). Concretely, we introduce two variants SynCSE: (1) SynCSE-partial which synthesizes (x," ,,,, given v,, and (2) SynCSE-scratch which synthesizes (I;; x x. from scratch. SynCSE- scratch is practically useful since large-scale unla- beled data are not always available in the domain of interest due to copyright restrictions, data distri- bution issues, or messy formats. We describe these variants below. In general, using SynCSE- scratch as an example, the complete data genera- tion process includes two parts: (1) generating unla- beled sentences in the target domain; (2) generating
natu-

---

showed in Table 1) and 18 few-shot exemplars for each of the prompt (generated using GPT-4). Dur- ing each data generation process, we sample one prompt and five exemplars to construct a distinct input prompt. Details of these pools can be found in Appendix A.
by SimCSE as SimCSE_NE NLI. In the creation of the SynCSE-partial dataset, for fair comparison, we utilized the unlabeled sentences x from Sim- CSE_NLI, and generated positive/hard negative examples for them using the algorithm detailed in $2.3. For SynCSE-scratch, we generate the same number of examples as in the SynCSE-partin case. as detailed in $2.4. While our method can eas- ily scale up the dataset, for fair comparison, we ensure the data volume used for SynCSE-scratch and SynCSE-partial is equivalent that of Sim- CSE CSE_NLI NLL. For the combination of the SynCSE- scratch and SimCSE_NE NLI datasets, we simply merge these two datasets to evaluate whether our generated dataset can aid the manually annotated one.
2.4 SynCSE-scratch
Creating synthetic dataset from scratch, where the necessary unlabeled sentences for annotation are absent, presents a substantial challenge. We address this problem in two stages: initially, we generate unlabeled sentences, and subsequently, we apply the procedure discussed in $2.3 to anno- tate positive and hard negative samples of these sentences.
To ensure data diversity during the generation of unlabeled sentences, we employ a strategy that specifies the genres and topics when generation, combined with the utilization of example and prompt pools. This strategy is intended to mini- mize repetition and redundancy between the new data and the generated data so far. More specifi- cally, as illustrated in Figure 1, given a text genre, we randomly select six topics from pre-defined list to be included in the prompt (the list of genres and topics used in this paper can be found Ap- pendix B). The term "etc." in the prompt ensures that the generated sentences are not strictly limited to these six topics. We adopt one-shot prompt- ing generate several sentences at once. As long given different genres or topics when adding data compared the existing data, the added data will likely have low redundancy with the existing data, thereby enhancing the overall diversity of the dataset. The examples used for generating raw sentences were produced by GPT-4.
Given that SimCSE serves as general method in contrastive learning, we consistently use SimCSE as the backbone method for SynCSE. We note that SynCSE is general and could be combined with more advanced algorithms well, such with PromCSE (Jiang al., 2022b) and CARDS (Wang et al., 2022c). We emphasize that, after training the models on the NLI dataset, we freeze the mod- els and directly evaluate our embeddings on all the different tasks and setting below we do not specifically train sentence embeddings each set- ting separately. For the STS and transfer learning tasks, use the same hyperparameters Sim- CSE. Since SimCSE did conduct reranking experiments, we directly use the default parameters of MTEB (Muennighoff et al., 2023) to evaluate embeddings on the reranking tasks.
3.2 Evaluation Settings
Following the procedure outlined in SimCSE, evaluate our model, trained on the synthetic NLI dataset, across seven semantic textual similarity (STS) tasks: STS 2012-2016 (Agirre et al., 2012, 2013, 2014, 2015, 2016), the STS Benchmark (Cer et al., 2017), and SICK Relatedness (Marelli et al., 2014). is impor- tant to note that no data from these STS tasks were used during training. Our model was trained solely on our synthetic NLI dataset. The sentence em- beddings, which we evaluate on the STS tasks, are obtained from the [CLS) representation. During the training process. we average the development scores from the STS Benchmark and SICK Relat- edness to form the evaluation matrix. This matrix is used to select the best models. The other hyper-
Semantic Textual Similarity Tasks:
3 Experiment
3.1 Training
We evaluate three different settings in the experi- ments, including SynCSE-partial SynCSE-scratch, as well as a combination of SynCSE-scratee scrattch with existing annotated datasets in a supervised setting. While both SynCSE-pame partial and SynCSE-scratch represent unsupervised settings, in the combination setting augment previous annotated datasets with the synthesized data produced in SynCSE- scratch, to examine whether SynCSE-scratch could provide help for a supervised scenario as well. We refer to the NLI dataset (MNLI+SNLI) used

---

Model
Method
STSI2
STS13
STS14
STSIS
STSI6
STSb
SICK-R
Avg
Unsupervised methods
unsup-SinCCE!
70.16
81.77
73.24
81.36
80.65
80.22
68.56
76.57
RankCSF iatNee
72.88
84.50
76.46
84.67
83.00
83.24
71.67
79.49
RankCSE LALLE
72.74
84.24
75.99
84,68
82.88
83.16
71.77
79.35
.2P-CSR**
74.97
83.63
78.28
84.86
82.03
82.77
71.26
79.69
RoBFRRa--bar
PromptRoBERTal
73.94
84.74
77.28
84.99
81,74
81.88
69.50
79.15
PCL
71.54
82.70
75.38
83.31
81,64
81.61
69.19
77.91
CARDS
72.49
84.09
76.19
82.98
82.11
82.25
70.65
78.68
ConPVP"
73.20
83,22
76.24
81.49
82.18
74.59
79.18
SynCSF- Part-aaraalll (SimCSF. based)
76.11
84.49
79.61
85.26
82,60
83.94
81.57
81.94
SynCSE-scratch (SimCSE based)
74.61
83.76
77.89
85.09
82.28
82.71
78.88
80.75
unsup-SimCSEt
72.86
83,99
75.62
84.77
81.80
81.98
71.26
78.90
RankCSL BaNN
73.23
85.08
77.50
85.67
82.99
84.20
72.98
80.24
RankCSF MankCSPieaAALE LLL
73,40
85.34
77.25
85.45
82.64
84.14
72.92
80.16
1.2P-CSR
73.65
84.08
78.29
85.36
82.15
83.70
73.47
80.10
RoBeKTa-large
PCL
73.76
84.59
76.81
85.37
81.66
82,89
80.33
79.34
CARDS
74.63
86.27
79.25
85.93
83.17
83,86
72.77
80.84
ConPVP"
74.75
84.09
77.88
83.13
83.44
83.64
74.31
80.18
SynCSF--prtiia (SimCSF. hased)
76.03
84.27
80.03
85.37
83.62
84,26
81.14
82.10
SynCSE-scratch (SimCSD based)
75.45
85.01
80.28
86,55
83.95
84.49
80.61
82.33
Supervised methods
sup-SimCSEE sup-SimCSE PromptRoBERTal
76.53
85.21
80.95
86.03
82.57
85.83
80.50
82.52
75.61
85.19
79.58
85.85
82.39
85.30
80.39
82.04
RoBFRTa-base
76.75
85.93
82.28
86.69
82.80
86, 14
80.04
82.95
PrompCSF: + EH
75.96
84,99
80.44
86.83
81,30
84,40
80.96
82.
SynCSE-scratch + SimCSE
76.79
84.93
80.14
86.28
83,38
85.75
81.02
82.61
sup-SimCSR sup-SimCSE PromCSE + E
77.46
87.27
82.36
86.66 66.66
83.93
86.70
81 81.95
83.76
76.62
86.90
82.05
86.10
83,97
86.10
82.04
83,40
RoBERTa-large
79.56
88.97
83.81
88.08
84.96
87.87
82,43
85.10
SynCSE-seratch SimCSE_NLI
77.13
87.61
82.82
87.67
85.66
87.22
82.45
84.37
Table 2: Results the STS benchmark. Spearman's correlation is reported. The "unsup-" and "sup-" correspond
unsupervised and supervised settings, respectively. "f": results from (Gao et al., 10; "3": results from (Liu et al.); results from (Zhou et al., 2023); "It": results from (Jiang et al., 2022a); "tt": results from (Wu et al., 2022a); "e": results from (Wang al., 2022e); "*": results from (Zeng et al., 2022); "Q"": results from (Jiang et al., 2022b). The term "SynCSF-scratch + SimCSE_ _I_'" represents our synthetic data combined with the NIJI dataset used in SimCSE. The SynCSE PartSS--paall//srracch experiments were implemented on the basis of SimCSE. Some baselines did not conduct some experimental setups. We report the results that exist in their papers.
parameters are kept consistent with those used in SimCSE.
Baselines: We compare our approach with state- of-the-art sentence embedding learning methods: RankCSE (Liu et al.), L.2P-SSR (Zhou et al., 2023), PCL (Wu et al., 2022a), CARDS (Wang et al., 2022c), ConPVP (Zeng et al., 2022), and PromptRoBERTa (Jiang et al., 2022a). While we base approach on SimCSE, emphasize that our approach is orthogonal the baseline algo- rithms and our synthesized datasets may be com- bined with them further boost the performance. We directly report the results from their respective papers.
Reranking tasks: We further evaluate the syn- thetic dataset on four reranking tasks: AskUbun- tuDupQuestions (Lei et al., 2016), MindSmallR- eranking (Wu et al., 2020), SciDocsRR (Cohan et al., 2020), and weweOOveerfooDuuuuueseeee et al., 2018). We directly evaluate the model, which is frozen after training on the NLI dataset, on reranking tasks, without using the training sets of reranking tasks. The resulting ranking is scored for each query and averaged across all queries. In line with the methodology of MTEB (Muennighoff et al., 2023), we utilize Mean Average Precision (MAP) as the primary metric.
(Liu
3.3 Semantic Texual Similarity
Main results: As shown Table 2, Both SynCSE-parias and SynCSE-scratch significantly

---

Model
Method
AskU.
MindSmall
SciDocsRR
StackO.
Avg
Unsupervised methods
unsup-Sim SimCSE
52.78
29.91
65.96
39.25
46.95
CARDS
52.94
27.92
64.62
41.51
46.75
RoBERTa-base
PCL.
51.85
27.92
64.70
41.18
46.41
SynCSE-partic (SimCSE based)
53.95
29.97
65.21
37.84
46.74
SynCSE-scratch (SimCSE based)
53.27
30.29
67.55
39.39
47.63
unsup-Simna SimCSE
55.10
29.23
68.54
42.56
48,86
CARDS
53.83
29.07
68.26
43.24
48.60
RoBERTa-lane
PCL
53.43
28.56
66.06
41.54
47.40
TUnCSE-partiag (SimCSE based)
54.78
30.23
68.90
38.28
48.05
SynCSE-scratch (SimCSE based)
55.48
30.27
70.85
40.00
49.15
Supervised methods
sup-SimCSE
52.55
29.87
68.43
37.52
47.09
RoBERTa-base
SynCSE-scratch SimCSE NLI
52.74
30.40
67.65
38.17
47.24
sup-SimCSE
54.72
30.89
71.69 71.53
38.24
48.89
RoBeRTa-large
SynCSE-seratch SimCSE_N _LI
55.26
30.40
39.84
49.26
Table 3: Results the reranking benchmark. Mean Average Precision (MAP) is reported.
Dataset
STS12
STS13
STS14
STS15
STS16
STSb
SICK-R
Avg
GenSE
72.09
85.24
79.84
83.25
82.88
83.24
75.33
80.27
DINO
70.27
81.26
71.25
80.49
77.18
77.82
68.09
75.20
SynCSE-partial (SimCSE based)
76.11
84,49
79.61
85.26
82.60
83.94
81.57
81.94
SynCSE-scratch (SimCSE based)
74.61
83.76
77.89
85.09
82.28
82.71
78.88
80.75
Table 4: Performance comparison of RoBERTa-base trained various datasets, using the STS benchmark for evaluation. The reported metric is Spearman's 's correlation. The "f" symbol is used to indicate results reported in DINO. For SimCSE, we adopted MNLI+SNLI dataset used (Gao al., 2021). "t": GenSE released an NI.I synthetic dataset comprising over 60 million samples, For fair comparison, randomly sampled from it the same number of samples used in the SimCSE dataset.
outperformed all the unsupervised baselines by more than 2 absolute points. Even when compared with supervised settings, our approach achieved performance near that manual annotation on RoBERTa-base, falling behind by only about 1 point on RoBERTa-large. It's worth noting that while the supervised SimCSE training dataset (SNLI) and STS test data share significant over- lap in domains (for instance, both STSb and SNLI extensively used Flicker30k data (Plum- mer al., 2015)), the domains were explic- itly known while generating the SynCth-scrateh dataset. Interestingly, SynCSE-partial does not always beat SynCSE-scratel as demonstrated in the RoBERTa-large case, which implies the poten- tial of SynCSh-scratch as promising approach to learn sentence embeddings without using any real data samples. By augmenting annotated NLI
data with the SynCSE-scratch synthetic dataset, approach outperformed sup-SimCSE signif- icantly, reaching performance of 84.37% with RoBERta-large, suggesting that our synthetic data complementary to Luman-labbeled NLI datasets. "PromptCSE+EEH"" (Jiang et al., 2022b) achieves competitive performance in the supervised setups. As an orthogonal contribution, however, SynCSE may be combined with the loss function they pro- posed to further advance the results.
3.4 Reranking
Table 3 shows the results of the reranking tasks. Compared the STS task, the domain of the reranking task data is more divergent from that of the NLI data used for training, as a result, SynCSE- scratch actually outperforms SynCSE-partin sig- nificantly, which implies the advantage of SynCSE-

---

Model
Method
STS12
STS13
STS14
STS15
STS16
STSb
SICK-R
Avg
ZeroGen
51.68
71.45
58.80
67.04
70.04
65.00
66.88
64.41
RoBERTa-base
SynCSE-scran (SimCSE based)
71.81
83.43
76,90
83.39
82.33
82.89
77.39
79.73
ZeroGen
50.97
70.90
59.97
69.59
68.79
65.43
65.72
64.48
RoBERTu-large
SynCSE-scratch (SimCSE bused)
74.61
83.76
77.89
85.09
82.28
82.71
78.88
80.75
Table Performance comparison of SynCSB-scratch and ZeroGen, using the STS benchmark for evaluation. The
Spearman's correlation is reported.
BIOSSES
StacKOverfoowDugguestio (Mean Average Precision)
Model
Method
(Spearman's 's correlation)
unsup-SimCSE, (Wikipedia domain)
68.86
39.25
RoBERTa-basc
SynCSF-scratch (SmCCSF hased)
80.12
43.22
unsup-SimCSS (Wikipedia domain)
71.96
42.56
RoBERTa-large
ScnCah-seratch (SimCSE based)
77.73
45.67
Table 6: Performance comparison of the RoBERTa trained on the Wikipedia domain (using the publicly available unsup-SimCSE checkpoint) and specialized domains data generated by SynCSI--craath..
scratch when in-domain unlabeled sentences are unavailable. SynCSE-scratch also surpasses other unsupervised baselines while SynCSE-partis partial un- derperforms them. Moreover, the combination of SynCSE-scratch with manually annotated datasets still facilitates further performance enhancement, substantiating that our method can aid augment- ing existing datasets.
erated by SynCSE-scratch. We compare the gener- ated sentences of our methods with them in Table 11. From the table, sentence can generate more diverse annotations. depicted Table 4, both SynCSE-scratch and SynCSE-partial have achieved performance on the STS task that surpasses that of DINO, GenSE. a practical setting when generating a dataset from scratch (SynCSE-scratch), we compare our method with ZeroGen (Table 5), and the results show our method significantly outperforms the baseline.
find that our generated
3.5 Comparison with Other Synthetic Datasets
In addition to comparing with the MNLI+SLII datasets used in SimCSE, we also compare our method with three other baselines that leverage synthetic NLI data: (1) GENSE (Chen et al., 2022) aims to automatically annotate the positive and hard negative examples with a LLM trained on ex- isting NLI labeled dataset. We sample the same number of examples from their published dataset those used in SynCSE; (2) The objective of DINO (Schick and Schütze, 2021) is generate synthetic data for sentence embeddings as well. In DINO's most effective configuration, they generate the positive or hard negative samples and assign a similarity score to them based on the prompts used. As they have not made an NLI-style dataset avail- able, we directly report results from their paper, and (3) ZeroGen (Ye et al., 2022a) proposes an effi- cienty unsupervised dataset generation method. We selected those examples that have been provided in both "entailment" and "oot_ennailmet" sentences to construct sentence pairs, totaling 46,31 pairs, as training data. To ensure a fair comparison, we ran- domly sampled an equal number of examples gen-
3.6 Applying to Specialized Domains
SynCSE is advantageous when dealing with spe- cialized domains where unlabeled data unavail- able. such cases, traditional methods are not directly applicable. To evaluate SynCSE this scenario, we conduct experiments on two another datasets focused on specialized domains the BIOSSES (Sogancioglu et al., 2017) dataset of a semantic textual similarity task for the biomedical domain, and the StocOOverloow.Duuuuussioo (Liu et al., 2018) dataset of reranking task for the programming questions domain. Specifically, our experimental design based on the assumption that we only have access to the names of the tar- get domains (i.e., "biomedicine" and "Stack Over- flow website") without any data available. We run SynCSE-scratch scrattc in these settings. Concretely, we first generate 37k unlabeled sentences in the spective domain following the procedure described in Section $2.4, then generate positive and hard negatives for these sentences, and train the mod- els. We use the publicly available unsupervised

---

Model
Method
MR
CR
SUBJ
MRQA
SST
TREC
MRPC
Avg
Unsupervised methods
unsup-SimCSE!
83.37
87.76
95.05
87.16
89.02
90.80
75.13
86.90
L2P-CSR*
79.67
88.30
94.27
87.70
87.50
81.14
76.47
85.01
PCL.1
81.83
87.55
92.92
87.21
87.26
85.20
76.46
85.49
RoBeRTa-hase
PrompRoBERTa
83.82
88.72
93.19
90.36
88.08
90.60
76.75
87.36
ConPVP"
82.44
88.30
93.20
88.74
87.70
87.33
76,15
86.27
SynCSE-partial (SimCSE based)
85.41
91.44
93.39
89.91
91.21
84.40
76.87
87.52
SynCSE-scratch (SimCSE based)
85.47
91.44
92.53
89.67
90.94
81.60
76.06
86.82
CSS
84.66
88.56
95.43
87.50
89.46
95.00
72.41
87.57
1.2P-CSR
80.12
88.53
94.07
88.92
87.04
83.05
76.84
85.51
RoBERTa-large
PCL
84.47
89.06
94.60
89.26
89.02
94.20
74.96
87.94
- (SiMCSH based)
87.18
92.02
94.16
90.76
91.65
86.80
76.87
88.49
SynCSE-scratch (SimCSE based)
87.24
92.16
93.75
90.81
91.87
84.00
76.29
88.02
Supervised methods
sup-SimCSEf
85.08 35.08
91.76
94.02
89.72
92.31
91.20
76.52
88
Sup-SimCSE
85.05
90.97
94.20
89.37
91.49
88.60
76.87
88.08
RoBERTa-base
PrompRoBERTa"
85.74
91.47
94.81
90.93
92.53
90.40
77.10
89.00
SynCSE-scratch SimCSE_NLI
85.51
91.52
93.33
89.87
92.48
83,40
76.06
87.40
sup-SimCSE!
88.12
92.37
95.11
90.49
92.75
91.80
76.64
89.61
RoBERTa-large
sup-SimCSB
87.89
92.61
95.20
90.77
92.86
90.80
77.22
89.62
SynCSE-scratch SimCSE NLI
88.22
92.56
94.76
90.98
93.08
88.00
76.81
89.20
Table 7: Transfer task results of different sentence embedding models (measured as accuracy). The labels "unsup-" and "sup-" correspond unsupervised and supervised settings, respectively. "t": results from (Gao et al., 2021);
results from (Zhou et al., 2023); "If": results from (Wu et al., 2022a); "ft": results from (Jiang et al., 2022a);
results from (Zeng et al., 2022). The term "SynCSE-scratte"" + SimCSE NLI [ represents our synthetic data combined human labeled NLI dataset used in SimCSE.
Method
STS12
STS13
STS14
STS15
STS16
STSb
SICK-R
Avg
Naive Generation
64.65
75.86
62.94
72.79
71.61
72.76
71.57
70.31
SynCSE-scratch
70.89
83.79
76,48
83.28
81.97
82.36
76.14
79.27
Table 8: Performance comparison of our synthetic dataset generation and the "Naive Generation" method.
SimCSE model checkpoint that was trained on the Wikipedia domain for comparison. This is because we assumed no access to unlabeled sentences in these domains, which is a practical setting. Our observations (Table 6) show that SynCSE-scratch outperforms the unsupervised SimCSE baseline sig- nificantly in both domains. This experiment further demonstrates the superiority of our method on new domains where no data is available traditional unsupervised approaches like SimCSE tend ex- perience a domain transfer drop in performance in such scenarios.
Transfer tasks: Following SimCSE, we execute seven transfer learning tasks: MR (Pang and Lee, 2005), CR (Hu and Liu, 2004), SUBJ (Pang and Lee, 2004), MPQA (Wiebe et al., 2005), SST- 2 (Socher et al., 2013), TREC (Voorhees and Tice, 2000), and MRPC (Voorhees and Tice, 2000). These experiments are carried out with the same settings as used in SimCSE. As shown in Table 7, JynCSE-partinn outperforms all unsupervised base- lines.
Comparion with the naive generation process: To validate the effectiveness of our data synthesis process, conduct ablation experiment, where (1) we do not specify topics genres when gener- ating unlabeled sentences, and (2) we do not vary the prompt and exemplars but fix them the same (that are randomly selected from the pools) when generating the positive and hard negative labels.
3.7 Analysis
In this subsection, we provide an in-depth analysis SynCSE. All results presented here are based on
the RoBERTa-B model.

---

Counsclor
H2
H3
H4
H5
Avg
uses prompts instead of the [CLS] token to extract
Fraction of ethically unsafe data
embeddings.
1%
0%
0%
1%
0%
0.4%
However, these unsupervised methods signif- icantly lag behind their supervised counterparts. Supervised approaches usually derive positive and hard negative samples from labeled NLI datasets (Wang and Lu, 2022; Gao al., 2021; Jiang et al., 2022a), but these datasets are limited in quantity and domain. Additionally, annotating a new NLI dataset is costly, especially in fields that require trained annotators. Chen et al. (2022) trained a T5 (Chung et al., 2022) model capable of producing positive and hard negative samples, while Ye et al. (2022b) implemented continuously updated model to modify prompts for generation. However, the performance of these algorithms is still constrained by the performance of generators, which need labeled NLI data for training. Differing from these methods, which necessitate training an additional model, Wang et al. (2022b) proposed rule-based algorithm capable of generating hard negative annotations. However, its diversity is lim- ited to the prescribed rules. Gilardi al. (2023) used ChatGPT for dataset annotation. However, their exploration was limited to tasks with explicit answer labels such as "RELEVANT" or "IRRELE- VANT". They did not attempt to annotate datasets that required diverse responses. Schick and Schützee (2021) also propose to generate both annotations and unlabeled sentences, while they do not focus on the contrastive learning framework.
Table 9: The result of the fraction of ethically unsafe data annotated by one psychological counselor and four postgraduate students. H* means the index of postgrad- uate annotators.
Other settings are kept the same as in SynCSE- scratch. We perform the ablation experiment on 22k examples. We denote the baseline without di- versity control "Naive Generation" and show them in the Table 8, our method outperforms the Naive Generation baseline by an average 8.96%. demonstrating the critical role diversity control in our data synthesis process.
Ethical considerations of the synthetic dataset:
To evaluate the safety of our synthetic dataset, we ask five annotators (one of which is psychologi- cal counselor and the other four are postgraduate students) to annotate whether the generated sen- tences have ethical problems. Specifically, we ran- domly select 100 sentences from those generated by SynCtE-scratch, and each sentence indepen- dently evaluated by the five people for potential ethical problems. As the Table 9 suggests, only a minor portion of the data is classified as ethi- cally unsafe, indicating that our synthetic dataset upholds certain level of safety concerning ethi- cal issues. This is not surprising since ChatGPT, the backend in our experiments, is already heav- ily aligned avoid producing text with ethical or safety issues.
Discussion
In this work, we propose SynCSE, a novel con- trastive learning framework for learning sentence embeddings with synthetic data. We prompt LLMs to synthesize unlabeled sentences and their positive and hard negative examples. Furthermore, by uti- lizing example and prompt pools, we can specify the genre and topic of generated sentences, thereby enhancing the quality of the synthetic dataset. Ex- periments on both sentence similarity and rerank- ing tasks demonstrate the effectiveness of SynCSE. The performance of SynCSE in this study strongly suggests the potential of synthetic datasets gen- crated by the increasingly advanced LLMs of to- day. We envision that, through the effective use of prompting strategies with LLMs, synthetic datasets produced by these models could potentially serve as promising alternatives to real-world data across a wide range of tasks.
4 Related Work
Prior approaches for sentence embedding fall into two main categories: (1) supervised learning with labeled sentences, and (2) unsupervised sentence embedding with unlabeled sentences. Among these, works based on contrastive learning have proven to be the most effective. For unsupervised methods, SimCSE uses dropout masks to construct positive pairs for learning, while negative exam- ples use in-batch negative examples. Some works employ data augmentation techniques input sen- tences, such word repetition (Wu al., 2022b), case flipping (Wang al., 2022c), combina- tion of multiple data augmentation strategies to offset the bias caused by mono-augmentattin (Wu et al., 2022a). PromptBERT (Jiang et al., 2022a)

---

References
Daniel Cer. Yinfei Yang. Sheng-yi Kong, Nan Hua, Nicole Limtiaco, Rhomni St John, Noah Constant, Mario Cuajardd-Cespeddss Steve Yuan, Chris Tar, et al. 2018. Universal sentence encoder. arXiv preprint arXiv:1803.11175
Eneko Agirre, Carmen Banea, Claire Cardie, Daniel Cer, Mona Diab. Aitor Gonzalez-Agire, Weiwei Guo, lñigo Lopez-Gazpio, Montse Maritxalar, Rada Mihalcea, German Rigau, Larraitz Uria, and Janyce Wiebe. 2015. SemEval-2015 task 2: Semantic tex- tual similarity, English, Spanish and pilot on inter- pretability. In Proceedings of the 9th International Workshop on Semantic Evaluation (SemEval 2015). pages 252-263, Denver, Colorado. Association for Computational Linguistics.
Yiming Chen, Yan Zhang. Bin Wang. Zuozhu Liu. and Haizhou Li. 2022. Generate, discriminate and contrast: A semi- sami--upervised sentence representation learning framework. In Proceedings of the 2022 Con- ference on Empirical Methods in Natural Language Processing, pages 8150-8161, Abu Dhabi, United Arab Emirates, Association for Computational Lin- guistics.
Eneko Agirre, Carmen Banea, Claire Cardie, Daniel Cer, Mona Diab, Aitor conzalez-Agirre, Weiwei Guo, Rada Mihalcea, German Rigau, and Janyce Wiebe. 2014. SemEval-2014 task 10: Multilingual semantic textual similarity. In Proceedings of the Sth Interna- tional Workshop on Semantic Evaluation (SemEval 2014), pages -1,91, Dublin, Ireland. Association for Computational Linguistics.
Hyung Won Chung, Le Hou, Shayne Longpre, Bar- ret Zoph, Yi Tay, William Fedus, Fric Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, al. 2022. Scaling instruction-finene arXiv preprint 22X1::2100.1416.
language models.
Arman Cohan, Sergey Feldman, Iz Beltagy. Doug Downey, and Daniel Weld. 2020. SPECTER: Document-level representation learning using citation-informed transformers. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 2270-2282, Online. Association for Computational Linguistics.
Eneko Agirre, Carmen Banea, Daniel Cer, Mona Diab, Aitor Konzalez-Agirre. Rada Mihalcea, German Rigau, and Janyce Wiebe. 2016. SemEval-2016 task 1: Semantic textual similarity, monolingual and cross-lingual evaluation. In Proceedings of the 10th International Workshop Semantic Evaluation (6emEval-20116, pages 497-511, San Diego, Califor- nia. Association for Computational Linguistics.
Tianyu Gao, Xingeheng Yao, and Danqi Chen. 2021. SimCSE: Simple contrastive learning of sentence em- beddings. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Process- ing. pages 6894-6910, Online and Punta Cana, Do- minican Republic. Association for Computational Linguistics.
Eneko Agirre, Daniel Cer, Mona Diab, and Aitor Gonzalez- Agirre. 2012. SemEval-2012 task 6: A pilot on semantic textual similarity. In *SEM 2072: The First Joint Conference on Lexical and Compu- lational Semantics- - Volume 1: Proceedings of the main conference und the shared task, and Volume 2: Proceedings of the Sixth International Workshop on Semantic Evaluation (SemEval 2012), pages 385- 393, Montréal, Canada. Association for Computa- tional Linguisties.
Fabrizio Gilardi, Meysam Alizadeh, and Maël Kubli. 2023. Chatgpt outperforms crowd-workers for text- annotation tasks. arXiv preprint ar35
Eneko Agirre, Daniel Cer, Mona Diab, Aitor Gonzalez- Agirre, and Weiwei Guo. 2013. *SEM 2013 shared task: Semantic textual similarity. In Second Joint Conference on Lexical and Computational Semantics (*SEM), Volume 1: Proceedings of the Main Confer- ence and the Shared Task: Semantic Textual Similar- ity, pages 32-43, Atlanta, Georgia, USA. Association for Computational Linguistics.
Hongliang He, Junlei Zhang, Zhenzhong Lan, and Yue Zhang. 2023. Instance smoothed contrastive learning for unsupervised sentence embedding. In Proceed- ings of the AAAI Conference on Artificial Intelligence. volume 37, pages 12863-12871.
Minqing Hu and Bing Liu. 2004. Mining and summa- rizing customer reviews. In Proceedings of the tenth ACM SIGKDD international conference on Knowl- edge discovery and data mining, pages 168-177.
Ken Barker, Parul Awasthy, Jian Ni, and Radu Florian. 2021. IBM MNLP IE CASE 2021 task 2: NLI reranking for zero-shot text classification. In Pro- ceedings of the 4th Workshop on Challenges and Ap- plications of Automated Extraction of pocii-political Events from Text (CASE 2027), ), pages 193-202, On- line. Association for Computational Linguistics.
Ting Jiang, Jian Jiao, Shaohan Huang, Zihan Zhang, Deqing Wang, Fuzhen Zhuang, Furu Wei, Haizhen Huang, Denvy Deng, and Qi Zhang. 2022a. Prompt- BERT: Improving BERT sentence embeddings with prompts. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 8826-88311 Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
Daniel Cer, Mona Diab, Eneko Agirre, Iñigo Lopez- Gazpio, and Lucia Specia. 2017. SemEval-2011 task I: Semantic textual similarity multilingual and crosslingual focused evaluation. In Proceedings of the 1 IIth International Workshop on Semantic Evaluation -SemEval-2077 pages 1-14, Vancouver, Canada, Association for Computational Linguistics.
Yuxin Jiang, Linhan Zhang, and Wei Wang. 2022b. Im- proved universal sentence embeddings with prompt- based contrastive learning and energy- learning. In Findings of the Association for Computational

---

Linguistics: EMNLP 2022, pages 3021-3035, Abu Dhabi, United Arab Emirates. Association for Com- putational Linguisties.
Graham Neubig and Zhiwei He. 2023. Zeno GPT Ma- chine Translation Report.
OpenAL. 2022. Chatgpt: Optimizing language models for dialogue. OpenAl Blog.
Ann Lec, Michael Auli, and Marc' Aurelio Ranzato. 2021. Discriminative reranking for neural machine translation. In Proceedings of the 59th Anmual Meet- ing of the Association for Computational Linguistics and the 11th International Joint Conference on Natu- ral Language Processing (Volume I: Long Papers), pages 7250-7264, Online. Association for Computa- tional Linguisties.
OpenAL 2023. Gpt-4 technical report. arXiv.
Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. 2022. Training language models to follow instruc- tions with human feedback. Advances in Neural Information Processing Systems, 35:27730-27744.
Tao Lei, Hrishikesh Joshi, Regina Barzilay, Tommi Jaakkola, Kateryna Tymoshenko, Alessandro Mos- chitti, and Lluís Märquez. 2016. Semi-supervised question retrieval with gated convolutions. In Pro- ceedings of the 2016 Conference of the North Amer- ican Chapter of the Association for Computational Linguistics: Human Language Technologies. pages 1279-1289, San Diego, California. Association for Computational Linguistics.
Bo Pang and Lillian Lec. 2004. A sentimental educa- tion: Sentiment analysis using subjectivity summa- rization based on minimum cuts. In Proceedings of the 42nd Annual Meeting of the Association for Computational Linguistics (ACL-04), pages 271-278, Barcelona, Spain.
Bo Pang and Lillian Lee. 2005. Seeing stars: Exploit- ing class relationships for sentiment categorization with respect to rating scales. In Proceedings of the 43rd Annual Meeting of the Association for Compu- tational Linguistics (ACI. '05), pages 115-124, Ann Arbor, Michigan. Association for Computational Lin- guistics.
Peerat Limkonchotiwat, Wuttikom Ponwitayarat, Lalita Lowphansirikul, Can Udomcharoenchaikit, Ekapol Chuangsuwanich, and Sarana Nutanong. 2022. Con- Gen: Unsupervised control and generalization distil- lation for sentence representation. In Findings of the Association for Computational Linguistics: EMNLP 2022, pages 6467-6480, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
Bryan A Plummer, Liwei Wang, Chris M Cervantes, Juan C Caicedo, Julia Hockenmaier, and Svetlana Lazebnik. 2015. Flickr30k entities: Collecting region-to-phrase correspondences for richer image- to-sentence models. In Proceedings of the IEEE international conference on computer vision, pages 2641-2649
Jiduan Liu, Jiahao Liu, Qifan Wang, Jingang Wang. Wei Wu, Dongyan Zhao, and Rui Yan. Rankese: Unsupervised sentence representations learning via learning to rank.
Xueqing Liu, Chi Wang, Yue Leng, and ChengXiang Zhai. 2018. Linkso: a dataset for learning to retrieve similar question answer pairs on software develop- ment forums. In Proceedings of the 4th ACM SIG- SOFT International Workshop on NLP for Software Engineering, pages 2-5.
Timo Schick and Hinrich Schütze. 2021. Generating datasets with pretrained language models. In Pro- ceedings of the 2021 Conference Empirical Meth- ods in Natural Language Processing, pages 6943- 6951, Online and Punta Cana, Dominican Republic. Association for Computational Linguisties.
Marco Marelli, Stefano Mcnini, Marco Baroni, Luisa Bentivogli, Raffaella Bernardi, and Roberto Zam- parelli. 2014. SICK cure for the evaluation of compositional distributional semantic models. In Proceedings of the Ninth International Conference on Language Resources and Evaluation (LREC' 14), pages 216-223, Reykjavik, Iceland. European Lan- guage Resources Association (ELRA).
Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D. Manning, Andrew Ng, and Christopher Potts. 2013, Recursive deep models for semantic compositionality over sentiment treebank. In Proceedings of the 2013 Conference on Empiri- cal Methods in Natural Language Processing, pages 1631-1642, Seattle, Washington, USA. Association for Computational Linguistics.
Amita Misra, Brian Ecker, and Marilyn Walker. 2016. Measuring the similarity of sentential arguments in dialogue. In Proceedings of the 17th Annual Meeting of the Special Interest Croup on Discourse and Dia- logue, pages 276-287, Los Angeles. Association for Computational Linguistics.
Gizem Sogancroğlu, Hakime Özüürk, and Arzucan Özgür. 2017. Biosses: semantic sentence simi- larity estimation system for the biomedical domain. Bioinformatics, 33(14):49-is) -18.
Niklas Muennighoff. Nouamane Tazi, Loic Magne, and Nils Reimers. 2023. MTEB: Massive text embedding henchmark. In Proceedings of the 17th Conference of the European Chapter of the Association for Com- patational Linguistics, pages 2034-2037, Dubrovnik, Croatia. Association for Computational Linguistics.
Nandan Thakur, Nils Reimers, Andreas Rücklé, Ab- hishek Srivastava, and Iryna Gurevych. 2021. Beir: A heterogeneous benchmark for zero-shot evaluation of information retrieval models. In Thirty-fifth Con- ference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 2).

---

Ellen M Voorhees and Dawn M Tice. 2000. Building a question answering test collection. In Proceedings of the 23rd annual international ACM SIGIR confer- ence on Research and development in information retrieval, pages 200-207.
Xing Wu, Chaochen Gao, Liangjun Zang, Jizhong Han, Zhongyuan Wang, and Songlin Hu. 2022b. ESim- CSE: Enhanced sample building method for con- trastive learning of unsupervised sentence embed- ding. In Proceedings of the 29th International Con- ference on Computational Linguistics, pages 3898- 3907, Gyeongju, Republic of Korea. International Committee on Computational Linguistics.
Bin Wang, C.-C. Jay Kuo, and Haizhou Li. 2022a. Just rank: Rethinking evaluation with word and sentence similarities. In Proceedings of the 60th Annual Meet- ing of the Association for Computational Linguistics (Volume I: Long Pupers), pages 6060-6077, Dublin, Ireland. Association for Computational Linguistics.
Jiacheng Ye, Jiahui Gao, Qintong Li. Hang Xu, Jiang- tao Feng, Zhiyong Wu, Tao Yu, and Lingpeng Kong, 2022a. ZeroGen: Efficient zero-shot learning via dataset generation. In Proceedings of the 2022 Con- ference on Empirical Methods in Natural Language Processing, pages 11653-1 1669, Abu Dhabi, United Arab Emirates. Association for Computational Lin- guistics.
Hao Wang, Yangguang Li. Zhen Huang, Yong Dou, Lingpeng Kong, and Jing Shao. 2022b. Snese: Con- trastive learning for unsupervised sentence embed- ding with soft negative samples. arXiv preprint arXiv:2201.05979
Jiacheng Ye, Jiahui Gao, Zhiyong Wu, Jiangtao Feng, Tao Yu, and Lingpeng Kong. 2022b. ProGen: Pro- gressive zero-shot dataset generation via in-context feedback. In Findings of the Association for Com- putational Linguistics: EMNLP 2022, pages 3671- 1- 3683, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
Tianduo Wang and Wei Lu. 2022. Differentiable data augmentation for contrastive sentence representation learning. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 7640-7653, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
Wei Wang, Liangzhu Ge, Jingqiao Zhang, and Cheng Yang. 2022c. Improving contrastive learning of sen- tence embeddings with augmented positives and retrieved negatives. In Proceedings of the 45th Inter- national ACM SIGIR Conference on Research and Development in Information Retrieval, pages 2159 2165.
Jiali Zeng, Yongjing Yin, Yufan Jiang, Shuangzhi Wu, and Yunbo Cao. 2022. Contrastive learning with prompt-derived virtual semannic prototypes for un- supervised sentence embedding, In Findings of the Association for Computational Linguistics: EMNLP 2022, pages 7042-7053, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
Janyce Wiebe, Theresa Wilson, and Claire Cardie. 2005. Annotating expressions of opinions and emotions in language. Language resources and evaluation, 39:165-2 210.
Kun Zhou, Beichen Zhang, Xin Zhao, and Ji- Ji-Rong Wen. 2022. Debiased contrastive learning of unsupervised sentence representations. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 6120- 61 6130, Dublin, Ireland. Association for Computational Linguistics.
Adina Williams, Nikita Nangia, and Samuel Bowman. 2018. A broad-coverage challenge corpus for sen- tence understanding through inference. In Proceed- ings of the 2018 Conference of the North American Chapter of the Association for Computational Lin- guistics: Human Language Technologies, Volume I (Long Papers), pages 1112-1122, New Orleans, Louisiana. Association for Computational Linguis- tics.
Kun Zhou, Yuanhang Zhou, Wayne Xin Zhao, and Ji- Rong Wen. 2023. Learning to perturb for contrastive learning of unsupervised sentence representations. IEEE/ACM Transactions on Audio, Speech, and Lan- guage Processing.
Fangzhao Wo, Ying Qiao, Jiun-Hung Chen, Chuhan Wu, Tao Qi, Jianxun Lian, Danyang Liu, Xing Xie, Jianfeng Gao, Winnie Wu, and Ming Zhou. 2020. MIND: A large-scald dataset for news recommenda- tion. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 3597-3606, Online. Association for Computational Linguistics.
Qiyu Wu, Chongyang Tao, Tao Shen, Can Xu, Xi- ubo Geng, and Daxin Jiang. 2022a. PCL: Peer- contrastive learning with diverse augmentations for unsupervised sentence embeddings. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 12052-12066. Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.

---

A Prompt pools
scientific theories, life stories of notable individu- als, COVID-19, immigration policies, and mental health.
In order to increase the diversity of imput prompts, we designed variety of prompts for generating positive samples, hard negative samples, and un- labeled data, which are adopted during generation based on certain probabilities. The specific prompts are displayed in Tables 12, 1, and 15. Given that generating image captions differs somewhat from generating other types of text, we have designed unique prompts for image captions to further en- hance diversity, as illustrated in Table 16.
Genre descriptions
in-person conversations
letters
reports, speeches, letters, and press releases from public domain government websites fiction
5
image descriptions
6 video descriptions
B Genres and Topics
7
news from websites or newspapers
8 reviews and critiques for shopping
When generating unlabeled sentences, to make the newly generated sentences as differ- ent possible from existing data, we specify the genre and topic of the new sentences. As long as the genre and topic of the new sentences are dif- ferent from existing ones, the probability of these
Genres:
headlines of news
10 dialogue of technical or Instructional tutorial
informative and expository texts which pro- vide guidance or explanations related to wide range of topics
12
STEM examination questions
new sentences providing more new information
13
travelogue
to the dataset becomes higher. In this paper, we use 20 different genres (Table 10) and 31 differ- ent topics (Table 2). Before generating sentences, we use GPT-4 to generate 30 examples for each genre one-shot example sentences. When using them, we first specify genre and fill it into the "[the description of the genre]" in the prompt of Table 15, then randomly choose from the topic list to fill into "[topic,]". These descriptions are adapted from the genre specifications provided by GPT-4, thus, creating new descriptions does not require a significant effort.
14
historical description
15
plots involving political intrigue and maneu- vering
16
formal writings that present research findings or scholarly discussion
17
speeches given by politicians, often with the intent of persuading or informing an audience about political topics
18
written works such poetry, drama, and nov- els
19
pieces of content shared on social media plat- forms
20
short image captions
Topics: We leveraged GPT-4 to generate an array of diverse topics, and 37 of these were randomly se- lected as the thematic grounding for our generation of unlabeled sentences. Concretely, these themes are: nature, technology, food, sports, culture, his- tory, animals, environment, politics, finance, edu- cation, social issues, global issues, entertainment, healthcare, war, mathematical and electrical engi- neering, crime, relationships and emotional bonds, magic and mythical creatures, personal life sto- ries, business strategies, fitness and mental health, global warming and conservation, various forms of art and cultural practices, teaching methodolo- gies and learning styles, recipes and culinary tech- niques, ethical dilemmas and existential questions, space exploration and celestial phenomena, legal issues and courtroom drama, examination of past events and civilizations, ancient myths and legends,
21 messages paid for by a business or individual to promote product, service, or event
Table 10: The list of genre descriptions.
C Hyperparameters
We employed gpt-3.5- -ttuboo000 for sentence gen- eration. For the generation of unlabeled sentences, we set the temperature to 1.3, top_p to 1.0, and both presence_penaae and ceqquency_penalyy to 0.3. The input prompts were one-shot prompts; in the example, 10 sentences were generated at once, and during the generation process, 20 sentences were generated at once. During the generation of positive sample annotations, we set the temperature to 1.0 and top_p to 0.9. In the generation of neg-

---

Method
Entailment
Contradiction
Input: A younn mam getting ready release red lite.
young man relensing
releasing red
DINO
A getiing ready releuse red kite.
hig deal to him and he didn't know how he would explain his
The man is prepared to fly the kite.
man playing baskethall. The woman is Bying kite
GenSE
A man is planning to fly a kite.
SynCSE--cratchtcouus)
young man preparing
young man gerting ready release
A prepares crimson kite.
young gets ready catch kite that heen released.
Input: One of the hotel's rooms
The hotel room
The other one is on fire
DINO
One the hotel roms
A mom inside hotel.
It's hotel' 's
GenSE
A hotel room.
There is no room the hocel.
A room in the hotel.
None of the hotel's rooms. hotel's
SynChE-ssracchhoours)
fully
Table 11: Comparison of different data synthesis methods. For samples of DINO and GenSE, cite the generation sentences reported in (Ye et al., 2022b).
Positive prompts pools
Prompt1:
Please paraphrase the input sentence phrase, providing an alternative expression with the same meaning.
Prompt2: Rewrite the following sentence phrase using different words and sentence structure while preserving its original meaning.
Prompt3: Create sentence or phrase that is also true, assuming the provided input sentence phrase is true.
Prompt4: Please provide a concise paraphrase of the input sentence or phrase, maintaining the core meaning while altering the words and sentence structure. Feel free to omit some of the non-essential details like adjectives adverbs.
Table 12: Positive prompts pools. During the generation of positive samples, a prompt is sampled with a certain probability and inserted into the few-shot input prompts Table 2, which are input in the form of multi-turn dialogues.
Unlabeled Sentence
Positive Label
Haad Negative Label
All
D
Cost the synthesize data
0,00007
0.00067
3U080766
per sentence)
We used gpt-3.5-turbo to synthesize data that is not very expensive, currently costing 0.0015 dol- lars per 1K tokens for input and 0.002 dollars per IK tokens for output. Concretely, there are three parts in the data generation process: unlabeled sen- tences, positive labels, and hard negative labels. Since the length of each input varies, to quantify the cost, we randomly sampled 40 inputs and cal- culated the average cost per sentence. As detailed in Table 13, our method cost total of around 1.5 $ for generating 1000 sentences, and the total cost of producing the 276k sentences used in our periments of SynCSEE-sraach in Table 2 is around 414 $. In the domain specialized task (Table 6), we just generate 37k sentence pairs and significantly surpass SimCSE the target domain, and the cost is around 55 $. We would like to highlight that
Table 13: The cost analysis of our method generating sentences with gpt-3.5-turbo.
ative sample annotations, we set the temperature to 1.0 and top_p to 0.95. Both positive and nega- tive sample generations were 5-shot. Our training framework is based on SimCSE, which forcibly truncates parts of the sentence exceeding 32 words during training. To maintain fair comparison, we filter out sentences with more than words before training with the SimCSE framework after generating sentences with Syrahh-scratch.

---

Data
0%
20%
40%
60%
80%
100%
Avg. STS
82.04
82.10
82.73
82.58
82.75
82.61
Table 14:
Performance of SimCSE NLI when com- bined with varying amounts of our synthetic SynCSE- scratch dataset. We report the performance on the avg STS results on the test set.
the rate per sentence above is much cheaper than manually labeling data; for instance, in machine translation tasks, human translation (around $0.1 per word) can be thousands of times costlier than using gpt-3.5-turbo (Neubig and He, 2023).
E Synthetic data amount
We also analyzed the impact on performance when augmenting the volume of generated data on the manually curated dataset, as shown in Table 14. Since the domain of SynCSE-scratch is established upon its completion, the performance ceases to in- crease after certain amount of SynCSE-scratee data is added to SimCSE. This may due to the fact that the added data is randomly sampled, which likely already covers the domain of SynCSE- scratch.

---

Prompt pool for generating unlabeled sentences with specified genres and topics.
Devise [number] distinct and diverse Sentences that may appear in [hhe description of the genre], covering a range of subjects ([opicı), [lopic2], [topics] [topica], [topics], [topica].ete.). These sentences should present mix of complexity levels, from elementary structures akin to "Birds fly in the sky." to more sophisticated ones. Aim for a low degree lexical overlap and an extensive vocabulary. Incorporate a variety of sentence modes declarative, interrogative, exclamatory, imperative, and descriptive. The sentences should oscillate in tone between informative, persuasive, descriptive, and narrative, and should present varied perspectives. Vary the length of the sentences, ranging from concise phrases of 3-5 words to longer sentences containing 20-40 words.
Prompt1:
Prompt2: Construct [number]] varied and diverse sentences that may appear in [the description of the genre] encompassing multitude of topics such as [topicı], [topicy], [topicy], [topica], [topicz], [topicş] and Aim for low lexical repetition and rich vocabulary variety. Ensure to blend different sentence structures declarative, interrogative, exclamatory, imperative, and descriptive. The sentences should convey different tones, including informative, persuasive, descriptive, and narrative, and should be expressed from various perspectives. Vary the length of the sentences, ranging from concise phrases of 3-5 words to longer sentences containing 20-30 words.
Prompt3: Create [number] varied and diverse sentences that may appear in the description of the genre]. spanning an array of topics such as [topici]. [topic2], [topics]. [topica], [topics], [topics] and so on. Include a mix of sentence styles declarative, interrogative, exclamatory, imperative, and descriptive. Vary the length of the sentences, ranging from concise phrases of 3-5 words to 25-35 words.
Prompt4:
Compose [number]] assorted and diverse sentences that may appear in the description of the genre), touching upon various themes like [topicı], [topicz], [topics], [lopica]. [Lopics], [Lopics] and so on. The sentences should reflect spectrum of complexity levels, from basic structures such as "The sets in the west." to more elaborate forms. Aspire for minimal lexical redundancy and wide array of vocabulary. Incorporate blend of sentence types declarative, interrogative, exclamatory, imperative, and descriptive. The sentences should shift tone, eycling between informative, persuasive, descriptive, and narrative, and should vary in perspective. Vary the length of the sentences, ranging from concise phrases of 3-5 words to longer sentences containing 25-45 words.
Table 15:
Prompt pool for generating unlabeled sentences with specified genres and topics. When generating unlabeled sentences, we randomly sample prompt. Here, "[number]] indicates the number of sentences generated each time, "[the description of the genre]" provides a specific description of the given genre, and "[topic;]" is a topie sampled from the topic list.

---

Prompt pool for generating image captions.
Please randomly generate [nummeer] diverse sentences in the style of [the description of the genre], similar to image captions in the Flickr30k dataset, covering wide range of subjects, actions, contexts, and settings. The sentences do not need to be semantically related. Please make sure generate sentences with uniform distribution in length, ranging from short phrases longer ones.
Prompt1:
Prompt2: Kindly generate [number]] unique sentences reflecting the style of [the description of the genre). drawing inspiration from the diverse scenarios found in the Flickr30k dataset. The sentences should cover an extensive array of themes, actions, surroundings, and situations. Please create mix simple sentences like "A man strums guitar", along with more elaborate ones.
Prompt3:
Kindly generate [nummee]] distinct sentences with simple structures, each may appear in [the description of the genre]. These sentences should touch on wide array of topies, actions, and environments without necessarily having a semantic link. Strive to provide sentences of various lengths, from short to moderately long, all while maintaining simplicity and clarity.
Prompt4:
Please randomly generate [number] diverse and descriptive sentences which may appear in [the description of the genre]. ensuring they mirror the structure the following image captions: [ezample]], [eramplez), (examplea]] [eramplc4]. Each sentence should vividly portray the primary activity in hypothetical image or segment of a video, articulating the main subject(s), their actions, and any important objects or secondary subjects involved.
The Prompt pool for image caption generation. We designed a separate prompt for generat- ing image captions to enhance diversity. Here, "[rumber)" denotes the number of sentences generate, "[the descriplion of the genre]" provides description for generating captions, and "[ezamplea]] is a randomly sampled example sentence.
Table 16:


Reviews:
[{"review_id": "v1Xlg3To9y", "reviewer": "Reviewer_P81C", "summary": "", "questions": "1. Can you provide more details about the synthetic data generation process? How did you ensure the diversity and quality of the generated data?\n2. Have you considered evaluating the performance of SynCSE on tasks that require specific domain knowledge? How do you think SynCSE would perform in such scenarios?", "limitations": "", "rating": 4, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your time and helpful comments! We address your concerns below:\n\n\n### Q1: Limited comparison with other synthetic datasets\nThank you for your advice! Methods based on contrastive learning have dominated this task, and they require NLI-style data. Therefore, we only compare with synthetic datasets that have NLI-style formats. During the rebuttal, we further compare our approach with the effective work of ZeroGen [1]. Specifically, we selected those examples that have been provided in both \"entailment\" and \"not_entailment\" sentences to construct sentence pairs, totaling 46,311 pairs, as training data. To ensure a fair comparison, we randomly sampled an equal number of examples generated by SynCSE-scratch. The experimental results are as follows:\n\n|                             | STS12 | STS13 | STS14 | STS15 | STS16 | STSb | SICK-R | Avg |\n|:-----------------------------|:-------|:-------|:-------|:-------|:-------|:------|:--------|:-----|\n| ZeroGen_RoBERTa_base             |      51.68 | 71.45 | 58.80 | 67.04 | 70.04 |    65.00     |      66.88      | 64.41      |\n| SynCSE-scratch_RoBERTa_base |  71.81 | 83.43 | 76.90 | 83.39 | 82.33 | 82.89 |  77.39 | 79.73      |\n| ZeroGen_RoBERTa_large             |    50.97 | 70.90 | 59.97 | 69.59 | 68.79 |    65.43     |      65.72      | 64.48     |\n| SynCSE-scratch_RoBERTa_large |     70.33 | 81.53 | 75.31 | 83.02 | 81.57 |    81.15    |      79.29      | 78.89    |\n\nResults show that our method consistently outperforms ZeroGen. This could be because ZeroGen is not designed specifically for this task, and there are significant differences in how to enhance data diversity. Additionally, in ZeroGen's implementation of generating QNLI, unlabeled sentences are sampled from a specific domain's corpus, which can also be costly. Conversely, our method only requires the target domain's name.\n\n\n\n### Q2: Lack of evaluation on specialized domains\nThanks for the advice! We conduct experiments on two another datasets focused on specialized domains – the BIOSSES[2] dataset of a semantic textual similarity task for the biomedical domain, and the StackOverflowDupQuestions[3] dataset of a reranking task for programming questions domain. Specifically, our experimental design is based on the assumption that we only have access to the names of the target domains (i.e., 'biomedicine' and 'Stack Overflow website') without any data available. We run SynCSE-scratch in these settings. Concretely, we first generate 37k unlabeled sentences in the respective domain following the procedure described in Section 2.4, then generate positive and hard negatives for these sentences, and train the models.   \n\nThe results are listed as follows:\n\n| Model     | Method     | BIOSSES (Spearman's correlation) |  StackOverflowDupQuestions  (Mean Average Precision) |\n|:-------------|:--------------|:------------------------------------|:----------------------------|\n|                             | unsup-SimCSE                  |     68.86        |       39.25    |\n| RoBERTa-base  | SynCSE-scratch      |   80.12  |  43.22    |\n|               | unsup-SimCSE                               |      71.96       |        42.56   |\n| RoBERTa-large | SynCSE-scratch        |  77.73     |     45.67  |\n\nWe use the publicly available unsupervised SimCSE model checkpoint that was trained on the Wikipedia domain for comparison. This is because we assumed no access to unlabeled sentences in these domains, which is a practical setting. Our observations show that SynCSE-scratch outperforms the unsupervised SimCSE baseline significantly in both domains. This experiment further demonstrates the superiority of our method on new domains where no data is available – traditional unsupervised approaches like SimCSE tend to experience a domain transfer drop in performance in such scenarios. We will add these experiments to the next revision of the paper.\n\n### Q3:  More details about the synthetic data generation process.\nUsing SynCSE-scratch as an example, the complete data generation process includes two parts: \n1. Performing one-shot generation to generate unlabeled sentences, specifying topics and genres as described in Appendix B and Table 8; \n2. Generating positive/hard negative labels in a 5-shot manner, as illustrated in Figures 3 and 4. \nWe keep the sentences less than 32 words following the hyperparameter of max sequence length in SimCSE during training (as described in line 931). The hyperparameters for generation are detailed in Appendix C. The version of the ChatGPT is: gpt-3.5-turbo-0301.\n\n### Q4: How to ensure the diversity and quality of the generated data\n\nIndeed, this is one of our core contributions. Specifically, \n\n1. For diversity, we ensure the diversity of the synthetic dataset primarily through two strategies: 1) Manually drafting and specifying the topics and genres (Appendix B) when generating data, so that the newly generated data's topics and genres differ from existing data to enhance diversity. 2) Constructing prompt and example pools, and varying the prompts and exemplars during generation to increase diversity;  \n\n2. For quality,  we found that ChatGPT’s performance in generating unlabeled sentences and positive/negative labels is reasonably good and satisfying by manually investigating a few examples. We just filter out unlabeled sentences of more than 32 words to match the max sequence length used in SimCSE (line 931)  \n\nTo validate the effectiveness of our data synthesis process, we conduct an ablation experiment, where (1)  we do not specify topics or genres when generating unlabeled sentences, and (2) we do not vary the prompt and exemplars but fix them the same (that are randomly selected from the pools) when generating the positive and hard negative labels.  Other settings are kept the same as in SynCSE-scratch. We perform the ablation experiment on 22k examples based on Roberta-base, and the results are shown below. We denote the baseline without diversity control as “Naive Generation”. \n\n| Method       | STS12 | STS13 | STS14 | STS15 | STS16 | STSb | SICK-R | Avg |\n|:-------------|:----------------|:---------|:---------|:---------|:---------|:---------|:--------|:----------|\n| Naive Generation |    64.65 | 75.86 | 62.94 | 72.79 | 71.61 |    72.76     |    71.57   | 70.31          |\n|     SynCSE-scratch   |    70.89  | 83.79 |  76.48  |  83.28  |  81.97  |  82.36   |  76.14  |  79.27      |\n\nOur method outperforms the Naive Generation baseline by an average of 8.96 points, demonstrating the critical role of diversity control in our data synthesis process. \n\nReference \n\n[1] Ye, J., Gao, J., Li, Q., Xu, H., Feng, J., Wu, Z., ... & Kong, L. (2022, December). ZeroGen: Efficient Zero-shot Learning via Dataset Generation. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing (pp. 11653-11669).  \n[2] Soğancıoğlu, G., Öztürk, H., & Özgür, A. (2017). BIOSSES: a semantic sentence similarity estimation system for the biomedical domain. Bioinformatics, 33(14), i49-i58.  \n[3] Liu, X., Wang, C., Leng, Y., & Zhai, C. (2018, November). Linkso: a dataset for learning to retrieve similar question answer pairs on software development forums. In Proceedings of the 4th ACM SIGSOFT International Workshop on NLP for Software Engineering (pp. 2-5).", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. Innovative methodology: The use of large language models (LLMs) to synthesize data for contrastive learning is a novel approach.\n2. Practical application: The framework addresses the challenge of acquiring labeled data in certain domains and provides a solution using synthetic data.\n3. Comprehensive evaluation: The paper includes experiments on sentence similarity and reranking tasks, demonstrating the effectiveness of SynCSE in different settings.\n4. Clear exposition: The paper provides a well-structured explanation of the proposed framework, making it easy to understand.", "reasons_to_reject": "1. Limited comparison with other synthetic datasets: The comparison with GenSE and DINO is limited, and a more detailed analysis and comparison with other approaches would strengthen the paper.\n2. Lack of evaluation on specialized domains: The paper does not evaluate the performance of SynCSE on tasks requiring specialized domain knowledge, which limits its applicability in certain fields.", "questions_to_authors": "1. Can you provide more details about the synthetic data generation process? How did you ensure the diversity and quality of the generated data?\n2. Have you considered evaluating the performance of SynCSE on tasks that require specific domain knowledge? How do you think SynCSE would perform in such scenarios?", "paper_topic_and_main_contributions": "The paper presents SynCSE, a contrastive learning framework that trains sentence embeddings using synthetic data. The authors explore two variants of SynCSE: SynCSE-partial, which synthesizes positive and hard negative examples given unlabeled sentences, and SynCSE-scratch, which generates sentences and their annotations from scratch. Experimental results show that both SynCSE-partial and SynCSE-scratch outperform unsupervised baselines and achieve comparable performance to supervised models in most settings.", "excitement": 4, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "L1n5HZ6xbt", "reviewer": "Reviewer_YEGU", "summary": "", "questions": "a) Can you provide more details on the ethical considerations taken into account during the synthesis of data using large language models? How do you ensure that the generated synthetic data is free from biases and representative of the underlying distribution?\n\nb) How does the computational cost of generating synthetic data using large language models impact the feasibility and scalability of the proposed approach? Have you conducted any analyses to quantify the resources required for training the models and generating the synthetic data?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your helpful comments on our work! Below we address your concerns:\n\n### Q1: Novelty of this work compared to  DINO[1] \n\nWe would like to highlight two main differences between this work and DINO: \n\n1. The synthesis methods are essentially different – we designed a top-down synthesis scheme to ensure diversity, where we first select topics/genres (Section 2.4, Appendix B), sample prompts, and exemplars (Section 2.3, Appendix A), and then produce the final data examples. Such a diversity control is distinct from DINO, and in fact a critical design of SynCSE. To validate the effectiveness of our diversity control, we conduct an ablation experiment, where (a)  we do not specify topics or genres when generating unlabeled sentences, and (b) we do not vary the prompt and exemplars but fix them the same when generating the positive and hard negative labels.  Other settings are kept the same as in SynCSE-scratch. We perform the ablation experiment on 22k examples based on Roberta-base, and the results are shown below. We denote the baseline without diversity control as “Naive Generation”. \n\n| Method       | STS12 | STS13 | STS14 | STS15 | STS16 | STSb | SICK-R | Avg |\n|:-------------|:----------------|:---------|:---------|:---------|:---------|:---------|:--------|:----------|\n| Naive Generation |    64.65 | 75.86 | 62.94 | 72.79 | 71.61 |    72.76     |    71.57   | 70.31          |\n|     SynCSE-scratch   |    70.89  | 83.79 |  76.48  |  83.28  |  81.97  |  82.36   |  76.14  |  79.27      |\n\n\nOur method outperforms the Naive Generation baseline by an average of 8.96 points, demonstrating the critical role of diversity control in our data synthesis process, which is one of the key novelties compared to DINO.\n\n2. As the reviewer pointed out, while DINO uses GPT-2 as the backend, we leverage a much stronger model, ChatGPT. Combining ChatGPT with specially designed prompting strategies (e.g. the diversity control as described above), we were able to push the performance of synthetic datasets to a new level, greatly outperforming traditional unsupervised methods and approaching the performance of supervised methods. Our results first mark that data synthesis from LLMs could be **practically** useful and superior to using real unlabeled data. We think that this is a non-trivial and “novel” empirical contribution compared to DINO.\n\n\n\n\n### Q2: The cost of generating synthetic dataset and training the models\nWe used gpt-3.5-turbo to synthesize data that is not very expensive, currently costing 0.0015 dollars per 1K tokens for input and 0.002 dollars per 1K tokens for output. Concretely, there are three parts in the data generation process: unlabeled sentences, positive labels, and hard negative labels. Since the length of each input varies, to quantify the cost, we randomly sampled 40 inputs and calculated the average cost per sentence:\n|                      | Unlabeled Sentence | Positive Label | Hard Negative Label | All |\n|:----------------------|:--------------------|:----------------|:---------------------|:-----|\n| cost($ per sentence) |        0.00007            |   0.00067            |         0.00076            |    0.0015 |\n\nThis makes a total of 1.5 dollars for generating 1000 sentences, and the total cost of producing the 276k sentences used in our experiments of SynCSE-scratch in Table 1 is 414 dollars. In our response to Q2 of Reviewer P81C, we just generate 37k sentence pairs and significantly surpass SimCSE in the target domain, and the cost is 55 dollars.  \nWe would like to highlight that the rate per sentence above is much cheaper than manually labeling data; for instance, in machine translation tasks, human translation (around $0.1 per word) can be thousands of times costlier than using gpt-3.5-turbo [2]. \n\n### Q3: Ethical considerations of the synthetic dataset\n\nThis is a good point. To evaluate the safety of our synthetic dataset. We ask five annotators (one of which is a psychological counselor and the other four are postgraduate students) to annotate whether the generated sentences have ethical problems. Specifically, we randomly select 100 sentences from those generated by SynCSE-scratch, and each sentence was independently evaluated by the five people for potential ethical problems. The results are as follows:\n|        | Human 1 (counselor)|Human 2| Human 3| Human 4|Human 5 | Avg |\n|:----------------------|:--------------|:--------------|:-------|:-------|:-------|:-------|\n| Fraction of ethically unsafe data |        1%        |           0%     |      0%       |   1%   | 0%  | 0.4%    |\n\nAs the table suggests, only a minor portion of the data was classified as ethically unsafe, indicating that our synthetic dataset upholds a certain level of safety concerning ethical issues. This is not surprising since ChatGPT, the backend in our experiments, is already heavily aligned to avoid producing text with ethical or safety issues. \n\nWe thank the reviewer for the advice and raising these good points, and we will add these additional results and analysis above to the next revision of the paper.\n\nReference\n\n [1] Schick, T., & Schütze, H. (2021, November). Generating Datasets with Pretrained Language Models. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing (pp. 6943-6951).  \n [2] Neubig, G., & He, Z. (2023). Zeno GPT Machine Translation Report", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "a) The paper addresses the issue of data availability, particularly in domains where labeled or large-scale unlabeled data is scarce or hard to obtain. By proposing a method to create synthetic data samples, it provides a practical solution for training effective sentence embeddings.\n\nb) Since the SynCSE approach is based on synthetic data generation, it is not restricted to specific domains or data types. This makes it potentially applicable to a wide range of NLP tasks, even in domains where acquiring large-scale unlabeled data is challenging.", "reasons_to_reject": "a) The novelty of the proposed method is limited. There is an existing work [1] that has a very similar objective and methodology with this paper. The main difference between this paper and [1] is that this paper adopts ChatGPT, a much more powerful language model, to produce the synthetic data. More insights on how ChatGPT makes improvements over previous language models are needed.\n\nb) Generating synthetic data using large language models can be computationally expensive and resource-intensive. The paper should provide insights into the computational requirements of the proposed approach, considering the scale of the language models used and the amount of data generated.\n\nReference\n[1] Timo Schick and Hinrich Schütze. 2021. Generating datasets with pretrained language models. In Proceedings of EMNLP.", "questions_to_authors": "a) Can you provide more details on the ethical considerations taken into account during the synthesis of data using large language models? How do you ensure that the generated synthetic data is free from biases and representative of the underlying distribution?\n\nb) How does the computational cost of generating synthetic data using large language models impact the feasibility and scalability of the proposed approach? Have you conducted any analyses to quantify the resources required for training the models and generating the synthetic data?", "paper_topic_and_main_contributions": "This paper discusses a novel approach for improving unsupervised sentence representation learning using contrastive learning methods. Sentence embeddings are vector representations of sentences that capture their semantic meaning, and contrastive learning is a popular technique used to learn meaningful representations from data. The authors propose a new contrastive learning framework called SynCSE. This framework leverages large language models to synthesize the required data samples for contrastive learning. The experimental results demonstrate the effectivenss of the proposed framework.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "NA", "Typos_Grammar_Style_and_Presentation_Improvements": "I think overall the paper is well-written and effectively communicates the research objectives, methodologies, and results."}, "extra_text": {}}, {"review_id": "26heprHtwp", "reviewer": "Reviewer_DJxQ", "summary": "", "questions": "How to ensure the quality of synthetic data? What is the volume?\nWhat is the improvement compared with Roberta or other pre-trained encoders like BERT without additional sentence embedding learning?\nWhat is the generalization of this embedding method to other languages?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate your insightful feedback. Here are our responses to your concerns:\n\n### Q1:The novelty of our paper.\n\nWe would like to highlight two main novel contributions of this work: \n\n1. Using LLMs to generate data is far from enough and our carefully designed synthesis methods play an important role – we designed a top-down synthesis scheme to ensure diversity, where we first select topics/genres (Section 2.4, Appendix B), sample prompts and exemplars (Section 2.3, Appendix A), and then produce the final data examples. Such a diversity control is in fact a critical design of SynCSE. To validate the effectiveness of our diversity control, we conduct an ablation experiment, where (a)  we do not specify topics or genres when generating unlabeled sentences, and (b) we do not vary the prompt and exemplars but fix them the same when generating the positive and hard negative labels.  Other settings are kept the same as in SynCSE-scratch. We perform the ablation experiment on 22k examples based on Roberta-base, and the results are shown below. We denote the baseline without diversity control as “Naive Generation”. \n\n| Method       | STS12 | STS13 | STS14 | STS15 | STS16 | STSb | SICK-R | Avg |\n|:-------------|:----------------|:---------|:---------|:---------|:---------|:---------|:--------|:----------|\n| Naive Generation |    64.65 | 75.86 | 62.94 | 72.79 | 71.61 |    72.76     |    71.57   | 70.31          |\n|     SynCSE-scratch   |    70.89  | 83.79 |  76.48  |  83.28  |  81.97  |  82.36   |  76.14  |  79.27      |\n\nOur method outperforms the Naive Generation baseline by an average of 8.96 points, demonstrating the critical role of diversity control in our data synthesis process, which is one of the key novelties presented in this work.\n\n2. Combining ChatGPT with specially designed prompting strategies (e.g. the diversity control as described above), we were able to push the performance of synthetic datasets to a new level, greatly outperforming traditional unsupervised methods and approaching the performance of supervised methods. Our results first mark that data synthesis from LLMs could be **practically** useful and superior to using real unlabeled data. We think that this is a non-trivial and “novel” empirical contribution compared to previous work.\n\n\n\n### Q2: How to ensure the quality of the data.\n\n\nFor diversity, we ensure the diversity of the synthetic dataset primarily through two strategies: 1) Manually drafting and specifying the topics and genres (Appendix B) when generating data, so that the newly generated data's topics and genres differ from existing data to enhance diversity. 2) Constructing prompt and example pools, and varying the prompts and exemplars during generation to increase diversity; \nFor quality,  we found that ChatGPT’s performance in generating unlabeled sentences and positive/negative labels is reasonably good and satisfying by manually investigating a few examples. We just filter out unlabeled sentences of more than 32 words to match the max sequence length used in SimCSE (line 931)\n\n\n\n\n### Q3: The volume of the generated dataset.\nFor a fair comparison, we generated a dataset similar in size to the one used in the supervised experiments with SimCSE (around 276K sentence pairs). \n\n### Q4:  The improvement compared with BERT without sentence embeddings learning.\n\nSimCSE (Table 5) provides the performances in comparison with a BERT model that does not include sentence embeddings learning.\n\n### Q5:  The generalization of this embedding method to other languages\nThat is a good point. ChatGPT supports multiple languages and can directly output in various languages; however, we have not conducted any experiments in this regard. We can explore this aspect as part of our future work.\n\n### Q6: Typos\nThanks for catching those typos! We will fixe them in the next revision of the paper.\n\nWe will add these experiments to the next revision of the paper.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The use of LLM in specific scenarios is a new trend in NLP and the authors designed lots of prompts. \nThere is an amount of experiments and have better performance than other models and other synthetic data.\nWriting of the paper is satisfying.", "reasons_to_reject": "Novelty of the work is limited. It seems that the core lies in the usage of LLM to generate training data.", "questions_to_authors": "How to ensure the quality of synthetic data? What is the volume?\nWhat is the improvement compared with Roberta or other pre-trained encoders like BERT without additional sentence embedding learning?\nWhat is the generalization of this embedding method to other languages?", "paper_topic_and_main_contributions": "This paper trains sentence embeddings in a contrastive learning manner, in which the contrastive training data is generated by LLM with many prompts in two mode. A series of experiments and analysis are conducted to prove the effectiveness of the work.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "Information (like year, conference name) of last reference is partly missing."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper introduces SynCSE, a contrastive learning framework for training sentence embeddings using synthetic data. It explores two variants: SynCSE-partial, which generates positive and hard negative examples from unlabeled sentences, and SynCSE-scratch, which creates sentences and their annotations from scratch. Experimental results show that both variants outperform unsupervised baselines and perform competitively with supervised models in most settings.

Reviewers have all found that the paper is sound and strong, therefore, I recommend acceptance of this paper.
Decision: Accept-Main