# Prefix Data Augmentation for Contrastive Learning of Unsupervised Sentence Embedding
## 5 Ablation Studies
### Progress of Prefix Data Augmentation  
We aimed to understand the extent of improvement PrdSimCSE could achieve over SimCSE and, specifically, the contributions of positive and negative prefixes to PrdSimCSE. As shown in Table 7, both positive and negative prefixes were found to be effective data augmentation methods, and their combination led to further improvement in PrdSimCSE. \begin{table}
\begin{tabular}{c c c} \hline \hline
**Model** & **BQ** & **LCQMC** \\ \hline SimCSE-Bert\({}_{\text{base}}\) & 45.38 & 56.46 \\ PrdSimCSE-Bert\({}_{\text{base}}\) & 47.01 & 58.63 \\ \hline SimCSE-RoBERT\({}_{\text{base}}\) & 47.97 & 62.41 \\ PrdSimCSE-RoBERT\({}_{\text{base}}\) & 48.82 & 63.78 \\ \hline \hline \end{tabular}
\end{table} 
Table 6: Sentence embedding performance on two Chinese datasets. \begin{table}
\begin{tabular}{l c c c c c c c c} \hline \hline
**Model** & **STS12** & **STS13** & **STS14** & **STS15** & **STS16** & **STS-B** & **SICK-R** & **Avg.** \\ \hline GloVe embedding (avg.) \(\blacktriangle\) & 55.14 & 70.66 & 59.73 & 68.25 & 63.66 & 58.02 & 53.76 & 61.32 \\ BERT\({}_{\text{base}}\) (first-last avg.) & 39.7 & 59.38 & 49.67 & 66.03 & 66.19 & 53.87 & 62.06 & 56.7 \\ BERT\({}_{\text{base}}\)-flow & 58.4 & 67.1 & 60.85 & 75.16 & 71.22 & 68.66 & 64.47 & 66.55 \\ BERT\({}_{\text{base}}\)-whitening & 57.83 & 66.9 & 60.9 & 75.08 & 71.31 & 68.24 & 63.73 & 66.28 \\ IS-BERT\({}_{\text{base}}\)\(\bigtriangledown\) & 56.77 & 69.24 & 61.21 & 75.23 & 70.16 & 69.21 & 64.25 & 66.58 \\ CT-BERT\({}_{\text{base}}\) & 61.63 & 76.8 & 68.47 & 77.5 & 76.48 & 74.31 & 69.19 & 72.05 \\ SimCSE-BERT\({}_{\text{base}}\) & 68.4 & 82.41 & 74.38 & 80.91 & 78.56 & 76.85 & 72.23 & 76.25 \\ PrdSimCSE-BERT\({}_{\text{base}}\)\(\bigtriangledown\) & 68.21 & 83.56 & 75.44 & 81.71 & 79.79 & 78.86 & 73.75 & 77.33 \\ \hline RoBERT\({}_{\text{base}}\)(first-last avg.) & 40.88 & 58.74 & 49.07 & 65.63 & 61.48 & 58.55 & 61.63 & 56.57 \\ RoBERT\({}_{\text{base}}\)-whitening & 46.99 & 63.24 & 57.23 & 71.36 & 68.99 & 61.36 & 62.91 & 61.73 \\ SimCSE-RoBERT\({}_{\text{base}}\) & 70.16 & 81.77 & 73.24 & 81.36 & 80.65 & 80.22 & 68.56 & 76.57 \\ PrdSimCSE-RoBERT\({}_{\text{base}}\)\(\bigtriangledown\) & 69.29 & 83.52 & 75.02 & 81.13 & 78.84 & 78.52 & 73.37 & 77.10 \\ \hline \hline \end{tabular}
\end{table} 
Table 4: Sentence embedding performance on STS tasks (Spearman correlation, “all” setting). We highlight the highest numbers among models with the same pretrained encoder. \(\blacktriangle\): results from [6]; \(\bigtriangledown\): results from [11]. \(\bigtriangledown\): results from ourselves; all other results are from [13]. \begin{table}
\begin{tabular}{l c c c c c c c} \hline \hline
**Model** & **MR** & **CR** & **SUBJ** & **MPQA** & **SST-2** & **TREC** & **MRPC** & **Avg.** \\ \hline GloVe embedding(avg.) \(\blacktriangle\) & 77.25 & 78.3 & 91.17 & 87.85 & 80.18 & 83 & 72.87 & 81.52 \\ BERT-[CLS]embedding \(\blacktriangle\) & 78.68 & 84.85 & 94.21 & 88.23 & 84.13 & 91.40 & 71.13 & 84.66 \\ SimCSE-BERT\({}_{\text{base}}\)\(\bigtriangledown\) & 81.18 & 86.46 & 94.45 & 88.88 & 85.5 & 89.8 & 74.43 & 85.81 \\ PrdSimCSE-BERT\({}_{\text{base}}\)\(\diamondsuit\) & 81.32 & 87.32 & 94.67 & 88.11 & 86.81 & 80.91 & 76.23 & 85.01 \\ \hline \hline \end{tabular}
\end{table} 
Table 5: Transfer task results of different sentence embedding models (measured as accuracy). \(\blacktriangle\): results from [6]; \(\bigtriangledown\): results from [13]. \(\diamondsuit\): results from ourselves. We highlight the highest numbers among models with the same pretrained encoder. Sentence embeddings play a pivotal role across a spectrum of downstream tasks including text matching, sentiment analysis, and question answering systems. The advancements facilitated by Predix data augmentation in sentence embedding models hold the promise of enhancing the efficacy of these specific downstream applications.