# Grouped Contrastive Learning of Self-Supervised Sentence Representation
## 4 Experiments
### Transfer Task  
Transfer task is used to evaluate the performance of downstream tasks using sentence representation [36]. Generally, a classifier is added on the top layer of a sentence representation model to evaluate the performance of the transfer task. Note that the classifier (consisting of linear layers) can be trained, while the sentence representation model needs to be frozen. Our proposed method underwent rigorous testing across a spectrum of tasks, including MR [44], CR [45], SUBJ [46], MPQA [47], SST-2 [48], TREC [49], and MRPC [50]. The pre-trained stage is the same as for STS tasks. The evaluation results are shown in Table 3. We find that the overall tendency of results is the same as STS tasks. However, there are two abnormalities that need to be explained. (1) The pre-trained model BERT\({}_{base}\) obtains a better result compared with our proposed model on transfer tasks. Firstly, we only chose \(10^{4}\) sentences from the wiki to perform the pre-training. Secondly, the number of parameters of our model is far less than BERT\({}_{base}\). Therefore, we take a short time to perform pre-training (about 1 h on 1 Tesla v100 GPU). (2) The application of self-attention and grouping harms the performance slightly compared with GCLSR\({}_{base}\)+PWVA. A possible explanation is that the implementation of PWVA is on word vectors, which could change the original semantic information of vectors. In addition, we do not perform joint training with tasks, which means the model could not digest the learned contrastive features.  
\begin{table}
\begin{tabular}{l c c c c c c c} \hline \hline
**Model** & **STS12** & **STS13** & **STS14** & **STS15** & **STS16** & **STS-B** & **SICK-R** & **Avg.** \\ \hline Word2 vec embeddings(avg.) & 33.75 & 43.20 & 36.95 & 55.23 & 54.85 & 36.24 & 48.90 & 44.16 \\ BERT\({}_{base}\)(first-last avg.) * & 39.70 & 59.38 & 49.67 & 66.03 & 66.19 & 53.87 & 62.06 & 56.70 \\ RoBERT\({}_{base}\)(first-last avg.) * & 40.88 & 58.74 & 49.07 & 65.63 & 61.48 & 58.55 & 61.63 & 56.57 \\ CLEAR **+** & 49.00 & 48.90 & 57.40 & 63.60 & 65.60 & **75.60** & **72.50** & 61.80 \\ BERT\({}_{base}\)-flow * & **58.40** & **67.10** & 60.85 & **75.16** & 71.22 & 68.66 & 64.47 & **66.55** \\ BERT\({}_{base}\)-whitening * & 57.83 & 66.90 & **60.90** & 75.08 & **71.31** & 68.24 & 63.73 & 66.28 \\ \hline GCLSR\({}_{base}\) & 57.47 & 68.70 & 64.03 & 72.84 & 67.90 & 65.64 & 59.55 & 65.16 \\ GCLSR\({}_{base}\)+EDA & 57.33 & 68.09 & 63.65 & 72.01 & 66.63 & 65.34 & 59.71 & 64.68 \\ GCLSR\({}_{base}\)+TransL & **60.53** & 66.09 & 63.50 & 72.83 & 67.39 & 66.09 & 59.78 & 65.17 \\ GCLSR\({}_{base}\)+PWVA & 58.92 & 67.94 & 64.41 & 73.54 & 68.72 & 66.16 & **59.85** & 65.65 \\ GCLSR\({}_{base}\)+PWVA+self-att. & 57.62 & 71.00 & 65.83 & 75.51 & 69.81 & 67.41 & 59.41 & 66.66 \\ GCLSR\({}_{base}\)+EDA+self-att.+GP & 58.13 & 68.85 & 64.14 & 73.40 & 66.92 & 65.31 & 59.61 & 65.19 \\ GCLSR\({}_{base}\)+TransL+self-att.+GP & 58.80 & 68.00 & 64.70 & 73.74 & 68.50 & 67.24 & 59.67 & 65.81 \\ GCLSR\({}_{base}\)+PWVA+self-att.+GP & 57.81 & **71.01** & **65.83** & **75.62** & **70.01** & **67.58** & 59.34 & **66.75** \\ \hline \hline \end{tabular}
\end{table}
Table 2: The evaluation of sentence representation in STS tasks. All results are computed with the Spearman correlation. *: results from [21]; **+:** results from [20]; the remaining results are evaluated by us. GCLSR\({}_{base}\) means that the model only consists of a backbone, projector, and predictor. The model receives the same two word vectors as the input, i.e., no data augmentation is used. EDA is a text data augmentation method proposed by [26]. TransL denotes the data augmentation back-translation. Self-att and GP denote the self-attention mechanism and feature grouping, respectively.