## 5 Experiments
### Third Experiment: Cosine Similarity Distribution
\begin{table}
\begin{tabular}{l c c c c c c c c} \hline \hline
**Model** & **MR** & **CR** & **SUBJ** & **MPQA** & **SST-2** & **TREC** & **MRPC** & **Avg.** \\ \hline \multicolumn{8}{c}{**Unsupervised models**} \\ \hline Glove embeddings (avg.) \({}^{\dagger}\) & 77.25 & 78.30 & 91.17 & 87.85 & 80.18 & 83.00 & 72.87 & 81.52 \\ Skip-thought \({}^{\spadesuit}\) & 76.50 & 80.10 & 93.60 & 87.10 & 82.00 & 92.20 & 73.00 & 83.50 \\ \hline Avg. BERT embedding \({}^{\dagger}\) & 78.66 & 86.25 & 94.37 & 88.66 & 84.40 & 92.80 & 69.54 & 84.94 \\ BERT-[CLS] embedding \({}^{\dagger}\) & 78.68 & 84.85 & 94.21 & 88.23 & 84.13 & 91.40 & 71.13 & 84.66 \\ IS-BERT\({}_{\text{base}}\)\({}^{\spadesuit}\) & 81.09 & 87.18 & 94.96 & 88.75 & 85.96 & 88.64 & 74.24 & 85.83 \\ CT-BERT\({}_{\text{base}}\)\({}^{\infty}\) & 79.84 & 84.00 & 94.10 & 88.06 & 82.43 & 89.20 & 73.80 & 84.49 \\ SimCSE\({}_{\text{unsup}}\)-BERT\({}_{\text{base}}\) & 81.18 & 86.46 & 94.45 & 88.88 & 85.50 & 89.80 & 74.43 & 85.51 \\ SimCSE\({}_{\text{unsup}}\)-BERT\({}_{\text{base}}\)-MLM & 82.92 & 87.23 & 95.71 & 88.73 & 86.81 & 87.01 & 78.07 & 86.64 \\ DiffCSE-BERT\({}_{\text{base}}\)\({}^{\blacksquare}\) & 82.69 & 78.23 & 95.23 & 89.28 & 86.60 & 90.40 & 76.58 & 86.86 \\ \hline \multicolumn{8}{c}{**Supervised models**} \\ \hline InferSent-GloVe \({}^{\dagger}\) & 81.57 & 86.54 & 92.50 & 90.38 & 84.18 & 88.20 & 75.77 & 85.59 \\ Universal Sentence Encoder \({}^{\dagger}\) & 80.09 & 85.19 & 93.98 & 86.70 & 86.38 & 93.20 & 70.14 & 85.10 \\ \hline SBERT\({}_{\text{base}}\)\({}^{\dagger}\) & 83.64 & 89.43 & 94.39 & 89.86 & 88.96 & 76.00 & **87.41** \\ SBERT\({}_{\text{base}}\)-nli-v2 & 83.09 & 89.33 & 94.98 & 90.15 & 87.92 & 87.00 & 75.25 & 86.82 \\ SelfCCL-SBERT\({}_{\text{base}}\) & 83.02 & 89.46 & 95.05 & 90.18 & 87.97 & 87.00 & 75.94 & 86.95 \\ \hline SG-BERT\({}_{\text{base}}\)\({}^{\diamond}\) & 82.47 & 87.42 & 95.40 & 88.92 & 86.20 & 91.60 & 74.21 & 86.60 \\ SimCSE\({}_{\text{sup}}\)-BERT\({}_{\text{base}}\) & 82.69 & 89.25 & 84.81 & 89.59 & 87.31 & 88.40 & 73.51 & 86.51 \\ SupMPN-BERT\({}_{\text{base}}\)\({}^{*}\) & 82.93 & 89.26 & 94.76 & 90.21 & 86.99 & 88.20 & 76.35 & 86.96 \\ SelfCCL-BERT\({}_{\text{base}}\) & 82.89 & 89.22 & 94.78 & 90.51 & 87.15 & 87.60 & 76.17 & 86.90 \\ \hline \hline \end{tabular}
\end{table} 
Table 5: Assessment results for the seven transfer learning tasks. The prediction accuracy is given for each task in the columns. The best average result is in bold in the last column. \({\dagger}\): [33], \({\spadesuit}\): [6], \(\diamondsuit\): [8], \(\infty\): [11], \(\blacksquare\): [13], \(*\): [15], SBERT-base-nil-v2: reproduced by ourselves, and the rest of the results are taken from [12]. We use \"bert-base-uncased\" ([[https://huggingface.co/bert-base-uncased](https://huggingface.co/bert-base-uncased)]([https://huggingface.co/bert-base-uncased](https://huggingface.co/bert-base-uncased)) (accessed on 13 December 2022)) and \"nli-bert-base\" ([[https://huggingface.co/sentence-transformers/nli-bert-base](https://huggingface.co/sentence-transformers/nli-bert-base)]([https://huggingface.co/sentence-transformers/nli-bert-base](https://huggingface.co/sentence-transformers/nli-bert-base)) (accessed on 13 December 2022)) checkpoints for BERT and SBERT, respectively. We have demonstrated the results in Figure 4, where the x-axis represents human relatedness scores, which are normalized in the range {0,1} via division by 5, and the y-axis represents cosine similarity between sentence-embedding pairs generated by models. The color coding corresponds to the human relatedness scores. **Results:** As show in Figure 4, almost all sentence pairs generated by BERT have similarities in the range between 0.6 and 1. The SBERT model, considering that it uses softmax-classification for classifying entailment, neutral, and contradiction pairs, pushes contradictions farther away from anchors. So, for sentence pairs of contradiction relation with a high relatedness score, SBERT embeddings have a low cosine similarity.