## 5 Experiments
### Third Experiment: Cosine Similarity Distribution  
In this experiment, considering that the supervised contrastive based models have a higher average improvement, we study the correlation of cosine similarity between sentence embeddings generated by our models and baseline models (BERT, SBERT) and human-scored on the SICK dataset [64]. \begin{table}
\begin{tabular}{l c c c c c c c} \hline \hline
**Model** & **STS12** & **STS13** & **STS14** & **STS15** & **STS16** & **STS-B** & **SICK-R** & **Avg.** \\ \hline \multicolumn{8}{c}{**Supervised models**} \\ \hline InferSent-GloVe \({}^{\dagger}\) & 52.86 & 66.75 & 62.15 & 72.77 & 66.87 & 68.03 & 65.65 & 65.01 \\ Universal Sentence Encoder \({}^{\dagger}\) & 64.49 & 67.80 & 64.61 & 76.83 & 73.18 & 74.92 & 76.69 & 71.22 \\ \hline SBERTbase\({}^{\dagger}\) & 70.97 & 76.53 & 73.19 & 79.09 & 74.30 & 77.03 & 72.91 & 74.89 \\ SBERTbase-flow & 69.78 & 77.27 & 74.35 & 82.01 & 77.46 & 79.12 & 76.21 & 76.60 \\ SBERTbase-whitening & 69.65 & 77.57 & 74.66 & 82.27 & 78.39 & 79.52 & 76.91 & 77.00 \\ CT-SBERTbase & 74.84 & 83.20 & 78.07 & 83.84 & 77.93 & 81.46 & 76.42 & 79.39 \\ SBERTbase-nli-v2 & 75.33 & 84.52 & 79.54 & 85.72 & 80.82 & 84.48 & 80.77 & 81.60 \\ SelfCCL-SBERTbase & 75.50 & 84.81 & 80.05 & 85.53 & 81.07 & 84.77 & 80.67 & **81.77** \\ \hline SG-BERTbase\({}^{\diamondsuit}\) & 75.16 & 81.27 & 76.31 & 84.71 & 80.33 & 81.46 & 76.64 & 79.41 \\ SimCSEsup-BERTbase & 75.30 & 84.67 & 80.19 & 85.40 & 80.82 & 84.25 & 80.39 & 81.57 \\ SupMPN-BERTbase\({}^{*}\) & 75.96 & 84.96 & 80.61 & 85.63 & 81.69 & 84.90 & 80.72 & **82.07** \\ SelfCCL-BERTbase & 75.61 & 84.72 & 80.04 & 85.44 & 81.37 & 84.63 & 80.82 & 81.80 \\ \hline \hline \end{tabular}
\end{table} 
Table 4: _Cont._SICK, an acronym for Sentences Involving Compositional Knowledge, contains about 10,000 sentence pairs rich in lexical, syntactic, and semantic phenomena. Each sentence pair is annotated in two forms: relatedness and entailment. The human relatedness score ranges from 1 to 5; the entailment relation is categorical, consisting of entailment, contradiction, and neutral. We use a version of SICK that is located in the SentEval Github repository. There are 4500 pairs in the train-split, 500 in the trial-split used for development, and 4927 in the test-split. We analyzed the test-split statistics. There are 720 pairs with a contradiction relation that average the human relatedness scores of them at about 3.58 (whose normalized value in the range of {0,1} via division by 5 is 71.60). That is, although they have contradiction relation, they are very similar to each other based on word embeddings. Table 6 shows some sentences pairs with contradiction relations from the SICK test dataset. \begin{table}
\begin{tabular}{l c c c} \hline \hline
**Sentence 1** & **Sentence 2** & **Relation** &
\begin{tabular}{c} **Human** \\ **Relatedness Scores** \\ \end{tabular} \\ \hline A girl is brushing her hair. & There is no girl brushing her hair. & Contradiction & 4.5 \\ A chubby faced boy is not wearing sunglasses. & Contradiction & 3.9 \\ The dog is on a leash and is walking out of the water. & Contradiction & 3.5 \\ A black sheep is standing far from three white dogs. & Contradiction & 3.5 \\ A man dressed in black and white is holding up & A man dressed in black and white is dropping & Contradiction & 4 \\ \hline \hline \end{tabular}
\end{table} 
Table 6: Example sentence pairs of the SICK [64] test-split data with the contradiction relation, which have high human relatedness scores.