## 5 Experiments
### First Experiment: Evaluation on STS Tasks  
We evaluate SupMPN on seven standard Semantic Textual Similarity (STS) tasks: STS 2012-2016 [74; 75; 76; 77; 78], STS Benchmark [79] and SICK-Relatedness [80]. These datasets provide gold labels between 0 and 5 on the semantic relatedness of sentence pairs. The Spearman's rank correlation between the cosine-similarity of the sentence embeddings and the gold labels are computed.  
For evaluation on the STS tasks, we use a modified version of the SentEval toolkit by [37]. SentEval [81] is a popular library for evaluating the quality of sentence embeddings ([[https://github.com/facebookresearch/SentEval](https://github.com/facebookresearch/SentEval)]([https://github.com/facebookresearch/SentEval](https://github.com/facebookresearch/SentEval)), accessed on 1 August 2022). It covers various tasks including binary and multi-class classification, Natural Language Inference and sentence similarity. Ref. [37] reported three differences in STS evaluation settings done by the SentEval toolkit. These differences are using additional regressors for training frozen sentence embedding on the STS-B [79] and SICK-R datasets [80], reported metrics that can be [PERSON]'s or [PERSON]'s correlation coefficient, and method of result aggregations. As has been argued in [37; 52], the [PERSON]'s correlation, which measures the ranking instead of the actual scores, it is appropriate for evaluation of sentence embeddings. We use [PERSON]'s rank correlation in our experiments. For several STS subsets each year, there are three different options for collecting results: all, mean, and wmean. In the \"all\" settings, all the topics are concatenated and then the overall Spearman's correlation is reported. In \"mean\" setting, results for the different subsets are calculated separately and then their simple average is reported. In the \"wmean\" setting, which is similar to the \"mean\" setting, the weighted average with subset sizes is reported.  
Since most papers do not state the method they take, ref. [37] reproduced the results of SBERT [52], BERT-flow [15] and BERT-whitening [72] and compared them to their original results. They reported two different settings, \"all\" and \"wmean\". Ref. [52] took the \"all\" setting, and [15; 72] took the \"wmean\" settings. Since SBERT usually is considered as the baseline in the literature, ref. [37] took the \"all\" setting and made two changes on the original SentEval: (1) added the \"all\" setting to all STS tasks, and (2) changed STS-B and SICK-R to not use an additional regressor. Ref. [37] requested researchers unify the so-called settings in evaluating sentence embeddings in their future works and released modification of SentEval in their GitHub repository ([[https://github.com/princeton-nlp/SimCSE](https://github.com/princeton-nlp/SimCSE)]([https://github.com/princeton-nlp/SimCSE](https://github.com/princeton-nlp/SimCSE)), accessed on 1 August 2022). Hence, we take their modified SentEval and use it in our first evaluation.  
**Results:** As shown in Table 2, the SupMPN model outperforms the state-of-the-art SimCSE model and all other previous unsupervised and supervised models on STS tasks. SupMPN achieves averages of 82.07% and 83.15% Spearman's correlation on STS tasks using BERT-base and BERT-large, respectively. The corresponding improvements are 7.18 points for SupMPN\({}_{\text{base}}\) and 8.26 points for SupMPN\({}_{\text{large}}\) on STS tasks, compared to baseline SBERT.