## 5 Experiments
### Baseline and Previous Supervised and Unsupervised Models for Comparison  
In our experiments, we compare our proposed SupMPN model to the previous state-of-the-art SimCSE and other supervised and unsupervised sentence encoder models. We list them separately as supervised and unsupervised models in Table 2. It is worthwhile emphasizing that our model (SupMPN) and state-of-the-art SimCSE are trained on the same data (entailments and contradictions from SNLI and MNLI) and other models are trained on various datasets with different sizes. We compared them in terms of type and size of training data in Appendix A.2. The non-BERT models are GloVe [46], InferSent [49], and USE [51] which we briefly explained in Section 2. We consider SBERT [52] as a baseline model. We introduce other BERT-based models briefly in the following:  
**BERT-flow:** Ref. [15] claim that sentence embeddings generated by BERT depend on the frequency of words that lead to the anisotropy of sentence representations. Therefore, to solve this problem, they feed embedding vectors to a flow network [71] and try fitting them to a standard Gaussian distribution. **BERT-whitening:** Ref. [72] proposed this model which, as with BERT-flow [15], tries to solve the problem of the anisotropy of the sentence representations of BERT using the whitening operation in machine learning. In addition, by applying the whitening operation, the dimensions of the sentence representation and the storage cost are reduced. Consequently, the model retrieval speed is accelerated. **CLEAR:** Ref. [33] proposed this model for fine-tuning BERT using combined loss as a combination of MLM (mask language model) and CL (contrastive loss) objectives, which rely on text augmentation techniques. **CT-BERT:** Ref. [57] proposed this model which uses two independent encoders and then tries to maximize/minimize the dot product between two identical/different sentences. **ConSERT:** Ref. [16] proposed this model to solve the collapse problem of BERT by generating different form of input samples using data augmentation and a contrastive objective on top of BERT-encoder. **DeCLUTR:** Ref. [35] proposed this model which learns via different spans from the same document as positive samples using contrastive learning. **IS-BERT:** Ref. [56] proposed this model which uses a contrastive objective based on mutual information maximization mechanism between the global and local sentence representation. **Mirror-BERT:** Ref. [36] proposed this model which uses self-supervised contrastive learning based on random span masking as data augmentation for the input space. **SBERT-base-nil-v2**: This checkpoint is a pre-trained model from the Sentence-Transformer packages ([[https://huggingface.co/kwang2049/SBERT-base-nli-v2](https://huggingface.co/kwang2049/SBERT-base-nli-v2)]([https://huggingface.co/kwang2049/SBERT-base-nli-v2](https://huggingface.co/kwang2049/SBERT-base-nli-v2)), accessed on 1 August 2022). SBERT-base-nli-v2 was trained on SNLI and MNLI data using the Multiple-Negative Ranking Loss (MNRL) in [73]. **SG-BERT:** Ref. [34] proposed this model which is a self-supervised contrastive learning method using the redesign of NT-Xent objective with self-guidance. They exploit similarities between different sentence embeddings made by BERT itself. **SimCSE:** Ref. [37] proposed SimCSE\({}_{\text{unsup}}\) and SimCSE\({}_{\text{sup}}\) models. SimCSE\({}_{\text{unsup}}\) is a self-supervised contrastive learning that takes an input sentence and predicts itself using the dropout noise. SimCSE\({}_{\text{sup}}\) uses entailment and contradiction pairs from NLI datasets and extends self-supervised to supervised contrastive learning. Additionally, they apply an auxiliary Masked Language Modeling (MLM) objective to its models and stated that adding MLM boosts performance on transfer-learning tasks (not on STS tasks). **TSDAE:** Ref. [73] proposed this model, which is an unsupervised method based on pre-trained transformers, and sequential denoising autoencoder.