## Appendix A  
### Result of Using Text Augmentation to Prepare Positive Samples  
We study text augmentation techniques for generating positive samples for each anchor sentence. As mentioned in Section 5.2, we use word deletion, word synonym replacement, and paraphrasing. We consider four options including none (No data augmentation is used), word deletion, synonym replacement using WordNet, and paraphrasing using Text-to-Text Transfer Transformer (T5) generative language model [68]. We briefly explain these four option settings in the following:  
* **None:** No data augmentation is used. We simply copy each anchor sentence several times as its positive samples (our implementation). * **Random Word Deletion (RD):** We randomly delete 10% of each entailment's words. * **Synonym Replacement (SR):** We randomly substitute 20% of each entailment's words with their synonyms using WordNet [89]. * **Paraphrasing (PP):** We paraphrase almost 50% of entailments. We use NLPAug ([[https://github.com/makcedward/nlpaug](https://github.com/makcedward/nlpaug)]([https://github.com/makcedward/nlpaug](https://github.com/makcedward/nlpaug)), accessed on 1 August 2022) library [90] for word deletion and synonym replacement. NLPAug is a straightforward data-augmentation library which implements 15 methods for text data augmentation based on word, sentence, and character. Additionally, we use Parrot ([[https://github.com/[PERSON]/Parrot_Paraphraser](https://github.com/[PERSON]/Parrot_Paraphraser)]([https://github.com/[PERSON]/Parrot_Paraphraser](https://github.com/[PERSON]/Parrot_Paraphraser)), accessed on 1 August 2022) paraphraser library [91] for paraphrasing. Parrot is a T5 model fine-tuned on some of paraphrase datasets such as MRPC Paraphrase ([[https://www.microsoft.com/en-us/download/details.aspx?id=52398](https://www.microsoft.com/en-us/download/details.aspx?id=52398)]([https://www.microsoft.com/en-us/download/details.aspx?id=52398](https://www.microsoft.com/en-us/download/details.aspx?id=52398)), accessed on 1 August 2022) [88], Google PAWS ([[https://github.com/google-research-datasets/paws](https://github.com/google-research-datasets/paws)]([https://github.com/google-research-datasets/paws](https://github.com/google-research-datasets/paws)), accessed on 1 August 2022) [92], ParaNMT ([[https://github.com/juvie/para-nmt-50m#readme](https://github.com/juvie/para-nmt-50m#readme)]([https://github.com/juvie/para-nmt-50m#readme](https://github.com/juvie/para-nmt-50m#readme)), accessed on 1 August 2022) [93], Quora Question Pairs ([[https://quoradata.quora.com/First-Quora-Dataset-Release-Question-Pairs](https://quoradata.quora.com/First-Quora-Dataset-Release-Question-Pairs)]([https://quoradata.quora.com/First-Quora-Dataset-Release-Question-Pairs](https://quoradata.quora.com/First-Quora-Dataset-Release-Question-Pairs)), accessed on 1 August 2022), SNIPS Commands ([[https://github.com/sonos/nlu-benchmark/tree/master/2017-06-custom-intent-engines](https://github.com/sonos/nlu-benchmark/tree/master/2017-06-custom-intent-engines)]([https://github.com/sonos/nlu-benchmark/tree/master/2017-06-custom-intent-engines](https://github.com/sonos/nlu-benchmark/tree/master/2017-06-custom-intent-engines)), accessed on 1 August 2022) [94]. As shown in Table 1, none of these augmentation produces acceptable average on all STS tasks and so, cannot outperform the None option (SupMPN-BERTbase-None). SupMPN-BERTbase-SR only works well on STS12 and STS15, SupMPN-BERTbase-RD only works well on STS16, and SupMPN-BERTbase-PP only works well on SICK-R. We think the underlying reason depends on the nature of STS as each task includes several sub-tasks from various sources with different properties. STS is related to both textual entailment (TE) and paraphrasing, but it differs in several ways. In STS, graded equivalence between a pair of texts is bidirectional but in TE the equivalence is directional, e.g., a car is a vehicle, but a vehicle is not necessarily a car.