## 2 Related Works  
The main and certainly the most common challenge for all NLP tasks, is the way to represent textual data as input. The traditional embedding techniques such as word2 vec [45] and GloVe [46] only work on the word level. After this, the authors in [47] introduced a sentence-embedding model, called Skip-thought, which is an extension of the word2 vec skip-gram method to apply sentences instead of words, i.e., instead of predicting the target word using surrounding words, Skip-thought predicts the target sentence by surrounding sentences. Later, ref. [48] presented fastText, which had the same goal as word2 vec with a small difference: fastText, unlike word2 vec, which treats each word in corpus such as an atomic entity, uses character n-grams and so can process out-of-vocabulary words. Although word2 vec, GloVe, and fastText encode words into a vector representation, there is still a need to represent whole sentences so that a computer can easily understand their semantic meanings in the entire text. Another simple way to create a single fixed-size sentence vector is averaging word vectors (ANV). This solution considers neither the interaction between words in a sentence, nor the order of words, while a good sentence encoder is expected to be able to create deep contextualized word representations that can handle polysemy (words with multiple meanings in different contexts). Recently, deep contextualized word-embedding models such as Facebook's InferSent [49], AllenAI's ELMo [50], Google's BERT [14], and USE [51] have been proposed and received significant attention. InferSent is a Siamese network that uses Bi-LSTM, a deep neural network with memory to remember the whole sentence to encode. It is a supervised model trained on the Natural Language Inference (NLI) dataset. ELMo uses the Bi-LSTM network trained on a huge corpus including billions of words. USE have two encoder variations, e.g., one is the transformers encoder and the other is the Deep Averaging Network (DAN) trained on supervised data (e.g., NLI), and unsupervised data which are drawn from a variety of web sources such as Wikipedia, web news, question-answer pages, and discussion forums. Finally, BERT, the contextualized word-embedding model, was proposed. BERT achieves the state-of-the-art performance in some of NLP tasks. It is a giant deep neural network with millions of parameters and uses a cross-encoder architecture. The cross-encoder architecture of BERT requires that the two sentences be passed to the network simultaneously which lead to greater computational overheads [52; 53]. Therefore, this feature makes training BERT from scratch very time-consuming for sentence-pair taskssuch as semantic similarity search. On the other hand, BERT uses Mask Language Model (MLM) and Next Sentence Prediction (NSP) objectives. The MLM objective enables BERT to learn word-level or phrase-level semantic relationships [54] and the NSP objective enables BERT to learn longer-term dependencies across sentences. With the NSP objective, BERT only answers to this question: given sentence A and B, is B the next sentence for A? Meanwhile, the STS task wants to answer this question: given sentence A and B, are A and B similar or not? As an effective solution to solve these problems, Sentence-BERT (SBERT), a modification of the BERT using Siamese network, has been proposed by [52]. SBERT uses bi-encoder architecture through training a Siamese network on top of the BERT model and pays attention to the sentence-level semantic relation in its training objective. Although SBERT increases BERT performance on STS tasks through training with human-labeled NLI datasets, it still fails to produce good sentence embeddings. Recently, several models have been proposed to enhance BERT and/or RoBERTa [55] on Semantic Textual Similarity (STS) using contrastive learning. In these models, contrastive learning is used to fine-tune BERT and improve the quality of sentence-embedding space [16; 33; 34; 35; 36; 57].