## 2 Related Works  
The traditional embedding models such as Word2 Vec [17], GloVe [18], and FastText [19] only work on the word level. After that, the same idea was extended to learn sentence embeddings instead of learning word embeddings. Sentence embedding models can be divided into those that produce non-contextualized embeddings, and those that produce contextualized embeddings [20]. Early sentence embedding models such as Doc2 Vec [21], Skip-thought [22], FastSent [23], Sent2 Vec [24], and Quick-thought [25] are non-contextualized embeddings and have been relatively successful in some of NLP tasks. However, these embeddings provide an exact meaning to words, which is a major downside of them, as the meaning of words changes based on context. For example, in the following three sentences, the meaning of the word \"date\" can be changed based on context:  
* Her favorite to eat is a date. * They went on a date tonight. * What is your date of birth? Thus, learning high-quality sentence representations can still be challenging, because a desirable sentence embedding model is expected to be able to model complex features of word use (e.g., syntax and semantics) and to use linguistic contexts in a way that can handle polysemy [26]. Recently, deep contextualized Sentence-embedding models such as Facebook's InferSent [27], AllenAI's ELMo [26], USE [28], and Google's BERT [1] have been proposed, which are considered as important milestone achievements in word and sentence representation. These pre-trained machine learning models can encode a sentence into deep contextualized embeddings that depend on its intra-sentential context [20]. Among contextualized sentence embedding models, BERT has established itself as the most effective NLP model, performing excellently in many NLP applications with appropriate fine-tuning for particular downstream tasks [29]. BERT is a contextualized language model based on the Transformer architecture developed by researchers at Google AI. The Transformer-based architecture of BERT uses the amazing attention mechanism that learns contextual relationships between words in a sequence of text. In fact, BERT solves the long-term dependencies problem of Recurrent Neural Networks (RNNs) by attention mechanism, which is a technique for paying attention to specific words. Recent works have shown that fine-tuning BERT has been very successful in many NLP tasks, including sentiment analysis [29], text classification [30], named-entity recognition (NER) [31], intent recognition [32], etc. For instance, [PERSON] et al. in Ref. [31] attemptedto solve the nested-NER problem using transfer learning through fine-tuning BERT. They performed fine-tuning of various pre-trained BERT models on the datasets in which the nested labels are connected by flattening (e.g., Tottenham: organization + location). Similarly, in Ref. [32], [PERSON] et al. fine-tuned BERT for intent recognition using vocabulary extension. They showed that adding adequate domain specific words to BERT's original tokenizer vocabulary can improve performance. The biggest advantage of BERT is that it generates \"contextualized\" word embeddings, which means that it can assign each word a representation based on its context. The biggest downside of BERT is that it is slow to train as it has millions of parameters and needs a prohibitively large dataset in order to train to reasonable accuracy. In addition, BERT is based on a cross-encoder architecture, which makes it too slow for sentence-pair tasks such as clustering, since both sentences must be fed into the network, which requires an enormous amount of computation. For example, clustering 10,000 sentences requires approximately 65 h of training. Ref. [33] have proposed a Bi-encoder BERT model, which is called Sentence-BERT (SBERT). SBERT aims to overcome this challenge through a siamese network, which creates sentence embeddings for each sentence and can then compare them using a cosine-similarity.