# Short-Text Semantic Similarity (STSS): Techniques, Challenges and Future Perspectives
## 4 What Are the Existing Deep Learning Techniques Used in Short-Text Semantic Similarity (STSS)?
### Transformer Learning Models
[PERSON] et al. [42] improved the BERT model in the ASAGs domain. They used fine-tuning techniques to use multi-domain resources. [PERSON] et al. [43] used a combined approach and added a multilayer recurrent unit, along with the BERT model, to develop the classification model. Moreover, [PERSON] et al. [44] compared the BERT model with ALBERT and Roberta and used the pretraining method to grade short answers. They also used optimization techniques to improve the BERT model performance. Various variants of BERT, such as SBERT, DistilBERT, KeyBERT, and Roberta, actively work on short-text semantic similarity tasks, which include, short text classification, short keyword extraction, sequence classification, semantic matching, and so on [45; 46; 47; 48; 49; 50]. [PERSON] et al. [51] used the BERT model for the context-sensitive representations. They further utilized the GCN model for the classification of short text. [PERSON] et al. [52] surveyed transformer models for short text similarity, and they dictated that the BERT model has the ability to understand the contextual meaning of words inside the sentences, and, based on context, can extract the similarity between two tasks. They further applied semantic matching to the short text. Moreover, [PERSON] et al. [53] utilized ELMO embeddings to understand the contextual embeddings from a short text. They further used the BERT model architecture to understand the structure of sentences on ATS and SNIPS datasets. Moreover, [PERSON] et al. [54] utilized the ELMO model to understand the level of sentences through their context. In the study, the ELMO model adopted the architecture of the Bi-LSTM model for entire sentences and mapped the sentences into the sequence of vectors. Further, they highlighted various entities from the text that represented conceptual embeddings. In the domain of ASAGs, the short text suffers from task-specific architectures, and extracting the data from multiple domains was difficult to encode. The NLP community developed a number of pre-trained and finetuned models on domain-specific and domain-independent datasets. These models can be transformed and utilized for many NLP tasks. Recently, the BERT model outperformed other pre-trained algorithms. It can learn the context of any sentence, left-to-right or right-to-left, simultaneously [55]. Moreover, various other models, such as BioBERT for biomedical tasks and SciBERT for science domain applications, have also been introduced to understand domain-specific language and tasks [56]. The following Table 9 depicts the summary of transformer learning models that exploit the similarity in the domain of short text applications. \begin{table}
\begin{tabular}{c c c c c c c} \hline \hline
**Study** & **Architecture** & **Trained** & **Category** & **Features** & **Dataset** & **Size** \\ \hline [PERSON] et al. & Stacked transformer & Fine-tuned (BERT) & Questions and answers & Short answers (SIM) & WikiQA & 2351 QA \\ (2019) & with attention & Fine-tuned (BERT) & Questions and answers & Short answers (SIM) & & QABank & 100 QA \\ (2021) & Stacked transformer & Pre-trained (BERT) & Questions and answers & Short answers (SIM) & & 2440 QA \\ (2020) & Stacked transformer & Pre-trained (BERT) & Questions and answers & Short answers (SIM) & & 2440 QA \\ (2020) & Stacked transformer & Fine-tuned (BERT) & Questions and answers & Short answers (annotated) & & 7605 \\ (2021) & Stacked transformer & Fine-tuned (BERT) & Gradients & Short answers  
Which Existing Deep Learning Techniques Are Most Appropriate for Generating High-Level Contextual Representations from Sentences in Order to Improve the Similarity? To answer this question, we have included approx. 76 research articles to review various techniques for understanding contextual information. Numerous machine learning algorithms in deep learning modified the representations of word vectors. These representations include many NLP applications, such as question-answering systems (QAS), sentiment analysis, textual entailment, and named entity recognition [57, 58].