# Short-Text Semantic Similarity (STSS): Techniques, Challenges and Future Perspectives
## 4 What Are the Existing Deep Learning Techniques Used in Short-Text Semantic Similarity (STSS)?
### Transformer Learning Models  
Currently, in deep learning techniques, the bi-directional encoder representation from the transformer (BERT) model displayed excellent achievement on many natural language processing (NLP) tasks [30]. In order to do this, [PERSON] et al. [31] proposed the BAS (BERT-answer selection model) to understand the syntactic and semantic information from short question answers. The study further used CNN, RNN, and BOW as classifiers to detect the correct answers. [PERSON] et al. [32] proposed a BERT model to automatically grade short answers. The model was trained by using the Indonesian language. However, they use Cohen's Kappa to check the inter-rater reliability among student answers. Moreover, [PERSON] et al. also [33] also proposed the BERT model to grade the student's short answers. He further trained the model on a short answer scoring V2.0 dataset and used the regression task function to check the linearity between answers. Furthermore, [PERSON] [34] presented the BERT model for the Arabic short text classification. The study synthesized the different Arabic BERT versions for text classification. Moreover, the study provides a comparison between ENGLISH and ARABIC text classification models. [PERSON] et al. [35] used domain-independent subjects, such as biology and geography, to grade the Indonesian short answers. Furthermore, they implemented the BERT model to detect the word embeddings from sentences and analyzed the contextual information. In relation to this, [PERSON] et al. [36] mentioned the difference between transformer learning models BERT, GPT, GPT2, and ELMO. BERT uses a transformer mechanism and takes the benefits of both ELMO and GPT to extract the contextual embeddings bi-directionally. It is trained on BookCorpus and Wikipedia datasets; the size of the dataset is 800 M and 2500 M words. Moreover, GPT and GPT2 also use a stacked transformer. GPT comprises the 800 M dataset word size and is pre-trained on BookCorpus. In contrast, GPT2 is pre-trained on WebText and utilizes the stacked transformer. In order to compare ELMO to these three transformers (BERT, GPT, and GPT2), ELMO uses the bi-LSTM architecture and a benchmark dataset with a word count of 1 billion in size. Furthermore, [PERSON] et al. [37] proposed a BERT regressor model to grade short answers. The model further used domain-specific datasets forkey-value pairs. Since automatic short answer grading (ASAG) gained a lot of attention in recent years, a number of methods have been proposed in this domain. In order to do this, [PERSON] et al. [38] also proposed a BERT-based framework to grade short answers. The method further used CNN, capsule, and triple hot loss strategy to encode the short key sentences. Moreover, [PERSON] et al. [39] researched 35 ASAG systems from 1996 to 2015 and classified them into five categories. The authors further presented various limitations inside the ASAG systems. However, [PERSON] et al. [40] used lexical semantic similarity to grade short answers. They further explained that deep learning techniques are more \begin{table}
\begin{tabular}{c c c c} \hline \hline
**Ref.** & **Model** & **Category** & **Feature Set** \\ \hline [PERSON] et al. (2018) [23] & RNN & Short text & Semantic matching \\ \hline [PERSON] et al. (2018) [24] & RNN \& LSTM & Short text & SIM \\ \hline [PERSON] (2017) [25] & RNN \& CNN & Short text & Gender classification SIM \\ \hline [PERSON] et al. (2019) [26] & RNN \& CNN & Short text & Character level representations \\ \hline [PERSON] et al. (2019) [27] & ABRNN & Short tweet & Asthma syndrome SIM \\ \hline [PERSON] et al. (2022) [28] & RNN & Short text & SIM \\ \hline [PERSON] et al. (2017) [29] & RNN & Short text & Vector representations \\ \hline \hline \end{tabular}
\end{table} 
Table 8: Summary of recurrent neural networks (RNN). useful to grade short answers. [PERSON] et al. [41] proposed the ml-BERT model to grade short answers. They targeted unlabeled data on the domain-specific dataset. They further used meta-learning to initialize the model parameters.