# Short-Text Semantic Similarity (STSS): Techniques, Challenges and Future Perspectives
## 4 What Are the Existing Deep Learning Techniques Used in Short-Text Semantic Similarity (STSS)?
### BERT Model for Contextual Word Embeddings (CWE)  
Word embeddings are a form of keyword representation. Many studies used to refer to the keyword or key phrase, which helps to identify the meaning of content from a document or sentences. Keywords refer to a unigram, while key phrase is an N-gram, which is usually concatenated with one or two sub-words. For example, the computer is a keyword, and the computer system is a key phrase [61, 62, 63]. BERT has shown many promising results for capturing contextual embeddings from long or short sentences. The model employed transformer architecture and the attention mechanism to extract the embeddings. In order to do this, [PERSON] et al. [64] proposed a self-attention mechanism in the BERT model to select the linguistic features and conduct various experiments to identify how these features co-related with one another. Moreover, the study used the GLUE benchmark and SQuAd dataset for the feature selection task and improved the model by an absolute gain of 3.2%. [PERSON] et al. [65] proposed the KeyBERT model for the impact analysis with RAKE, YAKE, and TF-IDF approaches. The model produced contextual word embeddings with the authors' provided keywords. The average similarity rate of the proposed model was 51%, which is higher than other baseline methods. Moreover, [PERSON] et al. [66] proposed a multilayer attention BERT model to extract the contextual \begin{table}
\begin{tabular}{c c c c c c c} \hline \hline
**Study** & **Architecture** & **Trained** & **Category** & **Features** & **Dataset** & **Size** \\ \hline \multirow{4}{*}{[PERSON] et al. (2022)} & Stacked & \multirow{4}{*}{Grading} & Short answers & \multirow{4}{*}{Mohler} & 2273 \\  & transformer & Pre-trained & & & Short answers (SIM) & Mahler \& \\  & and & [PERSON] & & & SemEval 2013 & \& \\  & Bi-LSTM & & & & & 2100 \\ \hline [PERSON] et al. (2019) & Stacked & Pre-trained & Grading & Short answers & \multirow{2}{*}{Biology} & \multirow{2}{*}{-} \\  [41] & transformer & ml-BERT & & & & \\ \hline [PERSON] et al. (2019) & Stacked & Pre-trained & & & & \\  [42] & transformer & [PERSON] & & & & \\ \hline [PERSON] et al. & \multirow{4}{*}{Stacked} & Fine-tuned & \multirow{4}{*}{Grading} & Short answers & \multirow{2}{*}{SemEval 2013} & 2100 \\  (2020) & transformer & [PERSON], & & & & \\  & & [PERSON] & & & & \\ \hline \multirow{4}{*}{[PERSON] et al. (2020)} & Bi-LSTM & \multirow{4}{*}{Short sentences} & Contextual & \multirow{4}{*}{ATS and SNIPS} & \multirow{4}{*}{-} \\  [53] & Transformer & ELMO & & & & \\ \cline{1-1}  & & & & & \\ \cline{1-1}  & & & & & \\ \cline{1-1}  & & & & & \\ \hline \hline \end{tabular}
\end{table} 
Table 9: _Cont._embeddings of clinical data from the EHR system. They improved the model by employing an additional layer of the BiLSTM model. This resulted in an improved accuracy of 97.6%, which is higher than the fine-tuned BERT model. Furthermore, [PERSON] et al. [67] introduced a linguistic knowledge-enhanced graph transformer to extract the ambiguity from short chine sentences. They further proposed the BERT model, along with the graph transformer model, to extract Chinese characters. However, they conducted the experiments on the Chinese dataset BQ and LCQMC; the model showed 88.38% accuracy. [PERSON] et al. [68] proposed a BERT model with feature fusion techniques to identify the sarcastic contextual-based features. The features were found from e-commerce and social media sites, and, for experimentation, they used the Twitter benchmark and achieved 98.0% precision. With an account of this, [PERSON] et al. [69] used a pre-trained BERT model to identify the polysemy words that provide sense embeddings in their contextual space. The KNN and POS were further utilized to capture the grammatical structure of sentences. They used the SenseEval dataset, and the F1 results of the model showed a value of 83.32%. [PERSON] et al. [70] used the pre-trained BERT model for conversational topic classification. The model captured the salient features from the representations.