# Short-Text Semantic Similarity (STSS): Techniques, Challenges and Future Perspectives
## 4 What Are the Existing Deep Learning Techniques Used in Short-Text Semantic Similarity (STSS)?
### ELMO Model for Contextual Word Embeddings (CWE)  
The ELMO (embeddings from language model) was introduced by [PERSON] et al. [77] for the purpose of extracting contextual and morphological representations. The embeddings of each word can change based on their syntactical and contextual structures. [PERSON] et al. [78] proposed the ELMO model for text summarizers. The model further represents the documents into a vector that includes syntax and the contextual dependent information of words. The model further used Kaggle datasets, and experimentation was completed by using Python programming. [PERSON] et al. [79] used the ELMO model to gather information from patients with schizophrenia. The authors used small statements, and, later, they extracted the features which included predicate information of patients. The model achieved 80% accuracy by using cross-validation. [PERSON] et al. [80] reported that it is difficult to extract the contextual information through the ELMO model for those words that contain very little information, such as'such as, somehow, some, well, etc'. [PERSON] et al. [81] use the LSTM as an input to name entity recognition with the ELMO model. They used a pre-trained version of the ELMO model on the corpus of medicine. Their results showed that the ELMO model is good for domain-specific tasks. Furthermore, the studies [82; 83; 84; 85; 86] suggest using convolutional layers with the ELMO model to improve its performance for extracting text features. These layers used max-pooling to represent the fixed length of the entire word. Similar to the convolutional layers, the ELMO model can utilize the layers of Fasttext to improve its performance in character-level representations. To improve the process for the input layers, these representations employ a two-layer network [87]. Additionally, the Bi-LSTM network layers are used by the ELMO language model to understand the previous word. In relation to this, [PERSON] et al. [88] used the ELMO model for ARABIC short Q2Q similarities. The representation of the model detects the high contextual embeddings for similarity. The model used character-level embeddings to overcome the morphological nature of answers. The model scored 71.30% for the Pearson correlation coefficient. The ELMO model uses a diverse approach to predict model similarity. The following are the three categories that the model utilizes for the contextual word and sentence representations [89; 90; 88]:  
Word Embeddings: Inside these embeddings, tokens usually convert into embeddings, and words from sentences represent as vectors. Sentence Representations: These embeddings are used to generate the Sentence2 Vec and build on top of the word embeddings. Prediction layer: The ELMO model uses this layer to predict the similarity. Traditional methods, including Bag-of-words [91], Sentence2 Vec [92], and Word2 Vec [93] models, developed the dictionary for source text. The available dictionary includes many words that are used as a vocabulary for the text. These models organize the text and developed the vectors for each sentence to capture the similarity. Furthermore, these models use documents, long, and short sentences for similarity tasks and are often known as context-independent models. The context-dependent model, such as ELMO, uses characters to understand the hidden meaning. In order to do this, [PERSON] et al. [94] state that, to extract the context meaning of words, the pertained version of the ELMO model is the most suitable. It employs the token embeddings with position embeddings and sends them to the transformer encoder. These encoders follow the self-attention mechanism, which uses feed word and pooling layers. As a result, these layers generate condensed vector representations. However, [PERSON] et al. [95] dictated that the result of these layers fused to domain-specific neural architectures. This integration is not straightforward for the pre-trained ELMO model.