<figure><img src="image_1.png" /></figure>

ifip

International Federation for Information Processing

This document is the original author manuscript of a paper submitted to an IFIP
conference proceedings or other IFIP publication by Springer Nature. As such, there
may be some differences in the official published version of the paper. Such
differences, if any, are usually due to reformatting during preparation for publication or
minor corrections made by the author(s) during final proofreading of the publication
manuscript.
<figure><img src="image_2.png" /></figure>

# Using Siamese BiLSTM Models for Identifying
Text Semantic Similarity

Georgios Fradelos<fnref n="1" />, Isidoros Perikos<fnref n="1" />,<fnref n="2" />, and Ioannis Hatzilygeroudis<fnref n="1" />(✉)

**Abstract.** The field of Natural Language Processing (NLP) has flourished during the past decades in computer science, and that is largely due to the exponential growth of internet applications, like search engines, social network platforms, chatbots and Internet of Things (IoT). On the other hand, the robotics and human computer interaction fields have been largely connected to NLP development, by exploring ways of human-robot or human-computer communication in natural language. In this work, we deal with the problem of semantic similarity between text passages, which is one of the problems faced in many NLP applications, like human-computer/robot communication through natural language text. More specifically, we developed three deep learning models to face the problem: two variations of the Siamese BiLSTM model and a variation of the Simple BiLSTM model. We used two different techniques of word embeddings, (a) classic token-to-vec embedding using GloVe, and (b) one implementing the encoder part of the BERT model. Finally, we train and compare each model in terms of performance, through experimental studies on two datasets, MRPC (MSRP) and Quora, and we draw conclusions about the advantages and disadvantages of each one of them. Siamese BERT-BiLSTM model achieves accuracy 83,03% on the Quora dataset, which is comparable to the state of the art.

**Keywords:** natural language processing · semantic similarity · siamese neural networks · transformers · BERT

## 1 Introduction

Determining the semantic relationship between texts is one of the most complex and challenging tasks in natural language processing. Social networks, search engines, recommendation systems and plagiarism recognition programs are just a few of the tools that base their operation on recognizing semantic similarity. On the other hand, semantic similarity plays an important role in the field human-machine interaction, where applications of text processing in human-robot or human-computer interaction are necessary [1, 2].

<footnote n="1">¹ Computer Engineering and Informatics Department, University of Patras,
26504 Patras, Greece</footnote>

<footnote>up1047143@upnet.gr, {perikos,ihatz}@ceid.upatras.gr</footnote>

<footnote n="2">² Computer Technology Institute and Press Diophantus, 26504 Patras, Greece</footnote>
The main objectives of this work are the study of the techniques proposed in the literature to approach the issue of semantic similarity, research of modern technological tools appropriate for this issue, and finally the practical engagement with the issue through the development of linguistic models capable of recognizing the semantic similarity between texts. In this paper, we propose three neural network models that we train with various datasets to capture the semantic similarity between two short passages of text. We experiment with both Siamese and non-Siamese architectures and we explore the text semantic similarity task both by measuring the similarity between two texts and by capturing the inference between them through classification.

We developed two Siamese architectures that differ in the embedding mechanism that they use. The first and most simple model makes use of an embedding layer that maps the tokens of the textual data to GloVe [3] word embeddings. These sequences are given to a Siamese BiLSTM network. The second Siamese architecture utilizes a BERT encoder [4] instead of a GloVe embedding layer. The encoder takes text as input and produces sequences of vectors. These sequences are then fed to a Siamese BiLSTM network. Both Siamese models consist of the same BiLSTM network, and they only differ in the embedding mechanism. The hidden vectors of the top BiLSTM layers are compared by measuring the Manhattan distance between them to calculate their similarity. The last model we developed is similar to the Siamese-BERT model, although it is a non-Siamese architecture, and it captures natural language inference via 3-way classification. The Siamese architecture with GloVe embeddings achieved an impressive 80.5% validation accuracy on Quora Duplicate Questions Dataset, however, this is a relatively static implementation with restricted practical flexibility. The best result yielded from the non-Siamese architecture using the largest BERT model that we managed to train, due to limited resources, was 87.44% validation accuracy on the SNLI Corpus. The models that use BERT, although demanding in memory and computational resources, are very flexible to use with great development dynamics. The main contributions of this work are a) an extensive literature review in the field of text semantic similarity and b) experimental results for DL models that haven't been used for this task before.

The paper is structured as follows. Section 2 presents related work. Section 3 presents the architectures of the three deep learning models used in the experiments. Section 4 deals with the experimental studies and their results, whereas Sect. 5 concludes the paper.

## 2 Related Work

In the literature, many works deal with the problem of recognizing and estimating the semantic similarity of texts. In the work presented in [5], Socher et al. use auto-encoders to model representations of local phrases in sentences and then calculate the similarity values of the phrases from the two sentences as features for binary sorting. A model is presented that incorporates the similarities between both single-word characteristics and multi-word phrases, which are extracted from the nodes of the analysis trees. The model is based on two elements. The first is a retrospective neural network (recursive autoencoder/RAE) that learns attribute representations for each node in the tree, the Euclidian distance of which fills a similarity matrix. However, simply extracting aggregate statistics from this table, such as the mean distance or a distance histogram, cannot accurately
capture the overall structure of the similarity of the sentences. Therefore, in order to preserve as much of the total information in this comparison as possible, the second part element of the model is then introduced, which is a pooling layer that produces a constant-sized representation. A softmax classifier is then used to categorize sentences as paraphrases or not. RAE architecture with dynamic pooling achieves state-of-the-art performance with 76.8% accuracy and 83.6% F1 score in MSRP.

Hu et al. [6] proposed neural network models (ARC-I), which adopt a convolutional architecture in order to semantically match two sentences, including vision and speech. In order to explore the relationship between the representation of sentences and their matching, they design a new model that can accommodate both the hierarchical synthesis of sentences and the simple-to-comprehensive fusion of the associated patterns of data with the same convergent network architecture. The model is general, does not require prior knowledge of the natural language (e.g. analysis tree) and does not impose any restrictions on the respective tasks. The sentence modeling architecture takes as input the pre-trained word embeddings aligned sequentially and in the order in which the terms appear in the sentence and summarizes their meaning through convolution and pooling layers, until it reaches a fixed vector representation length in the final layer. The convolution in the first level works with sliding-windows, as well as the convulsions in the deeper layers. After each convolution, max-pooling is performed in each window of two units. The first architecture proposed in the article takes a conventional approach. It first finds the representation of each sentence and then compares the representation for the two sentences with a Multi-Layer Perceptron [7]. It is essentially a Siamese architecture. The downside to ARC-I is general to Siamese architectures and lies in ignoring any semantic interaction between the sentences. An additional architecture is proposed, ARC-II, which aims to solve this problem. It has the property of letting two sentences "meet" before creating the individual high-level representations for each one, while maintaining the space for the individual development of the abstraction of each sentence. Both architectures were tested experimentally in paraphrase recognition, sentence completion, and tweet response matching. They achieved an accuracy of 0.696 (ARC-I) and 0.699 (ARC-I I) at Microsoft Research Paraphrase Corpus.

Authors in [8] proposed a Bi-CNN-MI (multigranular interaction features). The model “learns” the representation of each sentence separately through a Siamese deep convolutional neural network (“Bi” denotes a Siamese CNN layout rather than a bidirectional CNN) and computes similarity tables between two sentences at four levels. Each of the levels represents the similarity between unigrams, short n-grams and long n-grams of the sentences and lastly between the sentences themselves. The similarity tables are constructed by measuring the Euclidean distance of the corresponding hidden states of the network and are followed by a dynamic pooling layer. The result of the pooling layer is a vector which is then inserted into a logistic classifier to perform paraphrase detection. Microsoft Research Paraphrase Corpus was used for the experiments where the proposed model achieved 78% accuracy.

Authors in [9] propose a convolutional neural network model consisting of two main levels: modeling input propositions and measuring similarity. The first part learns vector representations of sentences to obtain information from different levels of abstraction, using multiple types of pooling (max, average, etc.), and different convergent filters.
The second part locally compares the representations of the first level using various similarity measurements achieving an accuracy of 78.6% at MRPC. Authors in [10] propose a model that combines CNN and LSTM networks to export three types of features, and then introduces them into a three-tier Multi-Layer Perceptron categorizer. The proposed model achieves an accuracy of 77.7% at MRPC.

Song et al. [11] propose an innovative RNN architecture, the fractional latent topic-based RNN model (FraLT-RNN) that largely retains the overall semantic information of the text. In FraLT-RNN, hierarchical attributes, i.e. word-level and subject-level attributes, as well as sequential word patterns, are integrated into the RNN for text representation via fractional latent topics. First, the fractional latent topic generator is used so that the model can learn the latent subjects according to the latent states of an RNN structure. An attention mechanism for the various topics is then designed to create an attentive vector at the topic level for each latent topic, which measures the perspective of the hidden topic and enhances the interactions between a pair of texts. Finally, hidden topics are checked based on this vector for text representation and similarity calculation. Given a text pair as X and Y, let x and y denote the embedded representations of the word x and y respectively, and assume that the sequential hidden state, as well as each hidden state, corresponds to a word in text X. Next, the text is represented by the method of observing hidden situations. Finally, the degree of similarity is calculated according to the two textual representations and a function of similarity, such as cosine similarity.

Authors in [12] study the subject of Natural Language Inference (NLI) citing a particular LSTM architecture. The model is based on the architecture proposed in [13] and is a neural network model with an observation mechanism for the NLI task but is based on a different idea. Instead of using actual sentence vectors to categorize them, a match-LSTM is developed which does word-to-word matching for each word in the sentence pair. The network can give more emphasis on more semantically important parts and ignore the less important ones. The model ‘remembers’ the important ‘matches’ which affect the final prediction. This architecture is called matching-LSTM or mLSTM. The SNLI Corpus is used for the tests, in which the proposed architecture achieves 86.1% accuracy of correct predictions in the testing set.

Authors in [14] propose a series of models for the representation of sentences encoded by GRUs (Gated Recurrent Units). The models are based on the decoder-decoder architecture, however the decoder is replaced by a categorizer that selects the desired sentence from a collection. The model achieves 76.9% accuracy.

Liu et al. [15] propose a Deep fusion LSTM (DF-LSTM) LSTM model with the aim of finding semantic similarity through the prism of textual entailment. The inference of text (or natural language inference), falls within the work of determining the semantic relationship between two sentences. A DF-LSTM consists of two interdependent LSTM networks, each of which models a sequence under the influence of the other. The output of the model is fed to an output level, at which the corresponding degree of similarity is calculated. In addition, researchers are using external memory to increase LSTMs' capacity. This model is compared to other neural networks of the LSTM architecture in two large datasets, one of which is the Stanford Natural Language Inference Corpus (SNLI). DF-LSTM outperformed other LSTMs by 2% in the SNLI Corpus, achieving 84.6% accuracy in the test data.
Vaswani et al. [16] propose an innovative model that follows the architecture of the transformer and is based on attention mechanisms in order to capture the semantic properties of words in vectors. The transformer consists of two parts, the “encoder” and the “decoder”. The encoder consists of multiple levels of “multi-head attention mechanisms” followed by a fully connected feed-forward neural network. The decoder is similar to the encoder with an additional multi-head attention level that captures the weights of observing the encoder output. Although this model was originally proposed for the work of machine translation, Devlin et al. [4] used it to create word vectors (BERT embeddings). The model is pre-trained on two tasks: Masked Language Model and Next Sentence Prediction. First, the authors randomly cover a certain percentage of terms (15%) in the corpus and train the linguistic representation model to predict them. To predict the next sentence, the model is trained to predict whether in a pair of sentences, the second one is the next of the first or not. The trained model is then fine-tuned for the individual downstream tasks. The authors report that the model gives state-of-the-art results in 11 works on natural language processing and achieves 89.3% accuracy in the MRPC dataset.

Zhang et al. [17] propose an innovative language model based on BERT which learns linguistic representations in fine-grained manner. The model consists of three parts. A semantic categorizer for data classification, a sequential encoder which uses a pre-trained language model to create vector representations of text input data and semantic tags, and a semantic integration element for combining the representation of a text with the semantic representation of the contexts and the acquisition of the common representation for later works. SemBERT has been evaluated on 11 reference datasets that include natural language inference, question answering, semantic similarity, and text classification. SemBERT achieves state-of-the-art performance at SNLI Corpus and improves semantic perception in GLUE Benchmark and SQuAD 2.0. Studies and analyzes confirm that the introduced explicit semantics is necessary to further improve performance and SemBERT functions effectively and efficiently as a single model of linguistic representation enriched with semantic information.

Parikh et al. [18] propose a simple neural network architecture for natural language inference. Their method utilizes attention mechanisms to break down the problem into separate sub-problems. Researchers take an approach in which they align pieces of the two texts with each other in order to draw semantic information from them. Having as input two propositions, each of which is represented by a vector of attributes, a soft alignment matrix is created primarily using observation mechanisms. This table is then used to break down the desired work into subproblems in order to address them separately. Finally, the results from the solution of the subproblems are combined to extract the final categorization. It achieved 86.8% accuracy in the Stanford Natural Language Inference dataset test data.

In [9] Radford et al. suggest a method that combines unsupervised pre-training and fine-tuning with supervision, for learning universal vector representations of sentences. The proposed model is based on the Transformer architecture [16] and can be used as a general model in various works in the field of natural language processing, through fine-tuning its parameters to the downstream task-specific data. Experimental results show
that the proposed method achieves state-of-the-art results by achieving 82% prediction accuracy in Microsoft Research Paraphrase Corpus.

# 3 Models and Their Architectures

## 3.1 Vector Space Representations

Through the study of the literature, we came to the development of neural network models, which we will train to recognize the conceptual relationship that characterizes two pieces of text. All language models developed in the context of our work are based on vector space. The models developed in this study represent the pairs of propositions with sequences of vectors, which pass as input to the various models that either measure the distance of these vectors - thus calculating the semantic relation of the corresponding propositions - or categorize the data according to the type of relationship that characterizes them (i.e. whether the two sentences have a common, conflicting, or unrelated meaning). In total, we have developed the following three models:

1. A Siamese BiLSTM model with a token-to-vec vector language representation mechanism using the GloVe collection.

2. A Siamese BiLSTM model integrating the BERT model for vector representation of language data.

The two (Siamese) models take as their inputs the pair of propositions to compare, represent them separately in vector sequences, and estimate their similarity by measuring the distance of the representations of the highest level of BiLSTM networks. The vector space model was applied to the experimental implementations of this study, according to which each word corresponds to a unique vector of size d, e.g. 'Tree': [0.326, 0.543, 0.013,...], which is made in such a way that it contains semantic information about the term. This vector is an instance of the d-dimensional vector space we use, and the d-dimensions of the space define the size of the vectors in it, e.g. if a vector space is d = 50 dimensional, each word corresponds to a vector of 50 elements. Thus, we can determine the distance between 2 vectors with different metrics (cosine similarity, Euclidian distance), and draw conclusions about how close or not are, to each other, the concepts of the corresponding words. We call vector space a mathematical structure consisting of a collection of elements called vectors. An example of a vector space is the three-dimensional space in which each of its elements is characterized by 3 values x, y and z. The number of dimensions of the space determines the size of the vectors that belong to it and in applications such as natural language processing we can define vector spaces with more than 3 dimensions e.g. d = 50.

## 3.2 Siamese BiLSTM Model

In the first and simplest implementation, a Siamese BiLSTM network was created that accepts proposal vectors as input using the GloVe collection. In the first stage of implementation, the lexical data is pre-processed. Specifically, the sentences are divided into the tokens from which the punctuation marks and infinitive parts of speech such as ‘and’,
‘or’, ‘the’ and so on are subtracted. Although these terms offer syntactic and semantic coherence in the sentence, they do not contain semantic information; for this reason we remove them to reduce the data volume. Then the suffixes of the inflected parts of speech such as ‘-ing’, ‘-ed’ and so on are removed, and the tenses of the verbs are replaced by the lemmas from which they came from, e.g. ‘Went’&#xF0E0;’go’. All these are data without semantic information and therefore we can safely remove them without losing vital semantics information. Finally, we save the processed tokens in a list. The next step is to create a dictionary with all the distinct terms of the processed data (let of number n) to create a matrix of size n × d, where d is the length of the word vectors. For each term of the dictionary, we find in the GloVe thesaurus the vector that corresponds to it and we add it to the register of vectors, so we end up with the final embedding matrix of our model. We end up at the output of the embedding layer taking the final sequences of vector representations of the sentences (sentence embedding sequences). Each sequence corresponds to a sentence, has length λ and each element of the sequence has length d = 300. These sequences are given as input to the two-way LSTM network that is trained on them. At each of the LSTM outputs, max pooling is applied to extract their basic characteristics, which pass to a Lambda layer which measures their distance and normalizes it with a sigmoid function within the space [0,1]. Thus, at the output of the network a prediction like ‘0.54’ is generated, which expresses the estimation of the degree of semantic similarity of the sentences.

## 3.3 Siamese BERT-BiLSTM Model

As a variation of the previously described model, in this model the embedding layer is replaced by the encoder part of BERT. Initially, the input data go through a custom data generator and in batches they pass as separate sequences to the Siamese BERT, which in turn represents them in the semantic vector space. The vector representations of the two sentences then pass as input to the Siamese BiLSTM network, which is trained on them. At each of the LSTM outputs, max pooling is applied to extract their basic characteristics. The two max-pooled vectors pass as input to a Lambda Layer which measures their distance and outputs the final prediction of the model.

# 4 Experimental Study

## 4.1 Datasets

The datasets that we used in this study to train and evaluate the proposed models are described in the following.

MRPC (or MSRP) is a dataset created by Microsoft and contains a total of 5801 pairs of sentences that have been collected from online news sources. Each pair of sentences is accompanied by a binary label, which categorizes them as paraphrases or not. ‘1’ defines the sentences that are a paraphrase of each other and therefore have a common semantic meaning, while ‘0’ means that those sentences are not a paraphrase of each other.

Quora is a page on the World Wide Web, where users quote questions of any kind that can be answered by other users of the page. In 2018, the Quora page decided to utilize
the volume of data it has collected over its years of operation and published a series of datasets oriented towards natural language processing tasks. One of these datasets is the Quora Duplicate Questions Dataset, which covers the task of semantic similarity of texts and consists of a total of 404,289 pairs of sentences/questions in English. All question pairs are accompanied by a label that indicates whether the two questions are duplicates or not (i.e. if they essentially ask the same thing). ‘0’ means pairs that are not duplicates, so they are not semantically similar, and ‘1’ means those pairs that are duplicates.

## 4.2 Experimental Setup

The Rectified-Adam optimizer, introduced in 2019 by Liu et al. in [21], was used to model the training. Rectified Adam (RAdam) is a variant of the classic Adam optimization algorithm used extensively in neural network training. The reason we used the RAdam instead of the classic Adam is that catastrophic forgetting was observed during the fine-tuning of the BERT models [22]. In the phenomenon of catastrophic forgetting the following paradox is observed: while the model, during the pre-training, is trained normally, ‘learns’ a percentage of information and achieves a score, e.g. valAccuracy: 0.65, afterwards, during fine-tuning, this percentage drops almost instantaneously and the model essentially ‘forgets’ the information it has learned and stops ‘learning’. After several tests the problem seems to disappear by replacing the Adam optimizer with the Rectified-Adam which ‘stabilizes’ the training process. Loss and accuracy metrics were used to evaluate the final performance of the models.

$$ \text{Loss: } \sum (\text{Targets} - \text{Predictions})^2 $$

$$ \text{Accuracy: } \frac{\text{True\_positive\_values}+\text{True\_negative\_values}}{(\text{True\_positive\_values}+\text{True\_negative\_values})+(\text{False\_positive\_values}+\text{False\_negative\_values})} $$

Accuracy shows the percentage of correct predictions (positive and negative) made by the model, while Loss measures the magnitude of the prediction error.

Each of the developed models was trained separately on the datasets mentioned above. The training process is as follows:

* Processing of the pair of sentences and their numerical coding (integer encoding).

* Representation of those sequences in vector representation of words (GloVe or BERT vector sequences).

* Learning the sequences by the neural network.

* Prediction generation (Distance measurement if it is a Siamese network, categorization if it is simple).

* Comparison of the prediction with the label/error calculation.

## 4.3 Results

In this subsection, we present the results of the experimental study. For the second Siamese model we implemented two variations. We present the performance results of our models on the above mentioned datasets (Table 1)
<table><caption>**Table 1.** Results for MRPC and Quora Datasets.</caption><thead><tr><th rowspan="2"></th><th colspan="3">MRPC</th><th colspan="3">Quora</th></tr><tr><th>Siamese<br>BiLSTM-GloVe</th><th>Siamese<br>BERT-BiLSTM<br>(L = 2, H = 256,<br>A = 4)</th><th>Siamese<br>BERT-BiLSTM<br>(L = 8, H = 128,<br>A = 2)</th><th>Siamese<br>BiLSTM-GloVe</th><th>Siamese<br>BERT-BiLSTM<br>(L = 2, H = 256,<br>A = 4)</th><th>Siamese<br>BERT-BiLSTM<br>(L = 8, H = 128,<br>A = 2)</th></tr></thead><tbody><tr><th>testAccuracy</th><td><strong>0.8055</strong></td><td>0.7856</td><td>0.7888</td><td>0.8103</td><td><strong>0.8303</strong></td><td>0.8227</td></tr><tr><th>testLoss</th><td>0.4318</td><td>0.4350</td><td>0.4216</td><td>0.4228</td><td>0.3956</td><td>0.4067</td></tr><tr><th>trainAccuracy</th><td>0.9118</td><td>0.8739</td><td>0.8493</td><td>0.9333</td><td>0.8023</td><td>0.7855</td></tr><tr><th>trainLoss</th><td>0.1932</td><td>0.2972</td><td>0.3356</td><td>0.1900</td><td>0.4175</td><td>0.4463</td></tr></tbody></table>

The best results were achieved by the Siamese BiLSTM-Glove on MRPC (MSRP) dataset, and by the Siamese BERT-BiLSTM (with L = 2, H = 256, A = 4) on the Quora dataset. We used exponential negative Manhattan distance to measure the distance between the two sentences in the Siamese models. The metric that achieved the best results is exponential negative Manhattan distance, and it also contributed to a more stable and predictable training process overall. In the following tables, we compare our results with the results of other attempts on the same datasets (Table 2).

<table><caption>**Table 2.** Comparative results – MRPC.</caption><thead><tr><th>Model</th><th>Accuracy</th></tr></thead><tbody><tr><td>MC-QT [23]</td><td>0.769</td></tr><tr><td>CNN-LSTM [10]</td><td>0.770</td></tr><tr><td>STN + Fr + De + NLI + L + STP + Par [24]</td><td>0.786</td></tr><tr><td>CNN [25]</td><td>0.8044</td></tr><tr><td><strong>Siamese BiLSTM-GloVe (our model)</strong></td><td><strong>0.8055</strong></td></tr><tr><td>FraLT-RNN [11]</td><td>0.812</td></tr><tr><td>Finetuned Transformer LM [19]</td><td>0.820</td></tr><tr><td>Transformer Mode [16]</td><td>0.893</td></tr><tr><td>Hybrid Siamese BiLSTM [29]</td><td>0.900</td></tr></tbody></table>

It seems that our models achieved results comparable to the state of the art, especially for the Quora dataset (Table 3).
<table><caption>**Table 3.** Comparative results - Quora Corpus.</caption><thead><tr><td>Model</td><td>Accuracy</td></tr></thead><tbody><tr><td>Siamese MALSTM [27]</td><td>0.7150</td></tr><tr><td>S-ARCNN [28]</td><td>0.8146</td></tr><tr><td>BERT [26]</td><td>0.8223</td></tr><tr><td><strong>Siamese BERT-BiLSTM (our model)</strong></td><td><strong>0.8303</strong></td></tr><tr><td>S-CNN [28]</td><td>0.8332</td></tr><tr><td>Paraphrase-MiniLM-L6-v2 + Siamese MaLSTM [30]</td><td>0.9000</td></tr></tbody></table>

# 5 Discussion and Conclusions

In this study we took a look at the complex topic of text semantic similarity. We developed two Siamese BiLSTM networks. The first Siamese BiLSTM model uses GloVe embedding vectors and achieved 80.5% accuracy on MRPC and 81% accuracy on the Quora Duplicate Questions dataset, both on ‘unknown’ data. The second Siamese model, instead of GloVe embeddings, utilizes the BERT encoder to produce the embedding sequences of the textual data. After fully training the model (pre-training and fine-tuning) it surpassed the performance of the first model achieving a maximum accuracy of 83% on the Quora dataset. The Siamese GloVe-BiLSTM model although small in size, achieved results that are surprisingly comparable with the other model, given that BERT is a really powerful language model responsible for state-of-the-art performances. The downside of the GloVe-BiLSTM model is that it is static, and incapable of training in multiple datasets over and over, in order to improve its performance. The model that uses BERT, on the other hand, can be trained in multiple datasets making it very flexible and easy to use. Although it is quite demanding in resources (both memory and computational power) due to its large size, and in order to achieve better state-of-the-art results it’s necessary to use the large version of it (e.g. BERT<sub>LARGE</sub>). Last but not least, the importance of quality data cannot be overlooked.

Concluding, the task of text semantic similarity is one of the most challenging tasks in Natural Language Processing and Understanding. It is important to emphasize that further research on the subject will provide opportunities for development in the field of NLU (Natural Language Understanding), thus offering innovative solutions that can be used in many practical applications, like text-based human-computer interaction.

# References

1. Tellex, S., Gopalan, N., Kress-Gazit, H., Matuszek, C.: Robots That Use Language. Ann. Rev. Control, Robot. Autono. Syst. 3, 25–55 (2020). https://doi.org/10.1146/annurev-control-10119-071628

2. Zhang, Y., et al.: Building Natural Language Interfaces Using Natural Language Understanding and Generation: A Case Study on Human-Machine Interaction in Agriculture. Appl. Sci. 12(22), 11830 (2022). https://doi.org/10.3390/app122211830
3. Pennington, J., Socher, R., Manning, C.D.: GloVe: Global Vectors for Word Representation. In: 2014 conference on empirical methods in natural language processing (EMNLP), Doha, Qatar, (2014)

4. Devlin, J., Chang, M.-W., Lee, K., Toutanova, K.: BERT: Pre-training of deep bidirectional transformers for language understanding. https://arxiv.org/abs/1810.04805. (2018)

5. Socher, R., Huang, E.H., Pennington, J., Ng, C.D.: Manning. Dynamic pooling and unfolding recursive autoencoders for paraphrase detection. In Advances in neural information processing systems. (2011)

6. Hu, B., Lu, Z., Li, H., Chen, Q.: Convolutional neural network architectures for matching natural language sentences. Adv. Neural. Inf. Process. Syst. 27, 2042–2050 (2014)

7. Bengio, Y.: Learning deep architectures for AI, Montreal. Now Publishers Inc, Canada (2009)

8. Yin, W., Schütze, H.: Convolutional neural network for paraphrase identification. In: Proceedings of the 2015 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (2015)

9. He, H., Gimpel, K., Lin, J.: Multi-Perspective Sentence Similarity Modeling with Convolutional Neural. In: Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, (2015)

10. Agarwal, B., Ramampiaro, H., Langseth, H., Ruocco, M.: A deep network model for paraphrase detection in short text messages. Inf. Process. Manage. 54(6), 922–937 (2018)

11. Song, Y., Hu, W., He, L.: Using fractional latent topic to enhance recurrent neural network in text similarity modeling. In: International Conference on Database Systems for Advanced Applications (2019)

12. Wang, S., Jiang, J.: Learning Natural Language Inference with LSTM. In: The 15th Annual Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, (2016)

13. Rocktäschel, T., Grefenstette, E., Hermann, K.M., Kočiský, T., Blunsom, P.: Reasoning about entailment with neural attention. In ICLR 2016 (2016)

14. Logeswaran, L., Lee, H.: An efficient framework for learning sentence representations. In: International Conference on Learning Representations (2018)

15. Liu, P., Qiu, X., Chen, J., Huang, X.: Deep fusion LSTMs for text semantic matching. In: Proceedings of ACL 2016, Berlin, Germany, (2016)

16. Vaswani, A., et al.: Attention is all you need. In: Advances in neural information processing systems (2017)

17. Zhang, Z., et al.: Semantics aware BERT for Language Understanding. AAAI-2020 (2020)

18. Parikh, A.P., Täckström, O., Das, D., Uszkoreit, D.: A Decomposable Attention Model for Natural Language Inference. In: MNLP 2016: Conference on Empirical Methods in Natural Language Processing, Austin. Texas, (2016)

19. Radford, A., Narasimhan, K., Salimans, T., Sutskever, I.: Improving language understanding by generative pre-training. (2018)

20. Bowman, S.R., Angeli, G., Potts, C., Manning, C.D: A large annotated corpus for learning natural language inference. In EMNLP, Lisbon, Portugal, (2015)

21. Liu, L., et al.: On the Variance of the Adaptive Learning Rate and Beyond. In: the 8th International Conference on Learning Representations (ICLR 2020) (2019)

22. Chen, S., Hou, Y., Cui, Y., Che, W., Liu, T., Yu, x.: Recall and learn: Fine-tuning deep pretrained language models with less forgetting. (2020) arXiv preprint arXiv:2004.12651

23. Logeswaran, L., Lee, H.: An efficient framework for learning sentence representations, In: International Conference on Learning Representations. (2018) arXiv:1803.02893 [cs.CL]

24. Subramanian, S., Trischler, A., Bengio, Y., Pal, C.: Learning general purpose distributed sentence representations via large scale multi-task learning. (2018) arXiv:1804.00079 [cs.CL]
25. Zhang, X., Rong, W., Liu, J., Tian, C., Xiong, Z.: Convolution neural network based syntactic and semantic aware paraphrase identification. International Joint Conference on Neural Networks (IJCNN) **2017**, 2158–2163 (2017)

26. Le, H.T., Cao, D.T., Bui, T.H., Luong, L.T., Nguyen, H.Q.: Improve Quora Question Pair Dataset for Question Similarity Task. RIVF International Conference on Computing and Communication Technologies (RIVF), pp. 1–5 (2021)

27. Poksappaiboon, N., Sundarabhogin, N., Tungruethaipak, N., Prom-on, S.: Detecting Text Semantic Similarity by Siamese Neural Networks with MaLSTM in Thai Language. In: Proceedings of the 2nd International Conference on Big Data Analytics and Practices (IBDAP), 07–11 (2021)

28. Han, S., Shi, L., Richie, R., Tsui, F.R.: Building siamese attention-augmented recurrent convolutional neural networks for document similarity scoring. Inf. Sci. **615**, 90–102 (2022)

29. Viji, D., Revathy, S.: A hybrid approach of Weighted Fine-Tuned BERT extraction with deep Siamese Bi – LSTM model for semantic text similarity identification. Multi. Tools Appl **81**, 6131–6157 (2022). https://doi.org/10.1007/s11042-021-11771-6

30. Gontumukkala, S.S.T., Godavarthi, Y.S.V., Gonugunta, B.R.R.T., Gupta, D., Palaniswam, S.: Quora Question Pairs Identification and Insincere Questions Classification. 2022 13th International Conference on Computing Communication and Networking Technologies (ICCCNT), Kharagpur, India, pp. 1–6 (2022) https://doi.org/10.1109/ICCCNT54827.2022.9984492