Experiments and Discussion	This section describes two experiments: i) compare our model against recent systems; ii) evaluate the efficiency of using multiple pre-trained word embeddings.

For measuring the similarity/relation between two sentence representations, we propose Multi-level comparison which consists of word-word level, sentence-sentence level, and word-sentence level. Through these levels, our model comprehensively evaluates the similarity/relation between two sentences. We evaluate our M-MaxLSTM-CNN model on three tasks: STS, textual entailment recognition, paraphrase identification. The advantages of M-MaxLSTM-CNN are: i) simple but efficient for combining various pre-trained word embeddings with different dimensions; ii) using Multi-level comparison shows better performances compared to using only sentence-sentence comparison; iii) does not require hand-crafted features (e.g., alignment features, Ngram overlaps, syntactic features, dependency features) compared to the state-of-the-art ECNU  on STS Benchmark dataset. Our main contributions are as follows: The remainder of this paper is organized as follows: Section 2 reviews the previous research, Section 3 introduces the architecture of our model, Section 4 describes the three tasks and datasets, Section 5 describes the experiment setting, Section 6 reports and discusses the results of the experiments, and Section 7 concludes our work.

Related work	Most prior research on modeling textual similarity relied on feature engineering.  extract INLINEFORM0 -gram overlap features and dependency-based features, while  employ features based on machine translation metrics.  propose a method using corpus-based and knowledge-based measures of similarity.  design a model which incorporates both syntax and lexical semantics using dependency grammars.  combine the fine-grained n-gram overlap features with the latent representation from matrix factorization.  develop a latent variable model which jointly learns paraphrase relations between word and sentence pairs. Using Dependency trees,  propose a robust monolingual aligner and successfully applied it for STS tasks. The recent emergence of deep learning models has provided an efficient way to learn continuous vectors representing words/sentences. By using a neural network in the context of a word prediction task,  and  generate word embedding vectors carrying semantic meanings. The embedding vectors of words which share similar meanings are close to each other. To capture the morphology of words,  enrich the word embedding with character n-grams information. Closest to this approach,  also propose to represent a word or sentence using a character n-gram count vector.

observed that different word embedding models capture different aspects of linguistic properties: a Bag-of-Words contexts based model tends to reflect the domain aspect (e.g., scientist and research) while a paraphrase-relationship based model captures semantic similarities of words (e.g., boy and kid). From experiments, we also observed that the performance of a word embedding model is usually inconsistent over different datasets. This inspired us to develop a model taking advantages of various pre-trained word embeddings for measuring textual similarity/relation. In this paper, we propose a convolutional neural network (CNN) to learn a multi-aspect word embedding from various pre-trained word embeddings. We then apply the max-pooling scheme and Long Short Term Memory (LSTM) on this embedding to form a sentence representation. In STS tasks,  shows the efficiency of the max-pooling scheme in modeling sentences from word embedding representations refined via CNN. However, the max-pooling scheme lacks the property of word order (e.g., sentence(“Bob likes Marry”) = sentence(“Marry likes Bob”)). To address this weakness, we use LSTM as an additional scheme for modeling sentences with word order characteristics.

Introduction	Measuring the semantic similarity/relation of two pieces of short text plays a fundamental role in a variety of language processing tasks (i.e., plagiarism detection, question answering, and machine translation). Semantic textual similarity (STS) task is challenging because of the diversity of linguistic expression. For example, two sentences with different lexicons could have a similar meaning. Moreover, the task requires to measure similarity at several levels (e.g., word level, phrase level, sentence level). These challenges give difficulties to conventional approaches using hand-crafted features. Recently, the emergence of word embedding techniques, which encode the semantic properties of a word into a low dimension vector, leads to the successes of many learning models in natural language processing (NLP). For example,  randomly initialize word vectors, then tunes them during the training phase of a sentence classification task. By contrast,  initialize word vectors via the pre-train word2vec model trained on Google News  .  train a word embedding model on the paraphrase dataset PPDB, then apply the word representation for word and bi-gram similarity tasks. Several pre-trained word embeddings are available, which are trained on various corpora under different models.

HCTI  , which is the current state-of-the-art in the group of neural representation models on STSB, also used the Glove embedding. However, the performance of HTCI in STSB ( INLINEFORM1 ) is lower than our model using the Glove embedding. In SICK-R, InferSent  achieves a strong performance ( INLINEFORM2 ) using the Glove embedding with transfer knowledge, while our model with only the Glove embedding achieves a performance close to the performance of InferSent. These results confirm the efficiency of Multi-level comparison. In STSB and MRPC, as employing the five pre-trained embeddings, the INLINEFORM0 is increased. This factor limits the number of random values when initializing word embedding representations because a word out of a pre-trained word embedding is assigned a random word embedding representation. In other words, a word out of a pre-trained word embedding is assigned a random semantic meaning. Therefore, the increase of the INLINEFORM1 improves the performance of measuring textual similarity. In STSB and MRPC, our multiple pre-trained word embedding achieves a significant improvement in performance compared against using one word embedding.

We observed that MaxLSTM-CNN outperforms Max-CNN in both of the settings (i.e., sentence-sentence comparison, Multi-level comparison). As mentioned in Section 1, Max-CNN ignores the property of word order. Therefore, our model achieves improvement compared to Max-CNN by additionally employing LSTM for capturing this property. We only applied Multi-level comparison on Max-CNN and MaxLSTM-CNN because these encoders generate multi-aspect word embeddings. The experimental results prove the efficiency of using Multi-level comparison. In the textual entailment dataset SICK-E, the task mainly focuses on interpreting the meaning of a whole sentence pair rather than comparing word by word. Therefore, the performance of Multi-level comparison is quite similar to sentence-sentence comparison in the SICK-E task. This is also the reason why LSTM, which captures global relationships in sentences, has the strong performance in this task.

Training Setting	In these tasks, we use the cross-entropy objective function and employ AdaDelta as the stochastic gradient descent (SGD) update rule with mini-batch size as 30. Details of Adadelta method can be found in  . During the training phase, the pre-trained word embeddings are fixed. To compute a similarity score of a sentence pair in the range INLINEFORM0 , where INLINEFORM1 is an integer, we replace Eq. (27) with the equations in  as follows: DISPLAYFORM0   where INLINEFORM0 , INLINEFORM1 , INLINEFORM2 and INLINEFORM3 are parameters; INLINEFORM4 ; INLINEFORM5 is a predicted similarity score. A sparse target distribution INLINEFORM0 which satisfies INLINEFORM1 is computed as: DISPLAYFORM0  for INLINEFORM0 , and INLINEFORM1 is the similarity score. To train the model, we minimize the regularized KL-divergence between INLINEFORM0 and INLINEFORM1 : DISPLAYFORM0  where INLINEFORM0 is the number of training pairs and INLINEFORM1 denotes the model parameters. The gradient descent optimization Adadelta is used to learn the model parameters.

Evaluation of exploiting multiple pre-trained word embeddings	In this section, we evaluate the efficiency of using multiple pre-trained word embeddings. We compare our multiple pre-trained word embeddings model against models using only one pre-trained word embedding. The same objective function and Multi-level comparison are applied for these models. In case of using one pre-trained word embedding, the dimension of LSTM and the number of convolutional filters are set to the length of the corresponding word embedding. Table TABREF57 shows the experimental results of this comparison. Because the approach using five word embeddings outperforms the approaches using two, three, or four word embeddings, we only report the performance of using five word embeddings. We also report INLINEFORM0 which is the proportion of vocabulary available in a pre-trained word embedding. SICK dataset ignores idiomatic multi-word expressions, and named entities, consequently the INLINEFORM1 of SICK is quite high. We observed that no word embedding has strong results on all the tasks. Although trained on the paraphrase database and having the highest INLINEFORM0 , the SL999 embedding could not outperform the Glove embedding in SICK-R.

Multi-aspect word embedding	Given a word INLINEFORM0 , we transfer it into a word vector INLINEFORM1 via INLINEFORM2 pre-trained word embeddings as follows: DISPLAYFORM0  where INLINEFORM0 is concatenation operator, INLINEFORM1 is the word embedding vector of INLINEFORM2 in the INLINEFORM3 th pre-trained embedding. To learn a multi-aspect word embedding INLINEFORM0 from the representation INLINEFORM1 , we design INLINEFORM2 convolutional filters. Each filter INLINEFORM3 is denoted as a weight vector with the same dimension as INLINEFORM4 and a bias value INLINEFORM5 . The INLINEFORM6 is obtained by applying these filters on the INLINEFORM7 as follows: DISPLAYFORM0   where INLINEFORM0 denotes a logistic sigmoid function. The next section explains how to model a sentence from its multiple-aspect word embeddings.

In SICK-R and SICK-E, although the INLINEFORM2 is not increased when employing five pre-trained embeddings, the performance of our model is improved. This fact shows that our model learned an efficient word embedding via these pre-trained word embeddings.

Sentence Modeling via Multiple Word Embeddings and Multi-level Comparison for Semantic Textual Similarity	Different word embedding models capture different aspects of linguistic properties. This inspired us to propose a model (M-MaxLSTM-CNN) for employing multiple sets of word embeddings for evaluating sentence similarity/relation. Representing each word by multiple word embeddings, the MaxLSTM-CNN encoder generates a novel sentence embedding. We then learn the similarity/relation between our sentence embeddings via Multi-level comparison. Our method M-MaxLSTM-CNN consistently shows strong performances in several tasks (i.e., measure textual similarity, identify paraphrase, recognize textual entailment). According to the experimental results on STS Benchmark dataset and SICK dataset from SemEval, M-MaxLSTM-CNN outperforms the state-of-the-art methods for textual similarity tasks. Our model does not use hand-crafted features (e.g., alignment features, Ngram overlaps, dependency features) as well as does not require pre-trained word embeddings to have the same dimension.

Pre-trained word embeddings	We study five pre-trained word embeddings for our model: word2vec is trained on Google News dataset (100 billion tokens). The model contains 300-dimensional vectors for 3 million words and phrases. fastText is learned via skip-gram with subword information on Wikipedia text. The embedding representations in fastText are 300-dimensional vectors. GloVe is a 300-dimensional word embedding model learned on aggregated global word-word co-occurrence statistics from Common Crawl (840 billion tokens). Baroni uses a context-predict approach to learn a 400-dimensional semantic embedding model. It is trained on 2.8 billion tokens constructed from ukWaC, the English Wikipedia and the British National Corpus. SL999 is trained under the skip-gram objective with negative sampling on word pairs from the paraphrase database PPDB. This 300-dimensional embedding model is tuned on SimLex-999 dataset  .

train a linear regression model with WordNet, alignment features and the word embedding word2vec.  develop an ensemble model with multiple boosting techniques (i.e., Random Forest, Gradient Boosting, and XGBoost). This model incorporates traditional features (i.e., n-gram overlaps, syntactic features, alignment features, bag-of-words) and sentence modeling methods (i.e., Averaging Word Vectors, Projecting Averaging Word Vectors, LSTM). MVCNN model  and MGNC-CNN model  are close to our approach. In MVCNN, the authors use variable-size convolution filters on various pre-trained word embeddings for extracting features. However, MVCNN requires word embeddings to have the same size. In MGNC-CNN, the authors apply independently CNN on each pre-trained word embedding for extracting features and then concatenate these features for sentence classification. By contrast, our M-MaxLSTM-CNN model jointly applies CNN on all pre-trained word embeddings to learn a multi-aspect word embedding. From this word representation, we encode sentences via the max-pooling and LSTM. To learn the similarity/relation between two sentences, we employ Multi-level comparison.

Conclusion	In this work, we study an approach employing multiple pre-trained word embeddings and Multi-level comparison for measuring semantic textual relation. The proposed M-MaxLSTM-CNN architecture consistently obtains strong performances on several tasks. Compared to the state-of-the art methods in STS tasks, our model does not require handcrafted features (e.g., word alignment, syntactic features) as well as transfer learning knowledge. In addition, it allows using several pre-trained word embeddings with different dimensions. Future work could apply our multiple word embeddings approach for transfer learning tasks. This strategy allows making use of pre-trained word embeddings as well as available resources.

Sentence modeling	Given an input sentence INLINEFORM0 , we obtain a sequence of multiple-aspect word embeddings INLINEFORM1 using Eq. (1-3). For modeling the sentence from the representation INLINEFORM2 , we use two schemes: max-pooling and LSTM. Max-pooling scheme: To construct a max-pooling sentence embedding INLINEFORM0 , the most potential features are extracted from the representation INLINEFORM1 as follows: DISPLAYFORM0  where INLINEFORM0 is the INLINEFORM1 th element of INLINEFORM2 . LSTM scheme: From Eq. (4), we find that the max-pooling scheme ignores the property of word order. Therefore, we construct a LSTM sentence embedding INLINEFORM0 to support the sentence embedding INLINEFORM1 . The representation INLINEFORM2 is transformed to a fix-length vector by recursively applying a LSTM unit to each input INLINEFORM3 and the previous step INLINEFORM4 .

Model description	Our model (shown in Figure FIGREF4 ) consists of three main components: i) learning a multi-aspect word embedding (Section 3.1); ii) modeling sentences from this embedding (Section 3.2); iii) measuring the similarity/relation between two sentences via Multi-level comparison (section 3.3).

Overall evaluation	Besides existing methods, we also compare our model with several sentence modeling approaches using multiple pre-trained word embeddings: Word Average: DISPLAYFORM0  where INLINEFORM0 is the sentence embedding of a INLINEFORM1 -words sentence, and INLINEFORM2 is from Eq. (1) Project Average: DISPLAYFORM0  where INLINEFORM0 is a INLINEFORM1 weight matrix, and INLINEFORM2 is a 1600 bias vector. LSTM: apply Eq. (5-11) on INLINEFORM0 to construct the 1600-dimension INLINEFORM1 sentence embedding. Max-CNN: apply Eq. (2-4) on INLINEFORM0 to construct the 1600-dimension INLINEFORM1 sentence embedding. We report the results of these methods in Table TABREF49 . Overall, our M-MaxLSTM-CNN shows competitive performances in these tasks. Especially in the STS task, M-MaxLSTM-CNN outperforms the state-of-the-art methods on the two datasets. Because STSB includes complicated samples compared to SICK, the performances of methods on STSB are quite lower.

In STSB, the prior top performance methods use ensemble approaches mixing hand-crafted features (word alignment, syntactic features, N-gram overlaps) and neural sentence representations, while our approach is only based on a neural sentence modeling architecture. In addition, we observed that InferSent shows the strong performance on SICK-R but quite low on STSB while our model consistently obtains the strong performances on both of the datasets. InferSent uses transfer knowledge on textual entailment data, consequently it obtains the strong performance on this entailment task. According to  , using Word Average as the compositional architecture outperforms the other architectures (e.g., Project Average, LSTM) for STS tasks. In a multiple word embeddings setting, however, Word Average does not show its efficiency. Each word embedding model has its own architecture as well as objective function. These factors makes the vector spaces of word embeddings are different. Therefore, we intuitively need a step to learn or refine a representation from a set of pre-trained word embeddings rather than only averaging them. Because Project Average model, LSTM model and Max-CNN model have their parameters for learning sentence embeddings, they significantly outperform Word Average model.

Multi-level comparison	In this section, we describe the process for evaluating the similarity/relation between two sentences. We compare two sentences via three levels: word-word, sentence-sentence and word-sentence. Given two input sentences INLINEFORM0 and INLINEFORM1 , we encode them into two sequences of multi-aspect word embeddings INLINEFORM2 and INLINEFORM3 (Section 3.2). We then compute a word-word similarity vector INLINEFORM4 as follows: DISPLAYFORM0   where INLINEFORM0 is the INLINEFORM1 th multi-aspect word embedding of sentence INLINEFORM2 ; INLINEFORM3 is a function to flatten a matrix into a vector; and INLINEFORM4 and INLINEFORM5 are respectively a weight matrix and a bias parameter. Given two input sentences INLINEFORM0 and INLINEFORM1 , we encode them into two sentence embeddings INLINEFORM2 and INLINEFORM3 (Section 3.1 and 3.2). To compute the similarity/relation between the two embeddings, we introduce four comparison metrics: Cosine similarity: DISPLAYFORM0  Multiplication vector & Absolute difference: DISPLAYFORM0   where INLINEFORM0 is element-wise multiplication.

Neural difference: DISPLAYFORM0   where INLINEFORM0 and INLINEFORM1 are respectively a weight matrix and a bias parameter. As a result, we have a sentence-sentence similarity vector INLINEFORM0 as follows: DISPLAYFORM0   where INLINEFORM0 and INLINEFORM1 are respectively a weight matrix and a bias parameter. Given a sentence embedding INLINEFORM0 and a sequence of multi-aspect word embeddings INLINEFORM1 , we compute a word-sentence similarity matrix INLINEFORM2 as follows: DISPLAYFORM0   where INLINEFORM0 is the multi-aspect word embedding of the INLINEFORM1 th word in sentence INLINEFORM2 ; INLINEFORM3 and INLINEFORM4 are respectively a weight matrix and a bias parameter. As a result, we have a word-sentence similarity vector INLINEFORM0 for the two sentences as follows: DISPLAYFORM0  where INLINEFORM0 is a function to flatten a matrix into a vector; INLINEFORM1 and INLINEFORM2 are respectively a weight matrix and a bias parameter.

Model configuration	In all of the tasks, we used the same model configuration as follows: Convolutional filters: we used 1600 filters. It is also the dimension of the word embedding concatenated from the five pre-trained word embeddings. LSTM dimension: we also selected 1600 for LSTM dimension. Neural similarity layers: the dimension of INLINEFORM0 , INLINEFORM1 , INLINEFORM2 and INLINEFORM3 are respectively 50, 5, 5 and 100. Penultimate fully-connected layer: has the dimension of 250 and is followed by a drop-out layer ( INLINEFORM0 ). We conducted a grid search on 30% of STSB dataset to select these optimal hyper-parameters.

We also use mini-batch size as 30 and keep the pre-trained word embeddings fixed during the training phase. We evaluate our models through Pearson correlation INLINEFORM2 .

At each time step INLINEFORM5 , the LSTM unit with INLINEFORM6 -memory dimension defines six vectors in INLINEFORM7 : input gate INLINEFORM8 , forget gate INLINEFORM9 , output gate INLINEFORM10 , tanh layer INLINEFORM11 , memory cell INLINEFORM12 and hidden state INLINEFORM13 as follows (from  ): DISPLAYFORM0 DISPLAYFORM1   where INLINEFORM0 respectively denote a logistic sigmoid function and element-wise multiplication; INLINEFORM1 are respectively two weights matrices and a bias vector for input gate INLINEFORM2 . The denotation is similar to forget gate INLINEFORM3 , output gate INLINEFORM4 , tanh layer INLINEFORM5 , memory cell INLINEFORM6 and hidden state INLINEFORM7 . Finally, the sentence embedding INLINEFORM0 is obtained by concatenating the two sentence embeddings INLINEFORM1 and INLINEFORM2 : DISPLAYFORM0

However, the objective function for learning these embeddings is based on paraphrase pairs. For modeling sentences, composition approach attracted many studies.  model each word as a matrix and used iterated matrix multiplication to present a phrase.  design a Dependency Tree-Structured LSTM for modeling sentences. This model outperforms the linear chain LSTM in STS tasks. Convolutional neural network (CNN) has recently been applied efficiently for semantic composition  ,  ,  . This technique uses convolutional filters to capture local dependencies in term of context windows and applies a pooling layer to extract global features.  use CNN to extract features at multiple level of granularity. The authors then compare their sentence representations via multiple similarity metrics at several granularities.  propose a hierarchical CNN-LSTM architecture for modeling sentences. In this approach, CNN is used as an encoder to encode an sentence into a continuous representation, and LSTM is used as a decoder.  train a sentence encoder on a textual entailment recognition database using a BiLSTM-Maxpooling network. This encoder achieves competitive results on a wide range of transfer tasks. At SemEval-2017 STS task, hybrid approaches obtain strong performances.

Acknowledgments	This work was done while Nguyen Tien Huy was an intern at Toshiba Research Center.

Tasks & Datasets	We evaluate our model on three tasks: Table TABREF30 shows the statistic of the three datasets. Because of not dealing with name entities and multi-word idioms, the vocabulary size of SICK is quite small compared to the others.

Finally, we compute a target score/label of a sentence pair as follows: DISPLAYFORM0   where INLINEFORM0 , INLINEFORM1 , INLINEFORM2 and INLINEFORM3 are model parameters; INLINEFORM4 is a predicted target score/label.

Question: which pretrained embeddings were experimented with?
Answer: word2vec , fastText , GloVe , Baroni , SL999