Venue: NAACL 2016

# Pairwise Word Interaction Modeling with Deep Neural Networks

for Semantic Similarity Measurement

 Hua He\({}^{1}\)  and Jimmy Lin\({}^{2}\)

\({}^{1}\) Department of Computer Science, University of Maryland, College Park

\({}^{2}\) David R. Cheriton School of Computer Science, University of Waterloo

huah@umd.edu, jimmylin@uwaterloo.ca

###### Abstract

Textual similarity measurement is a challenging problem, as it requires understanding the semantics of input sentences. Most previous neural network models use coarse-grained sentence modeling, which has difficulty capturing fine-grained word-level information for semantic comparisons. As an alternative, we propose to explicitly model pairwise word interactions and present a novel similarity focus mechanism to identify important correspondences for better similarity measurement. Our ideas are implemented in a novel neural network architecture that demonstrates state-of-the-art accuracy on three SemEval tasks and two answer selection tasks.

## 1 Introduction

Given two pieces of text, measuring their semantic textual similarity (STS) remains a fundamental problem in language research and lies at the core of many language processing tasks, including question answering [3], query ranking [5], and paraphrase generation [6].

Traditional NLP approaches, e.g., developing hand-crafted features, suffer from sparsity because of language ambiguity and the limited amount of annotated data available. Neural networks and distributed representations can alleviate such sparsity, thus neural network-based models are widely used by recent systems for the STS problem [7, 8, 9].

However, most previous neural network approaches are based on _sentence modeling_, which first maps each input sentence into a fixed-length vector and then performs comparisons on these representations. Despite its conceptual simplicity, researchers have raised concerns about this approach [4]: Will fine-grained word-level information, which is crucial for similarity measurement, get lost in the coarse-grained sentence representations? Is it really effective to "cram" whole sentence meanings into fixed-length vectors?

In contrast, we focus on capturing fine-grained word-level information directly. Our contribution is twofold: First, instead of using sentence modeling, we propose _pairwise word interaction modeling_ that encourages explicit word context interactions across sentences. This is inspired by our own intuitions of how people recognize textual similarity: given two sentences \(sent_{1}\) and \(sent_{2}\), a careful reader might look for corresponding semantic units, which we operationalize in our pairwise word interaction modeling technique (Sec. 5). Second, based on the pair-wise word interactions, we describe a novel _similarity focus layer_ which helps the model selectively identify important word interactions depending on their importance for similarity measurement. Since not all words are created equal, important words that can make more contributions deserve extra "focus" from the model (Sec. 6).

We conducted thorough evaluations on ten test sets from three SemEval STS competitions [1, 10, 11] and two answer selection tasks [12, 13]. We outperform the recent multiperspective convolutional neural networks of he2015deep and demonstrate state-of-the-art accuracy on all five tasks. In addition, we conducted ablation

studies and visualized our models to show the clear benefits of modeling pairwise word interactions for similarity measurement.

## 2 Related Work

Feature engineering was the dominant approach in most previous work; different types of sparse features were explored and found useful. For example, \(n\)-gram overlap features at the word and character levels Madnani et al. (2012); Wan et al. (2006), syntax features Das Smith (2009); Xu et al. (2014), knowledge-based features using WordNet Fellbaum (1998); Fern and Stevenson (2008) and word-alignment features Sultan et al. (2014).

The recent shift from sparse feature engineering to neural network model engineering has significantly improved accuracy on STS datasets. Most previous work use sentence modeling with a "Siamese" structure Bromley et al. (1993). For example, Hu et al. (2014) used convolutional neural networks that combine hierarchical structures with layer-by-layer composition and pooling. Tai et al. (2015) and Zhu et al. (2015) concurrently proposed tree-structured long short-term memory networks, which recursively construct sentence representations following their syntactic trees. There are many other examples of neural network-based sentence modeling approaches for the STS problem Yin and Schutze (2015); Huang et al. (2013); Andrew et al. (2013); Weston et al. (2011); Socher et al. (2011); Zarrella et al. (2015).

Sentence modeling is coarse-grained by nature. Most recently, despite still using a sentence modeling approach, He et al. (2015) moved toward fine-grained representations by exploiting multiple perspectives of input sentences with different types of convolution filters and pooling, generating a "matrix" representation where rows and columns capture different aspects of the sentence; comparisons over local regions of the representation are then performed. He et al. (2015) achieves highly competitive accuracy, suggesting the usefulness of fine-grained information. However, these multiple perspectives are obtained at the cost of increased model complexity, resulting in slow model training. In this work, we take a different approach by focusing directly on pairwise word interaction modeling.

## 3 Model Overview

Figure 1 shows our end-to-end model with four major components:

1. Bidirectional Long Short-Term Memory Networks (Bi-LSTMs) Graves et al. (2005); Graves et al. (2006) are used for context modeling of input sentences, which serves as the basis for all following components (Sec. 4).
2. A novel pairwise word interaction modeling technique encourages direct comparisons between word contexts across sentences (Sec. 5).
3. A novel similarity focus layer helps the model identify important pairwise word interactions across sentences (Sec. 6).

Figure 1: Our end-to-end neural network model, consisting of four major components.

 

4. A \(19\)-layer deep convolutional neural network (ConvNet) converts the similarity measurement problem into a pattern recognition problem for final classification (Sec. 7).

To our best knowledge this is the first neural network model, a novel hybrid architecture combining Bi-LSTMs and a deep ConvNet, that uses a similarity focus mechanism with selective attention to important pairwise word interactions for the STS problem. Our approach only uses pretrained word embeddings, and unlike several previous neural network models Yin and Schutze (2015); Tai et al. (2015), we do not use sparse features, unsupervised model pretraining, syntactic parsers, or external resources like WordNet. We describe details of each component in the following sections.

## 4 Context Modeling

Different words occurring in similar semantic contexts of respective sentences have a higher chance to contribute to the similarity measurement. We therefore need word context modeling, which serves as a basis for all following components of this work.

LSTM Hochreiter and Schmidhuber (1997) is a special variant of Recurrent Neural Networks Williams and Zipser (1989). It can capture long-range dependencies and nonlinear dynamics between words, and has been successfully applied to many NLP tasks Sutskever et al. (2014); Filippova et al. (2015). LSTM has a memory cell that can store information over a long history, as well as three gates that control the flow of information into and out of the memory cell. At time step \(t\), given an input \(x_{t}\), previous output \(h_{t-1}\), input gate \(i_{t}\), output gate \(o_{t}\) and forget gate \(f_{t}\), \(LSTM(x_{t},h_{t-1})\) outputs the hidden state \(h_{t}\) based on the equations below:

\[i_{t}=\sigma(W^{i}x_{t}+U^{i}h_{t-1}+b^{i}) \tag{1}\] \[f_{t}=\sigma(W^{f}x_{t}+U^{f}h_{t-1}+b^{f})\] (2) \[o_{t}=\sigma(W^{o}x_{t}+U^{o}h_{t-1}+b^{o})\] (3) \[u_{t}=tanh(W^{w}x_{t}+U^{v}h_{t-1}+b^{u})\] (4) \[c_{t}=i_{t}\cdot u_{t}+f_{t}\cdot c_{t-1}\] (5) \[h_{t}=o_{t}\cdot tanh(c_{t})\] (6) \[LSTM(x_{t},h_{t-1})=h_{t}\] (7) \[BiLSTMs(x_{t},h_{t-1})=\{LSTM^{f},LSTM^{b}\} \tag{8}\]

where \(\sigma\) is the logistic sigmoid activation, \(W^{*}\), \(U^{*}\) and \(b^{*}\) are learned weight matrices and biases. LSTMs are better than RNNs for context modeling, in that their memory cells and gating mechanisms handle the vanishing gradients problem in training.

We use bidirectional LSTMs (Bi-LSTMs) for context modeling in this work. Bi-LSTMs consist of two LSTMs that run in parallel in opposite directions: one (forward \(LSTM^{f}\)) on the input sequence and the other (backward \(LSTM^{b}\)) on the reverse of the sequence. At time step \(t\), the Bi-LSTMs hidden state \(h_{t}^{bi}\) is a concatenation of the hidden state \(h_{t}^{for}\) of \(LSTM^{f}\) and the hidden state \(h_{t}^{back}\) of \(LSTM^{b}\), representing the neighbor contexts of input \(x_{t}\) in the sequence. We define the \(unpack\) operation below:

\[h_{t}^{for},h_{t}^{back}=unpack(h_{t}^{bi}) \tag{9}\]

Context modeling with Bi-LSTMs allows all the following components to be built over word contexts, rather than over individual words.

## 5 Pairwise Word Interaction Modeling

From our own intuitions, given two sentences in a STS task, a careful human reader might compare words and phrases across the sentences to establish semantic correspondences and from these infer similarity. Our pairwise word interaction model is inspired by such behavior: whenever the next word of a sentence is read, the model would compare it and its context against all words and their contexts in the other sentence. Figure 2 illustrates this model.

We first define a comparison unit for comparing two hidden states \(\overrightarrow{h_{1}},\overrightarrow{h_{2}}\) of Bi-LSTMs.

\[\textit{co}\textit{U}(\overrightarrow{h_{1}},\overrightarrow{ h_{2}})=\{\cos(\overrightarrow{h_{1}},\overrightarrow{h_{2}}),L_{2}\textit{Euclid}( \overrightarrow{h_{1}},\overrightarrow{h_{2}}),\\ \textit{DotProduct}(\overrightarrow{h_{1}},\overrightarrow{h_{2}})\} \tag{10}\]

Cosine distance (\(\cos\)) measures the distance of two vectors by the angle between them, while \(L_{2}\) Euclidean distance (\(L_{2}\textit{Euclid}\)) and dot-product distance (\(\textit{DotProduct}\)) measure magnitude differences. We use three similarity functions for richer measurement.

Algorithm 1 provides details of the modeling process. Given the input \(x_{t}^{a}\in sent_{a}\) at time step \(t\) where \(a\in\{1,2\}\), its Bi-LSTMs hidden state \(h_{at}^{bi}\) is the concatenation of the forward state \(h_{at}^{for}\) and the 

backward state \(h_{at}^{back}\). Algorithm 1 proceeds as follows: it enumerates all word pairs \((s,t)\) across both sentences, then perform comparisons using the \(coU\) unit four times over: 1) Bi-LSTMs hidden states \(h_{1t}^{bi}\) and \(h_{2s}^{bi}\); 2) forward hidden states \(h_{1t}^{for}\) and \(h_{2s}^{for}\); 3) backward hidden states \(h_{1t}^{back}\) and \(h_{2s}^{back}\); and 4) the addition of forward and backward hidden states \(h_{1t}^{add}\) and \(h_{2s}^{add}\). The output of Algorithm 1 is a similarity cube \(simCube\) with size \(R^{13\cdot|sent_{1}|\cdot|sent_{2}|}\), where \(|sent_{*}|\) is the number of words in the sentence \(sent_{*}\). The \(13\) values collected from each word pair \((s,t)\) are: the \(12\) similarity distances, plus one extra dimension for the padding indicator. Note that all word interactions are modeled over word contexts in Algorithm 1, rather than individual words.

Our pairwise word interaction model shares similarities with recent popular neural attention models Bahdanau et al. (2014); Rush et al. (2015). However, there are important differences: For example, we do not use attention weight vectors or weighted representations, which are the core of attention models. The other difference is that attention weights are typically interpreted as soft degrees with which the model attends to particular words; in contrast, our word interaction model directly utilizes multiple similarity metrics, and thus is more explicit.

## 6 Similarity Focus Layer

Since not all words are created equal, important pairwise word interactions between the sentences (Sec. 5) that can better contribute to the similarity measurement deserve more model focus. We therefore develop a similarity focus layer which can identify important word interactions and increase their model weights correspondingly. This similarity focus layer is directly incorporated into our end-to-end model and is placed on top of the pairwise word interaction model, as in Figure 1.

Figure 3 shows one example where each cell of the matrix represents a pairwise word interaction. The similarity focus layer introduces re-weightings to word interactions depending on their importance for similarity measurement. The ones tagged with black dots are considered important, and are given higher weights than those without.

Algorithm 2 shows the forward pass of the similarity focus layer. Its input is the similarity cube \(simCube\) (Section 5). Algorithm 2 is designed to incorporate two different aspects of similarity based on \(cosine\) (angular) and \(L2\) (magnitude) similarity, thus it has two symmetric components: the first one is based on \(cosine\) similarity (Line 5 to Line 13); and the second one is based on \(L2\) similarity (Line 15 to Line 23). We also aim for the goal that similarity values of all found important word in

Figure 3: The similarity focus layer helps identify important pairwise word interactions (in black dots) depending on their importance for similarity measurement.

Figure 2: Pairwise word interaction modeling. Sentences are encoded by weight-shared Bi-LSTMs. We construct pairwise word interactions for context comparisons across sentences.

 

teractions should be maximized. To achieve this, we sort the similarity values in descending order (Line 5 for \(cosine\), Line 15 for \(L2\)). Note channels \(10\) and \(11\) of the \(simCube\) contain \(cosine\) and \(L2\) values, respectively; the padding indicator is in Line 24.

We start with the \(cosine\) part first, then \(L2\). For each, we check word interaction candidates moving down the sorted list. Function \(calcPos\) is used to calculate relative sentence positions \(pos_{s*}\) in the \(simCube\) given one interaction pair. We follow the constraint that no word in both sentences should be tagged to be important more than once. We increase weights of important word interactions to \(1\) (in Line 11 based on \(cosine\) and Line 21 based on \(L2\)), while unimportant word interactions receive weights of \(0.1\) (in Line 2).

We use a mask matrix, \(mask\), to hold the weights of each. The final output of the similarity focus layer is a focus-weighted similarity cube \(focusCube\), which is the element-wise multiplication (Line 25) of the matrix \(mask\) and the input \(simCube\).

The similarity focus layer is based on the following intuition: given each word in one sentence, we look for its semantically similar twin in the other sentence; if found then this word is considered important, otherwise it contributes to a semantic difference. Though technically different, this process shares conceptual similarity with finding translation equivalences in statistical machine translation [3].

The backward pass of the similarity focus layer is straightforward: we reuse the \(mask\) matrix as generated in the forward pass and apply the element-wise multiplication of \(mask\) and inflow gradients, then propagate the resulting gradients backward.

## 7 Similarity Classification with Deep Convolutional Neural Networks

The \(focusCube\) contains focus-weighted fine-grained similarity information. In the final model component we use the \(focusCube\) to compute the final similarity score. If we treat the \(focusCube\) as an "image" with \(13\) channels, then semantic similarity measurement can be converted into a pattern recognition (image processing) problem, where we are looking for patterns of strong pairwise word interactions in the "image". The stronger the overall pairwise word interactions are, the higher similarity the sentence pair will have.

Recent advances from successful systems at ImageNet competitions [8, 9] show that the depth of a neural network is a critical component for achieving competitive performance. We therefore use a deep homogeneous architecture which has repetitive convolution and pooling layers.

Our network architecture (Table 1) is composed of spatial max pooling layers, spatial convolutional layers (Conv) with a small filter size of \(3\times 3\) plus stride \(1\) and padding \(1\). We adopt this filter size because it is the smallest one to capture the space of left/right, up/down, and center; the padding and stride is used to preserve the spatial input resolution. We then use fully-connected layers followed by the final softmax layer for the output. After each spatial convolutional layer, a rectified linear units (ReLU) non-linearity layer [6] is used.

The input to this deep ConvNet is the \(focusCube\), which does not always have the same size because 

the lengths of input sentences vary. To address this, we use zero padding. For computational reasons we provide two configurations in Table 1, for length padding up to either \(32\times 32\) or \(48\times 48\). The only difference between the two configurations is the last pooling layer. If sentences are longer than the padding length limit we only use the number of words up to the limit. In our experiments we found the \(48\times 48\) padding limit to be acceptable since most sentences in our datasets are only \(1-30\) words long.

## 8 Experimental Setup

**Datasets.** We conducted five separate experiments on ten different datasets: three recent SemEval competitions and two answer selection tasks. Note that the answer selection task, which is to rank candidate answer sentences based on their similarity to the questions, is essentially the similarity measurement problem. The five experiments are as follows:

1. Sentences Involving Compositional Knowledge (SICK) is from Task 1 of the 2014 SemEval competition [7] and consists of 9,927 annotated sentence pairs, with 4,500 for training, 500 as a development set, and 4,927 for testing. Each pair has a relatedness score \(\in[1,5]\) which increases with similarity.
2. Microsoft Video Paraphrase Corpus (MSRVID) is from Task 6 of the 2012 SemEval competition [1] and consists of 1,500 annotated pairs of video descriptions, with half for training and the other half for testing. Each sentence pair has a relatedness score \(\in[0,5]\) which increases with similarity.
3. Task 10 of the 2014 SemEval competition on Semantic Textual Similarity [1] [1] provided six different test sets from different domains. Each pair has a similarity score \(\in[0,5]\) which increases with similarity. Following the competition rules, our training data is only drawn from previous STS competitions in 2012 and 2013. We excluded training sentences with lengths longer than the 48 word padding limit, resulting in 7,382 training pairs out of a total of 7,592. Table 2 provides a brief description of the test sets.
4. The open domain question-answering WikiQA data is from Bing query logs by yang2015bert. We followed the same pre-processing steps as yang2015bert, where questions with no correct candidate answer sentences are excluded and answer sentences are truncated to 40 tokens. The resulting dataset consists of 873 questions with 8,672 question-answer pairs in the training set, 126 questions with 1,130 pairs in the development set, and 243 questions with 2,351 pairs in the test set.
5. The TrecQA dataset [23] from the Text Retrieval Conferences has been widely used for the answer selection task during the past decade. To enable direct comparison with previous work, we used the same training, development, and test sets as released by Yao et al. [2013].

\begin{table}
\begin{tabular}{|c|l|} \hline \multicolumn{2}{|c|}{**Deep ConvNet Configurations**} \\ \hline \hline
**Input Size: \(32\) by \(32\)** & **Input Size: \(48\) by \(48\)** \\ \hline \hline Spatial Conv 128: size \(3\times 3\), stride \(1\), pad \(1\) \\ \hline ReLU \\ \hline Max Pooling: size \(2\times 2\), stride \(2\) \\ \hline Spatial Conv 164: size \(3\times 3\), stride \(1\), pad \(1\) \\ \hline ReLU \\ \hline Max Pooling: size \(2\times 2\), stride \(2\) \\ \hline Spatial Conv 192: size \(3\times 3\), stride \(1\), pad \(1\) \\ \hline ReLU \\ \hline Max Pooling: size \(2\times 2\), stride \(2\) \\ \hline RealU \\ \hline Max Pooling: size \(2\times 2\), stride \(2\) \\ \hline Total = & **3,750** \\ \hline \end{tabular}
\end{table}
Table 1: Deep ConvNet architecture given two padding size configurations for final classification.

\begin{table}
\begin{tabular}{|p{113.8pt}|p{113.8pt}|p{113.8pt}|} \hline
**STS2014** & **Domain** & **Pairs** \\ \hline deft-forum & discussion forums & 450 \\ \hline deft-news & news articles & 300 \\ \hline headlines & news headlines & 750 \\ \hline images & image descriptions & 750 \\ \hline OnWN & word sense definitions & 750 \\ \hline tweet-news & social media & 750 \\ \hline \hline
**Total** & & **3,750** \\ \hline \end{tabular}
\end{table}
Table 2: Description of STS2014 test sets.

 

The TrecQA data consists of 1,229 questions with 53,417 question-answer pairs in the _TRAIN-ALL_ training set, 82 questions with 1,148 pairs in the development set, and 100 questions with 1,517 pairs in the test set.

**Training.** For experiments on SICK, MSRVID, and STS2014, the training objective is to minimize the KL-divergence loss:

\[loss(\theta)=\frac{1}{n}\sum_{k=1}^{n}\mathit{KL}\left(f^{k}~{}||~{}\widehat {f}_{\theta}^{k}\right) \tag{11}\]

where \(f\) is the ground truth, \(\widehat{f}_{\theta}\) is the predicted distribution with model weights \(\theta\), and \(n\) is the number of training examples.

We used a hinge loss for the answer selection task on WikiQA and TrecQA data. The training objective is to minimize the following loss, summed over examples \(\langle x,y_{gold}\rangle\):

\[loss(\theta,x,y_{gold})=\\ \sum_{y^{\prime}\neq y_{gold}}\max(0,1+f_{\theta}(x,y^{\prime})- f_{\theta}(x,y_{gold})) \tag{12}\]

where \(y_{gold}\) is the ground truth label, input \(x\) is the pair of sentences \(x=\{S_{1},S_{2}\}\), \(\theta\) is the model weight vector, and the function \(f_{\theta}(x,y^{\prime})\) is the output of our model.

In all cases, we performed optimization using RMSProp (Tieleman and Hinton, 2012) with backpropagation (Bottou, 1998), with a learning rate fixed to \(10^{-4}\).

**Settings.** For the SICK and MSRVID experiments, we used 300-dimension GloVe word embeddings (Pennington et al., 2014). For the STS2014, WikiQA, and TrecQA experiments, we used 300-dimension paramgram-sl999 embeddings from Wieting et al. (2015) and the paramgram-phrase embeddings from Wieting et al. (2016), trained on word pairs from the Paraphrase Database (PPDB) (Ganitkevitch et al., 2013). We did not update word embeddings in all experiments.

We used the SICK development set for tuning and then applied exactly the _same_ hyperparameters to _all_ ten test sets. For the answer selection task (WikiQA and TrecQA), we used the official trec_eval scorer to compute the metrics Mean Average Precision (MAP) and Mean Reciprocal Rank (MRR) and selected the best development model based on MRR for final testing. Our timing experiments were conducted on an Intel Xeon E5-2680 CPU.

Due to sentence length variations, for the SICK and MSRVID data we padded the sentences to 32 words; for the STS2014, WikiQA, and TrecQA data, we padded the sentences to 48 words.

## 9 Results

**SICK Results** (Table 3). Our model outperforms previous neural network models, most of which are based on sentence modeling. The ConvNet work (He et al., 2015) and TreeLSTM work (Tai et al., 2015) achieve comparable accuracy; for example, their difference in Pearson's \(r\) is only 0.1%. In comparison, our model outperforms both by 1% in Pearson's \(r\), over 1.1% in Spearman's \(\rho\), and 2-3% in MSE. Note that we used the same word embeddings, sparse distribution targets, and loss function as in He et al. (2015) and Tai et al. (2015), thereby representing comparable experimental conditions.

**MSRVID Results** (Table 4). Our model outperforms the work of He et al. (2015), which already reports a Pearson's \(r\) score of over \(0.9\),

**STS2014 Results** (Table 5). Systems in the competition are ranked by the weighted mean (the of

\begin{table}
\begin{tabular}{|l|c|c|c|} \hline

- \\ \hline LSTM & 0.8477 & 0.7921 & 0.2949 \\ Bi-LSTM & 0.8522 & 0.7952 & 0.2850 \\
2-layer LSTM & 0.8411 & 0.7849 & 0.2980 \\
2-layer Bi-LSTM & 0.8488 & 0.7926 & 0.2893 \\ Tai et al. (2015) Const. LSTM & 0.8491 & 0.7873 & 0.2852 \\ Tai et al. (2015) Dep. LSTM & 0.8676 & 0.8083 & 0.2532 \\ \hline He et al. (2015) & 0.8686 & 0.8047 & 0.2606 \\ \hline
**This work** & **0.8784** & **0.8199** & **0.2329** \\ \hline \end{tabular}
\end{table}
Table 3: Test results on SICK grouped as: (1) RNN variants; (2) SemEval 2014 systems; (3) Sequential LSTM variants; (4) Dependency and constituency tree LSTMs. Evaluation metrics are Pearson’s \(r\), Spearman’s \(\rho\), and mean squared error (MSE). Rows in grey are neural network models.

 

ficial measure) of Pearson's \(r\) scores calculated based on the number of sentence pairs in each test set. We show the 1st ranked Sultan et al. (2014), 2nd Kashyap et al. (2014), 3rd Lynum et al. (2014) systems in the STS2014 competition, all of which are based on heavy feature engineering. Our model does not use any sparse features, WordNet, or parse trees, but still performs favorably compared to the STS2014 winning system Sultan et al. (2014).

**WikiQA Results** (Table 6). We compared our model to competitive baselines prepared by Yang et al. (2015) and also evaluated He et al. (2015)'s multi-perspective ConvNet on the same data. The neural network models in the table, paragraph vector (PV) Le and Mikolov (2014), CNN Yu et al. (2014), and PV-Cnt/CNN-Cut with word matching features Yang et al. (2015), are mostly based on sentence modeling. Our model outperforms them all.

**TreeQA Results** (Table 7). This is the largest dataset in our experiments, with over 55,000 question-answer pairs. Only recently have neural network approaches Yu et al. (2014) started to show promising results on this decade-old dataset. Previous approaches with probabilistic tree-edit techniques or tree kernels Wang and Manning (2010); Heilman and Smith (2010); Yao et al. (2013) have been successful since tree structure information permits a fine-grained focus on important words for similarity comparison purposes. Our approach essentially follows this intuition, but in a neural network setting with the use of our similarity focus layer. Our model outperforms previous work.

## 10 Analysis

**Ablation Studies.** Table 8 shows the results of ablation studies on SICK and WikiQA data. We removed or replaced one component at a time from the full system and performed re-training and re-testing. We found large drops when removing the context modeling component, indicating that the context information provided by the Bi-LSTMs is crucial for the following components (e.g., interaction modeling). The use of our similarity focus layer is also beneficial, especially on the WikiQA data. When we replaced the entire similarity focus layer with a random dropout layer (\(p=0.3\)), the dropout layer hurts accuracy; this shows the importance of directing the model to focus on important pairwise word interactions, to better capture similarity.

**Model Efficiency and Storage.**He et al. (2015)'s

\begin{table}
\begin{tabular}{|l|c|c|} \hline
**Model** & **MAP** & **MRR** \\ \hline \hline   Vori et al. (2005) & 0.4271 & 0.5259 \\ \hline   Wang et al. (2007) & 0.6029 & 0.6852 \\ \hline   Heilman and Smith (2010) & 0.6091 & 0.6917 \\ \hline   Wang and Manning (2010) & 0.5951 & 0.6951 \\ \hline   Yao et al. (2013) & 0.6307 & 0.7477 \\ \hline   Severyn and Moschitti (2013) & 0.6781 & 0.7358 \\ \hline   Yih et al. (2013) & 0.7092 & 0.7700 \\ \hline   Wang and Nyberg (2015) & 0.7134 & 0.7913 \\ \hline   Severyn and Moschitti (2015) & 0.7459 & 0.8078 \\ \hline   This **work** & **0.7588** & **0.8219** \\ \hline  & \multicolumn{1}{c}{} & \multicolumn{1}{c}{} \\ \end{tabular}
\end{table}
Table 6: Test results on WikiQA data.

\begin{table}
\begin{tabular}{|l|c|} \hline
**Model** & **Pearson’s \(r\)** \\ \hline \hline Beltagy et al. (2014) & 0.8300 \\ \hline Bär et al. (2012) & 0.8730 \\ \hline Saric et al. (2012) & 0.8803 \\ \hline He et al. (2015) & 0.9090 \\ \hline
**This work** & **0.9112** \\ \hline \end{tabular}
\end{table}
Table 4: Test results on MSRVID data.

\begin{table}
\begin{tabular}{|l|c|c|} \hline
**Model** & **MAP** & **MRR** \\ \hline \hline   Vori et al. (2005) & 0.4271 & 0.5259 \\ \hline   Wang et al. (2007) & 0.6029 & 0.6852 \\ \hline   Heilman and Smith (2010) & 0.6091 & 0.6917 \\ \hline   Wang and Manning (2010) & 0.5951 & 0.6951 \\ \hline   Yao et al. (2013) & 0.6307 & 0.7477 \\ \hline   Severyn and Moschitti (2013) & 0.6781 & 0.7358 \\ \hline   Yih et al. (2013) & 0.7092 & 0.7700 \\ \hline   Wang and Nyberg (2015) & 0.7134 & 0.7913 \\ \hline   Severyn and Moschitti (2015) & 0.7459 & 0.8078 \\ \hline   This **work** & **0.7588** & **0.8219** \\ \hline \end{tabular}
\end{table}
Table 7: Test results on TreeQA data.

\begin{table}
\begin{tabular}{|l|c|c|c|} \hline
**STS2014** & **3rd** & **2nd** & **1st** & **This work** \\ \hline \hline   deft-forum & 0.5305 & 0.4711 & 0.4828 & **0.5684** \\ \hline   deft-news & **0.7813** & 0.7628 & 0.7657 & 0.7079 \\ \hline headlines & **0.7837** & 0.7597 & 0.7646 & 0.7551 \\ \hline image & **0.8343** & 0.8013 & 0.8214 & 0.8221 \\ \hline OnWN & 0.8502 & 0.8745 & 0.8589 & **0.8847** \\ \hline tweetnews & 0.6755 & **0.7793** & 0.7639 & 0.7469 \\ \hline
**Wt. Mean** & 0.7549 & 0.7605 & 0.7610 & **0.7666** \\ \hline \end{tabular}
\end{table}
Table 5: Test results on all six test sets in STS2014.

 

ConvNet model uses multiple types of convolution and pooling for sentence modeling. This results in a wide architecture with around \(10\) million tunable parameters. Our approach only models pairwise word interactions and does not require such a complicated architecture. Compared to that previous work, Table 9 shows that our model is 3.4\(\times\) faster in training and has 83% fewer tunable parameters.

**Visualization.** Table 10 visualizes the cosine value channel of the \(focusCube\) for pairwise word interactions given two sentence pairs in the SICK test set. Note for easier visualization, the values are multiplied by \(10\). Darker red areas indicate stronger pair-wise word interactions. From these visualizations, we see that our model is able to identify important word pairs (in dark red) and tag them with proper similarity values, which are significantly higher than the ones of their neighboring unimportant pairs. This shows that our model is able to recognize important fine-grained word-level information for better similarity measurement, suggesting the reason why our model performs well.

## 11 Conclusion

In summary, we developed a novel neural network model based on a hybrid of ConvNet and Bi-LSTMs for the semantic textual similarity measurement problem. Our pairwise word interaction model and the similarity focus layer can better capture fine-grained semantic information, compared to previous sentence modeling approaches that attempt to "cram" all sentence information into a fixed-length vector. We demonstrated the state-of-the-art accuracy of our approach on data from three SemEval competitions and two answer selection tasks.

#### Acknowledgments

This work was supported by the U.S. National Science Foundation under awards IIS-1218043 and CNS-1405688. Any opinions, findings, conclusions, or recommendations expressed are those of the authors and do not necessarily reflect the views of the sponsor. We would like to thank Kevin Gimpel, John Wieting and TS for all the support.

## References

* Agirre et al. [2012] Eneko Agirre, Mona Diab, Daniel Cer, and Aitor Gonzalez-Agirre. 2012. SemEval-2012 task 6: A pilot on semantic textual similarity. In _Proceedings of the First Joint Conference on Lexical and Computational Semantics_, pages 385-393.
* Agirre et al. [2017] Eneko Agirre, Carmen Banea, Claire Cardie, Daniel Cer, Mona Diab, Aitor Gonzalez-Agirre, Weiwei Guo, Rada Mihalcea, German Rigau, and Janyce Wiebe.

\begin{table}
\begin{tabular}{l c c c c c c} \hline  & A & man & is & playing & the & drum \\ A & 8.99 & 0.69 & 0.43 & 0.32 & 0.38 & 0.22 \\ man & 0.70 & 0.03 & 0.62 & 0.45 & 0.46 & 0.38 \\ is & 0.64 & 0.80 & 8.50 & 0.62 & 0.58 & 0.36 \\ practicing & 0.46 & 0.67 & 0.66 & 6.51 & 0.62 & 0.48 \\ the & 0.35 & 0.56 & 0.66 & 0.64 & 7.85 & 0.52 \\ drum & 0.27 & 0.47 & 0.46 & 0.55 & 0.64 & 8.52 \\ \hline  & A & man & is & carrying & a & tree \\ A & 0.53 & 0.33 & 0.32 & 0.33 & 5.53 & 0.49 \\ tree & 0.32 & 0.30 & 0.19 & 0.20 & 0.38 & 8.73 \\ is & 0.35 & 0.31 & 0.21 & 0.06 & 0.03 & 0.40 \\ being & 0.28 & 0.37 & 2.60 & 0.18 & 0.13 & 0.38 \\ picked & 0.15 & 0.18 & 0.10 & 1.60 & 0.07 & 0.27 \\ up & 0.26 & 0.27 & 0.06 & 0.13 & 0.05 & 0.21 \\ by & 0.43 & 0.36 & 0.08 & 1.33 & 0.15 & 0.29 \\ a & 6.50 & 0.45 & 0.03 & 0.08 & 0.16 & 0.23 \\ man & 0.50 & 8.40 & 0.45 & 0.34 & 0.34 & 0.34 \\ \hline \end{tabular}
\end{table}
Table 10: Visualization of cosine values (multiplied by 10) in the \(focusCube\) given two sentence pairs in the SICK test set.

\begin{table}
\begin{tabular}{|p{113.8pt}|p{113.8pt}|} \hline
* Remove context modeling (Sec. 4)
- Remove entire focus layer (Sec. 6)
- Replace entire focus layer with dropout
- Roplace entire focus layer with dropout & -0.0314 \\ \hline \hline
* Remove context modeling (Sec. 4)
- Remove entire focus layer (Sec. 6)
- Replace entire focus layer with dropout & -0.0403 \\ \hline \end{tabular}
\end{table}
Table 8: Ablation studies on SICK and WikiQA data, removing each component separately.
 

2014. SemEval-2014 task 10: Multilingual semantic textual similarity. In _Proceedings of the 8th International Workshop on Semantic Evaluation_, pages 81-91.
* Al-onaizan et al. (1999) Yaser Al-onaizan, Jan Curin, Michael Jahr, Kevin Knight, John Lafferty, Dan Melamed, Franz-Josef Och, David Purdy, Noah A. Smith, and David Yarowsky. 1999. Statistical machine translation. Final report, JHU Summer Workshop on Language Engineering.
* Andrew et al. (2013) Galen Andrew, Raman Arora, Jeff Bilmes, and Karen Livescu. 2013. Deep canonical correlation analysis. In _Proceedings of the 30th International Conference on Machine Learning_, pages 1247-1255.
* Bahdanau et al. (2014) Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2014. Neural machine translation by jointly learning to align and translate. _CoRR_, abs/1409.0473.
* Bar et al. (2012) Daniel Bar, Chris Biemann, Iryna Gurevych, and Torsten Zesch. 2012. UKP: Computing semantic textual similarity by combining multiple content similarity measures. In _Proceedings of the First Joint Conference on Lexical and Computational Semantics_, pages 435-440.
* Beltagy et al. (2014) Islam Beltagy, Katrin Erk, and Raymond Mooney. 2014. Probabilistic soft logic for semantic textual similarity. In _Proceedings of 52nd Annual Meeting of the Association for Computational Linguistics_, pages 1210-1219.
* Bjerva et al. (2014) Johannes Bjerva, Johan Bos, Rob van der Gotot, and Malvina Nissim. 2014. The meaning factory: Formal semantics for recognizing textual entailment and determining semantic similarity. In _Proceedings of the 8th International Workshop on Semantic Evaluation_, pages 642-646.
* Bottou (1998) Leon Bottou. 1998. Online learning and stochastic approximations. In David Saad, editor, _Online Learning and Neural Networks_. Cambridge University Press.
* Bromley et al. (1993) Jane Bromley, James W Bentz, Leon Bottou, Isabelle Guyon, Yann LeCun, Cliff Moore, Eduard Sackinger, and Roopak Shah. 1993. Signature verification using a "Siamese" time delay neural network. _International Journal of Pattern Recognition and Artificial Intelligence_, 7(4):669-688.
* Burges et al. (2005) Chris Burges, Tal Shaked, Erin Renshaw, Ari Lazier, Matt Deeds, Nicole Hamilton, and Greg Hullender. 2005. Learning to rank using gradient descent. In _Proceedings of the 22nd International Conference on Machine Learning_, pages 89-96.
* Cui et al. (2005) Hang Cui, Renxu Sun, Keya Li, Min-Yen Kan, and TatSeng Chua. 2005. Question answering passage retrieval using dependency relations. In _Proceedings of the 28th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval_, pages 400-407.
* Das and Smith (2009) Dipanjan Das and Noah A. Smith. 2009. Paraphrase identification as probabilistic quasi-synchronous recognition. In _Proceedings of the 47th Annual Meeting of the Association for Computational Linguistics_, pages 468-476.
* Fellbaum (1998) Christiane Fellbaum. 1998. _WordNet: An Electronic Lexical Database_. MIT Press.
* Fern and Stevenson (2008) Samuel Fern and Mark Stevenson. 2008. A semantic similarity approach to paraphrase detection. In _Proceedings of the 11th Annual Research Colloquium of the UK Special-Interest Group for Computational Linguistics_, pages 45-52.
* Filippova et al. (2015) Katja Filippova, Enrique Alfonseca, Carlos A. Colmenares, Lukasz Kaiser, and Oriol Vinyals. 2015. Sentence compression by deletion with LSTMs. In _Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing_, pages 360-368.
* Ganitkevitch et al. (2013) Juri Ganitkevitch, Benjamin Van Durme, and Chris Callison-Burch. 2013. PPDB: The Paraphrase Database. In _Proceedings of the 2013 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_, pages 758-764.
* Part II_, pages 799-804.
* Graves et al. (2006) Alex Graves, Santiago Fernandez, Faustino Gomez, and Jurgen Schmidhuber. 2006. Connectionist temporal classification: Labelling unsegmented sequence data with recurrent neural networks. In _Proceedings of the 23rd International Conference on Machine Learning_, pages 369-376.
* He et al. (2015) Hua He, Kevin Gimpel, and Jimmy Lin. 2015. Multiperspective sentence similarity modeling with convolutional neural networks. In _Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing_, pages 1576-1586.
* Heilman and Smith (2010) Michael Heilman and Noah A. Smith. 2010. Tree edit models for recognizing textual entailments, paraphrases, and answers to questions. In _Human Language Technologies: The 2010 Annual Conference of the North American Chapter of the Association for Computational Linguistics_, pages 1011-1019.
* Hochreiter and Schmidhuber (1997) Sepp Hochreiter and Jurgen Schmidhuber. 1997. Long short-term memory. _Neural Computation_, 9(8):1735-1780.
* Hu et al. (2014) Baotian Hu, Zhengdong Lu, Hang Li, and Qingcai Chen. 2014. Convolutional neural network architectures for matching natural language sentences. In _Advances 

in Neural Information Processing Systems 27_, pages 2042-2050.
* Huang et al. (2013) Po-Sen Huang, Xiaodong He, Jianfeng Gao, Li Deng, Alex Acero, and Larry Heck. 2013. Learning deep structured semantic models for web search using clickthrough data. In _Proceedings of the 22nd ACM International Conference on Information & Knowledge Management_, pages 2333-2338.
* Jimenez et al. (2014) Sergio Jimenez, George Duenas, Julia Baquero, Alexander Gelbukh, Av Juan Dios Batiz, and Av Mendizabal. 2014. UNAL-NLP: Combining soft cardinality features for semantic textual similarity, relatedness and entailment. In _Proceedings of the 8th International Workshop on Semantic Evaluation_, pages 732-742.
* Kashyap et al. (2014) Abhay Kashyap, Lushan Han, Roberto Yus, Jennifer Sleeman, Taneeya Satyapanich, Sunil Gandhi, and Tim Finin. 2014. Meerkat mafia: Multilingual and cross-level semantic textual similarity systems. In _Proceedings of the 8th International Workshop on Semantic Evaluation_, pages 416-423.
* Krizhevsky et al. (2012) Alex Krizhevsky, Ilya Sutskever, and Geoffrey E. Hinton. 2012. ImageNet classification with deep convolutional neural networks. In _Advances in Neural Information Processing Systems 25_, pages 1097-1105.
* Lai and Hockenmaier (2014) Alice Lai and Julia Hockenmaier. 2014. Illinois-LH: A denotational and distributional approach to semantics. In _Proceedings of the 8th International Workshop on Semantic Evaluation_, pages 329-334.
* Le and Mikolov (2014) Quoc V. Le and Tomas Mikolov. 2014. Distributed representations of sentences and documents. In _Proceedings of the 31th International Conference on Machine Learning_, pages 1188-1196.
* Lin (2007) Jimmy Lin. 2007. An exploration of the principles underlying redundancy-based factoid question answering. _ACM Transactions on Information Systems_, 25(2):1-55.
* Lynum et al. (2014) Andre Lynum, Partha Pakray, Bjorn Gamback, and Sergio Jimenez. 2014. NTNU: Measuring semantic similarity with sublexical feature representations and soft cardinality. In _Proceedings of the 8th International Workshop on Semantic Evaluation_, pages 448-453.
* Madnani et al. (2012) Nitin Madnani, Joel Tetreault, and Martin Chodorow. 2012. Re-examining machine translation metrics for paraphrase identification. In _Proceedings of the 2012 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_, pages 182-190.
* Marelli et al. (2014) Marco Marelli, Luisa Bentivogli, Marco Baroni, Rafaella Bernardi, Stefano Menini, and Roberto Zamparelli. 2014. SemEval-2014 task 1: Evaluation of compositional distributional semantic models on full sentences through semantic relatedness and textual entailment. In _Proceedings of the 8th International Workshop on Semantic Evaluation_, pages 1-8.
* Mooney (2014) Raymond J. Mooney. 2014. Semantic parsing: Past, present, and future. In _ACL Workshop on Semantic Parsing Presentation slides_.
* Pennington et al. (2014) Jeffrey Pennington, Richard Socher, and Christopher D. Manning. 2014. GloVe: Global vectors for word representation. In _Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing_, pages 1532-1543.
* Rush et al. (2015) Alexander M. Rush, Sumit Chopra, and Jason Weston. 2015. A neural attention model for abstractive sentence summarization. _CoRR_, abs/1509.00685.
* Saric et al. (2012) Frane Saric, Goran Glavac, Miladen Karan, Jan Snajder, and Bojana Dalbelo Basic. 2012. TakeLab: systems for measuring semantic text similarity. In _Proceedings of the First Joint Conference on Lexical and Computational Semantics_, pages 441-448.
* Severyn and Moschitti (2013) Aliaksei Severyn and Alessandro Moschitti. 2013. Automatic feature engineering for answer selection and extraction. In _Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing_, pages 458-467.
* Severyn and Moschitti (2015) Aliaksei Severyn and Alessandro Moschitti. 2015. Learning to rank short text pairs with convolutional deep neural networks. In _Proceedings of the 38th International ACM SIGIR Conference on Research and Development in Information Retrieval_, pages 373-382.
* Simonyan and Zisserman (2014) Karen Simonyan and Andrew Zisserman. 2014. Very deep convolutional networks for large-scale image recognition. _CoRR_, abs/1409.1556.
* Socher et al. (2011) Richard Socher, Eric H. Huang, Jeffrey Pennington, Andrew Y. Ng, and Christopher D. Manning. 2011. Dynamic pooling and unfolding recursive autoencoders for paraphrase detection. In _Advances in Neural Information Processing Systems 24_, pages 801-809.
* Socher et al. (2014) Richard Socher, Andrej Karpathy, Quoc V. Le, Christopher D. Manning, and Andrew Y. Ng. 2014. Grounded compositional semantics for finding and describing images with sentences. _Transactions of the Association for Computational Linguistics_, 2:207-218.
* Sultan et al. (2014) Md Arafat Sultan, Steven Bethard, and Tamara Sumner. 2014. Dis\(@\)cu: Sentence similarity from word alignment. In _Proceedings of the 8th International Workshop on Semantic Evaluation_, pages 241-246.
* Sutskever et al. (2014) Ilya Sutskever, Oriol Vinyals, and Quoc V Le. 2014. Sequence to sequence learning with neural networks. In _Advances in Neural Information Processing Systems 27_, pages 3104-3112.
* Szegedy et al. (2015) Christian Szegedy, Wei Liu, Yangqing Jia, Pierre Sermanet, Scott Reed, Dragomir Anguelov, Dumitru Erhan, Vincent Vanhoucke, and Andrew Rabinovich.

 

2015. Going deeper with convolutions. In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_, pages 1-9.
* Tai et al. (2015) Kai Sheng Tai, Richard Socher, and Christopher D. Manning. 2015. Improved semantic representations from tree-structured long short-term memory networks. In _Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics_, pages 1556-1566.
* Tieleman and Hinton (2012) Tijmen Tieleman and Geoffrey E. Hinton. 2012. Lecture 6.5--RMSProp: Divide the gradient by a running average of its recent magnitude. Coursera: Neural Networks for Machine Learning.
* Wan et al. (2006) Stephen Wan, Mark Dras, Robert Dale, and Cecile Paris. 2006. Using Dependency-based Features to Take the "Para-farce" out of Paraphrase. In _Australasian Language Technology Workshop_, pages 131-138.
* Wang and Manning (2010) Mengqiu Wang and Christopher D. Manning. 2010. Probabilistic tree-edit models with structured latent variables for textual entailment and question answering. In _Proceedings of the 23rd International Conference on Computational Linguistics_, pages 1164-1172.
* Wang and Nyberg (2015) Di Wang and Eric Nyberg. 2015. A long short-term memory model for answer sentence selection in question answering. In _Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics_, pages 707-712.
* Wang et al. (2007) Mengqiu Wang, Noah A. Smith, and Teruko Mitamura. 2007. What is the Jeopardy model? A quasi-synchronous grammar for QA. In _Proceedings of the 2007 Joint Conference on Empirical Methods in Natural Language Processing and Computational Natural Language Learning_, pages 22-32.
* Weston et al. (2011) Jason Weston, Samy Bengio, and Nicolas Usunier. 2011. Wsabie: Scaling up to large vocabulary image annotation. In _Proceedings of the 22nd International Joint Conference on Artificial Intelligence_, pages 2764-2770.
* Wieting et al. (2015) John Wieting, Mohit Bansal, Kevin Gimpel, Karen Livescu, and Dan Roth. 2015. From paraphrase database to compositional paraphrase model and back. _Transactions of the Association for Computational Linguistics_, 3:345-358.
* Wieting et al. (2016) John Wieting, Mohit Bansal, Kevin Gimpel, and Karen Livescu. 2016. Towards universal paraphrastic sentence embeddings. In _Proceedings of the 4th International Conference on Learning Representations_.
* Williams and Zipser (1989) Ronald J. Williams and David Zipser. 1989. A learning algorithm for continually running fully recurrent neural networks. _Neural Computation_, 1(2):270-280.
* Xu et al. (2014) Wei Xu, Alan Ritter, Chris Callison-Burch, William B. Dolan, and Yangfeng Ji. 2014. Extracting lexically divergent paraphrases from Twitter. _Transactions of the Association for Computational Linguistics_, 2:435-448.
* Xu (2014) Wei Xu. 2014. _Data-Drive Approaches for Paraphrasing Across Language Variations_. Ph.D. thesis, Department of Computer Science, New York University.
* Yang et al. (2015) Yi Yang, Wen-tau Yih, and Christopher Meek. 2015. WikiQA: A challenge dataset for open-domain question answering. In _Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing_, pages 2013-2018.
* Yao et al. (2013) Xuchen Yao, Benjamin Van Durme, Chris Callison-burch, and Peter Clark. 2013. Answer extraction as sequence tagging with tree edit distance. In _Proceedings of the 2013 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_, pages 858-867.
* Yih et al. (2013) Wentau Yih, Ming-Wei Chang, Christopher Meek, and Andrzej Pastusiak. 2013. Question answering using enhanced lexical semantic models. In _Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics_, pages 1744-1753.
* Yin and Schutze (2015) Wenpeng Yin and Hinrich Schutze. 2015. Convolutional neural network for paraphrase identification. In _Proceedings of the 2015 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_, pages 901-911.
* Yu et al. (2014) Lei Yu, Karl Moritz Hermann, Phil Blunsom, and Stephen Pulman. 2014. Deep Learning for Answer Sentence Selection. In _NIPS Deep Learning Workshop_.
* Zarrella et al. (2015) Guido Zarrella, John Henderson, Elizabeth M. Merkhofer, and Laura Strickhart. 2015. MITRE: Seven systems for semantic similarity in tweets. In _Proceedings of the 9th International Workshop on Semantic Evaluation_, pages 12-17.
* Zhao et al. (2014) Jiang Zhao, Tian Tian Zhu, and Man Lan. 2014. ECNU: One stone two birds: Ensemble of heterogenous measures for semantic relatedness and textual entailment. In _Proceedings of the 8th International Workshop on Semantic Evaluation_, pages 271-277.
* Zhu et al. (2015) Xiaodan Zhu, Parinaz Sobhani, and Hongyu Guo. 2015. Long short-term memory over recursive structures. In _Proceedings of the 32nd International Conference on Machine Learning_, pages 1604-1612