Venue: ACL 2017

# Revisiting Recurrent Networks for Paraphrastic Sentence Embeddings

John Wieting Kevin Gimpel

Toyota Technological Institute at Chicago, Chicago, IL, 60637, USA

{jwieting,kgimpel}@ttic.edu

###### Abstract

We consider the problem of learning general-purpose, paraphrastic sentence embeddings, revisiting the setting of wieting2016b. While they found LSTM recurrent networks to underperform word averaging, we present several developments that together produce the opposite conclusion. These include training on sentence pairs rather than phrase pairs, averaging states to represent sequences, and regularizing aggressively. These improve LSTMs in both transfer learning and supervised settings. We also introduce a new recurrent architecture, the gated recurrent average network, that is inspired by averaging and LSTMs while outperforming them both. We analyze our learned models, finding evidence of preferences for particular parts of speech and dependency relations. 1

Footnote 1: Trained models and code are available at [http://ttic.uchicago.edu/~wieting](http://ttic.uchicago.edu/~wieting).

## 1 Introduction

Modeling sentential compositionality is a fundamental aspect of natural language semantics. Researchers have proposed a broad range of compositional functional architectures Mitchell and Lapata (2008); Socher et al. (2011); Kalchbrenner et al. (2014) and evaluated them on a large variety of applications. Our goal is to learn a general-purpose sentence embedding function that can be used unmodified for measuring semantic textual similarity (STS) Agirre et al. (2012) and can also serve as a useful initialization for downstream tasks. We wish to learn this embedding function such that sentences with high semantic similarity have high cosine similarity in the embedding space. In particular, we focus on the setting of wieting2016b, in which models are trained on noisy paraphrase pairs and evaluated on both STS and supervised semantic tasks.

Surprisingly, wieting et al. found that simple embedding functions--those based on averaging word vectors--outperform more powerful architectures based on long short-term memory (LSTM) Hochreiter and Schmidhuber (1997). In this paper, we revisit their experimental setting and present several techniques that together improve the performance of the LSTM to be superior to word averaging.

We first change data sources: rather than train on noisy phrase pairs from the Paraphrase Database (PPDB; Ganitkevitch et al. (2013), we use noisy _sentence_ pairs obtained automatically by aligning Simple English to standard English Wikipedia Coster and Kauchak (2011). Even though this data was intended for use by text simplification systems, we find it to be efficient and effective for learning sentence embeddings, outperforming much larger sets of examples from PPDB.

We then show how we can modify and regularize the LSTM to further improve its performance. The main modification is to simply average the hidden states instead of using the final one. For regularization, we experiment with two kinds of dropout and also with randomly scrambling the words in each input sequence. We find that these techniques help in the transfer learning setting and on two supervised semantic similarity datasets as well. Further gains are obtained on the supervised tasks by initializing with our models from the transfer setting.

Inspired by the strong performance of both averaging and LSTMs, we introduce a novel recurrent neural network architecture which we call

the gated recurrent averaging network (GRAN). The GRAN outperforms averaging and the LSTM in both the transfer and supervised learning settings, forming a promising new recurrent architecture for semantic modeling.

## 2 Related Work

Modeling sentential compositionality has received a great deal of attention in recent years. A comprehensive survey is beyond the scope of this paper, but we mention popular functional families: neural bag-of-words models Kalchbrenner et al. (2014), deep averaging networks (DANs) Iyyer et al. (2015), recursive neural networks using syntactic parses Socher et al. (2011, 2012, 2013); Irsoy and Cardie (2014), convolutional neural networks Kalchbrenner et al. (2014); Kim (2014); Hu et al. (2014), and recurrent neural networks using long short-term memory Tai et al. (2015); Ling et al. (2015); Liu et al. (2015). Simple operations based on vector addition and multiplication typically serve as strong baselines Mitchell and Lapata (2008, 2010); Blacoe and Lapata (2012).

Most work cited above uses a supervised learning framework, so the composition function is learned discriminatively for a particular task. In this paper, we are primarily interested in creating general purpose, domain independent embeddings for word sequences. Several others have pursued this goal Socher et al. (2011); Le and Mikolov (2014); Pham et al. (2015); Kiros et al. (2015); Hill et al. (2016); Arora et al. (2017); Pagliardini et al. (2017), though usually with the intent to extract useful features for supervised sentence tasks rather than to capture semantic similarity.

An exception is the work of Wieting et al. (2016). We closely follow their experimental setup and directly address some outstanding questions in their experimental results. Here we briefly summarize their main findings and their attempts at explaining them. They made the surprising discovery that word averaging outperforms LSTMs by a wide margin in the transfer learning setting. They proposed several hypotheses for why this occurs. They first considered that the LSTM was unable to adapt to the differences in sequence length between phrases in training and sentences in test. This was ruled out by showing that neither model showed any strong correlation between sequence length and performance on the test data.

They next examined whether the LSTM was overfitting on the training data, but then showed that both models achieve similar values of the training objective and similar performance on _in-domain_ held-out test sets. Lastly, they considered whether their hyperparameters were inadequately tuned, but extensive hyperparameter tuning did not change the story. Therefore, the reason for the performance gap, and how to correct it, was left as an open problem. This paper takes steps toward addressing that problem.

## 3 Models and Training

### Models

Our goal is to embed a word sequence \(s\) into a fixed-length vector. We focus on three compositional models in this paper, all of which use words as the smallest unit of compositionality. We denote the \(t\)th word in \(s\) as \(s_{t}\), and we denote its word embedding by \(x_{t}\).

Our first two models have been well-studied in prior work, so we describe them briefly. The first, which we call avg, simply averages the embeddings \(x_{t}\) of all words in \(s\). The only parameters learned in this model are those in the word embeddings themselves, which are stored in the word embedding matrix \(W_{w}\). This model was found by Wieting et al. (2016) to perform very strongly for semantic similarity tasks.

Our second model uses a long short-term memory (LSTM) recurrent neural network Hochreiter and Schmidhuber (1997) to embed \(s\). We use the LSTM variant from Gers et al. (2003) including its "peephole" connections. We consider two ways to obtain a sentence embedding from the LSTM. The first uses the final hidden vector, which we denote \(h_{-1}\). The second, denoted LSTMayG, averages all hidden vectors of the LSTM. In both variants, the learnable parameters include both the LSTM parameters \(W_{c}\) and the word embeddings \(W_{w}\).

Inspired by the success of the two models above, we propose a third model, which we call the gated recurrent averaging network (GRAN). The gated recurrent averaging network combines the benefits of avg and LSTMs. In fact it reduces to avg if the output of the gate is all ones. We first use an LSTM to generate a hidden vector, \(h_{t}\), for each word \(s_{t}\) in \(s\). Then we use \(h_{t}\) to compute a gate that will be elementwise-multiplied with \(x_{t}\), resulting in a new, gated hidden vector \(a_{t}\) for each step \(t\):

\[a_{t}=x_{t}\odot\sigma(W_{x}x_{t}+W_{h}h_{t}+b)\] 

where \(W_{x}\) and \(W_{h}\) are parameter matrices, \(b\) is a parameter vector, and \(\sigma\) is the elementwise logistic sigmoid function. After all \(a_{t}\) have been generated for a sentence, they are averaged to produce the embedding for that sentence. This model includes as learnable parameters those of the LSTM, the word embeddings, and the additional parameters in Eq. (1). For both the LSTM and GRAN models, we use \(W_{c}\) to denote the "compositional" parameters, i.e., all parameters other than the word embeddings.

The motivation for the GRAN is that we are contextualizing the word embeddings prior to averaging. The gate can be seen as an attention, attending to the prior context of the sentence.2

Footnote 2: We tried a variant of this model without the gate. We obtain \(a_{t}\) from \(f(W_{x}x_{t}+W_{h}h_{t}+b)\), where \(f\) is a nonlinearity, tuned over tanh and ReLU. The performance of the model is significantly worse than the GRAN in all experiments.

We also experiment with four other variations of this model, though they generally were more complex and showed inferior performance. In the first, GRAN-2, the gate is applied to \(h_{t}\) (rather than \(x_{t}\)) to produce \(a_{t}\), and then these \(a_{t}\) are averaged as before.

GRAN-3 and GRAN-4 use two gates: one applied to \(x_{t}\) and one applied to \(a_{t-1}\). We tried two different ways of computing these gates: for each gate \(i\), \(\sigma(W_{x_{i}}x_{t}+W_{h_{i}}h_{t}+b_{i})\) (GRAN-3) or \(\sigma(W_{x_{i}}x_{t}+W_{h_{i}}h_{t}+W_{a_{i}}a_{t-1}+b_{i})\) (GRAN-4). The sum of these two terms comprised \(a_{t}\). In this model, the last average hidden state, \(a_{-1}\), was used as the sentence embedding after dividing it by the length of the sequence. In these models, we are additionally keeping a running average of the embeddings that is being modified by the context at every time step. In GRAN-4, this running average is also considered when producing the contextualized word embedding.

Lastly, we experimented with a fifth GRAN, GRAN-5, in which we use two gates, calculated by \(\sigma(W_{x_{i}}x_{t}+W_{h_{i}}h_{t}+b_{i})\) for each gate \(i\). The first is applied to \(x_{t}\) and the second is applied to \(h_{t}\). The output of these gates is then summed. Therefore GRAN-5 can be reduced to either word-averaging or averaging LSTM states, depending on the behavior of the gates. If the first gate is all ones and the second all zeros throughout the sequence, the model is equivalent to word-averaging. Conversely, if the first gate is all zeros and the second is all ones throughout the sequence, the model is equivalent to averaging the LSTM states. Further analysis of these models is included in Section 4.

### Training

We follow the training procedure of Wieting et al. (2015) and Wieting et al. (2016), described below. The training data consists of a set \(S\) of phrase or sentence pairs \(\langle s_{1},s_{2}\rangle\) from either the Paraphrase Database (PPDB; Ganitkevitch et al., 2013) or the aligned Wikipedia sentences Coster and Kauchak (2011) where \(s_{1}\) and \(s_{2}\) are assumed to be paraphrases. We optimize a margin-based loss:

\[\underset{W_{x}^{*},W_{w}}{\text{min}}\frac{1}{|S|}\bigg{(}\sum _{(s_{1},s_{2})\in S}\max(0,\delta-\cos(g(s_{1}),g(s_{2}))\] \[+\cos(g(s_{1}),g(t_{1})))+\max(0,\delta-\cos(g(s_{1}),g(s_{2}))\] \[+\cos(g(s_{2}),g(t_{2})))\Big{)}+\lambda_{c}\left\lVert W_{c} \right\rVert^{2}+\lambda_{w}\left\lVert W_{w_{wind}}-W_{v}\right\rVert^{2} \tag{2}\]

where \(g\) is the model in use (e.g., avg or LSTM), \(\delta\) is the margin, \(\lambda_{c}\) and \(\lambda_{w}\) are regularization parameters, \(W_{winital}\) is the initial word embedding matrix, and \(t_{1}\) and \(t_{2}\) are carefully-selected negative examples taken from a mini-batch during optimization. The intuition is that we want the two phrases to be more similar to each other (\(\cos(g(s_{1}),g(s_{2}))\)) than either is to their respective negative examples \(t_{1}\) and \(t_{2}\), by a margin of at least \(\delta\).

#### 3.2.1 Selecting Negative Examples

To select \(t_{1}\) and \(t_{2}\) in Eq. (2), we simply choose the most similar phrase in some set of phrases (other than those in the given phrase pair). For simplicity we use the mini-batch for this set, but it could be a different set. That is, we choose \(t_{1}\) for a given \(\langle s_{1},s_{2}\rangle\) as follows:

\[t_{1}=\operatornamewithlimits{argmax}_{t:\langle t,\cdot\rangle\in S_{b} \backslash\{\langle s_{1},s_{2}\rangle\}}\cos(g(s_{1}),g(t))\]

where \(S_{b}\subseteq S\) is the current mini-batch. That is, we want to choose a negative example \(t_{i}\) that is similar to \(s_{i}\) according to the current model. The downside is that we may occasionally choose a phrase \(t_{i}\) that is actually a true paraphrase of \(s_{i}\).

## 4 Experiments

Our experiments are designed to address the empirical question posed by Wieting et al. (2016): why do LSTMs underperform avg for transfer 

learning? In Sections 4.1.2-4.2, we make progress on this question by presenting methods that bridge the gap between the two models in the transfer setting. We then apply these same techniques to improve performance in the supervised setting, described in Section 4.3. In both settings we also evaluate our novel GRAN architecture, finding it to consistently outperform both avg and the LSTM.

### Transfer Learning

#### 4.1.1 Datasets and Tasks

We train on large sets of noisy paraphrase pairs and evaluate on a diverse set of 22 textual similarity datasets, including all datasets from every SemEval semantic textual similarity (STS) task from 2012 to 2015. We also evaluate on the SemEval 2015 Twitter task Xu et al. (2015) and the SemEval 2014 SICK Semantic Relatedness task Marelli et al. (2014). Given two sentences, the aim of the STS tasks is to predict their similarity on a 0-5 scale, where 0 indicates the sentences are on different topics and 5 indicates that they are completely equivalent. We report the average Pearson's \(r\) over these 22 sentence similarity tasks.

Each STS task consists of 4-6 datasets covering a wide variety of domains, including newswire, tweets, glosses, machine translation outputs, web forums, news headlines, image and video captions, among others. Further details are provided in the official task descriptions Agirre et al. (2012, 2013, 2014, 2015).

#### 4.1.2 Experiments with Data Sources

We first investigate how different sources of training data affect the results. We try two data sources. The first is phrase pairs from the Paraphrase Database (PPDB). PPDB comes in different sizes (S, M, L, XL, XXL, and XXXL), where each larger size subsumes all smaller ones. The pairs in PPDB are sorted by a confidence measure and so the smaller sets contain higher precision paraphrases. PPDB is derived automatically from naturally-occurring bilingual text, and versions of PPDB have been released for many languages without the need for any manual annotation Ganitkevitch and Callison-Burch (2014).

The second source of data is a set of sentence pairs automatically extracted from Simple English Wikipedia and English Wikipedia articles by Coster and Kauchak (2011). This data was extracted for developing text simplification systems, where each instance pairs a simple and complex sentence representing approximately the same information. Though the data was obtained for simplification, we use it as a source of training data for learning paraphrastic sentence embeddings. The dataset, which we call SimpWiki, consists of 167,689 sentence pairs.

To ensure a fair comparison, we select a sample of pairs from PPDB XL such that the number of tokens is approximately the same as the number of tokens in the Simpwiki sentences.3

Footnote 3: The PPDB data consists of 1,341,188 phrase pairs and contains 3 more tokens than the SimpWiki data.

We use paragraph-sl999 embeddings Wieting et al. (2015) to initialize the word embedding matrix (\(W_{w}\)) for all models. For all experiments, we fix the mini-batch size to 100, and \(\lambda_{c}\) to 0. We tune the margin \(\delta\) over \(\{0.4,0.6,0.8\}\) and \(\lambda_{w}\) over \(\{10^{-4},10^{-5},10^{-6},10^{-7},10^{-8},0\}\). We train avg for 7 epochs, and the LSTM for 3, since it converges much faster and does not benefit from 7 epochs. For optimization we use Adam Kingma and Ba (2015) with a learning rate of 0.001. We use the 2016 STS tasks Agirre et al. (2016) for model selection, where we average the Pearson's \(r\) over its 5 datasets. We refer to this type of model selection as _test_. For evaluation, we report the average Pearson's \(r\) over the 22 other sentence similarity tasks.

The results are shown in Table 1. We first note that, when training on PPDB, we find the same result as Wieting et al. (2016b): avg outperforms the LSTM by more than 13 points. However, when training both on sentence pairs, the gap shrinks to about 9 points. It appears that part of the inferior performance for the LSTM in prior work was due to training on phrase pairs rather than on sentence pairs. The avg model also benefits from training on sentences, but not nearly as much as the LSTM.4

[FOOTNOTE:
\begin{table}
\begin{tabular}{|l|c|c|c|} \cline{2-4} \multicolumn{1}{c|}{} & avg & LSTM & LSTMaxg \\ \hline PPDB & 67.7 & 54.2 & 64.2 \\ \hline SimpWiki & 68.4 & 59.3 & 67.5 \\ \hline \end{tabular}
\end{table}
Table 1: Test results on SemEval semantic textual similarity datasets (Pearson’s \(r\times 100\)) when training on different sources of data: phrase pairs from PPDB or simple-to-standard English Wikipedia sentence pairs from Coster and Kauchak (2011).

 

Our hypothesis explaining this result is that in PPDB, the phrase pairs are short fragments of text which are not necessarily constituents or phrases in any syntactic sense. Therefore, the sentences in the STS test sets are quite different from the fragments seen during training. We hypothesize that while word-averaging is relatively unaffected by this difference, the recurrent models are much more sensitive to overall characteristics of the word sequences, and the difference between train and test matters much more.

These results also suggest that the SimpWiki data, even though it was developed for text simplification, may be useful for other researchers working on semantic textual similarity tasks.

#### 4.1.3 Experiments with LSTM Variations

We next compare LSTM and LSTMavg. The latter consists of averaging the hidden vectors of the LSTM rather than using the final hidden vector as in prior work Weting et al. (2016). We hypothesize that the LSTM may put more emphasis on the words at the end of the sentence than those at the beginning. By averaging the hidden states, the impact of all words in the sequence is better taken into account. Averaging also makes the LSTM more like avg, which we know to perform strongly in this setting.

The results on avg and the LSTM models are shown in Table 1. When training on PPDB, moving from LSTM to LSTMavg improves performance by 10 points, closing most of the gap with avg. We also find that LSTMavg improves by moving from PPDB to Simpwiki, though in both cases it still lags behind avg.

### Experiments with Regularization

We next experiment with various forms of regularization. Previous work Wieting et al. (2016, 2016) only used L\({}_{2}\) regularization. Wieting et al. (2016) also regularized the word embeddings back to their initial values. Here we use L\({}_{2}\) regularization as well as several additional regularization methods we describe below.

We try two forms of dropout. The first is just standard dropout Srivastava et al. (2014) on the word embeddings. The second is "word dropout", which drops out entire word embeddings with some probability Iyyer et al. (2015).

We also experiment with scrambling the inputs. For a given mini-batch, we go through each sentence pair and, with some probability, we shuffle the words in each sentence in the pair. When scrambling a sentence pair, we always shuffle both sentences in the pair. We do this before selecting negative examples for the mini-batch. The motivation for scrambling is to make it more difficult for the LSTM to memorize the sequences in the training data, forcing it to focus more on the identities of the words and less on word order. Hence it will be expected to behave more like the word averaging model.5

Footnote 5: We also tried some variations on scrambling that did not yield significant improvements: scrambling after obtaining the negative examples, partially scrambling by performing \(n\) swaps where \(n\) comes from a Poisson distribution with a tunable \(\lambda\), and scrambling individual sentences with some probability instead of always scrambling both in the pair.

We also experiment with combining scrambling and dropout. In this setting, we tune over scrambling with either word dropout or dropout.

The settings for these experiments are largely the same as those of the previous section with the exception that we tune \(\lambda_{w}\) over a smaller set of values: \(\{10^{-5},0\}\). When using L\({}_{2}\) regularization, we tune \(\lambda_{c}\) over \(\{10^{-3},10^{-4},10^{-5},10^{-6}\}\). When using dropout, we tune the dropout rate over \(\{0.2,0.4,0.6\}\). When using scrambling, we tune the scrambling rate over \(\{0.25,0.5,0.75\}\). We also include a bidirectional model ("Bi") for both LSTMavg and the gated recurrent averaging network. We tune over two ways to combine the forward and backward hidden states; the first simply adds them together and the second uses a single feedforward layer with a \(\tanh\) activation.

We try two approaches for model selection. The first, _test_ , is the same as was done in Section 4.1.2, where we use the average Pearson's \(r\) on the 5 2016 STS datasets. The second tunes based on the average Pearson's \(r\) of all 22 datasets in our evaluation. We refer to this as _oracle_.

The results are shown in Table 2. They show that dropping entire word embeddings and scram 

bling input sequences is very effective in improving the result of the LSTM, while neither type of dropout improves avg. Moreover, averaging the hidden states of the LSTM is the most effective modification to the LSTM in improving performance. All of these modifications can be combined to significantly improve the LSTM, finally allowing it to overtake avg.

In Table 3, we compare the various GRAN architectures. We find that the GRAN provides a small improvement over the best LSTM configuration, possibly because of its similarity to avg. It also outperforms the other GRAN models, despite being the simplest.

In Table 4, we show results on all individual STS evaluation datasets after using STS 2016 for model selection (unidirectional models only). The LSTMavg and gated recurrent averaging network are more closely correlated in performance, in terms of Spearman's \(\rho\) and Pearson'r \(r\), than either is to avg. But they do differ significantly in some datasets, most notably in those comparing machine translation output with its reference. Interestingly, both the LSTMavg and gated recurrent averaging network significantly outperform avg in the datasets focused on comparing glosses like _OnWN_ and _FNWN_. Upon examination, we found that these datasets, especially 2013 _OnWN_, contain examples of low similarity with high word overlap. For example, the pair \(\langle\textit{the act of preserving or protecting something}.\rangle\), _the act of decreasing or reducing something._\(\rangle\) from 2013 _OnWN_ has a gold similarity score of 0.4. It appears that avg was fooled by the high amount of word overlap in such pairs, while the other two models were better able to recognize the semantic differences.

### Supervised Text Similarity

We also investigate if these techniques can improve LSTM performance on supervised semantic textual similarity tasks. We evaluate on two supervised datasets. For the first, we start with the 20 SemEval STS datasets from 2012-2015 and then use 40% of each dataset for training, 10% for validation, and the remaining 50% for testing. There are 4,481 examples in training, 1,207 in validation, and 6,060 in the test set. The second is the SICK 2014 dataset, using its standard training, validation, and test sets. There are 4,500 sentence pairs

\begin{table}
\begin{tabular}{|l||c|c|c|} \hline Dataset & LSTMavg & avg & GRAN \\ \hline MSRpur & **49.0** & 45.9 & 47.7 \\ MSRvid & 84.3 & 85.1 & **85.2** \\ SMT-our & **§1.2** & 47.5 & 49.3 \\ OnWN & **71.5** & 71.2 & 71.5 \\ SMT-news & **68.0** & 58.2 & 58.7 \\ \hline STS 2012 Average & **64.8** & 61.6 & 62.5 \\ \hline headline & **77.3** & 76.9 & 76.1 \\ OnWN & 81.2 & 72.8 & **81.4** \\ FNWN & 53.2 & 50.2 & **55.6** \\ SMT & **40.7** & 38.0 & 40.3 \\ \hline STS 2013 Average & 63.1 & 59.4 & **63.4** \\ \hline detft forum & **56.6** & 55.6 & 55.7 \\ deft news & 78.0 & **78.5** & 77.1 \\ headline & 74.5 & **75.1** & 72.8 \\ images & 84.7 & 85.6 & **85.8** \\ OwnN & 84.9 & 81.4 & **85.1** \\ tweet news & 76.3 & **78.7** & 78.7 \\ \hline STS 2014 Average & 75.8 & 75.8 & **75.9** \\ \hline answers-forums & 71.8 & 70.6 & **73.1** \\ answers-students & 71.1 & **75.8** & 72.9 \\ belief & 75.3 & 76.8 & **78.0** \\ headline & 79.5 & **80.3** & 78.6 \\ images & 85.8 & **86.0** & 85.8 \\ \hline STS 2015 Average & 76.7 & **77.9** & 77.7 \\ \hline
2014 SICK & 71.3 & 72.4 & **72.9** \\ \hline
2015 Twitter & **52.1** & **52.1** & 50.2 \\ \hline \end{tabular}
\end{table}
Table 4: Results on SemEval textual similarity datasets (Pearson’s \(r\times 100\)). The highest score in each row is in boldface.

\begin{table}
\begin{tabular}{|l|l|c|c|} \hline Model & Regularization & Oracle & 2016 STS \\ \hline  & none & 68.5 & 68.4 \\ AVG & dropout & 68.4 & 68.3 \\  & word dropout & 68.3 & 68.3 \\ \hline  & none & 60.6 & 59.3 \\ L\({}_{2}\) & 60.3 & 56.5 \\  & dropout & 58.1 & 55.3 \\  & word dropout & 66.2 & 65.3 \\  & scrambling & 66.3 & 65.1 \\  & dropout, scrambling & 68.4 & 68.4 \\ \hline LSTMavg & none & 67.7 & 67.5 \\  & dropout, scrambling & 69.2 & 68.6 \\ \hline BiLSTMavg & dropout, scrambling & **69.4** & **68.7** \\ \hline \end{tabular}
\end{table}
Table 2: Results on SemEval textual similarity datasets (Pearson’s \(r\times 100\)) when experimenting with different regularization techniques.

\begin{table}
\begin{tabular}{|l|c|c|} \hline Model & Oracle & STS 2016 \\ \hline GRAN (no reg.) & 68.0 & 68.0 \\ GRAN & 69.5 & **68.9** \\ GRAN-2 & 68.8 & 68.1 \\ GRAN-3 & 69.0 & 67.2 \\ GRAN-4 & 68.6 & 68.1 \\ GRAN-5 & 66.1 & 64.8 \\ BigRAN & **69.7** & 68.4 \\ \hline \end{tabular}
\end{table}
Table 3: Results on SemEval textual similarity datasets (Pearson’s \(r\times 100\)) for the GRAN architectures. The first row, marked as (no reg.) is the GRAN without any regularization. The other rows show the result of the various GRAN models using dropout and scrambling.

 

in the training set, 500 in the development set, and 4,927 in the test set. The SICK task is an easier learning problem since the training examples are all drawn from the same distribution, and they are mostly shorter and use simpler language. As these are supervised tasks, the sentence pairs in the training set contain manually-annotated semantic similarity scores.

We minimize the loss function6 from Tai et al. (2015). Given a score for a sentence pair in the range \([1,K]\), where \(K\) is an integer, with sentence representations \(h_{L}\) and \(h_{R}\), and model parameters \(\theta\), they first compute:

Footnote 6: This objective function has been shown to perform very strongly on text similarity tasks, significantly better than squared or absolute error.

\[h_{\times}=h_{L}\odot h_{R},~{}h_{+}=|h_{L}-h_{R}|,\] \[h_{s}=\sigma\left(W^{(\times)}h_{\times}+W^{(+)}h_{+}+b^{(h)} \right),\] \[\hat{p}_{\theta}=\operatorname*{softmax}\left(W^{(p)}h_{s}+b^{( p)}\right),\] \[\hat{y}=r^{T}\hat{p}_{\theta},\]

where \(r^{T}=[1~{}2~{}\ldots~{}K]\). They then define a sparse target distribution \(p\) that satisfies \(y=r^{T}p\):

for \(1\leq i\leq K\). Then they use the following loss, the regularized KL-divergence between \(p\) and \(\hat{p}_{\theta}\):

\[J(\theta)=\frac{1}{m}\sum_{k=1}^{m}\mathrm{KL}\Big{(}p^{(k)}\,\big{\lVert} \,\hat{p}_{\theta}^{(k)}\Big{)},\]

where \(m\) is the number of training pairs.

We experiment with the LSTM, LSTMAvG, and avg models with dropout, word dropout, and scrambling tuning over the same hyperparameter as in Section 4.2. We again regularize the word embeddings back to their initial state, tuning \(\lambda_{w}\) over \(\{10^{-5},0\}\). We used the validation set for each respective dataset for model selection.

The results are shown in Table 5. The gated recurrent averaging network has the best performance on both datasets. Dropout helps the word-averaging model in the STS task, unlike in the transfer learning setting. The LSTM benefits slightly from dropout, scrambling, and averaging on their own individually with the exception of word dropout on both datasets and averaging on the SICK dataset. However, when combined, these modifications are able to significantly improve the performance of the LSTM, bringing it much closer in performance to avg. This experiment indicates that these modifications when training LSTMs are beneficial outside the transfer learning setting, and can potentially be used to improve performance for the broad range of problems that use LSTMs to model sentences.

In Table 6 we compare the various GRAN architectures under the same settings as the previous experiment. We find that the GRAN still has the best overall performance.

We also experiment with initializing the supervised models using our pretrained sentence model parameters, for the avg model (no regularization), LSTMavg (dropout, scrambling), and gatedrecurrent averaging network (dropout, scrambling) models from Table 2 and Table 3. We both initialize and then regularize back to these initial values, referring to this setting as "universal".7

\begin{table}
\begin{tabular}{|l|c|c|c|} \hline Model & STS & SICK & Avg. \\ \hline GRAN & **81.6** & 85.3 & **83.5** \\ GRAN-2 & 77.4 & 85.1 & 81.3 \\ GRAN-3 & 81.3 & 85.4 & 83.4 \\ GRAN-4 & 80.1 & **85.5** & 82.8 \\ GRAN-5 & 70.9 & 83.0 & 77.0 \\ \hline \end{tabular}
\end{table}
Table 6: Results from supervised training on the STS and SICK datasets (Pearson’s \(r\times 100\)) for the GRAN architectures. The last column is the average result on the two datasets.

\begin{table}
\begin{tabular}{|l|l|c|c|c|} \hline Model & Regularization & STS & SICK & Avg. \\ \hline  & none & 79.2 & 85.2 & 82.6 \\ AVg & dropout & 80.7 & 84.5 & 82.6 \\  & word dropout & 79.3 & 81.8 & 80.6 \\ \hline  & none & 68.4 & 80.9 & 74.7 \\ LSTM & dropout & 69.6 & 81.3 & 75.5 \\  & word dropout & 68.0 & 76.4 & 72.2 \\  & scrambling & 74.2 & 84.4 & 79.3 \\  & dropout, scrambling & 75.0 & 84.2 & 79.6 \\ \hline  & none & 69.0 & 79.5 & 74.3 \\ LSTMavg & dropout & 69.2 & 79.4 & 74.3 \\  & word dropout & 65.6 & 76.1 & 70.9 \\  & scrambling & 76.5 & 83.2 & 79.9 \\  & dropout, scrambling & 76.5 & 84.0 & 80.3 \\ \hline  & none & 79.7 & 85.2 & 82.5 \\ GRAN & dropout & 79.7 & 84.6 & 82.2 \\  & word dropout & 77.3 & 83.0 & 80.2 \\  & scrambling & 81.4 & **85.3** & **83.4** \\ \hline  & \multicolumn{1}{c}{} & \multicolumn{1}{c}{} & \multicolumn{1}{c}{} & \multicolumn{1}{c}{} \\ \cline{3-5}  & \multicolumn{1}{c}{} & \multicolumn{1}{c}{} & \multicolumn{1}{c}{} & \multicolumn{1}{c}{} \\ \cline{3-5}  & \multicolumn{1}{c}{} & \multicolumn{1}{c}{} & \multicolumn{1}{c}{} & \multicolumn{1}{c}{} \\ \cline{3-5}  & \multicolumn{1}{c}{} & \multicolumn{1}{c}{} & \multicolumn{1}{c}{} & \multicolumn{1}{c}{} \\ \cline{3-5}  & \multicolumn{1}{c}{} & \multicolumn{1}{c}{} & \multicolumn{1}{c}{} & \multicolumn{1}{c}{} \\ \cline{3-5}  & \multicolumn{1}{c}{} & \multicolumn{1}{c}{} & \multicolumn{1}{c}{} & \multicolumn{1}{c}{} \\ \cline{3-5}  & \multicolumn{1}{c}{} & \multicolumn{1}{c}{} & \multicolumn{1}{c}{} & \multicolumn{1}{c}{} \\ \cline{3-5}  & \multicolumn{1}{c}{} & \multicolumn{1}{c}{} & \multicolumn{1}{c}{} & \multicolumn{1}{c}{} \\ \cline{3-5}  & \multicolumn{1}{c}{} & \multicolumn{1}{c}{} & \multicolumn{1}{c}{} & \multicolumn{1}{c}{} \\ \cline{3-5}  & \multicolumn{1}{c}{} & \multicolumn{1}{c}{} & \multicolumn{1}{c}{} & \multicolumn{1}{c}{} \\ \cline{3-5}  & \multicolumn{1}{c}{} & \multicolumn{1}{c}{} & \multicolumn{1}{c}{} & \multicolumn{1}{c}{} \\ \cline{3-5}  & \multicolumn{1}{c}{} & \multicolumn{1}{c}{} & \multicolumn{1}{c}{} & \multicolumn{1}{c}{} \\ \cline{3-5}  & \multicolumn{1}{c}{} & \multicolumn{1}{c}{} & \multicolumn{1}{c}{} & \multicolumn{1}{c}{} \\ \cline{3-5}  & \multicolumn{1}{c}{} & \multicolumn{1}{c}{} & \multicolumn{1}{c}{} & \multicolumn{1}{c}{ 

The results are shown in Table 8. Initializing and regularizing to the pretrained models significantly improves the performance for all three models, justifying our claim that these models serve a dual purpose: they can be used a black box semantic similarity function, and they possess rich knowledge that can be used to improve the performance of downstream tasks.

## 5 Analysis

### Error Analysis

We analyze the predictions of avg and the recurrent networks, represented by LSTMavg, on the 20 STS datasets. We choose LSTMavg as it correlates slightly less strongly with avg than the GRAN on the results over all SemEval datasets used for evaluation. We scale the models' cosine similarities to lie within \([0,5]\), then compare the predicted similarities of LSTMavg and avg to the gold similarities. We analyzed instances in which each model would tend to overestimate or underestimate the gold similarity relative to the other. These are illustrated in Table 7.

We find that avg tends to overestimate the semantic similarity of a sentence pair, relative to LSTMavg, when the two sentences have a lot of word or synonym overlap, but have either important differences in key semantic roles or where one sentence has significantly more content than the other. These phenomena are shown in examples 1 and 2 in Table 7. Conversely, avg tends to underestimate similarity when there are one-word-to-multiword paraphrases between the two sentences as shown in examples 3 and 4.

LSTMavg tends to overestimate similarity when the two inputs have similar sequences of syntactic categories, but the meanings of the sentences are different (examples 5, 6, and 7). Instances of LSTMavg underestimating the similarity relative to avg are relatively rare, and those that we found did not have any systematic patterns.

### GRAN Gate Analysis

We also investigate what is learned by the gating function of the gated recurrent averaging network. We are interested to see whether its estimates of importance correlate with those of traditional syntactic and (shallow) semantic analysis.

We use the oracle trained gated recurrent averaging network from Table 3 and calculate the L\({}_{1}\) norm of the gate after embedding 10,000 sentences from English Wikipedia.8 We also automatically tag and parse these sentences using the Stanford dependency parser Manning et al. (2014). We then compute the average gate L\({}_{1}\) norms for particular part-of-speech tags, dependency arc labels, and their conjunction.

Footnote 8: We selected only sentences of less than or equal to 15 tokens to ensure more accurate parsing.

Table 9 shows the highest/lowest average norm tags and dependency labels. The network prefers nouns, especially proper nouns, as well as cardinal numbers, which is sensible as these are among the most discriminative features of a sentence.

Analyzing the dependenc

\begin{table}
\begin{tabular}{l|p{113.8pt} p{113.8pt}|c|c|c} \hline \# & Sentence 1 & Sentence 2 & & Avg & Gold \\ \hline
1 & the lamb is looking at the camera. & a cat looking at the camera. & **3.42** & 4.13 & 0.8 \\
2 & he also said shockey is “living the dream life of a new york athlete. & “jeremy’s a good guy,” barber said, adding:“jèremy is lying the dream life of the new york athlete. & **3.55** & 4.22 & 2.75 \\ \hline
3 & bloomberg chips in a billion & bloomberg gives S1.1 b to university & **3.99** & 3.04 & 4.0 \\
4 & in other regions, the sharia is imposed. & in other areas, sharia law is being introduced by force. & **4.44** & 3.72 & 4.75 \\ \hline
5 & three men in suits sitting at a table. & two women in the kitchen looking at a object. & 3.33 & **2.79** & 0.0 \\
6 & we ever get out of it in the first place! & where does the money come from in the first place? & 4.00 & **3.33** & 0.8 \\
7 & two birds interacting in the grass. & two dogs play with each other outdoors. & 3.44 & **2.81** & 0.2 \\ \hline \end{tabular}
\end{table}
Table 7: Illustrative sentence pairs from the STS datasets showing errors made by LSTMavg and avg. The last three columns show the gold similarity score, the similarity score of LSTMavg, and the similarity score of AVG. Boldface indicates smaller error compared to gold scores.

\begin{table}
\begin{tabular}{l|l|r|c} \hline Model & Regularization & STS & SlICK \\ \hline  & dropout & 80.7 & 84.5 \\ avg & dropout, universal & **82.9** & 85.6 \\ \hline  & dropout, scrambling & 76.5 & 84.0 \\  & dropout, scrambling, universal & 81.3 & 85.2 \\ \hline GRAN & dropout, scrambling & 81.6 & 85.1 \\  & dropout, scrambling, universal & 82.7 & **86.0** \\ \hline \end{tabular}
\end{table}
Table 8: Impact of initializing and regularizing toward universal models (Pearson’s \(r\times 100\)) in supervised training.

 

that nouns in the object position tend to have higher weight than nouns in the subject position. This may relate to topic and focus; the object may be more likely to be the "new" information related by the sentence, which would then make it more likely to be matched by the other sentence in the paraphrase pair.

We find that the weights of adjectives depend on their position in the sentence, as shown in Table 10. The highest norms appear when an adjective is an xcomp, acomp, or root; this typically means it is residing in an object-like position in its clause. Adjectives that modify a noun (amod) have medium weight, and those that modify another adjective or verb (advmod) have low weight.

Lastly, we analyze words tagged as VBG, a highly ambiguous tag that can serve many syntactic roles in a sentence. As shown in Table 11, we find that when they are used to modify a noun (amod) or in the object position of a clause (xcomp, pcomp) they have high weight. Medium weight appears when used in verb phrases (root, vmod) and low weight when used as prepositions or auxiliary verbs (prep, auxpass).

## 6 Conclusion

We showed how to modify and regularize LSTMs to improve their performance for learning paraphrastic sentence embeddings in both transfer and supervised settings. We also introduced a new recurrent network, the gated recurrent averaging network, that improves upon both avg and LSTMs for these tasks, and we release our code and trained models.

Furthermore, we analyzed the different errors produced by avg and the recurrent methods and found that the recurrent methods were learning composition that wasn't being captured by avg. We also investigated the GRAN in order to better understand the compositional phenomena it was learning by analyzing the L\({}_{1}\) norm of its gate over various inputs.

Future work will explore additional data sources, including from aligning different translations of novels Barzilay and McKeown (2001), aligning new articles of the same topic Dolan et al. (2004), or even possibly using machine translation systems to translate bilingual text into paraphrastic sentence pairs. Our new techniques, combined with the promise of new data sources, offer a great deal of potential for improved universal paraphrastic sentence embeddings.

#### Acknowledgments

We thank the anonymous reviewers for their valuable comments. This research used resources of the Argonne Leadership Computing Facility, which is a DOE Office of Science User Facility supported under Contract DE-AC02-06CH11357. We thank the developers of Theano Theano Development Team (2016) and NVIDIA Corporation for donating GPUs used in this research.

\begin{table}
\begin{tabular}{|l|c|} \hline Dep. Label & Weight \\ \hline xcomp & 170.6 \\ accomp & 167.1 \\ root & 157.4 \\ \hline amod & 143.1 \\ \hline advmod & 121.6 \\ \hline \end{tabular}
\end{table}
Table 10: Average L\({}_{1}\) norms for adjectives (JJ) with selected dependency labels.

\begin{table}
\begin{tabular}{|l|l|l|l|} \hline  & POS & \multicolumn{2}{c|}{Dep. Label} \\ \hline top 10 & bot. 10 & top 10 & bot. 10 \\ \hline NNP & TO & number & possessive \\ NNPS & WDT & nn & cop \\ CD & POS & num & det \\ NNS & DT & acomp & auxpass \\ VBG & WP & appos & prep \\ NN & IN & probj & cc \\ JJ & CC & vmod & mark \\ UH & PRP & dobj & aux \\ VBN & EX & amod & expl \\ JJS & WRB & conj & neg \\ \hline \end{tabular}
\end{table}
Table 9: POS tags and dependency labels with highest and lowest average gated recurrent averaging network gate L\({}_{1}\) norms. The lists are ordered from highest norm to lowest in the top 10 columns, and lowest to highest in the bottom 10 columns.

 



## References

* Agirre et al. (2015) Eneko Agirre, Carmen Banea, Claire Cardiie, Daniel Cer, Mona Diab, Aitor Gonzalez-Agirre, Weiwei Guo, Inigo Lopez-Gazpio, Montse Maritxalar, Rada Mihalcea, German Rigau, Larraitz Uria, and Janyce Wiebe. 2015. SemEval-2014 task 2: Semantic textual similarity, English, Spanish and pilot on interpretability. In _Proceedings of the 9th International Workshop on Semantic Evaluation (SemEval 2015)_.
* Agirre et al. (2016) Eneko Agirre, Carmen Banea, Claire Cardiie, Daniel Cer, Mona Diab, Aitor Gonzalez-Agirre, Weiwei Guo, Rada Mihalcea, German Rigau, and Janyce Wiebe. 2016. SemEval-2014 task 10: Multilingual semantic textual similarity. In _Proceedings of the 8th International Workshop on Semantic Evaluation (SemEval 2014)_.
* Agirre et al. (2016) Eneko Agirre, Carmen Banea, Daniel Cer, Mona Diab, Aitor Gonzalez-Agirre, Rada Mihalcea, German Rigau, and Janyce Wiebe. 2016. Semeval-2016 task 1: Semantic textual similarity, monolingual and cross-lingual evaluation. _Proceedings of SemEval_ pages 497-511.
* Agirre et al. (2013) Eneko Agirre, Daniel Cer, Mona Diab, Aitor Gonzalez-Agirre, and Weiwei Guo. 2013. *SEM 2013 shared task: Semantic textual similarity. In _Second Joint Conference on Lexical and Computational Semantics (*SEM), Volume 1: Proceedings of the Main Conference and the Shared Task: Semantic Textual Similarity_.
* Agirre et al. (2012) Eneko Agirre, Mona Diab, Daniel Cer, and Aitor Gonzalez-Agirre. 2012. SemEval-2012 task 6: A pilot on semantic textual similarity. In _Proceedings of the First Joint Conference on Lexical and Computational Semantics-Volume 1: Proceedings of the main conference and the shared task, and Volume 2: Proceedings of the Sixth International Workshop on Semantic Evaluation_. Association for Computational Linguistics.
* Arora et al. (2017) Sanjeev Arora, Yingyu Liang, and Tengyu Ma. 2017. A simple but tough-to-beat baseline for sentence embeddings. In _Proceedings of the International Conference on Learning Representations_.
* Barzilay and McKeown (2001) Regina Barzilay and Kathleen R McKeown. 2001. Extracting paraphrases from a parallel corpus. In _Proceedings of the 39th annual meeting on Association for Computational Linguistics_.
* Blacoe and Lapata (2012) William Blacoe and Mirella Lapata. 2012. A comparison of vector-based representations for semantic composition. In _Proceedings of the 2012 Joint Conference on Empirical Methods in Natural Language Processing and Computational Natural Language Learning_.
* Coster and Kauchak (2011) William Coster and David Kauchak. 2011. Simple english wikipedia: a new text simplification task. In _Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies: short papers-Volume 2_.
* Dolan et al. (2004) Bill Dolan, Chris Quirk, and Chris Brockett. 2004. Unsupervised construction of large paraphrase corpora: Exploiting massively parallel news sources. In _Proceedings of the 20th international conference on Computational linguistics_.
* Ganitkevitch and Callison-Burch (2014) Juri Ganitkevitch and Chris Callison-Burch. 2014. The multilingual paraphrase database. In _Proceedings of the Ninth International Conference on Language Resources and Evaluation (LREC-2014)_.
* Gantikevitch et al. (2013) Juri Ganitkevitch, Benjamin Van Durme, and Chris Callison-Burch. 2013. PPDB: The Paraphrase Database. In _Proceedings of HLT-NAACL_.
* Gers et al. (2003) Felix A. Gers, Nicol N. Schraudolph, and Jurgen Schmidhuber. 2003. Learning precise timing with LSTM recurrent networks. _The Journal of Machine Learning Research_ 3.
* Hill et al. (2016) Felix Hill, Kyunghyun Cho, and Anna Korhonen. 2016. Learning distributed representations of sentences from unlabelled data. In _Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_.
* Hochreiter and Schmidhuber (1997) Sepp Hochreiter and Jurgen Schmidhuber. 1997. Long short-term memory. _Neural computation_ 9(8).
* Hu et al. (2014) Baotian Hu, Zhengdong Lu, Hang Li, and Qingcai Chen. 2014. Convolutional neural network architectures for matching natural language sentences. In _Advances in Neural Information Processing Systems_.
* Irsoy and Cardie (2014) Ozan Irsoy and Claire Cardie. 2014. Deep recursive neural networks for compositionality in language. In _Advances in Neural Information Processing Systems_.
* Iyyer et al. (2015) Mohit Iyyer, Varun Manjunatha, Jordan Boyd-Graber, and Hal Daume III. 2015. Deep unordered composition rivals syntactic methods for text classification. In _Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)_.
* Kalchbrenner et al. (2014) Nal Kalchbrenner, Edward Grefenstette, and Phil Blunsom. 2014. A convolutional neural network for modelling sentences. In _Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_.
* Kim (2014) Yoon Kim. 2014. Convolutional neural networks for sentence classification. In _Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP)_.
* Kingma and Ba (2015) Diederik P. Kingma and Jimmy Ba. 2015. Adam: A method for stochastic optimization. In _Proceedings of International Conference on Learning Representations_

Ryan Kiros, Yukun Zhu, Ruslan R Salakhutdinov, Richard Zemel, Raquel Urtasun, Antonio Torralba, and Sanja Fidler. 2015. Skip-thought vectors. In _Advances in Neural Information Processing Systems_.
* Le and Mikolov (2014) Quoc V. Le and Tomas Mikolov. 2014. Distributed representations of sentences and documents. In _Proceedings of the 31st International Conference on Machine Learning_.
* Ling et al. (2015) Wang Ling, Chris Dyer, Alan W Black, Isabel Trancoso, Ramon Fermandez, Silvio Amir, Luis Marujo, and Tiago Luis. 2015. Finding function in form: Compositional character models for open vocabulary word representation. In _Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing_.
* Liu et al. (2015) Pengfei Liu, Xipeng Qiu, Xinchi Chen, Shiyu Wu, and Xuanjing Huang. 2015. Multi-timescale long short-term memory neural network for modelling sentences and documents. In _Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing_.
* Manning et al. (2014) Christopher D. Manning, Mihai Surdeanu, John Bauer, Jenny Finkel, Steven J. Bethard, and David McClosky. 2014. The Stanford CoreNLP natural language processing toolkit. In _Proceedings of 52nd Annual Meeting of the Association for Computational Linguistics: System Demonstrations_.
* Marelli et al. (2014) Marco Marelli, Luisa Bentivogli, Marco Baroni, Raffaella Bernardi, Stefano Menini, and Roberto Zamparelli. 2014. SemEval-2014 task 1: Evaluation of compositional distributional semantic models on full sentences through semantic relatedness and textual entailment. In _Proceedings of the 8th International Workshop on Semantic Evaluation (SemEval 2014)_.
* Mitchell and Lapata (2008) Jeff Mitchell and Mirella Lapata. 2008. Vector-based models of semantic composition. In _Proceedings of the 46th Annual Meeting of the Association for Computational Linguistics_.
* Mitchell and Lapata (2010) Jeff Mitchell and Mirella Lapata. 2010. Composition in distributional models of semantics. _Cognitive Science_ 34(8).
* Pagliardini et al. (2017) Matteo Pagliardini, Prakhar Gupta, and Martin Jaggi. 2017. Unsupervised Learning of Sentence Embeddings using Compositional n-Gram Features. _arXiv preprint arXiv:1703.02507_ .
* Pham et al. (2015) Nghia The Pham, German Kruszewski, Angeliki Lazaridou, and Marco Baroni. 2015. Jointly optimizing word representations for lexical and sentential tasks with the c-phrase model. In _Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)_.
* Socher et al. (2011) Richard Socher, Eric H. Huang, Jeffrey Pennington, Andrew Y. Ng, and Christopher D. Manning. 2011. Dynamic pooling and unfolding recursive autoencoders for paraphrase detection. In _Advances in Neural Information Processing Systems_.
* Socher et al. (2012) Richard Socher, Brody Huval, Christopher D. Manning, and Andrew Y. Ng. 2012. Semantic compositionality through recursive matrix-vector spaces. In _Proceedings of the 2012 Joint Conference on Empirical Methods in Natural Language Processing and Computational Natural Language Learning_.
* Socher et al. (2013) Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D. Manning, Andrew Ng, and Christopher Potts. 2013. Recursive deep models for semantic compositionality over a sentiment treebank. In _Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing_.
* Srivastava et al. (2014) Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. 2014. Dropout: A simple way to prevent neural networks from overfitting. _The Journal of Machine Learning Research_ 15(1).
* Tai et al. (2015) Kai Sheng Tai, Richard Socher, and Christopher D. Manning. 2015. Improved semantic representations from tree-structured long short-term memory networks. In _Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)_.
* Team (2016) Theano Development Team. 2016. Theano: A Python framework for fast computation of mathematical expressions. _arXiv e-prints_ abs/1605.02688. [http://arxiv.org/abs/1605.02688](http://arxiv.org/abs/1605.02688).
* Wieting et al. (2016a) John Wieting, Mohit Bansal, Kevin Gimpel, and Karen Livescu. 2016a. Charagram: Embedding words and sentences via character \(n\)-grams. In _Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing_.
* Wieting et al. (2016b) John Wieting, Mohit Bansal, Kevin Gimpel, and Karen Livescu. 2016b. Towards universal paraphrastic sentence embeddings. In _Proceedings of the International Conference on Learning Representations_.
* Wieting et al. (2015) John Wieting, Mohit Bansal, Kevin Gimpel, Karen Livescu, and Dan Roth. 2015. From paraphrase database to compositional paraphrase model and back. _Transactions of the ACL (TACL)_ .
* Xu et al. (2015) Wei Xu, Chris Callison-Burch, and William B Dolan. 2015. SemEval-2015 task 1: Paraphrase and semantic similarity in Twitter (PIT). In _Proceedings of the 9th International Workshop on Semantic Evaluation (SemEval)_