Venue: NAACL 2018

# Character-based Neural Networks for Sentence Pair Modeling

 Wuwei Lan & Wei Xu

Department of Computer Science and Engineering

Ohio State University

{lan.105, xu.1265}@osu.edu

###### Abstract

Sentence pair modeling is critical for many NLP tasks, such as paraphrase identification, semantic textual similarity, and natural language inference. Most state-of-the-art neural models for these tasks rely on pretrained word embedding and compose sentence-level semantics in varied ways; however, few works have attempted to verify whether we really need pretrained embeddings in these tasks. In this paper, we study how effective subword-level (character and character n-gram) representations are in sentence pair modeling. Though it is well-known that subword models are effective in tasks with single sentence input, including language modeling and machine translation, they have not been systematically studied in sentence pair modeling tasks where the semantic and string similarities between texts matter. Our experiments show that subword models without any pretrained word embedding can achieve new state-of-the-art results on two social media datasets and competitive results on news data for paraphrase identification.

## 1 Introduction

Recently, there have been various neural network models proposed for sentence pair modeling tasks, including semantic similarity Agirre et al. (2015), paraphrase identification Dolan et al. (2004); Xu et al. (2015), natural language inference Bowman et al. (2015), etc. Most, if not all, of these state-of-the-art neural models Yin et al. (2016); Parikh et al. (2016); He and Lin (2016); Tomar et al. (2017); Shen et al. (2017) have achieved the best performances for these tasks by using pretrained word embeddings, but results without pretraining are less frequently reported or noted. In fact, we will show that, even with fixed randomized word vectors, the pairwise word interaction model He and Lin (2016) based on contextual word vector similarities can still achieve strong performance by capturing identical words and similar surface context features. Moreover, pretrained word embeddings generally have poor coverage in social media domain where out-of-vocabulary rate often reaches over 20% Baldwin et al. (2013).

We investigated the effectiveness of subword units, such as characters and character n-grams, in place of words for vector representations in sentence pair modeling. Though it is well-known that subword representations are effective to model out-of-vocabulary words in many NLP tasks with a single sentence input, such as machine translation Luong et al. (2015); Costa-jussa and Fonollosa (2016), language modeling Ling et al. (2015); Vania and Lopez (2017), and sequence labeling dos Santos and Guimaraes (2015); Plank et al. (2016), they are not systematically studied in the tasks that concern pairs of sentences. Unlike in modeling individual sentences, subword representations have impacts not only on the out-of-vocabulary words but also more directly on the relation between two sentences, which is calculated based on vector similarities in many sentence pair modeling approaches (more details in Section 2.1). For example, while subwords may capture useful string similarities between a pair of sentences (e.g. spelling or morphological variations: _sister_ and _sista_, _teach_ and _teaches_), they could introduce errors (e.g. similarly spelled words with completely different meanings: _ware_ and _war_).

To better understand the role of subword embedding in sentence pair modeling, we performed experimental comparisons that vary (1) the type of subword unit, (2) the composition function, and (3) the datasets of different characteristics. We also presented experiments with language modeling as an auxiliary multi-task learning objective, showing consistent improvements. Taken together, subword and language modeling establish

new state-of-the-art results in two social media datasets and competitive results in a news dataset for paraphrase identification without using any pretrained word embeddings.

## 2 Sentence Pair Modeling with Subwords

The current neural networks for sentence pair modeling Yin et al. (2016); Parikh et al. (2016); He and Lin (2016); Liu et al. (2016); Tomar et al. (2017); Wang et al. (2017); Shen et al. (2017, etc) follow a more or less similar design with three main components: (a) contextualized word vectors generated via Bi-LSTM, CNN, or attention, as inputs; (b) soft or hard word alignment and interactions across sentences; (c) and the output classification layer. Different models vary in implementation details, and most importantly, to capture the same essential intuition in the word alignment (also encoded with contextual information) - the semantic relation between two sentences depends largely on the relations of aligned chunks Agirre et al. (2016). In this paper, we used pairwise word interaction model He and Lin (2016) as a representative example and staring point, which reported robust performance across multiple sentence pair modeling tasks and the best results by neural models on social media data Lan et al. (2017).

### Pairwise Word Interaction (PWI) Model

Let \({\mathbf{w}}^{a}=({\mathbf{w}}_{1}^{a},...,{\mathbf{w}}_{m}^{a})\) and \({\mathbf{w}}^{b}=({\mathbf{w}}_{1}^{a},...,{\mathbf{w}}_{n}^{b})\) be the input sentence pair consisting of \(m\) and \(n\) tokens, respectively. Each word vector \({\mathbf{w}}_{i}\in\mathbb{R}^{d}\) is initialized with pretrained \(d\)-dimensional word embedding Pennington et al. (2014); Wieting et al. (2015, 2016), then encoded with word context and sequence order through bidirectional LSTMs:

\[\overrightarrow{{\mathbf{h}}}_{i}=LSTM^{f}({\mathbf{w}}_{i}, \overrightarrow{{\mathbf{h}}}_{i-1}) \tag{1}\] \[\overleftarrow{{\mathbf{h}}}_{i}=LSTM^{b}({\mathbf{w}}_{i}, \overleftarrow{{\mathbf{h}}}_{i+1})\] (2) \[\overleftrightarrow{{\mathbf{h}}}_{i}=[\overrightarrow{{\mathbf{h}}}_{ i},\overleftarrow{{\mathbf{h}}}_{i}]\] (3) \[{\mathbf{h}}_{i}^{+}=\overrightarrow{{\mathbf{h}}}_{i}+\overleftarrow{{ \mathbf{h}}}_{i} \tag{4}\]

where \(\overrightarrow{{\mathbf{h}}}_{i}\) represents forward hidden state, \(\overleftarrow{{\mathbf{h}}}_{i}\) represents backward hidden state, and \(\overrightarrow{{\mathbf{h}}}_{i}\) and \({\mathbf{h}}_{i}^{+}\) are the concatenation and summation of two directional hidden states.

For all word pairs \(({\mathbf{w}}_{i}^{a},{\mathbf{w}}_{j}^{b})\) across sentences, the model directly calculates word pair interactions using cosine similarity, Euclidean distance, and dot product over the outputs of the encoding layer:

\[D(\overrightarrow{{\mathbf{h}}}_{i},\overrightarrow{{\mathbf{h}}}_{j})=[ cos(\overrightarrow{{\mathbf{h}}}_{i},\overrightarrow{{\mathbf{h}}}_{ j}), \tag{5}\] \[L2 Euclid(\overrightarrow{{\mathbf{h}}}_{i}, \overrightarrow{{\mathbf{h}}}_{j}),\] \[ DotProduct(\overrightarrow{{\mathbf{h}}}_{i},\overrightarrow{{\mathbf{ h}}}_{j})].\]

The above equation can also apply to other states \(\overleftarrow{{\mathbf{h}}}\), \(\overleftrightarrow{{\mathbf{h}}}\) and \({\mathbf{h}}^{+}\), resulting in a tensor \(\mathbf{D}^{13\times m\times n}\) after padding one extra bias term. A "hard" attention is applied to the interaction tensor to further enforce the word alignment, by sorting the interaction values and selecting top ranked word pairs. A 19-layer-deep CNN is followed to aggregate the word interaction features and the softmax layer to predicate classification probabilities.

### Embedding Subwords in PWI Model

Our subword models only involve modification of the input representation layer in the pairwise interaciton model. Let \(c_{1},...,c_{k}\) be the subword (character unigram, bigram and trigram) sequence of a word \(w\). The subword embedding matrix is \(\mathbf{C}\in\mathbb{R}^{d^{\prime}*k}\), where each subword is encoded into the \(d^{\prime}\)-dimension vector. The same subwords will share the same embeddings. We considered two different composition functions to assemble subword embeddings into word embedding:

Char C2W(Ling et al., 2015) applies Bi-LSTM to subword sequence \(c_{1},...,c_{k}\), then the last hidden state \(\overrightarrow{{\mathbf{h}}}_{k}^{char}\) in forward direction and the first hidden state \(\overleftarrow{{\mathbf{h}}}_{0}^{char}\) of the backward direction are linearly combined into word-level embedding \({\mathbf{w}}\):

\[{\mathbf{w}}={\mathbf{W}}_{f}\cdot\overrightarrow{{\mathbf{h}}}_{k}^{char}+{\mathbf{W}}_{b} \cdot\overleftarrow{{\mathbf{h}}}_{0}^{char}+{\mathbf{b}} \tag{6}\]

where \({\mathbf{W}}_{f}\), \({\mathbf{W}}_{b}\) and \({\mathbf{b}}\) are parameters.

Char CNNKim et al. (2016) applies a convolution operation between subword sequence matrix \(\mathbf{C}\) and a filter \(\mathbf{F}\in\mathbb{R}^{d^{\prime}\times l}\) of width \(l\) to obtain a feature map \(\mathbf{f}\in\mathbb{R}^{k-l+1}\):

\[\mathbf{f}_{j}=tanh(\langle\mathbf{C}[*,j:j+l-1],\mathbf{F}\rangle+b) \tag{7}\]

where \(\langle A,B\rangle=Tr({\mathbf{A}}{\mathbf{B}}^{T})\) is the Frobenius inner product, \(b\) is a bias and \(\mathbf{f}_{j}\) is the \(j\)th element of \(\mathbf{f}\). We then take the max-over-time operation to select the most important element:

\[y_{\mathbf{f}}=\max_{j}\mathbf{f}_{j}.\] 

After applying \(q\) filters with varied lengths, we can get the array \({\mathbf{w}}=[y_{1},...,y_{q}]\), which is followed by a one-layer highway network to generate final word embedding.

### Auxiliary Language Modeling (LM)

We adapted a multi-task structure, originally proposed by Rei et al. (2017) for sequential tagging, to further improve the subword representations in sentence pair modeling. In addition to training the model for sentence pair tasks, we used a secondary language modeling objective that predicts the next word and previous word using softmax over the hidden states of Bi-LSTM as follows:

\[\overrightarrow{E}_{LM} =-\sum_{t=1}^{T-1}(\log(P(w_{t+1}|\overrightarrow{{\mathbf{m}}_{t}})) \tag{9}\] \[\overleftarrow{E}_{LM} =-\sum_{t=2}^{T}(\log(P(w_{t-1}|\widehat{{\mathbf{m}}_{t}})) \tag{10}\]

where \(\overrightarrow{{\mathbf{m}}_{t}}=tanh(\overrightarrow{{\mathbf{W}}}_{hm} \overrightarrow{{\mathbf{h}}_{t}})\) and \(\widehat{{\mathbf{m}}_{t}}=tanh(\overleftarrow{{\mathbf{W}}}_{hm}\overleftarrow{{ \mathbf{h}}_{t}})\). The Bi-LSTM here is separate from the one in PWI model. The language modeling objective can be combined into sentence pair modeling through a joint objective function:

\[E_{joint}=E+\gamma(\overrightarrow{E}_{LM}+\overleftarrow{E}_{ LM}), \tag{11}\]

which balances subword-based sentence pair modeling objective \(E\) and language modeling with a weighting coefficient \(\gamma\).

## 3 Experiments

### Datasets

We performed experiments on three benchmark datasets for paraphrase identification; each contained pairs of naturally occurring sentences manually labeled as paraphrases and non-paraphrases for binary classification: **Twitter URL**Lan et al. (2017) was collected from tweets sharing the same URL with major news outlets such as @CNN. This dataset keeps a balance between formal and informal language. **PIT-2015**Xu et al. (2014, 2015) comes from the Task 1 of Semeval 2015 and was collected from tweets under the same trending topic, which contains varied topics and language styles. **MSRP**Dolan and Brockett (2005) was derived from clustered news articles reporting the same event in formal language. Table 1 shows vital statistics for all three datasets.

### Settings

To compare models fairly without implementation variations, we reimplemented all models into a single PyTorch framework.1 We followed the setups in He and Lin (2016) and Lan et al. (2017) for the pairwise word interaction model, and used the 200-dimensional GloVe word vectors Pennington et al. (2014), trained on 27 billion words from Twitter (vocabulary size of 1.2 milion words) for social media datasets, and 300-dimensional GloVe vectors, trained on 840 billion words (vocabulary size of 2.2 milion words) from Common Crawl for the MSRP dataset. For cases without pretraining, the word/subword vectors were initialized with random samples drawn uniformly from the range [0.05, 0.05]. We used the same hyperparameters in the C2W Ling et al. (2015) and CNN-based Kim et al. (2016) compositions for subword models, except that the composed word embeddings were set to 200- or 300- dimensions as the pretrained word embeddings to make experiment results more comparable. For each experiment, we reported results with 20 epochs.

Footnote 1: The code and data can be obtained from the first and second author’s websites.

### Results

Table 2 shows the experiment results on three datasets. We reported maximum F1 scores of any point on the precision-recall curve Lipton et al. (2014) following previous work.

Word ModelsThe word-level pairwise interaction models, even without pretraining (randomzied) or fine-tuning (fixed), showed strong performance across all three datasets. This reflects

\begin{table}
\begin{tabular}{c c c c c c c} \hline Dataset & Training Size & Test Size & \# INV & \# OOV & OOV Ratio & Source \\ \hline PIT-2015 & 11530 & 838 & 7771 & 1238 & 13.7\% & Twitter trends \\ Twitter-URL & 42200 & 9324 & 24905 & 11440 & 31.5\% & Twitter/news \\ MSPR & 4076 & 1725 & 16226 & 1614 & 9.0\% & news \\ \hline \end{tabular}
\end{table}
Table 1: Statistics of three benchmark datasets for paraphrase identification. The training and testing sizes are in numbers of sentence pairs. The number of unique in-vocabulary (INV) and out-of-vocabulary (OOV) words are calculated based on the publicly available GloVe embeddings (details in Section 3.2).

 

the effective design of the BiLSTM and word interaction layers, as well as the unique character of sentence pair modeling, where n-gram overlapping positively signifies the extent of semantic similarity. As a reference, a logistic regression baseline with simple n-gram (also in stemmed form) overlapping features can also achieve good performance on PIT-2015 and MSRP datasets. With that being said, pretraining and fine-tuning word vectors are mostly crucial for pushing out the last bit of performance from word-level models.

Subword Models (+LMs)Without using any pretrained word embeddings, subword-based pairwise word interaction models can achieve very competitive results on social media datasets compared with the best word-based models (pretrained, fixed). For MSRP with only 9% of OOV words (Table 1), the subword models do not show advantages. Once the subword models are trained with multi-task language modeling (Subword+LM), the performance on all datasets are further improved, outperforming the best previously reported results by neural models Lan et al. (2017). A qualitative analysis reveals that subwords are crucial for out-of-vocabulary words while language modeling ensures more semantic and syntactic compatibility (Table 3).

### Combining Word and Subword Representations

In addition, we experimented with combining the pretrained word embeddings and subword models with various strategies: concatenation, weighted average, adaptive models Miyamoto and Cho (2016) and attention models Rei et al. (2016). The weighted average outperformed all others but only showed slight improvement over word-based models in social media datasets; other combination strategies could even lower the performance. The best performance was 0.763 F1 in Twitter-URL and 0.671 in PIT-2015 with a weighted average: 0.75 \(\times\) word embedding \(+\) 0.25 \(\times\) subword embedding.

## 4 Model Ablations

In the original PWI model, He and Lin (2016) performed pattern recognition of complex semantic relationships by applying a 19-layer deep convolutional neural network (CNN) on the word pair interaction tensor (Eq. 5). However, the SemEval task on Interpretable Semantic Textual Similarity

\begin{table}
\begin{tabular}{c c c c c c c} \hline  & Model Variations & pre-train & \#parameters & Twitter URL & PIT-2015 & MSRP \\ \hline \multirow{8}{*}{Word Models} & Logistic Regression & – & – & 0.683 & 0.645 & 0.829 \\  & Lan et al. (2017) & Yes & 9.5M & 0.749 & 0.667 & 0.834 \\ \cline{2-7}  & pretrained, fixed & Yes & 2.2M & 0.753 & 0.632 & 0.834 \\  & pretrained, updated & Yes & 9.5M & 0.756 & 0.656 & 0.832 \\  & randomized, fixed & – & 2.2M & 0.728 & 0.456 & 0.821 \\  & randomized, updated & – & 9.5M & 0.735 & 0.625 & 0.834 \\ \hline \hline \multirow{8}{*}{Subword Models} & CZW, unigram & – & 2.6M & 0.742 & 0.534 & 0.816 \\  & C2W, bigram & – & 2.7M & 0.742 & 0.563 & 0.825 \\ \cline{1-1}  & C2W, trigram & – & 3.1M & 0.729 & 0.576 & 0.824 \\ \cline{1-1}  & CNN, unigram & – & 6.5M & 0.756 & 0.589 & 0.820 \\ \cline{1-1}  & CNN, bigram & – & 6.5M & 0.760 & 0.646 & 0.814 \\ \cline{1-1}  & CNN, trigram & – & 6.7M & 0.753 & 0.667 & 0.818 \\ \hline \hline \multirow{8}{*}{Subword+LM} & LM, CZW, unigram & – & 3.5M & 0.760 & **0.691** & 0.831 \\ \cline{1-1}  & LM, CZW, bigram & – & 3.6M & **0.768** & 0.651 & 0.830 \\ \cline{1-1}  & LM, CZW, trigram & – & 4.0M & 0.765 & 0.659 & 0.831 \\ \cline{1-1}  & LM, CNN, unigram & – & 7.4M & 0.754 & 0.665 & **0.840** \\ \cline{1-1}  & LM, CNN, bigram & – & 7.4M & 0.761 & 0.667 & 0.835 \\ \cline{1-1}  & LM, CNN, trigram & – & 7.6M & 0.759 & 0.667 & 0.831 \\ \hline \end{tabular}
\end{table}
Table 2: Results in F1 scores on Twitter-URL, PIT-2015 and MSRP datasets. The best performance figure in each dataset is denoted in **bold** typeface and the second best is denoted by an underline. Without using any pretrained word embeddings, the Subword+LM models achieve better or competitive performance compared to word models.

\begin{table}
\begin{tabular}{|c|c|l l|l|} \hline Model & \multicolumn{2}{c|}{INV Words} & \multicolumn{2}{c|}{OOV Words} \\ \cline{2-5}  & _any_ & _walking_ & _Fairport_ & _break_ \\ \hline \multirow{4}{*}{Word} & _anything_ & _walk_ & _salomon_ & _bolocks_ \\  & _anosine_ & _running_ & _163_ & _missographic_ \\  & _other_ & _dead_ & _pharmacobed_ & _paratology_ \\  & _there_ & _around_ & _hallways_ & _sexist_ \\ \hline \multirow{4}{*}{Subword} & _analogy_ & _waffling_ & _buni Airport_ & _grexi_ \\  & _any_ & _sleepsing_ & _Häglrobert_ & _hert_ \\  & _any_ & _scaling_ & _placeombo_ & _jullet_ \\  & _andy_ & _shocking_ & _guangeusoum_ & _thrust_ \\ \hline \multirow{4}{*}{Subword + LM} & _any1_ & _warning_ & _airport_ & _phoneit_ \\  & _may_ & _wagging_ & _fairports_ & _brri_ \\ \cline{1-1}  & _ang_ & _using_ & _support_ & _officext_ \\ \cline{1-1}  & _annoy_ & _waiting_ & _Bautoceptureport_ & _drought-hit_ \\ \hline \end{tabular}
\end{table}
Table 3: Nearest neighbors of word vectors under cosine similarity in Twitter-URL dataset.

 

(Agirre et al., 2016) in part demonstrated that the semantic relationship between two sentences depends largely on the relations of aligned words or chunks. Since the interaction tensor in the PWI model already encodes word alignment information in the form of vector similarities, a natural question is whether a 19-layer CNN is necessary.

Table 4 shows the results of our systems with and without the 19-layer CNN for aggregating the pairwise word interactions before the final softmax layer. While in most cases the 19-layer CNN helps to achieve better or comparable performance, it comes at the expense of \(\sim\)25% increase of training time. An exception is the character-based PWI without language model, which performs well on the PIT-2015 dataset without the 19-layer CNN and comparably to logistic regression with string overlap features (Eyecioglu and Keller, 2015). A closer look into the datasets reveals that PIT-2015 has a similar level of unigram overlap as the Twitter URL corpus (Table 5),2 but lower character bigram overlap (indicative of spelling variations) and lower word bigram overlap (indicative of word reordering) between the pairs of sentences that are labeled as paraphrase.

Footnote 2: See more discussions in Lan et al. (2017).

The 19-layer CNN appears to be crucial for the MSRP dataset, which has the smallest training size and is skewed toward very high word overlap.3 For the two social media datasets, our subword models have improved performance compared to pretrained word models regardless of having or not having the 19-layer CNN.

Footnote 3: See more discussions in Lan et al. (2017).

## 5 Conclusion

We presented a focused study on the effectiveness of subword models in sentence pair modeling and showed competitive results without using pretrained word embeddings. We also showed that subword models can benefit from multi-task learning with simple language modeling, and established new start-of-the-art results for paraphrase identification on two Twitter datasets, where out-of-vocabulary words and spelling variations are profound. The results shed light on future work on language-independent paraphrase identification and multilingual paraphrase acquisition where pretrained word embeddings on large corpora are not readily available in many languages.

#### Acknowledgments

We thank John Wieting and Mikhail Belkin for valuable discussions, and Ohio Supercomputer Center Center (Center, 2012) for computing resources. Funding was provided by the U.S. Army Research Office (ARO) and Defense Advanced Research Projects Agency (DARPA) under Contract Number W911NF-17-C-0095. The content of the information in this document does not necessarily reflect the position or the policy of the Government, and no official endorsement should be inferred. The U.S. Government is authorized to reproduce and distribute reprints for government purposes notwithstanding any copyright notation here on.

\begin{table}
\begin{tabular}{c c c c c c c c} \hline  & Model Variations & CNN\({}^{19}\) & \#parameters & \#hours/epoch & Twitter URL & PIT-2015 & MSRP \\ \hline \multirow{2}{*}{Word Models} & Logistic Regression & – & – & – & 0.683 & 0.645 & 0.829 \\ \cline{2-8}  & pretrained, fixed & Yes & 2.2M & 4.5h & 0.753 & 0.632 & 0.834 \\  & & – & 1.4M & 3.2h & 0.741 & 0.602 & 0.827 \\ \hline \hline \multirow{3}{*}{Subword Models} & C2W, unigram & Yes & 2.6M & 5.8h & 0.742 & 0.534 & 0.816 \\  & & – & 1.4M & 4.6h & 0.741 & 0.655 & 0.808 \\ \cline{2-8}  & CNN, unigram & Yes & 6.5M & 5.4h & 0.756 & 0.589 & 0.820 \\  & & – & 5.3M & 4.2h & 0.759 & 0.659 & 0.809 \\ \hline \hline \multirow{3}{*}{Subword+LM} & LM, C2W, unigram & Yes & 3.5M & 6.5h & **0.760** & **0.691** & 0.831 \\  & & – & 2.3M & 5.3h & 0.746 & 0.625 & 0.811 \\ \cline{1-1} \cline{2-8}  & LM, CNN, unigram & Yes & 7.4M & 5.8h & 0.754 & 0.665 & **0.840** \\ \cline{1-1} \cline{2-8}  & LM, CNN, unigram & – & 6.2M & 4.6h & 0.758 & 0.659 & 0.809 \\ \hline \end{tabular}
\end{table}
Table 4: Comparison of F1 scores between the original PWI model with 19-layer CNN for aggregation and the simplified model without 19-layer CNN on Twitter-URL, PIT-2015 and MSRP datasets. The number of parameters and training time per epoch shown are based on the Twitter URL dataset and a single NVIDIA Pascal P100 GPU.

\begin{table}
\begin{tabular}{l|r r r|r} \multicolumn{4}{l}{} & \multicolumn{1}{c|}{Twitter URL} & PIT-2015 & MSRP \\ \hline \#char unigrams in shorter sentence (all) & 67.5 & 36.2 & 109.0 \\ \#chur unigrams in longer sentence (all) & 97.7 & 50.5 & 128.5 \\ \#chr unigrams of the unin (all) & 101.1 & 53.0 & 130.0 \\ \#char unigrams of the intersection (all) & 64.1 & 37.7 & 107.4 \\ char unigram overlap (ungleram-only) & 63.4 & 63.5u & 82.6g \\ char unigram overlap (paraphrase-only) & 68.8f & 67.0k & 84.7a \\ \hline char bigram overlap (all) & 30.8v & 31.6v & 67.4v \\ char bigram overlap (paraphrase-only) & 48.2v & 42.4x & 71.6v \\ \hline word unigram overlap (pair) & 13.3v & 21.7e & 54.8v \\ \hline word higram overlap (paraphrase-only) & 32.0m & 30.2v & 59.1v \\ \hline word higram overlap (all) & 5.3v & 8.4v & 33.2v \\ word bigram overlap (paraphrase-only) & 17.9v & 12.3v & 36.8v \\ \hline \end{tabular}
\end{table}
Table 5: Character and word overlap comparison.

 



## References

* Agirre et al. (2015) Eneko Agirre, Carmen Banea, Claire Cardiie, Daniel Cer, Mona Diab, Aitor Gonzalez-Agirre, Weiwei Guo, Inigo Lopez-Gazpio, Montse Maritxalar, Rada Mihalcea, German Rigau, Larraitz Uria, and Janyce Wiebe. 2015. SemEval-2015 task 2: Semantic textual similarity, English, Spanish and pilot on interpretability. In _Proceedings of the 9th International Workshop on Semantic Evaluation (SemEval)_.
* Agirre et al. (2016) Eneko Agirre, Aitor Gonzalez-Agirre, Inigo Lopez-Gazpio, Montse Maritxalar, German Rigau, and Larraitz Uria. 2016. SemEval-2016 task 2: Interpretable semantic textual similarity. In _Proceedings of the 10th International Workshop on Semantic Evaluation (SemEval)_.
* Baldwin et al. (2013) Timothy Baldwin, Paul Cook, Marco Lui, Andrew MacKinlay, and Li Wang. 2013. How noisy social media text, how diffrnt social media sources? In _Proceedings of the Sixth International Joint Conference on Natural Language Processing (IJCNLP)_.
* Bowman et al. (2015) Samuel R. Bowman, Gabor Angeli, Christopher Potts, and Christopher D. Manning. 2015. A large annotated corpus for learning natural language inference. In _Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing (EMNLP)_.
* Center (2012) Ohio Supercomputer Center. 2012. Oakley supercomputer. [http://osc.edu/ark](http://osc.edu/ark);/19495/hpc0cvqn.
* Costa-jussa and Fonollosa (2016) Marta R. Costa-jussa and Jose A. R. Fonollosa. 2016. Character-based neural machine translation. In _Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (ACL)_.
* Dolan et al. (2004) Bill Dolan, Chris Quirk, and Chris Brockett. 2004. Unsupervised construction of large paraphrase corpora: Exploiting massively parallel news sources. In _Proceedings of the 20th International Conference on Computational Linguistics (COLING)_.
* Dolan and Brockett (2005) William B Dolan and Chris Brockett. 2005. Automatically constructing a corpus of sentential paraphrases. In _Proceedings of the Third International Workshop on Paraphrasing (IWP)_.
* Santos and Guimaraes (2015) Cicero dos Santos and Victor Guimaraes. 2015. Boosting named entity recognition with neural character embeddings. In _Proceedings of the Fifth Named Entity Workshop_.
* Eyecioglu and Keller (2015) Asli Eyecioglu and Bill Keller. 2015. ASOBEK: Twitter paraphrase identification with simple overlap features and SVMs. In _Proceedings of the 9th International Workshop on Semantic Evaluation (SemEval)_.
* He and Lin (2016) Hua He and Jimmy Lin. 2016. Pairwise word interaction modeling with deep neural networks for semantic similarity measurement. In _Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NACL-HLT)_.
* Kim et al. (2016) Yoon Kim, Yacine Jerinte, David Sontag, and Alexander M Rush. 2016. Character-aware neural language models. In _Proceedings of the Thirtieth AAAI Conference on Artificial Intelligence (AAAI)_.
* Lan et al. (2017) Wuwei Lan, Siyu Qiu, Hua He, and Wei Xu. 2017. A continuously growing dataset of sentential paraphrases. In _Proceedings of The 2017 Conference on Empirical Methods on Natural Language Processing (EMNLP)_.
* Ling et al. (2015) Wang Ling, Tiago Luis, Luis Marujo, Ramon Fernandez Astudillo, Silvio Amir, Chris Dyer, Alan W Black, and Isabel Trancoso. 2015. Finding function in form: Compositional character models for open vocabulary word representation. In _Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing (EMNLP)_.
* Lipton et al. (2014) Zachary C. Lipton, Charles Elkan, and Balakrishnan Narayanamy. 2014. Optimal thresholding of classifiers to maximize F1 measure. In _Proceedings of the 2014th European Conference on Machine Learning & Principles and Practice of Knowledge Discovery in Databases (ECML-PKDD)_.
* Liu et al. (2016) Pengfei Liu, Xipeng Qiu, Yaqian Zhou, Jifan Chen, and Xuanjing Huang. 2016. Modelling interaction of sentence pair with coupled-LSTMs. In _Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing (EMNLP)_.
* Luong et al. (2015) Thang Luong, Ilya Sutskever, Quoc Le, Oriol Vinyals, and Wojciech Zaremba. 2015. Addressing the rare word problem in neural machine translation. In _Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (ACL-IJCNLP)_.
* Miyamoto and Cho (2016) Yasumasa Miyamoto and Kyunghyun Cho. 2016. Gated word-character recurrent language model. In _Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing (EMNLP)_.
* Parikh et al. (2016) Ankur Parikh, Oscar Tackstrom, Dipanjan Das, and Jakob Uszkoreit. 2016. A decomposable attention model for natural language inference. In _Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing (EMNLP)_.
* Pennington et al. (2014) Jeffrey Pennington, Richard Socher, and Christopher D. Manning. 2014. GloVe: Global vectors for word representation. In _Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP)_.
* Plank et al. (2016) Barbara Plank, Anders Sogaard, and Yoav Goldberg. 2016. Multilingual part-of-speech tagging with bidirectional long short-term memory models and auxiliary loss. In _Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics_

* Rei (2017) Marek Rei. 2017. Semi-supervised multitask learning for sequence labeling. In _Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (ACL)_.
* Rei et al. (2016) Marek Rei, Gamal Crichton, and Sampo Pyysalo. 2016. Attending to characters in neural sequence labeling models. In _Proceedings of the 26th International Conference on Computational Linguistics: Technical Papers (COLING)_.
* Shen et al. (2017) Gehui Shen, Yunlun Yang, and Zhi-Hong Deng. 2017. Inter-weighted alignment network for sentence pair modeling. In _Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing (EMNLP)_.
* Tomar et al. (2017) Gaurav Singh Tomar, Thyago Duque, Oscar Tackstrom, Jakob Uszkoreit, and Dipanjan Das. 2017. Neural paraphrase identification of questions with noisy pretraining. In _Proceedings of the First Workshop on Subword and Character Level Models in NLP (SCLeM)_.
* Vania and Lopez (2017) Clara Vania and Adam Lopez. 2017. From characters to words to in between: Do we capture morphology? In _Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (ACL)_.
* Wang et al. (2017) Zhiguo Wang, Wael Hamza, and Radu Florian. 2017. Bilateral multi-perspective matching for natural language sentences. In _Proceedings of the Twenty-Sixth International Joint Conference on Artificial Intelligence (IJCAI)_.
* Wieting et al. (2015) John Wieting, Mohit Bansal, Kevin Gimpel, and Karen Livescu. 2015. From paraphrase database to compositional paraphrase model and back. _Transactions of the Association for Computational Linguistics (TACL)_ 3:345-358.
* Wieting et al. (2016) John Wieting, Mohit Bansal, Kevin Gimpel, and Karen Livescu. 2016. Towards universal paraphrastic sentence embeddings. In _Proceedings of the 4th International Conference on Learning Representations (ICLR)_.
* Xu et al. (2015) Wei Xu, Chris Callison-Burch, and William B. Dolan. 2015. SemEval-2015 task 1: Paraphrase and semantic similarity in Twitter. In _Proceedings of the 9th International Workshop on Semantic Evaluation (SemEval)_.
* Xu et al. (2014) Wei Xu, Alan Ritter, Chris Callison-Burch, William B. Dolan, and Yangfeng Ji. 2014. Extracting lexically divergent paraphrases from Twitter. _Transactions of the Association for Computational Linguistics (TACL)_ 2(1).
* Yin et al. (2016) Wenpeng Yin, Hinrich Schutze, Bing Xiang, and Bowen Zhou. 2016. ABCNN: Attention-based convolutional neural network for modeling sentence pairs. _Transactions of the Association for Computational Linguistics (TACL)_ 4:259-272.