# Paraphrase Identification with Lexical, Syntactic and Sentential Encodings
## 4 Experiments
### Qualitative Analysis of Errors  
We performed an error analysis by using the MRPC dataset to provide feedback for further improvement of our method. The number of false-positive and false-negative pairs of sentences was 61 and 38, respectively. These errors have occurred even though we used all the features or any combination of these features. We found that there are mainly three types of errors. 1. **Inclusion relation between sentences:** As we mentioned in Table 3, this error is that two sentences share the same contents but one sentence has more detailed information of the other. 1. \"There's a Jeep in my parents' yard right now that's not theirs\", said [PERSON], whose parents are vacationing in North Carolina. 2. \"There's a Jeep in my parents' yard right now that's not theirs\", she said. Sentence (1) and (2) are similar content and our model identified these sentences as paraphrases. However, according to the Microsoft Research definitions, [[https://www.microsoft.com/en-us/download/details.aspx?id=52398](https://www.microsoft.com/en-us/download/details.aspx?id=52398)]([https://www.microsoft.com/en-us/download/details.aspx?id=52398](https://www.microsoft.com/en-us/download/details.aspx?id=52398)) these sentences should be identified as \"non-paraphrase\" because the sentence (1) includes the information marked with the underlined that \"[PERSON]'s parents are vacationing in North Carolina\" and it is a significantly larger superset of the sentence (2). We observed that 39 pairs were classified into this type. 2. **Dependency relation:** Dependency relation within a sentence is not correctly analyzed. For example, in the sentence (3), \"<.DDJ>\" is divided into four tokens(\"<\", \"..\", \"DDJ\", and \">\") by BERT tokenizer. As a result, the Stanford parser incorrectly analyzed that \">\" modifies \"added\" with adverb modifier (advmod) relation. In total, 10 pairs of sentences were classified into this type. \begin{table}
\begin{tabular}{l c c} \hline \hline \multicolumn{3}{c}{**MRPC Dataset**} \\ \hline
**Model** & **Acc** & **F** \\ \hline LSSE (Our model) & **86.3** & **89.9** \\ \hline -PE & 85.3 & 89.0 \\ -SentE & 84.2 & 88.3 \\ -SentE and -PE & 83.5 & 88.1 \\ -R-GCNs & 85.7 & 89.5 \\ -R-GCNs and -SentE & 83.0 & 87.3 \\ -R-GCNs and -BERT tokenE & 84.3 & 88.1 \\ \hline \hline \end{tabular}
\end{table} 
Table 6: Ablation test: “PE” refers to position encoding and “SentE” indicates sentential encoding. “BERT tokenE” stands for lexical encoding by BERT. “-X” indicates the result by using LSSE without “X”. Bold font shows the best result. 3. The Dow Jones industrial average <.DJI> added 28 points, or 0.27 percent, at 10,557, hitting its highest level in 21 months. 3. **Inter-sentential relations:** Two sentences which have inter-sentential relations are difficult to interpret correctly whether these sentences are paraphrase or not. (4) British Airways' _New York-to-London_ runs will end in October. (5) British Airways plans to retire its seven _Concordes_ at the end of October. Sentences (4) and (5) have the same sense, while different expressions such as \"New York-to-London\" and \"Concordes\" are used and they are co-referred entities. To identify these sentences as \"paraphrases\" correctly, it requires not only local dependency, i.e., dependency structure within a sentence but also non-local dependency between sentences. There were nine pairs classified into this type. Apart from these observations, we found that when the number of arcs from other nodes is small, the performance of R-GCNs has not improved because convolution mixes the features of a node and its neighbors. One solution is to incorporate more linguistics information such as tree-based structure [62; 63], Named Entity Recognition, and Co-Reference Resolution into our framework to represent rich relations among nodes. This is a rich space for further exploration. We recall that our model for lexical and sentential encodings are based on the BERT.