# Paraphrase Identification with Lexical, Syntactic and Sentential Encodings
## 4 Experiments
### Main Results  
Table 2 shows the results by using MRPC data (Supplementary Materials). We can see from Table 2 that our model outperformed the baseline, BERT sentential encodings, by 2.0% accuracy and 1.8% F1 on the MRPC and 1.9% accuracy and 2.0% F1 on the QQP data. Why did our LSSE perform particularly strong on the dataset QQP? We notice that the volume of this dataset is larger than that of the MRPC dataset. This confirms our intuition that deep learning typically requires more training data to achieve high performance, and our model could successfully take this advantage on the QQP dataset. \begin{table}
\begin{tabular}{l c c c c} \hline \hline
**Data** & **\#1 ID** & **\#2 ID** & **\#1 String** & **\#2 String** & **Label** \\ \hline \multirow{6}{*}{MRPC} & \multirow{6}{*}{2108705} & \multirow{6}{*}{2108831} & Yucaipa owned Dominick’s before selling the chain to Safeway in 1998 for USD 693 million and sold it Safeway for USD 1.8 billion in 1998. & \multirow{6}{*}{0} \\  & & & 2.5 billion. & 1.8 billion in 1998. & \\ \cline{2-4}  & \multirow{6}{*}{702876} & \multirow{6}{*}{702977} & [PERSON] accused his brother, whom he called “the witness”, of deliberately distorting his evidence. & Referring to him as only “the witness”, [PERSON] accused his brother of deliberately distorting his evidence. & \multirow{6}{*}{1} \\ \cline{2-4}  & & & What causes stool color to change to yellow? & What can cause stool to come out as little balls? & \multirow{6}{*}{0} \\ \cline{2-4}  & \multirow{6}{*}{536040} & \multirow{6}{*}{536041} & How do I control my horny emotions? & How do you control your horniness? & \\ \cline{1-1} \cline{3-4}  & & & How do I control my horny emotions? & How do you control your horniness? & \\ \cline{1-1} \cline{3-4}  & & & & your horniness? & \\ \hline \hline \end{tabular}
\end{table} 
Table 1: Sentence pairs from MRPC and QQP datasets: Label indicates the ground-truth labels that the sentences are paraphrase (1) or non- paraphrase (0). \begin{table}
\begin{tabular}{l c c c c} \hline \hline \multirow{2}{*}{**Model**} & \multicolumn{2}{c}{**Baseline**} & \multicolumn{2}{c}{**LSSE**} \\ \cline{2-5}  & **Acc** & **F1** & **Acc** & **F1** \\ \hline MRPC & 84.3 & 88.1 & **86.3** & **89.9** \\ QQP & 88.7 & 88.4 & **90.6** & **90.4** \\ \hline \hline \end{tabular}
\end{table} 
Table 2: Main result by using test dataset: Baseline shows the result obtained by BERT sentential encodings [11]. Bold font shows the best result in each dataset. Table 3 shows some examples obtained by both of the models. In Table 3, TP, FP, TN, and FN refer to an abbreviation of true positive, false positive, true negative, and false negative, respectively. \"N\" indicates the number of instances from the test data. For example \"N = 70\" in Table 3 shows that the number of \"LSSE(TP) and BERT(FN)\", i.e., the sentence pairs that were classified by LSSE as true positive and classified by BERT as false negative is 70. We can see that the number of \"LSSE(TP) and BERT(FN)\" is larger than that of \"LSSE(FN) and BERT(TP)\" in both datasets. However, the number of \"LSSE(FP) and BERT(TN)\" is larger than that of \"LSSE(TN) and BERT(FP)\". Most of the errors of FP in our model are in the case that two sentences share the same contents but one sentence has more detailed information of the other. For example in the MRPC dataset, one sentence (#1 String) includes additional information, \"private creditors\", while it is not mentioned in the second sentence (#2 String). BERT sentential encodings is a simple paraphrase identification compared to our model. But why such a relatively simple model leads to a better prediction for particular test data is not clear at this point. Answering this question requires future research. We also examined how the percentage of training data affects overall performance. Figure 6 shows an F1-score against the percentage of the MRPC training data. We run ten times for each volume of training data size except for 100% and obtained the average F1-score.