# Automatic Construction of Fine-Grained Paraphrase Corpora System Using Language Inference Model
## 5 Fine-Grained Label Correctness and Accuracy Investigation
### Original Label vs. Fine-Grained Label  
Table 10 shows the properties of fine-grained classes with respect to each dataset's original class or value. For STS-B, the average similarity score of each new label is calculated. For MRPC and QQP, the positive paraphrase percentage of each new label is calculated. The results on STS-B are consistent with the definition of new labels. The pairs with equivalence (Label 3) have the highest average similarity score: 4.62. Directional entailment pairs have lower average similarity scores and there is not much difference in similarity score between forward (3.69) and reverse entailment (3.71). Label 4 has a lower average than the other three as the relation is either independence or alternation. Label 0 represents negation relation. It has the lowest similarity score. This suggests that our relabelling rule is able to identify the semantic difference among fine-grained classes. The results on MRPC and QQP are in general as expected. Both datasets contain pairs belonging to various entailment relations in the positive class. We expect to see a very high percentage of positive paraphrases in the newly assigned equivalent class (\(\equiv\)). We also expect to see a relatively high percentage of positive paraphrases in the other two entailment classes (\(\sqsubset\) and \(\sqsupset\)). In the MRPC dataset, 99.1% of the pairs labelled as \(\equiv\) are positive paraphrases, while 81% of the pairs labelled as \(\sqsubset\) and 82% of the pairs labelled as \(\sqsupset\) are positive paraphrases. In the QQP dataset, 86.2% of the pairs labelled as \(\equiv\) are positive paraphrases, while 52.6% of the pairs labelled as \(\sqsubset\) and 56.7% of the pairs labelled as \(\sqsupset\) are positive paraphrases. Overall, the results confirms that MRPC tolerates more semantic divergence in positive labels. It has a higher percentage of forward and reverse entailment pairs labelled as positive paraphrases. This might be the results of different annotation practices adopted while constructing the dataset. Theoretically, all pairs labelled as \(\equiv\) should have a positive label in the original dataset. Yet, there are 0.9% \(\equiv\) pairs in MRPC and 13.8% \(\equiv\) pairs in QQP that have a negative label in the original dataset. Our further analysis shows that some are caused by original labelling issue and some are caused by classifier noise. We do find that QQP contains a fair bit of false negative samples. We suspect this could be caused by relying on Quora users to indicate if questions are a duplicate. Since users are not expected to have seen all questions, the dataset is bound to have relatively high false negative samples. We also find that both the MNLI and SNLI classifier tend to make a wrong prediction in sentences with an ambiguous pronoun reference. An interesting example is the sentence pair from MRPC with a negative label: \"NBC will probably end the season as the second most popular network behind CBS, although it's first among the key 18-to-49-year-old demographic.\" and \"NBC will probably end the season as the second most-popular network behind CBS, which is first among the key 18-to-49-year-old demographic.\" Both the original and swapped pairs are classified as \"entailment\". \begin{table}
\begin{tabular}{c c c c c c c} \hline \hline  & \multicolumn{6}{c}{**Fine-Grained Class Label**} \\ \hline
**Metrics** & **Data Sets** & **0 (\(\wedge\))** & **1 (\(\sqsubset\))** & **2 (\(\sqsubset\))** & **3 (\(\equiv\))** & **4 (\(|\)or\#)** \\ \hline
**Average Similarity Score** & **STS-B** & 1.19 & 3.69 & 3.71 & 4.62 & 3.06 \\ \hline \multirow{2}{*}{**Positive Paraphrase Percent**} & **MRPC** & 39.6\% & 81\% & 82.0\% & 99.1\% & 49.9\% \\ \cline{2-7}  & **QQP** & 6.4\% & 52.6\% & 56.7\% & 86.2\% & 20.8\% \\ \hline \hline \end{tabular}
\end{table} 
Table 10: Positive paraphrase percentage of each fine-grained paraphrase label. We also expect a very small percentage of positive labels in the negation (\(\wedge\)) class.