# Automatic Construction of Fine-Grained Paraphrase Corpora System Using Language Inference Model
## 3 Auto Relabelling Methods
### Observations from Language Inference Datasets  
Instead of manually labelling, we propose a method utilising the existing labels in language inference dataset. In particular, we utilise datasets with three labels: \"entailment\", \"neutral\" and \"contradiction\". There are two such datasets: the Stanford Natural Language Inference corpus (**SNLI**) [18] and the Multi-Genre Natural Language Inference (**MNLI**) corpus [19]. They are constructed following a similar process where AMTs are given a prompt sentence (the premise) and are asked to write three sentences (the hypotheses): one that is definitely true (\"entailment\"; one that is probably true (\"neutral\") and one that is definitely not true (\"contradiction\"). The SNLI corpus contains 570,000 sentence pairs sourced from image captions. The MNLI corpus [19] is of an approximately similar size but with various genres of spoken and written text, including transcripts, government reports and fictions. The construction process ensures that both datasets maintain balanced samples in the three classes.  
\begin{table}
\begin{tabular}{c c c} \hline \hline
**Symbol** & **Name** & **Example** \\ \hline \(x\sqsubset y\) & forward entailment & _crow_\(\sqsubset\)_bird_ \\ \hline \(x\sqsupset y\) & reverse entailment. & _Asian_\(\sqsupset Thai\) \\ \hline \(x\equiv y\) & equivalence & _couch_\(\equiv sofa\) \\ \hline \(x\mid y\) & alternation & _cat_\(\mid\)_dog_ \\ \hline \(x\wedge y\) & negation & _able_\(\wedge\)_unable_ \\ \hline \(x\smile y\) & cover & _animal_\(\smile\)_non-_ope_ \\ \hline \(x\mathbin{\#}y\) & independence & _huugry_\(\mathbin{\#}\)_hippo_ \\ \hline \hline \end{tabular}
\end{table}
Table 2: Word level entailment relation examples in [PERSON]’s thesis [17].  
\begin{table}
\begin{tabular}{c l l} \hline \hline
**Relation** & **First** & **Second** \\ \hline \(\sqsubset\) & A young child is riding a horse. & A child is riding a horse. \\ \hline \(\sqsupset\) & Three men are playing guitars. & Three men are on stage playing guitars. \\ \hline \(\equiv\) & The man cut down a tree with an axe. & A man chops down a tree with an axe. \\ \hline \multirow{4}{*}{\(\mid\)} & The report also claims that there will be & There will be 9.3 million visitors to hot spots \\  & up to 9.3 million visitors to hot spots & spots in 2003, up from 2.5 million in \\ \cline{1-1}  & this year, up again from the meagre & 2002, [PERSON] said. \\ \cline{1-1}  & 2.5 million in 2002. & \\ \hline \hline \end{tabular}
\end{table}
Table 3: Fine-grained paraphrase example.  
The three classes only represent a subset of all possible sentence relations. Each class may contain sentence pairs belonging to a few fine-grained relations. In particular, we observe that:  
* The \"entailment\" label contains forward entailment (\(\sqsubseteq\) ) and equivalent (\(\equiv\)) pairs. This conforms to the definition of \"entailment\" class. The first two rows in Table 4 show sample \"entailment\" pairs from MNLI. The first pair is an example of forward entailment while the second one is an example of bidirectional entailment.
* The \"neutral\" class contains reverse entailment pairs, alternate pairs, independent pairs and pairs of other relations. It seems to be the result that \"neutral\" is designed as a catch-all class. There are also cases where event and entity coreference could make the distinction between \"neutral\" and \"contradiction\" ambiguous. The last three rows in Table 4 shows three sample \"neutral\" pairs from MNLI.  
Such observations enable us to define rules for extracting fine-grained sentence pair relations from existing datasets.