Venue: NAACL 2016

# Learning to Recognize Ancillary Information for

Automatic Paraphrase Identification

 Simone Filice

DICII,

University of Roma, Tor Vergata

filice@info.uniroma2.it &Alessandro Moschitti\({}^{*}\)

ALT, Qatar Computing Research Institute,

HBKU

amoschitti@qf.org.qa

Professor at DISI, University of Trento.

###### Abstract

Previous work on Automatic Paraphrase Identification (PI) is mainly based on modeling text similarity between two sentences. In contrast, we study methods for automatically detecting whether a text fragment only appearing in a sentence of the evaluated sentence pair is important or ancillary information with respect to the paraphrase identification task. Engineering features for this new task is rather difficult, thus, we approach the problem by representing text with syntactic structures and applying tree kernels on them. The results show that the accuracy of our automatic Ancillary Text Classifier (ATC) is promising, i.e., 68.6%, and its output can be used to improve the state of the art in PI.

## 1 Introduction

Automatic PI is the task of detecting if two texts convey the same meaning. For example, the following two sentences from the Microsoft Research Paraphrase Corpus (MSRP) Dolan et al. (2004):

S\({}_{1a}\)_: Although it's unclear whether Sobig was to blame, The New York Times also asked employees at its headquarters yesterday to shut down their computers because of "system difficulties."_

S\({}_{1b}\)_: The New York Times asked employees at its headquarters to shut down their computers yesterday because of "computing system difficulties."_

are paraphrases, while these other two are not:

S\({}_{2a}\)_: Dr. Anthony Fauci, director of the National Institute of Allergy and Infectious Diseases, agreed._

S\({}_{2b}\)_: "We have been somewhat lucky," said Dr. Anthony Fauci, director of the National Institute of Allergy and Infectious Diseases._

Most previous work on automatic PI, e.g., Madnani et al. (2012); Socher et al. (2011), is based on a direct comparison between the two texts, exploiting different similarity scores into a machine learning framework. However, these methods consider sentences as monolithic units and can thus be misled by ancillary information that does not modify the main meaning expressed in the text.

For example, the additional text fragment (ATF), "_Although it's unclear whether Sobig was to blame_", from \(S_{1a}\) expresses ancillary information, which does not add much to the message of \(S_{1b}\), thus the sentences are considered paraphrases. In contrast, S\({}_{2b}\) contains the ATF, "_We have been somewhat lucky_", whose meaning is not linked to any constituent of S\({}_{1b}\). Since such text expresses relevant information, the two sentences are not considered paraphrases.

In this paper, we study and design models for extracting ATFs from a sentence with respect to another one and classifying if their meaning is ancillary or important. For this purpose, we built a corpus of sentence pairs using MSRP, where at least one pair member always contains ATFs. We use SVMs with tree kernels applied to syntactic representations Severyn and Moschitti (2012) of ATFs for learning automatic ATCs.

The results derived on MSRP show (_i_) a promising accuracy of our ATC and (_ii_) the output of ATC

can be used as a feature for improving the state-of-the-art PI model.

## 2 Ancillary clauses in PI

Our main purpose in studying computational approaches to the detection of ancillary information is its practical application to PI. Thus, given a pair of sentences (in general two texts), we define ATFs as ancillary information if their semantics:

1. only appears in one of the two sentences and
2. does not change the main meaning of the sentence, i.e., either the sentences are paraphrases or, if they are not, such ATFs are not the reason for their different meaning.

The definition above along with a syntactic representation of the sentences can be applied to a paraphrase corpus to build a dataset of ancillary vs. important ATFs. For example, Fig. 1 shows the shallow tree representation1 we proposed in Severyn and Moschitti (2012) of the sentences, S\({}_{2a}\) and S\({}_{2b}\), reported in the introduction, where the red label \(\rho\) indicates that there is a link between the lemmas of the two sentences (also shown by the dashed edges). \(\rho\) can be propagated to the upper nodes to mark the related constituents. For example, the lemma _National_ is matched by the two sentences, thus both its father node, NNP, and its grandfather constituent, NP, are marked.

Footnote 1: The shallow trees are constituted by four levels: (_i_) word lemmas as leaves, (_ii_) POS-tags as parent of lemmas, (_iii_) phrases grouping POS-tag nodes and (_iv_) a final root S.

Such representation makes the extraction of ATFs easier. For instance, Fig. 1 shows the text fragment of \(S_{2b}\), "_We have been somewhat lucky_", on the right. This is an ATF since it is not aligned with any fragments of \(S_{2a}\). Moreover, since it expresses a central information to the sentence meaning, \(S_{2b}\) cannot be in paraphrase relation with \(S_{2a}\). Conversely, the ATF of \(S_{1a}\), "_Although it's unclear whether Sobig was to blame_", is ancillary to the main meaning of the sentence, indeed, the annotators marked \(S_{1a}\) and \(S_{1b}\) as a valid paraphrase.

## 3 Building the ATC corpus

The previous section has shown an approach to extract ATFs that can be potentially ancillary. This uses an alignment approach based on lexical similarity, which may fail to align some text constituents. However, these mistakes only affect the precision in extracting ATFs rather than the recall. In other words, we can build a corpus that considers most cases of additional information.

In particular, we design the following simple heuristic: let \(F_{i}\) and \(F_{j}\) be the largest not aligned (possibly discontinuous) word sequences appearing in the sentence pair \((S_{i}\), \(S_{j})\), where \(F_{i}\in S_{i}\) and \(F_{j}\in S_{j}\). We define ATF as the largest text between \(F_{i}\) and \(F_{j}\) subject to \(d=|size(F_{i})-size(F_{j})|>\tau\), where \(size(F)\) is the number of words2 appearing in \(F\). If the condition is not satisfied no ATF is extracted.

Footnote 2: Only verbs, nouns, adjectives, adverbs and numbers were considered, assuming all the others as “not informative words”.

The condition over \(d\) is important because the sentence aligner may fail to match some subsequences, creating false ATFs. However, what is missed from one sentence will be missed also in the other sen

Figure 1: A pair of non-paraphrase sentences and its corresponding additional fragment.

 

tence. Thus, in general, if we set a small \(d\) then \(F_{i}\) and \(F_{j}\) misalignments may generate false ATFs. In contrast, a large \(d\) would clearly prevent this problem, although small ATFs (of size \(<d\)) may be discarded. More precisely, smaller values of \(\tau\) may cause the selection of fragments that have corresponding fragments in the other sentence, expressed with dissimilar words (i.e., the aligner failed to match those constituents). Larger values of \(\tau\) make the heuristic more precise, but less effective in retrieving smaller ATFs.

The ATF corpus.We applied the heuristic above to extract an ATF (if exists) from each sentence pair of MSRP. The number of the extracted ATFs depends on \(\tau\) as reported in Table 1. A manual inspection of the retrieved fragments revealed that: (_i_) small values of \(\tau\), namely, 1 and 2, cause the extraction of many fragments from one sentence corresponding to fragments expressed with different words in the other sentence: these are not ATFs. (_ii_) With \(\tau=3\), the heuristic is very precise and captures most ATFs appearing in the sentence pairs. (_iii_) Higher values of \(\tau\) cause many valid fragments to be missed.

Once ATFs are generated, we need to label them as ancillary or important for PI such that this data can be used for training and testing ATC. Interestingly, the data can be automatically labeled exploiting the MSRP annotation: given a sentence pair from MSRP, we (_i_) extract the ATF from it and (_ii_) automatically annotate it as ancillary if the pair is a paraphrase and not ancillary otherwise. In other words, an ATF is considered ancillary only if it is extracted from a paraphrase pair. To verify the correctness of this approach, two experts manually labeled the obtained data extracted with \(\tau=3\) and found that only 3.3% of the data was mislabeled with respect to one annotator. The Cohen's kappa agreement between the annotators was 85%.

## 4 Experiments

In these experiments, we first evaluate state-of-the-art PI models to create our baseline, then we experiment with our ATC and finally, we combine them to show that ATC can improve PI.

### Deriving PI baselines

Dataset.We used MSRP, which consists of 4,076 sentence pairs in the training set and 1,725 sentence pairs in test set. About 66% of the pairs are paraphrases. The pairs were extracted from topically similar Web news articles, applying some heuristics that select potential paraphrases to be annotated by human experts. We represent the sentence pairs using shallow trees generated with the Stanford parser3.

Footnote 3: nlp.stanford.edu/software/corenlp.shtml

Models.We adopted our state-of-the-art PI approach we proposed in Filice et al. (2015). This, given two pairs of sentences, \(p_{a}=\langle a_{1},a_{2}\rangle\) and \(p_{b}=\langle b_{1},b_{2}\rangle\), represents instances as shown Fig. 1, and applies tree kernels to them. In particular, we used our best kernel com derived in the work above:

\[\text{SMK}(p_{a},p_{b})= sf\big{(}\text{SPTK}(a_{1},b_{1})\times\text{SPTK}(a_{2},b_{2}),\] \[\qquad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad \quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad \quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad \quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad \quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad \quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad \quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad \quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad \quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad \quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad \quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad \quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad \quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad \quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad \quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad \quad 

skip-gram model applied to the UkWaC corpus [1].

Results.As illustrated in Table 2, a binary Support Vector Machine equipped with SMK achieves a very high accuracy. Moreover, SMK combined with a linear kernel (LK) over similarity metrics6 attains the state of the art in PI.

Footnote 6: These include _cosine similarities_ of lemmas, POS-tags, and n-grams, _longest common substring_ and _longest common subsequence_ measures and Tree Kernel intra-pair similarities.

### Experimental Evaluation on ATC

Dataset and models.We created an ATC dataset with \(\tau=3\) as described in Sec. 3. We make this dataset available7. We learned ATC with the C-SVM algorithm [1] inside KeLP8. The examples are represented using the shallow tree like the one on the right of Fig. 1. We used three different tree kernels: the Syntactic Tree Kernel (STK) by Collins and Duffy [50], the Partial Tree Kernel (PTK) by Moschitti [49] and SPTK using the word2vec similarity defined before.

Footnote 7: [http://alt.qcri.org/resources/ancillary](http://alt.qcri.org/resources/ancillary)

Footnote 8: [https://github.com/SAG-KeLP](https://github.com/SAG-KeLP)

Given the small size of such dataset (Only 8% of MSRP instances have additional fragments), we performed a 5-fold cross validation. Table 3 illustrates the Accuracy, Precision, Recall and F\({}_{1}\) of our models. ATC based on SPTK provides the best accuracy, i.e., 68.6%, which is a promising result for this research. The second most accurate classifier uses PTK, which is more flexible than STK.

### Using ATC in PI

We carried out error analysis on PI and observed that the used classifier commits a systematic error: when two sentences share a very similar large part (identical in the extreme case) and one sentence has an ATF, it almost always classifies the sentences as paraphrases, even if the ATF contains important information that invalidates the paraphrase relation. This kind of mistakes can be corrected by ATC.

Thus, we created the following ensemble model: given a pair to be classified, we apply our heuristic for ATF extraction. If the heuristic does not find any fragment in the pair, we only rely on the prediction provided by PI. Otherwise, we combine the prediction of ATC applied to ATF with the one of the PI classifier using a stacking strategy [23], i.e., the two predictions become the input features of a third classifier that makes the final decision.

To train this meta-classifier, we need the predictions from ATC and PI computed on a validation set. Hence, we split the training set in two parts: one part is used for training ATC and PI, while the other is classified with the trained models to produce the predictions for the meta-classifier. Then, the roles of the two parts are inverted. The meta-classifier is a linear SVM [13] implemented with KeLP.

Note that: (_i_) since we use 5-fold cross-validation, for each fold, we needed to apply the process described above to each fold; and (_ii_) all the learning algorithms and kernels adopt default parameters to also facilitate the reproducibility of our results.

Results.Table 4 reports the comparison between PI and PI combined with ATC (trained with SPTK). The performance is derived only on sentence pairs with ATFs.

The first column indicates the kernel used by the PI classifier, while the second column reports '+' or '-' to indicate if PI is combined with ATC or not, respectively. We note that ATC produces a great improvement, ranging from 8 absolute percent points over LK to about 3 points over SMK+LK, i.e., the state-of-the-art model. As expected, the more accurate the baseline is, the lower the improvement is produced.

It should be noted that only a relative small subset

\begin{table}
\begin{tabular}{|c|c|c|c|c|} \hline Model & Acc (\%) & P & R & F1 \\ \hline LK & 75.9 & 78.4 & 88.1 & 82.9 \\ \hline SMK & 76.4 & 76.6 & 92.9 & 83.9 \\ \hline SMK+LK & **77.7** & 79.4 & 8.99 & **84.4** \\ \hline [1] & 76.8 & \(-\) & \(-\) & 83.6 \\ \hline [1] & 77.4 & \(-\) & \(-\) & 84.1 \\ \hline \end{tabular}
\end{table}
Table 2: Results on Paraphrase Identification.

\begin{table}
\begin{tabular}{|c|c|c|c|c|} \hline Kernel & Acc (\%) & P & R & F1 \\ \hline STK & 65.1 \(\pm\) 6.5 & 65.4 \(\pm\) 8.0 & 58.3 \(\pm\) 5.6 & 61.5 \(\pm\) 5.8 \\ \hline PTK & 67.4 \(\pm\) 8.2 & 69.7 \(\pm\) 8.8 & 56.5 \(\pm\) 7.7 & 62.4 \(\pm\) 7.8 \\ \hline SPTK & **68.6**\(\pm\) 9.4 & 71.0 \(\pm\) 9.0 & 57.9 \(\pm\) 9.7 & 63.7 \(\pm\) 9.3 \\ \hline \end{tabular}
\end{table}
Table 3: Results of Ancillary Text Classifiers 

of MSRP contains additional fragments (about 8% when \(\tau=3\)). Thus, the impact on the entire PI testset cannot be large. Tab. 5 reports the accuracy of the previous models on the entire testset. An improvement over all models, state-of-the-art included, can be still observed, although it is less visible.

## 5 Conclusions

In this paper, we study and design models for learning to detect ancillary information in the context of PI. We used a heuristic rule for selecting additional fragments from paraphrase pairs, which, applied to MSRP, generates our ATF dataset. We manually annotated the latter for training and testing our ATCs.

Our experiments using several kernel models show that ATC can achieve a good accuracy (about 69%) and significantly impact the PI accuracy. Our results suggest that:

* it is possible to recognize information humans believe is ancillary; and
* to go beyond the current results and technology for high-level semantic tasks (e.g., PI), we cannot just rely on shallow similarity features, but we rather need to build components that analyze different aspects of text and then combine the output of the different modules.

In the future, it would be interesting to use methods similar to those successfully used in question answering research, e.g., matching entities in the sentence trees using linked open data [21, 20] or enriching trees with semantic information automatically produced by classifiers, e.g., [47, 48].

## Acknowledgements

This work has been partially supported by the EC project CogNet, 671625 (H2020-ICT-2014-2, Research and Innovation action) and by an IBM Faculty Award.

## References

* Baroni et al. (2009) Marco Baroni, Silvia Bernardini, Adriano Ferraresi, and Eros Zanchetta. 2009. The wacky wide web: a collection of very large linguistically processed web-crawled corpora. _Language Resources and Evaluation_, 43(3):209-226.
* Chang and Lin (2011) Chih-Chung Chang and Chih-Jen Lin. 2011. LIBSVM: A library for support vector machines. _ACM Transac

\begin{table}
\begin{tabular}{|c|c|c|c|c|} \hline PI & ATC & Acc (\%) & P & R & F\({}_{1}\) \\ \hline LK & - & \(62.2\pm 5.4\) & \(57.8\pm 7.0\) & \(75.1\pm 7.4\) & \(65.3\pm 6.9\) \\ \hline LK & + & \(70.4\pm 5.5\dagger\) & \(69.2\pm 6.8\) & \(68.2\pm 4.7\) & \(68.7\pm 5.7\) \\ \hline SMK & - & \(64.7\pm 6.0\) & \(59.0\pm 6.5\) & \(84.9\pm 4.6\) & \(69.5\pm 5.9\) \\ \hline SMK & + & \(69.1\pm 5.5\ddagger\) & \(66.9\pm 6.4\) & \(69.7\pm 5.8\) & \(68.2\pm 5.9\) \\ \hline SMK+LK & - & \(70.5\pm 4.0\) & \(66.3\pm 6.7\) & \(77.3\pm 6.1\) & \(71.2\pm 5.3\) \\ \hline SMK+LK & **+** & \(\textbf{73.2}\pm 5.2\ddagger\) & \(72.5\pm 5.4\) & \(71.3\pm 3.9\) & \(\textbf{71.8}\pm 3.9\) \\ \hline \end{tabular}
\end{table}
Table 4: PI classifier performance using ATC. The test set is restricted to examples having additional fragments. \(\dagger\) and \(\ddagger\) mark statistically significant differences in accuracy compared to the counterpart model not using ATC with confidence levels of 95% and 90%, respectively (t-test).

\begin{table}
\begin{tabular}{|c|c|c|c|c|c|} \hline PI & ATC & Acc (\%) & P & R & F\({}_{1}\) \\ \hline LK & - & \(75.5\pm 0.5\) & \(78.6\pm 0.9\) & \(87.6\pm 1.9\) & \(82.8\pm 0.4\) \\ \hline LK & + & \(76.2\pm 1.0\dagger\) & \(79.5\pm 0.1\) & \(87.2\pm 2.2\) & \(83.1\pm 0.8\) \\ \hline SMK & - & \(75.6\pm 0.8\) & \(77.1\pm 0.4\) & \(90.7\pm 1.2\) & \(83.3\pm 0.7\) \\ \hline SMK & + & \(75.9\pm 0.9\dagger\) & \(77.9\pm 1.3\) & \(89.7\pm 1.2\) & \(83.4\pm 0.6\) \\ \hline SMK+LK & - & \(78.1\pm 1.1\) & \(80.7\pm 0.6\) & \(88.6\pm 2.1\) & \(84.4\pm 0.9\) \\ \hline SMK+LK & + & \(\textbf{78.3}\pm 1.1\ddagger\) & \(81.1\pm 0.9\) & \(88.2\pm 1.8\) & \(\textbf{84.5}\pm 0.8\) \\ \hline \end{tabular}
\end{table}
Table 5: PI classifier performance using ATC on the testset. \(\dagger\) and \(\ddagger\) mark statistically significant differences in accuracy compared to the counterpart model not using ATC with confidence levels of 95% and 90%, respectively (t-test).

 

tions on Intelligent Systems and Technology_, 2:27:1-27:27.
* Collins and Duffy (2001) Michael Collins and Nigel Duffy. 2001. Convolution kernels for natural language. In _Advances in Neural Information Processing Systems 14_, pages 625-632. MIT Press.
* Croce et al. (2011) Danilo Croce, Alessandro Moschitti, and Roberto Basili. 2011. Structured lexical similarity via convolution kernels on dependency trees. In _Proceedings EMNLP_.
* Dolan et al. (2004) Bill Dolan, Chris Quirk, and Chris Brockett. 2004. Unsupervised construction of large paraphrase corpora: Exploiting massively parallel news sources. In _Proc. of COLING '04_, Stroudsburg, PA, USA.
* Fan et al. (2008) Rong-En Fan, Kai-Wei Chang, Cho-Jui Hsieh, Xiang-Rui Wang, and Chih-Jen Lin. 2008. Liblinear: A library for large linear classification. _J. Mach. Learn. Res._, 9:1871-1874, June.
* Filice et al. (2015) Simone Filice, Giovanni Da San Martino, and Alessandro Moschitti. 2015. Structural representations for learning relations between pairs of texts. In _Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)_, pages 1003-1013, Beijing, China, July. Association for Computational Linguistics.
* Madnani et al. (2012) Nitin Madnani, Joel Tetreault, and Martin Chodorow. 2012. Re-examining machine translation metrics for paraphrase identification. In _Proceedings of NAACL HLT '12_. ACL.
* Mikolov et al. (2013) Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. 2013. Efficient estimation of word representations in vector space. _arXiv preprint arXiv:1301.3781_.
* Moschitti (2006) Alessandro Moschitti. 2006. Efficient convolution kernels for dependency and constituent syntactic trees. In _Proc. of ECML'06_, pages 318-329.
* Severyn and Moschitti (2012) Aliaksei Severyn and Alessandro Moschitti. 2012. Structural relationships for large-scale learning of answer re-ranking. In _Proceedings of the 35th international ACM SIGIR conference on Research and development in information retrieval_, pages 741-750. ACM.
* Severyn et al. (2013a) Aliaksei Severyn, Massimo Nicosia, and Alessandro Moschitti. 2013a. Building structures from classifiers for passage reranking. In _CIKM_.
* Severyn et al. (2013b) Aliaksei Severyn, Massimo Nicosia, and Alessandro Moschitti. 2013b. Learning adaptable patterns for passage reranking. In _CoNLL_.
* Socher et al. (2011) Richard Socher, Eric H. Huang, Jeffrey Pennington, Andrew Y. Ng, and Christopher D. Manning. 2011. Dynamic pooling and unfolding recursive autoencoders for paraphrase detection. In _Advances in Neural Information Processing Systems 24: 25th Annual Conference on Neural Information Processing Systems 2011. Proceedings of a meeting held 12-14 December 2011, Granada, Spain._, pages 801-809.
* Tymoshenko and Moschitti (2015) Kateryna Tymoshenko and Alessandro Moschitti. 2015. Assessing the impact of syntactic and semantic structures for answer passages reranking. In _CIKM_, pages 1451-1460. ACM.
* Tymoshenko et al. (2014) Kateryna Tymoshenko, Alessandro Moschitti, and Aliaksei Severyn. 2014. Encoding semantic resources in syntactic structures for passage reranking. In _Proceedings of EACL_.
* Wolpert (1992) David H. Wolpert. 1992. Stacked generalization. _Neural Networks_, 5:241-259.