Estimating Lexical Resources Impact in Text-to-Text Inference Tasks
Authors: Simone Magnolini, Bernardo Magnini (Università di Brescia, Fondazione Bruno Kessler, 2014)
Topic: Distributional Semantics

Abstract
This paper provides an empiri-cal analysis of both the datasets and the lexical resources that are commonly used in text-to-text inference tasks (e.g. textual entailment, semantic similarity). Accord-ing to the analysis, we define an index for the impact of a lexical resource, and we show that such index significantly corre-lates with the performance of a textual en-tailment system.

Introduction
In the last decade text-to-text semantic inference has been a relevant topic in Computational Lin-guistics. Driven by the assumption that language understanding crucially depends on the ability to recognize semantic relations among portions of text, several text-to-text inference tasks have been proposed, including recognizing paraphrasing (Dolan and Brockett., 2005), recognizing textual entailment (RTE) (Dagan et al., 2005), and se-mantic similarity (Agirre et al., 2012). A common characteristic of such tasks is that the input are two portions of text, let’s call them Text1 and Text2, and the output is a semantic relation between the two texts, possibly with a degree of confidence of the system. For instance, given the* following text fragments: *Text1: George Clooneys longest relationship ever might have been with a pig. The actor owned Max, a 300-pound pig. Text2: Max is an animal.* *a system should be able to recognize that there is an ”entailment” relation among Text1 and Text2.* *While the task is very complex, requiring in principle to consider syntax, semantics and also pragmatics, current systems adopt rather sim-plified techniques, based on available linguistic resources. For instance, many RTE systems (Da-gan et al., 2012) would attempt to take advantage of the fact that, according to WordNet, the word animal in Text2 is a hypernym of the word pig in Text1. A relevant aspect in text-to-text tasks is that datasets are usually composed of textual pairs for positive cases, where a certain relation occurs, and negative pairs, where a semantic relation doesn’t appear. For instance, the following pair:* *Text1: John has a cat, named Felix, in his farm, it’s a Maine Coon, it’s the largest domesticated breed of cat. Text2: Felix is the largest domesticated animal in John’s farm.* shows a case of ”non-entailment”. In the paper we systematically investigate the re-lations between the distribution of lexical associa-tions in textual entailment datasets and the system performance. As a result we define a ”resource impact index” for a certain lexical resource with respect to a certain dataset, which indicates the capacity of the resource to discriminate between positive and negative pairs. We show that the ”re-source impact index” is homogeneous across sev-eral datasets and tasks, and that it correlates with the performance of the algorithm we chose in our experiments.

Conclusion
We have proposed a method for estimating the im-pact of a lexical resource on the performance of a text-to-text semantic inference system. The start-ing point has been the definition of the RID index, which captures the intuition that in current datasets useful resources need to discriminate between pos-itive and negative pairs. We have then shown that the RID index is highly correlated with the accu-racy of the system for balanced datasets and with the F1 for the unbalanced one, a result that allows to use the RID as a reliable indicator of the im-pact of a resource. As for future work, we intend to further general-ize our current findings applying the same method-ology to different text-to-text inference algorithms, starting from those already available in the EX-CITEMENT Open Platform. We also want to con-duct experiment on operation, like summing, with this index to describe to combined effect of differ-ent resources.