Creating a standard for evaluating Distant Supervision for Relation Extraction
Authors: Azad Abad, Alessandro Moschitti (Università di Trento, Qatar Computing Research Institute, 2014)
Topic: Lexical and Semantic Resources and Analysis

Abstract
This paper defines a standard for comparing relation extraction (RE) systems based on a Distant Supervision (DS). We integrate the well-known New York Time corpus with the more recent version of Freebase. Then, we define a simpler RE system based on DS, which exploits SVMs, tree kernels and a simple one-vs-all strategy. The resulting model can be used as a baseline for system comparison. We also study several example filtering techniques for improving the quality of the DS output.

Introduction
Currently, supervised learning approaches are widely used to train relation extractors. However, manually providing large-scale human-labeled training data is costly in terms of resources and time. Besides, (i) a small-size corpus can only contains few relation types and (ii) the resulting trained model is domain-dependent. Distance Supervision (DS) is an alternative approach to overcome the problem of data annotation (Craven et al., 1999) as it can automatically generate training data by combining (i) a structured Knowledge Base (KB), e.g., Freebase1 with a large-scale unlabeled corpus, C. The basic idea is: given a tuple r<e1,e2> contained in a referring KB, if both e1 and e2 appear in a sentence of C, that sentence is assumed to express the relation type r, i.e., it is considered a training sentence for r. For example, given the KB relation, president(Obama,USA), the following sentence, Obama has been elected in the USA presidential campaign, can be used as a positive training example for president(x,y). However, DS suffers from two major drawbacks: first, in early studies, Mintz et al. (2009) assumed that two entity mentions cannot be in a relation with different relation types r1 and r2. In contrast, Hoffmann et al. (2011) showed that 18.3% of the entities in Freebase that also occur in the New York Times 2007 corpus (NYT) overlap with more than one relation type. Second, although DS method has shown some promising results, its accuracy suffers from noisy training data caused by two types of problems (Hoffmann et al., 2011; Intxaurrondo et al., 2013; Riedel et al., 2010): (i) possible mismatch between the sentence semantics and the relation type mapped in it, e.g., the KB correct relation, located in(Renzi,Rome), cannot be mapped into the sentence, Renzi does not love the Rome soccer team; and (ii) coverage of the KB, e.g., a sentence can express relations that are not in the KB (this generates false negatives). Several approaches for selecting higher quality training sentences with DS have been studies but comparing such methods is difficult for the lack of well-defined benchmarks and models using DS. In this paper, we aim at building a standard to compare models based on DS: first of all, we considered the most used corpus in DS, i.e., the combination of NYT and Freebase (NYT-FB). Secondly, we mapped the Freebase entity IDs used in NYT-FB from the old version of 2007 to the newer Freebase 2014. Since entities changed, we asked an annotator to manually tag the entity mentions in the sentence. As the result, we created a new dataset usable as a stand-alone DS corpus, which we make available for research purposes. Questo e-book appartiene a AlessandroLenci Finally, all the few RE models experimented with NYT-FB in the past are based on a complex conditional random fields. This is necessary to encode the dependencies between the overlapping relations. Additionally, such models use very particular and sparse features, which make the replicability of the models and results complex, thus limiting the research progress in DS. Indeed, for comparing a new DS approach with the previous work using NYT-FB, the researcher is forced to re-implement a very complicated model and its sparse features. Therefore, we believe that simpler models can bevery useful as (i) a much simpler reimplementation would enable model comparisons and (ii) it would be easier to verify if a DS method is better than another. In this perspective, our proposed approach is based on convolution tree kernels, which can easily exploit syntactic/semantic structures. This is an important aspect to favor replicability of our results. Moreover, our method differers from previous state of the art on overlapping relations (Riedel et al., 2010) as we apply a modification of the simple one-vs-all strategy, instead of the complex graphical models. To make our approach competitive, we studied several parameters for optimizing SVMsandfiltering out noisy negative training examples. Our extensive experiments show that our models achieve satisfactory results.

Conclusion
We have proposed a standard framework, simple RE models and an upgraded version of NYT-FB for more easily measuring the research progress in DS research. Our RE model is based on SVMs, can manage overlapping relations and exploit syntactic information and lexical features thanks to tree kernels. Additionally, we have shown that filtering techniques applied to DS data can discard noisy examples and significantly improve the RE accuracy.