SlowGuess/ACL_2020_OCR huggingface dataset:
TACRED (Zhang et al., 2017) is one of the largest, most widely used crowdsourced datasets in Relation Extraction (RE). But, even with recent advances in unsupervised pretraining and knowledge enhanced neural RE, models still show a high error rate. In this paper, we investigate the questions: Have we reached a performance ceiling or is there still room for improvement? And how do crowd annotations, dataset, and models contribute to this error rate? To answer these questions, we first validate the most challenging 5K examples in the development and test sets using trained annotators. We find that label errors account for  $8\%$  absolute F1 test error, and that more than  $50\%$  of the examples need to be relabeled. On the relabeled test set the average F1 score of a large baseline model set improves from 62.1 to 70.1. After validation, we analyze misclassifications on the challenging instances, categorize them into linguistically motivated error groups, and verify the resulting error hypotheses on three state-of-the-art RE models. We show that two groups of ambiguous relations are responsible for most of the remaining errors and that models may adopt shallow heuristics on the dataset when entities are not masked.
Relation Extraction (RE) is the task of extracting relationships between concepts and entities from text, where relations correspond to semantic categories such as per:spouse, org:founded-by or org:subsidiaries (Figure 1). This makes RE a key part of many information extraction systems, and its performance determines the quality of extracted facts for knowledge base population (Ji and Grishman, 2011), or the quality of answers in question answering systems (Xu et al., 2016). Standard benchmarks such as SemEval 2010 Task 8 (Hendrickx et al., 2010) and the more recent TACRED
Figure 1: Relation example from TACRED. The sentence contains the relation org:subsidiaries between the head and tail organization entities 'Aerolineas' and 'Austral'.
(Zhang et al., 2017) are essential to evaluate new RE methods and their limitations, and to establish open challenges.
TACRED is one of the largest and most widely used RE datasets. It contains more than 106k examples annotated by crowd workers. The methods best performing on the dataset use some form of pre-training to improve RE performance: finetuning pre-trained language representations (Alt et al., 2019; Shi and Lin, 2019; Joshi et al., 2019) or integrating external knowledge during pre-training, e.g. via joint language modelling and linking on entity-linked text (Zhang et al., 2019; Peters et al., 2019; Baldini Soares et al., 2019); with the last two methods achieving a state-of-the-art performance of 71.5 F1. While this performance is impressive, the error rate of almost  $30\%$  is still high. The question we ask in this work is: Is there still room for improvement, and can we identify the underlying factors that contribute to this error rate? We analyse this question from two separate viewpoints: (1) to what extent does the quality of crowd based annotations contribute to the error rate, and (2) what can be attributed to dataset and models? Answers to these questions can provide insights for improving crowdsourced annotation in RE, and suggest directions for future research.
To answer the first question, we propose the following approach: We first rank examples in the development and test sets according to the misclas
Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 1558-1569 July 5 - 10, 2020. ©2020 Association for Computational Linguistics
sifications of 49 RE models and select the top 5k instances for evaluation by our linguists. This procedure limits the manual effort to only the most challenging examples. We find that a large fraction of the examples are mislabeled by the crowd. Our first contribution is therefore a extensively relabeled TACRED development and test set.
To answer the second question, we carry out two analyses: (1) we conduct a manual explorative analysis of model misclassifications on the most challenging test instances and categorize them into several linguistically motivated error categories; (2) we formulate these categories into testable hypotheses, which we can automatically validate on the full test set by adversarial rewriting - removing the suspected cause of error and observing the change in model prediction (Wu et al., 2019). We find that two groups of ambiguous relations are responsible for most of the remaining errors. The dataset also contains clues that are exploited by models without entity masking, e.g. to correctly classify relations even with limited access to the sentential context.
We limit our analysis to TACRED, but want to point out that our approach is applicable to other RE datasets as well. We make the code of our analyses publicly available.<sup>1</sup> In summary, our main contributions in this paper are:
- We validate the 5k most challenging examples in the TACRED development and test sets, and provide a revised dataset<sup>2</sup> that will improve the accuracy and reliability of future RE method evaluations.
- We evaluate the most challenging, incorrectly predicted examples of the revised test set, and develop a set of 9 categories for common RE errors, that will also aid evaluation on other datasets.
- We verify our error hypotheses on three state-of-the-art RE models and show that two groups of ambiguous relations are responsible for most of the remaining errors and that models exploit cues in the dataset when entities are unmasked.
Split	# Examples	# Neg. examples
Train	68,124	55,112
Dev	22,631	17,195
Test	15,509	12,184
Table 1: TACRED statistics per split. About  $79.5\%$  of the examples are labeled as no Relation.
2 The TACRED Dataset
The TAC Relation Extraction  $Dataset^3$ , introduced by Zhang et al. (2017), is a fully supervised dataset of sentence-level binary relation mentions. It consists of 106k sentences with entity mention pairs collected from the TAC KBP evaluations 2009-2014, with the years 2009 to 2012 used for training, 2013 for development, and 2014 for testing. Each sentence is labeled with one of 41 person- and organization-oriented relation types, e.g. per:title, org:founded, or the label no Relation for negative instances. Table 1 summarizes key statistics of the dataset.
All relation labels were obtained by crowdsourcing, using Amazon Mechanical Turk. Crowd workers were shown the example text, with head (subject) and tail (object) mentions highlighted, and asked to select among a set of relation label suggestions, or to assign no Relation. Label suggestions were limited to relations compatible with the head and tail types.[5]
The data quality is estimated as relatively high by Zhang et al. (2017), based on a manual verification of 300 randomly sampled examples (93.3% validated as correct). The inter-annotator kappa label agreement of crowd workers was moderate at  $\kappa = 0.54$  for 761 randomly selected mention pairs.
3 An Analysis of TACRED Label Errors
In order to identify the impact of potentially noisy, crowd-generated labels on the observed model performance, we start with an analysis of TACRED's label quality. We hypothesize that while comparatively untrained crowd workers may on average produce relatively good labels for easy relation mentions, e.g. those with obvious syntactic and/or
$^{3}$ https://catalog.ldc.upenn.edu/LDC2018T24
4https://tac.nist.gov/2017/KBP/index.html
See the supplemental material provided by Zhang et al. (2017) for details of the dataset creation and annotation process.
<https://github.com/DFKI-NLP/tacrev>
Due to licensing restrictions, we can not publish the dataset but instead provide a patch to the original TACRED.
lexical triggers, or unambiguous entity type signatures such as per:title, they may frequently err on challenging examples, e.g. highly ambiguous ones or relation types whose scope is not clearly defined.
An analysis of the complete dataset using trained annotators would be prohibitively expensive. We therefore utilize a principled approach to selecting examples for manual analysis (Section 3.1). Based on the TAC-KBP annotation guidelines, we then validate these examples (Section 3.2), creating new Dev and Test splits where incorrect annotations made by crowd workers are revised (Section 3.3).
Since we are interested in identifying potentially incorrectly labeled examples, we implement a selection strategy which is based upon ordering examples by the difficulty of predicting them correctly. We use a set of 49 different RE models to obtain predictions on the development and test sets, and rank each example according to the number of models predicting a different relation label than the ground truth. Intuitively, examples with large disagreement, between all models or between models and the ground truth, are either difficult, or incorrectly annotated.
We select the following examples for validation: (a) Challenging – all examples that were misclassified by at least half of the models, and (b) Control – a control group of (up to) 20 random examples per relation type, including no Relation, from the set of examples classified correctly by at least 39 models. The two groups cover both presumably hard and easy examples, and allow us to contrast validation results based on example difficulty. In total we selected 2,350  $(15.2\%)$  Test examples and 3,655  $(16.2\%)$  Dev examples for validation. Of these, 1,740 (Test) and 2,534 (Dev) were assigned a positive label by crowd workers.
3.2 Human Validation
We validate the selected examples on the basis of the TAC KBP guidelines. We follow the approach of Zhang et al. (2017), and present each example by showing the example's text with highlighted head and tail spans, and a set of relation label suggestions. We differ from their setup by showing
more label suggestions to make the label choice less restrictive: (a) the original, crowd-generated ground truth label, (b) the set of labels predicted by the models, (c) any other relation labels matching the head and tail entity types, and (d) no Relation. The suggested positive labels are presented in an alphabetical order and are followed by no Relation, with no indication of a label's origin. Annotators are asked to assign no Relation or up to two positive labels from this set. A second label was allowed only if the sentence expressed two relations, according to the guidelines, e.g. per:city_of_birth and per:city_of_residence. Any disagreements are subsequently resolved by a third annotator, who is also allowed to consider the original ground truth label. All annotators are educated in general linguistics, have extensive prior experience in annotating data for information extraction tasks, and are trained in applying the task guidelines in a trial annotation of 500 sentences selected from the development set.
3.3 The Revised TACRED Dev and Test Sets
Table 2 shows the results of the validation process. In total, the annotators revised 960 (49.9%) of the Challenging Test examples, and 1,610 (52.1%) of the Challenging Dev examples, a very large fraction of label changes for both dataset splits. Revision rates for originally positive examples are lower at  $47.3\%$  (Test) and  $49.1\%$  (Dev). Approximately  $57\%$  of the negative examples were relabeled with a positive relation label (not shown). Two labels were assigned to only  $3.1\%$  of the Test, and  $2.4\%$  of the Dev examples. The multi-labeling mostly occurs with location relations, e.g. the phrase "[Gross]head:per, a 60-year-old native of [Potomac]tail:city" is labeled with per:city_of_birth and per:city_of_residence, which is justified by the meaning of the word native.
As expected, the revision rate in the Control groups is much lower, at  $8.9\%$  for Test and  $8.1\%$  for Dev. We can also see that the fraction of negative examples is approximately one-third in the Challenging group, much lower than the dataset average of  $79.5\%$ . This suggests that models have more difficulty predicting positive examples correctly.
The validation inter-annotator agreement is shown in Table 3. It is very high at  $\kappa_{Test} = 0.87$  and  $\kappa_{Dev} = 0.80$ , indicating a high annotation quality. For both Test and Dev, it is higher for the easier Control groups than for the Challenging
$^{6}$ A similar approach was used e.g. by Barnes et al. (2019).  
 $^{7}$ See the supplemental material for details on the models, training procedure, hyperparameters, and task performance.  
 $^{8}$ https://tac.nist.gov/2014/KBP/ColdStart/guidelines/TAC_KBP_2014_Slot_Description_V1.4.pdf
Dev	Test
Challenging	Control	Challenging	Control
# Examples (# positive)	3,088 (1,987)	567 (547)	1,923 (1,333)	427 (407)
# Revised (# positive)	1,610 (976)	46 (46)	960 (630)	38 (38)
# Revised (% positive)	52.1 (49.1)	8.1 (8.4)	49.9 (47.3)	8.9 (9.3)
Table 2: Re-annotation statistics for TACRED Dev and Test splits.
IAA	Dev	Test
H1,H2	H,C	H1,H2	H,C
Challenging	0.78	0.43	0.85	0.44
Control	0.87	0.95	0.94	0.96
All	0.80	0.53	0.87	0.55
Table 3: Inter-Annotator Kappa-agreement for the relation validation task on TACRED Dev and Test splits (H1,H2 = human re-annotators, H = revised labels, C = original TACRED crowd-generated labels).
groups. In contrast, the average agreement between our annotators and the crowdsourced labels is much lower at  $\kappa_{Test} = 0.55$ ,  $\kappa_{Dev} = 0.53$ , and lowest for Challenging examples (e.g.,  $\kappa_{Test} = 0.44$ ).
Frequently erroneous crowd labels are per:cities_of_residence, org:alternate_names, and per:other_family. Typical errors include mislabeling an example as positive which does not express the relation, e.g. labeling "[Alan Gross]head:per was arrested at the [Havana]tail:loc airport." as per:cities_of_residence, or not assigning a positive relation label, e.g. per:other_family in "[Benjamin Chertoff]head:per is the Editor in Chief of Popular Mechanics magazine, as well as the cousin of the Director of Homeland Security, [Michael Chertoff]tail:per". Approximately  $49\%$  of the time an example's label was changed to no Relation during validation,  $36\%$  of the time from no Relation to a positive label, and the remaining  $15\%$  it was changed to or extended with a different relation type.
To measure the impact of dataset quality on the performance of models, we evaluated all 49 models on the revised test split. The average model F1 score rises to  $70.1\%$ , a major improvement of  $8\%$  over the  $62.1\%$  average F1 on the original test split, corresponding to a  $21.1\%$  error reduction.
Discussion The large number of label corrections and the improved average model performance show
that the quality of crowdsourced annotations is a major factor contributing to the overall error rate of models on TACRED. Even though our selection strategy was biased towards examples challenging for models, the large proportion of changed labels suggests that these examples were difficult to label for crowd workers as well. To put this number into perspective – Riedel et al. (2010) showed that, for a distantly supervised dataset, about  $31\%$  of the sentence-level labels were wrong, which is less than what we observe here for human-supervised data.[9]
The low quality of crowd-generated labels in the Challenging group may be due to their complexity, or due to other reasons, such as lack of detailed annotation guidelines, lack of training, etc. It suggests that, at least for Dev and Test splits, crowdsourcing, even with crowd worker quality checks as used by Zhang et al. (2017), may not be sufficient to produce high quality evaluation data. While models may be able to adequately utilize noisily labeled data for training, measuring model performance and comparing progress in the field may require an investment in carefully labeled evaluation datasets. This may mean, for example, that we need to employ well-trained annotators for labeling evaluation splits, or that we need to design better task definitions and task presentations setups as well as develop new quality control methods when using crowd-sourced annotations for complex NLP tasks like RE.
4 An Analysis of Model Errors
After revising the dataset, we focus on the two open questions: which of the remaining errors can be attributed to the models, and what are potential reasons for misclassifications? To answer these, we first create an annotation task instructing the
Riedel et al.'s estimate is an average over three relations with 100 randomly sampled examples each, for similar news text. Two of the relations they evaluated, nationality and place_of_birth, are also contained in TACRED, the third is contains (location).
linguists to annotate model misclassifications with their potential causes (Section 4.1). We then categorize and analyze the causes and formulate testable hypotheses that can be automatically verified (Section 4.2). For the automatic analysis, we implemented a baseline and three state-of-the-art models (Section 4.3).
4.1 Misclassification Annotation
The goal of the annotation is to identify possible linguistic aspects that cause incorrect model predictions. We first conduct a manual exploratory analysis on the revised Control and Challenging test instances that are misclassified by the majority of the 49 models. Starting from single observations, we iteratively develop a system of categories based on the existence, or absence, of contextual and entity-specific features that might mislead the models (e.g. entity type errors or distracting phrases). Following the exploration, we define a final set of categories, develop guidelines for each, and instruct two annotators to assign an error category to each misclassified instance in the revised test subset. In cases where multiple categories are applicable the annotator selected the most relevant one. As in the validation step, any disagreements between the two annotators are resolved by a third expert.
4.2 Error Hypotheses Formulation and Adversarial Rewriting
In a next step, we extend the misclassification categories to testable hypotheses, or groups, that are verifiable on the whole dataset split. For example, if we suspect a model to be distracted by an entity in context of same type as one of the relation arguments, we formulate a group has_distractor. The group contains all instances, both correct and incorrect, that satisfy a certain condition, e.g. there exists at least one entity in the sentential context of same type as one of the arguments. The grouping ensures that we do not mistakenly prioritize groups that are actually well-handled on average. We follow the approach proposed by Wu et al. (2019), and extend their Errudite framework<sup>10</sup> to the relation extraction task. After formulating a hypothesis, we assess the error prevalence over the entire dataset split to validate whether the hypothesis holds, i.e. the group of instances shows an above average error rate. In a last step, we test the error hypothesis explicitly by adversarial rewriting of a group's ex
amples, e.g. by replacing the distracting entities and observing the models' predictions on the rewritten examples. In our example, if the has_distractor hypothesis is correct, removing the entities in context should change the prediction of previously incorrect examples.
We evaluate our error hypotheses on a baseline and three of the most recent state-of-the-art RE models. None of the models were part of the set of models used for selecting challenging instances (Section 3.1), so as not to bias the automatic evaluation. As the baseline we use a single layer CNN (Zeng et al., 2014; Nguyen and Grishman, 2015) with max-pooling and 300-dimensional GloVe (Pennington et al., 2014) embeddings as input. The state-of-the-art models use pre-trained language models (LM) fine-tuned to the RE task and include: TRE (Alt et al., 2019), which uses the unidirectional OpenAI Generative Pre-Trained Transformer (GPT) (Radford et al., 2018); SpanBERT (Joshi et al., 2019), which employs a bidirectional LM similar to BERT (Devlin et al., 2019) but is pre-trained on span-level; and KnowBERT (Peters et al., 2019), which is an extension to BERT that integrates external knowledge. In particular, we use KnowBERT-W+W, which is trained by joint entity linking and language modelling on Wikipedia and WordNet.
5 Model Error and Dataset Analysis
In this section, we present our analysis results, providing an answer to the question: which of the remaining errors can be attributed to the models, and what are the potential reasons for these errors? We first discuss the findings of our manual misclassification analysis (Section 5.1), followed by the results of the automatic analysis (Section 5.2).
5.1 Model Error Categories
Table 4 summarizes the linguistic misclassification categories we developed. We distinguish between errors resulting from (1) relation argument errors, and (2) context misinterpretation.[11] The category relation argument errors refers to misclassifications resulting from incorrectly assigned
11 The manual analysis focused on the sentence semantics, and left aspects such as sentence length, distance between entities, etc. for the automatic analysis, which can handle the analysis of surface features more effectively.
10https://github.com/uwdata/errudite
Error Type	Examples	Prediction	Freq.
Arguments
Span	This is a tragic day for the Australian [Defence Force]head:org ([ADF]tail:org)	org:alt._nam	12
Entity Type	[Christopher Bollyn]head:per is an [independent]tail:religion journalist	per:religion	31
The company, which [Baldino]head:org founded in [1987]tail:date sells a variety of drugs	org:founded
Context
Inverted Args	[Ruben van Assouw]head:per, who had been on safari with his 40-year-old father	per:children	25
[Patrick]tail:per , mother Trudy , 41 , and brother Enzo , 11 .
Wrong Args	Authorities said they ordered the detention of Bruno's wife , [Dayana Rodrigues]tail:per , who was found with [Samudio]head:per's baby .	per:spouse	109
Ling. Distractor	In May , [he]head:per secured $ 96,972 in working capital from [GE Healthcare Financial Services]tail:org .	per:employ._of	35
Factuality	[Ramon]head:per said he hoped to one day become an [astronaut]head:title	per:title	11
Neither he nor [Aquash]head:per were [American]tail:nationality citizens .	per:origin
Relation Def.	[Zhang Yinjun]tail:per , spokesperson with one of China 's largest charity organi-zation , the [China Charity Federation]head:org	org:top_mem.	96
Context Ignored	[Bibi]head:per , a mother of [five]tail:Number, was sentenced this month to death .	per:age	52
No Relation	[He]head:per turned a gun on himself committing [suicide]tail:causeofdeath .	no Relation	646
Total	1017
Table 4: Misclassification types along with sentence examples, relevant false predictions, and error frequency. The problematic sentence parts are underlined (examples may be abbreviated due to space constraints).
entity spans or entity types of arguments. We always labeled type annotation errors, but tolerated minor span annotation errors if they did not change the interpretation of the relation or the entity.