The State of Relation Extraction Data Quality: Is Bigger Always Better? (2024)

Abstract: Relation extraction (RE) extracts structured tuples of relationships (e.g. friend, enemy) between entities (e.g. Sherlock Holmes, John Watson) from text, with exciting potential applications. Hundreds of RE papers have been published in recent years; do their evaluation practices inform these goals? We review recent surveys and a sample of recent RE methods papers, compiling 38 datasets currently being used. Unfortunately, many have frequent label errors, and ones with known problems continue to be used. Many datasets focus on producing labels for a large number of relation types, often through error-prone annotation methods (e.g. distant supervision or crowdsourcing), and many recent papers rely exclusively on such datasets. We draw attention to a promising alternative: datasets with a small number of relations, often in specific domains like chemistry, finance, or biomedicine, where it is possible to obtain high quality expert annotations; such data can more realistically evaluate RE performance. The research community should consider more often using such resources.

heading: 1 Introduction
text: Relation extraction (RE) methods extract tuple structures from unstructured text, where structures consist of types of relationships, e.g. is a member of an organization, and entities involved with them, e.g. Samuel Gompers, American Federation of Labor. Ding et al. (2021); Nadgeri et al. (2021); Xu and Barbosa (2019); Trisedya et al. (2019); Han et al. (2021) claim that applications of these methods include populating knowledge graphs, and we find that realistic downstream applications also benefit from these methods: for example, automatic extraction of relations could replace manual extraction of {friend, enemy} relations in Icelandic sagas (Mac Carron and Kenna, 2013), and could replace manual extraction of financial acquisition relations from news text in Gugler et al. (2003); Clougherty
et al. (2014), helping to decrease labor in social science and humanities studies.
However, significantly noisy dataset labels and evaluation on datasets using different performance metrics than intended may impact estimation of RE method performance for a realistic setting. While some literature acknowledges the issues, various recent evaluations and efforts to construct datasets ignore them. We explore characteristics and application of 38 datasets for evaluating RE methods. While our review is preliminary, we find some persistent patterns:
• Finding (F1): Datasets with larger numbers of relation types often have ground truth labels assigned using distant supervision, which aligns entities in text to relation tuples in a knowledge base. Further, such datasets—whether their ground truth labels are assigned through distant supervision or through full manual annotation— are vulnerable to significant labelling errors (Tan et al., 2022; Stoica et al., 2021; Huang et al., 2022; Alt et al., 2020; Wang et al., 2022b).
• Finding (F2): Many recent studies (at least 40 in ACL/EMNLP/Findings, since 2021; details in §4, §A.1) exclusively evaluate RE on datasets with larger numbers (24+) of relation types.
• Finding (F3): We find that datasets with fewer relation types, although less widely used, are more likely to be annotated by experts and have domain-specific text (Herrero-Zazo et al., 2013; Luan et al., 2018).
Although many evaluations are exclusively on datasets with larger numbers of relation types (F2), we encourage more of the research community to broaden evaluation to datasets with fewer relation types, given F3. Evaluating on various datasets which are more likely to have expert-annotated labels and domain-specific text may have a higher chance to provide a more accurate estimation of RE performance for a realistic setting, given F1.
7893

heading: 2 Potential of relation extraction methods for realistic applications
text: Instead of automatically extracting relations, studies in various applications manually extract relations from text for downstream analysis. Mac Carron and Kenna (2013) compare social network structure in Icelandic saga text and modern-day social networks by manually extracting friend and enemy relationships between saga characters. Gugler et al. (2003) and Clougherty et al. (2014) use manually extracted financial acquisition relations from various news texts, even though automatic extraction has been investigated for these relations (Freitag, 1998).
Why automatically extract relations? In addition to decreasing labor, relation extraction methods in the NLP literature seem increasingly promising for realistic applications, departing from the traditional fully supervised, sentence-level setting on general news and Wikipedia text. Recent work explores low resource methods which require few or no labelled instances for use (Han et al., 2018b; Sabo et al., 2021; Zhang and Lu, 2022), and new datasets and models target specific domains, e.g. financial, legal, and biomedical (Gurulingappa et al., 2012; Herrero-Zazo et al., 2013; Peng et al., 2019; Hendrycks et al., 2021; Sharma et al., 2022). Further, more recent methods extract relation instances across many sentences at the document-level instead of from single sentences (Yao et al., 2019; Jain et al., 2020; Xiao et al., 2020; Li et al., 2023).

heading: 3 Disconnect between datasets and realistic application
text: Given promise of RE for realistic applications (§2), we ask: Does performance on the datasets that RE methods evaluate on give an accurate indication of how such methods will perform in a variety of realistic use cases?
We explore, for datasets: (1) potential sources of error when assigning relation instance labels, (2) literature about labelling errors and evaluation on widely used datasets, (3) persistence of ignoring literature that addresses the issues.

heading: 3.1 Sources of labelling error
text: To clarify sources of labelling issues, we first define the RE task as having inputs R of relation types, text T from which to extract and classify relations, and some labelled examples of relation instances in
text for training. The output are relations 〈e1, r, e2〉 where r ∈ R is a relation type (e.g. part of ), and e1, e2 are head and tail entities respectively in a sentence or document of T (e.g. Neolithic, Stone Age) as in the example:
Discovery of late Stone Age jugs suggest that intentionally fermented beverages existed at least as early as the Neolithic period (c. 10000 BC). (Han et al., 2018b)
⟨e1 = Neolithic, r = part-of, e2 = Stone Age⟩
We selected 38 datasets to analyze (§4), finding that label error may occur from distant supervision and manual annotation as follows:
Missing labels from distant supervision. Distant supervision assigns labels to relation instances in text by aligning entity pairs e1, e2 in text to relation triples ⟨e1, r, e2⟩ in a knowledge base (KB), e.g. Wikipedia text to Wikidata entries (Mintz et al., 2009; Bunescu and Mooney, 2007). While distant supervision is a common practice because it allows automatic assignment of labels on large amounts of data, it is prone to significant errors (§3.2). Many efforts aim to address false positive labels by manually verifying them (Han et al., 2018b; Huguet Cabot et al., 2023) or designing models to be more selective about assigning labels (Riedel et al., 2010; Bing et al., 2015; Xiao et al., 2020; Jia et al., 2019; Zeng et al., 2018; Feng et al., 2018; Surdeanu et al., 2012; Qin et al., 2018a,b). However, few efforts (Chen et al., 2021; Hao et al., 2021; Tan et al., 2022; Xu et al., 2013; Roller et al., 2015; Xie et al., 2021) aim to address missing labels, which occurs when related entities in text are not part of a KB relation triple, and therefore are not assigned a label. Missing labels may be common; for the DocRED dataset with distant supervision-assigned labels, Huang et al. (2022) and Tan et al. (2022) find up to 2/3 of true labels are missing.
Ambiguous annotation guidelines. For full manual annotation, label quality depends on whether annotation guidelines specify relation type definitions clearly and unambiguously with respect to other relation types, and on annotator quality. For example, Stoica et al. (2021) and Alt et al. (2020) observe ambiguous documentation in the TACRED dataset for the pair of relation types “Person:Other_Family” and “No_Relation”, which they find to be responsible for many label errors.

heading: 3.2 Discussion on widely-used datasets: Label and evaluation errors
text: We provide examples of discussion on labelling errors described in §3.1 that may affect estimation of RE performance for realistic applications. We review use of four popular datasets over years 2019-2024, chosen based on citation count from Semantic Scholar: NYT-FB (Riedel et al., 2010) (778 cit.) and TACRED (Zhang et al., 2017) (657 cit.) for sentence-level RE, where entities in relation triples belong to the same sentence, DocRED (Yao et al., 2019) (312 cit.) for document-level RE, where entities could be anywhere in a document, and FewRel 1.0 (Han et al., 2018b; Gao et al., 2019) (460 cit.) for few-shot RE, where the number of training examples is limited.
Discussion on errors in DocRED (312 citations since 2019). DocRED has distant supervision assigned labels and annotations for 96 relation types on 5053 Wikipedia documents. To reduce false positive labels assigned through distant supervision (§3.1), annotators review entities and relation types, filtering out incorrect labels. However, the missing label issue related to distant supervision approaches persists: Huang et al. (2022) identify that almost two-thirds of ground truth relations are not labelled from their re-annotation of 96 documents. Tan et al. (2022) independently find that approximately 64.6% of ground truth relations are missing. They further replace DocRED with ReDocRED which has 4053 documents, by training RE models on the original distant supervised data and manually validating relation instances.
Discussion on errors in NYT-FB (778 citations since 2019). NYT-FB has distant supervision assigned labels for 24 relations, aligning New York Times article text (Sandhaus, 2008) over years 2005-2007 with Freebase relations (Bollacker et al., 2008). While precision of labels is 91%, their recall struggles, and many efforts aim to address this in diverse ways. Wang et al. (2022b) find issues on labels of 40 out of 100 randomly selected sentences. Hoffmann et al. (2011) add more labelled relations by joining tables in Freebase. Zeng et al. (2015) alternatively manually annotate the test set. Han et al. (2018a) add more ground truth labels by linking the text with another knowledge graph, FB60K. Zhu et al. (2020) manually annotate a larger test set as NYT-H.
Mismatch between intended versus actual use of FewRel 1/2 (460 citations since 2019). FewRel
1/2 has distant supervision assigned and manually verified labels for 80 relation types, and uses accuracy as a performance metric. Each relation type has 700 instances, with a one-to-one mapping of each instance to a sentence. Since a sentence may have more ground truth relation instances that are not labelled, precision and recall metrics are not able to accurately assess performance. However, among others, Zhao et al. (2023a); Lv et al. (2023); Zhao et al. (2023b); Wang et al. (2022a); Najafi and Fyshe (2023); Chen and Li (2021) use FewRel for computing precision and recall.
Discussion on errors in TACRED (667 citations since 2019). Fully manually annotated datasets are also vulnerable to labelling issues, such as TACRED, where Alt et al. (2020) found at least 50% of samples need to be relabelled and Stoica et al. (2021) found 23.9% of labels were incorrect. A revised version, Re-TACRED, has annotators from Amazon Mechanical Turk and improved annotation guidelines that remove ambiguity of relation definitions. On analysis using several RE methods, Stoica et al. (2021) found an average improvement of 14 F1 score on Re-TACRED, suggesting that label quality heavily impacts performance.

heading: 3.3 Propagation of errors despite discussion
text: Despite these multiple papers that reveal labelling and evaluation issues in relation extraction, we find that many recent works ignore and continue to propagate the issues. We investigate two categories of widespread evaluation issues in current work: (1) persistent use of original versions of datasets or of unintended evaluation metrics, and (2) continued introduction of datasets that face the same issues as previous ones (e.g., missing labels from distant supervision). In this section, the papers that we cite are from ACL/EMNLP/Findings venues.
Persistent use of original versions of datasets. However, recent evaluations still use original versions of these datasets or use unintended performance metrics. Many evaluations use TACRED (467 cit., since 2021) as opposed to Re-TACRED (72 cit., since 2021), without noting labelling issues (Wan et al., 2023; Zhao et al., 2023c; Chen et al., 2023b; Wang et al., 2022b; Sainz et al., 2021). Some methods still use FewRel, designed to measure performance using accuracy, to evaluate precision and recall, which are not appropriate performance metrics for the dataset Zhao et al. (2023a); Lv et al. (2023); Zhao et al. (2023b); Wang et al.
(2022a); Najafi and Fyshe (2023); Chen and Li (2021). Despite revised versions of NYT-FB, several evaluations still use original NYT-FB (Wu and Shi, 2021; Hao et al., 2021; Hu et al., 2020). For document-level relation extraction methods, some evaluations use original DocRED (Li et al., 2023) but luckily, many evaluations are using ReDocRED.
New datasets are as vulnerable to missing labels. While more literature points out issues of various datasets, new datasets such as CodRED (Yao et al., 2021), T-rex (Elsahar et al., 2018), and REDFM (Huguet Cabot et al., 2023) still do not consider the recall issue of distant supervision-assigned labels discussed in (§3.1) that has caused many labelling issues for other datasets, e.g. DocRED (Yao et al., 2019), NYT-FB (Riedel et al., 2010).

heading: 4 Which datasets are more susceptible to noise?
text: To help determine if any characteristics lead a dataset to be more susceptible to noise discussed in §3, we find:
• Datasets with larger numbers of relation types, which tend to have labels assigned using distant supervision, are vulnerable to significant labelling and evaluation errors.
• Many evaluations exclusively use datasets with larger numbers (24+) of relation types for evaluation (Zhang et al., 2023a; Wang et al., 2023a; Lu et al., 2023a), and 37 more papers in 2021-2023 ACL/EMNLP/Findings venues (§A.1).
• Datasets with labels for fewer relation types are more likely to be annotated by experts and have domain-specific text (Herrero-Zazo et al., 2013; Luan et al., 2018).
We compile a list of English datasets that have been used for RE evaluation from two sources. First, we search for papers at several NLP and machine learning venues1 over years 2019-2023 that have the keyword “relation extraction” in their title, read a random sample of 100 such papers, and record all datasets used by each paper in evaluations. Second, we add all datasets mentioned in (Zhao et al., 2023e)’s relation extraction survey. This results in 38 datasets.
For each dataset, we read its original paper and/or documentation to record metadata including
1Proceedings of ACL, Proceedings of EMNLP, AAAI Conference Proceedings, or Findings of the ACL.
how labels were assigned, the type of annotator (if any), the domain, the dataset’s size, and number of citations on Semantic Scholar (with some attempt to restrict to uses of the dataset).2 See §A.2 for more details, including metadata for all 38 (Table 2). Table 1 shows a portion of this information for the 21 most-cited datasets.
Trends: On larger numbers of relation types. Tables 1 and 2 show that the more relation types a dataset has labels for, the more likely that labels are assigned through distant supervision. Such datasets are susceptible to various labelling issues— §3.1 discusses sources of potential issues and §3.2 provides examples of significance of the issues. Further, we find at least 40 papers in ACL/EMNLP/Findings since 2021 (§A.1) that exclusively evaluate on such (24+ rel. types) datasets.
Trends: On fewer relation types. Datasets with fewer relation types are more likely to avoid distant supervision labelling issues and be manually annotated by experts (Herrero-Zazo et al., 2013; Luan et al., 2018; Hendrycks et al., 2021; Gurulingappa et al., 2012; Peng et al., 2019). Originally, such datasets contain general-purpose text, e.g. ACE (Doddington et al., 2004), SemEval Task 8 (Hendrickx et al., 2010), and Conll04 (Roth and Yih,
2www.semanticscholar.org, accessed February 2024.
2004). Increasingly, new datasets cover other domains such as DDI (biomedical, 5 rel types), where text is from the DrugBank database and Medline abstracts and relations involve drug-drug interactions (Herrero-Zazo et al., 2013), and SciERC (scientific, 6 rel types), where text is 500 scientific abstracts (Luan et al., 2018).

heading: 5 Recommendations
text: Despite data quality challenges that may affect estimation of RE method performance on realistic applications (§3), we find potential for using RE methods in real-world applications such as those described in §2. Based on findings in §4, we provide two types of recommendations: (1) on selecting datasets to use for evaluation, and (2) on constructing future datasets to use for evaluation.
On selecting datasets for evaluation. We encourage the research community to broaden evaluation to include datasets with smaller numbers of relation types, which are more likely to be annotated by experts, and ideally to use multiple such datasets. This helps test the flexibility of a method across diverse relation types and domains. To further strengthen confidence in dataset quality, researchers can also manually check correctness of a sample of labels on familiar relation types (if any), and check the literature for potential revised versions of a dataset.
While we advocate for smaller and higher quality data, we note a counterargument that larger datasets—even with label noise—are crucial for training many relation extraction methods. However, the point of relation extraction research is to support applications, and we believe training data will be sparse and very expensive to obtain in most realistic settings, since annotations require significant domain expertise—heavy supervision is not a feasible modeling approach. Therefore evaluation ought to be the primary role of relation extraction annotation, where more accurate labels are ideal, even if there are fewer of them.
On future construction of datasets. While evaluation on noisy datasets may help to provide a rough indication of RE method performance, noisy labels render datasets unhelpful for accurately estimating performance in a real world application. Therefore, considering strategies to increase recall of labels assigned through distant supervision using approaches such as Xie et al. (2021) could help to label larger datasets more accurately and more
efficiently—we find recent datasets (Huguet Cabot et al., 2023) aim to increase precision, but do not check or address recall of labels. Further, defining relation types unambiguously is helpful for avoiding manual labelling issues such as in Zhang et al. (2017).

heading: 6 Conclusion
text: Relation extraction is a popular task with hundreds of relevant papers published in recent years. Methods continue to improve and are becoming more promising for real world applications. First, we examined factors that potentially undermine relation extraction evaluation. Next, we provided recommendations to overcome these challenges, involving broadening evaluation to include smaller and higher quality datasets, and considering strategies to increase the recall of labels in new datasets, to improve estimation of relation extraction performance for realistic settings.

heading: 7 Limitations
text: This paper reviews 38 English datasets for evaluating RE methods, but will have missed datasets if they did not appear in the random sample of 100 papers from the four NLP/AI venues, or the survey paper (§4,§A.2). In particular, this data collection procedure may be more likely to miss datasets that are less frequently used.
In our final 38 datasets reported in the metadata tables, we restrict entries to original versions of datasets, and as mentioned in §3, do not list or analyze revised variants such as Re-TACRED for TACRED, Re-DocRED for DocRED, and NYTH for NYT-FB; these three are instead described within §3.2). We found that revised versions of datasets tend to be not widely cited, and leave further analysis for future work.
The paper points out several patterns—that datasets with annotations for smaller numbers of relation types are more likely to be annotated by experts and be domain-specific, and that datasets that have annotations for larger numbers of relation types are more likely to have labels assigned from distant supervision. These statements are often, but not always, true.

heading: Acknowledgements
text: We thank the UMass NLP group for feedback. This material is based upon work supported by National Science Foundation award 1845576. Any opinions,
findings, and conclusions or recommendations expressed in this material are those of the authors and do not necessarily reflect the views of the National Science Foundation.

heading: A Appendix
text: A.1 List of 40 papers that exclusively evaluate on datasets with 24+ relation types
The following papers exclusively evaluate RE methods on datasets with 24+ relation types; these papers are drawn from Proceedings of ACL, Proceedings of EMNLP, or Findings of the ACL, between 2021 and 2023: Wang et al. (2023a); Lu et al. (2023a); Wang et al. (2022b); Zhang et al. (2023a); Zhang and Lu (2022); Wang et al. (2022a); Lin et al. (2021); Sainz et al. (2021); Han et al. (2021); Wu and Shi (2021); Brody et al. (2021); Zhao et al.
(2021); Zeng et al. (2020); Hu et al. (2020); Rosenman et al. (2020); Zhao et al. (2023d,b,a); Chen et al. (2023a); Zhao et al. (2023c); Zhang et al. (2023b); Sun et al. (2023); Picco et al. (2023); Lv et al. (2023); Xia et al. (2023); Lu et al. (2023b); Wang et al. (2023b); Liu et al. (2022a); Rathore et al. (2022); Liu et al. (2022b); Zhao et al. (2022); Cui et al. (2021); Liu et al. (2021); Ma et al. (2021); Zheng et al. (2021b); Yang et al. (2021); Zhang et al. (2021); Nadgeri et al. (2021); Verlinden et al. (2021); Yellin and Abend (2021).
The papers involve various types of RE methods, including sentence-level, which could be evaluated using many datasets, document-level, for which there are fewer datasets to evaluate on yet still some datasets with small and some with large numbers of relation types, and low-resource, which could be evaluated using many datasets. This is not a comprehensive list of papers within 2021–2023 that exclusively evaluate on 24+ relation type datasets.
A.2 Preliminary analysis details on 38 datasets
Table 2 column description. Each row of Table 2 corresponds to a dataset, listing its name, year of introduction, method of construction, domain of text and relation types, number of relations that it has labels for, size (with units of number of sentences, abstracts, or documents), and citation count from Semantic Scholar since 2019 of all papers that are not a “background" citation according to Semantic Scholar and have the word “relation" in their title. To verify that these filtering rules extract papers that use the dataset to evaluate an RE method, we manually checked 20 such papers that matched these filtering requirements and found that all of them use the dataset to evaluate an RE method.
Datasets in Table 1. To select a subset of datasets for Table 1, we chose all datasets that have 12+ citation counts, where the citation counts pass the filtering rules above. We also added datasets that were published in 2023 to Table 1 since they do not have time to accumulate citations yet.
On selected datasets for tables 1 and 2. Tables 1 and 2 show original versions of datasets; as mentioned in §3, some datasets have multiple revised versions such as TACRED with Re-TACRED, DocRED with Re-DocRED, and NYT-FB with NYT-H. Although revised versions of datasets are not in the tables, we discussed well known ones in §3.2. Many revised versions are not well-cited.
On trends in Table 2. The trends in Table 1 of §4 are also in Table 2: datasets with labels for larger numbers of relation types tend to have labels assigned using distant supervision, and tend to have general purpose text.
We observe a stronger correlation between the number of relation types and the method of assigning labels than between the size of a dataset and the method of assigning labels, but we observe a correlation for both comparisons.
On exceptions in Table 2. Two datasets have labels assigned in other ways than full manual annotation and distant supervision, noted with an ‘Oth’ entry in Table 2: WebNLG (Gardent et al., 2017), where human annotators manually convert one or several sets of triples from a KB into sentences, and where other annotators verify if each resulting sentence is faithful to the triple/s and seems natural; and FOBIE (Kruiper et al., 2020), which uses the Journal of Experimental Biology and Biomed Central Journal as text, and where annotators manually correct all initial annotations that pass an automated search for relations through trigger word keywordmatching. Additionally, DWIE (Zaporojets et al., 2020), noted with a ‘Both’ entry, uses distant supervision to assign some labels and performs full manual annotation to assign others.
Some of the datasets in Table 2 do not provide labels for the exact relation extraction task defined in the paper (with output ⟨e1, r, e2⟩), but for a similar task — WIKITIME (Yan et al., 2019) includes a “time" component in its output relation tuple ⟨e1, r, e2, t⟩. The WikiReading (Hewlett et al., 2016) task is to predict entities and properties of text given the text and a relation type such as original language of work or country. The CUAD (Hendrycks et al., 2021) task is not binary, but nary, outputting tuples of the form ⟨r, e1, e2, e3...⟩ where more than two entities could be part of a tuple.

heading: Future Work
text: We found that revised versions of datasets tend to be not widely cited, and leave further analysis for future work. The paper points out several patterns—that datasets with annotations for smaller numbers of relation types are more likely to be annotated by experts and be domain-specific, and that datasets that have annotations for larger numbers of relation types are more likely to have labels assigned from distant supervision. These statements are often, but not always, true