SlowGuess/NAACL_2022_OCR huggingface dataset:
Nicholas Popovic and Michael Färber
Karlsruhe Institute of Technology (KIT), Germany
{popovic, michael.faerber}@kit.edu
We present FREDo, a few-shot document-level relation extraction (FSDLRE) benchmark. As opposed to existing benchmarks which are built on sentence-level relation extraction corpora, we argue that document-level corpora provide more realism, particularly regarding none-of-the-above (NOTA) distributions. Therefore, we propose a set of FSDLRE tasks and construct a benchmark based on two existing supervised learning data sets, DocRED and sciERC. We adapt the state-of-the-art sentence-level method MNAV to the document-level and develop it further for improved domain adaptation. We find FSDLRE to be a challenging setting with interesting new characteristics such as the ability to sample NOTA instances from the support set. The data, code, and trained models are available online<sup>1</sup>.
The goal of relation extraction is to detect and classify relations between entities in a text according to a predefined schema. The schema, defining which relation types are relevant is highly dependent on the specific application and domain. Supervised learning methods for relation extraction (Soares et al., 2019; Zhou et al., 2021; Zhang et al., 2021; Xu et al., 2021; Xiao et al., 2022), which have advanced rapidly since the introduction of pretrained language models such as BERT (Devlin et al., 2019), need large corpora of annotated relation instances to learn a schema. Since annotating data sets for relation extraction manually is expensive and time consuming, few-shot learning for relation extraction represents a promising solution for relation extraction at scale.
While the general  $N$ -way  $K$ -shot few-shot learning framework is relatively well defined and appears easy to apply to relation extraction, constructing realistic benchmark tasks has proven to be challenging. One of the core difficulties of establishing
a realistic benchmark task for few-shot relation extraction is correctly modelling the most frequent situation a relation extraction system will encounter, none-of-the-above (NOTA) detection. NOTA refers to the case in which a candidate pair of entities does not hold any of the relations defined in the schema, a situation which is far more common than its reverse (for the document-level data set DocRED Yao et al., 2019),  $96.84\%$  of candidate entity pairs are NOTA cases). While initial benchmarks (Han et al., 2018) ignored this scenario altogether, researchers working on few-shot relation extraction have pushed for more realistic NOTA modeling in tasks and developed methods that can better detect NOTA instances (Gao et al., 2019; Sabo et al., 2021).
Parallel to the outlined efforts towards realistic few-shot relation extraction benchmarks, research into supervised relation extraction has moved from sentence-level tasks, relation extraction within single sentences, to document-level relation extraction. The push towards document-level relation extraction is motivated by (1) extracting more complex, cross-sentence relations and (2) information extraction at scale. The latter is driven by an inherent challenge when increasing the scope from single sentences to multiple sentences: The number of entities involved increases and with that comes a quadratic increase in candidate entity pairs. While sentence-level approaches typically evaluate each candidate entity pair individually, this strategy is infeasible at the document-level (DocRED contains an average of 393.5 candidate entity pairs per document, compared to only 2 for many sentence level-tasks). In addition to the increased computation requirements, this results in a drastic increase in the amount of NOTA examples in a given query, demanding new methods of handling the imbalances that come with this change of distribution (Han and Wang, 2020; Zhou et al., 2021).
All current few-shot relation extraction bench
'https://github.com/nicpopovic/FREDo
Human Language Technologies, pages 5733 - 5746
marks are based on sentence-level tasks. We argue that moving few-shot relation extraction from the sentence-level to the document-level: (1) brings with it as an inherent characteristic the more realistic NOTA distribution which prior work has looked to emulate and (2) will make the resulting methods more suitable for large scale information extraction.
In this work, we therefore define a new set of few-shot learning tasks for document-level relation extraction and design a strategy for creating realistic benchmarks from annotated document corpora. Applying the above to the data sets DocRED (Yao et al., 2019) and sciERC (Luan et al., 2018), we construct a few-shot document-level relation extraction (FSDLRE) benchmark, FREDo, consisting of two main tasks, an in-domain and a cross-domain task requiring domain adaptation. Finally, building on the state-of-the-art few-shot relation extraction approach MNAV (Sabo et al., 2021) and document-level relation extraction concepts (Zhou et al., 2021), we develop two approaches for tackling the above tasks.
We begin by outlining key related work in section 2. In section 3 we give a description of the proposed tasks. Next, in section 4 we explain the construction of the benchmark, FREDo, followed by an overview of the proposed methods (section 5), an analysis and discussion of the observed results (section 6), and the conclusion (section 7).
To the best of our knowledge, all current few-shot relation extraction benchmarks (Han et al., 2018; Gao et al., 2019; Sabo et al., 2021) focus on extracting relations from single sentences. FewRel (Han et al., 2018) introduces a relation extraction benchmark in the  $N$ -way  $K$ -shot setting (Vinyals et al., 2016; Snell et al., 2017) in which a relation instance is to be assigned to one of  $N$  classes given only  $K$  examples for each of the classes. In this setting human performance was quickly surpassed (Soares et al., 2019), leading Gao et al. (Gao et al., 2019) to create FewRel 2.0 in an effort to increase the difficulty by adding a domain adaptation task, as well as a NOTA detection task. Sabo et al. (Sabo et al., 2021) argue that the way FewRel 2.0 models NOTA cases is not realistic due to the way NOTA instances are sampled, develop a framework for creating more realistic benchmarks and propose building such a benchmark using the sentence-level data
set TACRED (Zhang et al., 2017). Tran et al. (Tran et al., 2021) forego labeled training data altogether and focus on the one-shot and weakly-supervised classification setting without NOTA cases.
While multiple relation extraction data sets based on annotated documents, rather than single sentences, are available in the form of CDR (Li et al., 2016), sciERC (Luan et al., 2018), SciREX (Jain et al., 2020), DialogRE (Yu et al., 2020), and GDA (Wu et al., 2019), the introduction of the large scale data set DocRED (Yao et al., 2019) seems to have significantly increased research interest into supervised relation extraction at the document-level more recently (Zhou et al., 2021; Zhang et al., 2021; Xu et al., 2021; Xiao et al., 2022).
Since documents contain considerably more entities than individual sentences and the amount of candidate entity pairs increases quadratically with the amount of entities, applying sentence-level methods to document-level tasks is not feasible. Document-level relation extraction approaches therefore use a different architecture (Wang et al., 2019) than sentence-level approaches. Another challenge is the large imbalance in the amount of positive and negative examples of relations encountered during training. Some researchers approach the problem by resampling training examples to counteract the imbalance (Han and Wang, 2020), while others use more specialized solutions, such as modified loss functions (Zhou et al., 2021).
3 Task Description
In document-level relation extraction the task is to return a set  $S$  containing all valid triples of the format  $(e_h, r_i, e_t)$  for a document  $D$ . Here,  $e_h$  and  $e_t$  are the head- and tail-entity of a relation instance, respectively, and  $r_i \in R$  is a relation type, with  $R$  being the set of relation types for which instances are to be extracted. The positions of any entity mentions, as well as any co-reference clusters are provided as part of the input<sup>2</sup>. In both supervised learning and few-shot learning the documents used at test time are sampled from a different corpus than those used at training time. The added complexity in few-shot learning is caused (1) by a change in the set of relation types  $R$  between training and test time, and (2) by a much smaller amount of annotated examples given for each relation type.
2The setting in which no such annotations are given is typically referred to as joint entity and relation extraction and is out of scope of this paper.
Figure 1: Illustration of an episode in the Few-Shot Document-Level Relation Extraction setting. Given a support document with annotated relation instances, the task is to return all instances of the same relation types for the query document. During testing a different corpus of documents, as well as a different set of relation types are used than during training.
3.1 Document-Level Few-Shot Relation Extraction
In figure 1 we give an illustration of the proposed task setting. We define as few-shot document-level relation extraction (FSDLRE) the following: Given a set of support documents  $\{D_{S,1},\dots,D_{S,k}\}$ , the corresponding sets  $\{T_{S,1},\dots,T_{S,k}\}$  containing all valid triples for each support document, and a query document  $D_{Q}$ , the task is to return the set  $T_{Q}$ , containing all valid triples in the query document. The sets  $\{T_{S,1},\dots,T_{S,k}\}$  and  $T_{Q}$  consist of triples for the relation types  $R_{\text{episode}}$ .  $R_{\text{episode}}$  is a subset of either  $R_{\text{train}}$  or  $R_{\text{test}}$ , two disjoint sets of relation types used for training or testing, respectively. The annotations of the support documents are complete, meaning that any candidate entity pair for which no relation type has been assigned can be considered NOTA.
3.1.1 In-Domain vs. Cross-Domain
For in-domain FSDLRE training and test documents are taken from the same domain. For cross-domain FSDLRE the test documents are taken from a different domain. Consequently, text style, text content, entity types, and relation types will all differ from those seen in the training documents. While this increases the difficulty of the challenge, this also resembles a more realistic application of few-shot relation extraction methods: A key motivation for few-shot learning is to develop methods which can be applied to new data without the need for large-scale manual annotation. Restricting the applicability of a method to a specific domain and annotation procedure does not fit this idea.
3.2 Differences to Existing Benchmarks
The tasks described above differ from existing few-shot relation extracting benchmarks in multiple ways. (1) Operating at document-level means the data now includes instances of relations expressed across multiple sentences and that models need to be able to evaluate candidate entity pairs more efficiently. (2) Like for FS-TACRED (Sabo et al., 2021), the amount of candidate entity pairs for which no relation is to be extracted is significantly larger than in other benchmarks (96.4% compared to 15%/50% for FewRel 2.0 (Gao et al., 2019)) and the distribution from which NOTA instances are sampled, is more realistic than in FewRel 2.0, where NOTA instances are always instances of other, valid relation types. (3) By requiring support annotations to be complete we have access to a support NOTA distribution, which is not the case for any existing benchmarks. (4) Our tasks do not follow the fixed  $N$ -way  $K$ -shot format that related work has followed. Instead,  $N$  and  $K$  are variable between documents and therefore between individual episodes.
4 FREDo: Few-Shot Document-Level Relation Extraction Benchmark
4.1 Selected Data Sets
In order to construct a benchmark based on the tasks described in section 3 we require fully annotated data sets from two separate domains. For the training set and the in-domain test set we use DocRED (Yao et al., 2019) due to it being, to the best of our knowledge, the largest and most widely used document-level relation extraction data set. For the
Data Set	#Docs	# RT1	# CP/Doc2	# Words/Doc	# Sents/Doc	Domain
DocRED	4051	96	394	172	8	Non-specific
sciERC	500	7	187	118	5.4	Scientific Publications
1 relation types
${}^{2}$  candidate pairs per document
Table 1: A comparison of DocRED (Yao et al., 2019) and sciERC (Luan et al., 2018), the datasets selected for the FREDo benchmark.
cross-domain test set we use sciERC (Luan et al., 2018) due to its domain (abstracts of scientific publications), which differs from DocRED (Wikipedia abstracts), and the fact that it contains annotations for 7 relation types. In table 1 we show a comparison of the selected datasets. Additional document-level relation extraction data sets, SciREX (Jain et al., 2020), DialogRE (Yu et al., 2020), GDA (Wu et al., 2019), CDR (Li et al., 2016), were considered but ultimately not used for the cross-domain set, due to the amount of relation types annotated (too few), missing coreference links, or differing relation format (SciREX annotations are based on  $N$ -ary relations, while the other datasets annotate only binary relations).
4.2 Training and Test Data
4.2.1 Document Corpora
We begin by building 3 separate corpora of documents, 1 for training and development and 1 for testing each task (in-/cross-domain). Since the annotated test corpus for DocRED is not publicly available we use the documents in the development corpus as the test corpus for our in-domain task (meta-test). The DocRED training corpus is therefore used as the basis for both our training, and development set (meta-train). For the cross-domain task we require only a test set. This is because the training and development set for this task are identical to that of the in-domain task. We therefore use all documents in sciERC as our cross-domain test set (meta-test).
4.2.2 Assigning Relation Types
For preprocessing, we begin by comparing the relation types annotated in sciERC to those in the DocRED corpus<sup>3</sup>. We find 2 relation types (P279, P361) which are annotated in both DocRED and sciERC. We remove these from the DocRED corpus in order to prevent data leakage between train and test sets.
For DocRED, we split the remaining 94 relation types into 4 disjoint sets, a training set (62)  $R_{train}$ , development set (16)  $R_{dev}$ , and in-domain test set (16)  $R_{test}$ . For the cross-domain test set we use all 7 relation types in the sciERC corpus. An overview of the relation types assigned to each set can be found in appendix A and B.
4.3 Test Episode Sampling
In few-shot learning, each training/testing step consisting of support documents and query documents is called an episode. Since evaluating every possible combination of support and query documents would result in too many episodes (approx. 1 million episodes for the in-domain and  $250\mathrm{k}$  episodes for the cross-domain test set) we need to sample a smaller amount of episodes from our corpora. We chose our sampling procedure with the goal of producing a representative measurement of the macro  $F_{1}$  score.
For few-shot learning tasks the episode sampling process can be split into 2 steps, the first step being the sampling of the support examples and the second step being the sampling of the query examples. Unlike the sentence-level scenario where each example contains exactly one relation instance, each document we sample contains multiple instances of different relation types. In order to balance the amount of times each relation type is seen as a support example during testing we use the following procedures for the first sampling step: We begin by selecting from the set  $R_{test}$  the relation type  $r_s$  which is currently least represented in the testing corpus. If there are multiple such relation types we randomly choose one. For this relation type we sample support documents which contain at least one instance of  $r_s$  each. Since the selected support documents may contain instances of other relation types in  $R_{test}$  we add all of the relation types contained in the support document to the episodes annotation schema. Following Sabo et al. (Sabo
3The mapping can be found in appendix A.
4The first, if there are multiple support documents.
Task	N	K (micro)	K (macro)
in-domain
1-Doc	2.18	2.36	2.24
3-Doc	3.47	4.30	4.31
cross-domain
1-Doc	4.26	2.73	2.40
3-Doc	6.08	5.55	5.27
Table 2: Average values for  $N$  and  $K$  across test episodes.  $K$  (micro) denotes the average across all episodes,  $K$  (macro) denotes the weighted average of mean  $K$  for each relation type.
et al., 2021), we randomly sample query documents from the test corpus<sup>5</sup> to realistically represent the NOTA distribution of the entire corpus.
4.3.1 Choosing Test Set Sizes
In order to choose a sufficiently large amount of test episodes for a representative  $F_{1}$  score we evaluate a trained model for 50k episodes, logging the macro  $f_{1}$  score at intervals of 100 episodes. We repeat this for 5 different random seeds. Using the variance between the 5 measurements as a guide, we choose a number of episodes which we deem to satisfy a good balance between low variance and manageable test set size. For robustness we sample episodes with 3 different random seeds for the final test sets. The resulting test set sizes are: 15k episodes for the in-domain tasks and 3k episodes for the cross-domain tasks.
4.4 Characteristics of Resulting Tasks
Existing few-shot benchmarks typically set 2 tasks, a single-shot and a  $K$ -shot (3/5/10-shot) challenge, in order to determine the way performance may scale when adding annotated training data. Due to the nature of our tasks,  $N$  and  $K$  are variable from episode to episode, depending on the specific support documents and relation types. We measure the scalability of approaches by defining a 1-Doc and 3-Doc challenge.
Therefore, the proposed benchmark, FREDo, consists of 2 main tasks with a 1-Doc and a 3-Doc subtask each:
- The in-domain tasks for which an approach which has been trained on documents sampled from DocRED is evaluated on 15k episodes generated using documents from DocRED.
- The cross-domain tasks for which an approach which has been trained on documents sampled from DocRED is evaluated on 3k episodes generated using documents from sciERC.
In order to better characterize our tasks in relation to the common  $N$ -way  $K$ -shot format we measure the distribution of  $N$  and  $K$  across our test sets. All the average values for  $K$  and  $N$  are shown in table 2. We find that the mean values of  $N$  are (2.18/3.47) for the in-domain tasks (1-/3-doc) and (4.26/6.08) for the cross-domain task. For  $K$  we calculate both the mean values across all episodes (micro), as well as the mean across the different relation types (macro).
A common approach to relation extraction in general is to compute the similarity between embeddings produced by a fine-tuned language model such as BERT (Devlin et al., 2019). In order to produce a relation embedding for a given pair of entities, most approaches concatenate embeddings corresponding to each entity. One way to generate an entity embedding from the output of a language model is to average the embeddings of all tokens belonging to an entity. Another way is the use of so-called entity markers, introduced by Soares et al. (Soares et al., 2019), which are tokens placed at the beginning and end of an entity mention within the input text. The embeddings of the tokens placed at the start of each entity mention are then used as the entity embeddings. In few-shot learning, a common way to use embedding similarity are prototypical networks (Snell et al., 2017). Here, the embeddings of all  $K$  support examples are averaged into a so-called prototype. Given a query embedding, the similarity to the  $N$  class-prototypes is then used for classification.
In order to assess the difficulty of our challenges we measure the performance of 3 approaches. We set an initial baseline using the pretrained language model  $\mathrm{BERT}_{\mathrm{BASE}}$  (Devlin et al., 2019) without fine-tuning. Next, we adapt the state-of-the-art sentence-level few-shot relation extraction method MNAV (Sabo et al., 2021) to the document-level (DL-MNAV). Finally, we make 2 modifications to DL-MNAV (SIE and SBN) to improve cross-domain performance. In figure 2 we show a comparison of how the different models handle deci
5Note that we exclude the previously sampled support documents.
(a) available for inference
$\bullet \bullet \bullet$  support relation instance
(b) DL-MNAV
$\odot$  support NOTA instance
(c) DL-MNAV $_{SIE}$
Figure 2: Embedding space illustration of the different models used for evaluation at inference time. Shown in (a) are all data points available from support documents, as well as the query instance. Class prototypes in (b) are computed as mean of support relation instances. Learned NOTA vectors, shown in red, are not based on support NOTA instances but learned during training and do not change based on the support document. Dotted lines indicate distances affecting the classification of the query instance, shown in white. Background colors illustrate approximate classification boundaries. For the baseline model (5.1.1), decision boundaries are the same as in (d).
$\bigcirc$  query instance
$\bullet$  class prototype
(d) DL-MNAV $_{SIE + SBN}$
learned NOTA vector
sion boundaries with respect to support and query instances.
5.1.1 Baseline
We set an initial baseline using the pretrained language model  $\mathrm{BERT}_{\mathrm{BASE}}$  (Devlin et al., 2019) without fine-tuning in the following way: We encode each document using the language model and then average the output tokens of each entity mention. Following Han and Wang (Han and Wang, 2020), we then average the mention representations for each entity. The resulting entity mentions are then concatenated for each candidate pair of entities to form relation embeddings. The similarity between a relation embedding in a query document to a relation embedding in a support document is calculated via their dot product. The relation type of the support embedding producing the highest dot product is output as the predicted relation type.
5.1.2 Adapting MNAV to Document-Level (DL-MNAV)
The current state-of-the-art few-shot sentence-level relation extraction method MNAV (Sabo et al., 2021) uses a combination of entity embeddings based on entity markers and prototypical networks. Furthermore, it introduces the idea of learning  $M$  prototypes to represent the NOTA class. In order to use MNAV at the document-level one key architectural change is required: Instead of only marking two entities, a head- and a tail-entity, with two different tokens we mark all spans with the same entity marker tokens. Furthermore, following other document-level approaches (Han and Wang, 2020;
Zhou et al., 2021) we apply a pooling step to create representations for entities which are mentioned more than once in a document.
One of the challenges in realistic relation extraction is the large imbalance in the amount of positive and negative examples of relations encountered during training. In document-level relation extraction this challenge is even more central to the task than in sentence-level relation extraction. Preliminary experiments showed that simply using cross-entropy loss, as is done for MNAV, yields sub-par results. To tackle this<sup>7</sup>, we adopt the adaptive thresholding loss function used by Zhou et al. (Zhou et al., 2021) which is an adaptation of categorical cross entropy loss designed specifically for classifiers which treat NOTA as a relation type during classification, as is the case for MNAV.
Finally, we modify the initialization procedure of the NOTA vectors. While Sabo et al. (2021) initialize the vectors using an averaged value of relation representations, we sample NOTA representations from the support documents during the first training step.