SlowGuess/ACL_2020_OCR huggingface dataset:
SCIREX: A Challenge Dataset for Document-Level Information Extraction
Sarthak Jain $^{2*}$  Madeleine van Zuylen $^{1}$  Hannaneh Hajishirzi $^{1,3}$  Iz Beltagy $^{1}$
Allen Institute for AI1 Northeastern University2 University of Washington3
{madeleinev,hannah,beltagy}@allenai.org
Extracting information from full documents is an important problem in many domains, but most previous work focuses on identifying relationships within a sentence or a paragraph. It is challenging to create a large-scale information extraction (IE) dataset at the document level since it requires an understanding of the whole document to annotate entities and their document-level relationships that usually span beyond sentences or even sections. In this paper, we introduce ScIREX, a document level IE dataset that encompasses multiple IE tasks, including salient entity identification and document level  $N$ -ary relation identification from scientific articles. We annotate our dataset by integrating automatic and human annotations, leveraging existing scientific knowledge resources. We develop a neural model as a strong baseline that extends previous state-of-the-art IE models to document-level IE. Analyzing the model performance shows a significant gap between human performance and current baselines, inviting the community to use our dataset as a challenge to develop document-level IE models. Our data and code are publicly available at https://github.com/allenai/SciREX
Extracting information about entities and their relationships from unstructured text is an important problem in NLP. Conventional datasets and methods for information extraction (IE) focus on within-sentence relations from general Newswire text (Zhang et al., 2017). However, recent work started studying the development of full IE models and datasets for short paragraphs (e.g., information extraction from abstracts of scientific articles as in SCIERC (Luan et al., 2018)), or only extracting
Figure 1: An example showing annotations for entity mentions (Dataset, Metric, Task, Method), coreferences (indicated by arrows), salient entities (bold), and  $N$ -ary relation (SQuaD, Machine Comprehension, BiDAF (ensemble), EM/F1) that can only be extracted by aggregating information across sections.
relations (given ground truth entities) on long documents (e.g. Jia et al. (2019)). While these tasks provide a reasonable testbed for developing IE models, a significant amount of information can only be gleaned from analyzing the full document. To this end, not much work has been done on developing full IE datasets and model for long documents.
Creating datasets for information extraction at the document level is challenging because it requires domain expertise and considerable annotation effort to comprehensively annotate a full document for multiple IE tasks. In addition to local relationships between entities, it requires identifying document-level relationships that go beyond sentences and even sections. Figure 1 shows an example of such document level relation (Dataset: SQuAD, Metric: EM, Method: BiDAF, Task:machine comprehension).
In this paper, we introduce SCIREX, a new comprehensive dataset for information extraction from
*Work done while at AI2
7506
Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 7506-7516 July 5 - 10, 2020. ©2020 Association for Computational Linguistics
scientific articles. Our dataset focuses on the task of identifying the main results of a scientific article as a tuple (Dataset, Metric, Task, Method) from raw text. It consists of three major subtasks, identifying individual entities, their document level relationships, and predicting their saliency in the document (i.e., entities that take part in the results of the article and are not merely, for example, mentioned in Related Work). Our dataset is fully annotated with entities, their mentions, their coreferences, and their document level relations.
To overcome the annotation challenges for large documents, we perform both automatic and manual annotations, leveraging external scientific knowledge bases. An automatic annotation stage identifies candidate mentions of entities with high recall, then an expert annotator corrects these extracted mentions by referring to the text of the article and an external knowledge base.<sup>1</sup> This strategy significantly reduces the time necessary to fully annotate large documents for multiple IE tasks.
In addition, we introduce a neural model as a strong baseline to perform this task end-to-end. Our model identifies mentions, their saliency, and their coreference links. It then clusters salient mentions into entities and identifies document level relations. We did not find other models that can perform the full task, so we evaluated existing state-of-the-art models on subtasks, and found our baseline model to outperform them. Experiments also show that our end-to-end document level IE task is challenging, with the most challenging subtasks being identifying salient entities, and to a lesser extent, discovering document level relations.
The contributions of our paper are as follows, 1. we introduce ScIREX, a dataset that evaluates a comprehensive list of IE tasks, including  $N$ -ary relations that span long documents. This is a unique setting compared to prior work that focuses on short paragraphs or a single IE task. 2. We develop a baseline model that, to the best of our knowledge, is the first attempt toward a neural full document IE. Our analysis emphasizes the need for better IE models that can overcome the new challenges posed by our dataset. We invite the research community to focus on this important, challenging task.
Scientific IE In recent years, there has been multiple attempts to automatically extract structured
information from scientific articles. These types of extractions include citation analysis (Jurgens et al., 2018;Cohan et al., 2019), identifying entities and relations (Augenstein et al., 2017; Luan et al., 2019, 2017), and unsupervised detection of entities and their coreference information (Tsai et al., 2013).
Most structured extraction tasks from among these have revolved around extraction from sentences or abstracts of the articles. A recent example is ScIERC (Luan et al., 2018), a dataset of 500 richly annotated scientific abstracts containing mention spans and their types, coreference information between mentions, and binary relations annotations. We use ScIERC to bootstrap our data annotation procedure (Section 3.2).
There has been a lack of comprehensive IE datasets annotated at the document level. Recent work by Hou et al. (2019); Jia et al. (2019) tried to rectify this by using distant supervision annotations to build datasets for document-level relation extraction. In both datasets, the task of relation extraction is formulated as a binary classification to check if a triplet of ground-truth entities is expressed in the document or not. Instead, our work focuses on a comprehensive list of information extraction tasks "from scratch", where the input is the raw document. This makes the IE model more interesting as it requires to perform entity extraction, coreference resolution, saliency detection in addition to the relation extraction.
General IE Most work in general domain IE focus on sentence-level information extraction (Stanovsky et al., 2018; Qin et al., 2018; Jie and Lu, 2019). Recently, however, Yao et al. (2019) introduced DocRED, a dataset of cross-sentence relation extractions on Wikipedia paragraphs. The paragraphs are of a comparable length to that of SCIERC, which is significantly shorter than documents in our dataset.
Previous IE work on the TAC KBP competitions (Ellis et al., 2017; Getman et al., 2018) comprise multiple knowledge base population tasks. Our task can be considered a variant of the TAC KBP "cold start" task that discovers new entities and entity attributes (slot filling) from scratch. Two aspects of our task make it more interesting, 1) our model needs to be able to extract facts that
Another approach is to perform entity extraction then use the binary classification approach with a list of all possible combinations of relation tuples. This might work for short documents, but it is intractable for long documents because of the large number of entities.
Papers with Code: paperswithcode.com
7507
are mentioned once or twice rather than rely on the redundancy of information in their documents (e.g. Rahman et al. (2016)), 2) TAC KBP relations are usually sentence-level binary relations between a query entity and an attribute (e.g. Angeli et al. (2015)), while our relations are 4-ary, span the whole document, and can't be split into multiple binary relations as discussed in Section 3.1.
End-to-End Neural IE models With neural networks, a few end-to-end models have been proposed that perform multiple IE tasks jointly (Miwa and Bansal, 2016; Luan et al., 2018; Wadden et al., 2019). The closest to our work is DYGIE++ (Wadden et al., 2019), which does named entity recognition, binary relation extraction, and event extraction in one model. DYGIE++ is a span-enumeration based model which works well for short paragraphs but does not scale well to long documents. Instead, we use a CRF sequence tagger, which scales well. Our model also extracts 4-ary relations between salient entity clusters, which requires a more global view of the document than that needed to extract binary relations between all pairs of entity mentions.
3 Document-Level IE
Our goal is to extend sentence-level IE to documents and construct a dataset for document-level information extraction from scientific articles. This section defines the IE tasks we address, and describe the details of building our ScIREX dataset.
Entity Recognition Our entities are abstract objects of type Method, Task, Metric, or Dataset that appear as text in a scientific article. We define "mentions" (or spans) as a specific instantiation of the entity in the text - this could be the actual name of the entity, its abbreviation, etc. The entity recognition task is to identify "entity mentions" and classify them with their types.
Salient Entity Identification Entities appear in a scientific article are not equally important. For example, a task mentioned in the related work section is less important than the main task of the article. In our case, salient entity identification refers to finding if an entity is taking part in the article evaluation. Salient Datasets, Metrics, Tasks, and Methods are those needed to describe the article's results. For the rest of this paper, we will use
the term salient to refer to entities that belong to a result relation tuple.
Coreference is the task of identifying a cluster of mentions of an entity (or a salient entity) that are coreferred in a single document.
Relation Extraction is the task of extracting  $N$ -ary relations between entities in a scientific article. We are interested in discovering binary, 3-ary, and 4-ary relations between a collection of entities of type (Dataset, Method, Metric, and Task). It is important to note that this 4-ary relation can't be split into multiple binary relations because, e.g., a dataset might have multiple tasks, and each one has its own metric, so the metric cannot be decided solely based on the dataset or the task.
Document-level information extraction requires a global understanding of the full document to annotate entities, their relations, and their saliency. However, annotating a scientific article is time-consuming and requires expert annotators. This section explains our method for building our ScIREX dataset with little annotation effort. It combines distant supervision from an existing KB and noisy automatic labeling, to provide a much simpler annotation task.
Existing KB: Papers with Code Papers with Code  $(\mathrm{PwC})^3$  is a publicly available corpus of 1,170 articles published in ML conferences annotated with result five-tuples of (Dataset, Metric, Method, Task, Score). The PwC curators collected this data from public leaderboards, previously curated results by other people, manual annotations, and from authors submitting results of their work.
This dataset provides us with distant supervision signal for a task that requires document-level understanding - extracting result tuples. The signal is "distant" (Riedel et al., 2010) because, while we know that the PwC result tuple exists in the article, we don't know where exactly it is mentioned (PwC does not provide entity spans, and PwC entity names may or may not appear exactly in the document).
PDF preprocessing PwC provides arXiv IDs for their papers. To extract raw text and section information, we use LaTeX (https://dlmf.nist.
3https://github.com/paperswithcode/paperswithcode-data
7508
Statistics (avg per doc)	SCIREX	SCIERC
Words	5,737	130
Sections	22	1
Mentions	360	16
Salient Entities	8	—
Binary Relations	16	9.4
4-ary Relations	5	—
Table 1: Comparison of SCIREX with next biggest ML Information Extraction dataset SCIERC. SCIREX consists of 438 documents. All dataset statistics are per-document averages.  $57\%$  of binary and  $99\%$  of 4-ary relations occur across sentences.  $20\%$  binary and  $55\%$  4-ary relations occur across sections. This highlight the need for document level models.
Dataset	Metric	Task	Method	Deleted
Dataset	3.55	0.01	0.07	0.16	0.03
Metric	0.02	7.95	0.00	0.03	0.00
Task	0.32	0.07	17.92	0.44	0.01
Method	0.65	0.21	0.24	53.27	0.02
Added	2.40	1.30	2.82	8.50	-
Table 2: Confusion Matrix for the mention-level corrections (change type, add span, or delete span). Values are average percentages "per document" (not per type). For example, cell at intersection of row Metric and column Task contains document-average percentage of span-type change from Metric to Task. The column Deleted represents percent spans that were deleted. The row Added represents percent spans added. Diagonal represent percent spans of each type that are correctly labeled by the automatic labeling and didn't need to change by the human annotator.
gov/LaTeX/) for papers with latex source (all 438 annotated papers), or use Grobid (GRO, 2008-2020) for papers in PDF format (only  $10\%$  of remaining papers did not have latex source). LaTeX allowed us to extract clean document text with no figures / tables / equations. We leave it as future work to augment our dataset with these structured fields. To extract tokens and sentences, we use the SpaCy (https://spacy.io/) library.
Automatic Labeling Given the length of the document is on the order of 5K tokens, we simplify the human annotation task by automatically labeling the data with noisy labels, then an expert annotator only needs to fix the labeling mistakes.
One possible way to augment the distant supervision provided by PwC is finding mention spans of PwC entities. Initial experiments showed that this did not work well because it does not provide
enough span-level annotations that the model can use to learn to recognize mention spans.
To get more dense span-level information, we want to label salient (corresponding to PwC entities) and also non-salient spans. We train a standard BERT+CRF sequence labeling model on the SCIERC dataset (described in Section 2). We run this model on each of the documents in the PwC corpus, and it provides us with automatic (but noisy) predictions for mention span identification.
The next step is to find mention spans that correspond to PwC entities. For each mention predicted by our ScIERC-trained model, we compute a Jaccard similarity with each of the PwC entities. Each mention is linked to the entity if the threshold exceeds a certain  $\epsilon$ . To determine  $\epsilon$ , two expert annotators manually went through 10 documents to mark identified mentions with entity names, and  $\epsilon$  was chosen such that the probability of this assignment is maximized. We use this threshold to determine a mapping for the remaining 1,170 documents. Given that Jaccard-similarity is a coarse measure of similarity, this step favors high recall over precision.
Human Annotation Given this noisily labeled data, we ask our annotator to perform necessary corrections to generate high-quality annotations. Annotators are provided with a list of papers-withcode entities that they need to find in the document, making their annotations deliberate (as opposed to not knowing which entities to annotate). Our annotator deleted and modified types of spans for salient entities (belong to PwC result tuple) and non-salient entities, while only adding missed spans for salient ones. Also, if a mention was linked to a wrong PwC entity, then our annotator was also asked to correct it. Full annotation instructions are provided in Appendix B.
3.3 Dataset and Annotation Statistics
Dataset statistics and Cross-section Relations Using the annotation procedure mentioned above, we build a dataset of 438 fully annotated documents. Table 1 provides dataset statistics and shows the proportion of relations in our dataset that requires reasoning across sentence/section. It shows that the majority of the relations, especially 4-ary relations span multiple sentences or even multiple sections. An example of such cross-section reasoning can be found in Figure 1.
7509
Corrections Table 2 provides information about the average number of changes made during the human annotation. It shows that  $83\%$  (sum of diagonal) are correct automatic labels,  $15\%$  (sum of bottom row) are newly added spans,  $2\%$  are type changes, and a negligible percentage is deleted entities (sum of the last column). Also, on average,  $12\%$  (not in the table) of the final mentions in the document had the wrong PwC links and needed to be corrected, with a majority of changes being removing links from Method spans.
Inter-annotator agreement We also asked four experts (Ph.D. students in ML/NLP field) to annotate five documents to compute the inter-annotator agreement. For mention classification, we achieve  $95\%$  average cohen-  $\kappa$  scores between each pair of experts and our main annotator.
Annotation Speed To measure if automatic labeling is making the human annotation faster, we also asked our annotator to perform annotations on five documents without automatic labeling. We compute the difference in time between these two forms of annotation per entity annotated. Note that here, we only ask our annotator to annotate salient mentions. With the automatic labeling, annotation speed is 1.34 sec per entity time vs. 2.48 sec per entity time on documents without automatic labeling (a 1.85x speedup). We also observe  $24\%$  improvement in recall of salient mentions by including non-salient mentions, further showing the utility of this approach.
We develop a neural model that performs document-level IE tasks jointly in an end-to-end fashion. This section details our model design (also summarized in Figure 2).
Document Representation An input document  $D$  is represented as a list of sections  $[s_1, \dots, s_{|S|}]$ . We encode the document in two steps, section-level, then document-level. We use pretrained contextualized token encodings using SciBERT (Beltagy et al., 2019) over each section separately to get embeddings for tokens in that section. To allow document-level information flow, we concatenate
the section-level token embeddings and add a BiLSTM on top of them. This allows the model to take into account cross-section dependencies. Thus for each token  $w_{i}$  in the document, this step outputs an embedding  $e_{i}$ .
Mention Identification and Classification Given token embeddings, our model applies a sequence tagger that identifies mentions and classifies their types. We train a BIOUL based CRF tagger on top of the BERT-BiLSTM embeddings of words to predict mention spans  $m_j$  and their corresponding types.
Mention Representation Given the words  $\{w_{j_1},\dots,w_{j_N}\}$  of a mention  $m_j$ , our model learns a mention embedding  $me_j$  of the mention, which will be used in later saliency identification and relation classification steps. The mention embedding is the concatenation of first token embedding  $e_{j_1}$ , last token embedding  $e_{j_N}$  and attention weighted average of all embeddings in the mention span  $\sum_{k = 1}^{N}\alpha_{jk}e_{jk}$ , where  $e_{jk}$  is the embedding of word  $w_{j_k}$  and  $\alpha_{j_k}$  are scalars computed by passing the token embedding through an additive attention layer (Bahdanau et al., 2015). We concatenate these embeddings with additional features — span's relative position in the document, an indicator showing if the sentence containing the mention also contains some marker words like 'experiment' or 'dataset' and the mention type.
Salient Mention Classification Each mention  $m_j$  is classified as being salient or not (i.e., should it belong in a relation tuple) by passing its span embedding  $me_j$  through a feedforward layer. Because saliency is a property of entities, not mentions, this mention saliency score is just an input to the salient entity cluster identifications.
Pairwise Coreference Resolution The coreference step is given a list of all pairs of identified mentions, and it decides which pair is coreferring. This component is separate from the end-to-end model. It concatenates the "surface forms" of two spans  $m_{i}$  and  $m_{j}$ , embed them using SciBERT, then use a linear classification layer on top of [CLS] embedding to compute the pairwise coreference score  $c_{ij}$ . We also tried integrating it into our model, where we classify pairs of "span embeddings" (not the surface form) but found the separate model that uses surface forms to work much better.
$^4$ with the exception of coreference resolution  
 $^5$ If the section is bigger than 512 tokens (SciBERT limit), it is broken into 512 token subsections, and each subsection is encoded separately.
Figure 2: Overview of our model; it uses a two-level BERT+BiLSTM method to get token representations which are passed to a CRF layer to identify mentions. Each mention is classified as being salient or not. A coreference model is trained to cluster these mentions into entities. A final classification layer predicts relationships between 4-tuple of entities (clusters).
Mention clustering Given a list of span pairs  $m_i$  and  $m_j$ , and their pairwise coreference scores  $c_{ij}$ , they are grouped into clusters that can be thought of as representing a single entity. We generate a coreference score matrix for all pairs and perform agglomerative hierarchical clustering (Ward, 1963) on top of it to get actual clusters. The number of clusters is selected based on the silhouette score (Rousseeuw, 1987) which optimizes for the cohesion and separation of clusters and does not depend on having gold standard cluster labels.
Salient Entity Cluster Identification This step filters out clusters from the previous step, and only keep salient clusters for the final relation task. To do so, we take a simple approach that identifies a salient cluster as the one in which there is at least one salient mention (as determined previously). The output of this step is a set of clusters  $C_1, \ldots, C_L$  where each cluster  $C_i$  is a set of mentions  $\{m_{i_1}, \ldots, m_{i_j}\}$  of the same type.
Relation Extraction Given all the clusters of mentions identified in a document from the previous step, our task now is to determine which of these belong together in a relation. To that end, we follow (Jia et al., 2019) methodology. We consider all candidate binary and 4-tuples of clusters and classify them as expressed or not expressed in the document. Here we describe the classification of 4-ary relations. For binary relation, the method is similar.
Consider such a candidate relation (4-tuple of
clusters)  $R = (C_1, C_2, C_3, C_4)$  where each  $C_i$  is a set of mentions  $\{m_{i_1}, \dots, m_{i_j}\}$  in the document representing the same entity. We encode this relation into a single vector by following a two-step procedure - constructing a section embedding and aggregating them to generate a document level embedding. For each section  $s$  of the document, we create a section embedding  $E_R^s$  for this relation as follows -