SlowGuess/ACL_2020_OCR huggingface dataset:
Ruben Kruiper*, Julian F.V. Vincent, Jessica Chen-Burger, Marc P.Y. Desmulliez, Ioannis Konstas
Heriot-Watt University Riccarton Campus, EH14 4AS Edinburgh, United Kingdom
*Corresponding author: rk22@hw.ac.uk
Information Extraction (IE) from scientific texts can be used to guide readers to the central information in scientific documents. But narrow IE systems extract only a fraction of the information captured, and Open IE systems do not perform well on the long and complex sentences encountered in scientific texts. In this work we combine the output of both types of systems to achieve Semi-Open Relation Extraction, a new task that we explore in the Biology domain. First, we present the Focused Open Biological Information Extraction (FOBIE) dataset and use FOBIE to train a state-of-the-art narrow scientific IE system to extract trade-off relations and arguments that are central to biology texts. We then run both the narrow IE system and a state-of-the-art Open IE system on a corpus of 10k open-access scientific biological texts. We show that a significant amount  $(65\%)$  of erroneous and uninformative Open IE extractions can be filtered using narrow IE extractions. Furthermore, we show that the retained extractions are significantly more often informative to a reader. $^{1}$
Identifying the central theme and concepts in scientific texts is a time-consuming task for experts and a hard task for laymen (Alper et al., 2004; El-Arini and Guestrin, 2011; Pain, 2016). This problem is even more pronounced in inter-disciplinary fields of study, where experts in a target domain often lack the deeper knowledge of a source domain (Carr et al., 2018). A specific example is biomimetics, an engineering problem-solving process in which one draws on analogous biological solutions (Kruiper et al., 2016). A major issue is that engineers (target domain) know little biology (source domain) or characteristics of plants or
Figure 1: Example of Semi-Open Relation Extraction in the Biology domain (Burgess et al., 2006). We first extract the TRADE-OFF expressed between the central concepts 'safety' and 'efficiency' (in blue), that takes place specifically in 'conifer species' (in green). We further explore the content of a paper by investigating the results of an Open Information Extraction (OIE) system (in red). The central concepts captured by a TRADE-OFF mechanism enable the filtering of many irrelevant OIE extractions, which are found to be error-prone in scientific texts. Further OIE extractions found in the document can shed light on the semantic meaning of relevant concepts, e.g., 'xylem', as depicted at the top of the Figure (in the red box).
animals (Vattam and Goel, 2013). This domain-mismatch complicates searching for and reasoning over relevant scientific information, rendering biomimetics adventitious and solutions serendipitous (Kruiper et al., 2018).
Recently, TRADE-OFF relations have become of interest to biomimetics (Adriaens, 2019) because a trade-off defined in technology can be directly used to search for relevant texts in biology (Vincent, 2016). TRADE-OFF relations express a problem space in terms of mutual exclusivity constraints between competing demands. Therefore, trade-offs play a prominent role in evolutionary thinking (Agrawal et al., 2010) and are the principal relation under investigation in a significant portion of biology research papers (Garland, 2014). The
<sup>1</sup>We release FOBIE and code at https://github. com/rubenkruiper/FOBIE.
Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 1489-1500 July 5 - 10, 2020. ©2020 Association for Computational Linguistics
functional demands that are traded off are usually abstract and domain-independent terms, such as 'safety' and 'efficiency' in Figure 1. A gap remains in quickly comprehending the central information in a text, e.g., the biological mechanisms that are used to manipulate a trade-off.
Information Extraction (IE), and specifically Relation Extraction (RE), can improve the access to central information for downstream tasks (Santos et al., 2015; Zeng et al., 2014; Jiang et al., 2016; Miwa and Bansal, 2016; Luan et al., 2018a). However, the focus of current RE systems and datasets is either too narrow, i.e., a handful of semantic relations, such as 'USED-FOR' and 'SYNONYM', or too broad, i.e., an unbounded number of generic relations extracted from large, heterogeneous corpora (Niklaus et al., 2018), referred to as Open IE (OIE) (Etzioni et al., 2005; Banko et al., 2007). Narrow approaches to IE from scientific text (Augenstein et al., 2017; Gabor et al., 2018; Luan et al., 2018a) cover only a fraction of the information captured in a paper – usually what is within an abstract. It has been shown that scientific texts contain many unique relation types and, therefore, it is not feasible to create separate narrow IE classifiers for these (Groth et al., 2018). On the other hand, OIE systems are primarily developed for the Web and news-wire domain and have been shown to perform poorly on scientific texts. What laymen really need is a bit of both: the accuracy of narrow RE systems to extract central relations from scientific texts and the flexibility of an OIE system to capture a much larger fraction of the possible relations expressed in scientific texts.
This work aims to enable rapid comprehension of a large scientific document by identifying a) the central concepts in a text and b) the most significant relations that govern these central concepts. To this end, we introduce the task of Semi-Open Relation Extraction (SORE); Figure 1 illustrates the SORE process. First, we find the central concepts 'safety' and 'efficiency' involved in a TRADE-OFF relation. Then, by using the argument concepts of the relation as anchor points, we can explore further concepts and relations, e.g., 'xylem' in Figure 1. Uncovering these relations can elucidate the meaning of unfamiliar concepts to a layperson (Mausam, 2016). The SORE approach is hypothesized to reduce the number of uninformative extractions without limiting RE to a finite set of relations, which could generally benefit IE from sci
entific articles, e.g., materials discovery (Kononova et al., 2019) and drug-gene-mutation interactions (Jia et al., 2019).
To address SORE we create the Focused Open Biological Information Extraction (FOBIE) dataset. FOBIE includes manually-annotated sentences that express explicit trade-offs, or syntactically similar relations, that capture the central concepts in full-text biology papers. We train a span-based RE model used in a strong scientific IE system (Luan et al., 2018a) to jointly extract these relation structures. We explore SORE and use the output of our model to filter the output of an OIE system (Saha and Mausam; Saha et al., 2017; Pal and Mausam, 2016; Christensen et al., 2011) on a corpus of biology papers. Qualitative analyses show that the output of a narrow RE model can speed up expert analysis of trade-offs in biological texts, and be used to filter out both erroneous and uninformative OIE extractions.
2.1 Open Information Extraction
OIE systems use a set of handcrafted or learned extraction rules and rely on dependency features to extract open-domain relational tuples from text (Yu et al., 2017; Niklaus et al., 2018). As OIE systems rely on syntactic features they require little fine-tuning when applied to different domains and the extraction rules work for a variety of relation types (Mausam, 2016). These properties can be especially useful on scientific texts where additional knowledge on unknown concepts can ease the textual comprehension for non-experts. Consider the example OIE extractions for 'xylem' in the top part of Figure 1.
Existing OIE systems have been shown to perform significantly worse on the longer and more complex sentences found in scientific texts than on Wikipedia texts (Groth et al., 2018). Common issues of OIE systems on Web, News, and Wikipedia texts include the correct identification of the boundaries of an argument, handling latent  $n$ -ary relations, difficulty handling negations, and generating uninformative extractions (Schneider et al., 2017). Groth et al. (2018) evaluate the output of two state-of-the-art OIE systems based on correctness, rather than, e.g., the number of missed extractions. They note that the crux of the IE challenge is that extractions reflect the consequence of the sentence. As an example of an uninformative extraction Fader et al.
(2011) note how ' (Faust, made, a deal)' captures the consequence, but not the critical information of whom Faust made a deal with in the sentence "Faust made a deal with the devil." In this work, we explore filtering both incorrect and uninformative OIE extractions from scientific texts using the central concepts that we extract through narrow IE (cf. Section 5.3).
2.2 Narrow Relation Extraction from scientific text
Narrow RE entails identifying two or more related entities in a text and classifying the relation that holds between them. Early works on the combined task of Named Entity Recognition and labeling of relations between extracted entities used precomputed dependency features (Liu et al., 2013; Chen et al., 2015; Lin et al., 2016), word position embeddings (Zeng et al., 2014), or considered only the Shortest Dependency Path between two entities as input (Bunescu and Mooney, 2005; Santos et al., 2015; Zeng et al., 2015). Later work aimed to reduce errors propagated by pre-computed dependency features (Nguyen and Grishman, 2015), or by joint modeling of entities and relations (Miwa and Bansal, 2016). Poor performance of these RE systems on scientific texts has led to the development of domain-specific  $\text{daset}^2$ .
The SCIENCEIE dataset focuses on the extraction of 3 types of key-phrases, rather than Named Entities, and hyponymy and synonymy relations between these (Augenstein et al., 2017). The SemEval 2018 task 7 dataset focuses on 6 narrow relations between 7 entity types (Gabor et al., 2018). And the SCIERC dataset focuses on 7 relation types, including co-reference, between 6 types of entities (Luan et al., 2018a). Top systems developed for both SemEval tasks adapt the LSTM-based approach of Miwa & Bansal (2016), combined with semi-supervised learning and ensembling (Ammar et al., 2018), as well as pre-trained concept embeddings (Luan et al., 2018b).
2.3 BioNLP and BioCreAtIvE
In the past, several BioNLP and BioCreAtIvE shared tasks were organized that aimed at identifying relations in the biology domain (Hirschman
#	FOBIE	SCIENCEIE	SE '18	SCIERC
Arg's	5834	9946	7483	8089
Rel's	4788	672	1595	4716
R/doc	3.09*	1.3	3.2	9.4
Table 1: Number of arguments, relations and relations per instance for FOBIE, SCIENCEIE, the SemEval 2018 task 7 dataset and SCIERC. R/doc stands for relations per sentence* for FOBIE (and per abstract or paragraph for the other datasets).
et al., 2005; Kim et al., 2009; Nédellec et al., 2013; Zhou et al., 2014). Many datasets focus primarily on a predefined set of biomedical relations, such as interactions between known proteins, genes, diseases, drugs, and chemicals (Kim et al., 2003; Krallinger et al., 2017; Cohen et al., 2017; Islamaj Dogan et al., 2019). Examples of more biology-oriented corpora include the BB corpus (Delèger et al., 2016) and the SEDEV corpus (Chaix et al., 2016). The BB corpus includes 4 entity types and 2 relation types that revolve around microorganisms of food interest. Besides abstracts and titles, it contains paragraphs and sentences from 20 full-text documents (Bossy et al., 2019). Similarly, SEDEV consists of 86 paragraphs from 20 full-text articles about seed development in a specific plant, the Arabidopsis thaliana. Considering the small size of the dataset, a relatively large number of many entity and relation types are used; 16 types of Named Entities and 21 types of relations. This results in an imbalanced dataset with 7 relations making up less than  $1\%$  of all relations. Furthermore, there is some overlap in source documents for the train/dev/test split (Chaix et al., 2016).
In contrast to the previously described datasets, FOBIE does not classify arguments of relations into specific entity-types. FOBIE contains annotations of key- phrases found in full-text scientific papers, similar to SCIENCEIE. The key- phrases and rela tions are annotated in 1,548 relatively long and complex sentences, which were sourced from 1,215 full-text scientific biological texts using a RuleBase System. Table 1 provides an overview of the size of FOBIE in comparison to SCIENCEIE, the SemEval 2018 task 7 dataset and ScIERC. Both the BB and SEEDEV corpus contain approximately 3,500 relations within a small sub-domain of biology, while FOBIE focuses more generally on the domain of biology. Section 3 describes the collection of FOBIE and dataset statistics in detail.
$^{2}$ SCIENCEIE SemEval 2017: 500 paragraphs from full-text Computer Science, Material Science, and Physics journal articles, SemEval 2018: 500 abstracts within the domain of Computational Linguistics. SCIERC: 500 abstracts from Artificial Intelligence conference and workshop proceedings.
3 Dataset description
3.1 Dataset collection
A variety of words are able to indicate a trade-off, e.g., compromise, optimization, balance, interplay and conflict (Kruiper et al., 2018). We adapt these terms as trigger words in a Rule-Based System (RBS) and run it on 10k open-access papers that were collected from the Journal of Experimental Biology (JEB) and BioMed Central (BMC) journals on 'Biology', 'Evolutionary Biology, and 'Systems Biology'. The selection of journals was made only to the extent that the articles focus on the biological domain. We retained the abstract, introduction, results, discussion and conclusion sections. We used  $\mathrm{spaCy}^3$  to split the texts into sentences and identify POS tags and dependency structure. The FOBIE dataset contains only sentences that the RBS identified as expressing a TRADE-OFF relation.
The initial annotations extracted by the RBS were manually corrected and extended by a biology expert using the BRAT interface (Stenetorp et al., 2012). We define three relation types: TRADE-OFF, ARGUMENT-MODIFIER and NOT-A-TRADE-OFF. The latter denotes phrases that are related to a trigger word, but not by a TRADE-OFF relation. These syntactically similar relations provide useful training signal as negative samples. Negative samples are important because possible trigger words can be contiguous, e.g., the phrase 'negative correlation' denotes a TRADE-OFF relation, whereas 'correlation' by itself does not. As a result, the annotation of training examples is harder, and lexical and syntactic patterns that correctly signify the relation are sparse (Peng et al., 2017). For simplicity's sake, with some abuse of terminology, we refer to all such relations collectively as trade-offs.
We found a substantial amount of arguments to be nested or in a non-projective relationship. In Figure 2 the prepositional phrase 'in jumping', conceptually refers to both central concept arguments of the relation, i.e., 'the need for energy storage' and 'the presence of resilin'. We adopt the following annotation heuristic: prepositional phrases are treated as modifying phrases when they apply to multiple arguments (as is the case in Figure 2) or can be distinctly separated from the argument, e.g., by punctuation.
# Sentences	1548
Avg. sent. length	37.77
% of sents ≥ 25 tokens	79.26%
Relations:
- TRADE-OFF	765
- NOT-A-TRADE-OFF	2502
- ARG-MODIFIER	1521
Triggers	1600
Arguments	4234
Spans	6309
Unique spans	2701
Unique triggers	41
Max triggers/sent	2
Max spans/sent	7
Spans w/ multiple relations	2075
# single-word arguments	498
Avg. tokens per argument	3.44
Table 2: Aggregated statistics for FOBIE.
We randomly selected 250 sentences (16.1%) for re-annotation and quality control by a second domain expert. The inter-annotator agreement Cohen  $k$  is found to be 0.93. Table 2 summarizes statistics on FOBIE. The final dataset consists of 1,548 single sentences from 1,292 unique documents, split into 1,248/150/150 train/dev/test. The split is controlled for source document overlap to avoid having identical arguments of relations appearing both during training and testing. FOBIE contains relatively long key-phrases with an average of 3.44 tokens and only 12% of them consist of a single token. In comparison SCIENCEIE and SCIERC both contain 31% singleton key-phrases, and the average entity length in SCIERC is 2.36. Furthermore, sentences taken from full-text documents are longer than those found in abstracts. The average sentence length in SCIERC is 24.31 tokens, while 79.26% of the sentences in FOBIE are longer than 25 tokens.
4 Narrow IE baseline
4.1 Task definition
Following Peng et al. (2017) we extract  $n$ -ary relations by (1) identifying the trigger and (2) extracting the binary relations between this trigger and the arguments - inspired by Davidsonian semantics. We define key-phrases as spans of consecutive words  $s \in S$ , with  $S$  all possible spans in a sentence, and relation-types as  $r \in R^d$ , with  $d$  the total number of unique relations. Then a binary relation is a triple  $< \text{governor}, \text{relation}, \text{dependent} >$  with governor and dependent elements of  $S$ . The union of the following binary relations found in a sentence may constitute a non-projective graph:
Figure 2: Example of an annotation in BRAT showing a trigger word, 'correlation', that is related to two arguments, which in turn are related to a single modifier. The trigger word does not indicate a TRADE-OFF relation, but a positive correlation.
Figure 3: We provide the SCIE system with single sentences  $D$  as input. For all possible spans up to width  $W$  a span label  $\in L_{E}$  is computed and a mention score  $\phi_{mr}$ . Spans with the lowest mention scores are pruned, with variable beam size  $\lambda_{n}$ . For combinations of remaining spans a relation label  $\in L_{R}$  is predicted. The set of span labels  $L_{E}$  and the set of relation labels  $L_{R}$  both contain a dummy class  $\epsilon$ .
Def. 1. An explicit trade-off is an instance of a directed relation  $t \in T^o$ , indicated by trigger word  $p \in P^u$  with  $u$  the set of unique trigger words and  $P \subset S$ . A trade-off is a binary relation,  $t \models o$ , with governor  $\in P$  and dependent  $\in S$ . A single trigger word  $p$  can be in  $n$  multiple relations.
Def. 2. An argument-modifier is a directed binary relation  $a \in A^m$ , where we omit the classification of  $a$  into a set of possible modification types  $\in m$ . An instance of  $a$  is then a tuple  $<governor, relation, dependent>$  where one of the arguments is related to a trigger word  $p$ , and both arguments  $\in S$ .
4.2 Baseline system
We adapt a span-based approach that has been used previously for the tasks of co-reference resolution (Lee et al., 2017), Semantic Role Labeling (He et al., 2018), and scientific IE (Luan et al., 2018a). The use of span representations as classifier features enables end-to-end learning by propagating information between multiple tasks without increasing the complexity of inference. We train the ScIE system (Luan et al., 2018a) on FOBIE to extract spans that constitute trigger words and key-phrases, as well as the binary relations between these spans.
Figure 3 illustrates the input that we provide to ScIE. All tokens are embedded using GloVe (Pennington et al., 2014) and ELMo embed
dings (original) (Peters et al., 2018). For a single sentence  $D = \{w_{1},\dots,w_{n}\}$  all possible spans  $S = \{s_1,\ldots ,s_N\}$  are computed, which are within-sentence word sequences.
The model deals with  $\mathcal{O}(n^4)$  possible combinations of spans, where  $n$  is the number of words in a sentence. Therefore, pruning is required to make the classification of span-pairs into relation labels tractable at both training and test time (Lee et al., 2017; He et al., 2018). First, a score  $\phi_{mr}$  of how likely a span is mentioned in a relation is computed. These mention scores enable beam pruning the number of spans considered for relation classification with a variable beam of size  $\lambda_{n}$ , where  $n$  is the number of tokens in the input sentence (Luan et al., 2018a). Second, the maximum width  $W$  of spans is limited to reduce the total number of spans. We set  $\lambda$  to .8 and  $W$  to 14 tokens, the maximum span length in FOBIE.
After pruning, a label  $e_i \in L_E$  is predicted for the remaining spans  $s_i$ . Here  $L_E$  is the set of possible span labels, including a non-span class  $\epsilon$ . For pairs of spans  $(s_i, s_j)$  the model predicts which relation  $r_{ij} \in L_R$  holds between them. The set of possible relation types is  $L_R$ , which includes a non-relation class  $\epsilon$ . The output consists of labeled spans and relation labels for pairs of spans. For a detailed description of the SciIE system we refer to Luan et al. (2018a).
TOindicator Arg0 Spans
These observations indicate a clear correlation between the presence of resinin and the need for energy storage in jumping.
Pred &
Gold:
There is an
Pred:
Duplication of the A3Z3 gene seems thus to be unfavoured, possibly reflecting the trade-off between
the advantages of an increased antiviral repertoire and the increased self-toxicity by gene dosage effect [18].
However, phenotypic traits that improve either locomotor economy or fighting ability may conflict with performance in the other (Carrier, 2002; Oufiero and Garland, 2007), resulting in a functional trade-off (Lewontin, 1978;
Figure 4: Examples of output from the SCIE system trained on FOBIE, in comparison to gold annotation.
Agu 10
Wg1
However, phenotypic traits that improve either locomotor economy or fighting ability may
(Carrier, 2002; Oufiero and Garland, 2007), resulting in a functional trade-off (Lewontin, 1978;
with performance in the other
4.3 Narrow IE results
We evaluate SCIE on two sub-tasks: (1) Argument Recognition, and (2) Relation Extraction. Table 3 summarizes the results on the sub-tasks of Argument Recognition and RE. With regards to the first sub-task, we train two SCIE models. One model only predicts whether a span is a valid span or not, while a second model predicts whether the span is a trigger word or a key-phrase. For the first sub-task we also report the results of the RBS described in Section 3.1. The RBS performs significantly worse; it identifies trigger words exceptionally well  $(\mathrm{F}1 = 95.89$  on test set) but does not correctly recognize many of the remaining key-phrases  $(\mathrm{F}1 = 22.36$  on test set), resulting in a low overall performance.
Figure 4 shows example outputs of the narrow RE model. The predicted relation (NOT-A-TRADE-OFF) and its accompanying structure for the first example are completely correct. Note how the argument modifiers result in a non-projective structure. The second example is more challenging, with a longer range dependency between the trade-off span and the second dependent argument. Our model predicts the correct relation, TRADE-OFF, but only extracts partial argument spans and essentially fragments them into several modifying
argument relations. The third example exhibits a relatively long argument – which is common in scientific literature – where only a small part of the span is predicted.
4.4 Supporting trade-off annotation