SlowGuess/NAACL_2021_OCR huggingface dataset:
Tom Hope\*, Aida Amini \* David Wadden
Madeleine van Zuylen\* Sravanthi Parasa\* Eric Horvitz
Daniel Weld\*, Roy Schwartz Hannaneh Hajishirzi\*,
$\clubsuit$  Paul G. Allen School for Computer Science & Engineering, University of Washington
$\clubsuit$  Allen Institute for Artificial Intelligence  $\checkmark$  Swedish Medical Group
Microsoft Research The Hebrew University of Jerusalem, Israel
{tomh, aidaa, danw, hannah}@allenai.org
The COVID-19 pandemic has spawned a diverse body of scientific literature that is challenging to navigate, stimulating interest in automated tools to help find useful knowledge. We pursue the construction of a knowledge base (KB) of mechanisms—a fundamental concept across the sciences, which encompasses activities, functions and causal relations, ranging from cellular processes to economic impacts. We extract this information from the natural language of scientific papers by developing a broad, unified schema that strikes a balance between relevance and breadth. We annotate a dataset of mechanisms with our schema and train a model to extract mechanism relations from papers. Our experiments demonstrate the utility of our KB in supporting interdisciplinary scientific search over COVID-19 literature, outperforming the prominent PubMed search in a study with clinical experts. Our search engine, dataset and code are publicly available. $^{1}$
"Some experts are familiar with one field, such as AI or nanotechnology [...] no one is capable of connecting the dots and seeing how breakthroughs in AI might impact nanotechnology, or vice versa."
-Yuval Noah Harari, Homo Deus, 2016
The effort to mitigate the COVID-19 pandemic is an interdisciplinary endeavor the world has rarely seen (Apuzzo and Kirkpatrick, 2020). As one recent example, expertise in virology, physics, epidemiology and engineering enabled a group of 200 scientists to understand and bring attention to the
Figure 1: Our COVID-19 Mechanism KB (COMB) is extracted from scientific papers and can be searched for diverse activities, functions and influences (1), retrieving relations from the literature (2).
airborne transmissibility of the SARS-CoV-2 virus (Morawska et al., 2020). The diverse and rapidly expanding body of past and present findings related to COVID-19 (Wang et al., 2020b) makes it challenging to keep up, hindering scientists' pace in making new discoveries and connections.
Research in natural language processing (NLP) has provided important resources to extract fine-grained relations from scientific papers in specific areas, such as certain subfields of biomedicine (Kim et al., 2013; Nye et al., 2018) or computer science (Wadden et al., 2019). However, these cover only a fraction of all concepts in the literature; in biomedicine alone, there are myriad concepts (Salvadores et al., 2013) not covered by NLP resources. For COVID-19 research, the challenge is especially pronounced due to diversity and emerging concepts; even reading just one paper may require background knowledge in multiple biomedical subfields, physics, chemistry, engineering, computer science and the social sciences. For example,
<sup>1</sup>https://covidmechanisms.apps.allenai.org/
Association for Computational Linguistics: Human Language Technologies, pages 4489-4503
consider a paper studying the indoor dynamics of aerosolized SARS-CoV-2 and the effect of ventilation on transmission by using simulation models, or work on economic impacts of COVID-19 on prices and consumption.
To make progress in consolidating such diverse information, we introduce a unified schema of mechanisms as a unified language covering activities, functions and influences across the sciences. These can be proteins that block viral binding, algorithms to design drugs, the effect heat has on viruses, or COVID-19 has on food prices (Fig. 1).
We build on the fact that mechanisms underlie much of the natural language of scientific papers (Röhl, 2012), and construct a unified schema with two coarse-grained mechanism relations:
- Direct Mechanisms: mechanistic activities (e.g., viral binding) or functions engendered by natural or artificial entities (e.g., a protein used for binding or algorithm used for diagnosis).
- Indirect Mechanisms: influences and associations such as economic effects of COVID-19 or complications associated with medical procedures.
Our coarse-grained relation schema, over freeform text spans, strikes a balance between the granular information extracted by Closed-IE approaches (Freitag, 1998; Hoffmann et al., 2010) and the schema-free breadth of Open IE approaches (Etzioni et al., 2008; Stanovsky et al., 2018), which often lead to generic and uninformative relations for scientific applications (Kruiper et al., 2020).
Furthermore, our schema facilitates construction of a high-quality KB that synthesizes interdisciplinary knowledge. We construct precisely this, releasing MECHANIC (Mechanisms ANotated in COVID-19 papers) – an annotated dataset of 2,400 mechanisms based on our schema. We train a state-of-the-art model to extract this information from scientific papers, and use it to build COMB (COVID-19 Open Mechanism Knowledge Base) – a broad-coverage KB of 1.5M mechanisms in COVID-19 papers. We analyze the characteristics of COMB, showing the distribution of relations across scientific subfields and comparing their quality to other IE approaches.
We demonstrate the utility of COMB in two studies with experts. In the first study, our system achieves high precision and recall in scientific search with structured queries on both diverse viral mechanisms and applications of AI in the literature. In the second study, we evaluate COMB
in a usability study with MDs active in treating and researching COVID-19. Our system is rated higher than PubMed search by the clinical experts, in terms of utility and quality.
- We introduce a unified schema for mechanisms that generalizes across many types of activities, functions and influences. We construct and distribute MECHANIC, an annotated dataset of papers related to COVID-19, with 2,400 instances of our mechanism relation.
- Using MECHANIC, we train an IE model and apply it to 160K abstracts in COVID-19 literature, constructing COMB, a KB of 1.5M mechanism instances. Manual evaluation of relations sampled from our KB shows them to have  $88\%$  accuracy. We also find a model trained on our data reaches roughly  $80\%$  accuracy on a sample of general biomedical papers from across the PubMed corpus, with no additional training, demonstrating the generalization of our approach.
- We showcase the utility of COMB in structured search for mechanisms in the literature. In a study with MDs working to combat COVID-19, our system is rated higher than PubMed search in terms of utility and quality.
Mechanisms in science The concept of mechanisms, also referred to as functional relations, is fundamental across the sciences. For example mechanisms are described in biomedical ontologies (Burek et al., 2006; Röhl, 2012; Keeling et al., 2019), engineering (Hirtz et al., 2002), and across science. Mechanisms can be natural (e.g., the mechanism by which amylase in saliva breaks down starch into sugar), artificial (electronic devices), non-physical constructs (algorithms, economic policies), and very often a blend (a pacemaker regulating the beating of a heart through electricity and AI algorithms).
Although seemingly intuitive, exact definitions of mechanisms are subject to debate in the philosophy of science (Röhl, 2012; Keeling et al., 2019). An Oxford dictionary definition of mechanisms refers to a natural or established process by which something takes place or is brought about. More intricate definitions discuss "complex systems producing a behavior", "entities and activities productive of regular changes", "a structure performing a function in virtue of its parts and operations", or the
Schema	Entity types	Relations	Example
SciERC	CS methods/tasks (free-form spans)	used-for	Use GNNs for relation extraction.
SemRep	Clinical (drugs, diseases, anatomy...)	causes, affects, treats, inhibits, interacts, used...	...intratympanic dexamethasone injections for patients with intractable Meniere's disease.
ChemProt	Chemicals, proteins	direct/indirect regulator, inhibitor, activator ...	Captopril inhibited MMP-9 expressions in right ventricles.
DDI	Drugs	interacts	Quinolones may enhance the effect of Warfarin.
GENIA	Proteins, cellular entities	binding, modification, regulation ...	BMP-6 induced phosphorylation of Smad1/5/8.
PICO	Clinical	Interventions, outcomes	The bestatin group achieved longer remission.
Ours: MECHANIC	Medicine, epidemiology, genetics, molecular bio., CS, math, ecology, economics ... (free-form)	direct (activities, functions) / indirect (influences, associations)	·RL can be used to learn mitigation policies in epidemiological models.
·Histophilus-somni causes respiratory, reproductive, cardiac and neuronal diseases in cattle.
Table 1: Our broad concept of mechanisms covers many relations within existing science-IE schemas. The table shows examples of representative schemas, and the types of entities and relations they capture.
distinction between "correlative property changes" and "activity determining how a correlative change is achieved" (Rohl, 2012).
Abstract definitions can help with generalization across many important types of mechanisms. The schema we propose (Sec. 3) is inspired by such definitions, operationalizing them and making them more concrete, and also simple enough for models and human annotators to identify.
Information extraction from scientific texts There is a large body of literature on extracting information from scientific papers, primarily in the biomedical sphere. This information often corresponds to very specific types of mechanisms, as shown in Tab. 1. Examples include ChemProt (Li et al., 2016) with mechanisms of chemical-protein regulation, drug interactions in the DDI dataset (Segura Bedmar et al., 2013), genetic and cellular activities/functions in GENIA (Kim et al., 2013), semantic roles of clinical entities (Kilicoglu et al., 2011), PICO interventions and outcomes (Wallace et al., 2016; Nye et al., 2018), and computer science methods/tasks in SciERC (Luan et al., 2018). Such schemas have been used, for example, to extract genomic KBs (Poon et al., 2014) and automate systematic reviews (Nye et al., 2020). Our schema draws on these approaches, but with a much broader reach across concepts seen in COVID-19 papers (Tab. 1, Fig. 2).
An important area in information extraction focuses on open concepts, with prominent approaches being Open IE (Etzioni et al., 2008) and Semantic Role Labeling (SRL; Carreras and Marquez,
2005), which share similar properties and predictions (Stanovsky et al., 2018). While such methods are intended to be domain independent, they perform significantly worse in the scientific domain (Groth et al., 2018). Kruiper et al. (2020) developed a multi-stage process to post-process Open IE outputs, involving trained models and humans to find a balance between generic and fine-grained clusters of relation arguments and omitting noisy clusters. In contrast, our unified schema enables annotating a dataset of mechanism relations between free-form spans and training IE models to automatically generalize across diverse relation types.
Our schema is also related broadly to the task of training reading comprehension models on procedural texts describing scientific processes (such as short paragraphs written by crowd workers to explain photosynthesis in simple language; Dalvi et al., 2018). Our representation of scientific texts in terms of a graph of causal relations can potentially help infer processes across science.
COVID-19 IE Recent work (Verspoor et al., 2020a) has focused on extracting information from the CORD-19 corpus (Wang et al., 2020b). PICO concepts are extracted and visualized in an exploratory interface in the COVID-SEE system (Verspoor et al., 2020b). In Wang et al. (2020a), genes, diseases, chemicals and organisms are extracted and linked to existing biomedical KBs with information such as gene-disease relations. Additional relations based on the GENIA schema are extracted from the text. To address the novel COVID-19 domain, the schema is enriched with new entity types
such as viral proteins and immune responses.
In this paper, we focus on a more general schema that captures diverse concepts appearing in literature related to COVID-19, an emerging domain with novel concepts coming from many fields and subfields. The mechanism KG we construct includes—as a subset—diverse biomolecular and clinical information (such as chemical-disease relations) as part of a general mechanism schema.
3 Mechanism Relation Schema
We present a schema that builds upon and consolidates many of the types of mechanisms discussed in Sec. 2. Our defined schema has three key properties: (1) it uses a generalized concept of mechanism relations, capturing specific types of mechanisms in existing schema and extending them broadly; (2) it includes flexible, generic entities not limited to predefined types, and (3) it is simple enough for human annotators and models to identify in the natural language of scientific texts. This schema enables forming our KB by identifying a set of mechanism relations in a corpus of scientific documents (Sec. 4.3).
We formally define each mechanism as a relation  $(E_1, E_2, \text{class})$  between entities  $E_1$  and  $E_2$ , where each entity  $E$  is a text span and the class indicates the type of the mechanism relation. Entities all share a single common type and can be either natural (e.g., protein functions, viral mechanistic activities) or artificial (e.g., algorithms, devices), to capture the generality of the concepts in science (see Fig. 2). We allow each entity to take part in multiple relations (tuples) within a given text, leading to a "mechanism graph". Mechanisms are categorized into two coarse-grained classes:
Direct mechanisms include activities of a mechanistic nature – actions explicitly performed by an entity, such as descriptions of a virus binding to a cell, and explicit references to a function (e.g., a use of a drug for treatment, or the use of AI for drug design as in Fig. 1).
Indirect mechanisms include influences or associations without explicit mechanistic information or mention of a function (such as describing observed effects, without the process involved). These relations correspond more to "input-output cor
Figure 2: MECHANIC covers a diverse set of scientific fields. Histogram of domains in MECHANIC (sample of 350 relations). Manually labeled relation entities, based on a list of scientific disciplines from Wikipedia.
relations” (Röhl, 2012), such as indicating that COVID-19 may lead to economic impacts but not how (Fig. 1), as opposed to direct mechanisms describing “inner workings” – revealing more of the intermediate states that lead from initial conditions (COVID-19) to final states (price inflation) or explicitly describing a function. As an example for the utility of this distinction between direct and indirect relations, consider an MD looking to generate a structured list of all uses of a treatment (direct mechanism), but not include side effects or complications (indirect).
4 KB Construction
We describe our approach (depicted in Fig. 3) for extracting a knowledge base of mechanisms using our unified schema. We first curate MECHANIC, an annotated dataset of general mechanisms from a small collection of scientific papers (Sec. 4.1). We then train a model on our annotated data to extract mechanism relations from the entire CORD-19 corpus of scientific papers; we use it to build COMB, a knowledge base of mechanisms across the entire CORD-19 corpus of (Sec. 4.2), which supports semantic search for relations (Sec. 4.3).
4.1 Collecting Mechanism Annotations
We construct a dataset of mechanism relations in texts randomly sampled from the CORD-19 corpus (Wang et al., 2020b) that includes scientific papers connected to COVID-19. To circumvent annotation challenges in scientific datasets (Luan et al., 2018) and ensure high-quality annotations, we follow a three-stage process of (1) annotating entities and relations using biomedical experts, (2) unifying span boundaries with an NLP expert, and (3) verifying annotations with a bio-NLP expert. Our annotation process is a relatively low-resource and generalizable approach for a rapid response to the
2We also provide a dataset and extraction model for ternary relations in the form of (subject, object, predicate). We focus on the coarse-grained mechanism schema due its broader flexibility and coverage. See App. A.1 for details.
Figure 3: Overview of our approach. We collect annotations of mechanisms (textual relations) from the CORD-19 corpus, which are used to train an IE model. We apply the model to over 160K documents in the corpus, extracting over 1.5M relations that are fed into our KB. Entity mention spans are embedded with a language model tuned for semantic similarity, and indexed with FAISS for fast similarity search as part of our search interface.
COVID-19 emergency.
In the first stage, five annotators with biomedical and engineering background annotate all mechanism relations as defined in Sec. 3 (full annotation guidelines are available in our code repository). Relations are annotated as either direct/indirect. Entities are annotated as the longest span of text that is involved in a relation with another entity, while not including redundant or irrelevant tokens. As in related tasks (Luan et al., 2018), annotators are guided to resolve doubt on span boundaries by selecting the longest relevant span.
Annotators had a one-hour training session. In the first part of the training session, annotation guidelines were reviewed. The guidelines included simple explanations of direct/indirect mechanisms along with introductory examples (e.g., "the virus makes use of spike protein to bind to a cell", "A virus leads to respiratory infection"). In the second part, annotators saw examples from papers in the annotation interface (see Fig. 6, App. A), and performed a few live training annotations.
We initially observed significant variation between annotators in identifying span boundaries for entity annotations, stemming from inherent subjectivity in such annotation tasks (Stanovsky et al., 2018; Luan et al., 2018) and from lack of NLP experience by some annotators. In the second stage, an NLP expert annotator conducted a round of style unification by viewing annotations and adjusting span boundaries to be more cohesive while preserving the original meaning, focusing on boundaries that capture essential but not redundant or generic information (e.g., adjusting the span substantial virus replication by unknown mechanisms to include only virus replication). Finally, in the third stage, a bio-NLP expert with experience in annotating scientific papers verified the annotations and corrected them as needed. The expert accepted  $81\%$
of the annotations from the second stage without modification, confirming the high quality of the stage-2 data. Relation label mismatches accounted for  $5\%$  of the remaining  $19\%$ . Other sources of disagreement were span mismatches and new relations added by the bio-NLP expert adjudicator.
The resulting dataset (MECHANIC: Mechanisms ANotated in COVID-19 papers) contains 2,370 relation instances (1645 direct, 725 indirect) appearing in 1,000 sentences from 250 abstracts. Average span length is 4 tokens, while the average distance between relation arguments is 11.40 tokens.
4.2 Extracting a KB of Mechanisms
Using MECHANIC, we train an IE model to extract mechanism relations from sentences in scientific documents. We train DyGIE++ (Wadden et al., 2019), a state-of-the-art end-to-end IE model which extracts entities and relations jointly (without assuming to have entity spans given), classifying each relation as one of {DIRECT, INDIRECT}.<sup>4</sup>
To form our corpus-level KB, we apply the trained model to each document in our corpus (all 160K abstracts in the CORD-19 corpus) to extract mechanism relations and then integrate the extracted relations. We find that our trained model achieves high precision scores for high confidence predictions (precision  $\geq 80\%$  within top-20 predicted relations; see  $P@K$  figure, App. B). Therefore, our corpus-level KB is constructed by filtering predictions with low confidence.
3The dataset is similar in size to related scientific IE datasets (Luan et al., 2018) which share related challenges in collecting expert annotations of complex or ambiguous concepts over difficult texts.
<sup>4</sup>We use DyGIE++ with SciBERT (Beltagy et al., 2019) embeddings fine-tuned on our task and perform hyperparameter grid search (for dropout and learning rate only) and select the best-performing model on the development set (7e-4 and 0.43, respectively). Full details are in App. B.3.
To integrate relations and entities across the corpus, we use standard surface-level string normalization (such as removing punctuation, lemmatizing, and lowercaseing) and unify and normalize entity mentions using coreference clusters of entities within a document. Each coreference cluster is assigned a representative entity as the mention with the longest span of text, and all other entities in that cluster are replaced with the representative entity. This is particularly useful for normalizing pronouns such as it with the original mention they referred to (e.g., a specific virus or method it refers to).
Our final KB (COMB) consists of 1.5M relations in the form of  $(E_1, E_2, \text{DIRECT/INDIRECT})$  filtered by high confidence score ( $>= 90\%$ ), where entities  $E_i$  are standardized free-form spans of text.
4.3 Semantic Relation Search
The constructed KB enables applications for retrieving relations across concepts from many disciplines. For example, searching for all documents that include mechanisms to incorporate  $AI$  in studies of heart disease ( $E_1 = AI, E_2 =$  heart disease, DIRECT) requires going beyond simply finding documents that mention  $AI$  and heart disease. Here, we describe our approach for searching over the KB by encoding entities and relations, capturing related concepts (such as cardiac disease and heart conditions), as well as simpler surface matches (artificial intelligence methods, artificial intelligence models).
Specifically, for a given query  $\mathbf{q} := (E_1^q, E_2^q, \text{class})$ , our goal is to find mechanisms  $r_i$  in COMB whose entities are free-form texts similar to  $E_1^q, E_2^q$  in the query. The class is used to filter for the type of relation—for example, when explicitly requiring DIRECT mechanisms.
Entity encoding We obtain an encoding function  $f: E \mapsto \mathbb{R}^d$  to encode all unique spans (entities) in the KB to a  $d$  dimensional vector space. The encoding function is derived by fine-tuning a language model (LM) originally trained on PubMed papers (Gururangan et al., 2020) on semantic similarity tasks. For fine-tuning, we use sentence pairs in STS (Cer et al., 2017) and SNLI (Bowman et al., 2015) following Reimers and Gurevych (2019), and add biomedical sentence pairs from the BIOSSES dataset (Sogancioglu et al., 2017).
Relation similarity Given a query  $\mathbf{q}$ , we rank the set of all COMB relations with the same class as the query. For each candidate relation  $r = (E_1, E_2, \text{class})$  in COMB, we compute its similarity to the query relation  $\mathbf{q}$  as the minimum similarity between encodings of their corresponding entities:  $\min_{j \in \{1, 2\}} f(E_j) \cdot f(E_j^q)$ . With this definition, a relation  $(E_1, E_2)$  with  $E_1$  very similar to the first entity of the query  $E_1^q$  but  $E_2$  distant from  $E_2^q$  will be ranked low. For example, with the query  $(E_1^q = \text{deep learning}, E_2^q = \text{drugs})$ , the relation  $(E_1 = \text{microscope}, E_2 = \text{drugs})$  will be ranked low due to the pair (deep learning, microscope). For efficient search, we create an index of embeddings corresponding to the 900K unique surface forms in COMB and employ a system designed for fast similarity-based search (Johnson et al., 2017).
5 Evaluating COMB