SlowGuess/NAACL_2021_OCR huggingface dataset:
In this section, we evaluate the constructed KB of mechanisms in terms of correctness and informativeness (Sec. 5.1), and its utility in searching for mechanisms (Sec. 5.2). Our main goal is to ensure the mechanism relations have high quality to support our large-scale KB and search applications. We further show that our schema is useful as compared to other schema.
5.1 KB Correctness and Informativeness
We employ two annotators with biomedical and CS backgrounds to judge the quality of the predicted relations in COMB. In particular, following Groth et al. (2018), annotators are given a predicted relation together with the sentence from which it was extracted. We collapse all entities/relations into one generic type for this analysis. Annotators are asked to label the predicted relation as correct if (1) it accurately reflects a mechanistic relation mentioned in the sentence (correctness), and (2) the extracted entities and relation label are sufficient to convey the meaning of the relation, without referring to the source sentence (informativeness). We collect human judgements for 300 predicted relations for our approach and baselines, sampled from 150 randomly selected sentences. Agreement is  $71\%$  by Cohen's Kappa and  $73\%$  by Matthew's Correlation Coefficient.
Comparing KB quality to other schemas To showcase the benefit of our approach, we compare the relations extracted using a DyGIE model trained on MECHANIC, versus a DyGIE model
<sup>5</sup>We use a pre-trained DyGIE++ model trained on SciERC to obtain coreference clusters.
Figure 4: Evaluating COMB in studies with experts. COMB is found to have high quality and utility, outperforming other approaches. Left: COMB outperforms external resources with either specific types of mechanisms or open relations, in human evaluation for correctness and usefulness of predictions, on a sample of 300 predicted relations. Center: Retrieved relations are ranked by query similarity (Sec. 4.3) and compared to human relevance labels to compute precision/recall. Higher-ranked results are overall judged as relevant by humans. Right: Results of COMB search study with five practicing MDs, using both our system and PubMed to search the literature. Experts were given a post-study questionnaire with questions grouped by subject (search, utility, interface). Our mechanism search system performed substantially better than PubMed.
Metric	PubMed	COMB
Search	71%	90%
Utility	69.5%	92%
Interface	78%	90%
Overall	74%	91%
trained on other resources that are most related to our mechanisms: SemRep (Kilicoglu et al., 2011) captures a wide range of biomedical relations (such as drug-drug interactions), and SciERC (Luan et al., 2018) contains relations relevant to computer science (such as "method-task" and "used-for" relations). In addition, we compare with a Semantic Role Labeling (SRL) method (Shi and Lin, 2019) that captures broad relations between free-form spans that focus on agents and actions, and a neural OpenIE model (Stanovsky et al., 2018).
Fig. 4 (left) shows that  $88\%$  of relations from COMB are marked as correct by human raters, demonstrating that our approach extracts mechanism relations with better quality than external resources. These results suggest that our predicted relations are of overall high quality and can be used to build our corpus-level KB and explore its utility.
Examining Generalization COVID-19 papers are highly diverse both topically and chronologically. We conduct a small-scale preliminary experiment examining whether a model trained on MECHANIC can generalize to capture mechanism relations in the general biomedical papers, from a much larger corpus of open access papers on PubMed Cen
tral (PMC). We randomly sample a set of 200 predicted relations from papers across the entire PMC corpus, and label them using the same criteria used above. As expected, we find that performance drops, but encouragingly is still considerably high: after filtering predictions with confidence lower than  $90\%$  in the same way we construct COMB,  $76\%$  of relations are considered correct. When filtering for confidence with a threshold of  $95\%$  (which captures  $70\%$  of the samples), the rate of correct predictions is  $78\%$ . In future work it would be interesting to fine-tune our model on a small set of labeled examples from the general PMC corpus to potentially improve these results.
5.2 COMB Utility
We design several search tasks and user studies to evaluate the utility of the constructed KB (Sec. 5.2.1) and compare it with the PubMed medical KB and search engine (Sec. 5.2.2), as judged by medical doctors working on the front lines of COVID-19 treatment and research. All tasks are designed to evaluate our framework's utility in helping researchers and clinicians looking to quickly search for mechanisms or cause-effect relations in the literature and retrieve a list of structured results.
5.2.1 Search Quality
We form search queries based on a wide range of topics pertaining to (1) SARS-CoV-2 mechanisms (such as modes of transmission, drug effects, climatic influences, molecular-level properties) and
<sup>6</sup>We use an expert annotator to align external resources to our direct or indirect mechanism annotations, e.g., USED-FOR is mapped to direct mechanism).
<sup>7</sup>We also experiment with automated evaluation. We split MECHANIC into train/dev/test sets (170/30/50 abstracts), and obtain  $F1 = 50.2$  for entity detection,  $F1 = 45.6$  for relation detection and  $F1 = 42.8$  for classification, on par with performance in other similar scientific IE tasks (Luan et al., 2018). See more details in App. B.4.
<sup>8</sup>https://www.ncbi.nlm.nih.gov/pmc/
Relation query	Example results from KB search interface
E1	Warm climate	Experimental data showed that coronavirus survival was negatively impacted by ozone and high temperature.
E2	Coronavirus
E1	COVID-19	The typical features of COVID-19 in chest CT include bilateral, peripheral, and multifocal ground-glass opacities with or without superimposed consolidations.
E2	Bilateral ground glass opacities
E1	Aerosols	SARS-CoV-2 is transmitted efficiently via the air (via respiratory droplets and/or aerosols) between ferrets.
E2	SARS-CoV-2 transmission
(a) Viral mechanism search. Queries for  $(E1,E2)$  relations, and example retrieved results.
E1Convolutional neural networks	3D patch - based convolutional neural networks were trained to predict conductivity maps from B1 transceive phase data.
E2
E1Computer vision	We present a computational pipeline using algorithms from computer vision to decompose ciliary motion into quantitative elemental components.
E2
E1Graph neural networks	We propose a new model called GraphDTA that represents drugs as graphs and uses graph neural networks to predict drug - target affinity.
E2
(b) AI search. Queries consists of only  $E_{1}$ , to find all applications of AI approaches/areas.
Table 2: Example search queries and results for the viral mechanism and AI applications tasks.
(2) applications of AI in this area. Tab. 2a and 2b show queries and example relations returned from COMB, along with the context sentences from which they were extracted.
Viral mechanism search Queries are formed based on statements in recent scientific claim-verification work (Wadden et al., 2020; see full list in App. C.2). For example, for the statement the coronavirus cannot thrive in warmer climates, we form the query as  $(E_{1} = \text{Warm climate}, E_{2} = \text{coronavirus})$  (see Tab. 2a row 1). For statements reflecting an indirect association/influence, we filter for INDIRECT relations (Tab. 2a row 2). For statements that reflect an undirected mechanism relation (e.g., Lymphopenia is associated with severe COVID-19 disease), we query for both directions.
AI applications search This task is designed to explore the uses of AI in COVID-19 papers (Tab. 2b). We use queries where the first entity  $E_{1}$  is a leading subfield or method within AI (e.g., deep reinforcement learning or text analysis), and the second entity  $E_{2}$  is left unspecified. Since all queries relate to uses of AI, we filter for DIRECT relations. These open-ended queries simulate an exploratory search scenario, and can potentially surface inspirations for new applications of AI against COVID-19 or help users discover where AI is being harnessed.
Evaluation Expert annotators are instructed to judge if a relation is related to the query or not and
if the sentence actually expresses the mechanism. These annotations are used as ground-truth labels to compute precision/recall scores of the relations extracted by our algorithm. Since it is not feasible to label every relation, annotators are shown a list of 20 relations for each query including high and low rank relations returned by our search algorithm.<sup>9</sup> In total, we use 5 annotators to obtain 1,700 relevance labels across both tasks. Inter-annotator agreement is high by several metrics, ranging from 0.7–0.8 depending on the metric and task; see App. C.2. Annotators have graduate/PhD-level background in medicine or biology (for the first task) and CS or biology (for the second task).
Results Fig. 4 (center) shows our results for both tasks. For biomedical search queries, we observe  $90\%$  precision that remains stable for recall values as high as  $70\%$ . For AI applications we observe a precision of  $85\%$  at a recall of  $40\%$  that drops more quickly. This lower precision is likely due to the fact that  $E_{2}$  is unspecified, leading to a wider range of results with more variable quality.
Overall, these results showcase the effectiveness of our approach in searching for mechanisms be
${}^{9}$  Specifically,for each query we retrieve the top-1000 similar relations from COMB,ranked as described in Sec. 4,and select the top and bottom 10 relations (20 per query,  ${200}\left( { = {20}\mathrm{x}{10}}\right)$  per task,  ${400}\left( { = {200}\mathrm{x}2}\right)$  in total),shuffle their order,and present to annotators together with the original sentence from which each relation was extracted.
tween diverse concepts in COVID-19 papers.
5.2.2 Comparing COMB with PubMed
This experiment compares the utility of COMB in structured search for causal relationships of clinical relevance to COVID-19 with PubMed<sup>10</sup>—a prominent search engine for biomedical literature that clinicians and researchers frequently peruse as their go-to tool. PubMed allows users to control structure (e.g., with MeSH terms or pharmacological actions), is supported by a KB of biomedical entities used for automatic query expansion, and has many other functions.
Expert evaluation We recruit five expert MDs—with a wide range of specialities including gastroenterology, cardiology, pulmonary and critical care—who are active in treating COVID-19 patients and in research. Each expert completed search randomly ordered tasks using both PubMed and our COMB UI, showing the full set of ranked relations, as well as the sentence snippet mentioning the relation, the paper title, and hyperlink to abstract. At the end of the study after all search tasks are completed for both our system and PubMed, experts are given a questionnaire of 21 7-point Likert-scale questions to judge system utility, interface, and search quality. The first 16 questions are taken from a Post Study System Usability Questionnaire (PSSUQ; Lewis, 2002) widely used in system quality research. The last 5 questions are designed by the authors to evaluate search quality such as overall result relevance and ranking (for the full question list, see App. C.2). Each question is asked twice, once for PubMed and once for our system, leading to  $21 \times 2 \times 5 = 210$  responses.
Search queries We provide experts with seven search queries that were created by an expert medical researcher, relating to causal links (e.g., between COVID-19 and cardiac arrhythmias) and functions (e.g., Ivermectin as a treatment). See full set of queries in App. C.
Results Fig. 4 (right) shows the average Likert scores (normalized to  $[0\%, 100\%]$ ) across all questions and users for COMB and PubMed. The results show that the medical experts strongly prefer COMB to PubMed (overall average of  $91\%$  vs.  $74\%$ , with non-normalized scores of 6.6 vs. 5.2). On average across the 21 questions, the majority of the five experts assigned our interface a higher score than PubMed, at an average rate of 3.5/5. This rate in
creases further when considering ties-on average 4.75/5 of the experts assigned our system a score equal or higher than PubMed.
Overall, our system significantly outperforms PubMed in this task, with an average gap of roughly  $20\%$  for search and utility-related questions (Wilcoxon signed rank test p-value is significant at  $4.77 \times 10^{-7}$ ). These results are particularly interesting and indicate the potential of COMB because of the experts' strong familiarity with PubMed and the simple nature of our UI.
Our system searches and retrieves relations—only texts explicitly mentioning relations that match the input query. This often more precisely reflects the query than results returned by PubMed, which do not have the additional layer of structured information in COMB. For example, for the query  $(E_{1} =$  cardiac arrhythmias,  $E_{2} =$  COVID-19), PubMed returns the following title of one paper: Guidance for cardiac electrophysiology during the COVID-19 pandemic [...] Electrocardiography and Arrhythmias Committee— $E_{1}$  and  $E_{2}$  are both mentioned, but not within a mechanism relation.
We introduced a unified schema for mechanisms that generalizes across many types of activities, functions and influences. We constructed and distributed MECHANIC, a dataset of papers related to COVID-19 annotated with this schema. We trained an IE model and applied it to COVID-19 literature, constructing COMB, a KB of 1.5M mechanisms. We showcased the utility of COMB in structured search for mechanism relations in COVID-19 literature. In a study with MDs active in the fight against the disease, our system is rated higher than PubMed search for both utility and quality. Our unified view of mechanisms can help generalize and scale the study of COVID-19 and related areas. More broadly, we envision a KB of mechanisms that enables the transfer of ideas across the literature (Hope et al., 2017), such as by finding relationships between mechanisms in SARS-CoV-2 and other viruses, and assists in literature-based discovery (Swanson and Smalheiser, 1996) by finding cross-document causal links.
Our knowledge-base and search system is primarily intended to be used by biomedical researchers working on COVID-19, and researchers from more
10 https://pubmed.ncbi.nlm.nih.gov/
general areas across science. Models trained and developed on our dataset are likely to serve researchers working on COVID-19 information extraction, and scientific NLP more broadly. We hope our system will be helpful for accelerating the pace of scientific discovery, in the race against COVID-19 and beyond.
Our knowledge-base can include incorrect information to the extent that scientific papers can have wrong information. Our KB includes metadata on the original paper from which the information was extracted, such as journal/venue and URL. Our KB can also miss information included in some papers.
Our data collection process respected intellectual property, using abstracts from CORD-19 (Wang et al., 2020b), an open collection of COVID-19 papers. Our knowledge-base fully attributes all information to the original papers. All annotators were given extensive background on our objectives, and told their annotations will help build and evaluate a knowledge-base and search engine over COVID-19 research. Graduate-student annotators were paid 25 USD per hour. MD experts helped evaluate the tool on a voluntary basis.
We like to acknowledge a grant from ONR N00014-18-1-2826. Authors would also like to thank anonymous reviewers, members of AI2, UW-NLP and the H2Lab at The University of Washington for their valuable feedback and comments.
Matt Apuzzo and David D. Kirkpatrick. 2020. Covid-19 changed how the world does science, together. New York Times.
Iz Beltagy, Kyle Lo, and Arman Cohan. 2019. Scibert: A pretrained language model for scientific text. In EMNLP.
Olivier Bodenreider. 2004. The unified medical language system (umlts): integrating biomedical terminology. *Nucleic acids research*.
Samuel R Bowman, Gabor Angeli, Christopher Potts, and Christopher D Manning. 2015. A large annotated corpus for learning natural language inference. In EMNLP.
Patryk Burek, Robert Hoehndorf, Frank Loebe, Johann Visagie, Heinrich Herre, and Janet Kelso. 2006. A top-level ontology of functions and its application in the open biomedical ontologies. Bioinformatics.
Xavier Carreras and Lluis Márquez. 2005. Introduction to the conll-2005 shared task: Semantic role labeling. In CoNLL).
Daniel Cer, Mona Diab, Eneko Agirre, Inigo Lopez-Gazpio, and Lucia Specia. 2017. Semeval-2017 task 1: Semantic textual similarity multilingual and crosslingual focused evaluation. In SemEval-2017.
Bhavana Dalvi, Lifu Huang, Niket Tandon, Wen-tau Yih, and Peter Clark. 2018. Tracking state changes in procedural text: a challenge dataset and models for process paragraph comprehension. In *NAACL*.
Jesse Dodge, Suchin Gururangan, Dallas Card, Roy Schwartz, and Noah A. Smith. 2019. Show your work: Improved reporting of experimental results. In EMNLP.
Oren Etzioni, Michele Banko, Stephen Soderland, and Daniel S Weld. 2008. Open information extraction from the web. Communications of the ACM.
D. Freitag. 1998. Toward general-purpose learning for information extraction. In COLING-ACL.
Paul Groth, Mike Lauruhn, Antony Scerri, and Ron Daniel Jr. 2018. Open information extraction on scientific text: An evaluation. In CoNLL.
Suchin Gururangan, Ana Marasovic, Swabha Swayamdipta, Kyle Lo, Iz Beltagy, Doug Downey, and Noah A Smith. 2020. Don't stop pretraining: Adapt language models to domains and tasks. In ACL.
Julie Hirtz, Robert B Stone, Daniel A McAdams, Simon Szykman, and Kristin L Wood. 2002. A functional basis for engineering design: reconciling and evolving previous efforts. Research in engineering Design.
R. Hoffmann, Congle Zhang, and Daniel S. Weld. 2010. Learning 5000 relational extractors. In ACL.
Tom Hope, Joel Chan, Aniket Kittur, and Dafna Shahaf. 2017. Accelerating innovation through analogy mining. In KDD.
Jeremy Howard and Sebastian Ruder. 2018. Universal language model fine-tuning for text classification. arXiv preprint arXiv:1801.06146.
Jeff Johnson, Matthijs Douze, and Hervé Jégou. 2017. Billion-scale similarity search with gpus. arXiv preprint arXiv:1702.08734.
Diane Marie Keeling, Patricia Garza, Charisse Michelle Nartey, and Anne-Ruxandra Carvunis. 2019. Philosophy of biology: The meanings of 'function' in biology and the problematic case of de novo gene emergence. *Elife*, 8:e47014.
Halil Kilicoglu, Graciela Roseblat, Marcelo Fiszman, and Thomas C Rindflesch. 2011. Constructing a semantic predication gold standard from the biomedical literature. BMC bioinformatics.
Jin-Dong Kim, Yue Wang, and Yamamoto Yasunori. 2013. The genia event extraction shared task, 2013 edition-overview. In BioNLP Shared Task Workshop.
Ruben Kruiper, Julian Vincent, Jessica Chen-Burger, Marc Desmulliez, and Ioannis Konstas. 2020. In layman's terms: Semi-open relation extraction from scientific texts. In ACL.
James R Lewis. 2002. Psychometric evaluation of the pssuq using data from five years of usability studies. International Journal of Human-Computer Interaction.
Jiao Li, Yueping Sun, Robin J Johnson, Daniela Sciaky, Chih-Hsuan Wei, Robert Leaman, Allan Peter Davis, Carolyn J Mattingly, Thomas C Wiegers, and Zhiyong Lu. 2016. Biocreative v cdr task corpus: a resource for chemical disease relation extraction. Database.
Chin-Yew Lin. 2004. Rouge: A package for automatic evaluation of summaries. In Text summarization branches out.
I. Loshchilov and F. Hutter. 2017. Fixing weight decay regularization in adam. ArXiv, abs/1711.05101.
Yi Luan, Luheng He, Mari Ostendorf, and Hannaneh Hajishirzi. 2018. Multi-task identification of entities, relations, and coreference for scientific knowledge graph construction. In EMNLP.
Ines Montani and Matthew Honnibal. 2018. Prodigy: A new annotation tool for radically efficient machine teaching. Artificial Intelligence.
Lidia Morawska, Donald K Milton, et al. 2020. It is time to address airborne transmission of Covid-19.
Benjamin Nye, Junyi Jessy Li, Roma Patel, Yinfei Yang, Iain J Marshall, Ani Nenkova, and Byron C Wallace. 2018. A corpus with multi-level annotations of patients, interventions and outcomes to support language processing for medical literature. In ACL.
Benjamin E Nye, Jay DeYoung, Eric Lehman, Ani Nenkova, Iain J Marshall, and Byron C Wallace. 2020. Understanding clinical trial reports: Extracting medical entities and their relations. arXiv preprint arXiv:2010.03550.
F. Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel, P. Prettenhofer, R. Weiss, V. Dubourg, J. Vanderplas, A. Passos, D. Cournapeau, M. Brucher, M. Perrot, and E. Duchesnay. 2011. Scikit-learn: Machine learning in Python. Journal of Machine Learning Research, 12:2825-2830.
Hoifung Poon, Chris Quirk, Charlie DeZiel, and David Heckerman. 2014. Literome: Pubmed-scale genomic knowledge base in the cloud. Bioinformatics.
Nils Reimers and Iryna Gurevych. 2019. Sentence-bert: Sentence embeddings using siamese bert-networks. In EMNLP.
Johannes Röhl. 2012. Mechanisms in biomedical ontology. In Journal of Biomedical Semantics. BioMed Central.
Manuel Salvadores, Paul R Alexander, Mark A Musen, and Natalya F Noy. 2013. Biportal as a dataset of linked biomedical ontologies and terminologies in rdf. Semantic web.
Isabel Segura Bedmar, Paloma Martínez, and María Herrero Zazo. 2013. Semeval-2013 task 9: Extraction of drug-drug interactions from biomedical texts (ddiextraction 2013). ACL.
Peng Shi and Jimmy Lin. 2019. Simple bert models for relation extraction and semantic role labeling. arXiv preprint arXiv:1904.05255.
Gizem Sogancioglu, Hakime Öztürk, and Arzucan Özgür. 2017. Biosses: a semantic sentence similarity estimation system for the biomedical domain. Bioinformatics.
Gabriel Stanovsky, Julian Michael, Luke Zettlemoyer, and Ido Dagan. 2018. Supervised open information extraction. In NAACL.
D. Swanson and N. Smalheiser. 1996. Undiscovered public knowledge: A ten-year update. In KDD.
Karin Verspoor, K Bretonnel Cohen, Michael Conway, Berry de Bruijn, Mark Dredze, Rada Mihalcea, and Byron C Wallace. 2020a. Proceedings of the 1st workshop on nlp for covid-19 (part 2) at emnlp 2020. In Workshop on NLP for COVID-19 at EMNLP 2020.
Karin Verspoor, Simon Šuster, Yulia Otmakhova, Shevon Mendis, Zenan Zhai, Biaoyan Fang, Jey Han Lau, Timothy Baldwin, Antonio Jimeno Yepes, and David Martinez. 2020b. Covid-see: Scientific evidence explorer for COVID-19 related research. arXiv preprint arXiv:2008.07880.
David Wadden, Shanchuan Lin, Kyle Lo, Lucy Lu Wang, Madeleine van Zuylen, Arman Cohan, and Hannaneh Hajishirzi. 2020. Fact or fiction: Verifying scientific claims. In EMNLP.
David Wadden, Ulme Wennberg, Yi Luan, and Hannaneh Hajishirzi. 2019. Entity, relation, and event extraction with contextualized span representations. In EMNLP.
Byron C Wallace, Joel Kuiper, Aakash Sharma, Mingxi Zhu, and Iain J Marshall. 2016. Extracting pico sentences from clinical trial reports using supervised distant supervision. *The Journal of Machine Learning Research*.
Jingqi Wang, Huy Anh Pham, Frank Manion, Masoud Rouhizadeh, and Yaoyun Zhang. 2020a. Covid-19 signsym: A fast adaptation of general clinical nlp tools to identify and normalize COVID-19 signs and symptoms to omop common data model. arXiv preprint arXiv:2007.10286.
Lucy Lu Wang, Kyle Lo, Yoganand Chandrasekhar, Russell Reas, Jiangjiang Yang, Darrin Eide, Kathryn Funk, Rodney Kinney, Ziyang Liu, William Merrill, et al. 2020b. Cord-19: The Covid-19 open research dataset. arXiv preprint arXiv:2004.10706.
A.1 Granular Relations