SlowGuess/NAACL_2021_OCR huggingface dataset:
In addition to the two coarse-grained relation classes, we also experimented with granular relations where the class represents a specific type of a mechanism relation explicitly mentioned in the text (we constrain the mention a single token for simplicity, e.g., binds, causes, reduces; see Fig. 5 for examples of granular relations). While more granular, these relations are also less general – as the natural language of scientific papers describing mechanisms often does not conform to this more rigid structure (e.g., long-range and implicit causal relations). We thus focus most of our work on coarse-grained relations. We release our dataset and a model for extraction of granular relations to support future research and applications, in our code repository.
Figure 5: Examples of granular relations.
A.2 Annotation Collection
We utilize the Prodigy (Montani and Honnibal, 2018) annotation platform which provides the ability to select span boundaries and relations with ease. Each annotator undergoes a training session in which we cover the definitions of spans and relations as well as use of the platform. See annotation guidelines in our code repository for more details and examples.
Tab. 3 shows examples of differences between annotations, with disagreements in the span boundaries. This reflects the challenging nature of our task with relations between flexible, open entities.
B IE Evaluations
B.1 Automated evaluation metrics
Entity detection Given a boolean span matching function  $\mathbf{m}(s_1, s_2) = \mathbb{1}(s_1 \text{ matches } s_2)$ , a predicted entity mention  $\hat{e}$  is correctly identified if there exists some gold mention  $e^*$  in  $\mathcal{D}$  such that  $\mathbf{m}(\hat{e}, e^*) = 1$  (since there is only one entity type, an entity mention is correctly classified as long as its span is correctly identified).
Following common practice in work on Open IE (Stanovsky et al., 2018), we report results using a partial-matching similarity function, in this case based on the widely-used Rouge score:  $\mathrm{m_{\mathrm{rouge}}(s_1,s_2)}$  is true if Rouge-L(s1,s2) > 0.5 (Lin, 2004).
Relation detection / classification Given a boolean span matching function, a predicted coarse-grained relation  $\hat{r} = (\hat{E}_1, \hat{E}_2, \hat{y})$  is correctly identified if there exists some gold relation  $r^* = (E_1^*, E_2^*, y^*)$  in  $\mathcal{D}$  such that  $\mathrm{m}(\hat{E}_1, E_1^*) = 1$  and  $\mathrm{m}(\hat{E}_2, E_2^*) = 1$ . It is properly classified if, in addition,  $\hat{y} = y^*$ .
Relation identification measures the model's ability to identify mechanisms of any type - direct or indirect - while relation classification aims to discriminate between direct and indirect types of mechanism mentions in the text.
SemRep The SemRep dataset (Kilicoglu et al., 2011), consisting of 500 sentences from MEDLINE abstracts and annotated for semantic predication. Concepts and relations in this dataset relate to clinical medicine from the UMLS biomedical ontology (Bodenreider, 2004), with entities such as drugs and diseases. Some of the relations correspond to mechanisms (such as X TREATS Y or X CAUSES Y); By the lead of domain experts, we map these existing relations to our mechanism classes and use them to train DyGIE. Other relations are even broader, such as PART-OF or IS-A – we do not attempt to capture these categories as they often do not reflect a functional relation.
Scierc SciERC dataset (Luan et al., 2018), consisting of 500 abstracts from computer science pa
Context	Annotator 1	Annotator 2
Predicted siRNAs should effectively silence the genes of SARS - CoV-2 during siRNA mediated treatment.	(predicted siRNAs, silence the genes of SARS - CoV-2, DIRECT)	(siRNAs, silence the genes of SARS - CoV-2 during siRNA mediated treatment, DIRECT)
Recent reports show that the inhibition of NSP4 expression by small interfering RNAs leads to alteration of the production and distribution of other viral proteins and mRNA synthesis , suggesting that NSP4 also affects virus replication by unknown mechanisms.	(NSP4, affects virus replication, INDIRECT)	(NSP4, virus replication by unknown mechanisms, INDIRECT)
Table 3: Examples of differences between two annotators. The core meaning of the relation is equivalent across both annotators.
Figure 6: Example of the annotation interface for coarse (left) and granular (right) mechanism relations.
pers that are annotated for a set of relations, including for USED-FOR relations between methods and tasks. We naturally map this relation to our DIRECT label and discard other relation types, and use this dataset to train DYGIE.
SRL Finally we also use a recent BERT-based SRL model (Shi and Lin, 2019). We select relations of the form (Arg0, verb, Arg1), and evaluate using our partial metrics applied to Arg0 and Arg1 respectively.
B.3 Hyperparameter Search
We perform hyperparameter search over these sets of parameters:
- Dropout is randomly selected from intervals [0, 0.5].
- Learning rate is randomly selected between  $[1e - 5, 1e - 2]$
- Hidden Size is randomly selected from interval [64, 512]
Hyperparameter search is implemented using grid search with the Allentune library (Dodge et al., 2019). For each experiment we set the search space to be among 30 total samples in hyperparameter
space. We select the best-performing parameters using the development set.
B.4 Best Performing Model over MECHANIC
We use the DYGIE package (Wadden et al., 2019) to train models for entity and relation extraction over MECHANIC and we utilize SciBERT (Beltagy et al., 2019) for our text embeddings and finetune upon that, with learning rate for finetuning set to  $5e - 5$  with weight decay of 0.01. The training was run for 100 epochs with the slanted_triangular (Howard and Ruder, 2018) learning rate scheduler. We used the AdamW (Loshchilov and Hutter, 2017) optimization algorithm. In our objective function we assign equal weights to relation and span loss terms. The maximum allowed length of spans is 12.
The hyperparameters achieving best performance over our development search are 0.43,  $7e - 4$  and 215 for dropout, learning rate and hidden size respectively. All other parameters are kept to default values (available in our code repository).
Tab. 4 compares the performance of our best model with the baselines introduced in Sec. B.2. Fig. 7 shows Precision@K results, with our model reaching high absolute numbers.
Model	RC	RD	ED
OpenIE	-	15.5	25.6
SRL	-	24.5	27.7
DYGIE(SemRep)	6.8	8.3	32.5
DYGIE(SciERC)	18.6	20.4	39.2
DYGIE(MECHANIC)	42.8	45.6	50.2
Table 4: F1 scores. Relations from SRL and OpenIE do not map directly to DIRECT MECHANISM and INDIRECT MECHANISM classes, and do not have relation classification scores.
Figure 7: Precision@K of our model compared with pre-trained SciERC and SemRep baselines. P@K for our model is high in absolute numbers.
B.5 Granular relation prediction
Granular relations are evaluated in the same fashion as coarse-grained relations, with the additional requirement that the predicted predicate token  $\hat{p}$  must match the gold  $p^*$ . Our evaluation shows that the model trained to predict granular triples achieves  $F1$  score of 44.0. When predicting relations without trigger labels (i.e.,  $(s, \circ)$ ), the model achieves  $F1$  scores of 53.4. These results are not comparable to those for MECHANIC, which includes more documents and relations that did not directly conform to the  $(s, \circ, p)$  schema.
B.6 Best Performing Model over
MECHANIC-G
Here too we use the DYGIE package (Wadden et al., 2019) with SciBERT (Beltagy et al., 2019). Due to technical equivalence in the annotation schema of our granular relations and the event extraction task in Wadden et al. (2019), we make use of the event extraction functionality of DYGIE. For fine-tuning the embedding weights of SciBERT we used the same learning rate weight as for MECHANIC, and the best hyperparameters found are
0.30,  $11e - 4$  and 372 for dropout, learning rate and hidden layer size respectively. All other parameters are kept to default values (available in our code repository).
C Human evaluation guidelines
C.1 KB Correctness and Informativeness evaluation guideline
Relation quality evaluations over various domains For the task involving the exploration of viral mechanisms, we used 10 recent scientific claims taken from (Wadden et al., 2020). These 10 claims, and the queries constructed for them, are as follows:
- Remdessevir has exhibited favorable clinical responses when used as a treatment for coronavirus.  $\mathrm{X} =$  [Remdessevir],  $\mathrm{Y} =$  [SARS-CoV-2, coronavirus, COVID-19]
- Lopinavir / ritonavir have exhibited favorable clinical responses when used as a treatment for coronavirus. X = [Lopinavir, Ritonavir], Y = [SARS-CoV-2, coronavirus, COVID-19]
- Aerosolized SARS-CoV-2 viral particles can travel further than 6 feet.  $\mathrm{X} =$  [Air, Aerosols, Droplets, Particles, Distance],  $\mathrm{Y} =$  [SARS-CoV-2 transmission]
- Chloroquine has shown antiviral efficacy against SARS-CoV-2 in vitro through interference with the ACE2-receptor mediated endocytosis.  $\mathrm{X} =$  [Chloroquine],  $\mathrm{Y} =$  [ACE2-receptor, Endocytosis, interference with the ACE2-receptor mediated endocytosis.]
- Lymphopenia is associated with severe COVID-19 disease. X = [Lymphopenia], Y = [severe COVID-19 disease, COVID-19]
- Bilateral ground glass opacities are often seen on chest imaging in COVID-19 patients.  $\mathrm{X} =$  [Bilateral ground glass opacities],  $\mathrm{Y} =$  [chest imaging in COVID-19 patients]
- Cardiac injury is common in critical cases of COVID-19. X = [COVID-19], Y = [Cardiac injury]
- Cats are carriers of SARS-CoV-2.  $\mathrm{X} =$  [Cats], Y = [SARS-CoV-2]
Figure 8: Average pairwise annotator agreement by several metrics. In the AI task human labels were more diverse but with overall high precision / recall.
Figure 9: List of post-study questions given to MDs.
- Diabetes is a common comorbidity seen in COVID-19 patients. X = [Diabetes], Y = [COVID-19]
- The coronavirus cannot thrive in warmer climates.  $\mathrm{X} =$  [warmer climates],  $\mathrm{Y} =$  [coronavirus]
- SARS-CoV-2 binds ACE2 receptor to gain entry into cells. X = [SARS-CoV-2], Y = [binds ACE2 receptor, binds ACE2 receptor to gain entry into cells]
For the AI open-ended search task, we used the following approaches/areas as queries (see guidelines and examples in our code repository): artificial intelligence, machine learning, statistical models, predictive models, Graph Neural Network model, Convolutional Neural Network model, Recurrent Neural Network model, reinforcement learning, image analysis, text analysis, speech analysis.
For both tasks, we use the following metrics to measure pairwise agreement between annotators (Fig. 8): standard accuracy (proportion of matching rating labels), F1 (taking into account both precision and recall symmetrically), balanced accuracy (with class weights to down-weight the higher proportion of positive ratings), and finally the Matthew Correlation Coefficient (MCC) score, using the corresponding functions in the Scikit-Learn Python library (Pedregosa et al., 2011).
Comparing KB quality to other schema We sampled the relations predicted by our model and the baseline models introduced in App. B.2. We randomly selected 20 abstracts from the MECHANIC
Topic	Question
IR	The overall search results accurately matched the query.
IR	I was satisfied by the overall ranking of results.
I found results that were both relevant and interesting or new to me, making this system useful for rapid explorations.
IR	I wanted to read the papers shown.
overall	Overall, I am satisfied with this system.
UI/UX	Overall, I am satisfied with how easy it is to use this system.
UI/UX	It was simple to use this system.
UI/UX	I felt comfortable using this system.
The information (such as on-screen messages and other documentation) provided with this system was clear.
UI/UX	The organization of information on the system screens was clear.
UI/UX	The system gave error messages that clearly told me how to fix problems.
UI/UX	Whenever I made a mistake using the system, I could recover easily and quickly.
UI/UX	The interface of this system was pleasant.
UI/UX	I liked using the interface of this system.
UI/UX	It was easy to learn to use this system.
I was able to understand and judge each individual result quickly and without effort.
utility
utility	I was able to complete the tasks and scenarios quickly using this system.
utility	The information was effective in helping me complete the tasks and scenarios.
utility	I believe I could become productive quickly using this system.
utility	It was easy to find the information I needed.
utility	This system has all the functions and capabilities I expect it to have.
test set and show at most two predictions (if available) for each sentence within that abstract. In total 300 relations are extracted. Each relation is shown separately to two bio-NLP expert annotators (with annotators blind to the condition), who label each relation with a  $0/1$  label (1 if the relation is both correct and informative).
C.2 KB Utility
MDs are instructed to search with our interface and with PubMed search, with the following 7 topics:
- Query 1: Cardiac arrhythmias caused by COVID 19
- Query 2: Hydroxychloroquine and its effect on COVID 19
- Query 3: Ivermectin and its role in management of COVID 19
- Query 4: Pulmonary embolism effect on complications related to COVID 19
- Query 5: Liver disease and COVID 19
- Query 6 : Inflammatory bowel disease and COVID -19
- Query 7 : Antibody therapy and its uses/effects on COVID-19
The full list of the post-study evaluation questions given to MDs is shown in Fig. 9.