SlowGuess/NAACL_2022_OCR huggingface dataset:
Domain Knowledge: Similarly, classifying relations in this dataset may require an understanding of domain knowledge. In "Growth inhibition and apoptosis were significantly higher in BxPC-3, HPAC, and PANC-1 cells treated with celecoxib and erlotinib than cells treated with either celecoxib or erlotinib", one must understand that having higher values of Growth inhibition and apoptosis in specific cells is a positive outcome, in order to classify this combination as positive.
Context related Complications: The following are kinds of complications found when the evidence lies in the wider part of the context.
Coreference: Anaphoric or coreferential reasoning is sometimes needed to understand the efficacy of the combination e.g. "it was demonstrated that they could be combined with acceptable toxicity".
Contradicting Evidence: the reader often must infer a conclusion given opposing claims within a given abstract. This can happen as combinations can be referred as e.g. toxic but effective.
Long Distance: The target sentence can be far—up to 41 sentences apart—from the evidence sentence,
making it difficult for even humans to process.
4.3 Quantitative Error Analysis
To probe this task, we analyze the performance of our strongest model—the one using a Pubmed-BERT base model tuned with domain-adaptive pretraining—along different partitions of test data. We trained with four random seeds and perform comparisons using a paired multi-bootstrap hypothesis test where bootstrap samples are generated by sampling hierarchically over the four random seeds and the subsets of the test set (Sellam et al., 2021). We use 1000 bootstrap samples in each test.
4.3.1 Do models leverage context effectively?
Each relation in our dataset consists of entities contained within a single sentence, but labeling the relation frequently requires extra-sentential context to make a decision. In our dataset, annotators record whether or not each relation requires paragraph-level context to label, reporting that  $67\%$  of drug combinations required context to annotate.
To understand the extent that models make use of paragraph-level context, we trained and evaluated our PubmedBERT-based model using varying amounts of extra-sentential context around the sentence containing drug entities. In Table 3, we see that adding context provides nearly identical performance to training a model with no extra-sentential context at all, with differences rarely exceeding one standard deviation of F1 score.
However, we see increased variability in "Positive Combination F1" performance when extrasentential context is used. To explain this, recall from §2.1 that determining the efficacy of a drug combination often requires paragraph-level context for annotators, while identifying any combination usually requires no context. From qualitative analysis of attention maps, we observed that our models are not able to consistently identify the salient parts of paragraph-level context, potentially causing instability with larger inputs.
Figure 4: Comparing models performance on binary vs higher-order  $N$ -ary relations, averaged over 4 seeds of the PubmedBERT-DAPT model. No consistent significant differences were observed;  $p$ -values for these comparisons are 0.456, 0.149, 0.240, and 0.276.
Figure 5: Comparing relation extraction on test set drug combinations that are observed in the training set or not, using the PubmedBERT-DAPT model. Paired multi-bootstrap test  $p$ -values for these four comparisons are 0.262, 0.025, 0.103, and 0.009, respectively.
These results suggest ample room for improvement in extracting document-level evidence. This makes our dataset a potentially useful benchmark for document-level language understanding.
4.3.2 Binary vs. higher-arity relations
Given that our dataset is the first relation extraction dataset with variable-arity relations, do higher-order relations pose a particular challenge for our models? To answer this, we separate all predicted and ground truth relations for the test set into binary relations and higher-arity relations. We then report precision among each subset of predicted relations and recall among each subset of ground truth relations. We perform this experiment across four different model seeds, and report results in aggregate using a paired multi-bootstrap procedure. In the results in Figure 4, we see no consistent significant differences between models of different arities, suggesting that our technique of computing relation representations by averaging entity representations scales well to higher-order relations.
4.3.3 Generalizing to new drug combinations
How well can relation extraction models classify drug combinations not seen during training? Sim
ilar to the setup in  $\S 4.3.2$ , we divide all predicted and ground truth relations for the test set into the set of drug combinations which are also annotated in our training set, and the set that have not been. In our dataset, over  $80\%$  of annotated test set relations are not found in the training set.
In Figure 5, performance is consistently better for relations observed in the training set than for unseen relations, by a margin of 10-15 points. Recall, in particular, is significantly worse for relations unseen during training (at  $95\%$  confidence), and precision is potentially also worse. Considering that unseen drug combinations are practically more valuable than already-known combinations, improving generalization to new combinations is a critical area of improvement for this task.
The DDI dataset (Herrero-Zazo et al., 2013) is the only work to our knowledge that annotates drug interactions for text mining. However, it fundamentally differs from our dataset in the type of annotations provided: the DDI annotates the type of discourse context in which a drug combination is mentioned, without providing explicit information about combination efficacy. In contrast, our dataset is focused on semantically classifying the efficacy of drug combinations as stated in text.
Other RE datasets exist in the biomedical field (Peng et al., 2017; Li et al., 2016; Wu et al., 2019; Krallinger et al., 2017), but do not focus on drug combinations. Similarly, several RE datasets tackle the  $N$ -arity problem in the scientific domain (Peng et al., 2017; Jain et al., 2020; Kardas et al., 2020; Hou et al., 2019), and in the non-scientific domain (Akimoto et al., 2019; Nguyen et al., 2016), however, all of them consider a fixed choice of  $N$ .
We present a new resource for drug combination and efficacy identification. We establish baseline models that achieve promising results but reveal clear areas for improvement. Beyond the immediate, application-ready value of this task, this task poses unique relation extraction challenges as the first dataset containing variable-arity relations. We also highlight challenges with document-level representation learning and incorporating domain knowledge. We encourage others to partici
pate in this task. Our processed dataset $^{10}$  and best baseline model $^{11}$  are available on Hugging Face, and our model training code is available to the public at https://github.com/allenai/drug-combo-extraction.
This project has received funding in part from the European Research Council (ERC) under the European Union's Horizon2020 research and innovation programme, grant agreement 802774 (iEXTRACT), and in part from the NSF Convergence Accelerator Award #2132318. We would also like to thank our annotators from the Shamay lab at the Faculty of Biomedical Engineering, Technion, including Shaked Launer-Wachs, Yuval Harris, Maytal Avrashami, Hagit Sason-Bauer and Yakir Amrusi
Kosuke Akimoto, Takuya Hiraoka, Kunihiko Sadamasa, and Mathias Niepert. 2019. Cross-sentence n-ary relation extraction using lower-arity universal schemas. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 6225-6231, Hong Kong, China. Association for Computational Linguistics.
Jun Araki, Lamana Mulaffer, Arun Pandian, Yukari Yamakawa, Kemal Oflazer, and Teruko Mitamura. 2018. Interoperable annotation of events and event relations across domains. In Proceedings 14th Joint ACL-ISO Workshop on Interoperable Semantic Annotation, pages 10-20.
Lora Aroyo and Chris Welty. 2013. Measuring crowd truth for medical relation extraction. In 2013 AAAI Fall Symposium Series.
John A Bartlett, Michael J Fath, Ralph Demasi, Ashwaq Hermes, Joseph Quinn, Elsa Mondou, and Franck Rousseau. 2006. An updated systematic overview of triple combination therapy in antiretroviral-naive hiv-infected adults. *Aids*, 20(16):2051-2064.
Iz Beltagy, Kyle Lo, and Arman Cohan. 2019. SciBERT: A pretrained language model for scientific text. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the
9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 3615-3620, Hong Kong, China. Association for Computational Linguistics.
Y Bhusal, CM Shiohira, and N Yamane. 2005. Determination of in vitro synergy when three antimicrobial agents are combined against mycobacterium tuberculosis. International journal of antimicrobial agents, 26(4):292-297.
Jennifer S Carew, Francis J Giles, and Steffan T Nawrocki. 2008. Histone deacetylase inhibitors: mechanisms of cell death and promise in combination cancer therapy. *Cancer letters*, 269(1):7-17.
Jacob Cohen. 1960. A coefficient of agreement for nominal scales. Educational and psychological measurement, 20(1):37-46.
VT DeVita, RC Young, and GP Canellos. 1975. Combination versus single agent chemotherapy: a review of the basis for selection of drug treatment of cancer. Cancer, 35(1):98-110.
Richard T Eastman and David A Fidock. 2009. Artemisinin-based combination therapies: a vital tool in efforts to eliminate malaria. Nature Reviews Microbiology, 7(12):864-874.
Joseph L Fleiss. 1971. Measuring nominal scale agreement among many raters. Psychological bulletin, 76(5):378.
Yu Gu, Robert Tinn, Hao Cheng, Michael Lucas, Naoto Usuyama, Xiaodong Liu, Tristan Naumann, Jianfeng Gao, and Hoifung Poon. 2020. Domain-specific language model pretraining for biomedical natural language processing.
Suchin Gururangan, Ana Marasovic, Swabha Swayamdipta, Kyle Lo, Iz Beltagy, Doug Downey, and Noah A. Smith. 2020. Don't stop pretraining: Adapt language models to domains and tasks. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 8342-8360, Online. Association for Computational Linguistics.
Xu Han, Hao Zhu, Pengfei Yu, Ziyun Wang, Yuan Yao, Zhiyuan Liu, and Maosong Sun. 2018. Fewrel: A large-scale supervised few-shot relation classification dataset with state-of-the-art evaluation.
Andrew F Hayes and Klaus Krippendorff. 2007. Answering the call for a standard reliability measure for coding data. Communication methods and measures, 1(1):77-89.
María Herrero-Zazo, Isabel Segura-Bedmar, Paloma Martínez, and Thierry Declerck. 2013. The DDI corpus: An annotated corpus with pharmacological substances and drug-drug interactions. Journal of biomedical informatics, 46(5):914-920.
Yufang Hou, Charles Jochim, Martin Gleize, Francesca Bonin, and Debasis Ganguly. 2019. Identification of tasks, datasets, evaluation metrics, and numeric scores for scientific leaderboards construction. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 5203-5213, Florence, Italy. Association for Computational Linguistics.
Aleksandr Ianevski, Rouan Yao, Svetlana Biza, Eva Zusinaite, Andres Mannik, Gaily Kivi, Anu Planken, Kristiina Kurg, Eva-Maria Tombak, Mart Ustav, et al. 2020. Identification and tracking of antiviral drug combinations. Viruses, 12(10):1178.
Aleksandr Ianevski, Rouan Yao, Hilde Lysvand, Gunnveig Grodeland, Nicolas Legrand, Valentyn Oksenych, Eva Zusinaite, Tanel Tenson, Magnar Bjorås, and Denis E. Kainov. 2021. Nafamostat-interferon- combination suppresses sars-cov-2 infection in vitro and in vivo by cooperatively targeting host tmprss2. Viruses, 13(9).
Sarthak Jain, Madeleine van Zuylen, Hannaneh Hajishirzi, and Iz Beltagy. 2020. Scirex: A challenge dataset for document-level information extraction.
Marcin Kardas, Piotr Czapla, Pontus Stenetorp, Sebastian Ruder, Sebastian Riedel, Ross Taylor, and Robert Stojnic. 2020. AxCell: Automatic extraction of results from machine learning papers. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 8580-8594, Online. Association for Computational Linguistics.
Itay Katzir, Murat Cokol, Bree B Aldridge, and Uri Alon. 2019. Prediction of ultra-high-order antibiotic combinations based on pairwise interactions. PLoS computational biology, 15(1):e1006774.
Martin Krallinger, Obdulia Rabal, Saber Ahmad Akhondi, Martin Pérez Pérez, Jesus Santamaría, Gael Pérez Rodríguez, Georgios Tsatsaronis, Ander Intxaurrondo, José Antonio Baso López, Umesh K. Nandal, Erin M. van Buel, Anjana Chandrasekhar, Marleen Rodenburg, Astrid Lægreid, Marius A. Doornenbal, Julien Oyarzábal, Analía Lourenço, and Alfonso Valencia. 2017. Overview of the BioCreative VI chemical-protein interaction track.
Jinhyuk Lee, Wonjin Yoon, Sungdong Kim, Donghyeon Kim, Sunkyu Kim, Chan Ho So, and Jaewoo Kang. 2020. BioBERT: a pretrained biomedical language representation model for biomedical text mining. Bioinformatics, 36(4):1234-1240.
Jiao Li, Yueping Sun, Robin Johnson, Daniela Sciaky, Chih-Hsuan Wei, Robert Leaman, Allan Peter Davis, Carolyn Mattingly, Thomas Wiegers, and Zhiyong lu. 2016. Biocreative V CDR task corpus: a resource for chemical disease relation extraction. Database, 2016:baw068.
Yi Luan, Luheng He, Mari Ostendorf, and Hannaneh Hajishirzi. 2018. Multi-task identification of entities, relations, and coreference for scientific knowledge graph construction. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 3219-3232, Brussels, Belgium. Association for Computational Linguistics.
Ines Montani and Matthew Honnibal. 2018. Prodigy: A new annotation tool for radically efficient machine teaching. Artificial Intelligence, to appear.
Kiem-Hieu Nguyen, Xavier Tannier, Olivier Ferret, and Romaric Besançon. 2016. A dataset for open event extraction in English. In Proceedings of the Tenth International Conference on Language Resources and Evaluation (LREC'16), pages 1939-1943, PortoRož, Slovenia. European Language Resources Association (ELRA).
Danna Niezni, Hillel Taub-Tabib, Yuval Harris, Hagit Sason-Bauer, Yakir Amrusi, Dana Azagury, Maytal Avrashami, Shaked Launer-Wachs, Jon Borchardt, M Kusold, Aryeh Tiktinsky, Tom Hope, Yoav Goldberg, and Yosi Shamay. 2022. Extending the boundaries of cancer therapeutic complexity with literature data mining. bioRxiv preprint.
Nanyun Peng, Hoifung Poon, Chris Quirk, Kristina Toutanova, and Wen tau Yih. 2017. Cross-sentence n-ary relation extraction with graph lstms.
Yifan Peng, Shankai Yan, and Zhiyong Lu. 2019. Transfer learning in biomedical natural language processing: An evaluation of BERT and ELMo on ten benchmarking datasets. In Proceedings of the 2019 Workshop on Biomedical Natural Language Processing (BioNLP 2019), pages 58-65.
Yogita Rochlani, Mohammed Hasan Khan, Maciej Banach, and Wilbert S Aronow. 2017. Are two drugs better than one? a review of combination therapies for hypertension. Expert opinion on pharmacotherapy, 18(4):377-386.
Shachar Rosenman, Alon Jacovi, and Yoav Goldberg. 2020. Exposing shallow heuristics of relation extraction models with challenge data.
Ofer Sabo, Yanai Elazar, Yoav Goldberg, and Ido Dagan. 2021. Revisiting few-shot relation classification: Evaluation data and classification schemes.
Thibault Sellam, Steve Yadlowsky, Jason Wei, Naomi Saphra, Alexander D'Amour, Tal Linzen, Jasmijn Bastings, Iulia Turc, Jacob Eisenstein, Dipanjan Das, Ian Tenney, and Ellie Pavlick. 2021. The multiberts: Bert reproductions for robustness analysis. ArXiv, abs/2106.16163.
Micah Shlain, Hillel Taub-Tabib, Shoval Sadde, and Yoav Goldberg. 2020. Syntactic search by example. In ACL.
Adam J Shuhendler, Richard Y Cheung, Janet Manias, Allegra Connor, Andrew M Rauth, and Xiao Yu Wu. 2010. A novel doxorubicin-mitomycin c coencapsulated nanoparticle formulation exhibits anticancer synergy in multidrug resistant human breast cancer cells. Breast cancer research and treatment, 119(2):255-269.
Hillel Taub-Tabib, Micah Shlain, Shoval Sadde, Dan Lahav, Matan Eyal, Yaara Cohen, and Yoav Goldberg. 2020. Interactive extractive search over biomedical corpora. arXiv preprint arXiv:2006.04148.
Ernesto Wasserman, William Sutherland, and Esteban Cvitkovic. 2001. Irinotecan plus oxaliplatin: a promising combination for advanced colorectal cancer. Clinical colorectal cancer, 1(3):149-153.
Ye Wu, Ruibang Luo, Henry CM Leung, Hing-Fung Ting, and Tak-Wah Lam. 2019. Renet: A deep learning approach for extracting gene-disease associations from literature. In International Conference on Research in Computational Molecular Biology, pages 272-284. Springer.
Yuhao Zhang, Victor Zhong, Danqi Chen, Gabor Angeli, and Christopher D. Manning. 2017. Position-aware attention and supervised data improve slot filling. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 35-45, Copenhagen, Denmark. Association for Computational Linguistics.
Zexuan Zhong and Danqi Chen. 2021. A frustratingly easy approach for entity and relation extraction. In North American Association for Computational Linguistics (NAACL).
A.1 Annotation Guidelines
Interaction Type	Context Required
POS1	○ Yes	○ No
POS2	○ Yes	○ No
POS3	○ Yes	○ No
Figure 6: Annotation instance in the Prodigy environment. The screen is constructed of the sentence where they should mark relations, a button to show the full context and a selection per relation to indicate the necessity of the context.
For this task we recruited 7 annotators all studying for advanced degrees in biomedical engineering. The annotators were paid by their advisor, an amount that is standard for annotation projects in their country of residence. All participating annotators were provided with annotation guidelines. The guidelines specified how the annotation process should be carried out and provided definitions and examples for the different labels used. As the task progressed, the guidelines were also expanded to include discussion of frequently encountered issues.
For a given instance, such as presented in the top of Figure 6 the annotator needs to first recognize any missing drugs and mark them, and then label any interactions they find among the drugs. In case they need to consult a wider context they can press on a 'show more context' button and a text box with the wider context will appear. This context can be again hidden by clicking the same button if needed. Lastly, in the bottom of the sample page, we present a table with questions regarding the necessity of using the context.
Then the annotator should decide if they need to ignore the current sample or to complete the current instance and accept it, by pressing the accept and ignore buttons.
The annotators are instructed as follows. They should read the sentence carefully, and try to answer a two phase question to themselves. First, if the drugs are mentioned in any form of combination or they should be given separately. Second, if indeed the annotator recognized the drugs as a combination can they determine the efficacy of the combination by the sole sentence.
In case they can not determine the efficacy they are instructed to press on the 'get more context' button and read the entire context in order to determine what is the correct efficacy. If after reading the context they can still not determine the efficacy then the label of the interaction should be OTHER_COMB (aside from negative label experimentation mentioned in Footnote 4). Otherwise it should be POS_COMB. In case that they recognized that there is no combination between the drugs in the sentence then they should not use any label and simply accept the current instance. Then they should answer the context related questions for the POS_COMB label in order to signal if the context was needed.
While reading the sentence if the annotators find unmarked drugs they can mark them before continuing
to the interaction-labeling phase and treat them the same as the other drugs, but, it is not required to mark a word as drug in order to use it in an interaction. If a drug is marked in a wrong manner they should try and fix it, e.g. the span of the drug is incorrect.
In order to achieve more consistent and accurate annotations, they are also instructed to annotate all the interactions that they can find in a given sentence. They should always use the accept button even if there are no interactions in the sentence. Only in cases where they want to skip a sentence (e.g. when there is an inherent problem with it) or leave it for a future discussion they should use the ignore button. An interaction can occur between more than two drugs, if so they should notice that they don't need each pair from this group to have a marked interaction, as long as they all connect to the same graph. e.g. "Drugs A, B and C are synergistic." connecting A to B and B to C is sufficient, no need to connect drug A to drug C. Each interaction should be marked with a different tag (POS_COMB1, POS_COMB2..., OTHER_COMB1, OTHER_COMB2...).
A.2 Evaluation Metric Discussion
For measuring the agreement, we chose to use our adaptation of F1 score and not other common metrics such as Cohen's Kappa (Cohen, 1960) or one of its variations (e.g. Feliss's Kappa (Fleiss, 1971) and Krippendorf's Alpha (Hayes and Krippendorff, 2007)). These metrics expect a setup where the relation candidates are already marked and the task is only to label them – a labeling task and not an extraction task. This causes two problems, one is that they inherently do not need to handle a partial match. So if for example there are three drugs in a sentence, the first annotator annotated a relation between drugs A and B, while a second annotator annotated the same relation between drugs A, B and C. So we will either underestimate or overestimate their agreement score if we considered this a mismatch or a match respectively. Moreover, their calculations depend on the hypothetical agreement by chance normalization factor, but this will not reflect the difficulty of random choosing in our setup as they ignore the size of the combinatorial set of relation candidates we can possibly have.
A.3 Trigger List
Figure 7: Abstracts percentage including each trigger word (1634 abstracts included; 44 words in the full word list; Words  $< 1\%$  were neglected from the figure.
In Figure 7 we show the triggers that we used in the Spike queries. We show the percentage of abstracts that included each trigger (others under  $1\%$ : conjunction, two-drug, first choice, additivity, combinational, synergetic, simultaneously with, supra-additive, five-drug, combinatory, over-additive, timed-sequential, co-blister, super-additive, synergisms, synergic, synergistical, less-than-additive, greater-than-additive, additivesynergistic, supraadditive, superadditive, overadditive, subadditive, first-choice, 2-drug, sub-additive, more-than-additive, 3-drug).