SlowGuess/ACL_2020_OCR huggingface dataset:
A qualitative analysis confirms the ability of the trained narrow IE system to support a domain expert during trade-off annotation. We predict trade-offs for 523 unlabeled, scientific papers that have been annotated with a trade-off in an ontology of biomimetics (Vincent, 2014, 2016). A domain expert compares the trade-offs found in the ontology of biomimetics against the output of the SCIE system, see Table 4. Narrow IE is found to locate the central TRADE-OFF relations and arguments for  $41.68\%$  of the total 523 papers. Explicit trade-offs were found in 243 documents. At least one of the extracted TRADE-OFF relations for each document is identical to the expert annotation in  $77.37\%$  of these documents. For  $89.71\%$  of the 243 documents a trade-off was found to be correct after some interpretation by the expert. Two main types of uninformative trade-offs were found: trade-offs from a cited source and trade-offs between generic terms, e.g., a trade-off between cost and benefit without defining what the cost and benefit are.
Argument Recognition	P	R	F1
RBS	44.31	35.32	39.31
SCIIE unlabeled spans	86.76	79.39	82.91
SCIIE labeled spans	83.86	80.83	82.32
Relation Extraction	P	R	F1
RBS	—	—	—
SCIIE unlabeled spans	68.53	65.48	66.97
SCIIE labeled spans	67.71	67.71	67.71
Table 3: Results on test set for our Rule-Based System (RBS) and SCIE (Luan et al., 2018a) w.r.t argument recognition and the combined task of extracting and classifying relations (RE). Providing the model with labels to distinguish trigger words from key-phrases slightly improves performance.
Documents with identified trade-offs	243
Exact match	77.37%
Match after interpretation	89.71%
Sentences with identified trade-off	998
Exact match	68.04%
Match after interpretation	84.47%
Table 4: Manual analysis of extractions from 523 scientific documents that were used in the creation of an ontology of biomimetics (Vincent, 2014, 2016).
5 Semi-Open Relation Extraction
5.1 Task description
We define the aim of SORE as extracting the relations and concepts in a text that capture the most central information. The application of SORE is especially of interest to scientific IE where OIE systems perform poorly and narrow IE systems are unable to cover the wealth of different relations types. One possible approach is to automatically filter out uninformative and incorrect extractions generated by OIE systems. In this approach, SORE relies on the output of both types of systems, providing a middle ground between precise, narrow IE and unbounded, but unreliable, OIE. The resulting extractions are expected to be useful for human readers, but can also be used to collect data for annotation and training of scientific IE systems.
We explore SORE on scientific biology texts using the output of the ScIE system trained on FOBIE, predicting trade-offs for the unlabeled 10k open access biology papers (see section 3.1). The narrow IE output consists of 2,216 trade-offs found in 1,279 documents. We pre-process arguments by appending their modifier, removing stop words, and embedding the remaining sequences using ELMo
(PubMed) $^4$ . We use the K-means algorithm to compute clusters on the IDF-weighted average of the resulting argument representations. A domain expert inspected the centroids qualitatively. Table 5 provides insight into some of the resulting argument clusters and their interrelations. The exact number of clusters does not seem to greatly affect SORE. For the given narrow IE output  $\pm 50$  clusters seems to provide a good balance between generic and more fine-grained topics. The IDF weights are computed over the subword units found in the dataset; we use SentencePiece $^5$  with a vocabulary of 16K. We then run OpenIE 5, a state-of-the-art OIE system (Saha and Mausam; Saha et al., 2017; Pal and Mausam, 2016; Christensen et al., 2011), on the same 1,279 documents that were found to contain one or more TRADE-OFF relations.
We retain only OIE extractions that contain one or more arguments that are classified into the same cluster as the TRADE-OFF arguments found in that text. Furthermore, we omit OIE arguments that belong to noisy clusters containing mostly math symbols or long nested phrases. We compute a simple IDF-weighted cosine similarity (Galárraga et al., 2014) between the vector representations of the remaining OIE and trade-off arguments.
5.3 Qualitative analysis of SORE output
We notice a striking drop in the number of irrelevant and noisy OIE arguments that remain after applying SORE. The total amount of OIE extractions reduces from 401k before filtering to 140k (34.95%) after filtering. As a result, the number of OIE extractions per document reduces from 314 to 110. The unfiltered OIE extractions are found in 170k sentences, of which 67k (39.55%) are retained after applying SORE.
To test our hypothesis that SORE can reduce the number of uninformative extractions, without limiting RE to a narrow set of relations, we randomly select representative samples of unfiltered and filtered OIE extractions (400 each). A domain expert manually annotated whether each extraction or sentence was thought to be informative, e.g., provides relevant information to understanding a biological text. As an example, consider the sentence "We have used this approach in a previous study to investigate the molecular factors governing the altered liver regeneration dynamics caused
4https://allennlp.org/elmo
5 https://github.com/google/sentencepiece
Cluster name	Immunity	Size	Locomotion
Top-5 arguments	immunity	size	swimming
immune function	number	sprinting
the immune system	volume	running
incompetence	age	locomotion
immune response	time	diving
Top-3 related clusters	Mating	Temperature	Attribute of Animal
Reproduction	Sperm Length	Verbs
Life History Traits	Offspring Number	Capacity/Endurance
Table 5: Examples of clusters found using the K-means algorithm on trade-off arguments from 1279 documents. For the related clusters only TRADE-OFF relations are taken into account.
by ablation of the gene adiponectin (Adn)" (Cook et al., 2015). OIE extractions such as ' (We, have used, this [...] study)' are considered uninformative, in contrast to ' (the molecular [...] dynamics, caused by, ablation [...] adiponectin)'.
Many OIE extractions are found to be poorly structured. Like Groth et al. (2018) we relax the requirement of extractions being well-formed, e.g., we consider extractions that incorrectly identify the boundaries of one or more arguments as possibly capturing relevant information. Different from their evaluation on correctness, we evaluate whether an extraction captures information that is relevant to understanding a text. As a result, we consider poorly structured OIE extractions that contain relevant information to be informative, e.g.:
- ('the resumption of respiration', 'can lead to an increase of superoxide anions in the cytosol perhaps driving', 'increased elevation of Cu-ZnSOD').
- ('transcriptional coregulation amongst many genes', 'will give', 'rise to indirect interaction effects in mRNA expression data').
The annotation relies on the correctness of the information captured by OIE extractions and whether this information is useful to a reader. However, this does not imply informative extractions are relevant to the central theme of the text captured in a trade-off. We consider OIE extractions uninformative if the extraction:
- contains an uninformative argument class, e.g., ('Miller et al. 2012', 'to minimize', 'their swimming effort').
- contains incomplete arguments, e.g., ('the RDME requirement', 'reactions', 'only fire').
- is non-sensible, e.g., ('P. magellanicus', 'would have resulted', 'in a 1.6-fold higher Vmax for the scallop muscle').
- is unlikely to help understand a text, e.g., ('DeepBind', 'was trained', 'on data from RNAcompete, CLIP - RIP - seq [ 10') and ('microlepidopteran superfamilies', 'are heavily entombed', 'L:in amber').
We also randomly select representative samples from the 170k unfiltered and 67k filtered sentences from which the OIE extractions are sourced. The reason is that erroneous OIE extractions, e.g., not well-formed tuples, can guide a reader to informative passages in a text. We see similar errors as described by Schneider et al. (2017) and Groth et al. (2018), e.g., long sentences lead to incorrect extractions and errors in argument boundaries. To illustrate the complexity of sentences that an OIE system encounters in scientific texts, consider the following examples:
- the arity of relations can be high, e.g., (49 tokens) "A large genome size tends to correlate with delayed mitotic and meiotic division [6-8] decreased plant invasiveness of disturbed sites [9] lower maximum photosynthetic rates in plants [2] and lower metabolic rates in mammals [10] and birds [11, 12]." (Warringer and Blomberg, 2006).
- many phrases are nested and express nonverbal relations, e.g., (45 tokens) "However, for arboreal animals that regularly jump between branches (often when elevated quite high above the ground), jumping accurately (which we define as the ability to land close to the intended target) may also be important to fitness." (Kuo et al., 2011).
Table 6 provides an overview of the annotation results. Filtering is found to increase the informativeness of both OIE extractions  $(\chi^2 = 6.39, p < .025)$  and sentences  $(\chi^2 = 11.75, p < .01)$ . The percentage of informative OIE extractions increases by  $5.75\%$  and of the percentage of informative sentences by
Filtering	#Nr.	% Informative
OIE extractions	before	401,588	29.25%
after	140,357	35.00%
Sentences	before	170,551	36.50%
after	67,460	44.75%
Table 6: Total number of OIE extractions before and after filtering, as well as the sentences that these extractions were found in. The % informative denotes the percentage of extractions and sentences annotated as informative by a domain expert, based on 400 randomly sampled instances from each group (95% confidence interval, margin of error 5%).
8.25%. A second domain expert annotated 25% of each set (400 total), the inter-annotator agreement Cohen  $k$  was found to be 0.84.
Manual inspection of the retained OIE extractions shows that many relevant extractions are retained, e.g., see Table 7. These extractions are useful to a reader in determining whether a document is worth reading in full, and can be used to identify informative sections in a text. The presented approach to SORE shows promising results w.r.t. automatically filtering out a large proportion of irrelevant, incorrect, or uninformative OIE extractions. Considering the poor quality of OIE extractions, however, we propose presenting a reader with the sentences that entail the filtered OIE extractions. Furthermore, SORE provides a method to collect data for annotation and training of scientific OIE systems.
We introduce the task of Semi-Open Relation Extraction (SORE) on scientific texts and the Focused Open Biological Information Extraction (FOBIE) dataset. We adapt off-the-shelf IE systems to show that SORE is feasible, and that our approach is worth improving upon - both in terms of performance, as well as reducing the system's complexity. A strong scientific IE system is used as a baseline, and its output is used to filter the relations found by a state-of-the-art OIE system.
OIE from scientific text is a hard task. The large number of errors that we find in OIE extractions from scientific texts render them near-useless to downstream computing tasks. A human reader may, nevertheless, find many incorrect extractions informative. An issue for humans is the sheer amount of OIE extractions and the high proportion of uninformative extractions. We show that our approach
TRADE-OFF relations
Trade-off arguments	Argument modifiers
sleep
cognitive abilities
energy conservation
memory retention	(the keeping of memory over prolonged periods of time)
memory consolidation	(in bats)(without a food reward)(shift from short- to long-term memory)(using torpor)
Examples filtered OIE extractions
(A memory; is normally formed; after repeated learning events; sleep enhances this process)
(learning; is associated; with a food reward)
(Sleep deprivation; has; negative effects on both memory consolidation)
(torpor; has; a negative influence on memory consolidation)
(digestion; prevents; the bats; from falling into torpor quickly)
(torpor ; indeed affects ; learning abilities)
Table 7: SORE extractions from a scientific biology text (Ruczyski et al., 2014). The TRADE-OFF relations are extracted by a narrow IE system trained on FOBIE. These relations capture the central theme and concepts of the text, and are used to filter the extractions that an OIE system outputs for the same document. The resulting extractions can support discerning the relevance of scientific documents.
to SORE reduces the number of OIE extractions by  $65\%$ , while increasing the relative amount of informative extractions by  $5.75\%$ . As a result, SORE improves the ability for a reader to quickly skim through the remaining extractions, or sentences that they are sourced from, and analyze how central concepts are related in a scientific text.
The presented approach is currently limited to the domain of biology and the use of trade-off relations, but we expect that central relations can be identified for other scientific domains that enable SORE. We show that creating a dataset for narrow RE can be done relatively cheaply by re-annotating the output of a simple RBS. Similarly, SORE may aid the collection of a dataset for scientific OIE.
The authors would like to gratefully acknowledge the financial support of the Engineering and Physical Sciences Research Council (EPSRC) Centre for Doctoral Training in Embedded Intelligence under grant reference EP/L014998/1 and the EPSRC Innovation Placement fund. We also thank Ben Trevett for proof-reading the document.
Dominique Adriaens. 2019. Evomimetics: the biomimetic design thinking 2.0. In Bioinspiration, Biomimetics, and Bioreplication IX, volume 1096509, page 8. SPIE.
Anurag A Agrawal, Jeffrey K Conner, and Sergio Rasmann. 2010. Tradeoffs and Negative Correlations in Evolutionary Ecology Why Are We Interested in Tradeoffs? In *Evolution since Darwin: the first 150 years*, chapter 10, pages 243–268.
Brian S Alper, Jason A Hand, Susan G Elliott, Scott Kinkade, Michael J Hauan, Daniel K Onion, and Bernard M Sklar. 2004. How much effort is needed to keep up with the literature relevant for primary care? Journal of the Medical Library Association: JMLA, 92(4):429-37.
Waleed Ammar, Matthew Peters, Chandra Bhagavatula, and Russell Power. 2018. The AI2 system at SemEval-2017 Task 10 (ScienceIE): semi-supervised end-to-end entity and relation extraction. pages 592-596.
Isabelle Augenstein, Mrinal Das, Sebastian Riedel, Lakshmi Vikraman, and Andrew McCallum. 2017. SemEval 2017 Task 10: ScienceIE - Extracting Keyphrases and Relations from Scientific Publications. In Proceedings of the 11th International Workshop on Semantic Evaluation (SemEval-2017), pages 546-555, Stroudsburg, PA, USA. Association for Computational Linguistics.
Michele Banko, M Cafarella, Stephen Soderland, M J Broadhead, and Oren Etzioni. 2007. Open information extraction from the web. Proceedings of the 20th International Joint Conference on Artificial Intelligence, pages 2670-2676.
Robert Bossy, Louise Deléger, Estelle Chaix, Mouhamadou Ba, and Claire Nédellec. 2019. Bacteria Biotope at BioNLP Open Shared Tasks 2019. In Proceedings of The 5th Workshop on BioNLP Open Shared Tasks, pages 121-131, Stroudsburg, PA, USA. Association for Computational Linguistics.
Razvan C Bunescu and Raymond J Mooney. 2005. A shortest path dependency kernel for relation extraction. Proceedings of the Human Language Technology Conference and Conference on Empirical Methods in Natural Language Processing, (October):724-731.
Stephen S. O. Burgess, Jarmila Pittermann, and Todd E. Dawson. 2006. Hydraulic efficiency and safety of branch xylem increases with height in Sequoia sem-pervirens (D. Don) crowns. Plant, Cell and Environment, 29(2):229-239.
Gemma Carr, Daniel P. Loucks, and Günter Blöschl. 2018. Gaining insight into interdisciplinary research and education programmes: A framework for evaluation. Research Policy, 47(1):35-48.
Estelle Chaix, Bertrand Dubreucq, Abdelhak Fatihi, Dialekti Valsamou, Robert Bossy, Mouhamadou Ba, Louise Deleger, Pierre Zweigenbaum, Philippe Bessieres, Loic Lepiniec, and Claire Nedellec. 2016. Overview of the Regulatory Network of Plant Seed Development (SeeDev) Task at the BioNLP Shared Task 2016. pages 1-11.
Yubo Chen, Liheng Xu, Kang Liu, Daojian Zeng, and Jun Zhao. 2015. Event Extraction via Dynamic Multi-Pooling Convolutional Neural Networks. In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing, pages 167-176.
Janara Christensen, Mausam, Stephen Soderland, and Oren Etzioni. 2011. An analysis of open information extraction based on semantic role labeling. In Proceedings of the sixth international conference on Knowledge capture - K-CAP '11, page 113.
K. Bretonnel Cohen, Karin Verspoor, Karen Fort, Christopher Funk, Michael Bada, Martha Palmer, and Lawrence E. Hunter. 2017. The Colorado Richly Annotated Full Text (CRAFT) Corpus: Multi-Model Annotation in the Biomedical Domain. In Handbook of Linguistic Annotation, pages 1379-1394. Springer Netherlands.
Daniel Cook, Babatunde A. Ogunnaike, and Rajanikanth Vadigepalli. 2015. Systems analysis of non-parenchymal cell modulation of liver repair across multiple regeneration modes. BMC Systems Biology, 9(1):71.
Louise Delegere, Robert Bossy, Estelle Chaix, Mouhamadou Ba, Arnaud Ferre, Philippe Bessières, and Claire Nédellec. 2016. Overview of the Bacteria Biotope Task at BioNLP Shared Task 2016. pages 12-22.
Khalid El-Arini and Carlos Guestrin. 2011. Beyond keyword search. In Proceedings of the 17th ACM SIGKDD international conference on Knowledge discovery and data mining - KDD '11, page 439, New York, New York, USA. ACM Press.
Oren Etzioni, Michael Cafarella, Doug Downey, Ana Maria Popescu, Tal Shaked, Stephen Soderland, Daniel S. Weld, and Alexander Yates. 2005. Unsupervised named-entity extraction from the Web: An experimental study. Artificial Intelligence, 165(1):91-134.
Anthony Fader, Stephen Soderland, and Oren Etzioni. 2011. Identifying relations for Open Information Extraction. In EMNLP 2011 - Conference on Empirical Methods in Natural Language Processing, Proceedings of the Conference, pages 1535-1545.
Kata Gabor, Davide Buscaldi, Anne-Kathrin Schumann, Behrang Qasemizadeh, Hafa Zargayouna, and Thierry Charnois. 2018. SemEval-2018 Task 7: Semantic Relation Extraction and Classification in Scientific Papers. In SemEval, pages 679-688.
Luis Galárraga, Geremy Heitz, Kevin Murphy, and Fabian M. Suchanek. 2014. Canonicalizing Open Knowledge Bases. In Proceedings of the 23rd ACM International Conference on Conference on Information and Knowledge Management - CIKM '14, pages 1679-1688.
Theodore Garland. 2014. Trade-offs. Current Biology, 24(2):R60-R61.
Paul Groth, Michael Lauruhn, Antony Scerri, and Ron Daniel. 2018. Open Information Extraction on Scientific Text: An Evaluation. In Proceedings of the 27th International Conference on Computational Linguistics, page 3414-3423. Association for Computational Linguistics.
Luheng He, Kenton Lee, Omer Levy, and Luke Zettlemoyer. 2018. Jointly Predicting Predicates and Arguments in Neural Semantic Role Labeling. pages 364-369.
Lynette Hirschman, Alexander Yeh, Christian Blaschke, and Alfonso Valencia. 2005. Overview of BioCreAtIvE: critical assessment of information extraction for biology. BMC Bioinformatics, 6(Suppl 1):S1.
Rezarta Islamaj Dogan, Sun Kim, Andrew Chatri-aryamontri, Chih-Hsuan Wei, Donald C Comeau, Rui Antunes, Sergio Matos, Qingyu Chen, Aparna Elangovan, Nagesh C Panyam, Karin Verspoor, Hongfang Liu, Yanshan Wang, Zhuang Liu, Berna Altunel, Zehra Melce Husunbeyi, Arzucan Ozgur, Aris Fergadis, Chen-Kai Wang, Hong-Jie Dai, Tung Tran, Ramakanth Kavuluru, Ling Luo, Albert Steppi, Jinfeng Zhang, Jinchan Qu, and Zhiyong Lu. 2019. Overview of the BioCreative VI Precision Medicine Track: mining protein interactions and mutations for precision medicine. Database, 2019.
Robin Jia, Cliff Wong, and Hoifung Poon. 2019. Document-Level N-ary Relation Extraction with Multiscale Representation Learning. In *NAACL HLT*, pages 3693–3704.
Xiaotian Jiang, Quan Wang, Peng Li, and Bin Wang. 2016. Relation Extraction with Multi-instance Multi-label Convolutional Neural Networks. *Coling*, (89):1471-1480.
J. D. Kim, T Ohta, Y Tateisi, and J Tsujii. 2003. GENIA corpus - A semantically annotated corpus for bio-textmining. In Bioinformatics, volume 19, pages 180-182.
Jin-Dong Kim, Tomoko Ohta, Sampo Pyysalo, Yoshi-nobu Kano, and Jun'ichi Tsujii. 2009. Overview of BioNLP'09 shared task on event extraction. In Proceedings of the Workshop on BioNLP Shared Task - BioNLP '09, page 1, Morristown, NJ, USA. Association for Computational Linguistics.
Olga Kononova, Haoyan Huo, Tanjin He, Wenhao Sun, Ziqin Rong, Tiago Botari, Vahe Tshitoyan, and Gerbrand Ceder. 2019. Text-mined dataset of inorganic materials synthesis recipes. Sci Data, 6(203).
Martin Krallinger, Obdulia Rabal, Saber A Akhondi, Martin Pérez Pérez, Jesús Santamaría, Gael Pérez Rodríguez, Georgios Tsatsaronis, Ander IntxAurrondo, José Antonio López, Umesh Nandal, Erin Van Buel, Akileshwari Chandrasekhar, Marleen Rodenburg, Astrid Laegreid, Marius Doornenbal, Julien Oyarzabal, Analia Lourenço, and Alfonso Valencia. 2017. Overview of the BioCreative VI chemicalprotein interaction Track. In Proceedings of BioCreative VI workshop, pages 141-146.
Ruben Kruiper, Jessica Chen-Burger, and Marc P Y Desmulliez. 2016. Computer-aided biomimetics. In Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics), volume 9793, pages 131-143.
Ruben Kruiper, Julian Vincent, Eitan Abraham, Rupert Soar, Ioannis Konstas, Jessica Chen-Burger, and Marc Desmulliez. 2018. Towards a Design Process for Computer-Aided Biomimetics. Biomimetrics, 3(3):14.
Chi Yun Kuo, Gary B. Gillis, and Duncan J. Irschick. 2011. Loading effects on jump performance in green anole lizards, Anolis carolinensis. Journal of Experimental Biology, 214(12):2073-2079.
Kenton Lee, Luheng He, Mike Lewis, and Luke Zettle-moyer. 2017. End-to-end Neural Coreference Resolution. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 188-197, Stroudsburg, PA, USA. Association for Computational Linguistics.
Yankai Lin, Shiqi Shen, Zhiyuan Liu, Huanbo Luan, and Maosong Sun. 2016. Neural Relation Extraction with Selective Attention over Instances. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 2124-2133.
Chunyang Liu, Wenbo Sun, Wenhan Chao, and Wanxiang Che. 2013. Convolution neural network for relation extraction. Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics), 8347 LNAI(PART 2):231-242.
Yi Luan, Luheng He, Mari Ostendorf, and Hannaneh Hajishirzi. 2018a. Multi-Task Identification of Entities, Relations, and Coreference for Scientific Knowledge Graph Construction. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 3219-3232, Stroudsburg, PA, USA. Association for Computational Linguistics.
Yi Luan, Mari Ostendorf, and Hannaneh Hajishirzi. 2018b. The UWNLP system at SemEval-2018 Task 7: Neural Relation Extraction Model with Selectively Incorporated Concept Embeddings. In Proceedings of The 12th International Workshop on Semantic Evaluation, pages 788-792, Stroudsburg, PA, USA. Association for Computational Linguistics.