1.1em1.1.1em1.1.1.1emJennifer D'Souza, Anett Hoppe, Arthur Brack, Mohamad Yaser Jaradeh, Sören Auer, Ralph EwerthTIB Leibniz Information Centre for Science and Technology,Hannover, Germany{jennifer.dsouza,anett.hoppe,arthur.brack,yaser.jaradeh,auer,ralph.ewerth}@tib.euWe introduce the STEM (Science, Technology, Engineering, and Medicine) Dataset for Scientific Entity Extraction, Classification, and Resolution, version 1.0 (STEM-ECR v1.0). The STEM-ECR v1.0 dataset has been developed to provide a benchmark for the evaluation of scientific entity extraction, classification, and resolution tasks in a domain-independent fashion. It comprises abstracts in 10 STEM disciplines that were found to be the most prolific ones on a major publishing platform. We describe the creation of such a multidisciplinary corpus and highlight the obtained findings in terms of the following features: 1) a generic conceptual formalism for scientific entities in a multidisciplinary scientific context; 2) the feasibility of the domain-independent human annotation of scientific entities under such a generic formalism; 3) a performance benchmark obtainable for automatic extraction of multidisciplinary scientific entities using BERT-based neural models; 4) a delineated 3-step entity resolution procedure for human annotation of the scientific entities via encyclopedic entity linking and lexicographic word sense disambiguation; and 5) human evaluations of Babelfy returned encyclopedic links and lexicographic senses for our entities. Our findings cumulatively indicate that human annotation and automatic learning of multidisciplinary scientific concepts as well as their semantic disambiguation in a wide-ranging setting as STEM is reasonable.Entity Recognition, Entity Classification, Entity Resolution, Entity Linking, Word Sense Disambiguation, Evaluation Corpus, Language ResourceBy starting with a STEM corpus of scholarly abstracts for annotating with scientific entities, we differ from existing work addressing this task since we are going beyond the domain restriction that so far seems to encompass scientific IE. For entity annotations, we rely on existing scientific concept formalisms BIBREF0, BIBREF1, BIBREF2 that appear to propose generic scientific concept types that can bridge the domains we consider, thereby offering a uniform entity selection framework. In the following subsections, we describe our annotation task in detail, after which we conclude with benchmark results.The corpus for computing inter-annotator agreement was annotated by two postdoctoral researchers in Computer Science. To develop annotation guidelines, a small pilot annotation exercise was performed on 10 abstracts (one per domain) with a set of surmised generically applicable scientific concepts such as Task, Process, Material, Object, Method, Data, Model, Results, etc., taken from existing work. Over the course of three annotation trials, these concepts were iteratively pruned where concepts that did not cover all domains were dropped, resulting in four finalized concepts, viz. Process, Method, Material, and Data as our resultant set of generic scientific concepts (see Table TABREF3 for their definitions). The subsequent annotation task entailed linguistic considerations for the precise selection of entities as one of the four scientific concepts based on their part-of-speech tag or phrase type. Process entities were verbs (e.g., “prune” in Agr), verb phrases (e.g., “integrating results” in Mat), or noun phrases (e.g. “this transport process” in Bio); Method entities comprised noun phrases containing phrase endings such as simulation, method, algorithm, scheme, technique, system, etc.; Material were nouns or noun phrases (e.g., “forest trees” in Agr, “electrons” in Ast or Che, “tephra” in ES); and majority of the Data entities were numbers otherwise noun phrases (e.g., “(2.5$\pm $1.5)kms$^{-1}$” representing a velocity value in Ast, “plant available P status” in Agr). Summarily, the resulting annotation guidelines hinged upon the following five considerations:To ensure consistent scientific entity spans, entities were annotated as definite noun phrases where possible. In later stages, the extraneous determiners and articles could be dropped as deemed appropriate.Coreferring lexical units for scientific entities in the context of a single abstract were annotated with the same concept type.Quantifiable lexical units such as numbers (e.g., years 1999, measurements 4km) or even as phrases (e.g., vascular risk) were annotated as Data.Where possible, the most precise text reference (i.e., phrases with qualifiers) regarding materials used in the experiment were annotated. For instance, “carbon atoms in graphene” was annotated as a single Material entity and not separately as “carbon atoms,” “graphene.”Any confusion in classifying scientific entities as one of four types was resolved using the following concept precedence: Method $>$ Process $>$ Data $>$ Material, where the concept appearing earlier in the list was preferred.After finalizing the concepts and updating the guidelines, the final annotation task proceeded in two phasesIn phase I, five abstracts per domain (i.e. 50 abstracts) were annotated by both annotators and the inter-annotator agreement was computed using Cohen's $\kappa $ BIBREF4. Results showed a moderate inter-annotator agreement at 0.52 $\kappa $.Next, in phase II, one of the annotators interviewed subject specialists in each of the ten domains about the choice of concepts and her annotation decisions on their respective domain corpus. The feedback from the interviews were systematically categorized into error types and these errors were discussed by both annotators. Following these discussions, the 50 abstracts from phase I were independently reannotated. The annotators could obtain substantial overall agreement of 0.76 $\kappa $ after phase II.In Table TABREF16, we report the IAA scores obtained per domain and overall. The scores show that the annotators had a substantial agreement in seven domains, while only a moderate agreement was reached in three domains, viz. Agr, Mat, and Ast.We discuss some of the changes the interviewer annotator made in phase II after consultation with the subject experts.In total, 21% of the phase I annotations were changed: Process accounted for a major proportion (nearly 54%) of the changes. Considerable inconsistency was found in annotating verbs like “increasing”, “decreasing”, “enhancing”, etc., as Process or not. Interviews with subject experts confirmed that they were a relevant detail to the research investigation and hence should be annotated. So 61% of the Process changes came from additionally annotating these verbs. Material was the second predominantly changed concept in phase II, accounting for 23% of the overall changes. Nearly 32% of the changes under Material came from consistently reannotating phrases about models, tools, and systems; accounting for another 22% of its changes, where spatial locations were an essential part of the investigation such as in the Ast and ES domains, they were decided to be included in the phase II set as Material. Finally, there were some changes that emerged from lack of domain expertise. This was mainly in the medical domain (4.3% of the overall changes) in resolving confusion in annotating Process and Method concept types. Most of the remaining changes were based on the treatment of conjunctive spans or lists.Subsequently, the remaining 60 abstracts (six per domain) were annotated by one annotator. This last phase also involved reconciliation of the earlier annotated 50 abstracts to obtain a gold standard corpus.Table TABREF17 shows our annotated corpus characteristics. Our corpus comprises a total of 6,127 scientific entities, including 2,112 Process, 258 Method, 2,099 Material, and 1,658 Data entities. The number of entities per abstract directly correlates with the length of the abstracts (Pearson's R 0.97). Among the concepts, Process and Material directly correlate with abstract length (R 0.8 and 0.83, respectively), while Data has only a slight correlation (R 0.35) and Method has no correlation (R 0.02).In Figure FIGREF18, we show an example instance of a manually created text graph from the scientific entities in one abstract. The graph highlights that linguistic relations such as synonymy, hypernymy, meronymy, as well as OpenIE relations are poignant even between scientific entities.In the second stage of the study, we perform word sense disambiguation and link our entities to authoritative sources.Aside from the four scientific concepts facilitating a common understanding of scientific entities in a multidisciplinary setting, the fact that they are just four made the human annotation task feasible. Utilizing additional concepts would have resulted in a prohibitively expensive human annotation task. Nevertheless, there are existing datasets (particularly in the biomedical domain, e.g., GENIA BIBREF6) that have adopted the conceptual framework in rich domain-specific semantic ontologies. Our work, while related, is different since we target the annotation of multidisciplinary scientific entities that facilitates a low annotation entrance barrier to producing such data. This is beneficial since it enables the task to be performed in a domain-independent manner by researchers, but perhaps not crowdworkers, unless screening tests for a certain level of scientific expertise are created.Nonetheless, we recognize that the four categories might be too limiting for real-world usage. Further, the scientific entities from stage 1 remain susceptible to subjective interpretation without additional information. Therefore, in a similar vein to adopting domain-specific ontologies, we now perform entity linking (EL) to the Wikipedia and word sense disambiguation (WSD) to Wiktionary.The same pair of annotators as before were involved in this stage of the study to determine the annotation agreement.During the annotation procedure, each annotator was shown the entities, grouped by domain and file name, in Google Excel Sheet columns alongside a view of the current abstract of entities being annotated in the BRAT interface stenetorp2012brat for context information about the entities. For entity resolution, i.e. linking and disambiguation, the annotators had local installations of specific time-stamped Wikipedia and Wiktionary dumps to enable future persistent references to the links since the Wiki sources are actively revised. They queried the local dumps using the DKPro JWPL tool BIBREF8 for Wikipedia and the DKPro JWKTL tool BIBREF9 for Wiktionary, where both tools enable optimized search through the large Wiki data volume.Through iterative pilot annotation trials on the same pilot dataset as before, the annotators delineated an ordered annotation procedure depicted in the flowchart in Figure FIGREF28. There are two main annotation phases, viz. a preprocessing phase (determining linkability, determining whether an entity is decomposable into shorter collocations), and the entity resolution phase.The actual annotation task then proceeded, in which to compute agreement scores, the annotators worked on the same set of 50 scholarly abstracts that they had used earlier to compute the scores for the scientific entity annotations.In this first step, entities that conveyed a sense of scientific jargon were deemed linkable.A natural question that arises, in the context of the Linkability criteria, is: Which stage 1 annotated scientific entities were now deemed unlinkable? They were 1) Data entities that are numbers; 2) entities that are coreference mentions which, as isolated units, lost their precise sense (e.g., “development”); and 3) Process verbs (e.g., “decreasing”, “reconstruct”, etc.). Still, having identified these cases, a caveat remained: except for entities of type Data, the remaining decisions made in this step involved a certain degree of subjectivity because, for instance, not all Process verbs were unlinkable (e.g., “flooding”). Nonetheless, at the end of this step, the annotators obtained a high IAA score at 0.89 $\kappa $. From the agreement scores, we found that the Linkability decisions could be made reliably and consistently on the data.While preference was given to annotating non-compositional noun phrases as scientific entities in stage 1, consecutive occurrences of entities of the same concept type separated only by prepositions or conjunctions were merged into longer spans. As examples, consider the phrases “geysers on south polar region,” and “plume of water ice molecules and dust” in Figure FIGREF18. These phrases, respectively, can be meaningfully split as “geysers” and “south polar region” for the first example, and “plume”, “water ice molecules”, and “dust” for the second. As demonstrated in these examples, the stage 1 entities we split in this step are syntactically-flexible multi-word expressions which did not have a strict constraint on composition BIBREF10. For such expressions, we query Wikipedia or Google to identify their splits judging from the number of results returned and whether, in the results, the phrases appeared in authoritative sources (e.g., as overview topics in publishing platforms such as ScienceDirect). Since search engines operate on a vast amount of data, they are a reliable source for determining phrases with a strong statistical regularity, i.e. determining collocations.With a focus on obtaining agreement scores for entity resolution, the annotators bypass this stage for computing independent agreement and attempted it mutually as follows. One annotator determined all splits, wherever required, first. The second annotator acted as judge by going through all the splits and proposed new splits in case of disagreement. The disagreements were discussed by both annotators and the previous steps were repeated iteratively until the dataset was uniformly split. After this stage, both annotators have the same set of entities for resolution.In this stage, the annotators resolved each entity from the previous step to encyclopedic and lexicographic knowledge bases. While, in principle, multiple knowledge sources can be leveraged, this study only examines scientific entities in the context of their Wiki-linkability.Wikipedia, as the largest online encyclopedia (with nearly 5.9 million English articles) offers a wide coverage of real-world entities, and based on its vast community of editors with editing patterns at the rate of 1.8 edits per second, is considered a reliable source of information. It is pervasively adopted in automatic EL tasks BIBREF11, BIBREF12, BIBREF13 to disambiguate the names of people, places, organizations, etc., to their real-world identities. We shift from this focus on proper names as the traditional Wikification EL purpose has been, to its, thus far, seemingly less tapped-in conceptual encyclopedic knowledge of nominal scientific entities.Wiktionary is the largest freely available dictionary resource. Owing to its vast community of curators, it rivals the traditional expert-curated lexicographic resource WordNet BIBREF14 in terms of coverage and updates, where the latter evolves more slowly. For English, Wiktionary has nine times as many entries and at least five times as many senses compared to WordNet. As a more pertinent neologism in the context of our STEM data, consider the sense of term “dropout” as a method for regularizing the neural network algorithms which is already present in Wiktionary. While WSD has been traditionally used WordNet for its high-quality semantic network and longer prevalence in the linguistics community (c.f Navigli navigli2009word for a comprehensive survey), we adopt Wiktionary thus maintaining our focus on collaboratively curated resources.In WSD, entities from all parts-of-speech are enriched w.r.t. language and wordsmithing. But it excludes in-depth factual and encyclopedic information, which otherwise is contained in Wikipedia. Thus, Wikipedia and Wiktionary are viewed as largely complementary.Given a scholarly abstract $A$ comprising a set of entities $E = \lbrace e_{1}, ... ,e_{N}\rbrace $, the annotation goal is to produce a mapping from $E$ to a set of Wikipedia pages ($p_1,...,p_N$) and Wiktionary senses ($s_1,...,s_N$) as $R = \lbrace (p_1,s_1), ... , (p_N,s_N)\rbrace $. For entities without a mapping, the corresponding $p$ or $s$ refers to Nil.The annotators followed comprehensive guidelines for ER including exceptions. E.g., the conjunctive phrase “acid/alkaline phosphatase activity” was semantically treated as the following two phrases “acid phosphatase activity” or “alkaline phosphatase activity” for EL, however, in the text it was retained as “acid” and “alkaline phosphatase activity.” Since WSD is performed over exact word-forms without assuming any semantic extension, it was not performed for “acid.” Annotations were also made for complex forms of reference such as meronymy (e.g., space instrument “CAPS” to spacecraft “wiki:Cassini Huygens” of which it is a part), or hypernymy (e.g., “parents” in “genepool parents” to “wiki:Ancestor”). As a result of the annotation task, the annotators obtained 82.87% rate of agreement in the EL task and a $\kappa $ score of 0.86 in the WSD task. Contrary to WSD expectations as a challenging linguistics task BIBREF15, we show high agreement; this we attribute to the entities' direct scientific sense and availability in Wiktionary (e.g., “dropout”).Subsequently, the ER annotation for the remaining 60 abstracts (six per domain) were performed by one annotator. This last phase also involved reconciliation of the earlier annotated 50 abstracts to obtain a gold standard corpus.In this stage 2 corpus, linkability of the scientific entities was determined at 74.6%. Of these, 61.7% were split into shorter collocations, at 1.74 splits per split entity. Detailed statistics are presented in Table TABREF36. In the table, the domains are ranked by the total number of their linkable entities (fourth column). Ast has the highest proportion of linked entities at 87.3% which comprises 10.4% of all the linked entities and disambiguated entities at 71.4% forming 8.5% of the overall disambiguated entities. From an EL perspective, we surmize that articles on space topics are well represented in Wikipedia. For WSD, Bio, ES, and Med predictably have the least proportion of disambiguated entities at 52.3%, 54.6%, and 55.5%, respectively, since of all our domains these especially rely on high degree scientific jargon, while WSD generally tends to be linguistically oriented in a generic sense. As a summary, linked and disambiguated entities had a high correlation with the total linkable entities ($R$ 0.98 and 0.89, respectively).In Table TABREF37, the ER annotation results are shown as POS tag distributions. The POS tags were obtained from Wiktionary, where entities that couldn't be disambiguated are tagged as SW (Single Word) or MWE (Multi-Word Expression). These tags have a coarser granularity compared to the traditionally followed Penn Treebank tags with some unconventional tagging patterns (e.g., “North Sea” as NNP, “in vivo” as ADJ). From the distributions, except for nouns being the most EL and WSD instances, the rest of the table differs significantly between the two tasks in a sense reflecting the nature of the tasks. While MWE are the second highest EL instances, its corresponding PHRASE type is least represented in WSD. In contrast, while adverbs are the second highest in WSD, they are least in EL.We do not observe a significant impact of the long-tailed list phenomenon of unresolved entities in our data (c.f Table TABREF36 only 17% did not have EL annotations). Results on more recent publications should perhaps serve more conclusive in this respect for new concepts introduced–the abstracts in our dataset were published between 2012 and 2014.The STEM-ECR v1.0 corpus of scientific abstracts offers multidisciplinary Process, Method, Material, and Data entities that are disambiguated using Wiki-based encyclopedic and lexicographic sources thus facilitating links between scientific publications and real-world knowledge (see the concepts enrichment we obtain from Wikipedia for our entities in Figure ). We have found that these Wikipedia categories do enable a semantic enrichment of our entities over our generic four concept formalism as Process, Material, Method, and Data (as an illustration, the top 30 Wiki categories for each of our four generic concept types are shown in the Appendix). Further, considering the various domains in our multidisciplinary STEM corpus, notably, the inclusion of understudied domains like Mathematics, Astronomy, Earth Science, and Material Science makes our corpus particularly unique w.r.t. the investigation of their scientific entities. This is a step toward exploring domain independence in scientific IE. Our corpus can be leveraged for machine learning experiments in several settings: as a vital active-learning test-bed for curating more varied entity representations BIBREF16; to explore domain-independence versus domain-dependence aspects in scientific IE; for EL and WSD extensions to other ontologies or lexicographic sources; and as a knowledge resource to train a reading machine (such as PIKES BIBREF17 or FRED BIBREF18) that generate more knowledge from massive streams of interdisciplinary scientific articles. We plan to extend this corpus with relations to enable building knowledge representation models such as knowledge graphs in a domain-independent manner.We thank the anonymous reviewers for their comments and suggestions. We also thank the subject specialists at TIB for their helpful feedback in the first part of this study. This work was co-funded by the European Research Council for the project ScienceGRAPH (Grant agreement ID: 819536) and by the TIB Leibniz Information Centre for Science and Technology.A.1. Proportion of the Generic Scientific EntitiesTo offer better insights to our STEM corpus for its scientific entity annotations made in part 1, in Figure FIGREF40 below, we visually depict the proportion of Process, Method, Material, and Data entities per domain.The Figure serves a complementary view to our corpus compared with the dataset statistics shown in Table TABREF17. It shows that the Ast domain has the highest proportion of scientific entities overall. On the other hand, per generic type, Bio has the most Process entities, CS has the most Method entities, Ast has the most Material closely followed by Agr, and Eng has the most Data.A.2. Cohen's $\kappa $ Computation Setup in Section 4.1.2Linkability. Given the stage 1 scientific entities, the annotators could make one of two decisions: a) an entity is linkable; or b) an entity is unlinkable. These decisions were assigned numeric indexes, i.e. 1 for decision (a) and -1 for decision (b) and can take on one of four possible combinations based on the two annotators decisions: (1,1), (1,-1), (-1,1), and (-1,-1). The $\kappa $ scores were then computed on this data representation.WSD Agreement. In order to compute the WSD agreement, the Wiktionary structure for organizing the words needed to be taken into account. It's structure is as follows. Each word in the Wiktionary lexicographic resource is categorized based on etymology, and within each etymological category, by the various part-of-speech tags the word can take. Finally, within each POS type, is a gloss list where each gloss corresponds to a unique word sense.Given the above-mentioned Wiktionary structure, the initial setup for the blind WSD annotation task entailed that the annotators were given the same reference POS tags within an etymology for the split single-word entities in the corpus. Next, as data to compute $\kappa $ scores, each annotator-assigned gloss sense was given a numeric index and agreement was computed based on matches or non-matches between indexes.A.3. Per-domain Inter-annotator Agreement for Entity ResolutionTo supplement the overall Inter-Annotator Agreement (IAA) scores reported in Section 4.1.2 `Entity Resolution (ER) Annotation' for the EL and WSD tasks, in Table TABREF43 below, we additionally report the IAA scores for our ER tasks (i.e., EL and WSD) per domain in the STEM-ECR corpus. First, considering the domains where the highest ER agreement scores were obtained. For EL, the IAA score was highest in the MS domain. While for WSD, the IAA score was highest in the Bio domain. Next, considering the domains where the agreement was least for the two tasks. We found the the EL agreement was least for CS and the WSD agreement was least for Mat. In the case of low EL agreement, it can be attributed to two main cases: only one of the annotators found a link; or the annotators linked to related pages on the same theme as the entity (e.g., wiki:Rule-based_modeling versus wiki:Rule-based_machine_learning for “rule-based system”). And in the case of the low WSD agreement obtained on Mat, we see that owing to broad terms like “set,” “matrix,” “groups,” etc., in the domain which could be disambiguated to more than one Wiktionary sense correctly, the IAA agreement was low.A.4. Babelfy's Precision ($P$) and Recall ($R$) Computation for Entity Resolution in FigureFor the $P$ and $R$ scores reported in Figure , the true positives (TP), false negatives (FN), true negatives (TN), and false positives (FP) were computed as follows:TP = human-annotated entities that have a EL/WSD match with Babelfy results (for Nil, a match is considered as no result from the automatic system);FN = human-annotated entities that have no EL/WSD match with Babelfy results;TN = spurious Babelfy-created strings as entities that do not have a EL/WSD result; andFP = spurious Babelfy-created entities that have a EL/WSD result.A.5. Top 30 Wikipedia Categories for Process, Method, Material, and DataIn part 1 of the study, we categorized the scientific entities by our four generic concept formalism, comprising Process, Method, Material, and Data. Linking the entities to Wikipedia further enables their broadened categorization. While in Figure is depicted the rich set of Wikipedia categories obtained overall, here, in Tables TABREF44 and TABREF45, we show the top 30 Wikipedia categories for the scientific entities by their four concept types. we observe the most of the Wikipedia categories pertinently broaden the semantic expressivity of each of our four concepts. Further that in each type, they are diverse reflecting the underlying data domains in our corpus. As examples, consider the Wikipedia categories for the Data scientific entities: “SIBaseQuantities” category over the entity “Kelvin” in Che; “FluidDynamics” in Eng and MS domains; and “SolarCalendars” in the Ast domain.

Q: How large is the dataset?