dataesr/scientific-paragraphs-categorization huggingface dataset:
These restrictions have led to authors evaluating their approaches on datasets  (1)  that are available to them and  (2)  for which writing a parser as well as of an evaluation tool can be carried out with reasonable effort. In addition, a large number of quality measures have been developed and used actively across the annotation research community to evaluate the same task, leading to the results across publications on the same topics not being easily comparable. For example, while some authors publish macro-F-measures and simply call them F-measures, others publish micro-F-measures for the same purpose, leading to significant discrepancies across the scores. The same holds for the evaluation of how well entities match. Indeed, partial matches and complete matches have been used in previous evaluations of annotation tools  [7, 39].
This heterogeneous landscape of tools, datasets and measures leads to a poor repeatability of experiments, which makes the evaluation of the real performance of novel approaches against the state of the art rather difficult. The insights above have led to a movement towards the creation of frameworks to ease the evaluation of solutions that address the same annotation problem  [5, 7]. In this paper, we present GERBIL -a general entity annotator benchmark -, a community-driven effort to enable the continuous evaluation of annotation tools. GERBIL is an open-source and extensible framework that allows evaluating tools against (currently) 9 different annotators on 11 different datasets within 6 different experiment types. By integrating such a large number of datasets, experiment types and frameworks, GERBIL allows users to evaluate their tools against other semantic entity annotation systems (short: entity annotation systems) by using exactly the same setting, leading to fair comparisons based on exactly the same measures.
While the evaluation core of GERBIL is based on the BAT-framework  [7] , our approach goes beyond the state of the art in several respects: • GERBIL provides persistent URLs for experimental settings. Hence, by using GERBIL for experiments, tool developers can ensure that the settings for their experiments (measures, datasets, versions of the reference frameworks, etc. ) can be reconstructed in a unique manner in future works. • Through experiment URLs, GERBIL also addresses the problem of archiving experimental results and allows end users to gather all pieces of information required to choose annotation frameworks for practical applications. • GERBIL aims to be a central repository for annotation results without being a central point of failure: While we make experiment URLs available, we also provide users directly with their results to ensure that they use them locally without having to rely on GERBIL.
• The results of GERBIL are published in a machinereadable format. In particular, our use of DataID  [2]  and DataCube  [9]  to denote tools and datasets ensures that results can be easily combined and queried (for example to study the evolution of the performance of frameworks) while the exact configuration of the experiments remains uniquely reconstructable. By these means, we also tackle the problem of reproducibility. • Through the provision of results on different datasets of different types and the provision of results on a simple user interface, GERBIL also provides means to quickly gain an overview of the current performance of annotation tools, thus providing (1) developers with insights pertaining to the type of data on which their accuracy needs improvement and (2) end users with insights allowing them to choose the right tool for the tasks at hand.
• With GERBIL we introduce the notion of knowledge base-agnostic benchmarking of entity annotation systems through generalized experiment types. By these means, we allow benchmarking tools against reference datasets from any domain grounded in any reference knowledge base. To ensure that the GERBIL framework is useful to both end users and tool developers, its architecture and interface were designed with the following principles in mind: • Easy integration of annotators: We provide a wrapping interface that allows annotators to be evaluated via their REST interface. In particular, we integrated 6 additional annotators not evaluated against each other in previous works (e. g. ,  [7] ). • Easy integration of datasets: We also provide means to gather datasets for evaluation directly from data services such as DataHub.
1 In particular, we added 6 new datasets to GERBIL. 1 http://datahub. io • Easy addition of new measures: The evaluation measures used by GERBIL are implemented as interfaces. Thus, the framework can be easily extended with novel measures devised by the annotation community. • Extensibility: GERBIL is provided as an open-source platform 2  that can be extended by members of the community both to new tasks and different purposes. • Diagnostics: The interface of the tool was designed to provide developers with means to easily detect aspects in which their tool(s) need(s) to be improved. • Portability of results: We generate human-and machine-readable results to ensure maximum usefulness and portability of the results generated by our framework. In the rest of this paper, we present and evaluate GERBIL.
We begin by giving an overview of related work. Thereafter, we present the GERBIL framework. We focus in particular on how annotators and datasets can be added to GERBIL and give a short overview of the annotators and tools that are currently included in the framework. We then present an evaluation of the framework that aims to quantify the effort necessary to include novel annotators and datasets to the framework. We conclude with a discussion of the current state of GERBIL and a presentation of future work. More information can be found at our project webpage http:// gerbil. aksw. org and at the code repository page https: //github. com/AKSW/gerbil. The online version of GERBIL can be accessed at http://gerbil. aksw. org/gerbil.
RELATED WORK Named Entity Recognition and Entity Linking have gained significant momentum with the growth of Linked Data and structured knowledge bases. Over the last few years, the problem of result comparability has thus led to the development of a hand full of frameworks. The BAT-framework  [7]  is designed to facilitate the benchmarking of named entity recognition (NER), named entity disambiguation (NED) -also known as linking (NEL) -and concept tagging approaches. BAT compares seven existing entity annotation approaches using Wikipedia as reference. Moreover, it defines six different task types, five different matchings and six evaluation measures providing five datasets. Rizzo et al. [35]  present a state-of-the-art study of NER and NEL systems for annotating newswire and micropost documents using well-known benchmark datasets, namely CoNLL2003 and Microposts 2013 for NER as well as AIDA/CoNLL and Microposts2014  [3]  for NED.
The authors propose a common schema, named the NERD ontology  3  , to align the different taxonomies used by various extractors. To tackle the disambiguation ambiguity, they propose a method to identify the closest DBpedia resource by (exact-)matching the entity mention. Over the course of the last 25 years several challenges, workshops and conference dedicated themselves to the compara-ble evaluation of information extraction (IE) systems. Starting in 1993, the Message Understanding Conference (MUC) introduced a first systematic comparison of information extraction approaches  [42]. Ten years later, the Conference on Computational Natural Language Learning (CoNLL) started to offer a shared task on named entity recognition and published the CoNLL corpus  [43]. In addition, the Automatic Content Extraction (ACE) challenge  [10] , organized by NIST, evaluated several approaches but was discontinued in 2008.
Since 2009, the text analytics conference hosts the workshop on knowledge base population (TAC-KBP)  [22]  where mainly linguistic-based approaches are published. The Senseval challenge, originally concerned with classical NLP disciplines, has wided it focus in 2007 and changed its name to SemEval to account for the recently recognized impact of semantic technologies  [19]. The Making Sense of Microposts workshop series (MSM) established in 2013 an entity recognition and in 2014 an entity linking challenge thereby focusing on tweets and microposts  [37]. In 2014, Carmel et al. [5]  introduced one of the first Web-based evaluation systems for NER and NED and the centerpiece of the entity recognition and disambiguation (ERD) challenge. Here, all frameworks are evaluated against the same unseen dataset and provided with corresponding results.
GERBIL goes beyond the state of the art by extending the BAT-framework as well as  [35]  in several dimensions to enhance reproducibility, diagnostics and publishability of entity annotation systems. In particular, we provide 6 additional datasets and 6 additional annotators. The framework addresses the lack of treatment of NIL values within the BAT-framework and provides more wrapping approaches for annotators and datasets. Moreover, GERBIL provides persistent URLs for experiment results, unique URIs for frameworks and datasets, a machine-readable output and automatic dataset updates from data portals. Thus, it allows for a holistic comparison of existing annotators while simplifying the archiving of experimental results. Moreover, our framework offers opportunities for the fast and simple evaluation of entity annotation system prototypes via novel NIF-based  [15]  interfaces, which are designed to simplify the exchange of data and binding of services.
Architecture Overview GERBIL abides by a service-oriented architecture driven by the model-view-controller pattern (see Figure  1 ). Entity annotation systems, datasets and configurations like experiment type, matching or metric are implemented as controller interfaces easily pluggable to the core controller. The output of experiments as well as descriptions of the various components are stored in a serverless database for fast deployment. Finally, the view component displays configuration options respectively renders experiment results delivered by the main controller communication with the diverse interfaces and the database.
This work differs from the previous one for experimenting in entity recognition, and on annotating entities to a RDF knowledge base. GERBIL reuses the six experiments provided by the BATframework and extends them by the idea to not only link to Wikipedia but to any knowledge base K. One major formal update of the measures in GERBIL is that in addition to implementing experiment types from previous frameworks, it also measures the influence of NIL annotations, i. e. , the linking of entities that are recognized as such but cannot be linked to any resource from the reference knowledge base K. For example, the string Ricardo Usbeck can be recognized as a person name by several tools but cannot be linked to Wikipedia/DBpedia, as Ricardo does not have a URI in these reference datasets. Our framework extends the experiments types of  [7]   We implement six types of experiments: 1.
C2KB: The concept tagging task C2KB aims to detect entities when given a document. Formally, the tagging  6. Rc2KB: In this particular extension of C2KB, the tagging function returns a sorted list of resources from K, i. e. , an element of K * , where K * = ∞ i=0 K i. With this extension, our framework can now deal with gold standard datasets and annotators that link to any knowledge base, e. g. , DBpedia, BabelNet  [30]  etc. , as long as the necessary identifiers are URIs. We were thus able to implement 6 new gold standard datasets usable for each experiment type, cf. Section 3. 3, and 6 new annotators linking entities to any knowledge base instead of solely Wikipedia like in previous works, cf. Section 3. 2. 4. With this extensible interface, GERBIL can be extended to deal with supplementary experiment types, e.
Metrics Currently, GERBIL offers six measures subdivided into two groups and derived from the BAT-framework, namely the micro-and the macro-group of precision, recall and f-measure. At the moment, those measures ignore NIL annotations, i. e. , if a gold standard dataset contains entities that are not contained in the target knowledge base K and an annotator detects the entity and links it to any URI, emerging novel URI or NIL, this will always result in a false-positive evaluation. To alleviate this problem, GERBIL allows adding additional measures to evaluate the results of annotators regarding the heterogeneous landscape of gold standard datasets.
Currently, GERBIL offers 9 entity annotation systems with a variety of features, capabilities and experiments. In the following, we present current state-of-the-art approaches both available or unavailable in GERBIL. 1. Cucerzan: As early as in 2007, Cucerzan presented a NED approach based on Wikipedia  [8]. The approach tries to maximize the agreement between contextual information of input text and a Wikipedia page as well as category tags on the Wikipedia pages. The test data is still available 6  but since we can safely assume that the Wikipedia page content changed a lot since 2006, we do not use it in our framework, nor we are aware of any publication reusing this data. Furthermore, we were not able to find a running webservice or source code for this approach. 2. Wikipedia Miner: This approach was introduced in  [25]  in 2008 and is based on different facts like prior probabilities, context relatedness and quality, which are then combined and tuned using a classifier.
The authors evaluated their approach based on a subset of the AQUAINT dataset  7. They provide the source code for their approach as well as a webservice  8  which is available in GERBIL. 3. Illinois Wikifier: In 2011,  [34]  presented an NED approach for entities from Wikipedia. In this article, the authors compare local approaches, e. g. , using string similarity, with global approaches, which use context information and lead finally to better results. The authors provide their datasets 9  as well as their software "Illionois Wikifier" 10  online. Since "Illionois Wikifier" is currently only available as local binary and GER-BIL is solely based on webservices we excluded it from GERBIL for the sake of comparability and server load. 4. DBpedia Spotlight: One of the first semantic approaches  [24]  was published in 2011, this framework combines NER and NED approach based upon DBpedia  11.
Based on a vector-space representation of entities and using the cosine similarity, this approach has a public (NIF-based) webservice  12  as well as its online available evaluation dataset 13. 5. TagMe 2: TagMe 2  [13]  was publised in 2012 and is based on a directory of links, pages and an inlink graph from Wikipedia. The approach recognizes named entities by matching terms with Wikipedia link texts and disambiguates the match using the in-link graph and the page dataset. Afterwards, TagMe 2 prunes identified named entities which are considered as noncoherent to the rest of the named entities in the input text. The authors publish a key-protected webservice  14  as well as their datasets  15  online. The source code, licensed under Apache 2 licence can be obtained directly from the authors.
The datasets comprise only fragments of 30 words and less of full documents and will not be part of the current version of GERBIL. 6. AIDA: The AIDA approach  [17]  relies on coherence graph building and dense subgraph algorithms and is based on the YAGO2 16  knowledge base. Although the authors provide their source code, a webservice and their dataset which is a manually annotated subset of the 2003 CoNLL share task  [43] , GERBIL will not use the webservice since it is not stable enough for regular replication purposes. 17. NERD-ML: In 2013,  [11]  proposed an approach for entity recognition tailored for extracting entities from tweets. The approach relies on a machine learning classification of the entity type given a rich feature vector composed of a set of linguistic features, the output of a properly trained Conditional Random Fields classifier and the output of a set of off-the-shelf NER extractors supported by the NERD Framework.
The authors published datasets 21 along with their source code and an API 22. AGDISTIS can only be used for the D2KB task. 11. Babelfy: The core of this approach lies in the use of random walks and a densest subgraph algorithm to tackle the word sense disambiguation and entity linking tasks in a multilingual setting  [27]  thanks to the BabelNet semantic network  [30]. Babelfy has been evaluated using six datasets: three from earlier Se-mEval tasks  [33, 29, 28] , one from a Senseval task  [38]  and two already used for evaluating AIDA  [17, 16]. All of them are available online but distributed throughout the web. Additionally, the authors offer a webservice limited to 100 requests per day which are extensible for research purposes 23  [26]. 12. Dexter: This approach  [6]  is an open-source implementation of an entity disambiguation framework.
The system was implemented in order to simplify the implementation of an entity linking approach and allows to replace single parts of the process. The authors implemented several state-of-the-art disambiguation methods. Results in this paper are obtained using an implementation of the original TagMe disambiguation function. Moreover, Ceccarelli et al. provide the source code 24 as well as a webservice. Table  1  compares the implemented annotation systems of GERBIL and the BAT-Framework. While AGDISTIS has been in the source code of the BAT-Framework provided by a third-party after publication of Cornolti et al. 's initial work  [7]  in 2014, GERBIL's community effort led to the implementation of overall 6 new annotators as well as the before mentioned generic NIF-based annotator. The AIDA annotator as well as the "Illinois Wikifier" will not be available in GERBIL since we restrict ourselves to webservices.
Datasets Table  2  shows the heterogeneity of datasets used for prior evaluations while Table  3  presents an overview of the datasets that were used to evaluate some well-known entity annotators in previous works. These tables make clear that the numbers and types of used datasets varies a lot, thus preventing a fast comparison of annotation systems. BAT allows evaluating the performance of different approaches using five datasets, namely AQUAINT, MSNBC, IITB, Meij and AIDA/CoNLL. With GERBIL, we activate one more dataset already implemented by the authors, namely ACE2004 from Ratinov et al. [34]. Furthermore, we implemented a dataset wrapper for the Microposts2014 corpus which has been used to evaluate NERD-ML  [35]. The dataset itself was introduced in 2014  [3]  and consists of 3500 tweets especially related to event data.
Moreover, we capitalize upon the uptake of publicly available, NIF based corpora over the last years  [40, 36]  25. To this end, GERBIL implements a Java-based NIF  [15]  reader and writer module which enables loading arbitrary NIF document collections, as well as the communication to NIF-based webservices. Additionally, we integrated four NIF corpora, i. e. , the RSS-500 and reuters-128 dataset 26 , as well as the Spotlight Corpus and the KORE 50 dataset 27. The extensibility of the datasets in GERBIL is furthermore ensured by allowing users to upload or use already available NIF datasets from DataHub. GERBIL will regularly check whether new corpora are available and publish them for benchmarking after a manual quality assurance cycle which ensures their usability for the implemented configuration options.
Additionally, users can upload their NIFcorpora directly to GERBIL avoiding their publication in publicly available sources. This option allows for rapid testing of entity annotation systems with closed source or licenced datasets. implemented due to size and server load limitations, i. e. , Wiki-Disamb30 and Wiki-Annot30, or due their original experiment type. In particular, the Senseval-3 as well as the different SemEval datasets demand as experiment type word sense disambiguation and thereby linking to BabelNet or Wordnet  [12] , which is not yet covered in GERBIL. Still, GERBIL offers currently 11 state-of-the-art datasets reaching from newswire and twitter to encyclopedic corpora of various amounts of texts and entities. Due to license issues we are only able to provide downloads for 9 of them directly but we provide instructions to obtain the others on our project wiki.
GER-BIL Experiment  [25]  Wikipedia Miner SA2KB [34] Illionois Wikifier () SA2KB [24] Spotlight SA2KB  [13]   persistent and time-stamped URL for each experiment, see Table  5. RDF DataCube is a vocabulary standard and can be used to represent fine-grained multidimensional, statistical data which is compatible with the Linked SDMX  [4]  standard. Every GERBIL experiment is modelled as qb:Dataset containing the individual runs of the annotators on specific corpora as qb:Observations. Each observation features the qb:Dimensions experiment type, matching type, annotator, corpus and time. The six evaluation measures offered by GERBIL as well as the error count are expressed as qb:Measures. To include further metadata, annotator and corpus dimension properties link DataID  [2]  descriptions of the individual components.
EVALUATION To ensure the practicability and convenience of the GERBIL framework, we investigated the effort needed to use GERBIL for the evaluation of novel annotators. To achieve this goal, we surveyed the workload necessary to implement a novel annotator into GERBIL compared to the implementation into previous diverse frameworks. Our survey comprised five developers with expert-level programming skills in Java. Each developer was asked to evaluate how much time he/she needed to write the code necessary to evaluate his/her framework on a new dataset. Overall, the developers reported that they needed between 1 29 http://gerbil. aksw. org/gerbil/overview   3. Importantly, all developers reported that they needed either the same or even less time to integrate their annotator into GERBIL. This result in itself is of high practical significance as it means that by using GERBIL, developers can evaluate on (currently) 11 datasets using the same effort they needed for 1, which is a gain of more than 1100%.
Moreover, all developers reported they felt comfortable-4 points on average on a 5-point Likert scale between very uncomfortable (1) and very comfortable (5)-implementing the annotator in GERBIL. Further developers were invited to complete the survey, which is available at our project website. Even though small, this evaluation suggests that implementing against GERBIL does not lead to any overhead. On the contrary, GERBIL significantly improves the timeto-evaluation by offering means to benchmark and compare against other annotators respectively datasets within the same effort frame previously required to evaluate on a single dataset. An interesting side-effect of having all these frameworks and datasets in a central framework is that we can now benchmark the different frameworks with respect to their runtimes within exactly the same experimental settings.
CONCLUSION AND FUTURE WORK In this paper, we presented and evaluated GERBIL, a platform for the evaluation of annotation frameworks. With GERBIL, we aim to push annotation system developers to better quality and wider use of their frameworks. Some of the main contributions of GERBIL include the provision of persistent URLs for reproducibility and archiving. Furthermore, we implemented a generic adaptor for external datasets as well as a generic interface to integrate remote annotator systems. The datasets available for evaluation in the previous benchmarking platforms for annotation was extended by 6 new datasets. Moreover, 6 novel annotators were added to the platform. The evaluation of our framework by contributors suggests that adding an annotator to GERBIL demands 1 to 2 hours of work.