dataesr/scientific-paragraphs-categorization huggingface dataset:
Hence, while keeping the implementation effort previously required to evaluate on a single dataset, we allow developers to evaluate on (currently) 11 times more datasets. The presented, web-based frontend for several use cases enabling laymen and expert users to perform informed comparisons of semantic annotation tools. The persistent URIs enhances the long term quotation in the field of information extraction. GER-BIL is not just a new framework wrapping existing technology. In comparison to earlier frameworks, it extends the state-of-the-art benchmarks by the capability of considering the influence of NIL attributes and the ability of dealing with data sets and annotators that link to different knowledge bases. More information about GERBIL and its source code can be found at the project's website.
Datasets and their formats. A indicates various inline or keyfile annotation formats. The experiments follow their definition in Section 3. 2 rect access to the evaluation corpora. Furthermore, ODRL license specifications in RDF are linked via dc:license, potentially facilitating automatically adjusted processing of licensed data by NLP tools. Licenses are further specified via dc:rights, including citations of the relevant publications. To describe annotators in a similar fashion, we extended DataID for services. The class Service, to be described with the same basic properties as dataset, was introduced. To link an instance of a Service to its distribution the datid:distribution property was introduced as super property of dcat:distribution, i. e. , the specific URI the service can be queried at.
Furthermore, Services can have a number of datid:Parameters and datid:Configurations. Datasets can be linked via datid:input or datid:output. Offering such detailed and structured experimental results opens new research avenues in terms of tool and dataset di- agnostics to increase decision makers' ability to choose the right settings for the right use case. Next to individual con- figurable experiments, GERBIL offers an overview of recent experiment results belonging to the same experiment and.
CIDOC-CRM is the ISO 21127:2014 formal model for describing cultural heritage developed by CIDOC, the International Committee for Documentation of the International Council of Museums (ICOM). CRM stands for Conceptual Reference Model. CRMinf stands for CRM Inference and is the acronym for the argumentation model. The outcome is a highlevel, generic process model that documents the reasoning behind such a process and allows it to be linked both to its digital outcome and to elements of an information system storing data regarding the site or monument under reconstruction.
Process-centered provenance tracking takes into consideration the mapping of a series of statements that explain the transition from the premises to the conclusion of an argument. It is used to show the differences and choices in data creation and elaboration. This kind of information helps to define the value of the data and can be used to enable the reuse of reasoning chains, both from a legal and from a scientific perspective, in addition to being helpful in identifying the context of information as a crucial aspect in the preservation of the identity of a dataset  (Doerr & Tzitzikas, 2012). The formalisation of process-centered provenance has been underway for the last thirty years. One of the first contemporary examples is IBIS (Issue-Based Information System)  (Kunz & Rittel, 1970) , an argumentation model developed to support planning in the political process.
Numerical Results All experiments are run under MATLAB R2012b on a Windows 7 operating system with i7 CPU 2GHz and 4GB of RAM. The computations are performed using CVX 2. 1 with semidefinite program solver SeDuMi. Tables  [1]  and [2 ] provide the computational results of Data 1 and Data 2 respectively. N presents the sampling number. k is the number of removal constraints and k/N is the corresponding ratio. We use the average linear size measure, which is defined as ALS(E(Z)) = (Vol n (E(Z)) 1/n )  (Cheung et al. , 2012) , to evaluate the volume of ellipsoid. The smaller the volume of ellipsoid is, the smaller the average linear size of ellipsoid is. The columns SC, Greedy, BMSP give the average linear size of ellipsoid obtained by scenario approach (4. 2. 1), greedy approach (4. 2. 2) and our method (4.
2013; Lee et al. 2010; Herron 2012; Allesina 2012; Day 2015; Park et al. 2014; Thurner and Hanel 2011). We aimed to develop an ABM, by adopting a unified view of peer review and publication systems. We attempted to embrace the full complexity of the scientific publication system and use empirical data for calibration. We simulated all the interactions between authors, reviewers and editors and took into account the complete path of scientific manuscripts from submission to the final decision, including resubmissions, rejections after in-house review (without external peer review) and multiple rounds of peer review. We implemented the model in the biomedical and life sciences domain and used empirical data from medical journals for calibration. Our results closely match the real life situation.
Collection of empirical data To guide the development and parameterization of the ABM and to perform calibration procedures, we collected empirical data pertaining to the medical domain. We contacted a sample of medical journals to obtain their acceptance rates, with a 40 % positive response rate. We consulted journal websites to obtain the remaining acceptance rates (when available). Finally, we collected the journal impact factors from Journal Citation Reports 2013 and the remaining data from a previously published international survey  (Mulligan et al. 2013).
Characterization of journals: survey of editors Among 119 journals indexed in the MEDLINE Core Clinical Journals subset, we surveyed 105. We excluded journals that invited only submissions (n = 11) and those no longer active (n = 2); a journal's special edition was considered among the regular issues. We searched the website for each journal for the contact details of the editor-in-chief or editorial office. On December 5, 2014 we sent an email asking for the number of manuscripts submitted to the journal in 2013, number of manuscripts rejected after in-house review (without external peer review) and number of articles published in 2014. We sent 2 reminders on December 12 and January 22, 2015 and closed our survey on February 1, 2015. We masked the data so that journals could not be matched to their acceptance rates.
Calibration procedures and main outputs of the model implementation We programmed the model using MATLAB (MATLAB and Statistics Toolbox Release 2014b, The MathWorks, Inc. , Natick, Massachusetts, United States). The code is available at http://www. clinicalepidemio. fr/peerreview_abm/. We programmed the model with a total population of researchers N = 25,000 and total population of journals J = 105. We ran the simulations for 10 years, with a burn-in period of 1 year for the initialization of the model. Results were averaged over 20 simulations. The main outputs measured were total number of publications per year, proportion of successfully published articles compared to all submissions, proportion of manuscripts revised before being published and proportion of manuscripts for which the peer review process improved their Q score after revision.
8 is a random number drawn from a uniform probability distribution. We randomly selected five journals that sent everything for peer review (excluding the top 10 % journals with the highest reputation). This process resulted in uniformly distributed rejection rates after in-house review that match the empirical data as seen in Fig. 4b  (mean value [0. 32 ± 0. 25] and Anderson-Darling p values [0. 068, 0. 152]).
Discussion Our ABM mimics the properties and functions and addresses different scenarios of behavior and interactions in the scientific publication system. The main strengths of our model are the use of empirical data, which allowed us to produce realistic outputs, and the unified view of evaluation and publishing systems. The main difficulty in the calibration was that we had to reproduce the whole journey of a manuscript from its submission to publication or until the authors give up on submitting it. From empirical data from the biomedical and life sciences domain, we calibrated the model so that the journals do not accept or reject too many manuscripts and so that the manuscripts are not resubmitted more than is required, before being published. We obtained an amount of publications very close to that estimated from our survey.
Acknowledgments Michail Kovanis is the recipient of a PhD grant from  Sorbonne Paris Cite. We would like to thank all the editors-in-chief and technical editors who provided us with data for using in calibrating this model. We thank  Prof Sandro Galea  ( Boston University, School of Public Health ) for helpful comments on earlier drafts of the manuscript. We thank  Elise Diard  ( Cochrane France ) for her help with the complex system diagram and  Laura Smales  ( BioMedEditing, Toronto, ON ) for copyediting the manuscript. We would also like to thank the two anonymous reviewers for their useful comments and observations. Open Access This article is distributed under the terms of the Creative Commons Attribution 4. 0 International License (http://creativecommons. org/licenses/by/4. 0/),which permits unrestricted use, distribution, and reproduction in any medium, provided you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made.
Linked Open Data (LOD) has emerged as one of the largest collections of interlinked datasets on the web. In order to benefit from this mine of data, one needs to access to descriptive information about each dataset (or metadata). This information can be used to delay data entropy, enhance dataset discovery, exploration and reuse as well as helping data portal administrators in detecting and eliminating spam. However, such metadata information is currently very limited to a few data portals where they are usually provided manually, thus being often incomplete and inconsistent in terms of quality. To address these issues, we propose a scalable automatic approach for extracting, validating, correcting and generating descriptive linked dataset profiles. This approach applies several techniques in order to check the validity of the metadata provided and to generate descriptive and statistical information for a particular dataset or for an entire data portal.
Introduction From 12 datasets cataloged in 2007, the Linked Open Data cloud has grown to nearly 1000 datasets containing more than 82 billion triples 1  [4]. Data is being published by both the public and private sectors and covers a diverse set of domains from life sciences to media or government data. The Linked Open Data cloud is potentially a gold mine for organizations and individuals who are trying to leverage external data sources in order to produce more informed business decisions  [8]. Dataset discovery can be done through public data portals like Datahub. io and publicdata. eu or private ones like quandl. com and enigma. io. Private portals harness manually curated data from various sources and expose them to users either freely or through paid plans. Similarly, in some public data portals, administrators manually review datasets information, validate, correct and attach suitable metadata information.
This information is mainly in the form of predefined tags such as media, geography, life sciences for organization and clustering purposes. However, the diversity of those datasets makes it harder to classify them in a fixed number of predefined tags that can be subjectively assigned without capturing the essence and breadth of the dataset  [21]. Furthermore, the increasing number of datasets available makes the metadata review and curation process unsustainable even when outsourced to communities. There are several Data Management Systems (DMS) that power public data portals. CKAN 2  is the world's leading open-source data portal platform powering web sites like DataHub, Europe's Public Data and the U. S Government's open data. Modeled on CKAN, DKAN 3  is a standalone Drupal distribution that is used in various public data portals as well.
Socrata 4  helps public sector organizations improve data-driven decision making by providing a set of solutions including an open data portal. In addition to these tradition data portals, there is a set of tools that allow exposing data directly as RESTful APIs like thedatatank. com. Metadata provisioning is one of the Linked Data publishing best practices mentioned in  [3]. Datasets should contain the metadata needed to effectively understand and use them. This information includes the dataset's license, provenance, context, structure and accessibility. The ability to automatically check this metadata helps in: -Delaying data entropy: Information entropy refers to the degradation or loss limiting the information content in raw or metadata. As a consequence of information entropy, data complexity and dynamicity, the life span of data can be very short.
Even when the raw data is properly maintained, it is often rendered useless when the attached metadata is missing, incomplete or unavailable. Comprehensive high quality metadata can counteract these factors and increase dataset longevity  [20]. -Enhancing data discovery, exploration and reuse: Users who are unfamiliar with a dataset require detailed metadata to interpret and analyze accurately unfamiliar datasets. A study conducted by the European Union commission  [29]  found that both business and users are facing difficulties in discovering, exploring and reusing public data due to missing or inconsistent metadata information. -Enhancing spam detection: Portals hosting public open data like Datahub allow anyone to freely publish datasets. Even with security measures like captchas and anti-spam devices, detecting spam is increasingly difficult.
In addition to that, the increasing number of datasets hinders the scalability of this process, affecting the correct and efficient spotting of datasets spam. Data profiling is the process of creating descriptive information and collect statistics about that data. It is a cardinal activity when facing an unfamiliar dataset  [24]. Data profiles reflect the importance of datasets without the need for detailed inspection of the raw data. It also helps in assessing the importance of the dataset, improving users' ability to search and reuse part of the dataset and in detecting irregularities to improve its quality. Data profiling includes typically several tasks: -Metadata profiling: Provides general information on the dataset (dataset description, release and update dates), legal information (license information, openness), practical information (access points, data dumps), etc.
-Statistical profiling: Provides statistical information about data types and patterns in the dataset (e. g. properties distribution, number of entities and RDF triples). -Topical profiling: Provides descriptive knowledge on the dataset content and structure. This can be in form of tags and categories used to facilitate search and reuse. In this work, we address the challenges of automatic validation and generation of descriptive dataset profile. This paper proposes Roomba, an extensible framework consisting of a processing pipeline that combines techniques for data portals identification, datasets crawling and a set of pluggable modules combining several profiling tasks. The framework validates the provided dataset metadata against an aggregated standard set of information. Metadata fields are automatically corrected when possible (e.
g. adding a missing license URL reference). Moreover, a report describing all the issues highlighting those that cannot be automatically fixed is created to be sent by email to the dataset's maintainer. There exist various statistical and topical profiling tools for both relational and Linked Data. The architecture of the framework allows to easily add them as additional profiling tasks. However, in this paper, we focus on the task of dataset metadata profiling. We validate our framework against a manually created set of profiles and manually check its accuracy by examining the results of running it on various CKAN-based data portals. The remainder of the paper is structured as follows. In Sect. 2, we review relevant related work. In Sect. 3, we describe our proposed framework's architecture and components that validate and generate dataset profiles.
Related Work Data Catalog Vocabulary (DCAT)  [25]  and the Vocabulary of Interlinked Datasets (VoID)  [11]  are concerned with metadata about RDF datasets. There exist several tools aiming at exposing dataset metadata using these vocabularies. In  [6] , the authors generate VoID descriptions limited to a subset of properties that can be automatically deduced from resources within the dataset. However, it still provides data consumers with interesting insights. Flemming's Data Quality Assessment Tool 5 provides basic metadata assessment as it computes data quality scores based on manual user input. The user assigns weights to the predefined quality metrics and answers a series of questions regarding the dataset. These include, for example, the use of obsolete classes and properties by defining the number of described entities that are assigned disjoint classes, the usage of stable URIs and whether the publisher provides a mailing list for the dataset.
However, deep schema information about the metadata is missing like description, license information or tags. Similarly on the LOD cloud, the Datahub LOD Validator 8  gives an overview of Linked Data sources cataloged on the Datahub. It offers a stepby-step validator guidance to check a dataset completeness level for inclusion in the LOD cloud. The results are divided into four different compliance levels from basic to reviewed and included in the LOD cloud. Although it is an excellent tool to monitor LOD compliance, it still lacks the ability to give detailed insights about the completeness of the metadata and overview on the state of the entire LOD cloud group and it is very specific to the LOD cloud group rules and regulations. Statistical Profiling: Calculating statistical information on datasets is vital to applications dealing with query optimization and answering, data cleansing, schema induction and data mining  [14, 17, 21].
Semantic sitemaps  [10]  and RDF-Stats  [22]  are one of the first to deal with RDF data statistics and summaries. ExpLOD  [19]  creates statistics on the interlinking between datasets based on owl:sameAs links. In  [24] , the author introduces a tool that induces the actual schema of the data and gather corresponding statistics accordingly. LODStats  [2]  is a stream-based approach that calculates more general dataset statistics. Pro-LOD++  [1]  is a Web-based tool that allows LOD analysis via automatically computed hierarchical clustering  [7]. Aether  [26]  generates VoID statistical descriptions of RDF datasets. It also provides a Web interface to view and compare VoID descriptions. LODOP  [13]  is a MapReduce framework to compute, optimize and benchmark dataset profiles. The main target for this framework is to optimize the runtime costs for Linked Data profiling.
In  [18]  authors calculate certain statistical information for the purpose of observing the dynamic changes in datasets. Topical Profiling: Topical and categorical information facilitates dataset search and reuse. Topical profiling focuses on content-wise analysis at the instances and ontological levels. GERBIL  [28]  is a general entity annotation framework that provides machine processable output allowing efficient querying. In addition, there exist several entity annotation tools and frameworks  [9]  but none of those systems are designed specifically for dataset annotation. In  [15] , the authors created a semantic portal to manually annotate and publish metadata about both LOD and non-RDF datasets. In  [21] , the authors automatically assigned Freebase domains to extracted instance labels of some of the LOD Cloud datasets.
The goal was to provide automatic domain identification, thus enabling improving datasets clustering and categorization. In  [5] , the authors extracted dataset topics by exploiting the graph structure and ontological information, thus removing the dependency on textual labels. In  [12] , the authors generate VoID and VoL descriptions via a processing pipeline that extracts dataset topic models ranked on graphical models of selected DBpedia categories. Although the above mentioned tools are able to provide various types of information about a dataset, there exists no approach that aggregates this information and is extensible to combine additional profiling tasks. To the best of our knowledge, this is the first effort towards extensible automatic validation and generation of descriptive dataset profiles.
Profiling Data Portals In this section, we provide an overview of Roomba's architecture and the processing steps for validating and generating dataset profiles. Figure  1  shows the main steps which are the following: (i) data portal identification; (ii) metadata extraction; (iii) instance and resource extraction; (iv) profile validation (v) profile and report generation. Roomba is built as a Command Line Interface (CLI) application using Node. js. Instructions on installing and running the framework are available on its public Github repository  9. The various steps are explained in detail below.
Data Portal Identification Roomba should be extensible to any data portal that exposes its functionalities via an external accessible API. Since every portal ca have its own data model, identifying the software powering data portals is a vital first step. We rely on several Web scraping techniques in the identification process which includes a combination of the following: -URL inspection: Various CKAN based portals are hosted on subdomains of the http://ckan. net. For example, CKAN Brazil (http://br. ckan. net). Checking the existence of certain URL patterns can detect such cases. The identification process for each portal can be easily customized by overriding the default function. Moreover, adding or removing steps from the identification process can be easily configured. After those preliminary checks, we query one of the portal's API endpoints.
Metadata Extraction Data portals expose a set of information about each dataset as metadata. The model used varies across portals. However, a standard model should contain information about the dataset's title, description, maintainer email, update and creation date, etc. We divided the metadata information into the following types: General Information: General information about the dataset. e. g. , title, description, ID, etc. This general information is manually filled by the dataset owner. In addition to that, tags and group information is required for classification and enhancing dataset discoverability. This information can be entered manually or inferred modules plugged into the topical profiler. Access Information: Information about accessing and using the dataset. This includes the dataset URL, license information i.
e. , license title and URL and information about the dataset's resources. Each resource has as well a set of attached metadata e. g. , resource name, URL, format, size. Ownership Information: Information about the ownership of the dataset. e. g. , organization details, maintainer details, author. The existence of this information is important to identify the authority on which the generated report and the newly corrected profile will be sent to. Provenance Information: Temporal and historical information on the dataset and its resources. For example, creation and update dates, version information, version, etc. Most of this information can be automatically filled and tracked. Building a standard metadata model is not the scope of this paper, and since we focus on CKAN-based portals, we validate the extracted metadata against the CKAN standard model 10.