dataesr/scientific-paragraphs-categorization huggingface dataset:
Ontology-based Checker Module In this phase, the CS-KG ontology is used to integrate entities from different tools and discard triples that do not comply with domain and range of the relations. All triples of the set T are then represented according to the CS-KG ontology. The types of entities returned by the DyGIEpp tool are mapped to the relevant classes in the ontology. Specifically, methods, tasks, materials, and metrics are mapped to the homonymous classes in the ontology (e. g. , material is mapped to the class cskg-ont:Material ), while other-scientific-terms and generic entities are mapped to cskg-ont:OtherEntity. The predicates are mapped to the ontology object properties. For instance, <cskg:semantic_interoperability, uses, cskg:ontology_matching>, considering that <cskg:ontology_matching, rdf:type, cskg-ont:Task>, becomes <cskg:semantic_interoperability, cskgont:usesTask, cskg:ontology_matching>.
In this phase, triples which do not comply with the semantics of the ontology are discarded. For example the triple <cskg:utk_face_dataset, uses, cskg:deep_learning>, where cskg:utk_face_dataset is a cskg-ont:Material and cskg:deep_learning is a cskg-ont:Method, is discarded because the class cskgont:Material is not in the domain of the property cskg-ont:usesMethod.
76. This demonstrates how the checker modules (described in Sections 4. 3 and 4. 4) are able to increase significantly the accuracy of the statements (+21% in precision), paying a relatively low price in recall. An inspection of the results shows also that 86% of the statements with support greater than 5 are correct, consistently with the intuition that support is an indicator of a triple correctness. The method which aggregates all the basic tools (DyGIEpp+OpenIE+PoST) performs second best (0. 69), highlighting the value of an hybrid approach that combines both unsupervised and supervised methods. Finally, DyGIEpp, OpenIE, and PoST obtain f-measures in the 0. 47-0. 51 range. Among them, DyGIEpp has the highest precision (0. 67), while PoST has the highest recall (0. 46). In summary, the evaluation suggests that i) CS-KG offers good quality statements, in particular when associated to a good support, ii) the performance of each of the three tools is unsatisfactory and, therefore, it is worth to produce a pipeline that is able to combine them, and iii) the components of the CS-KG pipeline used to discriminate valid statements (i.
DyGIEpp [43]. This tool extracts a set of entities E Dy of six pre-defined types (Method, Task, Material, Metric, Other-Scientific-Term, and Generic) and seven kinds of relationship (Used-for, Hyponym-Of, Compare, Part-of, Conjunction, Feature-of , Evaluate-for ). It is used to yield a set of entities E Dy and a set of triples among them, T Dy. -Computer Science Ontology Classifier (CSO-C) [34]. CSO-C is a classifier which exploits syntactic and semantic similarity to map text spans to topics in CSO. It extracts the set of entities E CSO. -OpenIE of the Stanford Core NLP suite [1]. This tool is used to extract open domain relationships from plain texts of the input dataset among the entities in the sets E Dy and E CSO. The module considers only triples whose relations are composed by only one verb and yields the set of triples T OIE.
Background The availability of data generated from different sources is increasing as well as the possibility to link these data sources with other databases. More e cient ways of data linkage and the use of arti cial intelligence (i. e. , machine learning techniques) are required to generate comparable and timely health information across European countries. Using these innovative techniques has several advantages such as data linkage improving completeness and comprehensiveness of information to guide health policy processes  [1]. New approaches more or less based on arti cial intelligence allow us to handle data with a large number of dimensions (features) and units (feature vectors) more e ciently and with high precision. Many countries have already invested in the linkage including both deterministic and probabilistic linkages and linking their traditional health administrative data with other types of data and has increased interoperability  [2].
Therefore, the InfAct project has proposed to develop these guidelines, which could guide those MSs who are planning to estimate health indicators using linked data and/or ML-techniques with new methods/techniques. InfAct (Information for Action) project is a joint action of Member States (MSs) aiming to develop a more sustainable EU (European Union) -health information system through improving the availability of comparable, robust and policy-relevant health status data and health system performance information  [5]. InfAct gathers 40 national health authorities from 28 MSs. The main objective of this study was to develop the methodological guidelines to estimate population-based health indicators using linked data and/or ML-techniques with new methods.
i. Literature review Firstly, we performed a literature search to identify published articles focusing on estimating health indicators using linked data and/or machine learning techniques in the eld of health surveillance and health care performance on August 1, 2020. We did not specify any time period to search for the related published articles as to obtain a wide range of studies published at any time. We included in our search peer-reviewed methodological articles, related guidelines and systematic reviews that were published in the English language. We excluded those studies published as protocols, scoping reviews or literature reviews, nonmethodological studies such as editorials, commentary or perspectives and studies related to life sciences such as RNAi or gene expression. Search strategies are reported in additional le 1.
Literature review We reviewed 215 citations from PubMed and 118 were included in our nal sample to develop these methodological guidelines (Fig. 1 ). Sixteen additional studies (i. e. , inspiring examples from European countries) identi ed from InfAct project were also included to the nal sample. The nal sample included 134 studies using linked data and/or machine learning techniques to address various research questions either to describe or estimate health indicators in the eld of health status monitoring or the evaluation of certain treatments in medical/health care. Among these citations, some guidelines were also identi ed to adopt the appropriate format of methodological guidelines  [7, 8]. We reviewed the methodologies applied in the selected studies and developed a check-list of various steps that could be adopted systematically to calculate health indicators using linked data and/or ML-techniques.
Study population/sample size The fourth step is to de ne the study population according to the proposed research question. Often, the study population is extracted from the national health administrative database linked either with a population-based cohort or disease-speci c registry or health survey or with any other administrative database. The linked database may allow having a large sample size and adds to large number of variables for analysis. The large sample size allows strati ed analysis among sub-groups. The inclusion and exclusion criteria of the study population should be clearly de ned according to the research question. The age, sex of included sample and the period of data collection should be clearly stated. The null and alternative hypothesis should be clearly stated based on the research question  [10].
iii. Split of nal data into training and test data sets: In most of the studies, the nal data set is split into 80% as a training data set and 20% as a test data set. If there is an imbalance of number of positive target - 1 group over the number of negative target - 0 group in the training dataset, a random down sampling or over sampling can be performed in the target 0 group to achieve the same number of individuals in both target groups. This helps to avoid the bias in ML-algorithm. Later, the selection of variables and the models is performed using the training data. The test data is used solely to test the nal model performance. According to the context of study and data available, different techniques of normalizing the data and crossvalidation could be used.
B. Statistical ii. ML-techniques: Several ML-techniques are applied, which focused on health care research. These techniques could be adopted to population health studies. Following are the most commonly used supervised ML-techniques: linear and logistic regression, Linear Discriminant Analysis (LDA) model  [12, 13] , partial least square discriminant analysis model  [24] , decision tree  [25] , random forest [26] and Gradient Boosting Classi er [GBC]  [27, 28] , k-nearest neighbours/k-means  [29] , support vector machine [SVM]  [30] , neural networks  [31] , convolutional neural networks, hierarchical clustering  [32]  and XGBoost  [33]. To develop and apply ML-techniques, following three main steps are used to train and select the nal model: a. Training various models: Some commonly used models are linear discriminant analysis, logistic regression, exible discriminant analysis and decision trees that are applied to the training data set.
c. Selection of nal model: After the model validation, the algorithm selection process is automated by giving the computer a speci c metrics including sensitivity, speci city, positive predictive value, negative predictive value, F1-score and kappa. Finally, a single model is retained based on its performance, computational parsimony and its transferability to other databases. C. Sensitivity/uncertainty analysis: After the selection of nal model, sensitivity analysis is performed. This analysis refers to identifying the most in uential assumptions or parameters for a given output of a mathematical computer model (i. e. , the sensitivity of output by changing the inputs) or to evaluate the effect of uncertainty in each uncertain computer input variable on a particular model output  [36]. It helps to understand the relationship between input and output variables and the robustness of the results of a computing model  [37].
The most common methods are: variance-based method [38], elementary effects method  [39]  and regression analysis. D. Potential issues during data analysis: During the data analysis, following are some common issues, which may encounter: missing data, imbalanced datasets and bias-variance tradeoff. i. Missing data: In datasets (small or big), missing values are often the main issue that can introduce a substantial amount of bias, make handling and data analysis harder and strongly in uence the model performance. There are three types of missing data  [40] : 1. Missing Completely At Random (MCAR): if subjects who have missing data are a random subset of the complete sample of subjects, 2. Missing Not At Random (MNAR): if the probability that an observation is missing depends on information that is not observed, like the value of the observation itself is missing, and 3.
Missing At Random (MAR): the probability that an observation is missing commonly depends on information for that subject that is present i. e. , the reason for missing data is based on other observed patient characteristics. Imputations of missing values: Imputation is a process of replacing missing values in a dataset. Following are some common approaches, which could be applied to both type of studies using classical statistical methods and ML-techniques a. For classical statistical methods: There are three most commonly used techniques i. e. , 1. listwise/complete case deletion, 2. single imputation and 3. multiple imputations. Simple/single imputation techniques (e. g. linear regression) for handling missing data (such as complete case analysis, overall mean/mode/median imputation, and the missing-indicator method) are more feasible to apply but may produce biased results.
II. Imbalanced datasets: Second issue is the imbalanced dataset (i. e. , the number of positive and negative targets/cases/values are unequal. ) that can skew in class distribution and may bias ML-algorithms. Many ML-techniques, such as neural networks, make more reliable predictions from being trained with balanced data  [45]. There are two commonly used approaches to create a balanced data set, rst is the down sampling and the second one is over sampling  [45, 46].
techniques: There are several statistical techniques that are applied to linked data either using classical statistical techniques or with ML-techniques. The former may be used for regression and later for classi cation purposes. In general, both of these techniques could be used to estimate, classify and predict the population health indicators or to evaluate the health care interventions according to the available linked datasets. The brief description of different techniques is reported in additional le 2. [17]19]sical statistical techniques: Several classical statistical techniques were identi ed in the selected studies to analyze the linked data set. Following are the most commonly used techniques: linear and logistic regression, Linear Discriminant Analysis (LDA) model[12,13], multilevel linear regression[14], multivariate logistic regression[15], multivariable hierarchical modi ed Poisson regression [16], Cox regression models[17], LASSO regression[18,19], Generalized Estimating Equation (GEE) models.
We describe the design, implementation, and impact of a data harmonization, data quality checking, and dynamic report generation application in an international observational HIV research network. The IeDEA Harmonist Data Toolkit is a web-based application written in the open source programming language R, employs the R/Shiny and RMarkdown packages, and leverages the REDCap data collection platform for data model definition and user authentication. The Toolkit performs data quality checks on uploaded datasets, checks for conformance with the network's common data model, displays the results both interactively and in downloadable reports, and stores approved datasets in secure cloud storage for retrieval by the requesting investigator. Including stakeholders and users in the design process was key to the successful adoption of the application.
Introduction The International epidemiology Databases to Evaluate AIDS (IeDEA) consortium is an international research network formed in 2006 by the U. S. National Institute of Allergy and Infectious Diseases (NIAID) to allow investigators to address high priority HIV/AIDS research questions through the combination and analysis of globally diverse clinical data  [1]. Merging observational data from participating sites in a research network like IeDEA presents challenges for data harmonization and data quality; assurance of data quality is critical to the integrity of any analysis  [2] [3] [4] [5] [6]. Like many disease-focused research consortia, IeDEA is a federated network with limited resources for technical support, so data quality solutions that require local software installation and dedicated data quality personnel are generally not feasible.
IeDEA IeDEA is composed of seven regional networks that collect data from over 380 HIV clinics worldwide, representing over 2 million adults and children living with and at risk for HIV  [1]  (Fig. 1 ). Clinics in IeDEA are grouped into these seven geographical regions, each with its own regional data center that is staffed by 1-4 data managers. These data managers are responsible for collecting and harmonizing observational data-in a wide variety of formats and languages-from the sites in their region. IeDEA regions collaborate by combining data from their regional databases for use in approved multiregional analyses  [8] [9] [10] [11] [12]. Due to data privacy concerns and the data sharing regulations of participating countries and institutions, IeDEA regions contribute subsets of the data collected from clinics in their region for approved projects rather than pooling data in a centralized database.
The U. S. National Institute of Allergy and Infectious Diseases purposefully developed this federalized structure for IeDEA to build a foundation of trust among global collaborators with differently resourced countries. Once a multiregional research study is approved by IeDEA regions, each participating region contributes data. Regional data managers select observations that match the study's inclusion and exclusion criteria and submit a dataset that has been mapped to the IeDEA Data Exchange Standard (DES), a common data model for observational HIV data  [13]. Historically, the work of mapping data to comply with the latest version of the data model and performing extensive data quality checking required substantial data manager effort. As a result, the research process could be slow and individual study data submissions often could not be readily merged into a global dataset by the data recipient.
They found that all six networks employed complex processes that included installing and updating required software-either open source or proprietary-and depended on the availability of both local and centralized data quality support, as well as familiarity with technical tools like GitHub. Liaw et al  [14]  conducted an extensive literature review of data quality practices in real world data and saw the need for an expanded data quality framework, one that includes meaningful but hard to measure contextual and technical indicators such as timeliness, trustworthiness, and traceability. These studies reveal that technically demanding data quality control practices are used by both centralized and distributed data networks. For example, data quality management is centralized at a coordinating center in the National Patient-Centered Clinical Research Network (PCORnet)  [15] [16] [17] , Pediatric Learning Health System (PEDSnet)  [5] , and the Sentinel Initiative  [18] [19] [20].
These three networks use the Observational Medical Outcomes Partnership (OMOP) common data model, and query their network sites about data quality using software that is installed and run at sites locally. Data quality issues are reviewed by coordinating center personnel who follow up with the sites. Data quality control is maintained centrally. In contrast, the Observational Health Data Science and Informatics (OHDSI) network has distributed data quality coordination  [21] [22] [23]. The original data quality tool developed for OHDSI, Achilles Heel  [21, 24] , functioned primarily as a dataset characterization tool and was recently supplemented by the OHDSI Data Quality Dashboard  [25]. The Data Quality Dashboard applies 20 "data quality ideas" to OMOP-formatted datasets: 8 conformance checks, 5 completeness rules, and 7 checks for data plausibility.
Depending on the number of tables and fields in a dataset, this can add up to as many as 3,300 data quality checks. These existing data preparation and data quality tools, although robust and well-suited to their networks, require high-level technical expertise and server infrastructure at either the participating sites or a central coordinating center. As a network with no coordinating center, a majority of sites located in resource-limited settings, and limited availability of IT support and infrastructure at the regional level, IeDEA needed an adaptable solution. In this paper, we present the collaborative design method and initial use results for the Harmonist Data Toolkit, an intuitive web-based data quality checking, report generation, and data exchange application designed for IeDEA but generalizable to other research domains with data models defined in REDCap.
Engaging stakeholders and identifying software needs Throughout the software development process, our informatics team sought both qualitative and quantitative input from IeDEA data managers and investigators through in-person and virtual meetings, email discussions, one-to-one calls, and REDCap surveys. We participated in monthly calls with IeDEA data managers (the IeDEA Data Harmonization Working Group) and hosted side meetings at international research conferences to collaborate in person with the community of stakeholders. Due to the small number of intended users of the IeDEA Harmonist Data Toolkit (1-4 data managers at each of the seven regional centers), it was not necessary to employ random sampling when distributing surveys or choosing participants for meetings and calls; we engaged data representatives from every region in the needs assessment stage as well as the software design, testing, and revision processes.
Designing data quality checks for IeDEA Members of the IeDEA Data Harmonization Working Group determined the scope of desired data quality checks based on their experience with IeDEA multiregional studies and common errors in IeDEA datasets. Those data quality checks were a combination of tests that individual data managers had implemented locally as well as checks used in other research networks  [2, 5, 15, 24, 27]. The Harmonist team mapped the data quality checks to the data quality categories proposed by Kahn  [4]. Next, we modeled the IeDEA common data model in a series of REDCap databases representing data tables, variables, and code lists. This allowed the data model to be easily edited and exported from REDCap in a machine-readable JSON format. We developed data checking scripts using the R statistical computing language  [26]  that processed the JSON data model as input and implemented general data quality checks based on the details from REDCap (Fig.
Implementing a user-centered, iterative software design approach After developing an initial prototype of the Harmonist Data Toolkit,  we iterated toward our current application design and report content through several cycles of demonstrations, feedback gathering, and software modifications (Fig. 3 ). This included two separate asynchronous user testing events with assigned tasks to complete, followed by surveys. In the first testing activity, users were asked to explore the features of the Toolkit and complete the survey. Instructions for Testing/ Training Activity #2 were more specific and included every activity that a data manager would need to perform, including uploading datasets, performing data quality checks, generating reports, submitting datasets, deleting datasets, and retrieving submitted datasets.
These tasks were chosen to test the full functionality of the Toolkit so that users could both learn and evaluate every aspect of the new workflow. Sample datasets were provided so that users could assess their feelings about Toolkit security without using real data. Both training events were introduced in a Data Harmonization Working Group call followed by emails to all data managers with details, sample datasets, and links to follow-up surveys in REDCap. Toolkit Survey #1 included a combination of open-ended and multiple-choice questions about which Toolkit features seemed most useful, how the Toolkit would impact the user's workflow, as well as what changes they would suggest making to the software and report content. In Toolkit Survey #2, users were asked to rate their comfort with using the Toolkit for IeDEA data tasks and to list any remaining questions or steps of the process they found confusing.
System architecture We implemented the Harmonist Data Toolkit as a web application written in the open source R statistical programming language  [26]. It leverages the Shiny  [28]  web framework for the user interface, the RMarkdown  [29]  package for reporting, and other R packages for support including ggplot2  [30]  and tidyverse  [31]. The Toolkit application interfaces with REDCap via the REDCap API to retrieve details about the common data model and data requests as well as for Toolkit usage tracking (Fig. 4 ). The application is deployed on an Amazon EC2 instance and is accessible via the web for data quality checking and report generation for IeDEA DES-compliant datasets. The Toolkit can also be used in conjunction with IeDEA multiregional research data requests for users who access the Toolkit by way of a REDCap-based consortium management portal, known as the IeDEA Hub, which requires user authentication.