dataesr/scientific-paragraphs-categorization huggingface dataset:
Instead of having several different classes of objects to detect, face detection has only one class to detect. Thus, the AP here is the average precision of the detection of different faces. As same as in  [60] , three types of AP with different Intersection over Union (IoU) values have been used as three types of metrics for evaluating the face detection method. The metric with three different IoU thresholds indicates three different levels to locate the faces in images. The IoU threshold from 0. 3 to 0. 7 indicates the localization requirement from easy to hard. That means AP with IoU of 0. 7 reward detectors with better localization than AP with IoU of 0. 5. The source code for the evaluation scripts are available at https://github. com/hengxyz/MIDV_2020_det. Table  11  shows the evaluation results of face detection on MIDV-2020 by MTCNN.
First, the faces presented on the documents are frontal faces no matter how the documents have been placed in space. Comparing to the profile or the faces with extreme pose included in WiderFace, the frontal faces are much easy to detect. Secondly, the ground truth of bounding boxes of faces in MIDV-2020 were semi-automatically annotated with MTCNN, which may also alleviate the detection error of MTCNN. Nevertheless, for the extreme complex cases such as the images with the strong reflect, extreme shooting angle or partial face as shown in Figure  12 , the general face detection model still fails to detect the faces. These extreme hard cases have seldom appeared in the general face detection datasets such as FDDB or WiderFace. Thus, MIDV-2020 provides a useful supplementary material to the current face analysis benchmarks enabling the face detection algorithms to be generalized in more different scenarios or domains.
Conclusion In this article, we presented a new identity documents dataset MIDV-2020, which consists of 2000 scans, 1000 photos, and 1000 video clips of different artificially generated identity documents. In total, the dataset contains 72409 annotated images, which makes it the largest publicly available dataset dedicated to identity documents. We have presented baseline evaluations of the methods of such identity document image analysis problems as document detection, location, and identification; text fields recognition; and face detection. The performed experiments show that the dataset contains examples of images and video clips captured in challenging conditions, which can serve as a motivation for further improvement of document analysis methods and algorithms. Although the MIDV-2020 dataset is in itself a dataset of fake documents, its usage for developing and evaluating methods of computational document forensics, presentation attacks detection, and other authenticity validation approaches, is limited.
Identifying cell types based on expression profiles is a pillar of single cell analysis. Existing machine-learning methods identify predictive features from annotated training data, which are often not available in early-stage studies. This can lead to overfitting and inferior performance when applied to new data. To address these challenges we present scROSHI, which utilizes previously obtained cell type-specific gene sets and does not require training or the existence of annotated data. By respecting the hierarchical nature of cell type relationships and assigning cells consecutively to more specialized identities, excellent prediction performance is achieved. In a benchmark based on publicly available PBMC data sets, scROSHI outperforms competing methods when training data are limited or the diversity between experiments is large.
We provide evidence for this by comparing the performance of scROSHI with three existing tools that scored among the best in a recent benchmark study  11. There, the authors used the same datasets for training, testing, and validation. Here, based on three annotated datasets, we show that scROSHI outperforms the competing methods when applied to data from a different source than the training data, which is a much more realistic scenario. Taken together, scROSHI is a conceptually simple, transparent, interpretable, and robust cell type classification approach particularly useful when previous knowledge about cell type-specific genes is available but annotated training data is scarce. scROSHI is available as an R package and can thus be seamlessly integrated into single cell analysis workflows.
Methods scROSHI: design considerations The most important design criterion for scROSHI was that the method should be capable of automated classification in the absence of labeled training data. This excluded any machine learning approach that would require training a model. Instead, the method should utilize and rely on the vast amount of validated cell type-specific gene sets available from previous bulk or single cell experiments, such as the widely acknowledged immune cell type gene sets (also known as "lm22") used by the cibersort algorithm  12  or the somewhat related resource for single cell melanoma data  13. Another important design criterion was that the method should avoid re-training on new datasets to avoid overfitting. Provided the originally chosen gene set was previously validated to be robust against changes in the experimental condition, re-training is not necessary.
While this approach is robust against expression outliers, it is relatively expensive in data availability, i. e. , it requires knowledge of the gene expression profile of the reference. We chose to follow an intermediate path by performing a quantitative and robust test, the Mann-Whitney rank sum test, to compare the expression ranks of the genes specific for one cell type with the ranks of the genes specific for all other cell types. The negative log of the test's p-value is then a measure of the association strength between the target cell and this cell type. Similarly, as in most classification problems, it is assumed here that a cell belongs to one and only one of a given set of classes. However, scROSHI goes one step further and allows for the introduction of two additional classes, unknown and uncertain, to deal with the unavoidable classification uncertainty.
Another step to improve on the classification efficiency is to utilize the hierarchical tree structure that is inherent to cell types due to developmental specialization and therefore apply a hierarchical classification approach. Instead of classifying all cell types at once, target cells are first assigned to a smaller number of major cell types and then consecutively to more specialized classes. This way, relatively similar cell types can be distinguished provided they belong to different branches in the tree. The scROSHI workflow 1. Find out which cell types to expect from field knowledge. 2. Obtain validated cell type-specific gene sets from the literature or learn marker genes based on other datasets. 3. Obtain a hierarchical tree structure to define cell type parent -kin relationships ("generations").
It is recommended to adapt the cell type selection to keep classification specificity high whenever possible: having closely related cell types in the candidate list may sometimes be required but should be avoided if possible. There are a number of resources available containing curated reference datasets, mostly assembled from bulk RNA-seq or microarray data of sorted cell types. Examples are the C8 set of the MSigDB collection 15 , the lm22 immune cell list of cibersort  12  , the BioGPS Human Cell Type and Tissue Gene Expression Profiles collection  16  from harmonizome  17  , or the Bioconductor 18 package celldex  19. These references are often good enough for most applications provided that they contain the cell types that are expected to be present in the data at hand. For our contribution to the Tumor Profiler Study  20  , working on melanoma patient biopsy samples, we used the curated gene set from Tirosh et al.
Benchmarking A detailed description of the datasets used, their origin, which preprocessing steps were applied, as well as the description of the pipeline and the competing tools is described in the Supplementary Material. To briefly summarize, we used public datasets with a similar cell type composition to benchmark scROSHI against high profile competitor methods. Data from three peripheral blood mononuclear cell experiments were retrieved, one from an adult human in which the cell types were pre-sorted (Zheng_sorted set), and one each of an adult (Adult set) and a newborn (Newborn set). Hence, the three sets are similar in content but differ in experimental setting and donor age. We defined a common set of matching cell type labels across the three datasets for comparisons between datasets (see Supplementary Material for further details on the ground truth dictionary).
Based on a previous benchmark of automatic cell identification methods  11  , we decided to compare scROSHI to three front runners: support vector machine (SVM), random forest (RF) and GARNETT  21. The main difference between scROSHI and its competitors is the fact that they use part of the data to train a model whereas with scROSHI there is no training involved once the cell type-specific gene sets are selected. While SVM and RF can capture non-linear relationships between the explanatory features (gene expression) as well as interactions between them, GARNETT is based on a penalized multivariate generalized linear prediction model (GLMNET). The competing methods were used under standard conditions, with default parameter settings. We trained a model for RF, SVM, and GARNETT and evaluated the performance of the classifiers by applying a 5-fold cross-validation for each dataset.
Validation scheme Each of the three datasets was split in training, validation, and testing sets. Three major validation runs were performed in which each of the three datasets served as the training/validation set. After the final model was obtained, it was tested once "in set" on the testing set that came from the same experiment as the training data, and two times "out of set" on the two remaining sets from which the model has not yet seen any data. Further details on the validation scheme can be found in the Supplementary Material.
Results & Discussion Performance evaluation We compared the performance of scROSHI on test datasets with the performance of supervised methods that had been trained with the test dataset (intra-dataset evaluation) and that had been trained with a different dataset (inter-dataset evaluation). There were three types of classifiers: (1) prior knowledge method (scROSHI) for which a marker gene list is required. (2) Supervised methods (RF, SVM), which require a training dataset labeled with corresponding cell labels. (3) Combined method (GARNETT), which requires both a marker gene list and a training dataset. We calculated the percentage of unlabeled cells across all cell populations per classifier. Further, we calculated the accuracy of only major cell types for scROSHI and GARNETT, since both methods perform a hierarchical cell typing with major and subtype labels (Suppl.
Tab. 5). Additionally, we determined the proportions of cells that only have a major cell type label, cells that have label "unknown", or are unclassified. Figure  2  shows the overall results of the inter and intra-dataset evaluation. Generally, scROSHI performs as well as the supervised methods if the supervised methods were trained with the test dataset (scROSHI accuracy: Adult 0. 823, Newborn 0. 879, Zheng 0. 715). However, scROSHI outperforms the supervised methods if they were trained with another dataset -in this case we observed a lower accuracy and/or a higher amount of unlabeled cells for all supervised methods. The supervised methods perform better if they were trained with a dataset that is closer to the test dataset (e. g. training data: Adult; test data: Newborn) but there is a strong decrease in performance if the test data is dissimilar (e.
g. training data: Adult; test data: Zheng). The subtype classification on the Zheng dataset was challenging for all classifiers (scROSHI accuracy: 0. 715). However, the accuracy of the major cell type label was 0. 952 for scROSHI indicating that even if it was not possible to find the correct subtype label the correct major cell type label could usually be determined. We presented the design and validation of the robust cell type identification method scROSHI that performs superior to competing tools provided a good quality marker gene list is available and annotated training data are limited or not available, which is often a realistic scenario in early stage projects. Similar to the scoring tool ucell  28  , the cell type score of scROSHI depends only on the relative rank of the gene expression signal, does not require normalization, and makes no assumptions about the distribution of the signal.
Consistency with estimations of copy number alterations In addition to these benchmark datasets with known ground truth but relatively simple cell type composition we used scROSHI for cell type identification in clinical samples, i. e. , biopsies from melanoma patients participating in the Tumor Profiler Study  20. Without ground truth we evaluated the classification results by consistency with copy number variation (CNV) estimations (Fig. 2 ), and by comparison to single cell CyTOF cell type composition analysis on the same samples. The three representative samples in Fig. 3A-C  show a diverse cell type composition, as illustrated by the two-dimensional UMAP representation based on gene expression in the top row. CNV states appear nearly exclusively in cells identified as melanoma cells, the only malignant cells present (insets).
Data Availability Availability of benchmark data: ask at scp-support@broadinstitute. zendesk. com. The three data sets from the TumorProfiler Study are available upon request at info@tu-pro. ch, according to the data sharing policy at the web site https://eth-nexus. github. io/tu-pro_website/data/ (in preparation by the consortium). Code availability: scROSHI will be available as R package on CRAN (in preparation). The source code for scROSHI is available on github at https://github. com/ETH-NEXUS/scROSHI.
Objectives. This paper addresses the question: How to design a sharable and interoperable data model for storing heterogeneous healthcare data and its semantic to support various applications? Method. This paper describes a method using an ontological knowledge model to automatically generate a data model for a domain of interest. The model can then be implemented in a relational database which efficiently enables the collection, storage, and retrieval of data while keeping semantic ontological annotations so that the same data can be extracted for various applications for further processing. Results. This paper (1) presents a comparison of existing methods for generating a relational data model from an ontology using 23 criteria, (2) describes standard conversion rules, and (3) presents , a prototype developed to demonstrate the conversion rules.
Furthermore, to control data integrity while manipulating a large amount of data, a relational database (RDB) is an appropriate system to store data due to its embedded access control, transaction management, data integrity control, efficiency, and performance  3. However, conventional techniques (entity-relationship, star schema, object-oriented techniques) do not provide sufficient semantic or contextual information for efficient use of a data model in a heterogeneous environment and ensure reliable reuse of data outside a restricted field of its original application  1, 4. Moreover, data generated across different systems (e. g. , health ministries, pharmacies, clinics, hospitals) is often stored in RDBs, but its semantic is rarely documented nor updated. In other words, conventional techniques do not offer adequate construct to allow the unification of data with its semantic.
Thus, the multiple stages of data processing and the exchanges that take place along the way can result in incomplete semantics of the extracted dataset. Therefore, a domain with such fragmentation and multiple data providers but with very low error tolerance needs a new approach to encapsulate the data and its semantics into the same structure that can be shared and reused for multiple applications. Healthcare is one such domain, as life and death decisions will be based on this data. Various approaches exists  5  to access or integrate data from multiple sources. These approaches use different kinds of inputs. Source-driven approaches use the data source's structure. Requirement-driven approaches use the user requirements for a specific application. Hybrid approaches combine the data structure of the sources and user requirements.
Thus, these ontologies can be used as a knowledge model to describe a specific domain of discourse objectively and formally. Therefore, deriving a relational data model directly from an ontology can be hypothesised to be the best way to leverage heterogeneous data and its semantic while ensuring integrity and concurrency access for various applications. Specifically, a relational data model generated from an ontology ensures explicit axiomatisation of the model structure enabling access to data and its associated semantic. Moreover, ontologies enable interactions with the database by referring to knowledge rather than ad hoc table and field structure and naming. Consequently, data storage, formulation, and calculation of queries can be more systematic and reliable. The presented method differs from other approaches involving mapping ontologies and databases described in the literature.
Namely, the goal is not to create an ontology from source databases or to store the ontology in a database. Instead, the ontology objectively captures the domain's semantic rather than the database's structure. In this way, the resulted RDB reflects the ontology and explicitly ties it with the data. Also, the method does not convert data into a triple store (Resource Description Framework -RDF triples) as it would not work in the use case of interest where the actors require a relational endpoint for data integrity reasons. Nevertheless, if a process benefits from transforming data from an RDB into a triple store, the work presented here would still greatly facilitate and simplify this task. This paper presents a method for deriving a relational data model from an ontology using conversion rules that covers more ontological constructs compared to existing methods including axiom complexity reduction rules (section 3), an implementation, OntoRelα, a freely available prototype used to demonstrate the conversion rules on various ontologies (section 4), a use case (section 5), a brief survey of existing methods (section 6), and a highlight of the contributions and remaining challenges (section 7).
Objective Many limitations remain with the existing methods to derive a sharable and interoperable relational data model based on a knowledge model in a heterogeneous environment. More specifically, every relational construct must be derived from a specific ontological construct uniformly in the same way, to reach uniformity and consistency through data integrity checks. Moreover, the generated data model must not reflect decisions based on specific query requirements to allow data usage outside the source database's scope or a specific project. Consequently, the standardisation and automation of the conversion through a set of rules increase the relevance and consistency of the derived relational data model and reduce the risk of errors and ambiguities that might be unnoticeably introduced by choices influenced by undocumented aspects of the designers' reality.
The conversion rules must include an axiom complexity reduction process and generate a highly  Object Property [op] links two individuals. An object property is converted to a relation including two iid attributes: the subject (subject_iid:iid_type) and the object (object_iid:iid_type). This conversion allows direct access to the links between the data of two relations and allows to store links that are not explicitly specified by an axiom of the ontology. Annotation describes an aspect of an ontological construct with a text in a specific natural language. An annotation is used to document the database and to provide multiple access interfaces in different languages using views. A definition annotation is converted to SQL comment so they can be integrated into the RDBMS catalog (if the target RDBMS supports it).
Results To illustrate the feasibility and the applicability of the presented method, a prototype, OntoRelα, was developed. As an input, OntoRelα takes an OWL ontology and some configuration files. It outputs an RDB script for the PostgreSQL database, a list of warnings to notify the user of problems in the conversion process, a mapping catalog between each ontological construct and relational construct (named OntoRelCat), and a normalised version of the ontology after the axiom complexity reduction. The resulted RDB scripts can be executed on PostgreSQL v9. 6+. A small example of the human body mass from the physiological measurement ontology is presented in Appendix C. The method is implemented through multiple processes (see Fig. 9 ): • The analysis process creates an instance of the normalised ontology model (µOnto) using a source ontology and an ontology configuration file that parameterises the process.
• The µOnto generator process generates a normalised ontology with reduced axioms formalised according to the µOnto language. • The conversion process converts an instance of µOnto into an ontological-relational model (OntoRel) according to the relational model (µRel) using the RDB configuration file that parameterises the process. • The OntoRelCat generator process the definitions of construct in the OntoRel to build a human and machine-readable mapping catalog. model was used in different use cases of the Quebec SPOR Support Unit and the SPOR Canadian data platform  16. The prototype and detailed results, including the RDB scripts, can be found on GitHub: https://github. com/OpenLHS/OntoRela.
Use case Software used to record clinical data generally does not provide explicit semantic to enable secondary use of data without ambiguity. In addition, there is currently no national standard for data exchange across institutions and provinces in Canada. This use case illustrates the context of drug prescriptions where researchers or physicians need to extract information about drug and laboratory prescriptions to make appropriate follow-up or identify inappropriate or missing prescriptions. Users are faced with many challenges, including and not limited to heterogeneity in levels of generality in drug administration and dispensing specification, homonymy, and dosing instructions  15. Moreover, merging data of drugs and laboratory results stored in different databases is not a straightforward task.
PDRO  15  , an ontology about drug prescriptions, is used to illustrate this use case with examples. OntoRelα generates the relational data model, and the database can be populated from data sources using Extract-Load-Transform processes or mediation systems. In this way, one query suffices to obtain needed data at various levels of generality without the need to explore each source separately to understand the content. Consider parts of two drug prescriptions termed "Drug administration specification" (DAS) and "Drug dispensing specification" (DDS) written on a paper or stored in a non-standard database: • (DAS1) 'Metoprolol 50 mg PO bid', instructs taking Metoprolol 50 mg per mouth ('PO') twice a day ('bid'), • (DDS1) 'Apo-Metoprolol 50 mg tab, 1 tab PO bid', instructs taking one tab of Apo-Metoprolol 50 mg per mouth twice a day.