dataesr/scientific-paragraphs-categorization huggingface dataset:
Task and material Each position was shown for 5 s, from the viewpoint of white. Next, players were required to evaluate it using the ordinal scale presented in Table  1. This scale is well established in the chess world and is for example used in chess 1 Chess players' skill is rated along a continuous scale called the Elo rating  (Elo, 1978). After each competitive game, a player's rating is adjusted as a function of performance (win, loss, or draw) and the rating difference between the two opponents. Experts have an Elo rating of 2000 or above. Below 2000 Elo, the following classes are used: Class A (1800-2000), Class B (1600-1800) and so on. books, search engines, and databases. It is, therefore, an ecological measure. In total, 56 positions were selected from experts'  (Elo > 2000)  games in a commercially available database (Chessbase, 2011).
Statistical analyses Response times and accuracy were analyzed separately by using the general linear model. The factorial design with four levels of skill (between-subject), two levels of complexity (within-subject) and seven levels of balance (withinsubject) was entered in a full-factorial analysis of variance. The model allowed estimating the influence of the main factors of expertise, complexity, and balance separately, plus estimating the impact of their interactions. We carried out the analysis first on response time and then, separately, on evaluation errors. The statistical analyses were carried out with IBM SPSS ®.
75 MHz. They are finally amplified by 70 dB and converted by analogue to digital converters (ADCs) operating at 250 MSa s -1 with a resolution of 14 bits. The distances are then calculated from the phase difference between these measurement signals and an electrical reference built from the direct mixing of the RF and LO carriers. The signals are processed digitally by a field-programmable gate array (FPGA) designed by our group and displayed using Matlab TM software. As shown in figure  2 (b), a motorized flip mirror has been implemented to deviate the free-space optical beams towards a small reference corner cube and thus compensate for the phase variations induced by temperature changes in the optoelectronic, fibre-optic and microwave components. In practice, this mechanical zero measurement is performed every second when the RF power received by the system is sufficient to perform a distance measurement.
According to formula  (8) , this value corresponds to a phase noise of 2π/14 800 for a 5 GHz modulation frequency. The uncertainty budget of the ADM is summarized in table  1. The dominant contribution is crosstalk. The uncertainty related to the ADM was determined for each wavelength by propagating the different sources of errors listed above to the optical path lengths (formula  (8)  for n = 1). This was done using Monte-Carlo simulations with 10 6 datasets randomly generated according to the probability distributions defined in table 1 and for a distance of 5 km to fix the uncertainty contribution of the RF frequency. The resulting error distributions are shown in figure  4  for SCRs of 60 dB and 50 dB. If the system is well optimized and an SCR higher than 60 dB is reached, a measurement uncertainty of 4.
Acknowledgments The authors are very grateful to  J Cali  from  École Supérieure des Géomètres et Topographes (ESGT, France ) for the availability of their interferometric bench and of their laser tracker Leica AT401. The authors would also like to thank  C Courde  and colleagues from  Observatoire de la Côte d'Azur (OCA, France)  for their hospitality and help with the measurements over 2. 6 km. The authors also thank  D Pesce  from  Institut national de l'information géographique et forestière (IGN, France ) for assistance and advice during the measurement campaigns, and  O Martin  from  IGN  for the design and construction of RF amplifiers. Lastly, the authors would like to thank their colleague  M D Plimmer  from  Conservatoire National des Arts et Métiers (Cnam, France ) for proofreading the article, which helped to improve the quality of the English.
The current rise of Open Science and Reproducibility in the Life Sciences requires the creation of rich, machine-actionable metadata in order to better share and reuse biological digital resources such as datasets, bioinformatics tools, training materials, etc. For this purpose, FAIR principles have been defined for both data and metadata and adopted by large communities, leading to the definition of specific metrics. However, automatic FAIRness assessment is still difficult because computational evaluations frequently require technical expertise and can be time-consuming. As a first step to address these issues, we propose FAIR-Checker, a web-based tool to assess the FAIRness of metadata presented by digital resources. FAIR-Checker offers two main facets: a "Check" module providing a thorough metadata evaluation and recommendations, and an "Inspect" module which assists users in improving metadata quality and therefore the FAIRness of their resource.
Introduction The production of scientific data intensifies from year to year, leading to a huge, ever-growing amount of data. Open Science is currently seen by many scientific communities and research funding organisations as a way to considerably simplify meta-analyses and improve the statistical power and validity of scientific models. It also contributes to more cumulative and reproducible science by speeding up the development of new hypotheses, data analysis methods, models, and their validation on various open and available datasets, as illustrated in the neuroimaging community  [1]  or more recently during the Covid-19 health crisis  [2, 3]. Open Science also appears to be an essential factor in limiting the energy and environmental impact of data production and storage, by limiting unnecessary duplication of data and experiences  [4].
Initially, the FAIR evaluation of a resource or a set of resources was mainly done through questionnaires (which have the disadvantage of being time-consuming and requiring a certain amount of experience). Rapidly enough, automatic tools have been developed, relying on various implementations of the FAIR principles. The first one was the FAIR Evaluation Services 3  [6] , but it was rapidly followed by FAIR-shake 4   [8]  and F-UJI 5   [9]. The scope of the FAIR principles is very broad and therefore impacts a wide range of scientific communities. For example, an expert in microscopy image analysis needs to share both analysis algorithms and reference datasets with perennial identifiers and a controlled vocabulary specific to his or her community. Biologists feeding a genomics database must be able to describe the provenance of their sequences and samples.
These same questions arise in the context of the study of marine biodiversity, for example. Thus, a facilitated access to a wide variety of semantic tools is needed if we want to engage a wide variety of communities towards FAIRer resources. Semantic web technologies seem to respond precisely to this need. Although some Findability and Accessibility principles are difficult to implement with Semantic Web technologies, such as long-term persistence of identifiers, or indexing by Web search engines (if we leave aside Google Dataset Search), most of the Interoperability, and Reuse principles can actually be addressed with already established Semantic Web standards technologies. In this paper, we propose FAIR-Checker as a tool, leveraging Knowledge Graphs and Semantic Web technologies, with the goal of making producers and developers of scientific digital resources more accountable and efficient in their FAIR implementation.
The main contributions of this paper are i) a collection of SPARQL queries aimed at evaluating FAIR principles for metadata, ii) a SHACL constraints generator aimed at evaluating metadata profiles and thus enhancing the completeness of metadata, and iii) an evaluation of our approach against typical bioinformatics resources. The paper is organized as follows. Section "Background" describes use cases and the state of the art. Our approach is described in section "Approach". The system implementation and its evaluation are reported in sections "Implementation" and "Results". Section "Discussion and conclusion" finally discusses our results and their perspectives.
Motivating scenarios Our main motivation for developing a new FAIR maturity assessment tool was to meet user needs and go beyond a simple scoring system. Improving the discoverability, interoperability, and reuse of a biological resource requires specific technical skills and an understanding of FAIR principles. We wanted to develop a tool based on Knowledge Graphs that facilitates the selection of appropriate ontology terms to enrich the metadata of web resources. Working in the context of a research infrastructure for bioinformatics developing services for health, plant sciences and agronomy communities, we identified 3 typical user profiles: data producers, software developers, and repository developers. These users typically use the web as an infrastructure to access and publish FAIR digital scientific resources.
Most of the time, these people are not trained to implement the FAIR principles and have to learn how to do it on the job. As a motivation, we consider the following three use cases. UC 1 : Selecting a data repository. In this use-case, data producers want to select a public data repository (e. g. Dataverse or Zenodo for instance) for publishing a given dataset. They can use FAIR-Checker to compare the FAIR compliance of several solutions. To do so, they simply need to submit to FAIR-Checker randomly selected identifiers of any dataset hosted by the repositories and compare FAIR-Checker analysis reports. This is very fast and does not require any particular semantic web expertise. UC 2 : Sharing a software. Software developers wishing to provide access to a particular tool on the web can create an early version of their website and submit its provisional URL for analysis by FAIR-Checker.
The analysis report will provide not only an initial FAIRness assessment of their resource, but also recommendations to improve its compliance with the FAIR principles. After making the necessary changes, the developers will be able to test their website again and see the improvements. In such a use-case, specific help is expected regarding 1 https:// doi. org/ 10. 15497/ rda00 050 2 https:// www. gofair. found ation/ inter preta tion 3 https:// fairs haring. github. io/ FAIR-Evalu ator-Front End 4 https:// fairs hake. cloud/ 5 https:// www. fairs fair. eu/f-uji-autom ated-fair-data-asses sment-tool the selection of appropriate classes and properties to enrich the metadata. Indeed, relevant ontologies can be very complex to comprehend for developers who are not familiar with. UC 3 : Enriching metadata for registries.
Related works A rapid survey of existing tools is necessary to assess whether any of them can provide a solution for the three use-cases. A first group of tools consist of questionnaires for manual assessment of FAIR metrics. One can first cite FAIR AWARE from FAIRsFAIR 6 which is a simple on-line questionnaire which helps researchers and data managers assess how much they know about FAIR requirements for datasets. Thus it cannot be used to score and compare resources with respect to their compliance with FAIR principles as in UC 1. Other questionnaires have been compared in 2019 by the RDA FAIR Data Assessment Working Group  [10]  showing a great heterogeneity in the way questions are expressed for the same principles (note that this preliminary work certainly inspired the RDA guideline document  [11]  released in 2020).
In the RDA 2019 survey, 12 tools were studied including 11 questionnaires and only one automated tool (FAIR Evaluator see below). From these 11 questionnaires only 3 have evolved to free on-line resources still available today. The Australian ANDS-NECTAR-RDS-FAIR data assessment tool has become the ARDC FAIR data self assessment tool proposed by the Australian Research Data Commons 7. It provides a FAIR indicator represented by a more-or-less fully colored green bar without any quantification. Moreover, no help is provided for improving the compliance to FAIR principles as expected in UC 2. The two Dutch tools: DANS-Fairdat and DANS-Fair enough are now merged in a single tool SATIFYD 8 (Self-Assessment Tool to Improve the FAIRness of Your Dataset) which contains 12 questions distributed on the four FAIR categories.
The answers are visualized by coloring more or less the corresponding letter and providing a percentage of compliance for each category. Moreover, suggestions for improving the FAIRness are provided when the score is not 100%. In addition to these few online questionnaires, the RDA-SHARC community (SHAring Rewards and Credit) also proposes a questionnaire  [12]  in the form of a spreadsheet downloadable from Zenodo 9 , to be completed by researchers. A quantitative score is computed based on the levels of importance (essential, recommended, desirable) for each criterion and on four possible answers (Never/NA, If mandatory, Sometimes, Always). In summary, manual FAIR assessment through questionnaires has a great value for training and propagating good understanding of FAIR principles. It should be noted that the field has evolved a lot and many early questionnaires are no more available on-line as they required constant updating of FAIR principles and FAIR metrics definition.
Regarding our use-cases however, questionnaires are clearly not the right solution mostly because manual answering would be tedious for UC 1 , but also because it would require a good prior knowledge of the FAIR principles by the software developer in UC 2 , and because in UC 3 , the resource publisher needs a detailed analysis of the metadata content associated with the web pages of his catalog which cannot be provided by a questionnaire. The second group of FAIR assessment tools are automatic tools which implement FAIR Maturity Index metrics. The FAIR Evaluator system is the result of a community-driven effort, promoted by the GO-FAIR initiative 10 with the goal of defining FAIR Metrics, soon renamed FAIR Maturity Indicators (MI), and to develop an automatable framework to discover, access and interpret the content of a data resource  [6].
dans. knaw. nl 9 https:// doi. org/ 10. 5281/ zenodo. 39220 68 10 https:// www. go-fair. org/ go-fair-initi ative/ 11 https:// fairs haring. github. io/ FAIR-Evalu ator-Front End start by creating projects to group several resources to be tested. Digital objects are then created, described through a form, and associated to one or more manual or automated metrics, provided by user submitted scripts. A gradient colored matrix (FAIR insignia) is then displayed as a summary of the assessments results. F-UJI  [9]  is a fully automated testing tool. It focuses on domainagnostic metadata which can either be embedded into the resources under assessment or provided by external services. F-UJI allows to test ressource URLs or DOIs and generates a graphical summary of the score as well as a detailed technical report of the evaluations.
Then, for each metadata entity (e. g. a person, a dataset, a software, etc. ), public KGs are queried to retrieve relevant associated RDF triples (➌). Since attribution or citation is a clear incentive to promote sharing and reuse in open sciences, we principally target scientific literature KGs such as OpenAire or OpenCitation. We also include general knowledge through WikiData. Following the Linked Data principles, our objective is, i) for data publisher, to limit their efforts in annotating individual web pages, and ii) for Knowledge Graph developers/maintainers to make them contribute to the implementation of FAIR principles. Then, ontology checks (➍) assess that used ontology classes or properties are part of community agreed standards. By leveraging community-specific registries such as BioPortal 12 or OLS 13 , or general registries such as LOV 14 , we can evaluate if common ontology terms are reused.
Public Knowledge Graphs supporting FAIR assessment Many public Knowledge Graphs already aggregate, and make accessible, a large number of metadata associated to digital resources such as databases, scientific literature or software. With FAIR-Checker, we propose to exploit these semantic data sources during the FAIRification process. From the assessed URL, FAIR-Checker generates a SPARQL DESCRIBE query to retrieve the RDF triples already accessible. Wikidata  [20]  is queried for general knowledge, through the wikidata:P356 property, used to cross reference digital object identifiers (DOIs). The properties openaire:resPersistentID and datacite:hasIdentifier are also exploited in SPARQL Describe queries to query respectively the SPARQL endpoints of OpenAIRE and OpenCitation which target scientific literature metadata.
Implementation FAIR-Checker is available at GitHub 23 under an MIT licence. It is a web application developed in Python and based on the Flask web framework. Requests, Selenium and Extruct libraries were used to access web pages and extract embedded RDF metadata, while supporting client-side HTML rendering. RDF data, SPARQL queries and SHACL shapes are handled thanks to the RDFlib and pySHACL libraries. Object-Oriented techniques were used to foster the extensibility of FAIR-Checker to multiple implementations of FAIR metrics. The Factory design pattern allows switching between multiple implementations of the same metrics. An abstract metrics class was designed and thus allowed to implement a generic FAIR metrics evaluation engine. Code generation techniques, leveraging the Jinja template-based text transformation engine, allow adapting multiple and evolving metadata profiles.
As shown in Fig. 3  only a single SHACL shape template needs to be maintained for validating multiple Bioschemas profiles. Regarding the selection of Bioschemas profiles for validating specific type of metadata, we first rely on declarations from data providers with the dct:conformsTo property, pointing to a Bioschemas profile URL. Since Bioschemas specification repository 24 provides a machine-readable specification of each profiles (JSON-LD), we then generate a SHACL shape for each available profile. When the dct:conformsTo is not specified, we propose a candidate profile based on the type (rdf:type) of a metadata entity. We used cache memory techniques to accelerate repetitive processes and alleviate external SPARQL endpoints from re-executing the same query in short periods of time. These caches were implemented through the flask_ caching and cachetools python modules.
The former one, of short duration (one minute), is used to store and share RDF metadata for the whole set of metrics, while the second one, updated less frequently (every 2 weeks), is used to store the result of SPARQL queries targeting stable external ontology services (OLS, LOV, and BioPortal) in order to avoid sending multiple identical requests. Figure  4  shows a sub part of the user interface displaying the result of a SHACL shape evaluation. RDF triples reporting the shape evaluation are queried and transformed into natural language. SHACL errors are reported as requirements ("must be") and warnings are reported as improvements ("should be").
Selecting FAIR data repositories Zenodo, Pangaea and Dryad are repositories aimed at sharing and preserving papers, datasets, software, and research artifacts in general. By identifying resources with DOIs and exposing metadata, they concretely 23 https:// github. com/ IFB-Elixi rFr/ FAIR-check er 24 https:// github. com/ BioSc hemas/ speci ficat ions 22 https:// fairc ookbo ok. elixir-europe. org/ conte nt/ home. html support Open Science. In this experiment, in line with use case UC 1 , we explore, with FAIR-Checker , the potential differences in the way metadata are exposed by several scientific repositories. For each repository, we randomly sampled and harvested 500 web pages describing datasets. Figure  5  shows three UpSet 25 plots aimed at comparing the compliance of the three investigated repositories with the FAIR metrics.
Each line represents a metric. A gray point means that the metrics is not validated while a black point means that it is validated. The second line of the left plot shows that none of the datasets in the Zenodo repository validate the R1. 2 metric (provenance metadata). The third line shows that, in this repository, 476 of the 497 tested datasets validate the R1. 1 metric. Reading these plots vertically, each column counts the number of datasets validating a subset of metrics. The second column of the left plot shows that a single dataset is failing the R1. 2 and F1B metrics. The FAIRest resources (475 over 497) are displayed in the rightmost column of this plot where only R1. 2 is not validated. The total of 497 datasets for Zenodo instead of 500 for Pangaea and Dryad is explained by the fact that two of them had invalid DOIs wich could not be resolved, while the last one had its DOI property incorrectly formatted.
In summary, Fig. 5  shows that none of the three tested repositories expose provenance metadata (R1. 2 principle) following the standard vocabularies we identified in Table  2. Although the coverage of FAIR metrics is very similar for these three registries, Dryad seems to be more normative due to the exposition of license metadata for all the 500 randomly sampled resources. This information can be valuable when a data producer wants to select a repository for scientific data sharing (UC 1 ).
Sharing a software. PhyML  [25]  is a bioinformatics software aimed at inferring phylogenetic trees. Its website 26 is a widely used resource for the phylogeneticists community. Its development started when FAIR principles were not as recommended as they are today, leading to a website without embedded metadata. In line with use-case UC 2 , we used the "Check" function of FAIR-Checker to analyze the PhyML early website. As shown in Fig. 6 , it complied with only two FAIR metrics (F1. A and A1. 1) associated to standard web protocols (URL, HTTP). To improve its FAIRness, we integrated metadata compliant with the Bioschemas ComputationalTool profile 27 for describing a software application. We also used the EDAM bioinformatics ontology  [26]  to semantically describe i) the disciplines (EDAM Topics) associated to PhyML, ii) the analyses done (EDAM Operations) on input data, and iii) the nature (EDAM Data) of output data.
FAIR assessment of a catalog of bioinformatics software. Bio. Tools is a large community registry cataloging bioinformatics software. In April 2022, the registry had 25, 048 tool descriptions. This registry has been instrumented to expose Schema. org semantic annotations to enhance the tools findability. In this experiment, we aimed at evaluating the complete collection of 25,048 bioinformatics software descriptions available from Bio. Tools. To speed up the evaluation process, we did not extract the metadata from all individual web pages but analyzed an RDF dataset assembled from all Schema. org annotations provided by the Bio. Tools registry 28. On a classical laptop workstation, 2. 7 GHz quad core Intel Core i7, the whole sequential evaluation of the FAIR metrics -including metrics relying on remote calls to SPARQL endpoints -lasted 42 minutes, which represents a throughput of around 10 evaluations per second.
com/ bio-tools/ conte nt/ tree/ master/ datas ets Finally, in line with use-case UC 3 , the FAIR-Checker "Inspect" function evaluated the Bioschemas ComputationalTool profile 29 on the whole collection of bio. tools entries. On the same laptop configuration as before, the generation and evaluation of SHACL shapes lasted 16 minutes on the whole software collection. The right panel of Fig. 7  reports on the compliance of these entries with respect to both mandatory and recommended properties. All mandatory properties for computational tools are available for all bio. tools entries. On the other hand, bio. tools provides access to an average of 54% of the recommended properties. These results deserve to be studied in detail by the editors of the bio. tools registry to continue to improve the metadata of the resources it contains.