dataesr/scientific-paragraphs-categorization huggingface dataset:
Discussion Codabench is a new meta-benchmark platform for data science communities. Codabench is compatible with diverse tasks (including supervised learning and reinforcement learning) and supports result, code, and dataset submission. It is easy to use Codabench and reproducibility is guaranteed by Dockers. Codabench has a public instance free for use, deployed at Université Paris-Saclay, but can also be deployed locally, with the technology stack provided in documentation. Hosting, maintaining, and further developing the platform is funded by grants and donations. As real-world scenarios, we introduce 4 benchmark use cases illustrating the flexibility, easiness in use, reproducibility and other features of Codabench. We also note that tremendous other tasks could be integrated into Codabench as well including EEG classification, drug discovery and property prediction, dynamic simulation for weather, traffic, fluid, etc.
, which are important tasks towards AI for Science. The current limitations of Codabench are mainly as follows. First, since it is relatively new, we do not have yet an active community of organizers and benchmark participants. We need more users' feedbacks to polish up our user-interface and documentation. Second, although supported by design, we have not had yet a distributed computation scenario, where complex multi-node compute workers are used. This could enrich our benchmark template library with benchmarks for algorithm parallelization. Thirdly, although Codabench supports both code submission and dataset submission, we do not currently allow users to extend the leaderboard in both directions simultaneously, i. e. , it does not allow users to submit both code and datasets at the same time.
This feature could largely increase the user experience of the platform. Lastly, Codabench doesn't support yet hardware related benchmarks or human-in-the-loop benchmarks which could be interesting to consider in the future. Potentially harmful uses of Codabench could result from poor benchmark designs (e. g. , no scientific question is asked by hosting a benchmark), or bad data collections (e. g. , data license, data quality), as in any machine learning project. We are working on an open-access book (to appear in 2022) on best practices for designing challenges and benchmarks including data preparation, task evaluation, benchmark analyse paper, etc. Further works include providing more comprehensive usage templates illustrating features such as: (1) splitting an algorithm workflow into submodules and scoring the effectiveness of the modules individually (e.
The ingestion program usually reads data and participant's submission. It calls participant's method on the dataset and produces predictions to a shared space. The scoring program usually reads ingestion program's output and evaluate w. r. t ground truth according to organizer customized metric. It finally writes scores to a text file which will be read by the platform and be displayed on a leaderboard. The data contain input data (in supervised learning, they are usually X_train, y_train, and X_test) and reference data (in supervised learning, it is usually y_test). Both are zipped into separate files. The additional files are just text or figure files for organizers to provide other information e. g. instructions, references, logo, etc. A final YAML file connects all previous parts and provides more configurations for the benchmark.
Acknowledgments The  Codabench project shares the same community governance as CodaLab Competitions. The openness of Codabench is total: the Apache 2. 0 licence is used, the source code is on GitHub; the development framework and all the used components are open-sourced. Codabench has received important contributions from many people who did not co-author this paper, and we would like to thank their efforts in making Codabench what it is today, including early CodaLab Competitions developers and contributors (alphabetically):  Pujun Bhatnagar ,  Justin Carden ,  Richard Caruana ,  Francis Cleary ,  Xiawei Guo ,  Ivan Judson ,  Lori Ada Kilty ,  Shaunak Kishore ,  Stephen Koo ,  Percy Liang ,  Zhengying Liu ,  Pragnya Maduskar ,  Simon Mercer ,  Arthur Pesah ,  Christophe Poulain ,  Lukasz Romaszko ,  Laurent Senta ,  Lisheng Sun ,  Sebastien Treguer Cedric Vachaudez ,  Evelyne Viegas ,  Paul Viola ,  Erick Watson ,  Tony Yang ,  Flavio Zhingri ,  Michael Zyskowski.
We would like thank particularly the people who contributed to the design, development, and testing of Codabench including (alphabetically):  Alexis Arnaud ,  Xavier Baró ,  Feng Bin ,  Yuna Blum ,  Eric Carmichael ,  Laurent Darré. Hugo Jair Escalante ,  Sergio Escalera ,  Eric Frichot ,  Yuxuan He ,  James Keith ,  Anne-Catherine Letournel ,  Shouxiang Liu ,  Zhenwu Liu ,  Adrien Pavao ,  Magali Richard ,  Tyler Thomas ,  Nic Threfts ,  Bailey Trefts ,  Catherine Wallez ,  Lanning Wei. Université Paris-Saclay  is hosting the main instance of Codabench. Funding and support have been received by several research grants, including Big Data Chair of  Excellence FDS Paris-Saclay, Paris Région Ile-de-France ,  EU EIT  projects HADACA and COMETH,  United Health Foundation   INCITE  project , ANR Chair of Artificial Intelligence HUMANIA   ANR-19-CHIA-0022 , the  Spanish  project  PID2019-105093GB-I00 ,  ICREA  under the  ICREA Academia programme ,  Inserm Cancer  project  ACACIA 232717, MIAI @Grenoble Alpes  ( ANR-19-P3IA-0003 ), 4Paradigm, ChaLearn, Microsoft, Google.
We also appreciate the following people and institutes for open sourcing datasets which are used in our use cases:  Andrew McCallum ,  C. Lee Giles ,  Ken Lang ,  Tom Mitchell ,  William L. Hamilton ,  Maximilian Mumme ,  Oleksandr Shchur ,  David D. Lewis ,  William Hersh ,  Just Research and Carnegie Mellon University, NEC Research Institute, Carnegie Mellon University, Stanford University, Technical University of Munich, AT&T Labs, Oregon Health Sciences University. We are also very grateful to  Joaquin Vanschoren  for fruitful discussions.
Without language restrictions, peer-reviewed systematic reviews of laboratory studies on dental materials published between 2015 and 2021 (last 5-year period) were included (Fig. 2 ). Additional references were also found through hand searching. The last search was conducted on June 8 th , 2021. Data extraction was carried out in the identified systematic reviews, with data selected and exported to a Microsoft Excel Spreadsheet (v. 16. 35, Microsoft, Boston, MA, USA). The data extracted were the RoB tool/instrument used to perform the quality assessment, specifically a list of the individual sources of bias used by the authors to classify each study. A second review was aimed to identify existing RoB tools in the literature, with RoB tools for basic research (e. g. , animal studies), interventional research and observational research being screened across the literature.
The same databases were searched as described above. Additionally, methodological checklists for in vitro studies such as the modified CONSORT developed by Faggion (2012)  [2]  were also considered at this stage, to aid the development of the tool. The relevant criteria were grouped in domains from the existing tools, and the most relevant domains for in vitro studies of dental materials were tailored and included in a preliminary form sheet, by two reviewers (A. D. and F. S) working independently. Each criterion was screened for applicability to assess bias in in vitro studies of dental materials. The team also proposed new criteria and/or domains that were not identified in any of the existing tools found or used in the literature and added for voting.
The RoBDEMAT tool outline The final RoBDEMAT tool contains four different domains: bias related to planning and allocation (D1), specimen preparation (D2), outcome assessment (D3) and data treatment and outcome reporting (D4; Table  3 ), and nine items pertaining to different sources of bias within the domains (Table  4 ), along with signalling questions and a guide that can be used for RoB judgement. The choice of four domains, was accomplished by taking into account the stages and domains identified in the clinical/animal RoB tools identified and used in 3. 1. The assessment with this tool should be undertaken by two reviewers, working independently after calibration and, when necessary, adjustments can be implemented for each systematic review. An individual RoBDEMAT should be completed for each laboratory study included in the systematic review.
The development phase was undertaken according to similar studies that recently developed RoB tools. The development of these recent tools also featured a review stage and a Delphi process stage within a systematized, step-by-step approach, to achieve a final, consented tool  [26] [27] [28]. The present tool was designed to assist researchers and authors, journal editors, reviewers, systematic review teams and readers to evaluate the methodological quality of dental materials studies. In fact, systematic reviews of dental material studies from pre-clinical laboratory data are frequently published in current literature  [19]. The clinical RoB model tools retrieved in the search helped to categorize the domains that were chosen and voted for inclusion in RoB-DEMAT. These tools, mentioned in 3.
OPEN DATA The data used in the study is available under an open license from the national research data repository Recherche Data Gouv: https://doi. org/10. 57745/AQ0OPT. The source code is available from Zenodo: https://doi. org/10. 5281/zenodo. 6940484. ln addition, each graph provides a link to the table of plotted values and the source code that generated it.
To address this within the scope of the fusion community the FAIR for fusion (FAIR4Fusion) project  [4]  was initiated. Here we discuss the technical requirements and developments needed to transition the current and future range of fusion research data to an open and FAIR data sharing structure guided by the principle 'as open as possible, as closed as necessary'. Here, we report on the current set of recommendations and the technical developments undertaken in the FAIR4fusion project, towards providing FAIR fusion data.
Restrictions in scope The European Fusion community has a long history of fusion research activities that mainly have developed organically via the domestic research programmes. It is only recently that it has moved towards a more project-oriented structure with a joint coordinating structure, EUROfusion  [5]. Several of the EUROfusion partners maintain strong domestic research activities that remain at least partially outside of the formal EUROfusion collaboration. In the following discussions we limit the definition of the 'European fusion community' to mean the activities within the EUROfusion consortium as it maintains a consistent set of agreed on rules for joint access to research results and intellectual property rights which may be more complicated with respect to data developed for the domestic projects.
On a technical level this distinction is superfluous as integration of the EUROfusion data within a facility in a beneficiary or laboratory, is trivially extended to the whole data set of that entity. In addition, focus is on modelling data and data from experimental (tokamak and stellarator) facilities where an emerging joint data description is already available through the integrated data interfaces (IDS) from the ITER ('The Way' in Latin) integrated modelling and analysis suite (IMAS)  [6]  which carries experimental data and modelling data within a single data model. IMAS has been developed by ITER, based on original developments by EUROfusion Integrated Tokamak Modelling Task force, and consists of a 'data dictionary' that defines the supported variables (definitions and conventions).
Outline In section 2, the data and policy surveys undertaken by the Fair4Fusion consortium is analysed in view of the FAIR requirements and recommendations are provided. In section 3, a blueprint architecture is outlined that provides a proposed implementation for an FAIR based data sharing infrastructure in Europe building on existing tools and practices within current experiments and based on a minimal impact philosophy for the devices. A key component in opening the data is to maintain and improve the provenance captured from the origins of the data collection (be it physical or digital) to the final users and usages, and an extended discussion of provenance management and implementation is provided in section 4. Section 5 discusses experiences with developing demonstrators used to encapsulate expressed user needs and the resources needed to move from the demonstrator implementation to a production level tool is indicated in section 6, ending with a closing summary section.
Experimental data Fusion data exists in various forms, but the one that most people think of immediately is the data arising from the major fusion experiments. Each of the major fusion experiments in the EU has developed a local infrastructure to acquire and then store data gathered during a plasma discharge. They have also developed procedures for converting the raw experimental data into scientifically useful data, and to produce derived data. The challenges to make this data FAIR are: the different formats that the data are stored in, the different definitions of what is stored, the variety of tools for processing and accessing the data and the policies that the data owner imposes. A common data format, with well-established conventions, such as the IMAS/IDS paradigm, supplemented with provenance capture is needed to resolve the situation.
An assessment of the FAIRness of present experimental data management practises has been made by contacting representatives of the following European experiments: ASDEX Upgrade, COMPASS, FTU, JET, MAST, W-7X, WEST: • Findable: all experiments have a metadata catalogue with 0D/1D quantities (time traces) and tools to browse it and formulate queries. However, each experiment has its own tool, capable of finding only the data of that experiment. There is no central metadata catalogue that would allow multi-machine searches, apart from some quite specific International Databases-which are not populated in a systematic way by the experiments. • Accessible (via authentication, so not open), for fusion researchers having an official link to an experiment, using access methods specific to that experiment • Not Interoperable between various experiments because each one is using its own ontology (both for data and metadata), partially justified by the differences in experimental diagnostic systems and data processing routines.
• Reusable, for fusion researchers having an official link to an experiment and being able to read provenance data and the experiment-specific data documentation. A major limitation of reusability for some applications (e. g. synthetic diagnostics) is the fact that machine descriptions and calibration data are sometimes not recorded in the local experiment's database. In summary, when considering a single experiment, its data has already today some degree of FAIRness in the context of that experiment. But when considering the whole potential dataset coming from the various fusion experiments, the fusion community has no simple means to exploit it in an FAIR way. A key objective for improving the FAIRness of the fusion data would be to provide to the EU fusion community a way to make scientific analysis interoperable across multiple fusion experiments, increasing the potential for new discoveries.
The benefits are to be found not only for usual manual database queries but would also enable the use of new methods of research with Data Mining and Machine Learning techniques at an unprecedented scale. Among the European experiments, only MAST-U has presently an active Open Data policy: by default, a three year embargo period is applied before public release of data, while 'immediate' openness is applied for data related to a publication: 'free access to all data behind published papers must be granted in a timely manner'. Licences for released data are not used yet by any experiment. The following recommendation towards evolution of policies and practises to make data more FAIR are promoted: • Findable: establish a central metadata catalogue, accessible and searchable (through a Web Portal), gathering data from multiple experiments.
The recommendation (in conjunction with the interoperability bullet below) is to use the IMAS Access Layer for this  [6] , although maybe through a simplified, more user-friendly interface for the public. • Interoperable between various experiments (both data and metadata) by using a standard ontology (IMAS)  [6]. This means mapping local ontologies to the IMAS data dictionary at some stage, before exposing it to users/public. Interoperable data must also carry information on provenance to track data processing and ontology mapping. More details about the use of IMAS for interoperability are given in section 3. 2. • Reusable, by making the access to the experiment documentation more systematic (e. g. machine description) and more open (to the public). Also, by increasing (when needed) the amount of provenance information contained within the data (see section 4).
Simulation and modelling data Parallel to the experimental data are modelling results that are used to predict future experimental results, to interpret present experimental results, or to make predictions for future devices. Whereas the number of EU experimental fusion devices is relatively limited, the number of codes in use or under development is large, and there is little standardisation in the output formats of the codes. While it was possible to engage with each of the major experiments directly, it was decided to distribute questionnaires to code developers and to people running codes ('data producers'). Forty responses from data producers were received, and thirty-one from code developers. Some responses are shown below in figures 1-4. Here Fair4Fusion has endorsed the recommendations of EUROfusion working groups for the creation of a Long Term Simulation Storage Facility where simulation results can be archived, and recommended that key simulation results be stored in IMAS IDSs, including the SUMMARY IDS that is the basis for making data findable.
Blueprint architecture The Blueprint architecture for Fusion Open Data Framework is one of the main outcomes of the Fair4Fusion project. It describes the current state of the art in terms of policies and data access and elaborates on the FAIRness status of the experiments and their repositories. Furthermore, it proposes, based on a set of user stories and requirements, architectural components along with their descriptions and technological options. It is complemented with a costs and benefits analysis, licensing options, gap analysis and an implementation roadmap. In this section we focus on community standards and the proposed licensing schema, more details can be found in the Blueprint architecture report  [7].
Architecture choices In the architecture we are assuming the use of the IMAS Data Dictionary as a standard ontology for making data and metadata interoperable across the various EU experiments, for the following reasons:  also placeholders for error bars and basic provenance information. The Summary IDS is defined in a machine-generic way and is usable for both experiments and simulations, we propose to use this ontology as the basis for the metadata standard in order to make European fusion experiments and simulation data FAIR. To guarantee that data generated by experiments are unique and can be referenced during its whole lifetime, the system will utilise Persistent Identifiers technology, such as DOI or ePIC, to register the data globally.
Architectural overview The proposed architecture, outlined in figure  5 , consists of three main building blocks, namely Metadata Ingests, Central Fair4Fusion Services, and Search and Access Services. Metadata Ingests are the entry point to the system for the metadata provided by Data Repositories associated with experiments. In the proposed design, Metadata Ingests stay within the administration of specific data repositories, thus the data repositories themselves can filter or amend data before they decide to expose it to the rest of the system. From Metadata Ingests, the metadata is transferred to the next block of the system, i. e. Central Fair4Fusion Services. The Core Metadata Services, being the heart of this block and the entire system in general, natively operate on the IMAS data format, but can accept different formats of metadata as input through the translation components.
Central Fair4Fusion Services provide supplementary functionality for the specification of data that is not strictly tied to experiments, such as user-level annotations or citations. The last main block of the system is a set of Search and Access Services. It contains all user-oriented client tools that integrate with the Central Fair4Fusion Services. At this level focus is given to the Web Portal that is expected to offer an extensive set of functionalities for searching, mining, filtering, or displaying metadata and data managed within the system. Once a particular dataset has been selected as a result of queries on metadata (e. g. in the Web Portal), the Data Access Service will enable automated client access to the corresponding data. This service is the final gate for users of the Fair4Fusion services to the original data that is referenced (by metadata) in the Fair4Fusion portal.
This data will not reside within the Fair4Fusion services but remain at the originating sites (e. g. experiment sites), therefore remote connections will have to be open to transfer data on the fly (upon user request). Depending on local policy, the requested data will not necessarily be open and accessible directly via the FAIR4Fusion portal. In such a case, the minimal requirement is that the data access service will provide the instructions for accessing available datasets, assuming the user has the credentials to access the needed resources (e. g. the cluster of the targeted experiment). A more convenient solution (still in case of non-open data) would be to embed an authentication mechanism in the data access service, so that the user of the FAIR4Fusion service can authenticate themselves towards the data server.
Proposed extension of community standards The Summary IDS provides a large coverage of the physics quantities that can be captured in fusion experiments but does not in general include more generic data entities which could help make the data more findable and accessible to non-fusion users, including funders, other researchers, and the public. We have thus decided to extend the Data Dictionary with additional FAIR information. A dedicated Dataset FAIR has been created as a placeholder for FAIR metadata that is not immutable but will evolve during the lifetime of the dataset, such as validity of the dataset, licensing, references attached to the dataset. Based on the requirements we have selected several Dublin Core Elements to include in this new IDS. In addition, we have extended the IDS_properties structure of all IDSs with a new structure to record the provenance of the data stored in the IDS.
Provenance in fusion The current situation in fusion is that aspects of provenance are recorded in myriad ways across the various sites. Through a combination of digital and analogue methods, provenance data is captured in ad-hoc formats, describing aspects of a given signal's history to various degrees of detail. Additionally, this information is usually not accessible via any standard protocol, e. g. MDSplus  [8]. Consequently, an objective comparison between sites as to how effectively provenance is recorded is difficult to make. The FAIR4Fusion project looked in detail at the provenance capture of MAST/MAST-U and found that it was possible to extract some provenance information from the session log files, but these files are not in any standard format, meaning that the extraction of provenance information had to be done manually, or by parsing the logs with regular expressions, a brittle method with often unexpected results.