dataesr/scientific-paragraphs-categorization huggingface dataset:
Discussion The RaDiCo program is intended to promote, through the support of RD e-cohorts' projects, the collection of phenotypic data for epidemiological and clinical research purposes in connection with basic and translational research. To this end, we implemented an operational team (cf. Material and Methods section) with the mission to establish a centralized platform of expert services and tools to ensure installation and follow-up of the RD e-cohorts, via consulting services, for instance legal & regulatory services, or clinical research quality tools, development and provision of innovative information technology tools (electronic case report forms (eCRF), interoperability solutions, or access to e-health tools). The preparatory period of the cohorts was slowed down by the uncontrollable delays required to obtain all the ethical and regulatory authorizations, as well as the evolving regulations both at the EU level, with the new European regulation on personal data safety and security (GDPR) [ 36 ], and at the national level, with modi cations of the law on research involving human subjects (French «Jardé law»)[ 37 ].
Legal contracts were set up for consortium and collaboration agreements, enabling data collection and sharing. At the cohort level, we industrialized our production processes, as well as tooling methods such as standardized operating procedures or documents shared online. Recurring di culties are the shortage of manpower for data entry, the di culties in identifying cases in the different hospital information systems, or the legal barriers to collect data from deceased children. We implemented solutions through identifying key data and prioritization of the elds related to the main objectives to be completed by the clinical research technicians with support from the residents for lling the medical information. Clinical research technicians have been recruited by the RDHNs and speci c resources have been obtained through targeted research projects supported by Public-Private Partnerships or applications to speci c grants.
We explored the impact of RaDiCo for the medical community. Based on current information provided by all the cohorts' participating teams, the program should contribute to signi cant improvements in patients' care and outcome. Indeed, the design of the cohorts is made in such a way that it allows structured collection of disease symptoms at various stages of the pathological processes. Expected bene ts include better knowledge of the natural history of all the investigated RDs, recognition of relevant comorbidities, novel proposals for healthcare management, and, ultimately, better quality of life for the patients and their families. In line with the objectives speci cally de ned for each cohort, other expectations are: identi cation of relevant disease biomarkers for diagnosis, disease severity and exacerbations; patient selection for clinical and therapeutic trials; and production of novel quality of life questionnaires.
In line with these needs, the Ministry of Health has set up a national bank for rare disease data (BNDMR). All the reference centers and their network of teams in France have the obligation to implement this data bank. Standardized and detailed patient information collected through the cohort programs support the production of economic indicators. Moreover, access to the French Health Data Hub [ 38 ] that is dedicated to cross-referencing the health databases and facilitating their use by research teams with full respect of user privacy of the health system, will open access to patients' health expenditures from three main databases: the National Health Insurance Fund ("Caisse All the teams of the RaDiCo cohorts are members of the RDCCs in France. Among their tasks is the production of the RDs National Protocols for Diagnosis and Care (PNDS) [ 42 ].
Therefore, through their organizations and their multidisciplinary teams, all the cohorts actively participate in the production of PNDS. The ERNs are networks of RD centers of expertise, healthcare providers and laboratories that are organized across borders. Most of the medical teams associated with the RaDiCo program are involved in the activities of the ERN in their eld. They participate in the implementation of European data registries, in the production of clinical practice recommendations, and in the setting of epidemiological and economic indicators.
Conclusion The RaDiCo program promotes the collection of RD phenotypic data of different types for epidemiological and clinical research purposes in connection with basic and translational research. The RaDiCo platform offers the cohorts an information system based on the cloud principle (Information as a Service) together with a common core of services and speci c procedures for some cohorts to ensure installation and follow-up of the RD e-cohorts. RaDiCo IS can drive a virtually unlimited number of RD ecohorts, a major strength compared to other IS that usually deal with only one cohort. Currently, 13 RD ecohorts are implemented and other national, European and international cohorts are targeted.
RaDiCo's national call for implementing RD cohorts In 2014, RaDiCo launched a national call for RD Cohort proposals organized as a two-stage procedure (letter of intention / full application). Guidelines and templates for candidate cohorts were designed and made accessible on line to applicants. The full dossiers were evaluated by independent international experts (at least 3 per project), with pre-established evaluation criteria. The RaDiCo Scienti c Committee (SC) for the whole Program is composed of 18 members with complementary expertise associating coordinators of RDRC, RDHN, epidemiologists, biostatisticians, experts in Information Systems, molecular geneticists, directors of Inserm Research Units in the eld of RDs, and representatives of Inserm Thematic Institutes. Its role is to support the Executive Committee concerning strategic issues and scienti c governance of the program, including management of the call for projects and speci c follow-up of cohorts; and to contribute to the national call for RD cohort projects launched by RaDiCo in 2014.
An Executive Committee (EC) is shared by all the cohorts. It is composed of 4 members with complementary expertise (SG, AC, PL and SA). Its role is to ensure the implementation, development and monitoring of the entire program. Its main missions are operational management and deployment planning of the sustainability and internationalization of the RaDiCo platform, the links with each RD cohort, and a factorization of the shares. The EC is responsible for the scienti c management of the program. The management of the scienti c program is implemented at different levels: by the Executive Committee, the Scienti c Committee, the Inserm Thematic Institutes of "Public health", "Technology for Health", "Genetics, Genomics and Bioinformatics", as well as by the 3 scienti c governance bodies of each RD cohort.
The RaDiCo platform RaDiCo platform uses exchange format and data security in compliance with the European directive on the General Data Protection (GDPR). The RaDiCo work plan for each cohort is structured to achieve four levels of interoperability: technological, semantic, syntactic, and institutional. This allows networking and optimizing the use of existing RD patient cohorts at the EU level, while allowing integration of new types of data and technologies. The data of each RaDiCo cohort and associated services have to be compliant with the FAIR principles (Findable, Accessible, Interoperable and Reusable) by both people and computers. An important step in the FAIR data approach is to publish existing and new datasets from RaDiCo cohorts in a semantically interoperable format that can be understood by computer systems.
Nationale d'Assurance Maladie") [ 39 ], hospital stays from the French hospital discharge database (PMSI standing for "Programme de Médicalisation du Système d'Information") [ 40 ] and to the causes of death from the Inserm Centre for Epidemiology on the Medical Causes of Death, in charge of the national statistics of medical causes of death ("CepiDC") [ 41 ]. Economic indicators aim at monitoring the development and implementation of national strategies for RDs, which imply homogeneity and standardization of patient management by the medical structures dedicated to RDs.
Acknowledgments: Inserm is the sponsor of the RaDiCo cohorts. Study data were collected and managed using REDCap (Research electronic data capture) hosted at the  Inserm/Sorbonne Université research unit UMR_S933 RaDiCo. REDCap is a secure, web-based application from the  Vanderbilt University, USA , designed to support data capture for research studies, providing an intuitive interface for validated data entry; audit trails for tracking data manipulation and export procedures; automated export procedures for seamless data downloads to common statistical packages; and procedures for importing data from external sources.
The consequence is that it drives the users in a closed infrastructure, made of customized storage and access services. situ observations with satellite or modelized environmental data on a long time period for the detection and the modeling of 40 any new threat on the animal habitat due to Climate and Environment changes (reference EO4wildlife). Cloud and bigdata technologies can help through their focus on horizontal scalability, by distributing the workload and I/O across a potentially resizable pool of resources. However, the effort to acquire new technical skills and to switch to this type of technology can be expensive and a barrier for scientists developing their data processing and artificial intelligence models by their own. 45 Many oceanographic, climate, Earth observations datasets are in NetCDF format.
This format is widely used to represent multidimensional arrays (see Unidata site). While the NetCDF format and the libraries ecosystem were initially designed for local, random-access files several approaches have been developed for enhancing the NetCDF libraries to make efficient use of cloud storage and parallel cloud processing. Support has also been added to the NetCDF-Java library to utilize HTTP byte-range reading into NetCDF files stored in S3-compatible object stores to emulate random-access file reading. Using the new S3 50 capabilities of the NetCDF-Java library, the THREDDS Data Server (TDS) can provide data access services to S3-stored NetCDF files. Work has also be done for adding support to read and write Zarr data in the NetCDF core libraries (NetCDF-C and -Java). Preliminary results based on this initial THREDDS S3 work are presented in this paper.
Pangeo is already widely used by scientists in the geoscience field to process data in bigdata architectures using Python (see https://speakerdeck. com/rabernat/pangeo-earthcube-tac-update?slide=13). The conversion of NetCDF data into Zarr data, 55 based on NumPy, and the use of Dask are logical in the Pangeo ecosystem. In the industry and many firms, such as CLS, we can still find places where the development environment can be Python oriented, but the operational environment is based on Spark/Java/Scala technologies. Thus the idea is to find a common storage format and provide a unique datalake for both environments. Several big data formats are candidate to extend the use of data in a Cloudera/Spark infrastructure. Cloud Optimized Geotiff, 60 HSDS object store, Parquet: each of them has pros and cons, (reference https://ntrs.
The objective of this study is to propose a light new open source solution which is able to store gridded datasets into a native big data format and make data available for parallel processing, analytics or artificial intelligence learning. There is a demand 20 for developing a unique storage solution that would be opened to different users: • Scientists, setting up their prototypes and models in their customized environment and qualifying their data to publish as Copernicus datatsets for instance; • Operational teams, in charge of the daily processing of data which can be run in another environment, to ingest the product in an archive and make it available to end-users for additional model and data science processing. 25 Data ingestion and storage are key factors to study to ensure good performances in further subsetting access services and parallel processing.
Through typical end users' use cases, four storage and services implementations are compared through benchmarks: • Unidata's THREDDS Data Server (TDS) which is a traditional NetCDF data access service solution built on the NetCDF-Java, 30 • an extension of the THREDDS Data Server using object store, • pangeo/Dask/Python ecosystem, • and the alternative Hadoop/Spark/Parquet solution, driven by CLS technical and business requirements.
Introduction 35 Thanks to the possibility to manage big and various datasets with velocity in big data architecture, there is an opportunity to quickly extract bigger data volumes, process them for new value-added products on larger geographical and temporal scales, with higher geographical and temporal resolutions, and provide more precise output data to build up new data science models intelligence from existing scientific data. For instance, in the animal monitoring field, scientists need to correlate locations, in- For the datasets, the following criteria were identified to stress these implementations: • Significant data volume: data is available over a one year duration, • Variability in the geographical distribution: the geographical coverage is typically on the ocean or on the land, 75 • Significant set of variables to estimate the impact on the performances of the services.
For the use cases, we identified 3 typical scenarios: • extraction of data in a geographical area over a time period for a subset of variables (3D or 4D) via a subsetter service, • enrichment of CSV locations with variables values as additional CSV columns, • Parallelized processing. 80 https://doi. org/10. 5194/gmd-2021-138 Preprint. Discussion started: 2 July 2021 c Author(s) 2021. CC BY 4. 0 License. The selected datasets are open data from the Copernicus Core Services. The modelized collection GLOBAL_ANALYSIS_FORECAST_PHY_001_024, from CMEMS, is covering the oceans, big continuous surfaces, with a diversity of variables and time steps resolution (see a short description of tested data in table 1 and for a complete description https://resources. marine. copernicus. eu/?option=com_csw&view=details&product_id=GLOBAL_ANALYSIS_FORECAST_ 85 PHY_001_024).
The lake water quality product, https://land. copernicus. eu/global/products/lwq, from CLMS, is made of small geographical spots over European and African regions. All these datasets are available as NetCDF files. According to each implementation, datasets can be converted into another 95 format to optimize the execution time of the scenarios. The first scenario is made of a set of 100 requests to a subsetter service. It is defined in order to run and replay the same extractions for each implementation. Storage size for inputs and outputs of the requests and execution times of the subsetter service are discussed in the results item for each implementation. 100 For the second scenario, an enrichment along tracks is performed. A set of additional variables, growing in numbers, is added to all positions in the tracks, over a specific region or worldwide.
Metrics • Global execution time: on a hot instance (service initialized, up and ready to request) For the marine dataset, the criteria of subsetting are: A first set of requests is performed to try to assess the incidence of the dataset characteristics on subsetting results and estimate 145 the limits of the implementation due to the time coverage or number of variables to extract. The system configuration for these extractions is a Linux server with 10 cores and 4G of memory. • A list A second set of requests is performed to see how the number of cores, for a given dataset and a given request, is impacting the subsetting performances and how scalable the solution is. The number of cores is increased for the Pangeo-Zarr and the Spark-Parquet environments. 150 All the requests are described in an excel file available with the tested datasets (see Data Availability).
THREDDS implementation Unidata's THREDDS Data Server (TDS) provides a variety of data access services that can be used to serve NetCDF datasets. The NetCDF Subset Service (NCSS) supports coordinate system requests and can return the requested data in several formats 195 including NetCDF-4. Unidata's TDS uses the NetCDF-Java library to read the datasets to be served by the TDS. This is the THREDDS implementation. By extending the NetCDF-Java library to support byte-range access to object stores that support the S3 API, the existing TDS services can now be used to serve NetCDF files stored as objects in an object store. It is the THREDDS-S3 implementation. While this new S3 access can be used to serve individual NetCDF files, extending the new S3 capability to TDS aggregation 200 of datasets is still being implemented.
This means that the year of daily files for the two CMEMS hourly and daily mean dataset collections cannot be served as a virtual aggregation of the existing files but must instead be concatenated into a single file per dataset collection. This concatenation was done using the NCO `ncrcat` command. TDS services (including NCSS) were designed for targeted subset data access and were not designed to handle requests that 205 result in very large responses. The size of results from some of the benchmark requests resulted in responses that exceeded size limits in place on the systems. The perhaps related second issue involved requests reaching timeout limits. While the TDS handles multiple requests in a multi-threaded manor, individual requests are not currently handled in a multithreaded manor.
Pangeo implementation Pangeo is, first and foremost, a community of people working collaboratively to develop software and infrastructure for geoscience research on large volume of data. Among the products developed by this community, there are interconnected https://doi. org/10. 5194/gmd-2021-138 Preprint. Discussion started: 2 July 2021 c Author(s) 2021. CC BY 4. 0 License. software that can be deployed in cloud computing or high-performance computing (HPC) environments. Such a deployment 220 is also known as a "Pangeo environment. " These environments are built around a software stack using the Python computer language and reference libraries, allowing large-scale processing. The central element of this ecosystem is the Dask library. This library enables to run calculations on a distributed or local cluster.
Dask uses task graphs and data structures to distribute tasks and memory. Data structures are handled in memory by Dask. Third-party libraries such as Zarr takes care of reading and writing data to the disk. Data 225 processing libraries, such as Dask, manage data internally by dividing it into smaller chunks to enable distributed access, efficient compression and to reduce input/output access time on the file system. In this implementation, the conversion of the different NetCDF datasets into Zarr format is performed by a specific Python script based on the NetCDF4 module. The written data is compressed with the LZ4 algorithm using a compression level of 5. When relevant for certain physical variables, filters (ex. , delta, scale factor, add offset) are added to improve the compression 230 rate of these variables.
Open the use of the solution to a large community of users For item 1: Existing data cubes are ingesting data to be able to process them in the same projection, if needed the same 240 geospatial resolution 'see open cube ingestion'. Scientists want to keep the original information or have a complete description of the ingestion mechanism, which is not clear once data is available through the data cube access services. So reprojections, or conversion of units, interpolations are prohibited in this solution. This introduces a limitation to manage X,Y geographical representation used in polar datasets for instance. The single recommendation in place is to homogenise the precision in latitude and longitude between datasets to be able to use such latitudes or longitudes key values during select or join operations.
The 245 decimal precision is set to 0. 00001 degrees, representing approximately a 1-meter precision at the Equator line. Depending on the variable type defining longitude and latitude, this rounding operation is also useful to avoid floating point error and failing jointures during requests. For item 2: a scalable docker ingestion process is in place to deal with inputs of GBytes and thousands of files. 250 For item 3: Parquet is a common big data format standard. Several libraries, In Java, Scala, Python or R, familiar to data scientist and educational communities, are reading it. Final storage can be done through additional local storage disks, HDFS or object storage like S3 buckets for big volumes available in big data infrastructures. Be aware that HDFS replicated storage can be typically three times the original size of a file.
The operational storage costs are not studied here. 255 For item 4: The proposed parquet cube solution can be the input format of a scalable subsetter service, to extract data for scientists matching their geographical, time coverage and environmental variables list of interest. The output of the subsetter service can load data in memory for further parallel processing or can generate a downloadable output file for any scientist who wants to process data directly on his computer. The first option is to write the output as a NetCDF file, but the storage 260   In addition to this parquet file, two separate files are generated: • A dimension index file: An index for each dimension (one file per dimension). This metadata file stores all the attributes extracted from the original NetCDF files of the dataset.
This can be useful to generate a NetCDF output in line with the NetCDF input but also to implement a description service of the dataset (not described here). Due to the size of the tested input files and some typical end user requests, the partitioning-per-day makes sense The partitioning key is calculated for each data point from the value of the time dimension in the format YYYYDDD (day of year). This allows to benefit from quick partition pruning based on dates, since almost all our use cases are applied to a pre-defined time interval.
285 During the data transformation from NetCDF to parquet format, the following basic transformations are performed to avoid additional operations in the subsetter service: • Normalization and rescaling of latitude/longitude values (latitudes are rescaled to the range  [-90, 90 ] and longitudes to the interval  [-180, 180] ). This enables the ability to join several datasets on the latitude/longitude without considering the ranges. 290 • Time dimension values are converted to unix epoch timestamps, to avoid disparities between datasets. • Variable values read from NetCDF files are rescaled using the offset and the scale factor declared in the NetCDF file metadata, to allow comparisons of values between datasets (this would otherwise needs calculating the real values during the processing).
• It is to be noted that we implement a special handling for dataset that mix 3d and 4d variables: Since all variables, 295 regardless of their dimensions are grouped in the same parquet table, data is stored as if the depth dimension was nullable and all variables were 4d variables. • the following example illustrates how the parquet table looks like in this specific case: The 4d variables are considered as "null" (empty cell) when the depth is "null", and the 3d variables are considered as "null" when depth is not. The parquet files structure is stored on HDFS, and Apache Spark is used to read and process data from parquet files in a 315 distributed manner. This Parquet storage can be accessed through a dedicated subsetter service or several technologies like Spark or Python Dask notebooks.