dataesr/scientific-paragraphs-categorization huggingface dataset:
The relation of trust between data providers and downstream users generated by these advantages is a necessary condition to foster a flourishing market for climate services  (Zeng et al. 2019). While the need for a quality assessment of the data available on the CDS is well recognised, the methodologies, the metrics, the assessment framework, and the way to present all this information have never been developed before in an operational service that disseminates all the main climate dataset categories. The proof-of-concept framework described in  Nightingale et al. (2019)  focused on observations only. Instead, here we implement the operational delivery of homogeneous EQC information across all the CDS climate dataset categories mentioned before. This objective, along with the task of building the underlying technical solutions, posed an unprecedented design challenge and makes the C3S EQC unique.
The framework for the operational EQC function of the CDS builds on past and present international research initiatives. A variety of concepts were developed in previous projects (mainly the EU FP7 funded QA4ECV 2 and the C3S_51 Lots 2-4 3 ), most of these concepts were integrated while moving the EQC function towards its operational phase. This paper describes the challenges, the development and the implementation of the operational EQC function providing an overarching quality assurance service for the whole CDS. The results of the dataset assessments are publicly available in the CDS Catalogue. 4 The manuscript focuses on the framework implemented for the CDS datasets only, leaving out other aspects of the EQC function, such as the assessment of the software made available by the CDS to explore the datasets (i.
e. Toolbox), the assessment of the CDS infrastructure and the assessment of the user's satisfaction and requirements. The EQC function regularly informs and recommends C3S about drawbacks, shortcomings and limitations related to the CDS datasets. These analyses are completed by a continuous user-engagement process to identify the user expectations that need to be addressed. The EQC team provides technical and scientific quality information of the CDS datasets via a set of homogeneous Quality Assurance Reports (QARs), helping to set the minimum requirements and baseline criteria for including new datasets in the CDS Catalogue. The QARs are filled templates called Quality Assurance Templates (QATs). Consistency across the QATs is obtained through the adoption of a vocabulary of homogeneous concepts and common practices.
The general strategy for assessing the CDS datasets consists of five steps: • designing QATs for all the dataset categories with a consistent terminology and a structure as similar as possible; • interacting with the data providers, who are the ones with the best knowledge of their datasets, and encouraging them to fill in the QATs or, in case the data provider is not available, fill in the templates using the documentation publicly available; • evaluating the content collected in the QATs, paying attention to ensure that the content is understandable for the users, the level of detail is similar across datasets, and the type of information is complete, correct and consistent with the template requests; • performing an independent quality assessment of the dataset, looking at aspects like (meta)data completion and integrity, scientific soundness and other characteristics that illustrate the multi-faceted nature of data quality; and • publishing the information in the CDS dataset catalogue, once the QAR is approved by the corresponding authority, which in this case is the C3S governance board.
The production of the QARs calls for setting the procedures to initiate, develop and update the QARs (e. g. , workflow), developing the software tools to support the assessments (e. g. , data checker), and engaging with a wide range of stakeholders to choose the most adequate options for the QARs. These steps lead to the creation of QARs, which provide users with comprehensive information about the technical and scientific quality of the datasets. The different sections of the QARs are made accessible to the users in the CDS web portal through a synthesis table. The synthesis table is devised as a tool to organise and homogenise the EQC information, which is made of atomic elements corresponding to the different entries of this table. These entries contain links leading the user to the respective subsection of the QAR, where the user can find the EQC information of interest.
The overall EQC framework is guided by homogeneity and scalability approaches. The former leads to consistency of the EQC information across the CDS dataset categories, the latter leads to the integration of automatic tools to produce timely and sustainable data assessments in an operational environment. In particular, the EQC framework is driven by: • a modular and flexible system able to consider new data/information sources and new actors involved; • as much as possible automation of information acquisition (e. g. , variable description, metadata checks) and its update in order to reduce human errors, speed up the QAR production, and make the system sustainable in the long-term; • an iterative and reproducible approach permeable to the evolving requirements of both users and C3S to ensure continuous improvement; • a user-friendly presentation of the quality information provided, clustered to facilitate its consultation and uptake to facilitate users in making their own decisions about climate data; • consistent provision of the CDS dataset quality information, recognising the existence of inherent differences across the dataset categories; • transparency and traceability of the quality assessments; • FAIR  (Wilkinson et al.
FAIR-DQI GUIDELINES (PENG ET AL. 2021) EQC FRAMEWORK DESCRIBED IN THIS PAPER Guideline 1: dataset The dataset is described with a comprehensive online page providing various information that includes DOI, rich metadata, and licence. Guideline 2: assessment model The assessment method is available online together with the quality information. This paper itself details further the assessment model used. The assessment model is versioned and publicly retrievable. Guideline 3: quality metadata The assessments are captured into a structured schema/template (QAT). The quality information is standardised in a machine-readable (in our case using the CMS) and reusable form. Guideline 4: assessment report The quality information is structured in a template and is accessible online, versioned, and human-readable.
DATASET QUALITY INFORMATION AND ITS DISSEMINATION The QAT is the tool used to gather information on the most relevant aspects of the CDS datasets, informing the user in a quicker way rather than accessing and reading several documents (e. g. , user guides, peer-reviewed papers, dataset descriptions). The QAT includes all the relevant quality information, in a concise and standardised form, with references and links leading to further details. The general strategy is to provide seamless QATs, which are as homogenous as possible across all dataset categories. The QATs for each dataset category (i. e. , satellite and insitu observations, reanalysis, seasonal forecasts, climate projections) are available as supplementary material. The QATs are regularly reviewed to gradually converge towards harmonisation.
Much improvement has been achieved by adopting a common terminology (see section 6) and common minimum requirement fields. The homogenisation of the QATs of different dataset categories ideally tends towards adopting one single QAT for all datasets. However, this goal is not feasible due to the diverse nature of the CDS dataset categories (concepts like 'processing level' or 'quality flag' are relevant for observational datasets, but not for other categories; along the same lines, the concept of 'ensemble size of the hindcast' is mostly relevant for seasonal forecasts). This homogenisation effort was pragmatically addressed by mapping the different QATs, one for each category, onto a general table agnostic of the dataset type. In practice, all the QAT fields were grouped under main sections and subsections with common names for all dataset categories.
An excerpt of the resulting QAT is reported in The synthesis table entries contain links leading the user to the respective subsection of the QAR (i. e. , filled QAT), where the user can find the EQC information of interest. Therefore, the structure of the synthesis table is agnostic of the dataset category, while the QAT fields, within each subsection (the information that is displayed when clicking on a cell in the synthesis table ), depend on the dataset category. The synthesis table offers an effective approach to guide the users into the documentation and homogenise the access to it. It addresses a typical user requirement: 'most of the time, the problems with the documentation are not due to the lack of it, but to the difficulty in finding it' (extracted from the C3S User Requirement Analysis Document 09/2019) or 'all documentation should be easy-toaccess'  (Nightingale et al.
• Independent assessments. An analysis of the dataset quality is performed independently of the provider, with the advantage of using the same metrics and tools nonpartisan of the source where the dataset was generated. This guarantees a uniform and impartial basic evaluation across datasets. Information here is the result of the technical and scientific assessments. See Appendix III for a definition of 'technical' and 'scientific' in this context. The table is characterized by fields grouped into columns (Figure  2 ). The column with the header 'introduction' gives a quick overview of the data characteristics (e. g. , name, provider, time resolution), as inspired by the WIGOS guide on metadata standards (WMO/WIGOS 2017). The column 'user documentation' provides the essential documentation for the effective use and understanding of the dataset (e.
MINIMUM REQUIREMENTS FOR PUBLICATION OF CDS DATASETS Some of the QAT entries are considered mandatory and some optional in the EQC framework. The content of the mandatory entries is considered so fundamental that, when missing, the dataset is not usable/understandable and thus unservable. These mandatory fields define the minimum requirements (MRs) for a dataset variable to be published (or withdrawn) by the CDS. The identification of these fields probably represents the first systematic effort towards the inclusion and development of an operational check of MRs, encompassing a wide range of dataset categories. Indeed, the identification of a suitable set of MRs was indicated among the 'Science Gaps' in assessing climate data quality by  Nightingale et al. (2019). The list of MRs is specifically thought to facilitate a timely publication of a dataset in the CDS, ensuring, at the same time, a sufficient (but not necessarily optimal) quality of the dataset.
The MRs cover several aspects ranging from the dataset documentation to the compliance of metadata with community standards. The fields were identified as a result of the interaction between the EQC team, data providers, C3S, and users. The analysis of the MRs leads to recommendations to the C3S governance board on whether a dataset shall be made public on (or withdrawn from) the CDS. To guarantee the maintainability of the MRs, they are an integral part of the QATs and are updated with the same frequency. See the supplementary material for a complete list of the MRs, indicated by an asterisk next to the QAT entry. Typical examples of MRs are 'data format', 'physical quantity name', 'user guide documentation' or 'validation activity description'. Beyond the mandatory text necessary to fill in the QAT entries, a number of documents are also requested to be linked in the QATs as minimum requirements.
These are: • dataset user guide (e. g. , seasonal forecasts SEAS5 user guide 5 ). In the satellite observations community, this is usually referred to as PUG (Product User Guide); • documentation describing processing of the dataset or a model/system technical documentation, including the description of the different components. In the QAT this document answers the question labelled 'model/system technical documentation' (e. g. , reanalysis UERRA 6 ). In the satellite observations community, this is usually referred to as ATBD (Algorithm Theoretical Baseline Document); • product traceability chain. Only mandatory for reference datasets (e. g. , in-situ observations GRUAN humidity 7 ). Definition of reference in this context is given by GCOS 8 ; and • uncertainty characterisation and validations reports.
In the QAT these documents answer questions about 'validation or inter-comparison or uncertainty characterisation activities performed' (e. g. climate projections CORDEX-CCLM 9 ). In the satellite observations community, this is usually referred to as PQAD (Product Quality Assurance Document) and PQAR (Product Quality Assessment Report). Before publication in the CDS, it is essential that the documents listed above are made available alongside the datasets they refer to. The current version of the MRs fits the existing technology infrastructure as well as available human resources. Ideally, the list shall be extended to include basic technical checks about the data and metadata, such as time and space consistency and completeness, physical plausibility. However, it would require a technical infrastructure that was not available on the CDS at the time.
DEVELOPMENT OF THE TECHNICAL SOLUTION Building the framework of the EQC requires designing protocols, software tools, QATs, and workflows for the QAR production as well as following common vocabularies and practices (e. g. , TRUST Principles for Data Repositories,  Lin et al. 2020). Among these, we focus now on the technical solutions underlying the EQC framework. Substantial technical developments have been undertaken during the onset of the operational phase of the EQC and more will be needed while it gets more mature over time: • a Content Management System (CMS) and its maintenance, more details below; • a Drupal-based module, inspired by the shiny-app R package, 10 to show dynamic plots resulting from the scientific assessments; • the integration of the EQC tab into the CDS infrastructure and its synchronisation with the other catalogue elements (e.
g. , download tab); • software packages adapted to perform the scientific assessment tailored to the CDS characteristics (e. g. , the ESMValTool 11 was adapted for climate projections analyses); • a data-checker software to scrutinise CDS data and metadata; • compatibility tests to check whether the data variables can be served through the CDS Toolbox; and • setting up the software infrastructure, such as a network of virtual machines with the right environment and a Git manager for software repository, data flow architecture, and so on.
CONTENT MANAGEMENT SYSTEM (CMS) At the heart of the EQC assessments is the Content Management System (CMS), an application used to manage content stored in a database and displayed in a presentation layer based on a set of templates, i. e. the QATs. Its objective is to ease the collaborative definition of the QAT structure and to facilitate and manage the creation of the QAT content. Creation of the QAT content is partially automated, as detailed in section 5. 2. The CMS facilitates the QAR production following a workflow that involves several roles, described in Table  2 , that access the CMS sequentially.
EQC main contact One EQC team member who acts as the main contact of a specific dataset category. As QAR production is a multi-actor process, it is important that there is a central person, the EQC main contact, to coordinate the QAR production. This member contacts the data providers to agree on when they are available to fill in the QAT. Once the link with the providers is established, the EQC main contact defines the QAR name, fills in the QAT entries that identify the QAR uniquely and selects the team involved in the QAR production. Eventually, the EQC team member lets the actors involved know where there is a potential issue before it impacts the production.
Reviewer An EQC member who scrutinises the whole QAR content for completeness and understandability. The reviewer is fundamental, because of her/his work in checking and verifying the correctness and consistency of all information introduced, while interacting with the evaluator to address any issue encountered. Approver Role covered by one C3S governance board member, who makes decisions about the publication of the dataset, (also) based on the QAR, and conducts a final check of the QAR before making it public together with the dataset. If the QAR requires further review, it will be sent back to the EQC team, commenting about what is still needed. Otherwise, it will be published in the CDS. 10 https://shiny. rstudio. com/. 11 https://www. esmvaltool. org/. Figure  3  sketches the interaction between the roles involved in the generation of the QARs.
The implementation of this workflow into the QAR production needs more consideration; it needs to distinguish between fast and in-depth assessment cycles as well as between common and non-common QAT fields. Details are provided in the next section. To complete the list of roles involved in the CMS, two additional roles are considered but they are not directly part of the QAR production workflow: • QAR support role: can edit any part of published QARs to fix simple issues like typos or broken links or update technical data checks performed on new parts of a dataset regularly extended over time. Depending on the task, this role is covered by either an EQC member or a CMS automatic functionality. • Observer: can read and comment in the CMS but is not granted the right to insert any content. This role, typically covered by a C3S technical officer, can intervene in case of blocking issues with the provider or whenever some aspects in the QARs need improvement.
WORKFLOW AND PROCEDURES FOR THE QAR PRODUCTION The trade-off between timely and detailed assessment is tackled by splitting the QAR production workflow into two phases: • A first phase (the fast assessment), mostly focused on verification of the minimum requirement fields. The dataset stewardship is scrutinised in terms of documentation, accessibility, and compliance with metadata standards. • If the C3S governance board decides to make the fast assessment part of the QAR public together with the associated dataset, a second phase (the in-depth assessment) starts. During this phase, the complete independent assessment is performed, and the other fields are updated in case of need. The in-depth assessment focuses on the technical, scientific, and maturity data evaluation. An additional element that makes the QAR production sustainable is the identification of QAT fields associated with common content across several QARs.
WORKFLOW IN A NUTSHELL In a nutshell, the process for the QAR production for both already published and submitted for publication datasets may be summarised as follows: • Start: o triggered by a QAR release calendar or new datasets available in the CDS • Input: o data provider and EQC team fill in the QAT according to the workflow in the CMS Throughout the QAR production process, the user engagement team of the EQC iterates with the users to harvest their feedback about the different steps or improvements taken in a coproduction process, making sure to advance in the direction of fulfilling the user's needs. These user's requests will result in reports to be discussed at regular EQC meetings, where they shall be further investigated and eventually trigger framework and QAR updates. User requirements also help to refine the QATs and to prioritise the performance metrics to be employed during the independent assessment.
DATASETS ALREADY PUBLISHED The EQC function has been implemented after many datasets were already published in the CDS. As a consequence, a workflow needed to be envisaged to produce the necessary QARs. In this case, the trigger of the QAR initiation is a QAR release calendar, defined by the EQC team together with the C3S governance board. Once the QARs are triggered, the workflow is managed in the CMS, as shown in Figure  4. Given the same roles identified in section 4, the QAT is filled in the private domain during the fast assessment cycle and then published. Once public, the QAR is completed with the independent assessment during the in-depth cycle and finally updated in the public domain. Each assessment cycle distinguishes between common and non-common fields: • The part associated with the common fields requires the data provider expertise and does not include the independent assessment, as such this part involves many members, but it does not need to go through the in-depth cycle.
• The part associated with the non-common fields is unique to each QAR, because it is tailored to each variable. During the fast assessment, almost 12  any non-common field (e. g. , variable description) can be extracted from precompiled tables validated with the data provider and extracted automatically to fill in the unique QAR. This part of the QAR production is nearly automatic, so it can involve fewer members, the EQC main contact to initiate the QAR and a reviewer to guarantee that the content is meeting expectations. Once the fast assessment cycle is complete, the non-common fields follow the in-depth cycle, where the evaluator includes the independent assessment material. Once published, one QAR might need to be updated. More details about the procedure in this case are given in Appendix II.
DATASETS READY TO BE PUBLISHED In the future, the evolution of EQC will need to consider a workflow for datasets ready to be published. So far, this workflow has not been implemented. Several options are considered based on the lessons learned during the ramp-up phase of EQC. Here we give some recommendations. A new dataset is a dataset the provider considers ready to be served through the CDS. At this stage, the provider and C3S officers iterate to ingest data information, like documentation or the location where data are stored. Much of this information could be collected in the CMS (or a tool connected with the EQC CMS), which facilitates the completion of part of the QARs shortly after. Instead of EQC asking for similar information again, we can leverage the content already stored in the CMS to streamline the flow of information exchanged among the various authors involved, by introducing a workflow starting with the fast assessment explained in the previous section.
Once the fast assessment cycle is complete, the dataset could be either rejected because, for instance, the minimum documentation required is not complete, or accepted for publication. When the dataset and the initial QARs are public, the in-depth assessment cycle starts. The logical flow, illustrated in Figure  5 , may be summarised as follows: • When the dataset is ready to be evaluated, the C3S governance board opens a ticket addressed to the EQC team. • The EQC team meets regularly to assign the work for the QAR production based on, among other sources, the tickets received. • Once the related QARs are completed of the fast assessment and approved in the CMS, the CMS closes automatically the ticket considered. • The C3S governance board decides whether to publish, postpone the publication or discard the dataset using, among other input, the QARs made available.