dataesr/scientific-paragraphs-categorization huggingface dataset:
• Once the dataset is published along with the preliminary QARs, the in-depth assessment to complete the QARs can start, as described in the previous section. The independent assessment is part of the in-depth cycle that always starts after the dataset is published with its QAR. However, for new datasets it would be convenient that the data provider performs the technical assessment, that is, data checks, and reports the evidence logs to the EQC team. The EQC team then controls that evidence is available for the entire dataset and performs random checks autonomously on a subset of the entire dataset. The reason for this logical flow is that there are technical limitations that make the data checks timely only when done by the provider. Indeed, the downloading and queuing time and the disk storage requirements are technical limitations that would demand more resources for EQC, while these resources are likely already allocated on the provider's side.
PROTOCOLS FOR GAP ANALYSIS Communication channels to inform C3S with recommendations to avoid gaps, address drawbacks and shortcomings and identify limitations have been established. These issues are reported via the EQC communication channels in the form of tickets sent to the rest of C3S. The tickets are sorted by resolution timing and priority as follows: • Short-term issues (<1 month) requiring quick attention, critical/blocking issues. These answer questions along the lines of 'Is it something that hampers the EQC work?', 'Is it something limiting the user experience significantly?', 'Is it an obvious bug, an error on the website?', such as a Catalogue entry lands to page-not-found. • Mid-term issues (1 to 6 months) identified problems and recommendations about the CDS data. These answer questions along the lines of 'Is it a problem that requires extensive analyses or impacts several aspects of the CDS?', such as unclear data licences.
• Long-term issues (>6 months) based on user requirements, non-blocking issues. These answer questions along the lines of 'Is it a user need that the EQC team is constantly facing when engaging with the users?', 'Is it a requirement coming from the EQC acting as a user?', such as the entry point to the Catalogue could be more efficient shifting from dataset category-based to variable-based. These requirements are inserted in a user requirement database and then analysed to become tickets. Usually, these tickets steer the evolution of the service over time. The different issues are analysed by C3S and may trigger internal processes to deal with them. In this respect, the EQC team supports the evolution of C3S through gap analysis of the current capabilities of the CDS and formulates recommendations.
COMMON PRACTICES TO ENSURE CONSISTENCY ACROSS DATASET CATEGORIES One key common practice to ensure consistency across dataset categories is to define a common vocabulary. The definition of shared vocabularies and common practices provides a foundation for interoperability, reduces interpretation ambiguities, and boosts an efficient communication exchange. The efforts to harmonise existing terminologies in a structured vocabulary aim to facilitate the C3S products usage by the downstream and upstream users, and it is also beneficial for the coordination with the rest of the C3S activities ensuring consistency when referring to specific CDS elements. It shall be noted that the lack of an overarching consistent EQC vocabulary was identified as one of the priority gaps in climate data quality  (Nightingale et al.
2019 ). Agreeing on a common terminology is by no means a simple task, as it is time-consuming and comes with a variety of challenges, especially in the case of C3S where datasets come from different communities adopting different conventions. For instance, numerous terms are interpreted differently across data communities. What is defined as 'product' in the satellite observations community 13 is way different to what the seasonal forecasts community 14 refers to or to what the ECMWF MARS (Meteorological Archival and Retrieval System) archive considers it to mean. 15 Some terms are very general (e. g. , 'observation') and lead to long discussions to reach an agreement. For these cases, a practical solution has been to include mostly CDSrelated terms, leaving out general terms as much as possible.
The definitions are continuously monitored and improved. According to the FAIR principles  (Wilkinson et al. 2016) , it is critical to use controlled vocabularies to describe and structure (meta)data in order to ensure findability and interoperability. A common vocabulary also refers to a set of common standards for data and metadata (formats and conventions) to be enforced by C3S. Indeed, gathering the metadata in a single system, as the CDS, with a common format, needs standardisation, as it is necessary to encourage data providers to convert their metadata inventories into formatted inventories that can be transferred to the C3S service. Including metadata in a consolidated and centralised system requires and/or encourages providers to agree to share the information with the community at large  (Aguilar et al.
2003; Brunet et al. 2020). Having a common metadata standard for the many communities gathered by C3S is not realistic, because these communities have their own standards. Thus, a first practical approach is that each dataset category follows a communityrecognised standard, as identified by the EQC team. Examples are CMIP, CORDEX, ESA-CCI, and obs4MIPs metadata conventions. Finally, consistency of the EQC framework is also achieved by commitment to transparency following the TRUST Principles for Data Repositories  (Lin et al. 2020). In particular, the methodology, the software and the assessments are made available to the users through the QARs public in the CDS. This helps to increase transparency and verifiability of the assessment, as well as resilience of the processes considered, which is open to further improvement.
LESSONS LEARNED The evolution of the EQC function would benefit from optimising the protocols, the templates and the workflow implemented so far, while tackling the gaps identified. The main issues encountered and more general considerations follow: • EQC workflow is a multi-actor process: collaborative iteration is key. Several lessons have been learned while working on the QAR production and made clear that the EQC framework is a multi-actor process that requires collaborative iterations with different stakeholders. Tied collaboration between the C3S contracts and approver, user engagement, and data provider is extremely important for sustaining the delivery of EQC information. Clear responsibilities and timelines have been defined. Perhaps, the most important interaction is with the data providers/producers for filling the QATs.
Responses from the providers are sometimes sparse or non-existent. Data providers/producers are considered the best source to fully describe their datasets and are the preferential choice to contribute to the QARs. This gap has been narrowed by: o including the officer in charge of the relationship with the data provider contractor (a technical officer in the case of C3S), who facilitates the interaction with the provider and ensures that specialistic knowledge about the data under scrutiny is fully accounted for; o ensuring that the EQC takes place earlier in the data ingestion process than now where the EQC work is done on already published data; and o making all EQC-related tasks a contractual obligation in the data provider commitments. For brokered datasets (i. e. , pre-existing dataset, not subject to the Copernicus license, to which C3S only acquires a license for the purpose of making it available in the CDS), the situation is not so different, because the contact point with EQC is the broker.
• Technical constraints limit the extension of the minimum requirements. The current version of the minimum requirements (MRs) fits the existing technological infrastructure as well as available human resources. The most important constraints to be faced are downloading and queueing time, memory disk space needs, lack of metadata information (that the provider should make available) about valid ranges for some variables, need for (land/sea) mask, and lack of automatic tools for data checks of each dataset category. for easier discovery and interoperability of the data (e. g. , spatial and time coverage description, keywords). • Capacity building both in terms of human resources and technologies. The implementation of the EQC requires cross-disciplinary knowledge (e. g. , science, data management, computer engineering) to design protocols, software, and workflows following best practices.
A constraint emerged during the work described in this paper is the importance of designing solutions sustainable for a large number of datasets. Otherwise, it is not possible to guarantee the necessary throughput with the available resources. As a consequence, some choices that seem to be straightforward and better than what has been implemented could not be considered because they do not scale for the large number of datasets under scrutiny (e. g. , writing individual QARs for each variable). It was necessary to automise as many parts of the workflow as possible to guarantee a timely production of the QARs. Based on our experience, it is also challenging to find a sufficient number of experts willing to regularly review in-depth all data streams. Considering that on demand requests for review do not necessarily guarantee that the same level of expertise could be kept over time, it would be appropriate to identify suitable funding mechanisms and contractual arrangements to keep the experts engaged for a longer period.
The EQC framework must be tailored to the service infrastructure to be successful both in terms of human resources, coordination, and technology capabilities. This should translate into an increased effort towards capacity building, the need of which was also highlighted by  Hewitt et al. (2020)  for climate services in general. Capacity building shall be closely taken into account during the implementation of a sustainable EQC framework. • Optimise the production of more insightful independent assessments. The scientific assessment needs to expand towards the diagnostics and standard metrics considered most insightful for the users. It would be beneficial to engage with the data providers to identify common baseline metrics to be applied independently. This will help to converge towards stronger provider engagement and satisfaction with the assessments performed and to reduce iterations during the in-depth cycle of the QAR production process.
The huge and increasing amount of CDS datasets will need to consider pragmatic approaches to streamline the production of the independent assessment. While the fast cycle of the EQC framework has been made very efficient, more work will be needed to establish sustainable mechanisms for the detailed in-depth cycle of the EQC framework. • Consistent and shared vocabulary. A common terminology should be shared and kept improving across the various C3S components. This facilitates consistency within the C3S (across the Catalogue entries for instance), it backs the user support desk when answering frequent questions by users about terminology, it gives a reference for the users to consult when jargon or acronyms are found, and it supports the project management to avoid ambiguities across C3S contracts.
As such, the common vocabulary is considered a fundamental guidance document to be integrated in the C3S portal to benefit both users and service. • Benchmarking and cross-service coordination. Consolidation of the EQC function also passes through the investigation of the most recent approaches. Given the challenges and opportunities that arose while implementing the EQC framework of the CDS C3S, exploration of the existing literature and coordination with other Earth data services, and in particular the other Copernicus services, is a key task to build a state-of-art EQC framework (i. e. , benchmarking). It is important to investigate the standards and best practices implemented in similar services operating around the world (e. g. ,  Leadbetter et al. 2020 , RfII 2020). This helps to assess the applicability of the different approaches to the EQC function.
• Scientific gaps warrant further research. There are clear scientific gaps that hinder smooth development of protocols for data quality assessments. Further research investments shall be considered by major funding bodies (e. g. , Horizon Europe) to fill the current scientific gaps. For instance, o the system maturity matrix CORE-CLIMAX (EUMETSAT 2014) was identified as a tool for the maturity assessments, but it exhibits limitations of scalability in an operational environment. An example, the in-situ observations GRUAN dataset Lacagnina et al. Data Science Journal DOI: 10. 5334/dsj-2022-010 required three months of work to complete the maturity assessment. In an environment like C3S, with increasing datasets and new versions, this effort is not practical. Scalability requires to detail the guidelines of the assessment (e.
g. , specify the metadata standards to check against or the source considered for citation to score usage). Moreover, the current scoring rules of the maturity matrix might require a peer-review process, possibly involving the data provider, to reduce subjectivity of the judgments. This poses once more challenges to produce scalable and timely assessments. In addition, the literature does not offer system maturity matrices for climate projections and seasonal forecasts, which opens intriguing questions about the different role maturity and verification play in modeling and observational communities. o Another case that would benefit from further research is the development of a metadata standard convention for seasonal forecasts and ERA5 data served in GRIB2 format. o Another example is the lack of well-defined ranges of physical plausibility for all the CDS variables, as well as a list of variable names and descriptions consistent across dataset categories.
For instance, so far it is not clear what reference to use to name and describe the surface temperature, it ranges from '(surface) temperature' in GCOS 21 to 'near-surface air temperature' in CMIP tables. 22 • User engagement as an integral component of the EQC. User engagement needs to be an integral part of the EQC evolution towards user-endorsed practices. It is inevitable that the users will drive the requirement for the provision of bespoke dataset quality information. The FP7 EUPORIAS project  (Buontempo et al. 2018 ) defined a set of principles for a successful climate service, which are particularly relevant for the design of an EQC framework in a user-driven context. The next phase of EQC would benefit from identifying the type of QAR content considered most useful from the user's perspective.
At present, our understanding of the usage of EQC assessments by users is still limited, but more attention to uncertainty characterisation, dataset stewardship, and ways the information is presented seem good candidates to start with. Thanks to the first QARs made available online, it will be insightful to test the efficacy of the communication strategies to ensure that appropriate and accurate quality assessments reach the users and are interpreted correctly. • Central repository of information. The tool used for the QAR production (the Content Management System in our case) may need to expand to consider the several data ingestion processes happening in C3S beyond EQC and may need to upgrade enhancing functionalities in relation to data import and synchronisation with the rest of the CDS information.
The tool may be better integrated into a single system for data ingestion and repository of information within the service to avoid duplication of effort and duplication of content describing the datasets. Better integration would also make the flow of information smoother across the actors involved in the service. Given the granularity of the datasets and their quality assessments, it may be convenient to make the quality information accessible through a structured API. • New challenges for the next phase. The next phase of EQC will need to evolve to tackle some new challenges. For instance, it remains to define the details of a workflow dealing with new datasets ready but not yet public in the CDS. Another example is defining how to trigger QAR updates and their maintenance due to dataset new versions, due to datasets extending over time and due to new documentation available.
Some practical directions have been suggested, but before adopting them operationally they would need to be assessed and tested. Evolution on how the quality information is disseminated (e. g. , synthesis table) and development of an advisory service about dataset robustness (e. g. , scoring scheme) will also deserve further exploration. A scoring system can be introduced so that users can quickly see which atomic elements of the EQC information have a good amount of detail. indicate the amount of quality information available. The scheme has to be simple and can be based on levels of increasing amount of detail/justification provided, as inspired by common practices in the literature (e. g. , Nightingale et al. 2018, GEO Label 23 ). While working on the EQC framework, some ideas have been already put forward, the levels of appraisal may depend on the fulfillment of the minimum requirements described in this paper, making the scoring objective and prone to automation.
SUMMARY AND CONCLUSIONS The current framework developed for the ramp-up operational phase of the Evaluation and Quality control (EQC) for the C3S CDS was presented. The framework considers the tasks, protocols and tools required to ensure that data are reliable and usable. It is inspired by the WMO GFCS guidelines, the ISOs 14090/1 and previous EU FP7/C3S projects. The framework is driven by a holistic approach aiming at homogenising the type of information made available across different climate datasets in a way that is both human and machine readable. It is characterised by a two-tier review system to assure the quality of dataset information released to the public. On the one hand, this approach enables fair and consistent comparison across datasets and facilitates guidance on the best use of data for the intended user's application; on the other hand, this approach makes the assessments sustainable and maintainable in an operational environment.
In doing so, the framework explored optimal mechanisms (e. g. , fast vs in-depth assessment and common vs non-common dataset information) for setting up sustaining delivery of EQC information, meeting the guidelines of the European Roadmap for Climate Services (EC 2015) and  Peng et al. (2021). The establishment of a quality management framework demonstrated benefits to the many actors involved: • the users: easy access and guidance to quality assurance information; • the data providers: feedback on data quality and incentive for improvement to increase data uptake and usability. This is in line with the good practice to connect the quality information with the dataset before release to reduce data misuse since this may also result in damage to the reputation of the data provider  (Peng et al.
2021 ); • the service itself: delivery of trusted and authoritative climate information, commitment to a user-driven evolution of the service. The service benefits from an established vehicle that triggers actions to improve the service itself; and • the funding agencies: to get a measure of how compliant the funded datasets or services are with specific requirements. As mentioned in the introduction, it is the first time that the methodologies, the metrics, the evaluation framework and the way to present all this information are being developed in an operational service that disseminates the majority of the climate dataset categories (including in-situ and satellite observations, seasonal forecasts, reanalysis, and climate projections). Building the underlying technical solutions makes the C3S EQC unique and requires pragmatic decisions for its implementation.
The first part of the EQC framework design focused on ensuring the robustness of the baselines and processes to collect the information required for the QATs, keeping their coherence and their comparability across all the datasets available in the CDS. Having a set of QATs covering consistently all the dataset categories is a unique endeavour. These activities needed continuous improvement to fine-tune the EQC framework, benefitting from the operational assessments (QARs) and user engagement process. During the second part of the EQC framework design, activities focused on defining the level of content required in the QARs and its homogeneity across. Optimisation of the QAR update by means of automation tools and workflow streamlining has also played an increasing role. The EQC framework was developed and implemented for all datasets published in the CDS at the beginning of 2020.
QARs have been generated at the granularity of the variable for each dataset and made available to the users via the CDS web platform. The implementation of the EQC function addressed most of the recommendations that arose during the preoperational phase (see  Nightingale et al. 2019)  QATs, (ii) enhancement of the CMS functionalities, (iii) identification of minimum requirements to publish a CDS dataset, (iv) establishment of an overarching consistent EQC vocabulary, (v) creation of guidance documents for evaluators and reviewers to guarantee consistency in the QAR production, (vi) regular benchmarking activities brought into the operational process, (vii) ability to track changes in the QAR content. Several constructive pieces of feedback from data providers, downstream users and C3S officers made the dissemination of the EQC information more robust over time.
Building on the  Lawrence et al. 's (2011)  checklist, the EQC developed a data checker software to detect whether the CDS data respond to the data models defined for the specific dataset category (i. e. community metadata standards, e. g. , ESA-CCI Data Standards V2. 1 24 ), have the expected format and metadata with no unforeseen gaps, and no suspicious outliers (physical plausibility). • Basic metrics (e. g. , bias, correlation, linear trends), appropriate for each dataset category, to check the scientific soundness and performance of the CDS datasets. Results are available through the synthesis table cell named 'expert evaluation'. These standard diagnostics represent a first step for more insightful scientific assessments to be developed over time. For instance, climate projections related metrics could include performance analyses of future climate projections simulations or more reference datasets could be considered.
Given the many different analyses options, priority shall be given according to the user needs. Based on our experience, it is anyway recommended to engage with the data providers to identify, together with EQC, common baseline metrics to be applied independently. This will help to converge towards stronger provider engagement and satisfaction with the assessments performed. • System maturity matrix (SMM) assessment, which is performed for these six clusters: metadata, user documentation, uncertainty characterisation, access/feedback/update, archive, and usage. The SMM model used is based on the CORE-CLIMAX approach and is intended to extend these assessments to all dataset categories in the CDS. However, the • users detecting and reporting via the helpdesk deficiencies or shortcomings with the dataset or with the QAR published.