dataesr/scientific-paragraphs-categorization huggingface dataset:
We can also see that NRPA is generally better than the SAT model, but SAT is surprisingly good for the difficult wap instances in Table  4 , unfortunately, we are unable to explain this result and further experiments are needed. However, a pair-wise comparison between the upper bounds discovered by NRPA and the ones discovered by HEAD demonstrates that HEAD is better in general. It is worth mentioning that algorithms such as HEAD include specialized graph coloring operators which have been extensively studied over the past decades. In contrast our algorithm is based on a general purpose implementation of NRPA, and only includes the specializations we have described in Section 4. Nevertheless, NRPA is the second best in all the datasets, and is often as good, or even better than HEAD. In Figure  1  we further analyze the behaviour of the two algorithms by comparing the execution times.
Comparison of the stability metrics In this section, experiments are conducted in order to compare the stability metrics. To that purpose, the benchmark used is described in Section 5. 1, a dominance function is introduced in Section 5. 2, then experimental results are presented and discussed in Section 5. 3. All experiments are conducted on a CPU Intel Xeon E5-1620 v3 @3. 5 Ghz with 64 GB RAM with a 600 seconds time limit. The subtour elimination constraints are added dynamically using Gurobi 9. 0. 0 as a solver with the lazyConstraints parameter. All models are implemented with Java in Eclipse IDE. Note that only three metrics here are compared, namely, sequence preservation, visit deviation and quantity deviation. As explained in Section 4 and shown in Table  6 , these metrics are the only ones judged both relevant and compatible with the IRP variant considered in this paper.
Methods We performed a bibliometric and lexical analysis of 95,415 scientific articles published between 1999 and 2018 in the highest impact factor journals within HBMS, using the Web of Science database and the CorText platform. HBMS's prevailing knowledge network of institutions was proxied with network maps where nodes represent affiliations and edges the most frequent co-authorships. The content of the prevailing HBMS research agenda was depicted through network maps of prevalent multi-terms found in titles, keywords, and abstracts.
For each step, we introduced a dynamic component by considering the two mentioned periods. A step-by-step methodology is provided in the following section. 1) Reconstruction of the prevailing HBMS knowledge network. Unlike other bibliometric databases, the WoS provides a separate field with authors' affiliations called "research institution". For each of the two selected periods, we used this field to map the most frequently connected affiliations. To build these network maps, we based our methodology on previous work. These previous studies showed that social network analyses using co-patenting and co-publication data allow for mapping relations between actors within a knowledge or innovation system  [23, 24]. This has become a standard way of measuring science-industry collaborations  [25].
Cooke  [26] , for example, analysed top impact factor journals from HBMS to proxy its prevailing knowledge network. However, this study focused on the geographical distribution of this network, overlooking the specific actors involved and their research agenda(s). Although the WoS presents an already cleaned database, affiliations frequently appeared spelled differently. To build a harmonized list, thus assuring that each institution appeared under only a single name, we followed the methodology presented in Tancoigne et al. [21]. First, we listed all institutions in alphabetical order and created a new field with a unified affiliation for each institution. We defined special criteria to determine each type of institution. In our retrieved corpus, each university was renamed as "univ" followed by the rest of its name in English.
Applied to our corpus, using the chi-square measure creates direct connections only for the most frequently co-authoring institutions from total co-authorships. Next, we used the Louvain community detection algorithm as our cluster detection method  [27]. A cluster means that in relation to the rest of the edges, those within the cluster represent the most frequent co-occurrences of the nodes that belong to that cluster. In the resulting maps (Figs  1  and 2 ), the nodes represent institutions (universities, research bodies, enterprises, etc. ). The size of the nodes represents the frequency of each institution's appearance in the dataset (thus each institution's publishing frequency for selected journals). As previously explained, we used chi-square, which is a normalized proximity measure that creates links (edges) towards higher degree nodes.
Acknowledgments We wish to thank  Hilary Rose ,  Steven Rose ,  Nicolas Rascovan ,  Pablo Nicolas Fernandez Larrosa ,  Diego Kozlowski ,  Javier Gasulla ,  Francisco Velazquez Duarte , and  Liliana Dain  for encouraging discussions that greatly helped to improve our manuscript. We also wish to thank  Alexandra Lamb ,  Katrina Slowski  and  Jeff Althouse  for the final editing of the manuscript. We are grateful to  Cortext Manager  team for their help.
The publication of this work was supported by  Carleton University  and the  LabEx Institut Francilien de Recherche Innovation Societe (IFRIS)  exploratory project " The Industry 4. 0 and the international division of labour ". Marc-Andre Ǵagnon  is an Associate Professor at  Carleton University. Cecilia Rikap was supported by an  IFRIS  (www. ifris. org)postdoctoral fellowship.
Knowledge about the quality of data and metadata is important to support informed decisions on the (re)use of individual datasets and is an essential part of the ecosystem that supports open science. Quality assessments reflect the reliability and usability of data. They need to be consistently curated, fully traceable, and adequately documented, as these are crucial for sound decision-and policy-making efforts that rely on data. Quality assessments also need to be consistently represented and readily integrated across systems and tools to allow for improved sharing of information on quality at the dataset level for individual quality attribute or dimension. Although the need for assessing the quality of data and associated information is well recognized, methodologies for an evaluation framework and presentation of resultant quality information to end users may not have been comprehensively addressed within and across disciplines.
Assessment of data quality is key for ensuring that the available data and information are credible and such assessments are essential when establishing trust for reuse of the data  (Callahan 2017). Trusted data are perceived as worthy of use in decision making environments where the metadata is sufficient to adequately describe the data, e. g. , information about the dataset author and data timeliness. Describing the quality of a data product and providing access to such quality information can support potential users of a particular dataset to determine whether it is appropriate for their planned usage, i. e. , fitness for purpose. A systematic analysis was carried out under European Commission to estimate the annual cost of not sharing data, which was found to be a minimum of €10. 2bn per year (European Commission and PwC EU Services 2018).
On average, data scientists spend 60-70% of their effort on dealing with data quality related issues (e. g. ,  Press 2016). Thus, not sharing data quality information will compound that loss, especially on productivity lost due to redundancy in assessing data quality. Although the importance of access to quality information is well recognized, methodologies for an evaluation framework and presentation of resultant quality information to end users may not have been comprehensively addressed. Access to this information is especially important for enabling data to be findable, accessible, interoperable, and reusable (FAIR). The FAIR data guiding principles defined by  Wilkinson et al. (2016)  emphasize the importance of data sharing in a machine-friendly environment. Since their inception, the FAIR data guiding principles have been adopted by global entities and have had a major impact in promoting data sharing and reuse globally (e.
g. ,  G20 Leaders 2016; Australia FAIR Access Working Group 2017; European Commission 2018 , 2020; Mons 2018; U. S. Public Law 115-435 2019; CODATA 2019). However, the FAIR data guiding principles are somewhat limited in that they call for only meta(data) to be associated with detailed provenance and "richly described with a plurality of accurate and relevant attributes"  (Wilkinson et al. 2016). They do not explicitly address the sharing of quality information of meta(data). For example, if the FAIRness of a dataset has been evaluated, what can be done to ensure that the method used and assessment results are readily findable, accessible, and (re)usable to end users? What can be done to ensure that the quality information can be readily integrated across different tools and systems within and out of individual organizations? Therefore, building on the direction that the FAIR guiding principles have provided for data sharing, we would like to go one step further and call for all dataset quality information to be FAIR to improve the sharing of quality information of individual datasets.
The optimal goal of this call is to allow for global access to and global harmonization of quality information of individual datasets as an important step towards open science in both machine-and humanfriendly environments. FAIR dataset quality information will also help improve data sharing as discussed in more detail in Section 4. Dataset quality can be affected by activities that are conducted throughout the data lifecycle. For example, lack of a quality assurance and control (QA/QC) procedure when generating the data product may impact the scientific quality of the data, while lack of the information about the QA/QC procedure will influence the quality of metadata and potentially reduce the level of user's confidence in trusting and using the data. In addition, data can be corrupted at any stage of the data lifecycle.
Moreover, managing quality throughout the entire dataset lifecycle is imperative for ensuring that the information and knowledge gained are not contaminated by inaccurate or corrupted data, as well as for facilitating accurate uncertainty estimates in the derived analyses. The value of lifecycle approaches to data quality has been recognized for various kinds of data, including remote sensing observations  (Barsi et al. 2019) , health services  (Kahn et al. 2015) , and health and biomedical citizen science  (Borda et al. 2020). Data lifecycle approaches to quality assessment also could be informed by lifecycle approaches to software quality  (Lenhardt et al. 2014). Another example where verifiable and consistent quality-controlled data are becoming increasingly important is in the domain of the emerging technologies of machine learning (ML) and artificial intelligence (AI).
Therefore, it is crucial to consistently record, curate, and represent quality information of individual datasets, and make it readily available and integrable. However, dataset quality information is not routinely curated and much less represented in a human-and machinereadable manner, despite the fact that international standards for describing the quality of geographic data have been in place since  2003 (e. g. , ISO 19157: 2013;; ISO 19115-1:2014). The lack of adoption of one or more data quality standards may in part reflect the diversity of approaches, availability of resources, technologies, networks, and research questions of investigators  (Leonelli 2017) , as well as the context for the planned purpose and use of the data  (Canali 2020; Illari 2014). Lack of motivation to document quality can be caused by the lack of prescriptiveness of existing standards -documentation of data quality metadata has always been optional in the ISO 19100 series and as of 2014, the ISO 19115-1 standard for metadata does not define a minimum set of discovery metadata, which used to suggest at least one data quality element (the provenance of a dataset).
Several other issues also may contribute to challenges for assessing and reporting data quality, and ultimately for the curation of dataset quality information. A frequently cited barrier against documenting the quality of spatial data is that it requires special domain-expert technical knowledge and across-domain expert knowledge integration, while documenting general metadata can be done automatically or by non-specialists  (Coetzee 2018; Peng et al. 2020a). Oftentimes, knowledge of the quality information resides with domain experts who need to contribute to the information but may not be fully aware of the importance of their contributions as the impact comes later on. The benefit of such knowledge may also be substantially less at an individual level than for the common good. Investigating barriers to assessing and reporting data quality information in medical bioinformatics,  Callahan et al.
(2017)  identified issues, at both organizational and individual levels, that contribute to deficiencies in quality assurance. Such organizational issues include inadequate support, unclear expectations and insufficient training such as the absence of best practices. Individual ones include consequential as well as process issues, such as unresolvable conditions. All these issues and challenges have led us to make this call-to-action statement to bring awareness about the needs and benefits of sharing quality information at the individual dataset level and rally people behind an important and challenging international and crossdisciplinary community effort. The paper is organized as follows. Section 2 provides definitions of some key terms used in this paper. Section 3 touches on multiplicity of dataset quality attributes and dimensions, which is one of the main challenges for assessing and curating dataset quality information.
Some potential benefits of having FAIR dataset quality information are described in Section 4, along with a brief summary of a real-life use case of how pre-vetted, timely and readily usable data and quality information is critical to disaster response efforts conducted by utility companies, with lives and billions of dollars at stake. A call for global community guidelines is then made in Section 5 which also includes an outline of community recommendations for developing such guidelines. A summary in Section 6 concludes the paper.
Dataset refers to an identifiable collection of data  (ISO 19115-1 2014) , and it can be published or curated by a single agent (W3C 2020). A dataset can be the digital rendition of a data product of a given version of an algorithm or model or experiment. A dataset may contain one or many data files or records in a database in an identical format, having the same variable(s) and product specification(s). Dataset quality information consists of information about the quality of data, metadata and documentation. Documentation can include descriptions of measurement methods and instruments, software, provenance, as well as that on the state of practices, workflows, frameworks, tools, and systems associated with the dataset and production, data and quality management, data services and usage, customer support and user engagement.
MULTI-DIMENSIONALITY OF DATASET QUALITY A dataset is associated with a number of distinct quality attributes or characteristics. For example,  Wang and Strong (1996)  identified over 179 individual data quality attributes through a survey from a data consumer perspective, attributes such as accuracy, correctness, freedom from bias. Furthermore, dataset quality attributes can be categorized into different perspectives or dimensions with emphasis on certain quality attributes (e. g. ,  Wang and Strong 1996; Redman 1996; Lee et al. 2002; Wilkinson et al. 2016; Ramapriyan et al. 2017). For instance,  Wang and Strong (1996)  prioritized the 179 quality attributes down to 15, and categorized them into four perspectives, i. e. , intrinsic, contextual, representational, accessibility. Redman (1996)  defined accuracy, completeness, consistency, and currency as four quality dimensions of data values.
Alternatively, based on the full dataset lifecycle,  Ramapriyan et al. (2017)  categorized quality attributes into four quality dimensions: science, product, stewardship, and service. These various groups of quality attributes are explicitly listed to demonstrate that they can differ greatly depending on the different perspectives. Assessment models are developed in Earth science to measure the maturity of different quality perspectives and dimensions at the dataset or collection level (see an overview by Peng 2018). However, there are very limited and sparse actionable guidelines on how to curate and represent dataset quality information in a way that is consistent with FAIR principles for improved sharing. Currently dataset quality information, when available, is published in science journals that are text-based and cannot be readily integrated into data management and stewardship processes or across different systems.
POTENTIAL BENEFITS OF FAIR DATASET QUALITY INFORMATION The FAIR data guiding principles emphasize the importance of data sharing by ensuring that data and data descriptions (metadata) are findable, accessible, interoperable, and reusable  (Wilkinson et al. , 2016). Findable data are discovered and understood by a search agent (e. g. , search engine or human user). Accessible data are rendered and used by machine and human end users via standard protocols within use and access constraints. Interoperable data can be readily used in conjunction with other data products or services and can also be integrated with other data to create new data products or services. Reusable data can be used, under the proper license and given well-documented dataset provenance, by diverse audiences beyond those who were initially envisioned as potential users by the original data producer(s).
Applying the FAIR guiding principles when curating and representing dataset quality information can help ensure the information is optimal for sharing and enabling global access. Successful reuse of data often depends critically on a potential user being able to access information about the quality of the data and determine its fitness for the intended application. Information about data quality also contributes to or improves the FAIRness of a dataset. For example, when data can be discovered based on information about certain quality attributes, the findability of the data is improved for users who need data that contain such attributes, and further, the quality information supports users to assess the relevance of a discovered dataset to a research or operational need. Global access to and sharing of dataset quality information will help improve data transparency and enable reproducibility, which is especially critical to highly-influential data that are used for decision-making (e.
g. ,  Tilmes et al. 2015b). Consistently curating and representing dataset quality information by following the FAIR principles could eventually lead to standardization within and across organizations, tools, and systems, which in turn will lead to harmonization of the information. In addition, describing quality information using standardized formats, schemas, and terminology with controlled vocabularies improves the interoperability and reusability of the data. Appendix A describes in detail a real-life use case of how trusted, timely and readily-integrable data and quality information is critical to disaster responses by utility companies with billions of dollars at stake. For disaster response managers, any information needs to be trusted and readily integrated, and understood in layman's terms in a matter of a minute.
Accuracy and timeliness of data and information is extremely important, and any datasets that are selected to be a part of their decision-making processes need to be trusted. Managers will not trust just any available datasets, since their decisions can have an impact on the safety and survival of at-risk populations, can cost up to millions of dollars, and influence the reputation of their organizations. For this use case, datasets are pre-vetted with an operational readiness level (ORL) ranking that is readily available and easily understood by decision makers who are generally non-data experts. An assigned ORL enables such decision makers to rapidly trust datasets. Data and information are integrated into a system which underpins an easy-to-understand dashboard for disaster response managers to allow them to make decisions promptly.
Thus, providing quality information along with the data establishes the trust needed for supporting such potentially life-saving emergency response activities, and maximizes the benefit of sharing data. More detailed information can be found in Appendix A. Pre-vetting datasets and developing the dashboard requires years of work and ongoing effort in addition to cultivated human relationships. Readily-available and consistently curated quality information of an individual dataset will help improve the process of establishing trust necessary to support tools and services provided to disaster responses, saving time and money. It will also support effective (re)use of the dataset for other applications, resulting in wide community utilization and therefore maximizing the value of the dataset.
CMIP6 data are published with persistent identifiers such as Digital Object Identifiers (DOIs)  (Stockhause and Lautenschlager 2017). Sharing and reusing dataset quality information is an even bigger challenge that requires global community effort. Issuing a call to the community for such an effort is the first step towards achieving that goal. To explore the needs for dataset quality information, approaches and challenges for consistently evaluating and representing the quality information, a one-day virtual workshop was held on Monday July 13, 2020. Domain experts from 9 countries across America, Europe and Oceania have participated in the workshop. It was followed by a report-out session on Wednesday July 22, 2020, during the virtual Earth Science Information Partners (ESIP) 2020 Summer Meeting (SM20), July 14-24, 2020.
Additional information can be found in the workshop report by  Peng et al. (2020a). A total of 14 presentations from organizations across 9 countries were given during the two live sessions of the workshop and the ESIP SM20 report-out session. Presenters summarized the data quality assessment approaches that have been developed and/or adopted by organizations representing the scope of national and international Earth science data producers, data management stewardship programs, data and service centers as well as data and information providers. Participants also included data users from academic and private sectors. The needs, challenges and approaches of consistently curating digital Earth science data and products were discussed during the live sessions as well as online during the following weeks.
QUALITY-ATTRIBUTE AGNOSTIC GUIDELINES As characterized in Section 2, assessing dataset quality is a multi-dimensional problem. The selection of the relevant attributes is context-dependent and it leads to different categorizations and practical dimensions  (Redman 1996). The complexity exists even within one discipline that a quality attribute can have different definitions, and be measured and represented differently. An example of this is data uncertainty as explored by  Moroni et al. (2019). The selection of the relevant attributes is context-dependent because datasets are often crafted for specific designated communities. Traditionally, designated communities of data consumers are domain literate and have some familiarity with the scientific context, data generation, or intended data use.