dataesr/scientific-paragraphs-categorization huggingface dataset:
The workflow was designed and applied in the production of the Status of Coral Reefs of the World: 2020 report, where more than two hundred individual datasets were integrated to assess the status and trends of hard coral cover at the global scale. The workflow was applied to two case studies and associated R codes, based on the experience acquired during the production of this report. The proposed workflow allows for the integration of datasets with different levels of taxonomic and spatial precision, with a high degree of reproducibility. It provides a conceptual and technical framework for the integration of ecological monitoring data, allowing for the estimation of temporal trends in biodiversity and ecosystems or to test ecological hypotheses at larger scales.
In contrast to other scientific fields such as physics, oceanography or genetics, ecology is based on a high diversity and heterogeneity of data collection methods and hence data formats  (Michener and Jones, 2012; Poisot et al. , 2019; Reichman et al. , 2011) , and this, despite the existence of data standards (e. g. DarwinCore  (Wieczorek et al. , 2012) ). This leads to a lack of interoperability between datasets, which represents a major challenge for wider data integration. This is likely one of the main reasons why data integration, which is needed for the "bottom-up" approach, remains poorly developed  (Henry et al. (2008)  but see  Miller et al. , 2019; O'Donnell et al. , 2021)  outside of large databases such as GBIF (GBIF: The Global Biodiversity Information Facility, 2021) or OBIS  (Ocean Biodiversity Information System, OBIS (2021) ).
An increased use of data integration may lead to a deeper understanding of status and trends in biodiversity and ecosystems, without having to acquire new data  (Carpenter et al. , 2009; Jones et al. , 2006). Moreover, as emphasized by  Borregaard and Hart (2016) , data preparation, which includes data integration, is barely considered and reported as part of data analysis and is rarely documented in code associated with published articles. This represents a major issue for transparent science and reproducibility as the data preparation step can also contain errors that other researchers must be able to track. Finally, this also limits the ability for other researchers to rely on existing methods to conduct similar studies. To address these issues, we propose a workflow which integrates ecological monitoring data from different data sources into a synthetic dataset, which can then be used to perform national, regional or global analyses on the status and trends of the considered ecological metric.
Data gathering The first step in the workflow is the selection of variables that will have to be present in the final synthetic dataset (hereafter "standard variables"; Fig. 2 -1. 1). We propose a classification of these variables into six groups: data descriptor (e. g. dataset ID, data source), spatial (e. g. latitude, longitude, depth), temporal (e. g. year, date, hour of sampling), methodological (e. g. length of the transect, number of quadrat), taxonomic (e. g. family, genus, species) and metric (e. g. percentage cover, abundance, size). The metric(s) variable(s) correspond to the response(s) variable(s) while all others correspond to potential explanatory variables or metadata. The variables selected in each of these groups depends on the goal of the project and on the future analyses that will be performed.
Among the data descriptor variable group, a variable corresponding to the ID of each dataset must be included in order to ensure the possibility of extracting or to perform sensitivity analyses on the individual datasets. We emphasize that the groups of spatial and taxonomic variables are nested. For example, in the case of spatial variables, we can have four variables with, for example, the country, the site, the transect and the quadrat. A country may have several sites, and each site can include several transects, which can then include multiple quadrats. These nested variables are very important as they make it possible to integrate datasets with different spatial and taxonomic precisions. Once the variables are selected, their types (e. g. character, numeric) and units (e. g. meters) must be defined.
As previously mentioned, data can originate from four main sources: databases, data papers, research articles with associated data and unpublished raw data from data providers. The acquisition of data from the first three sources (Fig. 2 -1. 5 to 1. 7) can be achieved through internet literature reviews and institutional or public repositories  (Michener, 2018a). In addition to these approaches, the fourth source of data may necessitate a call for contribution using existing mailing lists or social networks. For this last data source, once the list of potential data providers is established, an email template is written (Fig. 2 -1. 2) to describe the context and the goal of the project, as well as the required data (by describing the standard variables). The way that people will be cited and acknowledged in the documents that will be produced by the project (e.
Individual data reformatting The next step is the individual dataset reformatting which corresponds to a standardization of the variables of each dataset gathered from the four data sources. The raw data are first imported into the software, either from their format of origin or by an intermediate step, where they are first exported in a plain text format (e. g. csv, txt). The importation of raw data in their format of origin ensures a complete reproducibility but requires the use of specific packages (e. g. "readxl" on R  (Wickham and Bryan, 2019) ) in order to work with all of the different raw data formats. If the intermediate step, where the raw data files are exported in plain text format, is chosen, all of the file paths and spreadsheet names must be written to ensure the traceability of the data.
Multiple data files can also occur when data are stored in one file but are divided into several spreadsheets. This division is usually done to separate the different years or sites within the monitoring programs. Two cases can be considered to address such formatting. If the data in the different spreadsheets share the exact same formatting (i. e. same columns names) they can be bound together using a loop into a single, long dataset. Otherwise, each spreadsheet must be treated as a different  Variables corresponding to the standard variables (see part 2. 1 Data gathering) are first selected. Then, if the data are presented in wide format (i. e. one variable divided into several columns) they must be transformed into long format (i. e. one variable by column). Next, the variable corresponding to the ID of the dataset is added.
Here we propose a code of several letters associated with one (e. g. DATA1) or more numbers if the data comes from multiple spreadsheets within a single file (e. g. DATA1. 1, DATA1. 2, etc. ). Then, all variables selected for the synthetic dataset but absent in the raw data or associated metadata files, are added (Fig. 2 -2. 3). For example, these variables may have been created specifically for the project purpose or may correspond to information given by the data provider. To ensure reproducibility, we recommend that any correspondences with data providers are tracked and that they are referenced by adding comments in the code. The variables are then renamed to match the standard variable names. All variables containing information on the taxonomic level are grouped together in a temporary variable named "Tax_ID".
Data grouping and taxonomic assignment Once all data files are individually reformatted, they are all bound together (Fig. 2 3. 1). This can be done automatically by retrieving all files with a specific naming structure in the storage folder and binding them using a loop. Some error messages can occur when the variable types differ between the individual reformatted files. While such errors can be avoided by converting all variables in character string, we argue that these error messages are useful to identify mistakes in variable types. If such errors happen, a modification of the code for individual datasets concerned in the previous step is necessary (Fig. 2 -2. 3). As mentioned above, all information related to the taxonomy is, at this stage, stored in the unique variable ("Tax_ID"). This variable is extracted from the main data and duplicates are removed to only keep unique levels of factor (Fig.
Taxonomic re-categorization The first pathway (Fig. 2  -path 3. A) corresponds to the case where the selected variables are not (or partially) included within the taxonomic classification system. Such cases can arise when the taxonomic identification in the field is difficult and therefore invokes the use of broad categories instead of species or genus level ID. This is the case for coral reef benthic monitoring data with categories such as algae, rock or hard coral. In this situation, levels of factor (i. e. names of categories) are likely to vary greatly among data providers. To achieve a homogeneous classification within the synthetic dataset it is thus necessary to perform a re-categorization of the levels of factor used by each data provider. To do so, a classification is first defined by choosing standardized levels of factor, which should make it possible to re-categorize all cases.
Taxonomic verification The second pathway (Fig. 2  -path 3. B) corresponds to the case where the selected variables are part of the taxonomic classification system (e. g. family, genus, species). First, an API (Application Programming Interface, i. e. a service which allows for the query and/or upload of data from the web in a standardized format) of an online taxonomic database (e. g. "rfishbase"  (Boettiger et al. , 2012)  and "taxize"  (Chamberlain et al. , 2020)  packages on R) is used to add the higher taxonomic variables for each level of the "Tax_ID" variable. The levels of factor of the "Tax_ID" variable for which upper taxonomic variables were completed through the database correspond to correct species names, while those which remained filled with NA correspond to incorrect species names.
2  -3. 6). Fig. 4  illustrates the taxonomic assignment for path 3. B. Following the taxonomic assignment, it is possible that some rows for the taxonomic variables were left empty (i. e. fill with NA) because they corresponded either to not required (e. g. "Shadow") or noncategorizable categories (e. g. "juvenile fish"). We recommend that the levels of factor of "Tax_ID" for which taxonomic variables were left empty be verified, and the corresponding rows then be deleted. Rows that had different levels of factor for "Tax_ID" before the taxonomic assignment (e. g. "Red algae" and "Brown algae") may have been re-categorized with a common level of factor for taxonomic variable (e. g. "Algae") after the taxonomic assignment. Hence, it is necessary to aggregate these rows (on the metric(s) variable(s)) to avoid having multiple rows which correspond to the same category in the same sampling unit, which unnecessarily increases the size of the dataset (Fig.
Site coordinates Errors in data entry or an incorrect transformation of coordinates (e. g. non-homogenized coordinate reference systems) may cause the presence of invalid site coordinates. The identification of some invalid positions can be assessed through visual inspection, such as by using an interactive map (e. g. R package "leaflet"  (Cheng et al. , 2019) ). However, the response time of interactive maps tends to increase with the number of site coordinates. Thus, an alternative is to perform this verification for each individual dataset (Fig. 2 -2. 3), when the number of coordinates is still manageable. The advantage of this method is that it allows for site positions to be confirmed to ensure that they are consistent within the geographic extent of each dataset. While some invalid site coordinates are easily identifiable (e.
g. located on land while the data come from marine monitoring programs) others can be more difficult to identify, for example when the error in latitude and longitude is so small that it leads to a slight modification of the site position. Finally, a broad automatic identification can be done using a polygon shapefile representing the area in which the sites are supposed to be present, at which point sites that do not fall inside this polygon can then be identified. Unfortunately, this method cannot totally replace a visual inspection. Once identified, the correction of invalid site coordinates must be done during the individual data reformatting step (Fig. 2 -2. 3). In addition to the QAQC step, we strongly encourage the data aggregator to send basic data visualisation and identified errors of individually integrated datasets to data providers and to ask for their feedback in order to ensure accuracy.
Export of the synthetic dataset Finally, when all of the errors are corrected, the final synthetic dataset is exported (Fig. 2 3 4. 3). We strongly recommend associating a metadata file to the synthetic dataset, where at least a description of the variables and their units are provided. However, if the synthetic dataset is to be shared, a more complete description of the data is required and the use of the Ecological Metadata Language (EML) can be considered for this purpose  (Fegraus et al. , 2005). Metadata can include the name and contact of data aggregator(s), the link for the code repository used for data integration, a full description of individual datasets integrated, as well an appropriate reference to cite the synthetic dataset.
Case studies To illustrate our workflow and to facilitate its use, we have provided an R code template for two case studies. These case studies are inspired by the Status of Coral Reefs of the World: 2020 report for which 248 datasets of coral reef benthic monitoring data were integrated to assess the global status and trends of hard coral cover over a period of more than three decades  (Souter et al. , 2021). However, as the data used in the frame of this report were gathered through DSA and were restricted in their use, we created example datasets that do not correspond to real datasets but illustrate the main types of data formats encountered in this project. The first case study illustrates the integration of data from the monitoring of benthic communities (sessile organisms) in coral reefs.
It corresponds to path 3. A of the workflow, with a taxonomic recategorization. The second case study illustrates the integration of data from the monitoring of coral reef fish communities (mobile organisms). It corresponds to path 3. B of the workflow, with a taxonomic verification. The code template provided illustrates steps 2 (individual data reformatting), 3 (data grouping and taxonomic assignment) and 4 (QAQC) of the workflow, as the first step doesn't involve code. Each of these three steps are associated with an R script. The. Rmd format (rmarkdown  (Xie et al. , 2018) ) was selected for the different R scripts because it allows for a better segmentation and annotation of the code and process (necessary for the second step) and for the exportation of code and output to an HTML file which may include interactive tables, plots and maps (necessary for the last two steps, see Supplementary material).
Lessons learned and limits of the workflow The workflow design was possible thanks to the development of recent packages in R which make data wrangling easier, and which facilitate the access to online taxonomic databases and promote interactive data visualisation. Despite these technical improvements, data integration remains a time consuming task, as some steps are difficult to automate, and a full reproducibility can sometimes be hard to achieve. Within the context of the Status of Coral Reefs of the World: 2020 report  (Souter et al. , 2021) , the vast majority of the 248 datasets that were integrated were unpublished data which came from numerous data providers. For this reason, over the entire year that was necessary to one person to complete the data integration, the data gathering has represented the longest step in the workflow, involving the identification and contact of potential contributors, signing DSAs, and finally engaging in discussions with contributors to ensure that their data was properly reformatted.
Despite their financial cost, the organization of several workshops in different countries increased the visibility of the project, which significantly increased the number of data providers and facilitated the individual data reformatting step. However, in contrast to data publication, data sharing, on which the Status of Coral Reefs of the World: 2020 report  (Souter et al. , 2021)  was mainly built, limits the traceability of integrated individual datasets (as they are not associated with a DOI), reducing the reproducibility of the workflow. The trend towards increased data publication should nevertheless help to reduce this problem in the coming years  (Costello et al. , 2013; Shin et al. , 2020). The reformatting of individual data has also constituted a timeconsuming step, requiring between thirty minutes to several hours per dataset, depending on the complexity and machine readability of the data format as well as the completeness of the metadata provided.
Lack of data standards (e. g. Darwin Core) and appropriate data management practices are thus the main factors which explain the time required for this step and the difficulty for a complete automation. With regards to the reproducibility of this step, it can be improved by describing decisions taken to correct errors for each individual dataset, either directly in the code, or in metadata associated with the synthetic dataset. Given the difficulty of identifying benthic organisms on coral reefs, each monitoring program defined and used its own nomenclature, using broad benthic categories instead of the taxonomic classification system. For that reason, in the context of the Status of Coral Reefs of the World: 2020 report  (Souter et al. , 2021) , and for the third step of the workflow (path 3.
A), we defined a nested classification which allowed for the recategorization all the broad benthic categories used in the individual datasets. This process required an ecological expertise and was iterative, as the chosen nested classification was updated several times, when new cases were encountered. Once the nested classification was defined, a manual stage was necessary to re-categorize categories used in the individual datasets into the categories selected. This operation can be more or less laborious depending on the number or categories to recategorize. However, the taxonomic assignment step can be quick and almost fully automated in cases where taxonomic verification is used (path 3. B, true taxonomic categories) instead of taxonomic recategorization (path 3. A, non-taxonomic categories).
Finally, for the QAQC step, as the data integration process used for the Status of Coral Reefs of the World: 2020 report  (Souter et al. , 2021)  aimed to provide a ready-to-use synthetic dataset for data analysis, we deleted all data that could not have been corrected, which sometimes led to a loss up to 10% of an individual dataset. Our first case study can be easily adapted for the data integration of sessile communities (with metrics such as percentage cover) in the marine and terrestrial realms. However, we caution that for mobile communities (with metrics such as abundance or size), like fish or birds, the QAQC process is more difficult to perform as the data quality depends heavily on the sampling strategy. Nonetheless, it is important to note that all issues raised by the data heterogeneity cannot be fully resolved during the data integration process but that some of them may be addressed during the analysis itself.
full-data analysis  (Spake et al. , 2020) ) instead of using effect-size, which is usually used in ecological syntheses through meta-analyses (e. g. Côté et al. (2005) ). Finally, this method guarantees a high level of reproducibility in facilitating the identification and correction of errors committed during the data integration process. Workflows are particularly relevant for reproducibility  (Botvinik-Nezer et al. , 2020; Cohen-Boulakia et al. , 2017; Poisot et al. , 2016)  as they provide a visual representation of the different steps, and the possibility to adapt each step depending on the goals of the project  (Jones et al. , 2006). Until now, a vast majority of bottom-up approaches for data integration were developed by large databases, in particular GBIF (GBIF: The Global Biodiversity Information Facility, 2021) or OBIS  (OBIS, 2021).
These databases typically include a web-based interface to allow data providers to publish their datasets, such as the Integrated Publishing Toolkit (IPT) developed by GBIF  (Robertson et al. , 2014) , and are based on data standards, such as the DarwinCore  (Wieczorek et al. , 2012). Unlike these databases, which seek to increase the accessibility and standardization of biodiversity datasets, the workflow presented here aims to produce a ready-to-use synthetic dataset, adapted to the analyses that will be performed, but also to integrate datasets whose formats are not yet suited for incorporation into existing databases. Outside of large databases, the number of studies related to data integration remains limited and have been mainly focused on quality control (e. g. Belitz et al. , 2018; Dou et al.
Conclusion The purpose of the workflow presented in this study is to encourage researchers to integrate multiple ecological monitoring data into a single synthetic dataset in order to perform analyses on the status and trends in biodiversity and ecosystems at larger scales. The results of these analyses are essential to inform policy makers and to measure the effectiveness of global (e. g. CBD Post-2020 Biodiversity Framework), regional or national programs, that aim to protect biodiversity and ecosystems  (Balmford et al. , 2003). Data integration could also be of great interest for research topics, such as macroecology and biogeography, by testing ecological hypotheses at larger scales  (Carpenter et al. , 2009; König et al. , 2019; Poisot et al. , 2016). The need for data integration will likely increase in the coming years  (Miller et al.
Acknowledgments We thank the  International Coral Reef Initiative (ICRI)  and the  Prince Albert II of Monaco Foundation  for the funding. We thank  Claire Bissery ,  Charlotte Moritz ,  Jason Vii ,  Mary Donovan  and all the people involved in the Status of Coral Reefs of the World: 2020 report for the valuable discussions we had during the design of the workflow. We thank two anonymous reviewers for their helpful comments.
Spectrum clustering is a powerful strategy to minimize redundant mass spectral data by grouping highly similar mass spectra corresponding to repeatedly measured analytes. Based on spectrum similarity, near-identical spectra are grouped in clusters, after which each cluster can be represented by its so-called consensus spectrum for downstream processing. Although several algorithms for spectrum clustering have been adequately benchmarked and tested, the influence of the consensus spectrum generation step is rarely evaluated. Here, we present an implementation and benchmark of common consensus spectrum algorithms, including spectrum averaging, spectrum binning, the most similar spectrum, and the best-identified spectrum. We have analyzed diverse public datasets using two different clustering algorithms (spectra-cluster and MaRaCluster) to evaluate how the consensus spectrum generation procedure influences downstream peptide identification.