dataesr/scientific-paragraphs-categorization huggingface dataset:
-The same ordering holds when the source is a flat graph-based database. In this case, adopting the graph-based model for the target as well is not the best choice, since using nodes and edges to model complex multidimensional elements can be seen as some sort of normalization-like the one carried out when the target is a star schema. Documentbased solutions are cheaper as flat data are significantly denormalized, thus building a document-based solution is just a matter of creating the JSON objects, while the star and flat schemata require some more aggregation and normalization steps. To compare the solutions in terms of ETL performance we use the same approximation used for formulation complexity: (i) we classify data sources into normalized, nested, and flat; and (ii) we write a sample ETL query for each type of multidimensional element, each type of source schema, and each target logical design solution.
-A relational solution is chosen for the temporal (plain) hierarchy as well. Indeed, the source data only include a simple date attribute (the other levels must be derived during ETL), and the relational solution is the best one in terms of storage (goal 2, no need to support variety in this case since the temporal hierarchy has a fixed structure). -To encourage flexibility in presence of variety and better evolvability (goal 4) while ensuring good query performance and formulation (goal 1), for the product and order hierarchies (for the latter, limitedly to levels OrderCode and ShipmentMode) we adopt a shattered schema (since source data are in JSON form). -For the Customer hierarchy (which includes the knows non-onto hierarchy) we adopt the graph-based model, specifically, a flat schema.
Patrick Marcel is an Associate Professor at the University of Tours, France. His current research focuses on database, OLAP and data warehousing, personalization, recommender systems, exploratory data analysis and data narration. He authored numerous publications in international conferences and journals on these subjects. He served as program committee member in top tier international conferences, including ER, VLDB, EDBT. He is a member of the steering committee of DOLAP and a member of the regular editorial board of DKE. Stefano Rizzi is a Full Professor at the University of Bologna, Italy. He has authored more than 150 papers in international journals and conferences mainly in the fields of data warehousing, business intelligence, and pattern recognition. He is member of the steering committee of DOLAP and of the editorial board of DKE, and has been a member of the steering committee of the ER Conference.
Bimonte is a researcher at INRAE-TSCF. He received his Ph. D. from INSA-Lyon, France, in 2007. From 2007 to 2008 he carried out research at IMAG, France. He is an editorial board member of the International Journal of Decision Support System Technology and of the International Journal of Data Mining, Modeling and Management, as well as a member of the Commission on GeoVisualization of the International Cartographic Association. His research activities concern spatial data warehouses and spatial OLAP, visual languages, geographic information systems, spatiotemporal databases, and geovisualization.
Sandro Bimonte is a researcher at INRAE-TSCF. He received his Ph. D. from INSA-Lyon, France, in 2007. From 2007 to 2008 he carried out research at IMAG, France. He is an editorial board member of the International Journal of Decision Support System Technology and of the International Journal of Data Mining, Modeling and Management, as well as a member of the Commission on GeoVisualization of the International Cartographic Association. His research activities concern spatial data warehouses and spatial OLAP, visual languages, geographic information systems, spatiotemporal databases, and geovisualization.
INTRODUCTION Thirty-five years ago, invertebrates were famously called "the little things that run the world," (1) highlighting the importance of organisms that often remain in the background while vertebrates take center stage. Today, the critical roles of invertebrates for ecosystem functioning and services  (2, 3)  are more widely recognized in both the public consciousness and in organized conservation efforts. However, we still lack an understanding of the patterns and dynamics of Earth's invertebrate biodiversity, including basic questions such as which areas have the most species, which areas harbor concentrations of small-ranged species, and even whether there is a major global decline in insect biomass underway  (4). Without comprehensive, high-resolution geographic datasets on invertebrate groups, our knowledge of global biodiversity patterns is highly biased toward one branch of the tree of life: vertebrates  (5).
Without a base of experts to consult for each species, we need to rely heavily on computational approaches to overcome several challenges in assembling a map of global diversity. First, we must synthesize information scattered in (often obscure) literature, museum collections, and specimen databases. Second, such data and metadata require vetting, updating, and interpretation. Third, we need a workflow that takes occurrence data for thousands of species spanning a broad spectrum of data size and quality and then returns reasonable estimates of these species' distributions. Last, we need to account for different levels of research attention across the globe, resulting in geographic variation in field sampling intensity, the uneven taxonomic study of those samples, and variable progress in the continued improvement of taxonomic frameworks over time  (11).
g. ,  (21, 22) ]. These traits, combined with recent data synthesis efforts  (23, 24) , make them an attractive test case to assess congruence between invertebrate and vertebrate diversity centers. We present the most comprehensive ant occurrence dataset to date and use a multifaceted informatics and modeling pipeline (fig. S1 ) to reconstruct a global biodiversity map at 10-arc min resolution (~20 km at the equator) encompassing nearly all described ant species. We use estimates of species ranges to calculate global maps of ant species richness (the number of species in an area) and rarity (richness weighted by range size to emphasize small-ranged species). Previous analyses of global invertebrate diversity either used point samples of alpha diversity to model geographic patterns of species richness  (9, 17, 25)  or were limited in resolution to large administrative regions  (10, 18) , neither of which allow for both fine-scale richness and rarity estimates.
RESULTS AND DISCUSSION Our data compilation (fig. S1 ) recovered 1,802,913 occurrence records for the native distributions of 15,463 valid species and subspecies. From these raw records, after data correction, georeferencing coordinates from existing locality metadata, error checking, and filtering out records with errors or high uncertainty, 1,479,293 records (representing 14,328 species) were used for analysis. This constitutes considerably more geographic coverage than previous global studies on invertebrates [e. g. , this study: 159,061 unique georeferenced coordinates, mapped onto 47,385 unique 10-arc min grid cells (~20 km); as compared to earthworms, (9): 9212 unique sites and nematodes  (25) : 6759 samples, 1876 unique 30-arc sec grid cells (~1 km)]. These data were used to make range estimates with different methods based on data available for each species [buffered point(s): 5168 species; polygon based on alpha hull: 1554 species; or species distribution model (SDM) prediction within the alpha hull (after further spatially thinning occurrence records): 7606 species; see fig.
First, we evaluate whether the species distribution modeling step [complexity-tuned Maxent models  (31, 32)  predicted within the range polygon] has undue influence on the recovered richness centers, and we find that while these models contributed more detailed estimations of species ranges relative to univalue polygons, their effects on the location of richness centers are minimal (fig. S2 ). Second, we compare our richness estimates to a separate dataset [Global Ants Database (GLAD)  (24) ] of local community measurements of alpha diversity that (unlike our analysis) includes morphospecies. Although there is considerable variation in richness among local communities even on small spatial scales, the grid-cell prediction is both positively correlated and forms an upper bound for local measurements (i.
e. , local richness mostly does not exceed modeled regional richness). Third, we evaluate the robustness of our primary method of estimating and stacking species range estimates [e. g. ,  (33) ] using three alternate methods: (i) rarefaction and extrapolation of sampling curves, (ii) taxonomic surrogacy models (modeling species richness as a function of genus richness), and (iii) a climate-based macroecological model based on the GLAD data. In general, we find consistency between these methods and the range-stacking approach (fig. S3 ). However, we note that the only method that did not use the Global Ant Biodiversity Informatics (GABI) occurrence data (i. e. the macroecological model based on community data) did have the lowest correlations with all other methods (Spearman's rho = ~0. 6, compared to >0.
For example, ant biodiversity patterns are most congruent with those of reptiles, echoing a similar finding for the global spread of introduced species (  41 ), but the reasons for this correlation are unknown. Moreover, although global plant datasets at a comparable resolution were not available to include in this study, illuminating the level of geographic covariation between plant and insect diversity is a critically important goal for both basic science and conservation. Last, while our models aim to fill geographic gaps in knowledge, they are at best a temporary stopgap while we continue our global inventory. Models are hypotheses, and predictions of hidden diversity must be tested with future targeted inventory efforts for specific areas. Our analyses highlight how focused efforts can greatly increase knowledge of regional biodiversity; for example, Mesoamerica, Madagascar, Colombia, and the Atlantic Forest of Brazil are now robustly documented as ant biodiversity centers, largely due to vigorous research activity in these areas in recent years.
Data compilation The GABI database is intended to consolidate and curate ant geographic biodiversity data in a single place, synthesizing data from literature, online databases, museum databases, and personal collections  (23)  (see table  S1  for detailed data sources and data S1 for the full dataset). At the time of this study (data downloaded 14 July 2020), it is composed of 2,466,704 total records, but only 1,802,913 (73%) have a valid species name (following species nomenclature from AntCat. org  (42) ) and are not dubious or exotic records. We put these records through a data-cleaning, georeferencing, and optimization pipeline. Of these, 766,854 records we compiled from the literature, 1,177,653 records from 81 public databases [the largest component being 420,251 records from AntWeb (version 8.
66) (43)], and 42,084 records from personal collections and communications, representing 15,463 known ant species and subspecies (not including dubious or exotic records). The compilation workflow is summarized in fig. S1 ; more details on the methods for data compilation and databasing are described in a previous publication  (23) , and the data can be viewed through https://antmaps. org/ (44). Of these raw records, most lack georeferenced occurrence points, and many existing point records have errors. While previous analyses using earlier versions of this dataset assigned these records manually to large administrative areas (e. g. , states, countries) for analysis [e. g. (18, 41) ], here we used an informatics pipeline to clean and convert these raw records into an extensive georeferenced dataset, which we then used for downstream range modeling and diversity mapping.
Data cleaning, georeferencing, and optimization Species occurrence data are typically fraught with errors, missing fields, and various biases that necessitate cleaning and verification steps before it is used in analyses  (45, 46). Although several established tools exist for processing occurrence data [e. g. ,  (47) ], including some introduced quite recently  (48, 49) , most focus on online biodiversity databases, and the diverse nature of occurrence data problems often leads to low effectiveness of any one tool when applied to a new dataset  (50). These issues are compounded for datasets like GABI with a global extent, which include areas of the world poorly represented in locality-validation databases and records that originate from varied data sources. We initially experimented with single tools to validate and clean the GABI occurrence data, but found this strategy resulted in many inaccuracies, especially for regions outside North America, Europe, or Australia.
As GABI has over 2 million data records, it proved infeasible to manually inspect each record and correct inaccuracies. Geocoding tools, such as those offered by Google and implemented through Python or R packages, are not used often in ecological and evolutionary studies, but they can help fix a variety of spatial errors and suggest correct positioning of coordinates. We thus opted to use a combination of functionality from existing occurrence data cleaning tools, geocoding tools, and custom scripts that automate cleaning and validation procedures existing tools did not offer and then make manual fixes when necessary to optimize data accuracy (fig. S1 ). Our custom geoprocessing pipeline (i) converts raw text and numeric data gathered from the literature or specimen databases into georeferenced occurrence points and (ii) cross-checks point accuracy using a variety of methods (numbers here refer to R and Python scripts and can be found in data S1, https://doi.
org/10. 5061/dryad. wstqjq2pp). We first retrieved 1,802,913 raw ant occurrence records from the GABI database (downloaded 14 July 2020), of which only 1,062,720 records had coordinate information  [1]. To prepare the data for geocoding, we pooled duplicate localities based on locality or archipelago information from multiple fields and latitude/longitude coordinates (when available) rounded to four decimals (~11 m;  [2] ), ensured proper encoding for numeric fields  [3] , ensured coordinates did not have obvious errors in location (e. g. , outside of ±180°) or formatting  [4] , corrected character-encoding errors for locality fields (Python package ftfy (51);  [5] ), measured the fuzzy distance between original and corrected country names (Python package FuzzyWuzzy (52);  [5] ), and detected additional coordinate information by scanning text in other fields  [6].
We then addressed any remaining character-encoding errors for locality fields and confirmed country attributions by checking against global locality-validation databases [GeoNames (<https://geonames. org/>), United Nations Trade Statistic Country Code (<https://unstats. un. org/unsd/tradekb/ Knowledgebase/Country-Code>), Geo-Locate (<http://geo-locate. org/>), and Google geocoding returns] to generate a standardized locality string  [7] , and, finally, produced the geocoding dataset representing all the unique localities in the data  [8]. We conducted reverse geocoding (finding locality information based on available coordinates) and forward geocoding (finding coordinates based on locality information) on the cleaned dataset via the Google Geocoding API (53) using the geocoder Python package (54) (9) and then cleaned and parsed the geocoding return data, assigned levels of spatial precision  [10] , and integrated these data with the original database  [11].
The optimal records were then run through the CoordinateCleaner R package  (48) , which calculates metrics to judge the validity of the point data. We used this package to remove records that were judged to be polygon centroids and then made manual edits based on identified outliers for final locational corrections as many of the flags identified by the package were not applicable for ants and needed expert opinions to verify. This process assigned optimal coordinates derived from existing record metadata for 988,331 records (112,256 unique coordinates) and generated optimal coordinates for 490,962 geocoded records (48,675 unique coordinates), resulting in an optimized georeferenced dataset of 1,479,293 total records. No geocoded records were retained in the optimized dataset with a precision radius greater than 100 km, and most (82%) of these records have a spatial precision under 20 km.
We buffered alpha hulls and points by 30 km to account for spatial uncertainty, using the R package geobuffer  (56)  to make geodesic buffers for point data. As geodesic buffering for polygons was not possible in R and difficult to implement otherwise, we used the package rgeos (57) to make Euclidean buffers for polygonal data after projecting the data to the World Behrmann equal-area projection (as it allowed for some tools to draw geometry past the prime meridian instead of cutting it off using the "+over" specification in the coordinate reference system definition), and then projected back to the original World Geodetic System (WGS) 1984 geographic coordinate system. For species with ≥5 occurrence localities (7606 species), we first spatially thinned occurrences by 10 km [R package spThin  (58) ] to reduce the effects of sampling bias and then built SDMs to predict suitability within their study extent, which was defined by their buffered alpha hull range estimate.
We used the presence-background machine learning algorithm Maxent 3. 4. 1, which fits a relationship between occurrence localities and environmental variables constrained to best match the background environmental distribution  (32). For predictor variables, we used 19 global bioclimatic variables at 10-arc min resolution (approximately 20 km at the equator) from the WorldClim 2. 0 dataset (59) representing long-term annual trends, seasonality, and extremes that are relevant to describing the spatial patterns of species' ranges. The resolution was chosen to reflect the precision of the compiled and georeferenced data. From these rasters, we removed areas corresponding to inland water bodies (from <https://naturalearthdata. com>) and then masked them to each species' alpha hull range estimate for modeling using the R package raster  (60).
As inspections of the ant occurrence data revealed strong signals of sampling bias for multiple species even after spatial thinning, we sampled background localities for each SDM with the same bias inherent in the occurrence data using a sampling density grid  (61). To make this grid, we calculated the kernel density of the occurrence localities for all ant species in the GABI database across the globe [fig. S6 ; ArcGIS Pro 2. 6. 0 (62)] and removed inland water bodies. Then, after transforming these density values to a range between 0 and 1, we used them as proxies for sampling effort and probabilistically sampled 10,000 background points for each species. Species with smaller study extents (<10,000 grid cells) were assigned full background samples (i. e. , all grid cells) to avoid using too few background localities for model training.
In essence, the addition of complex feature classes (here, Q and H) adds potential complexity to the model, but increasing regularization will result in the increasing removal of both predictor variable features (i. e. , quadratic or hinge fits) and the variables themselves [i. e. , coefficients can be reduced to zero, dropping the variable from the model  (32) ]. We used the R package ENMeval 2. 0. 0 (31) to iteratively construct models with all combinations of these settings and evaluated models using k-fold cross-validation on withheld data: leave-one-out (i. e. , delete-one jackknife) for species with <25 occurrence localities [as this method maximizes sample size of validation data (64)], and random (k = 5) for all others. We did not use spatial cross-validation for evaluation, as our aim was simply to choose the models that best predicted current data and not to transfer models to different areas or time periods  (38).
We used the following sequential criteria for model selection based on both threshold-dependent and threshold-independent performance metrics  (63, 65). First, we filtered out all models without nonzero coefficients (possible because high regularization can reduce all predictor variable coefficients to zero). Next, we filtered out models that performed poorly (≤0 or NA) for the Continuous Boyce Index [R package ecospat  (66) ] calculated on the full dataset, as positive values indicate that the model's predictions are consistent with the distribution of occurrences  (67). We chose to calculate the Continuous Boyce Index on the full dataset as a standard measure across all species as opposed to on validation data because for many species, the validation scores were NA because of low sample size for partitions.
Comparison of richness estimates to community-level point observations As a test of the performance of our ant richness estimates, we used simple linear models to determine how well our grid cell richness corresponds to maximum observed community richness from a mostly independent ant community database. While regional richness is not necessarily expected to match community richness, it should form an upper bound and be correlated. The Global Ants Database (GLAD) (downloaded 20 February 2020) consists of over 50,000 ant occurrence records (>2300 unique localities) from over 200 community ecology studies around the world  (24). The GLAD database partially overlaps with the GABI database, in that occurrence records from community surveys may have been represented in published literature or specimen databases that were entered into GABI.
However, GLAD also contains many unpublished datasets and, importantly, includes morphospecies information in richness measurements. As the range-modeling approach using GABI point occurrence data can only be performed with described species (because morphospecies cannot be matched across different studies or localities), it allows us to check whether our grid-cell estimates are much too low because of a low fraction of described species in some areas. We used the existing fields to subset an analysis dataset consisting only of native species found in undisturbed areas, used the corrected versions of species names, and retained the identities of morphospecies by giving them unique names linked to the study that identified them. We also removed all occurrences of "?" from species names, assuming that the presumed identifications were correct.
Richness from rarefaction/extrapolation moving window We estimated richness using a rarefaction/extrapolation approach by calculating Hill numbers (76) (q = 0) based on the preprocessed GABI occurrence data (before removing point duplicates and spatial thinning for modeling). We used the R package iNEXT (77) to calculate the observed and estimated richness, as well as sampling completeness, for each grid cell based on the GABI occurrence data within a moving window (fig. S3 ). After experimenting with several different sizes, we decided to use 10-arc min grid cell windows of 60 × 60 (i. e. , 10° × 10°) to balance the smoothness of the mapped result with an appropriate spatial resolution to visualize global patterns. We summarized the GABI occurrences for all cells within each window as a frequency table that represents the number of occurrences per grid cell for each species (i.