dataesr/scientific-paragraphs-categorization huggingface dataset:
4b ). As expected, the most prevalent sequences were also the most abundant (Extended Data Fig. 4c ). Our analyses were carried out using a modest sequencing depth of 5,000 observations per sample after Deblur and rarefaction. To inves tigate how prevalence estimates were affected by sequencing depth, we focused on four major environment types for which we had the greatest number of samples with more than 50,000 observations (soil, saltwater, freshwater, and animal distal gut). The relationship between average tag sequence prevalence and sequencing depth differed among these environments (Extended Data Fig. 4d ) but was generally positive, suggesting that our global analysis underestimated true prevalence. Animalassociated microbiomes were a notable exception, with an upper bound on prevalence apparently imposed by host specificity when all host species were considered (Extended Data Fig.
4e ); this bound disappeared when considering only humanderived samples (Extended Data Fig. 4f ). Although contamination remains an issue in microbiome studies  24  , most of the very highly abundant and prevalent sequences here had higher mean relative abundances among samples than among notemplate controls (Supplementary Table  3 ), suggesting that they did not originate from reagents. Matches between our sequences and existing 16S rRNA gene reference databases highlight the novelty captured by the EMP. Exact matches to 46% of Greengenes  25  and 45% of SILVA  26  rRNA gene databases were found in our dataset, indicating that we 'recaptured' nearly half of the reference sequence diversity with just under 100 environmental surveys. These matches accounted for 10% and 13%, respectively, of the tag sequences in our dataset, indicating that large swathes of microbial community diversity are not yet captured in full length sequence databases.
Patterns of diversity reflect environment We used a structured categorical variable of microbial environments, EMPO, to analyse diversity in the EMP catalogue in the context of lessons from previous investigations 1,2. We observed environment dependent patterns in the number of observed tag sequences (alpha diversity), turnover and nestedness of taxa (betadiversity), and predicted genome properties (ecological strategy). Derived from a more standardized methodology, our dataset confirms the previous finding 2 that host association is a fundamental environmental factor that differentiates microbial communities (Fig. 2c , Extended Data Fig. 2d ). We build on this pattern by showing that there is less rich ness in hostassociated communities than in freeliving communi ties (Fig. 2a ), with the noted exception of plant rhizosphere samples, which resemble freeliving soil communities in both richness (Fig.
2a ) and composition (Fig. 2c ). Our findings also confirm the major com positional distinction between saline and nonsaline communities 1 (Fig. 2c ). The effect sizes of environmental factors on alpha and betadiversity generally showed large contributions of environment type and (for hostassociated samples) host species to both types of diversity (Extended Data Fig. 5a, b ). The ability to identify sample provenance using only a microbial community profile has applications ranging from criminal forensics to mistaken sample identification; these applications will require large curated datasets, such as the EMP. Supervised machine learning demonstrated that samples could be distinguished as being animal associated, plantassociated, saline freeliving, or nonsaline freeliving with 91% accuracy based solely on community composition, and to finescale environment with 84% accuracy (Extended Data Fig.
2d ). ACN can be predicted from 16S rRNA amplicon data  27  ; this method has been used, for example, to link the taxonomic groups associated with copiotrophic and oligotrophic behaviours in soils to high and low rRNA gene copy numbers, respectively  28. Approximately half the dataset centred on an ACN of 2. 2 (freeliving and plantassociated samples) and the other half on an ACN of 3. 4 (animalassociated samples) (Fig. 2d ). Greater pergenome rRNA operon copy number has been found to be associated with rapid maximum growth rates  29  , which may provide a selective advantage when resources are abundant, such as in animal hosts. While ACN is an estimate rather than a measurement of average rRNA copy number and is subject to potential biases in the underlying reference database, the distributions we observed are consistent with 16S rRNA copy number reflecting differences in ecological strategies among environments.
A resource for theoretical ecology The coordinated accumulation of data across studies allows investiga tions of patterns within (alphadiversity) and among (beta diversity) microbial communities at scales that vastly exceed what could be measured in any individual study. Patterns of alphadiversity in meta analyses have revealed global trends that have been key to the development of major ideas in macroecological theory, but fundamental patterns have been more difficult to discern in microbial ecology. For example, a nearly ubiquitous tendency towards greater diversity in the tropics is evident in macroecology  30  , but there is substantial variation among studies examining latitudinal trends of microbial diversity  [31] [32] [33]. The large EMP dataset analysed here reveals a weak but significant trend towards increasing diversity at lower latitudes in nonhost associated environments (Extended Data Fig.
5h ). An effect of latitude was apparent both within and across studies, consistent with global trends in latitudinal microbial diversity being an emergent function of locally selective environmental heterogeneity  34. However, substantial studytostudy variation in richness highlights the caveats inherent in metaanalysis; more coordination of sample collections from similar environments across larger gradients is necessary to better address this question. The EMP has the potential to link global patterns of microbial diversity with physicochemical parameters-if appropriate metadata are provided by researchers. Microbial community richness has been found to correlate with environmental factors, including pH and temperature  3, 33, 35, 36. For example, richness has been shown to increase up to neutral pH  36  and often to decrease above neutral pH 3,35 in soil communities.
Richness has been shown to increase with tempera ture up to a limit and then to decrease beyond that limit in seawater (maximum at about 19 °C)  33  and to increase with temperature in soil (up to at least around 26 °C)  36. However, general relationships of rich ness to temperature and pH remain unresolved  37. Here, across samples from nonhostassociated environments where pH or temperature were measured (mostly freshwater and soil environments), richness was greatest near neutral pH (around 7) and relatively cool temperatures (about 10 °C) (Fig. 2b ). We observed apparent upper bounds on richness with both temperature and pH that were best fit by twosided exponen tial (Laplace) curves. Thus, the present dataset suggests that maximum microbial richness occurs within a relatively narrow range of interme diate pH and temperature values.
These patterns, while robust in the context of the EMP dataset, necessarily reflect only the subset of sam ple types for which variables were measured (Supplementary Table  2 ); they should therefore be interpreted with caution. Understanding universal relationships between richness and environmental factors will require information from more studies with detailed and carefully collected physicochemical metadata. Beyond measured physical covariates, the breadth of environments in the EMP catalogue allows a detailed exploration of how microbial diversity is distributed across environments. Diversity among commu nities (betadiversity) is driven by turnover (replacement of taxa) and nestedness (gain or loss of taxa resulting in differences in richness)  38. If turnover dominates, then disparate communities will harbour unique taxa.
If nestedness dominates, then communities with fewer taxa will be subsets of communities with more taxa. We tested for nestedness using a 2,000sample subset with even representation across environ ments and studies. Given the contrasting environments and geographic separation among the many studies in the EMP, we expected different environments to contain unique sets of taxa and to show little nest edness. However, we found that communities across environments were significantly nested (Fig. 3a, b ; P < 0. 05) in comparison to null models (Fig. 3c ), accounting for the observed patterns of richness. At coarse taxonomic levels, an average of 84% of phyla, 73% of classes, and 58% of orders that occurred in less diverse samples also occurred in more diverse samples. Nestedness was observed even when the most prevalent taxa were removed and was robust across randomly chosen subsets of samples (Extended Data Fig.
4a ). Entropy distributions across all tag sequences at each taxonomic level show that this pattern is general (Fig. 4b ). Seeking a more precise measure of the divergence at which a taxon is specific for environments, we next investigated how entropy changes as a function of phylogenetic distance. We calculated entropy for each node of the phylogeny and visualized it as a function of maximum tiptotip branch length (Fig. 4c ). While entropy decreased gradually at finer phyloge netic resolution, it dropped sharply at the tips of the tree. We conclude that environment specificity is best captured by individual 16S rRNA sequences, below the typical threshold defining microbial species (97% identity of the 16S rRNA gene). The EMP dataset provides the ability to track individual sequences across the Earth's microbial communities.
Using a representative subset of the EMP (Extended Data Fig. 7a ), we produced a table of sequence counts and distributions, including among environments (EMPO) and along environmental gradients (pH, temperature, salinity, and oxygen). From this we generated 'EMP Trading Cards' , which promote explora tion of the dataset and here highlight the distribution patterns of three prevalent or environmentcorrelated tag sequences (Extended Data Fig. 7b , Supplementary Table  3 ). The entire EMP catalogue can be que ried using the Redbiom software, with commandline (https://github. com/biocore/redbiom) and webbased (http://qiita. microbio. me) interfaces to find samples based on sequences, taxa, or sample meta data, and to export selected sample data and metadata (instructions at https://github. com/biocore/emp).
User data generated from the EMP protocols can be readily incorporated into this framework: because Deblur operates independently on each sample  21  , additional tag sequences can be added to this dataset from new studies without repro cessing existing samples. Future combinations of datasets targeting the same genomic region but sequenced using different methods may be admissible but would require considerations to account for methodo logical biases. The growing EMP catalogue is expected to have applications for research and industry, with tag sequences used as environmental indicators and representing targets for cultivation, genome sequencing, and laboratory study. In addition, these tools and approaches, although developed for bacteria and archaea, could be expanded to all domains of life  43.
Conclusions and future directions Here we have used crowdsourced sample collection and standardized microbiome sequencing and metadata curation to perform a global metaanalysis of bacterial and archaeal communities. Using exact sequences in place of OTUs and a learned structure of microbial envi ronments, we have shown that agglomerative sampling can reveal basic biogeographic patterns of microbial ecology, with resolution and scope rivaling data compilations currently available for 'macrobial' ecology  44, 45. Our results point to key organizing principles of micro bial communities, with lessrich communities nested within richer communities at higher taxonomic levels, and environment specificity becoming much more evident at the level of individual 16S rRNA sequences. The EMP framework and global synthesis presented here represent value added to the scientific community beyond the substantial contri butions of the constituent studies (Supplementary Table  1 ).
At a time when both academic and governmental agencies increasingly recognize the value of communal biodiversity monitoring efforts  47, 48  , the EMP provides one example of a logistically feasible standardization framework to maximize inter operability across diverse and independent studies, in particular using stable identifiers (exact sequences) to enable enduring utility of environmental sequence data. Given current global sequencing efforts, the use of coordinated protocols and submission to this and other public databases should allow rapid accumulation of new data, providing an ever more diverse reference catalogue of microbes and microbiomes on Earth. Online Content Methods, along with any additional Extended Data display items and Source Data, are available in the online version of the paper; references unique to these sections appear only in the online paper.
For ease of application, we gathered these contributions into an application ontology, the EMP Ontology (EMPO) (Fig. 1a ). The EMP community will continue to work with ontology engineers to shape ENVO and other ontologies around the EMPO application ontology. EMPO will be maintained as a logical subset of ENVO and integrated into the ENVO release cycle to maximize interoperation. Metadata curation was automated using Pandas (http://pandas. pydata. org). The size of the dataset also required extensive software development to support analysis at this scale, leading to tools including the data and analysis portal Qiita (http:// qiita. microbio. me), the BIOM format  52  , new 'OTU picking' methods Deblur  21  and a subsampled openreference procedure  53  , a scalability improvement of Fast UniFrac phylogenetic inference software  54  , speed improvements to sequence insertion tree method SEPP  55  , and speed and feature improvements to Emperor ordination visualization software  56  (http://biocore.
github. io/emperor). Sample collection. The global community of microbial ecologists was invited to submit samples for microbiome analysis, and samples were accepted for DNA extraction and sequencing provided that scientific justification and highquality sample metadata were provided before sample submission. Standardized sampling procedures for each sample type were used by contributing investigators. Samples were collected fresh and, where possible, immediately frozen in liquid nitrogen and stored at -80 °C. Detailed sampling protocols are described in publications of the individual studies (Supplementary Table  1 ). Bulk samples (for example, soil, sediment, faeces) and fractionated bulk samples (for example, sponge coral surface tissue, centrifuged turbid water) were taken using microcentrifuge tubes.
Swabs (BD SWUBE dual cotton swabs or similar) were used for biofilm or surface samples. Filters (Sterivex cartridges, 0. 2 μ m, Millipore) were used for water samples. Samples were sent to laboratories in the United States for DNA extraction and sequencing: water samples to Argonne National Laboratory, soil samples to Lawrence Berkeley National Laboratory (pre2014) or Pacific Northwest National Laboratory (2014 onward), and faecal and other samples to the University of Colorado Boulder (pre2015) or the University of California San Diego (2015 onward). Metadata curation and EMP ontology. Metadata were collected in compliance with MIMARKS  10  , EBI (https://www. ebi. ac. uk/ena), and Qiita (http://qiita. microbio. me) standards, as described in the EMP Metadata Guide (http://www. earthmicrobiome.
org/ protocolsandstandards/metadataguide). QIIME mapping files (metadata) were downloaded from Qiita, merged, and refined using Python with Pandas, generating qualitycontrolled mapping files. Mapping file columns are described in Supplementary Table  2. Mapping files for the full EMP dataset and subsets (see below) are available at ftp://ftp. microbio. me/emp/release1/ mapping_files/. The EMP Ontology (EMPO) for microbial environments was devised to facilitate the present analysis while preserving interoperability. Coordinated by the ENVO team, annotations from ENVO  11, 51  , UBERON (metazoan anatomy)  57  , PO (plant ontology)  58  , FAO (fungal anatomy ontology, http://purl. obolibrary. org/obo/fao. owl), and OMP (ontology of microbial phenotypes)  59  were mapped to our EMPO levels 2 and 3 (empo_2 and empo_3).
Additionally, the free living or hostassoci ated lifestyles were captured in level 1 categories (empo_1). Descriptions of empo_3 categories are provided at http://www. earthmicrobiome. org/ protocolsand standards/empo. The W3C Web Ontology Language (OWL) document is available at http://purl. obolibrary. org/obo/envo/subsets/envoEmpo. owl. Map data were derived from the opensource project MatPlotLib package Basemap, which dis tributes map data from Generic Mapping Tools data (http://gmt. soest. hawaii. edu) released under the GNU Lesser General Public License v3. DNA extraction, amplicon PCR, sequencing, and sequence pre-processing. DNA extraction and 16S rRNA amplicon sequencing was done using EMP standard protocols (http://www. earthmicrobiome. org/protocolsand standards/16s)  14. In brief, DNA was extracted using the MO BIO PowerSoil DNA extraction kit (Carlsbad, CA), chosen because of its versatility with diverse sample types (rather than high yields with any given sample type).
Sequence data were demultiplexed and minimally quality filtered using the QIIME 1. 9. 1 script split_libraries_fastq. py  60 ith Phred quality threshold of 3 and default parameters to generate perstudy FASTA sequence files. Tag sequence and OTU picking and subsets. Sequence data were errorfiltered and trimmed to the length of the shortest sequencing run (90 bp) using the Deblur software  21  ; the resulting 90bp Deblur BIOM table was used for all analyses unless otherwise noted. Deblur tables trimmed to 100 bp and 150 bp were also generated and provided, which contain greater sequence resolution but fewer samples. Deblur observation tables were filtered to keep only tag sequences with at least 25 reads total over all samples. For comparison to existing OTU tables, traditional closedreference OTU picking was done against 16S rRNA databases Greengenes 13.
8  25  and SILVA 123  26  using SortMeRNA  61  , and subsampled openreference OTU picking  53  was done against Greengenes 13. 8. These unfiltered tables and the filtered and subset tables described below are available at ftp://ftp. microbio. me/ emp/release1/otu_tables. A total of 97 studies and 27,742 samples are included in the present study and in the unfiltered BIOM tables. The QCfiltered subset used in core diversity analyses (Fig. 2 ) contains 96 studies and 23,828 samples, and it was subset further for some analyses. In the provided BIOM tables (ftp://ftp. microbio. me/emp/release1/otu_ tables/ and https://zenodo. org/record/890000), the 'release1' set contains all sam ples in the 97 studies that have at least one sequence per sample; this set includes controls (blanks and mock communities).
006 per position  62. Comparison against reference databases. To compare the unique sequence diversity in this study to that in existing databases, sequences from the complete Deblur 90bp observation table were compared to the set of unique fulllength sequences from Greengenes 13. 8 and the noneukaryotic fraction of Silva 128 data bases using the opensource sequence search tool VSEARCH  63  in global alignment search mode, requiring 100% similarity across the query sequence and allowing multiple 100% reference matches. Prevalence as a function of sequencing depth. The QCfiltered Deblur 90bp observation table was additionally filtered to samples that had at least 50,000 sequences (observations). We chose to focus on four environment types (EMPO level 3) where there were many hundreds of samples with more than 50,000 sequences: soil (n = 2,279), saltwater (n = 478), freshwater (n = 1,508), and animal distal gut (n = 695) environments.
tree), dividing it into subsets of size 5,000. The branch lengths on the Greengenes tree were recomputed using RAxML 66 under the GTRCAT model before the placement. The pipeline used, including the reference trees and alignments can be found at ftp://ftp. microbio. me/emp/release1/otu_info/greengenes_sepp_pipeline, and the bash script is available at https://github. com/biocore/emp/blob/master/code/03 otupickingtrees/deblur/run_sepp. sh. Fast UniFrac. Unweighted and weighted UniFrac were computed using the Cythonized 67 implementation of Fast UniFrac 54 in scikitbio  68. Fast UniFrac by itself was not scalable for the EMP dataset owing to an intermediary data structure required by the algorithm, which scales in space by O(NM), where N is the number of nodes in the phylogeny and M is the number of samples.
A workaround was designed and implemented in scikitbio (skbio. diversity. block_beta_diversity) which computes partial distance matrices as opposed to all samples pairwise, enabling large reductions within the intermediary data structure by shrinking M and, in tandem, shrinking N to only the relevant nodes of the phylogeny. This decomposition also allows a classic mapreduce parallel approach with low perprocess space requirements. Further space and time reductions were obtained through the implementation and use of a balancedparentheses tree representation  69  (https://pypi. python. org/pypi/iow). Core diversity analyses: alpha-and beta-diversity. Alphararefaction was computed using single_rarefaction. py in QIIME 1. 9. 1 60 using as input the Deblur 90bp BIOM table and rarefaction depths of 1,000, 5,000, 10,000, 30,000, and 100,000.
Covariation of richness with latitude, pH, and temperature. Measurements of alphadiversity were compared to absolute latitude using a linear mixedeffects model incorporating study ID as a random variable and the interaction of envi ronment and latitude as fixed effects; this was performed on a dataset filtered to include only studies comprising samples that spanned at least 10° of absolute latitude. Correlation of richness with pH and temperature were fitted with a Laplace distribution. The Laplace distribution is a continuous probability distribution that simultaneously captures exponential increase and exponential decrease around a modal value (μ). This distribution is also referred to as the double exponential or twosided exponential because it represents two symmetrical exponential distributions backtoback.
0. 1 (http://github. com/biota/sourcetracker2) to define the degree to which tag sequences are shared among environmental samples, using the 2,000sample subset of the Deblur 90bp observation table (~ 20% of each sample type) as source samples to train the model, and the remainder as sink samples to test the model. Additionally, we used leaveoneout crossvalidation to predict the sample type of each source sample when that sample type is excluded from the model, in order to evaluate the homogeneity of source samples and independence of each source type. Source and sink samples were rarefied to 1,000 sequences per sample before feature selection and testing. Nestedness of taxonomic composition. Nestedness captures the degree to which elements of a large set are contained within progressively smaller sets.