dataesr/scientific-paragraphs-categorization huggingface dataset:
All sample types were stored at -80 °C if possible, otherwise -20 °C. To track the provenance of sample aliquots, we employed a QR coding scheme. Labels were affixed to aliquot tubes before shipping when possible. QR codes had the format 'name. 99. s003. a05', where 'name' is the PI name, '99' is the study ID, 's003' is the sample number and 'a05' is the aliquot number. QR codes (version 2, 25 pixels × 25 pixels) were printed on 1. 125' × 0. 75' rectangular and 0. 437' circular cap Cryogenic Direct Thermal labels (GA International, DFP-70) using a Zebra model GK420d printer and ZebraDesigner Pro 3 software for Windows. After receipt but before aliquots were stored in freezers, QR codes were scanned into a sample inventory spreadsheet using a QR scanner. Sample metadata. Environmental metadata were collected for all samples on the basis of the EMP Metadata Guide, which combines guidance from the Genomics Standards Consortium MIxS (Minimum Information about any Sequence) standard  74  and the Qiita Database (https://qiita.
LC-MS/MS data processing. The mass spectrometry data were centroided and converted from the proprietary format (. raw) to the m/z extensible markup language format (. mzML) using ProteoWizard (ver. 3. 0. 19, MSConvert tool)  75. The mzML files were then processed with MZmine 2 toolbox 76 using the ion-identity networking modules  77  that allow advanced detection for adduct/isotopologue annotations. The MZmine processing was performed on Ubuntu 18. 04 LTS 64-bits workstation (Intel Xeon E5-2637, 3. 5 GHz, 8 cores, 64 Gb of RAM) and took ~3 d. The MZmine project, the MZmine batch file (. XML format) and results files (. MGF and. CSV) are available in the MassIVE dataset MSV000083475. The MZmine batch file contains all the parameters used during the processing. In brief, feature detection and deconvolution was performed with the ADAP chromatogram builder  78  and local minimum search algorithm.
The isotopologues were regrouped and the features (peaks) were aligned across samples. The aligned peak list was gap filled and only peaks with an associated fragmentation spectrum and occurring in a minimum of three files were conserved. Peak shape correlation analysis grouped peaks originating from the same molecule and annotated adduct/isotopologue with ion-identity networking  77. Finally, the feature quantification table results (. CSV) and spectral information (. MGF) were exported with the GNPS module for feature-based molecular networking analysis on GNPS  79  and with SIRIUS export modules.
LC-MS/MS data annotation. The results files of MZmine (. MGF and. CSV files) were uploaded to GNPS (http://gnps. ucsd. edu)  52  and analysed with the feature-based molecular networking workflow  79. Spectral library matching was performed against public fragmentation spectra (MS 2 ) spectral libraries on GNPS and the NIST17 library. For the additional annotation of small peptides, we used the DEREPLICATOR tools available on GNPS  80, 81. We then used SIRIUS (v. 4. 4. 25, headless, Linux) to systematically annotate the MS 2 spectra. Molecular formulae were computed with the SIRIUS module by matching the experimental and predicted isotopic patterns  83  , and from fragmentation trees analysis 84 of MS 2. Molecular formula prediction was refined with the ZODIAC module using Gibbs sampling on the fragmentation spectra (chimeric spectra or those with poor fragmentation were excluded).
In silico structure annotation using structures from biodatabase was done with CSI:FingerID  86. Systematic class annotations were obtained with CANOPUS 41 and used the NPClassifier ontology  87. The parameters for SIRIUS tools were set as follows, for SIRIUS: molecular formula candidates retained, 80; molecular formula database, ALL; maximum precursor ion m/z computed, 750; profile, orbitrap; m/z maximum deviation, 10 ppm; ions annotated with MZmine were prioritized and other ions were considered (that is,.
[M+H3N+H]+, [M+H]+, [M+K]+, [M+Na]+, [M+H-H2O]+, [M+H-H4O2]+, [M+NH4]+); for ZODIAC: the features were split into 10 random subsets for lower computational burden and computed separately with the following parameters: threshold filter, 0. 9; minimum local connections, 0; for CSI:FingerID: m/z maximum deviation, 10 ppm; and biological database, BIO. To establish putative microbially related secondary metabolites, we collected annotations from spectral library matching and the DEREPLICATOR+ tools and queried them against the largest microbial metabolite reference databases (Natural Products Atlas 88 and MIBiG  89  ). Molecular networking  79  was then used to propagate the annotation of microbially related secondary metabolites throughout all molecular families (that is, the network component).
In parallel, we used the differential abundance results from 'songbird' described above to identify specific microbially related secondary metabolite pathways and superclasses that varied strongly across environments. We then went back to our metabolite feature-table to visualize differences in the relative abundances of those pathways and superclasses within each environment by first selecting features and calculating log-ratios using 'qurro'  94  , and then plotting using the 'ggplot2' package  95  in R 96 v4. 0. 0. We tested for significant differences in relative abundances across environments using Kruskal-Wallis tests implemented with the base 'stats' package in R  96.
GC-MS data processing and annotation. The data were converted from vendor's format to the. mzML format and processed using GNPS GC-MS data analysis workflow (https://gnps. ucsd. edu)  97. The compounds were identified by matching experimental spectra to the public libraries available at GNPS, as well as NIST 17 and Wiley libraries. The data are publicly available at the MassIVE depository (https://massive. ucsd. edu); dataset ID: MSV000083743. The GNPS deconvolution is available in GNPS (https://gnps. ucsd. edu/ProteoSAFe/status. jsp?t ask=d5c5135a59eb48779216615e8d5cb3ac), as is the library search (https://gnps. ucsd. edu/ProteoSAFe/status. jsp?task=59b20fc8381f4 ee6b79d35034de81d86).
GC-MS data analysis. For multi-omics analyses including GC-MS data, we first removed noisy (that is, suspected background contaminants and artifacts) features by excluding those with balance scores <50%. Balance scores describe compositional consistency of deconvoluted spectra across the dataset, where high values indicate reproducible spectral patterns and thus high-quality spectra. We then used QIIME 2's 'deicode'  91  plugin to estimate beta-diversity for each dataset using robust Aitchison distances. The final feature-table for GC-MS beta-diversity analysis included 460 samples and 216 features.
The PCR product was purified using the custom bead purification protocol above. Batches of 25 amplicon libraries were barcoded and sent for PacBio Sequel II library preparation and sequencing (Sequel II SMRT Cell 8M and 30 h collection time) at the DNA Sequencing Center at Brigham Young University. Circular consensus sequencing (CCS) reads were generated using CCS v. 3. 4. 1 (https:// github. com/PacificBiosciences/ccs) using default settings. UMI consensus sequences were generated using the longread_umi pipeline (https://github. com/SorenKarst/longread_umi) with the following command: longread_umi pacbio_pipeline -d ccs_reads. fq -o out_dir -m 3500 -M 6000 -s 60 -e 60 -f CAAGCAGAAGACGGCATACGAGAT -F AGRGTTYGATYMTGGCTCAG -r AATGATACGGCGACCACCGAGATC -R CGACATCGAGGTGCCAAAC -U '0. 75;1. 5;2;0' -c 2.
Amplicon data analysis. For multi-omics analyses including amplicon sequence data, we processed each dataset for comparison of beta-diversity. For all amplicon data except that for bacterial full-length rRNA amplicons, raw sequence data were converted from bcl to fastq, and then multiplexed files for each sequencing run uploaded as separate preparations to Qiita (study: 13114). For each 16S sequencing run, in Qiita, data were demultiplexed, trimmed to 150 bp and denoised using Deblur 122 to generate a feature-table of sub-operational taxonomic units (sOTUs) per sample, using default parameters. We then exported feature-tables and denoised sequences from each sequencing run, used QIIME 2's 'feature-table' plugin to merge feature-tables and denoised reads across sequencing runs, and placed all denoised reads into the GreenGenes 13_8 phylogeny 123 via fragment insertion using QIIME 2's  90  SATé-Enabled Phylogenetic Placement (SEPP) 124 plugin to produce a phylogeny for diversity analyses.
To allow for phylogenetically informed diversity analyses, reads not placed during SEPP (that is, 513 sOTUs, 0. 1% of all sOTUs) were removed from the merged feature-table. We then used QIIME 2's 'feature-table' plugin to exclude singleton sOTUs and rarefy the data to 5,000 reads per sample. Rarefaction depths for all amplicon analyses were chosen to best normalize sampling effort per sample while maintaining ≥75% of samples representative of Earth's environments, and also to maintain consistency with the analyses from EMP release 1. We then used QIIME 2's  90  'diversity' plugin to estimate alpha-diversity (that is, sOTU richness) and beta-diversity (that is, unweighted Uni-Frac distances). The final feature-table for 16S beta-diversity analysis included 681 samples and 93,260 features. We performed a comparative analysis of the data including and excluding the reads not placed during SEPP, and note that both alpha-diversity (that is, sOTU richness) and beta-diversity (that is, sample-sample RPCA distances) were highly correlated between datasets (Spearman r = 1.
0) (Supplementary Fig. 5 ). We thus proceeded with the SEPP-filtered dataset and used phylogenetically informed diversity metrics where applicable. For 18S data, we used QIIME 2's  90  'demux' plugin's 'emp-paired' method  125, 126  to first demultiplex each sequencing run, and then the 'cutadapt' plugin's 127 'trim-paired' method to trim sequencing primers from reads. We then exported trimmed reads, concatenated R1 and R2 read files per sample, and denoised reads using Deblur's 122,128 'workflow' with default settings, trimming reads to 90 bp, and taking the 'all. biom' and 'all. seqs' output, for each sequencing run. We then used Resource https://doi. org/10. 1038/s41564-022-01266-x QIIME 2's 'feature-table' plugin to merge feature-tables and denoised sequences across sequencing runs, and then the 'feature-classifier' plugin's 'classify-sklearn' method to classify taxonomy for each sOTU via pre-fitted machine-learning classifiers 129 and the SILVA 138 reference database  130.
We then used QIIME 2's 90 'feature-table' plugin to exclude reads assigned to bacteria and archaea, singleton sOTUs and samples with a total frequency of <5,500 reads, and the 'deicode'  91  plugin to estimate beta-diversity for each dataset using robust Aitchison distances  91. The final feature-table for 18S beta-diversity analysis included 461 samples and 14,839 features. For fungal ITS data, we used QIIME 2  90  to generate and merge feature-tables and denoised sequences across sequencing runs, as for 18S data but trimming reads to 150 bp. We then classified taxonomy for each sOTU as for 18S data, but using the UNITE 9 reference database  131. We then used QIIME 2's 'feature-table' plugin to exclude singleton sOTUs and samples with a total frequency of <500 reads, and the 'deicode'  91  plugin to estimate beta-diversity for each dataset using robust Aitchison distances  91.
The final feature-table for fungal ITS beta-diversity analysis included 500 samples and 10,966 features. For full-length rRNA operon data, per-sample fasta files were reformatted for importing to QIIME 2 as 'SampleData[Sequences]' (that is, with each header as '>{sample_identifier}_{sequence_identi-fier}'), concatenated into a single fasta file and imported. We then used QIIME 2's 'vsearch' plugin 132 to dereplicate sequences and then cluster them at 65% similarity (that is, due to rapid evolution at bacterial ITS regions). The 65% OTU feature-table had 365 samples and 285 features. The concatenated fasta file and 65% OTU feature-table were uploaded to Qiita as distinct preparations (study: 13114). We then used QIIME 2's  90  'feature-table' plugin to exclude singleton OTUs and samples with a total frequency of <500 reads, and the 'deicode'  91  plugin to estimate beta-diversity for each dataset using robust Aitchison distances  91.
The final feature-table for full-length rRNA operon beta-diversity analysis included 242 samples and 196 features. Shotgun metagenomic sequencing. One round of DNA extraction was performed as above for shotgun metagenomic sequencing. Sequencing libraries were prepared using a high-throughput version of the HyperPlus library chemistry (Kapa Biosystems) miniaturized to approximately 1:10 reagent volume and optimized for nanolitre-scale liquid-handling robotics  133. An exhaustive step-by-step protocol and accompanying software can be found in ref. 133. Briefly, DNA from each sample was transferred to a 384-well plate and quantified using the Quant-iT PicoGreen dsDNA Assay kit (P7589, Thermo Fisher), and then normalized to 5 ng in 3. 5 µl of molecular-grade water using an Echo 550 acoustic liquid-handling robot (Labcyte).
Trimmed reads were then mapped to the Web of Life database of microbial genomes (release 1) 136 using bowtie2 v2. 3. 2 137 in very-sensitive mode to produce alignments that were used for taxonomic and exploratory functional analysis of microbial communities. Bowtie2 settings included maximum and minimum mismatch penalties (mp= [1,1]), a penalty for ambiguities (np=1; default), read and reference gap open-and extend penalties (rdg=[0,1], rfg=[0,1]), a minimum alignment score for an alignment to be considered valid (score-min=[L,0,-0. 05]), a defined number of distinct valid alignments (k=16), and the suppression of SAM records for unaligned reads, as well as SAM headers (no-unal, no-hd). The Web of Life database is particularly attractive as it includes a phylogeny that can be used for diversity analyses, and was curated to represent phylogenetic breadth of Bacteria and Archaea 136 , ideal for analyses across diverse environments.
We compared mapping to the Web of Life to Rep200, a curated database of NCBI representative and reference microbial genomes (that is, corresponding to RefSeq release 200, released 14 May 2020) and found little difference across environments (Supplementary Fig. 6 ). We therefore chose the Web of Life as it allows for phylogenetically informed analyses. For taxonomic analysis, we generated a feature-table of counts of operational genomic units (OGUs) for each sample using a reference-based approach. We chose this method over the de novo or reference-free approach, as the latter uses assembly/clustering to deconvolute short reads into larger sequence units; the reference-free approach allows for the direct observation of the actual organisms in the community, but alone does not allow their meaningful characterization  6.
For diversity analyses, to best normalize sampling effort per sample while maintaining ≥75% of samples representative of Earth's environments, we rarefied the OGU feature-table to 6,550 reads per sample. The final feature-table for analyses of shotgun metagenomic taxonomic diversity included 612 samples and 8,692 OGUs. For alpha-diversity, we quantified three metrics, in part to see which had the strongest correlations with microbially related metabolite richness. We used the R package 'geiger' 141 to quantify weighted Resource https://doi. org/10. 1038/s41564-022-01266-x Faith's PD for each sample following the method of Swensen 142. We used QIIME 2's 'diversity' plugin  90  to quantify richness and Faith's PD (that is, unweighted), as well as beta-diversity (that is, using weighted UniFrac distance) between each pair of samples.
With that table, we used QIIME 2's 'deicode'  91  plugin to estimate beta-diversity for each dataset using robust Aitchison distances  91  and EMPeror 93 to visualize differences in microbial community composition among samples. We then performed PERMANOVA as above to test for significant differences in microbial functional composition across the various levels of EMPO. Nestedness analysis of metabolites and microbial taxa. As our analysis of turnover (replacement) of microbial taxa suggested a degree of nestedness (gain or loss of taxa promoting differences in richness) among environments in line with previous observations based on EMP 16S release 1, we tested for nestedness in our shotgun metagenomics data for microbial taxa. We used the NODF statistic 150 to quantify nestedness on the basis of the degree to which less diverse communities are subsets of more diverse communities, which we quantified at each major taxonomic level from phylum to species.
The final feature-table included 598 samples (representing 15 environments) and 8,587 microbial taxa (that is, Woltka OGUs). For the shotgun metagenomic feature-table for functional analysis, we used QIIME 2's 90 'feature-table' plugin to exclude samples in three EMPO environments (that is, 'Animal corpus (non-saline)', 'Surface (saline)' and 'Subsurface (non-saline)'), exclude singleton features and normalize the total count per sample to 10,000 sequences. The final feature-table included 706 samples (representing 16 environments) and 1,133 enzymes (that is, KEGG ECs). For each feature-table, we trained an auto-AI classifier 152 with SHAP explanations 153 and the hyper-tuned XGBoost method 154 for predicting environments (on the basis of EMPO 4). Each dataset was split into a training set (80%) and a testing set (20%), with similar environmental distributions in each iteration for the classification of samples.
Metabolite-microbe co-occurrence analysis. To begin to explore co-occurrences between microbes and metabolites across environments, we implemented an approach that generates co-occurrence probabilities between all metabolite and microbial features, clusters Resource https://doi. org/10. 1038/s41564-022-01266-x metabolites on the basis of their co-occurrence with the microbial community and highlights individual microbial features driving global patterns in metabolite distribution in this space. For co-occurrence analyses of LC-MS/MS metabolites and genomes profiled from shotgun metagenomic data, feature-tables were further filtered to retain only the 434 samples found in both datasets. For the LC-MS/ MS feature-table of microbially related secondary metabolites, we excluded 172 samples lacking shotgun metagenomics data, resulting in a final set of 6,501 microbially related metabolites.
Finally, we visualized differences in the log-ratio of the focal group to the reference group between the focal environment and all other environments, separately for metabolites and microbes  154. Mantel correlations between datasets. To explore the relationships between sample-sample distances for any two datasets (for example, LC-MS/MS vs shotgun metagenomics for taxonomy), we used QIIME 2's 'diversity' plugin  90  to perform Mantel tests on all pairings of the datasets using Spearman correlations. Input distance matrices are those described above for each dataset.
For samples from Singapore, permit (No:NP/RP18-086) was granted by National Parks Board (NParks) and sampling was conducting according to stipulations of the permit. All coral samples were collected by AAUS-certified scientific divers, in accordance with local regulations. Relevant permit numbers are: CITES (PWS2014-AU-002155, 12US784243/9), Great Barrier Reef Marine Park Authority (G12/35236. 1, G14/36788. 1), Lord Howe Island Marine Park (LHIMP/R/2015/005), New South Wales Department of Primary Industries (P15/0072-1. 0, OUT 15/11450), US Fish and Wildlife Resource https://doi. org/10. 1038/s41564-022-01266-x Extended Data Fig. 1  | Diagrammatic overview of multi-omics analyses performed using the EMP500 dataset. The process begins with data generation for both the microbiome and metabolome, which is then followed by analysis of differential abundance of both microbial taxa and microbiallyrelated metabolites across environments.
Acknowledgements We thank  G. Milivenvsky ,  A. Møller ,  I. Chizhevsky ,  S. Kirieiev ,  A. Nosovsky  and  M. Ivanenko  for logistic support with fieldwork in Ukraine;  L. Goldasich  and  J. Toronczak  for assistance with sample processing for sequencing;  M. Fedarko ,  R. Diner ,  E. Wood-Charlson ,  S. Nayfach ,  D. Udwary  and  E. Eloe-Fadrosh  for reviewing the manuscript. This work was supported in part by the  Samuel Freeman Charitable Trust ,  US National Institute of Health (NIH)  (awards  1RF1-AG058942-01 ,  1DP1AT010885 ,  R01HL140976 ,  R01DK102932 ,  R01HL134887 ,  U19AG063744  and  U01AI124316  to R. K. ),  US Department of Agriculture -National Institute of Food and Agriculture (USDA-NIFA)  (award  2019-67013-29137  to R. K. ), the  US National Science Foundation (NSF) -Center for Aerosol Impacts on Chemistry of the Environment, Crohn's & Colitis Foundation Award (CCFA)  (award  675191  to R.
analogue mode https://gnps. ucsd. edu/ProteoSAFe/status. jsp?task=fafdbfc058184c2b8c87968a7c56d7aa. The DEREPLICATOR jobs can be accessed here: https:// gnps. ucsd. edu/ProteoSAFe/status. jsp?task=ee40831bcc314bda928886964d853a52 and https://gnps. ucsd. edu/ProteoSAFe/status. jsp? task=1fafd4d4fe7e47dd9dd0b3d8bb0e6606. The SIRIUS results are available on the GitHub repository (https://github. com/biocore/emp/tree/master/data/ metabolomics/FBMN/SIRIUS).
Data availability The mass spectrometry method and data (. RAW and. mzML) were deposited on the MassIVE public repository and are available under the dataset accession number MSV000083475. The processing files were also added to the deposition (updates/2019-08-21_ lfnothias_7cc0af40/other/1908_EMPv2_INN/). GNPS molecular networking job is available at https://gnps. ucsd. edu/ProteoSAFe/ status. jsp?task=929ce9411f684cf8abd009670b293a33 and was also performed in analogue mode https://gnps. ucsd. edu/ProteoSAFe/ status. jsp?task=fafdbfc058184c2b8c87968a7c56d7aa. The DEREP-LICATOR jobs can be accessed at https://gnps. ucsd. edu/ProteoSAFe/ status. jsp?task=ee40831bcc314bda928886964d853a52 and https:// gnps. ucsd. edu/ProteoSAFe/status. jsp?task=1fafd4d4fe7e47dd9d d0b3d8bb0e6606. The SIRIUS results are available on the GitHub repository (emp/data/metabolomics/FBMN/SIRIUS).
The notebooks for metabolomics data preparation and microbially related molecules establishment are available at https://github. com/lfnothias/emp_ metabolomics. Amplicon and shotgun metagenomic sequence data were submitted to the European Nucleotide Archive under Project PRJEB42019 (https://www. ebi. ac. uk/ena/browser/view/PRJEB42019). Raw and demultiplexed amplicon and shotgun sequence data, the feature-table for full-length rRNA operon analysis, feature-tables for LC-MS/MS classical molecular networking and feature-based molecular networking, and the feature-table for GC-MS molecular networking data are available for download and analysis through Qiita at https://www. qiita. ucsd. edu (study: 13114). The GreenGenes database for 16S rRNA can be accessed at https://greengenes. secondgenome.
com. The SILVA 138 database for 16S and 18S rRNA can be accessed at https://www. arb-silva. de. The UNITE 9 database for fungal ITS sequences can be accessed at https://unite. ut. ee. The Web of Life database can be accessed at https://biocore. github. io/wol/. The Rep200 database can be accessed at https://www. ncbi. nlm. nih. gov/refseq/. The Natural Products Atlas database can be accessed at https://www. npatlas. org. The MIBiG database can be accessed at https://mibig. secondarymetabolites. org.