dataesr/scientific-paragraphs-categorization huggingface dataset:
However we expect this issue to be minor as the total number of species identified stabilized at 100,000 reads (Supp. Figure  S9B ) compared to an average of 6. 01M reads per sample. Samples collected from North America and Europe were distinct from those collected in East Asia, but the separation between other regions was less clear. A similar trend was found in an analogous analysis based on functional pathways rather than taxonomy (Supp. We quantified the degree to which metadata covariates influence the taxonomic composition of our samples using MAVRIC, a statistical tool to estimate the sources of variation in a count-based dataset  (Moskowitz and Greenleaf, 2018). We identified covariates which influenced the taxonomic composition of our samples: city, population density, average temperature in June, region, elevation above sea-level, surface type, surface material, elevation above or below ground and proximity to the coast.
(2018) , albeit a very minor one. To reduce bias that could be introduced by samples  taken from precisely the same object we excluded all pairs of samples within 1km of one another. At a global level, we examined the prevalence and abundance of taxa and their functional profiles between cities and continents. These data showed a fairly stable phyla distribution across samples, but the relative abundance of these taxa is unstable (Figure  2C ) with some continental trends. In contrast to taxonomic variation, functional pathways were much more stable across continents, showing relatively little variation in the abundance of high level categories (Figure  2D ). This pattern may also be due to the more limited range of pathway classes and their essential role in cellular function, in contrast to the much more wide-ranging taxonomic distributions examined across metagenomes.
Microbial Signatures Reveal Urban Characteristics To facilitate characterization of novel sequences we created GeoDNA, a high-level web interface (Figure  3A ) to search raw sequences against our dataset. Users can submit sequences to be processed against a k-mer graph-based representation of our data. Query sequences are mapped to samples and a set of likely sample hits is returned to the user. This interface will allow researchers to probe the diversity in this dataset and rapidly identify the range of various genetic sequences. We sought to determine whether a samples taxonomy reflected the environment in which it was collected. To this end we trained a Random Forest Classifier (RFC) to predict a sample's city of origin from its taxonomic profile. We trained an RFC with 100 components on 90% of the samples in our dataset and evaluated its classification accuracy on the remaining 10%.
We repeated this procedure with multiple subsamples of our data at various sizes and with 5 replicates per size to achieve a distribution (Fig. 3B ). The RFC achieved 88% on held out data which compares favorably to the 7. 01% that would be achieved by a randomized classifier. These results from our RFC demonstrate that city specific taxonomic signatures exist and can be predictive. We expanded our analysis of environmental signatures in taxonomy to the prediction of features in cities not present in our training set. To do this we collated a set of 7 features for each city: population, surface material, elevation, proximity to the coast, population density, region, ave June temperature, and Koppen climate classification. We trained a RFCs to predict each feature based on all samples that were not taken from a given city then used the relevant RFC to predict the feature for samples from the held out city and recorded the classification accuracy (Figure  3D ).
We performed a rarefaction analysis on the set of all resistance genes in the dataset, which we call the "panresistome" (Figure (Supp. Figure  S6B ). Similar to the rate of detected species, the panresistome also shows an open slope with an expected rate of discovery of 1 previously unobserved AMR gene per 10 samples. Given that AMR gene databases are rapidly expanding and that no AMR genes were found in some samples, it is likely that future analyses will identify many more resistance genes in this data. Additionally, AMR genes show a "neighbourhood" effect within samples that are geographically proximal analogous to the effect seen for taxonomic composition (Supp. Figure  S6C ). Excluding samples where no AMR genes were detected, the Jaccard distance between sets of AMR genes increases with distance for pairs of samples in the same city.
, 2019) at the species level for a total of 10,928 novel viruses. We note that this number is surprisingly high but was obtained using a conservative pipeline (99. 6% precision) and corresponded well with our identified CRISPR arrays (below). This suggests that urban microbiomes contain significant diversity not observed in other environments. Next, we attempted to identify possible bacterial and eukaryotic hosts for our predicted viral MAGs. For the 686 species with similar sequences in IMG/VR, we projected known host information onto 2,064 MetaSUB viral MAGs. Additionally, we used CRISPR-Cas spacer matches in the IMG/M system to assign possible hosts to a further 1,915 predicted viral species. Finally, we used a database of 20 million metagenome-derived CRISPR spacers to provide further rough taxonomic assignments.
Our predicted viral hosts aligned with our taxonomic profiles, 41% of species in the core microbiome (Section 2) had predicted viral-host interactions. Many of our viral MAGs were found in multiple locations (Figure  5D ). Many viruses were found in South America, North America and Africa. Viral MAGs in Japan often corresponded to those in Europe and North America. We identified 838,532 CRISPR arrays in our data of which 3,245 could be annotated for specific systems. The annotated CRISPR arrays were principally type 1-E and 1-F btu a number of type two and three systems were identified as well (Figure  5E, F ). A number of arrays had unclear or ambiguous type assignment. Critically the spacers in our identified CRISPR arrays closely matched our predicted viral MAGs. We aligned spacers to both our viral MAGs and all viral sequences in RefSeq.
Moreover, in order to study the transmission of AMRs on a global scales this dataset represents only focuses on some of the sources and vectors of the built environment. Indeed, datasets from rural and suburban areas with livestock and farms, sewage from cities  (Fresia et al. , 2019; Joseph et al. , 2019) , and other notable sources of AMRs need to be integrated together to truly capture AMR mechanisms at the global scale  (Singer et al. , 2016; Thanner et al. , 2016). Previous studies have already demonstrated a role for precision clinical metagenomics in managing infectious disease and global health  (Afshinnekoo et al. , 2017; Gardy and Loman, 2018; Ladner et al. , 2019). As demonstrated by the coronavirus disease 2019 (COVID-19) pandemic, as an atlas this data has the potential to aid physicians, public health officers, government officials, and others in tracing, diagnosis, clinical decision making, and policy within their communities.
Open Science The MetaSUB dataset is built and organized for full accessibility to other researchers. This is consistent with the concept of Open Science. Specifically, we built our study with the FAIR principles in mind: Findable, Accessible, Interoperable and Reusable. To make our study reproducible, we released an open source version-controlled pipeline called the MetaSUB Core Analysis Pipeline (CAP). The CAP is intended to improve the reproducibility of our findings by making it easy to apply a number of analyses consistently to a large dataset. This pipeline includes all steps from extracting data from raw sequence data to producing refined results like taxonomic and functional profiles. The CAP itself is principally composed of other open peer-reviewed scientific tools, with only a few custom scripts for mundane tasks.
Every tool in the CAP is open source with a permissive license. The CAP is available as a docker container for easier installation in some instances and all databases used in the CAP are available for public download. The CAP is versioned and includes all necessary databases allowing researchers to replicate results. The CAP is not designed to produce highly novel results but is meant to be a good practice agglomeration of open source tools. However, the output of the CAP still consists of a number of different output formats with multiple files for each sample. To make our results more reproducible and accessible, we have developed a program to condense the outputs of the Core Analysis Pipeline into a condensed data-packet. This data packet contains results as a series of Tidy-style data tables with descriptions.
The advantage of this set-up is that result tables for an entire dataset can be parsed with a single command in most high level analysis languages like Python and R. This package also contains Python utilities for parsing and analyzing data packets which streamlines most of the boilerplate tasks of data analysis. All development of the CAP and data packet builder (Capalyzer) package is open source and permissively licensed. In addition to general purpose data analysis tools essentially all analysis in this paper is available as a series of Jupyter notebooks. Our hope is that these notebooks allow researchers to reproduce our results, build upon our results in different contexts, and better understand precisely how we arrived at our conclusions. By providing the exact source used to generate our analyses and figures, we also hope to be able to quickly incorporate new data or correct any mistakes that might be identified.
For less technical purposes, we also provide web-based interactive visualizations of our dataset (typically broken into city-specific groups). These visualizations are intended to provide a quick reference for major results as well as an exploratory platform for generating novel hypotheses and serendipitous discovery. The web platform used, MetaGenScope, is open source, permissively licensed, and can be run on a moderately powerful machine (though its output relies on results from the MetaSUB CAP). Our hope is that by making our dataset open and easily accessible to other researchers the scientific community can more rapidly generate and test hypotheses. One of the core goals of the MetaSUB consortium is to build a dataset that benefits public health. As the project develops we want to make our data easy to use and access for clinicians and public health officials who may not have computational or microbiological expertise.
Data Access Raw sequencing reads from this study contain significant amounts of human DNA and cannot yet be made public. However, reads with the majority of human DNA filtered and low quality bases removed are available for download from Wasabi (an Amazon S3 clone) with individual URLs located here: https: //github. com/MetaSUB/metasub_utils. In addition to raw reads higher level results (e. g. taxonomic profiles, functional pathways, etc. ) are available in the MetaSUB data packet also available for download from Wasabi. For instructional purposes we also provide a simplified data packet for teaching which includes balanced numbers of samples from each city and completely filled metadata tables. Interactive We thank GitHub for providing private repositories to the MetaSUB consortium at no cost.
Metadata Collection and Cleaning Metadata from individual cities was collected from a standardized form and set of fields. The principle fields collected were the location of sampling, the material being sampled, the type of object being sampled, the elevation above or below ground, and the station or line where the sample was collected. However, several cities were unable to use the provided apps for various reasons and submitted their metadata as separate spreadsheets. Additionally, certain metadata features, such as those related to sequencing and quality control, were added after initial sample collection. To collate various metadata sources, we built a publicly available program which assembled a large master spreadsheet with consistent sample UUIDs. After assembling the originally collected data attributes we added normalized attributes based on the original metadata to account for surface material, control status, and features of individual cities.
Moreover, a significant amount of metadata was recorded throughout sample collections to ensure as much information regarding the samples was captured. All cities also developed sampling plans for their collections and submitted them for review to have swabs sent to them, this was to ensure consistency across the various sites. All principal investigators and MetaSUB city leaders were trained in the sampling instructions and this training was further disseminated to the respective sampling teams to ensure consistent and quality control sampling. Swabbers were instructed to put on gloves before each sample collection. The swab was dipped in the preservative medium to be pre-moistened before collection and sampling was timed to 3 minutes to ensure highest yield. Other key points in training included ensure highest surface area was used for collection (i.
Sequence Preprocessing Sequence data were processed with AdapterRemoval (v2. 17, Schubert et al. (2016) ) to remove low quality reads and reads with ambiguous bases. Subsequently reads were aligned to the human genome (hg38, including alternate contigs) using Bowtie2 (v2. 3. 0, fast preset, Langmead and Steven L Salzberg (  2013 )). Read pairs where both ends mapped to the human genome were separated from read pairs where neither mate mapped. Read pairs where only one mate mapped were discarded. Hereafter, we refer to the read sets as human reads and non-human reads. 6. 5. 9 Unmapped DNA is not similar to any known sequence A large proportion of the reads in our samples were not mapped to any references sequences. There are three major reasons why a fragment of DNA would not be classified in our analysis 1) The DNA originated from a non-human and non-microbial species which would not be present in the databases we used for classification 2) Our classifier (KrakenUniq) failed to classify a DNA fragment that was in the database due to slight mismatch 3) The DNA fragment is novel and not represented in any existing database.
Explanations (1) and (  2 ) are essentially drawbacks of the database and computational model used, and we can quantify them by mapping reads using a more sensitive aligner to a larger database, such as BLASTn  (Altschul et al. , 1990) , or ensemble methods for analysis  (McIntyre et al. , 2017). To estimate the proportion of reads which could be assigned, we took 10k read subsets from each sample and mapped these to a set of large database using BLASTn (see 2 for details). This resulted in 34. 6% reads which could not be mapped to any external database compared to 41. 3% of reads mapped using our approach with KrakenUniq. We note that our approach to estimate the fraction of reads that could be classified using BLASTn does not account for hits to low quality taxa which would ultimately be discarded in our pipeline, and so represents a worst-case comparison.
Taxonomic Analysis We generated taxonomic profiles by processing non-human reads with  KrakenUniq (v0. 3. 2 Breitwieser et al. (2018) ) using a database based on all draft and reference genomes in NCBI/RefSeq Microbial (bacteria/archaea, fungi and virus) ca. March 2017. KrakenUniq was selected because its high performance, as it has been demonstrated to be comparable or having higher sensitivity than the best tools identified in a recent benchmarking study  (McIntyre et al. (2017) ) on the same comparative dataset. In addition, KrakenUniq allows for tunable specificity and identifies k-mers that are unique to particular taxa in a database. Reads are broken into k-mers and searched against this database. Finally, the taxonomic makeup of a sample is given by identifying the taxa with the greatest leaf to ancestor weight.
We used MASH's built-in Jaccard distance function to generate distances between our samples and HMP samples. We then took the distribution of distances to each particular human commensal community as a proxy for the similarity of our samples to a given human body site. We also compared our samples to HMP and soil samples using taxonomic profiles generated by MetaPhlAn v2. 0  (Segata et al. , 2012). We generated taxonomic profiles from non-human reads using MetaPhlAn v2. 0 and found the cosine similarity between all pairs of samples. We used the Microbe Directory  (Shaaban et al. , 2018)  to annotate taxonomic calls. The Microbe Directory is a hand curated, machine readable, database of functional annotations for 5,000 microbial species.
Analysis of Antimicrobial Resistance Genes We generated profiles of antimicrobial resistance genes using MegaRes (v1. 0. Known AMR genes can come from gene families with homologous regions of sequence. To reduce spurious mapping from gene homology we used BLASTn to align all MegaRes AMR genes against themselves. We considered any connected group of genes with an average nucleotide identity of 80% across 50% of the gene length as a set of potentially confounded genes. We collapsed all such groups into a single pseudo-gene with the mean abundance of all constituent genes. Before clustering genes we removed all genes which were annotated as requiring SNP verification to predict resistance. In addition to MegaRes we mapped non-human reads from all samples to the amino acid gene sequences in the Comprehensive Antibiotic Resistance Database  (McArthur et al.
Alpha Diversity Intra-sample (alpha) diversity was measured by using Species Richness and Shannon's Entropy. We took species richness as the total number of detected species in a sample after rarefaction to 1 million reads. Shannon's entropy is robust to sample read depth and accounts for the relative size of each group in diversity estimation. Shannon's entropy is typically defined as H = a i log 2 a i where a i is the relative abundance of taxon i in the sample. For alpha diversity based on k-mers or pathways, we simply substitute the relative abundance of a species for the relative abundance of the relevant type of object. was created for each reference genome via Mash (v2. 1. 1) with default parameters to find Mash distances and select candidate "best matches" from each reference database.
Then, dnadiff (v1. 3) was used to further quantify differences between each OTU and its best match from either database. ANI between OTUs and their matches was found as "M-to-M AvgIdentity" in the query report column (ANI 95% over 60% OTU sequence qualified as a match). OTU Taxonomic Assignment OTUs were placed into a bacterial or archaeal reference tree (based on the Genome Database Taxonomy, GTDB) and then assigned taxonomic classifications using GTDB-Tk (v1. 0. 2 Chaumeil et al. (  2019 )). GTDB-Tk relies on 120 bacterial and 122 archaeal marker genes; domain assignment is chosen based on domain-specific marker content of the OTU sequence. Using the GTDB-Tk placements, we built an OTU-only bacterial phylogeny with FastTree (v2. 1. 10 Price et al. (  ) ). The tree was visualized using iTOL (v5.
Viral Discovery We followed the protocol described by  Paez-Espino et al. (2017). Briefly, we used an expanded and curated set of viral protein families (VPFs) as bait in combination with recommended filtering steps to identify 16,584 UViGs directly from all MetaSUB metagenomic assemblies greater than 5kb. Then, the UViGs were clustered with the content of the IMG/VR system (a total of over 730k viral sequences including isolate viruses, prophages, and UViGs from all kind of habitats). The clustering step relied on a sequence-based classification framework (based on 95% sequence identity across 85% of the shortest sequence length) followed by the markov clustering (mcl). This approach yielded 2,009 viral clusters (ranging from 2-611 members) and 9,605 singletons (or viral clusters of 1 member), sequences that failed to cluster with any sequence from the dataset or the references from IMG/VR, resulting in a total of 11,614 vOTUs.
Identifying Host Virus Interactions We used two computational methods to reveal putative host-virus connections  (Paez-Espino et al. , 2016a). (1) For the 686 vOTUs that clustered with viral sequences from the IMG/VR system, we projected the known host information to all the members of the group (total of 2,064 MetaSUB UViGs). (2) We used bacterial/archaeal CRISPR-Cas spacer matches (from the IMG/M 1. 1 million isolate spacer database) to the UViGs (allowing only for 1 SNP over the whole spacer length) to assigned a host to 1,915 MetaSUB vOTUs. Additionally, we also used a database of over 20 million CRISPR-Cas spacers identified from metagenomic contigs from the IMG/M system with taxonomy assigned. Since some of these spacers may derive from short contigs these results should be interpreted with caution.
CRISPR Array Detection and Annotation Using CRISPRCasFinder the MetaSUB database was investigated to predict CRISPR arrays and annotate them with their corresponding predicted type based on CRISPR-Cas genes in their vicinity. CRISPRCasFinder was run with default parameters, "-so" and "-cas" options to identify cas genes. The precision and recall of the virus detection was 99. 6% and 37. 5% respectively, as previously reported by  (Paez-Espino et al. , 2016b). CRISPR-Cas types were assigned to arrays based on detected cas genes within a 10 kilobases vicinity. Cases where CRISPRCasFinder associated several cas genes of contradicting CRISPR-Cas types with the same CRISPR array were regarded as unclear annotation. This procedure yielded 838,532 predicted CRISPR arrays (with additional CRISPR arrays predicted with default parameters for PILER-CR), of which, 3,245 CRISPR arrays had unambiguous annotation, resulting in 43,656 unique spacers queried against genomic databases using BLASTN.
The contig sets of each sample were then transformed into annotation columns (one column per sample) by mapping them onto an implicit complete de Bruijn graph of order k. All annotation columns were then merged into a joint annotation matrix and transformed into Multi-BRWT format. Finally, the topology of the Multi-BRWT representation was optimized by relaxing its internal tree arity constraints to allow for a maximum arity of 40. OTU to Reference Genome Matching OTUs were compared against reference genomes from Ref-Seq (release 96 from November 2019, complete bacterial and archaeal genomes only, with "Exclude anomalous" and "Exclude derived from surveillance project" applied) as well as the full Integrated Gut Genomes (IGG) dataset (v1. 0Nayfach et al. (2019); 23,790 representative genomes).
Scale bars 10 nm. The average is shown in side view, whereas the slices represent perpendicular slices. The 5 pairs of pillars formed a narrow central channel and separated into 10 arches towards the base (base not visible at the used threshold). The division of the structure into subparts was adapted from  Durand et al (2015). Subvolumes were extracted from cryotomograms of ghost cells and FIB-milled intact Escherichia coli BL21 cells heterologously expressing TssJLM. A 0 Slice (0. 69 nm) through the non-symmetrized average. Note that the C5 symmetry was visible. ª To better characterize the flexibility of the base of the complex, we collected tomograms on the same frozen EM grids that were used to collect the SPA dataset (Appendix Fig  S4A-C ). The base of individual complexes appeared as very heterogeneous, with single arches often pointing in opposite directions, or on the contrary several arches clumping together  (Appendix Fig S4A).