dataesr/scientific-paragraphs-categorization huggingface dataset:
The MiniPhy framework for phylogenetic compression Here, we describe the specific design choices of our implementation of phylogenetic compression for assemblies and de Bruijn graphs. More information and relevant links, including specific tools such as MiniPhy and Phylign and the resulting databases, can be found on the associated website (https://brinda. eu/mof). Clustering/batching. As genome collections encountered in practice can vary greatly in their properties as well as the available metadata, clustering is expected to be performed by the user. The recommended procedure is to identify species clusters using standard metagenomic approaches, such as those implemented in the Kraken software suite  55  (i. e. , Kraken 2  56  and Bracken 57 applied on the original read sets), as the obtained abundance profiles can also be used for quality control to filter out those samples that are likely contaminated.
If the total size of a collection is small enough, the clustering/batching step may be skipped entirely and the entire collection treated as a single batch. Inference of a compressive phylogeny. Users have the option to provide a custom tree generated by an accurate inference method such as RAxML 53. However, in most practical scenarios, such trees are not available, and MiniPhy then employs Attotree (https://github. com/karel-brinda/attotree), an efficient reimplementation of the Mashtree algorithm  34  , to generate a compressive phylogeny through sketching. Both Mashtree and Attotree first use Mash  58  to estimate the evolutionary distances between all pairs of genomes, which are then used to infer a compressive phylogeny employing the Neighbor-Joining algorithm  59, 60  as implemented in QuickTree  61.
The distance computation in Mash is based on estimating the Jaccard indexes of the corresponding k-mer sets and then estimating the likely mutation rate under a simple evolutionary model  62. Finally, MiniPhy post-processes the obtained tree using standard tree-transformation procedures implemented in the ETE3 library  63  , involving tree standardization, setting a midpoint outgroup, ladderization, and naming the internal nodes. MiniPhy (https://github. com/karel-brinda/miniphy). This is a central package for phylogenetic compression, including support for batching, and for calculating the associated statistics (see below). MiniPhy is implemented as a Snakemake 64 pipeline, offering three protocols for phylogenetic compression: 1) Compression of assemblies based on left-to-right reordering.
e. , the sizes of set and multiset unions across all batches. MiniPhy-COBS. MiniPhy-COBS (https://github. com/leoisl/miniphy-cobs) is a Snakemake 64 pipeline designed to create phylogenetically compressed ClaBS COBS indexes  37  (Classical Bit-sliced index) from assemblies already phylogenetically compressed by MiniPhy. ClaBS is a variant of COBS analogous to the original BIGSI data structure  17  , using Bloom filters of the same size; this property is important for ensuring that the order of Bloom filters is preserved and that the neighboring Bloom filters are mutually compressible (Supplementary Note 2). The workflow for each batch involves three main steps: 1) Renaming input assemblies to align their lexicographic and phylogenetic orders within each batch,.
Acquisition of the test collections GISP. The GISP collection was obtained from the https://github. com/c2-d2/rase-db-ngonorrhoeaegisp repository (version 04a132c) as published in ref  51. The assemblies (n=1,102) were obtained from the "isolates/contigs" subdirectory of Github repository (containing the original genomes including the plasmids), and the associated RAxML phylogenetic tree was downloaded from the "tree/" subdirectory of the same repository. The original data had originally been analyzed in ref  70    The command was run for individual prefixes ranging from 000 to 661, which resulted in n=661,405. fa. gz files.
4 ). The SC2 collection was provided in the left-to-right order according to Sarscov2phylo phylogeny. The genomes were progressively uniformly subsampled, stored as EOL-separated lists of sequences (without sequence headers), and then compressed using individual compressors, namely: 1) XZ: 'xz -9 -T1', 2) BZip2: 'bzip2 --best', 3) GZip: 'gzip -9', and 4) Re-Pair  83, 84  (https://github. com/rwanwork/Re-Pair, version as of 2021-10-.
7 TB. Comparison to Metagraph  38. The size of the phylogenetically compressed BIGSIdata collection was compared to the size of an analogous Metagraph index from the original paper  38  , based on the statistics in Table  1  and Supplementary Table  1  therein (the SRA-Microbe collection): n=446,506 indexed datasets, 39. 5 G canonical k-mers (with the same k-mer size k=31), and the size of the annotated de Bruijn graph being 291 GB (graph 30 GB + annotations 261 GB). This index was constructed from the same datasets as in the original BIGSI paper 17 but using a slightly different computational methodology. Consequently, the index of Metagraph contained approximately 4% fewer distinct canonical k-mers (k=31) compared to BIGSIdata as used in this paper. To compare the two compression approaches (MiniPhy with bottom-up k-mer propagation and XZ as a low-level tool vs.
The compact variant of the COBS index (default in COBS), based on adaptive adjustments of Bloom filter sizes through subindexes of different heights, was used as a baseline in our comparisons. For the 661k collection, we used the original index as provided (http://ftp. ebi. ac. uk/pub/databases/ENA2018-bacteria-661k/661k. cobs_compact, retrieved on 2022-09-08, 937 GB). For building a COBS index for 661k-HQ, we used the same construction protocol as in ref  16. Both indexes were then compressed on a highly performant server by XZ using 32 cores ('xz -9 -T32'). All of the obtained data points are provided in Supplementary Table  5.
Phylign pipeline for alignment against all pre-2019 bacteria from ENA Overview. The Phylign pipeline (https://github. com/karel-brinda/phylign) uses phylogenetically compressed assemblies (661k) and COBS indexes (661k-HQ) as described above to align queries against the entire 661k-HQ collection in a fashion similar to BLAST (Supplementary Note 4). The search procedure consists of two phases: matching the queries against the k-mer indexes using COBS  37  to identify the database's most similar genomes for each query, followed by an alignment of the queries to their best-matching genomes using Minimap 2  41. Phylign is developed as a Snakemake 64 pipeline, using Bioconda 69 for an automatic software management and the standard Snakemake resource management  64  to control the CPU cores assignments and limit RAM usage according to user-specified parameters.
Upon its first execution, Phylign downloads its phylogenetically compressed reference database from the Internet (102 GB), consisting of 29. 2 GB of assemblies and 72. 8 GB of COBS indexes. Matching. The matching step involves k-mer matching of all user queries against the entire 661k-HQ database using a modified version of COBS (v0. 3, see below), based on the principle that the number of k-mer matches between a genome and a query correlates with the alignment score  87. Each phylogenetically compressed COBS index is decompressed in memory and queried for the input user sequences, reporting all matches between the queries and genomes in the current batch with a sufficient (user-specified) proportion of matching k-mers. The computed matches are then aggregated across all batches, and for each query, only a (user-specified) number of best matches, plus ties, are retained and passed to the subsequent alignment step.
Evaluating Phylign Overview of the benchmarking procedure. The search using Phylign was evaluated on three datasets, representative of different query scenarios: a database of antibiotic resistance genes, a database of plasmids, and an Oxford nanopore sequencing experiment. In all cases, the search parametersincluding the number of hits of interest, the COBS k-mer threshold, and the Minimap preset -were tailored to each specific query type. The experiments were conducted on an iMac with a Quad-Core Intel CPU i7, 4. 2 GHz with 4 physical (8 logical) cores and 42. 9 GB (40 GiB) RAM. Time measurements. The wall clock and CPU time were measured using GNU time and calculated as real and usr+sys, respectively. The measurements were done for the matching and alignment steps separately. Memory measurements.
We have not found any reliable way of measuring peak memory consumption on macOS: both GNU time and the psutil Python library were significantly underestimating the memory footprint of our Snakemake pipeline. Therefore, we performed additional measurements on a Linux cluster using the SLURM job manager, using jobs allocated with a configuration similar to the parameters of our iMac computer. For 'max_ram_gb' set to 30 GB, we observed a peak memory consumption of 26. 2 GB, thus by 12. 7% lower compared to the specified maximum. Such a discrepancy is expected because the 'max_ram_gb' parameter defines an upper bound for the Snakemake resource management  64  , representing the worst-case scenario for parallel job combinations. Resistance genes -ARGannot. The resistance genes search was performed using the ARG-ANNOT database 42 comprising 1,856 genes/alleles, as distributed within the SRST2 software toolkit  89  (https://github.
com/katholt/srst2/blob/master/data/ARGannot_r3. fasta, retrieved on 2022-07-24). The search parameters were set to require a minimum of 50% matching k-mers, with 1,000 best hits plus ties taken for every gene/allele query. Alignment was performed with the Minimap preset for short reads ('sr'). Plasmids -the EBI plasmid database. The list of EBI plasmid was downloaded from the associated EBI website (https://www. ebi. ac. uk/genomes/plasmid. details. txt, retrieved on 2022-04-03), and individual plasmids were subsequently downloaded from the ENA using curl and GNU parallel  80. The search parameters were set to require at least 40% matching k-mers (the threshold previously used in ref  17  ), with 1,000 best hits plus ties taken for every plasmid. Alignment was performed with the Minimap preset for long, highly divergent sequences ('asm20').
Oxford Nanopore reads. The ERR9030361 experiment, comprising 159k nanopore reads from an isolate of M. tuberculosis, was downloaded from SRA NCBI. The search parameters were set to require at least 40% matching k-mers, with 10 best hits plus ties taken for every read. Alignment was performed with the Minimap preset for nanopore reads ('map-ont'). Comparison to BIGSI. As we were unable to reproduce the original plasmid search experiment  17  with BIGSI on our iMac computer (due to the required database transfer of 1. 43 TB over an unstable FTP connection), we used the values provided in the original publication  17. To ensure a fair comparison, we focused on evaluating the total CPU time (sys+usr) and verified that our parallelization efficiency was close to the maximal one (680% out of 800% possible achieved, based on the values in Supplementary Table  7 ).
ACKNOWLEDGEMENTS This work was partially supported by the  NIGMS  of the  National Institutes of Health  ( R35GM133700 ), the  David and Lucile Packard Foundation , the  Pew Charitable Trusts , and the  Alfred P. Sloan Foundation. R. C. was partially supported by the  European Union 's  Horizon 2020 research and innovation programme  (grants agreements No. 872539 ,  956229 , and  101047160 ) and  ANR Transipedia, SeqDigger, Inception , and  PRAIRIE grants  ( ANR-18-CE45-0020 ,  ANR-19-CE45-0008 ,  PIA/ANR16-CONV-0005 ,  ANR-19-P3IA-0001 ). Portions of this research were conducted on the O2 highperformance compute cluster, supported by the  Research Computing Group at Harvard Medical School , and on the  GenOuest bioinformatics core facility  (https://www. genouest. org).
DATA AVAILABILITY STATEMENT All data supporting the findings of this study and the developed software are available within the paper and its Supplementary Materials (Supplementary Table  4 ). Software Phylign Snakemake pipeline https://github. com/karel-brinda/phylign MiniPhy Snakemake pipeline https://github. com/karel-brinda/miniphy MiniPhy-COBS Snakemake pipeline https://github. com/leoisl/miniphy-cobs De-MiniPhy-BIGSIdata Client program to download and decompress de Bruijn graphs from the BIGSIdata collection https://github. com/karel-brinda/de-miniphybigsidata ProPhyle (modified, v0. 3. 3) ProPhyle metagenomic classifier https://github. com/prophyle/prophyle COBS (modified, v0. 3) COBS k-mer indexer https://github. com/iqbal-lab-org/cobs Attotree A fast reimplementation of Mashtree functionality https://github.
com/karel-brinda/attotree Phylogenetically compressed genome collections NCTC3k Assemblies (XZ) https://doi. org/10. 5281/zenodo. 5533354 BIGSIdata De Bruijn graphs (simplitigs after kmer propagation; XZ) https://doi. org/10. 5281/zenodo. 5555253 661k Assemblies (XZ) https://doi. org/10. 5281/zenodo. 4602622 Assemblies (MBGC) https://doi. org/10. 5281/zenodo. 6347064 k-mer index (COBS; XZ) https://doi. org/10. 5281/zenodo. 7313926 https://doi. org/10. 5281/zenodo. 7313942 https://doi. org/10. 5281/zenodo. 7315499 661k-HQ k-mer index (COBS; XZ) https://doi. org/10. 5281/zenodo. 6849657 https://doi. org/10. 5281/zenodo. 6845083.
of BLAST NT database size. The size of the BLAST NT database for the time period between 2002-01-01 and 2022-11-01 was estimated using five types of online resources, resulting in n=27 values. First, file sizes were manually recorded from the official NCBI website https://ftp. ncbi. nih. gov/blast/db/FASTA/ (n=11, between 2020-04-05 and 2022-11-01). Second, additional values were obtained from the snapshots of this website and its NCBI mirrors on http://web. archive. org (n=7, between 2012-10-11 and 2022-06-06). Third, archived versions of the NT database were found in diverse online repositories (n=3, between 2017-10-26 and 2021-01-15). Fourth, the NT database size was documented in a software documentation (n=1, 2013-12-03). Finally, the number of base pairs in the NT database was also documented in literature (n=5, between 2002-01-01 and 2010-01-01) (Supplementary Table.
This resulted in n=425,161 de Bruijn graphs (out of the original n=463,331 genomes from the FTP or n=447,833 genomes reported in ref 17 ). 661k. The 661k collection was downloaded in March 2021 from the official FTP repository specified in ref 16 , using RSync by rsync -avp rsync://ftp. ebi. ac. uk/pub/databases/ENA2018-bacteria-661k/Assemblies/{pref}.
These isolates had originally been collected and sequenced by various laboratories worldwide, and their raw-read data deposited to repositories synchronized with the ENA (ENA, NCBI SRA, and DDBJ Sequence Read Archive). The assemblies were generated using a single unified pipeline (https://github. com/iqbal-lab-org/assemble-all-ena) based on Shovill (https://github. com/tseemann/shovill) by EBI, and provided on an HTTP/FTP website (https://ftp. ebi. ac. uk/pub/databases/ENA2018-bacteria-661k/), along with metadata on FigShare 78. collection provides an example of a high-quality collection of draft genomes of a single low-diversity bacterial species, generated using a standardized sequencing and assembly protocol. NCTC3k. The NCTC3k collection comprises 1,065 draft and complete assemblies of isolates of various bacterial species, derived from strains in the National Collection of Type Cultures (NCTC) collection and analyzed by Public Health England, the Wellcome Sanger Institute, and Pacific Biosciences as part of the NCTC 3000 Project BIGSIdata.
The BIGSIdata collection comprises 425,160 cleaned de Bruijn graphs representing nearly all bacterial and viral isolates available in the European Nucleotide Archive (ENA) as of December 28,29 was released (v0. 3. 3. 1, https://github. com/prophyle/prophyle). The main 2016 17. These isolates had originally been collected and sequenced by various laboratories worldwide, improvement compared to previous versions includes the possibility to stop after k-mer propagation, deposited as raw-read data or genome assemblies to repositories synchronized with the ENA (ENA, without proceeding to the construction of an FM-index, as such an index is unnecessary for phylogenetic NCBI SRA, and DDBJ Sequence Read Archive), and later downloaded and transformed into cleaned de compression using MiniPhy. The new version of ProPhyle is provided in the form of a Github release Bruijn graphs using McCortex 40,76 (k=31) by the European Bioinformatics Institute (EBI).
The resulting (https://github. com/iqbal-lab-org/cobs/releases) and pre-built packages on Bioconda 69. Overview of the five test microbial collections GISP. 73   (https://www. culturecollections. org. uk/collections/nctc-3000-project. aspx). The isolates were sequenced using the PacBio Single Molecule, Real-Time (SMRT) DNA Sequencing technology, and assembled using automated pipelines. The assembled genomes are publicly available from the https://www. sanger. ac. uk/resources/downloads/bacteria/nctc/ website. The NCTC3k collection provides an example of a collection of high-quality, nearly complete genomes from diverse bacterial species. SC2. The SC2 collection comprises 590,779 complete assemblies of SARS-CoV-2 isolates obtained from the GISAID database 74   as of 2021-05-18. These isolates were collected, sequenced, and assembled by various laboratories worldwide between 2020 and 2021 using various protocols.
The phylogeny of the isolates is known and was computed by the sarscov2phylo software (https://github. com/roblanf/sarscov2phylo/, ref 75   ). The SC2 collection provides an example of a large collection of genomes of varying quality obtained from epidemiological surveillance of a single viral species at a global scale. graphs were provided on an HTTP/FTP website (http://ftp. ebi. ac. uk/pub/software/bigsi/nat_biotech_2018), along with metadata on Figshare 77   , although not all of the original 447,833 graphs could be retrieved in this study (see below). The.
and provided for download on Zenodo 79. www. sanger. ac. uk/resources/downloads/bacteria/nctc/, retrieved on 2022-09-14). The HTML table was manually exported to XLSX and used to construct a translation table from NCTC accession numbers to corresponding species. The accessions of the assemblies in our collection were then extracted from file names, translated to species, and the species counted. Overall, this resulted in n=1,065 assemblies of 259 species. SC2. The SARS-Cov-2 data were downloaded from the GISAID website (https://www. gisaid. org/, as of 2021-05-18) in the form of an assembly file ('sequences_fasta_2021_05_18. tar. xz', n=1,593,858) and a Sarscov2phylo phylogeny 82 ('gisaid-hcov-19-phylogeny-2021-05-11. zip', n=590,952). After converting both files to the same set of identifiers and removing isolates with missing data, we obtained n=590,779 genome assemblies organized in a phylogenetic tree.
NCTC3k. The assemblies were obtained in the GFF format from ftp://ftp. sanger. ac. uk/pub/project/pathogens/NCTC3000 by wget -m -np -nH --cut-dirs 3 -retr-symlinks \ ftp://ftp. sanger. ac. uk/pub/project/pathogens/NCTC3000. The obtained files were converted them to the FASTA format by any2fasta (https://github. com/tseemann/any2fasta, v0. 4. 2) parallelized by GNU Parallel 80 and uploaded to Zenodo (ref 81   , http://doi. org/10. 5281/zenodo. 4838517). The number of species in the collection was determined based on the data provided in the main Sanger/Public Health England assembly table for NCTC 3000 (https://BIGSIdata. The BIGSI collection data 17   were downloaded from the associated FTP (http://ftp. ebi. ac. uk/pub/software/bigsi/nat_biotech_2018/), including cleaned de Bruijn graphs, taxonomic information inferred using Kraken 26 , and abundance reports computed using Bracken 57.
68  (v2. 2. 10) by To compare the compressibility of the 661k collection in the ENA accession lexicographic order to the full phylogenetic compression, we streamed the genomes from the main collection file provided on http://ftp. ebi. ac. uk/pub/databases/ENA2018-bacteria-661k/661_assemblies. tar, decompressed them on-the-fly, converted them to the one-line FASTA format using SeqTK 67 , and compressed them using XZ with 32 threads by pv 661_assemblies. tar | tar -xOf -| gunzip -c | seqtk seq | xz -9 -T32 jellyfish count --mer-len 31 --size 200G --threads 32 \ --output kmer_counting. jf --out-counter-len=1 --canonical which resulted in 44,349,827,744 distinct k-mers (28,706,296,898 unique k-mers) for the 661k collection and in 35,524,194,027 distinct k-mers (22,904,412,202 unique k-mers) for the 661k-HQ collection (as described below).
The files uploaded to https://doi. org/10. 5281/zenodo. 4602622 are higher by approximately 0. 2 GB (approx. 0. 7% of the total size) compared to the value Supplementary Table 3 as the Zenodo submission was done with an older version of compressive phylogenies without their post-processing. Recompression using MBGC. Individual phylogenetically compressed batches from the previous step were converted to single FASTA files by 'tar -xOvf {input. xz}' and then compressed using MBGC 18 (v1. 2. 1) with 8 threads and the maximum compression level by mbgc -i {input. fa} -c 3 -t 8 {output. mbgc} Compression in the lexicographic order. Data in ENA and other similar repositories have identifiers assigned in the order in which they are uploaded, individual uploads typically proceed by uploading entire projects, and these typically involve phylogenetically very close genomes.
the plasmid experiment in Supplementary Tab. 6). Updated COBS. To integrate COBS into Phylign, new versions of COBS 37 were created (v0. 2, v0. 3, https://github. com/iqbal-lab-org/cobs). The updates include support for macOS, streaming of indexes into memory, and multiple bug fixes. The new versions of COBS are provided in the form of Github releases (https://github. com/iqbal-lab-org/cobs/releases) and pre-built packages on Bioconda 69. Benchmarking of the decompression time.
This climatological moisture convergence can be further decomposed into its time-mean (    ) and eddy (   ′  ′ ) components. The time-mean component is calculated using the climatological mean data provided in the DeepMIP data set while the eddy component is calculated as the residual between P-E and the time-mean component given that the high temporal resolution data required to calculate this term explicitly is not available as part of the DeepMIP data set.
Proxy Compilation Here we compile paleobotanical MAP estimates for the late Paleocene (59. 2-56 Ma; Thanetian) to early Eocene (56. 0-47. 8 Ma; Ypresian). Our compilation builds upon previous EECO-  (Carmichael et al. , 2016)  and PETMaged  (Carmichael et al. , 2017)  compilations. We supplement this with (a) published MAP estimates generated since, and (b) newly generated MAP estimates using CLAMP and NLR on published palynological and macrofloral (predominantly leaf-based) datasets. Our new proxy synthesis (n = 322) contains 133 MAP estimates (41%) from  Carmichael et al. (2016) , 106 data points (33%) from other published sources, and 83 new data points (26%) (Figure  1 ; Tables  S1  and S2  in Supporting Information S1). The new data in the compilation helps to improve geographical coverage in previously data-poor regions, including central west coast and eastern Africa (e.