dataesr/scientific-paragraphs-categorization huggingface dataset:
For example, jD2Stat  (Chan et al. , 2014)  gives results for 5000 sequences of 1500 nucleotides in 14 min, which would clearly make computationally intractable the dereplication of hundreds of thousands of whole prokaryotic genomes. As of January 2021, we only found two programs that were made to dereplicate genomes, dRep  (Olm et al. , 2017)  and Assembly-Dereplicator  (Wick & Holt, 2019). These two programs are presented below. Considering the limitations of the existing tools for assembling representative sets of prokaryotic genomes, the present article describes our own program called ''ToRQuEMaDA'' (abbreviated TQMD in the following for convenience) for Tool for Retrieving Queried Eubacteria, Metadata and Dereplicating Assemblies. TQMD is a wordbased alignment-free dereplicating tool for both public and private prokaryotic genomes designed for both high-performance computing (HPC) clusters and powerful single-node computing servers.
TQMD is available on Bitbucket and can be installed on any HPC with SGE/OGE (Sun/Open Grid Engine) installed as a scheduler. Few modifications are needed to adapt the scripts to most local setups. A Singularity  (Kurtzer, Sochat & Bauer, 2017)  container is also available for single-node computers without a scheduler. TQMD works both in parallel and iteratively. Using default parameter values, each elemental job takes two to three hours to complete (see Materials and Methods for test hardware specifications), and if enough CPUs are available to run all jobs of a given round at the same time, such a round should only take two to three hours. Usually, four to five rounds are sufficient to achieve the dereplication. Therefore, a single run of TQMD against ∼60,000 Bacteria in NCBI RefSeq takes 8 to 15 h to complete.
Hardware Almost all the computational work was performed on a grid computer IBM/Lenovo Flex System composed of one big computing node (x440) and nine smaller computing nodes (x240), featuring a total of 196 physical cores, 2. 5 TB of RAM and 160 TB of shared mass storage, and operating under CentOS 6. 6. Beyond ''bignode'' (running the scheduler and the MySQL database; see below), only four of the smaller computing nodes were used when testing TQMD; their specifications are as follow: 2 CPUs Intel Xeon E5-2670 (8 cores at 2. 6 GHz with Hyper-Threading enabled), 128 GB of RAM. For the dRep test (see below), we had to use a desktop workstation (Ubuntu Linux 16. 04) featuring 2 CPUs Intel Xeon E5-2620 v4 (8 cores at 2. 1 GHz with Hyper-Threading enabled) and 64 GB of RAM. Based on the comparator found on the website http://cpubenchmark.
Software architecture TQMD is composed of a database and includes two main phases: (1) a periodic preparation phase in which newly available genome assemblies (''genomes'' for short) are downloaded (or locally imported for private genomes) and individual genome metrics are computed, and (2) an ''on-demand'' dereplicating phase in which genomes (both new and old) are dereplicated on the fly to provide a list of high-quality representative genomes as a result (Fig. 1 ). The database stores the paths to the individual genome (FASTA) files, the individual genome metrics and the list of representative genomes produced by each TQMD run. Each piece of data is computed independently; if a dereplication request is issued during the computation of newly available genomes, TQMD only uses the genomes for which all the data is available in the database.
Parallelized steps are shown as overlaid boxes. Full-size DOI: 10. 7717/peerj. 11348/fig-  1  During the preparation phase, we download the genomes and proteomes and precompute all the data required by the dereplication phase to store them in the database: indexes of nucleotide k-mers for single genomes, genome assembly quality metrics, genome annotation richness metrics, Small Subunit ribosomal RNA (SSU (16S) rRNA) predictions, contamination level and completeness level, whereas during the dereplication phase, we cluster the genomes based on these k-mer indexes and select a representative for each cluster based on a user-modifiable ranking formula taking into account assembly quality, annotation richness, contamination and completeness level. These criteria were chosen, so as to select the best representative genomes  (Bowers et al.
Preparation phase As shown in Fig. 1A , we first download the prokaryotic genomes from NCBI RefSeq (O'  Leary et al. , 2016)  (or from GenBank  (Sayers et al. , 2020) ). For the sake of data traceability, TQMD never gets rid of older genomes; newly released genomes are simply added to its internal database. The genomes from RefSeq and GenBank are kept physically separate. As TQMD was developed over five years, we have progressively accumulated several different versions of the RefSeq database, starting with release 79 (85,465 prokaryotic genomes, including 713 Archaea), then 79+92 (126,959 prokaryotic genomes, including 1,037 Archaea) and finally 79+92+203 (223,785 prokaryotic genomes, including 1,312 Archaea). Once RefSeq is up to date locally, we compute single-genome k-mer indexes and other metrics.
For each of these computations, we use third-party programs and scripts (JELLYFISH, QUAST, RNAmmer, CD-HIT and Forty-Two or CheckM), except for the richness of the annotations, which we evaluate using an in-house script. JELLYFISH (v1. 1. 12) (Marçais & Kingsford, 2011)  is used to compute the k-mer indexes for single genomes (TQMD can also work with JELLYFISH v2. x and Mash  (Ondov et al. , 2016) ; see below). We tested several sizes for our k-mers. While JELLYFISH v1. x used to crash when using a size below 11 nucleotides, thus setting a hard lower bound on k-mer size, it is no longer an issue in JELLYFISH v2. x. On the other hand, while longer k-mers improve the specificity, they also require longer computing times  (Zielezinski et al. , 2017). With a size of 11, there are almost 4. 2 millions (4 11 ) possible words.
If GenBank is used instead of RefSeq, it is highly recommended to enable the canonical option in TQMD due to the presence of genomes likely to be not assembled (still at the scaffold stage) or only very poorly assembled. Yet, one has to remember that canonical k-mers are twice less numerous for a given k-mer size than strand-specific k-mers, which might become an issue for distinguishing large genomes. QUAST (v4. 4)  (Gurevich et al. , 2013)  is used to estimate the quality of genome assemblies (QUAST v5. x is also supported). We retrieve several quality metrics (13 in total) for each genome, these being the number of DNA sequences, the number of DNA sequences (or contigs) > 1 kbp, the size of the complete genome, the size of the complete genome composed of DNA sequences > 1 kbp, the number of contigs, the largest DNA sequence, the size of the complete genome composed of DNA sequences > 500 bp, the GC content, the N50, N75, L50 and L75 values, and the number of ''N'' per 100 kbp (N is the symbol used to scaffold contigs without matching ends).
While it is not an issue with RefSeq genomes, for which such predictions are always available, if TQMD is provided with an input genome set from a different source (GenBank or private genomes) with missing predicted proteomes, the related genomes will be automatically discarded (at least if the annotation metrics are used in the ranking formula). Our ''certainty'' metric corresponds to the proportion of sequences in a given proteome that we deem uncertain. To this end, we first count the number of sequence descriptions (in FASTA definition lines) with words indicating uncertainty, such as ''probable'', ''hypothetical'' or ''unknown'', then we compute a relative score as follows: Certainty = 1 -count of uncertain proteins total count of proteins For ''completeness'', instead of counting the number of uncertain proteins, we count the number of proteins without any description: Completeness = 1 -count of unannotated proteins total count of proteins Regarding genome contamination, RNAmmer (v1.
2)  (Lagesen et al. , 2007)  is used to predict the SSU (16S) rRNA of the genomes. By using cd-hit-est (v4. 6)  (Li & Godzik, 2006; Fu et al. , 2012)  with an identity threshold of 97. 5%  (Taton et al. , 2003) , TQMD optionally creates a list of genomes featuring at least two SSU (16S) rRNA sequences belonging to different species (i. e. , clustered in distinct CD-HIT clusters). This list of likely chimerical (or at least contaminated) genomes can be provided to filter out the genomes given as input to TQMD or produced in output by TQMD (see below). Another (more recent) possible threshold for species delineation based on SSU (16S) rRNA identity would be 99%  (Edgar, 2018)  and TQMD also supports such a setting. Finally, another contamination metric is also available for the ranking: the genome contamination level estimated by the program Forty-Two (Van Vlierberghe, 2021) (v0.
210570 or higher ''42'') based on the comparison of the genome ribosomal proteins to the reference sequences of the RiboDB database  (Jauffrit et al. , 2016). While this is the recommended approach for probing genome-wide contamination due to its speed, TQMD also supports CheckM  (Parks et al. , 2015) (v1. 1. 3) to predict ''genome completeness'' and ''genome contamination''. The contamination assessment of the latter is based on lineage-specific marker genes in addition to ribosomal proteins. Once all these individual k-mer indexes and metrics are computed for all individual genomes, the genomes are ranked in a global ranking from the best to the worst genome (to be selected as a cluster representative), using an equal-weight sum-of-ranks approach available in the Perl module Statistics::Data::Rank.
For each metric, a ranking is produced across all genomes and the final rank of a specific genome is computed as the sum of each of those individuals ranks without favoring one metric over another. For now, we do not consider all the metrics stored in the TQMD database, since all are optional and some are redundant. The five metrics (in TQMD syntax) used to compute the default ranking are: (1) assembly quality: quast. N. per. 100. kbp; (2) assembly quality: quast. largest. contig. ratio (= quast. largest. contig / quast. total. len. 1000. bp); (3) annotation richness: annot. certainty; (4) contamination level: 42. contam. perc; (5) contamination level: 42. added. ali. The first two metrics are obtained from QUAST, the third from our in-house script, and the fourth and fifth from ''42''. Finally, it is worth noting that TQMD allows the user to devise a custom ranking formula involving any combination of the 30 supported metrics (see details in TQMD manual).
Dereplication phase Genome clustering can be carried out on the full set of genomes stored in the TQMD database or only on one or more taxonomic subsets of them. Moreover, both positive (inclusion and/or representativeness priority) and negative (exclusion) lists of GCA/GCF numbers can be provided to alter TQMD input and output genome sets. TQMD itself optionally produces such a negative list to exclude genomes featuring multiple SSU (16S) rRNA sequences (see above). Furthermore, both public (from RefSeq/GenBank) and private (i. e. , custom) genomes can be dereplicated simultaneously. Moreover, the presence of at least one SSU (16S) rRNA predicted by RNAmmer can be used as a requirement for the genome to be selected, which would rule out some metagenome-assembled genomes (MAGs), for which rRNA genes are often missing  (Cornet et al.
, 2018a). Consequently, this option is recommended when working with RefSeq but not GenBank, at least if the selection of some lesser quality MAGs is important for the user. Regarding priority lists, they can be useful in comparative genomics, when one wants to include model organism genomes without sacrificing dereplication. As shown in Fig. 1B , the dereplication process is iterative and stops once it deems itself finished. Its decision is based on three different convergence criteria, for which we provide default threshold values but these can be modified by the user (see below). TQMD stops cycling as soon as one criterion is satisfied. Two different distances can be used for clustering genomes with TQMD, each one derived from a distinct similarity metric, the Jaccard Index (JI; see  (Real & Vargas, 1996) ) or the Identical Genome Fraction (IGF; see  (Cornet et al.
, 2018b) ), both applied to shared k-mers at the nucleotide level. The effective distance used by TQMD is then obtained by subtracting the corresponding similarity metric from 1. The JI is a measure of the similarity between two finite datasets. It is defined as the intersection over the union of the two datasets A and B: JI(A,B) = |A ∩ B| |A ∪ B| The JI can be computed in two different manners: (1) exact computation using JELLYFISH (default option) and (2) approximate estimation using Mash  (Ondov et al. , 2016) (v1. 1. 1). If Mash is to be used, precomputation of single-genome k-mers is not required. The IGF, for Identical Genome Fraction, replaces the union in the JI by the size of the smallest of the two datasets A and B: IGF(A,B) = |A ∩ B| min(A,B) The TQMD algorithm works similarly for both distances and is inspired by the greedy clustering approach implemented in packages such as CD-HIT  (Jones, Pevzner & Pevzner, 2004; Li & Godzik, 2006; Fu et al.
, 2018b) ). Thus, this strategy allowed us to simultaneously assess the completeness and the contamination level of each representative proteome while providing widely sampled ribosomal proteins for phylogenomic analyses (Table  1 ). For the bacterial dataset (B), the largest of the nine TQMD selections, this step took less than three hours to complete. For each TQMD run, we assembled a supermatrix from the ribosomal proteins retrieved earlier (Table  1 ). Briefly, sequences were aligned with MAFFT v7. 453  (Katoh & Standley, 2013) , then the alignments were cleaned using ali2phylip. pl from the Bio::MUST::Core software package (D. Baurain, https://metacpan. org/release/Bio-MUST-Core), which implements the BMGE  (Criscuolo & Gribaldo, 2010)  filter (min=0. 3, max=0. 5, bmge-mask=loose). This step reduced the proportion of missing sites in the alignments.
RESULTS AND DISCUSSION The TQMD workflow has two separate phases: a preparation phase (Fig. 1A ) and a dereplication phase (Fig. 1B ). The objective of the preparation phase is to compute the genome-specific data that will be needed during the dereplication phase. These operations are embarrassingly parallel and very easy to speed up. In contrast, the dereplication phase considers all genomes at once, with the aim of clustering similar genomes based on pairwise distances and selecting the best representative for each cluster. To achieve this in the presence of many genomes, TQMD resorts to a greedy iterative heuristic in which each round is parallelized through a divide-and-conquer approach. The two phases are interconnected by the means of a relational database (see 'Materials and Methods' for details).
Another point to consider are TQMD heuristics and their parameterization. Since TQMD is iterative, one can always decide to dereplicate genomes that are themselves representatives obtained in one or more previous runs. When trying to dereplicate very large and taxonomically broad genome sets, this raises the possibility to ''guide'' the dereplication by first clustering several phylum-wide subsets before merging the selected representatives in a single dataset to be dereplicated once more. This ''indirect strategy'' is to be contrasted with the ''direct strategy'', in which TQMD is left dealing with the whole dataset from the very beginning. Regarding the divide-and-conquer algorithm operating during a single round, four parameters might be relevant: the pack size (e. g. , 200 to 500), the clustering mode (loose or strict) and the dividing scheme (random or taxonomically-guided).
A word about the genome source In addition to RefSeq genomes, TQMD can also download and cluster GenBank genomes, along with (optional) custom genomes provided by the user. To test the effect of the source database, we studied the dereplication of RefSeq and GenBank Archaea (release 203), which have the advantage of combining a small number of genomes (941 and 4129 genomes, respectively) while featuring a lot of unclassified organisms, candidate phyla and metagenomic assemblies in GenBank (Table  5 ). Beyond the speed penalty due to sheer difference in the number of genomes, which influences the number of comparisons TQMD has to perform, switching to GenBank as the genome source also requires using canonical k-mers to account for the lesser assembly quality of many genomes (see Materials and Methods for details) and/or selecting Mash as the k-mer engine.
Moreover, with GenBank, the diversity of representative genomes is expanded with candidate phyla, but at the cost of more unclassified genomes and also (meta)genomes of lesser assembly quality. Unclassified genomes are genomes without higher-level taxonomic taxa, which hinders the taxonomic sort heuristic and makes it harder for TQMD to dereplicate them (since they can start in packs distinct from those including the genomes they are the most similar to). Regarding genomes of lesser quality, some can act as a bridge between two clusters that should not be clustered together (as discussed above with the polyphyletic groups) if they are chimerical in any way (either genuinely or due to the mixing of different organisms). In the worst case, all genomes end up lumped together in a single large cluster (last row of Table  5 ).
dRep is a less aggressive program than TQMD, which is unsurprising as the former is meant to be used on sets of MAGs and to dereplicate at the species level, while the latter is meant to be used on every completely sequenced prokaryotic genome available and to dereplicate at the phyla/class level. Moreover, from the very start, TQMD was designed with scalability in mind, so as to accommodate the ever growing number of sequenced genomes. In principle, dRep could be used aggressively like TQMD, by fine-tuning two different thresholds (primary and secondary clusters), but this would need dRep to allow the user to choose a different Mash k-mer size, which does not appear to be possible (for the average user). On the other hand, TQMD can be used to dereplicate down to the species level more easily (only one threshold to specify) but it would take a longer time to  , not yet published; its last update dates from November 2019.
Its main advantage is ease of use, since it is a simple (no-installation) script that only needs Mash as a prerequisite. A-D takes as input the path to a folder containing the genomes to be dereplicated and rearranges them randomly and separated into smaller packs (500 genomes per pack by default). The next step is the clustering of each pack serially using Mash. A-D stops as soon as it cannot dereplicate at least one genome from the current pack. However, at least in our hands, A-D revealed to be unstable and/or to perform poorly on our test datasets (see Supplementary Materials for details). TQMD allows the use of two different k-mer engines, JELLYFISH and Mash. With JELLYFISH, TQMD can compute a distance that is based on the exact JI (or the exact IGF), whereas with Mash, it relies on a distance based on the estimate of the JI.
CONCLUSION TQMD is an efficient dereplication tool initially designed for the assembly of phylum-level datasets of representative prokaryotic genomes. It manages to maintain the taxonomic diversity of input genomes while being fast, owing to its aggressive dereplication heuristics, which makes it able to scale with the ever growing number of genome assemblies in public repositories, such as NCBI RefSeq and GenBank. At lower taxonomic levels, TQMD becomes slower, probably because it has to compare more genomes before finding pairs close enough to be clustered and dereplicated. However, the use of the ''strict'' mode for the clustering can at least partially offset this effect. To dereplicate at the lowest taxonomic levels (species or strains), a longer k-mer would be better suited. While this is computationally intractable with the JELLYFISH engine, the support of the faster Mash engine makes it possible.
Figure2Illustration of the divide and conquer strategy of the dereplication phase. From a list of Bacteria downloaded from RefSeq (or GenBank), TQMD either sorts (based on the NCBI taxonomic lineage of each genome) or randomizes the list and splits it into packs of a given size. This allows each pack to be separately dereplicated, especially in parallel. Then all resulting lists of representative genomes are merged back and TQMD decides if it can stop or must refeed the merged list for another round. Full-size DOI: 10. 7717/peerj. 11348/fig-2.
For each dataset, TQMD was launched with the Jaccard Index as a distance, a pack size of 200, the loose clustering mode, and was allocated a maximum of 50 CPUs. Other parameters (direct or indirect strategy and distance threshold) are provided in the table, along with the total running time in CPU hours (h. CPU), the initial number of genomes (# starting), the number of representatives obtained (# repr. ), the number of ribosomal protein alignments used in the supermatrix (# prot. ), and the number of unambiguously aligned amino acids in the supermatrix (# AA). Further details (taxonomy and download links, Krona taxonomic plots, Forty-Two reports, supermatrices and trees) are available at https://doi. org/10. 6084/m9. figshare. 13238936.
The runs carried out on GenBank Archaea used canonical k-mers. The JI runs used a distance threshold of 0. 90 and the IGF runs a threshold of 0. 80. The super-phyla are the Asgard group, the TACK group and the DPANN group. Unclassified genomes are genomes without a phylum in the NCBI Taxonomy. JI: Jaccard Index; IGF: Identical Genome Fraction. Because of technical difficulties with the installation of dRep, we had to use a workstation less powerful than the grid computer used to run TQMD (see 'Materials and Methods'). That is why we did not use all the available bacterial genomes in these tests. Regarding Bacteroidetes, dRep required five hours (using 10 CPUs and default parameters) to select 835 genomes. With TQMD, we used a threshold of 0. 6 on the JI to obtain comparable results. TQMD run lasted 10 h (on at most 6 CPUs) and selected 789 representative genomes, of which 707 were in common with those of dRep.
Data Availability The following information was supplied regarding data availability: TQMD software is available at Bitbucket: https://bitbucket. org/phylogeno/tqmd. The datasets are available at figshare: Léonard, Raphaël R. ; Leleu, Marie; Van Vlierberghe, Mick; Cornet, Luc; Kerff, Frédéric; BAURAIN, Denis (2020): Datasets for Léonard et al. ToRQuEMaDA: Tool for Retrieving Queried Eubacteria, Metadata and Dereplicating Assemblies. figshare. Dataset. https://doi. org/10. 6084/m9. figshare. 13238936. v2.
org/10. 6084/m9. figshare. 13238936. Label Dataset Strategy Threshold h. CPU # starting # repr. # prot. # AA A Bacteria (49) indirect 0. 900 656 63,863 49 53 6338 B Bacteria (151) indirect 0. 880 656 63,863 151 53 6187 C Actinobacteria direct 0. 900 96 8859 20 51 6562 D Bacteroidetes direct 0. 850 16 1225 37 49 6605 E Chlamydia direct 0. 800 6 360 32 44 6131 F Cyanobacteria direct 0. 800 8 428 46 48 6314 G Firmicutes direct 0. 900 242 21,544 22 52 6536 H Proteobacteria direct 0. 885 310 30,690 36 53 6471 I Archaea direct 0. 850 8 432 86 57 7810.
With TQMD, we used a threshold of 0. 6 on the JI to obtain comparable results. TQMD run lasted 10 h (on at most 6 CPUs) and selected 789 representative genomes, of which 707 were in common with those of dRep. Source # super-phyla # phyla # unclassified # genomes Clustering genomes mode RefSeq 3 7 0 941 NA GenBank 3 24 265 4129 NA JI RefSeq 2 6 0 46 strict JI RefSeq 2 6 0 29 loose IGF RefSeq 2 6 0 38 strict IGF RefSeq 1 3 0 16 loose JI GenBank 3 17 38 313 strict JI GenBank 3 15 18 145 loose IGF GenBank 2 10 6 34 strict IGF GenBank 1 1 0 1 loose (648 genomes; phylum Actinobacteria).