dataesr/scientific-paragraphs-categorization huggingface dataset:
Several of these were specifically developed for metagenomics, while others are more broadly used (Table  1 , Supplementary Table  1 ). The assembly results were evaluated using the MetaQUAST  17  metrics and the reference genome and circular element sequences of the benchmark datasets (Supplementary Table  2 , Supplementary methods "Assembly metrics"). As performance metrics, we focused on genome fraction and assembly size, as well as on the number of unaligned bases and misassemblies. Genome fraction measures the assembled percentage of an individual reference genome, assembly size denotes the total length in bp for an assembly (including misassembled contigs), and the number of misassemblies and unaligned bases are error metrics reflective of the assembly quality. Combined, they provide an indication of the performance of a program, while individually, they are not sufficient for assessment.
For instance, while assembly size might be large, a high-quality assembly also requires the number of misassemblies and unaligned bases to be low. To assess how much metagenome data was included in each assembly, we also mapped all reads back to them. Across all datasets (Supplementary Table  3 ) the assembly statistics varied substantially by program and parameter settings (Supplementary Figures  SA1-SA12 ). The gold standard coassembly of the five samples from the high complexity data set has 2. 80 Gbp in 39,140 contigs. For the assemblers, values for this data set ranged from 12. 32 Mbp to 1. 97 Gbp assembly size (0. 4% -70% of the gold standard co-assembly, respectively), 0. 4% to 69. 4% genome fraction, 11 to 8,831 misassemblies and 249 bp to 40. 1 Mbp unaligned contigs (Supplementary Table  2 , Supplementary Fig.
Analysis of the low and medium complexity data sets delivered similar results (Supplementary Figs SA4-SA6, SA7-SA9). Closely related genomes-To assess how the presence of closely related genomes affects the performance of assembly programs, we divided genomes according to their Average Nucleotide Identity (ANI  18  ) to each other into "unique strains" (genomes with < 95% ANI to any other genome) and "common strains" (genomes with closely related strains present; all genomes with an ANI >= 95% to any other genome in the dataset). When considering the fraction of all reference genomes recovered, Meraga, Megahit and Minia performed best (Fig. 1a ). For the unique strains, Minia and Megahit had the highest genome recovery rate (Fig. 1c ; median over all genomes 98. 2%), followed by Meraga (median 96%) and VELOUR_k31_C2.
0 20 , MetaBAT 21 , MetaWatt-3. 5  22  , CONCOCT  23  , PhyloPythiaS+  24  , taxatortk  25  , MEGAN 6  26  and Kraken  27. Submitters could choose to run their program on the provided gold standard assemblies or on individual read samples (MEGAN 6), according to their suggested application. We then determined their performance for addressing important questions in microbial community studies: do they allow the recovery of high quality genome bins for individual strains, i. e. with high average completeness (recall), and purity (precision), i. e. low contamination levels? How does strain level diversity affect performance? How is performance affected by the presence of non-bacterial sequences in a sample, such as viruses or plasmids? Do current taxonomic binners allow recovery of higher-ranking taxon bins with high quality? How does their performance vary across taxonomic ranks? Which programs are highly precise in taxonomic assignment, so that their outputs can be used to assign taxa to genome bins? Which software has high recall in the detection of taxon bins from low abundance community members, as is required for metagenomes from ancient DNA and for pathogen detection? Finally, which programs perform well in the recovery of bins from deep-branching taxa, for which no sequenced genomes yet exist? Recovery of individual genome bins-We first investigated the performance of each program in the recovery of individual genome (strain-level) bins.
We calculated completeness and purity (Supplementary Methods) for every bin relative to the genome that was most abundant in that bin in terms of assigned sequence length. In addition, we calculated the Adjusted Rand Index (ARI) as measure of assignment accuracy for the portion of the data assigned by the different programs. As not all programs assigned the entire data set to genome bins, these values should be interpreted under consideration of the fraction of data assigned (Fig. 2d ). These two measures complement completeness and purity averaged over genome bins, as assignment accuracy is evaluated per bp, with large bins contributing more than smaller bins in the evaluation. For the genome binners, both average bin completeness (ranging from 34% to 80%) and purity (ranging from 70% to 97%) varied substantially across the three datasets (Supplementary Table  4 , Supplementary Fig.
B1 ). For the medium and low complexity datasets, MaxBin 2. 0 had the highest average values (70-80% completeness, more than >92% purity), followed by other programs with comparably good performance in a narrow range (completeness ranging with one exception from 50-64%, more than 75% purity). Notably, other programs assigned a larger portion of the datasets in bp than MaxBin 2. 0, though with lower ARI (Fig. 2d ). For applications where binning a larger fraction of the dataset at the cost of some accuracy is important, therefore, programs such as MetaWatt-3. 5, MetaBAT and CONCOCT could be a good choice. The high complexity dataset was more challenging to all programs, with average genome completeness decreasing to around 50% and more than 70% purity, except for MaxBin 2. 0 and MetaWatt-3.
5, which showed an outstanding purity of above 90%. The programs either assigned only a smaller dataset portion (>50% of the sample bp, MaxBin 2. 0) with high ARI or assigned a larger fraction with lower ARI (more than 90% with less than 0. 5 ARI). The exception was MetaWatt-3. 5, which assigned more than 90% of the dataset with an ARI larger than 0. 8, thus best recovering abundant genomes from the high complexity dataset. Overall, MetaWatt-3. 5, closely followed by MaxBin 2. 0, recovered the most genomes with high purity and completeness from the three datasets (Fig. 2c , Supplementary Table  17 ). Effect of strain diversity-When considering only unique strains, the performance of all genome binners improved substantially, both in terms of average purity and completeness per genome bin (Fig.
2a ). For the medium and low complexity datasets, all had a purity of above 80%, while completeness was more variable. MaxBin 2. 0 performed best across all three datasets, showing more than 90% purity and 70% or more completeness. An almost equally good performance for two datasets was delivered by MetaBAT, CONCOCT and MetaWatt-3. 5. For the "common strains" of all three datasets, however, completeness decreased substantially (Fig. 2b ), similarly to purity for most programs. MaxBin 2. 0 still stood out, with more than 90% purity on all datasets. Interestingly, when considering the value of bins reconstructed by taxon binners for genome reconstruction, taxonomic binners had lower completeness than genome binners, but reached a similar purity, thus delivering high quality, partial genome bins (Supplementary Material 1.
In addition, we determined the overall classification accuracy for each dataset, as measured by total assigned sequence length, and misclassification rate for all assignments. While the former two measures allow assessing performance averaged over bins, where all bins are treated equally, irrespective of their size, the latter are influenced by the actual sample taxonomic constitution, with large bins having a proportionally larger influence. For the low complexity data set, PhyloPythiaS+ had the highest sample assignment accuracy, average taxon bin completeness and purity, which were all above 75% from domain to family level. Kraken followed, with average completeness and accuracy still above 50% until family level. However, purity was notably lower, mostly caused by prediction of many small false bins, which affects purity more than overall accuracy, as explained above (Supplementary Fig.
B3 ). Removing the smallest predicted bins (1% of the data set) increased purity for Kraken, MEGAN, and, most strongly, for taxator-tk, for which it was close to 100% until the order level, and above 75% until the family level (Supplementary Fig. B4 ). This shows that small predicted bins by these programs are not reliable, but otherwise, high purity can be reached for higher ranks. Below the family level no program performed very well, with all either assigning very little data (low completeness and accuracy, accompanied by a low misclassification rate), or performing more assignments with a substantial amount of misclassification. Another interesting observation is the similar performance for Kraken and Megan, which was not observed on the other datasets, though. These programs employ different features of the data (Table  1 ), but rely on similar algorithms.
B5, B6 ). Analysis of low abundance taxa-We determined which programs had high completeness also for low abundance taxa. This is relevant when screening for pathogens in diagnostic settings  28  , or for metagenome studies of ancient DNA samples. Even though a high completeness was achieved by PhyloPythiaS+ and Kraken until the rank of family (Fig. 2e, f ), it degraded for lower ranks and low abundance bins (Supplementary Fig. B7 ), which are of most interest for these applications. It therefore remains a challenge to further improve the predictive performance. Deep-branchers-Taxonomic binning methods commonly rely on comparisons to reference sequences for taxonomic assignment. To investigate the effect of increasing evolutionary distances between a query sequence and available genomes, we partitioned the challenge datasets by their taxonomic distances to sequenced reference genomes as genomes of new strains, species, genus or family (Supplementary Fig.
B8 ). For genomes representing new strains from sequenced species, all programs performed well, with generally high purity and oftentimes high completeness, or with characteristics observed also for other datasets (such as low completeness for taxator-tk). At increasing taxonomic distances to the reference, performance for MEGAN and Kraken dropped substantially, in terms of both purity and completeness, while PhyloPythiaS+ decreased most notably in purity and taxatortk in completeness. For deep branchers at larger taxonomic distances to the reference collections, PhyloPythiaS+ maintained the best overall purity and completeness. Influence of plasmids and viruses-The presence of plasmid and viral sequences had almost no effect on the performance for binning bacterial and archaeal organisms.
7±6. 9% and 8. 8±26. 4% for recall and precision, respectively. The other error metrics showed similar performance trends for all samples and methods (Figs 3c and Supplementary Figs. P13-P17 ). Parameter settings and software versions-Several profilers were submitted with different parameter settings or versions (Supplementary Table  1 ). For some, this had little effect: for instance, the variance in recall among 7 different versions of FOCUS on the low complexity sample at the family level was only 0. 002. For others, this caused large changes in performance: for instance, one version of DUDes had twice the recall compared to another at the phylum level on the pooled high complexity sample (Supplementary Figs. P13-P17 ). Interestingly, a few developers chose not to submit results beyond a fixed taxonomic rank, such as for Taxy-Pro and Quikr.
These submissions generally performed better than default program versions submitted by the CAMI team; indicating that, not surprisingly, experts can generate better results than when using a program's default setting. Performance for viruses and plasmids-In addition to microbial sequence material, the challenge datasets also included sequences of plasmids, viruses and other circular elements (Supplementary Table  7 ). We investigated the effect of including these data in the gold standard profile for the taxonomic profilers (Supplementary Figs  P18-P20 ). Here, the term "filtered" is used to indicate the gold standard did not include these data, and the term "unfiltered" indicates use of these data. The metrics affected by the presence of these data were the abundance-based metrics (L1 norm at the superkingdom level and Unifrac), and precision and recall (at the superkingdom level).
All methods correctly detected Bacteria and Archaea, indicated by a recall of 1. 0 at the superkingdom level on the filtered samples. The only methods to detect viruses in the unfiltered samples were MetaPhlAn 2. 0 and CLARK. Averaging over all methods and samples, the L1 norm at the superkingdom level increased from 0. 051 for the filtered samples to 0. 287 for the unfiltered samples. Similarly, the Unifrac metric increased from 7. 213 for the filtered to 12. 361 for the unfiltered datasets. Thus, a substantial decrease in the fidelity of abundance estimates was caused by the presence of viruses and plasmids in a sample. Taxonomic profilers vs. profiles derived from taxonomic binning-We compared the profiling results to those generated by several taxonomic binners using a simple coverage-approximation conversion algorithm for deriving profiles from taxonomic bins (Supplementary Methods, Figs  P21-P24 ).
CONCLUSIONS Determination of program performance is essential for assessing the state of the art in computational metagenomics. However, a lack of consensus about benchmarking datasets and evaluation metrics has complicated comparisons and their interpretation. To tackle this problem, CAMI has engaged the global developer community in a benchmarking challenge, with more than 40 teams initially registering for the challenge and 19 teams handing in submissions for the three different challenge parts. This was achieved by providing benchmark datasets of unprecedented complexity and degree of realism, generated exclusively from around 700 newly sequenced microbial genomes and 600 novel viruses, plasmids and other circular elements. These spanned a range of evolutionary divergences from each other and from publicly available reference collections.
We implemented commonly used metrics in close collaboration with the computational and applied metagenomics communities and agreed on the metrics most important for common research questions and biological use cases in microbiome research using metagenomics. To be of practical value to researchers, the program submissions have to be reproducible, which requires knowledge of reference data, parameter settings and program versions. In CAMI, we have taken steps to ensure reproducibility by development of docker-based bioboxes  45  and encouraging developer submissions of bioboxes for the benchmarked metagenome analysis tools, enabling their standardized execution and format usages. The benchmark datasets, along with the CAMI benchmarking platform allow further result submissions and their evaluation on the challenge data sets, to facilitate benchmarking of further programs.
A sequence-derived reference phylogeny might represent a more suitable framework for -in that case -"phylogenetic" binning. When comparing the average performance of taxon binners for taxa with similar surroundings in the SILVA and NCBI reference taxonomies to those with less agreement, we observed a significant difference for taxa with discrepant surroundings, primarily a decreased performance for lower taxonomic ranks until family level (Supplementary Methods, Section 1. 4. 5; Supplementary Table  S24 ). Thus, the use of SILVA might further improve taxon binning, though the lack of associated genome sequence data represents a practical hurdle  46. Another challenge for all programs is the deconvolution of strain-level diversity, which we found to be substantially less effective than binning of genomes without close relatives present.
For the typically covariance of read coverage based genome binners it may require substantially larger numbers of replicate samples than those analyzed here (up to 5) to attain a satisfactory performance. Despite of a large variability in performance amongst the submitted profilers, most profilers performed well with good recall and low errors in abundance estimates until the family rank, with precision being the most variable of these metrics (Table  1 ). The use of different classification algorithms, reference taxonomies, reference databases and information sources (marker gene versus genome wide k-mer based) are likely contributors to the observed performance differences. To enable more systematic analyses of their individual impacts, software developers could provide configurable options for use of databases, k-mer sizes or other specialized settings, instead of having these hard-coded.
Similarly to taxonomic binners, performance across all metrics substantially decreased for the genus level and below. Also when taking plasmids and viruses into consideration for abundances estimates, the performance of all programs decreased substantially, indicating a need for further development to enable a better analysis of datasets with such content, as plasmids are likely to be present and viral particles are not always removed by size filtration  47. As both the sequencing technologies and the computational metagenomics programs continue to evolve rapidly, CAMI will provide further benchmarking challenges to the community. Long read technologies such as those by Oxford Nanopore, Illumina and PacBio  48  are expected to become more common in metagenomics, which will in turn require other assembly methods and may allow a better resolution of closely related genomes from metagenomes.
In the future, we also plan to tackle assessment of runtimes and RAM requirements, to determine program suitability for different use cases, such as execution on individual desktop machines or as part of computational metagenome pipelines provided by MG-RAST  49  , EMG  50  or IMG/M 51. We invite everyone interested to join and work with CAMI on providing comprehensive performance overviews of the computational metagenomics toolkit, to inform developers about current challenges in computational metagenomics and applied scientists of the most suitable software for their research questions.
Standardization and reproducibility For performance assessment, we developed several standards: we defined output formats for profiling and binning tools, for which no widely accepted standard existed. Secondly, standards for submitting the software itself, along with parameter settings and required databases were defined and implemented in docker container templates named bioboxes  45. These enable the standardized and reproducible execution of submitted programs from a particular category. Challenge participants were encouraged to submit the results together with their software in a docker container following the bioboxes standard. In addition to 23 bioboxes submitted by challenge participants, we generated 13 additional bioboxes and ran them on the challenge datasets (Supplementary Table  1 ), working with the developers to define the most suitable execution settings, if possible.
Genome sequencing and assembly Draft genomes of 310 type strain isolates were generated for the Genomic Encyclopedia of Type Strains at the DOE Joint Genome Institute (JGI) using Illumina standard shotgun libraries and the Illumina HiSeq 2000 platform. All general aspects of library construction and sequencing performed at the JGI can be found at http://www. jgi. doe. gov. Raw sequence data was passed through DUK, a filtering program developed at JGI, which removes known Illumina sequencing and library preparation artifacts [Mingkun L, Copeland A, Han J. DUK, unpublished, 2011]. The genome sequences of isolates from culture collections are available in the JGI genome portal (Supplementary Table  9 ). Additionally, 488 isolates from the root and rhizosphere of Arabidopsis thaliana were sequenced  9.
Challenge datasets We simulated three metagenome datasets of different organismal complexities and sizes by generating 150 bp paired-end reads with an Illumina HighSeq error profile from the genome sequences of 689 newly sequenced bacterial and archaeal isolates and 598 sequences of plasmids, viruses and other circular elements (Supplementary Methods "Metagenome simulation"; Supplementary Tables  3, 6 ; Supplementary Figs  D1, D2 ). These datasets represent common experimental setups and specifics of microbial communities. They consist of a 15 Gb single sample dataset from a low complexity community with log normal abundance distribution (40 genomes and 20 circular elements), a 40 Gb differential log normal abundance dataset with two samples of a medium complexity community (132 genomes and 100 circular elements) and long and short insert sizes, as well as a 75 Gb time series dataset with five samples from a high complexity community with correlated log normal abundance distributions (596 genomes and 478 circular elements).
Some important properties of the benchmark datasets are: All included species with strain-level diversity (Supplementary Fig. D1 ), to explore its' effect on program performances. They also included viruses, plasmids and other circular elements, to assess their impact on program performances. All datasets furthermore included genomes at different evolutionary distances to those in reference databases, to explore their effect on taxonomic binning. For every individual metagenome sample and for the pooled data set samples, gold standard assemblies, genome bin and taxon bin assignments, as well as taxonomic profiles were generated (available at https://data. cami-challenge. org/participate). The data generation pipeline is available on GitHub and as a docker container at https://hub. docker. com/r/cami/ emsep/.
Challenge Organization The first CAMI challenge benchmarked software for sequence assembly, taxonomic profiling and (taxonomic) binning. To allow developers to familiarize themselves with the data types, biobox-containers and in-and output formats, we provided simulated datasets from public data together with a standard of truth before the start of the challenge (Supplementary Figs  M1, M2 , https://data. cami-challenge. org/). Reference datasets of RefSeq, NCBI bacterial genomes, SILVA  53  , and the NCBI taxonomy from 04/30/2014 were prepared for taxonomic binning and profiling tools, to allow performance comparisons for reference-based tools based on the same reference datasets. For future benchmarking of reference-based programs with the challenge datasets, it will be important to use these reference datasets, as the challenge data have subsequently become part of public reference data collections.
The CAMI challenge started on 03/27/2015. Challenge participants had to register on the website for download of the challenge datasets, with 40 teams registered at that time. They could then submit their predictions for all datasets or individual samples thereof. Optionally, they could provide an executable biobox implementing their software together with specifications of parameter settings and reference databases used. Submissions of assembly results were accepted until 05/20/2015. Subsequently, a gold standard assembly was provided for all datasets and samples, which was suggested as input for taxonomic binning and profiling. This includes all genomic regions from the genome reference sequences and circular elements covered by at least one read in the pooled metagenome datasets or individual samples (Supplementary Methods, Section 1.