BACKGROUND
complex microbial communities colonize and affect a variety of environments, including our own bodies.
next-generation sequencing of amplicons from a taxonomically informative gene  is useful for estimating the composition of microbial communities and has been widely applied in diverse environments.
evaluating and optimizing the accuracy of this technique requires a gold standard for which one knows the true composition of the community.
popular software packages for microbiome studies include qiime  <cit>  and mothur  <cit> .
the flow for most microbiome software is similar.
the amplicon sequences are clustered into operational taxonomic units —sequences with sufficient similarity to be considered as arising from the same organism in the initial community.
analysis can proceed at that level, associating clinical outcomes with the presence or absence of a given otu, calculating microbial alpha-diversity  of the community, or beta-diversity  between communities, with the otu as a marker.
researchers often proceed to a classification step to identify each otu as representing a given already-known organism in a shared reference database.
this process can connect the otu sequences to the larger body of microbiological research, converting associations into a deeper understanding of the members of the community and their capabilities.
even within a given analysis pipeline, there are a variety of settings to be selected: which otu generating strategy should be used; which clustering algorithm; which classifier and reference database?
using constructed mock-communities as a gold-standard allows for a detailed assessment of the effects of dna storage, extraction, pcr enzymes and primers, sequencing technique and classification software.
the dna extraction technique and pcr conditions dramatically affect accuracy of the technique more than sequencing platform, and in ways that are not easily addressed by software .
community composition can affect the reliability of the results  <cit>  and result in bias, with more complex communities particularly challenging  <cit> .
spiked in dna into real samples has been successfully employed to test beta-diversity measuring techniques  <cit> .
standardized mock communities have been created to facilitate future work in this productive area  <cit> .
in-silico data can serve as a gold standard as well, allowing uncultivated organisms and more complex communities to be considered, something not possible or practical with mock communities.
using in-silico simulations, early clustering algorithms were found to be overly stringent when generating otus  <cit> .
the different alignments produced by references databases affected the quality of the downstream results  <cit> .
average neighbor clustering algorithms performed better in otu generation  <cit> , with large differences in output between algorithms  <cit> .
the clostridiales order was identified as particularly challenging for software to properly cluster  <cit> .
in-silico data has been used to optimize the pcr primer selection process  <cit>  and identify misidentified sequences  <cit> .
despite all of this excellent work, it remains a challenge for a researcher performing a microbiome experiment, a reviewer critically evaluating a study for publication, or a reader considering the validity of the study to determine which pipeline, selected otu strategy, reference database and classification tactics are the best—or even adequate in accuracy and precision—to support the conclusions of the study.
in most papers, the standard methods described in the tutorials for the respective pipelines are used.
here, we developed a software package decard  to generate realistic synthetic datasets for which we have a known source of the sequences to be used as a gold standard when evaluating microbiome analysis software.
we used decard to synthesize in-silico communities that approximate those we observe in healthy human stool to test the colligative performance of different microbiome analysis pipelines and settings in an idealized setting of no novel organisms and perfect pcr and sequencing or limited simulated sequencing and pcr errors.
we performed in-silico pcr followed by simulated sequencing of the amplicons.
the resultant amplicons were classified with qiime, mothur and a pplacer-based  <cit>  classifier.
we compared the outputs of each classification method against the true origins of the amplicons.
we assessed for robustness, accuracy and resolution.
all experiments were done with simulated miseq and 454-style amplicons, with and without simulated sequencing errors.
unless specified, results were similar for  <dig> and miseq, with or without simulated sequencing errors.
RESULTS
synthetic community generation
we generated  <dig> communities with a composition  and diversity  similar to our estimates of normal stool.
we used data from the healthy-gut cohort of the human microbiome project and our own samples from healthy donors to estimate the composition of a typical gut microbiota and define mathematical parameters  suitable for the decard “generate target module” .
figure 1a shows the community profile of the real stool microbiome data as compared to the synthetic communities, demonstrating similar representations of clades between our synthetic and real data.
for diversity we used the approach suggested by  <cit> , calculating diversity scores across hill values of − <dig> to  <dig>  with results shown in fig.
1b.
as we intended, at the extremes of the hill value , our simulated populations had a higher diversity than the estimates from real data from healthy human stool .
in the core range of hill numbers from zero to one  our synthetic data closely matches that of the real data.fig.
<dig> estimated real versus synthetic health human stool microbiota.
a each column represents one sample.
each band represents one organism.
the height of each band of color is proportional to the relative abundance of each sequence type.
taxonomically similar organisms are closer in color.
colors are by phylum : blue and purple for firmucutes; orange for bacteroides; tan and pinks for proteobacteria.
estimated relative abundances from real data are on the left and underlined in purple for healthy donor human stool microbiota, blue for the human microbiome project samples; synthetic data is on the right, and underlined in green.
b the diversity of the each microbiota  in blue) for hill numbers varying from − <dig> to  <dig>  in  <dig>  intervals.
solid lines are the mean, and dashed lines span the 95% confidence interval after bootstrapping  <dig> iterations  for the mean
for each amplicon we know the true origin organism , with an associated full taxonomy.
otu generation
we then asked how well the various pipelines were at forming operational taxonomic units or otus.
each otu  is meant to represent an organism in the initial community, suitable for unit measures of community diversity, for correlation analysis and for classification to a named organism.
there are three broad strategies used to generate otus: closed otu generation strategies align to a reference set, and cluster all amplicons aligning to the same reference sequence.
de novo otu-generation uses pairwise clustering to assemble amplicons into groups—often with some sort of identity thresholding or difference metric.
subsampled  otu generation  <cit>  is a hybrid of the two techniques, starting with a closed strategy, and then taking all of the unmatched amplicons remaining and assembling them into otus via a de novo otu-generation process.
to test otu generation we took the amplicons generated from our  <dig> communities through qiime, mothur and a pplacer-based classification pipeline to generate otus.
for qiime, we attempted several different methods of otu generation available in that package.
mothur uses a unique approach, including dereplication, alignment to the silva reference database, further dereplication and finally clustering with uclust; we consider this a closed strategy, given the discarding of sequences that do not align to the silva reference.
the pplacer-based pipeline uses an open otu generation strategy via the swarm algorithm  <cit>  .
for each amplicon we know the true origin organism.
we can use this knowledge to ask if pairs of amplicons from the same organism are paired into otus by the classifier , or not .
similarly, for pairs of amplicons from different organisms we can ask if the pipeline correctly split these reads , or incorrectly matched them into otus .
these results  are suited to the familiar sensitivity  and specificity  metrics used to evaluate tests.
in this situation, sensitivity drops as incorrect splitting of amplicons increases.
conversely, specificity declines as amplicons are incorrectly matched by a pipeline.
figure  <dig> shows the distribution of sensitivity, specificity and percentage of amplicons dropped for the different pipelines, settings and strategies used for otu generation for miseq data, without  and with simulated error , respectively, as a set of box-and-whiskers plots.
not surprisingly, in the idealized circumstance of perfect sequencing and pcr, the rate of false splitting of amplicons from the same organism into different otus was rare to non-existent, resulting in most sensitivities at  <dig>  specificity also approached  <dig>  demonstrating that sequences from different organisms were only rarely lumped together.
while the differences between settings and communities were significant by a paired student’s t-test, the practical differences were slight.fig.
<dig> assessment of otu performance.
on the left are the various conditions tested.
the first column specifies the pipeline, the second the strategy, the third the methodological details .
abbreviations: gg is greengenes.
sub is subsetted otu generation.
a no sequencing error.
b simulated sequencing error
with the addition of simulated sequencing errors in fig.
2b, both the sensitivity  and specificity  worsen, but remain modest.
de-novo otu generation with uclust-based methods consistently performed more poorly than swarm-based methods, particularly as reflected by more incorrect splitting of amplicons during classification .
with and without simulated sequencing error, closed otu generation resulted in some dropped amplicons, a feature either non-existent or minimal in the sub or de novo otu generation strategies.
classification
classification is the process by which the clusters of amplicons generated in the otu step are taxonomically assigned .
all of these pipelines take consensus amplicon sequences from each otu, aligned against a set of  reference sequences; based on the alignment scores, names and taxonomies are selected for each otu.
differences between pipelines arise in the selection of reference set, in how the alignments are completed and judged, and in how ties or similarly scoring alignments are settled with different names or taxonomies.
all of the source amplicons on our synthetic dataset have a name  and a defined taxonomy.
for each true organism, we have a set of associated amplicons.
each of these amplicons can be: correctly classified ; under-called in the correct clade but not down to the desired rank; miscalled as a sibling, with the correct parent but wrong final identification ; overcalled down the right clade but overconfidently ; miscalled down the entirely wrong clade; or dropped, and lost at this or an earlier stage.
tables  <dig> and  <dig> summarize the performance of the pipelines using miseq data with simulated sequencing error, and targeting to species-level  or genus-level  resolution.
genus-level classification is correctly done for  <dig> – <dig> % of the source organisms, with qiime, de-novo otu generation and a curated subset of the silva  <dig> reference set  as the most successful strategy.
genus-level miscall rates varied from a low of  <dig> – <dig> %.
species-level classification was significantly less successful, ; when targeting species-level classification, miscall rates were comparable to those of genus-level targets .table  <dig> species level classification
summary of classification performance.
on the left are the various conditions tested.
the first column specifies the pipeline, the second the otu strategy, the third the methodological details .
table  <dig> is for species-level classification, table  <dig> is for genus-level.
source organisms can be correctly called, undercalled , or miscalled .
we present both the percentage in each category  and the median  taxonomic ranks off for underacalled and miscalled source organisms
summary of classification performance.
on the left are the various conditions tested.
the first column specifies the pipeline, the second the otu strategy, the third the methodological details .
table  <dig> is for species-level classification, table  <dig> is for genus-level.
source organisms can be correctly called, undercalled , or miscalled .
we present both the percentage in each category  and the median  taxonomic ranks off for underacalled and miscalled source organisms
table  <dig> shows the relative performance of all the pipelines  broken down by the order of the source organism.
the ability of pipelines to correctly resolve organisms varied by the clade of the organism, particularly when considering the magnitude of error .
among the orders heavily represented in a typical stool sample, all pipelines struggled when attempting to classify enterobacteriales and clostridiales; performance for bacteroidales was consistently stronger.table  <dig> classification outcomes by order for all pipelines
classification performance by order of source organism.
combined performance for all pipelines and settings, broken down by the order of the organism.
correct are correctly classified organisms.
miscalled are organisms that are classified into the wrong clade.
undercalled are organisms placed into the correct clade, but at the higher order than species
additional file 2: figure s <dig> shows the true as compared to estimated relative abundance from three randomly selected synthetic communities and subjectively demonstrates the integrated effects of both misestimating in otu generation and classification on complexity and composition of the community.
shannon index estimation
the shannon index  <cit>  is a commonly used metric for describing the alpha-diversity  of a community.
diversity is a key feature of microbial communities, and a meaningful way to compare communities.
as diversity is mostly used as a comparator between communities, what we wish is for our estimates to be monotonic with the true diversity.
to test how well each classifier estimates diversity, for each community we calculated a spearman’s correlation coefficient when comparing the true diversity of the community to that estimated for a given pipeline as a test of monotonicity.
monotonicity is allows for systematic under or overestimation of true diversity, but retains the ability to accurately compare communities—and thus is a realistic and meaningful means of evaluating the pipeline output.
figure  <dig> graphically shows the results as scatter plots for miseq data with simulated error.
the pplacer-based classifier achieved the best results with spearman’s r <dig> of  <dig> ; the poorest performance was from uclust-based de novo otu generation, with a spearman’s r <dig> of  <dig> .
overall, de novo otu generation via swarm resulted in significantly better results  than other methods .fig.
<dig> true versus estimated shannon diversity.
in each scatter plot, the x-axis is the true shannon diversity for a community, and the y-axis is the estimated for the given pipeline.
the top graph is true-versus-true for comparison in the others.
we used spearman’s correlations coefficients  to test for monotonicity  of the estimates to true
pairwise distance estimation
the pairwise distance between two communities is a frequently used beta-diversity metric employed in clustering, multidimensional scaling, principle component analysis and other methods to demonstrate the relationships between communities.
again, as a comparator, ideally the estimated pairwise distance between communities would be monotonic as compared to the true pairwise distance.
some means of calculating distance consider the relationships between organisms phylogenetically when weighting the differences in their abundance, such as unifrac  <cit>   and double principle coordinate analysis   <cit> .
the rationale is phylogenetically-related organisms contribute similar functions to communities and the functional similarity should be considered as part of a distance between communities.
weighted unifrac has become the dominant method in the field for pairwise distance measurement.
we used the spearman’s correlation coefficient to test the monotonicity between the true pairwise distance between communities and the estimated pairwise distance by the different pipelines.
figure  <dig> shows the results as a series of density plots for weighted unifrac and dpcoa.fig.
<dig> true versus estimated pairwise distance.
in each density plot, the x-axis is the true pairwise distance and the y-axis is the estimated pairwise distance between communities.
we used spearman’s correlations coefficients  to test for monotonicity  of the estimates to true.
the left column is pairwise distance as calculated by weighted unifrac distance.
the right column is pairwise distances as calculated by double principle coordinate analysis
qiime with closed otu generation against the green genes database  has a distinctive method for phylogeny generation.
as per the tutorial, one prunes the pre-made phylogenetic tree from greengenes  down to the leaves recruited in the classification step.
for the case of the pplacer-based pipeline, the recruited full-length 16s sequences are used to generate a de novo phylogeny.
the other methods construct a de novo phylogeny from the amplicon sequences.
the greengenes phylogeny performed distinctly and particularly poorly when compared to the true phylogenetic-based distance , regardless of distance metric .
for settings resulting in a spearman r <dig> around  <dig>  , dpcoa proved significantly more robust than weighted unifrac.
for setting resulting in a spearman r <dig> in the  <dig> ’s  weighted unifrac was significantly better as a technique.
discussion
amplicon-based approaches to describe complex microbial communities have theoretical limitations, including limited information available in some variable regions of taxonomically informative genes , and horizontal gene transfers scrambling the relationship between taxonomy and phylogeny.
with a careful selection of a proper computational pipeline and settings for the pipeline one can achieve results close to theoretical limits for a given community type.
a lack of close attention to these variables when selecting computational tools and settings can lead to skewed results.
constructed communities remain an invaluable tool for optimizing methods for dna storage, extraction, pcr and sequencing.
decard and other in-silico techniques to generate a gold standard are complementary, with an ability to objectively evaluate the computational aspects of amplicon-based microbiome studies.
in the current iteration, decard tests a relatively idealized circumstance in which there is no novel organism  in the communities.
decard cannot assess how pipelines handle novel organisms, nor is it ideal for testing pcr or sequencing errors.
even with these limits, for healthy human stool-like communities we discovered careful selection of reference sets, curation of reference sets and improved otu generation techniques can all improve the accuracy of results.
shannon for alpha-diversity proved quite robust with the more optimal settings .
for beta-diversity, dpcoa was superior to weighted unifrac when otu generation was less robust.
classification and taxonomic assignment to the species level remains a challenge for all of the pipelines, particularly in highly relevant orders like enterobacteriales and clostridiales.
we hypothesize the clade-dependent performance to be primarily related to phylogenetic and taxonomic  divergence in these clades—where the 16s sequence has less correlation with the overall function of the organism.
we were surprised at the significant challenges in classification.
in our preliminary studies, we used 16s ssu rrna exclusively from reference organisms or complete genomes to generate our synthetic reads without simulated pcr or sequencing errors; even in this very idealized circumstance, classification success was limited in a similar way to the data presented here.
we speculate duplicated, misannotated and imperfectly sequenced entries in reference databases contribute to classification errors.
further, an amplicon sequence can match multiple reference database entries with different taxonomic classifications, due to duplicated sequences and the amplicon region sequence being shared between distinct full-length sequences.
how a pipeline handles this ambiguity can affect the result quality.
we favor classifiers that reflect the ambiguity and offer higher rank classifications in this situation.
it’s imperative for reproducibility and interpretability of results that researchers include the specific method details in microbiome studies: the version of the software used; the specific otu-generation strategy  and details ; and the specific tactic used for classification and the version or date of the reference set selected.
we demonstrate here that seemingly minor differences in these details can have a meaningful and statistically significant impact on the validity of the outputs.
it is insufficient for good science to simply specify the software pipeline used.
nor is it sufficient to use the settings in the tutorials or standard operating procedures of a computational pipeline and assume the results will be optimal.
we demonstrate here that with some optimization of the settings selected, the amplicon-sequence based estimation of microbial communities remains a valuable technique.
but investigators should strive to optimize the reliability of their results and understand how the computational pipeline selected and specific settings chosen may influence results as they design and interpret experiments.
CONCLUSIONS
amplicon-based methods for describing complex microbial communities can be accurate and precise, but only with careful attention to settings and method details.
synthetic datasets and constructed communities will help researchers select these settings and details.
the methods and classification details must be included when microbiome studies are published to ensure reproducibility and validity.

 complex microbial communities colonize and affect a variety of environments, including our own bodies. next-generation sequencing of amplicons from a taxonomically informative gene is useful for estimating the composition of microbial communities and has been widely applied in diverse environments. evaluating and optimizing the accuracy of this technique requires a gold standard for which one knows the true composition of the community. popular software packages for microbiome studies include qiime and mothur. the flow for most microbiome software is similar. the amplicon sequences are clustered into operational taxonomic units —sequences with sufficient similarity to be considered as arising from the same organism in the initial community. analysis can proceed at that level, associating clinical outcomes with the presence or absence of a given otu, calculating microbial alpha-diversity of the community, or beta-diversity between communities, with the otu as a marker. researchers often proceed to a classification step to identify each otu as representing a given already-known organism in a shared reference database. this process can connect the otu sequences to the larger body of microbiological research, converting associations into a deeper understanding of the members of the community and their capabilities. even within a given analysis pipeline, there are a variety of settings to be selected: which otu generating strategy should be used; which clustering algorithm; which classifier and reference database? using constructed mock-communities as a gold-standard allows for a detailed assessment of the effects of dna storage, extraction, pcr enzymes and primers, sequencing technique and classification software. the dna extraction technique and pcr conditions dramatically affect accuracy of the technique more than sequencing platform, and in ways that are not easily addressed by software. community composition can affect the reliability of the results and result in bias, with more complex communities particularly challenging. spiked in dna into real samples has been successfully employed to test beta-diversity measuring techniques. standardized mock communities have been created to facilitate future work in this productive area. in-silico data can serve as a gold standard as well, allowing uncultivated organisms and more complex communities to be considered, something not possible or practical with mock communities. using in-silico simulations, early clustering algorithms were found to be overly stringent when generating otus. the different alignments produced by references databases affected the quality of the downstream results. average neighbor clustering algorithms performed better in otu generation, with large differences in output between algorithms. the clostridiales order was identified as particularly challenging for software to properly cluster. in-silico data has been used to optimize the pcr primer selection process and identify misidentified sequences. despite all of this excellent work, it remains a challenge for a researcher performing a microbiome experiment, a reviewer critically evaluating a study for publication, or a reader considering the validity of the study to determine which pipeline, selected otu strategy, reference database and classification tactics are the best—or even adequate in accuracy and precision—to support the conclusions of the study. in most papers, the standard methods described in the tutorials for the respective pipelines are used. here, we developed a software package decard to generate realistic synthetic datasets for which we have a known source of the sequences to be used as a gold standard when evaluating microbiome analysis software. we used decard to synthesize in-silico communities that approximate those we observe in healthy human stool to test the colligative performance of different microbiome analysis pipelines and settings in an idealized setting of no novel organisms and perfect pcr and sequencing or limited simulated sequencing and pcr errors. we performed in-silico pcr followed by simulated sequencing of the amplicons. the resultant amplicons were classified with qiime, mothur and a pplacer-based classifier. we compared the outputs of each classification method against the true origins of the amplicons. we assessed for robustness, accuracy and resolution. all experiments were done with simulated miseq and 454-style amplicons, with and without simulated sequencing errors. unless specified, results were similar for and miseq, with or without simulated sequencing errors.  synthetic community generation we generated communities with a composition and diversity similar to our estimates of normal stool. we used data from the healthy-gut cohort of the human microbiome project and our own samples from healthy donors to estimate the composition of a typical gut microbiota and define mathematical parameters suitable for the decard “generate target module”. figure 1a shows the community profile of the real stool microbiome data as compared to the synthetic communities, demonstrating similar representations of clades between our synthetic and real data. for diversity we used the approach suggested by, calculating diversity scores across hill values of − to with results shown in fig. 1b. as we intended, at the extremes of the hill value, our simulated populations had a higher diversity than the estimates from real data from healthy human stool. in the core range of hill numbers from zero to one our synthetic data closely matches that of the real data.fig. estimated real versus synthetic health human stool microbiota. a each column represents one sample. each band represents one organism. the height of each band of color is proportional to the relative abundance of each sequence type. taxonomically similar organisms are closer in color. colors are by phylum: blue and purple for firmucutes; orange for bacteroides; tan and pinks for proteobacteria. estimated relative abundances from real data are on the left and underlined in purple for healthy donor human stool microbiota, blue for the human microbiome project samples; synthetic data is on the right, and underlined in green. b the diversity of the each microbiota in blue) for hill numbers varying from − to in intervals. solid lines are the mean, and dashed lines span the 95% confidence interval after bootstrapping iterations for the mean for each amplicon we know the true origin organism, with an associated full taxonomy. otu generation we then asked how well the various pipelines were at forming operational taxonomic units or otus. each otu is meant to represent an organism in the initial community, suitable for unit measures of community diversity, for correlation analysis and for classification to a named organism. there are three broad strategies used to generate otus: closed otu generation strategies align to a reference set, and cluster all amplicons aligning to the same reference sequence. de novo otu-generation uses pairwise clustering to assemble amplicons into groups—often with some sort of identity thresholding or difference metric. subsampled otu generation is a hybrid of the two techniques, starting with a closed strategy, and then taking all of the unmatched amplicons remaining and assembling them into otus via a de novo otu-generation process. to test otu generation we took the amplicons generated from our communities through qiime, mothur and a pplacer-based classification pipeline to generate otus. for qiime, we attempted several different methods of otu generation available in that package. mothur uses a unique approach, including dereplication, alignment to the silva reference database, further dereplication and finally clustering with uclust; we consider this a closed strategy, given the discarding of sequences that do not align to the silva reference. the pplacer-based pipeline uses an open otu generation strategy via the swarm algorithm. for each amplicon we know the true origin organism. we can use this knowledge to ask if pairs of amplicons from the same organism are paired into otus by the classifier, or not. similarly, for pairs of amplicons from different organisms we can ask if the pipeline correctly split these reads, or incorrectly matched them into otus. these results are suited to the familiar sensitivity and specificity metrics used to evaluate tests. in this situation, sensitivity drops as incorrect splitting of amplicons increases. conversely, specificity declines as amplicons are incorrectly matched by a pipeline. figure  shows the distribution of sensitivity, specificity and percentage of amplicons dropped for the different pipelines, settings and strategies used for otu generation for miseq data, without and with simulated error, respectively, as a set of box-and-whiskers plots. not surprisingly, in the idealized circumstance of perfect sequencing and pcr, the rate of false splitting of amplicons from the same organism into different otus was rare to non-existent, resulting in most sensitivities at specificity also approached demonstrating that sequences from different organisms were only rarely lumped together. while the differences between settings and communities were significant by a paired student’s t-test, the practical differences were slight.fig. assessment of otu performance. on the left are the various conditions tested. the first column specifies the pipeline, the second the strategy, the third the methodological details. abbreviations: gg is greengenes. sub is subsetted otu generation. a no sequencing error. b simulated sequencing error with the addition of simulated sequencing errors in fig. 2b, both the sensitivity and specificity worsen, but remain modest. de-novo otu generation with uclust-based methods consistently performed more poorly than swarm-based methods, particularly as reflected by more incorrect splitting of amplicons during classification. with and without simulated sequencing error, closed otu generation resulted in some dropped amplicons, a feature either non-existent or minimal in the sub or de novo otu generation strategies. classification classification is the process by which the clusters of amplicons generated in the otu step are taxonomically assigned. all of these pipelines take consensus amplicon sequences from each otu, aligned against a set of reference sequences; based on the alignment scores, names and taxonomies are selected for each otu. differences between pipelines arise in the selection of reference set, in how the alignments are completed and judged, and in how ties or similarly scoring alignments are settled with different names or taxonomies. all of the source amplicons on our synthetic dataset have a name and a defined taxonomy. for each true organism, we have a set of associated amplicons. each of these amplicons can be: correctly classified; under-called in the correct clade but not down to the desired rank; miscalled as a sibling, with the correct parent but wrong final identification; overcalled down the right clade but overconfidently; miscalled down the entirely wrong clade; or dropped, and lost at this or an earlier stage. tables  and summarize the performance of the pipelines using miseq data with simulated sequencing error, and targeting to species-level or genus-level resolution. genus-level classification is correctly done for – % of the source organisms, with qiime, de-novo otu generation and a curated subset of the silva reference set as the most successful strategy. genus-level miscall rates varied from a low of – %. species-level classification was significantly less successful,; when targeting species-level classification, miscall rates were comparable to those of genus-level targets.table species level classification summary of classification performance. on the left are the various conditions tested. the first column specifies the pipeline, the second the otu strategy, the third the methodological details. table  is for species-level classification, table  is for genus-level. source organisms can be correctly called, undercalled, or miscalled. we present both the percentage in each category and the median taxonomic ranks off for underacalled and miscalled source organisms summary of classification performance. on the left are the various conditions tested. the first column specifies the pipeline, the second the otu strategy, the third the methodological details. table  is for species-level classification, table  is for genus-level. source organisms can be correctly called, undercalled, or miscalled. we present both the percentage in each category and the median taxonomic ranks off for underacalled and miscalled source organisms table  shows the relative performance of all the pipelines broken down by the order of the source organism. the ability of pipelines to correctly resolve organisms varied by the clade of the organism, particularly when considering the magnitude of error. among the orders heavily represented in a typical stool sample, all pipelines struggled when attempting to classify enterobacteriales and clostridiales; performance for bacteroidales was consistently stronger.table classification outcomes by order for all pipelines classification performance by order of source organism. combined performance for all pipelines and settings, broken down by the order of the organism. correct are correctly classified organisms. miscalled are organisms that are classified into the wrong clade. undercalled are organisms placed into the correct clade, but at the higher order than species additional file 2: figure s shows the true as compared to estimated relative abundance from three randomly selected synthetic communities and subjectively demonstrates the integrated effects of both misestimating in otu generation and classification on complexity and composition of the community. shannon index estimation the shannon index is a commonly used metric for describing the alpha-diversity of a community. diversity is a key feature of microbial communities, and a meaningful way to compare communities. as diversity is mostly used as a comparator between communities, what we wish is for our estimates to be monotonic with the true diversity. to test how well each classifier estimates diversity, for each community we calculated a spearman’s correlation coefficient when comparing the true diversity of the community to that estimated for a given pipeline as a test of monotonicity. monotonicity is allows for systematic under or overestimation of true diversity, but retains the ability to accurately compare communities—and thus is a realistic and meaningful means of evaluating the pipeline output. figure  graphically shows the results as scatter plots for miseq data with simulated error. the pplacer-based classifier achieved the best results with spearman’s r of; the poorest performance was from uclust-based de novo otu generation, with a spearman’s r of. overall, de novo otu generation via swarm resulted in significantly better results than other methods.fig. true versus estimated shannon diversity. in each scatter plot, the x-axis is the true shannon diversity for a community, and the y-axis is the estimated for the given pipeline. the top graph is true-versus-true for comparison in the others. we used spearman’s correlations coefficients to test for monotonicity of the estimates to true pairwise distance estimation the pairwise distance between two communities is a frequently used beta-diversity metric employed in clustering, multidimensional scaling, principle component analysis and other methods to demonstrate the relationships between communities. again, as a comparator, ideally the estimated pairwise distance between communities would be monotonic as compared to the true pairwise distance. some means of calculating distance consider the relationships between organisms phylogenetically when weighting the differences in their abundance, such as unifrac and double principle coordinate analysis. the rationale is phylogenetically-related organisms contribute similar functions to communities and the functional similarity should be considered as part of a distance between communities. weighted unifrac has become the dominant method in the field for pairwise distance measurement. we used the spearman’s correlation coefficient to test the monotonicity between the true pairwise distance between communities and the estimated pairwise distance by the different pipelines. figure  shows the results as a series of density plots for weighted unifrac and dpcoa.fig. true versus estimated pairwise distance. in each density plot, the x-axis is the true pairwise distance and the y-axis is the estimated pairwise distance between communities. we used spearman’s correlations coefficients to test for monotonicity of the estimates to true. the left column is pairwise distance as calculated by weighted unifrac distance. the right column is pairwise distances as calculated by double principle coordinate analysis qiime with closed otu generation against the green genes database has a distinctive method for phylogeny generation. as per the tutorial, one prunes the pre-made phylogenetic tree from greengenes down to the leaves recruited in the classification step. for the case of the pplacer-based pipeline, the recruited full-length 16s sequences are used to generate a de novo phylogeny. the other methods construct a de novo phylogeny from the amplicon sequences. the greengenes phylogeny performed distinctly and particularly poorly when compared to the true phylogenetic-based distance, regardless of distance metric. for settings resulting in a spearman r around, dpcoa proved significantly more robust than weighted unifrac. for setting resulting in a spearman r in the ’s weighted unifrac was significantly better as a technique. discussion amplicon-based approaches to describe complex microbial communities have theoretical limitations, including limited information available in some variable regions of taxonomically informative genes, and horizontal gene transfers scrambling the relationship between taxonomy and phylogeny. with a careful selection of a proper computational pipeline and settings for the pipeline one can achieve results close to theoretical limits for a given community type. a lack of close attention to these variables when selecting computational tools and settings can lead to skewed results. constructed communities remain an invaluable tool for optimizing methods for dna storage, extraction, pcr and sequencing. decard and other in-silico techniques to generate a gold standard are complementary, with an ability to objectively evaluate the computational aspects of amplicon-based microbiome studies. in the current iteration, decard tests a relatively idealized circumstance in which there is no novel organism in the communities. decard cannot assess how pipelines handle novel organisms, nor is it ideal for testing pcr or sequencing errors. even with these limits, for healthy human stool-like communities we discovered careful selection of reference sets, curation of reference sets and improved otu generation techniques can all improve the accuracy of results. shannon for alpha-diversity proved quite robust with the more optimal settings. for beta-diversity, dpcoa was superior to weighted unifrac when otu generation was less robust. classification and taxonomic assignment to the species level remains a challenge for all of the pipelines, particularly in highly relevant orders like enterobacteriales and clostridiales. we hypothesize the clade-dependent performance to be primarily related to phylogenetic and taxonomic divergence in these clades—where the 16s sequence has less correlation with the overall function of the organism. we were surprised at the significant challenges in classification. in our preliminary studies, we used 16s ssu rrna exclusively from reference organisms or complete genomes to generate our synthetic reads without simulated pcr or sequencing errors; even in this very idealized circumstance, classification success was limited in a similar way to the data presented here. we speculate duplicated, misannotated and imperfectly sequenced entries in reference databases contribute to classification errors. further, an amplicon sequence can match multiple reference database entries with different taxonomic classifications, due to duplicated sequences and the amplicon region sequence being shared between distinct full-length sequences. how a pipeline handles this ambiguity can affect the result quality. we favor classifiers that reflect the ambiguity and offer higher rank classifications in this situation. it’s imperative for reproducibility and interpretability of results that researchers include the specific method details in microbiome studies: the version of the software used; the specific otu-generation strategy and details; and the specific tactic used for classification and the version or date of the reference set selected. we demonstrate here that seemingly minor differences in these details can have a meaningful and statistically significant impact on the validity of the outputs. it is insufficient for good science to simply specify the software pipeline used. nor is it sufficient to use the settings in the tutorials or standard operating procedures of a computational pipeline and assume the results will be optimal. we demonstrate here that with some optimization of the settings selected, the amplicon-sequence based estimation of microbial communities remains a valuable technique. but investigators should strive to optimize the reliability of their results and understand how the computational pipeline selected and specific settings chosen may influence results as they design and interpret experiments.  amplicon-based methods for describing complex microbial communities can be accurate and precise, but only with careful attention to settings and method details. synthetic datasets and constructed communities will help researchers select these settings and details. the methods and classification details must be included when microbiome studies are published to ensure reproducibility and validity. 
beta-diversity estimates with double principle coordinate analysis were more robust against errors introduced in processing than weighted unifrac. even when using the same classification pipeline, the specific otu-generation strategy, reference database and downstream analysis methods selection can have a dramatic effect on the accuracy of taxonomic classification, and alpha- and beta-diversity estimation. our synthetic data closely matched the composition and complexity of actual healthy human stool microbiomes. the degree of miscall varied by clade of organism, pipeline and specific settings used. here we evaluate the accuracy of operational taxonomic unit generation, taxonomic classification, alpha- and beta-diversity measures for different settings in qiime, mothur and a pplacer-based classification pipeline, using a novel software package: decard. species-level classification was less successful,; miscall rates were comparable to those of genus-level targets. even minor changes in settings adversely affected the accuracy of the results, bringing them far from the best-observed result. miscall rates varied from to %. genus-level taxonomic classification was correctly done for only – % of the source organisms. in-silico we generated synthetic bacterial communities approximating human stool microbiomes to be used as a gold-standard for evaluating the colligative performance of microbiome analysis software. otu generation accuracy varied by strategy, reference database, algorithm and software implementation. microbiome studies commonly use 16s rrna gene amplicon sequencing to characterize microbial communities. thus, specific details of how a pipeline is used should be specified in the methods section of all microbiome studies.
 microbiome studies commonly use 16s rrna gene amplicon sequencing to characterize microbial communities. errors introduced at multiple steps in this process can affect the interpretation of the data. here we evaluate the accuracy of operational taxonomic unit generation, taxonomic classification, alpha- and beta-diversity measures for different settings in qiime, mothur and a pplacer-based classification pipeline, using a novel software package: decard.  in-silico we generated synthetic bacterial communities approximating human stool microbiomes to be used as a gold-standard for evaluating the colligative performance of microbiome analysis software. our synthetic data closely matched the composition and complexity of actual healthy human stool microbiomes. genus-level taxonomic classification was correctly done for only – % of the source organisms. miscall rates varied from to %. species-level classification was less successful,; miscall rates were comparable to those of genus-level targets. the degree of miscall varied by clade of organism, pipeline and specific settings used. otu generation accuracy varied by strategy, reference database, algorithm and software implementation. shannon diversity estimation accuracy correlated generally with otu-generation accuracy. beta-diversity estimates with double principle coordinate analysis were more robust against errors introduced in processing than weighted unifrac. the settings suggested in the tutorials were among the worst performing in all outcomes tested.  even when using the same classification pipeline, the specific otu-generation strategy, reference database and downstream analysis methods selection can have a dramatic effect on the accuracy of taxonomic classification, and alpha- and beta-diversity estimation. even minor changes in settings adversely affected the accuracy of the results, bringing them far from the best-observed result. thus, specific details of how a pipeline is used should be specified in the methods section of all microbiome studies. researchers should evaluate their chosen pipeline and settings to confirm it can adequately answer the research question rather than assuming the tutorial or standard-operating-procedure settings will be adequate or optimal. electronic supplementary material the online version of this article contains supplementary material, which is available to authorized users. keywords microbiomeclassificationoperational taxonomic unitoptimizationunifracqiimemothurissue-copyright-statement© the author 2017 