BACKGROUND
since its inception, next-generation sequencing  techniques have transformed the way scientists extract multifaceted biological information from complex systems, fostering research in the fields of human disease, environmental science, evolutionary science, etc.
<cit>  the exploration of bacterial ecology by sequencing the small-subunit 16s rrna gene, or more commonly, portions thereof, is being used as a gold-standard technique for various human/environmental microbiome studies .
thus, the predictable conservation and variability of different portions of the prokaryotic 16s rrna gene have been exploited to provide high resolution identification and quantification of the bacterial source  <cit> .
this strategy for phylogenetic analysis and quantification has proven to be significantly more efficient than more traditional cloning and sequencing or rt-pcr based approaches.
these massively parallel, cost-effective, and high-throughput sequencing technologies can now produce up to 15gb of genomic data in one run  <cit> .
the sheer volume and complexity of data generated by these ngs systems necessitates the development of bioinformatics tools that streamline the downstream analyses.
the illumina platforms provide paired-end sequencing, in which a dna sequence is read from both ends up to a specified read length.
depending on the read length selected, currently up to  <dig> bases, a target dna segment that is longer than the sum of the forward and reverse reads would result in a gap of missing sequence between them, and a shorter target segment will result in an overlap between the reads.
moreover, since quality tends to degrade towards the ends of the reads, reliable merging of overlapping paired-end reads can results in a combined dna sequence that might permit bioinformatics correction of these 3′-end sequencing errors and yield higher quality sequence output.
in paired-end sequencing, accurate merging of the forward and reverse reads is a crucial first step that affects the results of a plethora of downstream analyses, especially, including but not limited to microbial taxonomic profiling.
various tools; e.g., shera  <cit> , flash  <cit> , pandaseq  <cit> , cope  <cit> , and others, have been developed for merging paired end data.
these tools generally apply sequence alignments to attempt to identify the best overlap between the paired-end reads.
thus, they resolve mismatching bases by considering their quality scores, with the higher quality base simply replacing one with poorer quality.
recently, a new method was proposed – casper, context-aware scheme for paired-end reads  <cit> .
casper uses the traditional quality-score based method to resolve mismatches, except when the difference in quality scores is not significant, relies on k-mer-based contexts surrounding the mismatch to make the decision.
another important aspect of sequence data that significantly impacts downstream analysis is read quality.
sequences of poor quality  need to be identified and removed prior to analysis to minimize sources of false positivity.
each ngs platform provides some quality control measures, and others have been developed for filtering of these sequencing artifacts .
most pipelines follow a qc protocol that filters reads based on their phred, or other equivalent quality  scores.
thus, reads of a specified length with greater than or equal to a specified, and often quite arbitrary average quality score are labeled as high-quality reads, and others with lower average quality scores are labeled low quality and removed from further analysis  <cit> .
however, phred scores are logarithmically related to the error probability of base-calling, which means that an average q-score of the read is not necessarily a good indicator of the expected accuracy/error within it  <cit> .
for example, consider two reads of length  <dig> nucleotides – read <dig> with 90 nt with q-score of  <dig> and 10 nt with q-score of  <dig> , and read <dig> with 100 nt with q-scores of  <dig> ).
however, the number of miscalls or maximum expected error  in read <dig> will be ~6; i.e.
<dig> bases each with a probability of being incorrect of  <dig> .
in contrast, the mee of read <dig> is  <dig>  .
thus, despite its lower average q-score, read <dig> has lower error probability, likely has many fewer sequence errors, and is for most purposes a higher quality read than read <dig>
the focus of research in our lab has been on applying high-throughput technologies to characterize the genome, transcriptome and proteome of the human microbiome and of the host-microbial interactions.
we are leading bioinformatics analysis on multiple human microbiome projects including the vaginal human microbiome project   <cit> , the multi-omics microbiome study: pregnancy initiatives   <cit> , oral and gut microbiome in neonates  <cit> .
each project involves the development of specialized computational methodologies and tools for analysis of large data sets generated by ngs sequencing on illumina platforms.
the accuracy of 16s rrna microbial profiling is highly dependent on the accuracy of pre-processing of sequencing data.
thus, we have developed a strategy entitled mefit  that efficiently merges overlapping paired-end sequence reads from the illumina miseq™ sequencing platform and quality filters them using the mee measure outlined above.
mefit invokes a version of casper  <cit>  for merging paired-end reads and extends it by including careful quality-filtering.
we provide users the option to quality filter the amplicons using the traditional average q-score metric or using a maximum expected error cut-off threshold.
we have also optimized mefit with the additional functionality of appending non-overlapping paired end reads; i.e., paired end reads from amplicons of greater than  <dig> bases which do not overlap are not discarded and are used to accurately determine the taxon from which they were derived.
most life science researchers lack an extensive it-infrastructure to support data analysis of the scale required for projects employing high throughput ngs technologies.
these investigators often cobble together a variety of software packages and pipelines to perform their analyses, often leading to a lack of quality control and computational efficiency.
herein, we describe our efforts to develop a fast and accurate in-house decision support tool for use as a first pre-processing step to obtain clean high-quality data for several ongoing projects generating microbiome profiles from targeting 16s rrna sequencing using paired end sequencing technology provided by the illumina miseq platform.
the mefit pipeline combines the essential first steps in analysis of paired-end sequencing data; i.e., merging of overlapping reads and quality filtering of the reads.
it is freely available  and can be easily and intuitively used by other groups dealing with similar challenges.
implementation
the mefit pipeline
the mefit pipeline performs following two steps:  merging of overlapping paired-end sequences, and  filtering data for quality .
the pipeline accepts the forward and reverse reads files in the fastq format and automates the merging and quality filtering of overlapping and non-overlapping  reads.
it also generates an extensive report on the quality and merging statistics.fig.
<dig> the mefit pipeline
merging of paired-end reads
mefit invokes a version of the casper algorithm  to merge the forward and reverse reads generated in paired-end sequencing.
briefly, the first step of the casper algorithm is to identify the best possible overlap region, with the least number of mismatching bases, between the forward and reverse reads.
any mismatches are then resolved by relying on the difference in their quality scores, with a lower quality base being replaced by the one with the higher score.
where the difference is not significant, casper makes partial decisions on k-mer-based contexts around the mismatch to make a final decision for resolving the mismatch.
the user, as needed, can modify the default casper parameters implemented into mefit.
non-overlapping reads
as outlined above, 2 ×  <dig> base pe reads will not overlap if the target amplicon is greater than  <dig> base pairs.
although targeted 16s amplicons are generally selected to be smaller than  <dig> base pairs, some bacterial taxa, including unknown taxa, occasionally exhibit target sequences exceeding the expected size.
most existing analysis packages tend to discard these sequences, resulting in the gross underrepresentation of some taxa.
mefit permits the user to specify to retain  non-overlapping reads by linking the forward and reverse reads with a user-specified patch of ns.
mefit reverse complements the reverse read and appends it to the 3′ end of the forward linked by a string of ns  with a assigned phred quality scores of  <dig>  which are subsequently ignored during the filtering step and calculation of quality statistics.
thus, taxa with unexpectedly large target amplicons are retained in the analysis.
quality filtering
as outlined briefly above, we were unsatisfied with the use of a simple average q score to identify good and bad reads.
thus, the program permits the user to select either the average read phred-quality score or q score, or the maximum expected error  as a percent of read length  .
the maximum expected error  within a read is the sum of error probabilities of each base, given by – p=10−q <dig>
readmee=∑pi
where, p is the error probability and q is the phred-quality score  of the i
th base of a read.
the meep-score is then computed from the mee of a read as the percentage of its read length.
thus, a meep cut-off of 1% permits a maximum error of  <dig> in a read of length 100 nt.
meep=readmee*100/readlength
RESULTS
with the advent and availability of illumina’s miseq system, capable of generating 2 ×  <dig> base paired-end reads and up to 15gb per run of  <dig> base-pair sequence information, it has become a go-to platform for targeted gene sequencing, metagenomics, small genome sequencing, etc.
accurate merging and quality control of raw miseq data is essential before any downstream analysis.
inaccurate merging or the presence of low-quality reads  result in an increase in the false negatives and false positives in subsequent analyses.
the mefit pipeline was developed to perform the two prerequisite steps – merging and quality filtering of the paired end reads, to obtain high-quality data prior to subsequent analyses.
as the first step of the pipeline, mefit uses casper to merge the forward and reverse reads.
as previously reported, casper, with its novel mismatch-resolving algorithm, showed a high level of accuracy when compared to cope, flash and pandaseq.
<cit>  although casper is not the fastest approach computationally, as expected from a k-mer based approach, the higher accuracy and robustness of the software compensates for this weakness.
one of the drawbacks of most current merging tools is that paired end reads that do not merge properly are not retained, possibly leading to the erroneous elimination of these reads from downstream analysis.
also, the target dna fragment may be longer than the sequenced read length resulting in a gap between the forward and reverse reads.
for example, in 16 s rrna-based taxonomic profiling, a targeted region of a known  bacterial taxon may exceed the  <dig> base length of the 2 ×  <dig> base paired end sequencing of the miseq platform.
such forward and reverse reads will not overlap, and discarding them as most analysis protocols do results in the loss of valuable information.
considering a typical paired-end sequencing experiment of 2 × 300 bp reads, one can sequence the 16s hypervariable regions up to ~540 bp in length .
our analysis of the lengths of hypervariable regions  of sequences in the silva database   <cit> , extracted using v-xtractor  <cit> , showed that a significant fraction of taxa have hypervariable regions that would likely result in amplicons of greater than 540 bp.
the most impacted taxa are the firmicutes and proteobacteria, which are abundant in gut microbiome samples .
a similar investigation of the stirrups database, which targets largely taxa of relevance to the female human urogenital tract, showed that ~1% of bacteria have v1–v <dig> regions in excess of 580 bp when the primer sequences are included  <cit> .
we believe that retaining such non-overlapping high-quality data will improve the accuracy of downstream phylogeny and taxonomy analysis.
the mefit pipeline addresses the above-mentioned shortcoming of other tools and provides users with the flexibility to append forward/reverse reads where no overlap was identified by casper.
the merged amplicons  are subjected to a quality control filter.
mefit invokes two alternative quality assessment metrics: the average q-score, or the meep-score.
amplicons that pass the user-defined filtering criteria  are saved as high quality in fastq format for further downstream analysis.
finally, mefit generates a detailed quality-statistics output for each processed sample that provides the total number of reads, the number of overlapping reads, the number of non-overlapping reads, the average amplicon length, the average q-score, the average meep-score in amplicons, the number of high-quality amplicons, the average read length in high-quality amplicons, the average q-score in high-quality amplicons, the average meep-score in high-quality amplicons, and the percent of overlapping reads in high-quality amplicons.
additional file 1: table s <dig> shows the detailed statistics report for  <dig> samples  processed through the mefit pipeline.
filtering amplicons for quality using an average q-score cutoff  retains > 99% reads for each sample, in contrast, filtering for quality using meep cutoff  only results in 83–0% high-quality amplicons.
this clearly indicates that quality filtering using average q-scores result in amplicons being labeled high-quality in spite of having higher error probabilities.
to demonstrate the utility of mefit, we generated a 2 × 300 bp simulated dataset from v1-v <dig> region of the 16s gene of six reference species: lactobacillus crispatus, l. iners, prevotella bivia, and gardnerella vaginalis, which have v1-v <dig> amplicons varying from 471–519 bp; and clostridium josui and campylobacter rectus which have amplicons of over 595 bp.
the simulated dataset, generated using grinder  <cit> , consisted of  <dig>  paired-end reads of 2 ×  <dig> read length configurations,  <dig>  from each of the six species.
sequencing errors were introduced using a modified 4th degree polynome illumina error model 3e-3 +  <dig> e- <dig> *i^ <dig>  <cit> .
mefit was run on this dataset, saving the non-overlapping reads appending them with a patch of  <dig> ns.
the merged amplicons were filtered using a meep cut-off of  <dig>  as seen in additional file 1: table s <dig>  paired-end reads simulated from species with shorter v1-v <dig> regions  result in overlapping amplicons, while those simulated from c. josui and campylobacter rectus are non-overlapping and the standard applications of most analyses would simply discard the reads from the two taxa with longer v1-v <dig> regions, resulting in a biased community profile.
otu clustering or taxonomic classification, on a set of only high-quality overlapping amplicons will result in a community profile composed of only the four species with shorter v1-v <dig> amplicons.
however, accurate identification and community abundance profiles are obtained if high quality non-overlapping amplicons are retained .
some of the common bioinformatics tools for microbiome analysis; including mothur  <cit> , qiime  <cit>  and usearch  <cit>  have their own implementations for merging paired-end reads.
the standard operating procedure for analyzing 16s rrna sequences generated using illumina’s miseq platform, demonstrated by mothur  <cit> , suggests using the ‘make.contigs’ command for preprocessing reads, and subsequently discards reads that cannot be assembled into contigs.
similarly, qiime and usearch  <cit>  have implementations that merge and quality filter overlapping paired-end reads.
however, these steps are not a part of the standard analysis pipelines, resulting in the need for users to incorporate these preprocessing steps in their custom pipelines to obtain high-quality data for analysis.
casper outperforms majority of merging tools in terms of sensitivity and specificity  <cit> .
characterizing a mock bacterial community  <cit>  using high-quality data obtained by mefit resulted in the identification of fewer spurious otus compared to that obtained by quality control steps implemented in mothur .
table  <dig> shows an overview of features of mefit compared to qiime, mothur and usearch.
we have developed a facile pipeline that takes sample-specific raw reads as input and provides a set of high-quality amplicons that can be directly fed into softwares/tools to perform otu analysis, taxonomic classification, functional profiling and other microbiome analysis.table  <dig> overview of features of mefit compared to other popular tools
CONCLUSIONS
the mefit pipeline combines open-source merging tool casper to a quality filtering step that automates the primary steps of analyzing overlapping paired-end sequencing data.
our pipeline reduces chances of human error due to minimal input requirements.
due to its simplified implementation and configuration, it can be easily incorporated into other analyses pipelines.
availability and requirements
project name: mefit
project home page: https://github.com/nisheth/mefit
operating system: unix-based platform
programming language: python v <dig>
other requirements: casper ; numpy ; htseq .
license: apache  <dig>
any restrictions to use by non-academics: apache  <dig>
additional files
additional file 1: supplementary methods, tables and figures.
additional file 2: otu-analysis commands.
abbreviations
caspercontext-aware scheme for paired-end reads
copeconnecting overlapped pair-end reads
dnadeoxyribonucleic acid
flashfast length adjustment of short reads
meemaximum expected error
mefitmerging and filtering tool
moms-pimulti-omic microbiome study-pregnancy initiative
ngsnext-generation sequencing
otuoperational taxonomic unit
pearpaired-end read merger
qiimequantitative insights into microbial ecology
rnaribonucleic acid
rt-pcrreverse transcriptase polymerase chain reaction
sherashortread error-reducing aligner
ssu rrnasmall subunit ribosomal rna
stirrupsspecies-level taxon identification of rdna reads using a usearch pipeline strategy
vahmpvaginal human microbiome project

 since its inception, next-generation sequencing techniques have transformed the way scientists extract multifaceted biological information from complex systems, fostering research in the fields of human disease, environmental science, evolutionary science, etc. the exploration of bacterial ecology by sequencing the small-subunit 16s rrna gene, or more commonly, portions thereof, is being used as a gold-standard technique for various human/environmental microbiome studies. thus, the predictable conservation and variability of different portions of the prokaryotic 16s rrna gene have been exploited to provide high resolution identification and quantification of the bacterial source. this strategy for phylogenetic analysis and quantification has proven to be significantly more efficient than more traditional cloning and sequencing or rt-pcr based approaches. these massively parallel, cost-effective, and high-throughput sequencing technologies can now produce up to 15gb of genomic data in one run. the sheer volume and complexity of data generated by these ngs systems necessitates the development of bioinformatics tools that streamline the downstream analyses. the illumina platforms provide paired-end sequencing, in which a dna sequence is read from both ends up to a specified read length. depending on the read length selected, currently up to bases, a target dna segment that is longer than the sum of the forward and reverse reads would result in a gap of missing sequence between them, and a shorter target segment will result in an overlap between the reads. moreover, since quality tends to degrade towards the ends of the reads, reliable merging of overlapping paired-end reads can results in a combined dna sequence that might permit bioinformatics correction of these 3′-end sequencing errors and yield higher quality sequence output. in paired-end sequencing, accurate merging of the forward and reverse reads is a crucial first step that affects the results of a plethora of downstream analyses, especially, including but not limited to microbial taxonomic profiling. various tools; e.g., shera, flash, pandaseq, cope, and others, have been developed for merging paired end data. these tools generally apply sequence alignments to attempt to identify the best overlap between the paired-end reads. thus, they resolve mismatching bases by considering their quality scores, with the higher quality base simply replacing one with poorer quality. recently, a new method was proposed – casper, context-aware scheme for paired-end reads. casper uses the traditional quality-score based method to resolve mismatches, except when the difference in quality scores is not significant, relies on k-mer-based contexts surrounding the mismatch to make the decision. another important aspect of sequence data that significantly impacts downstream analysis is read quality. sequences of poor quality need to be identified and removed prior to analysis to minimize sources of false positivity. each ngs platform provides some quality control measures, and others have been developed for filtering of these sequencing artifacts. most pipelines follow a qc protocol that filters reads based on their phred, or other equivalent quality scores. thus, reads of a specified length with greater than or equal to a specified, and often quite arbitrary average quality score are labeled as high-quality reads, and others with lower average quality scores are labeled low quality and removed from further analysis. however, phred scores are logarithmically related to the error probability of base-calling, which means that an average q-score of the read is not necessarily a good indicator of the expected accuracy/error within it. for example, consider two reads of length nucleotides – read with 90 nt with q-score of and 10 nt with q-score of, and read with 100 nt with q-scores of ). however, the number of miscalls or maximum expected error in read will be ~6; i.e. bases each with a probability of being incorrect of. in contrast, the mee of read is. thus, despite its lower average q-score, read has lower error probability, likely has many fewer sequence errors, and is for most purposes a higher quality read than read the focus of research in our lab has been on applying high-throughput technologies to characterize the genome, transcriptome and proteome of the human microbiome and of the host-microbial interactions. we are leading bioinformatics analysis on multiple human microbiome projects including the vaginal human microbiome project, the multi-omics microbiome study: pregnancy initiatives, oral and gut microbiome in neonates. each project involves the development of specialized computational methodologies and tools for analysis of large data sets generated by ngs sequencing on illumina platforms. the accuracy of 16s rrna microbial profiling is highly dependent on the accuracy of pre-processing of sequencing data. thus, we have developed a strategy entitled mefit that efficiently merges overlapping paired-end sequence reads from the illumina miseq™ sequencing platform and quality filters them using the mee measure outlined above. mefit invokes a version of casper for merging paired-end reads and extends it by including careful quality-filtering. we provide users the option to quality filter the amplicons using the traditional average q-score metric or using a maximum expected error cut-off threshold. we have also optimized mefit with the additional functionality of appending non-overlapping paired end reads; i.e., paired end reads from amplicons of greater than bases which do not overlap are not discarded and are used to accurately determine the taxon from which they were derived. most life science researchers lack an extensive it-infrastructure to support data analysis of the scale required for projects employing high throughput ngs technologies. these investigators often cobble together a variety of software packages and pipelines to perform their analyses, often leading to a lack of quality control and computational efficiency. herein, we describe our efforts to develop a fast and accurate in-house decision support tool for use as a first pre-processing step to obtain clean high-quality data for several ongoing projects generating microbiome profiles from targeting 16s rrna sequencing using paired end sequencing technology provided by the illumina miseq platform. the mefit pipeline combines the essential first steps in analysis of paired-end sequencing data; i.e., merging of overlapping reads and quality filtering of the reads. it is freely available and can be easily and intuitively used by other groups dealing with similar challenges. implementation the mefit pipeline the mefit pipeline performs following two steps: merging of overlapping paired-end sequences, and filtering data for quality. the pipeline accepts the forward and reverse reads files in the fastq format and automates the merging and quality filtering of overlapping and non-overlapping reads. it also generates an extensive report on the quality and merging statistics.fig. the mefit pipeline merging of paired-end reads mefit invokes a version of the casper algorithm to merge the forward and reverse reads generated in paired-end sequencing. briefly, the first step of the casper algorithm is to identify the best possible overlap region, with the least number of mismatching bases, between the forward and reverse reads. any mismatches are then resolved by relying on the difference in their quality scores, with a lower quality base being replaced by the one with the higher score. where the difference is not significant, casper makes partial decisions on k-mer-based contexts around the mismatch to make a final decision for resolving the mismatch. the user, as needed, can modify the default casper parameters implemented into mefit. non-overlapping reads as outlined above, 2 ×  base pe reads will not overlap if the target amplicon is greater than base pairs. although targeted 16s amplicons are generally selected to be smaller than base pairs, some bacterial taxa, including unknown taxa, occasionally exhibit target sequences exceeding the expected size. most existing analysis packages tend to discard these sequences, resulting in the gross underrepresentation of some taxa. mefit permits the user to specify to retain non-overlapping reads by linking the forward and reverse reads with a user-specified patch of ns. mefit reverse complements the reverse read and appends it to the 3′ end of the forward linked by a string of ns with a assigned phred quality scores of which are subsequently ignored during the filtering step and calculation of quality statistics. thus, taxa with unexpectedly large target amplicons are retained in the analysis. quality filtering as outlined briefly above, we were unsatisfied with the use of a simple average q score to identify good and bad reads. thus, the program permits the user to select either the average read phred-quality score or q score, or the maximum expected error as a percent of read length. the maximum expected error within a read is the sum of error probabilities of each base, given by – p=10−q readmee=∑pi where, p is the error probability and q is the phred-quality score of the i th base of a read. the meep-score is then computed from the mee of a read as the percentage of its read length. thus, a meep cut-off of 1% permits a maximum error of in a read of length 100 nt. meep=readmee*100/readlength  with the advent and availability of illumina’s miseq system, capable of generating 2 ×  base paired-end reads and up to 15gb per run of base-pair sequence information, it has become a go-to platform for targeted gene sequencing, metagenomics, small genome sequencing, etc. accurate merging and quality control of raw miseq data is essential before any downstream analysis. inaccurate merging or the presence of low-quality reads result in an increase in the false negatives and false positives in subsequent analyses. the mefit pipeline was developed to perform the two prerequisite steps – merging and quality filtering of the paired end reads, to obtain high-quality data prior to subsequent analyses. as the first step of the pipeline, mefit uses casper to merge the forward and reverse reads. as previously reported, casper, with its novel mismatch-resolving algorithm, showed a high level of accuracy when compared to cope, flash and pandaseq. although casper is not the fastest approach computationally, as expected from a k-mer based approach, the higher accuracy and robustness of the software compensates for this weakness. one of the drawbacks of most current merging tools is that paired end reads that do not merge properly are not retained, possibly leading to the erroneous elimination of these reads from downstream analysis. also, the target dna fragment may be longer than the sequenced read length resulting in a gap between the forward and reverse reads. for example, in 16 s rrna-based taxonomic profiling, a targeted region of a known bacterial taxon may exceed the base length of the 2 ×  base paired end sequencing of the miseq platform. such forward and reverse reads will not overlap, and discarding them as most analysis protocols do results in the loss of valuable information. considering a typical paired-end sequencing experiment of 2 × 300 bp reads, one can sequence the 16s hypervariable regions up to ~540 bp in length. our analysis of the lengths of hypervariable regions of sequences in the silva database, extracted using v-xtractor, showed that a significant fraction of taxa have hypervariable regions that would likely result in amplicons of greater than 540 bp. the most impacted taxa are the firmicutes and proteobacteria, which are abundant in gut microbiome samples. a similar investigation of the stirrups database, which targets largely taxa of relevance to the female human urogenital tract, showed that ~1% of bacteria have v1–v regions in excess of 580 bp when the primer sequences are included. we believe that retaining such non-overlapping high-quality data will improve the accuracy of downstream phylogeny and taxonomy analysis. the mefit pipeline addresses the above-mentioned shortcoming of other tools and provides users with the flexibility to append forward/reverse reads where no overlap was identified by casper. the merged amplicons are subjected to a quality control filter. mefit invokes two alternative quality assessment metrics: the average q-score, or the meep-score. amplicons that pass the user-defined filtering criteria are saved as high quality in fastq format for further downstream analysis. finally, mefit generates a detailed quality-statistics output for each processed sample that provides the total number of reads, the number of overlapping reads, the number of non-overlapping reads, the average amplicon length, the average q-score, the average meep-score in amplicons, the number of high-quality amplicons, the average read length in high-quality amplicons, the average q-score in high-quality amplicons, the average meep-score in high-quality amplicons, and the percent of overlapping reads in high-quality amplicons. additional file 1: table s shows the detailed statistics report for samples processed through the mefit pipeline. filtering amplicons for quality using an average q-score cutoff retains > 99% reads for each sample, in contrast, filtering for quality using meep cutoff only results in 83–0% high-quality amplicons. this clearly indicates that quality filtering using average q-scores result in amplicons being labeled high-quality in spite of having higher error probabilities. to demonstrate the utility of mefit, we generated a 2 × 300 bp simulated dataset from v1-v region of the 16s gene of six reference species: lactobacillus crispatus, l. iners, prevotella bivia, and gardnerella vaginalis, which have v1-v amplicons varying from 471–519 bp; and clostridium josui and campylobacter rectus which have amplicons of over 595 bp. the simulated dataset, generated using grinder, consisted of paired-end reads of 2 ×  read length configurations, from each of the six species. sequencing errors were introduced using a modified 4th degree polynome illumina error model 3e-3 +  e- *i^. mefit was run on this dataset, saving the non-overlapping reads appending them with a patch of ns. the merged amplicons were filtered using a meep cut-off of as seen in additional file 1: table s paired-end reads simulated from species with shorter v1-v regions result in overlapping amplicons, while those simulated from c. josui and campylobacter rectus are non-overlapping and the standard applications of most analyses would simply discard the reads from the two taxa with longer v1-v regions, resulting in a biased community profile. otu clustering or taxonomic classification, on a set of only high-quality overlapping amplicons will result in a community profile composed of only the four species with shorter v1-v amplicons. however, accurate identification and community abundance profiles are obtained if high quality non-overlapping amplicons are retained. some of the common bioinformatics tools for microbiome analysis; including mothur, qiime and usearch have their own implementations for merging paired-end reads. the standard operating procedure for analyzing 16s rrna sequences generated using illumina’s miseq platform, demonstrated by mothur, suggests using the ‘make.contigs’ command for preprocessing reads, and subsequently discards reads that cannot be assembled into contigs. similarly, qiime and usearch have implementations that merge and quality filter overlapping paired-end reads. however, these steps are not a part of the standard analysis pipelines, resulting in the need for users to incorporate these preprocessing steps in their custom pipelines to obtain high-quality data for analysis. casper outperforms majority of merging tools in terms of sensitivity and specificity. characterizing a mock bacterial community using high-quality data obtained by mefit resulted in the identification of fewer spurious otus compared to that obtained by quality control steps implemented in mothur. table  shows an overview of features of mefit compared to qiime, mothur and usearch. we have developed a facile pipeline that takes sample-specific raw reads as input and provides a set of high-quality amplicons that can be directly fed into softwares/tools to perform otu analysis, taxonomic classification, functional profiling and other microbiome analysis.table overview of features of mefit compared to other popular tools  the mefit pipeline combines open-source merging tool casper to a quality filtering step that automates the primary steps of analyzing overlapping paired-end sequencing data. our pipeline reduces chances of human error due to minimal input requirements. due to its simplified implementation and configuration, it can be easily incorporated into other analyses pipelines. availability and requirements project name: mefit project home page: https://github.com/nisheth/mefit operating system: unix-based platform programming language: python v other requirements: casper; numpy; htseq. license: apache any restrictions to use by non-academics: apache additional files additional file 1: supplementary methods, tables and figures. additional file 2: otu-analysis commands. abbreviations caspercontext-aware scheme for paired-end reads copeconnecting overlapped pair-end reads dnadeoxyribonucleic acid flashfast length adjustment of short reads meemaximum expected error mefitmerging and filtering tool moms-pimulti-omic microbiome study-pregnancy initiative ngsnext-generation sequencing otuoperational taxonomic unit pearpaired-end read merger qiimequantitative insights into microbial ecology rnaribonucleic acid rt-pcrreverse transcriptase polymerase chain reaction sherashortread error-reducing aligner ssu rrnasmall subunit ribosomal rna stirrupsspecies-level taxon identification of rdna reads using a usearch pipeline strategy vahmpvaginal human microbiome project 
for this analysis on the illumina platform, merging and quality filtering of paired-end reads are essential first steps in data analysis to ensure the accuracy and reliability of downstream analysis. mefit invokes casper for merging paired-end reads and provides users the option to quality filter the reads using the traditional average q-score metric or using a maximum expected error cut-off threshold. 16s rrnamicrobiomepaired-end sequencingquality filteringpre-processinghttp://dx.doi.org/ /100000002national institutes of health8u54hd080784buck gregory a. issue-copyright-statement© the author 2016 16s rrna amplicon sequencing using paired-end sequencing on the miseq platform from illumina, inc., is being used to characterize the composition and dynamics of extremely complex/diverse microbial communities.
 recent advances in next-generation sequencing have revolutionized genomic research. 16s rrna amplicon sequencing using paired-end sequencing on the miseq platform from illumina, inc., is being used to characterize the composition and dynamics of extremely complex/diverse microbial communities. for this analysis on the illumina platform, merging and quality filtering of paired-end reads are essential first steps in data analysis to ensure the accuracy and reliability of downstream analysis.  we have developed the merging and filtering tool to combine these pre-processing steps into one simple, intuitive pipeline. mefit invokes casper for merging paired-end reads and provides users the option to quality filter the reads using the traditional average q-score metric or using a maximum expected error cut-off threshold.  mefit provides an open-source solution that permits users to merge and filter paired end illumina reads. the tool has been implemented in python and the source-code is freely available at https://github.com/nisheth/mefit. electronic supplementary material the online version of this article contains supplementary material, which is available to authorized users. keywords 16s rrnamicrobiomepaired-end sequencingquality filteringpre-processinghttp://dx.doi.org/ /100000002national institutes of health8u54hd080784buck gregory a. issue-copyright-statement© the author 2016 