BACKGROUND
targeted capture and sequencing of human exons  is an increasingly popular addition to genotyping microarrays and a lower cost alternative to whole genome sequencing for researchers investigating heritable traits  <cit> .
the complete human protein-coding exome comprises only 1% of the human genome, and it is typically in what would be considered the easier fraction to sequence and align uniquely to a reference genome.
most of the causal disease variants identified to date have been in protein-coding exonic sequence  <cit> , and there are well established experimental paradigms to explore the functional consequences of amino acid changing variants.
as the approach is sequence based, there is the potential to ascertain all simple sequence variants within the targeted regions in each member of a study cohort.
exome-seq has been successfully applied to identify causal variants in a number of mendelian genetic disorders  <cit> .
these study designs often require only tiny cohorts with little or no pedigree information, making many diseases tractable to genetic study for the first time.
exome-seq is also being used to study somatic mutations in cancers  <cit>  and has been proposed as a method to study complex traits where the ability to detect rare variants along with new variant discovery make it an ideal complement to association studies  <cit> .
causal variants for complex diseases previously identified through microarray-based association studies have been shown to localize in or near exonic sequence  <cit> , and human exomes contain an excess of rare non-synonymous coding variants that could explain some part of the missing heritability problem  <cit> .
for all re-sequencing projects , the total amount of usable sequence is considerably less than the amount that comes off the sequencing machines.
two standard analysis steps reduce the amount.
first, reads that cannot be uniquely mapped to the genome cannot be used.
a read may be unmappable because a) it contains too many sequencing errors, b) it is relatively error free but contains too many non-reference bases, c) it maps to an insertion, deletion, or structural variant, or d) it maps to multiple positions in the genome equally well.
second, duplicate reads are removed.
duplication is generally defined as having identical position in the genome, but may or may not include identical read sequence.
removing duplicate reads minimises issues with pcr amplification and improves the specificity of variant calling  <cit> .
exome sequencing suffers from the additional problem of off-target reads, those generated from the genomic sample but outside the regions targeted for capture  <cit> .
these reads can certainly be used for polymorphism discovery, but they are unlikely to map to protein-coding sequence or even regulatory features.
after applying all three of these reductions, the percentage of usable on-target reads can be as low as 35%  <cit> .
read mapping also demonstrates a bias towards the reference sequence: reads that contain too many alternate alleles align with lower scores to the reference genome than reads with only reference alleles  <cit> .
as a consequence of heterogeneity in target capture efficiency, amplification and the mapping of reads to the reference genome, usable reads are unevenly distributed over the targeted regions.
there has rightly been a great deal of focus on accurately measuring the specificity of single nucleotide variant  calls in next-generation sequence data .
but it is also hugely important for study design and the interpretation of results to understand the sensitivity attained.
how many snvs are likely to have been missed and where are they most likely to be located?
ajay and colleagues  <cit>  have gone some way to addressing this problem by showing that at the commonly used threshold of 30x coverage, 30% of protein coding regions have insufficient read depth to confidently call genotypes.
we set out to extend this observation to a wide range of read depths and compare the consistency of snv calling sensitivity between genomic regions.
to measure the sensitivity of snv detection we have applied down-sampling to deeply sequenced exomes and asked how well the down-sampled alignments perform at calling known snvs that are both present in the full alignment and genotyped by the hapmap phase iii project  <cit> .
this approach provides us with a gold-standard set of snvs that can be generated for any deeply sequenced human target and allows cross-comparison of platforms and laboratories without the need to repeatedly and deeply sequence a known reference.
with this gold-standard we have investigated the relationship of snv detection sensitivity to both average coverage over the entire targeted exome and per-nucleotide coverage for each variant.
calibrating snv detection sensitivity with per-nucleotide coverage allows us to estimate how many genuine snvs may have been missed in a specific targeted region and evaluate whether an exon or gene can be considered thoroughly screened for snvs in an individual.
it also provides the potential to correct the observed rate of polymorphism for biases in coverage between genomic regions.
this is crucial for the robust application of population based measures of mutation, selection and demography  <cit>  that are becoming increasingly important for the functional interpretation of non-coding regulatory sequences  <cit> .
the relationship between mean on-target read depth and snv sensitivity is important for study design where a researcher must generally decide a priori how to handle the trade-off between the number of individuals to study and the depth at which to sequence them.
to explore this relationship we have collected exome sequence data, generated at unusually deep coverage, from separate laboratories and utilising four different capture technologies.
our principal aim here is not a comprehensive comparison of competing exome capture technologies, such studies have been published elsewhere  <cit> , rather to understand real world variances in target capture efficiency and uniformity.
in the case of rare genetic disorders, the focus is on how much sequencing is required to confidently identify the majority of variants in the targeted regions of an individual.
we address the problem in the context of both autosomal dominant and recessive disorders, in which both heterozygous and homozygous mutations may be implicated.
in this work, we asked what level of mean on-target read depth is required to accurately identify a given percentage of snvs, where accuracy includes position, alleles, and genotype.
RESULTS
simulating shallower sequencing
thirty capture-targeted and sequenced human exomes, encompassing two different laboratories and four different capture methods, were aligned to the reference genome using a standard protocol .
twelve of these exomes are part of an ongoing disease study, subsequently denoted hw samples and the remaining  <dig> have been previously published  <cit> .
of the previously published exomes, eight were from individuals genotyped in the hapmap phase iii project  <cit> , .
the deep sequencing of these exomes enabled us to explore snv detection sensitivity by random down-sampling of reads to simulate shallower sequencing.
the down-sampled alignments were randomly generated from the original “full alignments” by including each read with a given probability .
as the exomes were captured using a mixture of technologies by different laboratories, we defined a common set of targets using ccds  <cit> .
snv detection sensitivity as a function of read depth at a polymorphic site
we called snvs on all full and down-sampled alignments using the genome analysis toolkit  unifiedgenotyper tool  <cit> .
snv calls were considered “true” if they were represented  in the set of polymorphic sites genotyped by the hapmap phase iii project  <cit> , located within the ccds target regions, and found in the full alignment for a given exome.
the cross-reference to known polymorphic sites allowed us to include all snv calls with a reasonable assumption of accuracy that did not require filtering on the variant quality score or read depth.
we validated our true set of snv calls for the eight hapmap exomes against the genotype calls from the hapmap project  <cit>  and found concordance of  <dig>  ±  <dig> % for heterozygous snvs and  <dig>  ±  <dig> % for homozygous snvs.
snv calls from the down-sampled alignments were labelled “positive” if they met the same criteria as the “true” snv calls in the full alignments.
we classified positive snv calls according to whether or not they were observed in the set of true snv calls for the exome, and whether or not the genotype of the calls matched.
positive snv calls observed in the true set with matching genotype were labelled true positives; those with mismatched genotype were labelled partial true positives.
partial true positives can be of two types.
the more common heterozygous true–homozygous positive snv calls occur when the down-sampled alignment has too few reference allele reads to call the position heterozygous.
homozygous true–heterozygous positive snv calls are the opposite case: when the down-sampled alignment has sampled by chance enough reads with the reference allele to call the position heterozygous, but given all the reads at that position, the variant calling algorithm identifies these as sequencing errors.
we measured sensitivity as a function of read depth at each true snv site  as described in methods.
within the subset of ccds targeted by all four capture methods, 95% detection sensitivity was reached for homozygous snvs at depths of ≥ 3x and for heterozygous snvs at depths of ≥ 13x.
at positions with 10x read depth, only  <dig> ± 3% of heterozygous snvs are correctly detected whereas the remaining 7–13% are typically called as non-polymorphic .
at these lower read depths it was also common to observe partial true positive calls where heterozygous sites were miscalled as homozygous for the non-reference allele.
our results show that empirical sensitivity is substantially worse than that predicted from the naive expectation of sampling from a binomial .
sensitivity in polymorphic sites used as our true set was higher than those identified in the full alignments within the shared ccds targets and not occurring in the hapmap phase iii set .
this is likely due to the higher rate of low quality variants in the latter set.
the application of a commonly used phred-scaled quality threshold of  <dig>  corresponding to an expected  <dig> % false call rate, to both the known and novel variants makes the two curves indistinguishable .
the rate of missed heterozygote calls at low read depth is therefore not an unusual feature of sites genotyped by the hapmap project, and polymorphisms are generally called in the full alignments with high precision.
no significant difference was observed between sensitivity of detecting homozygous snvs in regions considered difficult  to capture, sequence, and map, and snvs in general; however, heterozygous snvs in these regions were more easily detected at lower depths .
this counterintuitive observation may relate to the typically higher g+c content of difficult regions  and a consequent reduced bias for capturing the reference allele.
the number of hapmap phase iii snv sites in these difficult regions was relatively small , and none had read depth over 9x.
there was no difference between the four capture methods in measured snv detection sensitivity given the depth at a polymorphic site .
per-site on-target read depth is thus a good predictor of polymorphism detection sensitivity across target regions, between capture platforms and amongst laboratories.
using these results, sensitivity can be calculated across a gene or target set of interest to identify regions which are under-covered for the purpose of variant detection .
software to achieve this is provided .
snv detection sensitivity as a function of mean on-target read depth
using the sensitivity curves described in figure  <dig>  we calculated the total potential detection sensitivity of targeted regions for the  <dig> exomes.
because of differences in target probe design, we used the ccds exon definitions .
total detection sensitivity for an exome was defined as the sum across all positions in the target regions of the depth at that position multiplied by the mean snv detection sensitivity for that depth, calculated separately for both heterozygous and homozygous snvs .
we estimated the mean on-target read depth required for achieving varying levels of total potential recall on the ccds target set for the four capture methods separately and together .
using the complete set of ccds target regions highlights the differences in probe design between the four capture methods, while examining the subset of regions covered by at least one exome for each method individually  gives a very similar result to the set of regions targeted by all four methods .
hw = human genetics unit/wellcome trust sanger institute.
different capture technologies have varying levels of target uniformity and efficiency.
exomes captured by the two custom array-based methods  had the most uniform coverage, followed by the hw-nimblegen exomes .
other researchers have also observed higher uniformity of coverage in exomes captured by nimblegen array kits compared to agilent kits  <cit> .
this difference is likely due in part to the different probe design strategies taken by nimblegen and agilent, as nimblegen’s seqcap ez solution-based kit also generates more uniform coverage than agilent kits  <cit> .
the lower uniformity of read depth in the hw-agilent exomes explains the higher levels of mean on-target read depth required to achieve similar levels of snv detection sensitivity.
replicate samples
two of the hw samples were captured once by each of the two capture methods .
we compared the snv calls in the subset of of ccds targeted by all four capture methods from the full alignments of the replicates.
for snvs in the hapmap phase iii set where genotypes were called in both samples, concordance was >99% for both replicate pairs and for both heterozygous and homozygous genotypes.
for novel sites, concordance was >98% for heterozygous sites and >95% for homozygous sites.
the main difference between the replicates was in the number of novel snvs called from the hw-agilent samples that did not appear in the set of snvs called from the hw-nimblegen samples.
66% of novel snvs called from hw <dig>  were not called from hw <dig> ; likewise 54% from hw <dig>  were not called from hw <dig> .
this was not due to lack of read depth in the nimblegen samples at these sites ; in fact, the nimblegen samples had greater depth at these sites on average .
sample hw <dig>  which was captured and sequenced at the same time as samples hw <dig> and hw <dig>  also has roughly  <dig> x as many novel heterozygous variant calls as the other  <dig> samples .
this appears to be an experimental batch effect.
characteristics of “difficult” target regions
the target regions were split into maximum 100bp non-overlapping tiles and classified as “difficult” or “easy” to capture, sequence and map for each exome .
consistent with earlier reports  <cit> , tiles classified as difficult in at least five of the sequenced exomes had higher g+c content compared to those classified as easy in at least five .
difficult tiles were more likely to be annotated with simple repeats than easy tiles .
bainbridge and colleagues also noted that low complexity regions were less likely to be well-covered  <cit> .
this is likely due to a combination of the difficulty in uniquely mapping reads to these regions, and in designing probes for them.
across all  <dig> exomes, on average  <dig> ± 21% of difficult target region tiles were shared between any given pair of exomes, implicating technology-agnostic issues such as read mappability.
there was more variation among easy target region tiles, with only  <dig> ± 22% of such tiles shared on average across all exomes.
however, exomes captured by the same or similar method were more likely to have difficulty with certain target tiles .
CONCLUSIONS
we used down-sampling of deeply sequenced exomes along with cross-referencing to known snv sites to measure how read depth influences snv detection sensitivity.
the considerable per-nucleotide sequence depth required to achieve 95% sensitivity for heterozygous snvs highlights a substantial missing-data problem that is often overlooked in next-generation sequence analysis.
for example, ng and colleagues  <cit>  used 8x coverage of a site as a threshold to call polymorphisms, but at this coverage we find that 16% of heterozygous polymorphisms are missed by the standard genotype calling strategies .
raising the coverage threshold to 10x  reduces the expected number of missing variant calls, but they remain substantial.
the problem of missing heterozygote snvs stems from the stochastic and possibly also biased sampling of alleles by sequence reads  <cit> .
if the reference genome allele is more sampled, non-reference alleles may be either not represented or at an insufficient frequency to produce a confident  genotype call.
in contrast, if the non-reference allele is more abundantly sampled, information in the reference sequence is used to bolster the confidence of a reference allele call.
consequently the missing variant calls are specifically biased to non-reference alleles in the heterozygous state, precisely the alleles that are most likely to be of interest in population based  studies of rare disease and somatic cancer mutations.
we are not advocating the use of an excessively deep threshold to call polymorphisms; it makes sense to maximally use the available sequence information in an attempt to call variants even in regions of low sequence coverage.
however it is important to quantify how likely a polymorphism is to remain undetected.
this quantification can be achieved with the down-sampled alignment based calibrated data  that we have shown is portable between laboratories, platforms and agnostic to sequence quality score filtering.
this will allow genes of interest to be scored according how thoroughly they have been screened for variants and identify sub-regions that may warrant follow-up targeted sequencing to improve coverage.
we have demonstrated that mean on-target read depth of 17–37x is required to identify 90% of heterozygous snvs in the targeted regions for a given capture method, depending primarily on uniformity of read coverage.
heterogeneity in read coverage is inherent in all sequencing technologies but it is particularly pronounced in target capture approaches.
this unevenness of coverage is the main reason it requires every target nucleotide to be sequenced dozens of times, to ensure sufficient coverage over the difficult targets to achieve reasonable sensitivity.
some exons are persistently difficult to capture for multiple methods.
our sensitivity estimates are based on the set of snvs that can be discovered in the alignments with the maximum available data, so they naturally exclude some real snvs in these difficult target regions.
it is possible that a second round of capture, focusing on difficult exons, such as that carried out by ng and colleagues  <cit> , may be necessary in rare disease projects if no probable candidates are found.
rates of polymorphism and allele frequencies within regions of the genome or categories of annotated sites can be used to help measure aspects of demography, selection and mutation  <cit> .
but if there is a systematic bias in polymorphism detection sensitivity between regions of a genome being compared , these important measures will be distorted and undermined.
again using the single nucleotide resolution calibrated data  the average variant detection sensitivity of genomic regions or categories of sites could be compared and if necessary a correction applied in the case that sensitivity is not balanced between comparators.
despite the missing polymorphism problem, from an analysis perspective the most immediate challenge is more likely to be too many probable candidates, and prioritisation of candidate causal variations and genes will be required.
the rare genetic disorders that have been investigated using exome-seq so far have proved amenable to stringent filtering of known variation followed by damage prediction.
as technology advances and sequencing becomes cheaper, researchers will need new methods to handle larger data sets and the inevitable inclusion of previously observed variants as disease or trait contributing candidates.
with rapid advances in sequencing technology there will soon come a cross-over point where whole genome sequencing is cheaper than exome-seq.
the methods and missing variant-call problem described here apply not only to exome-seq but also to whole genome sequencing.
the capturing of smaller target sets but across many more patients, for example to validate a substantial genetic effect contributed by multiple rare variants at a locus  <cit> , is likely to be common in the future.
estimating the likelihood of missing heterozygous snvs in a sequenced sample will be an integral part of analyzing the results of all such studies.

 targeted capture and sequencing of human exons is an increasingly popular addition to genotyping microarrays and a lower cost alternative to whole genome sequencing for researchers investigating heritable traits. the complete human protein-coding exome comprises only 1% of the human genome, and it is typically in what would be considered the easier fraction to sequence and align uniquely to a reference genome. most of the causal disease variants identified to date have been in protein-coding exonic sequence, and there are well established experimental paradigms to explore the functional consequences of amino acid changing variants. as the approach is sequence based, there is the potential to ascertain all simple sequence variants within the targeted regions in each member of a study cohort. exome-seq has been successfully applied to identify causal variants in a number of mendelian genetic disorders. these study designs often require only tiny cohorts with little or no pedigree information, making many diseases tractable to genetic study for the first time. exome-seq is also being used to study somatic mutations in cancers and has been proposed as a method to study complex traits where the ability to detect rare variants along with new variant discovery make it an ideal complement to association studies. causal variants for complex diseases previously identified through microarray-based association studies have been shown to localize in or near exonic sequence, and human exomes contain an excess of rare non-synonymous coding variants that could explain some part of the missing heritability problem. for all re-sequencing projects, the total amount of usable sequence is considerably less than the amount that comes off the sequencing machines. two standard analysis steps reduce the amount. first, reads that cannot be uniquely mapped to the genome cannot be used. a read may be unmappable because a) it contains too many sequencing errors, b) it is relatively error free but contains too many non-reference bases, c) it maps to an insertion, deletion, or structural variant, or d) it maps to multiple positions in the genome equally well. second, duplicate reads are removed. duplication is generally defined as having identical position in the genome, but may or may not include identical read sequence. removing duplicate reads minimises issues with pcr amplification and improves the specificity of variant calling. exome sequencing suffers from the additional problem of off-target reads, those generated from the genomic sample but outside the regions targeted for capture. these reads can certainly be used for polymorphism discovery, but they are unlikely to map to protein-coding sequence or even regulatory features. after applying all three of these reductions, the percentage of usable on-target reads can be as low as 35%. read mapping also demonstrates a bias towards the reference sequence: reads that contain too many alternate alleles align with lower scores to the reference genome than reads with only reference alleles. as a consequence of heterogeneity in target capture efficiency, amplification and the mapping of reads to the reference genome, usable reads are unevenly distributed over the targeted regions. there has rightly been a great deal of focus on accurately measuring the specificity of single nucleotide variant calls in next-generation sequence data. but it is also hugely important for study design and the interpretation of results to understand the sensitivity attained. how many snvs are likely to have been missed and where are they most likely to be located? ajay and colleagues have gone some way to addressing this problem by showing that at the commonly used threshold of 30x coverage, 30% of protein coding regions have insufficient read depth to confidently call genotypes. we set out to extend this observation to a wide range of read depths and compare the consistency of snv calling sensitivity between genomic regions. to measure the sensitivity of snv detection we have applied down-sampling to deeply sequenced exomes and asked how well the down-sampled alignments perform at calling known snvs that are both present in the full alignment and genotyped by the hapmap phase iii project. this approach provides us with a gold-standard set of snvs that can be generated for any deeply sequenced human target and allows cross-comparison of platforms and laboratories without the need to repeatedly and deeply sequence a known reference. with this gold-standard we have investigated the relationship of snv detection sensitivity to both average coverage over the entire targeted exome and per-nucleotide coverage for each variant. calibrating snv detection sensitivity with per-nucleotide coverage allows us to estimate how many genuine snvs may have been missed in a specific targeted region and evaluate whether an exon or gene can be considered thoroughly screened for snvs in an individual. it also provides the potential to correct the observed rate of polymorphism for biases in coverage between genomic regions. this is crucial for the robust application of population based measures of mutation, selection and demography that are becoming increasingly important for the functional interpretation of non-coding regulatory sequences. the relationship between mean on-target read depth and snv sensitivity is important for study design where a researcher must generally decide a priori how to handle the trade-off between the number of individuals to study and the depth at which to sequence them. to explore this relationship we have collected exome sequence data, generated at unusually deep coverage, from separate laboratories and utilising four different capture technologies. our principal aim here is not a comprehensive comparison of competing exome capture technologies, such studies have been published elsewhere, rather to understand real world variances in target capture efficiency and uniformity. in the case of rare genetic disorders, the focus is on how much sequencing is required to confidently identify the majority of variants in the targeted regions of an individual. we address the problem in the context of both autosomal dominant and recessive disorders, in which both heterozygous and homozygous mutations may be implicated. in this work, we asked what level of mean on-target read depth is required to accurately identify a given percentage of snvs, where accuracy includes position, alleles, and genotype.  simulating shallower sequencing thirty capture-targeted and sequenced human exomes, encompassing two different laboratories and four different capture methods, were aligned to the reference genome using a standard protocol. twelve of these exomes are part of an ongoing disease study, subsequently denoted hw samples and the remaining have been previously published. of the previously published exomes, eight were from individuals genotyped in the hapmap phase iii project,. the deep sequencing of these exomes enabled us to explore snv detection sensitivity by random down-sampling of reads to simulate shallower sequencing. the down-sampled alignments were randomly generated from the original “full alignments” by including each read with a given probability. as the exomes were captured using a mixture of technologies by different laboratories, we defined a common set of targets using ccds. snv detection sensitivity as a function of read depth at a polymorphic site we called snvs on all full and down-sampled alignments using the genome analysis toolkit unifiedgenotyper tool. snv calls were considered “true” if they were represented in the set of polymorphic sites genotyped by the hapmap phase iii project, located within the ccds target regions, and found in the full alignment for a given exome. the cross-reference to known polymorphic sites allowed us to include all snv calls with a reasonable assumption of accuracy that did not require filtering on the variant quality score or read depth. we validated our true set of snv calls for the eight hapmap exomes against the genotype calls from the hapmap project and found concordance of ± % for heterozygous snvs and ± % for homozygous snvs. snv calls from the down-sampled alignments were labelled “positive” if they met the same criteria as the “true” snv calls in the full alignments. we classified positive snv calls according to whether or not they were observed in the set of true snv calls for the exome, and whether or not the genotype of the calls matched. positive snv calls observed in the true set with matching genotype were labelled true positives; those with mismatched genotype were labelled partial true positives. partial true positives can be of two types. the more common heterozygous true–homozygous positive snv calls occur when the down-sampled alignment has too few reference allele reads to call the position heterozygous. homozygous true–heterozygous positive snv calls are the opposite case: when the down-sampled alignment has sampled by chance enough reads with the reference allele to call the position heterozygous, but given all the reads at that position, the variant calling algorithm identifies these as sequencing errors. we measured sensitivity as a function of read depth at each true snv site as described in methods. within the subset of ccds targeted by all four capture methods, 95% detection sensitivity was reached for homozygous snvs at depths of ≥ 3x and for heterozygous snvs at depths of ≥ 13x. at positions with 10x read depth, only ± 3% of heterozygous snvs are correctly detected whereas the remaining 7–13% are typically called as non-polymorphic. at these lower read depths it was also common to observe partial true positive calls where heterozygous sites were miscalled as homozygous for the non-reference allele. our results show that empirical sensitivity is substantially worse than that predicted from the naive expectation of sampling from a binomial. sensitivity in polymorphic sites used as our true set was higher than those identified in the full alignments within the shared ccds targets and not occurring in the hapmap phase iii set. this is likely due to the higher rate of low quality variants in the latter set. the application of a commonly used phred-scaled quality threshold of corresponding to an expected % false call rate, to both the known and novel variants makes the two curves indistinguishable. the rate of missed heterozygote calls at low read depth is therefore not an unusual feature of sites genotyped by the hapmap project, and polymorphisms are generally called in the full alignments with high precision. no significant difference was observed between sensitivity of detecting homozygous snvs in regions considered difficult to capture, sequence, and map, and snvs in general; however, heterozygous snvs in these regions were more easily detected at lower depths. this counterintuitive observation may relate to the typically higher g+c content of difficult regions and a consequent reduced bias for capturing the reference allele. the number of hapmap phase iii snv sites in these difficult regions was relatively small, and none had read depth over 9x. there was no difference between the four capture methods in measured snv detection sensitivity given the depth at a polymorphic site. per-site on-target read depth is thus a good predictor of polymorphism detection sensitivity across target regions, between capture platforms and amongst laboratories. using these results, sensitivity can be calculated across a gene or target set of interest to identify regions which are under-covered for the purpose of variant detection. software to achieve this is provided. snv detection sensitivity as a function of mean on-target read depth using the sensitivity curves described in figure  we calculated the total potential detection sensitivity of targeted regions for the exomes. because of differences in target probe design, we used the ccds exon definitions. total detection sensitivity for an exome was defined as the sum across all positions in the target regions of the depth at that position multiplied by the mean snv detection sensitivity for that depth, calculated separately for both heterozygous and homozygous snvs. we estimated the mean on-target read depth required for achieving varying levels of total potential recall on the ccds target set for the four capture methods separately and together. using the complete set of ccds target regions highlights the differences in probe design between the four capture methods, while examining the subset of regions covered by at least one exome for each method individually gives a very similar result to the set of regions targeted by all four methods. hw = human genetics unit/wellcome trust sanger institute. different capture technologies have varying levels of target uniformity and efficiency. exomes captured by the two custom array-based methods had the most uniform coverage, followed by the hw-nimblegen exomes. other researchers have also observed higher uniformity of coverage in exomes captured by nimblegen array kits compared to agilent kits. this difference is likely due in part to the different probe design strategies taken by nimblegen and agilent, as nimblegen’s seqcap ez solution-based kit also generates more uniform coverage than agilent kits. the lower uniformity of read depth in the hw-agilent exomes explains the higher levels of mean on-target read depth required to achieve similar levels of snv detection sensitivity. replicate samples two of the hw samples were captured once by each of the two capture methods. we compared the snv calls in the subset of of ccds targeted by all four capture methods from the full alignments of the replicates. for snvs in the hapmap phase iii set where genotypes were called in both samples, concordance was >99% for both replicate pairs and for both heterozygous and homozygous genotypes. for novel sites, concordance was >98% for heterozygous sites and >95% for homozygous sites. the main difference between the replicates was in the number of novel snvs called from the hw-agilent samples that did not appear in the set of snvs called from the hw-nimblegen samples. 66% of novel snvs called from hw were not called from hw; likewise 54% from hw were not called from hw. this was not due to lack of read depth in the nimblegen samples at these sites; in fact, the nimblegen samples had greater depth at these sites on average. sample hw which was captured and sequenced at the same time as samples hw and hw also has roughly x as many novel heterozygous variant calls as the other samples. this appears to be an experimental batch effect. characteristics of “difficult” target regions the target regions were split into maximum 100bp non-overlapping tiles and classified as “difficult” or “easy” to capture, sequence and map for each exome. consistent with earlier reports, tiles classified as difficult in at least five of the sequenced exomes had higher g+c content compared to those classified as easy in at least five. difficult tiles were more likely to be annotated with simple repeats than easy tiles. bainbridge and colleagues also noted that low complexity regions were less likely to be well-covered. this is likely due to a combination of the difficulty in uniquely mapping reads to these regions, and in designing probes for them. across all exomes, on average ± 21% of difficult target region tiles were shared between any given pair of exomes, implicating technology-agnostic issues such as read mappability. there was more variation among easy target region tiles, with only ± 22% of such tiles shared on average across all exomes. however, exomes captured by the same or similar method were more likely to have difficulty with certain target tiles.  we used down-sampling of deeply sequenced exomes along with cross-referencing to known snv sites to measure how read depth influences snv detection sensitivity. the considerable per-nucleotide sequence depth required to achieve 95% sensitivity for heterozygous snvs highlights a substantial missing-data problem that is often overlooked in next-generation sequence analysis. for example, ng and colleagues used 8x coverage of a site as a threshold to call polymorphisms, but at this coverage we find that 16% of heterozygous polymorphisms are missed by the standard genotype calling strategies. raising the coverage threshold to 10x reduces the expected number of missing variant calls, but they remain substantial. the problem of missing heterozygote snvs stems from the stochastic and possibly also biased sampling of alleles by sequence reads. if the reference genome allele is more sampled, non-reference alleles may be either not represented or at an insufficient frequency to produce a confident genotype call. in contrast, if the non-reference allele is more abundantly sampled, information in the reference sequence is used to bolster the confidence of a reference allele call. consequently the missing variant calls are specifically biased to non-reference alleles in the heterozygous state, precisely the alleles that are most likely to be of interest in population based studies of rare disease and somatic cancer mutations. we are not advocating the use of an excessively deep threshold to call polymorphisms; it makes sense to maximally use the available sequence information in an attempt to call variants even in regions of low sequence coverage. however it is important to quantify how likely a polymorphism is to remain undetected. this quantification can be achieved with the down-sampled alignment based calibrated data that we have shown is portable between laboratories, platforms and agnostic to sequence quality score filtering. this will allow genes of interest to be scored according how thoroughly they have been screened for variants and identify sub-regions that may warrant follow-up targeted sequencing to improve coverage. we have demonstrated that mean on-target read depth of 17–37x is required to identify 90% of heterozygous snvs in the targeted regions for a given capture method, depending primarily on uniformity of read coverage. heterogeneity in read coverage is inherent in all sequencing technologies but it is particularly pronounced in target capture approaches. this unevenness of coverage is the main reason it requires every target nucleotide to be sequenced dozens of times, to ensure sufficient coverage over the difficult targets to achieve reasonable sensitivity. some exons are persistently difficult to capture for multiple methods. our sensitivity estimates are based on the set of snvs that can be discovered in the alignments with the maximum available data, so they naturally exclude some real snvs in these difficult target regions. it is possible that a second round of capture, focusing on difficult exons, such as that carried out by ng and colleagues, may be necessary in rare disease projects if no probable candidates are found. rates of polymorphism and allele frequencies within regions of the genome or categories of annotated sites can be used to help measure aspects of demography, selection and mutation. but if there is a systematic bias in polymorphism detection sensitivity between regions of a genome being compared, these important measures will be distorted and undermined. again using the single nucleotide resolution calibrated data the average variant detection sensitivity of genomic regions or categories of sites could be compared and if necessary a correction applied in the case that sensitivity is not balanced between comparators. despite the missing polymorphism problem, from an analysis perspective the most immediate challenge is more likely to be too many probable candidates, and prioritisation of candidate causal variations and genes will be required. the rare genetic disorders that have been investigated using exome-seq so far have proved amenable to stringent filtering of known variation followed by damage prediction. as technology advances and sequencing becomes cheaper, researchers will need new methods to handle larger data sets and the inevitable inclusion of previously observed variants as disease or trait contributing candidates. with rapid advances in sequencing technology there will soon come a cross-over point where whole genome sequencing is cheaper than exome-seq. the methods and missing variant-call problem described here apply not only to exome-seq but also to whole genome sequencing. the capturing of smaller target sets but across many more patients, for example to validate a substantial genetic effect contributed by multiple rare variants at a locus, is likely to be common in the future. estimating the likelihood of missing heterozygous snvs in a sequenced sample will be an integral part of analyzing the results of all such studies. 
the targeted capture and sequencing of genomic regions has rapidly demonstrated its utility in genetic studies. using down-sampling of deeply sequenced exomes and a set of gold-standard single nucleotide variant genotype calls for each sample, we developed an empirical model relating the read depth at a polymorphic site to the probability of calling the correct genotype at that site. we find that measured sensitivity in snv detection is substantially worse than that predicted from the naive expectation of sampling from a binomial. to fully interpret the polymorphisms identified in a genetic study it is often essential to both detect polymorphisms and to understand where and with what probability real polymorphisms may have been missed. non-reference alleles in the heterozygote state have a high chance of being missed when commonly applied read coverage thresholds are used despite the widely held assumption that there is good polymorphism detection at these coverage levels. such alleles are likely to be of functional importance in population based studies of rare diseases, somatic mutations in cancer and explaining the “missing heritability” of quantitative traits. at a mean on-target read depth of 20x, commonly used for rare disease exome sequencing studies, we predict 5–15% of heterozygous and 1–4% of homozygous snvs in the targeted regions will be missed.
 the targeted capture and sequencing of genomic regions has rapidly demonstrated its utility in genetic studies. inherent in this technology is considerable heterogeneity of target coverage and this is expected to systematically impact our sensitivity to detect genuine polymorphisms. to fully interpret the polymorphisms identified in a genetic study it is often essential to both detect polymorphisms and to understand where and with what probability real polymorphisms may have been missed.  using down-sampling of deeply sequenced exomes and a set of gold-standard single nucleotide variant genotype calls for each sample, we developed an empirical model relating the read depth at a polymorphic site to the probability of calling the correct genotype at that site. we find that measured sensitivity in snv detection is substantially worse than that predicted from the naive expectation of sampling from a binomial. this calibrated model allows us to produce single nucleotide resolution snv sensitivity estimates which can be merged to give summary sensitivity measures for any arbitrary partition of the target sequences. these metrics are directly comparable between platforms and can be combined between samples to give “power estimates” for an entire study. we estimate a local read depth of 13x is required to detect the alleles and genotype of a heterozygous snv 95% of the time, but only 3x for a homozygous snv. at a mean on-target read depth of 20x, commonly used for rare disease exome sequencing studies, we predict 5–15% of heterozygous and 1–4% of homozygous snvs in the targeted regions will be missed.  non-reference alleles in the heterozygote state have a high chance of being missed when commonly applied read coverage thresholds are used despite the widely held assumption that there is good polymorphism detection at these coverage levels. such alleles are likely to be of functional importance in population based studies of rare diseases, somatic mutations in cancer and explaining the “missing heritability” of quantitative traits. 