PAGE 7 OF 14 Pilot CNV SNPs Pilot CNVs Pilot SNPs/CNV Analysis CNV SNPs Analysis CNVs Analysis SNPs/CNV Sim 1 CNV SNPs Sim 1 CNVs Sim 1 SNPs/CNV Sim 2 CNV SNPs Sim 2 CNVs Sim 2 SNPs/CNV CBS 1046 127 8.24 35 650 5680 6.28 24 249 1319 18.38 21 225 3587 5.92 1855 210 8.83 119 780 13 110 9.14 47 549 5084 9.35 21 077 2921 7.22 Table 4. The relationship between CNV SNPs and CNVs Nucleic Acids Research, 2010, Vol. 38, No. 9 e105 CNVFinder cnvPartitiona GLAD Nexus rank Nexus SNPRank PennCNV QuantiSNP 1060 64 16.56 18 520 1380 13.42 785 62 12.66 32 300 1520 21.25 24 370 1368 17.81 19 209 3144 6.11 273 311 42 468 6.44 3 179 600 544 410 5.84 28 178 1448 19.46 23 037 2320 9.93 7640 1405 5.44 43 370 1960 22.13 24 059 1202 20.02 20 564 3346 6.15 2055 405 5.07 29 560 6030 4.90 24 080 1266 19.02 21 066 3337 6.31 3694 369 10.01 70 190 29 560 8.22 24 825 1256 19.77 23 041 3696 6.23 aN/A is used here because cnvPartition was not evaluated in the simulations. optimal and suboptimal parameters for four DNA types in the pilot dataset. As with the training dataset, the Nexus algorithms overcalled CNV SNPs with as many as three (SNPRank) to 100 (Rank) times the number of calls of other methods (Table 2). The number of SNPs of the Nexus CNVs in the saliva data was small compared to the size of other methods’ CNVs, especially GLAD and cnvPartition, which were more than double the size of the Nexus CNVs. The pattern across DNA types was similar between Nexus and other methods. Ampliﬁed buccal DNA consistently showed the highest number of CNV SNPs (2032–225 597) across all methods in comparison to other DNA types (135–39 759). The excessive number of CNV calls in ampliﬁed buccal DNA caused bias of the ROC curve residuals, resulting in it having the best per- formance across all methods. The relationship between the number of CNV and CNV SNP calls in the pilot dataset is found in Table 4. If ampliﬁed buccal DNA is disregarded because of overcalling, the ROC residuals were best in blood-derived DNA samples in half of the methods. Overall, the performance of DNA from saliva, blood and buccal swabs was comparable in terms of the number of CNV SNP calls and the value of the ROC curve residuals, which is consistent with the observation of genotype call rates (data not shown). Method performance by analysis dataset at optimal parameters The Nexus Rank algorithm again overcalled CNV SNPs (31 796 per sample), and the cnvPartition program detected the least number of CNV SNPs (185 per sample) with at least 40% fewer calls than other methods (Supplementary Table S5). The segmentation methods CBS and GLAD had similar call numbers. The HMM-based PennCNV and QuantiSNP algorithms had divergent CNV SNP call detection numbers (average 295.6 vs. 701.9 CNV SNPs, respectively). Call numbers were correlated to sensitivity, speciﬁcity and kappa. Given this correlation, GLAD and CNVFinder did not perform as well as expected for sensitivity, speciﬁcity and kappa given their number of CNV SNP calls (Supplementary Table S5). CNVs from GLAD and SNPRank (21 and 22 SNPs long) were four times as big as PennCNV (5 SNPs long) on average. Rank and CBS CNVs were also small at an average of 6 SNPs per CNV. Further details on the relationships between CNVs and CNV SNPs are in Table 4. Nexus Rank, QuantiSNP and Nexus SNPRank were the top three programs on our DGV and HapMap Asian CNV databases (ROC residuals 0.0058, 0.0023 and 0.0017, respectively, on the HapMap Asian database) (Table 3, Figure 2). However, Rank falls to the bottom rank in the HapMap Conﬁrmed database. The high ROC curve residuals in the Nexus algorithms were primarily due to overcalling CNVs, which results in sensitivity increasing more quickly than 1-speciﬁcity (Table 3). Nexus Rank and CNVFinder had wide distri- butions of residuals, demonstrating a lot of sample- to-sample variation. The SD of Rank’s sensitivity and speciﬁcity distributions were approximately equivalent (0.045 and 0.04, respectively), indicating that the primary factor in the variance is the number of CNV calls per sample. When the total number of CNV calls was ﬁxed to the same level for these methods, we observed smaller ROC curve residuals for the Nexus programs than that of the QuantiSNP and CBS programs (data not shown). Method performance by simulation 1 at optimal parameters Ranking by ROC curve residual. Parameters other than those displayed in Supplementary Tables S4 and S6 were tested. Parameters displayed reﬂect ROC optimal param- eters for simulation 1, NSR optimized parameters or parameters with measurable diﬀerences from the optimal parameter. Methods ranked from ﬁrst to last using CNVs were as follows: QuantiSNP and Nexus SNPRank, PennCNV (average ROC residual = 0.707), PennCNV (0.703), CBS (0.675), GLAD (0.631), Nexus Rank (0.598) and CNVFinder (0.061) (Figure 4c; Supplementary Table S4). The interquartile ranges (IQRs) of the highest ranking methods: CBS, Nexus SNPRank, PennCNV and QuantiSNP have the highest ROC residuals and are overlapping (Figure 3a). These methods also displayed comparable sensitivity and speciﬁcity rates. Power and false positive rates diﬀered between the CNV and CNV SNP levels, and so ROC values and rankings also diﬀered (Figure 4; Supplementary Table S6).