throughput data constitute an important area for study. The present paper showed that choices of error metric, classifier, error estimator and event balancing have large and compounding effects on statistical power. They can further be combined with inappropriate hypotheses to yield ineffective analysis proto- cols. An experimental comparison of data analysis protocols reveals that previous highly-cited claims that microarray assays may not be able to predict clinical outcomes better than chance are byproducts of data analysis limitations. Research designs of high-throughput studies will benefit by using the most powerful data analysis protocols available combined with appropriate statistical tests and doing so leads to substantial economies of required sample. New data analysis protocols should be tested for statistical efficiency before deploying for building molecular signatures. We recommend testing against existing protocols (such as one presented in this paper) in simulated or real data with known predictive signal using datasets in which the experimenter varies sample sizes [44,45]. Materials and Methods Microarray Datasets The characteristics of the human datasets analyzed [8,9,18–22] are summarized in Table 1. Error / prediction performance metric The area under the receiver operating characteristic (ROC) curve (AUC) is calculated by the formula provided in [46]. Proportion of misclassifications is calculated as the ratio: number of wrong classification divided by total number of classifications. Classifiers & gene selection Protocol I [23] involves selection of 50 genes with the highest correlation in training data with outcome variable according to Pearson’s correlation coefficient. Then molecular signatures are developed based on these genes using a nearest-centroid prediction method [47]. Protocol II uses the LibSVM implementation of Support Vector Machines (SVMs) to build molecular signatures with a fixed misclassification penalty parameter C = 100, and a linear kernel [48]. Gene selection is not employed to avoid increased computational costs. We note that SVMs have built-in regulari- zation however, which means that the learning algorithm penalizes large weights of predictors thus favoring simpler models by implicitly selecting genes, without using explicit gene selection procedures [49,50]. Power Molecular Signatures Statistical analysis Statistical significance of the molecular signatures in Protocol I replicates the procedure of [23]. Namely, 500 training datasets of size n are obtained by sampling without replacement the original dataset (of size N) such that each training set has n/2 subjects with each outcome. For each training set, the testing set is defined as its complement (of size N-n). The molecular signatures are then fitted on the training sets and their classification performance is assessed on the corresponding testing sets. The above procedure is repeated for different training set sizes ranging from 10 to a maximum value which was chosen so that the testing set has at least one subject representing each outcome. Given a distribution of classification performances for each training set size, the corresponding 95% intervals are constructed. The original dataset is considered to contain predictive signal if the upper 95% interval limit is less than 0.5 proportion of misclassifications. Notice that the published description of this method [23] does not explicitly state whether the above condition for significance should hold in at least one training set size n, or all possible training set sizes, or the majority of them. Thus we examine all three possibilities in the present work. For Protocol II, we use outcome value-permutation to test in each dataset the hypothesis of no predictive signal [12,13]. This is also known as a randomization test or a Monte-Carlo permutation test. We construct the distribution corresponding to the hypothesis by randomly permuting the values of the outcome variable (400 times) and then using SVMs (as described above) to compute the signature and repeated 10-fold cross- validation estimate of AUC for each permuted dataset. The repeated 10-fold cross-validation estimate from the original data is then compared to this distribution, and p-values correspond to the proportion of permuted estimators (under the hypothesis) that are more extreme than the repeated 10-fold cross-validation estimate from the original (non-outcome value permuted) data. Supporting Information File S1 Comparison of proportion of misclassifications with area under ROC curve (AUC) Found at: doi:10.1371/journal.pone.0004922.s001 (0.06 MB DOC) File S2 Demonstration of pitfalls of non-balanced data Found at: doi:10.1371/journal.pone.0004922.s002 (0.10 DOC) File S3 Details of simulation experiments Found at: doi:10.1371/journal.pone.0004922.s003 (0.08 DOC) MB MB Table 1. Characteristics of gene expression microarray datasets analyzed in this study. Dataset authors and reference Beer et al [18] Bhattacharjee et al [19] Iizuka et al [21] Pomeroy et al [22] Rosenwald et al [20] Veer et al [9] Yeoh et al [8] Sample size and number of events 86 (24 events) 62 (31 events) 60 (20 events) 60 (21 events) 240 (138 events) 97 (46 events) 233 (32 events) doi:10.1371/journal.pone.0004922.t001 7129 12600 7070 7129 7399 24188 12240 Number of variables (genes) Predicted event (outcome) Lung adenocarcinoma survival Lung adenocarcinoma 4-year survival Hepatocellular carcinoma 1-year recurrence-free survival Medulloblastoma survival Non-Hodgkin lymphoma survival Breast cancer 5-year metastasis-free survival Acute lymphocytic leukemia relapse-free survival PLoS ONE | www.plosone.org 6 March 2009 | Volume 4 | Issue 3 | e4922