Factors Influencing the Statistical Power of Complex Data Analysis Protocols for Molecular Signature Development from Microarray Data Constantin F. Aliferis1,2,3*, Alexander Statnikov2, Ioannis Tsamardinos2,4, Jonathan S. Schildcrout3, Bryan E. Shepherd3, Frank E. Harrell Jr.3 1 Center of Health Informatics and Bioinformatics, New York University, New York, New York, United States of America, 2 Department of Biomedical Informatics, Vanderbilt University, Nashville, Tennessee, United States of America, 3 Department of Biostatistics, Vanderbilt University, Nashville, Tennessee, United States of America, 4 Department of Computer Science, University of Crete, Iraklio, Greece Abstract Background: Critical to the development of molecular signatures from microarray and other high-throughput data is testing the statistical significance of the produced signature in order to ensure its statistical reproducibility. While current best practices emphasize sufficiently powered univariate tests of differential expression, little is known about the factors that affect the statistical power of complex multivariate analysis protocols for high-dimensional molecular signature development. Methodology/Principal Findings: We show that choices of specific components of the analysis (i.e., error metric, classifier, error estimator and event balancing) have large and compounding effects on statistical power. The effects are demonstrated empirically by an analysis of 7 of the largest microarray cancer outcome prediction datasets and supplementary simulations, and by contrasting them to prior analyses of the same data. Conclusions/Significance: The findings of the present study have two important practical implications: First, high- throughput studies by avoiding under-powered data analysis protocols, can achieve substantial economies in sample required to demonstrate statistical significance of predictive signal. Factors that affect power are identified and studied. Much less sample than previously thought may be sufficient for exploratory studies as long as these factors are taken into consideration when designing and executing the analysis. Second, previous highly-cited claims that microarray assays may not be able to predict disease outcomes better than chance are shown by our experiments to be due to under-powered data analysis combined with inappropriate statistical tests. Citation: Aliferis CF, Statnikov A, Tsamardinos I, Schildcrout JS, Shepherd BE, et al. (2009) Factors Influencing the Statistical Power of Complex Data Analysis Protocols for Molecular Signature Development from Microarray Data. PLoS ONE 4(3): e4922. doi:10.1371/journal.pone.0004922 Editor: Vladimir B. Bajic, University of the Western Cape, South Africa Received July 18, 2008; Accepted February 5, 2009; Published March 17, 2009 Copyright: 2009 Aliferis et al. This is an open-access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited. Funding: The work was in part supported by grant 2R56LM007948-04A1. The funding agencies had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript. Competing Interests: The authors have declared that no competing interests exist. * E-mail: constantin.aliferis@nyumc.org Introduction Microarrays and other high-throughput assaying technologies have generated immense opportunities for discovery spanning the spectrum from basic research to clinical studies [1–3]. As the field moves from simpler analyses (e.g., differential expression of single genes and clustering) to more complex analyses such as developing multivariate molecular signatures in supervised fashion, the interpretation of microarray data involves multifaceted analysis protocols with many sophisticated and interacting analytic steps [4]. Developing molecular signatures in particular, is playing an increasingly important role in a variety of research design objectives both in basic and translational studies. Such objectives include, for example, detecting complex and coordinated patterns of transcriptional response to chemotherapeutic agents on cell lines and predicting subsequent patient treatment response on the basis of this information [5], discovery of new drug targets [6], discovery of biomarkers [7], subtyping diseases [8] and personalizing treatments [9]. The reproducibility of gene expression microarrays across laboratories for individual gene expression measurements and the ability to differentiate between disease subtypes are well established in recent studies [2,10,11]. Essential to developing molecular signatures is not only assay reproducibility however, but also statistical reproducibility. The latter can be directly assessed by tests of statistical significance of the produced signatures. These tests are usually permutation based and were introduced in bioinformatics by [12,13] based on foundational works of [14,15]. Although substantial efforts have been invested in studying the statistical power of differential gene expression [16,17], much less is known currently about the power of testing molecular signatures for statistically significant (hence reproducible, ‘‘real’’) signal. The present work shows that four specific components of data analysis (error metric, error estimator, classifier, and event balancing) have PLoS ONE | www.plosone.org 1 March 2009 | Volume 4 | Issue 3 | e4922