Reliability and Statistical Power: How Measurement Fallibility Affects Power and Required Sample Sizes for Several Parametric and Nonparametric Statistics

Duquesne University Ohio University The relationship between reliability and statistical power is considered, and tables that account for reduced reliability are presented. A series of Monte Carlo experiments were conducted to determine the effect of changes in reliability on parametric and nonparametric statistical methods, including the paired samples dependent t test, pooled-variance independent t test, one-way analysis of variance with three levels, Wilcoxon signed-rank test for paired samples, and Mann-Whitney-Wilcoxon test for independent groups. Power tables were created that illustrate the reduction in statistical power from decreased reliability for given sample sizes. Sample size tables were created to provide the approximate sample sizes required to achieve given levels of statistical power based for several levels of reliability. Key words: Pseudorandom generation, effect size, Monte Carlo simulations. Introduction Students of statistics usually become familiar with the factors that affect statistical power. For example, most students learn that sample size, level of significance, and estimated effect size all determine the a priori power of a statistical analysis. Some know that how effectively a particular design reduces error variance affects power, as does the directionality of the alternative hypothesis. However, many students do not realize that the reliability of measurements may also affect the statistical power (Hopkins & Hopkins, 1979). Light, Singer, and Willett (1990) provided tables to illustrate the point. Unfortunately, their tables provide only a very few situations and are therefore limited in their usefulness. It is not Gibbs Y. Kanyongo is an Assistant Professor of Education. He teaches educational statistics and research. Email: kanyongog@duq.edu. Gordon Brooks is an Associate Professor in the College of Education at Ohio University. He teaches educational statistics and research, and Monte Carlo simulations. He is also an author of several computer simulation programs. Lydia Kyei-Blankson and Gulsah Gocmen are graduate student in the College of Education. clear how the Light et al. tables were developed. The present study extends their tables and provides such information for additional statistical methods. Using the information provided in these tables, researchers can account for different levels of reliability as they determine sample sizes for their studies. Perhaps the converse approach is even more useful; however, that is, researchers might be encouraged to improve the reliability of their instruments in order to need fewer participants in their studies. These tables can also be useful tools in teaching students the relationship between reliability of a survey instrument and statistical power. Background One of the chief concerns of research design is to ensure that a study has adequate statistical power to detect meaningful differences, if indeed they exist. There is a very good reason researchers should worry about power a priori: If researchers are going to invest a great amount of money and time in carrying out a study, then they would certainly want to have a reasonable chance, perhaps 70% or 80%, to find a difference between groups if it does exist. Thus, a priori power (the probability of rejecting a null hypothesis that is false) will inform researchers how many subjects per group