Power, Effect Size, and Practical Significance: How the Reporting in Journal of Visual Impairment & Blindness Articles Has Changed in the past 20 Years

Dae Shik Kim The topics of effect size and statistical power have been neglected in social science research for many decades despite their importance. Partly helped by the American Psychological Association’s (2001) call for reporting effect sizes, some journals have started to make efforts to require their authors to report or interpret effect sizes and indicate whether the selected study design allowed sufficient statistical power to detect a target effect size of the intervention. However, many journal editors and reviewers still seem to be neglecting this topic (Fritz, Morris, & Richler, 2012). This neglect is of serious concern, because the appropriate use of and interpretation of effect size, along with information on statistical power, is essential in determining practical (clinical) significance of any applied research findings (Fritz et al., 2012; Keselman et al., 1998). The term effect size can be defined as “the degree to which the phenomenon is present in the population” (Cohen, 1988, p. 9). In an a priori (prospective) statistical power analysis, effect size means “the magnitude of the real effect to be detected” (Lipsey, 1990, p. 14). On the other hand, when interpreting the result of a study, effect size (post hoc) indicates the “obtained” size of the impact produced by the treatment or intervention (Huck, 2008). As briefly indicated above, effect size can be used for two different purposes when conducting research. First, it can be used in an a priori statistical power analysis. Effect size in this context refers to a predetermined specification of what constitutes the smallest difference that researchers consider to be practically (clinically) meaningful (Huck, 2008). For example, researchers may consider a reading speed difference of 30 or more words per minute as practically meaningful, whereas a difference smaller than that may be seen as trivial and thus not noteworthy in a practical sense. In this case, an effect size of 30 words or more per minute can be used as a target effect size, which the study should be designed to detect. An a priori statistical power analysis allows researchers to calculate an adequate sample size to achieve a desired level of statistical power for a target effect size so as not to inflate type II error (failure to reject a null hypothesis when the null hypothesis is actually false). Second, measures of effect size can serve as yardsticks for determining the practical significance of the obtained result (Huck, 2008). Although a p-value indicates whether or not the obtained difference or correlation is statistically significant (whether or not we obtained the difference due to chance alone), an effect size of the result allows us to see whether the obtained difference (for instance, a difference of one point on a 0 to 100 scale in a history test between male and female freshman students) is meaningful for practitioners and policy makers (for instance, whether we need to implement an intervention program to narrow the gap in history competency between male and female students). A p-value alone is not sufficient when interpreting a study result because even an effect too small to be practically meaningful (as was the case in the example above) may turn out to be statistically significant if the sample size is very large. Effect sizes can be expressed in two forms: raw effect size and standardized effect size. Raw effect size is often a simple difference in scores between the two groups (for instance, the mean height of men [5 10 ] minus the mean height of women [5 6 ] or between the