The Use of Predicted Confidence Intervals When Planning Experiments and the Misuse of Power When Interpreting Results

The increasing statistical sophistication of medical researchers has heightened their awareness of the importance of having appropriate statistical power in a clinical experiment. Power is the probability that, given a specified true difference between two groups, the quantitative results of a study will be deemed statistically significant. Freiman and colleagues [1] sensitized researchers to the possibility that many so-called negative trials of medical interventions might have too few participants to produce statistically significant findings even if clinically important effects actually existed. These authors calculated the statistical power of 71 negative studies that compared two treatments. They showed that for most of the studies, the power to detect a 50% improvement in success rates was quite low. This kind of analysis has been applied in various specialty fields, with similar results [2-6]. Studies with low statistical power have sample sizes that are too small, producing results that have high statistical variability (low precision). Confidence intervals are a convenient way to express that variability. Numerous articles, commentaries, and editorials [7-13] have appeared in the biomedical literature during the past decade showing how confidence intervals are informative data summaries that can be used in addition to or instead of P values in reporting statistical results. Although there is a growing understanding of the role of power in designing studies and the role of confidence intervals in study interpretation, confusion exists about the reverse arrangement, the role of confidence intervals in designing studies and of power estimates in interpreting study findings. Confidence intervals should play an important role when setting sample size, and power should play no role once the data have been collected, but exactly the opposite is widely practiced. When no significant difference between two compared treatments is reported, a common question posed in journal clubs, on rounds, in letters to the editor, and in reviews of manuscripts is, What was the power of the study to detect the observed difference? A widely used instrument for assessing the quality of a clinical trial penalizes studies if post hoc power calculations of this type are omitted [14], and reviews of the literature are frequently critical of the failure to include them [15, 16]. Although several writers [12, 17-19] have pointed out the error implicit in the concept of post hoc power, such caveats have not had great impact. This is perhaps because the error implicit in post hoc power is created by a subtle inconsistency in the logic of standard statistical methods. This inconsistency derives from applying a pre-experiment probability of a hypothetical group of results to the one result that is observed. Most discussions of this problem focus on purported errors in interpreting statistical rules, rather than on the problems with the statistical framework itself. We elucidate the problems with the framework in the course of explaining why post hoc estimates of power are of little help in interpreting results and why the focus of attention should be exclusively on confidence intervals. In addition, we show how the size of a confidence interval can be predicted in the planning stages of an experiment and how that can be a great help in understanding the implications of different sample size choices. Confidence Intervals A confidence interval can be thought of as the set of true but unknown differences that are statistically compatible with the observed difference. The standard convention for this statistical compatibility is the two-sided 95% confidence interval. A confidence interval is typically reported in the following way: There was a 10% (95% CI, 2% to 18%) difference in mortality. This means that even though the observed mortality difference was 10% (for example, 70% compared with 60%), the data are statistically compatible with a true mortality difference as small as 2% or as large as 18%. True differences that lie outside the 95% confidence interval are not impossible; they merely have less statistical evidence supporting them than values within it. The choice of 95% as the standard convention is somewhat arbitrary and corresponds to the use of a threshold of P 0.05 for statistical significance. However, a confidence interval should not be treated simply as a surrogate for a significance test, that is, to declare nonsignificance when the null hypothesis is included within it and to declare significance when it is not [20]. The location and width of the confidence interval have important information [9, 11]. Before the Experiment: The Proper Use of Statistical Power The meaning of power before an experiment is straightforward. Suppose we are evaluating a medical treatment that has a 45% cure rate. A surgical procedure is proposed but because of the surgical morbidity, we judge that it would have to achieve a 70% cure rate (25 percentage points better than medical therapy) to justify its use. If we are designing a clinical trial to compare the two therapies, and the treatment difference was that big, 80 participants would have to be randomized to each therapy to produce a study with a 90% chance of achieving a statistically significant difference. This 90% chance is the power of the study with respect to an underlying cure rate difference of 25%. The jargon is that we have 90% power to detect a 25% difference. Conventionally, power should be no lower than 80% and preferably around 90%, akin to a diagnostic test with a sensitivity of 80% to 90%. For any given underlying difference, a larger sample size produces greater power. Although the role of power in the planning stage of an experiment may be fairly clear, it comes with baggage that can lead to confusion when interpreting the results of that experiment. Implicit in its use is the assumption that we will report the results of a study only as statistically significant or not statistically significant, without further detail. Although that is how medical research results are sometimes simplistically summarized, that is rarely the only information provided in a research report. Researchers typically report at least the size of the observed effect (for example, an average blood pressure decrease of 10 mm Hg) and the precise P value (for example, P = 0.03, not just P < 0.05). As we show in the next section, once that information is provided, neither power nor its concomitant notion of detecting a difference remains a meaningful concept. Power is exclusively a pretrial concept; it is the probability of a group of possible results (namely, all statistically significant outcomes) under a specified alternative hypothesis. A study produces only one result. We discuss the methods that must be adopted after that result has been observed. After the Experiment: The Improper Use of Statistical Power The perspective after the experiment differs from that before the experiment simply because the result is known. That may seem obvious, but what is less apparent is that we cannot cross back over the divide and use pre-experiment numbers to interpret the result. That would be like trying to convince someone that buying a lottery ticket was foolish (the before-experiment perspective) after they hit a lottery jackpot (the after-experiment perspective). We will show the problem with combining those perspectives by a close examination of the concept of detection. This concept is derived from the field of signal detection, whose purposes differ somewhat from those in medical research. In the signal detection model, we are concerned with deciding whether there is a message within random noise; it is either there or it is not. Results fall into one of only two possible categories: A signal is detected or a signal is not detected. The parallel with the hypothesis test paradigm of statistics is direct; we report that either a statistically significant result is observed (that is, the observed difference is distinguishable from random variation and the null hypothesis is rejected) or a statistically nonsignificant result is observed (that is, the observed difference cannot be distinguished from random variation and the null hypothesis is accepted). Problems are produced when we apply this paradigm to medical research. We will continue with the previous example, in which there was 90% power to detect a 25% difference between two treatments, with n = 80 in each of the two groups. Suppose we observed a cure rate of 60/80 (75%) in one group and of 49/80 (61%) in the other, a difference of 14%. This has a P value of 0.06, not quite statistically significant at = 0.05. There are two ways we can report this result, one in which the notion of detection makes some sense, and one in which it does not. We could say simply that a statistically nonsignificant result was observed (P > 0.05) without reporting the observed 14% difference or the exact P value of 0.06. With this information, one would conclude that the true difference was probably less than 25% because there was a 90% power chance of achieving statistical significance (that is, detecting a difference) if a 25% true difference existed. Conversely, if statistical significance (P < 0.05) had been achieved, we would claim that the null hypothesis was unlikely because there was only a 5% chance of observing such a result if the null hypothesis were true. Thus, all the components of the sample size calculationthe detectable difference, the notion of detection, power, and remain relevant for the interpretation of the result if we reduce it to just one of two possible outcomes. However, this relevance comes at a steep price: It means that our degree of confidence in the no difference conclusion is the same for every nonsignificant result, and our confidence in a some difference conclusion is the same for every significant result. This price is not one that most sci