Statistics for clinicians: 7: Sample size

Studies that are too small or too large both present problems in clinical investigation. It is intuitive to most readers that a study based on a small number of patients has little chance of producing clear-cut conclusions about potentially important differences within or between groups. Even if a difference is observed, the uncertainty around the result will be so wide that the study can be interpreted to support almost any clinical interpretation. Consider a randomized trial for the comparison of two drugs, Drug A and Drug B, in the treatment of a certain disorder. (As an aside, both drugs need not be ‘active’ – one could be a placebo, as is common in controlled clinical trials). The results of such a trial should be expressed in terms of a primary outcome measure, and the type of this outcome variable will determine the statistical analysis. The simplest possibilities are continuous and dichotomous outcomes. Previous articles in this series have discussed the standard statistical approaches to such analyses. In the fourth 1 and fifth 2 articles, we explained how the difference between two treatment groups can be presented with a confidence interval (CI), usually 95% CI, that reflects the amount of sampling variability or uncertainty associated with the estimated difference. A range of results is shown schematically in Fig. 1. In the first scenario, the results favour Drug A, but the difference is not statistically significant (the CI for the difference crosses zero) and it would be difficult to argue that these findings on their own provide convincing support for Drug A’s superiority. In the second scenario, the results favour Drug B, but again the difference is not statistically significant. In the third and fifth scenarios, the results favour Drug A, and the differences are statistically significant (CIs do not cross zero). However, the finding of statistical significance reflects quite different results in each case. In scenario 3, the size of the difference is larger than in scenario 1, but the width of its 95% CI is the same. In scenario 5, the size of the difference is the same as in scenario 1, but the width of the 95% CI is narrower. Scenarios 1 and 5 produced the same results, in terms of actual (average) difference between the groups, but the evidence from scenario 5 is more convincing: Why? The answer is that scenario 5 involved a larger study. In the fourth and sixth scenarios, the results favour Drug B, and although the differences are statistically significant in both cases, again the results tell a rather different story. Both scenarios produced the same ‘point estimate’ for the difference, but the evidence from scenario 6 (which must have had a larger sample size) is more definitive: the CI from scenario 4 suggests that the average advantage of Drug B could be quite small, as the upper end of the CI comes close to 0. The statistical analysis illustrated by these scenarios can be understood in terms either of precision of estimation of the difference between groups (using the CI), or hypothesis testing: is the observed difference large enough (relative to sampling variability) for us to conclude that it may reflect a non-zero true difference? We will see in the rest of this article that approaches to sample size estimation can be framed using either perspective. The hypothesis-testing perspective is the more familiar one, and it involves the concept of power (how likely is it that my study will reject the null hypothesis? ). The precision perspective is simpler, however, and is the only approach that is available when the study has a fundamentally descriptive purpose, rather than comparison between two groups. In the fourth article in the series 1 we explained that statistical significance may reflect either a large observed difference between groups or a smaller difference accompanied by less variation within the groups (small standard error of the difference), as shown by the ‘signal-to-noise ratio’: