When Positive Isn't Positive: The Hopes and Disappointments of Clinical Trials

in bipolar disorder. While the trial published in this issue found a statistically significant but modest effect in favor of armodafinil 150 mg after the seventh week of an 8-week trial, the company’s 2 other trials (currently unpublished) failed to find a statistically significant effect—and Teva has abandoned its efforts to gain FDA approval for the bipolar depression indication. How then, should we interpret the results of this single study with regard to our clinical practice? Perhaps the biggest difficulty in interpreting studies is in understanding the relationship between the statistical power inherent in a given study and whether the results of that study can be trusted. We have come to rely on P values as the determination of whether treatments are effective or not: P .05, it’s not.3 No matter how often statisticians publish about this oversimplification and its dangers, this sense of a magic threshold persists. It is the power of the study, however, rather than the P value itself that determines whether the finding is likely to be a true finding. That is to say, an underpowered study is likely to result in both a negative finding (P > .05) that is actually true (false-negative) and a positive finding (P < .05) that is actually false (false-positive). Most findings in neuroscience are likely to be false-positive findings, as Button et al4 have pointed out, yet positive findings are the ones that are most likely to be published. It is rare that the a priori power calculations for a study are reported in publications.5 Instead, what tends to be reported are assumptions about power that were used to design the study and justify sample size—the predicted effect, its standard deviation, and the calculated number of subjects required to have 80% or 90% power—although in the case of this study, it is not reported at all. There is some controversy about the importance of post hoc power analyses, with some arguing that, for positive trials, they are not important at all, but the work of John Ioannidis6 and others4 has quite elegantly demonstrated that there is a strong relationship between lack of power and false-positive results. The post hoc power of this study, calculated from the reported results, appears to be somewhere between 40% and 68% and suggests that the study is, in fact, underpowered. This would mean that there is a high probability that the authors are falsely rejecting the null hypothesis, even when their P value is < .05. P values do not measure the size of the effect, whether the effect is large enough to be of importance to patients and treatment providers.3 For that, we rely on effect size or number needed to treat (NNT). The authors in this study calculated the effect size of the treatment and found a Cohen d therapeutic effect size of 0.28 in favor of modafinil 150 mg. This is, at best, a modest effect size, one that would Submitted: August 4, 2014; accepted August 4, 2014. Corresponding author: Michael J. Ostacher, MD, MPH, MMSC, Department of Psychiatry and Behavioral Sciences, Stanford University School of Medicine, 3801 Miranda Ave, Palo Alto, CA 94304 (ostacher@Stanford.edu).