Seeing the Positive in Negative Studies

Identifying ineffective health interventions can be as important as identifying effective ones (1). Consequently, Annals is interested in publishing “negative” studies. We base our editorial decisions not on the “positivity” of findings (that is, their statistical significance) but on the importance of the research question and methodological quality. Unfortunately, few negative studies provide strong evidence for treatment ineffectiveness (2). To do so, they must have results statistically compatible only with a narrow range of small effects. If we can exclude sources of bias and agree that clinically important effects lie outside this range, we can conclude that the intervention is clinically ineffective. Although it is impossible to “prove” a null hypothesis of no effect, such studies provide strong evidence that any nonzero effect of the intervention is unlikely to be clinically meaningful. Many negative studies, however, are imprecise, and their results are compatible with a wide range of effects, thereby providing weak evidence that the intervention is clinically ineffective. Regrettably, imprecise results whose margin of error (represented by their CIs) include a null effect are often misinterpreted as providing strong evidence of ineffectiveness. In their article, Wang and colleagues (3) provide a good example of properly interpreting a negative study. These investigators conducted a randomized clinical trial (ACTIV-4C [COVID-19 Thrombosis Prevention Trials: Post-hospital Thromboprophylaxis]) to evaluate the effect of extended thromboprophylaxis in patients hospitalized for COVID-19 in 107 U.S. hospitals. Patients were randomly assigned to 2.5 mg of apixaban or placebo twice daily at hospital discharge and continued treatment for 30days. The primary end point was a composite of all-cause mortality, venous thrombosis, and arterial thrombosis within 30days of randomization. On the basis of the assumptions that 4% of patients in the placebo group would develop the primary end point and that apixaban would have an efficacy of 35%, the investigators estimated that they needed 5320 patients (2660 in each group) to reliably evaluate the efficacy of the intervention. ACTIV-4C began recruitment on 15 February 2021. The study’s data and safety monitoring board met on 15 June 2022 and noted that the rate of the primary end point was lower than anticipated and that the waning of COVID-19 hospitalizations in the United States would make achieving recruitment targets difficult. On 21 June 2022, study leadership and the National Heart, Lung, and Blood Institute decided to halt new enrollment while continuing treatment and observation of the 1217 patients enrolled through 23 June 2023. At completion of follow-up, only 27 participants experienced a primary end point (14 in the placebo group and 13 in the treatment group). The relative risk for the primary end point comparing thromboprophylaxis with placebo was 0.92 (95% CI, 0.44 to 1.95; P = 0.85). A simplistic interpretation of such negative findings is that thromboprophylaxis was ineffective. A proper interpretation would be more nuanced. The risk for allcause mortality, venous thrombosis, and/or arterial thrombosis after discharge was lower than anticipated. In the placebo group, the 30-day cumulative incidence of the combined end point was only 2.31%, including 5 cases of venous thromboembolism (0.82%) and 3 cases of arterial thromboembolism (0.49%). The risk for these end points in hospitalized patients with COVID-19 was similar to that in other medically ill hospitalized patients (4), and the potential absolute benefit of thromboprophylaxis in this setting is small. The lower than anticipated event rate and the inability to reach target sample size meant that the treatment effect was too imprecisely estimated to conclude that the treatment was clinically ineffective. Wang and colleagues stated that the results were imprecise and “highly compatible, with estimates of efficacy ranging from a 56% reduction to a 95% increase in risk for apixaban compared with placebo. ” Such descriptions should be routine when reporting negative studies (Table). The 95% CI represents the range of true effects consistent with the observed effect based only on random variability—that is, if the estimate is unbiased (2, 5). A wide CI around a null effect indicates that the data are statistically compatible with a wide range of effects—from harm to no effect to benefit. How narrow should the CI be to conclude that an intervention is clinically ineffective? That depends on the size of effects considered clinically important. Researchers should refrain from identifying findings as negative or positive. Instead, they should clearly state the range of effects compatible with the observed data and help the reader to determine if the results are compatible with clinically meaningful effects. Focusing on the clinical implications of confidence bounds further underscores the need for careful study design and correct variance estimation (6). This problem is hardly new (2, 7, 8), but the situation has been hard to change. In 1978, Freiman and colleagues (7) evaluated 71 published trials whose investigators concluded that the test intervention was not superior to control. Many of these studies had 95% CIs compatible with clinically important beneficial and