Prespecified falsification end points: can they validate true observational associations?

AS OBSERVATIONAL STUDIES HAVE INCREASED IN NUMber—fueled by a boom in electronic recordkeeping and the ease with which observational analyses of large databases can be performed—so too have failures to confirm initial research findings. Several solutions to the problem of incorrect observational results have been suggested, emphasizing the importance of a record not only of significant findings but of all analyses conducted. An important and increasingly familiar type of observational study is the identification of rare adverse effects (defined by organizations such as the Council for International Organizations and Medical Sciences as occurring among fewer than 1 per 1000 individuals) from population data. Examples of these studies include whether macrolide antibiotics such as azithromycin are associated with higher rates of sudden cardiac death; whether proton pump inhibitors (PPIs) are associated with higher rates of pneumonia; or whether bisphosphonates are associated with an increased risk of atypical (subtrochanteric) femur fractures. Rare adverse events, such as these examples, occur so infrequently that almost by definition they may not be identified in randomized controlled trials (RCTs). Postmarketing data from thousands of patients are required to identify such low-frequency events. In fact, the ability to conduct postmarketing surveillance of large databases has been heralded as a vital step in ensuring the safe dissemination of medical treatments after clinical trials (phase 4) for precisely this reason. Few dispute the importance of observational studies for capturing rare adverse events. For instance, in early studies of whether bisphosphonate use increases the rate of atypical femur fractures, pooled analysis of RCTs demonstrated no elevated risk. However, these data were based on a limited sample of 14 000 patients with only 284 hip or femur fractures and only 12 atypical fracture events over just more than 3.5 years of follow-up. In contrast, later observational studies addressing the same question were able to leverage much larger and more comprehensive data. One analysis that examined 205 466 women who took bisphosphonates for an average of 4 years identified more than 10 000 hip or femur fractures and 716 atypical fractures. This analysis demonstrated an increased risk of atypical fractures associated with bisphosphonate use and was validated by another large population-based study. However, analyses in large data sets are not necessarily correct simply because they are larger. Control groups might not eliminate potential confounders, or many varying definitions of exposure to the agent may be tested (alternative thresholds for dose or duration of a drug)—a form of multiple-hypothesis testing. Just as small, true signals can be identified by these analyses, so too can small, erroneous associations. For instance, several observational studies have found an association between use of PPIs and development of pneumonia, and it is biologically plausible that elevated gastric pH may engender bacterial colonization. However, it is also possible that even after statistical adjustment for known comorbid conditions, PPI users may have other unobserved health characteristics (such as poor health literacy or adherence) that could increase their rates of pneumonia, apart from use of the drug. Alternatively, physicians who are more likely to prescribe PPIs to their patients also may be more likely to diagnose their patients with pneumonia in the appropriate clinical setting. Both mechanisms would suggest that the observational association between PPI use and pneumonia is confounded. In light of the increasing prevalence of such studies and their importance in shaping clinical decisions, it is important to know that the associations identified are true rather than spurious correlations. Prespecified falsification hypotheses may provide an intuitive and useful safeguard when observational data are used to find rare harms. A falsification hypothesis is a claim, distinct from the one being tested, that researchers believe is highly unlikely to be causally related to the intervention in question. For instance, a falsification hypothesis may be that PPI use increases the rate of soft tissue infection or myocardial infarction. A confirmed falsification test—in this case, a positive association between PPI use and risks of these conditions—