Comment on: “Desideratum for Evidence-Based Epidemiology”

We agree with Overhage et al. [1] from the Observational Medical Outcomes Partnership (OMOP) that better reproducibility is needed in pharmacoepidemiology. In addition to epidemiology in general [2], the need for better reproducibility has been highlighted across many scientific fields including psychology [3], economics [4], laboratory-based biologic research [5, 6], genomics [7], computational science [8], clinical trials [9, 10], and meta-analyses of clinical trials [11]. OMOP’s goal of rigorously evaluating the approaches and databases used in pharmacoepidemiology is therefore important and laudable. However, we have three major concerns about OMOP’s recent attempts to achieve these goals. Our first major concern is that most of the ‘base truths’ against which OMOP assessed pharmacoepidemiologic methods may not actually be true. To wit, 48 % of the ‘positive control’ drug–health outcome of interest (HOI) pairs examined by OMOP were based solely on case reports or case series [12]—hardly a sound basis for evaluating the validity of controlled epidemiologic studies. An additional 19 % of the positive controls had nonrandomized epidemiologic studies as their basis [12]. Of course, validating a method against the results of studies that used the same method is circular. In contrast, by OMOP’s count, only 32 % of the ‘positive control’ pairs were supported by randomized clinical trials [12]. Surely we need better reason to believe that ‘base truths’ against which research methods are evaluated are actually true. Our second major concern is the potentially poor validity of three of the four HOI algorithms used by OMOP [13]. OMOP’s studies examined acute myocardial infarction (AMI), acute kidney injury (AKI), acute liver injury (ALI), and gastrointestinal bleeding (GIB). OMOP’s algorithm for GIB would probably perform well given that positive predictive values (PPVs; the proportion of putative cases identified by an algorithm that are actually true cases of that HOI) of 71 to 88 % have been found for similar algorithms [14–16]. However, OMOP’s algorithm for AMI did not restrict cases to those for which the AMI diagnosis was in the principal position (i.e., the ostensible reason for the hospitalization), as is recommended [17], nor did it match an algorithm that has been widely used in recent years [17]. Given this, it is difficult to estimate the PPV of OMOP’s primary AMI algorithm, although it might range from 52 to 95 % [17]. OMOP’s primary algorithm for AKI would be expected to have a PPV ranging from 48 to 96 %, depending on the reference standard used to define a true case [18–21]. These variable PPVs for AKI highlight the inconsistent accuracy of diagnoses of renal conditions, which can lead to unacceptably low PPVs [22]. Identifying ALI from healthcare claims is even more challenging [23]. Some of the diagnostic codes used in OMOP’s primary ALI algorithm have poor-to-moderate PPVs (range 7 to 54 %) [23]. In addition to chart-based validation studies cited above, OMOP investigators reported that their primary and alternate HOI algorithms had poor validity when evaluated against expert review of claims and laboratory S. Hennessy C. E. Leonard (&) Center for Pharmacoepidemiology Research and Training, Center for Clinical Epidemiology and Biostatistics, Perelman School of Medicine at the University of Pennsylvania, Philadelphia, PA, USA e-mail: celeonar@mail.med.upenn.edu