ClinicalTrials.gov and Drugs@FDA: A Comparison of Results Reporting for New Drug Approval Trials

Sponsors are required by federal law to submit summary results of applicable clinical trials (including those beyond phase 1 supporting U.S. Food and Drug Administration [FDA] new drug approvals) to ClinicalTrials.gov for public posting (1). Submissions consist of minimum basic results data elements in tabular format, including results for all primary and secondary outcomes prespecified in the study protocol and all anticipated and unanticipated serious adverse events observed during the trial (2). This law also requires ClinicalTrials.gov to assess ways to verify the accuracy of sponsor-submitted results information, including using public sources, such as FDA advisory committee summary documents and FDA action package approval documents (3). Although ClinicalTrials.gov currently determines internal consistency through quality checks (4), the validity of posted results can be assessed only by comparing submitted data with external reference standards. Recent studies comparing ClinicalTrials.gov data with peer-reviewed journal publications suggest that discrepancies in reported primary and secondary outcomes, numerical results, and adverse events are relatively common, although which source is more likely to be correct is unclear (57). Drug approval packages from the FDA may represent a better reference standard than publications for validating results posted on ClinicalTrials.gov, because journal editors and peer reviewers typically lack access to individual-participant data. Consequently, investigators may choose to publish outcomes based largely on statistical significance or other criteria (810). In contrast, FDA statisticians, who have access to individual-participant data, can analyze sponsor-submitted trial results independently on the basis of what they believe are the best statistical practices (11, 12). Independent analysis of individual-participant data from a trial may yield treatment effects that range in direction, magnitude, and statistical significance according to the particular outcome selected and how it is analyzed (for example, discretion in selecting measurement populations, such as intention-to-treat vs. per-protocol population; accounting for missing data; or timing for outcomes assessment). For example, on the basis of 6-month results, a high-profile journal article concluded that celecoxib reduced major bleeding compared with ibuprofen and diclofenac (13). However, the FDA reviews, which included results for the protocol-specified 1-year end points, indicated that celecoxib did not reduce major bleeding (14). We compared sponsor-submitted definitions and results posted on ClinicalTrials.gov with corresponding FDA-generated information posted on Drugs@FDA for trials used to support new drug approvalsspecifically, how often efficacy and adverse event outcomes could be compared and whether posted data were consistent. Methods Sample Collection To identify 100 parallel-group, randomized trials that were the basis for FDA new drug approvals (that is, new molecular entities), we searched Drugs@FDA (15) beginning with approvals on 1 January 2013 (Figure). Each FDA approval package includes review documents written by FDA staff (such as physicians, statisticians, and pharmacologists). These documents, which summarize analyses of clinical and other data submitted in new drug applications, are used by the FDA to determine whether to approve marketing of new drugs or biologic products for a particular use (16). Although the FDA has made some drug approval packages and component review documents publicly available on Drugs@FDA since 1997 (12), recent federal law now requires systematic posting of action packages for original new drug applications (1). Figure. Trial search and selection. FDA = U.S. Food and Drug Administration. * The 20 unmatched trials were from 8 new drug reviews: 4 had other matched trials, 4 did not. The 50 trials without results in both sources were from 21 new drug reviews: 12 had other matched trials, 9 did not. We manually searched FDA medical and statistical reviews to find all trials designated as pivotal and supportive by the FDA reviewer. We then sought to match these trials with the corresponding results in ClinicalTrials.gov, downloaded on 15 March 2015. Although ClinicalTrials.gov and Drugs@FDA were created for different purposes, their content overlaps substantially (Table 1). Because FDA review documents do not list ClinicalTrials.gov identifiers (NCT numbers), we used the process described in the Figure to match trials between sources. We searched Drugs@FDA through July 2014 until reaching our target: 75 pivotal and 25 supportive parallel-group, randomized trials with some results data in both sources, comprising all trials that could be compared during this time frame. We hypothesized that documents available from Drugs@FDA would contain less results information for supportive trials than for pivotal trials, which provide the primary evidence for approval. Table 1. Comparison of ClinicalTrials.gov and FDA Reviews on Drugs@FDA Data Extraction We created a structured data extraction form to capture detailed trial information from ClinicalTrials.gov and Drugs@FDA and revised it after a pilot test extracting information for 5 trials. The 6 major domains (Appendix Table 1) were as follows: 1) trial characteristics, including drug indication, development phase, blinding, comparator, and basic trial data (number of patients randomly assigned, number of patients completing the study, age, and sex distribution); 2) primary outcome, including definition using the following framework: domain, specific measurement, specific metric, and method of aggregation (4) plus time frame, analysis (measurement population and methods to account for missing data), result values (consistency of number analyzed and results for each study group), and treatment effect (consistency in treatment effect and associated CI and P value between experimental and control groups); 3) secondary outcomes (number, data availability, and outcome); 4) serious adverse events (number analyzed and consistency of results); 5) deaths (whether they were reported or it was mentioned that no deaths occurred; consistency of results); and 6) number of other adverse events. Appendix Table 1. Overview of Data Extracted From ClinicalTrials.gov and FDA Reviews on Drugs@FDA and Definitions of Discrepancies Between These Sources In contrast to ClinicalTrials.gov records, which typically present only a single set of analyses per outcome, Drugs@FDA review documents often present multiple analyses, including those conducted by the sponsor and separately by the FDA statistician (such as sensitivity analyses with different measurement populations or different imputation methods for missing data). We extracted results from the FDA statistical reviewer's independent analyses (available for two thirds of primary outcomes) or, if unavailable, from the drug company's analyses, provided that the statistical reviewer explicitly indicated agreement with them (remaining primary outcomes). Two assessors systematically extracted, and another verified, the trial design, primary and secondary outcomes, adverse events, and deaths from both ClinicalTrials.gov and Drugs@FDA. Data Comparison Consistency of Outcome Definitions and Analyses The number and definitions of primary and secondary outcomes posted on ClinicalTrials.gov (and concordance in outcome level) were compared with those listed in Drugs@FDA. Definitions of primary outcome were considered discordant if a mismatch occurred at any level of the following framework: domain (such as anxiety), specific measurement (such as Beck Anxiety Inventory), specific metric (such as change from baseline), or method of aggregation (such as mean). We also used an alternative definition of discordance that excludes the method of aggregation to account for researchers who feel it is unnecessary to prespecify statistical analysis plans before trial unblinding (11). In addition, we compared timing of the outcome assessment plus 3 key aspects of the primary outcome analyses: measurement population, crude or adjusted analysis, and method of handling missing data. Consistency of Results We assessed the consistency of results reporting between ClinicalTrials.gov and Drugs@FDA using the approach adopted from Hartung and colleagues (7) (detailed in Appendix Table 1). For example, results for the outcome measure change from baseline HbA1c (hemoglobin A1c) were considered discordant if the reported values were not consistent to 1 decimal place (for example, 0.094 is not consistent with 0.12 because it rounds to 0.09, but 0.115 would be consistent because it rounds to 0.12). We analyzed the data at 2 levels: numbers of trials, and numbers of primary outcomes or named serious adverse events, including death. Although the latter approach explicitly shows the frequency of discrepancies for individual measures, it may overstate the distribution of discrepancies among trials, because the number of potential discrepancies is the product of the outcomes times the number of study groups. Reporting numbers of trials mitigates this problema few discordant trials (even with many outcomes) would not overwhelm most concordant trials. This approach, however, created an important challenge: How many discordant outcomes (or study groups) does it take to deem a trial discordant between the 2 sources? We used the Hartung approach and called a trial discordant if data from ClinicalTrials.gov and Drugs@FDA were inconsistent for 1 or more results; concordant if all were consistent; and cannot compare if, in both sources, the outcomes did not match or the data were not posted. Role of the Funding Source Drs. Woloshin and Schwartz were funded by a contract from the National Library of Medicine. Drs. Tse and Zarin were supported in part by the Intramural Research Program of the National Library of Medicine, National Institutes of He