Reporting Discrepancies Between the ClinicalTrials.gov Results Database and Peer-Reviewed Publications

Medical decision makers who use clinical trial evidence most often rely on findings that are published in peer-reviewed journals. Selective reporting of clinical trial results is a well-documented problem that raises concerns about using journal publications (1). Clinical trial registration is one mechanism aimed at reducing the effect of dissemination biases. Although many clinical trial registries exist, the single largest publicly accessible trial registry, and the only one with a results database, is ClinicalTrials.gov (2). Administered through the National Library of Medicine, ClinicalTrials.gov was developed to provide the public with a Web-based, searchable source of information about trials conducted within the United States. In September 2007, the Food and Drug Administration Amendments Act (FDAAA) was passed, greatly expanding the legal requirements for trial registration and mandating the creation of a publicly accessible clinical trial results database (3). According to FDAAA section 801, as of September 2008, basic summary results must be submitted for certain trials (called applicable clinical trials in the statute). Applicable clinical trials include most phase 2 through 4 trials of drugs, devices, or biologics regulated by the FDA having at least 1 site in the United States or conducted under an investigational new drug application or investigational device exemption (4). Several elements are required to be reported, including number of participants entering and completing the study; number of participants analyzed; demographic data, such as age and sex; summary results for all prespecified primary and secondary outcome measures; and anticipated and unanticipated adverse events by organ system. Results are generally required to be reported within 1 year of study completion, although submission may be delayed if the drug or device is not yet approved or if an application for a new use is to be submitted. The ClinicalTrials.gov results database has the potential to be a great asset for clinicians, patients, and researchers, but the ultimate validity of posted results is unclear. In contrast to the scientific scrutiny trials undergo during peer review for journals, results posted to ClinicalTrials.gov go through a quality assurance process focusing on internal consistency and logic. Although a gold standard repository of clinical trial results does not exist, inconsistencies between the ClinicalTrials.gov results database and other sources of clinical trial data suggest validity problems in 1 or both sources. The goal of this study was to assess the consistency of results reported in the ClinicalTrials.gov results database compared with those summarized in peer-reviewed journal publications. Methods Trial Selection Studies were eligible for inclusion if they posted results to ClinicalTrials.gov, were interventional, and were phase 3 or 4. To allow sufficient time for publication, we limited our search to trials with a primary completion date before 1 January 2009 or a start date before 1 July 2008 if the primary completion date field was not populated. Completed trials with results were sequenced in random order using Excel 2010 (Microsoft, Redmond, Washington) and were screened for the presence of a matching publication until a 10% random sample of trials with results was obtained. Trials were excluded if they did not assign participants to 2 or more interventional groups. We identified matching publications in a sequential process, first examining citations provided within ClinicalTrials.gov and then using a manual search of 2 electronic bibliographic databases. PubMed citations embedded within ClinicalTrials.gov can be provided by the investigator or the National Library of Medicine on the basis of matching National Clinical Trial (NCT) identifiers (5). We considered a publication to be a match if the intervention was the same and 1 or more groups in the trial had an identical number of participants. If relevant studies were not identified using citations provided within ClinicalTrials.gov, an electronic search of MEDLINE and the Cochrane Central Register of Controlled Trials was conducted using the study interventions, condition, principal investigator (if supplied), and date of trial completion as search criteria. Data Abstraction and Comparisons The following elements were abstracted and compared between the ClinicalTrials.gov results record and its corresponding publications: trial design, number of groups, primary outcome measure (POM) descriptions, secondary outcome measure (SOM) descriptions, total enrollment, and primary outcome results. We also abstracted the number of individuals affected by at least 1 adverse event (AE) and the number of individuals at risk, as reported to ClinicalTrials.gov. Comparisons of counts (such as enrollment, participants analyzed for the primary outcome, and number with an AE) were considered discrepant if they were not an exact match. The primary outcome result was required to be consistent to 1 decimal place. In cases of multiple publications, inconsistencies between the ClinicalTrials.gov results record and descriptions in any of the associated publications were considered a discrepancy. We classified POM description inconsistencies by using an existing framework describing the specificity of outcome reporting in ClinicalTrials.gov (6). The POM could deviate entirely in the domain measured or the number reported, the measurement tool used (for example, change in low-density lipoprotein cholesterol level vs. change in total cholesterol level), how the measure was used (for example, percentage change from baseline vs. absolute value), or the method of aggregation (for example, hemoglobin A1c level <7% vs. <8%). We considered SOMs to be consistent if they were mentioned in the results or methods section of the publication and were listed in the ClinicalTrials.gov results record. For trials with multiple publications, we considered the aggregate number of SOMs across all associated publications. When evaluating POM reporting consistency, we first determined whether the descriptions were consistent in both sources. When they were, we looked for discrepancies in the reported value (for example, mean response or count with outcome) or the number of individuals analyzed for the outcome (for example, denominator or number analyzed). For trials with more than 1 POM specified in both sources, any inconsistency in the result numerator or denominator was considered a discrepancy. If discrepancies in trial features resulted in downstream inconsistencies, we compared only the highest-order feature to avoid double counting. Adverse events did not become a mandatory reporting element until September 2009 and are summarized in the ClinicalTrials.gov results record in 2 tables: serious AEs (SAEs) and other (nonserious) AEs (OAEs). The FDA defines an SAE as any event that results in death, is life-threatening, requires or extends hospitalization, results in significant incapacity or interferes with normal life functions, or causes a congenital anomaly or birth defect (7). We compared the total number of SAEs reported in ClinicalTrials.gov with the total reported in the corresponding publications. In cases where the SAE counts differed, we compared the risk difference (experimental group risk minus control group risk) reported in ClinicalTrials.gov with the published estimate. For trials with multiple experimental groups, we selected the group of primary interest stated in the paper; if multiple FDA-approved dosing groups were assessed, we combined these for comparison with the control group. For OAEs, we restricted our comparison to specific AEs that could be matched to the publication without ambiguity and that were not also reported as an SAE in order to eliminate the possibility of double counting participants who may have had both a serious and nonserious AE. We distinguished publications reporting only treatment-related (attributable) AEs because ClinicalTrials.gov requires reporting of AEs regardless of attribution. Finally, we compared the number of deaths reported in each source. In ClinicalTrials.gov, deaths can be reported as an outcome, in the participant flow section, or as an SAE. If death was not a primary or secondary outcome, we compared the number of deaths reported in the participant flow or SAE section of ClinicalTrials.gov with the number reported in the publication. We classified the sources as discrepant only if counts of death differed between them. A second reviewer independently assessed reporting discrepancies between the ClinicalTrials.gov results record and the matched publication in a 20% random sample (22 trials) for all comparisons. Agreement between the primary and secondary abstractors was high, with an average of 0.98 across categories and no single category with a less than 0.91. Role of the Funding Source This work was supported by the National Library of Medicine and the Agency for Healthcare Research and Quality. The funding sources had no role in the design or execution of the study. Results The Figure describes the flow of trials from the initial ClinicalTrials.gov candidate pool to the final study sample. A total of 1669 phase 3 and 4 trials with posted results were initially identified through a query of ClinicalTrials.gov on 15 February 2011. After exclusion of trials with a primary completion date after 1 January 2009 and those not completed or terminated, 1120 trials remained. We randomly screened 357 potentially includable trials until a 10% sample (n= 110) was achieved. Three trials reported results in multiple publications. Table 1 describes the characteristics of the 110 matched trials and the 195 unmatched trials. Most studies were industry-funded, parallel-design trials of drugs. Unmatched trials were more likely to investigate something other than a drug or device and less likely to be cardiovascular trials. Twenty-nine t