Reproducible Research: Moving toward Research the Public Can Really Trust

The context for our paper is every journal's nightmare: publishing fraudulent research. Cases of scientific misconduct have surfaced with alarming frequency, tarnishing public trust in journals, researchers, and science itself. To much fanfare, the journal Science published promising developments in stem cell research only to learn months later that the findings were bogus (1). Lancet editors retracted a paper on oral cancer when the editors learned that the researchers had fabricated their data (2). The New England Journal of Medicine published a statement of concern on a drug toxicity study when evidence suggested that the authors had withheld important adverse outcomes (3). Incidents of authors who have not disclosed potential conflicts of interest have prompted JAMA to reinforce its disclosure policies (46). Here at Annals, we retracted an article after learning that the lead author had fabricated the data. (79). In the midst of these accumulating scandals, a New York Times article declared, Reporters Find Science Journals Harder to Trust, but not Easy to Verify (10). Reporters are not alone; journal editors also find biomedical research harder to trust and not easy to verify. Fraud gets headlines and drives editors to despair but fortunately occurs rarely. We at Annals spend much of our time evaluating the validity and generalizability of every article that we consider for publication. Like our colleagues at other biomedical journals, we strive to assure ourselves and our readers that the information we publish is accurate. Yet, because we do not personally witness every step of the research process, we can never be entirely certain of validity. We are less concerned with outright fraud, which can elude even the most astute reviewers and editors, than with the subtler but much more prevalent biases that are present in one form or another in most clinical research. Prompted by both the recent wave of scientific misconduct and our own experience in trying to minimize the impact of bias in our pages, we review here how we protect against publishing research that is misleading or outright wrong. Existing Strategies to Guard against Research Impropriety Annals relies on a variety of processes to evaluate the validity of work that we consider for publication (Table). These processes span the life of the article from submission through postpublication. Table. Existing and New Strategies That Annals Uses to Guard against the Publication of Biased or Invalid Research To facilitate the detection of potential sources of bias or threats to validity, Annals requires authors to include several pieces of information with submitted manuscripts. They must disclose the contributions and potential conflicts of interest of all authors on the byline, including but not limited to any financial relationship that involves diseases, tests, or treatments discussed in the manuscript or competing products. This alerts editors, reviewers, and readers to situations in which competing interests might bias content or threaten validity (11). Authors know that this information will be public, which we hope increases their sense of public accountability for unbiased reporting. We also require authors of clinical trials to register the trials (12) and submit copies of the trial protocol. These requirements give editors access to the detailed, prespecified research plan, which can be useful if the manuscript lacks key details or when we need to reconcile what the researchers planned with what they reported. Annals' next layer of defense against publishing erroneous information is a multilayered review that includes scrutiny by both journal staff and external reviewers. We obtain external review on about 50% of submitted manuscripts, turning to the more than 12000 individuals in our reviewer database to augment the range of skills and knowledge of our editors. Although the peer-review process is not perfect, these reviews provide important information that we weigh during the decision-making process; reviewers influence our decisions on a daily basis by identifying problems that we missed. Many authors believe (incorrectly) that reviewers' opinions always dictate the editorial decision. Sometimes, the external reviewer's opinion is decisive; however, usually the reviewer's comments are incorporated into the insights and judgments of our own editorial team, which includes scientists and clinicians with a range of methodological and clinical skills. We then make a preliminary decision at an editorial conference, during which we discuss all papers under consideration by the associate and deputy editors. After the preliminary decision, the journal's processes are a mix of evaluative and editorial functions, with the editorial function taking on greater importance as we near final acceptance of the manuscript. We never rely solely on the statistical expertise of the authors. Any papers selected for possible publication must pass through yet another filter, a methods conference in which Annals' 6 statistical editors, 4 deputy editors, and editor-in-chief scrutinize a study's methodology. The statistician assigned to the manuscript presents the main statistical issues for discussion at this conference. Often, we ask for statistical code or data to clarify or validate methods. We may ask authors to provide supplementary analyses to assess the sensitivity of results to different assumptions or methods, work with authors to improve analyses, or on occasion, perform our own analyses of the data with the authors' cooperation. Usually, the revised paper reflects these clarifications. These discussions, or subsequent inquiries, occasionally diminish our confidence in the work to the point that we choose not to publish it. More often, they improve the clarity and completeness of the manuscript, making the strengths and weaknesses of the study more evident to the reader and ensuring that the tone of the conclusions matches the strength of the evidence. These outcomes of editing and peer review have been documented in empirical studies at Annals (13) and, more recently, in a preliminary report of a study that included several journals (14). Arguably, the most important function of prepublication peer review and editing is to make effective postpublication peer review possible. The most stringent internal processes cannot guarantee truth in reporting or reliably detect fraud; however, they can provide the scientific community with the tools to assess the strength and validity of published evidence. We strive to publish methods sections that provide sufficient detail so that readers can assess all the potential threats to validity. Reporting standards like CONSORT for randomized, controlled trials (15, 16) also facilitate thorough reporting of study methodology. When the methods are transparent, fraud, bias, and weak research methods are easier to spot. In this age of instant electronic dissemination and reader feedback, features like rapid response letters to the editors provide a medium for postpublication peer review and discussion of the research. Reproducible Research: A New Strategy to Increase Confidence in Published Research Even with these safeguards, experience teaches us that we must seek more innovative strategies to promote truth in publication. Independent replication by independent scientists in independent settings provides the best assurance that a scientific finding is valid; however, the resources and time required for high-quality clinical studies makes literal replication of published studies a slow corrective to any errors in the original publication. However, scientists and journal editors can promote reproducible research, described recently by Peng and colleagues (17). The term has its roots in highly technical forms of statistical programming and data presentation, but its requirement for sharing data and analytic code is applicable to clinical research. Reproducibility involves methods to ensure that independent scientists can reproduce published results by using the same procedures and data as the original investigators. It also requires that the primary investigators share their data and methodological details. These include, at a minimum, the original protocol, the dataset used for the analysis, and the computer code used to produce the results. Reproducibility requires a level of openness that is rare in the current competitive environment of biomedical research. Investigators might be understandably concerned that sharing data would make them vulnerable to unwanted scrutiny or enable other researchers to make discoveries using the data that took them years to assemble. Recognizing this, Peng and colleagues describe forms of data sharing that are between unlimited access and no access (17). Their proposals, adapted in part from commercial models of intellectual property protection, involve limitations on the permitted use of the data set, as set forth in a license or written agreement between an investigator and a requestor. At one extreme, researchers could only have access to portions of the analytic data set and solely for the purpose of reproducing published results; at the other extreme, researchers could have unrestricted access to the entire file of raw data. Programs incorporated into commercial software that create technical barriers to unauthorized use or widespread dissemination could also protect authors' interests. We believe that the reproducible research model is important. By strengthening the scientific process, it can also protect the public against any long-term adverse effects of research gone wrong. For this reason, Annals is launching a reproducible research initiative. Every original research article will include a statement that indicates whether the study protocol, data, or statistical code is available to readers and under what terms authors will share this information. Sharing will not be mandatory, but