Publishing replication studies to support excellence in physiological research

No one working in the field of biomedical research can be unaware of what has been termed the “reproducibility crisis”. For example, Prinz et al (2011) reported that research scientists in a major pharmaceutical company were unable to reproduce 65% of the results published in peer-reviewed scientific journals, supporting concerns that had been emerging over several years (Ioannidis, 2008). As a result of these concerns and the ensuing debate, funders and journal editors have moved to reinforce and improve standards in data recording and reporting (US National Institutes of Health, Principles and Guidelines for Reporting Preclinical Research, 2014). A central tenet of all scientific research is that new findings must be subsequently replicated by others. A discovery cannot be valuable if it is not judged sufficiently important to require attempted replication. Most research that has not been successfully replicated will be of limited value. Yet, despite widespread agreement that replication of experimental results is a central requirement of the scientific process, there are considerable barriers to undertaking replication studies. These include difficulty in funding such studies and reluctance by journals to publish them (Ioannidis et al. 2014). When submitted, such work is often rejected on the grounds of ‘lack of novelty’, because novelty is an essential requirement for consideration of a manuscript by many journals. To support researchers who undertake worthwhile replication studies in the field of physiological research, Experimental Physiology is introducing an additional, new category of research article. The intent of this new article category is to encourage the publication of replication studies that have adequate experimental group sizes to make a useful, meaningful contribution to the body of publicly available research data. The pillar underpinning this new article type is that a manuscript that achieves this aim will be published regardless of whether it agrees with or contradicts the original report. Put another way, no specific P value is required when testing the significance of the differences between control and test groups. To identify articles that achieve this objective, we have established three straightforward further criteria that must be satisfied in addition to all our normal standards (see our Information for Authors). The first criterion is that the replication study seeks to reproduce a novel finding that is considered an important advance in the field of integrative and translational physiology, and thus should be confirmed. The introduction to the manuscript reporting the replication study must make this case convincingly. The novel finding must have been previously published in a reputable peer-reviewed journal, and that original publication must be cited in the introduction of the replication study. The replication study must be undertaken by an independent research group and must be the first bona fide attempt to replicate the novel finding of interest, using the same experimental design and methods, in so far as it is reasonably possible to do so. This may require obtaining unique reagents, including genetically manipulated animals, from the original research group. In that event, the authors of the replication study must confirm that the unique reagents are active in the way previously published (e.g. a specific gene is successfully deleted). The replication study need not contain every element reported in the original publication. However, the central experiments leading to the results that formed the novel findings of the original study must be repeated. The second criterion that we will require of a replication study is that all of the underlying data are made freely accessible, where ethically permissible, in open access repositories or as supplemental files. This is to permit the use of the data in any subsequent studies in which future researchers undertake analyses using combined data sets, another approach that has been recommended to facilitate confirmation and assessment of the importance of novel findings. This is called individual participant data meta-analysis in the case of human studies. The availability of original data in this way would allow others to use additional analyses that are useful in replication studies, such as confidence, credibility or prediction intervals, Bayesian methods or false-discovery rates (Wasserstein & Lazar, 2016). The final criterion that a replication study must satisfy is that it must report data from experimental groups large enough to make a useful, meaningful contribution to the body of publicly available research data. We define such groups as either (i) having n = 10 in each experimental group or (ii) that the number in each group provides adequate statistical power to detect the important effect reported in the original study; the smaller of these two numbers will be accepted. Determining the number required in each group to give adequate statistical power is not as straightforward as might at first appear and may not have received sufficient attention in previous replication studies. To illustrate this issue, consider conducting a replication study when the effect size (difference between the test and control samples following some intervention) that was reported in the original study was exactly the same as that between the two populations from which those experimental samples were drawn, and α = 0.05 (two tailed). It is worth noting that this precise result is in fact very unlikely but is assumed here for the purpose of illustration. A random sample from a population rarely reflects exactly the population from which it is drawn. Thus, the reported effect size in any study is only an estimate of the true difference between the populations. If we undertook the replication experiment with the same sample size in each experimental group as in the original study, there is a probability of 0.50 that we