Replication initiative: beware misinterpretation.

Two new initiatives are exploring the reproducibility of findings within psychology (“Psychology's bold initiative,” S. Carpenter, News Focus, 30 March, p. [1558][1]). One approach provides an online outlet for replication attempts, regardless of success or failure. The other approach evaluates reproducibility with new empirical replications. Proponents of these initiatives should be careful, because it is easy to misinterpret replication successes and failures in a field that uses statistics ([ 1 ][2]). Psychologists establish empirical findings by using hypothesis tests that determine the probability ( P ) of data more extreme than observed under a null hypothesis. Equally important is power, which is the probability of rejecting a false null hypothesis. These probabilities reflect the inherent uncertainty in conclusions that can be drawn from a sampled set of data. Because of random sampling, even experiments that measure true positive effects will sometimes fail to reject the null hypothesis. For example, an experiment that just barely rejects the null hypothesis (e.g., P = 0.05) is expected to yield insignificant results almost half the time in future replications with the same sample size. A failure to replicate an established finding often ignites a debate about the differing conclusions, but in many cases there is no real conflict. Given the relatively low power of most psychological experiments, a failure to replicate should be common. It is also possible to have too much replication success. Rejection of the null hypothesis at a rate inconsistent with the experimental power values indicates the possibility of publication bias ([ 2 ][3], [ 3 ][4]). In such cases, the experimental findings are nonscientific because relevant data is missing. The significance of replication studies should be assessed by carefully structured meta-analysis studies ([ 4 ][5]). 1. [↵][6]1. V. Hedges, 2. I. Olkin , Statistical Methods for Meta-Analysis (Psychology Press, New York, 1985). 2. [↵][7]1. J. P. A. Ioannidis, 2. T. A. Trikalinos , Clini. Trials 4, 245 (2007). [OpenUrl][8][CrossRef][9] 3. [↵][10]1. G. Francis , Psychonomic Bull. Rev. 19, 151 (2012). [OpenUrl][11][CrossRef][12] 4. [↵][13]1. G. Cumming , Understanding the New Statistics: Effect Sizes, Confidence Intervals, and Meta-Analysis (Routledge, New York, 2012). [1]: pending:yes [2]: #ref-1 [3]: #ref-2 [4]: #ref-3 [5]: #ref-4 [6]: #xref-ref-1-1 "View reference 1 in text" [7]: #xref-ref-2-1 "View reference 2 in text" [8]: {openurl}?query=rft.jtitle%253DClini.%2BTrials%26rft.volume%253D4%26rft.spage%253D245%26rft_id%253Dinfo%253Adoi%252F10.1177%252F1740774507079441%26rft.genre%253Darticle%26rft_val_fmt%253Dinfo%253Aofi%252Ffmt%253Akev%253Amtx%253Ajournal%26ctx_ver%253DZ39.88-2004%26url_ver%253DZ39.88-2004%26url_ctx_fmt%253Dinfo%253Aofi%252Ffmt%253Akev%253Amtx%253Actx [9]: /lookup/external-ref?access_num=10.1177/1740774507079441&link_type=DOI [10]: #xref-ref-3-1 "View reference 3 in text" [11]: {openurl}?query=rft.jtitle%253DPsychonomic%2BBull.%2BRev.%26rft.volume%253D19%26rft.spage%253D151%26rft_id%253Dinfo%253Adoi%252F10.3758%252Fs13423-012-0227-9%26rft.genre%253Darticle%26rft_val_fmt%253Dinfo%253Aofi%252Ffmt%253Akev%253Amtx%253Ajournal%26ctx_ver%253DZ39.88-2004%26url_ver%253DZ39.88-2004%26url_ctx_fmt%253Dinfo%253Aofi%252Ffmt%253Akev%253Amtx%253Actx [12]: /lookup/external-ref?access_num=10.3758/s13423-012-0227-9&link_type=DOI [13]: #xref-ref-4-1 "View reference 4 in text"