Using prediction markets to estimate the reproducibility of scientific research
Authors: Magnus Johannesson
Thomas Pfeiffer
Anna Dreber
Siri Isaksson
Yiling Chen
Johan Almenberg
Brian A Nosek
Brad Wilson
Journal: Proceedings of the National Academy of Sciences of the United States of America (2015)
Discipline: Multidisciplinary
Keywords: prediction markets; replications; reproducibility.
Affiliations: Department of Economics, Stockholm School of Economics, SE-113 83 Stockholm, Sweden; ., New Zealand Institute for Advanced Study, Massey University, Auckland 0745, New Zealand; Wissenschaftskolleg zu Berlin-Institute for Advanced Study, D-14193 Berlin, Germany;, Sveriges Riksbank, SE-103 37 Stockholm, Sweden;, Department of Economics, Stockholm School of Economics, SE-113 83 Stockholm, Sweden;, Consensus Point, Nashville, TN 37203;, John A. Paulson School of Engineering and Applied Sciences, Harvard University, Cambridge, MA 02138;, Department of Psychology, University of Virginia, Charlottesville, VA 22904; Center for Open Science, Charlottesville, VA 22903.

Abstract:  Concerns about a lack of reproducibility of statistically significant results have recently been raised in many fields, and it has been argued that this lack comes at substantial economic costs. We here report the results from prediction markets set up to quantify the reproducibility of 44 studies published in prominent psychology journals and replicated in the Reproducibility Project: Psychology. The prediction markets predict the outcomes of the replications well and outperform a survey of market participants' individual forecasts. This shows that prediction markets are a promising tool for assessing the reproducibility of published scientific results. The prediction markets also allow us to estimate probabilities for the hypotheses being true at different testing stages, which provides valuable information regarding the temporal dynamics of scientific discovery. We find that the hypotheses being tested in psychology typically have low prior probabilities of being true (median, 9%) and that a "statistically significant" finding needs to be confirmed in a well-powered replication to have a high probability of being true. We argue that prediction markets could be used to obtain speedy information about reproducibility at low cost and could potentially even be used to determine which studies to replicate to optimally allocate limited resources into replications. 

Retracted: No

PubPeer comments:
 Here, authors used survey of beliefs and prediction markets to aggregate reproducibility estimates of 44 studies replicated in the Reproducibility Project: Psychology. Over 90 researchers from various fields of psychology participated in the markets, ranging from graduate students to professors. Prediction markets correctly predicted most replication outcomes and could be useful to generate reproducibility estimates of scientific evidence at low cost. , 
 The authors actually conducted two prediction markets. They only report the combined results across 41 studies, but differences in performance across the two markets was statistically significantly different. Whereas performance for the first market was exceptionally good (over 90% correct), performance for the second market was at chance level (flip a coin). The market also did not do significantly better than a simple  don t trust anything rule. A simpler way to assess replicability is to disregard all p-values below .005 and to bet on studies with p-values less than this criterion value. Future methods of predicting replication outcomes should be evaluated against this simple rule. https: replicationindex.com 2021 05 10 osc-pm ,