The Reliability of Replications: A Study in Computational Reproductions
Authors: Breznau Nate , Rinke Eike Mark , Wuttke Alexander , Nguyen Hung Hoang Viet , Adem Muna , Adriaans Jule , Akdeniz Esra , Alvarez-Benjumea Amalia , Andersen Henrik Kenneth , Auer Daniel , Azevedo Flavio , Bahnsen Oke , Bai Ling , Balzer Dave , Bauer Gerrit , Bauer Paul , Baumann Markus , Baute Sharon , Benoit Verena , Bernauer Julian , Berning Carl , Berthold Anna , Bethke Felix S. , Biegert Thomas , Blinzler Katharina , Blumenberg Johannes , Bobzien Licia , Bohman Andrea , Bol Thijs , Bostic Amie , Brzozowska Zuzanna , Burgdorf Katharina , Burger Kaspar , Busch Kathrin , Castillo Juan Carlos , Chan Nathan , Christmann Pablo , Connelly Roxanne , Czymara Christian S. , Damian Elena , de Rooij Eline Adriane , Ecker Alejandro , Edelmann Achim , Eder Christina , Eger Maureen A. , Ellerbrock Simon , Forke Anna , Forster Andrea Gabriele , Freire Danilo , Gaasendam Chris , Gavras Konstantin , Gayle Vernon , Gessler Theresa , Gnambs Timo , Godefroidt Amélie , Grömping Max , Groß Martin , Gruber Stefan , Gummer Tobias , Hadjar Andreas , Halbherr Verena , Heisig Jan Paul , Hellmeier Sebastian , Heyne Stefanie , Hirsch Magdalena , Hjerm Mikael , Hochman Oshrat , Höffler Jan H. , Hövermann Andreas , Hunger Sophia , Hunkler Christian , Huth-Stöckle Nora , Ignacz Zsofia , Israel Sabine , Jacobs Laura , Jacobsen Jannes , Jaeger Bastian , Jungkunz Sebastian , Jungmann Nils , Kanjana Jennifer , Kauff Mathias , Khan Salman , Khatua Sayak , Kleinert Manuel , Klinger Julia , Kolb Jan-Philipp , Kołczyńska Marta , Kuk John Seungmin , Kunißen Katharina , Sinatra Dafina Kurti , Greinert Alexander , Lee Robin C. , Lersch Philipp M. , Liu David , Löbel Lea-Maria , Lutscher Philipp , Mader Matthias , Madia Joan Eliel , Malancu Natalia , Maldonado Luis , Marahrens Helge , Martin Nicole , Martinez Paul , Mayerl Jochen , MAYORGA OSCAR Jose , McDonnell Robert Myles , McManus Patricia A. , Wagner Kyle , Meeusen Cecil , Meierrieks Daniel , Mellon Jonathan , Merhout Friedolin , Merk Samuel , Meyer Daniel , Micheli Leticia , Mijs Jonathan J.B. , Moya Cristóbal , Neunhoeffer Marcel , Nüst Daniel , Nygård Olav , Ochsenfeld Fabian , Otte Gunnar , Pechenkina Anna , Pickup Mark , Prosser Christopher , Raes Louis , Ralston Kevin , Ramos Miguel , Reichert Frank , Roets Arne , Rogers Jonathan , Ropers Guido , Samuel Robin , Sand Gergor , Petrarca Constanza Sanhueza , Schachter Ariela , Schaeffer Merlin , Schieferdecker David , Schlueter Elmar , Schmidt Katja , Schmidt Regine , Schmidt-Catran Alexander , Schmiedeberg Claudia , Schneider Jürgen , Schoonvelde Martijn , Schulte-Cloos Julia , Schumann Sandy , Schunck Reinhard , Seuring Julian , Silber Henning , Sleegers Willem W. A. , Sonntag Nico , Staudt Alexander , Steiber Nadia , Steiner Nils , Sternberg Sebastian , Stiers Dieter , Stojmenovska Dragana , Storz Nora , Striessnig Erich , Stroppe Anne-Kathrin , Suchow Jordan , Teltemann Janna , Tibajev Andrey , Tung Brian B. , Vagni Giacomo , Van Assche Jasper , van der Linden Meta , van der Noll Jolanda , Van Hootegem Arno , Vogtenhuber Stefan , Voicu Bogdan , Wagemans Fieke , Wehl Nadja , Wernere Hannah , Wiernik Brenton M. , Winter Fabian , Wolf Christof , Wu Cary , Yamada Yuki , Zakula Björn , Zhang Nan , Ziller Conrad , Zins Stefan , Żółtak Tomasz 
Published: May 2021

This paper reports findings from a crowdsourced replication. Eighty-five independent teams attempted a computational replication of results reported in an original study of policy preferences and immigration by fitting the same statistical models to the same data. The replication involved an experimental condition. Random assignment put participating teams into either the transparent group that received the original study and code, or the opaque group receiving only a methods section, rough results description and no code. The transparent group mostly verified the numerical results of the original study with the same sign and p-value threshold (95.7%), while the opaque group had less success (89.3%). Exact numerical reproductions to the second decimal place were far less common (76.9% and 48.1%), and the number of teams who verified at least 95% of all effects in all models they ran was 79.5% and 65.2% respectively. Therefore, the reliability we quantify depends on how reliability is defined, but most definitions suggest it would take a minimum of three independent replications to achieve reliability. Qualitative investigation of the teams’ workflows reveals many causes of error including mistakes and procedural variations. Although minor error across researchers is not surprising, we show this occurs where it is least expected in the case of computational reproduction. Even when we curate the results to boost ecological validity, the error remains large enough to undermine reliability between researchers to some extent. The presence of inter-researcher variability may explain some of the current “reliability crisis” in the social sciences because it may be undetected in all forms of research involving data analysis. The obvious implication of our study is more transparency. Broader implications are that researcher variability adds an additional meta-source of error that may not derive from conscious measurement or modeling decisions, and that replications cannot alone resolve this type of uncertainty.