Many Labs 5: Testing pre-data collection peer review as an intervention to increase replicability
Authors: Ebersole Charles R. , Mathur Maya B , Baranski Erica , Bart-Plange Diane-Jo , Buttrick Nick , Chartier Christopher R. , Corker Katherine S. , Corley Martin , Hartshorne Joshua K. , IJzerman Hans , Lazarevic Ljiljana B. , Rabagliati Hugh , Ropovik Ivan , Aczel Balazs , Aeschbach Lena Fanya , Andrighetto Luca , Arnal Jack Dennis , Arrow Holly , Babincak Peter , Bakos Bence Endre , Baník Gabriel , Baskin Ernest , Belopavlović Radomir , Bernstein Michael , Bialek Michal , Bloxsom Nicholas , Bodroža Bojana , Bonfiglio Diane B. V. , Boucher Leanne , Brühlmann Florian , Brumbaugh Claudia Chloe , Casini Erica , Chen Yiling , Chiorri Carlo , Chopik William J. , Christ Oliver , Claypool Heather M. , coary Sean , Čolić Marija V. , Collins W. Matthew , Curran Paul G , Day Chris , Dering Benjamin , Dreber Anna , Edlund John , Falcão Filipe , Fedor Anna , Feinberg Lily , Ferguson Ian , Ford Máire , Frank Michael C. , Fryberger Emily , Garinther Alexander , Gawryluk Katarzyna , Giacomantonio Mauro , Giessner Steffen Robert , Grahe Jon E. , Guadagno Rosanna Elizabeth , Hałasa Ewa , Hancock Peter , Hüffmeier Joachim , Hughes Sean , Idzikowska Katarzyna , Inzlicht Michael , Jern Alan , Jimenez-Leal William , Johannesson Magnus , Joy-Gaba Jennifer Alana , Kauff Mathias , Kellier Danielle , Kidwell Mallory , Kimbrough Amanda , King Josiah , Kołodziej Sabina , Kovacs Marton , Krasuska Karolina , Kraus Sue , Krueger Lacy Elise , Kuchno Katarzyna , Lage Caio Ambrosio , Langford Eleanor V. , Levitan Carmel , Lima Tiago Jessé Souza , Lin Hause , Lins Samuel , Loy J E , Manfredi Dylan , Markiewicz Lukasz , Menon Madhavi , Mercier Brett , Metzger Mitchell , Millen Ailsa E , Miller Jeremy K. , Montealegre Andres , Moore Don A , Nave Gideon , Nichols Austin Lee , Novak Sarah Ann , Orlic Ana , Panno Angelo , Parks Kimberly P. , Pedović Ivana , Pękala Emilian , Penner Matthew R. , Pessers Sebastiaan , Petrovic Boban , Pfeiffer Thomas , Pieńkosz Damian , Preti Emanuele , Purić Danka , Ramos Tiago Silva , Ravid Jon , Razza Timothy , Rentzsch Katrin , Richetin Juliette , Rife Sean Chandler , Dalla Rosa Anna , Salamon Janos , Saunders Blair , Sawicki Przemyslaw , Schmidt Kathleen , Schuepfer Kurt , Schultze Thomas , Schulz-Hardt Stefan , Schütz Astrid , Shabazian Ani , Silva Rúben Filipe Lopes , Sioma Barbara , Skorb Lauren , Souza Luana Elayne Cunha , steegen sara , Stein LAR , Sternglanz R. Weylin , Stojilović Darko , Storage Daniel , Sullivan Gavin Brent , Szaszi Barnabas , Szecsi Peter , Szoke Orsolya , Szuts Attila , Thomae Manuela , Tidwell Natasha Davis , tocco Carly , Torka Ann-Kathrin , tuerlinckx francis , vanpaemel wolf , Vaughn Leigh Ann , Vianello Michelangelo , Viganola Domenico , Vlachou Maria , Walker Ryan J. , Weissgerber Sophia Christin , Wichman Aaron Lee , Wiggins Bradford Jay , Wolf Daniel , Wood Michael James , Zealley David A. , Zezelj Iris , Zrubka Mark , Nosek Brian A. 
Published: October 2019

Replications in psychological science sometimes fail to reproduce prior findings. If replications use methods that are unfaithful to the original study or ineffective in eliciting the phenomenon of interest, then a failure to replicate may be a failure of the protocol rather than a challenge to the original finding. Formal pre-data collection peer review by experts may address shortcomings and increase replicability rates. We selected 10 replications from the Reproducibility Project: Psychology (RP:P; Open Science Collaboration, 2015) in which the original authors had expressed concerns about the replication designs before data collection and only one of which was “statistically significant” (p &lt; .05). Commenters suggested that lack of adherence to expert review and low-powered tests were the reasons that most of these RP:P studies failed to replicate (Gilbert et al., 2016). We revised the replication protocols and received formal peer review prior to conducting new replications. We administered the RP:P and Revised protocols in multiple laboratories (Median number of laboratories per original study = 6.5; Range 3 to 9; Median total sample = 1279.5; Range 276 to 3512) for high-powered tests of each original finding with both protocols. Overall, Revised protocols produced similar effect sizes as RP:P protocols following the preregistered analysis plan (Δr = .002 or .014, depending on analytic approach). The median effect size for Revised protocols (r = .05) was similar to RP:P protocols (r = .04) and the original RP:P replications (r = .11), and smaller than the original studies (r = .37). The cumulative evidence of original study and three replication attempts suggests that effect sizes for all 10 (median r = .07; range .00 to .15) are 78% smaller on average than original findings (median r = .37; range .19 to .50), with very precisely estimated effects.