Many Labs 2: Investigating Variation in Replicability Across Sample and Setting
Authors: Klein Richard Anthony , Vianello Michelangelo , Hasselman Fred , Adams Byron Gregory , Adams Reginald B. , Alper Sinan , Aveyard Mark , Axt Jordan , Babalola Mayowa Tosin , Bahník Štěpán , Berkics Mihaly , Bernstein Michael Jason , Berry Daniel R. , Bialobrzeska Olga , Bocian Konrad , Brandt Mark , Busching Robert , Cai Huajian , Cambier Fanny , Cantarero Katarzyna , Carmichael Cheryl L , Cemalcilar Zeynep , Chandler Jesse J. , Chang Jen-Ho , Chatard Armand , CHEN Eva , Cheong Winnee , Cicero David C , Coen Sharon , Coleman Jennifer A. , Collisson Brian , Conway Morgan Anna , Corker Katherine S. , Curran Paul G , Cushman Fiery Andrews , Dalgar Ilker , Davis William E. , de Bruijn Maaike Jolise , de Vries Marieke , Devos Thierry , Doğulu Canay , Dozo Nerisa , Dukes Kristin Nicole , Dunham Yarrow , Durrheim Kevin , Easterbrook Matthew , Ebersole Charles R. , Edlund John , English Alexander Scott , Eller Anja , Finck Carolyn , Freyre Miguel-Ángel , Friedman Mike , Frankowska Natalia , Galliani Elisa Maria , Ghoshal Tanuka , Giessner Steffen Robert , Gill Tripat , Gnambs Timo , Gomez Angel , Gonzalez Roberto , Graham Jesse , Grahe Jon E. , Grahek Ivan , Green Eva , Hai Kakul , Haigh Matthew , Haines Elizabeth L. , Hall Michael P. , Heffernan Marie E. , Hicks Joshua A. , Houdek Petr , van der Hulst Marije , Huntsinger Jeffrey R. , Huynh Ho Phi , IJzerman Hans , Inbar Yoel , Innes-Ker Åse , Jimenez-Leal William , John Melissa-Sue , Joy-Gaba Jennifer Alana , Kamiloglu Roza Gizem , Kappes Andreas , Kappes Heather Barry , Karabati Serdar , Karick Haruna , Keller Victor N. , Kende Anna , Kervyn Nicolas / , Knezevic Goran , Kovacs Carrie , Krueger Lacy Elise , Kurapov German , Kurtz Jaime , Lakens Daniel , Lazarevic Ljiljana B. , Levitan Carmel , Lewis Neil Anthony , Lins Samuel , Maassen Esther , Maitner Angela , Malingumu Winfrida , Mallett Robyn , Marotta Satia Adele , McIntyre Jason Cameron , Međedović Janko Mileta , Milfont Taciano L , Morris Wendy , Myachykov Andriy , Murphy Sean Christopher , Neijenhuijs Koen Ilja , Nelson Anthony J. , Neto Felix Fernando Monteiro , Nichols Austin Lee , O'Donnell Susan L. , Oikawa Masanori , Orosz Gabor , Osowiecka Malgorzata , Packard Grant , Pérez Rolando , Petrovic Boban , Pilati Ronaldo , Pinter Brad , Podesta Lysandra Maria , Pollmann Monique Maria Henriettte , Dalla Rosa Anna , Rutchick Abraham M. , Saavedra Patricio , Sacco Airi , Saeri Alexander K , Salomon Erika , Schmidt Kathleen , Schönbrodt Felix D. , Sekerdej Maciek Borys , Sirlopu David Ricardo , Skorinko Jeanine , Smith Michael A. , Smith-Castro Vanessa , Sobkow Agata , Sowden Walter J. , Spachtholz Philipp , Steiner Troy G. , Stouten Jeroen , Street Chris N. H. , Sundfelt Oskar , Szumowska Ewa , Tang Andrew , Tanzer Norbert K. , Tear Morgan James , Theriault Jordan E. , Thomae Manuela , Torres-Fernández David , Traczyk Jakub , Tybur Joshua M. , Ujhelyi Adrienn , van Assen Marcel A. L. M. , van 't Veer Anna Elisabeth , Vásquez-Echeverría Alejandro , Vaughn Leigh Ann , Vázquez Alexandra . , Vega Diego , Verniers Catherine , Verschoor Mark , Voermans Ingrid , Vranka Marek Albert , Welch Cheryl Alyssa , Wichman Aaron Lee , Williams Lisa A. , Woodzicka Julie A. , Wronska Marta Katarzyna , Young Liane , Zelenski John M. , Nosek Brian A. 
Published: November 2018

We conducted preregistered replications of 28 classic and contemporary published findings with protocols that were peer reviewed in advance to examine variation in effect magnitudes across sample and setting. Each protocol was administered to approximately half of 125 samples and 15,305 total participants from 36 countries and territories. Using conventional statistical significance (p &lt; .05), fifteen (54%) of the replications provided evidence in the same direction and statistically significant as the original finding. With a strict significance criterion (p &lt; .0001), fourteen (50%) provide such evidence reflecting the extremely high powered design. Seven (25%) of the replications had effect sizes larger than the original finding and 21 (75%) had effect sizes smaller than the original finding. The median comparable Cohen’s d effect sizes for original findings was 0.60 and for replications was 0.15. Sixteen replications (57%) had small effect sizes (&lt; .20) and 9 (32%) were in the opposite direction from the original finding. Across settings, 11 (39%) showed significant heterogeneity using the Q statistic and most of those were among the findings eliciting the largest overall effect sizes; only one effect that was near zero in the aggregate showed significant heterogeneity. Only one effect showed a Tau &gt; 0.20 indicating moderate heterogeneity. Nine others had a Tau near or slightly above 0.10 indicating slight heterogeneity. In moderation tests, very little heterogeneity was attributable to task order, administration in lab versus online, and exploratory WEIRD versus less WEIRD culture comparisons. Cumulatively, variability in observed effect sizes was more attributable to the effect being studied than the sample or setting in which it was studied.