Same data, different analysts: variation in effect sizes due to analytical decisions in ecology and evolutionary biology
Authors: Gould Elliot , Fraser Hannah , Parker Timothy , Nakagawa Shinichi , Griffith Simon , Vesk Peter , Fidler Fiona , Abbey-Lee Robin , Abbott Jessica , Aguirre Luis , Alcaraz Carles , Altschul Drew , Arekar Kunal , Atkins Jeff , Atkinson Joe , Barrett Meghan , Bell Kristian , Bello Suleiman , Berauer Bernd , Bertram Michael , Billman Peter , Blake Charlie , Blake Shannon , Bliard Louis , Bonisoli-Alquati Andrea , Bonnet Timothée , Bordes Camille , Bose Aneesh , Botterill-James Thomas , Boyd Melissa , Boyle Sarah , Bradfer-Lawrence Tom , Brand Jack , Brengdahl Martin , Bulla Martin , Bussière Luc , Camerlenghi Ettore , Campbell Sara , Campos Leonardo , Caravaggi Anthony , Cardoso Pedro , Carroll Charles , Catanach Therese , Chen Xuan , Chik Heung Ying Janet , Choy Emily , Christie Alec , Chuang Angela , Chunco Amanda , Clark Bethany , Cox Murray , Cressman Kimberly , Crouch Connor , D'Amelio Pietro , de Sousa Alexandra , Döbert Timm , Dobler Ralph , Dobson Adam , Doherty Tim , Drobniak Szymon , Duffy Alexandra , Dunn Robert , Dunning Jamie , Eberhart-Hertel Luke , Elmore Jared , Elsherif Mahmoud , English Holly , Ensminger David , Ernst Ulrich , Ferguson Stephen , Ferreira-Arruda Thalita , Fieberg John , Finch Elizabeth , Fiorenza Evan , Fisher David , Forstmeier Wolfgang , Fourcade Yoan , Francesca Santostefano Francesca , Frank Graham , Freund Cathryn , Gandy Sara , Gannon Dustin , García-Cervigón Ana , Géron Charly , Gilles Marc , Girndt Antje , Gliksman Daniel , Goldspiel Harrison , Gomes Dylan , Goslee Sarah , Gosnell J. , Gratton Paolo , Grebe Nicholas , Greenler Skye , Griffith Daniel , Griffith Frances , Grossman Jake , Güncan Ali , Haesen Stef , Hagan James , Harrison Natasha , Hasnain Sarah , Havird Justin , Heaton Andrew , Hsu Bin-Yan , Iranzo Esperanza , Iverson Erik , Jimoh Saheed , Johnson Douglas , Johnsson Martin , Jorna Jesse , Jucker Tommaso , Jung Martin , Kačergytė Ineta , Ke Alison , Kelly Clint , Keogan Katharine , Keppeler Friedrich , Killion Alexander , Kim Dongmin , Kochan David , Korsten Peter , Kothari Shan , Kuppler Jonas , Kusch Jillian , Lagisz Malgorzata , Larkin Daniel , Larson Courtney , Lauck Katherine , Lauterbur M. , Law Alan , Léandri-Breton Don-Jean , Lievens Eva , Lima Daniela , Lindsay Shane , Macphie Kirsty , Mair Magdalena , Malm Lisa , Mammola Stefano , Manhart Michael , Mäntylä Elina , Marchand Philippe , Marshall Benjamin , Martin Dominic , Martin Jake , Martin Charles , Martinig April , McCallum Erin , McNew Sabrina , Meiners Scott , Michelangeli Marcus , Moiron Maria , Moreira Bruno , Mortensen Jennifer , Mos Benjamin , Muraina Taofeek , Nelli Luca , Nilsonne Gustav , Nolazco Sergio , Nooten Sabine , Novotny Jessie , Olin Agnes , Organ Chris , Ostevik Kate , Palacio Facundo , Paquet Matthieu , Pascall David , Pasquarella Valerie , Payo-Payo Ana , Pedersen Karen , Perez Grégoire , Perry Kayla , Pottier Patrice , Proulx Michael , Proulx Raphaël , Pruett Jessica , Ramananjato Veronarindra , Randimbiarison Finaritra , Razafindratsima Onja , Rennison Diana , Riva Federico , Riyahi Sepand , Roast Michael , Rocha Felipe , Roche Dominique , Román-Palacios Cristian , Rosenberg Michael , Ross Jessica , Rowland Freya , Rugemalila Deusdedith , Russell Avery , Ruuskanen Suvi , Saccone Patrick , Sadeh Asaf , Salazar Stephen , sales kris , Salmón Pablo , Sanchez-Tojar Alfredo , Santos Leticia , Schilling Hayden , Schmidt Marcus , Schmoll Tim , Schneider Adam , Schrock Allie , Schroeder Julia , Schtickzelle Nicolas , Schultz Nick , Scott Drew , Shapiro Julie , Sharma Nitika , Shearer Caroline , Sitvarin Michael , Skupien Fabrício , Slinn Heather , Smith Jeremy , Smith Grania , Sollmann Rahel , Stack Whitney Kaitlin , Still Shannon , Stuber Erica , Sutton Guy , Swallow Ben , Taff Conor , Takola Elina , Tanentzap Andrew , Thawley Christopher , Tortorelli Claire , Trlica Andrew , Turnell Biz , Urban Lara , Van de Vondel Stijn , van Oordt Francis , Vanderwel Mark , Vanderwel K. , Vanderwolf Karen , Verrelli Brian , Vieira Marcus , Vollering Julien , Walker Xanthe , Walter Jonathan , Waryszak Pawel , Weaver Ryan , Weller Daniel , Whelan Shannon , White Rachel , Wolfson David , Wood Andrew , Yanco Scott , Yen Jian , Youngflesh Casey , Zilio Giacomo , Zimmer Cédric , Zitomer Rachel , Villamil Nora , Tompkins Emily 
Published: October 2023

Although variation in effect sizes and predicted values among studies of similar phenomena is inevitable, such variation far exceeds what might be produced by sampling error alone. One possible explanation for variation among results is differences among researchers in the decisions they make regarding statistical analyses. A growing array of studies has explored this analytical variability in different (mostly social science) fields, and has found substantial variability among results, despite analysts having the same data and research question. We implemented an analogous study in ecology and evolutionary biology, fields in which there have been no empirical exploration of the variation in effect sizes or model predictions generated by the analytical decisions of different researchers. We used two unpublished datasets, one from evolutionary ecology (blue tit, Cyanistes caeruleus, to compare sibling number and nestling growth) and one from conservation ecology (Eucalyptus, to compare grass cover and tree seedling recruitment), and the project leaders recruited 174 analyst teams, comprising 246 analysts, to investigate the answers to prespecified research questions. Analyses conducted by these teams yielded 141 usable effects for the blue tit dataset, and 85 usable effects for the Eucalyptus dataset. We found substantial heterogeneity among results for both datasets, although the patterns of variation differed between them. For the blue tit analyses, the average effect was convincingly negative, with less growth for nestlings living with more siblings, but there was near continuous variation in effect size from large negative effects to effects near zero, and even effects crossing the traditional threshold of statistical significance in the opposite direction. In contrast, the average relationship between grass cover and Eucalyptus seedling number was only slightly negative and not convincingly different from zero, and most effects ranged from weakly negative to weakly positive, with about a third of effects crossing the traditional threshold of significance in one direction or the other. However, there were also several striking outliers in the Eucalyptus dataset, with effects far from zero. For both datasets, we found substantial variation in the variable selection and random effects structures among analyses, as well as in the ratings of the analytical methods by peer reviewers, but we found no strong relationship between any of these and deviation from the meta-analytic mean. In other words, analyses with results that were far from the mean were no more or less likely to have dissimilar variable sets, use random effects in their models, or receive poor peer reviews than those analyses that found results that were close to the mean. The existence of substantial variability among analysis outcomes raises important questions about how ecologists and evolutionary biologists should interpret published results, and how they should conduct analyses in the future.