Variability in the analysis of a single neuroimaging dataset by many teams
Authors: Botvinik-Nezer Rotem , Holzmeister Felix , Camerer Colin F. , Dreber Anna , Huber Juergen , Johannesson Magnus , Kirchler Michael , Iwanir Roni , Mumford Jeanette A. , Adcock Alison , Avesani Paolo , Baczkowski Blazej , Bajracharya Aahana , Bakst Leah , Ball Sheryl , Barilari Marco , Bault Nadège , Beaton Derek , Beitner Julia , Benoit Roland , Berkers Ruud , Bhanji Jamil , Biswal Bharat , Bobadilla-Suarez Sebastian , Bortolini Tiago , Bottenhorn Katherine , Bowring Alexander , Braem Senne , Brooks Hayley , Brudner Emily , Calderon Cristian , Camilleri Julia , Castrellon Jaime , Cecchetti Luca , Cieslik Edna , Cole Zachary , Collignon Olivier , Cox Robert , Cunningham William , Czoschke Stefan , Dadi Kamalaker , Davis Charles , De Luca Alberto , Delgado Mauricio , Demetriou Lysia , Dennison Jeffrey , Di Xin , Dickie Erin , Dobryakova Ekaterina , Donnat Claire , Dukart Juergen , Duncan Niall W. , Durnez Joke , Eed Amr , Eickhoff Simon , Erhart Andrew , Fontanesi Laura , Fricke G. Matthew , Galvan Adriana , Gau Remi , Genon Sarah , Glatard Tristan , Glerean Enrico , Goeman Jelle , Golowin Sergej , González-García Carlos , Gorgolewski Krzysztof , Grady Cheryl , Green Mikella , Guassi Moreira João , Guest Olivia , Hakimi Shabnam , Hamilton J. Paul , Hancock Roeland , Handjaras Giacomo , Harry Bronson , Hawco Colin , Herholz Peer , Herman Gabrielle , Heunis Stephan , Hoffstaedter Felix , Hogeveen Jeremy , Holmes Susan , Hu Chuan-Peng , Huettel Scott , Hughes Matthew , Iacovella Vittorio , Iordan Alexandru , Isager Peder , Isik Ayse Ilkay , Jahn Andrew , Johnson Matthew , Johnstone Tom , Joseph Michael , Juliano Anthony , Kable Joseph , Kassinopoulos Michalis , Koba Cemal , Kong Xiang-Zhen , Koscik Timothy , Kucukboyaci Nuri Erkut , Kuhl Brice , Kupek Sebastian , Laird Angela , Lamm Claus , Langner Robert , Lauharatanahirun Nina , Lee Hongmi , Lee Sangil , Leemans Alexander , Leo Andrea , Lesage Elise , Li Flora , Li Monica , Lim Phui Cheng , Lintz Evan , Liphardt Schuyler , Losecaat Vermeer Annabel , Love Bradley , Mack Michael , Malpica Norberto , Marins Theo , Maumet Camille , McDonald Kelsey , McGuire Joseph , Melero Helena , Méndez Leal Adriana , Meyer Benjamin , Meyer Kristin , Mihai Paul , Mitsis Georgios , Moll Jorge , Nielson Dylan , Nilsonne Gustav , Notter Michael , Olivetti Emanuele , Onicas Adrian , Papale Paolo , Patil Kaustubh , Peelle Jonathan E. , Pérez Alexandre , Pischedda Doris , Poline Jean-Baptiste , Prystauka Yanina , Ray Shruti , Reuter-Lorenz Patricia , Reynolds Richard , Ricciardi Emiliano , Rieck Jenny , Rodriguez-Thompson Anais , Romyn Anthony , Salo Taylor , Samanez-Larkin Gregory , Sanz-Morales Emilio , Schlichting Margaret , Schultz Douglas , Shen Qiang , Sheridan Margaret , Shiguang Fu , Silvers Jennifer , Skagerlund Kenny , Smith Alec , Smith David , Sokol-Hessner Peter , Steinkamp Simon , Tashjian Sarah , Thirion Bertrand , Thorp John , Tinghög Gustav , Tisdall Loreen , Tompson Steven , Toro-Serey Claudio , Torre Juan , Tozzi Leonardo , Truong Vuong , Turella Luca , van’t Veer Anna E. , Verguts Tom , Vettel Jean , Vijayarajah Sagana , Vo Khoi , Wall Matthew , Weeda Wouter D. , Weis Susanne , White David , Wisniewski David , Xifra-Porxas Alba , Yearling Emily , Yoon Sangsuk , Yuan Rui , Yuen Kenneth , Zhang Lei , Zhang Xu , Zosky Joshua , Nichols Thomas E. , Poldrack Russell A. , Schonberg Tom 
Published: November 2019

SummaryData analysis workflows in many scientific domains have become increasingly complex and flexible. To assess the impact of this flexibility on functional magnetic resonance imaging (fMRI) results, the same dataset was independently analyzed by 70 teams, testing nine ex-ante hypotheses. The flexibility of analytic approaches is exemplified by the fact that no two teams chose identical workflows to analyze the data. This flexibility resulted in sizeable variation in hypothesis test results, even for teams whose statistical maps were highly correlated at intermediate stages of their analysis pipeline. Variation in reported results was related to several aspects of analysis methodology. Importantly, meta-analytic approaches that aggregated information across teams yielded significant consensus in activated regions across teams. Furthermore, prediction markets of researchers in the field revealed an overestimation of the likelihood of significant findings, even by researchers with direct knowledge of the dataset. Our findings show that analytic flexibility can have substantial effects on scientific conclusions, and demonstrate factors related to variability in fMRI. The results emphasize the importance of validating and sharing complex analysis workflows, and demonstrate the need for multiple analyses of the same data. Potential approaches to mitigate issues related to analytical variability are discussed.