Variability in the analysis of a single neuroimaging dataset by many teams
Authors: Jean-Baptiste Poline
Edna C Cieslik
Richard C Reynolds
Matthew E Hughes
Sangil Lee
Felix Hoffstaedter
Emanuele Olivetti
Sergej A E Golowin
Steven H Tompson
Russell A Poldrack
Angela R Laird
Benjamin Meyer
Tom Schonberg
Jeanette A Mumford
Bronson B Harry
Remi Gau
Sarah M Tashjian
Ruud M W J Berkers
J Paul Hamilton
Xu Zhang
Peer Herholz
Alexander Bowring
Tom Verguts
Flora Li
Alba Xifra-Porxas
Gustav Nilsonne
Erin W Dickie
Adriana Galván
Margaret A Sheridan
Matthew B Wall
Andrew Jahn
Anna E van &#x27;t Veer
Jenny R Rieck
Joseph W Kable
Luca Cecchetti
Timothy R Koscik
Nadège Bault
Xiang-Zhen Kong
Adriana S Méndez Leal
Julia Beitner
Glad Mihai
Michael J E Joseph
Gabrielle Herman
Katherine L Bottenhorn
Yanina Prystauka
Julia A Camilleri
Blazej M Baczkowski
Cemal Koba
Nuri Erkut Kucukboyaci
Michael P Notter
Anthony C Juliano
Juergen Huber
Khoi Vo
Leonardo Tozzi
Juergen Dukart
Derek Beaton
Aahana Bajracharya
Monica Y C Li
Alexander Leemans
Luca Turella
Cristian B Calderon
Joseph T McGuire
David J White
Felix Holzmeister
Niall W Duncan
Michael L Mack
Loreen Tisdall
Simon R Steinkamp
Juan Jesus Torre Tresols
Shabnam Hakimi
Tristan Glatard
Gregory R Samanez-Larkin
Susanne Weis
Tiago Bortolini
Krzysztof J Gorgolewski
Emily G Brudner
Hongmi Lee
Patricia A Reuter-Lorenz
Michalis Kassinopoulos
Paolo Avesani
Joke Durnez
R Alison Adcock
Jennifer A Silvers
Theo Marins
Anais M Rodriguez-Thompson
Vuong Truong
Peder M Isager
Bharat B Biswal
Robert Langner
Robert W Cox
Xin Di
John N Thorp
Bertrand Thirion
Alexandre Pérez
Alberto De Luca
Norberto Malpica
Roeland Hancock
Jamil P Bhanji
Sagana Vijayarajah
Shiguang Fu
Peter Sokol-Hessner
Joshua E Zosky
Sarah Genon
Enrico Glerean
Evan N Lintz
G Matthew Fricke
Sheryl Ball
Laura Fontanesi
Tom Johnstone
Colin F Camerer
Schuyler W Liphardt
Olivia Guest
Qiang Shen
Sebastian Kupek
Roni Iwanir
Charles P Davis
Sangsuk Yoon
Amr Eed
Kristin N Meyer
David Wisniewski
Thomas E Nichols
Margaret L Schlichting
Cheryl L Grady
Jelle J Goeman
Kenny Skagerlund
Wouter D Weeda
Douglas H Schultz
Ayse I Isik
Stephan Heunis
Annabel B Losecaat Vermeer
Magnus Johannesson
Bradley C Love
Helena Melero
Alexandru D Iordan
Kaustubh R Patil
Simon B Eickhoff
Stefan Czoschke
Dylan M Nielson
Paolo Papale
Olivier Collignon
Andrea Leo
Alec Smith
Rotem Botvinik-Nezer
Jorge Moll
Jonathan E Peelle
Kenneth S L Yuen
Phui Cheng Lim
Michael Kirchler
Shruti Ray
Jean M Vettel
Claus Lamm
Kamalaker Dadi
Lei Zhang
Brice A Kuhl
Matthew R Johnson
Adrian I Onicas
Carlos González-García
Elise Lesage
Ekaterina Dobryakova
Jeremy Hogeveen
Nina Lauharatanahirun
Susan Holmes
Sebastian Bobadilla-Suarez
Jeffrey B Dennison
Leah Bakst
João F Guassi Moreira
Hayley R Brooks
Marco Barilari
Doris Pischedda
Lysia Demetriou
Emiliano Ricciardi
Claudio Toro-Serey
Vittorio Iacovella
Camille Maumet
Taylor Salo
Scott A Huettel
Anthony Romyn
William A Cunningham
Mauricio R Delgado
Emilio Sanz-Morales
Senne Braem
Claire L Donnat
Mikella A Green
Giacomo Handjaras
Rui Yuan
Andrew Erhart
David V Smith
Zachary J Cole
Georgios D Mitsis
Anna Dreber
Emily A Yearling
Kelsey McDonald
Roland G Benoit
Colin Hawco
Gustav Tinghög
Jaime J Castrellon
Chuan-Peng Hu
Journal: Nature (2020)
Discipline: Multidisciplinary
Keywords: Data analysis workflows; scientific domains; flexibility; functional magnetic resonance imaging; analytical approaches
Affiliations: Sagol School of Neuroscience, Tel Aviv University, Tel Aviv, Israel., Department of Neurobiology, The George S. Wise Faculty of Life Sciences, Tel Aviv University, Tel Aviv, Israel., Department of Psychological and Brain Sciences, Dartmouth College, Hanover, NH, USA., Department of Banking and Finance, University of Innsbruck, Innsbruck, Austria., HSS and CNS, California Institute of Technology, Pasadena, CA, USA., Department of Economics, Stockholm School of Economics, Stockholm, Sweden., Department of Economics, University of Innsbruck, Innsbruck, Austria., Center for Healthy Minds, University of Wisconsin-Madison, Madison, WI, USA., Center for Cognitive Neuroscience, Duke University, Durham, NC, USA., Department of Psychiatry and Behavioral Sciences, Duke University, Durham, NC, USA., Neuroinformatics Laboratory, Fondazione Bruno Kessler, Trento, Italy., Center for Mind/Brain Sciences - CIMeC, University of Trento, Rovereto, Italy., Department of Neurology, Max Planck Institute for Human Cognitive and Brain Sciences, Leipzig, Germany., Department of Otolaryngology, Washington University in St. Louis, St. Louis, MO, USA., Department of Psychological and Brain Sciences, Boston University, Boston, MA, USA., Center for Systems Neuroscience, Boston University, Boston, MA, USA., Department of Economics, Virginia Tech, Blacksburg, VA, USA., School of Neuroscience, Virginia Tech, Blacksburg, VA, USA., Crossmodal Perception and Plasticity Laboratory, Institutes for Research in Psychology (IPSY) and Neurosciences (IoNS), UCLouvain, Louvain-la-Neuve, Belgium., School of Psychology, University of Plymouth, Plymouth, UK., Rotman Research Institute, Baycrest Health Sciences Centre, Toronto, Ontario, Canada., Department of Psychology, University of Amsterdam, Amsterdam, The Netherlands., Department of Psychology, Goethe University, Frankfurt am Main, Germany., Max Planck Research Group: Adaptive Memory, Max Planck Institute for Human Cognitive and Brain Sciences, Leipzig, Germany., Department of Psychology, Rutgers University-Newark, Newark, NJ, USA., Department of Biomedical Engineering, New Jersey Institute of Technology, Newark, NJ, USA., School of Life Science and Technology, University of Electronic Science and Technology of China, Chengdu, China., Department of Experimental Psychology, University College London, London, UK., D'Or Institute for Research and Education (IDOR), Rio de Janeiro, Brazil., Department of Psychology, Florida International University, Miami, FL, USA., Oxford Big Data Institute, Li Ka Shing Centre for Health Information and Discovery, Nuffield Department of Population Health, University of Oxford, Oxford, UK., Department of Experimental Psychology, Ghent University, Ghent, Belgium., Department of Psychology, Vrije Universiteit Brussel, Brussels, Belgium., Department of Psychology, University of Denver, Denver, CO, USA., Institute of Neuroscience and Medicine, Brain and Behaviour (INM-7), Research Centre Jülich, Jülich, Germany., Institute of Systems Neuroscience, Medical Faculty, Heinrich Heine University Düsseldorf, Düsseldorf, Germany., Department of Psychology and Neuroscience, Duke University, Durham, NC, USA., MoMiLab Research Unit, IMT School for Advanced Studies Lucca, Lucca, Italy., Department of Psychology, University of Nebraska-Lincoln, Lincoln, NE, USA., National Institute of Mental Health (NIMH), National Institutes of Health, Bethesda, MD, USA., Department of Psychology, University of Toronto, Toronto, Ontario, Canada., Institute of Medical Psychology, Goethe University, Frankfurt am Main, Germany., Inria, CEA, Université Paris-Saclay, Palaiseau, France., Department of Psychological Sciences, University of Connecticut, Storrs, CT, USA., Brain Imaging Research Center, University of Connecticut, Storrs, CT, USA., Connecticut Institute for the Brain and Cognitive Sciences, University of Connecticut, Storrs, CT, USA., PROVIDI Lab, Image Sciences Institute, University Medical Center Utrecht, Utrecht, The Netherlands., Section of Endocrinology and Investigative Medicine, Faculty of Medicine, Imperial College London, London, UK., Nuffield Department of Women's and Reproductive Health, University of Oxford, Oxford, UK., Department of Psychology, Temple University, Philadelphia, PA, USA., Krembil Centre for Neuroinformatics, Campbell Family Mental Health Research Institute, Centre for Addiction and Mental Health, Toronto, Ontario, Canada., Department of Psychiatry, University of Toronto, Toronto, Ontario, Canada., Center for Traumatic Brain Injury Research, Kessler Foundation, East Hanover, NJ, USA., Department of Statistics, Stanford University, Stanford, CA, USA., Graduate Institute of Mind, Brain and Consciousness, Taipei Medical University, Taipei, Taiwan., Brain and Consciousness Research Centre, TMU-ShuangHo Hospital, New Taipei City, Taiwan., Department of Psychology and Stanford Center for Reproducible Neuroscience, Stanford University, Stanford, CA, USA., Instituto de Neurociencias, CSIC-UMH, Alicante, Spain., Faculty of Psychology, University of Basel, Basel, Switzerland., Computer Science Department, University of New Mexico, Albuquerque, NM, USA., School of Management, Zhejiang University of Technology, Hangzhou, China., Institute of Neuromanagement, Zhejiang University of Technology, Hangzhou, China., Department of Psychology, University of California Los Angeles, Los Angeles, CA, USA., Department of Computer Science and Software Engineering, Concordia University, Montreal, Quebec, Canada., Department of Neuroscience and Biomedical Engineering, Aalto University, Espoo, Finland., Department of Biomedical Data Sciences, Leiden University Medical Center, Leiden, The Netherlands., Department of Psychology, Stanford University, Stanford, CA, USA., Research Centre on Interactive Media, Smart Systems and Emerging Technologies - RISE, Nicosia, Cyprus., Center for Social and Affective Neuroscience, Department of Biomedical and Clinical Sciences, Linköping University, Linköping, Sweden., The MARCS Institute for Brain, Behaviour and Development, Western Sydney University, Sydney, New South Wales, Australia., Campbell Family Mental Health Research Institute, Centre for Addiction and Mental Health, Toronto, Ontario, Canada., McConnell Brain Imaging Centre, The Neuro (Montreal Neurological Institute-Hospital), Faculty of Medicine, McGill University, Montreal, Quebec, Canada., Department of Electrical Engineering, Eindhoven University of Technology, Eindhoven, The Netherlands., Department of Research and Development, Epilepsy Centre Kempenhaeghe, Heeze, The Netherlands., Department of Psychology, University of New Mexico, Albuquerque, NM, USA., Psychology Clinical Neuroscience Center, University of New Mexico, Albuquerque, NM, USA., Leibniz-Institut für Resilienzforschung (LIR), Mainz, Germany., School of Health Sciences, Swinburne University of Technology, Hawthorn, Victoria, Australia., Department of Psychology, University of Michigan, Ann Arbor, MI, USA., Department of Industrial Engineering and Innovation Sciences, Eindhoven University of Technology, Eindhoven, The Netherlands., Department of Neuroscience, Max Planck Institute for Empirical Aesthetics, Frankfurt am Main, Germany., fMRI Laboratory, University of Michigan, Ann Arbor, MI, USA., Center for Brain, Biology and Behavior, University of Nebraska-Lincoln, Lincoln, NE, USA., Center for Neuropsychology and Neuroscience Research, Kessler Foundation, East Hanover, NJ, USA., Department of Psychology, University of Pennsylvania, Philadelphia, PA, USA., MindCORE, University of Pennsylvania, Philadelphia, PA, USA., Graduate Program in Biological and Biomedical Engineering, McGill University, Montreal, Quebec, Canada., Language and Genetics Department, Max Planck Institute for Psycholinguistics, Nijmegen, The Netherlands., Department of Psychiatry, University of Iowa Carver College of Medicine, Iowa City, IA, USA., Department of Physical Medicine and Rehabilitation, Rutgers New Jersey Medical School, Newark, NJ, USA., Department of Psychology, University of Oregon, Eugene, OR, USA., Faculty of Economics and Statistics, University of Innsbruck, Innsbruck, Austria., Department of Physics, Florida International University, Miami, Florida, USA., Department of Cognition, Emotion, and Methods in Psychology, Faculty of Psychology, University of Vienna, Vienna, Austria., Vienna Cognitive Science Hub, University of Vienna, Vienna, Austria., US CCDC Army Research Laboratory, Human Research and Engineering Directorate, Aberdeen Proving Ground, MD, USA., Annenberg School for Communication, University of Pennsylvania, Philadelphia, PA, USA., Department of Psychological and Brain Sciences, Johns Hopkins University, Baltimore, MD, USA., Fralin Biomedical Research Institute, Roanoke, VA, USA., Economics Experimental Lab, Nanjing Audit University, Nanjing, China., Haskins Laboratories, New Haven, CT, USA., Biology Department, University of New Mexico, Albuquerque, NM, USA., The Alan Turing Institute, London, UK., Laboratorio de Análisis de Imagen Médica y Biometría (LAIMBIO), Universidad Rey Juan Carlos, Madrid, Spain., Inria, Univ Rennes, CNRS, Inserm, IRISA UMR 6074, Empenn ERL U 1228, Rennes, France., Departamento de Psicobiología, División de Psicología, CES Cardenal Cisneros, Madrid, Spain., Northeastern University Biomedical Imaging Center, Northeastern University, Boston, MA, USA., Neuroimaging Center (NIC), Focus Program Translational Neurosciences (FTN), Johannes Gutenberg University Medical Center Mainz, Mainz, Germany., Department of Psychology and Neuroscience, University of North Carolina at Chapel Hill, Chapel Hill, NC, USA., Max Planck Research Group: Neural Mechanisms of Human Communication, Max Planck Institute for Human Cognitive and Brain Sciences, Leipzig, Germany., Chair of Cognitive and Clinical Neuroscience, Faculty of Psychology, Technische Universität Dresden, Dresden, Germany., Department of Bioengineering, McGill University, Montreal, Quebec, Canada., Data Science and Sharing Team, National Institute of Mental Health, National Institutes of Health, Bethesda, MD, USA., Department of Clinical Neuroscience, Karolinska Institutet, Stockholm, Sweden., Department of Psychology, Stockholm University, Stockholm, Sweden., The Laboratory for Investigative Neurophysiology (The LINE), Department of Radiology, University Hospital Center and University of Lausanne, Lausanne, Switzerland., Department of Vision and Cognition, Netherlands Institute for Neuroscience, Amsterdam, The Netherlands., Bernstein Center for Computational Neuroscience and Berlin Center for Advanced Neuroimaging and Clinic for Neurology, Charité Universitätsmedizin, corporate member of Freie Universität Berlin, Humboldt Universität zu Berlin, and Berlin Institute of Health, Berlin, Germany., Cluster of Excellence Science of Intelligence, Technische Universität Berlin and Humboldt Universität zu Berlin, Berlin, Germany., NeuroMI - Milan Center for Neuroscience, Milan, Italy., Henry H. Wheeler, Jr. Brain Imaging Center, Helen Wills Neuroscience Institute, University of California Berkeley, Berkeley, CA, USA., Scientific and Statistical Computing Core, National Institute of Mental Health, National Institutes of Health, Bethesda, MD, USA., Department of Behavioural Sciences and Learning, Linköping University, Linköping, Sweden., Center for Social and Affective Neuroscience, Department of Clinical and Experimental Medicine, Linköping University, Linköping, Sweden., Institute of Neuroscience and Medicine, Cognitive Neuroscience (INM-3), Research Centre Jülich, Jülich, Germany., Department of Psychology, Columbia University, New York, NY, USA., Department of Management and Engineering, Linköping University, Linköping, Sweden., Department of Health, Medicine and Caring Sciences, Linköping University, Linköping, Sweden., Center for Cognitive and Decision Sciences, University of Basel, Basel, Switzerland., Department of Psychiatry and Behavioral Sciences, Stanford University School of Medicine, Stanford, CA, USA., Methodology and Statistics Unit, Institute of Psychology, Leiden University, Leiden, The Netherlands., US Combat Capabilities Development Command Army Research Laboratory, Aberdeen, MD, USA., University of California Santa Barbara, Santa Barbara, CA, USA., University of Pennsylvania, Philadelphia, PA, USA., Invicro, London, UK., Faculty of Medicine, Imperial College London, London, UK., Clinical Psychopharmacology Unit, University College London, London, UK., Centre for Human Psychopharmacology, Swinburne University, Hawthorn, Victoria, Australia., Department of Management and Marketing, School of Business, University of Dayton, Dayton, OH, USA., Biomedical Engineering Department, University of Connecticut, Storrs, CT, USA., Oxford Big Data Institute, Li Ka Shing Centre for Health Information and Discovery, Nuffield Department of Population Health, University of Oxford, Oxford, UK. ., Department of Psychology, Stanford University, Stanford, CA, USA. ., Sagol School of Neuroscience, Tel Aviv University, Tel Aviv, Israel. ., Department of Neurobiology, The George S. Wise Faculty of Life Sciences, Tel Aviv University, Tel Aviv, Israel. .

Abstract:  Data analysis workflows in many scientific domains have become increasingly complex and flexible. Here we assess the effect of this flexibility on the results of functional magnetic resonance imaging by asking 70 independent teams to analyse the same dataset, testing the same 9 ex-ante hypotheses1. The flexibility of analytical approaches is exemplified by the fact that no two teams chose identical workflows to analyse the data. This flexibility resulted in sizeable variation in the results of hypothesis tests, even for teams whose statistical maps were highly correlated at intermediate stages of the analysis pipeline. Variation in reported results was related to several aspects of analysis methodology. Notably, a meta-analytical approach that aggregated information across teams yielded a significant consensus in activated regions. Furthermore, prediction markets of researchers in the field revealed an overestimation of the likelihood of significant findings, even by researchers with direct knowledge of the dataset2-5. Our findings show that analytical flexibility can have substantial effects on scientific conclusions, and identify factors that may be related to variability in the analysis of functional magnetic resonance imaging. The results emphasize the importance of validating and sharing complex analysis workflows, and demonstrate the need for performing and reporting multiple analyses of the same data. Potential approaches that could be used to mitigate issues related to analytical variability are discussed. 

Retracted: No

PubPeer comments:
 I found a number of the Western Blots in this paper to have odd features, but show only one that I feel is the best example of my concerns. In Figure 5A, the SQSTM1 p62 band appears to include evidence of one splice (yellow arrow) and two adjacent smears have multiple similarities (red boxes). The B-actin bands directly below the red boxes also appear quite similar. Could the authors post here the original uncropped scan to reassure readers like me? *** Shown in inverse ![file] *** Inverse without markups ![file] *** Original ![file] , 
 #1 Fig 3(a) and 5(a) - shenanigans in the loading control. ![file] ,