Small effect size leads to reproducibility failure in resting-state fMRI studies
Authors: Jia Xi-Ze , Zhao Na , Barton Barek , Burciu Roxana , Carrière Nicolas , Cerasa Antonio , Chen Bo-Yu , Chen Jun , Coombes Stephen , Defebvre Luc , Delmaire Christine , Dujardin Kathy , Esposito Fabrizio , Fan Guo-Guang , Federica Di Nardo , Feng Yi-Xuan , Fling Brett W. , Garg Saurabh , Gilat Moran , Gorges Martin , Ho Shu-Leong , Horak Fay B. , Hu Xiao , Hu Xiao-Fei , Huang Biao , Huang Pei-Yu , Jia Ze-Juan , Jones Christy , Kassubek Jan , Krajcovicova Lenka , Kurani Ajay , Li Jing , Li Qian , Liu Ai-Ping , Liu Bo , Liu Hu , Liu Wei-Guo , Lopes Renaud , Lou Yu-Ting , Luo Wei , Madhyastha Tara , Mao Ni-Ni , McAlonan Grainne , McKeown Martin J. , Pang Shirley YY , Quattrone Aldo , Rektorova Irena , Sarica Alessia , Shang Hui-Fang , Shine James , Shukla Priyank , Slavicek Tomas , Song Xiao-Peng , Tedeschi Gioacchino , Tessitore Alessandro , Vaillancourt David , Wang Jian , Wang Jue , Wang Z. Jane , Wei Lu-Qing , Wu Xia , Xu Xiao-Jun , Yan Lei , Yang Jing , Yang Wan-Qun , Yao Nai-Lin , Zhang De-Long , Zhang Jiu-Quan , Zhang Min-Ming , Zhang Yan-Ling , Zhou Cai-Hong , Yan Chao-Gan , Zuo Xi-Nian , Hallett Mark , Wu Tao , Zang Yu-Feng 
Published: March 2018

AbstractThousands of papers using resting-state functional magnetic resonance imaging (RS-fMRI) have been published on brain disorders. Results in each paper may have survived correction for multiple comparison. However, since there have been no robust results from large scale meta-analysis, we do not know how many of published results are truly positives. The present meta-analytic work included 60 original studies, with 57 studies (4 datasets, 2266 participants) that used a between-group design and 3 studies (1 dataset, 107 participants) that employed a within-group design. To evaluate the effect size of brain disorders, a very large neuroimaging dataset ranging from neurological to psychiatric isorders together with healthy individuals have been analyzed. Parkinson’s disease off levodopa (PD-off) included 687 participants from 15 studies. PD on levodopa (PD-on) included 261 participants from 9 studies. Autism spectrum disorder (ASD) included 958 participants from 27 studies. The meta-analyses of a metric named amplitude of low frequency fluctuation (ALFF) showed that the effect size (Hedges’ g) was 0.19 - 0.39 for the 4 datasets using between-group design and 0.46 for the dataset using within-group design. The effect size of PD-off, PD-on and ASD were 0.23, 0.39, and 0.19, respectively. Using the meta-analysis results as the robust results, the between-group design results of each study showed high false negative rates (median 99%), high false discovery rates (median 86%), and low accuracy (median 1%), regardless of whether stringent or liberal multiple comparison correction was used. The findings were similar for 4 RS-fMRI metrics including ALFF, regional homogeneity, and degree centrality, as well as for another widely used RS-fMRI metric namely seed-based functional connectivity. These observations suggest that multiple comparison correction does not control for false discoveries across multiple studies when the effect sizes are relatively small. Meta-analysis on un-thresholded t-maps is critical for the recovery of ground truth. We recommend that to achieve high reproducibility through meta-analysis, the neuroimaging research field should share raw data or, at minimum, provide un-thresholded statistical images.