Multidimensional Signals and Analytic Flexibility: Estimating Degrees of Freedom in Human-Speech Analyses
Authors: Coretta Stefano , Casillas Joseph V. , Roessig Simon , Franke Michael , Ahn Byron , Al-Hoorie Ali H. , Al-Tamimi Jalal , Alotaibi Najd E. , AlShakhori Mohammed K. , Altmiller Ruth M. , Arantes Pablo , Athanasopoulou Angeliki , Baese-Berk Melissa M. , Bailey George , Sangma Cheman Baira A , Beier Eleonora J. , Benavides Gabriela M. , Benker Nicole , BensonMeyer Emelia P. , Benway Nina R. , Berry Grant M. , Bing Liwen , Bjorndahl Christina , Bolyanatz Mariška , Braver Aaron , Brown Violet A. , Brown Alicia M. , Brugos Alejna , Buchanan Erin M. , Butlin Tanna , Buxó-Lugo Andrés , Caillol Coline , Cangemi Francesco , Carignan Christopher , Carraturo Sita , Caudrelier Tiphaine , Chodroff Eleanor , Cohn Michelle , Cronenberg Johanna , Crouzet Olivier , Dagar Erica L. , Dawson Charlotte , Diantoro Carissa A. , Dokovova Marie , Drake Shiloh , Du Fengting , Dubuis Margaux , Duême Florent , Durward Matthew , Egurtzegi Ander , Elsherif Mahmoud M. , Esser Janina , Ferragne Emmanuel , Ferreira Fernanda , Fink Lauren K. , Finley Sara , Foster Kurtis , Foulkes Paul , Franzke Rosa , Frazer-McKee Gabriel , Fromont Robert , García Christina , Geller Jason , Grasso Camille L. , Greca Pia , Grice Martine , Grose-Hodge Magdalena S. , Gully Amelia J. , Halfacre Caitlin , Hauser Ivy , Hay Jen , Haywood Robert , Hellmuth Sam , Hilger Allison I. , Holliday Nicole , Hoogland Damar , Huang Yaqian , Hughes Vincent , Icardo Isasa Ane , Ilchovska Zlatomira G. , Jeon Hae-Sung , Jones Jacq , Junges Mágat N. , Kaefer Stephanie , Kaland Constantijn , Kelley Matthew C. , Kelly Niamh E. , Kettig Thomas , Khattab Ghada , Koolen Ruud , Krahmer Emiel , Krajewska Dorota , Krug Andreas , Kumar Abhilasha A. , Lander Anna , Lentz Tomas O. , Li Wanyin , Li Yanyu , Lialiou Maria , Lima Ronaldo M. , Lo Justin J. H. , Lopez Otero Julio Cesar , Mackay Bradley , MacLeod Bethany , Mallard Mel , McConnellogue Carol-Ann Mary , Moroz George , Murali Mridhula , Nalborczyk Ladislas , Nenadić Filip , Nieder Jessica , Nikolić Dušan , Nogueira Francisco G. S. , Offerman Heather M. , Passoni Elisa , Pélissier Maud , Perry Scott J. , Pfiffner Alexandra M. , Proctor Michael , Rhodes Ryan , Rodríguez Nicole , Roepke Elizabeth , Röer Jan P. , Sbacco Lucia , Scarborough Rebecca , Schaeffler Felix , Schleef Erik , Schmitz Dominic , Shiryaev Alexander , Sóskuthy Márton , Spaniol Malin , Stanley Joseph A. , Strickler Alyssa , Tavano Alessandro , Tomaschek Fabian , Tucker Benjamin V. , Turnbull Rory , Ugwuanyi Kingsley O. , Urrestarazu-Porta Iñigo , van de Vijver Ruben , Van Engen Kristin J. , van Miltenburg Emiel , Wang Bruce Xiao , Warner Natasha , Wehrle Simon , Westerbeek Hans , Wiener Seth , Winters Stephen , Wong Sidney G.-J. , Wood Anna , Wottawa Jane , Xu Chenzi , Zárate-Sández Germán , Zellou Georgia , Zhang Cong , Zhu Jian , Roettger Timo B. 
Published: July 2023

Recent empirical studies have highlighted the large degree of analytic flexibility in data analysis that can lead to substantially different conclusions based on the same data set. Thus, researchers have expressed their concerns that these researcher degrees of freedom might facilitate bias and can lead to claims that do not stand the test of time. Even greater flexibility is to be expected in fields in which the primary data lend themselves to a variety of possible operationalizations. The multidimensional, temporally extended nature of speech constitutes an ideal testing ground for assessing the variability in analytic approaches, which derives not only from aspects of statistical modeling but also from decisions regarding the quantification of the measured behavior. In this study, we gave the same speech-production data set to 46 teams of researchers and asked them to answer the same research question, resulting in substantial variability in reported effect sizes and their interpretation. Using Bayesian meta-analytic tools, we further found little to no evidence that the observed variability can be explained by analysts’ prior beliefs, expertise, or the perceived quality of their analyses. In light of this idiosyncratic variability, we recommend that researchers more transparently share details of their analysis, strengthen the link between theoretical construct and quantitative system, and calibrate their (un)certainty in their conclusions.