Variation in effect sizes due to analytical decisions in ecology and evolutionary biology
Channel: Center for Open Science
Date: 2023-05-25

Session: Same data, different analysts: Variation in effect sizes due to analytical decisions in ecology and evolutionary biology

Speaker: Tim Parker, Professor of Biology and Environmental Studies, Whitman College

Abstract: Multiple recent ‘many-analyst’ projects have demonstrated that when different analysts seek to answer the same question with the same data, they often produce a substantially heterogeneous set of answers. We evaluated the extent of this phenomenon in ecology and evolutionary biology with a registered report in which several hundred biologists analyzed data from either of two data sets to answer a predetermined question associated with each data set. We found substantial heterogeneity in results from these analyses, but this heterogeneity differed qualitatively and quantitatively between the two data sets. Surprisingly, very little of the observed heterogeneity in results among analyses appeared related to the assessments of the individual analyses submitted by peer reviewers. Our results have major implications for how ecologists and evolutionary biologists conduct and interpret statistical analyses.