User-Centered Design of Statistics for Researchers: The Case of Ronald A. Fisher

Ronald A. Fisher is acknowledged as the founder of experimental design and statistical methods for experiments with small samples. His 1925 and 1935 books, which went through many editions, set the stage for the modern era of research in cognitive psychology and human factors/ ergonomics (HF/E) with its emphasis on group designs. His null hypothesis testing approach of determining whether a null hypothesis can be rejected (typically at a .05 level) has been used extensively from the 1950s to the present. However, this approach to statistics was, and continues to be, criticized by various statisticians and quantitative psychologists. Many of them favor instead a decision analysis approach developed by Jerzy Neyman and Egon S. Pearson (1933a, 1933b). Their approach relies on specification of a particular alternative hypothesis a priori and views the statistical decision as accepting the null hypothesis or the alternative hypothesis. Although concepts from both are customarily intermixed without differentiation in contemporary statistics texts, the philosophies of these two approaches with regard to statistics and experimental design are antithetical. Statisticians tend to favor the Neyman-Pearson approach, which Perezgonzalez (2015) describes as deductive and less flexible than that of Fisher, because of its emphasis on planning about the effect size of a specified alternative hypothesis in advance. But most researchers tend to continue doing null hypothesis testing, roughly in line with Fisher’s approach, which Perezgonzalez depicts as inferential and flexible, with most work done a posteriori. What tends to be overlooked in most criticisms of null hypothesis significance testing is that Fisher worked as a researcher at an agricultural research institution for many years. Consequently, he developed his approach to statistics specifically to address the wants and needs of the researchers. Fisher emphasized throughout his career that statistical analysis plays a supportive role to experimental design. He stressed that the researchers’ knowledge and goals must be considered and given primacy when prescribing statistical methods for scientific research (Fisher, 1974). This user-centered design attitude adopted by Fisher, with its emphasis on researchers’ wants and needs, explains to a large extent why his applied approach to statistics has predominated throughout the history of contemporary cognitive psychology and HF/E research: It fits the way that many researchers design and conduct their experiments. This point was noted by Acree (1978) who concluded that, although Neyman and Pearson had the more sophisticated statistical theory, Fisher’s approach was closer to the needs of psychological researchers, including HF/E specialists. Most current “best practices” to remedy a perceived replication crisis are mainly based in the Neyman-Pearson approach, which places an emphasis on a priori research protocols (Perezgonzalez, 2015). These include calls for preregistration of experiments, a priori power analyses to be mandatory, and elimination of tests of statistical significance (Cumming, 2014; Gigerenzer, 2018). If we are correct that most researchers approach their studies in a manner consistent with Fisher (as Nickerson, 2000, concluded), then these recommendations are a mismatch as to how research should be conducted. The incompatibility between the advocated practices and researchers’ wants and needs likely will not produce meaningful improvement in the quality of research, but rather may have deleterious effects on scientific progress.