Why Hypothesis Tests Are Essential for Psychological Science

In a much-anticipated move, Psychological Science recently announced important changes in its publication practices (Eich, 2014); specifically, the changes promote open science (i.e., open data, open materials, and preregistration) and recommend that researchers report confidence intervals (CIs) instead of p values. As part of these initiatives, Cumming (2014) argued that inference by CI should constitute a “new statistics” in psychology (see also Grant, 1962) and will avoid the pitfalls of null-hypothesis significance testing (NHST). Embracing the new statistics, Cumming recommended a shift to estimation as “usually the most informative approach” (p. 11) and stated that “interpretation [of data] should be based on estimation” (p. 26). We broadly agree with the recommendations for more quantitative thinking, more openness and transparency, and an end to p values. Nonetheless, the benefits of estimation have been overstated, and the mistaken idea that estimation is superior to hypothesis testing is unfortunately becoming the conventional wisdom in psychology. This conventional wisdom is being perpetuated by the American Psychological Association’s (2010) publication manual, the new statistical guidelines for journals of the Psychonomic Society (n.d.), the Society for Personality and Social Psychology Task Force on Publication and Research Practices (Funder et al., 2014), and now also Psychological Science. However, estimation alone is insufficient to move psychology forward as a science; proper hypothesis-testing methods are crucial. Scientific research is often driven by theories that unify diverse observations and make clear predictions. For instance, in physics, one might test for the existence of the Higgs boson (Low, Lykken, & Shaughnessy, 2012). In biology, one might compare various phylogenetic hypotheses about how species are related (Huelsenbeck & Ronquist, 2001). In psychology, one might test whether fluid intelligence can be improved by training (Harrison et al., 2013). Testing a theory requires testing hypotheses that are consequences of the theory, but unfortunately, this is not as simple as looking at the data to see whether they are consistent with the theory. There are three necessary components to testing a theory: First, one must know what one would expect of the data if the theory were true; second, one must know what one would expect if the theory were false; and third, one must have a principled method for using the data to make an inference about the theory. The second and third components are crucial. Inferring support for a theory on the sole basis of agreement between the observations and the theory is a logical fallacy (known as the converse error; Popper, 1935/2005); having no principled method for inferring support leaves one with only ad hoc rules subject to one’s own biases. The difficulties inherent in using estimation to test theory can be illustrated by the example chosen by Cumming (2014, p. 21). Velicer et al. (2008) tested a theoretically motivated model of smokers’ readiness to stop smoking. They predicted the strength of association between 15 inventory scales and the stage of change from the transtheoretical model of behavioral change (Prochaska & Velicer, 1997). To assess the model, they determined whether CIs contained the predicted values. Eleven of the 15 predicted values were included in the 95% CIs, and Velicer et al. concluded that these results provide “overall support for the theoretical model” (p. 602). This assessment raises two issues. First, it is arbitrary. If 10 of the 15 CIs included the predicted values, would the results also support the theory, or instead refute it? If one instead used 99% CIs, would positive results for 12 of the 15 predictions be enough to support the theory? This arbitrariness arises because CIs offer no 525969 PSSXXX10.1177/0956797614525969Morey et al.Hypothesis Testing Versus Estimation research-article2014