The Rise of Statistical Significance Testing in Public Administration Research and Why This Is a Mistake (2013)

ABSTRACT:The growth of statistical significance testing in articles published in the Public Administration Review for the period 1945 through 2008 is examined. Comparisons with sister journals, the American Political Science Review and the American Journal of Political Science, show this growth to be less emphatic and of more recent origin than theirs. That public administration researchers are not yet quite entrenched in this practice is good because statistical significance testing, with its focus on p-values, is largely ritualistic and adds almost nothing of scientific value to a study. The justification for this conclusion is presented. Instead of the infatuation with p-values, we encourage public administration researchers to report and interpret sample statistics, effect sizes, and their confidence intervals in empirical work. This offers a better prospect for developing cumulative knowledge within the field.INTRODUCTIONIt is evident that the current practice of focusing exclusively on a dichotomous reject-nonreject decision strategy of null hypothesis testing can actually impede scientific progress. I suspect that the continuing appeal of null hypothesis significance testing is that it is considered to be an objective scientific procedure for advancing knowledge. In fact, focusing on p values and rejecting null hypotheses actually distracts us from our real goals: deciding whether data support our scientific hypotheses and are practically significant. The focus of research should be on our scientific hypotheses, what data tell us about the magnitude of effects, the practical significance of effects, and the steady accumulation of knowledge (Kirk, 2003, p. 100).A heated debate over the merits of quantitative versus qualitative research methodologies has occurred recently in a number of public administration and public policy journals, especially the Journal of Public Administration Research and Theory and Administration and Society (see, e.g., Gill, J., & Meier, K. J. (2000); Luton, 2007, 2008; Lynn, Heinrich, and Hill, 2008; Meier and O'Toole, 2007). Because our paper deals with methodology, we would like for readers to know of our own perspective on this debate from the outset.Both of us are staunch advocates of a postpositivist philosophy of science, credited to Bhaskar (1978, 1979), called critical realism. While incorporating aspects of both, critical realism provides an alternative philosophy to those found wanting-positivism/empiricism on the one hand and relativism/interpretivism on the other (Sayer, 2000). In essence, this philosophy states that, first, the world exists independently of our knowledge of it. Second, the goal of science is to create genuine, but always fallible, knowledge about the world. Because knowledge is produced socially, and hence is theory laden, does not make it theory determined (Sayer, 2000). Third, all theories concerning knowledge claims must be subject to critical evaluation; knowledge is not immune to empirical check (Sayer, 1992). It is through this critical evaluation of competing theories that the scientific community, over time, is able to decide on which ones to retain and which to discard. That is, not all knowledge is equally fallible (Smith, 2006).Critical realism, then, sees value in both quantitative and qualitative research approaches to the examination of knowledge claims. Both camps can help in the triangulation of findings. It must be acknowledged, however, that throughout our careers we have been engaged chiefly with quantitative research. But this should not be read as a ringing endorsement of a quantitative research orientation. This brings us to the subject of this paper, one which finds much at fault with mainstream quantitative work. Especially its obsession with p-values.In many social science journals the p-value from a statistical significance test is a staple of empirical research. This same index is now appearing with greater frequency in the pages of the Public Administration Review (PAR). …