The connection between varying treatment effects and the crisis of unreplicable research : A Bayesian perspective

We connect the replication crisis in social science to the default model of constant effects coupled with the flawed statistical approach of null-hypothesis significance testing and the related problems arising from the default model of constant treatment effects. We argue that Bayesian modeling of interactions could lead to a general improvement in the communication and understanding of research results. Moving to Bayesian methods offers an opportunity for introspection into how social scientists interact with social phenomena to produce knowledge. 1. Living with variation and uncertainty For the past few years, there has been a growing crisis in social science, and in biology, medicine, and other statistics-dependent fields, that many claimed research findings are fragile, unreliable, cannot be replicated, and do not generalize outside of the lab to real-world settings (Ioannidis, 2005). Arguably the crisis is most pronounced within psychology (see Pashler and Wagenmakers, 2012). While some view this crisis as being due to factors such as publication bias (see Francis, 2012), we see many of these problems as the result of applying twentieth-century statistical models in a twenty-first-century research environment. An essential part of science is that we can come to agreement about new knowledge. But this is threatened by a torrent of published studies of varying quality. To simplify somewhat, we feel the world should be real with objective properties (indeed, even children have been found to have an “essentialist” view of the world as having deep structure despite surface appearances; see S. Gelman, 2005) and it is natural for us to think these properties can be studied in a deterministic fashion. When previously trusted studies do not replicate, this is upsetting: science should give us facts because that’s how we want it to work, and we are unsatisfied when the world doesn’t conform to our obsession with using modern tools for dicing up the world into individualized knowledge-objects (or, as we say, “stylized facts”). Beyond all this, we as social scientists have difficulty following scientific practice as generalized from the world of classical physics, for (at least) two reasons: • Psychological and social processes show much more variability than the usual phenomena in the physical sciences. The strength of an iron bar might be calculable to a high degree of accuracy from a formula derived from first principles, but there will not be anything like the same precision when predicting prices, political opinions, the contact patterns of friends, and so on. • Even to the extent that patterns can be discovered from social data (for example, the YerkesDodson curve in psychology, the Phillips curve in macroeconomics, the seats-votes curve in politics, and various predictive models of consumer behavior), these patterns can and do ∗For Journal of Management. Editors Michael Zyphur and Fred Oswald provided so many suggestions that they should be considered as coauthors of this paper. We also thank the National Science Foundation for partial support of this work. †Department of Statistics, Columbia University, New York, N.Y. change over time, and they can look different in different countries and for different groups of people. Social science even at its best is contingent in the sense that physical models are not. These concerns relate to the problem of non-replication of social science research. Instead of focusing on ways to procure the certainty often attributed to the hard sciences, perhaps social scientists can address the non-replication problem by changing what we want to get out of our research, accepting that we can gain knowledge without the certainty we might like, either for individual predictions or for the larger patterns we seek to discover. If effects are different in different places and at different times, then episodes of non-replication are inevitable, even for very well-founded results. Once we realize that effects are contextually bound, a next step is to study how they vary. In policy analysis, this implies different effects for different people under different scenarios (Dehejia, 2005), and it connects to goals such as personalized medicine (or, less pleasantly, individuallytargeted marketing pitches). From the perspective of statistical inference, however, fitting models of varying treatment effects presents technical challenges, as interactions are typically estimated with less accuracy than main effects. Bayesian methods are relevant in this effort for several reasons. Most directly, Bayesian analysis uses the prior distribution, allowing the direct combination of different sources of information, which is particularly relevant for inferences about parameters that are not well estimated using data from a single study alone. Bayes plays well with uncertainty: the posterior distribution represens uncertainty about any set of unknowns in a model. And, in a setting where the appropriateness of the model is itself uncertain, it is useful that Bayesian models are “generative”; that is, one can simulate replicated datasets from the model and compare them to the observed data (Gelman et al., 2013, chapter 6). Before considering Bayesian methods, we first discuss general concerns about unreplicable research. 2. Unreliable findings We can roughly divide the problems of non-replicability into three categories: 1. Joke research, for example the attempts of Bem (2011) to prove that he found ESP and the claims of Kanazawa (2005, 2007) that nurses are more likely to have daughters, big strong men are more likely to have sons, etc. Undoubtedly some of these sorts of taboo-busting efforts will turn out to be correct, but we can consider them as joke research because they fall outside the usual bounds of scientific communication. The individual research articles look real enough (indeed, Bem’s article was, notoriously, published in a top psychology journal) but the research programs run in parallel with, rather than as part of, mainstream science: as sociologist Jeremy Freese writes, these hypotheses are “perhaps more ‘vampirical’ than ‘empirical’—unable to be killed by mere evidence.” 2. Outright fraud, scientific misconduct, and simple mistakes, for example Diederik Stapel’s fake data, Mark Hauser’s questionable data interpretation, and the data misalignment that Anderson and Ones notoriously refused to retract (see Goldberg, Lee, and Ashton, 2008a, 2008b, for the story and Anderson and Ones, 2008, for the non-retraction). The concern here is that most published work never gets checked. Add to this the pressure to publish in high-prestige journals and you get a motivation to publish false or sloppy claims. We have published mistakes ourselves (Gelman, 2013b), not from any pressure but because we can’t