Experimental and Quasi-Experimental Designs for Generalized Causal Inference

I know a social psychologist who carries in his wallet a laminated copy of Campbell and Stanley’s (1966) list of threats to the internal validity of causal inferences. The list is not long—surely he has memorized it by now. But I suspect that the card offers epistemological comfort in times of doubt. Though ostensibly about education evaluation, Campbell and Stanley’s rarified analysis of causal inference, with its tables of X’s, O’s, +’s, and –’s, has an almost Platonic timelessness. Though a dozen years younger, Donald Campbell’s sequel with Thomas Cook (Cook and Campbell, 1979) seems more outdated, perhaps because its aspirations were more encompassing. Campbell, who died in 1996, envisioned an “experimenting society” that would make steady progress through systematic trial-anderror experimentation—in both the popular sense of “trying something new” and in the methodological sense of “random assignment to alternative treatments” (e.g., Campbell, 1969). But, as Campbell and Stanley recognized, there are often major political, ethical, or logistical barriers to random assignment, especially when no-treatment control groups or risky interventions are involved. Thus, Cook and Campbell (1979) devoted an entire monograph to quasi-experimentation— the use of designs that lack random assignment to condition, yet creatively use time and circumstance to permit stronger inferences than a simple observational analysis would allow. Where Campbell and Stanley provided a concise and focused overview of research designs, Cook and Campbell attempted a state-of-the-art treatise, drawing on then-recent work in the philosophy of science, the politics of bureaucracies, and the statistical analysis of quasi-experimental correlational data. Its perspective has been challenged by the growth of econometrics, the Reagan revolution, and the post-modernism “turn” in the academia. Random assignment is still the “gold standard” for causal inference, but field experiments rarely resolve uncertainties about program effects. Even with high levels of cooperation, randomization is rarely implemented with complete fidelity. Through the vagaries of real life, participants are misassigned or reassigned or they drop out, undermining many of the inferential benefits of the experiment. Ideally, we would like results to speak for themselves, but it seems that program evaluation will always need sophisticated statistical methods. Since Cook and Campbell’s (1979) book was published, there have been major developments in econometrics and statistical inference, including instrumental variable models, multi-level regression analysis, intention-to-treat modeling, the propensity score approach, selection bias modeling, and the computation of identification bounds on effect sizes. (Winship and Morgan [1999] provide a nice overview of these approaches for non-specialists. The relative merit of these approaches is debated in a series of comments by Rosenbaum, Manski, Robins, and Shadish and Cook in the 1999 and 2000 volumes of Statistical Science.) In Experimental and Quasi-Experimental Designs for Generalized Causal Inference, Shadish, Cook, and Campbell applaud these developments rather than resisting them. But they argue persuasively that such methods are a second-best solution when rigorous research designs are precluded or undermined. The stronger the design, the more transparent and explicit the results. In the end,