The External Validity of Treatments: A Comparison of Natural and Survey Experiments

Survey experiments help establish causality, but scholars do not know how closely the treatments mimic natural phenomena. This study compares survey experiments and a natural experiment on the same topic. In two survey experiments providing information about Medicare, we observe double-digit learning effects. In contrast, most respondents in our contemporaneous natural experiment show little evidence of learning. Consistent with our expectations, the only people who showed comparable levels of learning to respondents in our survey experiment were individuals exposed to Medicare facts in their media source of choice as well as people who were uncertain about the facts from the very beginning. Our conclusion is that survey experiments, at least on this topic, generate effects that are only observed among parts of the population who are likely to be exposed to treatment messages or predisposed to accept them. 1 Assistant Professor, Department of Political Science, Florida State University, jjerit @ fsu.edu. 2 Assistant Professor, Department of Political Science, Florida State University, jason.barabas @ fsu Randomized social science experiments permit causal inference, but a common criticism concerns the lack of external validity when, as is often the case, the subjects are university undergraduates. Selecting individuals randomly from a national pool offsets these concerns and allows generalizations to the larger population. However, even in nationally representative survey experiments we should be concerned about the external validity of the treatments—i.e., the correspondence between the stimuli and the political phenomena they seek to emulate. Unlike some disciplines (e.g., economics or psychology) where the primary goal of experiments is testing some aspect of a theory, “Political scientists use survey experiments to identify how citizens make decisions and respond to real-world political objects, in order to enhance understanding of politics” (Gaines, Kuklinski, and Quirk 2007, 2; emphasis original; also see Kinder and Palfrey 1993, 27). In this way, the correspondence between the experimental stimuli and real world referent is assumed. A significant treatment effect in a survey experiment is viewed as saying something about the direction, if not the rough magnitude, of effects that might be expected to occur in the real world. This paper examines whether this is in fact the case. We do so by taking advantage of a naturally occurring event that was expected to increase media attention to Medicare. Each spring, the trustees overseeing Medicare and Social Security release their future funding estimates. We leveraged this situation by bracketing the event with survey experiments that delivered information about Medicare’s financial status to randomly selected samples of the American public. As a result, this study offers a rare opportunity to compare survey experiments with the real world phenomena they seek to emulate. In the case we examine, the only people who showed comparable levels of learning to respondents in our survey experiment were individuals exposed to Medicare facts in their media source of choice as well as people who were uncertain