Cognitive Aspects of Survey Measurement and Mismeasurement

During the past  years, survey methodology has undergone a paradigm shift. The old paradigm was based on a statistical model that focused on the effects of survey errors on the estimates derived from survey data. The new paradigm is based on a social scientific model that focuses on the causes of survey errors. Several developments have helped bring about this shift—the application of methods and concepts from cognitive psychology to the reduction of survey measurement error, the development of new computerized methods of data collection, and the increase in concern about measurement and nonresponse as sources of error in survey estimates. The new paradigm has little to say about the topics, such as sampling error, which were central to the old one; similarly, the old paradigm had little to say about how to reduce or prevent errors, a major concern for the new one. Thus, the two paradigms do not clash so much as complement each other. One of the deepest changes in survey research over the last  years has been the shift in the reigning paradigm guiding methodological research on surveys. The earlier paradigm, which was completely dominant when I joined the field some  years ago, was based on a statistical model of survey error. Perhaps the fullest expression of that model is found in the work of Hansen, Hurwitz, and Madow (), and Hansen, Hurwitz, and Bershad (). This has been a very successful paradigm and the fact that it’s still useful, even vital, can be seen in more recent work, such as Lessler and Kalbeek’s () Nonsampling Error in Surveys or Groves’s () Survey Costs and Survey Errors, where the outlines of the old paradigm can still be discerned just under the surface. Like any paradigm, the statistical paradigm has its limitations and, from our current vantage point, one of its key limitations is its focus on the consequences rather than the causes of survey errors. The central concepts in the statistical paradigm—variance and bias—refer to the different effects that survey errors This article is based on the speech that the author gave on the occasion of his receiving the  Helen Dinerman Award, during the WAPOR’s th Annual Conference in St Petersburg Beach, Florida, in May .  World Association for Public Opinion Research                 can have on survey estimates. These are extraordinarily important and useful concepts but they don’t tell us much about how errors arise or about how to prevent (or at least reduce) the errors in the first place. To address those issues requires well-developed theories about the sources of survey errors, theories about how people decide whether to take part in surveys and theories about how they come up with answers to the questions. The new paradigm—which I’ll call the scientific paradigm for surveys—attempts to provide such theories. One manifestation of the new paradigm has been the attempt by Groves, Couper, and their colleagues to develop detailed theories about the sources of nonresponse (e.g., Groves, Cialdini, & Couper, ; Groves & Couper, ; Groves, Singer, & Corning, ). These theories aren’t statistical models about the impact of nonresponse on survey estimates, although they do have implications for the question of when nonresponse is likely to bias survey results. Instead, the theories focus on why nonresponse occurs—on who is likely to be hard to reach, on how interviewers try to extend the interaction with potential respondents and how they tailor what they say to fend off objections, on how interest in the topic affects willingness to take part in a survey. This work is beginning to have real impact on survey practice; for example, it has led to the development of new training procedures for interviewers to reduce the refusal rates in surveys. It’s hard to imagine how something similar would have come out of the older statistical paradigm. But an even more striking manifestation of the conceptual shift from a statistical to a scientific paradigm for surveys has been the movement to apply concepts and methods from cognitive psychology and related disciplines to reducing measurement error. This movement is sometimes referred to as the Cognitive Aspects of Survey Methodology movement, or CASM for short. It’s hard to provide an exact date for the beginning of the paradigm shift—I suspect that’s true for most fields—but it’s clear that CASM has been gathering steam for about  years or so. My own involvement in the movement began in , when I attended a seminar sponsored by the Committee on National Statistics here in the United States that brought together survey methodologists and cognitive scientists to look at the underlying problems that produced errors in surveys (see Jabine, Straf, Tanur, & Tourangeau, , for a summary of the CNSTAT seminar). Similar conferences were held about the same time in the United Kingdom and in what was then West Germany (both Sudman, Bradburn, & Schwarz, , and Tourangeau, Rips, & Rasinski, , provide brief histories of the CASM movement). My own work in this area culminated in a book, The Psychology of Survey Response (), with Lance Rips and Ken Rasinski. This book tries to summarize the theoretical and empirical work carried out under the CASM banner. I’m a little embarrassed to admit it, but this work can be accurately summed A    S M   M     up in a single sentence: reporting errors in surveys arise from problems in the underlying cognitive processes through which respondents generate their answers to survey questions. To unpack this just a little, the basic insight of the CASM movement is that respondents give inaccurate or unreliable answers because they don’t really understand the questions, can’t remember the relevant information, used flawed judgment or estimation strategies, have trouble mapping their internal judgments onto one of the response options, or edit their answers in a misleading way before they report them. These problems reflect both the lifetime of experience in answering questions in everyday life that we bring to surveys (and the habits we’ve built up to get us through everyday conversations smoothly) and the shortcuts we take to reduce the cognitive burden imposed by interviews and similar tasks. Different researchers have given more or less emphasis to different portions of this picture, but whether they have emphasized the conversational roots of the cognitive processes in surveys (e.g., Schober, ; Schober & Conrad, ; Schwarz, ), the cognitive processes themselves (e.g., Burton & Blair, ; Conrad, Brown, & Cashman, ), or motivational and ability factors affecting how the cognitive processes are carried out (e.g., Krosnick, ; Krosnick & Alwin, ), a lot of very prominent survey researchers have jumped aboard the CASM bandwagon. This new paradigm has both encouraged and reflected a shift in concern about the most important sources of error in surveys. The old paradigm mostly focused on various forms of sampling error; the new paradigm emphasizes measurement error and, to a lesser degree, nonresponse error instead. Of course, the old paradigm simply didn’t have that much to say about measurement and nonresponse error, particularly about how they arise. Likewise, the new paradigm is unlikely to have much impact on our understanding of sampling error or on the practice of sampling. This nonoverlap between the two paradigms is probably a good thing—the new paradigm complements the old one rather than replacing it. Similarly, I think that a shift in emphasis from sampling to nonsampling errors will also prove to be a useful development. My own guess is that until we get to very low levels of aggregation, measurement error is a far more important contributor to total survey error than sampling error is; sampling errors still get the lion’s share of the attention and resources because we know how to measure them and how to reduce them, but things are gradually changing. If anything, this shift in emphasis is likely to accelerate in the coming years as we continue to invent new tools for collecting survey data. The new computer-assisted methods of data collection—including audio computerassisted self-interviewing (or audio-CASI), its telephone counterpart interactive voice response, and Web and e-mail surveys—raise a host of new measurement issues. We are only just beginning to have a sense of when different methods of data collection yield similar results and when they diverge and of the key                variables that determine whether there is agreement or disagreement across modes of data collection. The issues raised by Web surveys are particularly hot right now, partly because Web surveys are primarily visual and use a much wider range of visual material (photographs, drawings, video clips) than has been true of surveys in the past. The issues raised by the new methods are likely to remain central to survey methodologists for some time to come. I personally look forward to continuing to work on them.