DO OPEN-ENDED QUESTIONS MEASURE “SALIENT” ISSUES?

Closed-ended questions dominate most interview schedules. Yet the almost exclusive use of this form did not arise because open-ended questions, its major competitor, proved to be weak indicators of public opinion. Instead, responses from open-ended questions proved more difficult and expensive to code and analyze than those from closed-ended questions. Although such practical concerns are important, the real task of survey researchers is to measure public opinion accurately. Using an experimental design, this article tests whether open-ended questions measure the important concerns of respondents-one of the long-claimed advantages of this format. The results, on balance, show that open-ended comments reflect such concerns, suggesting that pollsters may want to include more of these questions in their surveys of public opinion. In the early days of survey research, controversy swirled around whether pollsters should use closed-ended or open-ended questions (Lazarsfeld 1944). That debate has long been settled with closed-ended questions emerging as the overwhelming choice of survey researchers. A major reason for this decision was that fixed-format questions were easier to ask, code, and analyze than their free-format counterparts (Schuman and Presser 1981). Such pragmatic concerns are important, but the crucial issue should be whether open-ended questions provide important insights about public opinion. Yet the preference for closedended questions did not arise because of any systematic data that cast doubt on the ability of the open-ended questions to measure the attitudes of the public accurately. This article does not seek to resurrect the debate between the open and the closed form, since the latter is an integral part of our efforts to track public opinion. Instead, it hopes JOHN G. GEER is associate professor of political science at Arizona State University. The author thanks James Geer, Kim Kahn, Pat Kenney, Warren Miller, referees, and the editor of this journal for help with the preparation of this manuscript. An earlier version of the article was presented at the Midwest Political Science Association, Chicago, April 5-7, 1990. I bear sole responsibility for the interpretations offered here. Public Opinion Quarterly Volume 55.360-370 C) 1991 by the American Association for Public Opinion Research All rights reserved 0033-362X/91/5503-0006/$2.50 Open-ended Questions and Salience 361 to provide a better idea of the capacity of open-ended questions to measure the attitudes of the public. Proponents of open-ended questions, for instance, have long claimed that this format taps concerns that are important to respondents' political calculations (Campbell et al. 1960; Kelley 1983; Knight 1985; RePass 1971). RePass (1971, p. 391), for instance, argues that these kinds of questions permit citizens "to define [their] own issue space by naming issues that were salient to [them]." Kelley (1983) concurs, contending that the open-ended format allows researchers to tap the diverse attitudes of the American electorate. If these proponents are correct, it would call into question the infrequent use of the free-format question in surveys. But not all scholars agree with these arguments, claiming instead that open-ended questions measure "superficial" concerns (Smith 1989, p. 84). Lodge, McGraw, and Stroh (1989, pp. 400-402) doubt that one set of open-ended questions asked by the National Election Studies (NES), the so-called like/dislike questions, accurately tap the electorate's salient attitudes about the candidates (see also Stroh, Lodge, and Callaghan 1990). Respondents, they argue, are unlikely to probe their memories in enough detail to remember correctly the pieces of information that generated their overall judgment of the candidates. Smith (1989, p. 84) expands on this point, writing that openended "comments do not reveal fundamental attitudes. Instead, the responses reveal more casual likes and dislikes, such as what the respondent has read in the papers recently or hears on television or in a conversation with a friend." Thus, disagreement exists over the kinds of insights open-ended questions can yield about public opinion. But it is unclear whether critics or proponents are right, since there are few data that explicitly address their respective contentions. Using an experimental design, I test whether responses to open-ended questions tap the "salient" or "superficial" concerns of respondents.' In an experiment, the presence of salient and superficial information can be manipulated, allowing one to determine whether open-ended questions reflect attitudes that are important to respondents. I relied on undergraduates as subjects, which, of course, limits the generalizability of these findings. But given the many threats to internal validity that might arise in distinguishing between salient and superficial comments, an initial experiment on students becomes an attractive way to test these rival claims. 1. Scholars have used the term "salient" to mean different things (see Schuman, Ludwig, and Krosnick 1986). Some view the term as indicating something in the immediate environment that may or may not be important. Others equate the term with importance. In this article, the latter meaning is used.