RESPONDENT CUES THAT SURVEY QUESTIONS ARE IN DANGER OF BEING MISUNDERSTOOD 1

INTRODUCTION In ordinary conversation, people who are uncertain about what their parmers mean don't necessarily ask for clarification explicitly. Sometimes they wait to hear more, as if all were well, in hopes that something their partners say later will clarify what was meant. Whether it is because people want to avoid challenging their partners, because they want to avoid looking stupid, because they want to maintain the flow of the conversation, or because they honestly don't recognize that they haven't understood, people don't ask for clarification on many occasions when it would be rational to do so. For example, college students in one study rarely asked questions about math problems even when the problems contained anomalous and irrelevant information (Graesser & McMahen, 1993). In another study (Schober & Conrad, 1997), survey respondents answering factual questions misunderstood what the survey designers meant by questions as much as 73% of the time, but they asked for clarification only 38% of the time. This poses a problem for survey researchers who want to be sure that respondents interpret questions as intended. The common wisdom is that interviewers should read questions exactly as worded, leaving the interpretation up to respondents so that interviewers won't bias the answers (e.g., Fowler & Mangione, 1990). But a common practice in survey organizations that consider themselves to follow standardization is for interviewers to read scripted definitions for key question concepts if respondents ask for them. (Technically, this practice isn't strictly standardized, in that some respondents get clarification and others don't, but it is nonetheless commonly considered to fall within the boundaries of standardized practice). The problem is that if respondents don't ask for clarification when they need it, how can interviewers provide clarification appropriately? The problem is greater than it at first appears. In order to ask an appropriate clarification question, survey respondents must overcome more than just their fears of looking ignorant. They must recognize that their own way of conceiving of a question concept may not coincide with the survey designer's. They must also recognize that their personal circumstances warrant their pursuing clarification (see Schober, 1999; Schober & Conrad, 2000). Consider a respondent to the Current Point of Purchase Survey who is asked "Last year, did you purchase or have expenses for household furniture? "; and imagine that during the past year she bought a floor lamp. According to the official definitions of the sponsoring agency (the Bureau of Labor Statistics), the floor lamp shouldn't be counted as a furniture purchase. For the respondent to ask the right clarification question (something like "Do you count a floor lamp as a piece of household furniture? ") she must recognize that her notion of furniture may differ from the survey designers'. She must also recognize that a floor lamp may be an instance that the survey designers might not count as a purchase of household furniture. From our point of view, knowing when to ask for clarification and asking the right question are substantial metacognitive feats, and much remains unknown about people's metacognition. It certainly doesn't seem particularly surprising, given the complexity of what survey respondents are faced with, that they don't always ask for clarification when they ought to.