Interviewer Expectation Effects: A Replication and Extension

Interviewers' evaluations of the general difficulty of a survey have no effect on the cooperation rate they obtain, but do affect responses to individual questions. Although these effects are not large, interviewers who expect more difficulty with a study, or who expect certain parts of it to be difficult to ask, tend to get higher nonresponse rates to sensitive questions and lower estimates of sensitive behavior. Such expectations affected one of the experimental manipulations in the informed consent study, but not enough to account for the effects of the experimental variable. Eleanor Singer is a Senior Research Associate at the Columbia Center for the Social Sciences and Editor of The Public Opinion Quarterly. Luane Kohnke-Aguirre is Sampling Manager at Yankelovich, Skelly, and White, Inc. Charles F. Cannell, Program Director at the Survey Research Center of the Institute for Social Research, University of Michigan, refereed the evaluation of this article. Public Opinion Quarterly ? 1979 by The Trustees of Columbia University Published by Elsevier North-Holland, Inc. 0033-362X/79/0043-0245/$1.75 This content downloaded from 157.55.39.243 on Thu, 06 Oct 2016 04:38:45 UTC All use subject to http://about.jstor.org/terms 246 SINGER AND KOHNKE-AGUIRRE However, in concluding their analysis, Sudman and his colleagues note: "Small sample sizes keep the data in this paper from being conclusive. This problem is inherent in research about interviewer expectations. ... Replication will be the only way to establish precisely the nature and magnitude of interviewer-expectation effects." This paper presents the results of such a replication, together with some extension of the findings previously reported. Two major findings were documented by Sudman and his colleagues. First, interviewers' expectations about the general difficulty of the study were related to levels of reporting. Those who anticipated difficulties with the questionnaire obtained lower percentages of respondents acknowledging certain activities than interviewers who thought the survey would be very easy. Second, interviewers who expected their respondents to underreport behavior also obtained lower levels of reporting. Other interviewer expectations-for example, expectations of difficulty with each section of the interview, and expected respondent uneasiness-had no additional impact on the data, nor did such expectations affect the response rate to the survey. The replication to be reported here matches the original study more closely than many other such attempts. It is based on (1) the same survey organization-NORC, (2) 27 of the 59 interviewers who worked on the earlier study (plus 43 additional interviewers), (3) a survey instrument virtually identical to that administered in the earlier study, and (4) identical questions for measuring interviewer expectations, plus some additional questions. The primary purpose of the later study was to assess the effect of informed consent procedures on response rate and response quality in social surveys. Specifically, it was designed to investigate the effects of three factors that together may be said to constitute "informed consent" procedures in such surveys. The first of these factors was the amount of information given to respondents ahead of time about the content of the interview. Half the respondents were given a brief, vague description of the survey as a study of leisure time and the way people are feeling. The other half were given a fuller description of the interview, which contained a large number of questions generally considered sensitive. The second factor that was experimentally varied in the study was the assurance of confidentiality given to respondents. One-third of the respondents were told nothing at all about the confidentiality of their replies; one-third were given an absolute assurance of confidentiality ("Of course, your answers will remain completely confidential"); one-third were given a qualified assurance of confidentiality ("Of course, we will do our best to protect the confidentiality of your answers, except as required by law"). This content downloaded from 157.55.39.243 on Thu, 06 Oct 2016 04:38:45 UTC All use subject to http://about.jstor.org/terms INTERVIEWER EXPECTATION EFFECTS 247 The final factor varied in the study was whether or not a signature was required to document consent, and if so, whether the request for a signature came before or after the interview. The three factors described above were combined in a 2 x 3 x 3 factorial design, yielding 18 different introductions to respondents. These introductions were stapled to the household enumeration folders assigned to each interviewer, along with instructions for answering questions and objections, also specifically tailored to each version of the introduction. Interviewers (all of them female) were instructed to read the introduction in its entirety to the selected respondent, and to give standardized replies to questions that might be raised about content, confidentiality, or signature. The sample was a national probability sample of 2,084, drawn within 50 primary sampling units of NORC's master sample. Each sample line was assigned a randomly selected experimental treatment. Interviewers were required to list each household and to select the appropriate respondent for a given household according to a sampling table. Each interviewer was assigned 31 sample lines, and each assignment included all 18 versions of the introduction. The interview schedule was substantially the same as that used a year earlier by Sudman and Bradburn in a study of the effects of question threat and question wording. Following some questions about conventional leisure activities were sections dealing with emotional well-being and mental health, drinking, marijuana use, and sexual behavior. A number of demographic items, among them income, were also included. The questionnaire was designed to permit investigation of the effect of informed consent procedures on eight types of questions: sensitive and nonsensitive survey-specific and general questions about attitudes, and the same types of questions about behavior.3 Before beginning field work, interviewers were asked to complete a questionnaire designed to measure their attitudes toward the survey instrument, the informed consent variables, and the expected difficulty of completing their assignment.4 We begin by describing these 3 For additional details about the design of the study as well as its major findings, see Singer (1978a, 1978b, 1979). 4 The research design called for 67 interviewers, each with an assignment of 31 cases in one geographical area (segment). However, several changes had to be made in this design. One segment was dropped without replacement; three segments had interviewing done by more than one person; and three interviewers worked in two segments rather than in one. The net effect of these various shifts was that 70, rather than 67, interviewers worked on the study. The number of eligible households (i.e., after elimination of vacant and no-dwelling units) in each interviewer's assignment varied from 1 to 57, with most hovering around 30. In the analysis of response rates below, four interviewers, each with fewer than nine eligible households, were eliminated; and seven interviewers, each with fewer than eight completed interviews, were eliminated from all subsequent analyses. This content downloaded from 157.55.39.243 on Thu, 06 Oct 2016 04:38:45 UTC All use subject to http://about.jstor.org/terms 248 SINGER AND KOHNKE-AGUIRRE initial attitudes and expectations, and then go on to examine the effect of these on selected outcome variables. Initial Attitudes and Expectations Asked how difficult they expected the survey as a whole to be, interviewers distributed their responses among four of the five answer categories, tending more frequently toward the less difficult end of the scale. Twenty-four percent expected the survey to be very easy; 42 percent, moderately easy; 18 percent neither easy nor difficult; and 15 percent, moderately difficult, with one interviewer selecting the "very difficult" category. Expectations about the difficulty of individual sections within the questionnaire showed a predictable pattern (Table 1), with questions about such topics as sports expected to elicit virtually no difficulty, and questions about intercourse and masturbation, considerable difficulty. Income was perceived as relatively easy to ask, although as it turned out the question about earned income elicited a nonresponse rate second only to the questions on masturbation. These findings for the most part parallel those reported by Sudman and his colleagues.5 Interviewers were also asked how uneasy they expected respondents to be made by the same topics. When these topics are ordered in terms of the percentage of interviewers saying "not at all uneasy," the question order is almost identical to that shown in Table 1. Table 1. Interviewers' Expectations of Difficulty with Different Sections of Questionnaire