Estimating the Nonresponse BiasDue to Refusals in Telephone Surveys

This paper attempts to aid the process of accumulating the necessary information for making more informed judgments about the effects of nonresponse under different conditions. Two measures, which permit quantifiable nonsubjective assessment of the effects of nonresponse on sample estimates, are introduced and are used to examine the effects of respondent refusals in a random-digit-dialed general population telephone survey of over 1,200 households as the response rate is increased from 74.5 percent to 86.8 percent. By applying these measures under a wide range of conditions, the adequacy of various response rates may be assessed and more rational decisions made about the costs and benefits of devoting extraordinary resources to minimizing nonresponse. Michael J. O'Neil is a postdoctoral fellow at the Survey Research Center of the Institute for Social Research and Lecturer in Sociology at the University of Michigan. This research was supported in part by a grant (EPP75-08963) from the National Science Foundation for which the author was principal investigator. Drafts of this paper have been read by Norman Bradburn, Don Dillman, Fredric DuBow, Andrew Gordon, Marilyn Johnson, Richard McCleary, Stanley Presser, Kent Smith, and Seymour Sudman. An earlier version of this paper was presented at the American Association for Public Opinion Research Annual Meetings, Roanoke, Virginia, June 1978. Public Opinion Quarterly ? 1979 by The Trustees of Columbia University Published by Elsevier North-Holland, Inc. 0033-362X/79/0043-0218/$1.75 This content downloaded from 157.55.39.116 on Thu, 02 Mar 2017 05:51:54 UTC All use subject to http://about.jstor.org/terms NONRESPONSE BIAS IN TELEPHONE SURVEYS 219 techniques tend not to be systematically applied, and when they are there is almost always residual nonresponse. Proposals for dealing with nonresponse have generally involved either (1) adjusting for nonresponse bias ex post facto, or (2) estimating the maximum potential distortion it can cause. Daniel (1975) reviewed several classic adjustment procedures.2 Recently suggested adjustments have included extrapolating observed differences between easy-to-reach and difficult-to-reach respondents (Filion, 1975) and weighting respondents according to known demographic population parameters (Mandell, 1974). While the particular assumptions differ, all these adjustments are problematic in that they assume precisely what is at issue; that is, the nature of the differences between respondents and nonrespondents. Cochran (1963) suggested an alternative approach for surveys with low nonresponse which requires no such assumptions. By assuming extreme values for nonrespondents, the maximum bias that could be caused by nonresponse under varying conditions can be specified. Fuller (1974) and Wayne (1975) used Cochran's formulas to estimate the size of these expanded "confidence intervals" under a variety of response rates, sample sizes, and estimated proportions. Looking at these under a range of realistic conditions, however, we find that the calculated "confidence intervals" are almost always far too wide to permit most substantively meaningful inferences. Furthermore, this approach also tells us nothing of the probable direction of the bias, only how large it could be. To reduce the width of these "confidence intervals" to the point where they might be usable or to specify the direction of the impact of nonresponse, empirical data are required on tion, cover letters, anonymity, and cash incentives. In addition, Kanuk and Berenson (1975) dealt with the effectiveness of including return envelopes, various forms of questionnaire reproduction, and specifying deadline dates. Among the most promising techniques in terms of the results obtained have been the methods developed by Dillman and his associates (Dillman, 1972; Dillman et al., 1974). A similar literature on increasing response rates for telephone surveys is almost nonexistent. The absence of a comparable body of knowledge on which to draw may help explain why attempts to increase response rates in telephone surveys have often been less successful (Dillman et al., 1976). The comparative absence of research on increasing response rates over the phone is probably due to the fact that mail surveys have traditionally been more extensively used and have usually experienced higher nonresponse rates. Even if more "tailored" methods for phone surveys were developed, however, there would inevitably be some nonresponse and little means of assessing its impact. None of these methods in any way address the question of the extent of the bias induced by various levels and sources of nonresponse. 2 Specifically, Daniel reviews the Hansen and Hurwitz (1946) proposal to subsample nonrespondents, the Politz-Simmons (1949, 1950) times-at-home procedure, the Kish and Hess (1959) replacement procedure, the Bartholomew (1961) two-call technique, Hendricks' (1956) extrapolation method, Ericson's (1967) Bayesian approach, and Dalenius' (1955, 1957) variable selection probability method for selection within the household. This content downloaded from 157.55.39.116 on Thu, 02 Mar 2017 05:51:54 UTC All use subject to http://about.jstor.org/terms 220 MICHAEL J. O'NEIL what actually does (not just what could) happen to various estimates under differing conditions with less than complete response. Most of the empirical examinations of the actual effects of nonresponse have almost invariably been based on specialized samples not easily generalizable: migratory duck hunters (Filion, 1975), elderly males in Iowa (Goudy, 1976), college alumni from a particular institution (Reuss, 1943), science talent search participants (Edgarton et al., 1947), and contributors to the National Committee for an Effective Congress (Bachrack and Scoble, 1967). It is not, however, only the samples that have been specialized. Almost all empirical examinations of nonresponse bias (including all the above) have been based on mail surveys. Not only was Dunkelberg and Day's (1973) examination of the effects of increasing numbers of callbacks in personal interviews a rare exception to this rule, it was one of the few empirical examinations of nonresponse to employ a general population sample. The objectives of this paper are twofold: first, to present comparable data for telephone surveys within the context of a general population survey; second, because there is considerable evidence that the characteristics of refusers and "not-at-homes" differ (Colombotos, 1962; Mayer and Pratt, 1966; Wilcox, 1977), to isolate the effects of refusals. Data Collection Procedures The data analyzed in this paper are taken from a general population telephone survey of 1,209 Chicago households. Several measures were employed to insure maintenance of equal selection probabilities. First, random-digit-dialing was used to include households with unlisted telephones. Second, verification of the number reached was required to discover misdials and account for the telephone company switching system's erratic disposition of nonworking numbers dialed. Third, exhaustive callbacks of persons not at home (up to 20 calls at staggered times) were undertaken to insure that all households were reached and to allow for separation of the effects of "refusals" and "unavailables." Finally, persons who refused to be interviewed on the first contact3 were mailed a "persuasion letter"4 and recontacted. How effective were these procedures in minimizing nonresponse? 3When it was unclear whether a respondent's request to "call me back later" represented a disguised refusal or a genuine request to be contacted at a more convenient time, respondents were taken literally and were not considered refusals. Only when it became obvious that this was really a refusal was it treated as such. For this reason, the "second contact" referred to in this discussion had been preceded in some cases by more than a single phone contact. 4 Because addresses could not be obtained for the unlisted telephone numbers, these letters could only be sent to the portion of the sample with listed telephones. Both listed and unlisted households, however, were recontacted by phone. This content downloaded from 157.55.39.116 on Thu, 02 Mar 2017 05:51:54 UTC All use subject to http://about.jstor.org/terms NONRESPONSE BIAS IN TELEPHONE SURVEYS 221 Results are presented in Figure 1. From 1,392 eligible households 5 there were 1,037 completed interviews6 after the first contact and 1,209 after the second. These represent completion rates of 74.5 percent and 86.8 percent, respectively. Ideally, one would like to compare groups C and D in Figure 1. Unfortunately, very little is known about group D except that they refused, at least once, to be interviewed. We attempt to approximate this comparison by comparing group C with group E, which was interviewed after having refused on the first attempt.7 We cannot be sure how groups E and F differ, Figure 1. Interview Attempt Outcomes