Editors and Researchers Beware: Calculating Response Rates in Random Digit Dial Health Surveys

When evaluating survey research, editors and reviewers often look for high response rates as evidence that the results are representative and valid. For example, 89 percent of journal editors consider response rates somewhat or very important when evaluating whether to publish an article that uses survey data (Carley-Baxter et al. 2005). Response rates are particularly important for random digit dial (RDD) surveys as rates for these surveys have been falling precipitously in recent years (Bunin et al. 2007). Because many health services researchers rely heavily on RDD surveys as an important source of data, there is significant incentive to report the highest possible response rate. Although the calculation of response rates for many types of surveys is straightforward, calculation of response rates in RDD surveys is more complicated and can be influenced by factors that are not readily apparent. The main problem is adjusting the response rate for cases in which the eligibility of the sampled unit is unknown. Eligibility might be unknown because the phone was never answered after repeated calls or the respondent hung up before any information could be obtained. It is important to estimate how many of these cases are likely to be eligible. Most methods estimate the proportion of these cases that are eligible based on information about the cases whose eligibility could be determined. The specifics of how eligibility is considered in response rate calculation formulas can lead to markedly different results (Smith 2009). The American Association for Public Opinion Research (AAPOR) (AAPOR, 2008) and the Council of American Survey Research Organizations (CASRO) (Centers for Disease Control and Prevention 2010) have developed the two most commonly used response rate calculation methods for RDD surveys. These methods are quite similar in many respects. However, they differ on how cases of unknown eligibility are treated. The AAPOR1 method is likely to yield substantially lower rates than those calculated using the CASRO method. These differences are particularly important for health services researchers because the Centers for Disease Control and Prevention's (CDC) Behavioral Risk Factor Surveillance System (BRFSS) survey uses the CASRO response rate calculation technique, and BRFSS is often used as a standard by which response rates are judged in other RDD health surveys. The AAPOR method is the approach recommended by the major survey research and polling professional association. Features of the sampling design used to generate the list of phone numbers to call might also affect the number of cases of unknown eligibility that will ultimately be considered in the final response rate calculations. Specifically, many survey sample vendors “prescreen” the samples that they sell to survey research centers. Common practices include choosing “blocks” of numbers with more known eligibles, using an automated system to precall numbers to detect a working tone, and screening out known business numbers. Some survey research companies and researchers will add those prescreened numbers back when estimating the proportion of phone numbers of unknown eligibility that are assumed to be eligible. However, these prescreened numbers are likely to have very different eligibility probabilities than cases of unknown eligibility. Including the cases that were removed by prescreening in the response rate calculations would likely underestimate the proportion of unknown eligibles that are considered to be eligible, leading to inflated response rates. When calculating or reporting response rates, researchers seldom mention, and may be unaware of, prescreening and its effect on the response rates. No published articles in the health services research literature investigate the implications of using different approaches for handling cases of unknown eligibility in RDD surveys. Using two separate samples from the Aligning Forces for Quality Consumer Survey (Alexander et al. 2011, 2012), this study demonstrates that different approaches to handling cases of unknown eligibility contribute to substantial differences in response rates. First, we calculate response rates using both the AAPOR and CASRO methods. Second, we calculate response rates for samples where prescreened numbers are included and excluded in the final response rate calculations. This study also discusses implications for editors and researchers, as well as recommendations on how response rates should be reported.