Socio-Demographic Study of Telephone Survey 

Nonrespondents 

 

Timothy Triplett, Natalie Abi-Habib 

Urban Institute 

2100 M Street, NW, Washington DC 

ttriplet@ui.urban.org, nabihabi@ui.urban.org 

Introduction 

Response rates for telephone surveys in the United States are declining (Curtin et al., 2005; Steeh et al., 

2001). The impact of nonresponse on estimates derived from a telephone survey depends on the extent to 

which nonrespondents differ from respondents on survey items of interest. Whether survey estimates are 

unbiased depends on the assumption that nonrespondents are missing at random (MAR). When 

nonrespondents differ from respondents the MAR assumption does not hold and nonresponse error may be 

present in some of the survey estimates. 

Little empirical evidence exists to evaluate the MAR assumption because any information about 

nonrespondents must come from administrative records or other auxiliary sources. The current research 

compares respondents and nonrespondents from the Urban Institute’s 2002 National Survey of America’s 

Families (NSAF). Auxiliary information about nonrespondents comes from 2000 census block group level 

data. A census block is the smallest geographical area for which census data are collected. A clustering of 

blocks forms the geographically larger census block group and at the next level several block groups 

combine to form a census tract. The Census Bureau’s goal is for each block group to contain 400 housing 

units, however block groups generally vary in size between 250 and 550 housing units. For confidentiality 

reasons census data is available publicly at the block group level 1

. 

Linking census block group data to the NSAF allows comparisons of block group characteristics of 

respondents to those of nonrespondents. For this research, neighborhood classifications are defined as 

being above or below the national averages of a set of characteristics. For example a neighborhood that has 

a Hispanic population above the national average of 12.5% is characterized as a Hispanic neighborhood. 

Learning the extent to which nonrespondents’ neighborhood characteristics differ from those of 

respondents sheds light on the appropriateness of the assumption that nonrespondents are missing at 

random and the presence of nonresponse bias in the sample. 

Data Sources

The 2002 NSAF survey has a dual-frame design (a large random-digit-dial of telephone households and 

small area sample of non-telephone households), features an over sample of low-income households with 

children, and is representative of the nation and of 13 states. The questionnaire consists of a short 

screening interview used to determine household eligibility and a longer extended interview, which gathers 

detailed information on the characteristics of sampled household members. The extended interview 

collects data about one or two sampled children, their parents or guardians as well as a randomly sampled 

childless adult if present in a subset of households (Abi-Habib et al., 2004). NSAF uses standard survey 

methods to reduce nonresponse, such as multiple contact attempts, refusal conversion, monetary incentives 

and an extended field period. Westat conducted the interviews from a random-digit-dial (RDD) sample of 

556,651 telephone numbers. This research uses data from only the RDD sample. 

 

1 United States Census Bureau. 2003. 2000 Geographic Terms and Concepts. 
The census block group data comes from both the census 2000 long and short form questionnaires. The 

data collected on the long form is available at the block group level, the smallest geographical level 

released publicly for confidentiality reasons. The long form is administered to 1-in-6 census households 

and collects data at both the household and person levels. The short form questionnaire collects information 

about gender, race, and age while the long form collects data about living arrangements, disability, personal 

and household income, educational enrollment, and other topics2

. 

Research Plan 

The current research seeks to explore differences between respondents and nonrespondents to the 2002 

NSAF by linking to data from the 2000 Census at the block group level. Census block group is identified 

for each NSAF household (respondent and nonrespondent households) using reverse directory call back 

services. These services identify a household’s address through its telephone number. Of the initial NSAF 

telephone sample of over half a million telephone numbers, 214,181 were identified as residential. Of these 

residential telephone numbers 75.7% (or 162,157) were matched to an address using reverse directory call 

back services and subsequently matched to the census block group in which the address is found. Future 

research may explore the remaining households by matching them to census data at the zip code level3

. 

However the current research only examines nonresponse in the block group matched NSAF households. 

Once matched to Census data, NSAF respondent and nonrespondent households were characterized as 

being located in block groups above or below the national average on the indicators below: 

• Racial composition 

• Linguistic isolation, foreign born and US citizenship 

• Poverty rate 

• Receipt of public assistance 

• Education 

• Urbanization indicator 

• Employment 

• Time spent living in neighborhood 

• Home ownership 

Neighborhood characteristics are explored for respondents and nonrespondents at the screener interview 

and extended interview levels. Nonrespondents at both levels are broken down into refusals and other 

nonresponse categories. The latter category includes passive refusals and also households that did not 

respondent because they were difficult to contact. Households selected for the extended interview are a 

subset of those who completed the screener interview. Criteria for selection into the extended interview 

sample include the presence of at least one householder under the age of 65, the presence of at least one 

child under the age of 17 and family income below 200% of the federal poverty line. Due to these 

sampling criteria analysis of nonresponse at the extended interview is less comparable to other national 

telephone surveys of the general population than analysis of nonresponse at the screener level. 

Comparison of Screener Respondents versus Nonrespondents 

Most telephone nonresponse occurs before the first question of a survey is even asked. Similar to other 

national telephone surveys the NSAF also experiences its highest rates of nonresponse when attempting to 

 

2 U.S. Census Bureau, 2000 Census of Population and Housing, Summary File 3: Technical Documentation

3

Using telephone exchange information the most populace zip code within an exchange could be assigned 

to each household then either use zip code level census data or assign the most populace block group within 

that zip code. 

2
complete the screener survey. Table 1 shows the variation in household screener completion, refusal, and 

other nonresponse rates by neighborhood type. The above average and below average sub-columns 

indicate the rate of survey response, refusal or other nonresponse for each neighborhood characteristic. 

Given the large sample size small differences found in this table are significant. 

Table 1. Household Screener Response, Refusal and Other Nonresponse Rates by Neighborhood 

 Characteristics

Neighborhood 

Characteristics 

Screener Completes 

Overall Rate= 65.9% 

(n = 106,804) 

Screener Refusals 

Overall Rate=26.6% 

(n = 43,134) 

Screener Other Nonresponse

Overall Rate = 7.5% 

(n = 12,219) 

 

 

 

 

Above Average Below Average Above Average Below Average Above Average Below Average 

Urban 64.4% 70.3% 27.3% 24.6% 8.4% 5.1% 

Rural 70.3 64.6 24.9 27.1 4.9 8.3 

White 65.9 65.8 27.3 24.9 6.9 9.3 

Black 66.0 65.8 25.5 26.9 8.5 7.3 

Asian 62.3 67.0 27.6 26.3 10.1 6.7 

Other Race 66.2 65.8 24.5 26.9 9.3 7.3 

Hispanic 64.5 66.2 25.8 26.8 9.7 7.0 

Spanish language isolation 64.4 66.2 25.9 26.7 9.7 7.1 

Asian language isolation 62.6 66.7 27.3 26.4 10.2 6.9 

Linguistically Isolated: 

Any Language 62.6 66.8 26.7 26.6 10.7 6.6 

Foreign Born 61.3 67.6 27.9 26.1 10.8 6.3 

Non-Citizen 61.8 67.2 27.2 26.4 11.0 6.4 

 

 

 

 

Same house in 1995 66.2 65.4 26.9 26.2 6.9 8.4 

Different house in 1995 65.1 66.4 26.5 26.7 8.4 6.9 

High school degree only 67.7 64.5 25.9 27.2 6.5 8.3 

College degree or higher 62.9 68.0 28.7 25.1 8.4 6.9 

Employed – all 65.1 66.5 27.3 26.0 7.6 7.5 

Employed – females 65.4 66.2 27.0 26.3 7.6 7.5 

HH receives public 

assistance 66.7 65.8 24.4 26.8 8.9 7.4 

 

 

Less than 50% of FPL 66.5 65.6 25.1 27.2 8.4 7.2 

Less than 100% of FPL 67.2 65.3 24.8 27.4 8.1 7.3 

Less than 200% of FPL 68.0 64.8 24.4 27.8 7.6 7.48 

Owner occupied 66.6 64.7 27.0 25.9 6.5 9.4 

Renter occupied 64.3 66.7 26.2 26.8 9.5 6.5 

 Rural neighborhoods tend to have higher household screener completion rates than urban neighborhoods, 

although the reason appears to be the result of having a lower other nonresponse rate. Over half of the 5.7 

percentage point increase in the rural completion rate can be accounted for by the 3.4 percentage point 

decrease in the other nonresponse rate. Maybe more telling is that the other nonresponse rate for urban 

neighborhoods is almost twice as high as it is in rural neighborhoods. This implies that nonresponse in 

urban neighborhoods is not only a problem of high refusal rates but of making contact with respondents. 

3
Surprisingly there is no difference in screener completion rates between White and Black neighborhoods. 

This is surprising given that NSAF weighting adjustments give greater weights to black non-Hispanics. 

This anomaly could mean that by increasing the share of black non-Hispanics through post stratification 

there becomes a slight over representation of black respondents living in black neighborhoods. This 

potential problem is similar to the 1997 NSAF nonresponse analysis (Groves and Wissoker, 1999) where 

there was some evidence that by treating black respondents and nonrespondents the same the weight may 

have overstated the number of black low-income households. Though this potential problem was 

investigated the report did not find evidence that this in fact occurred. While completion rates are roughly 

the same in Black and White neighborhoods the source of nonresponse varied as White neighborhoods had 

a higher refusal rate while Black neighborhoods suffered from a higher other nonresponse rate. This 

indicates that potential respondents in Black neighborhoods are more difficult to reach but once reached 

they can be more cooperative. 

 

Both Asian and Hispanic neighborhoods had lower completion rates than White and Black neighborhoods. 

These differences may be explained by the low completion rates found in neighborhoods with high levels 

of linguistically isolated households and by the low completion rates of neighborhoods with above average 

numbers of foreign-born and non-citizen residents. In fact the screener completion, refusal and other 

nonresponse rates of Hispanic and Asian neighborhoods are mirrored by the language isolation measures 

for those groups. The screener completion rate in Spanish language linguistically isolated neighborhoods is 

higher than in Asian language linguistically isolated neighborhoods but may be attributable to the 

availability of a Spanish language translation of the NSAF survey. The NSAF is available in English and 

Spanish only. Presumably households that are linguistically isolated in a language other than Spanish 

would be unable to respond or even to refuse the survey request, explaining the higher other nonresponse 

rate and lower refusal rate of Asian language linguistically isolated neighborhoods. 

Transient households, those that had moved to the current neighborhood in the last 5 years, did not differ 

much in their willingness to complete the NSAF screener than more established households. However, 

neighborhoods with above average numbers of transient household had a higher other nonresponse rate 

than neighborhoods where people had lived for more than 5 years. Here the small finding is fairly intuitive, 

since often more transient respondents are thought to be more difficult to contact and interview. While this 

research supports the notion that nonresponse increases in transient neighborhoods, the difference of 1.1 

percentage points is modest. 

Education tended to have a negative impact on response rates. The data show that screener completion 

rates in less educated neighborhoods, where respondents tended to have only a high school degree or 

equivalent, are nearly 5 percentage points higher than in neighborhoods with above average numbers of 

college graduates. Highly educated respondents tend to not only refuse the survey request at a higher rate 

than less educated respondents but are also more difficult to contact as evidenced by their higher refusal 

and other nonresponse rates. 

Neighborhoods with higher overall employment and higher female employment had only slightly lower 

screener completion rates than the overall average for the survey. This is a positive finding given that 

previous NSAF nonresponse analysis (Black and Safir, 2000; Triplett et al., 2002) found that employment 

negatively impacted response rates. Both the long field period (9 months) and the selection of any adult to 

complete the screener probably offset the difficulty that most national telephone surveys experience in 

attempting to reach employed respondents. 

Neighborhoods with more than the national average number of households receiving public assistance as 

well as poor neighborhoods had slightly higher completion rates than the overall average screener 

completion rate for the survey. This finding seems somewhat counterintuitive however; there may be topic 

interest effect among poorer households that increases their propensity to respond to the NSAF. Research 

has demonstrated that respondent interest the topic of a survey can affect survey participation decisions 

(Groves et al., 2004). 

4
Finally, neighborhoods that have a high concentration of homeowners tend to have higher screener 

completion rates than high rental occupancy neighborhoods. This difference may be explained by the 

higher other nonresponse rate in homeowner neighborhoods and indicate that renters are more difficult to 

contact. This finding is similar to the trend seen among respondents in transient neighborhoods where home 

rental rather than home ownership is likely to be more common. 

 

 

 

 

Comparison of Extended Interview Respondents versus Nonrespondents

The households that completed an extended interview are a subset of the households that completed the 

screener interview. The research on extended interview nonresponse on the NSAF is less comparable to 

other household surveys since not all-high income and childless households are asked to complete an 

extended interview. In addition, the respondent was often the most knowledgeable adult for a child in the 

household. However, the findings in this section should be useful for those studies collecting data on 

children or low-income populations. 

Table 2 shows variation in response rates by neighborhood characteristics at the extended interview level. 

As before the above and below average columns indicate the rates of response, refusal or other nonresponse 

to the extended interview by neighborhood type. The sample sizes are still large so that again even the 

small differences found in this table are significant. 

Table 2. Household Extended Response, Refusal and Other Nonresponse Rates by Neighborhood 

 Characteristics 

Neighborhood 

Characteristics 

Extended Completes 

Overall Rate= 84.3% 

(n=33,919) 

Extended Refusals 

Overall Rate=10.4% 

(n = 4,179) 

Extended Other Nonresponse

Overall Rate = 5.3% 

(n = 2,137) 

Above Average Below Average Above Average Below Average Above Average Below Average 

Urban 83.4% 86.6% 10.6% 9.7% 6.0% 3.7% 

Rural 86.9 83.4 9.7 10.6 3.5 6.0 

White 85.4 81.9 10.4 10.4 4.2 7.7 

Black 82.2 84.9 10.9 10.2 6.9 4.9 

Asian 82.1 85.0 10.9 10.2 7.0 4.8 

Other Race 82.3 84.7 9.0 10.6 8.8 4.7 

Hispanic 82.4 84.8 9.3 10.7 8.3 4.5 

Spanish language isolation 82.3 84.8 9.3 10.7 8.5 4.6 

Asian language isolation 82.3 84.8 10.2 10.4 7.5 4.8 

Linguistically Isolated: 

Any Language 81.7 85.1 9.8 10.6 8.5 4.4 

Foreign Born 80.8 85.6 10.7 10.3 8.6 4.2 

Non-Citizen 81.1 85.3 10.3 10.4 8.6 4.3 

 

 

 

 

Same house in 1995 84.1 84.5 11.0 9.6 4.9 5.9 

Different house in 1995 84.6 84.1 9.6 11.0 5.9 4.9 

High school degree only 85.1 83.7 10.2 10.5 4.7 5.8 

College degree or higher 83.6 84.7 11.5 9.8 4.9 5.5 

Employed – all 84.8 83.9 10.6 10.2 4.5 5.9 

Employed – females 85.0 83.7 10.4 10.4 4.6 5.9 

5
HH receives public 

assistance 84.0 84.3 9.0 10.5 7.0 5.2 

 

 

Less than 50% of FPL 83.8 84.5 9.5 10.8 6.7 4.7 

Less than 100% of FPL 84.1 84.4 9.5 10.8 6.4 4.8 

Less than 200% of FPL 84.5 84.1 9.3 11.1 6.2 4.7 

Owner occupied 84.9 83.3 10.7 10.0 4.4 6.8 

Renter occupied 83.2 84.9 10.0 10.6 6.9 4.5 

A comparison of the screener and extended response and nonresponse rates shows that completion rates are 

much higher for the extended interview even though the extended interview often takes more than 45 

minutes to complete. This is evidence that length of the questionnaire has little impact on response rate. 

Additionally, other nonresponse makes up over half of the overall extended nonresponse but only 28% of 

the overall screener nonresponse. This is likely a result of the added difficulty of scheduling a long 

interview with a chosen respondent whereas the screener interview was completed by any householder 18 

years of age or older. 

While differences in the extended completion rate by neighborhood characteristics were generally smaller 

than the differences found at the screener level, they tend to follow the same patterns as the screener 

interview completion rates. There were a few exceptions such a decline in the completion rates of Black 

neighborhoods and very poor neighborhoods (below 50% of Federal Poverty Level). However, 

neighborhoods with high employment rates maintained response rates above the overall rate for the survey 

while at the screener level the opposite is true. This finding may be attributable to the NSAF data 

collection design of administering the survey to the adult in the family who is most knowledgeable about 

the sampled child(ren). This adult also provides information about him or herself and any spouse or partner 

in the household, thereby relieving the need for scheduling time to administer the survey to the spouse or 

partner. 

Regression Analysis 

In our descriptive analysis of neighborhood characteristics most of the findings have shown that some 

relationship between the neighbor type and the propensity to responds exists. To further the research a 

regression model is used to answer the question “Does not knowing the percentage of neighbor that has a 

certain characteristic help us predict a household’s likelihood of participating. The result of this model is 

shown in Table 3 at the top next page. 

The results indicate that knowing any of the block group characteristics used in this model would provide 

some insight in predicting nonresponse. These results still show that rural areas predict higher response 

but the factor seems smaller when compared with some of the other predictors of higher response (below 

poverty, owner occupancy, and receipt of public assistance). The reason households located in rural areas 

are more likely to respond may have more to do with characteristics about rural areas rather than the 

household simply being located in a rural area. The only real surprise is that the sign of the high school 

only coefficient is the opposite of what one would have guessed based on the descriptive analysis. In this 

model having a greater percentage of either more college graduates or more high school only respondents 

predicts lower response rates. Finally, the descriptive analysis did not find much association between 

being Black or employed. The regression confirms this in that these two characteristics have only slight 

predictive powers. 

6
Table 3. Linear Regression 

 

 

Unstandardized 

Coefficients 

Standardized 

Coefficients t Sig. 95% Confidence Interval for B 

B Std. Error Beta Lower Bound Upper Bound 

(Constant) .567 .016 34.758 .000 .535 .598

Black -.034 .007 -.012 -4.670 .000 -.049 -.020

Rural .044 .004 .028 11.493 .000 .036 .051

Hispanic -.115 .015 -.034 -7.587 .000 -.144 -.085

Different house 

in 1995 .058 .010 .015 5.615 .000 .038 .079

Less than 100% 

of FPL .124 .020 .023 6.247 .000 .085 .163

Other Race .151 .030 .022 5.022 .000 .092 .211

Owner occupied .098 .007 .043 13.193 .000 .083 .112

High school 

degree only -.123 .022 -.022 -5.594 .000 -.166 -.080

HH receives 

public assistance .299 .110 .008 2.725 .006 .084 .513

College degree 

or higher -.212 .014 -.066 -15.171 .000 -.239 -.185

Employed - all .053 .012 .011 4.280 .000 .029 .077

 Dependent Variable: Screener Disposition (0=nonrespondent, 1=respondent) 

 Independent Variables: Percentage of block with that that characteristic (values 0 to 100) 

 

 

Summary & Conclusions 

Most of the findings only show modest differences in completion rates by neighborhood type. However, 

even differences of a couple percentage points could introduce enough bias into some survey estimates so 

as to affect the overall findings. In addition, the differences in completion rates by neighborhood 

characteristics are probably somewhat conservative in that the households that were not matched to block 

group data are probably more transient and therefore would tend to not respond to the survey. 

Some of the potential bias associated with different response by neighborhood type is dealt with by the 

under coverage and post stratification weights (Bethlehem, 2002) developed for the NSAF. The NSAF 

methods and response rate evaluation (Brick et al., 2003) report did some evaluation of the potential bias 

owing to nonresponse. However, this report was fairly vague as to the degree of how much increasing 

nonresponse rates are biasing survey estimates. This paper goes one step further in that it identifies types 

of neighborhoods that have differing response propensity (rural, Asian, Hispanic educated, owner 

occupancy rates). A natural next step would be to look at the completed cases block group characteristics 

using the survey weights. Unfortunately, for the screener where most nonresponse occurs poststratification information is unavailable However, post-stratification is not as effective if the 

nonrespondents differ from respondents which does appear at least partially true. 

References 

Abi-Habib, Natalie, Adam Safir, and Timothy Triplett. 2004. NSAF 2002 Methodology Series Report No. 

1: Survey Methods and Data Reliability. Washington D.C.: Urban Institute. 

Bethlehem, Jelke G. 2002. Weighting Nonresponse Adjustments Based on Auxiliary Information. In 

Survey Nonresponse, edited by Robert M. Groves, Don Dillman, John L. Eltinge, and Roderick 

J.A. Little, Chapter 18, 275-87. New York: Wiley. 

Black, Tamara, and Adam Safir. 2000. Assessing Nonresponse Bias in the National Survey of America's 

Families. In Survey Methodology Proceedings. American Statistical Association Meetings. 

7
Brick, Michael J., D. Ferraro, Rauch C., and T. Strickler. 2003. 2002 NSAF Response Rates and Methods 

Evaluation. Tech. Rept. no. 8. Washington, D.C.: Urban Institute. 

Curtin, Richard, Stanley Presser, and Eleanor Singer. 2005. Changes in Telephone Survey Nonresponse 

over the Past Quarter Century. Public Opinion Quarterly 69, no. 1, Spring: 87-98. 

Groves, Robert M., Stanley Presser, and S. Dipko. 2004. The Role of Topic Interest in Survey Participation 

Decisions. Public Opinion Quarterly 68, no. 1, Spring: 2-31. 

Groves, Robert M., and Douglas Wissoker. 1998. 1997 NSAF Methods Report No. 7 - Early Nonresponse 

Studies of the 1997 National Survey of America's Families. Washington, D.C.: Urban Institute. 

Steeh, C.G., Nicole Kirgis, Brian Cannon, and Jeff DeWitt. 2001. Are They Really as Bad as They Seem? 

Nonresponse Rates at the End of the 20th Century. Journal of Official Statistics 17: 227-47. 

Triplett, Timothy, Adam Safir, Kevin Wang, Rebecca Steinback, and Simon Pratt. 2002. Using a Short 

Follow-up Survey to Compare Respondents and Nonrespondents. In Survey Methodology 

Proceedings. American Statistical Association Meetings. American Statistical Association. 

U.S. Census Bureau. 2002. 2000 Census of Population and Housing, Summary File 3: Technical 

Documentation. United States Census Bureau. 

———. 2003. 2000 Geographic Terms and Concepts. United States Census Bureau.

8