Modeling the Difference in Interview Characteristics for 

Different Respondents October 2009

John Dixon 

Bureau of Labor Statistics, Room 1950 

2 Massachusetts Ave., NE 

Washington, DC 20212-0001 

Abstract 

The survey process as measured by paradata may be different for different 

respondents. Those initially reluctant to participate may be convinced by the 

interviewer to cooperate, or the reluctance may produce attrition. Difficulty to 

contact respondents may be related to interviewer effort and busy respondent 

schedules, or it may be a form of reluctance. This paper will use mixed models to 

attempt to identify subsets of respondents whose paradata relates differently to 

survey outcomes and measures of survey quality. 

Key Words: Survey, paradata, interviewer

1. Introduction 

 The Consumer Expenditure Survey (quarterly) is a household survey which 

provides part of the “market basket” of consumer expenditures, which are the 

basis of the CPI as well as other indices. Sampled housing units in the Quarterly 

are interviewed for 5 consecutive quarters. These interviews are referred to as 

“time-in-sample” (TIS) 1 to 5. 

 A very useful feature was added in 2005 to collect detailed call history data 

(Bates, 2004). The interviewer records times and outcomes of attempted contacts, 

problems or concerns reported by reluctant households, and strategies used to gain 

contact or overcome reluctance. This provides a very rich source for studying the 

interview process, which is only lightly used in this study. 

 Dixon (2006) found that estimates of nonresponse bias weren't impacted much 

by the addition of call history variables. Those interviews which required a larger 

number of calls where the interviewer changed modes had lower expenditures (-

39.4). This effect was partially offset by those interviews that required more calls 

but where respondents who reported no problems had higher expenditures (27.0). 

Those interviewers who reported "no strategy" for attempted contact ended up 

with lower expenditures (-66.2) and those who changed modes during the contact 

process ended up with higher expenditures. 

 Multilevel models have been used to examine effects of interviewers on survey 

outcomes (Tucker and Dixon, 2000). The current study will investigate if there 

are patterns of differences in the interview process and the interview outcome. 

This relationship may be different because several factors; differences in personal 

style of the interviewer (some strategies may work better for some interviewers 

than for others), differences in the local characteristics (urban vs other), and 

differences in the respondents. 

Section on Government Statistics – JSM 2009

160
2. Data Sources 

 The Call History Instrument used in conjunction with the Consumer 

Expenditure Survey (quarterly) will be combined from 2006 through 2008 to 

provide the paradata for this study. Since interviewers were one of the levels of 

analysis those interviewers which only had a few interviews were excluded from 

the analysis. Many were probably supervisors stepping in where needed. The 

interviews from the second interview were used for the bulk of the data, with 

refusals in that or subsequent interviews used as an outcome variable. 97317 

households were used for the analysis, with 617 interviewers. Interviewers which 

had fewer than 5 households or didn’t provide CHI data were excluded. 

3. Methods 

 A mixed model (using MPlus) was be used to examine if interviewer effects 

varied in terms of the relationship between paradata and survey refusal. The 

paradata consisted of the outcomes of each attempted interview. The coefficients 

for each interviewer were explored using a two-stage density based cluster 

analysis (SAS 9.1).. The patterns selected from the cluster analysis were 

examined for differences in household characteristics. 

4. Results 

 The most common concern expressed by respondents was “busy” (Table 3), 

followed by “schedule difficulties”, and “not interested”, which was also most 

predictive of a refusal outcome. Other notable concerns were “time the interview 

takes” and “privacy concerns”. 

 The multilevel model showed moderate effects for interviewers (variance of 

0.156 with a standard error of 0.066) in terms of their coefficients relating 

respondent concerns to refusal. The logistic models for the relationships between 

concerns and refusal (Table 1), shows some strong effects. The univariate logistic 

models showed positive relationships between most of the concerns and refusal 

during some of the interviews. “Family issues” (issues, which was not 

significantly related to refusal) and “intends to quit” are the two related to not 

refusing. The multivariate model showed some coefficients which reversed 

direction or became non-significant when adjusting for the other variables. The 

univariate estimates could be interpreted as the relationship between those 

concerns and refusal, while the multivariate estimates could be interpreted as the 

unique relationship of those concerns beyond the other concerns. The 

combination gives a more complete picture of the relationship between concerns 

and refusal. The most common concern “busy”, showed a strong relationship 

with refusal, but didn’t contribute anything beyond the other variables. “Not 

interested”, which also was a frequent concern (almost 10% of cases) had a strong 

relationship with refusal even after adjusting for the other variables. Counter to 

expectations, the concern “planning to quit” (quit) showed a strong negative 

relationship to refusal. This might have been related to increased efforts by the 

interviewer to persuade the respondent to stay with the survey. 

 The cluster analysis of the coefficients indicated 3 clusters. The first cluster 

contained 84% of the interviewers and would have dominated any analysis which 

Section on Government Statistics – JSM 2009

161
ignored interviewer effects. “Not interested” was most strongly related to refusal, 

while “No concerns” was most related to cooperation. The first cluster differed 

from the other clusters in having a lower relationship with “survey doesn’t apply 

to me” (notapp). The interviews were more likely to be done primarily by 

personal visit rather than telephone compared with the other clusters. 

 The second cluster contained 2.34% of interviewers. It had interviewers who 

were more successful at overcoming concerns about being “too busy”, but were 

less successful at overcoming concerns about the survey “not applying to me” 

(notapp) or “planning to quit” (quit). Note that the relationship at the household 

level between planning to quit was negative, at the interviewer level it becomes 

positive, indicating that differences in interviewers and the composition of the 

households they contact would be expected to produce different results. The 

second cluster interviewers had a higher percentage of households which planned 

to quit, so it would be interesting to see if it is the households or the interviewers 

where the effect is taking place. The second cluster had a slightly higher 

proportion of household which owned their own home, but a lower percentage 

which used records to respond to the expenditure questions. The respondent was 

also less likely to be male than for the other clusters. 

 The third cluster had 13.66% of interviewers. They which were more 

successful at dealing with “privacy”, but had more difficulty with respondents 

which “broke appointments” (noshow), “didn’t understand the survey” (question), 

“not apply to me” (notapp), or were “hostile” (hostile). The respondent was more 

likely to use records than the other clusters. 

Table 1: Multilevel analysis 

Logistic MODEL RESULTS 

 Name Bivariate 

Estimate(SE) 

Multivariate 

Estimate 

Two-Tailed 

SE 

 Est/SE PValue 

 NOTINT 1.016(0.057) 0.472 0.072 6.579 0.000 

 HUNGUP 2.927(0.191) 1.617 0.223 7.233 0.000 

 HOSTILE 1.612(0.163) 0.640 0.214 2.990 0.003 

 VOLUNTAR 1.099(0.068) -0.266 0.098 -2.721 0.007 

 PRIVACY 1.916(0.111) 0.776 0.142 5.464 0.000 

 ANTIGOV 0.520(0.080) -0.300 0.105 -2.862 0.004 

 QUESTION 1.214(0.187) 0.166 0.256 0.647 0.518 

 NOTAPP 2.359(0.143) 1.405 0.168 8.388 0.000 

 OTHHH 0.627(0.120) -0.151 0.163 -0.928 0.353 

 SAMEINF 0.685(0.257) -0.225 0.324 -0.693 0.488 

 BUSY 0.898(0.084) -0.108 0.113 -0.960 0.337 

 NOSHOW 0.464(0.067) -0.275 0.091 -3.014 0.003 

 SCHEDULE 1.776(0.079) 0.860 0.100 8.572 0.000 

 TIME 1.226(0.136) 0.729 0.194 3.769 0.000 

 MEMBER 1.024(0.123) 0.229 0.158 1.447 0.148 

 ISSUES -0.464(0.433) -0.812 0.521 -1.558 0.119 

 SAMEFR 1.440(0.528) 0.141 0.799 0.176 0.860 

 TOOPERS 0.580(0.174) -0.564 0.236 -2.392 0.017 

 TOOMANY 0.108(0.358) -1.281 0.464 -2.761 0.006 

 TOOLONG 2.002(0.171) 1.008 0.221 4.559 0.000 

 QUIT -1.833(0.056) -1.538 0.065 -23.532 0.000 

Section on Government Statistics – JSM 2009

162
Table 2; Bivariate Logistic coefficients relating concerns to refusal 

aggregated at the interviewer level 

Cluster1 Cluster2 Cluster3 Overall 

 notint 5.91 8.53 6.23 6.02 

 busy 1.16 -2.77 1.18 1.06 

 time 0.60 -1.41 2.20 0.78 

 noshow -0.09 0.49 4.72 0.64 

 schedule -0.96 -5.64 0.68 -0.84 

 voluntary 4.22 4.82 7.17 4.67 

 privacy 2.65 4.22 2.40 2.65 

 antigov 4.22 1.01 4.56 4.19 

 question -0.25 -1.55 3.00 0.20 

 notapp 0.97 7.95 5.36 1.79 

 hungup 4.64 8.04 4.02 4.64 

 hostile 5.18 2.44 7.67 5.48 

 othhh 2.29 -1.61 2.93 2.29 

 member -0.53 -1.41 0.44 -0.41 

 issues 0.69 -2.77 0.44 0.57 

 samefr -0.55 0.28 0.56 -0.37 

 sameinf -0.66 -1.43 -2.19 -0.91 

 toopers 0.84 0.50 0.09 0.72 

 toomany 0.08 0.19 -0.92 -0.07 

 toolong -0.56 0.80 -0.83 -0.56 

 quit 2.64 9.14 2.53 2.79 

 noconc -3.64 -5.43 -3.40 -3.65 

 other -0.86 -3.76 1.51 -0.58 

Table 3; Mean rates of Chi responses 

Cluster 1 Cluster 2 Cluster 3 Overall 

 notint 0.09 0.11 0.08 0.09 

 busy 0.20 0.28 0.19 0.20 

 time 0.08 0.10 0.08 0.08 

 noshow 0.02 0.01 0.03 0.02 

 schedule 0.13 0.11 0.16 0.13 

 voluntar 0.05 0.04 0.05 0.05 

 privacy 0.07 0.08 0.09 0.07 

 antigov 0.02 0.03 0.03 0.02 

 question 0.03 0.07 0.04 0.03 

 notapp 0.01 0.01 0.01 0.01 

 hungup 0.02 0.01 0.02 0.02 

 hostile 0.01 0.01 0.01 0.01 

 othhh 0.01 0.02 0.01 0.01 

 member 0.03 0.01 0.03 0.03 

 issues 0.03 0.04 0.03 0.03 

 samefr 0.00 0.00 0.01 0.01 

 sameinf 0.02 0.02 0.02 0.02 

 toopers 0.02 0.02 0.02 0.02 

 toomany 0.03 0.03 0.04 0.04 

 toolong 0.02 0.01 0.02 0.02 

 quit 0.02 0.05 0.03 0.03 

 noconc 0.69 0.59 0.65 0.68 

Section on Government Statistics – JSM 2009

163
5. Discussion 

 Other studies have found a relationship between CHI data and refusal on both 

the National Health Interview Survey and the Consumer Expenditure surveys 

(Bates 2004, Bates et. al. 2008). This study extended their research to examine 

differences in the interview experience and subsequent refusal based on the 

concerns expressed by the respondents. 

 The studies by Bates (2004), and Henley and Bates (2006) found that the 

number of concerns was a more important predictor of refusal than particular 

concerns for the NHIS. They found privacy concerns, the voluntary nature of the 

survey, “not interested”, and “Survey takes too long” to be the primary concerns 

for refusers. This study found a similar overall pattern, but added “schedule 

difficulties” to the list. Some unexpected effects were found with negative 

relationships to refusal for respondents which had “family issues” (issues) or 

“intends to quit survey” (quit). The family issues were thought to make the 

difficult process of reporting expenditures more difficult, but although nonsignificant, that didn’t seem to be a problem for respondents. The respondent 

“intending to quit” probably triggered a strong anti-refusal response from the 

interviewer (which isn’t captured by the CHI instrument), resulting in a decrease 

in the likelihood of refusal. When aggregated at the interviewer level, the 

concerns were more predictive of refusal, so interviewers which had a higher 

proportion of respondents wanting to quit had more refusals. Similarly, 

“scheduling difficulties” (schedule) showed a positive relationship to refusal at 

the respondent level, but interviewers who had a higher proportion of scheduling 

difficulties had fewer refusals. 

 The cluster analysis had large within cluster variances, so there was a lot of 

variability between interviewers and their experiences with respondents concerns. 

The clusters reflect different profiles of concerns and their relationship to refusal. 

The second cluster represented a small group of interviewers with busy 

homeowners who didn’t think the survey applied to them. The third cluster 

reported more privacy and scheduling concerns, and were more likely to refuse 

after complaining about not having time, breaking appointments, asking questions 

about the survey, and making hostile comments. 

Limitations and future research 

 The lack of CHI data for some interviewers (100 out of 97317 households) 

may mask some of the interaction patterns experienced by the survey participants, 

but it represents enough of the interactions to be useful. The rarity of some of the 

concerns pose a problem in modeling, but since they are so rare they don’t present 

much of a problem for possible new approaches for avoiding refusal. 

 Future research could include an examination of strategies the interviewers use 

in persuading reluctant households to see if there are similar groupings in terms of 

outcomes. The effect of reluctance on data quality may also be of interest, since 

one of the groups used fewer records in reporting expenditures. 

Section on Government Statistics – JSM 2009

164
References 

Bates, N., Contact Histories: A Tool for Understanding Attrition in Panel 

Surveys., A paper presented at the 59th Annual AAPOR Conference, May 

11-13, 2004 

Bates, N., Dahlhamer, J., and Singer, E., “Privacy Concerns, Too Busy, or Just 

Not Interested? Exploring the Relationship Between Doorstep Concerns 

and Nonresponse Bias”, Journal of Official Statistics, Vol. 24.4, 

December, 2008. 

Chopova, B., Edgar, J., Gonzalez, J., King, S., McGrath, D., and Tan, L., 

“Assessing nonresponse bias in the CE Interview Survey: A summary of 

four studies”, presented at the Federal Committee for Statistical 

Methodology Workshop on How to Do Nonresponse Bias Analyses in 

Household and Establishment Surveys, June 10, 2009. 

Dahlhamer, J., Simile, C., and Taylor, B., “Exploring the Impact of Participant 

Reluctance on Data Quality in the National Health Interview Survey 

(NHIS)”, paper presented at the Statistics Canada Symposium; 

Methodological Issues in Measuring Health, 2006. 

Dixon, J., “The Effects of Item and Unit Nonresponse on Estimates of Labor 

Force Participation”, Paper presented at the Joint Statistical Meetings, 

N.Y., N.Y., 2002. 

Dixon, J., “Models of Nonresponse Bias, Survey Quality, Paradata, and 

Household Characteristics in the Consumer Expenditure Survey.”, A paper 

presented at the International Survey Nonresponse Workshop, Omaha, 

Nebraska, 2006. 

Henley, M., and Bates, N., “Using Call Records to Understand Response in 

Longitudinal Surveys”, paper presented at the American Association for 

Public Opinion Research, 2006. 

Tucker, C., and Dixon, J., “Predicting Interviewer Nonresponse Rates from 

Household and Regional Characterstics”, Paper presented at AAPOR, 

2000. 

Section on Government Statistics – JSM 2009

165