Evaluating the Association of Initial Expressions of 

Reluctance with Panel Survey Behavior

Brett McBride

Brandon Kopp

Lucilla Tan

U.S. Bureau of Labor Statistics

69th Annual AAPOR Conference

May 16, 2014
Outline

 Background

 Research Questions

How does initial contact data inform survey behavior?

 Data and Methods

 Findings

Variability in households’ overall concerns over survey panel

Response patterns

Respondent effort, reporting quality

Perceived survey burden

 Conclusions & Caveats

2
Background

Consumer Expenditure (CE) Survey

 National household panel survey that collects 

information about buying habits of consumers 

 Used to calculate the weights of items tracked by 

the Consumer Price Index (CPI), also used by 

economists, academics, and market researchers 

 Study focus: Quarterly Interview Survey (CEQ) –

survey of larger, easier-to-recall purchases 

Designed as CAPI survey, with proxy reporting, conducted 

quarterly for 5 waves

About 1-hour in duration

3
Background: 

Doorstep Concern Data

 Contact History Instrument’s Doorstep (DS) Concerns

 Interviewer-Assessed DS Concern(s) at Contact

4

DS Concern Items (‘Mark All That Apply’)

1 Too busy 8 Anti-government 15 Hostile/threatening

2 Not interested 9 Talk to specific HH member 16 Survey content not applicable

3 Privacy concerns 10 Intends to quit survey 17 Other HH members say don’t do survey

4 Scheduling difficulty 11 Family issues 18 Too many questions previously

5 Survey voluntary 12 Broke appt-puts off int. indefinitely 19 R requests same interviewer

6 Does not understand survey 13 Hangs up/slams door 20 Interview too long previously

7 Interview too time consuming 14 Too many interviews 21 Information previously given

No concerns Other specify
Research Questions

1. Do households’ overall DS concerns vary over survey 

panel?

2. Are respondents with DS concerns responding 

throughout survey panel?

3. Do these respondents provide poorer data quality?

4. Do pre-survey DS concerns lead to greater 

perceptions of survey burden after panel completion?

5
Data

 Sample units combined from 6 panels from October 

2011 to March 2013 (N=3,000)

6

Wave 1 (N=3,000) Wave 5 (N=3,000)

Unit response 

(completed interviews)

86% 83%

No DS concerns 52% 46%

Top DS concerns (of

HH with DS concerns)

-Too busy (22%)

-Not interested (14%)

-Privacy concerns (13%)

-Scheduling difficulty (12%)

-Too busy (24%)

-Not interested (15%)

-Privacy, scheduling difficulty, 

too many interviews (12%)
Methods

 Developed composite scores for each wave based on 21 DS 

concern items using principal components analyses (per 

Henderson et al., 1990), and divided scores into 3 groups:

Wave 1 DS concern groups:

7

No concerns (N=1,561)

Low concerns (N=742)

High concerns (N=715)
Findings

1. Change in DS Concern Groups Over Survey Panel

 Across survey panel, some households moved to a 

higher DS concern group (27%) but most remained in 

same group at Wave 5 as at Wave 1 (52%). 

8

21%

52%

27%

0% 10% 20% 30% 40% 50% 60%

Lower group (N=603)

Same group (N=1,650)

Higher group (N=747)
Findings

2. Panel Response Patterns for DS Concern Groups

 Respondents in High DS concern group completed 

fewer interviews

9

0

1 2

3

4

5

No concerns (N=1,561)

# of interviews completed

0

1

2

3

4

5

Low concerns (N=742)

# of interviews completed

0

1

2

3

4

5

High concerns (N=715)

# of interviews completed
Findings

3. Respondent Effort and Data Quality (Wave 5)

Interviewer rating measures:

1) Effort: “How much effort would you say this respondent put 

into answering the expenditure questions during this 

interview?” 

A lot of effort, a moderate amount of effort, a bare minimum of effort

2) Record Use: “Did respondent use records for questions?” 

90% of time or more, 50% to 89% of time, 10% to 49% of time, less than 10% of time

3) Consult Others: “Did the respondent get information from 

other household members when answering the questions?”

Yes, No, Don’t Know

10
Findings

3. Respondent Effort and Data Quality (Wave 5)

Interviewer ratings for respondents in High DS concern 

group:

1) Effort – Half as many (19%) rated as exerting ‘a lot of effort’

2) Record Use – Almost half as many (11%) reported to use 

records at least 90% of the time

3) Consult Others – Significantly fewer (11%) reported to get 

(expenditure) information from other household members

11
Findings

3. Respondent Effort and Data Quality (Wave 5)

Data quality measures for respondents in High DS 

concern group:

 Significantly higher proportion of edited expenditure records 

(21%)

 More likely to require refusal conversion (26%)

 Almost twice as likely to be reassigned to a different 

interviewer (11%) 

12
Findings

4. Perceived Survey Burden (Wave 5)

 Used burden composite scores (similar to DS composite 

scores) based on questions respondents answered in Wave 5

 Found significant correlation between respondents’ burden 

composite scores and their DS concern scores: 

Spearman ρ=0.38 (p<0.0001)

13
Conclusions

 Most households did not exhibit significant change in 

DS concern ranking over survey panel 

 Households with higher concern rankings in Wave 1 

were less likely to complete all 5 interviews

 Among responding households, those with higher 

concern rankings were less likely to exhibit optimal 

reporting behavior and provide high-quality data

 Respondents with higher concern rankings at last 

interview also perceived survey burden to be higher

14
Caveats

 Unclear how consistently interviewers record various 

DS concerns that are expressed

 No data available on the construct validity of DS 

concern items 

 Unclear how often person expressing DS concern is 

the same person responding to survey

 Did not control for household characteristics in this 

study

15
Contact Information

Brett McBride

(202) 691-5136

mcbride.brett@bls.gov

Disclosure: Any opinions expressed in this paper are those of the authors, and do 

not constitute policy of the Bureau of Labor Statistics.
DS Concern Themes

 Examined DS concern themes (“No concerns,” “not interested/ 

hostile,” “survey content/privacy,” “time”) and panel response

17

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

No concerns (N=1,561) Not interested/ hostile

(N=433)

Survey content/ privacy

(N=697)

Time (N=904)

Percent of households

0 1 2 3 4 5
Themes by DS Concern Group

18