1 — U.S. BUREAU OF LABOR STATISTICS • bls.gov

Using early-wave interviewer 

observations to predict respondent 

burden and survey nonresponse in 

later panel survey waves

Scott Fricker, Arcenis Rojas, Lucilla Tan 

Bureau of Labor Statistics

AAPOR 2016

May 14th
2 — U.S. BUREAU OF LABOR STATISTICS TATISTICS • bls.gov bls.gov

Background

 Objective

Examine associations between early panel wave 

interviewer observations about respondent reluctance 

and...

– Later-wave interviewer observations of reluctance

– Later-wave survey outcome (i.e., refusals)

– Later-wave respondent burden
3 — U.S. BUREAU OF LABOR STATISTICS TATISTICS • bls.gov bls.gov

Background, cont.

 Interviewer-observation paradata inform our understandings 

of nonresponse

 Interviewer observations about R concerns significantly related to 

nonresponse propensity (Bates, 2004; Henley and Bates, 2006; Dixon, 

2009; Kopp et al., 2013; McBride and Tan, 2014) 

 Wave 1 observations were related to Wave 2 attrition (Henley and 

Bates, 2006)

 Little examination of effects across panel waves

 Potential applications to responsive design/interventions
4 — U.S. BUREAU OF LABOR STATISTICS TATISTICS • bls.gov bls.gov

Methods

 Data Sources

Consumer Expenditure Quarterly Interview Survey (CEQ) 

(http://www.bls.gov/cex)

– Five-wave panel survey*

• Collects information on expenditures, income, and demographics for 

a “consumer unit” (CU)

• Respondents’ self-reported burden measured in 5th wave

– Linked records of Wave 1 sample units (4/12 – 9/12) who were 

eligible to receive the burden question in their final interview 

(4/13 – 9/13)

– Includes both interviews and non-interviews (n = 3,529)

Contact History Instrument (CHI)

– Collects information about each contact attempt

– Includes interviewer observations of sample members’ reluctance 

(“doorstep concerns”)
5 — U.S. BUREAU OF LABOR STATISTICS TATISTICS • bls.gov bls.gov

Formation of CHI Doorstep Concerns Themes

Doorstep concern code number

used to form theme

Doorstep Concern Theme

6, 7, 8, 9, 10 Survey Content / Privacy

1, 11, 12 Not Interested / Hostile

2, 3, 4, 5 Time

13, 14, 15 Gatekeeper

23 Other
Methods, cont.

 Doorstep concern theme indicators

 “1” if concern was observed at least once in wave X

 “0” if concern was not observed

 ISSUE: concerns recorded at CU-level, but different respondents within 

the CU can report in different waves

 Respondent burden dependent variable

 Question has four response choices (not at all, a little, somewhat, 

very)

 Dichotomized for this analysis (“1” = very; “0” = all others)

 Other predictors (Wave 1 data)

 Change in interviewer

 Contact attempts (less than 7 vs. 7 or more attempts)

 Household composition (single vs. multiple-person)

 Use of records and Information booklet (more than 50% of time, not)

 Length of interview (minutes)
7 — U.S. BUREAU OF LABOR STATISTICS TATISTICS • bls.gov bls.gov

Methods, cont.

 Analytic approach

Correlations between Wave 1 and Wave 5 doorstep 

concern themes

Compare Single- vs. Multiple-person CUs

– Sample characteristics

– Prevalence/distribution of doorstep concerns

Examine bivariate associations between Wave 1 individual 

predictors and Wave 5 outcome measures (refusal, 

burden)

Estimation of effects of Wave 1 predictors on Wave 5 

outcome measures

– Two-staged censored Probit model

– STATA 13.1, heckprob algorithm
8 — U.S. BUREAU OF LABOR STATISTICS TATISTICS • bls.gov bls.gov

Results

 Tetrachoric correlations of doorstep concern 

themes in Waves 1 and 5

Doorstep Concern

T. Correlation 

(1st – 5th Interview)

Content/Privacy 0.387

Not interested/Hostile 0.507

Time 0.350

Gatekeeper 0.387

Other 0.317
9 — U.S. BUREAU OF LABOR STATISTICS TATISTICS • bls.gov bls.gov

Doorstep Concern Prevalence, Waves 1 and 5: 

Single vs. Multiple Person Households

14.3%

12.2%

17.4% 17.1%

10.6% 11.1%

Single person CU (n=540) Multi-person CU (n=2002)

Not interested/Hostile

W1yes_W5no W1no_W5yes W1yes_W5yes
10 — U.S. BUREAU OF LABOR STATISTICS TATISTICS • bls.gov bls.gov

Doorstep Concern Prevalence, Waves 1 and 5: 

Single vs. Multiple Person Households

23.5% 24.6% 24.3%

22.3%

15.9%

23.6%

Single person CU (n=540) Multi-person CU (n=2002)

Time 

W1yes_W5no W1no_W5yes W1yes_W5yes
11 — U.S. BUREAU OF LABOR STATISTICS TATISTICS • bls.gov bls.gov

Doorstep Concern Prevalence, Waves 1 and 5: 

Single vs. Multiple Person Households

24.3% 24.9%

14.1% 13.7%

10.9%

12.6%

Single person CU (n=540) Multi-person CU (n=2002)

Survey content/Privacy
12 — U.S. BUREAU OF LABOR STATISTICS TATISTICS • bls.gov bls.gov

Doorstep Concern Prevalence, Waves 1 and 5: 

Single vs. Multiple Person Households

3.5%

10.7%

3.3%

8.0%

0.4%

2.8%

Single person CU (n=540) Multi-person CU (n=2002)

Gatekeeping
13 — U.S. BUREAU OF LABOR STATISTICS TATISTICS • bls.gov bls.gov

Doorstep Concern Prevalence, Waves 1 and 5: 

Single vs. Multiple Person Households

14.3% 14.3%

11.3%

9.7%

4.1% 3.8%

Single person CU (n=540) Multi-person CU (n=2002)

Other Concerns
Results: Wave 5 Refusal

22.0

22.2

22.4

22.6

22.8

23.0

No Yes

% Refusal

Wave 1: Single-person CU

0.0

10.0

20.0

30.0

40.0

No Yes

% Refusal

Wave 1: 7 or More Contacts

0.0

10.0

20.0

30.0

40.0

50.0

No Yes

% Refusal

Wave 1: Ever Changed Interviewer

0.0

10.0

20.0

30.0

40.0

50.0

60.0

No Yes

% Refusal

Wave 1 "Not Interested/Hostile"

0.0

10.0

20.0

30.0

40.0

No Yes

% Refusal

Wave 1: Survey Content/Privacy

0.0

5.0

10.0

15.0

20.0

25.0

30.0

35.0

No Yes

% Refusal

Wave 1: Time Concern

0.0

5.0

10.0

15.0

20.0

25.0

30.0

35.0

No Yes

% Refusal

Wave 1: Gatekeeping Concerns

0.0

5.0

10.0

15.0

20.0

25.0

30.0

35.0

40.0

No Yes

% Refusal

Wave 1: Other Concerns

0.0

2.0

4.0

6.0

8.0

10.0

12.0

14.0

16.0

No Yes

% Refusal

Wave 1: Record Use GT 50%
Results: Wave 5 Burden

8.5

9.0

9.5

10.0

10.5

11.0

% Very Burdened

No Yes

Wave 1: Single-person CU

0.0

5.0

10.0

15.0

% Very Burdened

No Yes

Wave 1: 7 of More Contacts

0.0

5.0

10.0

15.0

% Very Burdened

No Yes

Wave 1: Ever Changed Interviewer

0.0

5.0

10.0

15.0

20.0

No Yes

% Very Burdened

Wave 1: Not Interested/Hostile

0.0

5.0

10.0

15.0

20.0

No Yes

% Very Burdened

Wave 1: Survey Content/Privacy 

Concerns

0.0

5.0

10.0

15.0

No Yes

% Very Burdened

Wave 1: Time Concerns

0.0

5.0

10.0

15.0

20.0

25.0

No Yes

% Very Burdened

Wave 1: Gatekeeping Concerns

0.0

2.0

4.0

6.0

8.0

10.0

12.0

14.0

16.0

18.0

No Yes

% Very Burdened

Wave 1: Other Concerns

0.0

2.0

4.0

6.0

8.0

10.0

12.0

No Yes

% Very Burdened

Wave 1: Record Use GT 50%
Results: Probit Model with Sample Selection

Wave 1 Predictors

Stage 1: 

Respondent in Wave 5

Stage 2: 

Wave 5 Respondent “Very 

Burdened”

Coefficient P-value Coefficient P-value

Completed Interview 2.99 0.000

Single-person CU 0.13 0.057 -0.04 0.664

7+ Contacts -0.77 0.340

Interviewer change -0.18 0.039

Record Use GT 50% -0.10 0.237

Info Book Use GT 50% -0.20 0.008

Survey Length -0.00 0.908

Concern Theme:

Not Interested/Hostile -0.21 0.059 0.17 0.200

Survey Content/Privacy -0.08 0.27 0.004

Time Concerns -0.14 0.044 0.12 0.142

Gatekeeping Concerns 0.38 0.006 0.27 0.040

Other Concerns -0.14 0.122 0.13 0.232
17 — U.S. BUREAU OF LABOR STATISTICS TATISTICS • bls.gov bls.gov

Summary

 Doorstep concerns correlated across panel waves

Despite year span and possible changes in R and Intv’r

Refined/segmented analysis may strengthen associations

 Wave 1 interviewer observations of R concerns were 

significant predictors of Wave 5 response and burden

Not Interested/Hostile – response

Survey Content/Privacy – burden

Time concerns expressed in Wave 1 were associated with 

Wave 5 (non)response, but not with perceptions of burden

Actual interview duration not associated with burden
18 — U.S. BUREAU OF LABOR STATISTICS TATISTICS • bls.gov bls.gov

Implications, Limitations, & Next Steps

 Paradata are key to understanding and predicting survey 

outcomes of interest

 Changes in interviewer, interviewer observations/R doorstep concerns

 Potential input for responsive-design decisions/interventions

 Study limitations

 Likely error in measurement of CHI and burden variables

 Single dataset, model specifications

 Future work

 Examine effects on other substantive CE outcomes

 Apply this analytic approach to other surveys where CHI paradata are 

available
Contact Information

19 — U.S. BUREAU OF LABOR STATISTICS • bls.gov

Scott S. Fricker

Sr. Research Psychologist

Office of Survey Methods Research

202-691-7390

fricker.scott@bls.gov