1 — U.S. BUREAU OF LABOR STATISTICS • bls.gov

Evaluating Perceived Burden of Household 

Survey Respondents

Daniel K. Yang

Office of Survey Methods Research

U.S. Bureau of Labor Statistics

AAPOR

May 16, 2018

The views expressed in this paper are those of the author and do not 

necessarily reflect the policies of the Bureau of Labor Statistics
2 — U.S. BUREAU OF LABOR STATISTICS TATISTICS • bls.gov bls.gov

Overview

I. Consumer Expenditure Surveys (CE) 

Redesign and burden measurement.

II. Data and other questions indicate burden.

III. Burden proxy indicators.

IV. Explore recursive partitioning models.
3 — U.S. BUREAU OF LABOR STATISTICS TATISTICS • bls.gov bls.gov

I. Respondents’ Burden Perception

 CE interview is almost an hour long, non-trivial questions

 Gemini: redesign the CE to improve data quality, through a 

verifiable reduction in measurement error.

 Important: able to measure respondent burden (could 

contribute to data quality).

 How to best evaluate respondents’ perceived level of burden is 

still an open question.
4 — U.S. BUREAU OF LABOR STATISTICS TATISTICS • bls.gov bls.gov

II. Burden Questions

 Between October 2012 and September 2013, a series of questions 

were asked in the interview survey at the end of the final wave, 

including ten questions assessing respondents’ perceived burden, e.g.

 How burdensome was this survey to you?

Not at all burdensome

A little burdensome

Somewhat burdensome

Very burdensome
5 — U.S. BUREAU OF LABOR STATISTICS TATISTICS • bls.gov bls.gov

Burden Questions (cont.)

 Would you say that this was too many interviews?

 A reasonable number

 Too many interviews

 Thinking about the amount of effort that you put forth into answering 

today's survey, would you say that you put forth:

 A little effort

 A moderate amount of effort

 A lot effort
6 — U.S. BUREAU OF LABOR STATISTICS TATISTICS • bls.gov bls.gov

Burden Measures

 We have three burden measures:

Single Burden Question (or item),

Likert Scales Summation Scores (or Likert scales sum): a simplified 

alternative computes a summation of burden questions (in Likert 

scales), and

Composite Burden Index Scores: weighted, involves a correlation 

matrix of burden questions (Yang 2015 & 2017).
7 — U.S. BUREAU OF LABOR STATISTICS TATISTICS • bls.gov bls.gov

Data Sample

 CE has 4 waves, burden questions were only collected from 

participants in their final wave.

 Attritions by the final wave:

 Excluded households with missing values in any of the burden 

questions (items), final sample total had 6,369 households.

Wave 2

Drop off

Wave 1 Wave 3 Wave 4

Drop off Drop off
8 — U.S. BUREAU OF LABOR STATISTICS TATISTICS • bls.gov bls.gov

What we found in previous studies ...

There is no conclusive evidence of differences in 

correlations in data quality measures with burden 

measurements.

For both the single burden question and burden scores, 

excluding most-burdened respondents does not appear 

to have much of an effect on selected expenditure 

variable mean estimates.
9 — U.S. BUREAU OF LABOR STATISTICS TATISTICS • bls.gov bls.gov

Other Questions Indicate Burden

CE collects respondent’s answer of burden, but there 

are other objective indicators, e.g. other sets of items 

people used to indicate burden or burden proxy 

indicators.

So, can burden measures be extrapolated from a set of 

variables that would indicate burden, e.g. by 

conditioning on subpopulations?
10 — U.S. BUREAU OF LABOR STATISTICS TATISTICS • bls.gov bls.gov

III. Burden Proxy Indicators

Income: household income before tax

Total Time: interview length in minutes

Num. Expn.: number of expenditures (unedited)

Mortgage: mortgage indicator

Conv. Ref.: whether it is a converted refusal

Mode: interview mode (personal visit or telephone)
11 — U.S. BUREAU OF LABOR STATISTICS TATISTICS • bls.gov bls.gov

Burden Proxy Indicators (cont.)

 Info. Booklet: information booklet usage

 5=Almost always (90% of the time or more)

 4=Most of the time (50% to 89% of the time)

 3=Occasionally (10% to 49% of the time)

 2=Never or almost never (less than 10% of the time)

 1=The respondent did not have access to the information booklet (ref.)

 Record: records usage

 4=Almost always (90% of the time or more)

 3=Most of the time (50% to 89% of the time)

 2=Occasionally (10% to 49% of the time)

 1=Never or almost never (less than 10% of the time) (ref.)

 Door Step Concerns (CHI Contact History Instrument)

 0=No concerns

 1=Privacy/govt. concerns

 2=Busy/logistics

 3=Other
12 — U.S. BUREAU OF LABOR STATISTICS TATISTICS • bls.gov bls.gov

IV. Recursive Partitioning

 Partitions the data space into subpopulations among independent 

variables to generate a decision tree until a predetermined criterion is 

met.

 A decision tree is a “forecasting model” to use input variables (“branch”) 

to predict a target variable (“leaf”). Classification trees for discrete 

target variables. Regression trees for continuous target variables.

 Respondent’s perception of burden could be very different for different 

subpopulations.

 Recursive Partitioning for Modeling Survey Data {rpms} R package

{rpms}: node sample size 200, permutation test p-value = 0.05
13 — U.S. BUREAU OF LABOR STATISTICS • bls.gov

Decision Tree: Single Burden Question

(1-5 point scale; higher score = greater burden)

Single Burden 

Question

Door Step Concern: 

Busy/logist., 

Privacy/Gov., Other

Converted 

Refusal (3.14)

Converted 

Refusal: No

Record Usage: 

Never

# of Expn. > 26 

(2.56)

# of Expn. ≤ 26 

(2.88) Record Usage: 

Almost, Most, 

Occas. (2.47)

Door Step Concern: 

No (2.02)

• Respondents with no door step concerns were the least burdened

• Respondents who expressed concerns and had to be convinced to participate reported the 

greatest burden

• For respondents with door step concerns, burden index scores were different among 

subgroups of converted refusal, record usage and number of expenditures.

Single Burden 

Question

Door Step Concern: 

Busy/logist., 

Privacy/Gov., Other

Converted 

Refusal (3.14)

Converted 

Refusal: No

Record Usage: 

Never

# of Expn. > 26 

(2.56)

# of Expn. ≤ 26 

(2.88) Record Usage: 

Almost, Most, 

Occas. (2.47)

Door Step Concern: 

No (2.02)
14 — U.S. BUREAU OF LABOR STATISTICS • bls.gov

Decision Tree: Likert Scales Summation Scores

(Scores range from 11 to 36; higher score = greater burden)

Likert Scales 

Summation 

Scores

Door Step Concern: 

Busy/logist., 

Privacy/gov., Other

Converted 

Refusal (27.48)

Converted 

Refusal: No

Record Usage: 

Never

Income > 

$20,400 (25.48)

Income ≤ 

$20,400 (24.57) Record Usage: 

Almost, Most, 

Occas. (23.96)

Door Step Concern: 

No (20.83)

• Respondents with no door step concerns were the least burdened

• Respondents who expressed concerns and had to be convinced to participate reported the 

greatest burden

• For respondents with door step concerns, burden index scores were different among 

subgroups of converted refusal, record usage, and household income.
15 — U.S. BUREAU OF LABOR STATISTICS • bls.gov

Decision Tree: Composite Burden Index Scores

(Scores range from 6.56 to 22.41; higher score = greater burden)

Composite 

Burden Index 

Scores

Door Step Concern: 

Busy/logist., 

Privacy/gov., Other

Door Step Concern: 

Privacy/gov. (17.25)

Door Step Concern: 

Busy/logist., Other

Door Step 

Concern: 

Busy/logist.

# of Expn. > 30 

(15.83)

# of Expn. ≤ 30 

(16.63) Door Step 

Concern: Other 

(13.97)

Door Step Concern: No 

(12.98)

• Once again, the “No door stop concern” group expressed the lowest level of burden.

• In this model, the specific type of door step concerns expressed by respondents were 

shown to be related to the composite burden index score.
16 — U.S. BUREAU OF LABOR STATISTICS TATISTICS • bls.gov bls.gov

Single Burden 

Question

Likert Scales 

Summation Scores

Composite Burden 

Index Scores

First Split Door Step Concerns 

vs. Not

Door Step Concerns 

vs. Not

Door Step Concerns vs. 

Not

Second Split Converted Refusal 

vs. Not

Converted Refusal 

vs. Not

Door Step: Privacy, or Gov. 

vs. Busy, Logistic, Other

Third Split Record Usage vs. 

Not

Record Usage vs. 

Not

Door Step: Busy or 

Logistic vs. other 

Fourth Split # of Expn > 26 vs. 

# of Expn ≤ 26

Income > $2.04k vs. 

Income ≤ $2.04k

# of Expn > 30 vs. 

# of Expn ≤ 30
17 — U.S. BUREAU OF LABOR STATISTICS TATISTICS • bls.gov bls.gov

Burden Proxy Indicators Main Take Away

 For all the three measures of burden, a few proxy 

measures were repeatedly identified to be associated 

with burden.

 These measures should be explored in future studies as 

they may be useful in understanding respondent 

behaviors that could be caused by burden (e.g., 

attrition, data quality).
18 — U.S. BUREAU OF LABOR STATISTICS TATISTICS • bls.gov bls.gov

Possible Next Steps

Could new burden proxy indicators be included in 

the recursive partitioning model? What about the 

prediction error?

Additional exploration of burden index scores 

regression tree models? (e.g. extrapolate into a 

new data set?)
19 — U.S. BUREAU OF LABOR STATISTICS TATISTICS • bls.gov bls.gov

THANK YOU!
Contact Information

20 — U.S. BUREAU OF LABOR STATISTICS • bls.gov

Daniel K. Yang

Research Mathematical Statistician

Office of Survey Methods Research (OSMR) 

www.bls.gov/osmr/home.htm 

yang.daniel@bls.gov

Disclaimer: Any opinions expressed in this paper are 

those of the author and do not constitute policy of the 

Bureau of Labor Statistics.