1 — U.S. BUREAU OF LABOR STATISTICS • bls.gov

Evaluating Respondents' Burden via 

Indirect Indicators of Data Quality: 

Item vs. Index Scores

Daniel K. Yang

Office of Survey Methods Research

U.S. Bureau of Labor Statistics

CE Survey Methods Symposium

July 18, 2017

The views expressed in this paper are those of the author(s) and do 

not necessarily reflect the policies of the Bureau of Labor Statistics
2 — U.S. BUREAU OF LABOR STATISTICS TATISTICS • bls.gov bls.gov

Overview

 Consumer Expenditure Surveys (CE) Redesign and 

burden measurement.

 Burden questions (items) and burden index 

scores.

 Correlations between burden and indirect 

indicators of data quality.

 Effects of burden on expenditure estimates.

 Conclusion and follow-ups.

2
3 — U.S. BUREAU OF LABOR STATISTICS TATISTICS • bls.gov bls.gov

Respondents’ Burden Perception

 Gemini: redesign the CE Surveys to improve 

data quality, through a verifiable reduction in 

measurement error.

 Important: able to measure respondent 

burden (could contribute to data quality).

 How to best evaluate respondents’ perceived 

level of burden is still an open question.

3
4 — U.S. BUREAU OF LABOR STATISTICS TATISTICS • bls.gov bls.gov

Burden Items

 Between October 2012 and September 2013, a 

series of questions were asked in the interview 

survey at the end of the final wave, including ten 

questions assessing respondents’ perceived 

burden:

e.g. How burdensome was this survey to you? (bbur)

Not at all burdensome

A little burdensome

Somewhat burdensome

Very burdensome

4
5 — U.S. BUREAU OF LABOR STATISTICS TATISTICS • bls.gov bls.gov

Burden Index Scores

 Composite burden index scores (weighted, 

involving a correlation matrix of level of 

measurements, Yang 2015).

 Likert scales summation scores: a simplified 

alternative by compute a simple summation 

(of burden items Likert scales).

5
6 — U.S. BUREAU OF LABOR STATISTICS TATISTICS • bls.gov bls.gov

Figure 1. Distribution of Composite Burden 

Index Scores

Not at all 

burdensome

A little 

burdensome

Somewhat 

burdensome

Very 

burdensome

5 10 15 20 25 0.00 0.02 0.04 0.06 0.08 0.10

Composite Burden Index Scores: 2012 Oct. - 2013 Sep. Probability Density

1st Quartile

2nd Quartile

3rd Quartile

4th Quartile
7 — U.S. BUREAU OF LABOR STATISTICS TATISTICS • bls.gov bls.gov

Figure 2. Distribution of Likert Scales 

Summation Scores

Not at all 

burdensome

A little 

burdensome

Somewhat 

burdensome

Very 

burdensome

5 10 15 20 25 30 35 40 0.00 0.02 0.04 0.06 0.08

Likert Scales Summation Scores: 2012 Oct. - 2013 Sep. Probability Density

1st Quartile

2nd Quartile

3rd Quartile

4th Quartile
8 — U.S. BUREAU OF LABOR STATISTICS TATISTICS • bls.gov bls.gov

Burden Measures

Now we have three burden measures:

(single) burden question or (single) burden 

item,

(composite) burden index (scores), and

Likert scales summation (scores) or Likert 

scales sum.

8
9 — U.S. BUREAU OF LABOR STATISTICS TATISTICS • bls.gov bls.gov

Single Burden Question vs. Burden 

Index Scores

 The single burden question (item) had been used 

to examine the impact of the respondent’s 

perceived burden on data quality (Yan 2015).

 Objective: whether we can use the single burden 

question (item) to monitor burden via comparing 

correlations between indirect indicators of data 

quality and the single burden question (item) vs. 

burden index scores.
10 — U.S. BUREAU OF LABOR STATISTICS TATISTICS • bls.gov bls.gov

Data Sample

10

Burden data were collected between 

October, 2012 (Q4) and September, 2013 

(Q3).

Excluded households with missing values 

in any of the burden questions (items).

Final sample total has 6,378 households.
11 — U.S. BUREAU OF LABOR STATISTICS TATISTICS • bls.gov bls.gov

Evaluation of Single Burden Question 

(Item) vs. Index Scores

I. Visualize the descriptive statistics.

II. Numerically examine how burden index 

scores are correlated to indirect indicators of 

data quality.

III. Compare methods to identify most-burdened 

respondents (via burden measures) and the 

effects of burden measures on statistical 

estimates of expenditures.
12 — U.S. BUREAU OF LABOR STATISTICS TATISTICS • bls.gov bls.gov

Evaluation of Single Burden Question 

(Item) vs. Index Scores

I. Visualize the descriptive statistics.
13 — U.S. BUREAU OF LABOR STATISTICS TATISTICS • bls.gov bls.gov

Figure 3. Example: Distribution of Single Burden 

Question and Burden Index Score Quartiles

13

Single Burden Item

Burden Index Scores

Likert Scales Summation

Proportion (%)

0 20 40 60 80 100 Not at all burdensome A little burdensome Somewhat burdensome Very burdensome 1st Quartile 2nd Quartile 3rd Quartile 4th Quartile
14 — U.S. BUREAU OF LABOR STATISTICS TATISTICS • bls.gov bls.gov

Table 1. Mean and Standard Deviation (SD) of Burden 

Index Scores at (Single) Burden Question (Item) Level

Comparing by (Level) of 

Single Burden Question 

(Item, bbur)

Composite 

Burden Index

Likerts Scale 

Sum

n Mean (SD) Mean (SD)

1 = Not at all burdensome 1944 10.39 (2.11) 17.17 (3.08)

2 = A little burdensome 1994 13.45 (2.11) 21.44 (3.08)

3 = Somewhat burdensome 1773 15.57 (2.15) 24.48 (3.18)

4 = Very burdensome 667 18.89 (2.24) 29.42 (3.43)

14

Note: SD – standard deviation
15 — U.S. BUREAU OF LABOR STATISTICS TATISTICS • bls.gov bls.gov

Figure 4. Boxplot of Composite Burden Index Scores at 

(Single) Burden Question (Item) Level

15

Composite Burden 

Index Scores

Burden item
16 — U.S. BUREAU OF LABOR STATISTICS TATISTICS • bls.gov bls.gov

Figure 5. Boxplot of Likerts Scale Summation Scores at 

(Single) Burden Question (Item) Level

Likert Scales 

Summation Scores

Burden item
17 — U.S. BUREAU OF LABOR STATISTICS TATISTICS • bls.gov bls.gov

Evaluation of Single Burden Question 

(Item) vs. Index Scores

II. Numerically examine how burden index 

scores are correlated to indirect indicators of 

data quality as compared to the correlation 

between the single burden question (item) 

and indirect indicators of data quality.
18 — U.S. BUREAU OF LABOR STATISTICS TATISTICS • bls.gov bls.gov

Indirect Indicators of Data Quality

18

 Two indirect indicator variables of data quality 

were adopted:

NUMDK: the number of “Don’t Know” answers 

and

NUMRF: the number of “Refused” answers 

provided by respondents. 
19 — U.S. BUREAU OF LABOR STATISTICS TATISTICS • bls.gov bls.gov

Figure 6. Average Number of “Don’t Know” Answers 

by Levels of Single Burden Question (Item) and Burden 

Index Scores Quartile Groups
20 — U.S. BUREAU OF LABOR STATISTICS TATISTICS • bls.gov bls.gov

Figure 7. Average Number of “Refused” Answers by 

Levels of Single Burden Question (Item) and Burden 

Index Scores Quartile Groups
21 — U.S. BUREAU OF LABOR STATISTICS TATISTICS • bls.gov bls.gov

Williams' t-test: Compare Two Nonindependent Correlations

 Let n be the sample size, let r demote 

correlation and let j, k and h index three 

separate samples, respectively, then

 tn−3 = (rjk − rjh)

(n−1)(1+rkh)

2

n−1

n−3

R +

(rjk+rjh)

2

4

(1−rkh)

3

 where R = 1 − rjk

2 − rjh2 − rkh

2 + 2rjkrjhrkh.
22 — U.S. BUREAU OF LABOR STATISTICS TATISTICS • bls.gov bls.gov

Table 2. Compare Non-Independent Correlations (ρ) 

with a Variable in Common (NUMDK/NUMRF among 

Burden Measures)

Common Variable: 

Indirect Indicators 

of Data Quality

Burden Measures ρ

(jk, jh)

Williams' ttest p-value 

of H0: ρjk = ρjh

Number of Do Not 

Know (NUMDK, j)

Burden Question (k) 0.07

(h): Composite

Burden Index 0.10 0.03

Likert Scale Sum 0.10 0.03

Number of Refused 

(NUMRF, j)

Burden Question (k) 0.12

(h): Composite

Burden Index 0.15 0.26

Likert Scale Sum 0.14 0.27

22
23 — U.S. BUREAU OF LABOR STATISTICS TATISTICS • bls.gov bls.gov

Evaluation of Single Burden Question 

(Item) vs. Index Scores

III. Compare methods to identify mostburdened respondents (via burden 

measures) and the effects of burden 

measures on statistical estimates of 

expenditures.
24 — U.S. BUREAU OF LABOR STATISTICS TATISTICS • bls.gov bls.gov

Effects of Burden Measures on 

Expenditure Estimates

24

 We examined whether or not the impact of 

burden index scores on expenditure 

estimates would be different from the single 

burden question (item).

 We computed the mean estimates with all 

respondents and the mean estimates without 

“most-burdened” respondents for each 

expenditure variable in two ways:
25 — U.S. BUREAU OF LABOR STATISTICS TATISTICS • bls.gov bls.gov

Effects of Burden Measures on 

Expenditure Estimates

25

 Excluding the “Very burdensome” level for the 

single burden question (item),

 Excluding the 4th quartile level for the composite 

burden index scores and the simple Likerts Scale 

summation scores, respectively.

Note: Standard error of mean (SE) was computed after 

accounting for complex design and weights.
26 — U.S. BUREAU OF LABOR STATISTICS TATISTICS • bls.gov bls.gov

Table 3A. Complex Weighted Estimates of Mean Expenditures 

with and without Most-Burdened Respondents

26

Mostly in Person

With

Single Burden 

Question (Item)

Composite Burden 

Index Scores

Likerts Scale 

Summation Scores

Expenditures 

($) Without* Difference Without† Difference Without‡ Difference

Total 

Expenditure 12268 12269 1 12399 131 12328 60

(SE) (204) (216) (236) (225)

Food 1812 1789 -23 1780 -32 1772 -40

(SE) (27) (25) (24) (22)

Housing 3871 3859 -12 3854 -17 3839 -32

(SE) (54) (56) (63) (61)

Transportation 2276 2300 24 2366 90 2357 81

(SE) (70) (84) (97) (95)

Note: for without Burdened-out Respondents:

* Burden Item excludes level 4 (Very burdensome),

† Burden Index scores < 16.08,

‡ Likerts Scale Sum < 25
27 — U.S. BUREAU OF LABOR STATISTICS TATISTICS • bls.gov bls.gov

Table 3B. Complex Weighted Estimates of Mean Expenditures 

with and without Most-Burdened Respondents

27

Mostly Telephone

With

Single Burden 

Question (Item)

Composite Burden 

Index Scores

Likerts Scale 

Summation Scores

Expenditures 

($) Without* Difference Without† Difference Without‡ Difference

Total 

Expenditure 12296 12213 -83 12365 69 12340 44

(SE) (228) (226) (246) (253)

Food 1868 1859 -9 1847 -21 1852 -16

(SE) (26) (29) (30) (27)

Housing 4219 4174 -45 4208 -11 4206 -13

(SE) (91) (99) (109) (109)

Transportation 2103 2079 -24 2081 -22 2074 -29

(SE) (107) (111) (1467) (145)
28 — U.S. BUREAU OF LABOR STATISTICS TATISTICS • bls.gov bls.gov

Conclusion

 The single burden question (item, 'bbur') can be 

used as a sole indicator of respondent’s perceived 

burden for monitoring respondent’s burden 

perception over time based on available data.

 If possible, include the composite burden index 

scores as a “check point” in case when the simple 

Likert Scale summation scores behave differently 

from the single burden question (item).
29 — U.S. BUREAU OF LABOR STATISTICS TATISTICS • bls.gov bls.gov

Proposed Follow-up Steps

29

 Exploring Burden Item's Proxy-Indicators: e.g. household 

income before tax, number of expenditures (unedited), 

mortgage indicator, whether it is a converted refusal, 

information booklet usage, records usage, interview mode 

and CHI variables: interview length and door step concerns?

 How does indirect indicators of data quality correlate with the 

single burden question (item) comparing to those proxyindicator candidates?

 How to quantify the associations? (e.g. extrapolate into a new 

data set?)
30 — U.S. BUREAU OF LABOR STATISTICS TATISTICS • bls.gov bls.gov

Acknowledgement

 Branch of Research and Program Development 

(BRPD), DCES.

Adam N. Safir, Chief (DCES)

Laura P. Erhard, Supervisory Economist (BRPD)

 Office of Survey Methods Research (OSMR)

Wendy L. Martinez, PhD, Director, Mathematical Statistics 

Research Center (MSRC)

Brandon M. Kopp, PhD, Research Psychologist

Scott S. Fricker, PhD, Research Psychologist

30
31 — U.S. BUREAU OF LABOR STATISTICS TATISTICS • bls.gov bls.gov

THANK YOU!

31
Contact Information

32 — U.S. BUREAU OF LABOR STATISTICS • bls.gov

Daniel K. Yang

Research Mathematical Statistician

Office of Survey Methods Research (OSMR) 

www.bls.gov/osmr/home.htm 

yang.daniel@bls.gov

Disclaimer: Any opinions expressed in this paper are 

those of the author(s) and do not constitute policy of 

the Bureau of Labor Statistics.