1 — U.S. BUREAU OF LABOR STATISTICS • bls.gov

Data quality in the Consumer Expenditure Survey’s 

Large Scale Online Diary Feasibility Test

Laura Erhard

Joint Statistical Meetings

August 8-12, 2021

Gray Jones, Parvati Krishnamurty, and Brett McBride
2 — U.S. BUREAU OF LABOR STATISTICS • bls.gov

Outline

Brief Background

Test Design

Results

Next steps
3 — U.S. BUREAU OF LABOR STATISTICS • bls.gov

Consumer Expenditure Survey

 Collects spending data on the U.S. Population

 Sponsored by the Bureau of Labor Statistics, 

collected by the Census Bureau

 Survey participants report dollar amounts for 

all non-investment purchases. Business 

expenses and reimbursements are excluded.

 Provide expenditure weights for the U.S. 

Consumer Price Index (CPI)

 Quarterly Interview and Diary Survey

 Diary Survey: two, consecutive one-week paper 

diaries* designed to collect small, frequently 

purchased items
4 — U.S. BUREAU OF LABOR STATISTICS • bls.gov

Large Scale Feasibility Test (LSF) of the Online Diary Design

 Starting sample size: 2,500 addresses

 Field period: October 2019-April 2020

 Two visits, one for diary placement and one at the end of the diary keeping period 

(“pickup”)

 Push to online diary if the household respondent is eligible, if they refuse or are 

ineligible-- they get paper diaries

 Summary reports for interviewers to monitor the progress of online diary participants

 Recall tab in the CAPI instrument for interviewer to record additional expenses 

reported by respondent at pickup interview (normally transcribed in the paper diary)

 Embedded incentives/postcard experiment
5 — U.S. BUREAU OF LABOR STATISTICS • bls.gov

Response Rates and Mode
6 — U.S. BUREAU OF LABOR STATISTICS • bls.gov

Estimation Response Rates: LSF Test vs. CE Production

Disposition CE Production

(2019q4-2020q1)

LSF Test

(2019q4-2020q1)

n % n %

Eligible Sample 12,690 100 4,268 100

Complete Cases 5,838 46.0 1,782 41.8

Overall Nonresponse 6,852 54.0 2,486 58.3

Noncontact 939 7.4 284 6.7

Refusal 3240 25.5 1108 26.0

Diary placed late 939 6.5 408 9.6

Minimal expenditure 481 3.8 253 5.9

Other 1,253 9.9 433 10.1
7 — U.S. BUREAU OF LABOR STATISTICS • bls.gov

Mode at Completion

LSF n %

Online 663 37.2

Paper 1074 60.3

Other 45 2.5
8 — U.S. BUREAU OF LABOR STATISTICS • bls.gov

Demographics of Mode

 Similar to other mode studies

 LSF online households: Younger, with higher household income, 

higher education

 Less likely to be Hispanic or Latino origin
9 — U.S. BUREAU OF LABOR STATISTICS • bls.gov

Expenditure Estimates and Quality

preliminary
10 — U.S. BUREAU OF LABOR STATISTICS • bls.gov

Earlier analysis quality takeaways

 Item Non-response: 

Lower non-response for business of expenditure in online diaries 

compared to production diaries (by design, since required field) 

Other item non-response rates similar across LSF modes and 

production

 Similar rates of Diary Fatigue (reporting more in the first week 

than the second week)

 Recalled expenses at pick-up handled differently for online 

diaries than in the paper diaries  potential missed recall

 Lower counts of expenditures in the LSF
11 — U.S. BUREAU OF LABOR STATISTICS • bls.gov

Expenditure Totals By Section

 Average weighted expenditures higher 

in Production than LSF.

 Most of difference attributable to 

reduction in “Other” category. 

Significant by minor difference in Food 

away from home

 No differences in Food at home and 

Clothing.

Food at Home, 

$102.64 

Food at Home, 

$104.94 

Food Away, $58.18 Food Away, $50.86 

Clothing, $33.22 Clothing, $33.32 

Other, $531.97 

Other, $440.17 

$0

$100

$200

$300

$400

$500

$600

$700

$800

Production (N=5,838) LSF (N=1,782)

Weighted Expenditures

Food at Home Food Away Clothing Other

13.3% reduction
12 — U.S. BUREAU OF LABOR STATISTICS • bls.gov

Expenditure Regression Results

Coefficient SE

Intercept -234.008 (239.25)

LSF online -114.704 (46.33)*

LSF paper -69.693 (67.04)

Log(pre-tax income) 106.645 (22.63)***

Non-homeowner CU -155.335 (40.31)***

Highest: HS grad or less -245.555 (75.33)**

Highest: Some college -205.940 (76.18)*

Highest: Assoc. degree -171.684 (77.13)*

Highest: Professional degree 108.471 (99.10)

The full model did not find the CU’s size, respondent’s age, sex, race or ethnicity, or the RO in which the 

interview occurred to be significant factors in reported expenditures.

* p < 0.05

** p < 0.01

*** p < 0.0001
13 — U.S. BUREAU OF LABOR STATISTICS • bls.gov

Editing Rates Production vs. LSF

CE Production LSF Test

n % n %

Minimal expenditure 481 3.79 253 5.93

Allocation rate 17,096 9.33 3,424 11.71
15 — U.S. BUREAU OF LABOR STATISTICS • bls.gov

Summary

 Preliminary research showed little to no impact on item nonresponse, diary fatigue

 Response rate slightly lower, but partly attributable to test effect

 Take up rate much lower than expected for the online diary

 Quality indicators show online diary may not provide as high quality data

Higher allocation rates

Higher number of removed diaries

Lower expenditures after controls

BUT....

 Earlier research identified concerns on how recalled expenses were captured during LSF 

test, particularly for online diaries  since mitigated with further interviewer training 

and revised procedures

 Online diary helped us “pandemic proof” our survey

 Some respondents expect an online diary, Interviewers are happy to have the option.
18 — U.S. BUREAU OF LABOR STATISTICS • bls.gov

Online Diary next steps

 Continued use due to the pandemic

 Go/No-Go decision on planned official 

implementation in July 2022

 Recommendations for changes to full implementation 

protocols

 Continuous improvement

autosuggest feature, prompts, trip based diary

more use of receipts

self-administration test
Contact Information

19 — U.S. BUREAU OF LABOR STATISTICS • bls.gov

Laura Erhard

Branch of Research and Program Development

Division of Consumer Expenditure Surveys

Erhard.Laura@bls.gov