1 — U.S. BUREAU OF LABOR STATISTICS • bls.gov

Going Online: Results from the 

Consumer Expenditure Survey’s Large 

Scale Online Diary Feasibility Test (LSF)

Laura Erhard

CE Symposium

July 21, 2020
2 — U.S. BUREAU OF LABOR STATISTICS • bls.gov

Outline

 Background

 LSF Test Design

 The LSF Online Diary

 Preliminary Results

 Lessons Learned

 Next Steps
3 — U.S. BUREAU OF LABOR STATISTICS • bls.gov

Background
4 — U.S. BUREAU OF LABOR STATISTICS • bls.gov

CE Diary Survey Characteristics

 CE Diary Survey (CED) currently uses two oneweek paper diaries to collect household 

expenditures 

 Interviewers place the diaries in-person and 

conduct one more visit to pick up the diaries 

 One household respondent reports on all 

household expenditures

 Expenditures grouped into four categories: 

Food and Drinks For Home Consumption

Meals, Snacks, and Drinks Away from Home

Clothing, Shoes, Jewelry and Accessories

All Other Products, Services and Expenses 
5 — U.S. BUREAU OF LABOR STATISTICS • bls.gov

Large Scale Feasibility Test Design
6 — U.S. BUREAU OF LABOR STATISTICS • bls.gov

Sample size and Procedures

 Starting sample size: 2,500 addresses

 Field period: October 2019-April 2020

 Two visits, one for diary placement and one at the end of the diary keeping period 

(“pickup”)

 Push to online diary if the household respondent is eligible, if they refuse or are 

ineligible-- they get paper diaries

 Recall tab in the CAPI instrument for interviewer to record additional expenses 

reported by respondent at pickup interview

 Embedded incentives/postcard experiment (see next slide)
7 — U.S. BUREAU OF LABOR STATISTICS • bls.gov

Embedded Incentive/Postcard Experiment

Advance 

Postcard No Postcard

$5 Cash with

Advance 

Letter 

Mailing

1, 250 N/A

No cash with

Advance 

Letter 

Mailing

625 625
8 — U.S. BUREAU OF LABOR STATISTICS • bls.gov

LSF research topics

 Operational Issues

 Impact of $5 incentive

 Impact of pre-notification postcard 

 Technical Issues with online diaries

 Usefulness of respondent materials

 Effectiveness of training and 

protocols

 Data quality

 Mode differences
9 — U.S. BUREAU OF LABOR STATISTICS • bls.gov

The LSF Online Diary
10 — U.S. BUREAU OF LABOR STATISTICS • bls.gov

LSF Online Diary Design

 Developed in Centurion (Census’ 

proprietary software for online surveys)

 Device-optimized based on screen size

Desktop/Tablet version

Mobile version

 Must be connected to the internet (no 

data stored on device)
11 — U.S. BUREAU OF LABOR STATISTICS • bls.gov

Login Screen
12 — U.S. BUREAU OF LABOR STATISTICS • bls.gov

Help and changing the password/username
13 — U.S. BUREAU OF LABOR STATISTICS • bls.gov

Desktop Expense Entry
14 — U.S. BUREAU OF LABOR STATISTICS • bls.gov

Mobile expense entry
15 — U.S. BUREAU OF LABOR STATISTICS • bls.gov

Expense categories
16 — U.S. BUREAU OF LABOR STATISTICS • bls.gov

Expense entry
17 — U.S. BUREAU OF LABOR STATISTICS • bls.gov

Entering an expense
18 — U.S. BUREAU OF LABOR STATISTICS • bls.gov

Preliminary Results
19 — U.S. BUREAU OF LABOR STATISTICS • bls.gov

A note on COVID-19 interruptions

 In mid-March, the Census Bureau stopped personal visits and the National Processing 

Center for Census was temporarily closed

 LSF Impact: 

New field procedures went into place in March, and all diaries were placed and/or picked up by 

phone

The Census National Processing Center closed so we have not yet received the last three months 

of data.
20 — U.S. BUREAU OF LABOR STATISTICS • bls.gov

Preliminary Results

(October – December)
21 — U.S. BUREAU OF LABOR STATISTICS • bls.gov

Participation

 Response rates

 Incentives

 Pre-notification Postcard
22 — U.S. BUREAU OF LABOR STATISTICS • bls.gov

Percent of LSF Eligible 

(n-eligible =1,062)

Percent of CE Production 

Diary Eligible 2018 q4

(n-eligible =5,072)

Complete Cases 47.2 % 51.5 %

Non-interview

(Noncontact)

10.4 % 7.3 %

Non-interview (Refused) 27.9 % 27.9 %

Non-interview

(Other*)

14.6 % 13.3 %

Overall Response

* For production, “non-interview (other)” includes reclassification, where an edit check results in reclassifying a 

completed interview into a nonresponse diaries. This step has not been completed for LSF data.

Percent of LSF Eligible 

(n-eligible =1,062)

Complete Cases 47.2 %

Non-interview

(Noncontact)

10.4 %

Non-interview (Refused) 27.9 %

Non-interview

(Other*)

14.6 %
23 — U.S. BUREAU OF LABOR STATISTICS • bls.gov

Embedded Incentive/Postcard Experiment

Advance 

Postcard No Postcard

$5 Cash with

Advance 

Letter 

Mailing

1, 250 N/A

No cash with

Advance 

Letter 

Mailing

625 625
24 — U.S. BUREAU OF LABOR STATISTICS • bls.gov

Response Rates

Advance 

Postcard No Postcard

$5 Cash with

Advance Letter 

Mailing

47.9 %

(n=251)

N/A

No cash with

Advance Letter 

Mailing

49.6 %

(n = 132)

42.9 %

(n = 115)

n represents the number of complete cases
25 — U.S. BUREAU OF LABOR STATISTICS • bls.gov

Response Rates

Advance 

Postcard No Postcard

$5 Cash with

Advance Letter 

Mailing

47.9 %

(n=251)

N/A 47.9 %

(n=251)

No cash with

Advance Letter 

Mailing

49.6 %

(n = 132)

42.9 %

(n = 115)

46.3 %

(n=247)

n represents the number of complete cases
26 — U.S. BUREAU OF LABOR STATISTICS • bls.gov

Response Rates

Advance 

Postcard No Postcard

$5 Cash with

Advance Letter 

Mailing

47.9 %

(n=251)

N/A

No cash with

Advance Letter 

Mailing

49.6 %

(n = 132)

42.9 %

(n = 115)

48.5 %

(n=383)

42.9 %

(n=115)

n represents the number of complete cases
27 — U.S. BUREAU OF LABOR STATISTICS • bls.gov

Online Diary Placement
28 — U.S. BUREAU OF LABOR STATISTICS • bls.gov

Online Diary Placement

 Push to online diary first, paper as a backup (ineligible or 

otherwise refused paper)

 Eligibility requirements: 

Online access at home

Go online at least 1-2 per week

English-speaking
29 — U.S. BUREAU OF LABOR STATISTICS • bls.gov

Online Diary Eligibility (n=498)

56.2%

14.1%

29.7%

Home Internet Access - Frequent Use Home Internet Access - Infrequent Use No Home Internet Access

Eligible for online diary
30 — U.S. BUREAU OF LABOR STATISTICS • bls.gov

Diary Placement Modes (n=501)

52.9%

47.1%

0.0%

10.0%

20.0%

30.0%

40.0%

50.0%

60.0%

70.0%

80.0%

90.0%

100.0%

Paper Online

Percent of all respondents
31 — U.S. BUREAU OF LABOR STATISTICS • bls.gov

Respondent Experience with the Online Diary
32 — U.S. BUREAU OF LABOR STATISTICS • bls.gov

Respondent experience – How easy or difficult was 

it to log in to the online diary? (n=215)

69%

14%

3% 3% 3%

7%

0%

10%

20%

30%

40%

50%

60%

70%

80%

Very easy Somewhat easy Neither easy nor

difficult

Somewhat

difficult

Very difficult I did not try to

log in

Percent of online respondents
33 — U.S. BUREAU OF LABOR STATISTICS • bls.gov

Paradata – login attempts/failures (n=256)

n percent

Failed login on first attempt 56 22.2%

At least one failed login during 113 44.8%

No Failed logins 139 55.2%
34 — U.S. BUREAU OF LABOR STATISTICS • bls.gov

Logins by Device

Device Type Respondents Average number

of logins

Desktop Only 102 7.4

Mobile Only 96 6.6

Both 45 8.7
35 — U.S. BUREAU OF LABOR STATISTICS • bls.gov

Time Spent in Diary (n= 228)

Avg Total Duration: 40.1 min

Median Total Duration: 27.6 min
36 — U.S. BUREAU OF LABOR STATISTICS • bls.gov

Respondent Daily Time Spent in Diary (n=210)

0

10

20

30

40

50

60

70

0 1 2 3 4 5 7 8 9 10 12 15 20 25 30 45 60

Number of Respondents

Respondent Estimated Daily Duration in Diary (in minutes)
37 — U.S. BUREAU OF LABOR STATISTICS • bls.gov

Respondent Total Time Spent in Diary (n=210)

0

10

20

30

40

50

60

70

0 14 28 42 56 70 98 112 126 140 168 210 280 350 420 630 840

Number of Respondents

Respondent Estimated Total Duration in Diary (in minutes)

Avg Total Duration: 133.7 min

Median Total Duration: 98 min
38 — U.S. BUREAU OF LABOR STATISTICS • bls.gov

Respondent experience – length (n=211)

32% 31%

27%

7%

2%

0%

5%

10%

15%

20%

25%

30%

35%

Very short Somewhat short Neither short nor

long

Somewhat long Very long

Percent of online respondents
39 — U.S. BUREAU OF LABOR STATISTICS • bls.gov

How burdensome was filling out the diary to you?

40%

34%

18%

4% 4%

38%

31%

18%

7% 6%

0%

5%

10%

15%

20%

25%

30%

35%

40%

45%

Not at all

burdensome

A little burdensome Somewhat

burdensome

Very burdensome Extremely

burdensome

Percent of respondents

Online (n=213) Paper (n=217)

Chi-Sq: 3.3875

Pr = 0.4952
40 — U.S. BUREAU OF LABOR STATISTICS • bls.gov

How difficult was it for you to complete the diary?

74%

16%

6%

2% 2%

58%

21%

7% 8% 6%

0%

10%

20%

30%

40%

50%

60%

70%

80%

Not at all difficult A little difficult Somewhat difficult Very difficult Extremely difficult

Percent of respondents

Online (n=213) Paper (n=216)

Chi-Sq: 15.763

Pr = 0.0034
41 — U.S. BUREAU OF LABOR STATISTICS • bls.gov

Lessons Learned
42 — U.S. BUREAU OF LABOR STATISTICS • bls.gov

Lessons Learned

 Online diary placement: lower rate than expected, despite 

screening. Barriers include language, tech savviness, and 

connectivity. 

 Incentives: minimal increase in response rate from $5 token 

incentive

 Postcards: larger increase in response rate

 Difficulty with logins – still a concern

 In general, respondent experience with the online diary has 

been positive
43 — U.S. BUREAU OF LABOR STATISTICS • bls.gov

Next Steps

 Go/No-Go decision on implementation in 2022 (pending full 

analysis)

Pending “Go” decision: 

 Continuous improvements to the online diary and protocols

Exploring ways to collect business name

Auto-filled response or response look up?

 Updates to training based on lessons learned
Contact Information

44 — U.S. BUREAU OF LABOR STATISTICS • bls.gov

Laura Erhard

Chief, Branch of Research and Program Development

Division of Consumer Expenditure Surveys

202-691-5119

Erhard.Laura@bls.gov