1 — U.S. BUREAU OF LABOR STATISTICS • bls.gov

Household Participation and Response Quality in a 

Survey Featuring Mobile and Desktop Modes

Brett McBride

Ian Elkin

Lucilla Tan

Division of Consumer Expenditure Survey

71st Annual AAPOR Conference

May 13, 2016
2 — U.S. BUREAU OF LABOR STATISTICS • bls.gov

Outline

1. Test design

a. Overview

b. Mode Assignment

c. Demographics

2. Performance

a. Within-HH participation

b. Data quality

i. Logins/Entries

ii. Consistency

3. Discussion
3 — U.S. BUREAU OF LABOR STATISTICS • bls.gov

1. Test Design
4 — U.S. BUREAU OF LABOR STATISTICS • bls.gov

Overview

 The Consumer Expenditure (CE) Diary includes a household 

paper diary

 Households record all expenditures over a 2 week period 

 CE tested feasibility of household individuals completing survey 

using web diaries 

 Test fielded by Census August – December, 2014 with a realized 

sample of 241 households in Philadelphia, Chicago, and Los 

Angeles Regional Offices
5 — U.S. BUREAU OF LABOR STATISTICS • bls.gov

Mode Assignment

Each eligible respondent (age 16+) assigned diary for expenditure 

entries (2 weeks):

Mode Assignment

1. Smartphone ‘Mobile’

2. No smartphone, but home internet ‘Desktop’ (laptop/tablet)

3. No home internet access Screened-out
6 — U.S. BUREAU OF LABOR STATISTICS • bls.gov

Mode Assignment
7 — U.S. BUREAU OF LABOR STATISTICS • bls.gov

Mobile-Optimized Diary Desktop-Optimized Diary
8 — U.S. BUREAU OF LABOR STATISTICS • bls.gov

Demographics

Mobile Desktop Mix

Household Size

- 1 member 17% 20% -

- 2-3 members 49% 62% 65%

- 4+ members 33% 18% 35%

Average members 2.1 2.1 2.6
9 — U.S. BUREAU OF LABOR STATISTICS • bls.gov

Demographics

Mobile Desktop Mix

Household Size

- 1 member 17% 20% -

- 2-3 members 49% 62% 65%

- 4+ members 33% 18% 35%

Average members 2.1 2.1 2.6

Age composition

-youngest 16-35 58% 30% 52%

-youngest 36-50 31% 21% 17%

-youngest 51+ 11% 49% 30%
10 — U.S. BUREAU OF LABOR STATISTICS • bls.gov

Demographics

Mobile Desktop Mix

Household Size

- 1 member 17% 20% -

- 2-3 members 49% 62% 65%

- 4+ members 33% 18% 35%

Average members 2.1 2.1 2.6

Age composition

-youngest 16-35 58% 30% 52%

-youngest 36-50 31% 21% 17%

-youngest 51+ 11% 49% 30%
11 — U.S. BUREAU OF LABOR STATISTICS • bls.gov

Demographics

Mobile Desktop Mix

Household Size

- 1 member 17% 20% -

- 2-3 members 49% 62% 65%

- 4+ members 33% 18% 35%

Average members 2.1 2.1 2.6

Age composition

-youngest 16-35 58% 30% 52%

-youngest 36-50 31% 21% 17%

-youngest 51+ 11% 49% 30%

Highest Education

-no high school 11% 15% 0%

-some college 26% 29% 35%

-college grad 63% 56% 65%

Own home 58% 74% 74%
12 — U.S. BUREAU OF LABOR STATISTICS • bls.gov

2. Performance
13 — U.S. BUREAU OF LABOR STATISTICS • bls.gov

Within-Household Participation

 Household member involvement:

Screened-out (no diary – no internet access)

Non-Participating (assigned diary, but no entries)

‘Total Recall’ (assigned diary, but telling interviewer expenses instead 

of making entries)

Participating (assigned diary, and making entries into mobile or 

desktop diary)
14 — U.S. BUREAU OF LABOR STATISTICS • bls.gov

Within-Household Participation

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

Mobile (N=81) Desktop (N=106) Mix (N=23)

Mean Proportion Household 

Members

Screened-out

Non-participating

Total recall

Participating
15 — U.S. BUREAU OF LABOR STATISTICS • bls.gov

Data Quality: Login Days

4.2

4.9

5.6

0

1

2

3

4

5

6

Mobile (N=66) Desktop (N=106) Mix (N=22)

Average No. of Days with a Login
16 — U.S. BUREAU OF LABOR STATISTICS • bls.gov

Reporting Consistency by Group
17 — U.S. BUREAU OF LABOR STATISTICS • bls.gov

Reporting Consistency by Group and HH Size
18 — U.S. BUREAU OF LABOR STATISTICS • bls.gov

Reporting Consistency by Group and HH Size
19 — U.S. BUREAU OF LABOR STATISTICS • bls.gov

More Total Recall in Mobile HHs

35.8%

12.3%

17.4%

0%

5%

10%

15%

20%

25%

30%

35%

40%

Mobile (N=81) Desktop (N=106) Mix (N=23)
20 — U.S. BUREAU OF LABOR STATISTICS • bls.gov

Difference in Diary Entries Due to Groceries

21.5 22 24

18

23

30

0

10

20

30

40

50

60

Mobile (N=52) Desktop (N=93) Mix (N=19)

Median HH Entries
21 — U.S. BUREAU OF LABOR STATISTICS • bls.gov

Difference in Diary Entries Due to Total Recall

15

41.5

34

0

5

10

15

20

25

30

35

40

45

Mobile (N=81) Desktop (N=106) Mix (N=23)

Median HH Entries
22 — U.S. BUREAU OF LABOR STATISTICS • bls.gov

3. Discussion
23 — U.S. BUREAU OF LABOR STATISTICS • bls.gov

Comparative Performance

Mobile: fewer within HH making entries 

Similar extent of days with log-in 

Mobile: HHs slightly more consistent in entries across weeks

Many mobile HHs had ‘total recall’ and no diary entries

Entries in diary: mobile HHs had fewer entries, due to fewer 

groceries entered
24 — U.S. BUREAU OF LABOR STATISTICS • bls.gov

What accounts for performance difference?

1

2

3

4

5

Ease of Use Security

Mobile (N=52) Desktop (N=81) Mix (N=17)

Very difficult

Very easy

Very unsecure

Very secure
25 — U.S. BUREAU OF LABOR STATISTICS • bls.gov

Summary

Why improved performance for desktop HHs? 

1. Lack of mode choice – total recall in mobile suggests tech difficulties 

2. Desktop may be better for some expenditures (groceries)

3. More widespread desktop participation meant greater self-reporting

4. Possible ‘peer effect’ in desktop HHs?
Contact Information

26 — U.S. BUREAU OF LABOR STATISTICS • bls.gov

Brett McBride

Economist

Division of Consumer Expenditure Survey

www.bls.gov/cex

202-691-5136

mcbride.brett@bls.gov
27 — U.S. BUREAU OF LABOR STATISTICS • bls.gov

Additional Findings: Daily Diary Login Patterns

0%

5%

10%

15%

20%

25%

30%

midnight-3am 3am-6am 6am-9am 9am-noon noon-3pm 3pm-6pm 6pm-9pm 9pm-midnight

Mobile (N=93) Desktop (N=149)