1 — U.S. BUREAU OF LABOR STATISTICS • bls.gov

Assessing the Impact of Device Type on Data 

Quality in the Consumer Expenditure (CE) 

Online Diary Survey

Gray Jones, BLS

AAPOR Annual Conference 2024

May 15, 2024
2 — U.S. BUREAU OF LABOR STATISTICS • bls.gov

Agenda

 CE Diary Survey Background

 CE Online Diary Background 

 Motivation for Analysis

• Past Diary Paradata Research 

 Device Use in the Online Diary

 Analysis Results:

• Devices and Demographics

• Reporting Behavior 

• Relationship to Data Quality

 Next Steps for CE Paradata use
3 — U.S. BUREAU OF LABOR STATISTICS • bls.gov

CE Diary Survey Background

Consumer Expenditure Surveys (CE)

The CE Interview 

Survey 

The CE Diary 

Survey

CE Paper Diary CE Online Diary
4 — U.S. BUREAU OF LABOR STATISTICS • bls.gov

CE Online Diary Background

Development of CE Online 

Diary Instrument

Large Scale Feasibility Test of 

Online Diaries (LSF) 10/2019 

– 4/2020

CE Online Diary 

Supplemental Use

6/2020

CE Online Diary 

in Production

7/2022
5 — U.S. BUREAU OF LABOR STATISTICS • bls.gov

CE Online Diary Paradata

 Data generated as a by-product of the survey’s a collection process

 Analysis of paradata can lead to...

 Improved data collection efficiency

 Reduced survey fielding costs

 Better understanding of measurement error
6 — U.S. BUREAU OF LABOR STATISTICS • bls.gov

Analysis of CE Online Diary Paradata

 BLS has already begun analyzing CE Diary Survey Paradata

Large Scale Feasibility Test Preliminary Report (Krishnamurty et al., 

2021)

2021 Consumer Expenditure Survey Online Diary Paradata Analysis 

(Jones, G., and P. Krishnamurty)
7 — U.S. BUREAU OF LABOR STATISTICS • bls.gov

Motivation for Research 

 Past analysis found differences by the device type used. 

 Primarily driven by “Mobile Only” device users. 

2021 Paradata Analysis 

Results by Device Type

Desktop Only 

(n=1,162)

Mobile Only 

(n=387)

Mixed Use 

(n=252)

Median Logins per CU 10 logins 6 logins 11 logins

Median Time Spent per CU 27.5 minutes 10.3 minutes 26.5 minutes

Median Expenditure Count 34 entries 27 entries 33 entries

Median Expenditure Total (In 

Dollars)

$555.31 $400.50 $559.00
8 — U.S. BUREAU OF LABOR STATISTICS • bls.gov

Motivation for Research 

 These findings led to questions about data quality for mobile 

users.

Are discrepancies in reporting driven mostly by demographic 

differences? (e.g., lower income  lower entries and expenditures)

Or are differences driven by mobile device users reporting behavior?
9 — U.S. BUREAU OF LABOR STATISTICS • bls.gov

Purpose of Analysis

 Disentangling the relationship 

between respondent 

characteristics, device use, and 

diary reporting behavior.

 Important for optimizing survey 

fielding procedures and 

understanding data quality. 

Reporting 

Behavior

Respondent 

Characteristics

Device Use
10 — U.S. BUREAU OF LABOR STATISTICS • bls.gov

Device Type Analysis
11 — U.S. BUREAU OF LABOR STATISTICS • bls.gov

Device Type Analysis: Data Use and Set Up

 Data Used

CE Diary Data from 2022q3 and 2022q4

 Pre-Analysis Steps

Generate device type information using CE Online Diary Paradata.

Generate expenditure counts and totals (in dollars) using unprocessed 

expenditure data. 

Combine these measures with Consumer Unit (CU) level respondent 

characteristics data associated with complete cases. 
12 — U.S. BUREAU OF LABOR STATISTICS • bls.gov

Device Type Analysis: Device Definitions

 Desktop Only User

Includes online diary users accessing the instrument with either a 

desktop, laptop, or tablet, but never a mobile device. 

 Mobile Only User

Includes online diary users accessing the instrument with only a mobile 

device (never a desktop, laptop, or tablet). 

 Mixed Users

Includes online diary users who accessed the instrument with a mobile 

device, as well as a desktop, laptop, or tablet. 
13 — U.S. BUREAU OF LABOR STATISTICS • bls.gov

Device Type Analysis

 Analytical Steps

1. Overview of Device Use

2. Comparison of Nonresponse

3. Comparison of Demographic Characteristics 

4. Multivariate analysis
14 — U.S. BUREAU OF LABOR STATISTICS • bls.gov

Overview of Device Use

Desktop Only User

• All actions recorded in the paradata associated with a desktop

• Accounted for 372 or 62.4% of CUs

Mobile Only User

• All actions recorded in the paradata associated with a mobile device

• Accounted for 131 or 22.0% of CUs

Mixed Device User

• Actions recorded in the paradata split between desktop and mobile devices. 

• Accounted for 93 or 15.6% of CUs
15 — U.S. BUREAU OF LABOR STATISTICS • bls.gov

Nonresponse Indicators by Device Type

 In addition to previously mentioned paradata metrics.

 Mobile Only device users have slightly higher rates of Item 

Nonresponse & Bundling.

Nonresponse Indicators Desktop Only 

(n=372)

Mobile Only 

(n=131)

Mixed Use 

(n=93)

Item Nonresponse (Blank Costs) 0.4% 1.0% 0.3%

Invalid Item Descriptions 1.7% 2.0% 2.8%

Average Bundling Rate 11.7% 13.4% 9.7%
16 — U.S. BUREAU OF LABOR STATISTICS • bls.gov

Main Research Questions

1. Are discrepancies in reporting driven by demographics? 

2. Or are differences driven by mobile device users reporting 

behavior?
17 — U.S. BUREAU OF LABOR STATISTICS • bls.gov

Comparison of Demographic Characteristics 
18 — U.S. BUREAU OF LABOR STATISTICS • bls.gov

Demographic Comparison: Device Use by Age 

Group

18.2

17.1

19.3

13.2

11.1

36.4

31.4

24.6

20.7

7.4

45.5

51.4

56.2

66.1

81.5

0.0 10.0 20.0 30.0 40.0 50.0 60.0 70.0 80.0 90.0

Under 25 years

25-34 years

35-49 years

50-64 years

65 years and older

Percentage of Device Use

Desktop (n=372) Mobile (n=131) Mixed (n=93)
19 — U.S. BUREAU OF LABOR STATISTICS • bls.gov

Demographic Comparison: Device Use by 

Education Level

18.2

6.6

16.5

17.2

45.5

44.7

27.7

14.8

36.4

48.7

56.2

68.1

0.0 10.0 20.0 30.0 40.0 50.0 60.0 70.0 80.0

Less than high school

High school graduate

Some college

College graduate

Percentage of Device Use

Desktop (n=372) Mobile (n=131) Mixed (n=93)
22 — U.S. BUREAU OF LABOR STATISTICS • bls.gov

Demographic Comparison: Device Use by PreTax Income Level

12.7

18.6

15.1

15.5

22.8

32.7

24.3

15.5

64.6

48.7

60.5

69.1

0.0 10.0 20.0 30.0 40.0 50.0 60.0 70.0 80.0

Less than $35,000

Between $35,001 and $70,000

Between $70,001 and $125,000

Greater than $125,000

Percentage of Device Use

Desktop (n=372) Mobile (n=131) Mixed (n=93)
23 — U.S. BUREAU OF LABOR STATISTICS • bls.gov

Multivariate Analysis
24 — U.S. BUREAU OF LABOR STATISTICS • bls.gov

Multivariate Analysis

 Regression model specified to control for demographic 

characteristics.

 Two Models:

Full Model

Reduced Model 

 Dependent variable: Diary Entry Counts 
25 — U.S. BUREAU OF LABOR STATISTICS • bls.gov

Multivariate Analysis: Full Model 

 Dependent Variable: Diary Entry Counts

Transformed to the Natural Log of Diary Entry Counts

 Main Explanatory Variable of Interest: Mobile Only Device Use

 Additional Control Variables: Age, Allocation, Area Type, CU 

Size, Data Collection Month, Diary Recall, Education, Ethnicity, 

Gender, Homeowner, Income, Race, and Region. 
26 — U.S. BUREAU OF LABOR STATISTICS • bls.gov

Multivariate Analysis: Full Model 

 Results from the full model show that the impact of “Mobile 

Only” device use on entry counts was significant.

 Elements of Age, CU Size, Race, and Region were also found to 

be significant in the full model. 

 These findings informed a reduced model.
28 — U.S. BUREAU OF LABOR STATISTICS • bls.gov

Multivariate Analysis: Reduced Model 

 “Mobile Only” diary use still 

strongly significant.

 Interpretation of the coefficient 

implies a negative relationship 

with expenditure counts. 

 Strengthens claim that “Mobile 

Only” diary use is associated 

with lower quality expenditure 

diary reporting. 

Variables in Reduced Model Coefficient Standard Error

Intercept 3.74 0.148

Online Diary Device: Mixed Use 0.01 0.103

Online Diary Device: Mobile Only*** -0.47 0.093

Race: Other* -0.24 0.111

Race: Black -0.24 0.138

Age: Under 25** -0.57 0.210

Age: 25 - 34* -0.29 0.126

Age: 35 - 49 -0.22 0.116

Age: 50 – 64 0.03 0.111

CU Size: 1*** -0.47 0.092

CU Size: 4+ 0.12 0.100

CU pre-tax income: $35K-<$70K -0.07 0.133

CU pre-tax income: $70K-<$125K 0.14 0.130

CU pre-tax income: >$125K 0.25 0.131

Region: Northeast* -0.28 0.107

Region: Midwest -0.01 0.106

Region: West -0.02 0.104

*p < 0.05, **p < 0.01, ***p < 0.001
29 — U.S. BUREAU OF LABOR STATISTICS • bls.gov

End of Device Type Results
30 — U.S. BUREAU OF LABOR STATISTICS • bls.gov

Conclusions and Recommendations

 Using only a mobile device appears to be associated with suboptimal diary reporting, even when controlling for other 

relevant variables.

 The rate of “Mobile Only” diarists should be monitored moving 

forward to as a data quality metric.

 Consider discouraging “Mobile Only” reporting as a quality 

assurance measure and pushing those respondents to paper.
31 — U.S. BUREAU OF LABOR STATISTICS • bls.gov

Contact Information

Gray Jones

Economist

Consumer Expenditure Surveys

jones.graham@bls.gov