1 — U.S. BUREAU OF LABOR STATISTICS • bls.gov

Results from a Test of Online 

Modes for the Consumer 

Expenditure Diary Survey

Ian Elkin - Bureau of Labor Statistics

Douglas Williams & Hanyu Sun - Westat

2018 Joint Statistical Meetings

07/31/2018

Any opinions expressed in this presentation are those of the authors and do 

not constitute policy of the Bureau of Labor Statistics.
2 — U.S. BUREAU OF LABOR STATISTICS • bls.gov

Outline

I. Background

II. Study Design

III. Usability Test Results

IV. Conclusions and 

Recommended Changes
3 — U.S. BUREAU OF LABOR STATISTICS • bls.gov

I. Background
4 — U.S. BUREAU OF LABOR STATISTICS • bls.gov

Gemini Redesign Project

 “... is the multi-year redesign 

project for the purpose of 

researching, developing, and 

implementing an improved 

survey design to improve data 

quality through a verifiable 

reduction in measurement 

error.” 
5 — U.S. BUREAU OF LABOR STATISTICS • bls.gov

Background

 CE Diary - one of two components of the CE Surveys used to capture smaller and more 

frequent expenditures. 

 Current diary is a paper diary with a household-level respondent and covers two oneweek reporting periods.

 Part of an iterative review and development of online platform for the CE Diary. 

 Web Diary Feasibility Test - 2013

 Individual Diary Feasibility Test (IDFT) - 2014

 Proof of Concept Test - 2015

 Electronic Diary Desktop Design Improvements - 2016 
6 — U.S. BUREAU OF LABOR STATISTICS • bls.gov

II. Study Design
7 — U.S. BUREAU OF LABOR STATISTICS • bls.gov

Online Diary Improvement Project - Overview

 Developed collaboratively with Westat building on prior research and diary data 

requirements. 

 Modernization – Flexible instrument that can add and adapt to new technologies

 Access across multiple platforms specifically mobile optimization

 Person-level diary assignment

 Instrument developed using open-source programming software. 

 Adaptive web design – optimized for mobile and desktop

 Design elements were the same across device types
8 — U.S. BUREAU OF LABOR STATISTICS • bls.gov

Online Diary Improvement Project – Design & 

Methods

 Household recruitment for usability test. 

 Desktop/Laptop and Smartphone ownership

 Multiple age eligible household members (71% of households)

 Procedure: placement interview; diary reporting; pick-up interview 

 In-person interviews conducted March 2017 – May 2017

 Completed 62 placement interviews and 61 pick-up interviews

 Incentive: $40 for the main diarist and $10 for the other household members
9 — U.S. BUREAU OF LABOR STATISTICS • bls.gov

Desktop Diary View
10 — U.S. BUREAU OF LABOR STATISTICS • bls.gov

Mobile Diary View
11 — U.S. BUREAU OF LABOR STATISTICS • bls.gov

III. Usability Test Results
12 — U.S. BUREAU OF LABOR STATISTICS • bls.gov

Research Questions

 Compliance and Usage

How well do different household members comply with diary task?

 Device Usage

What are the characteristics between diary usage by device?

Are there differences in types of expenditures entered by device type?

Are there any barriers to accessing the online diary?

 Data Quality

Do expenditures indicate problems or category mismatches?
13 — U.S. BUREAU OF LABOR STATISTICS • bls.gov

Compliance

 How well do different household members comply with the 

diary task?

Composition of recruited sample

– 62 main diarist – responsible for encouraging other HHM participation

• 48.% above the age of 40, predominantly female, education evenly distributed

– 72 other household members

 Access Use

Access – log-on and create password

Use – enter an expense
14 — U.S. BUREAU OF LABOR STATISTICS • bls.gov

Compliance - Results

 Access and Use

61 of 62 (98%) main diarist accessed the diary 

50 of 72 (69%) other household members accessed the diary 

Similar results for use 

 Comply with directions to access and change password day of 

visit 

Days from visit 0 1 2 3 4 5 6 7+

Main Diarist 51% 16% 8% 7% 5% 5% 2% 7%

Other 22% 18% 10% 10% 6% 14% 14% 6%
15 — U.S. BUREAU OF LABOR STATISTICS • bls.gov

Distribution of Initial Diary Access
16 — U.S. BUREAU OF LABOR STATISTICS • bls.gov

Expense Reporting Behavior
17 — U.S. BUREAU OF LABOR STATISTICS • bls.gov

Expense Entry Timeliness

 Timely entry of expenses 

Suggest attentiveness to the diary task 

Better data: less time for recall; less likely to forget expense

Distributional difference between expense date and entry date

Difference 0 1 2 3 4 5 6+

Main Diarist 31.0 33.2 11.3 11.2 4.6 2.5 6.2

Other 15.1 36.4 9.0 9.2 9.0 6.7 14.6

All 27.0 34.0 10.7 10.7 5.7 3.6 8.3
18 — U.S. BUREAU OF LABOR STATISTICS • bls.gov

Device Usage - Characteristics

 What are the characteristics of diary usage by device? 

Half of main diarists used mobile with one-third only using mobile

Why not more mobile usage: smaller screen, lack of tactile keyboard 

Desk/Laptops used more (1,522;679), mobile more timely 

Device Type Desk/Laptop Mobile Both

Main Diarist (n=61) 43 31 13

Other (n=49) 37 20 7

Difference 0 1 2 3 4

Desk/Laptop 24% 31% 12% 12% 21%

Mobile 33% 41% 8% 8% 10%
19 — U.S. BUREAU OF LABOR STATISTICS • bls.gov

Device Usage – Expense Types
20 — U.S. BUREAU OF LABOR STATISTICS • bls.gov

Barriers to Access – Failed Login Attempts

 Are there any barriers to accessing the online diary? 

Setting/remembering password most reported barrier 

Password 

– Eight characters: number, upper & lower-case letter, special character 

(!@#$%^&*) 

– One-half of all failed attempts occurred before setting a new password 

Total # of failed attempts 0 1 2 3 4 5 6 7+ 

Percent 46.9 12.6 8.1 3.6 3.6 3.6 4.5 17.1
21 — U.S. BUREAU OF LABOR STATISTICS • bls.gov

Data Quality

 Do expenditure descriptions indicate problems or category 

mismatches? 

Description field: open-ended text field for describing expense 

– Red polo shirt; 2 liter coke; Bananas; Gasoline; Etc... 

Type of issues identified – independent coding and review 

– Including multiple items in one entry 

– Vague descriptions: item or expense is unclear (e.g., food, clothes, cleaning) 

– Establishment name: (e.g., Wegmans) 

– Incorrect category 

– Unidentifiable
22 — U.S. BUREAU OF LABOR STATISTICS • bls.gov

Data Quality – Expense Description

 Overall, poor quality descriptions were low (of n = 2,255 total expenses) 

 Issues clustered within respondents, or affect specific category 

 14 respondents account for over half (55%) of coded issues 

Description Issue Type Count Percent/All Expenses

Multiple items 52 2.3% 

Vague description 37 1.6% 

Establishment name 59 2.6% 

Incorrect category 41 1.8% 

Unidentifiable 13 0.6% 
23 — U.S. BUREAU OF LABOR STATISTICS • bls.gov

IV. Conclusions and Recommended Changes
24 — U.S. BUREAU OF LABOR STATISTICS • bls.gov

Conclusions

 Personal Diaries 

Near uniform use of online diary by main diarist (household 

respondent) 

– Usage lower for other HHM, but encouraging 

– Not ideal to rely on the main diarist to encourage and motivate other HHM to 

participate 

– Indications that other HHM start later and abandon sooner 
25 — U.S. BUREAU OF LABOR STATISTICS • bls.gov

Conclusions

 Online Diaries 

Offers dimensions not available with paper: use of multiple devices, 

mobility 

Mobile devices were frequently used, but not as expected 

– Most respondents used at home – still convenient 

– Receipt complexity influencing device type used? 

Expenses entered via mobile device were more timely 

Password the biggest barrier to access – and mobility 

Data quality – low incidence of issues, but still problematic 

– E.g., 52 instances of multiple entries can equal 100-150 missed expenses 
26 — U.S. BUREAU OF LABOR STATISTICS • bls.gov

Recommended Changes

 Diary Placement

Obtain main diarist commitment, collect other HHM contact 

information, and incentivize recruitment

 Design, layout, and function of the online diary

Simplify access

– Passwords & Usernames: simplify requirements, allow username changes, 

allow saving

Provide summary statistics of spending for HH

Active feedback

– E.g., Modal pop-up when entry saved
27 — U.S. BUREAU OF LABOR STATISTICS • bls.gov

Recommended Changes

 Future Enhancements

Features using the smartphone camera

– Receipt scanning, barcode scanning, item detection/machine learning and 

pictures of products

Features using the smartphone microphone

– Speech-to-text functionality

Features using smartphone GPS or geolocation

– Geofencing 
Contact Information

28 — U.S. BUREAU OF LABOR STATISTICS • bls.gov

Ian J. Elkin

Senior Economist

Division of the Consumer Expenditure Surveys

www.bls.gov/cex

202-691-6865

Elkin.Ian@bls.gov