2017 CE Survey Methods Symposium: U.S. Bureau of Labor Statistics

July 18, 2017

Online Diaries for Everyone: 

Data Quality, Device Usage and 

Compliance with Personal 

Expenditure Diaries

Douglas Williams, Westat

Hanyu Sun, Westat
• Background & Online Diary Improvement Project

• Usability test design and methods

• Findings: compliance

• Findings: diary and device usage

• Findings: data quality

• Summary

Outline
• CE Diary one of two components of the CE Survey capturing minor and 

frequent expenditures.

• Current diary uses PAPI and a household respondent covering two oneweek reporting periods.

• Continued review and development of online platform for CE Diary

(see Elkin, To, and Williams, 2017 – FedCASIC).

– Web Diary Feasibility Test – 2013

– Individual Diaries Feasibility Test (IDFT) – 2014

– Proof of Concept Test – 2015

– Electronic Diary Desktop Design Improvements – 2016

Background
• Developed collaboratively with BLS building on prior research and diary 

data requirements

– Modernization – flexible instrument that can add and adapt to new 

technologies

– Accessible across multiple platforms and mobile optimized

• Smartphone ownership: 55% in 2013; 77% in 2016

– Designed for personal diary assignment

• Instrument developed using open-source programming software

– Adaptive (web) design – optimized for mobile

– Design elements (look and feel) same across device types used to access

Online Diary Improvement Project
• Recruited households for usability test

– Desktop/Laptop and Smartphone ownership

– Multiple age eligible household members (71% of households)

• Procedure: placement interview; diary reporting; pick-up interview

– In-person interviews conducted March, 2017 – May, 2017

– Completed 62 placement interviews; 61 pick-up interviews

– Incentive: $40 for the main diarist; $10 for other HHM

Design & Methods:

Online Diary Improvement Project
Desktop Diary View
Mobile Diary View
Usability Test Results
• Compliance & Use

– How well do different household members comply with the diary task? 

• Device Usage

– What are the characteristics diary usage by device?

– Are there differences in types of expenditures entered by device type?

– Are there any barriers to accessing the online diary?

• Data Quality

– Do expenditure descriptions indicate problems or category mismatches? 

Research Questions
• How well do different household members comply with the diary task?

– Composition of recruited sample

• 62 main diarist – responsible for encouraging other HHM participation

• 72 other household members

• Access & Use

– Access: log-in and create password

– Use: enter any expense

Compliance
• Access & Use

– 61 of 62 (98%) main diarist accessed the diary

– 50 of 72 (69%) other household members accessed the diary

– Similar results for use

• Comply with directions to access and change password day of visit

Compliance: Results

Days from visit 0 1 2 3 4 5 6 7+

Main Diarist 51% 16% 8% 7% 5% 5% 2% 7%

Other 22% 18% 10% 10% 6% 14% 14% 6%
Expense Reporting Behavior

4.3

6.4 6.9

8.3

6.9

5.5

9.9

11.3

0.0

2.0

4.0

6.0

8.0

10.0

12.0

1 2 3 4 5 6 7 8+

Reporting Day

4.0

2.4

4.2

8.1

2.8

5.3

6.0 5.9

0.0

2.0

4.0

6.0

8.0

10.0

12.0

1 2 3 4 5 6 7 8+

Reporting Day

Main Diarist Other Household Members
• Timely entry of expenses

– Suggest attentiveness to the diary task

– Better data: less time for recall; less likely to forget expense

Expense Entry Timeliness

Difference 0 1 2 3 4 5 6+

Main Diarist 31.0 33.2 11.3 11.2 4.6 2.5 6.2

Other 15.1 36.4 9.0 9.2 9.0 6.7 14.6

All 27.0 34.0 10.7 10.7 5.7 3.6 8.3
• What are the characteristics of diary usage by device?

– Half of main diarist used mobile / one-third only used mobile

– Why not more mobile: smaller screen, lack of tactile keyboard

– Desk/Laptop used more (1,522; 679), mobile entries were more timely

Device Usage: Characteristics

Device Type Desk / Laptop Mobile Both

Main Diarist (n = 61) 43 31 13

Other (n = 49) 37 20 7

Difference 0 1 2 3 4

Desk/Laptop 24% 31% 12% 12% 21%

Mobile 33% 41% 8% 8% 10%
• Are there differences in types of expenditures entered by device type?

– ODIP – CE expense categories expanded from 4 to 10

Device Usage: Expense Types

• Food and Drinks Away from Home

• Food and Drinks for Home Consumption

• Clothing, Shoes, Jewelry, and Accessories

• All Other Products and Services

• Meals, Snacks, and Drinks Away From Home

• Food and Drinks for Home Consumption

• Clothing, Shoes, Jewelry, and Accessories

• Medical Expenses and Supplies

• Entertainment and Recreation

• Home Furnishings and Decorative Items

• Education Expenses and Supplies

• Transportation Expenses

• Personal Care or Hygiene Items

• All Other Products, Services, and Expenses
Expense Type Desk/Laptop Mobile All

1 - Meals, Snacks, and Drinks Away From Home 15.0% 30.2% 19.7%

2 - Food and Drinks For Home Consumption 52.9 21.5 43.2

3 - Clothing, Shoes, Jewelry, and Accessories 6.0 6.8 6.2

4 - Medical Expenses and Supplies 1.9 2.4 2.0

5 - Entertainment and Recreation 2.1 5.7 3.2

6 - Home Furnishings and Decorative Items 0.3 2.5 1.0

7 - Education Expenses and Supplies 0.4 0.7 0.5

8 - Transportation Expenses 6.6 9.7 7.6

9 - Personal Care, or Hygiene Items 3.7 8.0 5.0

10 - All Other Products, Services, and Expenses 11.1 12.5 11.5

Table: Expense Type by Device Type
• Are there any barriers to accessing the online diary?

– Setting/remembering password most reported barrier

– Password

• Eight characters: number, upper & lower-case letter, special character 

(!@#$%^&*)

• One-half of all failed attempts occurred before setting password

Barriers to Access: Failed Login Attempts

Total number of 

failed attempts 0 1 2 3 4 5 6 7+

Percent 46.9 12.6 8.1 3.6 3.6 3.6 4.5 17.1
• Do expenditure descriptions indicate problems or category mismatches?

– Description field: open-ended text field for describing expense

• Red polo shirt; 2 liter coke; Bananas; Gasoline; Etc... 

– Type of issues identified – independent coding and review

• Including multiple items in one entry

• Vague descriptions: item or expense is unclear (e.g., food, clothes, 

cleaning)

• Establishment name: (e.g., restaurant name)

• Incorrect category

• Unidentifiable: gibberish 

Data Quality
Description Issue Type Count Percent/All Expenses

Multiple items 52 2.3%

Vague description 37 1.6%

Establishment name 59 2.6%

Incorrect category 41 1.8%

Unidentifiable 13 0.6%

Data Quality: Expense Description

• Overall, poor quality descriptions low (of n = 2,255 total expenses)

• Issues clustered within respondents, or affect specific category

– 14 respondents account for over half (55%) of coded issues

– Establishment name generally used for meals away from home (restaurant 

name)
Summary
• Personal Diaries

– Near uniform use of online diary by main diarist (household respondent)

• Usage lower for other HHM, but encouraging

• Minimum level? Estimate the proportion of expenses lost for household 

diaries versus loss due to nonparticipation in personal diaries.

• Not ideal to rely on the main diarist to encourage and motivate other 

HHM to participate

• Indications that other HHM start later and abandon sooner

Summary
• Online Diaries

– Offer dimensions not available with paper: use of multiple devices, mobility

– Mobile devices were frequently used, but not as expected

• Most respondents used at home – still convenient

• Receipt complexity influencing device type used?

– Expenses entered via mobile device were more timely – better data?

– Password the biggest barrier to access – and mobility

– Data quality – low incidence of issues, but still problematic

• E.g., 52 instances of multiple entries can equal 100-150 missed expenses

Summary
Thank you!

DouglasWilliams@Westat.com