Usability Testing for the 

Consumer Expenditure Survey

Electronic Diary

Brandon Kopp

CE Symposium

July 14, 2015
The Design Process

Concept 

Development

Usability 

Testing

Feasibility

Testing Implementation
2016

2011

2014

2012

2013

2015

Mobile

Desktop

Timeline

3

Desktop Diary Usability Testing

Gemini Proof of 

Concept Test

Desktop Diary Feasibility Test

Individual Diaries 

Feasibility Test

Concept 

Development Usability Testing FeasibilityTesting Implementation
2016

2011

2014

2012

2013

2015

Mobile

Desktop

Timeline

4

Desktop Diary Feasibility Test

Desktop Diary Usability Testing

Gemini Proof of 

Concept Test Individual Diaries 

Feasibility Test

Mobile Diary (Phase I)

Mobile Diary (Phase II)

Desktop & Mobile Diary

Usability Testing

Concept 

Development Usability Testing FeasibilityTesting Implementation
Concept Development

5

Early Ideas Requirements & 

Specifications become

Concept 

Development Usability Testing FeasibilityTesting Implementation
Mobile Diary Phase I

 Get feedback on initial design

 Test on wireframe instrument

6
7
8

Mobile Diary (Phase I)

Materials

Login Initial Setup Home Screen
9

Mobile Diary (Phase I)

Materials

General Data Entry Food and Drink 

Away From Home

Food and Drink For 

Home Consumption

Clothing, Shoes 

Jewelry, & 

Accessories

All Other Products & 

Services
Mobile Diary (Phase I)

Methods

 10 Participants

3 with “A Moderate Amount” of smartphone 

experience

7 with “A Lot” of experience

 14 tasks that covered the basic operations a 

respondent would be expected to perform

Logging in, performing setup, navigation (X 3)

Entering expenditure data (X 8)

Editing/deleting data (X 3)

10
Mobile Diary (Phase I)

Results

11

Task Name N

Percent

Successful

Initial Setup

Login 10 90%

Start Date 10 100%

Info Screen 10 50%

Data Entry

Own Non-Food Purchase 10 70%

Clothing 10 70%

Other Cleaners 10 100%

Food Away 10 80%

Other DVD 10 90%

Own Food Purchase 10 70%

Grocery List 10 50%

Other Bill 10 100%

Data Editing

Edit Price 10 90%

Delete Item 10 90%

Edit Date 10 90%
Mobile Diary (Phase I)

Recommendations

 12 recommendations

 Make minor 

instrument changes

 Allow users to set 

their own password

 Emphasize 

placement 

procedures in Phase 

II testing

13

CES Individual Diary

Cancel Save

Enter Description

$0.00

Select Category

May 29, 2013

Logout CE Diary i

Enter Description

$0.00

May 29, 2013

INFO

Simplify title

INFO

Replace buttons with 

something more 

descriptive

Left-align text boxes 

for uniform look
Mobile Diary (Phase II)

14

 Took place five months after Phase I

 Used functional prototype instrument

 Participants used their own smartphone
Mobile Diary (Phase II)

Materials

15

Login Initial Setup Home Screen Change Password 

Screen
Mobile Diary (Phase II)

Materials

16

General Data Entry Food and Drink 

Away From Home

Food and Drink For 

Home Consumption

Clothing, Shoes 

Jewelry, & 

Accessories

All Other Products & 

Services
Mobile Diary (Phase II)

Methods

17

 29 Participants

Screened by operating system 

& smartphone experience

 13 tasks were similar to 

Phase I

 New, change password task

 Emphasized diary placement 

instructions
Mobile Diary (Phase II)

Placement Instructions

 Walked through pamphlet with participants

Discussed navigation

Emphasized itemization of entries

Discussed categories and data needs for each

18
Mobile Diary (Phase II)

Results

19

Task Name N

Percent

Successful

Initial Setup

Navigate & Login 28 46%

Start Date 28 96%

Set Personalized Password 9 33%

Data Entry

Own Food Purchase 25 80%

Book for Friend, Jeans for Self 24 63%

Enter Car Insurance Bill 22 95%

Own Non-Food Purchase 28 93%

Dinner, Movie 26 35%

Drinks 25 48%

Long Receipt 22 27%

Data Editing

Delete Pants 21 90%

Change Book Details 18 100%

Edit Price 24 58%
Mobile Diary (Phase II)

Recommendations

 22 recommended changes

10 relating to the password 

change process

 Entering a field should bring up 

the optimal virtual keyboard

 Reformat change password 

screen to emphasize the 

requirements

21
Desktop & Mobile Diary

 The Desktop & Mobile Diary are now 

combined and can be accessed at the same 

URL

 Website formats the screen for the device 

being used

 Goals:

Conduct usability testing for the new Desktop 

Diary

See whether participants have difficulty switching 

back and forth between modes

22
Desktop & Mobile Diary

Desktop Diary Changes

23

Desktop Diary (2015) Desktop Diary (2013)
Desktop & Mobile Diary

Mobile Diary Changes

24

Mobile Diary (2013) Mobile Diary (2015)
Desktop & Mobile Diary

Materials

25

Desktop Mobile
Desktop & Mobile Diary

Methods

 20 participants

 Screened to have at least some 

experience with both a smartphone and 

a computer

 Used their own smartphone for Mobile 

and lab computer for Desktop

26
Desktop & Mobile Diary

Methods

 Participants received 

placement instructions 

emphasizing data 

requirements for diary

 Used an early version 

of the User Guide that 

will be used in the POC 

test

27
Desktop & Mobile Diary

Tasks

28

Task Name Task Description Group 1 Group 2

1. Navigate to Website and Log In Setup Mobile Desktop

2. Set Start Date Setup Mobile Desktop

3. Enter Own Non-Food Purchase Data Entry Mobile Desktop

Device Change

4. Navigate to Website and Log In Setup Desktop Mobile

5. Enter Own Food Purchase Data Entry Desktop Mobile

6. Enter Book for Friend, Clothes for Self Data Entry Desktop Mobile

7. Enter Dinner, Movie Data Entry Desktop Mobile

8. Enter Car Insurance Data Entry Desktop Mobile

9. Change Book Details Editing Desktop Mobile

Device Change

10. Delete Pants Editing Mobile Desktop

11. Enter Shoes Data Entry Mobile Desktop

12. Enter Five Items Data Entry Mobile Desktop

13. Enter Food Away Expense Data Entry Mobile Desktop

Device Change

14. Enter Seven Items Data Entry Choice Choice
Desktop & Mobile Diary

Results

 Several participants had trouble 

navigating to the site

 Only one participant was unable to 

enter the username and password

This participant was not skilled with using 

a smartphone

29

Task Name N

Mobile %

Successful

Desktop %

Successful

Initial Setup

Navigate & Login 20 90% 80%

Start Date 20 90% 100%

Navigate & Login 20 60% 100%
Desktop & Mobile Diary

Results

 42 of the 51 issues were with data 

requirements and not with data entry

 The remaining issues were due to design

30

Task Name N

Mobile %

Successful

Desktop %

Successful

Data Entry

Own Non-Food Purchase 20 73% 100%

Own Food Purchase 20 75% 83%

Book For Friend, Clothes For Self 20 63% 58%

Dinner, Movie 20 38% 67%

Car Insurance Bill 20 88% 100%

Shoes 19 100% 78%

5-Item Receipt 19 70% 67%

Coffee Shop 19 60% 56%

7-Item Receipt 18 100%* 59%
Desktop & Mobile Diary

Results

 One person was unable to find the 

delete button in the Desktop Diary

Several others took an extended time to 

find it

31

Task Name N

Mobile %

Successful

Desktop %

Successful

Data Editing

Change Book Details 18 88% 73%

Delete Pants 19 90% 67%
Desktop & Mobile Diary

Recommendations

 Total of 33 recommendations

Most for the Desktop Diary

Almost all have been made leading up to 

POC

 Use an easy to type URL (e.g., 

cediary.gov) to redirect people to the 

CE Diary

32
Desktop & Mobile Diary

Recommendations

33

Add an Expense

Summary of Expenses

February 11, 2014 Enter Description $0.00

Select One Select One

Date Description Cost Category

Clothing, Shoes, Jewelry, and 

Was this item for: Age: Purchased for someone outside your household

February 09, 2014 Tennis Shoes $120.00

Male 16 & Over

Date Description Cost Category

Clothing, Shoes, Jewelry,

Was this item for: Gender: Age: Purchased for someone outside your household

Save

Cancel Save X

Edit February 07 Strawberries $4.20 Food and Drink for Home Consumption

Cancel

Clothing, Shoes, Jewelry, and Accessories

Delete

Gender:

Clothing, Shoes, Jewelry, and Acc

Enter Your Expense Below
Going Forward

 Contract with Westat to conduct the Electronic 

Desktop Diary Design Improvements (ED3I)

 Propose alternative designs

 Usability test two of those designs

 Provide recommendations for future

 Work with Census to review and usability test 

alternative login requirements

 7-Digit PIN as Password

 Other feasible alternatives

35
Contact Information

Brandon Kopp

Research Psychologist

Office of Survey Methods Research

www.bls.gov/osmr

202-691-7514

kopp.brandon@bls.gov