Testing the Feasibility of 

Integrating Outlets into the CE Diary

Erica Yu and Brandon Kopp

CE Methods Symposium

July 14, 2015

All views expressed in this presentation are those of the 

authors and do not necessarily reflect the views or 

policies of the U.S. Bureau of Labor Statistics
Outline

 Background: CE, TPOPS, and CPI

 Research questions

 Study methods

 Findings: CE data quality and burden

 Findings: Outlet data quality

 Summary

2
CPI

Consumer 

Price 

Index

CE

Consumer 

Expenditure 

Survey

Other Sources

TPOPS

Telephone Pointof-Purchase 

Survey

Background: 

CE, TPOPS, and CPI

3
Background: 

Telephone Point-of-Purchase Survey

 Primary source for outlet frame for CPI

 Interview is organized around 

expenditure categories

Expenditures collected for all in household

Categories are similar to those used in CE

Reference periods can be different to those 

used in CE; based on purchase frequency

4
Research Questions

If outlet questions are integrated into 

the CE Diary...

 What is the impact on data quality? 

 What is the impact on respondent 

burden?

Answer: “It depends.”

5
Two Ways of Integrating Outlets 

in the CE Diary

6

Item-Based

1. Shirt

a. July 13th

b. $49.99

c. Macy’s

d. Age 16+

e. Female

2. Blender

a. July 13th

b. $59.99

c. Macy’s

• Embed the outlet 

field within the 

other follow-up 

questions

• Collect outlet for 

every item
Two Ways of Integrating Outlets 

in the CE Diary

7

Item-Based

1. Shirt

a. July 13th

b. $49.99

c. Macy’s

d. Age 16+

e. Female

2. Blender

a. July 13th

b. $59.99

c. Macy’s

Advantages

• Minimal disruption

to original design

Disadvantages

• Additional burden

for every item
Two Ways of Integrating Outlets 

in the CE Diary

8

Transaction-Based

1. Macy’s on July 13th

a. Shirt

i. $49.99

ii. Age 16+

iii. Female

b. Blender

i. $59.99

c. Sports watch

i. $32.99

ii. Age 16+

• Organize the diary 

around transactions

• Collect outlet once 

per transaction 

regardless of the 

number of items
Two Ways of Integrating Outlets 

in the CE Diary

9

Transaction-Based

1. Macy’s on July 13th

a. Shirt

i. $49.99

ii. Age 16+

iii. Female

b. Blender

i. $59.99

c. Sports watch

i. $32.99

ii. Age 16+

Advantages

• Potentially better

match with the way 

people recall 

expenses

Disadvantages

• Major disruption to 

the original Diary 

design
Experiment Design

 “Control” diary

Current CE Diary, item-based, without outlets

 Control “Plus” diary

Current CE Diary, item-based, plus an outlet 

field embedded in the other follow-up fields

 “Transactions” diary

Redesigned CE Diary organized around 

transactions, with outlet as an initial field

10
Participant Task

1. Introduction to CE

2. Told about a fictional 

expense

 Item description, price, 

date, outlet name and 

location

 Paper receipt to refer to 

at any time

3. Record expense into 

diary

4. Repeat steps 2-3

11
Screenshot: 

“Control” Diary

12
Screenshot: 

“Control” Diary

13
Screenshot: 

“Plus” Diary

14
Screenshot: 

“Plus” Diary

15
Screenshot: 

“Transactions” Diary

16
Screenshot: 

“Transactions” Diary

17
Screenshot: 

“Transactions” Diary

18
Methods

 Cognitive lab study at the BLS

 n = 60 participants; 20 per diary

Convenience sample

Qualitative test of feasibility

Recruited from our database of 

participants, local community, contractors 

working in the building

 Task duration about 60 minutes

 Compensation of $40 

19
RESULTS: 

REPORTING EXPENDITURES

20
Data Quality:

Item-level reporting

21

• Bundle “household

goods”?

• Make one entry for total 

expense rather than 

separate entries for items?
Data Quality:

Item-level reporting

 No differences in understanding across 

diary conditions

22

Condition

n 

Total

n 

Adequate

n 

Inadequate

Proportion 

Adequate

Control 20 17 3 85%

Plus 20 15 5 75%

Transactions 20 19 1 95%

Fisher’s exact test, ns
Data Quality:

Meal-level reporting

23

• Itemize dishes?

• Omit tax or gratuity?
Data Quality:

Meal-level reporting

 No differences in understanding across 

diary conditions

24

Condition

n 

Total

n 

Adequate

n 

Inadequate

Proportion 

Adequate

Control 20 11 9 55%

Plus 18 11 7 61%

Transactions 18 13 5 72%

X2(2, N = 56) = 1.22, ns
Data Quality:

Follow-up questions

25

• Pay more attention 

to entering outlets?

• Skip over CE followup questions?
Data Quality:

Follow-up questions

 No differences in reporting across diary 

conditions

26

Condition

n 

Total

n 

Adequate

n 

Inadequate

Proportion 

Adequate

Control 20 18 2 90%

Plus 19 17 2 89%

Transactions 20 20 0 100%

Fisher’s exact test, ns
Respondent Burden:

Time taken for single-item transaction

27

• Task: report a single item 

purchased from a previously 

reported outlet

• How much time is added by 

collecting outlets?
Respondent Burden:

Time taken for single-item transaction

 Significantly more time in Transactions 

condition

28

Condition

n 

Total Mean Median SD

Control 20 53.2 54.3 15.2

Plus 17 62.4 59.7 25.6

Transactions 20 81.1 70.7 38.2

(F(2, 54) = 5.13, p < 0.009); Transactions > Control, Bonferroni p < 0.008. 
Respondent Burden:

Time taken for multi-item transaction

29

• Task: report multiple items 

purchased from a new 

outlet

• How much time is added by 

collecting outlets?
Respondent Burden:

Time taken for multi-item transaction

 No difference in time taken across diary 

conditions

30

Condition

n 

Total Mean Median SD

Control 15 250.5 213.0 85.9

Plus 16 296.5 263.0 117.9

Transactions 19 252.8 223.0 113.3

(F(2, 47) = 0.94, ns). 
Respondent Burden:

Participant ratings overall

 No difference in burden ratings across 

diary conditions

31

Transactions

(n=18)

Plus

(n=19)

Control

(n=19)

Very easy

Somewhat easy

Neither easy nor

difficult

Somewhat difficult

Very difficult
Preference for Diary Design

 Out of 20 participants asked for their 

preference

16 participants preferred Transactions diary

4 participants preferred Plus diary

32
RESULTS: 

REPORTING OUTLETS

33
Outlet Data Quality:

Unusual outlet name

34

• Spell the outlet name 

accurately?
Outlet Data Quality:

Unusual outlet name

35

Reported Outlet Name Frequency

Val U Mart 34

grocery store 1

VAL U MART STORE 326 1

VAL-U-MART 1

ValUMart 1

Val U Maart 1

Vall U Mart 1
Outlet Data Quality: 

Unknown outlet name

36

While you’re out that night 

[in Washington, DC], you 

had to pay for parking. You 

don’t get a receipt so you 

enter the information as 

soon as you get home –

the meter cost $2.25
Outlet Data Quality:

Unknown outlet name

37

Inadequate Outlet Names n

Parking 3

Park Now 1

Parking fee 1

J Street Cinema Parking 1

HARVEST TABLE 1

BTC PARKING 1

Jay's Parking Garage 1

j street cinema 1

don't know 1

PARKING 1

J Street Cinema 1

other 1
Outlet Data Quality:

Unknown outlet name

 No differences in reporting across diary 

conditions

38

Condition

n 

Total

n 

Adequate

n 

Inadequate

Proportion 

Adequate

Control - - - -

Plus 18 9 9 50%

Transactions 16 7 9 44%

X2(1, N = 34) = .13, ns
Summary

 Outlet data quality collected here is 

similar to data quality from TPOPS

 Outlet data quality and error types may 

change due to change in administration 

mode

39
Summary

 The collection of outlet information did 

not substantively increase or decrease 

CE data quality

 Participant ratings of burden show no 

large effects due to outlets

 Possible increase in time taken to enter 

items in the diary

40
Next Steps

 Would these results differ for the mobile 

diary?

 Can usability of the diary designs be 

improved?

 What is the impact of outlet collection 

on the CE recall interview?

41
Contact Information

Erica Yu

Research Psychologist

Office of Survey Methods Research

www.bls.gov/osmr

202-691-7924

yu.erica@bls.gov