1 — U.S. BUREAU OF LABOR STATISTICS • bl s.gov

Exploring the Impact of Outlet Questions 

on Data Quality and Respondent Burden

Yezzi Angi Lee

Safia Abdirizak

Economist

Division of Consumer Expenditure Surveys

AAPOR

May 18th, 2019
2 — U.S. BUREAU OF LABOR STATISTICS • bl s.gov

Outline

 Background

 Objective

 Data 

 Results

Burden

Data Quality 

 Summary

 Next Steps

Where? Grocery 

Store!

Bought

HERE!
3 — U.S. BUREAU OF LABOR STATISTICS • bl s.gov

Background

CPI

Consumer 

Price 

Index

CE

Consumer 

Expenditure 

Surveys

TPOPS

Telephone 

Point-of-Purchase

Survey
4 — U.S. BUREAU OF LABOR STATISTICS • bl s.gov

Background

CPI

Consumer 

Price 

Index

CE

Consumer 

Expenditure 

Surveys

TPOPS

Telephone 

Point-of-Purchase

Survey
5 — U.S. BUREAU OF LABOR STATISTICS • bl s.gov

Surveys

 Interview (CEQ)

 Large purchases and recurring payments

 3 months recall

 Rotating panel 

 4 waves

 In-person and telephone surveys 

 Diary (CED)

 Smaller purchases

 Diary is placed for two weeks

 Respondents record expenditures 

 Collects where consumers purchase goods 

and services and how much they are 

spending

 Random Digit Dialed telephone survey

 Will be discontinued starting 2020

Consumer Expenditure Surveys (CE) Telephone Point-of-Purchase Survey 

(TPOPS)
6 — U.S. BUREAU OF LABOR STATISTICS • bl s.gov

Integration 

 CPI market basket will have the 

same target population 

 Serves as memory cues for 

recalling purchases

 Additional information during 

editing process 

 Respondent Burden

 Data Quality

Benefits Risk
7 — U.S. BUREAU OF LABOR STATISTICS • bl s.gov

Test Outlet Questions in CEQ

 Asked in the final wave for 2017 CEQ data (Testing Phase)

 Are on path for those households assigned to an outlet group and those who 

reported expenditures 

 Outlet group is assigned based on a rotation matric to BLS defined non-rural PSUs

 Included in 10 sections

Where did you purchase 

this?

(Point of purchase)

Was this purchase 

online or in-person?

(Purchase mode)

City and State?

(Location)
9 — U.S. BUREAU OF LABOR STATISTICS • bl s.gov

Objective

 What is the effect of introducing outlet questions into CEQ?

On data quality?

On respondent burden?

 Test Group

Households who were eligible to receive the outlet questions 

 Control group

Households who were NOT eligible to receive the outlet questions
10 — U.S. BUREAU OF LABOR STATISTICS • bl s.gov

Data

 Sources:

From July 2017—March 

2018 

– 4,517 completed final 

(4th ) wave interviews

Audit Trail Tables
11 — U.S. BUREAU OF LABOR STATISTICS • bl s.gov

Data

Demographic Not Eligible for 

Outlet Q (Control)

Eligible for Outlet 

Q(Test)

N 3,417 1,100

Median Respondent Age 52 53

Household Size

1 29.7% 30.5%

1 33.5% 34.8%

2 15.4% 13.5%

4+ 21.4% 21.2%

Housing Tenure

Owner 63.7% 64.3%

Renter 34.4% 34.1%

Oth_Tenure 2.0% 1.6%

Not statistically significant at p < 0.05, Chi-squared test & Wilcoxon 

Rank Sum and Signed Rank tests

 Other demographics that were 

not statistically significant 

 Respondents’ race

 Respondents’ sex 

 Respondents’ Education 
12 — U.S. BUREAU OF LABOR STATISTICS • bl s.gov

Data

 Percent of CUs that were designated in outlet group that 

actually received outlet questions 

N Received Outlet Q Did not Received

Outlet Q

Eligible for Outlet Q 1,100 46.3% 53.7%
13 — U.S. BUREAU OF LABOR STATISTICS • bl s.gov

Burden: Time Spent by Outlet Types

 Data based on looking at each CE item, summing how much time was spent by each 

household that had an expense, then grouping by question type and finally taking 

averages

 The time estimate may be higher than actual time spent per question because 

households may have been asked to report more than one outlet for some CE items

Question Type N

Mean Time 

(sec)

Median Time

(sec)

Min Time

(sec)

Max Time

(sec)

Store name 562 33.17 23 0 557

Purchase Mode 224 7.15 6 0 44

Location 447 16.04 12 1 129
14 — U.S. BUREAU OF LABOR STATISTICS • bl s.gov

Burden: Outlet Question on Survey Length, 

Income and Expenditures 

Group N Median Survey 

Length (mins)

Median 

Income

Median Total 

Quarterly 

Expenditures 

Not Eligible for 

Outlet Q (Control)

3,417 56.98 $40,464 $10,654

Eligible for 

Outlet Q (Test)

1,100 55.37 $39,060 $10,056

Not statistically significant at p < 0.05, Wilcoxon Rank Sum and Signed Rank test
15 — U.S. BUREAU OF LABOR STATISTICS • bl s.gov

Burden: Number of Questions Added to the 

Interview for those who received Outlet Questions

 Data based on looking at each household and summing how many outlet questions were asked, and 

then taking averages of the households 

 Note that not all of the outlet questions have follow up questions

 Ex) Purchase mode was not asked for taxi, transportation, salon, parking, etc

N Median Mean SD Min Max

Eligible for 

Outlet Q (Test) 1,100 0.00 1.43 2.15 0.00 17.00

Received

Outlet Q 509 2.00 3.10 2.19 1.00 17.00

Max amount of questions that could be received by a given CU: 25
16 — U.S. BUREAU OF LABOR STATISTICS • bl s.gov

Burden: Outlet Question on Perceived Burden

N No Burden Burden

Not Eligible for

Outlet Q (Control)

3,305 33.2% 66.8%

Eligible for 

Outlet Q (Test)

1,071 35.5% 64.5%

Not statistically significant at p < 0.05, t test

 Data based on those who responded to burden question

“How burdensome was this survey to you?”
17 — U.S. BUREAU OF LABOR STATISTICS • bl s.gov

Burden: Respondent’s Cooperation Level 

(FR reported N=473)

0%

5%

10%

15%

20%

25%

30%

35%

More Same Less Unsure

q173 q174 q181

 Data based on those who responded to outlet and burden questions
18 — U.S. BUREAU OF LABOR STATISTICS • bl s.gov

Data Quality: Outlet Questions on 

Reported Expenditures 

N Median Mean SD Min

Not Eligible for 

Outlet Q (Control) 3,417 $410.00 $728.15 $1,338.48 $0.00

Eligible for 

Outlet Q (Test) 1,100 $385.00 $664.84 $972.74 $0.00

Not statistically significant at p < 0.05 , Wilcoxon Rank Sum and Signed Rank Test
19 — U.S. BUREAU OF LABOR STATISTICS • bl s.gov

Data Quality: Item Nonresponse by Household

CE Item Household Outlet name Purchase mode Location

Pre-paid phone 216 1 0 0

Appliances, Household Equipment: computers, 

Bicycles, etc.

619 4 0 0

Household Item Repairs and Service : Garbage 

disposal, washer, oven, etc.

308 0 0 0

Home Furnishings and Related Household item: 

renting, leasing, or repair of furniture 

308 0 0 0

Clothing and Sewing Materials 311 4 0 0

Other Vehicle Operating Expenses: Gas, Parking, tolls 446 16 - 1

Subscriptions and Memberships 216 0 0 0

Miscellaneous Expenses: babysitting, toys, pets, etc. 524 3 0 0

Expense Patterns for Selected Services and Goods: 

Taxi, salon, and public transportation 

446 18 - 0
20 — U.S. BUREAU OF LABOR STATISTICS • bl s.gov

Results

 Overall, introducing additional outlet questions did NOT reduce the data quality 

nor increase respondent burden

 Median respondents spent 41 seconds to answer a series of outlet questions 

(Store name, Purchase mode, and Location) 

 Respondents were asked an average of 2 additional outlet questions

 There were no statistical differences between Eligible Vs. Not eligible for Outlet Q 

groups on the following :

 Survey length 

 Income

 Total expenditures

 Total reported expenditures associated with outlet questions

 Perceived burden
21 — U.S. BUREAU OF LABOR STATISTICS • bl s.gov

Moving Forward

 From July 2018 on, outlet questions have been asked in all 

interview waves and added to an an additional 12 sections 

 “Last Purchase” questions are asked for unreported expenses to 

capture additional outlets

Extended recall period: 3 months -> 5 years 
Contact Information

22 — U.S. BUREAU OF LABOR STATISTICS • bl s.gov

Yezzi Angi Lee

Economist

Division of Consumer Expenditure Surveys

202-691-5154

lee.yezzi@bls.gov