1 — U.S. BUREAU OF LABOR STATISTICS • bls.gov

Gray Jones, Nikki L. Graf, Tucker Miller, and Wendy Carlton

U.S. Bureau of Labor Statistics

2023 FCSM Research & Policy Conference 

October 25, 2023

Data Quality in the Online Panel SelfAdministered Diary Test of the Consumer 

Expenditure Survey
2 — U.S. BUREAU OF LABOR STATISTICS • bls.gov

Presentation Outline

 Consumer Expenditure Surveys 

(CE) background

 Online Panel Self-Administered 

Diary (OPSAD) background 

 Comparison OPSAD and CE

 Analysis of Data Quality

 Summary & Next steps
3 — U.S. BUREAU OF LABOR STATISTICS • bls.gov

Consumer Expenditure Surveys (CE) 

Background
4 — U.S. BUREAU OF LABOR STATISTICS • bls.gov

Consumer Expenditure Survey (CE) Background

Consumer Expenditure Surveys (CE)

The CE Interview 

Survey 

The CE Diary 

Survey

Household 

Characteristics 

Questionnaire

Spending Diary
5 — U.S. BUREAU OF LABOR STATISTICS • bls.gov

CE Spending Diary Interviewer-Aided Protocol

Collect any forgotten expenditures at the end of the diary period

Follow-up to encourage diary keeping

Train respondents to use the diary

Diary placement
6 — U.S. BUREAU OF LABOR STATISTICS • bls.gov

CE Spending Diary Online Mode

The CE Diary Survey

Household 

Characteristics 

Questionnaire

Spending Diary

Paper Diary Online Diary
7 — U.S. BUREAU OF LABOR STATISTICS • bls.gov

Online Panel Self-Administered Diary (OPSAD) 

Test Background 
8 — U.S. BUREAU OF LABOR STATISTICS • bls.gov

OPSAD Test Background

OPSAD Test Design

• Test conducted on the Ipsos KnowledgePanel® 

• Fielded between November 2021 – January 2022

• Total starting sample of 12,700 adults representing their household

• Two stage survey design (based on the CE Diary Survey)

• Invitation and reminder emails sent for each stage

• Conditional incentives used at each stage
9 — U.S. BUREAU OF LABOR STATISTICS • bls.gov

OPSAD Test Background: OPSAD Survey 

Components

Household 

Characteristics 

Survey (HCS)

• $2 Incentive

Two-week 

Consumer Spending 

Diary

• $50 Incentive
10 — U.S. BUREAU OF LABOR STATISTICS • bls.gov

CE Diary Survey and OPSAD Comparison
11 — U.S. BUREAU OF LABOR STATISTICS • bls.gov

CE Diary and OPSAD Comparison: Structure

CE Diary Survey

• Components of the survey design

• Household Characteristics Questionnaire

• Spending Diary 

• Data fielded between November 2021 and 

January 2022

• Interviewer-Aided

• No incentives

• Regular CE Sample

OPSAD Test

• Components of the survey design

• Household Characteristics Survey 

• Spending Diary 

• Data fielded between November 2021 and 

January 2022

• Fully Self-Administered

• Conditional Incentives

• Sample from Ipsos KnowledgePanel® 
12 — U.S. BUREAU OF LABOR STATISTICS • bls.gov

CE Diary and OPSAD Comparison: Completion by 

Diary Week

Households with at 

least one entry in... CE Diary Survey OPSAD

Both diary weeks 1,586 2,702

Only week 1 15 424

Only week 2 21 28

Total with entries in 

either or both weeks 1,622 3,154
13 — U.S. BUREAU OF LABOR STATISTICS • bls.gov

Analysis of OPSAD Data Quality & Comparison 

to the CE Diary Survey Data Quality 
14 — U.S. BUREAU OF LABOR STATISTICS • bls.gov

CE Diary and OPSAD Comparison: Sample 

Composition 

 Comparison to focus on differences between CE “Online” Diary 

users and OPSAD users to control for mode effects. 

 Past CE research has found meaningful differences between 

“Paper” and “Online” Diary users (McBride & Graf 2022).

 Preliminary results shown here are weighted. 
15 — U.S. BUREAU OF LABOR STATISTICS • bls.gov

OPSAD Data Quality: Item Nonresponse

% of Diary Entries with... Online CE Diary OPSAD

Blank Cost Values 0.3% 2.07%

Blank Item Descriptions (Excluding 

Meals Away from Home) 0.0% 1.05%

Invalid Item Descriptions (Excluding 

Meals Away from Home) 0.1% 1.58%
16 — U.S. BUREAU OF LABOR STATISTICS • bls.gov

OPSAD Data Quality: Average Expenditure 

Counts

24.1

17.0

14.1

10.0

0

5

10

15

20

25

30

Average Expenditure Counts Median Expenditure Counts

Online CE Diary (N=639) OPSAD (N=5,864)
17 — U.S. BUREAU OF LABOR STATISTICS • bls.gov

OPSAD Data Quality: Average Total 

Expenditures

$976.89 

$545.00 

$892.00 

$368.35 

 $0

 $200

 $400

 $600

 $800

 $1,000

 $1,200

Average Expenditures Median Expenditures

Online CE Diary (N=639) OPSAD (N=5,864)
18 — U.S. BUREAU OF LABOR STATISTICS • bls.gov

OPSAD Data Quality: Average Expenditure 

Amounts

$62.37 

$34.99 

$82.69 

$37.50 

 $0

 $20

 $40

 $60

 $80

 $100

Average Expense Amount Median Expense Amount

Online CE Diary (N=639) OPSAD (N=5,864)
19 — U.S. BUREAU OF LABOR STATISTICS • bls.gov

OPSAD Data Quality: Bundled Expenses

 Item descriptions containing the following strings were flagged as 

“bundled” entries:

 GROCERIES

 FOOD

 CLOTHES

 CLOTHING

 HOME IMPROVEMENT

 HOUSEHOLD

 MISCELLANEOUS

 MISC
20 — U.S. BUREAU OF LABOR STATISTICS • bls.gov

OPSAD Data Quality: Expenditure Entry Bundling

 The rate of diaries with any 

bundled expenses

 # oooo DDDDDDDDDD CC CCCCCC ww www BBBBBBBBBB BB EEEEEE

TT # oooo DDDDDDDDDD CC CCCCCC

31.8%

29.3%

0%

10%

20%

30%

40%

Online CE Diary (N=639) OPSAD (N=5,864)

Percentage of Diary Cases with Any Bundled 

Expenses
21 — U.S. BUREAU OF LABOR STATISTICS • bls.gov

OPSAD Data Quality: Expenditure Entry Bundling

28.3%

2.8%

0.7%

26.5%

2.5%

0.3%

0%

10%

20%

30%

Food at Home Any Bundled Expenses Clothing Any Bundled Expenses Other Any Bundled Expenses

Percentage of Diaries with Bundled Expenses by Expenditure Category

Online CE Diary (N=639) OPSAD (N=5,864)
22 — U.S. BUREAU OF LABOR STATISTICS • bls.gov

OPSAD Data Quality: Cost and Bundling

$1,166.59 

$888.30 

$1,169.87 

$876.28 

 $0

 $200

 $400

 $600

 $800

 $1,000

 $1,200

 $1,400

Yes Bundling No Bundling

Comparison of Average Total Expenditures

Average Total Expenditures CE Online Diary

Average Total Expenditures OPSAD

$83.77 

$52.38 

$118.01 

$68.59 

 $0

 $20

 $40

 $60

 $80

 $100

 $120

 $140

Yes Bundling No Bundling

Comparison of Average Expenditure Cost

Average Expenditure Cost CE Online Diary

Average Expenditure Cost OPSAD
23 — U.S. BUREAU OF LABOR STATISTICS • bls.gov

Summary

 Measures of data quality were generally more favorable within 

the CE Production Diary data.

Lower Rates of Item Nonresponse

Higher Average Expenditure Totals

Higher Average Entry Counts 

Similar Rates of Bundling Expenses 

 These differences may be attributable to the lack of interviewer 

guidance in the OPSAD test. 
24 — U.S. BUREAU OF LABOR STATISTICS • bls.gov

Next Steps

 Further investigate and account for the differences in the 

underlying samples.

As the OPSAD sample and the subset of Online Diaries from the CE 

sample are not 1:1

Comparisons to the full CE production sample.

Will involve a more detailed analysis of data quality, using multivariate 

models to introduce controls.

 Final OPSAD Analysis report is forthcoming. 
25 — U.S. BUREAU OF LABOR STATISTICS • bls.gov

Thank you!

Gray Jones, Economist

 Email: Jones.graham@bls.gov

 Phone: 202-691-6869