1 — U.S. BUREAU OF LABOR STATISTICS • bl s.gov

Testing a Probability-Based Online Panel 

Self-Administration of the Consumer 

Expenditure Diary Survey

Nikki L. Graf, Graham Jones, Tucker Miller, Wendy Carlton

(U.S. Bureau of Labor Statistics), and Ryan Tully (Ipsos)

2022 Joint Statistical Meetings

August 7, 2022
2 — U.S. BUREAU OF LABOR STATISTICS • bl s.gov

Outline

◼ Background

◼ Online Panel Self-Administered 

Diary (OPSAD) test design

◼ Completion and consent rates

By demographics

◼ Data quality

Item nonresponse

Entry counts

Itemization

Drop-off

◼ Next steps
3 — U.S. BUREAU OF LABOR STATISTICS • bl s.gov

Consumer Expenditure Diary Survey

◼ CE Diary Survey traditionally uses two one-week paper diaries to collect household 

expenditures

◼ Interviewers place the diaries in person, train respondents how to record the 

household’s daily expenditures, contact them during the week to encourage diary 

keeping, and collect any forgotten expenditures at the end of the diary period

◼ Online diary mode available in 2020 using these protocols, with interviewers training 

respondents on diary-keeping in person, or by telephone due to COVID-19 restrictions

◼ How would a fully self-administered mode compare to our production?

 Conducted test in a probability-based online panel to assess the performance of an online 

diary administered without the aid of an interviewer
4 — U.S. BUREAU OF LABOR STATISTICS • bl s.gov

Online Panel Self-Administered Diary (OPSAD) Test Design

◼ Conducted on the Ipsos KnowledgePanel®, a probability-based web panel 

designed to be representative of the U.S.

◼ Field period: November 2021 – January 2022

◼ Independent samples drawn for each of the 3 months

◼ Total starting sample: 12,700 adults representing their households

◼ Stages:

Initial Household Characteristics Survey (developed based on CE CAPI instrument)

Two-week Consumer Spending Diary

◼ Incentives:

$2 cash equivalent for completing Household Characteristics Survey

$50 cash equivalent for qualifying as a complete for the Spending Diary
5 — U.S. BUREAU OF LABOR STATISTICS • bl s.gov

OPSAD Test Design – Email Communications

◼ Household Characteristics Survey (HCS)

 Refers to days within a given month of the field period

◼ Spending Diary

 Refers to days after completing Household Characteristics Survey and agreeing to participate in Spending Diary 

3 Days Prior

PreNotification 

Email

Day #8

Reminder 

Email #2

Day #12

Reminder 

Email #3

Day #1

Invitation 

Email

Day #4

Reminder 

Email #1

Day #1

Week #1 

Invitation 

Email

Day #11

Week #2 

Reminder 

Email

Day #15

Closing 

Email

Day #4

Week #1 

Reminder 

Email

Day #8

Week #2 

Invitation 

Email
6 — U.S. BUREAU OF LABOR STATISTICS • bl s.gov

Household Characteristics Survey (HCS)

◼ In the current Consumer Expenditure Diary Survey, the interviewer records 

information on the household, including member demographics, work experience, 

income, and typical household spending.

◼ For OPSAD, a self-administered version of this questionnaire was developed based on 

the CE CAPI instrument.

◼ For some respondent demographics, previously collected Ipsos profile data was 

available, allowing these questions to be eliminated.

◼ Median completion time for OPSAD HCS: 11 minutes

◼ One of the final HCS questions describes the Diary task, informs respondents of the 

$50 incentive, and asks if they are willing to participate.

 Only respondents who consented to participate in the Diary received an invitation.
7 — U.S. BUREAU OF LABOR STATISTICS • bl s.gov

Diary Survey

◼ One household respondent reports on 

all household expenditures

◼ Expenditures grouped into four 

categories: 

 Food and Drinks For Home Consumption

 Meals, Snacks, and Drinks Away from Home

 Clothing, Shoes, Jewelry, and Accessories

 All Other Products, Services, and Expenses 
8 — U.S. BUREAU OF LABOR STATISTICS • bl s.gov

Diary Instructional Materials for Respondents

◼ In the online diary instrument:

User Guide pdf

Instructions, FAQs, Definitions, Examples

Video Tutorial Links

◼ Help Desk for users to contact with any questions or concerns

Access and navigation

Survey concepts (e.g., what level of detail to include in reporting, 

where to report items)
9 — U.S. BUREAU OF LABOR STATISTICS • bl s.gov

Completion and Consent Rates
10 — U.S. BUREAU OF LABOR STATISTICS • bl s.gov

Step 1: HCS Completion
11 — U.S. BUREAU OF LABOR STATISTICS • bl s.gov

Step 2: Diary Consent
12 — U.S. BUREAU OF LABOR STATISTICS • bl s.gov

Step 3: Diary Completion

Overall 

Completion Rate:

25.3%
13 — U.S. BUREAU OF LABOR STATISTICS • bl s.gov

Completion and Consent Rates – by Age

30

45

54

64

Age 18-29 30-44 45-59 60+

% Completed HCS % Consented to Diary

(among HCS completes)

% Completed Diary

(among consenters)
14 — U.S. BUREAU OF LABOR STATISTICS • bl s.gov

Completion and Consent Rates – by Age

30

45

54

64

73

68

62

52

Age 18-29 30-44 45-59 60+

% Completed HCS % Consented to Diary

(among HCS completes)

% Completed Diary

(among consenters)
15 — U.S. BUREAU OF LABOR STATISTICS • bl s.gov

Completion and Consent Rates – by Age

30

45

54

64

73

68

62

52

86 80 83 80

Age 18-29 30-44 45-59 60+

% Completed HCS % Consented to Diary

(among HCS completes)

% Completed Diary

(among consenters)
16 — U.S. BUREAU OF LABOR STATISTICS • bl s.gov

Completion and Consent Rates – by Education

39

50 49

57

Less than high school High school graduate Some college Bachelor's+

% Completed HCS % Consented to Diary

(among HCS completes)

% Completed Diary

(among consenters)
17 — U.S. BUREAU OF LABOR STATISTICS • bl s.gov

Completion and Consent Rates – by Education

39

50 49

57

63

58 58 62

Less than high school High school graduate Some college Bachelor's+

% Completed HCS % Consented to Diary

(among HCS completes)

% Completed Diary

(among consenters)
18 — U.S. BUREAU OF LABOR STATISTICS • bl s.gov

Completion and Consent Rates – by Education

39

50 49

57

63

58 58 62

82 80 79 83

Less than high school High school graduate Some college Bachelor's+

% Completed HCS % Consented to Diary

(among HCS completes)

% Completed Diary

(among consenters)
19 — U.S. BUREAU OF LABOR STATISTICS • bl s.gov

Completion and Consent Rates – Summary

◼ Wide variation in HCS completion rates across demographic groups

Lower rates among typically harder-to-reach groups (e.g., young adults)

◼ But – among those who do complete the HCS, after being informed of the 

$50 incentive, the diary consent rates of these groups rebound, and in 

some cases exceed the rates of other groups 

◼ At final stage, minimal demographic differences in Diary completion rates 

(among those who completed the HCS and consented to the Diary)
20 — U.S. BUREAU OF LABOR STATISTICS • bl s.gov

Data Quality

Preliminary Results
21 — U.S. BUREAU OF LABOR STATISTICS • bl s.gov

Comparisons with Large Scale Feasibility Test (LSF) of the 

Online Diary

◼ CE production data from a comparable time period are not publicly 

released yet, so this analysis uses the LSF as a comparison

◼ LSF included both online and paper diaries

Push to online diary if the household respondent was eligible

If they refused or were ineligible, they received paper diaries

◼ Online and paper diaries were placed and picked up by interviewers in 

person

◼ LSF starting sample size: 2,500 addresses

◼ LSF field period: October 2019 - April 2020
22 — U.S. BUREAU OF LABOR STATISTICS • bl s.gov

Item Nonresponse

◼ For each item, respondents were 

required to provide:

Date of purchase

Business where item was 

purchased

Expense category (e.g., ‘Food and 

Drinks for Home Consumption’)

◼ But other fields could be left blank:

Cost

Item Description

◼ Overall, item nonresponse rates 

were low, but...

◼ Leaving fields blank was more 

common in OPSAD.

◼ Share of entries with blank fields:

OPSAD LSF

Cost 1.73% 0.42%

OPSAD LSF

Item Description (excl. 

Food Away category)

1.68% 0.32%
23 — U.S. BUREAU OF LABOR STATISTICS • bl s.gov

Expenditure Entry Counts

OPSAD

LSF 

Overall

LSF 

Online

LSF 

Paper

Number of 

weekly diaries

5,856 1,518 646 872

Mean 13.5 26.8 23.9 28.9

Median 10 20 16 24

◼ Average weekly entry 

counts (for non-blank 

diaries) lower in OPSAD 

than in LSF
24 — U.S. BUREAU OF LABOR STATISTICS • bl s.gov

Counts by Section

15.0

5.6

3.2

2.2

1.0

.7

7.7

5.0

LSF

Overall

OPSAD

Food at Home Food Away Clothing Other

Total: 

13.5

Total: 

26.8

(1,518

diary weeks)

Average number of weekly diary entries, by expense category:

(5,856

diary weeks)
25 — U.S. BUREAU OF LABOR STATISTICS • bl s.gov

Adherence to Itemization Instructions

◼ For all expense 

categories except 

Food Away, 

respondents are 

instructed to 

itemize –

enter each item 

purchased as a 

separate entry
26 — U.S. BUREAU OF LABOR STATISTICS • bl s.gov

Itemization in OPSAD

◼ Item descriptions containing the following strings were flagged as “nonitemized” entries:

GROCERIES

FOOD

CLOTHES

CLOTHING

HOME IMPROVEMENT

HOUSEHOLD

MISCELLANEOUS

MISC
27 — U.S. BUREAU OF LABOR STATISTICS • bl s.gov

◼ Comparison of 3 data sources:

 Online Panel Self-Administered Diary

– N=68,894 expenses

 Large Scale Feasibility Test (LSF Overall)

– N=25,750 expenses

 CE Production Diary Data (CEDPROD) 

– N=195,131 expenses

◼ Much higher rate of non-itemized 

purchases in LSF

◼ Compared to CEDPROD, rates were 

slightly higher in OPSAD across all 

expenditure categories

Itemization – Comparison Between OPSAD, LSF, 

and CE Production

% of Expenses NOT Itemized

13.2 14.7

2.4

39.8

28.2

2.9

11.0

13.9

0.7

Food at Home Clothing Other

Expenditure Categories (excl. Food Away)

OPSAD LSF CEDPROD
28 — U.S. BUREAU OF LABOR STATISTICS • bl s.gov

Drop-Off in Entry Counts

◼ Respondents may experience 

fatigue from the reporting process 

and report fewer expenditure 

entries over the diary period

◼ Drop-off measure: Averaged 

percentage of each household’s 

two-week entries reported in 

Week 1

◼ Calculated for households having 

at least one expenditure in both 

weeks

OPSAD

(n=2,702)

LSF 

Overall 

(n=780)

LSF 

Online 

(n=334)

Week 1 mean entries 15.1 27.9 24.6

Week 2 mean entries 13.2 24.2 21.6

Mean entries (Weeks 1 + 2) 28.2 52.1 46.2

Avg. % of two-week entries 

reported in Week 1 54.4% 53.3% 54.4%
29 — U.S. BUREAU OF LABOR STATISTICS • bl s.gov

Summary of Preliminary OPSAD Results

◼ Higher rates of blanks in cost and item description fields, though still low overall

◼ Fewer entries (particularly ‘Food at Home’ section entries)

◼ Lower rates of non-itemization than in prior test that included an online diary 

option, suggesting stronger adherence to diary-keeping instructions to itemize

 Further research needed to explore relationship between non-itemization and entry counts

◼ Drop-off in entries across weeks similar to prior tests
30 — U.S. BUREAU OF LABOR STATISTICS • bl s.gov

Next Steps

◼ Comparisons with CE production data, including online diaries, 

from the same time period

◼ Itemization analysis

◼ Expenditure amounts

◼ Analysis of paradata

◼ Improvements to the CE online diary instrument and protocols
Contact Information

31 — U.S. BUREAU OF LABOR STATISTICS • bl s.gov

Nikki L. Graf

Branch of Research and Program Development

Division of Consumer Expenditure Surveys

Graf.Nikki@bls.gov