1 — U.S. BUREAU OF LABOR STATISTICS • bls.gov

Developing Metrics to Evaluate 

Streamlined Questionnaire Changes in the 

Consumer Expenditure Interview Survey

Nikki L. Graf, Erica Yu, and Ariana Welsh

U.S. Bureau of Labor Statistics

AAPOR Annual Conference

May 17, 2024
2 — U.S. BUREAU OF LABOR STATISTICS • bls.gov

Outline

◼ Overview of the Consumer Expenditure (CE) Interview Survey

◼ Streamlined questionnaire implementation and evaluation

◼ Defining question text for analysis: considerations and challenges

◼ Documentation system

◼ Metrics methodology

◼ Example changes – Results

◼ Summary and next steps
3 — U.S. BUREAU OF LABOR STATISTICS • bls.gov

Consumer Expenditure Surveys (CE)

◼ Collect spending data on the U.S. 

population

◼ Sponsored by the Bureau of Labor 

Statistics, collected by the Census Bureau

◼ Provide expenditure weights for the U.S. 

Consumer Price Index (CPI)

◼ Two separate surveys:

Interview Survey 

Diary Survey
4 — U.S. BUREAU OF LABOR STATISTICS • bls.gov

CE Survey Redesign: Streamlined Questionnaire

◼ Objectives of CE redesign

Primary: Address measurement error

Secondary: Reduce respondent burden

◼ Streamlined questionnaire features

 Reduce the level of detail: aggregate expenditure questions to the extent possible

 Records focus: facilitate the use of records, including a records path for certain 

groups of questions

 Question order/sections: intuitive changes to the order and section organization to 

improve flow
5 — U.S. BUREAU OF LABOR STATISTICS • bls.gov

Streamlined Questionnaire Sections

Phased implementation started in April 2023:

◼ Family Care and Education section

 Childcare, education, school meals, and adult care

◼ Expense Patterns section

 Grocery expenses, alcohol and tobacco, food away 

from home, and gifts

◼ Transportation section

 Vehicle operating expenses and public transportation
6 — U.S. BUREAU OF LABOR STATISTICS • bls.gov

Evaluating Streamlined Questionnaire Changes

◼ Have the questionnaire changes made progress toward reducing 

complexity and length?

Estimate metrics to compare old vs. new wording

Focus on text read aloud to respondent

Examine changes both at the question level and among questions grouped 

at the path or section level

◼ Separate research will analyze:

 Response outcomes to gauge impact on data quality

 Actual time spent for interviews
7 — U.S. BUREAU OF LABOR STATISTICS • bls.gov

Defining the Question Text for Analysis 

Considerations and Challenges

◼ CE uses a variety of question formats

 Including “laundry list questions” where a list of items share a common question stem

◼ Skip patterns

 Reported purchases lead to follow-up questions, e.g., month, amount spent

 Follow-ups and their wording vary by item

◼ New sections integrate items previously in different sections and do not 

have a direct counterpart for comparison

◼ Tailored question fills, with wording dependent on household characteristics, 

previous responses, etc.
8 — U.S. BUREAU OF LABOR STATISTICS • bls.gov

Example: Family Care and Education Section

“In this section, I'm going to ask about any payments for education and caring for 

[family and non-household members/your family, other household members, and 

non-household members]. Please remember to include payments you made online 

or by an automatic payment. Please only report your final out-of-pocket payments.”

“Since the first of [reference month], not including this month, have [you/you or any 

members of your household] paid for any –

▪ Education savings plans, including 529s?

▪ Daycare, nursery, or preschool tuition or fees?

▪ Babysitting, nanny services, or other childcare inside or outside your home?

▪ Elementary through high school tuition or fees?

▪ Vocational or technical program tuition or fees?

▪ College or university tuition or fees?

▪ Housing while attending school? ...”

Section Intro
9 — U.S. BUREAU OF LABOR STATISTICS • bls.gov

Example: Family Care and Education Section

“In this section, I'm going to ask about any payments for education and caring for 

[family and non-household members/your family, other household members, and 

non-household members]. Please remember to include payments you made online 

or by an automatic payment. Please only report your final out-of-pocket payments.”

“Since the first of [reference month], not including this month, have [you/you or any 

members of your household] paid for any –

▪ Education savings plans, including 529s?

▪ Daycare, nursery, or preschool tuition or fees?

▪ Babysitting, nanny services, or other childcare inside or outside your home?

▪ Elementary through high school tuition or fees?

▪ Vocational or technical program tuition or fees?

▪ College or university tuition or fees?

▪ Housing while attending school? ...”

Section Intro
10 — U.S. BUREAU OF LABOR STATISTICS • bls.gov

Example: Family Care and Education Section

“In this section, I'm going to ask about any payments for education and caring for 

[family and non-household members/your family, other household members, and 

non-household members]. Please remember to include payments you made online 

or by an automatic payment. Please only report your final out-of-pocket payments.”

“Since the first of [reference month], not including this month, have [you/you or any 

members of your household] paid for any –

▪ Education savings plans, including 529s?

▪ Daycare, nursery, or preschool tuition or fees?

▪ Babysitting, nanny services, or other childcare inside or outside your home?

▪ Elementary through high school tuition or fees?

▪ Vocational or technical program tuition or fees?

▪ College or university tuition or fees?

▪ Housing while attending school? ...”

Section Intro

Question Stem
11 — U.S. BUREAU OF LABOR STATISTICS • bls.gov

Example: Family Care and Education Section

“In this section, I'm going to ask about any payments for education and caring for 

[family and non-household members/your family, other household members, and 

non-household members]. Please remember to include payments you made online 

or by an automatic payment. Please only report your final out-of-pocket payments.”

“Since the first of [reference month], not including this month, have [you/you or any 

members of your household] paid for any –

▪ Education savings plans, including 529s?

▪ Daycare, nursery, or preschool tuition or fees?

▪ Babysitting, nanny services, or other childcare inside or outside your home?

▪ Elementary through high school tuition or fees?

▪ Vocational or technical program tuition or fees?

▪ College or university tuition or fees?

▪ Housing while attending school? ...”

Section Intro

Question Stem

Item List
12 — U.S. BUREAU OF LABOR STATISTICS • bls.gov

▪ “Education savings plans, including 529s?

▪ Daycare, nursery, or preschool tuition or fees?

▪ Babysitting, nanny services, or other childcare 

inside or outside your home?

▪ Elementary through high school tuition or fees?

▪ Vocational or technical program tuition or fees?

▪ College or university tuition or fees?

▪ Housing while attending school?

▪ School meals for preschool through college, 

including meal plans?*

▪ Private school bus service?

▪ Purchase or rental of any school books, supplies, or 

equipment, which has not already been reported?

▪ Tutoring or test prep?

▪ Adult day care centers?

▪ In-home care for convalescing, disabled, or elderly 

persons, excluding medical care?”

Asked for each item that respondent reports 

paying for:

“In what month or months did [you/your 

household] pay for [item description]?”

“How much did [you/your household] pay? 

Please include any sales tax with the expense.”

“Did you pay for any other [item description]?”

“Was this for... Prepaid tuition plans? Other 

education savings plans?”

*School meals item leads to a different set of 

follow-up questions (e.g., who, how often)
13 — U.S. BUREAU OF LABOR STATISTICS • bls.gov

Documenting Questionnaire Changes

Change Types:

◼ Section introduction

◼ Question

 Question stem text, along with any instructional or clarifying language read aloud

◼ Item grouping

 Aggregated item list text from laundry lists

◼ Question set

 Captures deletions/additions of follow-up questions

◼ Section

 For sections/subsections with comparable content across questionnaire versions
14 — U.S. BUREAU OF LABOR STATISTICS • bls.gov

Classifying the Purpose of Changes

◼ Capture information not 

previously collected

◼ Disaggregate a 2022 

question/response option/item

◼ Add instructions

◼ Update language or examples

◼ Improve specificity

◼ Reduce number of questions/ 

response options/items

◼ Simplify language

◼ Make language consistent across 

survey

◼ Match respondent cognition

Purpose Categories:
15 — U.S. BUREAU OF LABOR STATISTICS • bls.gov

Metrics – Changes in...

◼ Length

Number of words

Number of characters

◼ Speaking time
16 — U.S. BUREAU OF LABOR STATISTICS • bls.gov

Metrics (cont.)

Changes in...

◼ Readability

Flesch-Kincaid Grade Level (FKG) 

Gunning-Fog Index (FOG)

◼ Text density

Average characters per word

For Questions, Item Groupings, and Section Introductions:
17 — U.S. BUREAU OF LABOR STATISTICS • bls.gov

Metrics (cont.)

◼ Term ambiguity

 Number of words flagged by QUAID as:

• Unfamiliar technical terms

• Vague or imprecise relative terms

• Vague or ambiguous noun phrases

For Questions, Item Groupings, and Section Introductions:

◼ Problematic sentence structure

 Number of sentences flagged by QUAID 

as having:

• Complex syntax

• Working memory overload

Changes in...

Comprehension difficulty – based on the Question Understanding Aid (QUAID) 

http://quaid.cohmetrix.com/ (Graesser et al. 2006) 
18 — U.S. BUREAU OF LABOR STATISTICS • bls.gov

Metrics (cont.)

◼ Length

Number of words

Number of characters

Number of questions

Number of sentences

For Question Sets and Sections:

Defined and analyzed both the longest and shortest paths possible

Changes in...
19 — U.S. BUREAU OF LABOR STATISTICS • bls.gov

Example Changes – Metrics Results
20 — U.S. BUREAU OF LABOR STATISTICS • bls.gov

Question Example

Transportation section – Vehicle operating expenses laundry list question stem

Metric Old 

Wording

New 

Wording

Change

(New - Old)

Length: # of Words 20 16 -4

Length: # of Characters 86 65 -21

Speaking Time (seconds) 10 8 -2

Readability: FKG 9.9 6.9 -3.0

Readability: FOG 14.0 8.9 -5.1

Text Density 4.3 4.1 -0.2

Term Ambiguity (QUAID) 2 1 -1

Problematic Sentence 

Structure (QUAID) 0 0 0

Old Wording

“Since the first of August, have you 

or any members of your household 

had expenses for any of the 

following?”

New Wording

“Since the first of August, have you 

or any members of your household 

paid for any –”
21 — U.S. BUREAU OF LABOR STATISTICS • bls.gov

Item Grouping Example

Comparable items from old Educational Expenses section vs. new Family Care and Education section

Metric Change

Length: # of Words +7

Length: # of Characters +25

Speaking Time (seconds) +3

Readability: FKG +0.04

Readability: FOG -4.9

Text Density -0.3

Term Ambiguity (QUAID) +2

Problematic Sentence 

Structure (QUAID) +4

Old Wording

▪ Preschool or child day care 

centers?

▪ Tuition, including pre-paid 

tuition?

▪ Housing while attending school?

▪ Private school bus service?

▪ Test preparation or tutoring 

services?

▪ Purchase of school books?

▪ Purchase of school supplies or 

equipment not already 

reported?

▪ Other school related expenses 

not already reported including 

the rental of any school books or 

equipment?

New Wording

▪ Education savings plans, including 

529s?

▪ Daycare, nursery, or preschool 

tuition or fees?

▪ Elementary through high school 

tuition or fees?

▪ Vocational or technical program 

tuition or fees?

▪ College or university tuition or fees?

▪ Housing while attending school?

▪ Private school bus service?

▪ Purchase or rental of any school 

books, supplies, or equipment, 

which has not already been 

reported?

▪ Tutoring or test prep?
22 — U.S. BUREAU OF LABOR STATISTICS • bls.gov

Item Grouping Example

Comparable items from old Educational Expenses section vs. new Family Care and Education section

Metric Change

Length: # of Words +7

Length: # of Characters +25

Speaking Time (seconds) +3

Readability: FKG +0.04

Readability: FOG -4.9

Text Density -0.3

Term Ambiguity (QUAID) +2

Problematic Sentence 

Structure (QUAID) +4

Old Wording

▪ Preschool or child day care 

centers?

▪ Tuition, including pre-paid 

tuition?

▪ Housing while attending school?

▪ Private school bus service?

▪ Test preparation or tutoring 

services?

▪ Purchase of school books?

▪ Purchase of school supplies or 

equipment not already 

reported?

▪ Other school related expenses 

not already reported including 

the rental of any school books or 

equipment?

New Wording

▪ Education savings plans, including 

529s?

▪ Daycare, nursery, or preschool 

tuition or fees?

▪ Elementary through high school 

tuition or fees?

▪ Vocational or technical program 

tuition or fees?

▪ College or university tuition or fees?

▪ Housing while attending school?

▪ Private school bus service?

▪ Purchase or rental of any school 

books, supplies, or equipment, 

which has not already been 

reported?

▪ Tutoring or test prep?
23 — U.S. BUREAU OF LABOR STATISTICS • bls.gov

Question Set Example

For comparable items from old Educational Expenses section vs. new Family Care and Education 

section that share the same set of follow-up questions in a given year

Metric Change

Length: # of Words -25

Length: # of Characters -112

Length: # of Questions -4

Length: # of Sentences -3

Speaking Time (seconds) -12

Old Wording

◼ What was the expense for?

◼ Was this expense for - Someone 

inside the household? Someone 

outside the household?

◼ In what month was the payment 

made?

◼ How much was paid?

◼ Has any of this amount been or 

will any of it be reimbursed by an 

employer, agency, or other 

person?

◼ Did you make any other 

payments for [ ]?

New Wording

◼ In what month or months did your 

household pay for [ ]? 

◼ How much did your household pay? 

Please include any sales tax with 

the expense.

◼ Did you pay for any other [ ]?
24 — U.S. BUREAU OF LABOR STATISTICS • bls.gov

Summary and Next Steps

◼ Defining the question text for analysis can present challenges

◼ Process of documenting changes proved useful in itself

◼ Data quality improvements sometimes involved adding language, and therefore 

increasing length

 Considerations in developing interpretations of the metrics

◼ Examining multiple metrics provides additional insight

◼ Future directions: 

 Evaluating data quality, actual time spent for interviews, perceived burden, and 

respondent comprehension

 Using metrics during question development process as streamlining continues
Contact Information

25 — U.S. BUREAU OF LABOR STATISTICS • bls.gov

Nikki L. Graf

Branch of Research and Program Development

Division of Consumer Expenditure Surveys

Graf.Nikki@bls.gov