1 — U.S. BUREAU OF LABOR STATISTICS • bls.gov

Data Quality Session
2 — U.S. BUREAU OF LABOR STATISTICS • bls.gov

Developing a Data Quality 

Profile for the Consumer 

Expenditure Survey

Yezzi Angi Lee

Veri Crain, Scott Fricker , Evan Hubener, 

Clayton Knappenberger, Brandon Kopp,

Julie Sullivan, and Lucilla Tan. 

July 18, 2017
3 — U.S. BUREAU OF LABOR STATISTICS TATISTICS • bls.gov bls.gov

Presentation Outline

To share the challenges encountered in the initial stages of this 

development process, report on interim progress, and thoughts for 

next steps.

 What is a Data Quality Profile (DQP)

 Challenges

 Iterative approach to development

 Interim results

 Moving forward 
4 — U.S. BUREAU OF LABOR STATISTICS TATISTICS • bls.gov bls.gov

What is a Data Quality Profile(DQP)?
5 — U.S. BUREAU OF LABOR STATISTICS • bls.gov

Survey Research Center (2010)

“A comprehensive report prepared by producers of 

survey data that provided information data users need 

to assess the quality of the data” 

“ To provide researchers and data users with a 

single source for a wide range of information on 

the quality of AHS data” 

Quality Profile of the American Housing Survey (1996)
6 — U.S. BUREAU OF LABOR STATISTICS • bls.gov

C

BRFSS 2013 Summary Data 

Quality Report

More Example: 

Vary in Breadth and Depth of Coverage

American Housing Survey 

1996 Quality Profile

https://www.cdc.gov/brfss/annual_data/2013/pdf/2013_dqr.pdf https://www.census.gov/content/dam/Census/program

s-surveys/ahs/publications/h12195-1.pdf

 RESPONSE RATES

 23 PAGE

 Annual publication

 TOTAL SURVEY ERROR 

DIMENSIONS

 80 + PAGE 

 1996
7 — U.S. BUREAU OF LABOR STATISTICS • bls.gov

Data Quality Profile for the CE

Internal External

M

“Monitoring; Establish baselines ” “

Fitness for Use ”
8 — U.S. BUREAU OF LABOR STATISTICS TATISTICS • bls.gov bls.gov

Total Quality 

Management 

Dimensions 

(TQM)

Relevance

Accuracy 

Coherence

Timeliness

Accessibility

Interpretability

Definition of Data Quality for CE

Multi-dimensional Definition of Data Quality adopted for CE

Total Survey Error Sources (TSE)

Frame (coverage) Specification 

(construct)

Sampling Measurement

Non-response Processing (data edit)

Post-survey

adjustment

(Gonzalez et al 2009)

https://www.bls.gov/cex/ovrvwdataqualityrpt.pdf
9 — U.S. BUREAU OF LABOR STATISTICS TATISTICS • bls.gov bls.gov

Challenges
10 — U.S. BUREAU OF LABOR STATISTICS TATISTICS • bls.gov bls.gov

Metric Documentation:

efficient and robust

Infrastructure :

Continuous and adaptable 

to change

To achieve reproducibility and interpretability of metrics
11 — U.S. BUREAU OF LABOR STATISTICS TATISTICS • bls.gov bls.gov

`

1.Requires participation 

and coordination 

across the survey 

program 

2.Resource intensive to 

develop and maintain

TSE

CE DQP Challenges

TQM
12 — U.S. BUREAU OF LABOR STATISTICS • bls.gov

CE Strategy to identify metrics
13 — U.S. BUREAU OF LABOR STATISTICS TATISTICS • bls.gov bls.gov

TQM: Survey as a manufacturing process

http://www.freepik.com/free-vector/industry-and-technologybackground_1048768.htm Designed by Freepik

Total Quality 

Management 

Dimensions 

(TQM)

Relevance

Accuracy 

Coherence

Timeliness

Accessibility

Interpretability
14 — U.S. BUREAU OF LABOR STATISTICS TATISTICS • bls.gov bls.gov

Proposed Framework

Activity 1

Issue 1

Monitoring method/ 

Metric

Quality dimension(s)

Activity 2

Issue 1

Monitoring method/ 

Metric

Quality dimension(s)

Identifying key stages in CE life cycle 

For each stage, identify major activity

For each activity, identify issue(s) of 

concern

Propose how to monitor issue 

identified

Identify quality dimension(s) affected

(Fricker et al 2012)
15 — U.S. BUREAU OF LABOR STATISTICS TATISTICS • bls.gov bls.gov

Example of metric metadata description using a template

 Metric Name

 Description

 Metric interpretation

 Survey

 Quality dimension

CALCULATION

 Formula

 Data source and variables

 Frequency

 Level of aggregation

 Maintained by

MONITORING

 Target / Threshold / Tolerance 

 Presentation / display

NOTES/COMMENTS
16 — U.S. BUREAU OF LABOR STATISTICS TATISTICS • bls.gov bls.gov

Proposed framework: 

Criteria for Metric Prioritization

Specific – targeted at identified risk

Measurable – can be used to determined progress

Achievable – realistically attainable

Relevant – not just “good to know”, actionable

Timely – available when needed

S.M.A.R.T
17 — U.S. BUREAU OF LABOR STATISTICS • bls.gov

Iterative approach to DQP development

“Learn by doing, Refine and Scale up!”
18 — U.S. BUREAU OF LABOR STATISTICS TATISTICS • bls.gov bls.gov

on Learned

2011

L E S S O N S L EA R N E D

• UNDERSTAND THE TASK FOR WHICH WE WANT TO DEVELOP METRIC 

• IMPORTANCE OF METRIC METATDATA DOCUMENTATION FOR 

REPRODUCIBILITY AND INTERPRETATION OVER TIME 

2012

• PROPOSE FRAMEWORK FOR DQP

• ENSURE CONSISTENCY IN DOCUMENTING KEY ELEMENTS OF METRIC METADATA 

USE OF A TEMPLATE

2013 - 1 4

MEASUREMENT ERROR STUDY (WESTAT CONTRACT)

• NO SINGLE ”BEST” METHOD 

MULTIPLE METHOD AND INDICATORS (MMI) APPROACH

2016 MMI FOLLOW-UP

• EXTERNAL INDICATORS FEASIBILITY STUDY

2017 DQP VERSION 2 IN PROGRESS

2015

DQP VERSION 1

• RESPONSE RATES AND EDIT RATES

I n 2 0 0 9 , 

D Q d e f i n i t i o n 

A d o p t e d f o r C E
19 — U.S. BUREAU OF LABOR STATISTICS TATISTICS • bls.gov bls.gov

on Learned

2011

L E S S O N S L EA R N E D

• UNDERSTAND THE TASK FOR WHICH WE WANT TO DEVELOP METRIC 

• IMPORTANCE OF METRIC METATDATA DOCUMENTATION FOR 

REPRODUCIBILITY AND INTERPRETATION OVER TIME 

2012

• PROPOSE FRAMEWORK FOR DQP

• ENSURE CONSISTENCY IN DOCUMENTING KEY ELEMENTS OF METRIC METADATA 

USE OF A TEMPLATE

2013 - 1 4

MEASUREMENT ERROR STUDY (WESTAT CONTRACT)

• NO SINGLE ”BEST” METHOD 

MULTIPLE METHOD AND INDICATORS (MMI) APPROACH

2016 MMI FOLLOW-UP

• EXTERNAL INDICATORS FEASIBILITY STUDY

2017 DQP VERSION 2 IN PROGRESS

2015 DQP VERSION 1

• RESPONSE RATES AND EDIT RATES

I n 2 0 0 9 , 

D Q d e f i n i t i o n 

A d o p t e d f o r C E
20 — U.S. BUREAU OF LABOR STATISTICS TATISTICS • bls.gov bls.gov

on Learned

2011

L E S S O N S L EA R N E D

• UNDERSTAND THE TASK FOR WHICH WE WANT TO DEVELOP METRIC 

• IMPORTANCE OF METRIC METATDATA DOCUMENTATION FOR 

REPRODUCIBILITY AND INTERPRETATION OVER TIME 

2012

• PROPOSE FRAMEWORK FOR DQP

• ENSURE CONSISTENCY IN DOCUMENTING KEY ELEMENTS OF METRIC METADATA 

USE OF A TEMPLATE

2013 - 1 4

MEASUREMENT ERROR STUDY (WESTAT CONTRACT)

• NO SINGLE ”BEST” METHOD 

MULTIPLE METHOD AND INDICATORS (MMI) APPROACH

2016 MMI FOLLOW-UP

• EXTERNAL INDICATORS FEASIBILITY STUDY

2017 DQP VERSION 2 IN PROGRESS

2015 DQP VERSION 1

• RESPONSE RATES AND EDIT RATES

I n 2 0 0 9 , 

D Q d e f i n i t i o n 

A d o p t e d f o r C E
21 — U.S. BUREAU OF LABOR STATISTICS TATISTICS • bls.gov bls.gov

Example of CE DQP Version 1

https://www.bls.gov/cex/ce_dqreport.pdf

1. Response Rates 

2. Nonresponse rates

3. Expenditure Edit Rates

4. Income Imputation rates

* Reporting period: 2010 - 2013
22 — U.S. BUREAU OF LABOR STATISTICS TATISTICS • bls.gov bls.gov

on Learned

2011

L E S S O N S L EA R N E D

• UNDERSTAND THE TASK FOR WHICH WE WANT TO DEVELOP METRIC 

• IMPORTANCE OF METRIC METATDATA DOCUMENTATION FOR 

REPRODUCIBILITY AND INTERPRETATION OVER TIME 

2012

• PROPOSE FRAMEWORK FOR DQP

• ENSURE CONSISTENCY IN DOCUMENTING KEY ELEMENTS OF METRIC METADATA 

USE OF A TEMPLATE

2013 - 1 4

MEASUREMENT ERROR STUDY (WESTAT CONTRACT)

• NO SINGLE ”BEST” METHOD 

MULTIPLE METHOD AND INDICATORS (MMI) APPROACH

2016 MMI FOLLOW-UP

• EXTERNAL INDICATORS FEASIBILITY STUDY

2017 DQP VERSION 2 IN PROGRESS

2015 DQP VERSION 1

• RESPONSE RATES AND EDIT RATES

I n 2 0 0 9 , 

D Q d e f i n i t i o n 

A d o p t e d f o r C E
23 — U.S. BUREAU OF LABOR STATISTICS TATISTICS • bls.gov bls.gov

CE DQP Version 2
24 — U.S. BUREAU OF LABOR STATISTICS TATISTICS • bls.gov bls.gov

DQP Version 2: Scale up from DQP version 1

► Updated metric reporting period: 2010-2015

► New metric added: Use of Records by Survey Mode

► Metrics refined:

• Reponses rates: Additional breakouts by collection wave (Internal) 

• Expenditure edit rates: Differentiated between processed and 

reported data (Internal)

► Addition of visual summary of metric trends

Contents
25 — U.S. BUREAU OF LABOR STATISTICS TATISTICS • bls.gov bls.gov

DQP Version 2: Scale up from DQP version 1

Production Process

► Coordinated team from 3 areas of the CE Program

► Use of metric metadata template for Documentation

► All coding for analysis of metrics and graphs produced within SAS
26 — U.S. BUREAU OF LABOR STATISTICS TATISTICS • bls.gov bls.gov

Moving forward
27 — U.S. BUREAU OF LABOR STATISTICS TATISTICS • bls.gov bls.gov

Lessons Learned from DQP 2

 Spend more time for creating and reviewing the data

 Spend more time for exploring and discussing metric ideas, and 

document!

 Consult “topic experts”

 Moving the DQP to routine production will need further 

consideration about the infrastructure needed to support that
28 — U.S. BUREAU OF LABOR STATISTICS TATISTICS • bls.gov bls.gov

Next

 Upcoming: Data Quality Profile version 2 will be available for 

public users on SEPTEMBER 

 We will appreciate your feedbacks and comments! 
Contact Information

29 — U.S. BUREAU OF LABOR STATISTICS • bls.gov

Yezzi Angi Lee

Economist

Division of Consumer Expenditure Surveys

www.bls.gov/cex

202-691-5154

Lee.Yezzi@bls.gov