Leveraging the Data 

Quality Framework 

Workshop

September 10, 2021

A Virtual Workshop Sponsored by the Interagency Council on Statistical Policy, Chief Data Officer Council, and Evaluation Officer Council
Opening Remarks

Hubert Hamer, USDA NASS
Mentimeter

• Go to 

https://www.menti.com/xau2spxsrs

• Go to menti.com and enter the code: 

4081 7831

• Scan the QR code with your mobile 

device
Leveraging the Data Quality Framework

September 10, 2021

The FCSM Framework for

Data Quality

Jennifer D. Parker, Ph.D.

National Center for Health Statistics
Background

• Understanding data quality is essential for data-driven 

decision making

– Data users who understand the “fitness-for-use” of data 

products are more likely to use them appropriately 

– Higher-impact uses of data require higher quality data

• All data have strengths and weaknesses

• Data quality for surveys is relatively well-established but data 

quality for integrated data and other non-statistical data are 

less developed
Data Quality Milestones 2001-2020 (A)

WP #31

• 2001 Measuring and Reporting Sources of Error in Surveys

• Focus on reporting accuracy of survey data outputs

IQ Act

• 2001 Information Quality Act/OMB Guidelines

• Provided a framework, with a call for more detailed OMB and 

Agency Guidelines

Standards

• 2006 OMB Statistical Policy Directive 2: Standards and 

Guidelines for Statistical Surveys

• Emphasis on survey data quality
Data Quality Milestones 2001-2020 (B)

Integrated

Data

• System-wide declining response rates, increasing costs

• Increased use of non survey data sources, alone or integrated with 

statistical survey data

Evaluations

• 2015-17 Two CNSTAT reports and the Commission on EvidenceBased Policymaking, integrated data

• New visions for Federal statistics; identified obstacles and provided 

recommendations for moving forward

2018 

Evidence Act

• Federal Data Strategy, Foundations for Evidence Based Policymaking 

Act, revised OMB Information Quality Act Guidelines

• Address data quality and compatibility with integrated data

Quality 

Framework

• 2018-2019 seminars on data quality and integrated data

• FCSM Data Quality Framework
FCSM Framework for Data Quality

• Builds on experience of the Federal Statistical 

System

• Organizes the elements of data quality around the 

structure of the Information Quality Act

• Explains for a broad audience the importance of 

understanding data quality to determine fitness 

for purpose, identifying and mitigating key data 

quality threats, and evaluating trade-offs

• Provides strategies for documenting and reporting 

data quality

8
FCSM Framework for Data Quality

9
Domains of Data Quality

• Utility - the extent to which information is welltargeted to valuable needs: it reflects the 

usefulness of the information to the intended users

• Objectivity - whether information is accurate, 

reliable, and unbiased, and is presented in an 

accurate, clear and interpretable, and unbiased 

manner 

• Integrity – the maintenance of rigorous scientific 

standards and the protection of information from 

manipulation or influence as well as unauthorized 

access or revision 

10
Dimensions of Utility - I

• Relevance: whether the data product is 

targeted to meet current and prospective user 

needs

• Credibility: the confidence that users place in 

data products based simply on the image of the 

data producer 

• Accessibility: the ease with which data users 

can obtain an agency’s products and 

documentation in forms and formats that are 

understandable to data users. 

11
Dimensions of Utility - II

• Timeliness: the length of time between the 

event or phenomenon the data describe and 

their availability

• Punctuality: the time lag between the actual 

release of the data and the planned target date 

for data release 

• Granularity: the amount of disaggregation 

available for key data elements. 

12
Dimensions of Objectivity

• Accuracy: the closeness of an estimate from a 

data product to its true value 

– Reliability: characterization of repeated estimates 

of accuracy over time 

• Coherence: the ability of the data product to 

maintain common definitions, classification, 

and methodological processes, to align with 

external statistical standards, and to maintain 

consistency and comparability with other 

relevant data 

13
Dimensions of Integrity

• Scientific Integrity: an environment that ensures the use 

of established scientific methods to produce and 

disseminate objective data products and shields these 

products from inappropriate political influence 

• Computer and Physical Security: the protection of 

information throughout the collection, production, 

analysis, and development process from unauthorized

access or revision to ensure that the information is not 

compromised through corruption or falsification

• Confidentiality: a quality or condition of information as 

an obligation not to disclose that information to an 

unauthorized party

14
Threats to Data Quality

• Threats can be identified for all dimensions

– Threats can be relevant for multiple dimensions

– Mitigating threats for one dimension can increase 

threats for another

• Managing trade-offs among quality dimensions 

is important

• Threats to quality for blended data combine 

threats for data inputs, blending methods, and 

data outputs

15
Assessing Data Quality

• Regularly identify threats to data quality for 

ongoing data collections, including when 

considering new source data for inclusion 

– Decisions on trade-offs among threats and mitigation 

measures should be considered in the context of the

data’s purpose and all identified threats

– Data quality for the intended use may differ from 

that for its original purpose

16
Conclusion

• Data quality has been long studied for 

statistical data, especially surveys, but is less 

developed for integrated and secondary-use 

data

• The FCSM Data Quality Framework can be used 

to evaluate quality for all data

17
September 2021

Using the Framework for

Data Quality

Rolf R. Schmitt, PhD

Bureau of Transportation Statistics
19

The Framework for Data Quality

• Organizes the many elements of data quality 

around the structure of the Information Quality 

Act

• Provides a comprehensive and consistent 

terminology to describe the many aspects of 

data quality

• Looks overwhelming to use and burdensome to 

report
20

Don’t panic

• Many data quality threats can be dismissed 

after brief consideration for a data program

• There are few universal rules for weighing 

importance of one data quality concern over 

another: tradeoffs are expected

• Documentation while planning and doing what 

you do is a good habit that helps your 

successors and supports transparency
21

Reporting data quality

• Data quality reports as a byproduct of documenting your work

• Applies to managers of data collection programs and to 

analysts

• Three audiences

– The data program manager / analyst

– The power user

– The occasional user or decisionmaker
22

Reporting data quality

• The cultural change for program managers and analysts: 

consider all threats and note how you address each relevant 

threat to inform your successor

• The manager’s notes provide a cornerstone for technical 

documentation for power users

• The elevator speech: describe in a few words how likely the 

data will misguide a decision
23

Tradeoffs change over time

• Covid-19 put a premium on timeliness over deliberative vetting 

of accuracy

• “It may be better, in the gross affairs of life, to be less precise 

and more prompt. Quick decisions, though they may contain a 

grain of error, are often better than precise decisions at the 

expense of time.”

– T.C. Chamberlin, President of the University of Wisconsin, 1890
24

Future work

• Additional tools to measure quality in blended data sets

• Best practices for identifying quality of data obtained from sources 

that lack transparency and from advanced (AI) algorithms

• Tools for harvesting data quality notes into metadata and into 

effective caveats for power users

• Effective labeling of carefully vetted data versus experimental data

• Communicating data quality while building trust

• Other ...
25

Conclusion

• All data have problems, but do the problems matter for the 

decision at hand?

• Data managers should consider all possible data quality problems, 

deal with problems that can reasonable be addressed, and 

document how they dealt each problem for their successors

• Include data quality in guides for power users and summarize the 

problems for an elevator speech to tell occasional users how far 

they can take the data without misguiding decisions that have 

important consequences
26

Conclusion

• By using the structure and terminology of the Framework, we 

will have a common basis for sharing information about data 

quality across agencies and with the public

• A common language will support transparency about our 

current data and analyses and a common basis for considering 

improvements in data and analysis
27

For the details

• The full report is available at:

https://nces.ed.gov/fcsm/pdf/FCSM.20.04_A_Framework_for_

Data_Quality.pdf
Avital Percher

Office of the Director

09/10/21

DESIGNING A DATA 

QUALITY POLICY 
NSF’s Frame of Reference 

Evidence Act

Federal Data Strategy The CDO shall “ensure that, to the extent practicable, 

the agency maximizes the use of data in the agency”

Learning Agendas: “Systematic way to identify the 

data agencies intend to collect, use, or acquire, as 

well as the methods and analytical approaches to 

facilitate the use of evidence in policymaking”

Mission 

(NSF Foundation Act, 1950) 

To promote the progress of science; to advance the 

national health, prosperity, and welfare; and to 

secure the national defense; and for other purposes. 
across the data, information, knowledge stages. 

“It applies to all data: data collections and data systems; restricted and public use 

microdata files; data products produced through data integration, modeling, 

harmonization and other statistical analyses; and analysis outputs, such as tables, 

estimates, graphics and reports.” FCSM Data Quality Framework 

To be a strategic asset, data must be transparent, verified, and documented...

Leveraging the Data Quality Framework 
Dimensions of the Data Quality Framework 
ATLAS Experiment at CERN 

Roads Transport Route
Roads Transport Route

Data 

Inventory 

Analytic 

Tools

Analytic 

Outputs

across the data, information, knowledge stages. 

To be a strategic asset, data must be transparent, verified, and documented...
Data Quality Policy

Data 

Inventory 

Analytic 

Tools

Data management 

lifecycle is standardized, 

accessible and detailed

Tools processing the 

data are documented 

and vetted

Queries and analyses 

are documented and 

reproducible

Analytic Tool 

Standards

Data Inventory 

Standards

Best Practices for Analytics 

Documentation

Analytic 

Outputs
Data Quality Policy - Process

Data 

Inventory 

Analytic 

Tools

Inventories prepared by 

Data Stewards 

Analytic 

Outputs

Feedback provided by 

internal user community

Reviewed and approved 

by Data Governing Body

Tools used at 

Enterprise level 

Methods and 

documentation 

reviewed by Data 

Governing Body

Output generated by 

office 

Documented and 

archived internally by 

office standards 
Thank You!

Avital Percher: apercher@nsf.gov

Dorothy Aronson (CIO/CDO): daronson@nsf.gov
Data Inventory Standards – Objectives

Defines metadata documentation standards and review process

Compliance

Support Agency 

compliance with 

federal 

mandates

Roles & 

Responsibilities

Define the roles 

and 

responsibilities in 

the Data 

Inventory 

Management 

Process

Documentation 

Maintenance

Define the 

requirements 

for maintaining 

metadata profiles 

and data 

dictionaries of 

NSF’s data 

repositories

Master Metadata 

Schema

Define a master 

metadata and 

dictionary 

schema as an 

agency standard

Validation

Define a user 

inclusive 

validation 

process
Analytic Tools Standards - Objectives

Defines documentation requirements and validation process for tools used on an ‘enterprise’ level. 

Community Standards

Define a community standard 

of excellence and support 

leadership’s need for 

trustworthy and vetted data 

tools. 

Tool Documentation 

Benchmark

Establish a benchmark for 

tool documentation to 

promote development and 

application practices that 

align with community best 

practices. 

Review and Approval

Describe a review and 

approval process by the 

EADGSC to support the NSF 

community’s need for tools 

vetted by data experts. 
Best practices for analytics documentation- Objectives

Improved Quality 

Standards

Enhance the quality and 

trustworthiness of the data 

collection and analysis. 

Replication 

Enable replication of the 

analysis as needed in the 

future, by both the office and 

others. 

Knowledge 

Dissemination

Allow the adaptation of the 

study to other needs of the 

community, increasing 

efficiency. 

Defines guidelines for documenting analytics outputs 
STAGE 1

Validate data with

domain data steward

expertise and submit 

for Data Governance 

(EAGDSC) approval

Output

Finalized metadata & data dictionary

Inputs for master data management

Publish validated 

and 

approved data for 

internal NSF use

Output

Published & searchable 

data inventory (metadata 

& data dictionary)

Collect data lineage 

information from

technical data 

stewards

Output

Draft metadata & data 

dictionary for RPTSQL tables

STAGE 2 STAGE 3

EDI RPTSQL PILOT STAGES

1Stage 1 corresponds with the beginning of Q3. Activities preceding Q3 are not included in the listed stages.

Data Inventory Standards
41 — U.S. BUREAU OF LABOR STATISTICS • bls.gov

Quality Considerations for Alternative Data:

A Case Study using CORP5 Data

John Bieler 

Senior Economist, CPI

September 10, 2021
42 — U.S. BUREAU OF LABOR STATISTICS TATISTICS • bls.gov bls.gov

Familiar?

December 2017
43 — U.S. BUREAU OF LABOR STATISTICS TATISTICS • bls.gov bls.gov

Scorecard for Alternative Data

Quality Metrics

Sample 

Frames Benchmarking Hedonics

Replace 

collection

Supplement 

Collection

Data 

Validation

Data Transparency- methods 

understood

Granularity- Level of detail

Quality of descriptive data

Scope, type of price

Coverage- items

Coverage- geography

Coverage- outlets

Sampling procedures

Data delivery reliable

Viability of data source

Data Usability

Data Frequency

Data Security

Data delivery timeliness

Data history

Data Cleanliness

Data Usability- mods to current 

system

Familiar?
44 — U.S. BUREAU OF LABOR STATISTICS • bls.gov

The Framework
45 — U.S. BUREAU OF LABOR STATISTICS TATISTICS • bls.gov bls.gov

Background on CORP5

 CORP5 is a secondary source of gas price data 

 Average of roughly 205,000 reported gas price observations 

every day

 Roughly 6.23 million gas prices every month! 

 Gas prices are updated in real-time

 CPI receives data the following day

 CORP5 data includes prices for three categories: Regular 

unleaded gasoline, Mid-grade, and Premium

 BLS obtained approval from CORP5 to use their data and 

began to voluntarily provide their data using a secure portal
46 — U.S. BUREAU OF LABOR STATISTICS TATISTICS • bls.gov bls.gov

CORP5 case study

Domain Dimension Definition Question Answer

Utility

Relevance

Relevance refers to 

whether the data product 

is targeted to meet 

current and prospective 

user needs.

Is the data a relevant input to our 

data products and measurement

our measurement objective?

CORP5 provides daily gasoline 

prices for thousands of stations 

across the U.S.

Produce indexes and average 

prices for gasoline and individual 

fuel types.

Accessibility

Accessibility relates to the 

ease with which data 

users can obtain an 

agency’s products and 

documentation in forms 

and formats that are 

understandable to data 

users.

Are the costs to access the data 

an effective use of resources?

Will the methodology limit our 

ability to release data to users?

How can we describe the 

methodology to data users?

CORP5 is providing the data on a 

voluntary basis.

Make a public announcement in 

advance and provide materials on 

line, such as factsheets and 

articles. 
47 — U.S. BUREAU OF LABOR STATISTICS TATISTICS • bls.gov bls.gov
48 — U.S. BUREAU OF LABOR STATISTICS TATISTICS • bls.gov bls.gov

CORP5 case study cont.

Domain Dimension Definition Question Answer

Utility

Timeliness

Timeliness is the length 

of time between the 

event or phenomenon 

the data describe and 

their availability.

Are the data representative of 

the index reference period?

Yes, daily prices across the 

month.

Punctuality

Punctuality is measured 

as the time lag between 

the actual release of the 

data and the planned 

target date for data 

release.

Can the methodology be 

implemented within the typical 

production processing schedule?

What is the probability and 

impact on the production 

schedule due to delayed delivery 

of data or unexpected time 

needed to process data? 

Yes, CORP5 will be implemented 

into the current production 

schedule. We are currently 

parallel testing. 

Based on multi year evaluation 

period, the probability of an 

impact is low. 
49 — U.S. BUREAU OF LABOR STATISTICS TATISTICS • bls.gov bls.gov

Corp5 case study cont.

Domain Dimension Definition Question Answer

Utility Granularity

Granularity refers to the 

amount of disaggregation 

available for key data 

elements. Granularity can 

be expressed in units of 

time, level of geographic 

detail available, or the 

amount of detail available 

on any of a number of 

characteristics (e.g. 

(demographic, socioeconomic).

Is there adequate data to support 

the current level of granularity in 

data products?

Is there sufficient data to 

adequately protect 

confidentiality?

Yes, we will produce price indexes 

and average price products at the 

same level of granularity.

Yes, thousands of gas stations 

protecting confidentiality.
50 — U.S. BUREAU OF LABOR STATISTICS TATISTICS • bls.gov bls.gov

Comparison of regular gas prices

CPI CORP 5
51 — U.S. BUREAU OF LABOR STATISTICS TATISTICS • bls.gov bls.gov

Comparing number of prices

2,516,768 

139,107 4,500 

 -

 500,000

 1,000,000

 1,500,000

 2,000,000

 2,500,000

 3,000,000

CORP5 Daily

Price

Observations

CORP5 Monthly

Average Prices

CPI Sample
52 — U.S. BUREAU OF LABOR STATISTICS TATISTICS • bls.gov bls.gov

CORP5 case study cont.

Domain Dimension Definition Question Answer

Objectivity

Accuracy and 

reliability

Accuracy measures the closeness of 

an estimate from a data product to 

its true value. Reliability, a related 

concept, characterizes the 

consistency of results when the same 

phenomenon is measured or 

estimated more than once under 

similar conditions.

Any concerns with the 

qualitative assessment of total 

measurement error?

No, research results compared 

favorably to the CPI Gasoline 

index at the U.S. Level.

Coherence

Coherence is defined as the ability of 

the data product to maintain 

common definitions, classification, 

and methodological processes, to 

align with external statistical 

standards, and to maintain 

consistency and comparability with 

other relevant data.

Does the methodology impact 

the ability to compare CPI data 

with external sources?

Is the methodology coherent 

with other CPI methodologies 

(not just what it is replacing)?

No, the methodology is still 

comparable with external 

sources. 

Yes, a mix of geomeans and 

Laspeyres index methodology. 

Added additional aggregation 

steps.
53 — U.S. BUREAU OF LABOR STATISTICS TATISTICS • bls.gov bls.gov

CORP5 Research – Differences never greater than 1% at 

U.S. level for gasoline
54 — U.S. BUREAU OF LABOR STATISTICS TATISTICS • bls.gov bls.gov

CORP5 Stored Meta Data

AREA AREA_DESC NUM_PR_OBS NUM_RELATIVES NUM_PHYS_LOCATIONS

0000 U.S. 9,576,611 129,755 50,049 

N000

Non-Self-Representing 

PSUs 3,745,235 42,976 16,753 

S000 Self-Representing PSUs 5,831,376 86,779 33,296 

S12A

New York-NewarkJersey City, NY-NJ-PA 265,885 10,187 3,932 

S23A

Chicago-NapervilleElgin, IL-IN-WI 901,102 6,855 2,462 

S49A

Los Angeles-Long 

Beach-Anaheim, CA 447,457 7,086 2,408 
55 — U.S. BUREAU OF LABOR STATISTICS TATISTICS • bls.gov bls.gov

CORP5 case study cont.

Domain Dimension Definition Question Answer

Integrity

Scientific 

integrity

Scientific integrity refers to 

an environment that 

ensures adherence to 

scientific standards and use 

of established scientific 

methods to produce and 

disseminate objective data 

products and one that 

shields these products from 

inappropriate political 

influence.

What is the probability and 

impact of the data provider 

(either maliciously or 

unintentionally) interfering 

with the data in a way that 

impacts estimates?

The probability is low and the 

impact is low. There is no 

incentive for the data 

provider to manipulate the 

data.

Credibility

Credibility characterizes the 

confidence that users place 

in data products based 

simply on the qualifications 

and past performance of 

the data producer.

Review the output of index 

simulations. The more a 

simulation deviates from 

production, the more of an 

understanding approvers 

would like to have of the 

cause of differences.

Often cited source in news 

organizations and widely 

accepted by users as a 

credible source of price 

information.
56 — U.S. BUREAU OF LABOR STATISTICS TATISTICS • bls.gov bls.gov

CORP5 case study cont.

Domain Dimension Definition Question Answer

Integrity Computer and 

physical security

Computer and physical security 

of data refers to the protection 

of information throughout the 

collection, production, 

analysis, and development 

process from unauthorized 

access or revision to ensure 

that the information is not 

compromised through 

corruption or falsification.

What is the probability and 

impact of risks of a loss of 

data or data quality issues 

due to technical issues?

The fallback plan is to use 

CPI collected data.
57 — U.S. BUREAU OF LABOR STATISTICS TATISTICS • bls.gov bls.gov

CORP5 case study cont.

Domain Dimension Definition Question Answer

Cost 

effectiveness 

(CPI 

addition)

Are the new data and methods 

cost effective relative to the data 

and methods they are replacing? 

Include development costs 

contracting costs, data collection 

costs, data storage, and 

maintenance costs.

Using the CORP5 data is cost neutral 

at this point. 
Contact Information

58 — U.S. BUREAU OF LABOR STATISTICS • bls.gov

John Bieler

Senior Economist

Consumer Price Index

(202) 691-5407

bieler.john@bls.gov
59

Data Quality Evolution

Dorothy Aronson

Chief Information Officer/Chief Data Officer

09/10/21
When I think about Data Quality for NSF...

Source: Data quality, framework, accessibility are key to implementing emerging technologies | Federal News Network

60
Ut oh...

61

When I think about Data Quality for NSF...

CIO
Ut oh...

62

When I think about Data Quality for NSF...

CDO

CIO
Ut oh...

63

When I think about Data Quality for NSF...

Ut oh...

CIO

CDO
Ut oh...

64

When I think about Data Quality for NSF...

What does “data 

quality” even mean?

CIO

CDO

Relevance

Accessibility

Timeliness

Punctuality

Granularity

Accuracy

Reliability

Coherence

Scientific Integrity

Credibility

Computer and Physical Security

Confidentiality
Ut oh...

65

FCSM offers a framework.

What does “data 

quality” even mean?

CIO

CDO

Relevance

Accessibility

Timeliness

Punctuality

Granularity

Accuracy

Reliability

Scientific Integrity

Credibility

Computer and Physical Security

Confidentiality
NSF aligns within the framework.

66
BLS demonstrates 

alignment with 

the framework.

67
Framework 

provides a 

common 

language...

...allowing 

necessary 

variation to fit 

mission.

68
NSF’s Data Quality: Lessons Learned

UNDERSCORE AGENCY GOALS

AND ALIGN WITH FEDERAL

POLICIES

CENTER THE END USER BOTTOM-UP AND

TOP-DOWN TACTICS

FOCUS ON PROGRESS

OVER PERFECTION

69
NSF’s Data Quality Initiatives: Challenges and Solutions

ISSUE CHALLENGE SOLUTION

Getting Started When creating a policy from scratch 

there is significant time spent 

collecting artifacts.

Use the numerous existing resources 

to create the basis for the policy.

Maintaining Scope Through the drafting and review 

process several other policy needs 

were identified.

Instead of incorporating the ideas 

into the draft policy, log the ideas for 

future policy development efforts.

Establishing an 

Inclusive Process

Numerous stakeholders have an 

interest in supporting the 

development of the policy.

Small teams assist in policy 

development. Iterative and inclusive 

review process. Tailored briefings for 

senior staff.

Making the Change 

Stick

Implementing a new policy requires 

buy-in across the agency.

(In process) Imbed Data Governance 

Group in policy implementation. 

Work to build a policy and tools that 

provide value to stakeholders.

70
BLS’ CORP5 Case Study: Standout Points

SHOWCASING SCALED IMPACT ACROSS THE ALTERNATIVE DATA SOURCES

INTEGRATING EXISTING FRAMEWORKS AND RESOURCES (E.G., THE FRAMEWORK AND THE

SCORECARD FOR ALTERNATIVE DATA)

STRENGTHENING EVIDENCE-BUILDING EFFORTS BY USING SECONDARY DATA SOURCES TO

SUPPLEMENT AGENCY DATA

71
Contact Information

Dorothy Aronson

CIO/CDO

NSF

daronson@nsf.gov

703.292.4299

72