Federal Committee on Statistical Methodology

Work Group on Transparent Reporting of Data Quality

November, 2020

A Framework for

Data Quality

Rolf R. Schmitt, PhD

Bureau of Transportation Statistics
Workgroup on Transparent

Reporting of Data Quality

A brief history

• Chamberlin on data quality tradeoffs

• Program evaluation research

• Total survey error

• The Information Quality Act of 2001

• The Evidence Act, Federal Data Strategy, 

CNStat research on blended data

• ICSP policy and charge to FCSM

• FCSM workshops and preliminary report

2
Workgroup on Transparent

Reporting of Data Quality

3

A Framework for Data Quality

• Builds on experience of the Federal Statistical 

System

• Explains for a broad audience the importance of 

understanding data quality to determine fitness for 

purpose

• Organizes the many elements of data quality 

around the structure of the Information Quality Act

• Provides strategies for documenting and reporting 

data quality
Workgroup on Transparent

Reporting of Data Quality

4

Organizing threats to data quality
Workgroup on Transparent

Reporting of Data Quality

5

Factors that affect data quality

• Provides an inventory of threats to each 

dimension of data quality and examples of ways 

to manage the threats

• Covers all types of data

– Surveys

– Administrative records

– Sensor data

– Integrated/blended data and estimates

• Touches on special topics such as geographic data
Workgroup on Transparent

Reporting of Data Quality

6

Organizing threats to data quality
Workgroup on Transparent

Reporting of Data Quality

7

Organizing threats to data quality
Workgroup on Transparent

Reporting of Data Quality

8

Organizing threats to data quality
Workgroup on Transparent

Reporting of Data Quality

9

With respect to the quality of 

integrated/blended data

• Quality of integrated/blended data is often 

different than the sum of the qualities of its 

parts

– Offsetting versus exacerbating quality problems

• Estimating the aggregate quality

– Comparisons of competing estimates

– Sensitivity analysis
Workgroup on Transparent

Reporting of Data Quality

10

Reporting data quality

• Avoids past emphasis on large and resourceconsuming data quality profiles

• Applies to managers of data collection 

programs and to analysts

• Three audiences

– The data program manager / analyst

– The power user

– The occasional user or decisionmaker
Workgroup on Transparent

Reporting of Data Quality

11

Reporting data quality

• The cultural change for program managers 

and analysts: consider all threats and note 

how you address each relevant threat to 

inform your successor

• The manager’s notes provide a cornerstone 

for technical documentation for power users

• The elevator speech: describe in a few words 

how likely the data will misguide a decision
Workgroup on Transparent

Reporting of Data Quality

12

Lessons and future work

• The Information Quality Act provides a useful framework for 

examining data quality

• Data quality tradeoffs change over time

– Covid-19 put a premium on timeliness over deliberative vetting of 

accuracy

• More work is needed

– Strategies for developing and updating data quality standards

– Additional tools to measure quality in blended data sets

– Best practices for identifying quality of data obtained from sources 

that lack transparency and from advanced (AI) algorithms

– Tools for harvesting data quality notes into metadata and into effective 

caveats for power users

– Effective labeling of carefully vetted data versus experimental data

– Communicating data quality while building trust

– Other ...
Workgroup on Transparent

Reporting of Data Quality

13

Conclusion

• All data have problems, but do the problems 

matter for the decision at hand?

• Data managers should consider all possible data 

quality problems, deal with problems that can 

reasonable be addressed, and document how 

they dealt each problem for their successors

• Include data quality in guides for power users and 

summarize the problems for an elevator speech 

to tell occasional users how far they can take the 

data without misguiding decisions that have 

important consequences
Workgroup on Transparent

Reporting of Data Quality

14

For the details

• The full report is available at:

https://nces.ed.gov/fcsm/pdf/FCSM.20.04_A_

Framework_for_Data_Quality.pdf

• Information about the Federal Committee on 

Statistical Methodology is posted at:

https://nces.ed.gov/fcsm/