Session A-4 Discussion – Data Quality: Communication of Uncertainty in Official Statistics  Jeffrey M. GonzalezEconomic Research Service 2021 Federal Committee on Statistical Methodology Research and Policy ConferenceTuesday, November 2, 2021

Disclaimer: The findings and conclusions in this presentation are those of the authors and should not be construed to represent any official USDA or U.S. Government determination or policy.

Outline

Background on themes of FCSM

Discussion of presentations

Evaluating Uncertainty in Multiple Dimensions of Data Quality (Eltinge)

Brief Summary

Relevance to themes

Additional considerations

More Fully Capturing Uncertainty Associated with Official Estimates (Young)

Tailored Transparency: Public Trust vs. Reproducibility (Miller) 

Concluding remarks 

‹#›

Trust as the Cornerstone of Federal Statistics and Evidence Building

Federal data inform and improve our decisions, policies, and lives.

Question: What does it take to build trust?

Communication 

Helpfulness

Honesty

Transparency 

Question: Why do we need trust in official statistics?

Credibility for the role that the Federal Statistical System plays in data collection and outcomes research

Fosters a culture and society that values data and evidence-based policymaking

‹#›

Communication of Uncertainty in Official Statistics

Inherent connection between trust in Federal Statistics and data quality.

Citation: Federal Committee on Statistical Methodology. 2020. A Framework for Data Quality.

Adapted from Groves et al 2009 Survey Methodology, 2nd Edition

‹#›

Evaluating Uncertainty in Multiple Dimensions of Data Quality (Eltinge)

Brief summary 

Provides a comprehensive introduction to the multiple dimensions of data quality, with distinctions between quantifiable and qualitative components

Illustrates concepts and poses a series of questions about quality components with an application to the integration of multiple data sources

Relevance to themes

Framework and questions give practitioners a model, or process, to systematically consider potential threats to the utility, objectivity, and integrity – all which impact trust

Openness about each data source, integration methods, and trade-offs being made 

Additional considerations

Extensions of “backbone and bridge” beyond supplemental surveys as “bridges” to crowd-sourced or other data sources compiled by a third party

All dimensions of data quality matter, but incorporating errors for all inputs and production steps may be unfeasible

‹#›

More Fully Capturing Uncertainty Associated with Official Estimates (Young)

Brief summary

Focus on the accuracy dimension of total survey error and quantifies errors due to sampling, nonresponse, undercoverage, misclassification, and calibration

Application to the 2017 Census of Agriculture to improve upon the 2012 methods by incorporating the impact of calibration and rounding in all measures of uncertainty

Relevance to themes

Communication and transparency about standard error calculations

Inaccuracies can impact confidence in the data product and can erode users' trust

Additional considerations

Models are imperfect characterizations and can be subject to some level of subjectivity, potential conflicts with objectivity as a domain of data quality

Accounting for and understanding the relative importance of other sources of uncertainty (e.g., linkage, editing, and imputation) is critical for reducing the overall error 

‹#›

Tailored Transparency: Public Trust vs. Reproducibility (Miller)

Brief summary

Discusses issues related to disclosure of statistical and survey information from legal/regulatory and Federal Statistical System perspectives

Outlines a procedure for tailored disclosure that considers different purposes and audiences for the documentation

Relevance to themes

Recognition that an aspect of effective communication is understanding the audience

Transparency relates to accessibility and utility

Additional considerations

Focus on disclosure/dissemination of data products and statistical information to data consumers/users, applications of tailored transparency in survey recruitment

Transparency may be at odds with making undesirable features more salient 

‹#›

Concluding remarks

All data products have strengths and weaknesses across each dimension of data quality

Honest and transparent documentation enables appropriate use of the data products and reinforces credibility and utility, impacting trust

Implementation of best practices helps build and maintain trust and involves understanding the threats to each dimension and their trade-offs 

Best practices and examples of applications of these concepts exist

FCSM-20-04 A Framework for Data Quality (ed.gov)

Data Quality and Data Comparisons in the Consumer Expenditure Surveys (bls.gov)

Session F-2: Examination of the Data Quality Properties of USDA and Proprietary Databases (Milburn) 

‹#›

Jeffrey M. GonzalezResearch Mathematical StatisticianEconomic Research ServiceJeffrey.Gonzalez@usda.gov

Thank you!

‹#›

# image4.png

# image5.png

# image1.png

# image2.png

# image3.png