USDA’s Economic Research Service

Data Product Quality 

Standards and Processes

Katherine Ralston

FCSM Research and Policy Conference

October 25, 2022

The findings and conclusions in this presentation 

are those of the author and should not be 

construed to represent any official USDA or U.S. 

Government determination or policy."
ERS Mission

Inform and enhance 

public and private 

decision making on 

economic and policy 

issues related to 

agriculture, food, the 

environment, and rural 

development.
Outline

• ERS’s data product quality 

standards

• Data product review 

process

• Keys to success

www.ers.usda.gov/data-products.aspx
More Than 70 Data Products

www.ers.usda.gov/data-products.aspx
Data Products include transformations of ‘raw survey data’ used as a 

research database and/or published to the ERS website, as well as 

development of data sets collected under ERS sponsorship or compiled from 

diverse sources where ERS adds value in the form of recompilation and 

subject-matter expertise.

Types of ERS Data Products

Primary Data 

• ARMS

• FoodAPS

• Scanner data 

(best example 

of big data)

Model Results

• Food Dollar

• Commodity 

Supply and 

Use

Summary 

Statistics

• Foreign 

Agricultural 

Trade of the 

U.S.

Repackaged 

Data 

• State Fact 

Sheets

• Food 

Environment

• Land use/ 

quality 

research data
Audiences and Data Users

• Policymakers and their staffs

• Policy influencers 

• Government

• Agribusiness professionals

• Academics/researchers

• News media

• Informed lay people

• Developers and digital professionals
ERS Data Product Review Committee Charge (2012)

Develop a data policy for core, high-value statistics:

• Consistent with OMB guidance

• Reflect best practices at ERS, USDA, and other principal 

statistical agencies to ensure objectivity and quality

Develop a hierarchy of ERS data and information 

systems that recognize differences in quality, usability, 

purpose, transparency, and utility:

• Attributes can be mapped to measureable characteristics

• Recommended practices and policies will recognize the 

various types of data produced. 

• Framework will form the basis for cost-benefit analysis to 

ensure efficient management of data resources. 
Considered OMB Directives, Policies, and 

Guidance: • Statistical Policy Directive Number 3

• Statistical Policy Directive Number 4

• Standards and Guidelines for Statistical Surveys

• Guidelines for Ensuring and Maximizing the Quality, Objectivity, Utility, and 

Integrity of Information Disseminated by Federal Agencies

• OMB Circular A-130

• Open Data Policy-Managing Information as an Asset

Emulated best practices from other 

Federal Statistical Agencies: • BTS Statistical Standards Manual

• BEA Information Quality Guidelines

• NASS 

• National Center for Science and Engineering Statistics/NSF

Formulation of 

Data Product Quality Standards 
Hierarchy: Three Tiers of Data Products

Criteria developed to rank data products in terms of adherence 

with the OMB and USDA definitions of Influential Scientific, 

Financial, or Statistical Information and quality guidelines, as 

well as the importance to the agency’s mission: 

• Premier data products: influential and central to the agency’s 

mission, as well as adhering to all components of quality 

guidelines as applicable.

• Core data products: central to the agency’s mission but do 

not meet the criteria for influential. Includes foundational 

data that serve as inputs to premier data products.

• Other data products: serve key agency stakeholders and the 

public. 
For Each OMB Attribute, 

Committee Developed Measurable Standards That Can 

Be Applied to Different Types and Tiers of ERS Data

UTILITY

External Methods 

Review

Accuracy Measures 

Reported

Modern IT Production 

Processes

TRANSPARENCY

Release Scheduled on 

ERS Calendar

Substantial Changes to 

Methods Announced

Production Methods 

Documented

OMB 

Attribute

Suggested 

Measure

Attributes: Purpose, Utility, Objectivity, Transparency, Integrity, and Accessibility
Data Product Quality Standards

• Operationalize OMB Directives

• Categories:

– Purpose/MOUs

– Utility

– Objectivity

– Transparency

– Integrity

– Accessibility

• 25 Standards (36 checklist 

items) https://www.ers.usda.gov/about-ers/policies- and-standards/data-product-quality/ers-data-product- quality-standards/
Purpose

1.1 MOU for 

data 

provided to 

others

1.2 MOU for 

data 

received 

from 

others

Utility

2.1 Preeminence 

and pertinence

2.2 Branding 

and data 

sources

2.3 Future 

releases 

2.4 Stakeholders

2.5 Quality of 

communication

2.6 Feedback

2.7 Web Stats

Objectivity

3.1 Data quality 

review 

procedure

3.2 External 

review of 

methods

3.3 Accuracy 

measures 

reported

3.4 Research 

on Methods 

and 

Operations

3.5 IT 

investment

Transparency

4.1 Public notice 

of changes

4.2 Good 

documentation

4.3 Explanation 

of similar data 

elsewhere

4.4 Update and 

revision history 

available

4.5 Archival 

capability

Integrity

5.1 Disclosure 

protection

5.2 Physical 

security

5.3 Staff 

training 

Accessibility

6.1 Machine 

readable

Open format

Section 508

6.2 Usability 

testing

6.3 Metadata
Initial Evaluation of Data Products 

Against Data Quality Attributes
Applying the Standards In-Depth:

Data Product Review Council

• Reviews Premier products every 5 years, 

others as time permits

• Develops guidance for meeting standards

• Helps track Action Plan progress for ERS 

management

• Improves review process to minimize time by 

researchers and reviewers
Data Product 

Review Process

• Selection of products, notification, kickoff meeting

• Data product managers submit response form 

describing how their product meets each ERS data 

quality standard

• Council provides comprehensive evaluation for 

adherence to the standards

• Council offers feedback and guidance to data product 

managers and their branch chiefs, and identifies areas 

for improvement if necessary

• Data product managers submit action plans for 

addressing findings, developed in consultation with 

branch chiefs, incorporated into annual workplan
Keys to Success

• Build from best practices and existing 

guidelines

• Build in flexibility to adapt to change

• Set feasible priorities 

• Invest in staff training

• Pay attention to incentives facing staff

• Collaborate with Data Product Managers
Questions?

ERS Data Product Quality 

Join us online: www.ers.usda.gov

Follow us on Twitter: @USDA_ERS