Updating Data Editing and Review for 

Large-scale Commodity Flow Survey 

Response Data

2023 Federal Committee on Statistical Methodology

October 24, 2023

College Park Marriott Hotel & Conference Center

3501 University Blvd E, Hyattsville, MD 20783

Gritiya Tanner
2

Any conclusions expressed herein are those of the authors and do not 

reflect the views of the U.S. Census Bureau. 

The Census Bureau has reviewed this data product to ensure appropriate 

access, use, and disclosure avoidance protection of the confidential 

source data (Project No. P-7504831, Disclosure Review Board (DRB) 

approval number: CBDRB-FY24-ESMD002-002).

Disclaimers
3

Challenges:

• 16 times data increase

• Computational costs

• Storage requirements

• Data review

Solution:

• Redesign and automate the process as much as you can

Today: 

• Highlight the implemented changes from 2017 CFS to 2022 CFS

Background
4

2022 CFS Data processing pipeline

Collect data Edit data Review 

microdata

Impute missing 

data

Apply disclosure 

avoidance and 

produce 

estimates

Review 

macrodata

Publish data 

products
5

2022 CFS Data processing pipeline

Collect data Edit data Review 

microdata

Impute missing 

data

Apply disclosure 

avoidance and 

produce 

estimates

Review 

macrodata

Publish data 

products
6

Data editing process

Edit flag to identify:

• Invalid data

• Missing data

• Common data issues

Type of edit flag:

• Shipment

• Establishment

• Quarter-to-quarter

• Hazmat

• Imputation

Data correction:

• Manually review 

each record and 

make changes 

through the user 

interface

• Programmatically 

apply changes 

based on set 

parameters 

through batch 

collection tool
7

Changes in data editing process

2017 CFS

• Data

• 6.5 million shipments

• 7 million edit failures

• Edit checks

• 3 categories

• Run every weekend

• Batch data correction

• Ad-hoc process

• Take 1 week to reflect 

data changes in database

• Data review tools

• Tables

2022 CFS

• Data

• 106 million shipments

• 170 million edit failures

• Edit checks

• 5 categories

• Run on demand

• Batch data correction

• Streamlined process

• Take 1 day to reflect data 

changes in database

• Data review tools

• Data visualization
8

2022 CFS Data processing pipeline

Collect data Edit data Review 

microdata

Impute missing 

data

Apply disclosure 

avoidance and 

produce 

estimates

Review 

macrodata

Publish data 

products
9

Review:

• Visualization tool

• Prioritize the review

• Quarter 

• Make it to tabulation

• Figure out the approach to make a correction

Make any correction or changes:

• Manually change each record through the user interface

• Programmatically apply changes through batch correction

Microdata review process
10

Microdata review tool – Visualization summary (New for 2022 CFS)
11

Microdata review tool – Visualization detail (New for 2022 CFS) 
12

CFS Production user interface
13

Batch correction tool 
14

2022 CFS Data processing pipeline

Collect data Edit data Review 

microdata

Impute missing 

data

Apply disclosure 

avoidance and 

produce 

estimates

Review 

macrodata

Publish data 

products
15

The changes on imputation process

2017 CFS

• Data

• 6.5 million shipments

• 7 million edit failures

• Imputation process

• Only two variables are 

automatically imputed. The rest 

are done through ad-hoc 

imputation.

2022 CFS

• Data

• 106 million shipments

• 170 million edit failures

• Imputation process

• All variables are automatically

imputed.

• Required a database redesign.
16

2022 CFS Data processing pipeline

Collect data Edit data Review 

microdata

Impute missing 

data

Apply disclosure 

avoidance and 

produce 

estimates

Review 

macrodata

Publish data 

products
17

Review:

• Visualization tool

• Review and learn macrodata

• Figure out the approach to make a correction

• Perform macrodata review prior to micro data review when estimation available

• Estimation available 1.5 year sooner than last cycle

Make any correction or changes:

• Manually review record and make changes through the user interface

• Programmatically apply changes through batch correction

Macrodata review process
18

Macrodata review tool - 2017 CFS version
19

Macrodata review tool – new in 2022 CFS
20

Macrodata review tool - new in 2022 CFS
21

To address the challenges associated with the large-scale data 

processing pipeline, including: 

• Automated the process as much as we can

• Redesign the database to support the automated process effectively

• Using the data visualization tool to speed up the data review

• Review macrodata before microdata when possible

These improvements can be adopted by other surveys whether 

dealing with a large-scale data set or small sets of data.

Summary
QUESTIONS?

22

Gritiya Tanner

Survey Statistician/Data scientist 

Business Development Staff

Economic Reimbursable Surveys Division

U.S. Census Bureau

gritiya.tanner@census.gov

Contacts

Updating Data Editing and Review for Large-scale 

Commodity Flow Survey Response Data