Score Functions for Selective Editing of the US Census Bureau Trade Data

The Foreign Trade Division at the US Census Bureau is responsible for the production and publication of monthly import and export statistics. These data are not survey based, but collected electronically through an online internet filing system upon arrival or departure of merchandise goods. The validity of these data is checked and verified at every step of collection, processing, and tabulation. In this report we present score functions for selective editing of these data. The scores have two components: a measure of how suspicious an incoming record is and a measure of the impact changes in the record may have on publication totals within a particular set of commodity groupings. We also present results of a feasibility study on the application of selective editing to the checking and correction phase of foreign trade exports data processing. I. Background on Foreign Trade Data Processing The Foreign Trade Division (FTD) at the Census Bureau processes monthly import and export transactions for the shipment of merchandise between the United States and its international trading partners and publishes the official international trade statistics for the country. Foreign trade transactions are filed via an online internet data collection system, mostly through the U.S. Customs and Border Protection. The collection of these data is unusual at the Census Bureau because they are filed upon arrival or departure of merchandise goods and are not based on surveys or censuses that are sent to respondents soliciting responses. Data items collected include commodity, country of origin or destination, port of arrival or dispatch, value, quantity, and shipping weight. Data processing begins with extensive micro-editing using the division’s automated edit and imputation system that uses a parameter file called the Edit Master. The Edit Master verifies that numeric data fall within the prescribed ranges and that the ratios of highly correlated items fall within prescribed commodity bounds. Records that do not pass the edits are automatically imputed. However, imputation may not be successful for a small portion of the edit failing records. Records for which imputation is not successful are marked as “rejects” and distributed by commodity and sent to subject matter experts for manual review. The analysts use their commodity expertise to manually adjust rejected records. They may also call back filers in an attempt to correct erroneous data. The commodity experts review a large number of records under tight timeconstraints before the publication of monthly statistics deadline. Due to the time and resource constraints, the division has an ongoing effort to improve the current procedures while preserving (or improving) data quality. To this aim, we are investigating the feasibility of a selective editing application to these data. 1 The authors thank Ryan Fescina and William Yancey for their review and helpful comments. This report is released to inform parties of ongoing research and to encourage discussion. The views expressed are those of the authors and not necessarily of the U. S. Census Bureau.