Assessing the Automated Imputation of Missing 

and Erroneous Survey Data: A Simulation-Based 

Approach

Larkin Terrie

Federal Committee on Statistical Methodology Research and Policy 

Conference

March 7, 2018
Outline

•Introduction to auto-editing at BEA

•Proposal of simulation-based testing 

framework

•Results regarding how successfully the 

simulation mimics reality and the accuracy of 

auto-editing imputations

•Conclusions

4/30/2018

1
Introduction: Auto-Editing at BEA

•Focused on annual direct investment surveys, 

which collect financial and operating data from:

–U.S. multinational enterprises and their foreign 

affiliates

–Foreign-owned U.S. companies

•Motivation: allow editors to spend more time 

on most complex/impactful responses, improve 

general efficiency of survey editing

4/30/2018

2
Approach to Auto-Editing

•Implementation of Banff system for data 

editing and imputation

•Key procedures:

–Error localization

–Donor imputation

–Estimator imputation

4/30/2018

3
The Research Question

•How should auto-editing be evaluated?

–BEA’s Iurreミt approaIh: Ioマpare to results 

of manual editing

–Ideal approach: compare to true values

4/30/2018

4
New Framework

•Fiミd さIleaミざ forマs

•Simulate missing/erroneous data

•Impute

•Compare imputations to reported values

4/30/2018

5
Testing of New Framework

•Data: 2015 BE-15C (8 numeric items)

•Key Issues: 

–Proximity of imputed values to reported 

values

–Comparison of different versions of 

imputation procedures

4/30/2018

6


How realistic are the simulated data?

• FTIs per form:

– Actual data: 0.234 

Simulated data: 0.237

4/30/2018

9

Field Selected as FTI

Observed 

Average 

Percent 

Share

Simulated 

Average 

Percent 

Share

Assets 0.3 0.3

Liabilities 1.6 1.8

Sales 23.2 23.6

Net Income 4.5 5.5

Employee 

Compensation 23.9 25.2

Gross PP&E 18.2 16.2

R&D 9.2 8.2

Employees 19.1 19.2

Distribution of FTIs among Survey Items 

in Actual vs. Simulated Data
The Tests

•Two versions of auto-editing system 

tested:

1. Base settings

2. Additional years of data used for donor and 

estimator imputation

4/30/2018

10

Summary of Test Results

4/30/2018

12

First Test Second Test

Field

Avg. % Diff. 

(y̅)

Avg. Abs. % 

Diff. (x̅)

Avg. % Diff. 

(y̅)

Avg. Abs. % 

Diff. (x̅)

Assets -0.01 0.06 -0.01 0.05

Liabilities 0.01 0.23 -0.01 0.20

Sales 0.09 1.47 0.08 1.31

Net Income -0.04 6.22 0.35 5.84

Employee 

Compensation -0.08 0.70 -0.23 0.71

Gross PP&E 0.12 1.17 0.26 1.20

R&D -1.81 5.04 -2.05 3.88

Employees 0.01 0.51 -0.03 0.52

Accuracy of Imputations by Field and Test
Are 5,000 runs enough?

4/30/2018

13

The NuマHer of Ruミs aミd the Measureマeミt of Iマputatioミs’ AIIuraIy
Comparing Versions 1 and 2

4/30/2018

14

Stability of Differences Between Versions 1 and 2 of Imputation Procedures
Summary and Conclusions

•Proposed ミe┘ マethod for assessiミg BEA’s 

auto-editing systems

•Found close agreement between imputed 

values and reported values

•Identified means of improving imputation 

procedures

4/30/2018

15
Contact Information

•Questions on the presentation?

–Larkin Terrie: Larkin.Terrie@bea.gov

•Questioミs oミ BEA’s direIt iミ┗estマeミt statistiIs?

–Internationalaccounts@bea.gov

4/30/2018

16

Thank You!