Bayesian Data Editing for Continuous Microdata

Hang Kim12, Jerome Reiter1, Alan Karr2, and Lawrence Cox2

Duke University1 and National Institute of Statical Sciences2

2013 FCSM Research Conference 

Washington, DC 

November 6, 2013 

Research supported by the National Science Foundation [SES-11-31897]
Contents

1

2

3

4

Statistical Data Editing

Bayesian Data Editing for Continuous Microdata

Simulation Study

Concluding Remarks
Contents

1

2

3

4

Statistical Data Editing

Bayesian Data Editing for Continuous Microdata

Simulation Study

Concluding Remarks
Statistical data editing

procedure of detecting and correcting errors in records to improve data quality

Manual editing vs. Automatic editing

Manual editing spends high costs and times with a large number of records

Some complex feature of data can be found by using computer power

Automatic editing can replace manual editing while preserving (or improving)

the released data quality

Two steps of automatic editing

1 Error localization step∗1 

identifies erroneous records and fields to be corrected for the records 

2 Imputation step 

replaces the identified fields with more accurate data 

∗1 De Waal, Pannekoek, and Scholtus (2011)

Statistical Data Editing 1/ 17
Automatic error localization

identifies erroneous records and fields to be corrected for the records

Two mathematical approaches of error localization∗1

1 Statistical modeling-based approach

identify unusual records (outliers) under a statistical model

extensively discussed in the literature but scarcely used in practice

2 Mathematical optimization-based approach

use logical conditions, edit rules to find inconsistent records 

often based on the (generalized) Fellegi-Holt paradigm (Fellegi and Holt 1976) 

best-known and most-used methods by statistical agencies 

ex) SPEER (U.S. Census Bureau), 

AGGIES (National Agricultural Statistics Service),

Banff (used to be called GEIS, Statistics Canada), . . .

∗1 De Waal, Pannekoek, and Scholtus (2011)

Statistical Data Editing 2/ 17
Edit rules for continuous data (using in optimization-based approach)

“Edit rule is a logical condition to the value of a data field which must be met if the

data is to be considered correct” (UNECE 2000)

x ̃i = {x ̃i1, . . . , x ̃ip}: record i with reported values of p fields and q balance edits

Range restriction 

Lj ≤ x ̃ij ≤ Uj where j = 1, . . . , p 

Ratio edit x ̃ij ' Ljj ≤ ≤ Ujj where j = j x ̃ij 

Balance edit

 x ̃ij = x ̃isl where l = 1, . . . , q

j∈Cl

Cl: the set of indices for reported components

sl: the index for the reported sum

Statistical Data Editing 3/ 17
 

The (generalized) Fellegi-Holt algorithm

1 Find all implicit edit rules from user-specified edits

2	 Define the latent variable sij

If sij = 1, field j is “flagged” and replaced with a reasonable value

If sij = 0, field j is released without editing

3 Minimal set of (weighted) fields to impute criterion

(in short, minimum change criterion). Solve the optimization problem to find

the values of {si1, . . . , sip} which minimizes

p

wj sij

j=1

where wj is the reliability weight of field j

4 Blank fields j with sij = 1 and impute them by imputation methods (e.g.

Hot-deck imputation)

Statistical Data Editing	 4/ 17
Issues for the F-H type optimization-based approach

1	 No closed form of a feasible region 

It is difficult to find all implied edits from balance edits and ratio edits 

(despite marginal solution of Draper and Winkler 1997) 

2	 Risk of the minimum change criterion 

especially when the number of erroneous fields is greeter than the assumed 

minimum number 

3 Using simple imputation methods

The usual imputation methods, such as Hot-deck imputation or regression

imputation, may fail to find a complex, multivariate feature of data

4 Unknown statistical quality

The optimization-based approach does not measure uncertainty introduced by

data editing procedure

∗1 the Nearest-neighbor Imputation Methodology

Statistical Data Editing	 5/ 17
···.············ 

~ 

10 

log(X1) 

. reported 

value 

15 

When is the minimum change criterion harmful?

Want to edit the reported value (blue dot) given the error-free values (black dots)
···.············ 

~ 

············ ---- ----

10 

log(X1) 

. reported 

value 

15 

When is the minimum change criterion harmful?

Supports of the imputed value (blue lines) under the minimum change criterion
--------------------- -~~-- !:: :: ---

.: .. ::· . ~ 

. ; · .... 

- - ~-. .::: : -:. -. -- -. - :: ! -:.-:.·. ·- · -· -· :: -- -- --,; :: •• ••••• 

10 

log(X1) 

. -:· 

.; • .... 

. reported 

value 

15 

When is the minimum change criterion harmful?

Supports of the imputed value without the minimum change criterion
Issues for the F-H type optimization-based approach

1	 No closed form of feasible region 

It is difficult to find all implied edits from balance edits and ratio edits 

(despite marginal solution of Draper and Winkler 1997) 

2	 Risk of the minimum change criterion 

especially when the number of erroneous fields is greeter than the assumed 

minimum number 

3 Using simple imputation methods

The usual imputation methods, such as Hot-deck imputation, may fail to find a

complex, multivariate feature of data

4	 Unknown statistical quality

The optimization-based approach does not measure uncertainty introduced by

data editing procedure

∗1 the Nearest-neighbor Imputation Methodology

Statistical Data Editing	 5/ 17
Contents

1 Statistical Data Editing

2 Bayesian Data Editing for Continuous Microdata

3 Simulation Study

4 Concluding Remarks
Key features of Bayesian data editing approach

1

2

3

4

Modeling the latent structure of reported records by introducing latent variables

for

1 unobserved error-free (true) values

2 unobserved location of errors

Incorporating a priori knowledge for reliability of data fields (if any)

Using nonparametric Bayesian imputation methods

Using multiply imputed values or posterior distributions for inference drawn

from MCMC

Bayesian Data Editing for Continuous Microdata 6/ 17
Error-free Value Model 

Balance 

edits 

Reported 

value 

x; 

Measurement Error Model 

D 

Framework of the Bayesian data editing Model 

Bayesian Data Editing for Continuous Microdata 7/ 17 
� � �

� �

Measurement error model

si = (si1, . . . , sip): latent variables to indicate error location

sij = 1 if field i needs be corrected

sij = 0 otherwise

Model for reported value x ̃i

� 1 � � f(x ̃i|xi, s) = f x ̃i |xi I [ ̃xij = xij ]

{j:sij=0}

1 def x ̃1

i = {x ̃ij : sij = 1, j = 1, . . . , p}

→ f x ̃i

1|xi : (p − sij )-dimensional density for the erroneous values j

2 f x ̃i

1|xi can be any form of probability distribution that models the

measurement error generating process (if any)

Bayesian Data Editing for Continuous Microdata 8/ 17
� 

Error-free value model

xi = (xi1, . . . , xip): latent error-free values for record i with reported values x ̃i

Model for xi with inequality constraints and q balance edits

⎡ ⎤ q

f(xi|θ) = f (xi,C |θ) · I x ⎣ ij = xisl

⎦ · I [xi ∈ X ]

l=1 j∈Cl

1 def xi,C = {xij : j ∈ Cl, l = 1, . . . , q}

→ f(xi,C |θ): (p − q)-dim. density for latent values for reported components

2 I[·] = 1 if the statement is true and I[·] = 0 otherwise

→ Calculate the latent value for reported sum by balance edit

3	 X : the set of convex regions with the inequality constraints (range restrictions

and ratio edits)

→ All latent error-free values must satisfy range restrictions and ratio edits

Bayesian Data Editing for Continuous Microdata	 9/ 17
 

Prior distributions

For error-free values xi for i = 1, . . . , n

use Dirichlet Process Gaussian mixture model

→ to reflect complex joint distributional features based on observed data

with minimum level of a priori distributional assumption

K

f (xi,C |θ) ∝ πkN(xi,C ; μk, Σk)

k=1

πk ∼ DirichletProcess

Bayesian Data Editing for Continuous Microdata 10/ 17
Prior distributions (cont.)

For error location variables si

reflect a priori knowledge about fields’ reliability

ex) If an agency finds that field 1 is twice as reliable as field 2, one may assume

τ1 = 1/3 and τ2 = 2/3 where the prior distribution of si is that

si = (1, 0) with prob. τ1 and si = (0, 1) with prob. τ2

For edit-failing records x ̃ij

assume uniform distribution over the space where an edit is violated 

if there is no available information about error-generating process 

→ to minimize the impact of model misspecification

Bayesian Data Editing for Continuous Microdata 11/ 17
Contents

1

2

3

4

Statistical Data Editing

Bayesian Data Editing for Continuous Microdata

Simulation Study

Concluding Remarks
 

Simulation study

Simulated data

introduce edits for p = 8 fields

range restrictions for each field 

ratio edits for some pairs of fields 

q = 2 balance edits, i.e., xi1 + xi2 + xi3 = xi4 and xi5 + xi6 = xi7 

generate n = 2000 error-free values xi from mixture of three normal dist’n

for 600 out of 2000 records, introduce edit-failing records x ̃i(= xi) which are

uniformly distributed over a compact region where at least an edit is violated

Implemented methods for comparison

1

2

Bayesian editing method 

Bayesian editing method with the minimum change criterion 

by restricting the support of si 

3 F-H based editing process currently used by agencies (not included) 

Simulation Study 12/ 17
q q ~ ~ 

<-i <-i 

'C'. 'C'. 

"!: "!: 

>< >< 

"" "" 0 0 

~ ~ 

"l "l 

q q 

,__; ,__; 

2.0 2.2 2.4 2.6 2.8 3.0 2.0 2.2 2.4 2.6 2.8 3.0 

logx;2 logx;2 

Simulated error-free values xi and reported values x ̃i

Left: Error-free values xi 

Right: Reported values x ̃i when sij = 0 (red dots) or sij = 1 (black dots) 

Simulation Study 13/ 17 
q q ~ ~ 

<-i <-i 

'C'. 'C'. 

"!: "!: 

>< >< 

"" "" 0 0 

~ ~ 

"l "l 

;; ;; 

,__; ,__; 

2.0 2.2 2.4 2.6 2.8 3.0 2.0 2.2 2.4 2.6 2.8 3.0 

logx;2 logx;2 

Result of the Bayesian editing method 

Right: Error-free values xi 

Left: Edited values when sij = 1 (blue) and unchanged values when sij = 0 (red)

Simulation Study 14/ 17 
q q ~ ~ 

:.: .. 

<-i 

'C'. 

"!: 

>< 

"" 0 

~ 

<-i 

'C'. 

"!: 

>< 

"" 0 

~ 

·. <· .. ~:§fiii ; , ..' ::·, :/1l' "l "l 

;; ;; 

,__; ,__; 

2.0 2.2 2.4 2.6 2.8 3.0 2.0 2.2 2.4 2.6 2.8 3.0 

logx;2 logx;2 

Result of Bayesian editing with the minimum change criterion 

Right: Error-free values xi 

Left: Edited values when sij = 1 (blue) and unchanged values when sij = 0 (red)

Simulation Study 15/ 17 
F-H based editing process used by agencies

Some difficulties to implement a real editing process with the simulation data for

comparison purpose

For example, the current editing process of the Census of Manufactures

needs reliability weights for error localization

cannot find a closed form of feasible region with balance edits and ratio edits

use different imputation methods for fields

Simulation Study 16/ 17
Contents

1 Statistical Data Editing

2 Bayesian Data Editing for Continuous Microdata

3 Simulation Study

4 Concluding Remarks
Concluding remarks

The proposed approach replaces the two step optimization-based approach with a

single probability based, data-driven approach in which

1 a stochastic model to identify values plausibly in error unlike the F-H routines is

suggested

reflecting uncertainty over the unknown faulty values when making corrected data

2

3

a flexible joint probability (DP Gaussian) model captures more complex

associations than typical hot deck imputation schemes

imputed values from the model are guaranteed to satisfy all linear constraints

(balance and ratio edits)

Concluding Remarks 16/ 17
Future research

Application to the Census of Manufactures data (in progress)

Study of measurement error models reflecting real error-generating mechanism

Contemplation of the role of edit rules 

Can it be replaced by a statistical outlier model? 

Concluding Remarks 17/ 17
References

Lisa R. Draper and William E. Winkler (1997)

Balancing and Ratio Editing With the New Speer System

Research Report RR97/05, Statistical Research Division, US Bureau of the Census

Ivan P Fellegi and D Holt (1976)

A Systematic Approach to Automatic Edit and Imputation

Journal of the American Statistical Association, 71, 17–35

Hang J. Kim, Jerome P Reiter, Quanli Wang, Lawrence H Cox, and Alan F Karr (2013)

Multiple Imputation of Missing or Faulty Values Under Linear Constraints

Technical Report No. 181, National Institute of Statistical Sciences

United Nations Statistical Commission and Economic Commission for Europe (UNECE,

2000)

Glossary of Terms on Statistical Data Editing

presented at Conference of European Statisticians

Ton De Waal, Jeroen Pannekoek, and Sander Scholtus (2011)

Handbook of Statistical Data Editing and Imputation

Wiley
Thank you! 

Hang J. Kim

hangkim@niss.org

Triangle Census Research Network (TCRN)

http://sites.duke.edu/tcrn/

Duke University and National Institute of Statistical Sciences
�

Appendix 1. Additional practical assumptions for measurement error

model

Let B be an arbitrary support such that x ̃i ∈ B for all records i = 1, . . . , n

Uniform measurement error model

� 1 � � f(x ̃i|xi, si) = Unif x ̃i ∈ B1 I [ ̃xij − xij ]

{j:sij=0}

where

def x ̃i

1 = {x ̃ij : sij = 1, j = 1, . . . , p}

B1: subspace of B on ( j sij )-dimension corresponding to the fields with errors
� � �

� � �

Appendix 1. Additional practical assumptions for measurement error

model (cont.)

Additional practical assumptions for measurement error model

1 When x ̃i satisfies all edit rules,

�p

f(x ̃i|xi, si) =

j=1

δ ( ̃xij − xij ).

2 When all inequality constraints but some balance edits are satisfied,

f(x ̃i | xi, si) = Unif x ̃i

1

; x ̃i ∈ X δ ( ̃xij − xij ).

{j:sij=0}

3 When at least one inequality constraint is violated,

f(x ̃i | xi, si) = Unif x ̃1

i ; x ̃i ∈ X/ δ ( ̃xij − xij ).

{j:sij=0}

Note X ⊂ B