Simultaneous Edit-Imputation for Categorical 

Microdata 

Daniel Manrique–Vallier 

Department of Statistics, Indiana University 

Jerome P. Reiter 

Department of Statistical Science, Duke University 

2013 FCSM Research Conference 

November 6th, 2013 

Research supported by NSF grant SES-11-31897. 
The problem 

Inconsistent Datasets 

Many individual level multivariate datasets, e.g. surveys, 

have consistency requirements specifying combinations of 

responses that are not allowed. 

In real-life, however, datasets often include errors. 

When the errors end up in a violation of a consistency rule, 

we can detect the error. 

When the error doesn’t result in a consistency rule 

violation, the error is not detectable. 

2 / 16 
The problem 

1 

2 

Inconsistent Datasets

Many individual level multivariate datasets, e.g. surveys, 

have consistency requirements specifying combinations of 

responses that are not allowed. 

In real-life, however, datasets often include errors. 

When the errors end up in a violation of a consistency rule, 

we can detect the error. 

When the error doesn’t result in a consistency rule 

violation, the error is not detectable. 

We Want

Detect and locate errors (even if they don’t result in the 

violation of a consistency rule.) 

Impute consistent values, respecting the distribution the 

data, and reflecting the uncertainty associated with the 

procedure. 

2 / 16 
Conceptualizing the Problem 

Data consists of vectors Yi = (Yi1, ...,YiJ ) , i = 1, ..., n (e.g. 

recorded responses to J survey questions) 

Each of the J components take values from a finite set 

Yij ∈ {1, 2, ..., Lj} . 

Entries in Yi might be inconsistent. Then 

Yi ∈ C =

 J

=1

{1, ..., Lj}.

j

Consistency rules are a collection of S s C that specify 

which values of Yi shouldn’t be present in the dataset. 

Connections to structural zeros in contingency tables. 

3 / 16 
A Generative Perspective 

The observed response Yiis a contaminated version of a 

“true” underlying response, Xi. 

Yi

is observed. Xiis unobserved. 

Pr(Yi ∈ S) > 0. Pr(Xi ∈ S) = 0. 

We assume a generation process for Xi 

iid Xi ∼ F,

which doesn’t allow for inconsistent values. Xi ∈ C \ S. 

Yis come from an “error process” 

Yi|Xi ∼ E(Xi).

which allows for inconsistent values. Yi ∈ C. 

4 / 16 
A Generative Perspective 

The observed response Yiis a contaminated version of a 

“true” underlying response, Xi. 

Yi

is observed. Xiis unobserved. 

Pr(Yi ∈ S) > 0. Pr(Xi ∈ S) = 0. 

We assume a generation process for Xi 

iid Xi ∼ F,

which doesn’t allow for inconsistent values. Xi ∈ C \ S. 

Yis come from an “error process” 

Yi|Xi ∼ E(Xi).

which allows for inconsistent values. Yi ∈ C. 

Our objective is to estimate F. 

4 / 16 
Error models 

Given true data, the error process determines what we 

observe. 

We differentiate two components: 

1 

2 

Location model: Which items are in error? 

Substitution model: Given that there’s an error at the (i, j)

location, how does Yij is generated from Xij? 

5 / 16 
Error models 

Given true data, the error process determines what we 

observe. 

We differentiate two components: 

1 

2 

Location model: Which items are in error? 

Substitution model: Given that there’s an error at the (i, j)

location, how does Yij is generated from Xij? 

Let Eij = 1 if there’s an error at the (i, j) location, and 0 

otherwise. We define the error mask

Ei = (Ei1, ...,EiJ ) ∈ {0, 1}

J

. 

5 / 16 
Error models 

Given true data, the error process determines what we 

observe. 

We differentiate two components: 

1 Location model: Which items are in error? 

2 Substitution model: Given that there’s an error at the (i, j) 

location, how does Yij is generated from Xij? 

Let Eij = 1 if there’s an error at the (i, j) location, and 0 

otherwise. We define the error mask

Ei = (Ei1, ...,EiJ ) ∈ {0, 1}

J

. 

The location model is the distribution of Ei. 

The substitution model is the conditional distribution of Yi

given Ei and Xi

(This separation allows to specify a priori which values we 

know are correct or incorrect.) 

5 / 16 
Specifying the Error Model 

Location: Independent Errors Model

indep Eij|Ej ∼ Bernoulli(Ej)

iid

Ej ∼ Beta(aE, bE)

Error locations are independent. 

Each item has its own error rate, Ej

. 

Other specifications possible. 

6 / 16 
�

Specifying the Error Model 

Location: Independent Errors Model

indep Eij|Ej ∼ Bernoulli(Ej)

iid

Ej ∼ Beta(aE, bE)

Error locations are independent. 

Each item has its own error rate, Ej

. 

Other specifications possible. 

Substitution: Uniform Substitution Model

if Eij = 0

Yij|Xij, Eij ∼

δXij

� � Uniform {1, ..., Lj} \ {Xij} if Eij = 1 

6 / 16 
Data Generation Models 

“True Responses” Distribution

Xi ∼ F

In principle it can be any distribution over C \ S. 

In practice we need a flexible enough specification, able to 

capture the nuances of the multivariate structure. 

Challenges: 

Sparsity (very high-dimensional tables with many 

zero-counts). 

Model selection. We want high prediction power. 

Handling of structural zeros! 

We use the Nonparametric Truncated Latent Class Model from 

Manrique-Vallier and Reiter, 2013 (JCGS, to appear) 

7 / 16 
Non Parametric Truncated Latent Class Models 

Truncated mixtures of discrete distributions:

0 J ∞ J

xi

|λ,π ∼ 1{xi ∈/ S} πk λjk(xij )

k=1 j=1 

iid with π = (π1, π2, ...) ∼ DP(α), λjk ∼ Dirichlet(1K ), and 

α ∼ Gamma(aα, bα). 

Very flexible models. 

Method by Manrique-Vallier and Reiter (2013) to obtain 

posterior parameter samples subject to truncated (to C \ S) 

data support. 

Several advantages: Automatic overfitting control. 

Computationally tractable. High tolerance to sparsity. 

Capacity to handle large collections of structural zeros. 

8 / 16 
Test Application - Data Based Simulation 

J = 10 variables from 5% public use microdata from 2000 U.S. 

census (NY) 

Variable Levels (Lj) Variable Levels (Lj) 

Ownership of dwelling 3 Mortgage status 4 

Age 9 Sex 2 

Marital status 6 Race 5 

Education 11 Employment 4 

Work disability 3 Veteran Status 3 

Take N = 953, 076 as a population. Compute statistics. 

Sub-sample n = 1, 000, introduce errors, fix them, and try 

to estimate population quantities back. 

Notes: 

Resulting contingency table has 2, 566, 080 cells. 

|S| = 2, 317, 030 possible inconsistent responses. 

Originally specified as 60 pair-wise rules (e.g. veteran 

toddlers). 

Original data without inconsistencies. 

9 / 16 
�

Test Application - Introducing Errors 

Contaminate the data using independent errors and uniform 

substitution, 

if Eij = 0

Yij|Xij, Eij ∼

δXij

� � Uniform {1, ..., Lj} \ {Xij} if Eij = 1 

iid Eij ∼ Bernoulli(ε)

Try with different error rates ε = 0.1, 0.3, 0.5. 

Pretend that we only observe Y. 

10 / 16 
Prior Specification for Error Model 

We use the independent errors / uniform substitution 

model. 

Need to specify prior distribution for item error rates: 

Ej ∼ Beta(aE, bE)

The method will always detect and correct detectable

errors. 

The prior specification determines how much we trust what 

we observe: 

af/bf = Prior expected rate of error. 

Large af + bf (relative to sample size) puts more weight on 

our beliefs than on the data. 

Small af + bf puts more weight on data. 

For variables that we don’t want to ever alter, we set 

Eij = 0 a priori. This forces Yij = Xij. (can have unintended 

consequences, though) 

11 / 16 
Results (1)- Two-Way margins (ε = 0.1) 

Two-way Margin Proportions

(Estimated vs. Population Values) 

0.0 0.2 0.4 0.6

0.0 0.1 0.2 0.3 0.4 0.5 0.6 0.7

Noisy sample (inconsistent)

Population

Noisy Sample

● ●

●

●

●

●

●

●

●●●

●●

●

●●

●

●●

●

●

●

●

●

●

●●●●

●

●

●

●

●

●

●

●

●

●●●

●●

●

●●

●

●

●

●

●

●

●

●

●

●

●

●●

●●●

●

●●

●●●●

●

●

●

●

●

●

●

●

●

●

●

●●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

● ●

●

●

●

●

●

●

●

●

●

●

●●

●

●

●

●

●

●●

●

●

●

●

●

●

●

●●

●

●

●

●

●●●

●

●

●

●

●

●

●

●

●

●

●

●●

●●●●●●

●●

●●

●

●

●

●●●●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●●

●

●

●

●

●

●

●

●

●

●●

●

●

●●

●

●

●

● ●●●

●

●●●

●

●●

●

●

●

●

●

● ●

●

●●●

●

●●

●●

●

●

●

●

●

●

●●

●

●● ●

●

●

●●●

●●●

●

●

●

●

●

●

●

●●●●

●

●

●

●

●

●

●

●●

●

●

●

●

●

●

●

●●

●●

●●

●

●

●●

●

●

●

●

●

●

●●

●

●

●

●

●

●

●

●●

●●

●●

●●

●

●

●●

●

●

●●●

●●

●●

●●

●●●●

●

●

●

●

●

●

●

●●

● ●

●●

●

●

●

●

●

●

● ●

●

●

●●

●

●

●●●●

●

●

●

●

●

●●●●

●

●

●

●

●●

●●

●

●

●

●

●●

●●●

●

●

●

●

●

●

●

●

●

●●

●

●

●

●

●

●

●

●●●

●

●

●

●

●

●●

●

●

●●

●

●

●

●●

●●

●

●

●●

●

●

●

●

●

●

●●

●

●●

●

●

●

●

●

●

●

●

●

●

●

●

●

●●

●

●

●

●

●

●

●

●

●

●●

●●

●

●●

●

●●

●

●●●●●

●

●

●

●●

●

●●●●

●

●

●

●●

●

●

●

●

●●●

●

●●●●

●

●

●

●

●

●●

●

●

●

●●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

0.0 0.2 0.4 0.6

0.0 0.1 0.2 0.3 0.4 0.5 0.6 0.7

Estimated (strong prior)

a = 1, b = 999

Population

Edited/Imputed Sample

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●●

●

●

●

●

●

●

●

●

●

●

●●

●

●

●●

●

●

●

●

●

●

● ●

●

●●

●●

●

●

●

●

●

●

●●●●●

●

● ●

●

●

●

●

●

●

●

●

●

● ●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●●

●

●

●

●

●

●

●

●

●

●●●

●

●

●

●

●●

●

●

●

●

●

●

●

●

●

●

●

●●●

●

●●●

●

●

●

●

●

●

●

●

●

●●●

●●●●

●

●●

●

●●● ●

●

●

●

●

●

●

●

●

●●

●

●

●

●

●

●

●●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

● ●

●

●

●

●

●●

● ●

●

●●

●

●

●

●

● ●●

●●●●●

●●

●

●

●

●

●

●

●

● ●

● ●

●

●●●●●● ●●

●● ●

●

●

●

●

●

●

●●●

●

●

●●

●

●

●

●●

●●

●●

●●

●●●●

●

●

●

●

●

●

●●

●●

●●

●

●

●

●

●

●

●

●

●

●

●

●

●●●

●●

●

●

●

●

●

●

●

●

●●

●

●

●

●

●

●

●●

●●

●●

●●

●

●

●●

●

●

●●●

●●

●●

●●

●●●●●

●

●●

●●

●

●

●

●

●

●

●

●

●

●

●

●

● ●

●

●

●●

●

●

●●●●

●

●

●

●

●

●●●

●

●

●

●

●●

●●

●

●

●

●

●

●

●●●

●

●●

●

●

●●●

●

●

●

●

●

●●

●●

●

●●

●

●

●

●

●

●

●

●

●●

● ●

●

●●

●●●

●

●

●●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

● ●●

●

●

●

●

●

●

●●

●

●

●

●

●

●

●

●

●●

●●●

●

●

●●

●

●●

●

●●●●

●●

●

●

●

●●●

●●

●●

●

●

●

●

●

●

●

●●

●

●

●●●

●

●●

●

●

●

●

●

●

●●

●

●

●

●

●

●

●

●

●

●

●

●●

●

●

●

●

●

●

●

●

●●

●

●

●

●

0.0 0.2 0.4 0.6

0.0 0.1 0.2 0.3 0.4 0.5 0.6 0.7

Estimated (weak prior)

a = 1, b = 1

Population

Edited/Imputed Sample

●

●

●

●

●

●●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●●●●

●●●

●

●

●

●

●

●

●

●

● ●

●●

●●

●

●

●

●●

●

●●●

●

● ●

●

●

●

●

●

●

●

●

●

●

● ●

●

●

●

●●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●●

●

●

●

●

●●

●

●

●

●

●

●

●

●

●

●

●

●●

●●●

●

●

●

●

●

●

●

●

●

●

● ●

●●●●

●

●●

●●

●

●●

●

●

●

●●●

●

●●●

●●

●

●

●

●

●

●

●●

●●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●●

●

●

●

●

●

● ●

●

●●

● ●

●

●

● ●

●●

●

●

●

●

●

●

●●●

●●●●

●●

●

●

●

●

●

●

●

●

●

●

●

●

● ●

●

●●●●● ● ●●

●●

●

●

●●

●

●

●

●

●

●

●

●●

●

●

●●

●●

●●●

●●

●●●●

●

●

●

●

●

●

●●●●

●●

●

●

●

●

●●

●

●

●

●

●

●

●

●●●●

●●●

●

●

●

●

●

●

●

●●●

●

●

●

●

●

●

●●●

●●

●

●●

●

●

●

●

●●

●

●●

●●

●

●

●●●

●

●

●

● ●

●

●

●

●

●

●

●

●

●

●

●● ●

●

●

●

● ●

●

● ●

●

●

●●

●

●

●

●●●

●

●

●

●

●

●●●

●

●

●

●

●●

●●

●

●

●

●

●

●

●●●

●

●●

●

●

●●●

●

●

●

●

●

●●

●

●

●

●●

●

●

●

●

●

●

●

●

●●

● ●

●

●●

●●●

●

●

●●

●

●

●

●

●

●

●

●

●

●●

●

●

●

●

●

●

● ●

●

●

●

●

●

●

●●

●

●

●

●

●

●

●

●●

●●●

●

●

●●

●

●●

●

●●●●

●●

●

●

●

●●●

●●

●

●

●

●

●

●

●

●

●

●●

●

●

●●●

●

●

●

●●●

●

●

●

●

●

●●

●

●

●

●

●

●

●

●

●

●

●

●●●

●

●

●

●

●

●

●

●

●●

●

● ●

●

Simulation Parameters: 

ε = 0.1, n = 1, 000 

Rows with errors = 626. Detectable errors = 306 

12 / 16 
Results (2)- Two-Way margins (ε = 0.3) 

Two-way Margin Proportions

(Estimated vs. Population Values) 

0.0 0.2 0.4 0.6

0.0 0.1 0.2 0.3 0.4 0.5 0.6 0.7

Noisy sample (inconsistent)

Population

Noisy Sample

l

l

l

l

l

l

l

l

l

l

l

l

l

l

l

l

ll

l

l

l

l

l

l

l

l l

l

l

l

l

l

l l

l

l

ll

l

l

l

l

l

l

l

l

l

l

l

l

l

l l

l

l

l

l

l

l

l

l

l

l

l

l

l l

l

l

l

l ll l

l l

l l l l

l

l

l

l

l

l

l

l

l

l l l

l

l

l

l l

l

l

l

l

l

l

l

l

l

l

ll

l

l

l

l

l

l

l

l

l

l

l

l

l

l l

l

l

l

l

l

l

l l

l

l

l

l

l

l

l

l

l l

l

l

l l l l

ll

l

l

ll

l

l

l

l

l

l

l

l

l

l

l

l

l

l

l

l l l

l

l

l

l l l

l

l

l

l

l

l

l

l

l l

l

l

l l

l

l

ll

l

l

l

l

l

l

ll

l

l

l

l l ll

l

l

ll

l l

l

l l l

l

l

l

l

ll

l

l

l ll

l

l

l

l

l

l

l

l

l

l l

l

l

l

l

l

l

l

l

l

l

l

l

l

l

l

l

l

l

l

l

l

l

l

l

l

l l

l

l

l

l l

l l

l

l

l l l

l

l

l l

l

l

l

l

l

l

l l

l

l

l

l

l l l l l

l l

l l l

l l

l ll l l

l l

l l l

l

l

ll

l

l

l

l

l

l

l

l

l

l l

l l l

l l ll

l

l l

ll

l

l l

l

l

l

l

l ll

l

l l l l

l l

l

l

l

l

l l lll l

l

ll l l l

ll

l

l l l

l lll ll

l

l

ll l

l

l

l

l

l

l l l

l

l

ll

l

l

l

l l

l

l

l

l

l l

l

l

l

l

l l

l

l

l

l

l

l l l

l

l l

l

l

l

l

l

l

l l

l

l

l l

l

l

l

l

l

l

l

l

l l

l

l

l

l

l

ll

ll l

l

l

l

l

l

l

l l

l

l

l

l l ll

l

l

l

l l

ll

l

l l

l

l

l

l

l l

l

l

l

l

l

l

l

ll

l

l

l

l

l l

l

l

l

l

l

l

l

l l l l

l

l

l

l

l

l

l

l

l

l

l

l

l

l

l ll

l

l

l l

l

l

l

l l l

l

l

l

l

l

l

ll

l

l

l

l l

l

l l

l

l l

l

l

l

l l

l

l

l

l

l

l

l

l

l l l

l

l

l

l l

l

l

l

l

ll

l

l

l

l

l

l

l l

l

l

l

l

l

l

l

l

l

l

l l l

l

l

l lll

l

l

l

l

ll

l

l

l

l

l

l

l

l

l

l

l l

l

l

l

l l

l

l

l

l

l

l

l

l

l

l

l

l

l

l

l

l

l

l

l

l

l

l

l

l

l

ll

l

l

l

l

l

l

l

l

l

l

l

l

l

ll l

l

l l

l

l

l

l l l

l l

l

l

l

l

l ll l

l

l

l

l

l l

l

l

l

lll

l

l

l

l

lll

l

l

l l l l

l

l

l

l

l

ll

l

ll

l

l

l

l

l

l ll

l

l

l l

l

l

l

l

l l

l

l

l

l

l

l

l

l

l

l

l

l

l

l

l

l

l

l

l

l

l

l

l

l

l

l

l

l

l

l

l

l

0.0 0.2 0.4 0.6

0.0 0.1 0.2 0.3 0.4 0.5 0.6 0.7

Estimated (strong prior)

a = 1, b = 999

Population

Edited/Imputed Sample

l

l

l

l

l

l

l

l

l

l

l

l

l l

l

l

l

l

l

l l l

l

l

l

llll

l

l

l

l

l

l

l

l

l

l

l

l

l

l

l

l l

l

l l

l

ll

l

l

l

l

l

l

l

l

l

l

l

l

l

l

l

l

l

l

l l l

l

l

l

l

l l

l

l

l

l

l

l l

l

l

l

l

l

l

l

l

l l

l

l

l

l

l

l

l

l

l

l

l l

l

l

l

l

l

l

l

l

l

l

l

l

l

l

l

l

l

l

l

l

l

l

l

l

l

l l

l

l

l l

l l l

l

l l

l

ll

ll

l

l

l

l

l

l

l

l

l

l

l

l

l

l

l l

l

l

l

l l

l

l

l

l

l

l

l

l

l

l

l

l

l l

l l

l

l

l

l

l

l l

l

l

l

l

l

l

l

l

l

l

l

l l l

l

l

l l

l

l

l

l l l

l

l

l

ll

l

l l

l

l l

l l

l

l

l

l

l

l

l

l

l

l

l

l

l

l l l

l

l

l

l

l

l

l

l l

l

l

l

l

l

l

l

l

l

l

l

l

l

l

l

l

l

l

l

l

l l

l l

l

l

l

l

l l

l l

l

l

ll

l

l

ll

l

ll ll

l

l

ll l

l

l

l

l

l l

l

l l

l

l

l

l

l

l

l

l

l

l

l

l

lll

l

l

l

l

l

l

l

lll

l

l

l

l

l

l

l

l l

l

l ll l l l

ll

l

l

ll

l

l

l l

l

l

l

l

l l

ll

l

l

ll l

l l

l

l ll l

l

l

l l

l lll

l

l l l

l

l l

l

l

l

l

l

l

l

ll

l

l

l

l

l

l

l l

l

l

l

l

l

l l

l

l l

l

l

l

l

l

l

l

ll

lll

l

l

l

l

l l

l

l

l

l

l

l

l

l

l

l l

l

l l

l

l

l

l

l

l

l

ll

l l l

l

l

l

l

l

l

l l

l

l

l

l ll

l

l

l

l l

ll

l

l l

l

l

ll

l

l

l

l

l

ll

l

l

l

l

l

l

l

l

l l

l

l

l

l

l

l

l

l l l l

l

l

l

l

l

l

l

l

l

l

l

l

l

l

l ll

l

l

l

l

ll l l

l

l

l

l

l

l

ll

l

l

l

l l

l

l l

l

l l

l l

l

l

l

l

l

l

l

l

l

l l l

l

l l l

l

l

l

l

ll

l

l

ll

l

l

l

l

l

l

l

l

l

l

l

l

l

l

l

l

l

l

lll

l

l

l

l

ll

l

l

l

l

l

l

l

l

l

l l

l

l

l

l l l

l

l ll

l

l

l

l

l

l

l

l

l

l

l

l l

l ll

l

l

l

l

l

l

l

ll

l

l

l ll

l

l

l

l

l

l

l

l

l ll ll l l

l

ll

l

l

l l l

l

l

l

l

ll l l l

l

l

l

l

l l

l ll ll l l

ll l

ll

l

l

l l l

l

ll

ll

l

l

l l l l l l l

l

ll l l

l

l

l

l l

l

l

l

l

l

l

ll

l

l l l

l

l

l

l

l

l

l

l

l

l

l

l

l

l

l

l

l

l

l l

l

l

l

l

l

l

0.0 0.2 0.4 0.6

0.0 0.1 0.2 0.3 0.4 0.5 0.6 0.7

Estimated (weak prior)

a = 1, b = 1

Population

Edited/Imputed Sample

l

l

l

l

l

l

l

l

l

l

l

l

l l

l

l l

l

l

l

l l

l

l

l

llll

l

l

l

l

l

l

l

l l

l

l

l

l

l

l

l

l l

l

l

l

l

l

l

l

l

l

l

l

l

l

l

l

l

l

l l

l

l

l

l

l l l

l

l l

l

l

l

l

l

l

l

l

l

l l

l

l l

l

l l

l

l l

l l

l

l

l

l

l

l

l

l

l

l

l l

l

l

l

l

l

l

l

l

l

l

l

l

l

l

l

l

l l

l

l

l l

l

l

l

l

l

l

l

l

l l l l

l

l l

l

ll

l

ll

l

l

l

l

l

l

l

l

l

l

l

l

l

l l

l

l

l

l l

l

l

l

l l

l

l

l

l

l

l

l

l

l

l l

l

l

l

l

l

l l

l

l

l

l l

l

l

l

l l l

ll

ll

l ll

l

l

l

l

l

l

l

l

ll l

l

l l l

l l l

l l

l

l

l

l

l

ll

l

l

l

l

l l

l l l

l

l

l

l

l

l

l

l

l

l

l

l

l

l

l

l

l

l

l

l

l

l

l l

l

l

l

l

l

l

l

l l

l

l

l

l

l l

l l

l

l

lll

l

l

ll

l

l

l

ll ll

l

l

l l

l

l

l

l

l

l l l

l

l

l

l

l

l

l

l

l

l

l

l

l

ll

l

l

l

l

l

l

l

ll

l

l

l

l

l

l

l

l l l

l

l

l l

l

l l l ll

l

l

lll

l

l

l

l

l

l

l

l l

ll

l

ll l

l l

l

l

l lll

l

l

l l l ll ll

l l l

l

l l

l

l

l

l

l

l

l

l

l

ll

l

l

l

l

l

l l

l

l

l

l

l l

l

l l

l

l

l

l

l

l

l

l

l

lll

l

l

l

l

l l

l

l

l

l

l

l

l

l

l

l

l

l

l l

l

l

l

l

l

l

ll

l

l l l l

l l

l

l

l l

l

ll l

ll

l l

l

l

l

l l l l l

l

l l

l

l

l

l

l

l

l

l

l

l

ll

l

l

l

l

l l

l

l

l

l

l

l

l

l

l

l

l

l

l

l

l

l

l

l

l

l

l

l

l

l

l

l

ll l l

l

l

l

l

l

l

ll

l

l

l l

l

l l

l

l l l

l

l

l

l

l

l

l

l

l

l

ll l l

l

l l

l

l

l

l

l

l

l

l

l

l

l

l

l l

l

l

ll

l

l

l

l

ll

l l

l ll

l

l

l

l

l l

l

l

l

l

l

l

l

l

l l

l

l

l

l

l

ll l

l

l

l

l

l

l

l

l

l

l

l l

ll l

l

l

l

l

l

l

l

l

l

ll l

l

l

l

l

l

l

l

l

l

ll l ll l l

l

l

l

l

l

l

l l l

l

l

l

l

ll l l ll

l

l

l l l

l ll l ll l l

ll l

ll

l

l

l l

l

l

l

ll

l

l

l

l l l l l l l

l

ll l l

l

l

l

l

l

l

l

l

l

l

ll

l

l l l

l

l

l

l

l

l

l

l l

ll

l

l

l

l

l

l

l

l l

l

l

l

l

l

l

Simulation Parameters: 

ε = 0.3, n = 1, 000 

Rows with errors = 980. Detectable errors = 685 

13 / 16 
Results (3)- Two-Way margins (ε = 0.5) 

Two-way Margin Proportions

(Estimated vs. Population Values) 

0.0 0.2 0.4 0.6

0.0 0.1 0.2 0.3 0.4 0.5 0.6 0.7

Noisy sample (inconsistent)

Population

Noisy Sample

l

l

l

l

l l

l

l

l

l

l

l

l

l

l

l

l

l

l

l

l

l

l

l l

l l l

l

l

l

l

ll

l

l l

l

l

l

l

l l

l

ll

l l l l

l l

l

l

l

l

l

l

l

l

l

l

l

l l

l

l

l

l

l

l

l

l l

l

l

l

l

l l

l

l

l

l

l l l

l

l

l l l l l l

l

l l

l

l

l

l

l

l

l

l l

l

l l

l

l

l

l

l

l

l

l

l

l

l l

l

l

l

l

l l l

l l l

l

l

l ll l

l

l

l

l

l

l l l ll

l

l

l

l

l

l

l

l

l

l

l

l

l

l

l

l

l l

l l

l

l

l

l

l

l

l

l

l

l

l

l

l

l

l

l l l l

l

l

l

l

l

l

l

l

l

l

l l l l

ll

l

l

l l l ll

l

l l l

l l

l l

l

ll ll l

l

l ll

l

l

l

l

l

l l

l

l

l

l

l

l

l

l

l

l

l

l

l

l

l

l

l

l

l

l

l

l

l

l

l

l

l

l

ll

l

l l

l

l l

l

l

l

l l l

ll

l

l

l

l l ll

l l

l

l

l

l

l

l

l

l

ll l

l

l l ll

l

l

l l

l l l

l

l

l l

l

ll

ll

l l l

ll l

l

l l

ll l

l

l

lll

ll

l

l

l l

l

l

l l

l

l

l

l

l l ll l l l l

l l l

l l l

ll lll ll l lll lll l

l lll l

l

l

l

l

l

l l

l

l l

l l

ll

ll

l

l

l l

l

l

l l l l l

l l

l

l

l

l l

l

l

l

l l

l l

l l

ll

l l

l l l

l l l

l

l

l

l

l l

l l

l l l

l

ll

ll

l l

l l

l

l

l

l

l

l

ll

l

l l l

l

l

l

l

l l

l

l

l

l

l

l

l

l l l

l

l

l

l

l

l

l

l ll

l l

l

ll

l

l

l

l

l

l

l

l

l

l

l

l

l

l

l

l

l

l

l

l

l l

l

l

l

l

l l

l

l

l

l l

l

l

l

l

l

l

l

l

l l

l

l

l

l

l ll

l l

l l

l

l

l

l

l l

l

l

l

l l l

l l l

l

l

l

l

l

l

l

l

l

l

l

l

l

l

l

l

l ll l

l l

l

l

l l

l

l

l

l

l

l

l

l

l

l

l

l

l

l

l

l

l

l

l

l

l

ll

l

l

l ll

l

l

l

l l

l l

l

l

l l

l

l l

l l

l lll

l

l

l

lll

l

l

l

l

l ll

l

l

l l l

l

l l l

l

l

l

l l

l

l l l

l

l

l

l

l

l l l

l

l

l l

l

l

l

l

l

l

l l l l

l l l

l

l

l

l l l l

l

l

l

l l l

l

l

l

l ll

l

ll

l

l ll

l l l

ll

l

l

ll

llll

l

l l l l

ll

l

lll l l l

l ll

l

l

l

l

ll l

ll

l

l

l l l l

l

l

l

ll

l

l l

l

l

l

l

l

l

l

l

l

l l l

l

l

l

l

l

l

l

l

l

l

l l

l

l

l

l

l

l

l

0.0 0.2 0.4 0.6

0.0 0.1 0.2 0.3 0.4 0.5 0.6 0.7

Estimated (strong prior)

a = 1, b = 999

Population

Edited/Imputed Sample

l

l

l

l

l

l

l

l

l

l l

l

l

l l

l

l l l

l

l

l

lll

l

l

ll

l

l

l

l l

l

l

l

l

l

l

l

l

l

l

l

l

l l

l

l

l

l

l

l

l

l

l

l

l l

l

l

l l

l

l

l

l l

l

l l

l

l l

l

l

ll

l

l

l

l

l l

l

l l

l

l

l

l

l l l

l

l

l

l

l

l

l

l

l

l l l

l

l

l

l

l

l

l

l

l

l

l

l

l

l

l

l

l l

l

l

l l

l

l l

l

l l

l l l l

l

l l l

ll

l

ll

l

l

l

l l

l

l

l l

l

l

l l

l l l

ll

l

l

l l

l

l

l l

l

l

l l

l l

l

l l

l

l

l

l l

l

l

l

l

l

l l l

l

l

l

l

l l l

lll

l

l

l l

l l

l l l

l

l l

l

l l

l

l l

l l l l

l

l l

l l

ll

l

l

l

l

l l l l

l

l l

l

l

l l

l

l

l l l

l

l l

l

l

l

l

l

l

l l

l

l

l

l

l

l

l l

l

l

l

l

l

l

l l l l

l

l

l

lll

l

l

ll

l

l

l

l

ll ll

l

l

l

l

l l

l

l

l

l

l l l

l

l

l

l

l

l

l l

l

l

l

l

l

l

l

ll

l

l

l

l

l

l

l

l

l

l

ll

l

l

l

l

l

l

l

l

l

l l l

l

l l

l

l

l

l

ll

l

l

lll

l

l

l

l

l l l l

l

l

l

l l

ll l

l

lll ll l

l

l

l lll

l

l

l l

l lll

l

l

l l l l

l

l

l l

l

l

l

ll

l

l

l

l

l

l l l

l

l

l

l l l l

l

l

l l

l

l

l

l

l

l l

l

l

l

l

l

l l l l

l

l

l

l l

l

l

l

l

ll l

l

l

l

l

l

l

l

ll

l

l l l

l

l

l

l

l l

l

l

l

l

l

l

l

l

l

l

l

l

l

l

l

l

l ll

l l l

ll

l

l

l

l

l

ll

l

l

l

l

ll

l

l

l

l

l

l

l l

l

l

l

l

l l

l

l

l

l l

l

l

l

l

l

l

l

l

l

l

l

l

l

l

ll

l l l

l

l

l

l

l

l

l

l

l l l

l l l

l

l

l

l l l l

l

l

l

l

l

l

l

l

l

l

ll

l l

l

l l l l l

l

l

l

l

l

l

ll

l

l

l

l

l

l

ll

l

l

l

ll

l

l

l

l

l ll

l

l

l

ll

l l

l

l

l

l

l l

l ll

l

l

l

l

ll

l

l

l l ll l

l

l

l

l l

l

l

l

l

l

l

l

l l

l ll l

l

l

l

l

l

l

l l

ll

l l ll

l

l

l

l

l

l

l

l

l

l l ll l ll

l

ll

ll

l

l

l l l

ll

l

l ll l ll

l

llll

l l ll l ll

l

l l

ll

l

l

l l l l

ll

l

l

l

l

l l l l ll

l

l l

ll l

ll

l

l

l l

l l

l

l

l

ll

l

l

l l l

l

l

l

l

l

l

l

l

l

ll

l

l

l

l

l

l

l

l

l

l

l

l

l l

l

0.0 0.2 0.4 0.6

0.0 0.1 0.2 0.3 0.4 0.5 0.6 0.7

Estimated (weak prior)

a = 1, b = 1

Population

Edited/Imputed Sample

l

l

l

l

l

l

ll

l

l l

l

l

l

l l

l l l l

l

l

l

lll

l

l

l

l

l

l

l

l

l

l

l

l

l

l

l

l

l

l ll

l

l

l l

l

l

l

l

l

l

l

l

l

l

l l

l

l

l

l

l

l

l

l

l

l l

l

l

l

l l l

l

l

l

l

l

l l

l

l

l

l

l

l

l

l

l

l

l l l l

l

l

l

l

l

l

l

l

l

l l l

l

l

l

l

l

l l

l

l

l

l

l

l

l

l

l

l

l

l l l

l l

l l

l

l

l

l

l l l l

l

l

l l

ll

l

ll

l

l

l

l

l

l

l

l

l

l

l

l

l

l

l

l

l l ll l

l l

l l

l l

l

l

l

l

l

l

l

l l l

l l

l

l

l

l l

l

l

l l

l

l l

l l

l l

l l l l l

l l

l

l

l

l l l

l l

l

l

l

l

l l

ll

l

l l l l

l

l

l

l l

l

ll

l

l

l l

l l l l

l

l

l

l

l

l

l

l l l

l l

l

l

l

l

l

l

l

l

l

l l l

l

l

l

l

l

l

l

l l

l

l

l

l

l

l

l l l

l

l

l

l

l

ll l

l

l ll

l

l

l

l l l

l

l

l l

l

l

l

l

l

l

l

l l

l

l

l

l

l

l

ll

l

l

l

l

l

l

l

l

l

l

l

l

l

l

l

l

ll

l

l

l

l

l

l

l

l

l

ll l l

l

l

l l

ll

l

l

l l l

l

l

l

l

l

ll l

l

l

l

l

lll l

l

l

l

lll

l

l

l l

l

lll l

l

l l l l

l

l

l l

l

l

l

l

l

ll

l

l

l

l

l

l l l

l

l

l

l

l l l l

l

l

l l

l

l

l

l

l

lll

l

l

l

l

l l l l

l

l

l

l l

l

l

l

l

ll l

l

l

l

l

l

l

l

l

l

l

l

l

l

l

l

l

l

l

l

l l

l

l

l

l

l

l

l

l

l

l

l

l

l

l

l

l

l

l

l

l

l

l

ll

l

l

l

l

l

l

ll

l

l

l

l

l

l

l

l

l

l

l

l

l

l

l

l

l

l

l

l

l

l

l

l

l

l

l l

l

l

l

l

l l

l

l

l

l

l ll

l

l l

l

l

l

l

l

l

l

l

l

l

l

l

l

l

l

l

l

l

l

l

l

l

l

l

l

l

l

l

l

l

l

ll

l

l

l

l l

l

l

l

l

l

l

l

l

l

l

l

l

l

l

l

l

l

l

l

l

l

l

l

l

l

l

l l

l

l

l

l l

l

l

l

l

l l

l

l l

l

l

l

lll

l

l l ll l

l

l

l

l l

l

l

l

l

l

l l l

l ll l

l

l

l

l

l

l

l

l

ll

l

l ll

l

l

l

l

l

l

l

l

l

l l ll l ll

l

ll

l

l

l

l l

l

l

l

l

l

l

l l ll

l

l l

ll

l l ll l ll

l

l l

l

l

l

l l

l

l

l

ll

l

l

l

ll

l

l l l l ll

l

l l

l

l l

l

ll

l

l l

l

l

l

l

l

l

l

ll

l

l

l l l

l

l

l

l

l

l

l

l l l

l

l

l

l

l

l

l

l

l

l

l

l

l l

l

Simulation Parameters: 

ε = 0.5, n = 1, 000 

Rows with errors = 999. Detectable errors = 833 

14 / 16 
Concluding Remarks 

Full Bayesian model-based approach to edit-imputation. 

Integrates data generation with measurement error. 

Automatic over-fitting protection. 

Edit and imputation based on joint distribution. Respects 

data distribution. 

Does not require full analysis of consistency rules. 

Guaranteed to generate consistent imputations. 

Computationally feasible, but can be demanding in tough 

problems. (runtime example = 1.6 min) 

Prior specification matters: 

Strong prior w/low error rate. 

Weak prior. 

Open issue: Which values do we really want to change? 

(prior for Ej and which Eij set to 0 a priori) 

15 / 16 
The End 

(Thanks!) 

For details about truncated latent structure models: 

http://mypage.iu.edu/ ̃dmanriqu/papers/lcm_zeros.pdf

For multiple imputation see: 

http://mypage.iu.edu/ ̃dmanriqu/papers/LCM_Zeros_

Imputation.pdf