A Latent Class Modeling Approach for

Differentially Private Synthetic Data for

Contingency Tables

Andres Felipe Barrientos  ́

Assistant Professor

Department of Statistics

Florida State University

Joint work with Michelle P. Nixon, Aleksandra Slavkovic, and Jerry P. Reiter.  ́

2021 FCSM conference

November 2021
Outline

1 Data privacy

2 Differentially private modeling approach

3 Illustrations with 2016 ACS data

4 Concluding remarks
Privacy and data sharing

◮ Agencies and companies often seek to share their data.

◮ Protection of individuals’ private information is a must.

◮ Traditional strategies: disclosure control methods

[Hundepool et al., 2012] or releasing synthetic data [Rubin,

1993].

◮ In recent years, agencies are looking for methods that

provide formally quantifiable privacy guarantees, e.g.,

those that rely on differential privacy.
Problem setup

◮ Confidential dataset X =



Xi = (X1i, . . . , Xpi)

	n

i=1

, where

Xij is categorical.

◮ Assume that the agency is willing to release summaries of

X denoted by M(X) = (M1(X), . . . , MT (X)).

◮ The goal is to generate a synthetic version of X using

M(X) and a formally private mechanism.
Illustration with ACS PUMS

◮ We selected a subset of 10,000 individuals from the 2016

one-year ACS PUMS.

◮ Each Mt(X), t = 1, . . . , 10, denotes a two-way marginal

table.

Age

Citizenship 0 1

0 11 596

1 443 8950

Race

Citizenship 0 1

0 299 308

1 1731 7662

Sex

Citizenship 0 1

0 273 334

1 4505 4888

Income

Citizenship 0 1

0 294 313

1 2916 6477

Race

Age 0 1

0 110 344

1 1920 7626

Sex

Age 0 1

0 239 215

1 4539 5007

Income

Age 0 1

0 445 9

1 2765 6781

Sex

Race 0 1

0 945 1085

1 3833 4137

Income

Race 0 1

0 827 1203

1 2382 5587

Income

Sex 0 1

0 1281 3497

1 1929 3293
Differential privacy

◮ Differential privacy is the best known formal privacy

framework in use.

◮ M(X) is a randomized version of M(X).

Definition

ǫ-Differential Privacy [Dwork et al, 2006]: A randomized

mechanism M satisfies ǫ-differential privacy if for all data sets

X and X

′ differing on at most one row, and S ⊆ Range(M),

Pr[M(X) ∈ S|X]

Pr[M(X′) ∈ S|X′]

≤ exp(ǫ).
Differentially private summary statistics

◮ M(X) = (M1(X), . . . ,MT (X)) is a randomized version of

M(X) = (M1(X), . . . , MT (X)).

Theorem

Geometric Mechanism [Ghosh et. al, 2012]: For

Mt(X) : D → Z

dt

, the mechanism Mtthat adds independently

drawn noise from a two-sided-Geom(exp{

−ǫt

∆Mt

}) distribution to

each of the dtterms of Mt(X) satisfies ǫt-differential privacy.

◮ Sensitivity ∆Mt = supX,X′ kMt(X) − Mt(X

′

)k1.
Illustration with ACS PUMS

◮ Sequential composition [Mcsherry, 2009]: If each Mt

provides ǫt-differential privacy. The sequence of

M(X) = (M1(X), . . . ,MT (X)) provides

(ǫ =

P

t

ǫt)-differential privacy. We can use ǫt = ǫ/T.

Age

Citizenship 0 1

0 11 596

1 443 8950

Race

Citizenship 0 1

0 299 308

1 1731 7662

Sex

Citizenship 0 1

0 273 334

1 4505 4888

Income

Citizenship 0 1

0 294 313

1 2916 6477

Race

Age 0 1

0 110 344

1 1920 7626

Sex

Age 0 1

0 239 215

1 4539 5007

Income

Age 0 1

0 445 9

1 2765 6781

Sex

Race 0 1

0 945 1085

1 3833 4137

Income

Race 0 1

0 827 1203

1 2382 5587

Income

Sex 0 1

0 1281 3497

1 1929 3293
Bayesian modeling approach

◮ The released summary statistic is of the form

M(X) = (M1(X) + ε1, . . . , MT (X) + εT ).

◮ Some counts based on M(X) will not necessary match.

◮ Ideal modeling approach:

Mt(X)|Mt(X)

ind∼ two-sided-Geomdt



Mt(X), exp 

−ǫ

∆MtT

 ,

M(X) = (M1(X), . . . , MT (X))|θ ∼ pM (·|θ),

θ ∼ pθ.

◮ It is not easy to characterize pM (·|θ).

◮ We know that Mt(X)|θ ∼ Multinomialrt

(n, Pt(θ)).
Bayesian modeling approach using composite

likelihood methods

◮ Proposed modeling approach:

Mt(X)|Mt(X)

ind∼ two-sided-Geomdt



Mt(X), exp 

−ǫ

∆MtT

 ,

Mt(X)|θ

ind∼ Multinomialdt

(n, Pt(θ)), t = 1, . . . , T,

θ ∼ pθ.

◮ Notice that the probabilities P1(θ), . . . , PT (θ) are related.

◮ We can define Pt(θ) by specifying a model for X|θ.
Illustration with ACS PUMS

M1(X) =

Age

Citizenship 0 1

0 11 596

1 443 8950

P1(θ) = 

p1,(0,0) p1,(0,1)

p1,(1,0) p1,(1,1)



M2(X) =

Race

Citizenship 0 1

0 299 308

1 1731 7662

P2(θ) = 

p2,(0,0) p2,(0,1)

p2,(1,0) p2,(1,1)



◮ Coherence: p1,(1,0) + p1,(1,1) = p2,(1,0) + p2,(1,1)

◮ We define Pt(θ) by specifying a model for X|θ.
Modeling X|θ

◮ We use the following mixture model [Dunson and Xing 2009]:

Xij|zi, {Ψ

(j)

h

}

∞

h=1

ind∼ Multinomial{1, Ψ

(j)

zi1

, . . . , Ψ

(j)

zidj

},

zi|{πh}

∞

h=1

ind∼ Discrete{(1, . . . , ∞),(π1, . . . , π∞)},

πh = Vh

Y

l<h

(1 − Vl), Vh ∼ β(1, α),

Ψ

(j)

h ∼ Dirichlet(aj1, . . . , ajdj

),

where θ =



πk = {πh}

k

h=1

, Ψk = {Ψ

(j)

h

}

k,p

h=1,j=1



.
Defining P1(θ), . . . , PT (θ)

◮ If M1(X) is the contingency table of the first two variables,

then

P1(θ) = 

p1,(0,0) p1,(0,1)

p1,(1,0) p1,(1,1)



where, e.g.,

p1,(0,0) = Pr (X·1 = 0, X·2 = 0|θ) = Xk

h=1

πhΨ

(1)

h0 Ψ

(2)

h0

X1

i=0

X1

j=0

X1

l=0

Ψ

(3)

hi Ψ

(4)

hj Ψ

(5)

hk .
Bayesian modeling approach and inference

◮ Proposed approach:

Mt(X)|Mt(X)

ind∼ two-sided-Geomdt



Mt(X), exp 

−ǫ

∆MtT

 ,

Mt(X)|θ

ind∼ Multinomialdt

(n, Pt(θ)), t = 1, . . . , T,

θ ∼ pθ.

◮ We use MCMC algorithms to sample from θ|M(X).

◮ Inferences are performed using (P1(θ), . . . , PT (θ))|M(X).
Bayesian modeling approach and inference

◮ Instead of using M(X)|M(X), we use M(X

S)|M(X).

◮ To make inferences about the confidential summary, we

use

Pr(X(n+1)1 = c1, . . . , X(n+1)p = cp|M(X)) =

Z

Pr(X(n+1)1 = c1, . . . , X(n+1)p = cp|θ)Pr(θ|M(X))dθ

to generate synthetic datasets X

S and induce a

distribution via X

S 7→ M(XS).
Illustrations with ACS PUMS

◮ We selected a subset of 10,000 individuals from the 2016

one-year ACS PUMS.

◮ Each Mt(X), t = 1, . . . , 10, denotes a two-way marginal

table.

Age

Citizenship 0 1

0 11 596

1 443 8950

Race

Citizenship 0 1

0 299 308

1 1731 7662

Sex

Citizenship 0 1

0 273 334

1 4505 4888

Income

Citizenship 0 1

0 294 313

1 2916 6477

Race

Age 0 1

0 110 344

1 1920 7626

Sex

Age 0 1

0 239 215

1 4539 5007

Income

Age 0 1

0 445 9

1 2765 6781

Sex

Race 0 1

0 945 1085

1 3833 4137

Income

Race 0 1

0 827 1203

1 2382 5587

Income

Sex 0 1

0 1281 3497

1 1929 3293
Illustrations with ACS PUMS

◮ True versus estimated two-way marginal tables.

●

●

●

●

●

●

●

●

●●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●●

●

●

●

●

●

●

●

●

●

●●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

0.00

0.25

0.50

0.75

0.00 0.25 0.50 0.75

True Probability

Estimated Probability

Epsilon

●

●

●

0.25

0.5

1
Illustrations with ACS PUMS

◮ True versus estimated full table.

●

●●

●

● ●

●

● ●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

0.0

0.1

0.2

0.0 0.1 0.2

True Probability

Estimated Probability

Epsilon

●

●

●

0.25

0.5

1
Comparisons with existing methods

◮ True versus estimated full table (ǫ = 0.5).

● ●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●●

● ●

● ●

●

●

●

●

●

●

●

●

●

●

●

●

0.0

0.1

0.2

0.0 0.1 0.2

True Probability

Estimated Probability

Method

●

●

●

GM

PrivBayes

Proposed Method
Concluding remarks

◮ We present a novel method to create differentially private

synthetic data for contingency tables based on marginal

counts.

◮ The simulation results indicate that our approach

preserves the summaries.

◮ The proposed approach is complementary to existing

releasing mechanisms.

◮ Our general strategy can be extended to more complex

data structures.

Thank you!