Data Deidentification Research and 

Resources from the NIST Collaborative 

Research Cycle

Gary Howarth, NIST

Christine Task, Knexus Research

with content from

Karan Bhagat, Knexus Research

Dhruv Kapur, University of Michigan 
Portions of this talk are presented by a guest speaker. The contents of this 

presentation do not necessarily reflect the views or policies of the National Institute 

of Standards and Technology or the U.S. Government. 

Certain commercial entities, equipment, or materials may be identified in this 

document in order to describe an experimental procedure or concept adequately. 

Such identification is not intended to imply recommendation or endorsement by 

the National Institute of Standards and Technology, nor is it intended to imply that 

the entities, materials, or equipment are necessarily the best available for the 

purpose. 

Please note, unless mentioned in reference to a NIST Publication, all information 

and data presented is preliminary/in-progress and subject to change.

Disclaimer
Measurements essential 

to commerce, trade, 

and innovation

Federal role 

established in the 

U.S. Constitution

National Institute of 

Standards and Technology
Why does NIST research privacy?

NIST seeks to facilitate organizations and individuals deriving benefits from data while 

simultaneously encouraging effective management of risks to individuals’ privacy. 

NIST seeks to be the world's leader in creating critical measurement solutions and 

promoting equitable standards.

NIST is a trusted leader in metrology and provides independent, and transparent 

technical guidance for the benefit of all. 
The NIST Collaborative Research Cycle Program

This talk focuses on a NIST metrology program for data 

deidnetification techniques. 

Deidentification includes any processing to microdata 

that produces microdata in the same schema and is 

intended to be resistant to individual reidentification: 

SDC, synthetic data, differential privacy. 

NIST Collaborative Research Cycle

https://pages.nist.gov/privacy_col

laborative_research_cycle/

This past year we’ve launched a massive community benchmarking and 

meta-analysis project, collecting metrics, algorithms and data samples from 

stakeholders, researchers and statistical agencies around the world— and 

making them all freely available and easy to use. We’ll give you a tour, and 

you can check the QR code to access it all yourselves. 
NIST Collaborative Research Cycle: Far more than four algorithms 

https://pages.nist.gov/privacy_

collaborative_research_cycle/
Differential Private Histogram (ε = 10) PATECTGAN Differential Private GAN (ε = 10)

CART-model Synthesis (non-DP synthetic) Cell Suppression (k = 6)

Algorithms: A Sample of Four Deidentification Approaches
Differential Private Histogram (ε = 10) PATECTGAN Differential Private GAN (ε = 10)

CART-model Synthesis (non-DP synthetic) Cell Suppression (k = 6)

Algorithms: A Sample of Four Deidentification Approaches

DP GAN: Add randomized 

noise while training an ML 

model to reproduce the 

distribution. 
Differential Private Histogram (ε = 10) PATECTGAN Differential Private GAN (ε = 10)

CART-model Synthesis (non-DP synthetic) Cell Suppression (k = 6)

Algorithms: A Sample of Four Deidentification Approaches

DP GAN: Add randomized 

noise while training an ML 

model to reproduce the 

distribution. 

CART: Use a sequence of decision trees to generate 

new values for every feature, one at a time. 
Differential Private Histogram (ε = 10) PATECTGAN Differential Private GAN (ε = 10)

CART-model Synthesis (non-DP synthetic) Cell Suppression (k = 6)

Algorithms: A Sample of Four Deidentification Approaches

DP GAN: Add randomized 

noise while training an ML 

model to reproduce the 

distribution. 

CART: Use a sequence of decision trees to generate 

new values for every feature, one at a time. 
The NIST Collaborative Research Cycle Program

Data: Diverse Community Excerpts Benchmark Data

https://github.com/usnistgov/SDNist/tree/

main/nist%20diverse%20communities%2

0data%20excerpts

Excerpts of 2019 American Community Survey Data

Tractable schema size for research: 22 Data Features + Weights 

Curated to focus on geographies (PUMA) with challenging distributions 
The NIST Collaborative Research Cycle Program

Data: Diverse Community Excerpts Benchmark Data

https://github.com/usnistgov/SDNist/tree/

main/nist%20diverse%20communities%2

0data%20excerpts

Excerpts of 2019 American Community Survey Data

Tractable schema size for research: 22 Data Features + Weights 

Curated to focus on geographies (PUMA) with challenging distributions

Recommended Feature Subsets provided for small schema approaches

The next few slides will use the “Demographic-focused” Feature Subset 
Metrics: Univariate 

Differential Private Histogram (ε = 10) PATECTGAN Differential Private GAN (ε = 10)

CART-model Synthesis (non-DP synthetic) Cell Suppression (k = 6)
Differential Private Histogram (ε = 10)

CART-model Synthesis (non-DP synthetic) Cell Suppression (k = 6)

Metrics: Pairwise Correlations

PATECTGAN Differential Private GAN (ε = 10)

Pairwise Correlations: A key 

goal of deidentified data is to 

preserve the feature 

correlations from the target 

data, so that analyses 

performed on the deidentified 

data provide meaningful insight 

about the target population. 

Which correlations are the 

deidentified data preserving, 

and which are being altered?

The Pearson Correlation

difference was a popular utility 

metric during the HLG-MOS 

Synthetic Data Test Drive. Note 

that darker highlighting 

indicates pairs of features 

whose correlations were not 

well preserved by the 

deidentified data.
Differential Private Histogram (ε = 10)

CART-model Synthesis (non-DP synthetic) Cell Suppression (k = 6)

Metrics: K-Marginal Similarity 

PATECTGAN Differential Private GAN (ε = 10)

977

Equivalent to a 20%

uniform random 

subsample of the input 

target data. 

K-marginal Similarity: checks 

how far the shape of the 

deidentified data distribution 

has shifted away from the 

target data distribution, using 

many 3-dimensional snapshots 

of the data, averaging the 

density differences across all 

snapshots. It was developed as 

an efficient scoring mechanism 

for the NIST Temporal Data 

Challenges, and can be applied 

to measure the distance 

between any two data 

distributions. A score of 0 

means zero overlap, while a 

score of 1000 means the two 

distributions match identically. 

More information can be found 

here.

998

Equivalent to a 90%

uniform random 

subsample of the input 

target data. 

984

Equivalent to a 40%

uniform random 

subsample of the input 

target data. 

805

Less than a 1% uniform 

random subsample of the 

input target data. 
Differential Private Histogram (ε = 10)

CART-model Synthesis (non-DP synthetic) Cell Suppression (k = 6)

Metrics: Propensity 

PATECTGAN Differential Private GAN (ε = 10)

Propensity Metrics: 

Can a decision tree classifier tell 

the difference between the 

target data and the deidentified 

data? If a classifier is trained to 

distinguish between the two 

data sets and it performs poorly 

on the task, then the 

deidentified data must not be 

easy to distinguish from the 

target data. If the green line 

matches the blue line, then the 

deidentified data is high quality. 

Propensity based metrics have 

been developed by Joshua 

Snoke and Gillian Raab and 

Claire Bowen
Differential Private Histogram (ε = 10)

CART-model Synthesis (non-DP synthetic) Cell Suppression (k = 6)

Metrics: Pairwise PCA 

PATECTGAN Differential Private GAN (ε = 10)

PCA Metric visually compares a synthetic 

data set with the original input data. It 

plots high dimensional data as a 2D 

scatterplot using the first two principal 

component axes; each point represents 

an individual in the data. Good synthetic 

data should recreate the shape of the 

original data with new points (new 

synthetic individuals). The plot above 

shows the shape of the original sensitive 

data; the synthetic data generators are 

trying to reproduce this distribution. To 

display more detail, we’ve used red

points to highlight records that 

represent children (MSP value = ‘N’)
Differential Private Histogram (ε = 10)

CART-model Synthesis (non-DP synthetic) Cell Suppression (k = 6)

Metrics: Consistency Checks 

PATECTGAN Differential Private GAN (ε = 10)

Age Inconsistencies: These 

inconsistencies deal with the 

AGE feature; records with agebased inconsistencies might 

have children who are married, 

or infants with high school 

diplomas

Work Inconsistencies: These 

inconsistencies deal with the 

work and finance features —

such as high incomes while 

being in poverty. 

Housing Inconsistencies: 

Records with household 

inconsistencies might have 

more children in the house than 

the total household size, or be 

residents of group quarters 

(such as prison inmates) who 

are listed as owning their 

residences.
Differential Private Histogram (ε = 10)

CART-model Synthesis (non-DP synthetic) Cell Suppression (k = 6)

Metrics: Unique Exact Match Rate 

PATECTGAN Differential Private GAN (ε = 10)

Percent of unique Target 

Data records exactly 

matched in Deid. Data: 

48.5%

Unique Exact Match Rate:

This is a count of unique 

records in the target data 

that were exactly reproduced 

in the deidentified data. 

Because these records were 

unique outliers in the target 

data, and they still appear 

unchanged in the 

deidentified data, they are 

potentially vulnerable to 

reidentification.

Percent of unique Target 

Data records exactly 

matched in Deid. Data: 

100%

Percent of unique Target 

Data records exactly 

matched in Deid. Data: 

20.32%

Percent of unique Target 

Data records exactly 

matched in Deid. Data: 

7.1%
Differential Private Histogram (ε = 10)

CART-model Synthesis (non-DP synthetic) Cell Suppression (k = 6)

Application Specific Metrics: Relationship between Education and Income 

PATECTGAN Differential Private GAN (ε = 10)

This data-specific metric looks at linear 

regression on adults (AGEP > 15) 

across two features: Income Decile and 

Educational Attainment. Higher values 

of EDU should lead to higher values of 

PINCP_DECILE, however the 

relationship is different for different 

demographic subgroups.

Here we show how well the deidentified 

data preserves the distribution of black 

women, using a deviation heatmap: 

Purple indicates the deidentified data 

contains too few individuals in that 

area, brown indicates too many. The 

original target distribution is shown 

above in blue. 
CART-model Synthesis (non-DP synthetic)

What Happens on the Full Feature Set?: Pairwise Correlations 

10 Feature Subset

21 Features 

10 Feature Subset

21 Features 

PATECTGAN Differential Private GAN (ε = 10)

Pairwise Correlations: A key 

goal of deidentified data is to 

preserve the feature 

correlations from the target 

data, so that analyses 

performed on the deidentified 

data provide meaningful insight 

about the target population. 

Which correlations are the 

deidentified data preserving, 

and which are being altered?

The Pearson Correlation

difference was a popular utility 

metric during the HLG-MOS 

Synthetic Data Test Drive. Note 

that darker highlighting 

indicates pairs of features 

whose correlations were not 

well preserved by the 

deidentified data.
CART-model Synthesis (non-DP synthetic)

What Happens on the Full Feature Set?: Pairwise Correlations 

10 Feature Subset

21 Features 

10 Feature Subset

21 Features 

PATECTGAN Differential Private GAN (ε = 10)

Pairwise Correlations: A key 

goal of deidentified data is to 

preserve the feature 

correlations from the target 

data, so that analyses 

performed on the deidentified 

data provide meaningful insight 

about the target population. 

Which correlations are the 

deidentified data preserving, 

and which are being altered?

The Pearson Correlation

difference was a popular utility 

metric during the HLG-MOS 

Synthetic Data Test Drive. Note 

that darker highlighting 

indicates pairs of features 

whose correlations were not 

well preserved by the 

deidentified data.
CART-model Synthesis (non-DP synthetic)

What Happens on the Full 24 Feature Set?: Pairwise PCA 

10 Feature Subset

21 Features 

10 Feature Subset

21 Features 

PATECTGAN Differential Private GAN (ε = 10)

10 Feature Subset Target

24 Feature Target
CART-model Synthesis (non-DP synthetic)

What Happens on the Full 24 Feature Set?: Pairwise PCA 

10 Feature Subset

21 Features 

10 Feature Subset

21 Features 

PATECTGAN Differential Private GAN (ε = 10)

10 Feature Subset Target

24 Feature Target
CART-model Synthesis (non-DP synthetic)

What Happens on the Full 24 Feature Set?: UEM 

10 Feature Subset

21 Features 

10 Feature Subset

21 Features 

PATECTGAN Differential Private GAN (ε = 10)

Percent of unique Target 

Data records exactly 

matched in Deid. Data: 

20.32%

Percent of unique Target 

Data records exactly 

matched in Deid. Data: 

7.1%

Percent of unique Target 

Data records exactly 

matched in Deid. Data: 

0%

Percent of unique Target 

Data records exactly 

matched in Deid. Data: 

2.4%

Unique Exact Match Rate:

This is a count of unique 

records in the target data 

that were exactly reproduced 

in the deidentified data. 

Because these records were 

unique outliers in the target 

data, and they still appear 

unchanged in the 

deidentified data, they are 

potentially vulnerable to 

reidentification.
NIST Collaborative Research Cycle: Far more than four algorithms 

https://pages.nist.gov/privacy_

collaborative_research_cycle/
Collaborative Research Cycle

The CRC is an ongoing 

NIST program that 

provides resources for 

researching the behavior 

of deidentification (data 

privacy) on diverse 

populations. 

Resources include: 

● Techniques 

Directory

● Evaluation 

Reports

● Archive of 

Deidentified Data 

Samples
Collaborative Research Cycle

The CRC is an ongoing 

NIST program that 

provides resources for 

researching the behavior 

of deidentification (data 

privacy) on diverse 

populations. 

Resources include: 

● Techniques 

Directory

● Evaluation 

Reports

● Archive of 

Deidentified Data 

Samples
Collaborative Research Cycle

The CRC is an ongoing 

NIST program that 

provides resources for 

researching the behavior 

of deidentification (data 

privacy) on diverse 

populations. 

Resources include: 

● Techniques 

Directory

● Evaluation 

Reports

● Archive of 

Deidentified Data 

Samples
Collaborative Research Cycle

The CRC is an ongoing 

NIST program that 

provides resources for 

researching the behavior 

of deidentification (data 

privacy) on diverse 

populations. 

Resources include: 

● Techniques 

Directory

● Evaluation 

Reports

● Archive of 

Deidentified Data 

Samples
NIST Collaborative Research Cycle: Far more than four algorithms 

Meta-analysis notebooks and tools available on the NIST CRC site make it easy to 

explore the archive
Pair-wise PCA Inspection 

Tool

Pairwise PCA is a relatively new visualization

metric that was introduced by the IPUMS

International team during the HLG-MOS

Synthetic Data Test Drive.

It lets us look at the high dimensional data

distribution using a set of 2D scatterplots along

principle component axes. The plots look at

the deidentified data and target data from the

same angle (ie, using axes from the target

data), so we can directly see where their

distributions differ from each other.

The pairwise PCA tool lets you interactively

explore these plots using a GUI interface.

You can install it by following the directions

here: https://github.com/usnistgov/pairwise_PCA

NIST Collaborative Research Cycle: Far more than four algorithms 
Pair-wise PCA Inspection 

Tool

Pairwise PCA is a relatively new visualization

metric that was introduced by the IPUMS

International team during the HLG-MOS

Synthetic Data Test Drive.

It lets us look at the high dimensional data

distribution using a set of 2D scatterplots along

principle component axes. The plots look at

the deidentified data and target data from the

same angle (ie, using axes from the target

data), so we can directly see where their

distributions differ from each other.

The pairwise PCA tool lets you interactively

explore these plots using a GUI interface.

You can install it by following the directions

here: https://github.com/usnistgov/pairwise_PCA

NIST Collaborative Research Cycle: Far more than four algorithms 
Collaborative Research Cycle (CRC) Strategy

● 12-month program that collects, disseminates, and analyzes synthetic data

● No prize money, low barrier to participation, emphasizes cooperation

Exploratory Phase (Feb. 2023 - July. 2023)

● NIST releases Diverse Community Excerpt Data

● Participants submit de-identified data and abstract on techniques

● NIST releases machine readable analysis of submissions (acceleration bundle)

Explanatory Phase (July. 2023 - Dec. 2023)

● Participants perform meta-analysis on the acceleration bundle.

● NIST hosts a seminar series and conducts outreach

● Participants submit papers on their analyses Nov 17th 2023

● NIST hosts conference Dec. 18 2023

https://pages.nist.gov/privacy_collaborative_research_cycle/

google keywords: NIST collaborative research cycle

or NIST CRC
Thank you!

Questions? 

gary.howarth@nist.gov

christine.task@knexusresearch.com

https://pages.nist.gov/privacy_collaborative_research_cycle/

google keywords: NIST collaborative research cycle

or NIST CRC