Privacy and Public Records: 

Perils and possible solutions of releasing 

public safety records 

December 6, 2023 
 

 

 

Disclaimer

Certain commercial entities, equipment, or materials may be identified in 

this document in order to describe an experimental procedure or concept 

adequately. Such identification is not intended to imply recommendation or 

endorsement by the National Institute of Standards and Technology, nor is it 

intended to imply that the entities, materials, or equipment are necessarily 

the best available for the purpose. 

Please note, unless mentioned in reference to a NIST Publication, all 

information and data presented is preliminary/in-progress and subject to 

change. 

2 
AGENDA 

1 Introduction 

2 Quick Poll 

3 Presentation 

4 Live Q&A 

3 
Gary S. Howarth, II 

• Physical Scientist and project 

manager for the Public Safety 

Communications Research 

Division (CTL) and the Privacy 

Engineering Program (ITL). 
Power in Data 

● Model outcomes 

● Identify risk factors 

● Distinguish subpopulations 

Medical Records 

Analysis 

5 
Power in Data 

● Find waste and fraud 

● Predict policy outcomes 

● Identify opportunities 

Financial Records 

Analysis 

6 
Power in Data 

Analysis ● Infrastructure planning 

● Optimize services 

● Traffic routing 

Location Data 

7 
Power in Data 

911 Calls for Service 

8 
Power in Data 

911 Calls for Service 

● Parse by water 

quality-related calls 

9 
Protecting PII with Redaction? 

10 
 

Can you protect PII with redaction? 

Redacted data is vulnerable to de-anonymization attacks with auxiliary data 

sources 

Diagnosis 

Procedure 

Medications 

Year of visit 

Ethnicity 

Redacted Medical Record Public Voter Roster 

Name 

Party affiliation 

Address 

Registration date 

Date last voted 

Zip code 

Birth year 

Sex 

Zip code 

Birth year 

Sex 

11 
 

 

 

Protecting PII with Redaction? 

Redacted data is vulnerable to de-anonymization attacks with auxiliary data 

sources 

Redacted Medical Record Public Voter Roster 

• Diagnosis • Name • Zip code • Procedure • Party affiliation • Birth year • Medications • Address • Sex • Date of Visit • Registration date

• Ethnicity • Date last voted

L. Sweeney. Weaving Technology and Policy 

Together to Maintain Confidentiality. 

Journal of Law, Medicine & Ethics, 25, nos. 

2&3 (1997): 98-110. 

L. Sweeney. Simple Demographics Often 

Identify People Uniquely. Carnegie Mellon 

University, Data Privacy Working Paper 3. 12 

Pittsburgh 2000. 

87% of people in U.S. can be re-identified using 3 quasi-identifiers. 
 

Know when and where what taxi was entered? 

J. Trotter. Public NYC Taxicab 

Database Lets You See How Celebrities 

Tip. Gawker. 14 Oct 2014. 13 
 

 

 

Know when and where what taxi was entered? 

• Pickup and Dropoff 

Address 

• Pickup and Dropoff 

Time 

• Intermediate stops

• Fare

• Tip (some of the time)

Taxi Database Paparazzo Photo 

• Rider(s)

• Taxi number 

• Pickup OR 

Dropoff address 

• Partial taxi number

• Time and date

14 
Traditional disclosure control 
Anonymization is not enough 

Practically all information is identifying. 

Field suppression, redaction, and anonymization 

techniques limit utility and may be highly vulnerable to 

attack. 

16 
 

Reidentification is bad 

Reidentification attacks fuel: 

• Discrimination, abuse, violence against minorities 

• SWATing 

• Predatory marketing, phishing, and cons 

• Distrust of information collection programs 
Adding noise to protect privacy 

18 
 

 

 

 

 

Adding noise to protect privacy 

• Phones makes suggestions. 

• Tech companies collect feedback. 

• Some collections involve privacy noise. 

Gary always chooses 🛌 to represent ‘sleep.’

Gary Phone transmits 

Selects feedback 

🛌 🛌

🛌 💤 (noise) 

🛌 🛌

🛌 🛌

🛌 😴 (noise) 

Sometimes the phone adds noise creating privacy (plausible deniability). 
 

Adding noise to protect privacy Selection frequency 

‘sleep’ →🛌 💤 😴

Phone provider can still analyze the noisy data for meaning 
Adding noise to protect privacy 

Sensitive survey examples: 

• Have you ever under-reported income on your taxes? 

• What’s your HIV status? 
 

 

Illustration of the privacy-utility trade off.

From Liu et al. “Privacy-Preserving Monotonicity of Differential

Privacy Mechanisms.” 2018.

Illustration of the privacy-utility trade off.

Privacy-utility trade off 

From Liu et al. “Privacy-Preserving 

Monotonicity of Differential Privacy 

Mechanisms.” 2018.

22 
Remember, Gary always chooses 🛌

Privacy-utility trade off 

Low privacy, high utility High privacy, low utility 

Selection frequency Selection frequency 

‘sleep’ →🛌 💤 😴 ‘sleep’ →🛌 💤 😴
Differential privacy 

Differential privacy is: 

• Rigorous mathematical definition of privacy 

• A framework to add privacy noise 

Differential privacy is not: 

• A specific algorithm 

• Silver bullet 

• Bogie man 
DP can be used to make synthetic records 

25 
DP can be used to make synthetic records 

ORIGINAL DATA SYNTHETIC DATA 

Person Age Income State 

O1 24 31,000 CO 

O2 88 45,000 NM 

... ... ...

O450 11 0 CO 

→

DP algorithm 

Person Age Income State 

S1 44 51,151 CO 

S2 22 33,232 CO 

... ... ...

S450 35 12,223 NM 

Aggregate Metric Original Synthetic 

Mean age 44 44 

Mean Income 51,231 51,244 

People in CO 250 249 

26 
P

DP can be used to make synthetic records 

erson Age Income State

O1 24 31,000 CO 

O2 88 45,000 NM 

... ... ...

O450 11 0 CO 

ORIGINAL DATA SYNTHETIC DATA 

→

DP algorithm 

Person Age Income State

S1 43 51,845 NM 

S2 22 31,412 NM 

... ... ...

S499 19 21,121 CO 

Differential privacy limits how much can be learned 

about an individual in the data. 

27 
DP can be used to make synthetic records 

ORIGINAL DATA SYNTHETIC DATA 

Person Age Income State

O1 

O2 

...

24 

88 

...

31,000 

45,000 

...

CO 

NM →

DP algorithm 

O450 11 0 CO 

Person Age Income State

S1 43 51,845 NM

S2 22 31,412 NM 

... ... ...

S499 19 21,121 CO 

Original Synthetic 

Metric All data O450 All data O450 

Mean age 44 45 44 44 

Mean Income 51,231 51,345 51,244 51,243 

People in CO 252 251 249 249 28 
DP is tunable for privacy 

29 
 

 

Data Publishing mandates 

Case study: 

U.S. Census Bureau is mandated to make accurate counts of people 

(U.S. Constitution Article I, Section 2) 

U.S. Census Bureau is required by law to protect respondent confidentiality at every stage of 

the data lifecycle with criminal penalties for violations 

(U.S. Code 13 § 8-9 / 141) 

“Differential privacy is the best science available to protect 2020 Census respondent 

confidentiality while minimizing the impact on statistical validity.”1 

1. Disclosure Avoidance and the 2020 Census Redistricting Data, U.S. Census Bureau 30 
Big Questions of Differential Privacy? 

• What types of data can we successfully de-identify? 

• How much noise must we add? 

• Are the noisy data still useful / accurate? 

• Are the output data actually private? 

• Are the noisy data accurate for all subgroups in the data? 

31 
 

NIST Innovates: 2019 Synthetic Data Challenge 

Goal: De-identify records from San Francisco Calls for Service portal. 

NIST gave Competitors: Competitors gave NIST: 

• training data • deidentified data 

• basic, ‘baseline’ algorithm • new, innovative algorithms 

• scoring methodology • mathematical proofs their algorithms were DP 

• public leaderboard 

https://data.sfgov.org/Public-Safety/Fire-Department-Calls-for-Service/ 32 
 

NIST Innovates: 2019 Synthetic Data Challenge 

Public leaderboard within a match 

(simulated example) 

TIME 

SCORE 

Team A 

Team B 

Team C 

33 
NIST Innovates: 2019 Synthetic Data Challenge 

Acknowledgements: 

- Terese Manley, NIST PSCR, Prize Manager 

- Christine Task, Knexus Research, Technical Lead 

https://doi.org/10.6028/NIST.TN.2151 34 
 

 

 

 

 

 

 

 

 

NIST Innovates: 2020 Temporal Map Challenge 

Sprint 1 

Baltimore 911 Incidents 

Highly variable PS data 

Training data: 2019 

Evaluation data: 2016 & 2020 

Sprint 2 Sprint 3 

American Community Survey (US Census) Chicago Taxi Rides 

Complex demographic information Linked trip information 

Training data: IL + OH Training data: 2019 

Evaluation data: NY + PA & NC+SC+GA Evaluation data: 2016 & 2020 

35 
Temporal Map Challenge Outcomes 

Score (higher better, 1000 = max) 36 
Temporal Map Challenge Outcomes 

Score (higher better, 1000 = max) 37 
Temporal Map Challenge Outcomes 

“NIST temporal map 

challenge”

Acknowledgements 

• Dr. Christine Task, Knexus 

Research, Technical Lead 

• John Garofolo, NIST ITL, 

Portfolio Lead 

• DrivenData and HeroX 

38 
 

Collaborative Research Cycle (CRC) 

NIST privacy prize challenges have: 

• Provided essential proof-of-concept experiments 

• Accelerated practical synthetic data generating techniques 

• Expanded the audience for and consumers of differential privacy 

NIST CRC seeks to: 

• Expand the scope and breadth of synthetic 

data evaluations 

• Compare different algorithms on the same 

underlying data 

• Provide a venue for cooperation 
The Diverse Communities Excerpt Data 

Data Features (excerpts of American Community Survey Data): 

40 
The Diverse Communities Excerpt Data 

Data PUMA and 

Postcard Descriptions: 

41 
The Diverse Communities Excerpt Data 

Data PUMA and 

Postcard Descriptions: 

42 
The Diverse Communities Excerpt Data 

Data PUMA and 

Postcard Descriptions: 

43 
The SDNist Evaluator 

44 
The SDNist Evaluator (sdnist v2.3) 

pip install sdnist 

45 
Algorithms: A Sample of Four Deidentification Approaches 

DP GAN: Add randomized 

noise while training an ML 

model to reproduce the 

distribution. 

Differential Private Histogram (ε = 10) PATECTGAN Differential Private GAN (ε = 10)

CART: Use a sequence of decision trees to generate 

new values for every feature, one at a time. 

Cell Suppression (k = 6) 

CART-model Synthesis (non-DP synthetic) 
Metrics: Univariate 

Differential Private Histogram (ε = 10) PATECTGAN Differential Private GAN (ε = 10)

CART-model Synthesis (non-DP synthetic) Cell Suppression (k = 6) 

47 
 

 

 

 

 

 

Metrics: Pairwise Correlations 

Differential Private Histogram (ε = 10) PATECTGAN Differential Private GAN (ε = 

10) 

CART-model Synthesis (non-DP synthetic) Cell Suppression (k = 6) 

Pairwise Correlations: A key 

goal of deidentified data is to 

preserve the feature 

correlations from the target 

data, so that analyses 

performed on the deidentified 

data provide meaningful 

insight about the target 

population. Which correlations 

are the deidentified data 

preserving, and which are 

being altered? 

The Pearson Correlation 

difference was a popular utility 

metric during the HLG-MOS 

Synthetic Data Test Drive. Note 

that darker highlighting 

indicates pairs of features 

whose correlations were not 

well preserved by the 

deidentified data. 

48 
 

 

 

 

 

 

 

 

 

 

Metrics: Propensity 

Differential Private Histogram (ε = 10)

CART-model Synthesis (non-DP synthetic) 

PATECTGAN Differential Private GAN (ε = 

10) 

Cell Suppression (k = 6) 

Propensity Metrics: 

Can a decision tree classifier 

tell the difference between the 

target data and the 

deidentified data? If a classifier 

is trained to distinguish 

between the two data sets and 

it performs poorly on the task, 

then the deidentified data 

must not be easy to distinguish 

from the target data. If the 

green line matches the blue 

line, then the deidentified data 

is high quality. Propensity 

based metrics have been 

developed by Joshua Snoke 

and Gillian Raab and Claire 

Bowen 

49 
 

 

 

 

 

 

 

 

 

 

Metrics: Pairwise PCA 

Differential Private Histogram (ε = 10) PATECTGAN Differential Private GAN (ε = 

10) 

CART-model Synthesis (non-DP synthetic) Cell Suppression (k = 6) 

PCA Metric visually compares a 

synthetic data set with the original 

input data. It plots high dimensional 

data as a 2D scatterplot using the first 

two principal component axes; each 

point represents an individual in the 

data. Good synthetic data should 

recreate the shape of the original data 

with new points (new synthetic 

individuals). The plot above shows the 

shape of the original sensitive data; the 

synthetic data generators are trying to 

reproduce this distribution. To display 

more detail, we’ve used red points to 

highlight records that represent 

children (MSP value = ‘N’)

50 
 

 

 

 

 

 

 

 

 

 

 

 

Differential Private Histogram (ε = 10)

CART-model Synthesis (non-DP synthetic) Cell Suppression (k = 6) 

Metrics: Consistency Checks 

PATECTGAN Differential Private GAN (ε = 

10) 

Age Inconsistencies: These 

inconsistencies deal with the AGE 

feature; records with age-based 

inconsistencies might have 

children who are married, or 

infants with high school diplomas 

Work Inconsistencies: These 

inconsistencies deal with the 

work and finance features —such 

as high incomes while being in 

poverty. 

Housing Inconsistencies: Records 

with household inconsistencies 

might have more children in the 

house than the total household 

size, or be residents of group 

quarters (such as prison inmates) 

who are listed as owning their 

residences. 

51 
 

 

 

Metrics: Unique Exact Match Rate 

Percent of unique Target 

Data records exactly 

matched in Deid. Data: 

100% 

Differential Private Histogram (ε = 10)

Percent of unique Target 

Data records exactly 

matched in Deid. Data: 

20.32% 

CART-model Synthesis (non-DP synthetic) 

Percent of unique Target 

Data records exactly 

matched in Deid. Data: 

7.1% 

PATECTGAN Differential Private GAN (ε = 

10) 

Percent of unique Target 

Data records exactly 

matched in Deid. Data: 

48.5% 

Cell Suppression (k = 6) 

Unique Exact Match Rate: 

This is a count of unique 

records in the target data that 

were exactly reproduced in the 

deidentified data. Because 

these records were unique 

outliers in the target data, and 

they still appear unchanged in 

the deidentified data, they are 

potentially vulnerable to 

reidentification. 

52 
NIST Collaborative Research Cycle: Far more than four algorithms 

https://pages.nist.gov/privacy_c 

ollaborative_research_cycle/ 

53 
Collaborative Research Cycle 

The CRC is an ongoing NIST 

program that provides 

resources for researching the 

behavior of deidentification 

(data privacy) on diverse 

populations. 

Resources include: ● Techniques Directory ● Evaluation Reports ● Archive of Deidentified 

Data Samples 

54 
 

 

 

CRC Workshop 

December 18: 10:30 AM – 2:00PM ET 

- Results of CRC submissions 

- Practical lessons on DP, reidentification, 

and other topics 

- Register and see the full agenda here: 

55 
Collaborative Research Cycle 

Acknowledgements 

• Christine Task (Knexus) 

• Karan Bhagat (Knexus) 

• Aniruddha Sen (U. Mass.) 

• Dhruv Kapur (U. Mich.) 

• Ashley Simpson (Knexus) 

56 
NIST Differential Privacy Guidelines 

57 
 

Have some data? Have some ideas? 

Contact me to talk about a potential pilot! 

• Guidance on how to try it at home 

• Internal-only sandbox to try out ideas 

• Help with potential public releases 

Gary Howarth 

gary.howarth@nist.gov 

(720)-360-9158 

58 
Resources 

Contact 

Gary Howarth 

gary.howarth@nist.gov 

(720)-360-9158 

Get Connected 

Subscribe to the NIST PSCR 

newsletter at 

nist.gov/ctl/pscr/get-connected 

59 
Thank You! 
61