SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Higher Order Similarity Metrics ( $k$ -marginal, equivalent subsample) Data analysts often use more than two features. SDNist provides higher order distributional similarity comparison using the  $k$ -marginal edit distance similarity score [7, 12, 13]. This compares the target data and deidentified data densities across  $k$ -marginal queries, with a maximum score of 1000 indicating identical data sets. The report provides overall and and per-PUMA 3-marginal scores. To aid interpretation, the report includes an Equivalent Subsample (ES) table that provides a comparison to edit distance induced by sampling error. A  $k$ -marginal score with an ES of  $15\%$  is about as dissimilar from the target data as a data set created by uniformly randomly discarding  $85\%$  of the target data.
Task Comparison Metrics (propensity, linear regression) The report includes two machine learning task-based metrics. The propensity metric (Figure 6) trains a classifier to predict whether a record
belongs to the target or deidentified data. The classifier's per-record prediction confidence (propensity distribution) is plotted for both data sets; when the classifier finds it impossible to distinguish the two, the traces will match with a spike at  $50\%$ . The linear regression metric focuses on a task very relevant to the subpopulation dispersal problem (see Figure 6). It uses linear regression to summarize the relationship between educational attainment and income decile for race + sex subgroups. The target regression line is given in red, the deidentified line is in green. Correlation strength differs for different subgroups; this causes dispersal and introduces challenges for deidentification. Some deidentification approaches artificially weaken the correlation, and others strengthen it.
Visual Diagnosis Metrics (PCA, regression heatmap) To help diagnose the causes of low similarity scores, and identify bias/artifacts introduced during deidentification, the SDNist report includes several visualizations. Pairwise Principle Component Analysis (PCA) supports direct comparison of the target and deidentified data distribution using scatterplots across principle component axes. An interactive PCA exploration tool is available. In Figure 6, the target data (ground truth) PCA scatterplot is shown on the left in blue, the deidentified data plot is shown on the right, in green. For additional insight, records with marital status = N/A (indicating children < 15) are highlighted in red in both plots. Algorithms that provide good privacy and utility will reproduce the shape of the target scatterplot using new points (i.e., deidentified records). The linear regression heatmaps (Figure 5) provide another resource for identifying artifacts. The target data distribution is shown as a red-blue heatmap of distribution densities (normalized by educational attainment level). The deidentified heatmaps show deviation from the target distribution: brown indicates the deidentified data contains too many individuals in that category, purple indicates it contains too few. Artifacts and bias are identifiable as large areas of blue or brown blocks, indicating where privacy techniques systematically erase or over emphasize parts of the distribution.
Empirical Privacy Metrics (Unique Exact Match) Privacy protection can be considered empirically, or formally. For techniques with parameterized guarantees (such as differential privacy or  $k$ -anonymity) we include parameter values in their metadata. However, any technique (including DP) may have catastrophic privacy failures. The Unique Exact Match (UEM) metric is a measure of unambiguous privacy leakage applicable to all deidentification techniques; it simply counts the percentage of unique individuals in the target data who remain present unaltered in the deidentified data. Being unique, these individuals are more vulnerable to reidentification; being unaltered, this reidentification may not be difficult. As Table 1 (and Appendix E.3) shows, a simple differentially private hisotogram technique with weak guarantee  $\epsilon = 10$  can reproduce its input data nearly exactly without violating DP. The UEM metric allows us to identify when one technique is performing very badly on privacy, or identify when one technique is performing significantly worse than another. Because UEM doesn't test non-trivial reidentification attacks, it cannot be used to determine whether a technique provides good privacy, or which of two similarly performing techniques is best. We plan to expand the SDNist privacy metrics; however, this starting point provides valuable insights.
5 The Collaborative Research Cycle Data and Metrics Archive
The CRC Data and Metrics Archive is comprised of deidentified data samples generated by applying deidentification algorithms to the Diverse Community Excerpts, using varied techniques, feature subsets, and parameters. It opened collection in March 2023 and is still actively accepting submissions; at time of writing it includes 453 samples of 33 techniques from 18 deidentification libraries. Below we provide a brief overview of its contents, based on meta-data available in the archive index file.
Privacy Types. We delineate three basic types: 56 samples use Statistical Disclosure Control Techniques that use perturbation or redaction to anonymize the target data while leaving it substantially intact; 139 are non-differentially private synthetic data techniques which generate new records to fit the target distribution; and 258 are Differentially Private techniques which add randomization to satisfy a formally private guarantee limiting an individual record's influence on the relative probability of possible outputs. [3]
Algorithm Types. To support meta-analysis, we provide a high-level categorization of approaches. All approaches can be implemented with or without formal privacy guarantees. In the archive, 166 deidentified samples use neural network modeling approaches, 143 fit traditional statistical models, 67 iteratively update a naive distribution to mimic query results on the target data, 56 use statistical disclosure control anonymization, 12 are simple histogram techniques that alter counts of record occurrences, and nine select new points using geometric interpolation between target records.
Exploring Diverse Subgroup Dispersal in the Archive. In Section 2 and Figure 1 we discussed the impact subpopulation dispersal and resulting small count cells can have on performance after deidentification. In the analyses below we look at how this impact plays out in practice in the archive. These deidentified samples used the National Excerpts target data and demographic-focused feature set. To compare subgroup utility, we use the  $k$ -marginal metric across 4-margins including the race + sex features. Only samples with  $15\%$  subsample equivalence or greater are included.
Figure 3 shows the distribution of  $k$ -marginal scores for the four largest race + sex subpopulations; the more dispersed group in general has poorer performance, but some deidentified data samples show good performance overall. Understanding what distinguishes deidentification algorithms that are more or less impacted by subgroup dispersal is a key goal of this benchmarking project, and identifying them is a first step. Figure 4a provides further breakdown, showing how subgroup performance varies with overall data fidelity. We see that the expected difference in performance appears regularly but not constantly, some algorithms overcome it and provide similar performance for all groups. Figure 4b supplements this with privacy leak information using the UME metric (Section 4). Some high performing algorithms are actually identi
Figure 3: 4-marginal edit distance similarity, by demographic subgroup, from the CRC Data and Metrics Archive
cally reproducing the target data, providing very little privacy, while others are able to maintain distributional fidelity for all subgroups using substantially new records. The CRC benchmarking program is designed to support the research that leads to a formal understanding of this behavior, and the development of high fidelity, equitable deidentification techniques with negligible privacy leaks. The detailed metrics for all samples appearing in these charts are provided in Appendix E2; no current archive algorithm fully satisfies all three requirements for this feature set.
(a)  $k$  -marginal similarity performance.
Figure 4: Subgroups utility/privacy performance, ordered by total population 4-marginal performance.
(b) UEM privacy performance.
5.1 Exploration of Selected Algorithms
Table 1 uses seven interesting Archive contributions to demonstrate the Excerpts' efficacy as a tool for exploring and understanding the behavior of deidentification algorithms.
The regression metric (Figure 5) allows us to explore the impact of variable subpopulation dispersal. Very poor performing algorithms show little impact (ADSGAN, PACSynth), while others have worse performance on the more dispersed subpopulation (black women). CART and MST perform well, but introduce a slight bias that strengthens the correlation between high educational attainment and high income in the dispersed group (brown artifact in upper right of heatmap). This could obscure the real disparity between the groups.
Library and Algorithm	Privacy Type	Algorithm Type	Priv. Leak (UEM)	Utility (ES)
DP Histogram (ε = 10)	differential privacy (DP)	simple histogram	100%	~90% (988)
R synthpop CART model [14]	non-DP synthetic data	multiple imputation decision tree	2.54%	~40% (935)
MOSTLY AI SDG [15] [16]	non-DP synthetic data	proprietary pre-trained neural network	0.03%	~30% (921)
SmartNoise MST (ε = 10) [17]	DP	probabilistic graphical model (PGM)	13.6%	= 10% (969)
SDV CTGAN [18] [19]	non-DP synthetic data	generative adversarial network (GAN)	0.0%	~5% (775)
SmartNoise PACSynth (ε = 10) [20]	DP + k-anonymity	constraint satisfaction	0.87%	~1% (551)
synthcity ADSGAN [21] [22]	custom noise injection	GAN	0.0%	< 1% (121)
Table 1: Performance of selected deidentification algorithms, see Appendix E.1 for additional details.
White Men	Black Women	White Men	Black Women
Target Data	synthcity ADSGAN
SmartNoise PACSynth (epsilon-10)	Synthetic Data Vault CTGAN
SmartNoise MST (epsilon-10)	R synthpop CART model
Differentially Private Histogram (epsilon-10)	MOSTLY AI Synthetic Data Platform
Figure 5: Linear regression metric showing how well the selected algorithms maintain the relationship between educational attainment  $(x)$ . and income decile  $(y)$  for different demographic groups.
Considering privacy, all algorithms in the left column guarantee differential privacy with an identical privacy parameter setting  $\epsilon = 10$  (weak privacy protection). However we see in Table 1 that techniques with the same formal guarantee can produce widely varied results in terms of both privacy and utility. PACSynth has additional  $k$ -anonymity protection which eliminates rare, dispersed records; this provides good privacy, but has much poorer utility with unusual impacts on the distribution that can be identified in this and other metrics. Meanwhile, the simple DP Histogram provides almost no privacy protection at all, reproducing the target data nearly exactly. MST is possibly good compromise of privacy and utility (and would provide better privacy at smaller  $\epsilon$ ), but non-differentially private techniques CART and MostlyAI have significantly less trivial privacy leakage (by UEM metric, Section 4) and better utility.
In figure 6 we present two more SDNist metrics: The PCA metric uses scatterplots along principle component axes to compare the shapes of distributions. The propensity metric trains a classifier to distinguish between real and synthetic data; if the data are indistinguishable the propensity distributions will peak at the center (indicating the classifier can only make a '50/50 guess' whether a given record is from the target data or deidentified data). If the deidentified data contains artifacts or bias these will be visible as mismatches in the shape of the scatterplot. This figure compares the three neural network synthesizers in our selection, and provides further explanation for the utility values noted in Table 1. CTGAN is not preserving constraints that hold for child records (highlighted in
Figure 6: Propensity and PCA metrics.
red), and is missing a constraint on housing features (visible as the three well-separated clusters in the target data, but blurred together in the CTGAN data). Examples of records violating constraints might be a 7-year-old widow or an inmate who owns her jail; these can be confidently classified as synthetic by the propensity metric. Meanwhile, the ADSGAN data contains no children at all, and has retained none of the structure of the target data. Note that these results are more informative than simple edit distance utility scores. We are able to identify and diagnose specific algorithm behaviors induced by the real world challenges in the Excerpt data.
Tool name	Univariate / Correlations	Higher Order Similarity	Task Comparison	Visual Diagnosis	Privacy analysis	Built-in synthesizer(s)
Anonymeter[23]	✓
Data Responsibly[24]	✓	✓	✓
NHS Synthetic Data Pipeline	✓	✓	✓	✓	✓	✓*
OpenDP[25]	✓	✓*
R Synthpop[14]	✓	✓	✓	✓	✓
SDGym / SDMetrics[18]	✓	✓	✓	✓	✓
SDNist	✓	✓	✓	✓	✓
Synthcity[22]	✓	✓	✓	✓	✓
Table Evaluator	✓	✓	✓	✓
TAPAS	✓
YData	✓	✓	✓	✓	✓
Table 2: Synthetic data evaluation tools
Table 2 provides a broad survey of comparable tools for benchmarking and evaluating synthetic data, including both library links and research citations. Libraries annotated with \* are specifically configured to evaluate data from their built-in synthesizer models. Visual diagnosis refers to tools that visualize and explore data within a high dimensional space (e.g., principal component analysis). Privacy analysis specifically refers to comparison of output records to input records (e.g., re-identification, replication). Many of these tools have built in data and all of them also have 'bring-your-own-data' capabilities. SDNist, and the CRC program in general, distinguish themselves with a more extensive evaluation set, and a specific goal to promote the state of research in the deidentification field as a whole, aiming at rigorously well understood high performance on diverse, real-world data. Unlike the CRC, none of these resources are designed to focus energy on improving synthetic data specifically by comparing algorithm performance on unified data with common metrics.
Rather than focus on evaluating arbitrary data, we present tools that are tailored to a specific dataset. We are motivated by previous evaluations of differentially private [26] and of GAN-based synthetic data [27] generators that show surprising differences in algorithm performance. The Excerpts, partnered with SDNist and theory presented here, provide a compact vehicle to address diverse subpopulations, which we show is a persistent problem facing deidentification technologies.
Disclaimer. The Collaborative Research Cycle (CRC), the Data and Metrics Archive, and SDNist are intended as tools to encourage investigation and discussion of deidentification algorithms, and they are not intended or suitable for product evaluation. The National Institute of Standards and Technology does not endorse any algorithm included in these resources. No mention of a commercial product in this paper or any CRC resource constitutes an endorsement.
[18] Neha Patki, Roy Wedge, and Kalyan Veeramachaneni. The Synthetic Data Vault. In 2016 IEEE International Conference on Data Science and Advanced Analytics (DSAA), pages 399-410, 2016. doi:10.1109/DSAA.2016.49.
[19] Lei Xu, Maria Skoularidou, Alfredo Cuesta-Infante, and Kalyan Veeramachaneni. Modeling Tabular data using Conditional GAN. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d' Alché-Buc, E. Fox, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 32. Curran Associates, Inc., 2019. URL https://proceedings.neurips.cc/paper_files/paper/2019/file/254ed7d2de3b23ab10936522dd547b78-Paper.pdf.
[20] OpenDP. Private aggregate seeded from pac-synth. https://docs.smartnoise.org/synth/ synthesizers/pac_synth.html, 2023.
[21] Jinsung Yoon, Lydia N. Drumright, and Mihaela Van Der Schaar. Anonymization Through Data Synthesis Using Generative Adversarial Networks (ADS-GAN). IEEE Journal of Biomedical and Health Informatics, 24(8):2378-2388, August 2020. doi:10.1109/JBHI.2020.2980262. URL https://ieeexplore.ieee.org/document/9034117/.
[22] Zhaozhi Qian, Bogdan-Constantin Cebere, and Mihaela van der Schaar. Synthcity: facilitating innovative use cases of synthetic data in different data modalities, 2023. URL https://doi.org/10.48550/arXiv.2301.07573.
[23] Matteo Giomi, Franziska Boenisch, Christoph Wehmeyer, and Borbála Tasnádi. A unified framework for quantifying privacy risk in synthetic data, 2023. URL https://doi.org/10.56553/popets-2023-0055.
[24] Haoyue Ping, Julia Stoyanovich, and Bill Howe. DataSynthesizer: Privacy-Preserving Synthetic Datasets. In Proceedings of the 29th International Conference on Scientific and Statistical Database Management, pages 1-5, Chicago IL USA, June 2017. ACM. doi:10.1145/3085504.3091117.
[25] The OpenDP Team. OpenDP Library. URL https://github.com/opendp/opendp.
[26] Claire McKay Bowen and Joshua Snake. Comparative Study of Differentially Private Synthetic Data Algorithms from the NIST PSCR Differential Privacy Synthetic Data Challenge. Journal of Privacy and Confidentiality, 11(1), February 2021. doi:10.29012/jpc.748.
[27] Yuchao Tao, Ryan McKenna, Michael Hay, Ashwin Machanavajhala, and Gerome Miklau. Benchmarking Differentially Private Synthetic Data Generation Algorithms, February 2022. URL https://doi.org/10.48550/arXiv.2112.09238.