Synthcity: a benchmark framework for diverse use cases of tabular synthetic data
Authors: Zhaozhi Qian
Rob Davis
Mihaela van der Schaar
Venue: NeurIPS 2023 Datasets and Benchmarks Poster (2023)
Keywords: Synthetic data
generative models
data augmentation
privacy

Abstract: Accessible high-quality data is the bread and butter of machine learning research, and the demand for data has exploded as larger and more advanced ML models are built across different domains. Yet, real data often contain sensitive information, are subject to various biases, and are costly to acquire, which compromise their quality and accessibility. Synthetic data have thus emerged as a complement to, sometimes even a replacement for, real data for ML training. However, the landscape of synthetic data research has been fragmented due to the diverse range of data modalities, such as tabular, time series, and images, and the wide array of use cases, including privacy preservation, fairness considerations, and data augmentation. This fragmentation poses practical challenges when comparing and selecting synthetic data generators in for different problem settings. To this end, we develop Synthcity, an open-source Python library that allows researchers and practitioners to perform one-click benchmarking of synthetic data generators across data modalities and use cases. Beyond benchmarking, Synthcity serves as a centralized toolkit for accessing cutting-edge data generators. In addition, Synthcity’s flexible plug-in style API makes it easy to incorporate additional data generators into the framework. Using examples of tabular data generation and data augmentation, we illustrate the general applicability of Synthcity, and the insight one can obtain.

This challenge especially prominent for tabular datasets, which are often curated in highly regulated industries including healthcare, finance, manufacturing etc. Synthetic tabular data has the potential fuel the development of AI by unleashing the information in datasets that are small, sensitive or biased. To achieve this, need high-perfonr nigh-performance generative models that both faithfully capture the Synthcity: a benchmark framework for diverse use cases of tabular synthetic data
Zhaozhi Qian University of Cambridge zq2240cam ac
Rob Davis University of Cambridge rd4738cam ac uk
Mihaela van der Schaar University of Cambridge mv4720cam. ac uk
Abstract
Accessible high-quality data is the bread and butter of machine learning research, and the demand for data has exploded as larger and more advanced MI models are built across different domains. Yet, real data often contain sensitive information, subject to various biases, and are costly to acquire, which compromise their quality and accessibility. Synthetic data have thus emerged as a complement, sometimes even a replacement, to real data for MI. training. However, the landscape of synthetic data research has been fragmented due the large number of data modalities (e.g.. tabular data, time series data, images, etc.) and various use cases (e.g., privacy, fairness, data augmentation, etc.). This poses practical challenges in comparing and selecting synthetic data generators in different problem settings. To this end, we develop Synthcity, an open-source Python library that allows researchers and practitioners to perform one-click benchmarking of synthetic data generators across data modalities and use cases. In addition, Syntheity's plug-in in style API makes it easy to incorporate additional data generators into the framework. Beyond benchmarking, it also offers single access point diverse range of cutting-edge data generators. Through examples tabular data generation and data augmentation, we illustrate the general applicability of Synthcity, and the insight one can obtain.
11
12
13
17
18
Introduction
19
20
Access to high quality data is the lifeblood of AI. Although AI holds strong promise in numerous high- stakes domains, the lack of high-quality datasets creates a significant hurdle for the development of
21
22
AI, leading missed opportunities. Specifically, three prominent issues contribute this challenge:
data scarcity, privacy, and bias [MMhrraii et al. 2021 Gianfrancesco et al. 2018 Tashea 2017
Dastin 2018). As result, the dataset may not be available, accessible, or suitable for building performant and socially responsible AI systems [Sambasivan et al. [021].
25
26
27
28
30 data distribution and satisfy additional constraints for the desired use cases.
31 To date, the landscape of synthetic data research has been fragmented because the combination of
32 use cases (i.e. fairness, privacy, and augmentation) and data modalities (e.g. static tabular data, time
33 series data, etc.) creates a plethora of problem settings. In response to the large problem space, the
Submitted to 37th Conference on Neural Information Processing Systems (NcurIPS 2023) Track on Datascts and Benchmarks. Do not distribute.

---

34
community has taken a divide-and-congee approach: highly-specializze generative models have been developed to fit in one particular setting. This has led to a proliferation of specialized generative
35
36
models Jordon et al.
2018
Yoon et al.
2020 Ho et al.
2021
Mehrabi et al. 2021 van Breugel
37
et al. 2021 Zhu al.
2017 Yoon et al. 2018 Saxena and Cao 2021].
This fragmented landscape has created four main challenges for benchmarking synthetic data genera- tors, which would hamper the research progress if left unaddressed.
39
40 1. Challenge in use case specifie evaluation. Most existing studies in generative model only focus
41 on the fidelity of the synthetic data, i.e. how they resemble the real data in distribution Wang et al.
2019]. Tucker al. 22000. Goncalves et al. [2020] Wang al. [02T]] Kokosi and Harron [022] 42
However, additional evaluation is needed to assess the specific cases. For example, the utility to
44
downstream models and the data privacy. This calls for the introduction of new metries as well as
45
new evaluation pipelines.
Challenge in off label uses. Although specialized generative models are developed for one use 47 case, the practical application often requires them to cover multiple use cases (e.g. data augmentation with privacy). Hence, generative models are often used outside the designed scope. Prior work has shown that this may lead to undesirable and unpredictable consequences [Pereira al.] 202 Ganev al. 2022]. As result, researchers need to comprehensively evaluate the generative model across variety of use cases to assess the risk of off label uses.
46
48
51
52
Challenge in comparing with a large number of baselines. In practice, it is often very challenging to systematically compare with a large number of existing baselines because the interfaces (API) of these models are often inconsistent and incompatible (e.g. they may require different formats of input data and conflicting software dependencies). As a result, the researcher usually needs to spend time and effort to harmonize the code rather than focusing on the research question itself.
53
55
56
57
Challenge in understanding the performance gain. Generative models are complex systems that involve many components, such as the model architecture, the objective function, and the hyper- parameters. These aspects all encode prior assumptions and inductive biases, which would bring unique strengths and weaknesses to the models [Bond-Taylor al.) 2021]. However, it is often difficult to pinpoint the exact component that leads to the performance gain. Most existing studies evaluate the model as whole and neglect the role of different components.
58
59
61
62
63
Contribution. In this work, we present Synthcity, an open-source Python library available on pip and GitHub) as a solution to these benchmark challenges. Synthcity offers diverse data modalities and supports various use cases. It provides an extensive set of evaluation metrics for assessing dataset fidelity, privacy, and utility, making it robust tool for evaluating synthetic data across different applications. With wide array of state-of-the-art generators and customizable architectures,
64
65
67
68 users can perform consistent comparisons with existing models, gaining insights into performance improvements. Accessible through an intuitive interface, Synthcity facilitates tabular data generation and augmentation, demonstrated through two case studies. Researchers can employ Synthcity for benchmarks and guidance in synthetic data research
69
70
71
2
The syntheity library
72
73
2.1
Overview of the synthcity workflow
74 Despite the fragmented landscape in synthetic data research, Synthcity implements a unified workflow
75 for benchmark studies. We formalize the process as follows. Let X be the random variable of
78 interest (which could be static, temporal or censored). The real data is composed with observations
77 I; P(X) drawn from the true (but unknown) distribution. For benchmark evaluation, the real data is split into a training (Drrran)) and test (D)est) set. The generator is trained using the training set P(X) in order to sample from it. After training, the generative model generates synthetic data Ds, which will be
79 Dirain During training, the generator (explicitly or implicitly) learns the distribution
81 evaluated with respect to the test set Diett (or in some special cases, the training set ID (.ra).)

---

3. Generate synthetic data
2. Training a generator plugin
4. Evaluate synthotic data
Loading
Plugin (generative models)
Input real data
Synthetic Data
DataLoader (static, time series, survival, etc.)
Metrics (fideliity utility, privacy, etc.)
sesehhittt-plggin
Metadata
wwatheity.plugins.com wwt.eeata......ddaeee...e.e.ee.. dataloader
syntacity metriic
Figure 1: Standard workflow of generating and evaluating synthetic data with synthcity.
82
The synthcity library captures the entire workflow of synthetic data benchmark four steps (Figure Tp. This workflow applies all cases, generative models and data modalities.
83
84
Loading the datasel using a DataLoader. The DataLoader class provides a consistent interface for loading, storing, and splitting different types of input data (e.g. tabular, time series, and survival data). Users can also provide meta-data to inform downstream algorithms, like specifying sensitive columns for
85
86
algorithms.
88
Training the generator using a Plugin. synthcity, the users instantiate, train, and apply different data generators via the Plugin class. Each Plugin represents a specific data generation algorithm. The generator can be trained using the fit() method of Plugin.
89
90
91
3. Generating synthetic data. After the Plugin is trained, the user can use the generate() method to generate synthetic data. Some plugins also allow for conditional generation.
92
93
Evaluating synthetic data. Syntheity provides large set of metries for evaluating different aspects of synthetic data. The Metrics class allows users to perform evaluation.
95 In addition, synthcity also has a Benchmark class that wraps around all four steps. This provides one-line interface for comparing and evaluating different generators and produces an evaluation report at the end of the process.
96
98
2.2
Evaluation for diverse use cases
Synthetic data has many different use cases including faireess, privacy, data augmentation. As benchmarking framework, Synthcity provides the users with a comprehensive list of metrics and routines to evaluate various aspects of synthetic data, including metrics that are specific to these use cases. In this section, we describe the use cases of synthetic data and how Synthcity performs its evaluation. full list of metrics can be found in Appendix.
100
101
102
103
104
2.2.1
Standard data generation
105
Standard data generation refers to the most basic generation task, where the synthetic data should be generated as faithfully as possible the real-data distribution (van Breugel
106
2023 Hansen et al.
2023).
This is captured by the fidelity metrics.
108
Fidelity. The fidelity of synthetic data captures how much the synthetic data resembles real data. The fidelity metrics typically evaluate the closeness between the true distribution and the distribution learned by the generator P using samples from these two distributions. Synthcity supports distribu- tional divergence measures (e.g. Sonsen-Shannon distance, Wasserstein distance, and maximal mean discrepancy) as well two sample detection scores (i.e. scores that measure how well classifier
109
110
111
112
can distinguish real versus synthetic data) (Gretton al. 2012 Lopez-PPee Pazz-Paz and Oquab 2016 Snoke
al. 2018

---

Use case
Method
Evaluation
Reference
Standard data generation
Generative model
Fidelity
Gretton al.
2012
Cross domain augmentation
Domain transfer
Utility
Bing ct al.
2022
Balancing distribution
Minority performance
Lu et al.
2018
ML fairness
Causal fairness
Algorithmic fairness
Xu ct al.
2018
Differential privacy
Privacy metrics
Abadi et al.
2016
Privacy preservation
Threat model
Attack simulation
Shokri al.
2011
Table I:
Synthcity is a unified framework to benchmark diverse use cases of synthetic data. It supports range of methods and evaluation metrics, and also allows evaluation of off label uses.
115
2.2.2
Cross domain augmentation
116
Here we consider a dataset that is collected from multiple domains or sources (c.g. data from different countries). Often the practitioner is interested in augmenting one particular data source that suffers from data scarcity issues (e.g. it difficult to collect data from remote areas) by leveraging other related sources. This challenge has been studied in the deep generative model literature [Antoniou al. 2017 Dina et al. 2022 Das et al. 2022 Bing et al. 2022). By learning domain-specific and domain- domain-agnostic representations, the generator is able to transfer knowledge across domains, making data augmentation more efficient. Synthcity offers a clean interface so that the user can benchmark the downstream utility cross-domain generation using only one line of code..
117
118
121
122
123
124
Utility. Synthcity measures the performance for cross domain augmentation through its utility to downstream tasks. Our approach adapts the common practice of cain-on--yntheete evaluate-on -n-rrel Beaulieu-Jones et al., 2019], where a downstream predictive model is trained on fully synthetic training data and then validated on real testing data.
126
127
128
For data augmentation, Synthcity augments the data-scarce domain in the training data Dr Frain with the synthetic data ID* A predictive model is then trained on this augmented dataset and evaluated on the domain of interest in the testing data Disst Synthcity supports various types of predictive tasks, including regression, classification and survival analysis. In addition to linear predictive models, syntheity supports Xgboost and neural nets as downstream models due to their wide adoption in data analytics. In practice, the user may average the performance of several predictive models to reduce the model uncertainty.
129
132
133
134
As naive baseline, Syntheity reports the predictive performance where data augmentation is performed. Syntheity provides pre-configured pipeline to automatically handle this entire procedure, reducing the code and preventing mistakes
136
138
2.2.3
Synthetic data for ML. fairness
139
Existing research has considered two different ways where Synthetic data could promote fairesss.
140
Table 2 shows the corresponding models in syntheity.
141
Balancing distribution. In this setting, certain groups of people are underrepresented in a dataset that is used for training downstream ML systems. This may lead to a bias being introduced into these
1.
142
143
ML systems [Lu et al., 2018 de Vassimon Manela et al., [222] Kadambi] [202T. As a remedy, one could generate synthetic records for the minority group to augment the real data, thereby achieving balance in distribution. This often requires the data generator to learn the conditional distribution
144
145
P (G), where G is the group label.
147
2.
Causal fairness. The second approach is to generate fairer synthetic data from a biased real dataset and to use synthetic data alone in downstream tasks [emel et al. 2013 Xu et al.. 2018 2019a van Breugel et al. 2021] In this setting, it postulated that the real distribution P(X) reflects existing learn distribution P(X)
148
149
150
biases (e.g. unequal access to healthcare). The task for the generator
that is free from such biases but also stay close to P(X) possible (to ensure high data fidelity).
Typically, notions of causality are employed in the bias removal process.

---

153
Fairness. Synthcity allows users to benchmark both use cases by training downstream predictive model on the fully synthetic or augmented data and presenting their performance or characteristics. For example, one can evaluate the performance gain on any specified (minority) group as an indicator of the utility of synthetic data. In addition, Synthcity also supports standard algorithmic fairness
154
155
156
157
metries for the trained predictive model, such as Fairness Through Unawareness, Demographic Parity and Conditional Fairness van Breugel et al., 20011
158
159
2.2.4
Synthetic data for privacy
Methods for generating that employ differential privacy, and the ones that are designed for specific threat models.
synthetic data mainly fall into two categories: the ones
162
Differential privacy (DP). DP is a formal way to describe how private a data generator is (Dwork] 2008). Typically, generators with DP property introduce additional noise the training procedure
165
Jordon al. 2022]. For example, adding noise in the gradient or using a noisy discriminator in
184
165
GAN architecture [Abadi et al. 2016 Jordon al. 2018 Long et al. 2019].
166
2. Threat model (TM). While DP focuses on giving formal guarantees, the TM approach is designed
167
for specific threat models, such as membership inference, attribute inference, and re-identification
168
Shokri et al. 2017 Kosinski et al. 2013 Dinur and Nissim [2003]. These models often involve regularization terms designed to mitigate privacy attack risk (Yoon et al. 2020].
169
Privacy. Synthcity evaluates the privacy of synthetic data using list of well privacy met-
rics (e.g. k-anonymity (Sweeney) [002] and I-diversity [Machanavaijhala et al.) (007]). Furthermore, it can measure the privacy of data by performing simulated privacy attacks (e.g. a re-identificatton attack). The success (or failure) of such an attack quantifies the degree of privacy preservation.
172
173
174
2.2.5
Evaluating off' label use cases
175
Synthcity allows users to conveniently evaluate the off label usage of generative models. For instance, one could evaluate the privacy of synthetic data even they are not generated by a privacy-enabled generative model. As another example, one could evaluate the fairness for generative models that are differentially private, thereby enabling studies like Ganev al. [2022]
176
177
178
179
Off label evaluation is made easy because Synthcity implements generative models and evaluation metrics in two separate modules (PlugIns and Metrics). The consistent interface enables mix and match of models and metrics to empower different benchmark studies.
180
181
182
2.3
Baseline generative models
183
As benchmarking framework, Synthcity is a one-stop-shop for state-of-the-art benchmarks with large collection of baselines covering both deep generative models and other types of generative models. In this way, the user can easily compare with range of existing methods, without the need to worry about implementation details or interfaces. Table 2 lists the generative models in synthcity for different data modalities.
184
187
188
Syntheity covers all major families of deep generative models, including Generative adversarial
189
networks (GAN) [oodffellow al. 2020). Variational Autoencoders (VAE) (Kingma et al. 2019).
190
Normalizing flows (NF) (Papamakarios et al. 2021], as well as Diffusson models (DDPM) Kingma
al.
1021]. In the GAN family, Synthcity currently supports GOGGLE [Liu et al. 2023). CTGAN
Xu et al. 2019b). DPGAN [Xie et al.
2018 . PATEGAN [Joddon et al. 2019] ADSGAN Yoon et al.
193
2020]. DECAF (van Breugel et al.) 2021 for static data, Survival GAN [Nocllilte et al. 2023 for
194
censored data, TimeGAN (Yoon et al. 2019 for time series data, as well as RadialGAN Yoon et al.
195
2018 for multi-souree data. In the VAE family, it supports TVAE [x et al. 2019b). RTVAE for static Akrami et al., 2020), Survival VAE [Noreliffe et al. 2023 for censored data, and VAE Yoon al. 2019] for time series data. In the NF family, Synthcity implements the standard NF Papamakarios et al. 2021] for static data. Survival NF [Norcliffe al. 2023] for censored data,
196
data
197
and FourierFlow Alaa et al. 2021 of time series data. Syntheity also includes the TabDDPM
5

---

Privacy
Fairness Causal
Data Modality
Model
Standard Gen
DP
TM
Balance
Bayesian Net
NF
GREAT
ARF
GOGGLE
TabDDPM
Static
TVAE
RTVAE
CTGAN
AIM
PrivBayes
DPGAN
PATEGAN
ADSGAN
J
DECAF
Survival GAN
V
V
Static (Censored)
Survival VAE
Survival NF
V
TimeGAN
Time Series (regular, irregular, censored)
VAE
FourierFlow
Probabilistic AR"
Multi-source
RadialGAN
Table 2: Generative models available in synthcity for different data modalities and use cases. Ab- breviations: Differential Privacy (DP), Threat Model (TM). *FourierFlow and Probabilistic AR is compatible with regular time series only while TimeGAN and Time VAE support both.
200
Kotelnikov al. 2022] in the diffusion model family, and GREAT Borisov al. 2022 which uses auto-regressive generative LLM model.
201
In addition to deep generative models, Synthcity also contains generative models that are not based on neural networks, such as Bayesian networks [Heckerman 1997]. AIM [McKenna et al. 2022].
203
204
Probabilistic Auto-regressive models [Deodatis and Shinozuka 1988 and Adversarial random forests
205
(ARF) Watson et al. 2023].
Synthcity implements all generative models using the PlugIn interface. This consistent approach makes it easy to add additional generative models into the benchmark. The GitHub repository includes tutorials and step-by-step instructions on how to add new models.
2.4
Architecture and hyper-parametert
210
To help researchers pinpoint the source of performance gain and conduct fair comparison, Synthcity allows the user incarnate all the deep generative models with different network architectures and hyper-parameters.
211
213
The architecture can be specified when the user creates model instance (i.e. a PlugIn). For example, each time-series generative model can be configured using twelve different architectures, including LSTM Hocchreier and Schmidhuber 1997]. GRU [Dey and Salem 2017), Transformer
214
215
216
Vaswani al. 2017). MLN (Karim et al. 2019]. TCN [Lea et al. 2017], Inception
217
and PiccptionTimePlu [Ismail Fawaz al. 2020 XceptionTime RRahimian
2020 ResCNN
Sun et al.] 2020), Omni-Scale CNN [Tng al. 2020], and XCM (Fauvel et al. 202 The network
architectures compatible with other data modalities are tabulated in the Appendix.

---

220
Synthcity also has a consistent interface for dealing with hyper-paramenn parameters. The library allows generative model. Furthermore, optimization libraries, such as Optuna Akiba et al. 2019]. J. In this way, Syntheity allows user to perform hyper-parameter search before evaluating on the best-performing setting to ensure like-for-lin comparison. Furthermore, Syntheity also allows the user to configure various early stopping rules to control and compare the training of generative models.
221
the user to list, set, and sample all relevant hyper-parameter
222
this interface is compatible with all popular
223
224
227
2.5
Data modalities
228
We emphasize that "tabular data" in fact encapsulates many different data modalities, including static tabular data, time series data, and censored survival data, all of which may contain a mix of 229
230
continuous and discrete features (columns). Synthcity can also handle composite datasets composed
231
of multiple subsets of data. We give detailed description of the diverse tabular data modalities Synthcity supports in Figure and and further discuss them below. In future versions, plan to include more data modalities including relational database-style data, richly annotated images, and texts.
234
2.5.1
Single dataset
235
We start by introducing the most fundamental case where there is single training dataset (e.g. single DataFrame in Pandas). We characterize the data modalities by two axes: the observation pattern and the feature type. Synthcity supports all combinations.
236
237
238
The observation pattern describes whether and how the data are collected over time. There are three most prominent patterns, static data, regular time series, and irregular time series, which are all supported by synthcity.
241
The second axis, feature type, describes the do- main of individual features. Synthcity supports multivariate tabular data with mixtures of con- tinuous, categorical, integer, and censored fea- tures. Censored features common in survival analysis applications (e.g. healthcare and insur- ance). They are represented as a tuple (z, c). where x E R represents the survival time and {0, 1} the censoring indicator.
242
Tabular Data
243
244
Single
Composite datacet
Multi-source
247
Obseevation Pattern
Feature Type
Static Time-series Etc.
248
249
Static data Regular time series Irregular series
Continuous Integer Categorical Censored
250
2.5.2
Composite dataset
composite dataset involves multiple sub datasets. Synthcity handle the benchmark- ing of different classes of composite datasets. Currently, it supports (1) static datasets with the same features, collected from different domains, (2) static and time series dataset. The latter setting is common in applications. For example, a patient's medical record may contain both static demographic information and longitudinal follow up data.
Figure 2: Supported tabular data modalities.
255
256
257
Comparison with existing libraries
258
259
In this section, we compare synthcity with other popular open source libraries for synthetic data generation to demonstrate its suitability as a comprehensive benchmark framework. Here we only consider the libraries that can generate synthetic data while preserving the statistical properties of
280
261
282
real data, which includes YData Synthetic Gretel Syntheties] SDV) DataaSythesizerr Smart and nbsynthetic. Libraries that generate "fake" data for software testing not considered because they do not attempt learn the distribution of real data.
7

---

Setting \ Software
Synthcity
YData
Gretel
SDV
DataSynthesizer
SmartNoise
nbsynthetic
Data modalities
Static data
Regular time series
Iregular time series
Censored features
Composite data
V
Use cases
Generation
VA
V
Faimesss (balance)
V
Fairness (causal)
Privacy (DP)
V
Privacy (TM)
Cross domain aug.
Table 3: The data modalities and use cases supported by synthcity and other open source synthetic data libraries. Comparisons are based on the software versions available at the time of writing.
285
Table shows that synthcity supports much broader use cases and data modalities than the alternatives. The existing libraries often focus single data modality
because they are intended solution to specific problem rather than benchmark framework. Furthermore, Synthcity includes many more data generators, including all major flavors of deep generative models as well as traditional generative models. It also contains a built-in evaluation module that assesses various aspects of the generator. more detailed comparison of the supported data generators and evaluation metrics are available in the Appendix. The broad coverage of data modalities, use cases, data generators and evaluation metries make Syntheity unique in its capacity for model evaluation and comparison.
268
269
270
271
272
4
Illustrative case studies
273
274
In this section, we present two illustrative use cases to show the type of benchmark studies that Synthcity can facilitate. We stress that these examples do not cover the full capability of Synthcity and they are used as illustrations.
275
278
277
4.1
Static tabular data generative model benchmark
We study which generative model has the strongest performance in generating synthetic tabular data. Synthcity allows us to compare variety of state-of-the-an algorithms in this study, including ARF, GOGGLE, TabDDPM, CTGAN and TVAE. These algorithms are representative of broader families of generative models such as GANs, VAEs, Diffusion models, and forest-based generative models.
279
280
281
Similar to prior tabular data benchmarks Grinsstajj al. 2022 have selected 18 datasets from the OpenML benchmark, which cover common regression and classification datasets encountered in data science projects [Vanschoren et al. 20144). The datasets cover a range of sample sizes (4,209 to 1 1,025,010) and feature counts (5 to 771).
285
286
Synthcity can automatically calculate more than 25 supported evaluation metrics benchmark. In this study, we focus on evaluating the fidelity of synthetic data. Similar to Liu al. 20033 we report the average of the three-dimensional metrics (a-precision, 3-recall, and authenticity). proposed in measure of data quality- the synthetic data realistic, cover the
287
Alaa al. [2022). true data distribution, and are generalized. Furthermore, we report the detection score, which reflects how often the synthetic data can be distinguished from the real data. To reduce the variability from
290
281
282
the classifiers, we report the average AUROC scores from three different post-hoe data classifiers, as Liu et al. [0033].
293
Table shows the experimental results averaged across all the datasets. We observe that the ARF model achieves strong performance in the quality score and stands out in terms of the detection score.

---

296 This suggests that the tree-based generative models are strong competitors to deep generative models
287 for static tabular data. And this area a promising avenue for further research.
Quality
Detection
ARF
0.5475
0.6721
GOGGLE
0.4054
0.9261
TabDDPM
0.5436
0.7074
CTGAN 0.5475
0.7758
TVAE
0.5487
0.7389
Table 4: Benchmark results for static tabular data generation. Quality: the higher the better; Detection: the lower the better.
298
4.2 Tabular data fairness and augmentation benchmark
We consider benchmark study cross-domain data augmentation for improving predictive perfor- mance on minority groups. We use the SIVEP-Gripe public dataset as an illustrative example. which contains anonymized records of COVID-19 patients in Brazil [Baqui et al.) 2021]. In this dataset, Mixed' and 'White" are the majority ethnicity groups while 'Black', 'East Asian" and 'Indigenous' are the minority groups (accounting for less than 10% of the total population). The dataset used for training a downstream model to predict COVID-19 mortality. Due the distributional imbalance, the downstream predictor likely under- perfo-p the minority groups, which may raise faimess issues (Section) 2.2.3) This study aims to benchmark the utility of different generative models for data augmentation by measuring the AUROC of mortality prediction on the minority groups.
300
301
302
303
304
307
308
Synthcity allows us to easily compare RadialGAN, which was designed for cross-domain data augmentation, and the conditional generative models (TabDDPM. CTGAN, and TVAE). We use Synthcity's 's pre-configured pipeline for data augmentation benchmark, which reduces the amount of code and prevents data leakage. Synthcity also allows to evaluate the performance gain for different downstream models, and we have selected multi-layer perceptron classifier and xgBoost classifier.
309
313
The results are listed in Table 5 We observe that data augmentation consistently improves the accuracy of mortality prediction for minority groups. TabDDPM, a novel diffusion model, achieves
314
315
best overall performance, followed by RadialGAN.
Neural net XgBoost
TabDDPM
0.7241
0.7786
RadialGAN
0.7137
0.7627
CTGAN
0.6477
0.7507
TVAE
0.3623
0.7794
Baseline
0.3244
0.7327
Table 5: Benchmark results for cross domain data augmentation. The metric reported is the AUROC of mortality prediction on the minority groups
Discussion
316
317
Synthetic data is an emerging field where many novel algorithms have been proposed; yet there lacks an easy way to benchmark generative models across different desired or off label use cases, compare them with diverse baselines, and explain their performance gain. this work, we present the open source Synthcity library as solution the benchmark challenge. Syntheity contains many built-in generative models, architectures and evaluation metries, which are easily accessible through end-to-end evaluation pipelines. It can help researchers to perform in-depth and comprehensive benchmark studies with minimal programming effort.
318
319
320

---

324
6
Funding Statement
325
This work is supported by Cancer Research UK. The computational experiments in this work was supported by Azure sponsorship credits granted by Microsoft's 's Al for Good Research Lab.
326
7 Acknowledgment
327
328
We thank all the open source contributors to the Synthcity library. The full list of contributors is available on GitHub Special thanks goes to Bogdan Cebere, who made major contributions to Synthcity while he was working as a Research Engineer at the van der Schaar Lab.
329
330
References
331
332
Martín Abadi, Andy Chu, Ian Goodfellow, H Brendan Mcmahan, Ilya Mironov, Kunal Talwar, and Li Zhang. Deep Learning with Differential Privacy. Proceedings of the 2016 ACM SIGSAC Conference on Computer and Communications Security, 2016. doi: 10. 10.1145/2976749 URL
333
334
335
http://dx.doi: doi. 10. 1e111111117744 2978318
336
Takuya Akiba, Shotaro Sano, Toshihiko Yanase, Takeru Ohta, and Masanori Koyama. Optuna: A next-generation hyperparameter optimization framework. In Proceedings of the 25th ACM SIGKDD international conference on knowledge discovery data mining, pages 2623 2623-2631,
337
338
339
2019.
340
Haleh Akrami, Sergul Aydore, Richard M Leahy, and Anand A Joshi. Robust variational autoencoder for tabular data with beta divergence. arXiv preprint arXiv:2006. 082004 2020.
341
342
Ahmed Alaa, Alex James Chan, and Mihaela van der Schaar. Generative time-series modeling with fourier flows International Conference on Learning Representations, 2021.
343
344
A Ahmed Alaa, Boris Van Breugel, Evgeny S Saveliev, and Mihaela van der Schaar. How faithful is your synthetic data? sample-level metries for evaluating and auditing generative models. In International Conference on Machine Learning, pages 290-306. PMLR, 2022.
345
346
347
Antreas Antoniou, Amos Storkey, and Harrison Edwards. Data Augmentation Generative Adversarial Networks. 11 2017. httt:://t:/://ii wttpaaawvv...oggbaaa////aaaa www./ra///////eeeeoo 1111. 17....4411.....011
348
349
Pedro Baqui, Valerio Marra, Ahmed M Alaa, Ioana Bica, Ari Ercole, and Mihaela van der Schaar. Comparing covid-19 risk factors in brazil using machine learning: the importance of socioeconomic, demographic and structural factors. Scientific reports, 11(1):1-10, 2021.
350
351
352
Brett K Beaulieu-Jones, Zhiwei Steven Wu, Chris Williams, Ran Lee, Sanjeev P Bhavnani, James Brian Byrd, and Casey S Greene. Privacy-preservve generative deep neural networks support clinical data sharing. Circulation: Cardiovascular Quality and Outcomes, 12(7):e005122,
353
354
355
2019.
356
Simon Bing, Andrea Dittadi, Stefan Bauer, and Patrick Schwab. Conditional Generation of Medical Time Series for Extrapolation to Underrepresented Populations. 2022.
357
358
Sam Bond-Taylor, Adam Leach, Yang Long, and Chris G. Willcocks. Deep Generative Modelling:
359
A Comparative Review of VAES, GANs, Normalizing Flows, Energy-Based and Autoregressive Models. IEEE Transactions on Pattern Analysis and Machine Intelligence, 3 2021. doi: 10.1 htttt::::///////// www.aaiww.aa...eeeeee.re.eoowele abs 2103 w4w.::1...9.t:..////1 doi. 1r1/0
360
TPAMI.2021.3116660
1109/TPAMI TPAMI 2022 3116668
363
Vadim Borisov, Kathrin Seßler, Tobias Leemann, Martin Pawelczyk, and Gjergji Kasneci. Language models are realistic tabular data generators. arXiv preprint arXiv:2210.0622 00.02880 2022.
364
365
Hari Prasanna Das, Ryan Tran, Japjot Singh, Xiangyu Yue, Geoffrey Tison, Alberto Sangiovanni- Vincentelli, and Costas J Spanos. Conditional synthetic data generation for robust machine learning applications with limited pandemic data. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 36, pages 1 11792-11800 11000 2022.
366
10

---

389
Jeffrey Dastin. Amazon scraps secret Al recruiting tool that showed bias against women. Reuters, 2018.
370
Daniel de Vassimon Manela, David Errington, Thomas Fisher, Boris van Breugel, and Pasquale Minervini. Stereotype and Skew: Quantifying Gender Bias in Pre-trained and Fine-tuned Language
372
373
Models. In Proceedings of the 16th Conference of the European Chapter of the Association for
374
Computational Linguistics: Main Volume, pages 2232-2242, Online, 4 2021. Association for
375
Computational Linguistics. URL httphtt//s:www http:///wwwwwwww..ccco aclweb. htw.a//.rrg/aaaoooo///////// http://2221ee111e--e.1111 eacl-main
378
190
377
George Deodatis and M Shinozuka. Auto-segressive model for nonstationary stochastic processes.
378
Journal of engineering mechanics, 114(11):195-2201,, 1988.
379
Rahul Dey and Fathi M Salem. Gate-variants of gated recurrent unit (gru) neural networks. In 2017
380
IEEE 60th international midwest symposium on circuits and systems (MWSCAS), pages 1597-1600.
IEEE, 2017.
382
Ayesha S. Dina, A. B. Siddique, and D. Manivannan. Effect of Balancing Data Using Synthetic Data the Performance of Machine Learning Classifiers for Intrusion Detection in Computer Networks. URL Ut..
4 2022. doi: 10. 00
Irit Dinur and Kobbi Nissim. Revealing information while preserving privacy. In Proceedings of the twenty-second ACM SIGMOD-SIGACTT-SIGAI symposium on Principles of database systems, pages 202-210, 2003.
387
Cynthia Dwork. Differential privacy: A survey of results. In International conference theory and applications of models of computation, pages 1-19. Springer, 2008
389
Kevin Fauvel, Tao Lin, Véronique Masson, Élisa Fromont, and Alexandre Termier. Xem: An explainable convolutional neural network for multivariate time series classification. Mathematics, 9(23):3137, 2021.
392
393
Georgi Ganev, Bristena Oprisanu, and Emiliano De Cristofaro. Robin hood and matthew effects: Differential privacy has disparate impact on synthetic data. In International Conference on Machine Learning, pages 6959-6959. PMIR, 2022.
394
385
396
Milena A Gianfrancesco, Suzanne Tamang, Jinoos Yazdany, and Gabriela Schmajuk. Potential biases in machine learning algorithms using electronic health record data. JAMA internal medicine, 178
397
398
(1 (11):1544-1547, 2018.
389
Andre Goncalves, Priyadip Ray, Braden Soper, Jennifer Stevens, Linda Coyle, and Ana Paula Sales. Generation and evaluation of synthetic patient data. BMC medical research methodology, 20(1):
400
401
40, 2020.
402
Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial networks. Communications of the ACM, 63(11):139-144, 2020.
405
404
405
Arthur Gretton, Karsten M Borgwardt, Malte J Rasch, Bernhard Schölkopf, and Alexander Smola. kernel two-sample test. The Journal of Machine Learning Research, 13(1):723-773, 2012.
406
407
Leo Grinsztajn, Edouard Oyallon, and Gael Varoquaux. Why do tree-based models still outperform deep learning on typical tabular data? In Thirty-sixth Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2022.
408
410
Lasse Hansen, Nabeel Seedat, Mihaela van der Schaar, and Petrovic Andrija. Reimagining syn- thetic data generation through data-centric ai: A comprehensive benchmark. Advances in neural information processing systems, 30. 2023.
413
David Heckerman. Bayesian networks for data mining, Data mining and knowledge discovery, 79-17 119, 1997.
415 Stella Ho, Youyang Qu, Bruce Gu, Longxiang Gao, Jianxin Li, and Yong Xiang. Dp-gan: Differen- tially private consecutive data publishing using generative adversarial nets. Journal of Network and Computer Applications, 185:103066, 2021.

---

418
Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural computation, 9()):
419
1735-1780, 1997.
420
Hassan Ismail Fawaz, Benjamin Lucas, Germain Forestier, Charlotte Pelletier, Daniel F Schmidt, Jonathan Weber, Geoffrey I Webb, Lhassane Idoumghar, Pierre-Alain Muller, and François Petít- jean. Inceptiontime: Finding alexnet for time series classification. Data Mining and Knowledge Discovery, (134(66)966-193
421
422
425
2020.
424
James Jordon, Jinsung Yoon, and Mihaela Van Der Schaar. Pate-gan: Generating synthetic data with differential privacy guarantees, In International conference on learning representations, 2018.
425
426
James Jordon, Jinsung Yoon, and Mihaela Van Der Schaar. Pate-gan: Generating synthetic data with differential privacy guarantees. In International conference on learning representations, 2019.
427
428
James Jordon, Lukasz. Szpruch, Florimond Houssiau, Mirko Bottarelli, Giovanni Cherubin, Carsten Maple, Samuel N. Cohen, and Adrian Weller. Synthetic Data what, why and how? 5 2022. doi: 10.485500amxx..220002222 0.3255. URL https://arxii... wwtpa/iii.rr//ssss.ssresss
429
430
431
Achuta Kadambi. Achieving fairness in medical devices. Science, 372(6537):30 31, 2021.
432
Fazle Karim, Somshubra Majumdar, Houshang Darabi, and Samuel Harford. Multivariate Istm-fens for time series classification. Neural networks, 11::237-245, 2019.
433
434
Diederik Kingma, Tim Salimans, Ben Poole, and Jonathan Ho. Variational diffusion models. Advances
435
in neural information processing systems, 34:21696-21707 2021.
436
Diederik P Kingma, Max Welling, et al. An introduction to variational autoencoders. Foundations
437
and Trends in Machine Learning, 3774):307-392, 2019.
438
Theodora Kokosi and Katie Harron. Synthetic data in medical research. BMJ Medicine, 1(I), 2022.
Michal Kosinski, David Stillwell, and Thore Graepel. Private traits and attributes are predictable from digital records of human behavior. Proceedings of the national academy of sciences, 110(15): 5802-5805, 2013.
440
441
442
Akim Kotelnikov, Dmitry Baranchuk, Ivan Rubachev, and Artem Babenko. Tabddpm: Modelling tabular data with diffusion models. arXiv preprint arXiv:2209. 15421, 2022.
443
444
Colin Lea, Michael D Flynn. Rene Vidal, Austin Reiter, and Gregory D Hager. Temporal convolutional networks for action segmentation and detection. In proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 156-165, 2017.
445
446
447
Tennison Liu, Zhaozhi Qian, Jeroen Berrevoets, and Mihaela van der Schaar. Goggle: Generative modelling for tabular data by learning relational structure. In The Eleventh International Conference on Learning Representations, 2023.
448
449
450
Yunhui Long, Boxin Wang, Zhuolin Yang. Bhavya Kailkhura, Aston Zhang, Carl A. Gunter, and Bo Li. G-PATE: Scalable Differentially Private Data Generator via Private Aggregation of Teacher Discriminators. In Advances in Neural Information Processing Systems, 6 2019. doi: 10.48550/
451
452
arxiv. 190.09396.. URL/https:: htr/arxvvvvvrr... org/ httpg/abb/1990..9933 http:pp:/.aaai..orrr/aa/////0... 09338v2
454
David Lopez-Pazz and Maxime Oquab. Revisiting classifier two-sample tests. arXiv preprint arXiv: /5f.:66610065455 2016.
455
456
Kaiji Lu, Piotr Mardziel, Fangjing Wu, Preetam Amancharla, and Anupam Datta, Gender Bias in Neural Natural Language Processing. CoRR, abs/1807.11714, 2018. URL http:://axii......// org/
abs/ 11714
459
Ashwin Machanavajjhala, Daniel Kifer, Johannes Gehrke, and Muthuramakrishnan Venkitasubrama- niam. I-diversity: Privacy beyond k-anonymity. ACM Transactions Knowledge Discovery from Data (TKDD), 1(1):3ess, 2007.
462
Ryan McKenna, Brett Mullins, Daniel Sheldon, and Gerome Miklau. Aim: An adaptive and iterative mechanism for differentially private synthetic data. arXiv preprint vrXiv:2201.126/1 2022.
Ninarch Mehrabi, Fred Morstatter, Nripsuta Saxena, Kristina Lerman, and Aram Galstyan. A survey on bias and fairness in machine learning. ACM Computing Surveys (CSUR), 54(6)1135, 2021.
12

---

488
Alexander Norcliffe, Bogdan Cebere, Fergus Imrie, Pietro Lio, and Mihaela van der Schaar. Survival- gan: Generating time-to-event data for survival analysis. In International Conference on Artificial Intelligence and Statistics, pages 10279-10304. PMLR, 2023.
467
468
George Papamakarios, Eric Nalisnick, Danilo Jimenez Rezende, Shakir Mohamed, and Balaji Lakshminarayanan. Normalizing flows for probabilistic modeling and inference. The Journal of Machine Learning Research, 22( 11 262172266., 2021.
470
471
Mayana Pereira, Meghana Kshirsagar, Sumit Mukherjee, Rahul Dodhia, and Juan Lavista Ferres. An analysis of the deployment of models trained on private tabular synthetic data: Unexpected surprises. arXiv preprint Stti::2006.124441 2021.
474
475
Elahe Rahimian, Soheil Zabihi, Seyed Farokh Atashzar, Amir Asif, and Arash Mohammadi. Xcep- tiontime: independent wineww xceptiontime architecture for hand gesture classification. In ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 1304-1308. IEEE, 2020.
478
479
Nithya Sambasivan, Shivani Kapania, Hannah Highfill, Diana Akrong, Praveen Paritosh, and Lora M Aroyo. "everyonee wants to do the model work, not the data work": Data cascades in high- ai. In proceedings of the 2021 CHI Conference on Human Factors in Computing Systems, pages 1-15, 2021.
482
483
Divya Saxena and Jiannong Cao. Generative adversarial networks (gans) challenges, solutions, and future directions. ACM Computing Surveys (CSUR), 54(3):1-42 2021.
485
Reza Shokri, Marco Stronati, Congzheng Song, and Vitaly Shmatikov. Membership inference attacks against machine learning models. In 2017 IEEE symposium on security and privacy (SP), pages 3-18. IEEE, 2017.
486
487
488
Joshua Snoke, Gillian M Raab, Beata Nowok, Chris Dibben, and Aleksandra Slavkovic. General and specific utility measures for synthetic data. Journal of the Royal Statistical Society: Series (Statistics in Sociery), 181(3):663-68) 2018.
489
490
481
Weitong Sun, Yuping Su, Xia Wu, and Xiaojun Wu. A novel end-to-end ld-rescnn model to remove artifact from eeg signals. Neurocomputing, 404:108-124 121, 2020.
492
Latanya Sweeney. k-anonymity: A model for protecting privacy. International journal of uncertainty, fuzziness and seowledge-based systems, 10(05):557-570, 2002.
494
485
Wensi Tang, Guodong Long, Lu Liu, Tianyi Zhou, Michael Blumenstein, and Jing Jiang. Omni-scale
cnns: simple and effective kernel size configuration for time series classification. arXiv preprint
2000010002..061,,,, 2020.
496
Jason
Tashea
Courts Are Using AI Now. WIRED, 4 2017. URL. https: /ttps:://ww.... www.wwired.cmm com/2017/04/
Sentence Criminals.
That Must
Stop
courts-using usw-u-si-nneeeeeeeee www.seeetenne-ciiinansssss-ss-se stousststop-oow
501
Allan Tucker, Zhenchen Wang, Ylenia Rotalinti, and Puja Myles. Generating high-fidelity synthetic patient data for assessing machine learning healthcare software. NPJ digital medicine, 3(1):1-13, 2020.
502
503
504
Boris van Breugel, Trent Kyono, Jeroen Berrevoets, and Mihaela van der Schaar. Decaf: Generating fair synthetic data using caus generative networks. Advances Neural Information Processing Systems, 54:22221-22233. 2021.
505
508
507
Boris van Breugel, Zhaozhi Qian, and Mihaela van der Schaar. Synthetic data, real errors: how (not) to publish and use synthetic data. 2023.
508
Joaquin Vanschoren, Jan N Van Rijn, Bernd Bischl, and Luis Torgo. Openml: networked science in machine learning. ACM SIGKDD Explorations Newsletter, 15(2):49 60, 2014.
510
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017.
13

---

514
Zhenchen Wang, Puja Myles, and Allan Tucker. Generating and evaluating synthetic uk primary care data: preserving data utility & patient privacy. In 20/9 IEEE 32nd International Symposium on Computer-Bass Based Medical Systems (CBMS), pages 126-131. IEEE, 2019.
515
516
517
Zhenchen Wang, Puja Myles, and Allan Tucker. Generating and evaluating cross-sectional synthetic electronic healthcare data: Preserving data utility and patient privacy. Computational Intelligence, 37(2):819-88 851, 2021.
518
519
520
David S Watson, Kristin Blesch, Jan Kapar, and Marvin N Wright. Adversarial random forests for density estimation and generative modeling. In International Conference on Artificial Intelligence and Statistics, pages 5357-5375. PMLR, 2023.
521
522
523
L Liyang Xie, Kaixiang Lin, Shu Wang, Fei Wang, and Jiayu Zhou. Differentially private generative adversarial network. arXiv preprint arXiv: 06739, 2018.
524
525
Depeng Xu, Shuhan Yuan, Lu Zhang, and Xintao Wu. Fairgan: Fairness-aware generative adversarial networks. In 2018 IEEE International Conference on Big Data (Big Data), pages 570-575, 2018.
526
527
Depeng Xu, Yongkai Wu, Shuhan Yuan, Lu Zhang, and Xintao Wu. Achieving causal fairness through generative adversarial networks. In IJCAI International Joint Conference on Artificial Intelligence, volume 2019-August, pages 1452-1458. International Joint Conferences on Artificial Intelligence, 2019a. ISBN 9780999241141. doi: 10.24963/ijeai22199200
528
529
Lei Xu, Maria Skoularidou, Alfredo Cuesta-Infante, and Kalyan Veeramachaneni. Modeling tabular data using conditional gan. Advances in Neural Information Processing Systems, 32, 2019b.
533
Jinsung Yoon, James Jordon, and Mihaela Schaar. Radialgan: Leveraging multiple datasets to improve target-specifie predictive models using generative adversarial networks. In International Conference Machine Learning, pages 5699-5707. PMLR, 2018
Jinsung Yoon, Daniel Jarrett, and Mihaela Van der Schaar. Time-series generative adversarial networks. Advances in neural information processing systems, 32, 2019.
Jinsung Yoon, Lydia N Drumright, and Mihaela Van Der Schaar. Anonymization through data synthesis using generative adversarial networks (ads-gan). IEEE journal of biomedical and health informatics, 24(8):2378-23) 2388,, 2020.
540
Rich Zemel, Yu Wu, Kevin Swersky, Toni Pitassi, and Cynthia Dwork. Learning fair representations. In International conference on machine learning, pages 325-333 2013.
542
Jun Yan Zhu, Taesung Park, Phillip Isola, and Alexei A Efros. Unpaired image-to-image translation using cycle-consistent adversarial networks. In Proceedings of the IEEE international conference on computer vision, pages 2223-2232, 2017.
544
545
14

---

548
A
Appendix
547
A.1
Code availability
The code for the illustrative
cases
available GitHub The synthcity library is avail-
550
able on pip and GitHub The tutorials folder contains additional illustrative examples.
551
A.2
Supported algorithms and metrics
Aspect
Evaluation SSofware
Synthcity
YDtt
Gretel
SDV
DataSynthesizr
SmariNNose
nbsynthetic
Jensen-Shannoe Shannon distance
Wasserstein distance
Total variation distance
KL divergence
Skewness
Max-mean discrepancy
KS test
Fodelity
PRDC
Alpha-precision precision
Survival Kaplan-Meii Meier dist.
Delection: linear
Detection: NN
Detection: XGB
Detection: Linear
Linear model
MLP
XGBoost
Utility
survival
Time-scries
Survival time-series
Correct attribution prob.
K-anonymity
K-map
Privacy
Delta-presence
L-diversity
DOMIAS
Identifiability score
Table The evaluation metries supported by syntheity and other open source synthetic data libraries. Comparisons are based on the software versions available at the time of writing.
Static
Censored
Time Series
Fully connected
Weibull AFT
LSTM
Residual network
Cox PH
GRU RNN
TabNet
Random Survival Forest
Survival Xgboost
Transformer
Deephit
MLSTM_ FCN
Tenn
TCN
Date
Inception Time
InceptionTimePlus
Xception Time
ResCNN
OmniScaleCNN XCM
Table 7: Available network architectures and survival models in synthcity for different data modalities. These components are compatible with multiple algorithms.
15

---

Algorithm ISoftware
Syntheity
YData
Gretel
SDV
DataSynthesizer
SmartNoise
nbsynthetic
AIM
GREAT
TabDDPM
ARF
GOGGLE
CTGAN
ACTGAN
TVAE
Bayesian Network
Normalizing Flows
Survial GAN
Survival VAE
DoppelGANger
TimeGAN
FourierFlows
Probabilistic AR
DECAF
RadialGAN
ADSGAN
DPGAN
PATEGAN
PrivBayes
Table 8: The data generating algorithms supported by synthcity and other open source synthetic data libraries. Comparisons are based on the software versions available at the time writing.
16


Reviews:
[{"review_id": "9QSm4blpn9", "reviewer": "Reviewer_T6Ey", "summary": "", "questions": "", "limitations": "Library -\n1. I tried two use cases (time series and tabular data generation) provided on their GitHub examples and despite following their code as is, I ran into errors and could not use the library for training models. Therefore, my first impression is that the library is not stable.\n2. The documentation is not well-written except for the first page and the GitHub README file.\n\nWriting - \n1. The paper can be significantly improved by organizing it in a more structured manner and improving the writing. To highlight a few issues - \nAt L65: \"to assess various aspects\" is too vague and could mean anything. The contribution paragraph generally could be improved by quantifying contributions concretely.\nAt L123: \"Synthcity can benchmark this mode of cross-domain generation via the synthetic data’s utility to downstream tasks.\" Why is that special to synthcity? Anyone can evaluate any benchmark by using synthetic data's utility for downstream tasks\nAt L138: \"Preventing mistakes such as data leakage\" I am not sure why preventing data leakage becomes particularly relevant for cross domain generation?\n\nIllustrative case studies do not appear to be comprehensive enough. The supplementary shows support for a lot of different models and techniques however, none of them have been evaluated in the paper. The same holds true for privacy benchmarking.", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "The main strength of this work is the vastness of functionality it supports such as domain augmentation, privacy, fairness etc.\nI am not aware of any prior works that support multiple such modalities in an API-like format. Therefore, if done well, Synthcity could foster research progress in synthetic data generation.\nThe authors have done a great job by including tutorials that enable contributors or users to get onboarded quickly.", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ""}, {"review_id": "m1ofC3spvZ", "reviewer": "Reviewer_qioZ", "summary": "", "questions": "", "limitations": "Authors addressed this point.", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "- **Diverse use cases**: The library covers a variety of different use-cases from evaluating synthetic data quality to privacy evaluations of generated data. \n- **Reasonable library structure**: The library structure 'load, train, generate, evaluate' appears to be reasonable. I like that the authors allow for meta-data to be supplied for downstream evaluations. How difficult is it for other researchers to include new models and/or evaluation measures?\n", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ""}, {"review_id": "IyTJvL5Dac", "reviewer": "Reviewer_rLVX", "summary": "", "questions": "", "limitations": "See \"Opportunities For Improvement\", the paper didn't include some of the state-of-the-art approaches.", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "The GitHub documentation of Synthcity is comprehensive, offering a wealth of resources including tutorials that facilitate quick learning. The library's APIs are user-friendly, allowing for effortless integration of new algorithms. Notably, the benchmark encompasses a diverse range of algorithms that are tailored to handle various data types such as tabular data, time series data, and images. Additionally, the benchmark provides an extensive array of metrics, enabling thorough comparisons of model performance with regard to privacy and fairness considerations. Overall, the strength of this library lies in its accessibility, versatility, and its ability to foster comprehensive evaluations and comparisons of synthetic data generation models.", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ""}]

Author rebuttal:

Meta-review: The presented library has a lot of tools to generate synthetic data, and do so with ease. The API appears to be quite generic and can have wide impact. All reviewers are unanimously in favor of the paper, and hence I recommend acceptance.
Decision: Accept (Poster)