SlowGuess/NeurIPS_2023_OCR huggingface dataset:
zq224@cam.ac.uk
Rob Davis
rd473@cam.ac.uk
Accessible high-quality data is the bread and butter of machine learning research, and the demand for data has exploded as larger and more advanced ML models are built across different domains. Yet, real data often contain sensitive information, subject to various biases, and are costly to acquire, which compromise their quality and accessibility. Synthetic data have thus emerged as a complement, sometimes even a replacement, to real data for ML training. However, the landscape of synthetic data research has been fragmented due to the large number of data modalities (e.g., tabular data, time series data, images, etc.) and various use cases (e.g., privacy, fairness, data augmentation, etc.). This poses practical challenges in comparing and selecting synthetic data generators in different problem settings. To this end, we develop Synthcity, an open-source Python library that allows researchers and practitioners to perform one-click benchmarking of synthetic data generators across data modalities and use cases. In addition, Synthcity's plug-in style API makes it easy to incorporate additional data generators into the framework. Beyond benchmarking, it also offers a single access point to a diverse range of cutting-edge data generators. Through examples on tabular data generation and data augmentation, we illustrate the general applicability of Synthcity, and the insight one can obtain.
Access to high quality data is the lifeblood of AI. Although AI holds strong promise in numerous high-stakes domains, the lack of high-quality datasets creates a significant hurdle for the development of AI, leading to missed opportunities. Specifically, three prominent issues contribute to this challenge: data scarcity, privacy, and bias [Mehrabi et al., 2021, Gianfrancesco et al., 2018, Tashea, 2017, Dastin, 2018]. As a result, the dataset may not be available, accessible, or suitable for building performant and socially responsible AI systems [Sambasivan et al., 2021].
This challenge is especially prominent for tabular datasets, which are often curated in highly regulated industries including healthcare, finance, manufacturing etc. Synthetic tabular data has the potential to fuel the development of AI by unleashing the information in datasets that are small, sensitive or biased. To achieve this, we need high-performance generative models that both faithfully capture the data distribution and satisfy additional constraints for the desired use cases.
To date, the landscape of synthetic data research has been fragmented because the combination of use cases (i.e. fairness, privacy, and augmentation) and data modalities (e.g. static tabular data, time series data, etc.) creates a plethora of problem settings. In response to the large problem space, the
1   
2   
3   
4   
5   
6   
7   
8   
9   
10   
11   
12   
13   
14   
15   
16   
17   
18   
19   
20   
21   
22   
23   
24   
25   
26   
27   
28   
29   
30   
31   
32   
33
community has taken a divide-and-conquer approach: highly-specialized generative models have been developed to fit in one particular setting. This has led to a proliferation of specialized generative models [Jordon et al., 2018, Yoon et al., 2020, Ho et al., 2021, Mehrabi et al., 2021, van Breugel et al., 2021, Zhu et al., 2017, Yoon et al., 2018, Saxena and Cao, 2021].
This fragmented landscape has created four main challenges for benchmarking synthetic data generators, which would hamper the research progress if left unaddressed.
1. Challenge in use case specific evaluation. Most existing studies in generative model only focus on the fidelity of the synthetic data, i.e. how they resemble the real data in distribution Wang et al. [2019], Tucker et al. [2020], Goncalves et al. [2020], Wang et al. [2021], Kokosi and Harron [2022]. However, additional evaluation is needed to assess the specific use cases. For example, the utility to downstream models and the data privacy. This calls for the introduction of new metrics as well as new evaluation pipelines.
2. Challenge in off label uses. Although specialized generative models are developed for one use case, the practical application often requires them to cover multiple use cases (e.g. data augmentation with privacy). Hence, generative models are often used outside the designed scope. Prior work has shown that this may lead to undesirable and unpredictable consequences [Pereira et al., 2021, Ganev et al., 2022]. As a result, researchers need to comprehensively evaluate the generative model across a variety of use cases to assess the risk of off label uses.
3. Challenge in comparing with a large number of baselines. In practice, it is often very challenging to systematically compare with a large number of existing baselines because the interfaces (API) of these models are often inconsistent and incompatible (e.g. they may require different formats of input data and conflicting software dependencies). As a result, the researcher usually needs to spend time and effort to harmonize the code rather than focusing on the research question itself.
4. Challenge in understanding the performance gain. Generative models are complex systems that involve many components, such as the model architecture, the objective function, and the hyperparameters. These aspects all encode prior assumptions and inductive biases, which would bring unique strengths and weaknesses to the models [Bond-Taylor et al., 2021]. However, it is often difficult to pinpoint the exact component that leads to the performance gain. Most existing studies evaluate the model as a whole and neglect the role of different components.
Contribution. In this work, we present Synthcity, an open-source Python library available on pip and GitHub, as a solution to these benchmark challenges. Synthcity offers diverse data modalities and supports various use cases. It provides an extensive set of evaluation metrics for assessing dataset fidelity, privacy, and utility, making it a robust tool for evaluating synthetic data across different applications. With a wide array of state-of-the-art generators and customizable architectures, users can perform consistent comparisons with existing models, gaining insights into performance improvements. Accessible through an intuitive interface, Synthcity facilitates tabular data generation and augmentation, demonstrated through two case studies. Researchers can employ Synthcity for benchmarks and guidance in synthetic data research
2 The synthcity library
2.1 Overview of the synthcity workflow
Despite the fragmented landscape in synthetic data research, Synthcity implements a unified workflow for benchmark studies. We formalize the process as follows. Let  $X$  be the random variable of interest (which could be static, temporal or censored). The real data is composed with observations  $x_{i} \sim P(X)$  drawn from the true (but unknown) distribution. For benchmark evaluation, the real data is split into a training  $(\mathbb{D}_{train}^{r})$  and test  $(\mathbb{D}_{test}^{r})$  set. The generator is trained using the training set  $\mathbb{D}_{train}^{r}$ . During training, the generator (explicitly or implicitly) learns the distribution  $\hat{P}(X)$  in order to sample from it. After training, the generative model generates synthetic data  $\mathbb{D}^s$ , which will be evaluated with respect to the test set  $\mathbb{D}_{test}^{r}$  (or in some special cases, the training set  $\mathbb{D}_{train}^{r}$ ).
34   
35   
36   
37   
38   
39   
40   
41   
42   
43   
44   
45   
46   
47   
48   
49   
50   
51   
52   
53   
54   
55   
56   
57   
58   
59   
60   
61   
62   
63   
64   
65   
66   
67   
68   
69   
70   
71   
72   
73   
74   
75   
76   
77   
78   
79   
80   
81
Figure 1: Standard workflow of generating and evaluating synthetic data with synthcity.
The synthcity library captures the entire workflow of synthetic data benchmark in four steps (Figure 1). This workflow applies to all use cases, generative models and data modalities.
1. Loading the dataset using a DataLoader. The DataLoader class provides a consistent interface for loading, storing, and splitting different types of input data (e.g. tabular, time series, and survival data). Users can also provide meta-data to inform downstream algorithms, like specifying sensitive columns for privacy-preserving algorithms.
2. Training the generator using a Plugin. In synthcity, the users instantiate, train, and apply different data generators via the Plugin class. Each Plugin represents a specific data generation algorithm. The generator can be trained using the fit() method of a Plugin.
3. Generating synthetic data. After the Plugin is trained, the user can use the generate() method to generate synthetic data. Some plugins also allow for conditional generation.
4. Evaluating synthetic data. Synthcity provides a large set of metrics for evaluating different aspects of synthetic data. The Metrics class allows users to perform evaluation.
In addition, synthcity also has a Benchmark class that wraps around all four steps. This provides a one-line interface for comparing and evaluating different generators and produces an evaluation report at the end of the process.
2.2 Evaluation for diverse use cases
Synthetic data has many different use cases including fairness, privacy, data augmentation. As a benchmarking framework, Synthcity provides the users with a comprehensive list of metrics and routines to evaluate various aspects of synthetic data, including metrics that are specific to these use cases. In this section, we describe the use cases of synthetic data and how Synthcity performs its evaluation. A full list of metrics can be found in Appendix.
2.2.1 Standard data generation
Standard data generation refers to the most basic generation task, where the synthetic data should be generated as faithfully as possible to the real-data distribution [van Breugel et al., 2023, Hansen et al., 2023]. This is captured by the fidelity metrics.
Fidelity. The fidelity of synthetic data captures how much the synthetic data resembles real data. The fidelity metrics typically evaluate the closeness between the true distribution  $P$  and the distribution learned by the generator  $\hat{P}$  using samples from these two distributions. Synthcity supports distributional divergence measures (e.g. Jensen-Shannon distance, Wasserstein distance, and maximal mean discrepancy) as well as two sample detection scores (i.e. scores that measure how well a classifier can distinguish real versus synthetic data) [Gretton et al., 2012, Lopez-Paz and Oquab, 2016, Snake et al., 2018].
82   
83   
84   
85   
86   
87   
88   
89   
90   
91   
92   
93   
94   
95   
96   
97   
98   
99   
100   
101   
102   
103   
104   
105   
106   
107   
108   
109   
110   
111   
112   
113   
114
Use case	Method	Evaluation	Reference
Standard data generation	Generative model	Fidelity	Gretton et al. [2012]
Cross domain augmentation	Domain transfer	Utility	Bing et al. [2022]
ML fairness	Balancing distribution	Minority performance	Lu et al. [2018]
Causal fairness	Algorithmic fairness	Xu et al. [2018]
Privacy preservation	Differential privacy	Privacy metrics	Abadi et al. [2016]
Threat model	Attack simulation	Shokri et al. [2017]
Table 1: Synthcity is a unified framework to benchmark diverse use cases of synthetic data. It supports a range of methods and evaluation metrics, and also allows evaluation of off label uses.
2.2.2 Cross domain augmentation
Here we consider a dataset that is collected from multiple domains or sources (e.g. data from different countries). Often the practitioner is interested in augmenting one particular data source that suffers from data scarcity issues (e.g. it is difficult to collect data from remote areas) by leveraging other related sources. This challenge has been studied in the deep generative model literature [Antoniou et al., 2017, Dina et al., 2022, Das et al., 2022, Bing et al., 2022]. By learning domain-specific and domain-agnostic representations, the generator is able to transfer knowledge across domains, making data augmentation more efficient. Synthcity offers a clean interface so that the user can benchmark the downstream utility of cross-domain generation using only one line of code..
Utility. Synthcity measures the performance for cross domain augmentation through its utility to downstream tasks. Our approach adapts the common practice of train-on-synthetic evaluate-on-real [Beaulieu-Jones et al., 2019], where a downstream predictive model is trained on fully synthetic training data and then validated on real testing data.
For data augmentation, Synthcity augments the data-scarce domain in the training data  $\mathbb{D}_{train}^r$  with the synthetic data  $\mathbb{D}^s$ . A predictive model is then trained on this augmented dataset and evaluated on the domain of interest in the testing data  $\mathbb{D}_{test}^r$ . Synthcity supports various types of predictive tasks, including regression, classification and survival analysis. In addition to linear predictive models, synthcity supports Xgboost and neural nets as downstream models due to their wide adoption in data analytics. In practice, the user may average the performance of several predictive models to reduce the model uncertainty.
As a naive baseline, Synthcity reports the predictive performance where no data augmentation is performed. Synthcity provides a pre-configured pipeline to automatically handle this entire procedure, reducing the code and preventing mistakes.
2.2.3 Synthetic data for ML fairness
Existing research has considered two different ways where Synthetic data could promote fairness. Table 2 shows the corresponding models in synthcity.
1. Balancing distribution. In this setting, certain groups of people are underrepresented in a dataset that is used for training downstream ML systems. This may lead to a bias being introduced into these ML systems [Lu et al., 2018, de Vassimon Manela et al., 2021, Kadambi, 2021]. As a remedy, one could generate synthetic records for the minority group to augment the real data, thereby achieving balance in distribution. This often requires the data generator to learn the conditional distribution  $P(X|G)$ , where  $G$  is the group label.
2. Causal fairness. The second approach is to generate fairer synthetic data from a biased real dataset and to use synthetic data alone in downstream tasks [Zemel et al., 2013, Xu et al., 2018, 2019a, van Breugel et al., 2021]. In this setting, it is postulated that the real distribution  $P(X)$  reflects existing biases (e.g. unequal access to healthcare). The task for the generator is to learn a distribution  $\hat{P}(X)$  that is free from such biases but also stay as close to  $P(X)$  as possible (to ensure high data fidelity). Typically, notions of causality are employed in the bias removal process.
115   
116   
117   
118   
119   
120   
121   
122   
123   
124   
125   
126   
127   
128   
129   
130   
131   
132   
133   
134   
135   
136   
137   
138   
139   
140   
141   
142   
143   
144   
145   
146   
147   
148   
149   
150   
151   
152
Fairness. Synthcity allows users to benchmark both use cases by training a downstream predictive model on the fully synthetic or augmented data and presenting their performance or characteristics. For example, one can evaluate the performance gain on any specified (minority) group as an indicator of the utility of synthetic data. In addition, Synthcity also supports standard algorithmic fairness metrics for the trained predictive model, such as Fairness Through Unawareness, Demographic Parity and Conditional Fairness [van Breugel et al., 2021]
2.2.4 Synthetic data for privacy
Methods for generating privacy-preserving synthetic data mainly fall into two categories: the ones that employ differential privacy, and the ones that are designed for specific threat models.
1. Differential privacy (DP). DP is a formal way to describe how private a data generator is [Dwork, 2008]. Typically, generators with DP property introduce additional noise in the training procedure [Jordon et al., 2022]. For example, adding noise in the gradient or using a noisy discriminator in a GAN architecture [Abadi et al., 2016, Jordon et al., 2018, Long et al., 2019].
2. Threat model (TM). While DP focuses on giving formal guarantees, the TM approach is designed for specific threat models, such as membership inference, attribute inference, and re-identification [Shokri et al., 2017, Kosinski et al., 2013, Dinur and Nissim, 2003]. These models often involve regularization terms designed to mitigate privacy attack risk [Yoon et al., 2020].
Privacy. Synthcity evaluates the privacy of synthetic data using a list of well-established privacy metrics (e.g. k-anonymity [Sweeney, 2002] and l-diversity [Machanavajjhala et al., 2007]). Furthermore, it can measure the privacy of data by performing simulated privacy attacks (e.g. a re-identification attack). The success (or failure) of such an attack quantifies the degree of privacy preservation.
2.2.5 Evaluating off label use cases
Synthcity allows users to conveniently evaluate the off label usage of generative models. For instance, one could evaluate the privacy of synthetic data even if they are not generated by a privacy-enabled generative model. As another example, one could evaluate the fairness for generative models that are differentially private, thereby enabling studies like Ganev et al. [2022].
Off-label evaluation is made easy because Synthcity implements generative models and evaluation metrics in two separate modules (PlugIns and Metrics). The consistent interface enables mix and match of models and metrics to empower different benchmark studies.
2.3 Baseline generative models
As a benchmarking framework, Synthcity is a one-stop-shop for state-of-the-art benchmarks with a large collection of baselines covering both deep generative models and other types of generative models. In this way, the user can easily compare with a range of existing methods, without the need to worry about implementation details or interfaces. Table 2 lists the generative models in synthcity for different data modalities.
Synthcity covers all major families of deep generative models, including Generative adversarial networks (GAN) [Goodfellow et al., 2020], Variational Autoencoders (VAE) [Kingma et al., 2019], Normalizing flows (NF) [Papamakarios et al., 2021], as well as Diffusion models (DDPM) [Kingma et al., 2021]. In the GAN family, Synthcity currently supports GOGGLE [Liu et al., 2023], CTGAN Xu et al. [2019b], DPGAN [Xie et al., 2018], PATEGAN [Jordon et al., 2019], ADSGAN [Yoon et al., 2020], DECAF [van Breugel et al., 2021] for static data, Survival GAN [Norcliffe et al., 2023] for censored data, TimeGAN [Yoon et al., 2019] for time series data, as well as RadialGAN [Yoon et al., 2018] for multi-source data. In the VAE family, it supports TVAE [Xu et al., 2019b], RTVAE for static data [Akrami et al., 2020], Survival VAE [Norcliffe et al., 2023] for censored data, and TimeVAE [Yoon et al., 2019] for time series data. In the NF family, Synthcity implements the standard NF [Papamakarios et al., 2021] for static data, Survival NF [Norcliffe et al., 2023] for censored data, and FourierFlow [Alaa et al., 2021] of time series data. Synthcity also includes the TabDDPM
153   
154   
155   
156   
157   
158   
159   
160   
161   
162   
163   
164   
165   
166   
167   
168   
169   
170   
171   
172   
173   
174   
175   
176   
177   
178   
179   
180   
181   
182   
183   
184   
185   
186   
187   
188   
189   
190   
191   
192   
193   
194   
195   
196   
197   
198   
199
Data Modality	Model	Standard Gen	Privacy	Fairness
DP	TM	Balance	Causal
Static	Bayesian Net	✓
NF	✓
GREAT	✓
ARF	✓
GOGGLE	✓
TabDDPM	✓
TVAE	✓	✓
RTVAE	✓	✓
CTGAN	✓	✓
AIM	✓	✓
PrivBayes	✓	✓
DPGAN	✓	✓
PATEGAN	✓	✓
ADSGAN	✓	✓
DECAF	✓	✓
Static (Censored)	Survival GAN	✓	✓	✓
Survival VAE	✓
Survival NF	✓
Time Series(regular, irregular,censored)	TimeGAN	✓	✓
TimeVAE	✓
FourierFlow*	✓
Probabilistic AR*	✓
Multi-source	RadialGAN	✓	✓
Table 2: Generative models available in synthcity for different data modalities and use cases. Abbreviations: Differential Privacy (DP), Threat Model (TM). *FourierFlow and Probabilistic AR is compatible with regular time series only while TimeGAN and TimeVAE support both.
[Kotelnikov et al., 2022] in the diffusion model family, and GREAT Borisov et al. [2022], which uses auto-regressive generative LLM model.
In addition to deep generative models, Synthcity also contains generative models that are not based on neural networks, such as Bayesian networks [Heckerman, 1997], AIM [McKenna et al., 2022], Probabilistic Auto-regressive models [Deodatis and Shinozuka, 1988] and Adversarial random forests (ARF) [Watson et al., 2023].
Synthcity implements all generative models using the PlugIn interface. This consistent approach makes it easy to add additional generative models into the benchmark. The GitHub repository includes tutorials and step-by-step instructions on how to add new models.
2.4 Architecture and hyper-parameters
To help researchers pinpoint the source of performance gain and conduct fair comparison, Synthcity allows the user to incarnate all the deep generative models with different network architectures and hyper-parameters.
The architecture can be specified when the user creates a model instance (i.e. a PlugIn). For example, each time-series generative model can be configured using twelve different architectures, including LSTM [Hochreiter and Schmidhuber, 1997], GRU [Dey and Salem, 2017], Transformer [Vaswani et al., 2017], MLSTM-FCN [Karim et al., 2019], TCN [Lea et al., 2017], InceptionTime and InceptionTimePlus [Ismail Fawaz et al., 2020], XceptionTime [Rahimian et al., 2020], ResCNN [Sun et al., 2020], Omni-Scale CNN [Tang et al., 2020], and XCM [Fauvel et al., 2021]. The network architectures compatible with other data modalities are tabulated in the Appendix.
200   
201   
202   
203   
204   
205   
206   
207   
208   
209   
210   
211   
212   
213   
214   
215   
216   
217   
218   
219
Synthcity also has a consistent interface for dealing with hyper-parameters. The library allows the user to list, set, and sample all relevant hyper-parameters of a generative model. Furthermore, this interface is compatible with all popular hyper-parameter optimization libraries, such as Optuna [Akiba et al., 2019]. In this way, Synthcity allows the user to perform hyper-parameter search before evaluating on the best-performing setting to ensure a like-for-like comparison. Furthermore, Synthcity also allows the user to configure various early stopping rules to control and compare the training of generative models.
2.5 Data modalities
We emphasize that "tabular data" in fact encapsulates many different data modalities, including static tabular data, time series data, and censored survival data, all of which may contain a mix of continuous and discrete features (columns). Synthcity can also handle composite datasets composed of multiple subsets of data. We give a detailed description of the diverse tabular data modalities Synthcity supports in Figure 2 and further discuss them below. In future versions, we plan to include more data modalities including relational database-style data, richly annotated images, and texts.
2.5.1 Single dataset
We start by introducing the most fundamental case where there is a single training dataset (e.g. a single DataFrame in Pandas). We characterize the data modalities by two axes: the observation pattern and the feature type. Synthcity supports all combinations.
The observation pattern describes whether and how the data are collected over time. There are three most prominent patterns, static data, regular time series, and irregular time series, which are all supported by synthcity.
The second axis, feature type, describes the domain of individual features. Synthcity supports multivariate tabular data with mixtures of continuous, categorical, integer, and censored features. Censored features are common in survival analysis applications (e.g. healthcare and insurance). They are represented as a tuple  $(x,c)$  where  $x\in \mathbb{R}^{+}$  represents the survival time and  $c\in \{0,1\}$  is the censoring indicator.
2.5.2 Composite dataset