SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Reimagining Synthetic Tabular Data Generation through Data-Centric AI:
A Comprehensive Benchmark
Lasse Hansen\*1,2 Nabeel Seedat\*3 Mihaela van der Schaar3 Andrija Petrovic4
$^{1}$ Department of Affective Disorders, Aarhus University Hospital - Psychiatry, Aarhus, Denmark  
 $^{2}$ Department of Clinical Medicine, Aarhus University, Denmark
$^{3}$ Department of Applied Mathematics and Theoretical Physics, University of Cambridge, UK  
 $^{4}$ Faculty of Organisational Sciences, University of Belgrade, Serbia  
lasse.hansen@clin.au.dk  
ns741@cam.ac.uk
Synthetic data serves as an alternative in training machine learning models, particularly when real-world data is limited or inaccessible. However, ensuring that synthetic data mirrors the complex nuances of real-world data is a challenging task. This paper addresses this issue by exploring the potential of integrating data-centric AI techniques which profile the data to guide the synthetic data generation process. Moreover, we shed light on the often ignored consequences of neglecting these data profiles during synthetic data generation — despite seemingly high statistical fidelity. Subsequently, we propose a novel framework to evaluate the integration of data profiles to guide the creation of more representative synthetic data. In an empirical study, we evaluate the performance of five state-of-the-art models for tabular data generation on eleven distinct tabular datasets. The findings offer critical insights into the successes and limitations of current synthetic data generation techniques. Finally, we provide practical recommendations for integrating data-centric insights into the synthetic data generation process, with a specific focus on classification performance, model selection, and feature selection. This study aims to reevaluate conventional approaches to synthetic data generation and promote the application of data-centric AI techniques in improving the quality and effectiveness of synthetic data.
Machine learning has become an essential tool across various industries, with high-quality data representative of the real world being a crucial component for training accurate models that generalize [1, 2, 3]. In cases where data access is restricted or insufficient synthetic data has emerged as a viable alternative [4, 5]. The purpose of synthetic data is to generate training data that closely mirrors real-world data, enabling the effective use of models trained on synthetic data on real data. Moreover, synthetic data is used for a variety of different uses, including privacy (i.e. to enable data sharing, [6, 7]), competitions [8] fairness [9, 10], and improving downstream models [11, 12, 13, 14].
Figure 1: Measures of data-centric profiling (A) better reflect the downstream performance of generative models (B) than measures of statistical fidelity (C). Assessed on the Adult dataset [16] using five different generative models A) Proportion easy examples in the generated datasets identified by Cleanlab, B) Supervised classification performance when training on synthetic, testing on real data, C) Inverse KL-divergence. (bn=bayesian_network)
However, generating high-quality synthetic data that adequately captures the nuances of real-world data, remains a challenging task. Despite significant strides in synthetic data with generative models, they sometimes fall short in replicating the complex subtleties of real-world data, particularly when dealing with messy, mislabeled or biased data. For instance, regarding fairness, [15] have shown that such gaps can lead to flawed conclusions and unreliable predictions on subpopulations, thereby restricting the practical usage of synthetic data.
The ability of synthetic data to capture the subtle complexities of real-world data is crucial, particularly in contexts where these issues might surface during deployment. Inaccurate synthetic data can not only hamper predictive performance but also result in improper model selection and distorted assessments of feature importance, thereby undermining the overall analysis. These challenges underscore the need to improve the synthetic data generation process.
One might wonder, surely, assessing fidelity via statistical divergence metrics [17, 5] such as MMD or KL-divergence is sufficient? We argue that such high-level metrics tell one aspect of the story. An overlooked dimension is the characterization of data profiles. In this approach, samples are assigned to profiles that reflect their usefulness for an ML task. Specifically, samples are typically categorized as easy to learn, ambiguous, or hard, which are proxies for data issues like mislabeling, data shift, or under-represented samples. In methods such as Data-IQ [18] and Data Maps [19] this is referred to as "groups of the data", however, we use "data profiles" for clarity.
While this issue has been well-studied for supervised tasks, it has not been explored in the generative setting. We highlight the issues of overlooking such data profiling in Figure 1, where despite near-perfect statistical fidelity (inverse KLD), we show the differing proportion of 'easy' examples identified in synthetic data generated by different generative models trained on the Adult dataset [16]. On the other hand, this data profile correlates with downstream classification performance.
To address this challenge of the representativeness of synthetic data, we explore the potential of integrating data-centric AI techniques and their insights to improve synthetic data generation. Specifically, we propose characterizing individual samples in the data and subsequently using the different data profiles to guide synthetic data generation in a way that better reflects the real world. While our work is applicable across modalities, our primary focus is tabular data given the ubiquity of tabular data in real-world applications [20, 21], with approximately  $79\%$  of data scientists working with it on a daily basis, vastly surpassing other modalities [22].
① Conceptually, we delve into the understanding of fundamental properties of data with respect to synthetic data generation, casting light on the impact of overlooking data characteristics and profiles when generating synthetic data.
$②$  Technically, we bring the idea of data profiles in data-centric AI to the generative setting and explore its role in guiding synthetic data generation. We introduce a comprehensive framework to facilitate this evaluation across various generative models.
$③$  Empirically, we benchmark the performance of five state-of-the-art models for tabular data generation on eleven distinct tabular datasets and investigate the practical integration of data-centric profiles to guide synthetic data generation. We provide practical recommendations for enhancing
synthetic data generation, particularly with respect to the 3 categories of synthetic data utility (i) predictive performance, (ii) model selection and (iii) feature selection.
We hope the insights of this paper spur the reconsideration of the conventional approaches to synthetic data generation and encourage experimentation on how data-centric AI could help synthetic data generation deliver on its promises.
This work engages with synthetic data generation and data characterization in data-centric AI.
Synthetic Tabular Data Generation uses generative models to create artificial data that mimics the structure and statistical properties of real data, and is particularly useful when real data is scarce or inaccessible [23, 24, 4]. In the following, we describe the broad classes of synthetic data generators applicable to the tabular domain. Bayesian networks [25] are a traditional approach for synthetic data generation, that represent probabilistic relationships using graphical models. Conditional Tabular Generative Adversarial Network (CTGAN) [26] is a deep learning method for modeling tabular data. It uses a conditional GAN to capture complex non-linear relationships. The Tabular Variational Autoencoder (TVAE) is a specialized Variational Autoencoder, designed for the tabular setting [26].
Normalizing flow models [27, 28] provide an invertible mapping between data and a known distribution, and offer a flexible approach for generative modeling. Diffusion models, which have gained recent popularity, offer a different paradigm for generative modeling. TabDDPM [29] is a diffusion model proposed for the tabular data domain. In this work, we evaluate these classes of generative models, considering various aspects of synthetic data evaluation.
Evaluation of Synthetic Data is a multifaceted task [17, 30], involving various dimensions such as data utility with respect to a downstream task, statistical fidelity, and privacy preservation [17, 30]. In this work, we focus on dimensions that impact model performance and hence, while important, we do not consider privacy aspects.
(1) Data Utility: refers to how well the synthetic data can be used in place of the real data for a given task. Typically, utility is assessed by training predictive models on synthetic data and testing them on real data [4, 17, 31, 32, 5]. We posit that beyond matching predictive performance, we also desire to retain both model ranking and feature importance rankings. We empirically assess these aspects in Sec. 5.
(2) Statistical Fidelity: measures the degree of similarity between synthetic data and the original data in terms of statistical properties, including the marginal and joint distributions of variables [17]. Statistical tests like the Kolmogorov-Smirnov test or divergence measures like Maximum Mean Discrepancy, KL-divergence or Wasserstein distance are commonly used for evaluation[17, 5].
Beyond statistical measures, the concept of data characterization and profiles of easy and hard examples has emerged in data-centric AI. These profiles serve as proxies for understanding real-world data, which is often not "perfect" due to mislabeling, noise, etc. The impact of these profiles on supervised models has been demonstrated in the data-centric literature [33, 34, 18]. In Figure 1, we show that data profiles are similarly important in the generative setting. Despite having almost perfect statistical fidelity, different generative models capture different data profiles (e.g. proportion of easy examples), leading to varying data utility as reflected in different performances. Consequently, we propose considering data profiles as an important dimension when creating synthetic data. We describe current data-centric methods that can facilitate this next.
Data profiling is a growing field in Data-Centric AI that aims to evaluate the characteristics of data samples for specific tasks [35, 36]. In the supervised learning setting, various methods have been developed to assign samples to groups, which we refer to as data profiles. These profiles, such as easy, ambiguous, or hard, often reveal issues such as mislabeling, data shifts, or under-represented groups [34, 18, 33, 37, 19, 38]. Various mechanisms are used in different methods for data characterization. For example, Cleanlab [34] models relationships between instances based on confidence, while Data Maps and Data-IQ [18] assess uncertainty through training dynamics. However, many existing methods are designed for neural networks and are unsuitable for non-differentiable models like XGBoost, which are commonly used in tabular data settings. Consequently, we focus on data characterization approaches such as Cleanlab, Data-IQ, and Data Maps which are more applicable to tabular data.
Figure 2: Illustration of the framework's process flow. Data partitioning: the dataset is divided into a training set,  $\mathbb{D}_{\mathrm{train}}$ , and a testing set,  $\mathbb{D}_{\mathrm{test}}$ . Data profiling: a data-centric preprocessing approach is employed on a duplicate of  $\mathbb{D}_{\mathrm{train}}$  to produce  $\mathbb{D}_{\mathrm{train}}^{\mathrm{pre}}$ . A generative model, trained on  $\mathbb{D}_{\mathrm{train}}^{\mathrm{pre}}$ , is then utilized to synthesize a dataset,  $\mathbb{D}_{\mathrm{synth}}$ , which is further processed using a data-centric postprocessing method to achieve the final synthetic dataset,  $\mathbb{D}_{\mathrm{synth}}^{\mathrm{post}}$ . Classification model training: various classification models are separately trained on  $\mathbb{D}_{\mathrm{train}}$  and  $\mathbb{D}_{\mathrm{synth}}^{\mathrm{post}}$  and applied to  $\mathbb{D}_{\mathrm{test}}$ . Evaluation: the generative and supervised models are appraised for their statistical fidelity and utility, focusing on classification accuracy, model selection, and feature selection.
We propose a unified framework that enables a thorough assessment of generative models and the synthetic data they produce. The framework encompasses the evaluation of the synthetic data based on established statistical fidelity metrics as well as three distinct tasks encompassing data utility.
At a high level, the framework proceeds as visualized in 2. The dataset is first divided into a training set, denoted as  $\mathbb{D}_{\mathrm{train}}$ , and a testing set, denoted as  $\mathbb{D}_{\mathrm{test}}$ . A duplicate of the training set  $(\mathbb{D}_{\mathrm{train}})$  undergoes a data-centric preprocessing approach to produce a preprocessed version of the training set, referred to as  $\mathbb{D}_{\mathrm{train}}^{\mathrm{pre}}$ . A generative model is then trained on  $\mathbb{D}_{\mathrm{train}}^{\mathrm{pre}}$ . This model is used to synthesize a new dataset, denoted as  $\mathbb{D}_{\mathrm{synth}}$ . The synthetic dataset is further processed using a data-centric postprocessing method to create the final synthetic dataset, denoted as  $\mathbb{D}_{\mathrm{synth}}^{\mathrm{post}}$ . Various classification models  $\mathcal{M}$  are then trained separately on the original training set  $\mathbb{D}_{\mathrm{train}}$  and the synthetic dataset  $\mathbb{D}_{\mathrm{synth}}^{\mathrm{post}}$ . These models are then applied to the testing set  $\mathbb{D}_{\mathrm{test}}$  for evaluation. The generative and supervised models are evaluated for their statistical fidelity and data utility. The focus is on classification performance, model selection, and feature selection. Further details on each process within the framework can be found in the following subsections.
3.1 Data profiling
Assume we have a dataset  $\mathcal{D} = \{(x^n, y^n) \mid n \in [N]\}$ . Data profiling aims to assign a score  $S$  to samples in  $\mathcal{D}$ . On the basis of the score, a threshold  $\tau$  is typically used to assign a specific profile group  $p^n \in \mathcal{P}$ , where  $\mathcal{P} = \{Easy, Ambiguous, Hard\}$  to each sample  $x^n$ .
Our framework supports three recent data characterization methods applicable to tabular data: Cleanlab [34], Data-IQ [18], and Data Maps [33]. They primarily differ based on their scoring mechanism  $S$ . For instance, Cleanlab [34] uses the predicted probabilities as  $S$  to estimate a noise matrix, Data-IQ [18] uses confidence and aleatoric uncertainty as  $S$ , and Data Maps uses confidence and variability (epistemic uncertainty) as  $S$ . Moreover, they differ in the categories in the data profiles derived from their scores. Data-IQ and Data Maps provide three categories of data profiles: easy; samples that are easy for the model to predict, ambiguous; samples with high uncertainty, and hard;
samples that are wrongly predicted with high certainty. Cleanlab provides two profiles: easy and hard examples.
We create data profiles with these three data-centric methods to evaluate the value of data-centric methods to improve synthetic data generation, both ex-ante and post hoc. We use the profiles in multiple preprocessing and postprocessing strategies applied to the original and synthetic data.
3.1.1 Preprocessing
Preprocessing strategies are applied to the original data  $\mathbb{D}_{\mathrm{train}}$  i.e., before feeding to a generative model. We investigate three preprocessing strategies: (1) baseline, which applies no processing, and simply feeds the  $\mathbb{D}_{\mathrm{train}}$  to the generative model. (2) easy-hard: Let  $S_{c}:\mathbb{D}_{\mathrm{train}}\to [0,1]$  denote the scoring function for data-centric method  $c$ . We partition  $\mathbb{D}_{\mathrm{train}}$  into  $\mathbb{D}_{\mathrm{train}}^{\mathrm{easy}}$  and  $\mathbb{D}_{\mathrm{train}}^{\mathrm{hard}}$  data profiles using a threshold  $\tau$ , such that  $\mathbb{D}_{\mathrm{train}}^{\mathrm{easy}} = \{x^n\mid S_c(x^n)\leq \tau \}$  and  $\mathbb{D}_{\mathrm{train}}^{\mathrm{hard}} = \{x^n\mid S_c(x^n) > \tau \}$ . (3) Analogously, easy_ambiguous-hard² splits the  $\mathbb{D}_{\mathrm{train}}$  on the easy, ambiguous, and hard examples. Further details are provided in Appendix A.
3.1.2 Generative model
We utilize the data profiles identified in the preprocessing step to train a specific generative model for each data segment, e.g. easy and hard examples separately. Let  $G: \mathbb{D}_{\mathrm{train}} \to \mathbb{D}_{\mathrm{synth}}$  denote the generative model trained on a dataset  $\mathbb{D}_{\mathrm{train}}$ , which produces synthetic dataset  $\mathbb{D}_{\mathrm{synth}}$ . In our framework, for each data profile in preprocessed dataset  $\mathbb{D}_{\mathrm{train}}^{\mathrm{pre}}$ , we train a separate generative model. We generate data using each generative model and the combined synthetic data is then  $\mathbb{D}_{\mathrm{synth}} = G_{\mathrm{easy}}(\mathbb{D}_{\mathrm{train}}^{\mathrm{easy}}) \cup G_{\mathrm{hard}}(\mathbb{D}_{\mathrm{train}}^{\mathrm{hard}})$ , with generation preserving the ratio of the data segments, to reflect their distribution in the initial dataset.
3.1.3 Postprocessing
We define postprocessing strategies as processing applied to the synthetic data after data generation but before supervised model training and task evaluation. We denote the set of postprocessing strategies as  $\mathcal{H}$ . Given the synthetic dataset  $\mathbb{D}_{\mathrm{synth}}$ , each postprocessing strategy  $h \in \mathcal{H}$  maps  $\mathbb{D}_{\mathrm{synth}}$  to a processed dataset  $\mathbb{D}_{\mathrm{synth}}^{\mathrm{post}} = h(\mathbb{D}_{\mathrm{synth}})$ . Two different postprocessing strategies were used: baseline: This is the identity function  $h_{\mathrm{baseline}}(\mathbb{D}_{\mathrm{synth}}) = \mathbb{D}_{\mathrm{synth}}$ . no-hard: We remove the hard examples from the synthetic data,  $\mathbb{D}_{\mathrm{synth}}^{\mathrm{post}} = \mathbb{D}_{\mathrm{synth}} \setminus \{x_{\mathrm{synth}}^{n} \mid S_{c}(x_{\mathrm{synth}}^{n}) > \tau\}$ , where  $x_{\mathrm{synth}}^{n}$  is generated synthetic data.
3.2 Classification model training
The training procedure of the supervised classification models  $\mathcal{M}$  comprises two steps, each minimizing a cost function  $\mathcal{L}$ . (1) Train on the real data, i.e.,  $\mathcal{M}_{\mathrm{real}} = \arg \min \mathcal{L}(\mathcal{M}(\mathbb{D}_{\mathrm{train}}))$ . (2) Train on synthetic data, i.e.  $\mathcal{M}_{\mathrm{syn}} = \arg \min \mathcal{L}(\mathcal{M}(\mathbb{D}_{\mathrm{synth}}^{\mathrm{post}}))$  We then compare utility of  $\mathcal{M}_{\mathrm{real}}$  and  $\mathcal{M}_{\mathrm{syn}}$  in the evaluation procedure. Our framework supports any machine learning model  $\mathcal{M}$  compatible with the Scikit-Learn API.
Finally, the framework includes automated evaluation tools for the generated synthetic data to evaluate the effect of pre- and postprocessing strategies, across datasets, random seeds, and generative models. To thoroughly assess our framework, we establish evaluation metrics that extend beyond statistical fidelity, encapsulating data utility through the inclusion of three tasks.
3.3.1 Statistical fidelity
The quality of synthetic data is commonly assessed using divergence measures between the real and synthetic data [5, 30]. Our framework allows for this assessment using widely adopted methods including inverse KL-Divergence [5], Maximum Mean Discrepancy [39], Wasserstein distance, as
2Only defined for data-centric methods that identify ambiguous examples, i.e. Data-IQ and Data Maps.
well as Alpha-precision and Beta-Recall [30]. However, as shown in Figure 1, such measures can only tell one aspect of the story. Indeed, despite all generative models providing near-perfect statistical fidelity based on divergence measures, the synthetic data captures the nuances of real data differently, as reflected in the varying data profiles (e.g. proportion easy examples). This motivates us to also assess the data utility and the potential implications of this variability.
3.3.2 Data utility
Three specific metrics were employed to assess data utility: classification performance, model selection, and feature selection.
Classification performance To explore the usefulness of the generated synthetic data for model training, we use the train-on-synthetic, test-on-real paradigm to fit a set of machine learning models  $\mathcal{M}$  on the synthetic data,  $\mathbb{D}_{\mathrm{synth}}$ , and subsequently evaluate their performance on a real, held-out test dataset,  $\mathbb{D}_{\mathrm{test}}$ . By using  $\mathbb{D}_{\mathrm{test}}$  we avoid potential issues from data leakage that might occur from an evaluation on the real training sets,  $\mathbb{D}_{\mathrm{train}}$ .
Model selection When using synthetic data for model selection, it is imperative that the ranking of classification models  $\mathcal{M}$  trained on synthetic data aligns closely with the ranking of classification models trained on the original data. To evaluate this, we first train a set of  $\mathcal{M}_{\mathrm{real}}$  on  $\mathbb{D}_{\mathrm{train}}$  and evaluate their classification performance on  $\mathbb{D}_{\mathrm{test}}$ . Next, we fit the same set of  $\mathcal{M}_{\mathrm{synth}}$  on  $\mathbb{D}_{\mathrm{synth}}^{\mathrm{post}}$  and evaluate their classification performance on  $\mathbb{D}_{\mathrm{test}}$ . The rank-ordering of the  $\mathcal{M}_{\mathrm{real}}$  in terms of a performance metric (e.g. AUROC) is compared with the ranking-order of the  $\mathcal{M}_{\mathrm{synth}}$  using Spearman's Rank Correlation.
Feature selection Feature selection is a crucial task in data analysis and machine learning, aiming to identify the most relevant and informative features that contribute to a model's predictive power. To evaluate the utility of using synthetic data for feature selection, a similar approach is followed as for model selection. First, a model  $\mathcal{M}_{\mathrm{real}}$  with inherent feature importance (e.g. random forest) is trained on  $\mathbb{D}_{train}$  and the rank-ordering of the most important features is determined. This ranking is then compared to the rank ordering of the most important features obtained from the same model type  $\mathcal{M}_{\mathrm{synth}}$  trained on  $\mathbb{D}_{\mathrm{synth}}^{\mathrm{post}}$  using Spearman's Rank Correlation.
3.4 Extending the framework
The framework presented in this paper is intentionally designed to be modular and highly adaptable, allowing for seamless integration of various generative models, pre- and postprocessing strategies, and diverse tasks. This flexibility enables researchers and practitioners to explore and evaluate e.g. different combinations of generative models alongside various pre- and post-processing strategies. Further, the framework is extensible, allowing for the incorporation of additional generative models, novel processing methods, and emerging tasks, ensuring that it remains up-to-date and capable of accommodating future advancements in the field of synthetic data generation.
To demonstrate the framework, we conduct multiple experiments, aiming to answer the following subquestions in order to investigate: Can data-centric ML improve synthetic data generation?:
Q1: Is statistical fidelity sufficient to quantify the utility of synthetic data?
Q2: Can we trust results from supervised classification models trained on synthetic data to generalize to real data?
Q3: Can data-centric approaches be integrated with synthetic data generation to create more realistic synthetic data?
Q4: Does the level of label noise influence the effect of data-centric processing for synthetic data generation?