Reimagining Synthetic Tabular Data Generation through Data-Centric AI: A Comprehensive Benchmark
Authors: Lasse Hansen
Nabeel Seedat
Mihaela van der Schaar
Andrija Petrovic
Venue: NeurIPS 2023 Datasets and Benchmarks Poster (2023)
Keywords: synthetic data generation
data-centric AI
tabular data
synthetic data evaluation
TL;DR: This paper proposes integrating data-centric AI into synthetic data generation, offering a new framework and practical recommendations based on an evaluation of state-of-the-art tabular data generation models.

Abstract: Synthetic data serves as an alternative in training machine learning models, particularly when real-world data is limited or inaccessible. However, ensuring that synthetic data mirrors the complex nuances of real-world data is a challenging task. This paper addresses this issue by exploring the potential of integrating data-centric AI techniques which profile the data to guide the synthetic data generation process. Moreover, we shed light on the often ignored consequences of neglecting these data profiles during synthetic data generation --- despite seemingly high statistical fidelity. Subsequently, we propose a novel framework to evaluate the integration of data profiles to guide the creation of more representative synthetic data. In an empirical study, we evaluate the performance of five state-of-the-art models for tabular data generation on eleven distinct tabular datasets. The findings offer critical insights into the successes and limitations of current synthetic data generation techniques. Finally, we provide practical recommendations for integrating data-centric insights into the synthetic data generation process, with a specific focus on classification performance, model selection, and feature selection. This study aims to reevaluate conventional approaches to synthetic data generation and promote the application of data-centric AI techniques in improving the quality and effectiveness of synthetic data.

Reimagining Synthetic Tabular Data Generation through Data-Centric AI: A Comprehensive Benchmark
Lasse Hansen" 1,2
Nabeel Seedat"
Mihaela van der Schaar2
Andrija Petrovic
DDpaarment Affective Disorders, Aarhus University Hospital Psychiatry, Aarhus, Denmark *Department of Clinical Medicine. Aarhus University, Denmark Department of Applied Mathematics and Theoretical Physics, University of Cambridge, UK 'Faculty of Organisational Sciences, University Belgrade, Serbia lasse wwwssnnenlln in. au dk n87410cam.o ac.
Abstract
Synthetic data serves as alternative in training machine learning models, par- ticularly when real-world data limited or inaccessible. However, ensuring that synthetic data mirrors the complex nuances of real-world data is challenging task. This paper addresses this issue by exploring the potential integrating data-centric AI techniques which profile the data to guide the synthetic data generation process. Morcover, we shed light on the often ignored consequences of neglecting these data profiles during synthetic data generation - despite seemingly high statistical fidelity. Subsequently, we propose novel framework to evaluate the integration of data profiles guide the creation of more representative synthetic data. In an empirical study, we evaluate the performance of five state-of-the--e models for tabular data generation on eleven distinct tabular datasets. The findings offer critical insights into the successes and limitations of current synthetic data gener- ation techniques. Finally, we provide practical recommendations for integrating data-centric insights into the synthetic data generation process, with a specific focus on classification performance, model selection, and feature selection. This study aims to reevaluate conventional approaches to synthetic data generation and promote the application of data-centric AI techniques in improving the quality and effectiveness of synthetic data.
Introduction
Machine learning has become an essential tool across various industries, with high-quality data representative of the real world being crucial component for training accurate models that generalize 222 3 In cases where data access restricted or insufficient synthetic data has emerged as The purpose of synthetic data is to generate training data that closely mirrors real-world data, enabling the effective use models trained on synthetic data on real data. Moreover, synthetic data is used for variety of different uses, including privacy (i.e. to enable data sharing, 10), and improving downstream models
viable alternative 4 5
7 competitions
8 fairness
'Equal Contributions
37th Conference on Neural Information Processing Systems (NeurIPS 2023) Track on Datasets and Benchmarks.

---

A
8
0.92
Kt Divorggnos
Properion E8e
0:89
0.96
90.88
88
0:87
50.85
0.80
Real data
bn
dlgan
drjpm
nifor
Roal data
bn
olgan
ddpm
niflow
haee
Roal data
bn
eppon
ddpm
nillow
bag
Generative Model
Generative
Modal
Gonerative
Model
Figure 1: Measures of data-centric profiling (A) better reflect the downstream performance of generative models (B) than measures of statistical fidelity (C). Assessed on the Adult dataset [16] using five different generative models A) Proportion easy examples in the generated datasets identified by Cleanlab, B) Supervised classification performance when training on synthetic, testing on real data, C) Inverse KL-divergence. (bn=bayesian, (
However, generating high-quality synthetic data that adequately captures the nuances of real-world data, remains a challenging task. Despite significant strides in synthetic data with generative models, they sometimes fall short in replicating the complex subtleties real-world data, particularly when dealing with messy, mislabeled or biased data. For instance, regarding fairness, [15] have shown that such gaps can lead to flawed conclusions and unreliable predictions on subpopulations, thereby restricting the practical usage synthetic data.
The ability of synthetic data to capture the subtle complexities of real-world data is crucial, particularly in contexts where these issues might surface during deployment. Inaccurate synthetic data not only hamper predictive performance but also result in improper model selection and distorted assessments of feature importance, thereby undermining the overall analysis. These challenges underscore the need to improve the synthetic data generation process.
One might wonder, surely, assessing fidelity via statistical divergence metrics [7,55 such as MMD or KL-divergence is sufficient? We argue that such high-level metrics tell one aspect of the story. An overlooked dimension is the characterization of data profiles. In this approach, samples assigned to profiles that reflect their usefulness for an ML task. Specifically, samples are typically categorized easy learn, ambiguous, hard, which proxies for data issues like mislabeling, data shift, under-represented samples. In methods such as Data-IQ [88 and Data Maps [19 this is referred to as "groups of the data", however, use "data profiles" for clarity.
While this issue has been well-studied for supervised tasks, has not been explored the generative setting. We highlight the issues of overlooking such data profiling in Figure [ where despite near-perfect statistical fidelity (inverse KIDD), we show the differing proportion identified in synthetic data generated by different generative models trained on the Adult dataset [16). On the other hand, this data profile correlates with downstream classification performance.
eaay" examples
To address this challenge of the representativeness of synthetic data, we explore the potential of integrating data-centric Al techniques and their insights to improve synthetic data generation. Specifically, we propose characterizing individual samples in the data and subsequently using the different data profiles guide synthetic data generation a way that better reflects the real world. While our work is applicable across modalities, our primary focus is tabular data given the ubiquity of tabular data real-world applications (20 211, with approximately 79% data scientists working with it on daily basis, vastly surpassing other modalities 22
Contributions:
i Conceptually, we delve into the understanding of fundamental properties of data with respect to synthetic data generation, casting light on the impact of overlooking data characteristics and profiles when generating synthetic data.
② Technically, we bring the idea of data profiles in data-centric Al the generative setting and explore its role in guiding synthetic data generation. We introduce comprehensive framework to facilitate this evaluation across various generative models.
3 Empirically, we benchmark the performance of five nate-of-thheart models for tabular data generation on eleven distinct tabular datasets and investigate the practical integration of data-centric profiles to guide synthetic data generation. We provide practical recommendations for enhancing

---

synthetic data generation, particularly with respect to the 3 categories of synthetic data utility (i) predictive performance, (ii) model selection and (iii) feature selection.
We hope the insights of this paper spur the reconsideration of the conventional approaches to synthetic data generation and encourage experimentation how data-centric AI could help synthetic data generation deliver on its promises.
2 Related work
This work engages with synthetic data generation and data characterization in data-centric AL.
Synthetic Tabular Data Generation uses generative models to create artificial data that mimics the structure and statistical properties of real data, and is particularly useful when real data is scarce or inaccessible [23] 24 4. In the following, we describe the broad classes of synthetic data generators applicable the tabular domain. Bayesian networks [25] are a traditional approach for synthetic data generation, that represent probabilistic relationships using graphical models. Conditional Tabular Generative Adversarial Network (CTGAN) (26] is a deep learning method for modeling tabular data. It uses a conditional GAN to capture complex non- non-linear relationships. The Tabular Variational Autoencoder (TVAE) a specialized Variational Autoencoder, designed for the tabular setting (26).
Normalizing flow models (7 28 provide an invertible mapping between data and a known distribu- tion, and offer flexible approach for generative modeling- Diffusion models, which have gained recent popularity, offer different paradigm for generative modeling, TabDDPM [29 is diffusion model proposed for the tabular data domain. In this work, we evaluate these classes of generative models, considering various aspects of synthetic data evaluation.
Evaluation of Synthetic Data is multifaceted task [17 (30], involving various dimensions such as data utility with respect to a downstream task, statistical fidelity, and privacy preservation [17 30 In this work, we focus on dimensions that impact model performance and hence, while important, we do not consider privacy aspects.
(1) Data Utility: refers to how well the synthetic data can be used in place of the real data for a given task. Typically, utility is assessed by training predictive models on synthetic data and testing them on real data [4] [7 (2] 5). We posit that beyond matching predictive performance, we also desire to retain both model ranking and feature importance rankings. We empirically assess these aspects in Sec. 5
(2) Statistical Fidelity: measures the degree of similarity between synthetic data and the original data in terms of statistical properties, including the marginal and joint distributions of variables [17]. Statistical tests like the Kolmogorov-Smimoo test or divergence measures like Maximum Mean Discrepancy, KL-divergence or Wasserstein distance are commonly used for evaluation [7 5
Beyond statistical measures, the concept of data characterization and profiles of easy and hard examples has emerged in data-centric AI. These profiles serve as proxies for understanding real-world data, which is often not "perfect" due to mislabeling, noise, etc. The impact of these profiles on supervised models has been demonstrated in the data-centric literature [33 34 18]. In Figure [ we show that data profiles are similarly important in the generative setting. Despite having almost perfect statistical fidelity. different generative models capture different data profies (e.g. proportion of easy examples), leading to varying data utility as reflected different performances. Consequently, we propose considering data profiles describe current data-centric methods that can facilitate this next.
important dimension when creating synthetic data. We
Data profiling is a growing field in Data-Centric AI that aims to evaluate the characteristics of data samples for specific tasks [35 36). In the supervised learning setting, various methods have been developed to assign samples to groups, which we refer to as data profiles. These profiles, such as casy, ambiguous. hard, often reveal issues such as mislabeling, data shifts, or under-represented groups
Various mechanisms are used in different methods for data characterization. models relationships between instances based on confidence, while Data Maps and Data-IQ IQ [8 assess uncertainty through training dynamics. However, many existing methods are designed for neural networks and are unsuitable for non-differentiable models like XGBoost, which are commonly used in tabular data settings. Consequently, we focus on data characterization approaches such as Cleanlab, Data-IQ, and Data Maps which are more applicable to tabular data.
34
18
33
37
19 38)
For example, Cleanlab
34
3

---

profiling
Preprocessing
Generative model
Postproceesing
baseline
TWAE
baseline
easy hard
D
TabDDPM
Syyk
no. hard
Data partitioning
Daw symt
Doohn
Clessification training
Evaluation
D
Synthetic data models
Statistical fidelity
Real data models
Classification performance
Model selection
Feature selection
Figure Illustration of the framework's process flow. Data partitioning: the dataset divided into training set, Driin and testing set, Dies- Data profiling: data--entric preprocessing approach employed on a duplicate of Drain produce D urin A generative model, trained on P is then utilized to synthesize a dataset, Syynth which is further processed using data-centric postprocessing method to achieve the final synthetic dataset, Dyynh Classification model training: various classification models are separately trained on Dirain and gost and applied to Desa- Evaluation: the generative and supervised models are appraised for their statistical fidelity and utility, focusing classification accuracy, model selection, and feature selection.
3 Framework
We propose a unified framework that enables a thorough assessment of generative models and the synthetic data they produce. The framework encompasses the evaluation of the synthetic data based on established statistical fidelity metrics as well as three distinct tasks encompassing data utility.
At high level, the framework proceeds visualized in The dataset is first divided into training set, denoted as Drrann and testing set, denoted as Dies- A duplicate of the training set (Darin) undergoes data-centric preprocessing approach produce preprocessed version of the training set, referred to as IDPe A generative model is then trained on D This model is used synthesize new dataset, denoted as (Dsyythh The synthetic dataset is further processed using
data-centric postprocessing method to create the final synthetic dataset, denoted as D) symth* Various classification models M are then trained separately on the original training set Drain and the synthetic dataset symth These models then applied the testing set Dist for evaluation. The generative and supervised models evaluated for their statistical fidelity and data utility. The focus is classification performance, model selection, and feature selection. Further details on each process within the framework can be found in the following subsections.
3.1 Data profiling
Assume we have a dataset D = ((x",''"' | e [N]]. Data profiling aims to assign a score $ to samples in D. On the basis of the score, threshold 7 is typically used to assign specific profile group p" E P, where
Eass, Ambigious, Hard] to each sample x".
Our framework supports three recent data characterization methods applicable to tabular data: Clean- lab [34. Data-IQ [18, and Data Maps [33]. They primarily differ based on their scoring mechanism S. For instance, Cleanlab [34] uses the predicted probabilities as S to estimate noise matrix, Data-IQ [18] uses confidence and aleatoric uncertainty as S, and Data Maps uses confidence and variability (epistemic uncertainty) as S. Moreover, they differ in the categories in the data profiles derived from their scores. Data-IQ and Data Maps provide three categories data profiles: easy: samples that are easy for the model to predict, ambiguous; samples with high uncertainty, and hard;

---

samples that are wrongly predicted with high certainty. Cleanlab provides two profiles: easy and hard examples.
We create data profiles with these three data-centric methods to evaluate the value of data-centric methods to improve synthetic data generation, both ex-anfe and post hoc. We use the profiles in multiple preprocessing and postprocessing strategies applied to the original and synthetic data.
3.1.1 Preprocessing
Preprocessing strategies are applied to the original data Durain i.e., before feeding to a generative model. We investigate three preprocessing strategies: (1) baseline, which applies no processing,
and simply feeds the Dunin to the generative model. (2) easy_hard:
hard: Let S. Il + [0, 1] denote
the scoring function for data-centric method We partition Dmin into M and D data profiles
using threshold such that eaa = {x* [e(x"") S ) and Dhhar = {xx" Se(a") T}. (3)
Analogously, asy_ambiguout _aaa__ardd splits the Irain on the easy, ambiguous, and hard examples.
Further details are provided in Appendix
3.1.2 Generative model
We utilize the data profiles identified in the preprocessing step to train specific generative model for each data segment, e.g. easy and hard examples separately. Let G D -> Deymth denote the generative model trained framework, for each data profile in preprocessed dataset DDpr train train separate generative model. We generate data using each generative model and the combined synthetic data is then Dyymh = Geasy ( U Ghard (Daard)) ) with generation preserving the ratio the data segments, to reflect their distribution in the initial dataset.
dataset Durin, which produces synthetic dataset ID synnh In
3.1.3 Postprocessing
We define postprocessing strategies as processing applied to the synthetic data after data generation but before supervised model training and task evaluation. We denote the set of postprocessing strategies as H. Given the synthetic dataset DD Synnh each postprocessing strategy h E H. maps 1ysh to processed dataset Depeet h(Dsyath). Two different postprocessing strategies were used: baselines This is the identity function hbusetine (Dsymth) Dyymth- no no_hard: We remove the hard examples from the synthetic data, Dyport Dyynthh {x Sc(2)yym)) > T}, where z-ymth is generated synthetic data.
3.2 Classification model training
The training procedure of the supervised classification models M comprises two steps, each mini- mizing a cost function C. (1) Train on the real data, i.c., Mreal = arg min ((M(Daa)))). (2) Train on synthetic data, i.e. M,yn arg min ((M(I)) We then compare utility of M,eat and Myy in the evaluation procedure. Our framework supports any machine learning model M compatible with the Scikit-Learn API.
3.3 Evaluation
Finally, the framework includes automated evaluation tools for the generated synthetic data to evaluate the effect of pre- and postprocessing strategies, across datasets, random seeds, and generative models. To thoroughly assess our framework, we establish evaluation metrics that extend beyond statistical fidelity, encapsulating data utility through the inclusion of three tasks.
3.3.1 Statistical fidelity
The quality of synthetic data is commonly assessed using divergence measures between the real and synthetic data [5] 30]. Our framework allows for this assessment using widely adopted methods
including inverse KL-Divergence (5), Maximum Mean Discrepancy [39 Wasserstein distance, as
Only defined for dena-centric methods that identify ambiguous examples, i.c. Data-IQ IQ and Data Maps.
5

---

well as ceeha-preisson and Beta-Recall [3). However, as shown in Figure such measures can only tell one aspect of the story. Indeed. despite all generative models providing near-perfect statistical fidelity based on divergence measures, the synthetic data captures the nuances of real data differently, as reflected the varying data profles (e.g. proportion easy examples). This motivates us to also assess the data utility and the potential implications of this variability.
3.3.2 Data utility
Three specific metrics were employed to assess data utility: classification performance, model selection, and feature selection.
Classification performance To explore the usefulness of the generated synthetic data for model training, we use the train-on-synthetic, test-on-real paradigm fit set of machine learning models M on the synthetic data, II syuth and subsequently evaluate their performance real, held-out test dataset, Drest By using Dosst we avoid potential issues from data leakage that might occur from an evaluation on the real training sets, Dunin-
Model selection When using synthetic data for model selection, it is imperative that the ranking of classification models .M trained on synthetic data aligns closely with the ranking of classification models trained the original data. To evaluate this, we first train their classification performance on Diett Next, fit the same set of Maynch D synth and evaluate their classification performance on West. The rank-ordering of the Meal in terms of performance metric (e.g. AUROC) compared with the ranking-order of the Msynch using Spearman's 's Rank Correlation.
of Maaa on Darinn and evaluate
Feature selection Feature selection a crucial task in data analysis and machine learning, aiming to identify the most relevant and informative features that contribute to model's predictive power. To evaluate the utility of using synthetic data feature selection, similar approach followed as for model selection. First, model Mreal with inherent feature importance (e.g. random forest) is trained on Merain and the rank-ordering of the most important features is determined. This ranking is then compared to the rank ordering of the most important features obtained from the same model type Msyath trained on syath using Spearman's 's Rank Correlation.
3.4 Extending the framework
The framework presented in this paper is intentionally designed to be modular and highly adaptable, allowing for seamless integration of various generative models, pre- and postprocessing strategies, and diverse tasks. This flexibility enables researchers and practitioners explore and evaluate e.g. different combinations of generative models alongside various pre- and post-processing strategies. Further, the framework is extensible, allowing for the incorporation of additional generative models, novel processing methods, and emerging tasks, ensuring that it remains up-to-date and capable of accommodating future advancements in the field of synthetic data generation.
Experiments
To demonstrate the framework, we conduct multiple experiments, aiming to answer the following subquestions in order to investigate: Can data-centrie ML improve synthetic data generation?:
Q1: Is statistical fidelity sufficient quantify the utility of synthetic data?
Q2: Can we trust results from supervised classification models trained on synthetic data to generalize to real data?
Q3: Can data-centric approaches be integrated with synthetic data generation create more realistic synthetic data?
Q4: Does the level of label noise influence the effect of data-centric processing for synthetic data generation?
All code for running the analysis and creating tables and graphs can be found at the following links: http:///ww////tttticcccaaaaaaaaa http://aeeecccaaeellaaaaaaaccccn
or https:://// hhttps://githt:

---

Table 1: Summarised performance for the baseline condition (no data-centric processing) across all datasets. Classification is measured by AUROC, model selection and feature selection by Spearman's s Rank Correlation, and statistical fidelity by inverse KI. divergence. Numbers show bootstrapped mean and 95% CI. The best-performing model by task is in bold.
Generative Model
Classification
Model Selection
Feature Selection
Statistical fidelity
Real data
0.866 (0.855, 0.877)
1.0
1.0
1.0
bayesian eeeetsiannnetwor
0.622 (0.588, 0,656)
0.155 (0.055, 0.264)
0.091 (-0.001, 0.188)
0.998 (0.998. 0.999)
ctgan
0,797 (0.769. 0.823)
0.519 (0,457, 0.579)
0.63 (0.557, 0.691)
0,979 (0.967, 0,987)
ddpm nflow
0.813 (0.781, 0.844)
0.508 (0.446, 0.573)
0.635 (0.546, 0.718)
0.846 (0.668, 0.972)
0.737 (0.713, 0.761)
0.354 (0.288, 0.427)
0.415 (0.34, 0.485)
0.975 (0.968, 0.981)
tvae
0.792 (0.764, 0.818)
0.506 (0.436, 0.565)
0.675 (0.63, 0.722)
0,966 (0.953, 0.978)
4.1 Data
We assess our framework on a filtered version of the Tabular Classification from Numerical features benchmark suite from [4)]. To reduce computational costs, we filter the benchmark suite only include datasets with less than 100.000 samples and less than 50 features which reduced the number of datasets from 16 to 11. The datasets span several domains and contain highly varied number of samples and features (see for more details.). Notably, the datasets have been preprocessed to meet series of criteria ensure their suitability for benchmarking tasks. For instance, the datasets have at least 5 features and 3000 samples, are not too easy to classify, have missing values removed, have balanced classes, and only contain low cardinality features.
4.2 Generative models
To cover a representative sample of the space of generative models, we evaluate different models with different architectures as reviewed i2 bayesian networks (bayesian _ne_aarrk)) conditional tabular generative adversarial network (ctgan), tabular variational autoencoder (tvae), normalizing flow (nflow), diffusion model for tabular data (ddpm).
4.3 Supervised classification model training
The variety of models employed in our study includes: extreme gradient boosting (xgboost), random forest, logistic regression, decision tree, k-nearest neighbors, support vector classifier, gaussian naive bayes, and multi-layer perception. It is the ranking of these models that is evaluated for the model selection task. Given the large number of models, we restrict the classification results in the main paper to be from the xgboost model. Feature selection results are reported for xgboost models. Classification and feature selection results for the other classifiers can be found in Appendix C
4.4 Experimental procedure
Main study The experimental process followed the structure outlined in Sec. Figuree for each of the 11 datasets, 5 generative models, 10 random seeds, and all permutations of pre- and postprocessing methods for each of the three data-centric methods (Cleanlab, Data-IQ, and Data Maps). We comprehensively evaluate the results across classification performance, model selection, feature selection, and statistical fidelity. In total, we fit more than 8000 generative models.
Figure repeated
Impact of label noise To assess the impact of label noise on the effect of data-centric pre- and postprocessing, carried out analogous experiment to the main study, the Covid mortality dataset [41] Here, we introduce label noise to Darain before applying any processing. We study the impact of adding [0, 2, 4, 6, 8, 10] percent label noise.
All results reported in the main paper use Cleanlab as the data-centric method for both pre- and postprocessing. This decision was made to ensure clarity in the reported results and because Cleanlab was found to outperform Data-IQ and Data Maps a simulated benchmark. For the benchmark of the data-centric methods well as results using Data-IQ and Data Maps, refer Appendix C
7

---

Table 2: Percentage increase in performance from baseline, i.e., no data-centric pre- or postprocessing (as seen in Table 1), per generative model for each pre- and postprocessing strategy, averaged across all datasets and seeds.
Generative Model
Preproensing
Posppeeessingg
Clasification
Model Selection
Festuone Selection
Strtitical Fidelity
Strategy
Stralegy
bayuer antnekorrk
(4.80,
(-86
-52.26
(4.054,
easy hard
taselime
6.03)
(31.99
(-10447 129.68)
0..13
no, hard
B8 6,34)
27:25 (-31,99, 90.288)
99.9 (2152.22 107729)
(-0.0) 0.021)
otgan
bareline
no, hardd
1,236-200, 4,64)
11.47 24,28)
3,66 [-6.48 12.61)1
0.054 (-1,23,, 0.805)
saay_hard
baseline
40778 2,88)
-L.12(-13334 11.47)
0.15 (-10.56. 10.13)
-0.06 (-1.661,00 (.800)
no_hard
0.37 (-2.66 3.8)
12.13 26.681
0.18 (-10.04, 9.43)
-0.119 (-1.218, 0.701)
ddpx
baseline
no, hard
1.63 4.24) f
-6.35 1-20.41, 7,86)
17 (-10.67 13,37)
00.16 (-19.424, 15 15.964)
anay hard
baseling
0.65 (-2.84 4.06)
701-5.87, 20 19)
1.58 (-10.92, 13,92)
0.356 (-16.583, 13.833)
hardard
1.32 1.68)
5.98 (:7.27, 18,63)
4.19 (-695, 16.16)
0.284 (-16.221, 14,567)
nflow
hnawline
no_hard
(:245, 3.97)
(-16.88 18.41)
5 (-11 21.68)
0.055 (-8.743,
eary.
taselime
(-2.611.)...81
0.022 (-0.752.
hard
1.64 4.81)
4,66 22.84)
7:28 (-11.54, 25.04)
0,022 (40.205 0.654)
trae
bareline
no, hardd
1.1 (-2:29 4.38)
-3.58 (-18A4, 11,01)
(-538 6,38)
0-053 1.J13)
easy_hard hard
baseline
.025 (:3.46,
.3.8 (-1945, 6,75)
1.67 1-603, 8.17)
(-,9441 1,296)
no no_hard
0.71 3,95
0.11 (-13.79, 14.38)
3.41 (-3.38,9 9.89)
0.199 (-0.929, 1.285)
4.4.1 Evaluation metrics
The classification performance is evaluated in terms of the area under the receiver operating char- acteristic curve (AUROC), model selection performance as Spearman's 's Rank Correlation between the ranking of the supervised classification models trained on the original data and the supervised classification models trained on the synthetic data, and feature selection performance as Spearman's 's Rank Correlation between the ranking of features in an xghoost model trained on the original data and an xgboost model trained on the synthetic data.
Results
Statistical fidelity is insufficient for evaluating generative models. Measures of statistical fidelity fail capture variability in performance on downstream tasks, as shown Table Surprisingly, the worst performing model across all tasks (bayesian network), has the highest inverse KL-divergence of all synthetic datasets, which should indicate a strong resemblance the original data. Conversely, the lowest inverse KL-divergence is found for ddpm which is one of the consistently best performing models.
Practical guidance: The benchmarking results illustrate that when selecting a generative model, even if the statistical fidelity appears similar, different generative models may perform differently on the 3 downstream tasks (classification, model selection, feature selection). Hence, beyond statistical fidelity, practitioners should understand which aspect is most crucial for their purpose to guide selection of the generative model.
Different generative models for different tasks. As shown in Table training on synthetic data leads to a marked decline in classification performance compared to real data, as well as highly differing model and feature rankings. The effect differs largely by generative model, where CTGAN, TabDDPM, and TVAE most closely retain the characteristics of the real data. No one model is superior across all tasks. Specifically, TabDPPM achieves the highest classification performance, CTGAN performs best in model selection, and TVAE excels in feature selection. These findings indicate that one should test range of generative models and consider the trade-offs in data utility before publishing synthetic data. Additionally, Appendix C reveals that although there are slight differences in performance based on the supervised model type. the overall pattern of results remains consistent across generative models.
Practical guidance: No generative model reigns supreme (highlighting the inherent challenge of synthetic tabular data). However, over tabular data sets, we show that CTGAN and TVAE offer the best trade-off between high statistical fidelity and strong performance the three downstream tasks.
Data-centric methods can improve the utility of synthetic data. The addition of data-centric pre- and postprocessing strategies has generally positive effect across all tasks as seen in Table 2 and Figure 33 despite resulting in lower statistical fidelity. In terms of classification performance, 13 out of 15 evaluations showed a net improvement, with gains up to 1.64% better classification
8

---

performance compared to not processing the data. Model selection exhibited more pronounced effects, particularly for bayesian networks, which demonstrated the greatest variabilityy overall. While the model selection results for TVAE decreased following data-centric processing, the other generative models saw positive effects, with performance improvements ranging from 4.66% to 92%. Regarding feature selection, 12 out of 15 evaluations demonstrated a net benefit of data-centric processing, resulting in improvements of 3.41 % to 9.79% in Spearman's 's rank correlation. The benefit of data- centric processing was found to be statistically significant for classification and feature selection (see Appendix D for details).
Before releasing a synthetic dataset, practitioners are advised to apply the data-centric methods studied in this paper as an add-on. This will ensure enhanced utility of the synthetic data in terms of classification performance, model selection, and feature selection.
Practical guidance:
Data-centric processing provides benefits across levels of label noise. Data-centric pre- and postprocessing lead to consistently higher performance across tasks for all augmented datasets. As shown in Figure the magnitude of the effect of data-centric processing decreases with higher levels of label noise, particularly above 8%, although this effect is not statistically significant. Even though the level of statistical fidelity decreased marginally by applying data-centric processing, data-centric processing led to statistically significant increases in performance on all three tasks.
Fitting generative models on noisy Freal-world" data can lead to sub- sub-optimal downstream performance despite seemingly high statistical fidelity. Data-centric methods are espe- cially useful at reasonable levels of label noise, typically below 8%. Therefore, we recommend their application when fitting generative models on real-world datasets.
Practical guidance:
5.1 Limitations and future work
Our work delves into the performance-driver aspects of synthetic data generation, focusing primarily on data utility and statistical fidelity, particularly within the realm of tabular data. While tabular data is highly diverse and contains many intricacies, we also recognize several directions for further exploration. Our current framework, while rooted in tabular data, hints the broader applicability to other data types such as text and images. Accommodating our framework to these modalities would require further work on modality-speeiff tasks. For instance, images or text do not possess direct analog to feature selection. Such disparities underscore the need for a bespoke benchmarking methodology tailored to each specific data type.
Clessification
Model Selection
Festure Selection
0.80
0.5
0.6
Performance
0.75
0.4
0.4
0.2
0.65
0.1
baseline-han baseine-baanllie baneline
basaline-no hard
eas hard batading
easy hard-no hard
baseline-iatatine bassalne
barsstine-tt no hard
easy hard Jatadinni
eusy hardino hard
baseline-ban laseline-barettee banetine
Iarasine-no no hard
mas hard basstina
dasy hardno hard
Preprocessing TreprocessinngPootrrcceeee
Postpracessing
bayesian, network
ctgan
ddpm
inlow
bac
Figure 3: Average performance across all datasets for each generative model by pre- and postprocess-
ing method.

---

Classiticatior
Model So ection
0.78
0.74
0.6
0.4
0.2
Performance
Feature Selection
Statistical Fidelity
0.6
0.9996
0.9994
0.9992
0.2
0.9990
20
Percent label noise
baseline-baseline baseline-no_ri hard
pasy_hard-bbasellee easy_hard-no hard
Figure 4: Performance of single generative model (TabDDPM) on the Covid mortality dataset with varying levels label noise across the pre- and postprocessing conditions.
6 Conclusion
This research provides novel insights into integrating data-centric AI techniques into synthetic tabular data generation. First, introduce framework to evaluate the integration of data profiles for creating more representative synthetic data. Second, we confirm that statistical fidelity alone is insufficient for assessing synthetic data's 's utility, as it may overlook important nuances impacting downstream tasks. Third, the choice of generative model significantly influences synthetic data quality synthetic data across varying levels of label noise. Our study demonstrates the potential of data-centric AI techniques to enhance synthetic data's representation of real-world complexities, opening avenues for further exploration at their intersection.
and utility. Last, incorporating data-centric methods consistently improves the utility of
Acknowledgements
This work was partially supported by DeiC National HPC (g.a. 2022-H2-10). NS supported by the Cystic Fibrosis Trust. LH was supported by travel grant from A.P. Møller Fonden til Lægevidenskabens Fremme and is supported by grants from the Lundbeck Foundation (grant num- ber: 2344-2020-1077), the Central Denmark Region Fund for Strengthening of Health Science (grant number: 1-36-72-4-20), and The Danish Agency for Digitisation Investment Fund for New Technologies (grant number: 2020 (020-6720).
10

---

References
[1] Abhinav Jain, Hima Patel, Lokesh Nagalapatti, Nitin Gupta, Sameep Mehta, Shanmukha Guttula, Shashank Mujumdar, Shazia Afzal, Ruhi Sharma Mittal, and Vitobha Munigala. Overview and importance of data quality for machine learning tasks. In Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, pages 3561-3562, 2020.
[2] Nitin Gupta, Hima Patel, Shazia Afzal, Naveen Panwar, Ruhi Sharma Mittal, Shanmukha Guttula, Abhinav Jain, Lokesh Nagalapatti, Sameep Mchta, Sandeep Hans, al. Data quality toolkit: Automatic assessment of data quality and remediation for machine learning datasets. arXiv preprint arXiv:2108. 05935, 2021.
[3] Cedric Renggli, Luka Rimanic, Nezihe Merve Gürel, Bojan Karlas, Wentao Wu, and Ce Zhang. A data quality-driven view of mlops. IEEE Data Engineering Bulletin, 2021.
[4] Yingzhou Lu, Huazheng Wang, and Wenqi Wei. Machine learning for synthetic data generation: a review. arXiv preprint arXiv:2302.04066 2023.
[5] Zhaozhi Qian, Coggaan-Constnttn Cebere, and Mihaela van der Schaar. Synthcity: facil- itating innovative use cases of synthetic data in different data modalities. arXiv preprint arXiv:2301.07573, 2023.
[6] James Jordon, Jinsung Yoon, and Mihaela Van Der Schaar. PATE-GAN: data with differential privacy guarantees. In International Conference on Learning Representa- lions, 2018.
Generating synthetic
[7] Samuel A Assefa, Danial Dervovic, Mahmoud Mahfouz, Robert E Tilllma,, Prashant Reddy, and Manuela Veloso. Generating synthetic data in finance: opportunities, challenges and pitfalls. In Proceedings of the First ACM International Conference on Al in Finance, pages 1-8, 2020.
[8] James Jordon, Jinsung Yoon, and Mihacla van der Schaar. Measuring the quality of synthetic data for use in competitions. arXiv preprint 1806v:1806.1 J 1345, 2018.
[9] Depeng Xu, Yongkai Wu, Shuhan Yuan, Lu Zhang, and Xintao Wu. Achieving causal fairness through generative adversarial networks. In Proceedings of the Twenty-Eight International Joint Conference on Artificial Intelligence, 2019.
[10] Boris van Breugel, Trent Kyono, Jeroen Berrevoets, and Mihaela van der Schaar. Decaf: Generating fair synthetic data using causally-aware generative networks. Advances in Neural Information Processing Systems, 34:22221-22233, 2021.
[11] Antreas Antoniou, Amos Storkey, and Harrison Edwards. Data augmentation generative adversarial networks. arXiv preprint arXiv: 2017.
[12] Ayesha S Dina, AB Siddique, and Manivannan. Effect of balaneing data using synthetic data on the performance of machine learning classifiers for intrusion detection in computer networks. arXiv preprint arXiv:2204.000 000400004444...4 /4, 2022.
[13] Hari Prasanna Das, Ryan Tran, Japjot Singh, Xiangyu Yue, Geoffrey Tison, Alberto Sangiovanni- Vincentelli, and Costas Spanos. Conditional synthetic data generation for robust machine learning applications with limited pandemic data. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 36, pages 11792-11 1800, 2022.
[14] Simon Bing, Andrea Dittadi, Stefan Bauer, and Patrick Schwab. Conditional generation of medical time series for extrapolation to underrepresented populations. PLOS Digital Health, 1(7):1-26, 07 2022.
[15] Georgi Ganev, Bristena Oprisanu, and Emiliano De Cristofaro. Robin hood and matthew effects: Differential privacy has disparate impact on synthetic data. In International Conference on Machine Learning, pages 6954-6959, PMLR, 2022.
[16) Barry Becker and Ronny Kohavi. Adult. UCI Machine Learning Repository, 1996. DOI: http:///ooior//122443/////22220

---

[17] Joshua Snoke, Gillian M Raab, Beata Nowok, Chris Dibben, and Aleksandra Slavkovic. General and specific utility measures for synthetic data. Journal of the Royal Statistical Society. Series A (Statistics in Society), 181(3):663-688, 2018
[18) Nabeel Seedat, Jonathan Crabbé, Ioana Bica, and Mihaela van der Schaar. Data-iq: Characteriz- ing subgroups with heterogeneous outcomes in tabular data. In Advances in Neural Information Processing Systems, 2022.
[19] Chirag Agarwal, Daniel souza, and Sara Hooker. Estimating example difficulty using variance of gradients. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 10368-10378, 2022.
[20] Vadim Borisov, Tobias Leemann, Kathrin Seßler, Johannes Haug, Martin Pawelczyk, and Gjergji Kasneci. Deep neural networks and tabular data: survey. arXiv preprint arXiv:2110.0888 2021.
[21] Ravid Shwartz-Zii and Amitai Armon. Tabular data: Deep learning not all you need. Information Fusion, 81:84-90, 2022.
[22] Kaggle. Kaggle machine learning and data science survey, 2017.
[23] Sam Bond-Taylor, Adam Leach, Yang Long, and Chris G Willcocks. Deep generative modelling:
A comparative review of vaes, gans, normalizing flows, energy-based and autoregressive models. IEEE transactions on pattern analysis and machine intelligence, 2021.
[24] Jessamyn Dahmen and Diane Cook. Synsys: A synthetic data generation system for healthcare applications. Sensors, 19(5): 118,, 2019.
[25] Jim Young, Patrick Graham, and Richard Penny. Using bayesian networks to create synthetic data. Journal of Official Statistics, 25(4):549, 2009.
[26] Lei Xu, Maria Skoularidou, Alfredo Cuesta-Infante and Kalyan Veeramachaneni, Modeling tabular data using conditional gan. Advances in Neural Information Processing Systems, 32, 2019.
[27] Jaewoo Lee, Minjung Kim, Yonghyun Jeong, and Youngmin Ro. Differentially private normal- izing flows for synthetic tabular data generation. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 36, pages 7345-7353, 2022.
[28] Danilo Rezende and Shakir Mohamed. Variational inference with normalizing flows. In International conference on machine learning, pages 1530-1538. PMLR, 2015.
[29] Akim Kotelnikov, Dmitry Baranchuk, Ivan Rubachev, and Artem Babenko. Tabddpm: Mod- elling tabular data with diffusion models. arXiv preprint arXiv:2209. /5421, 2022.
[30] Ahmed Alaa, Boris Van Breugel, Evgeny S Saveliev, and Mihaela van der Schaar. How faithful is your synthetic data? sample-level metrics for evaluating and auditing generative models. In International Conference Machine Learning, pages 290-306. PMLR, 2022.
[31] Tariq Alkhalifah, Hanchen Wang, and Oleg Ovcharenko. Mlreal: Bridging the gap between training synthetic data and real data applications in machine learning. Artificial Intelligence Geosciences, 3:101-114, 2022.
[32] Tingwei Shen, Ganning Zhao, and Suya You. A study on improving realism of synthetic data for machine learning. arXiv preprint arXiv:2304.124 12463,, 2023.
[33] Swabha Swayamdipta. Roy Schwartz, Nicholas Lourie, Yizhong Wang, Hannaneh Hajishirzi, Noah A Smith, and Yejin Choi. Datasct cartography: Mapping and diagnosing datasets with training dynamics. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 2275-9293, 2020.
[34] Curtis Northcutt, Lu Jiang, and Isaac Chuang. Confident learning: Estimating uncertainty in dataset labels. Journal of Artificial Intelligence Research, 70:1373-1411, 2011.
12

---

[35] Weixin Liang, Girmaw Abebe Tadesse, Daniel Ho, L Fei-Fei, Matei Zaharia, Ce Zhang, and James Zou. Advances, challenges and opportunities in creating data for trustworthy ai. Nature Machine Intelligence, ((8))669-677, 2022.
[36] Nabeel Seedat, Fergus Imrie, and Mihaela van der Schaar. De-check: A data-centric ai checklist to guide the development of reliable machine learning systems. arXiv preprint arX1v:2211.05764,
2022.
[37] Nabeel Seedat, Jonathan Crabbé, and Mihaela van der Schaar. Data-suite: Data-centric iden- tification of in-distribution incongruous examples. In International Conference on Machine Learning, pages 19467-19499. PMLR, 2022.
[38] Nabeel Seedat, Jonathan Crabbé, Zhaozhi Qian, and Mihaela der Schaar. Triage: Character- izing and auditing training data for improved regression. In Advances in Neural Information Processing Systems, 2023.
[39] Arthur Gretton, Karsten Borgwardt, Malte J Rasch, Bernhard Schölkopf, and Alexander Smola. A kernel two-sample test. The Journal of Machine Learning Research, 13(1):723-77, 2012.
[40] Léo Grinsztajn, Edouard Oyallon, and Gaël Varoquaux. Why do tree-based models still outperform deep learning tabular data?
[41] Pedro Baqui, Ioana Bica, Valerio Marra, Ari Ercole, and Mihaela van Der Schaar. Ethnic and regional variations in hospital mortality from covid-19 brazil: a cross-sectional observational study. The Lancet Global Health, 88)xc1018-e1026 2020.
[42] Takuya Akiba, Shotaro Sano, Toshihiko Yanase, Takeru Ohta, and Masanori Koyama. Optuna:
A next-generation hyperparameter optimization framework.
[43] Kevin Bello, Bryon Aragam, and Pradeep Ravikumar. DAGMA: Learning DAGs via m-matrices and log-determinant acyclicity characterization.
13

---

Appendix
A
Additional details
This appendix provides additional details on the data-centric methods used in our study, as well as outlining limitations and future work.
A.1 Data-centric Al methods
Our paper considers different data-centric Al methods perform data characterization, which forms the basis of our data profiles. As discussed in the main manuscript, the primary difference between these methods their scoring mechanism S. The three approaches considered in this paper next, i.e. Cleanlab, Data-IQ and Data Maps will be described next. Importantly, these methods are applicable to tabular data models like XGBoost, unlike several other data-centric methods which are only applicable to differentiable models. Recall that the data characterization methods apply threshold S. The implementation of r in this work is described in Appendix
Cleanlab [34]. Cleanlab estimates the joint distribution of noisy and true labels, thereby character- izing data into profiles easy and hard, where hard data points might indicate mislabeling. It operates on the output of classification models and can be applied to any modality. We focus on Cleanlab in the main manuscript, however, our framework could be applied with any of the following data-centric tools.
Data-IQ IQ 18
Data-IQ is training dynamics-based method, which characterizes data based the aleatoric uncertainty (inherent data uncertainty), i.e. E(P(rr, @1 - P(x, &))]. We are able to extract three data profiles: easy, ambiguous and hard. Typically, these "ambiguous" and "hard" examples harmful to model performance and might be mislabeled or "dirty".
Data Maps training dynamics-based method, which characterizes data based on the variabilitylepistemiic uncertainty (model uncertainty), i.e. V[P(x, +)]. We are able to extract three data profiles: easy, ambiguous and hard. Typically, these "ambiguous" and "hard" examples harmful to model performance and might be mislabeled or "dirty".
Data Maps 33
B Experimental details
By accident, the results reported in the main paper were based on data from 8 random seeds instead of the intended 10, as stated in the paper. To maintain consistency in the results, the results reported in the Appendix are also presented with 8 seeds.
B.1 Computational resources
All experiments were conducted using NVIDIA T4 16GB GPUs. The total number of GPU hours spent across all experiments is approximately 1000. All experiments were performed on the UCloud platform.
B.2 Hyperparameter tuning of generative models
In order to reduce the computational costs rumning the main experiment, we conducted a hy- perparameter search the Adult dataset instead of tuning the parameters for each dataset in benchmark. For each model (bayesian network, baayesian_network, ctgan, ddpm, nflow, tvae), we conducted a
search the hyperparameter space defined for each model in the synthcity [5] implementation.
We used Optuna [42] to conduct 20 trials for each model. The best hyperparameters for each model
listed in Table 3
B.3 Classification model set
Throughout the experiments, we fit a set of models from the seikit-learn li- brary, namely, XGBClassifier, RandomforestClassfffee, LogisticRegression.
14

---

Table 3: Hyperparameters used for the generative models.
Model
Parameters
bayesian network baytsikn.network
struct Laa__leannng__sardeee method: hillclimb,
struct aruut_learming__scon bic
ctgan
generator_n_ Laciio______ae____iia_____ hidden: 2,
generator_n_unit _utiiorr___nuisss_ee hidden: 50,
generator_nonling tanh, n_iter: 1000,
generator_ 0.0575,
httprmiiintto___aaes_________ hiddde:d: 4,
satiaattoor___nnnssshsihhhheeee hidden: 150, discriminator_n__me
discriminator_nnnne relu Ir: 0,000993750805526677667 batch_ size: 2929.
ddpm
num_timesteps. 998, n_iter: 1051, is ss_clasiffication: True n iter: 1000.
nflow
n_layers_hidden 10,
p_units_hiddent 98,
dropout: 0. 0.114996883367499366 batch_norm: True,
batch norm:
Ir: 0.0001,
linear mneear_transfomm_uge permutation, aase_transform _ttp:: rq autoregressivve
batch size: 512,
tvae
n_iter: 300,
Ir: 0.0002,
decoder layersss hidden:
weight_ dechht__eeay___ 0.001,
batch natch_size: 256,
u__units_om cunit_emmbedding: 200,
decoder_n_unie hidden: 300, decoder_nonling elu,
decoder_dropout: 19432251 19117226,
ancoder_n_layers___ hhidden:hidnn: 1.
sesoderrn_unit__See hidden: 450,
encoder encoder_nonling leaky relu,
encoder sncoder_dropout: 0.042885667000994711,,
DecissionTreeClasiifeer KlleighoorsCasssifeeee SVC, GaussianNB, and MLPClassifier. All hyperparameters were kept at their default value.
B.4 Datasets
Main experiments As described Sec we used the "Tabular benchmark numerical classification" downloaded from OpenML (suite id 337), filtered to only include datasets with less than 100.000 samples and less than 50 features for our main experiments. The number of samples, features, and links the datasets provided in Table
Noise dataset The dataset used to investigate the impact of label noise was the Covid mortality dataset from [41). The data includes data patients with Covid with label for whether the patient will die within 14 days. The dataset contains 6882 samples and 21 features.
Adult dataset The adult dataset is classic machine learning dataset, containing 48842 and 14 samples, with the task of predicting whether an individual's 's income exceeds $50k per year based on census data.
15

---

Table 4: Datasets used in the benchmark.
Task id
Damnet name
F_Jostures
n_samples
URL
361055
cradit
16714
hispar (opuns) /wwarch of &i task Teet Toct i8-3576xer Lecung S 36-36163
361060
electiiity
7
38474
361062
26
10083
361063
house_ 100
13438
Teleseape
13576
mankmmnneiing
SSTRBOL
1rpe TaPASCO1 20 Re001
defuit-of-rredh-enn---ii
B.Pp:: Tapentl ond Sypee walken
361277
califomia
20534
R1126 7/prm 002 ao 1eepee ST 19
301278
helee
10000
B..pa:7 /oppsss oag www. 7pp www.ECo Coe coreaor 2 1i0511 591278
Decis on TreeCaassiee
KNeighborsClassifeer
PClassifier
SVC
GaussianNB
LogisticRegression
XGBClassifier
bayesian_netweer
ctgan
ddpm
0.80
0.75
0.65
0.60
AUROC
0.55
nflow
baseline baseline
reselime-nt hard
easy hard baseline
easy hard-noorroooooo hard
0.80
0.70
0.65
0.60
0.55
baseline baseline
baseli ne no hard
easy hard-baselinee
easy hard no hard
baseline-baseline
baseline no hard
easy bard-basellnee
hard-n hard
Asea
pegrrocessing-posprreeees
Figure 5: Average classification performance across all datasets and generative models by pre- and postprocessing strategy and classification model type, using Cleanlab as the data-centric method.
C
Additional experiments
C.1 Performance other models
Jsbles classification performance across all classification models, averaged over all datasets. Random Forest and XGBoost performs the best across all combinations of pre- and postprocessing. hence, we chose to focus our results in the main manuscript on the XGBoost model. As evident from Table and Figure the benefit of data-centric pre- and postprocessing consistent across model types.
16

---

Table 5: Average classification performance across all datasets and generative models by pre- and postprocessing strategy and classification model type, using Cleanlab as the data-centric method. Numbers in parentheses indicate 95% bootstrapped CI.
Preprocessing
Postprocessing Classification model Strategy
AUROC
Strategy
baseline
baseline
Decision CreeClassfierr
0.64 (0.63, 0.64)
GaussianNB
0.7 (0.69, 0.71)
KNeighborsClassifer
0.65 (0.64, 0.66)
0.72 (0.7, 0.73)
MLPClassifier
0.7 (0.68, 0.71)
0.76 (0.75, 0.78)
SVC
0.69 (0.67, 0.7)
XGBClassifier
0.76 (0.75, 0.77)
no_ jo_hard
Treesssiee
0.66 (0.65, 0.67)
GaussianNB
0.71 (0.69, 0.72)
ClassiborsClassifier
0.66 (0.65, 0.67)
LogisticRegression
0.72 (0.7, 0.73)
MLPClassifier
0.72 (0.7, 0.73)
0.77 (0.75, 0.78)
SVC
0.69 (0.68, 0.71)
XGBClassifier
0.77 (0.75, 0.78)
easy_hard
baseline
Decision TreeClassffr
0.64 (0.63, 0.65)
GaussianNB
0.7 (0.69, 0.72)
KNeighborsClassffer
0.65 (0.64, 0.66)
LogisticRegression
0.72 (0.7, 0.73)
MLPClassifier
0.71 (0.69, 0.72)
RandomForeste Classsiee
0.77 (0.75, 0.78)
SVC
0.69 (0.68, 0.71)
XGBClassifier
0.76 (0.75, 0.77)
no hard
DecisionTreeClasse
0.67 (0.66, 0.68)
GaussianNB
0.71 (0.69.
KNeighhorsClassifer
0.66 (0.65,
LogisticRegression
0.72 (0.71,
MLPClassifier
0.72 (0.71, 0.73)
BandomForestClassee
0.77 (0.76, 0.78)
SVC
0.7 (0.68, 0.71)
XGBClassifier
0.77 (0.75, 0.78)
Real data
Decision TreeClassfer
0.74 (0.73, 0.75)
GaussianNB
0.73 (0.72, 0.74)
KNeighborsClassifer
0.73 (0.71, 0.74)
LogisticRegression
0.78 (0.77, 0.79)
MLPClassificr
0.77 (0.76, 0.79)
0.86 (0.85, 0.87)
SVC
0.74 (0.72, 0.75)
XGBClassifier
0.87 (0.86, 0.88)
17

---

Classification
361055
361060
361062
361083
0.7
361065
361066
361070
361273
0.9
bayesian_network nntwork
AUROC
ctgan
ddpm
0.7
nfoow
0.5
361275
361277
361278
baseline losselliee
baseline no hard
easy. hard baseline
easy hard no hard
0.9
0.7
0.5
baseline baseline
beseline hard
easy easy_hard haadh--a-asaleeee
easy hard-no hard
base leeeine--aelleeee
baseline-nd hard
eosy hard boseline
eassy hard hard
baseline baseline
baseline hard
easy, hard hww.haaarr--eeeee hard-baselline
easy hard-no hard
Preprocessing Pastprocessing Preproceesing-Potrroeesiii
Figure 6: Classification performance by pre- and postprocessing strategy for XGBoost across all
datasets using Cleanlab as the data-centric method.
C.2 Performance by dataset
As shown in Figures to the relative ranking of the five generative models remains largely consistent datasets. Specifically, the classification performance results shown in Figure (ouure6linder inddiiate that ddpm produces synthetic data that best retains the ability to train classification models.
The model selection results shown in Figure 7 display higher variance across datasets and pre- and postprocessing methods. This variance can likely be attributed to the findings presented in Table [] which demonstrate that the performance of several classification models is relatively similar. Consequently, minor performance variations caused by data-centric processing can significantly change the models' relative ranking.
As shown in Figure g data generated with the ddpm model tends to lead to the best ranking of important features. Consistently with the findings in classification performance and model selection, bayesian, bayesian_network model tends to perform the worst across datasets.
18

---

Model Selection
361055
361060
361062
361083
Spearman's Rank Correlation
361065
361066
361070
361273
bayesian_network network
ctgan
ddpm
nfoow
361275
361277
361278
baseline, lossellleeee
baseline no hard
easy. hard baseline
easy hard no hard
bascling baseline
besteline hard
easy www.haad-bslllin
easy hard-no hard
base eeeiiie--aellnee
baseline-nd hard
eosy hard bosoline
easy. haw.haadddo hard
base baseline
baseline hard
essy, hard hard-basellng
easy hard-no hard
Preprocessing Postprocessing PreproceesinyyPottpreessss
Figure 7: Model selection performance by pre- and postprocessing strategy across all datasets using Clcanlab as the data-centric method.
C.3 Identifying optimal thresholds for the data-centric methods
To identify the optimal threshold r for the scoring function S' for each of the data-centric methods, we constructed four simulated datasets with varying levels of label noise. In particular, we generated datasets with 10 features and 10 samples, 10 features and 50 samples, 50 features and 10 samples, and 50 features and 50 samples. The correlation of each feature with the outcome label was sampled from uniform distribution ranging from -0.7 to 0.7.
For each dataset, randomly flipped proportion (0, 0.02, 0.04, 0.06, 0.08, 0.1) of the outcome labels and recorded the indices of the flipped indices.
Data-IQ. Following the Data-IQ implementation, explored two methods for setting one based on whether the aleatoric uncertainty of the sample was below a certain percentile, and the other based on whether the alcatoric uncertainty below certain value. In both cases, to categorize sample as 'hard', the average predicted probability of the correct label (throughout the model training) had to be 0.25 and the aleatoric uncertainty had be below 'Easy' samples were defined as samples where the average predicted probability of the correct label was 2 0.75 and the aleatoric uncertainty categorized as 'ambiguous', We tested setting the [20, 30, 40, 50, 60, 70, 80) percentile, and using raw values of [0.1, 0.1 0.15, 0.175, 0.2].
was below Any remaining samples
Data Maps. The data-profiling procedure for Data Maps followed the same approach as Data-IQ, with the exception that the epistemic uncertainty was used instead of the aleatoric uncertainty. We tested the same values of used for Data-IQ..
19

---

Feature Selection
361055
361060
361062
361063
1
0.5
0-
Spearnan's Rank Correlation
-0.5
361063
361066
361070
361273
1-
bayesinn network network
cigan
ddpm
nllow
tvoe
-0.5
361275
361277
361278
baseline baseline
bavolline no, hard
baseelme
easy. hard hard-basellii
eess posee-ioo han
0.5
0-
-0.5
baseline reee-asellieee
baseline no hard
easy hard-- srrd-baselln line-
easy hard--nd hard
baseline baselime
baseline-no hard
sssy, hard-aa hard
baseline saloslllse
baselime hard
sasy hardaarrnn hard
easy Jaaad---aassee
easy.ha hard-baaaee
Figure 8: Feature selection performance by pre- and postprocessing strategy for XGBoost across all datasets using Cleanlab as the data-centric method.
Cleanlab. The Cleanlab package provides a 'label quality' score which was used to set categorize samples as "easy" or "hard'. We tested using the default score as well as the values [0.1, 0.125, 15, 0.175, 0.2]. For details of the theoretical basis of Cleanlab's 's thresholds, we refer to 34 Section 3. 1.
Experiment. For each augmented dataset (varying levels of label noise), we applied each of the data-centric methods and tested all the specified values of т. We recorded the number of samples identified as 'hard', Ahard as well as the number samples that were correctly identified as hard (i.e., correct
correct
flipped label) correct, Recall was calculated as pylppped
precision
and the Fl score as Ahard
2 precision recall
D-coll---rocision
Results. As shown in 6 Cleanlab achieves markedly higher Fl scores and recall compared to Data Maps and Data-IQ, albeit with slightly lower precision. Given substantially higher and recall scores, chose to focus the main analysis Cleanlab. We selected the threshold that maximised Fl for the main experiments, resulting 7 of 0,2 for all methods.
20

---

Table Average results across all simulated datasets and levels of noise. Numbers in parentheses indicate the standard deviation. The highest value per data-centric method and metric is highlighted bold. The value in bold for r indicates our chosen threshold for the main experiments.
Method
Type
F1
Recall
Precision
Cleanlab
cutoff
Default
0.407 (0.29)
0.633 (0.30)
0.332 (0.28)
0.100
0.384 (0.30)
0.354 (0.30)
0.5 (0.33)
0.125
0.403 (0.30)
0.393 (0.30)
0.478 (0.33)
0.150
0.41 (0.30)
0.42 (0.31)
0.45 (0.32)
0.175
0.417 (0.30)
0.45 (0.32)
0.428 (0.31)
0.200
0.42 (0.29)
0.478 (0.32)
0,408 (0.30)
Data-IQ
cutoff
0.100
0.083 (0.15)
0.051 (0.10)
0.507 (0.42)
0.125
0.118 (0.19)
0.076 (0.13)
0.562 (0.40)
0.150
0.16 (0.22)
0.11 (0.16)
0.547 (0.40)
0.175
0.206 (0.24)
0.153 (0.18)
0,523 (0.40)
0.200
0.221 (0.24)
0.17 (0.19)
0.513 (0.40)
percentile
20
0.033 (0.05)
0.019 (0.03)
0.279 (0.41)
30
0.057 (0.07)
0.038
0,433 (0.43)
40
0.075
(0.06)
0,371 (0.39)
50
0.099 (0.10)
0.067 (0.07)
0.373 (0.39)
60
0.138 (0.12)
0.093 (0.08)
0.407 (0.40)
70
0.17 (0.16)
0.12
0.4 (0.40)
80
0.199 (0.21)
0.146 (0.15)
0.438 (0.40)
Data Maps
cutoff
0.100
0.21 (0.24)
0.16 (0.18)
0.426 (0.39)
0.125
0.216 (0.24)
0.166 (0.19)
0.47 (0.40)
0.150
0.219 (0.24)
0.169 (0.19)
0.513 (0.40)
0.175
0.221 (0.24)
0.17 (0.19)
0.513 (0.40)
0.200
0.221 (0.24)
0.17 (0.19)
0.513 (0.40)
percentile
20
0.058 (0.07)
0.033 (0.04)
0.327 (0.37)
30
0.082 (0.10)
0.048 (0.06)
0.331 (0.37)
40
0.103 (0.12)
(0.07))
0.334 (0.37)
50
0.124 (0.14)
0.08 (0.09)
0.336 (0.37)
60
0.144 (0.17)
(0.77
0.376 (0.38)
70
0.163 (0.19)
0.114
0,419 (0.39)
80
0.183 (0.21)
0.133 (0.15)
0.423 (0.39)
C.4 Development of Figure
The results shown in Figure D were obtained performing the following steps: First, the Adult dataset was split into an 80% training set and a 20% test set. Second, each of the five generative models (with the hyperparameters specified in Table 3 was trained on the training set. Third, A synthetic dataset, D Deynth: with the same dimensions as training set was generated using each generative model. Fourth, the inverse KL divergence between Deynth and the training data was calculated, an XGBoost model was trained on Daynth and evaluated on the test set to measure the performance, and Cleanlab was applied to D Dyyuth to extract the proportion of easy and hard examples from each synthetic dataset.
21

---

Table 7: Percentage increase in performance from baseline, i.e. no data-centric pre- or postprocessing (as seen in Table 1,, per generative model for each pre- and postprocessing strategy, averaged across all datasets and seeds, using Data-IQ as the data-centric method.
Generative Model
Preprocesing
Postprocessing
Classification
Model Selection
Feature Selection
Strategy
Strategy
beyesisn_ network
buseline
no hard
0.27 (-6.25, 6.66)"
-3.49 (-98.06, 92.25).
-9.71 (-61.47, 44,447444,,94) 44,94))
easy amybi
baseline
00.92
2.33 115.12)1
-27.86 (-90.12, 33.36)
nn hard
-0.64 7.64))
63.18 (-35.66, 164.73)
-20.55 (-76.47, 35.22)
easy_hard
buseline
-0.33 (-7.34, 6.76)l
61.24 (-9.69, 131,4))
-64.55 (-128.52,- -4.18)
hard
.0.1 (-6.77, 6.99)
29.84 (:33.33, 93.02)1
-81.7 (-135.04, 223466)
ctgan
baseline
no hard
0.29 (-2.26, 2.75)"
2.13 2.13 (-9.95, 13.86)7
0.43 0.43 (-7.16, 7.78)7
easy_ambi_
(-3.24, 2.37))
8.51 (-5.0, 21.51)1
40.56
8.99X]
no hard
-0.17 (-2.7, 2.38),
6.96 (-5.12. 18. 18.29y7
0.56 (-9.08, 10.0y7
easy_hard
haseline
.0.08 (-2.83, 2.44)]
-12.02 (-25, 1.38))
-.02 (-10.59, 2.31)).
nard
(-2.5,, 2.83)
(-25.88. 0.12),
-3.97 (-11.06, 2.72),
ddpm
baseline
no no_hard
0.2448.88.4.02)
-0.23 (-16.27 15.02).
1.39 (-11.55, 13.66)1
easy_ambe hard
baseline
-5.08 (-9.82, 0.18)]
28.29 (-47,34, 9.988L
44.45 (-18.54, 9.27))
hard
-5.06 (-9.84 40.29).
-18.2 (-34.6, .0.06))
-5.43 (-20.07, 8.01))
easy_hard
baseline
(-7.9 0.98)L
-11.51 (-26.59, 3.12)
-10.6 (-22.83, 0.19).
no hard
(-7.3 1.19)
-17.66 (-34.07, -1.47)
-11.44 (-23.96, 0.79)
nflow
baseline
no hard
0.35 (-2.39, 3.07Y"
(-15.26 19.65)1
4.66 10,09, 19.15)1
easy_ambi_has _hrd
baseline
0.16 (-2.58, 2.79)
6.32 (-15.85, 27.4)
-1.57 (-17.11, 15.53)L
no
(-2.14 3.24)
(-14.67,
(-15.45, 16.06)1
easy hard
haseline
(-1.91, 3.62)"
5.65 (-13.41, 25, 15,13)1
6,18 (-9.79,11.522 21,52)
no no_hard
1.03 (-2.06, 3.67)*
3.46 (-15.85, 22.85)
4,82 (-10.25, 18.61)1
baseline
hard
0.19 (-2.63, 3.11y
3.27 (-11.38, 16.11
0.97 (-4.51, 6.27y?
easy easy_ambi_nte hard
baseline
-0.43 (:3.47, 2.35)
-6.48 ( 7.53)
-1.28 (9.16, 6.16).
no
(-3.37, 2.71))
(-20.01, 10.97)
(-7.57, 6.89).
easy_hard
baseline
1.02 (-1.93, 3.63)
(-32.66. 15)1
-0.8 (-5,49, 3.83)L
no hard
1.07 (-1.66, 3.97
-19.84 (-34.66, -5.19).
0.9 (-.11, 5.73V
C.5 Results with Data-IQ and Datamaps
The following subsections show the main results using Data-IQ and Data Maps as the data-centric method. Note, that statistical fidelity metries for Data-IQ and Data Maps are included in the table to reduce the complexity of the table, as they do not demonstrate any correspondence with task-related performance.
C.5.1 Data-IQ
As shown in Table reported Table However, the impact of data-centric processing is weaker and more variable when using Data-IQ. This effect be attributed to Data-IQ's tendency to identify fewer 'hard' examples (as shown in Table 6 by markedly lower recall) and thereby being a less strong intervention than Cleanlab.
and Figure 9 the results from using Data-IQ are relatively similar to what's
C.5.2
Data Maps
The main results using Data Maps as the data-centric method are presented Table and Figure 10 Similarly to Data-IQ, the variation across performance metrics is generally higher when using Data Maps compared Cleanlab. The benefits of Data Maps compared to no processing varies to some extent, with models like nflow seeming draw some benefit from it, while other models do not see major gains. This might again be attributed to the relatively low recall and the small number of identified 'hard' samples with Data Maps, which reduces the strength of the intervention.
22

---

Table 8: Percentage increase in performance from baseline, i.e., no data-centric pre- or postprocessing (as seen in Table D. per generative model for each pre- and postprocessing strategy, averaged across all datasets and seeds, using Data-IQ as the data-centric method.
Generative Model
Preprocessing
Posiprocessing
Classification
Model Selection
Feature Selection
Stralegy
Strategy
bayesian_netiern
baseline
no. no_l
(-.48.5508. 6.41)
-14.67 (+179,33, 148.67)
(-243.7, 116.631
easy_amble aulammii_ha hard
baseline
0.87 (:5.888 7.4)1
190.67 (39.33, 340.67)1
135.71 (:33.95, 301.53)1
no hard
(.4.4,,6 8.79)1
199.33 (52.67, 341.33)1
104.23 (-63.28, 272.74)|
easy_hard
baseline
(:3.58 8.95)T
(64.6774.67, 443.33)1
-5.7 (.223.73, 137.24)
no hard
(-3.46, 9.15)
213.33 (26.0, 400.0)"
-80.31 (-249.14 118.38)
ctgaan
haseline
no.
0.17 2.72)
(-7.26, 14.19)7
(-666, 6.91)1
basy_amb_hhar hard
baseline
-0.57 (-3.39, 2.17)
7.54 (-4.08, 18.38)*
(-16,4, 5,99)L
no,
-0.32 (:3.111, 2,47)
(1.34, 24.25)1
(-.. 5.62)).
casy_hard
baseline
-0.75 (-3.8, 2,14)
-1..2 (-28.21, -1.9).
-2.47 (-9.A, (-9.,, 4,63)_
no no_hard
-0.65 (-3.62, 2.23)
-10.34 (-22.79, 1.91L
-3.13 (-9.47, 3.35)L
ddpu
baseline
hard
(-3.2. 3.73)
100 (-24.66, 4.66)
1-32 13.9y,
easy__ hal.mbii_aaa hard
baseline
-3.98 (-8.9,
-1.77 (-25.59. 1.71.
(-18,22 8.67)1
hard
-4003 (-8.96, 0.77)
-9.23 (-35.29, 1
(-14 9.47))
easy. easy_hard hard
baseline
(-7.05, 1.58).
-0.93 -5.99..
4-16 6,67)]
no, no_hard
(-7.99, 1.12)L
16.277 (:28.99, -318))
(-12.95, 9.81)]
nflov
baseline
no_ no_hard
(-2.48. 2.75)
3.83 (-13.8, 21.77)*
(-1..33. 13.22N]
eaxy_ambi_ hard
easy, ___amiijjuu
baseline
0.77 (-1.9,, 3.22)
8.13 25.68y1
4,43 (-10.51 19:46)
hard
(-1.33, 3.75)
-0.22 (-16.59, 14.75)
5.48 (-9.09, 19.53y?
easy_hand
baseline
1,23 (-1.5, 4.15)1
-3.03 (-18.74, 12.84))
9.02 (-5.26, 22.42y"
ho_h
(-1.02
(-10.3, 22.49)
9:53 (-4.3, 21.967
tvae haseline
no hard
0.21 (-2.65, 2.92)
(-10.44 18.35)*
(16.94, 4.55))
easy_ cay__obi__hard
baseline
-0.45 (-3.26, 2.43)L
-14,39 (-28.0.0 0.3).
-1.25 (-7.76, 5.62))
no hard
-0.26 (-3.14, 2.68).
-14.68 0.
.0.67 (-8.08, 5.57))
easy_hard
baseline
0-2.13, 3,5yt
-947 (-24.33, 6,34))
-0.61 (-5.74, 4.4).
no_hard
0.88 3.551)
-10.3 4.321
0,31 4.711.
C.6 Performance on data with added label noise across all tasks and generative models
The performance of all generative models across all tasks (of which a subset is presented in Figure is shown in Figure II The overall trend is consistent across generative models, although not all models are affected equally hard by varying levels of noise. For instance, tvae and ctgan seem be more robust to noise than e.g. ddpm.
Classification
Model Selection
Feature Selection
0.80
0.6
Performance
0.75
0.4
0.70
0.2
taxsaline-to Rard
ensy ambi hard hasdline
B8Sy ambi hard no hard
easy bant baseline
ausy, handina no hard
bassline-busatine
laveline no hard
eas, ambi hard baseliiee
aay ambi no hard
easy harth baseliine
easy hardino hard
lassiins-baielne
laseline--e no hard
easy aimbl hans baseline
eas ambi hard-na hard
ediy. hard baselie
easy hardnn hard
Seapoocessinn-Pssroocesss
bayesian, network
cagan
ddom
nfoow
tvae
Figure 9: Average performance across all datasets for each generative model by pre- and postprocess- ing method using Data-IQ as data-centric method.
23

---

Table 9: Feature selection performance by classification model, averaged across all datasets and generative models using Cleanlab as the data-centric method. Numbers in parentheses indicate 95% bootstrapped CI.
Preprocessing
Postprocessing Strategy
Classification Model
Spearman's 's Rank Correlation
Strategy
baseline
baseline
DecisionTreeClaseee
0.68 (0.65, 0.71)
PandomForesiClasiffer
0.71 (0.68, 0.74)
XGBClassifier
0.51 (0.47, 0.55)
no_ mo_hard
Decision Clessssiie
0.67 (0.63, 0.69)
RandomFooestClassife
0.69 (0.66, 0.73)
XGBClassifier
0.51 (0.48, 0.56)
easy_hard
baseline
Decision TreeClassifie
0.69 (0.66, 0.72)
GandomFooestClasiine
0.73 (0.7,
XGBClassifier
0.52 (0.49,
no hard
Decision TreeClassifie
0.67 (0.63, 0.7)
0.71 (0.68, 0.74)
XGBClassifier
0.53 (0.49, 0.56)
C.7 Feature selection results with other models
As indicated in Table [and Figure 2
the feature selection results reported in the main manuscript stable across generative models and classification models. Across all comparisons, Spearman's rank correlation is calculated between the feature rankings of the same model type trained on mrai and syath In general, random forest and decision trees models tend to display higher concordance their feature rankings between models trained on synthetic data and models trained on real data compared to xgboost models.
Classification
Model Salaction
Feature Salection
0.6
0.80
0.6
Performnnc
0.75
0.65
0.2
0.2
basetine-baseeli Basiine-baseliiee
baseline-te 11 hard
easy, ambi hard tasotine
easy ambi hard hard
essy. hamt- tzapeine
awsy hard- hard
basetine hasallne
basseline no hard
easy anbbl hard bavelline
easy ambi hard hard
elay hard sassline
ewsy, hard-no hard
basetine basslinee
tasseline no hand
easy ambbl hard haseline
easy ambl hard no hard
GRay hart taseine
easy hardna hard
Proprocessing Postprocessing PorpesssiiipPPsspppccessssnn
bayesisnnneeenne network
oigan
depm nflow
tyse
Figure 10: Average performance across all datasets for each generative model by pre- and postpro- cessing method using Data-IQ as the data-centric method.
24

---

jayesian_netwer network
ctgan
ddpm
nflow
tvae
0.76
0.74
classifcation
0.72
0.70
0.68
0.8
Model Selection
0.6
0.4
Performance
0.2
Feature Selection
0.2
Statistical Fidelity
0.995
0.990
0.985
10
10
10
Percent label noise
baseline-baselle baseline-no hard
hasy_hard-bassllne
easy_hardee heryynord-no__ae hard
Figure 11: Performance across models and tasks on the Covid mortality data with added label
noise. Using Cleanlab as the data-centric method.
C.8 Causal Discovery
As proof of concept, we investigated whether data-centric processing might be useful for causal discovery. For this purpose, we used DAGMA 43 to estimate causal graph, DAGiriin based on
D DAGuai was used as the reference graph, and compared to causal graph estimated from syath Results for single dataset are shown in Figure 13 in terms of Structural Hamming Distance (SHD). The results appear promising in this particular example, however, further work is required.
25

---

DeclsionTreeClassffee
RandoifrrestClassffeer
XGBClassifler
bayesian_netwae nettoink
ctgan
ddpm
Spearman's Rank Correlation
0.6
0.4
nflow
tvae
base baseline: finelieeee re-baallliee
easy easy_hard har.hardd-.. rec--baleinne
easy easy_haaee hard-no hard
baseline-mi
0.6
0.4
baseline baseline
baseline no hard
easy hww.haarraaaa arddbbaseline
easy easy_hare har_aaadddoo haadd
baseline baseline
baseline ne- hard
easy hard baseline
easy hard no hard
Figure 12: Mean feature selection performance across all datasets by classification model and generative model, using Cleanlab the data-centric method.
govesian_netwonn ctgan
ddpm
nflow
tvae
20
SHD
18
baseline baseline
baseline-ne no hard
easy hard baseline
easy hard-no, hard
Preprocessing Postprocessing
Figure 13: Causal graph discovery performance using Cleanlab as the data-centric method on single dataset (id=361055).
26

---

C.9 Distribution of Scores
C.9.1 Main Experiment
To investigate the variability of the performance across the three main tasks, we show the distribution of performance metrics across all generative models, datasets, and processing strategies. The findings for classification performance, feature selection, and model selection are shown in Figure [1] Figure
15 and Figure 16] respectively. All figures are shown using Cleanlab as the data-centric method. Overall, the difficulty of the datasets highly variable, and the variance by generative model heterogeneous. In general, the best performing models, such as TabDDPM, exhibit comparatively lower variability across different seeds. In contrast, models with poorer performance, like Bayesian networks and normalizing flows, display higher variability across seeds.
The variance the performance metries higher for the tasks of feature selection and model selection than classification. This discrepancy can likely be partly attributed to the inherent smoothness of the performance metrics. The model ranking score is established by accurately ranking the eight supervised classification models listed in Appendix [B3] Similarly, the feature selection score is determined by correctly ranking the metrics for feature importance across all features within the dataset quantity that naturally varies by dataset.
In contrast, classification performance relies AUROC values computed the test dataset, which encompasses a relatively large number of samples (1,141 in the smallest dataset). Due to the dissimilarity in the number of samples, models, and features, the metrics for model and feature ranking are inevitably more susceptible to larger variance. This occurs because disparities in a single element lead to more substantial effects the performance metries in comparison to the impact of single elements for classification metric.
27

---

Classification
361055
361060
361062
361083
361065
361086
361070
361273
361275
361277
361278
150
bayesian network
100
50
200
150
cigan
100
50
400
Density
300
28
ddpm
200
100
100
nnoo
50
150
100
50
0.4
0.6
0.8
10.4
0.6
0.8
10.4
0.6
0.8
10.4
0.6
0.8
10.4
0.6
0.8
10.4
0.6
0.8
10.4
0.6
0.8
10.4
0.6
0.8
10.4
0.6
0.8
10.4
0.6
0.8
10.4
0.6
0.8
AUROC
Preprocessing postprocessing
baseline baseline
baseline no_hard
hary baseline
easy_het b_r_hardd- no no_hard
Figure 14: Distribution of classification performance across the random seeds by generative model (rows) and dataset ids (columns).

---

Feature Selection
361055
361060
361062
361063
361065
361066
361070
361273
361275
361277
361278
25
bayesian network
20
10
30
20
cigan
Density
29
40
0
6
nnoow
2
0
15
w80
5
0 0.5
-0.5 0.5
-0.5
0.5
-0.50
0.5
00.50
0.5
-0.50
0.5 1
-0.5
0
0.5
-0.50 0.5
-0.50 0.5
-0.5
0.5
-0.5
0.5
Spearman's Rank Correlation
Preprocessing postprocessing
baseline bascline
baseline no_ hard
easy_hard Sard_ baseline
easy_hard bard_ no_ norhard
Figure 15: Distribution of feature selection performance across the random seeds by generative model (rows) and dataset ids (columns).

---

Model Selection
361055
361060
361062
361063
361065
361066
361070
361273
361275
361277
361278
bayesian, network
6
4
2
90
cigan
30
30
Density
30
20
10
0
20
now
10
0
90
30
-0.5
0.5
-0.5
0.5
-0.5
0.5
0.50
0.5
-0.5
0.5
-0.5
0.5
1
-0.5
0
0.5
-.5
0.5
-05
0.5
-0.5
0.5
-0.5
0.5
Spearman's Rank Correlation
Preprocessing postprocessing
baseline bascline
baseline _o_ hard
easy_hard hard baseline
easy_har Sard_ no_ mord
Figure 16: Distribution model selection performance across the random seeds by generative model (rows) and dataset ids (columns).

---

C.9.2 Label noise
To investigate the variability in the label noise experiments, we show the distribution of performance metrics across all generative models, proportions of label noise, and processing strategies. The findings for classification performance, feature selection, and model selection are shown in Figure 17 Figure 18 and Figure 19] respectively. Further, we show the performance across the generative models, summarized over the levels of label noise, in Table and show the performance by the level of label noise, summarized over all generative models, in Table IT using Cleanlab as the data-centric method.
All figures and tables are shown
31

---

Classification
0
4
8
10
bayesian network
200
100
80
oggan
40
20
40
Density
30
33
10
120
nnlow
60
60
20
0.65
0.70
0.75
0.80
0.65
0.70
0.75
0.80
0.65
0.70
0.75
0.80
0.65
0.70
0.75
0.80
0.65
0.70
0.75
0.80
0.65
0.70
0.75
0.80
AUROC
Preprocessing postprocessing
baseline baseline
baseline no hard
casy_hard baseline
easy_hard _ard_ no mo_hard
Figure 17: Distribution of classification performance across the random seeds by generative model (rows) and percent label noise (columns).

---

Feature Selection
2
10
bayesian network
7.5
5
2.5
20
15
Density
33
5
2.5
0
25
20
15
nloow
10
0
20
10
5
-0.25
0
0.25 0.50
0.75
-0.25
0.25
0.50
0.75
-0.25 0
0.25
0.50
0.75
-0.25
0.25
0.50
0.75
-0.25
0
0.25
0.50
0.75
-0.25
0.25 0.50
0.75
Spearman's Rank Correlation
Preprocessing postprocessing
baseline baseline
baseline
hard
easy_hard baseline
masy_hard- hard
Figure 18: Distribution of feature selection performance across the random seeds by generative model (rows) and percent label noise (columns).

---

Model Selection
2
10
bayesian network
7.5
5
2.5
15
10
cigan
5
15
Density
10
34
0
15
MOU
5
0
6
2
0.25
0.50
0.75
1
0
0.25
0.50
0.75
0.25
0.50
0.75
0
0.25
0.50
0.75
0.25
0.50
0.75
1
0
0.25
0.50
0.75
Spearman's Rank Correlation
Preprocessing postprocessing
baseline baseline
baseline hard
easy_hard baseline
easy_hard hard
Figure 19: Distribution of model selection performance the random seeds by generative model (rows) and percent label noise (columns).

---

Table 10: Summarised performance for the label noise experiments across all levels of label noise. Classification is measured by AUROC, feature selection and model selection by Spearman's 's Rank Correlation, and statistical fidelity by inverse KI. divergence. Numbers show bootstrapped mean and 95% CL.
Generative Model
Prepracessing
Posppocessing
Clussification
Feature Selection
Model Selection
Statistical Pidelity
Stralogy
Stralegy
Real data
None
None
0.766 (0.764, 0.767)
Twyesian_netwook network
baseline
baseline
0.69 0.69 (0.678, 0,704)
0.393 (0.349, 0,435)
0.376 (0.284, 0,46)
1.0 1.0)
no
0.697 (0.684,
0.417 (0.371. 0,462)
0.539)
(1.0. 1.0)
hand
buseline
0.742 (0.7377 0.747)
0.535 (0.505, 0.566)
0.44 (0.336, 0.526)
(1.0, 1.D)
no hard
0.751 (0.744, 0,756)
0.522 (0.4778, 0.563)
0.608 (0.536, 0.683)
(.0. 1.0)
otgan
baseline
0.741 (0.734, 0.746)
(0.62, 0.7)
(0:346, 0,559)
0.995 (0:994, 0.996)
no hard
0.752 (0.7477 0,757)
0.63 (0.601. 0.6611
0.56 (0.467, 0.651)
0.995 (0.994, 0.996
eary_aard
buseline
0.734 (0.73. 0.738)
0.62 (0.581, 0.666)
0.366 (0.292, 0.442)
0.998 (0.997. 0.998)
no hard
0.746 (0.74, 0.751)
0.58 (0.54, 0.625)
0.544 (0.46, 0,626)
0.998 (0.997, 0.998)
dlppm
baseline
haseline
0.685 (0.677, 0.693)
0.272 (0.208, 0 0.338)
0.294 (0.23, 0.36)
(1.0, 1.D)
no,
(0.212. 0.356)
(0.229.
1.0)
easy hand
baseline
0.7.10.71.0.7200
0.43 (0. 347. 0.511)
0.309 (0.217, 0.398)
0.999 (0.999, 0.999)
no, hard
0.728 (0.717, 0,739)
0,439 (0.365, 0.51)
0.518 (0.421, 0.626)
0.999 (0.999, 0.999)
aflow
baseline
baseline
0.686 (0.674, 0.697)
0.325 (0. (68268, 0.378)
0.393 (0. (0.328. 0,457)
0.995 (0.995, 0.996)
no hard
0.699 (0.686, 0,711)
0,344 (0.294, 0.396)
0.467 (0.37, 0.556)
0.995 (0.995. 0.996)
easy_han hand
haseline
0.707 (0.599, 0.716)
0.342 (0.286, 0.395)
0.521 (0.45, 0.586)
0.995 (0.995, 0.996)
no hard
0.717 (0.71, 0.724)
0.324 (0.26, 0.383)
0.577 (0.488, 0.663)
0.995 (0.995, 0.996)
baseline
baseline
0.735 (0,731, 0,74)
0.64 (0.605, 0,671)
0.41 (0.327, 0.498)
0.987 (0.986, 0.989)
no
0.747 (0.743, 0.752)
0.679)
0.537
0.987 (0.985,
eary_hard
buseline
0.722 (0.715, 0.729)
0,608 (0.558, 0,663)
0.437 (0.356, 0.519)
0.988 (0.987. 0.99)
hard
0.732 (0.725, 0.738)
0.601 (0.547, 0.658)
0.415 (0.315, 0.506)
0.987 (0.986. 0.989)
C.10
Other Measures of Statistical Fidelity
Table 12 shows additional measures of statistical fidelity (Wasserstein Distance (WD) and Maximum Mean Discrepancy (MMD) for the main experiment.
35

---

Table 11: Summarised performance for the label noise experiments across all generative models. Classification is measured by AUROC, feature selection and model selection by Spearman's 's Rank Correlation, and statistical fidelity by inverse KL divergence. Numbers show bootstrapped mean and 95% CI. A value of "None' in Pre- and postprocessing strategy indicates that the results are for models trained on the real data.
Prop.
Preprocessing
Postprocessing Strategy
Classification
Feature Selection
Model Selection
Statistical Fidelity
hel noise
Strategy
0.0
None baseline
None
0.775 (0.773, 0.777)
baseline
0.728 (0.717, 0.738)
0.482 (0.39%, 0.559)
0.475 (0.365, 0.588)
0.996 (0.994, 0.997)
no hard
0.739 (0.728, 0.749)
0.481 (0.411, 0.558)
0.657 (0.545. 0.748)
0.996 (0. 0.997)
easy_hue hard
baseline
0.74 (0.733, 0,747)
0.546 (0.46, 0.621)
0,581 (0.474, 0.688)
0.997 (0.995, 0.998)
bo_hard
0.749 (0.741, 0.756)
0,546 (0.471, 0.62)
0.703 (0.626, 0.769)
0.997 (0.995, 0.998)
0.02
None baseline
None
0,771 (0.768, 0.774)
baseline
0.715 (0.705, 0.725)
0.441 (0.344, 0.529)
0.459
0.536)
0.995 (0,993, 0,997)
no, hard
0.727 (0.716, 0.737)
0.489 (0.411, 0.558)
0.503 (0.796. 0.604)
0.995 (0.993, 0,997)
easy easy_hard
baseline
0.732 (0.724. 0.739)
0.568 (0.492, 0.631)
0.434 (0.346, 0.53)
(0.994, 0.998)
po_ hard
0,746 (0.738, 0.753)
0.557 (0.498, 0.611)
0.607 (0.529. 0.68)
0.996 (0.994, 0.998)
0.04
Nome
0.767 (0.765, 0.77)
baseline
haseline
0.704 (0.691, 0.718)
0.457 (0.37, 0.54)
0.397 (0.321, 0.484)
0.996 (0.994, 0.997)
no, hard
0.714 (0.699. 0.728)
0.447 (0.369, 0,525)
(0.34, 0.526)
0.995 (0.993, 0,997)
easy. easy_hard
baseline
0.729 (0.723, 0.735)
0.513 (0.4417 0,581)
0,362 (0.276. 0.449)
0.996 (0.995, 0.9988
bo_hard
0.738 (0.732, 0.745)
0.485 (0.41, 0.555)
0,599 (0.507. 0.704)
0.996 (0.993, 0.998)
0.06
None baseline
None
0,765 (0.762, 0.767)
haseline
0.7 (0 (0.6688.00 0.713)
0.449 (0.367, 0,534)
0.389 (0.303, 0,481)
0.996 (0.993, 0.998)
no hard
0,71 (0.697, 0.726)
0.443 (0.351, 0,526)
0,536 (0.463, 0.61)
0.995 (0.993, 0,997)
easy_ hard
baseline
0.723 (0.712, 0.733)
(0:418. 0.565)
(0.355, 0.519)
(0,995, 0,998)
no hard
0,733 (0.723, 0.742)
0.468 (0.395, 0,535)
0.597 (0.498, 0.689)
0.996 (0.993, 0.998)
0.08
None baseline
None
0.759 (0.756, 0.762)
haseline
0,701 (0.686, 0.71.5)
0.501 (0.428, 0.577)
0.317 (0.232, 0.409)
0.995 (0.993, 0,997)
no_ hard
0,709 (0.695, 0.724)
0.463 (0.389, 0,532)
0,343 (0 (0.26. 0,454)
0.995 (0.993, 0.997)
easy hard
haseline
0.718 (0.709. 0.726)
0.47 (0.376, 0,554)
0,356 (0.262, 0.463)
0.996 (0.994, 0.998)
no_ no_hard
0.729 (0.72, 0.738)
(0.39, 0.558)
0.351 (0.266, 0.441)
0.996 (0.994, 0,997)
0.1
None
None
0,756 (0.753, 0.76)
baseline
baseline
0.697 (0.68, 0.714)
0.42 (0.329. 0.504)
0.267 (0.19, 0.354)
0.995 (0.993, 0.997)
no_ hard
0,703 (0.686, 0.721)
0.455 (0,373, 0,534)
0.296 (0.212, 0.385)
0.995 (0,993, 0.997)
easy_ hard
baseline
0.706 (0.697, 0.715)
0.452 (0.387, 0.513)
0.319 (0.238, 0.408)
0.996 (0.994, 0,998)
no. hard
0.713 (0.703, 0.722)
0.42 (0.348, 0.487)
0.326 (0.255. 0.406)
0.996 (0.994, 0.998)
Table 12: Additional measures of statistical fidelity for the main experiment, summarized across all datasets. All numbers have been rounded to decimals. Inv. KL-D = inverse Kullback-Leibler Divergence, WD Wasserstein Distance, MMD Maximum Mean Discrepancy.
Generative Model
Preprocessing
Postprocessing
Inw. KI.-D
WD
MMD
Strategy
Strategy
bayesian_nctwee network baseline
baseline
0.998 (0.998, 0.999)
0.041 (0.018, 0.067)
0.001 (0.0, 0.003)
no hard
0.998 (0.998, 0.999)
0.148 (0.034, 0.34)
0.001 (0.0. 0.003)
easy hard
baseline
0.998 (0.998, 0.999)
0.04 (0.017. 0.068)
0.001 (0.0, 0.003)
no_ no_hard
0.998 (0.998, 0.999)
0.146 (0.035, 0.332)
0.001 (0.0, 0.003)
ctgan
baseline
baseline
0.979 (0.967, 0.987)
0.022 (0.01, 0.037)
0.001 (0.0, 0.002)
no mo_hard
0,978 (0.967, 0.986)
0.534 (0.041, 1,481)
0.001 (0.0. 0.002)
easy_hard
baseline
0.979 (0.968, 0.986)
0.022 (0.01, 0.036)
0.001 (0.0, 0.002)
no hard
0.977 (0.967, 0.985)
0.546 (0.039, 1.518)
0.001 (0.0, 0.002)
ddpm
baseline
basclinc
0,846 (0.685, 0.979)
0.717 (0.17. 1.409)
0.0 0.001)
no hard
0.845 (0.688, 0.971)
1.085 (0.326, 1.966)
0.0 (0.0, 0.001)
easy_hard hard
baseline
0.849 (0.709. 0.967)
0.744 (0.137. 1.42)
0.0 (0.0. 0.001)
no hard
0.848 (0.714, 0.967)
1.066 (0.363, 1.874)
0.0 (0.0, 0.001)
nflow
baseline
baseline
0.975 (0.968, 0.981)
0.074 (0.038. 0.115)
0.001 (U.0. 0.002)
no_ no_hard
0.975 (0.968, 0.981)
0.519 (0.079, 1.351)
0.001 (0.0, 0.002)
hard
baseline
0.975 (0.967, 0.981)
0.075 (0.04, 0.113)
0.001 (0.0, 0.003)
no mordard
0.975 (0.967, 0.982)
0.546 (0.076. 1.439)
0,001 (0.0. 0.003)
Ivae
baseline
baseline
0.966 (0.952, 0.977)
0.026 (0.012, 0.041)
0.0 0.001)
no hard
0.965 (0.952, 0.977)
0.531 (0.046, 1.465)
0.0 (0.0, 0.001)
easy_hard
haseline
0.968 (0.957, 0.978)
0.026 (0.012, 0.041)
0.0 (0.0, 0.001)
no hard
0.968 (0.957, 0.978)
0.483 (0.047. 1.32)
0.0 (0.0.
36

---

Table 13: Summarised performance across all datasets, with an additional row showing the perfor- mance of the real data with data data-centric postprocessing.
Generative Model
Preprocessing
Postprocessing
Classification
Feature Selection
Model Selection
Strategy
Strategy
None
None
None
0.866 (0.856, 0.876)
no h__r
0.859 (0.837, 0.88)
0.937 (0.926, 0.946)
0.697 (0.647, 0.747)
prvessannnntwoote
buseline
baseline
0.622 (0,588, 0.656)
0.091 (-0.004, 0.188)
0.155 (0.047, 0,263)
no no_dard
0.624 (0.59, 0.659)
0.043 (-0.065, 0.143)
0.112 (0.021. 0.207)
hary_hard
baseline
0.624 (0.592, 0.66)
0.1 (-0.015, 0.213)
0.297 (0.202, 0.387)
no no_hard
0.627 (0.594, 0.661)
0,082 (-0.033, 0.195)
0.197 (0.104, 0.291)
ctgan
baseline
bascline
0.797 (0.771, 0.822)
0.63 (0.568, 0.699)
0.519 (0:457. 0.581)
no hard
0.807 (0.782, 0.829)
0.653 (0.592, 0.714)
0.578 (0.512, 0,648)
easy_hard
baseline
0.791 (0.763, 0.818)
0.63 (0.566, 0.692)
0.513 (0.441, 0.576)
mo_hard
(0.772, 0.828)
0.631 (0.566, 0.691)
0.594 (0.531, 0.655)
ddpm
baseline
baseline
0.813 (0.784, 0.842)
0.635 (0.552, 0.718)
0.508 (0.442. 0,571)
hard
0.818 (0.787, 0.849)
0.645 (0.569, 0.726)
0,476 (0.406, 0.543)
hurd
baseline
0.818 (0.788, 0.844)
0.645 (0.564, 0.723)
0.544 (0.476, 0.61)
no. hard
0.824 (0.796, 0.85)
0.661 (0.584, 0.738)
0,539 (0.466, 0.604)
aflow
bascline
bascline
0.737 (0.713, 0.761)
0.415 (0.342, 0.486)
0,348 (0.275. 0.415)
no hard
0.745 (0.718, 0.768)
0.439 (0.372, 0.503)
0.357 (0.292, 0.425)
easy_hard
baseline
0.742 (0.718, 0.768)
0.455 (0.384, 0.527)
0.362 (0.297, 0.428)
no_hard
0.749 (0.725, 0.772)
0.445 (0.365, 0.511)
0.371 (0.31, 0.436)
baseline
baseline
0.792 (0.766, 0.816)
0.675 62.62, 0.723)
0.506 (0.438, 0,575)
no_ no_hard
0.8 (0.775, 0.828)
0.675 (0.63, 0.721)
0.488 (0.419, 0.561)
easy_hard
baseline
0.79 (0.763, 0.814)
0.687 (0.642, 0.73)
0.477 (0.416, 0,538)
no. nardard
0.797 (0.771, 0.822)
0.699 (0.653, 0.742)
0.507 (0:437, 0.578)
C.11 Postprocessing the real data
Tables 13 and 14 investigate the effect of postprocessing the original data (i.e. removing hard examples) on the main experimental benchmark data and on the datasets with added label noise, respectively. The performances for feature and model selection are obtained by comparing the feature and model rankings against the potpposppooeeeeed version the real data, similar manner as the performance of the generative models. 'he results indicate that postprocessing has minor impact on classification: slight negative effect on the purportedly clean benchmark datasets, and a slight positive effect the datasets with added label noise. Spearman's 's Rank Correlation for feature selection remains high across both the main benchmark datasets (0.94) and the datasets with label noise (0.88), whereas the ranking of models is lower at approximately 0.7 for both types of data.
37

---

Table 14: Summarised performance across all datasets with added label noise, with an additional row showing the performance of the real data with data-centric postprocessing.
Generative Model
Preprocessing
Postprocessing
Classification
Feature Selection
Model Selection
Strategy
Strategy
None
None
None
0.766 (0.764, 0.767)
no. hard
0.776 (0.773, 0.779)
0.883 (0.868, 0.896)
0.704 (0.662, 0,744)
bayesian_nctwork network
baseline
basclinc
0.69 (0.678, 0.703)
0.393 (0.348, 0.439)
0.376 (0.284, 0.47)
no_ no_hard hard
0.697 (0.685, 0.71)
0.417 (0.371, 0.462)
0.45 (0.359, 0.55)
easy_hard
baseline
0.742 (0.736, 0.747)
0.535 (0.503, 0.565)
0.44 (0.346, 0.54)
no_ no_hard
0.751 (0.745, 0.756)
0.522 (0.485, 0.562)
0.608 (0.533, 0.685)
ctgan
baseline
baselinc
0.741 (0.734, 0.746)
0.662 (0.621, 0.7)
0.451 (0.352, 0.565)
haad
0.752 (0.747, 0.757)
0.63 (0.6, 0.658)
0.56 (0.463, 0.646)
easy_ hard
baseline
0.734 (0.73, 0.738)
0.62 (0.58, 0.665)
0,366 (0.29, 0.443)
no_ hard
0.746 (0.74, 0.751)
0.58 (0.539, 0.624)
0.544 (0.46, 0.624)
ddpm
baseline
baseline
0.685 (0.677, 0.693)
0.272 (0.21. 0,333)
0.294 (0.232, 0,357)
no, hard
0.69 (0.682. 0,699)
0.28 (0.203, 0.35)
0.298 (0.228, 0.367)
casy casy_hard hard
basclinc
0.72 (0.711, 0.73)
0.43 (0.345, 0.518)
0.309 (0.222, 0.404)
pordard
0.728 (0.717, 0.738)
0.439 (0.365, 0.508)
0.518 (0.417, 0.635)
nflow
baselime
baseline
0.686 (0.675, 0.698)
0.325 (0.266, 0.384)
0.393 (0.328, 0.457)
no no_hard
0.699 (0.685, 0,711)
0.344 (0.295, 0.396)
0,467 (0.363, 0.56)
cusy hard
baseline
0.707 (0.699, 0.715)
0,342 (0.279. 0.395)
0.521 (0,449, 0,394)
no. ho_hard
0.717 (0.71, 0.725)
0.324 (0.261, 0.385)
0.577 (0.491, 0.667)
tvae
baseline
baseline
0.735 (0.731, 0.739)
0.64 (0.609, 0.673)
0.41 (0.331, 0.494)
no_ no_hard
0.747 (0.743, 0.752)
0.643 (0.608, 0.676)
0.537 (0.459, 0.621)
easy hard
baseline
0.722 (0.715, 0.729)
0.608 (0.56, 0.66)
0.437 (0.355, 0.523)
no_ no_hard
0.732 (0.725, 0.738)
0.601 (0.546, 0.653)
0,415 (0.325, 0.521)
D
Statistical tests
This appendix provides details on the statistical significance of the effect of data-centric processing on both the datasets in the main experimental benchmark and on the datasets with added noise. Subsection D.I provides a summary of the insights gained from the tests, and subsections and D.3 provide details on the main experiment and label noise experiment, respectively.
and
D.1 Summary
On the label noise experiments, the data-centric processing strategies have beneficial effect across all tasks (see Table [6), with significant effects of pre- and postprocessing strategies across all tasks, except postprocessing for feature selection. The effect of data-centric processing is not significantly different across varying levels of label noise. There is slight tendency towards an inverted V shape effect of the processing strategies, i.c., larger benefits with moderate levels of label noise, however, this is not significant.
For the main experimental benchmark data, data-centric processing led to significant improvements in performance for classification (postprocessing only), and model selection (preprocessing only), as shown in Table [5] The diminished effect of data-centric processing in the main experimental benchmark compared to the datasets with added label noise might be caused by the extensive preprocessing of the datasets in the benchmark as mentioned in See 4.1 The datasets could therefore be speculated to too "clean" to receive substantive benefits from data-centric processing.
Measures of model fit (R²) of the models in the main benchmark data indicate that the variance in task performance is best explained in the classification task (1²² = 0.98), slightly worse in the feature selection task (Rt² = 0.85), and substantially lower in the model selection task (It2 = 0.57). This might be due to the smoothness of the performance metries. The model ranking score is established by accurately ranking the eight supervised classification models listed in Appendix B.3] Similarly, the feature selection score is determined by correctly ranking the metrics for feature importance across all features within the dataset a quantity that naturally varies by dataset. In contrast, classification performance relies on AUROC values computed on the test dataset, which encompasses a relatively large number of samples (1,141 in the smallest dataset). Due to the dissimilarity in the number of samples, models, and features, the metrics for model and feature ranking are inevitably more susceptible to larger variance. This occurs because disparitics in single element can lead to more
38

---

Table 15: Main effect of the pre- and postprocessing strategies of the main experiment, controlling for the interaction between dataset and generative model. The estimates represent the difference in the corresponding performance metric (AUROC or Spearman's Rank Correlation depending on the task), compared to the baseline strategy, i.e. no processing.
Task
Processing
Estimate
Std. Error
p-value
Significant
Classification
preprocessing strategy[[.ccasy _ardd
patrrocssinng_sttatgyTT...om
0.001
0.001
0.526
0
postrocessing_straatggyTTTnan strategy[T.no__mrg hard]
0.007
0.001
<0.000
Feature Selection
preprocessing_.ttraiee strgsstttrtttyyeeeeeeee hard]
0.013
0.008
0.091
0
0.003
0.008
0,706
0
Model Selection
wreposcesing__straeeee
_aad]]
0.031
0.012
0.008
1
postprocessing strategy[T [T.no hard]
0.003
0.012
0.822
0
substantial effects on the performance metrics in comparison to the impact on classification. In summary, this means that the performance metrics for feature and model selection more sensitive to minor deviations or random noise than classification.
D.2
Main experiment
Three linear models were constructed to statistically assess the effect of the data-centric processing strategies on the performance each task (classification, feature selection, model selection). The models controlled for the main effects of the datasets and generative models, as well as the interaction between the two, as the different datasets might have characteristies that make them more or less difficult for specific generative models. The models took the following formula:
metric ~ dataset_ic id senerative_modee + preprocessing strategy + postprocessing_sstraeee
Where metric represents the performance metric corresponding the task, i.e., AUROC for classifi- cation or Spearman's 's Rank Correlation for feature and model selection.
Table 15 shows the main effect of the processing strategies for each of the tasks. Tables 17 18 and 19 show the full output of the models.
D.3 Label noise
Similarly to the main experiment, three linear models were constructed to statistically assess the effect of the data-centric processing strategies each task's performance with different label noise levels (see Figure(). The models controlled for the main effects of level of label noise and generative model, and took the following formula:
metric prop_label_none generative_ Senerative_model-e + preprocessing_straine sttratggyggttggyygeeeee + postprocessiing_strace strattgy
Where metric represents the performance metric corresponding to the task, i.e., AUROC for classifi- cation Spearman's 's Rank Correlation for feature and model selection.
Table 16 shows the main effect of the processing strategies for each of the tasks. Tables 20 and 22 show the full output of the models.
To assess whether the effectiveness of the data-centric processing strategies is modulated by the level of label noise in the data, we constructed a linear model similar to the one above, but with an interaction effect between the proportion of label noise and the data-centric processing strategies. The models took the following form:
metric ~ generative_moddee
label noise* http://ssssnnsttttttttog-sssorro
The output of these models is shown in Tables 23
24 and
25
Notably, the model with interaction effects had slightly higher Akaike's Information Criteria than the model without, indicating worse fit to the data.
39

---

Table 16: Main effect of the pre- and postprocessing strategies of the label noise experiment, controlling for the effect of the proportion of label noise and generative model. The estimates represent the difference in the corresponding performance metric (AUROC or Spearman's Rank Correlation depending on the task), compared to the baseline strategy, L.c. no processing.
Task
Processing
Estimate
Std. Error
p-valuc
Significant
Classification
preprccesinng_strateeyy Stray hard]
0.017
0.002
40.000
postprocessing stralegy[T.no hard]
0.010
0.002
<0.000
Feature Selection
seeprocesingg_strateee Strategy[Teeasy hard]
0.039
0.013
0.002
postprocessing strategy[[.no hard]
-0.005
0,013
0.710
0
Model Selection
dreprocesinng_strateee
Strategy[T.easy hard]
0.050
0.019
0.008
1
feiipoocesing_straeeeyy [T.ne _ard]]
0.098
0.019
40.000
Table 17: Full output of the linear model assessing the impact of data-centric processing the classification task. The model has the following formula: auroc dataset generative model + preprocessing strategy + postprocessing strategy.
sblor
1406
687
0131
301
113
9.30
Adonn Mooen
035
0286
1180
Adlon 13011
11279
0.58
1100
0101
prender (Drlen
19124
035
prandic sode
01190
033
636
0101
430
dalan 18 T1ol plansh TadeT
0156
035
20314
030
116
0333
Adaa 18 Linoa rali 204 (eagg))
638
033
4236
2383
4U66
430
Stdiit
0LS
ML Bequnnd:
007
0000
030
1:50
1338
080
008
Duchie Maske-
184114
Log Laaikedd
636
11313
115
12300
3.2tv113
DE Nadd::
656
11:44
138
036
1180
130
0176
Adan T2000
038
138
636
11114
030
130
030
Eilan 18 Thutt piarai sodellabe
4250
011
-290
080
430
fslmn Thi1t To
molelTabe
035
4130
030
406
430
0120
038
2350
080
340
066
0140
636
131
186
500
063
000
056
0.00
130
119
40

---

Table 18: Full output of the linear model assessing the impact of data-centric processing on the feature spearmans_ r ~ dataset_id id generative_model-
selection task. The model has the following formula: preprocessinggstrratee strategy + postprocessing_sttaaeeee
HA25
8101
30.13
833
0.24
dasa 1ET seet
8:00
310m
8.30
1188
ditm 18 Tisoe
-1140
6.09
-9471
0.30
-1270
-1M4
datan 18T NC17
4.300
-1.00
8.30
4111
0.00
8.30
2477
0.36
0.24
gredio tie Lnad)
4.313
6300
1882
0334
0118
130
ramppta bed)
030)
8.07
1506
633
191
63:31
6:30
5.00
633
Regamee
2:00
2314
A13
Ad Rrquind:
0300
1130
6.64
1334
180
Deeden
17730
16390
6330
Cxad No.
IF Made:
Akod all Accenn qie odee aad 12.kp:
6mn
0061
6:34
Aston LET gornate =edeTT.0ee
Em
-148
033
dismt 18 LETiual
Cocuelaaaann ausnaas padellater
6561
633
0460
099
Aaton LeT AeThuuelinnn ponettte mede
65:3
600
558
0330
013
0104
4:131
6.01
-38
630
0332
085
633
6.04
198
638
186
Abvd atl sureet
6mm
5m
150
630
013
LET STTNL19 poenahe sedelElvel
6.08
633
Table 19: Full output of the linear model assessing the impact of data-centric processing on the model selection task. The model has the following formula: spearmans_ r dataset id gennerative pooereemooee---eee preprocessing_sttreee strategy + Seaaprocessing_strateee
300
-1000
300
11.14
3400
-115
4.40
3000
-1.84
1400
4.50)
4333
Caase Al 136122
08238
3104
0.63
1136
4.004
3063
-180
3186
6.55
300
dbe
4.30
200
2.33
1400
4.43
4111
al 1-M1a
E-lped
3187
3114
cranc
1802
1000
1201
Cranc gaarahh Expes
110
1.0
309
1301
310
36det
0L.
51.11
2110
1001
1001
1181
15300
1101
1000
1390
115
Kuttel:
13:012
Dasikn Piltow
25.11
DE Nadi:
caan munrato ze coadiiaa
1010
2101
6301
2000
13mm
1101
1311
1196
E31
4151
Iatan al momrato ce slur
1101
2101
4001
197h
1161
1160
1000
860
ratan ol Trar27 Sorrabu ce salee
1181
130
1100
110
feaee of siarshe re wiee
106
1.41
116
4905
415
1181
171
1100
0.83
frant af TASIP searrhh padt Froe
2561
124
1011
0389
0180
ream T14777 wwwreh ow hoar
198
1981
1124
100
(111
1790
41

---

Table 20: Full output of the linear model assessing the impact of data-centric processing on the classification task on the label noise data. The model has the following formula: auroc ~ pere perc_labe label noise + generative_ model + preprocessing strategy postprocessing Strattggy.ttt.......
seaf
2--5%
JKAS5
(.913]
61214
100
38:17
1.00
1717
1171
Yortale:
Ragarreh
21
Roqured:
pir liee (idetE4
43222
-483
133
4005
021044
Look Balleed
13N00
purk Lad, Jeda
-6343
-U37
1.30
-007
Prbillentaa:
0.00
Jangedder Ulo
A383
Dratekt
DI Bodbude
-305
sneket reak fepe Eniges
-0311
1300
1880
(100
JUSS
Coaal No.
1.4
Cerertine Trpet
craburk saal sypd [
0.1155
Jard)
6783
1.30
2014
(UC)
Table 21: Full output of the linear model assessing the impact of data-centric processing on the feature selection task on the label noise data. The model has the following formula: spearmans ~ pere label noise + generative model + preprocessing strategy + posiprocessing sttrttgyy__.....
4.59
01831
Dep Tanialle
0.121
0122
<186
1.90
-2843
2.83
Mathed:
Laa Squars
20.13
0183
3.08
1.94
2108
gratddt
Tami
(9.000
-35.3
crabaak abl leee Lokiea)
pere bie medtEE
403310
J0077
0128
0103
-3823
.3505
1.30
1.00
A133
3158
JUB3
Kirade
0407
3.399
Coed. No.
5.0
5145
Type
srathak sadd Lovel
vnidegt Jued]
00991
0011
3120
polpremmiin
Table 22: Full output of the linear model assessing the impact of data-centric processing on the model selection task on the label noise data. The model has the following formula: spearmans annerattve_model
perc_ label noise + generative model + preprocessing starressssnnsttttttyyggg + nostprocessing_strateeee
owf
P- H1
14.15 6.9%
Isterigt
5156
B132
15 233
8.30
0:33
$:10
Madhed:
Dusc
09:19:33
Prubo (
40,755
56.37
parc labi padec (-4)
pir loke rene 1141
.1361
<11254
6131
-4385
.114
8.30
8:30
4.10
4mh
4.24
4236
B457
Janga-dur
Watues
L515
544
1453
oudee audal byed Totpea
-U1080
6029
-3485
0571
436
B524
Nartal
3.5g
Lowel Ne.
8.50
extte opee Latkel
Cersrianco Typ::
awde
0013
B139
0164
850
8:00
paogenooiigg Tta Sard]
0,375
B209
5.80
8.30
8.61
8.34
Table 23: Full output of the linear model assessing the impact of data-centric processing on the classification task on the label noise data with interaction effects. The model has the following formula: auroc senerative_model. pere_ larrc_labl__noiseeeeee (preprocessing strategy + postprocessing_straeeeee
eef
10825
8:361
0.35
17.18
0220
1.30
4370
1104
4120
part andal added 12)
45129
0.60
3.54
2311
tere and T4)
Dog, Tataalic
Beuuurdd
3980
0350
07/3
1121
19511
The 30 Kug 1013
131o
pre
5101
033
186)
10.8
1810
OceBac
5,00
Danikio Madenn 1185
RXC
140
por, and adel LAlon pot Thunkas 1a Juli
40017
033
0.72
080
4881
IU85
Krrvdd
2.96
Cord No.
343
por and adel Tobn, pot Tan hand- bunilal
033
0468
0633
0014
I004
por Mut theee pott Tran bunt buddae)
E0814
039
048
019
0111
fon, Lotn, pott Taay en Sard
630
138
0-32
11:1
896
bon. pn Tater 2ndd kiril
030
055
42

---

Table 24: Full output of the linear model assessing the impact of data-centric processing on the feature selection task on the label noise data with interaction effects. The model has the follow- ing formula: spearmans r generative_ Locileee__eelleeeeeee + perc label noise (preprocessing_staaiiee strateey
postprocessing_ strategy)
green foe warl Trigerd
0.133
1351
184
0.34
0:48
1334
0331
630
8:15
per wedd
48811
534
0.01
61:14
Requared
pn, par Daedka- Jurd)
Sear
14:23
0001
1,88
0145
0313
No. Oborrttin
ABC
1543
pro wodf 2oae sederra es
11
1140
680
8:34
31-2
Preie
0111
and sad Belie sun ber)
1956
0993
0.30
31W
1391
2303
679
8:33
prr pulftoat booke]
130
133
191
636
119
por Sooe Lan
1a11 wnd
land kert
131
-181
016
033
20M
Table 25: Full output of the linear model assessing the impact of data-centric processing the model selection task on the label noise data with interaction effects. The model has the follow- ing formula: spearmans_r generative_mode perc_label noise ppreprocessing__tratee mostprocessing__trateee
650
21040
0.911
m1:01
3mm
1913
216
01244
0.81
11103
par. (abel13)
41581
276
0.455
0314
0131
-800
Bequrd:
Lead Scarcs
1,422
03907
2183
15/9
11633
No: Ober aiee
MlC
-0.40
per. gn kunkka
41430
2180
1364
0186
Peatedtty
per. pn Thwwler Swef)
Orgdlas Typ
21311
2101
1955
196
0310
530
per. (144 are bach baakaet
13:00
2181
6011
135
133
0314
pan. paaelal part Teaer (handaa))
4.5
438
035
0333
E.0
par. am Jwet
12.0
(181
130
1384
43


Reviews:
[{"review_id": "Fbor2NvM4n", "reviewer": "Reviewer_yA6T", "summary": "", "questions": "", "limitations": "The authors provide limitations in the appendix. Discussion is minimal and only focuses on the idea of expanding future work to include privacy preservation.", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "* Very strong motivation for an important issue that is of broad importance. Evaluating the quality of generated data is crucial and an active research direction that is appropriate for the datasets & benchmarks track.\n* The inclusion of relative model ranking and feature importance for each classification model provides an insightful additional point of comparison beyond just accuracy, and allows for deeper understanding of the strengths and weaknesses of the generated data. \n* The experimental manipulations around both label noise and different pre/post-processing steps are important for understanding data-level interventions that affect model performance.", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ""}, {"review_id": "jHeNu9zeTv", "reviewer": "Reviewer_P9Bf", "summary": "", "questions": "", "limitations": "I think the limitations should be in the Main Paper, not the Appendix.\n", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "These results will be useful for tasks requiring data augmentation, e.g., applications in which large annotated datasets are difficult to obtain, e.g., cybersecurity challenges. \n\n", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ""}, {"review_id": "9HyfVlsfud", "reviewer": "Reviewer_g5vc", "summary": "", "questions": "", "limitations": "yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "1. The motivation of the paper is good, as it addresses a crucial and practical challenge in the field of synthetic data generation by proposing the integration of data-centric AI techniques.\n\n2. The code/data are available and will potentially benefit the community of synthetic data generation.\n\n", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ""}]

Author rebuttal:

Meta-review: The paper addresses the important problem of how data-centric AI methods can be used to carefully build synthetic data and evaluates the performance of  five state-of-the-art models for tabular data generation on eleven distinct tabular datasets.  Practical recommendations for integrating data-centric insights are offered for the synthetic data generation process, with a specific focus on classification performance, model selection, and feature selection. The authors find that though synthetic data is inferior to real data, careful design choices of the generative model and pre and post processing steps can impact the utility of the data for the learning task.

Pros: Addresses an important issue, Good illustration on how relative model ranking and feature importance for each classification model provides insights beyond accuracy and sheds light on generated data strengths/weaknesses, data level interventions such as label noise and pre/post processing steps allow for more detailed performance characterization.
Cons: Most results do not appear to be robust. The paper was oversold in the first version as a generic approach that can be applied for all data modalities. This was then changed to only tabular data. Thus the scope of applicability of the work is limited.  Besides this, the type of insights offered in this paper is well known to statisticians who practice careful stratification of data samples during model estimation.

After discussion between the area chair, reviewers, the rating of the reviewer g5vc was increased to 5 (even though it is not reflected in the final review yet). Thus the overall average rating is: 19/3 = 6.33.    The paper is a borderline accept and may be considered for a poster.
Decision: Accept (Poster)