SlowGuess/NeurIPS_2023_OCR huggingface dataset:
All code for running the analysis and creating tables and graphs can be found at the following links: https://github.com/HLasse/data-centric-synthetic-data or https://github. com/vanderschaarlab/data-centric-synthetic-data.
Table 1: Summarised performance for the baseline condition (no data-centric processing) across all datasets. Classification is measured by AUROC, model selection and feature selection by Spearman's Rank Correlation, and statistical fidelity by inverse KL divergence. Numbers show bootstrapped mean and  $95\%$  CI. The best-performing model by task is in bold.
Generative Model	Classification	Model Selection	Feature Selection	Statistical fidelity
Real data	0.866 (0.855, 0.877)	1.0	1.0	1.0
bayesian_network	0.622 (0.588, 0.656)	0.155 (0.055, 0.264)	0.091 (-0.001, 0.188)	0.998 (0.998, 0.999)
ctgan	0.797 (0.769, 0.823)	0.519 (0.457, 0.579)	0.63 (0.557, 0.691)	0.979 (0.967, 0.987)
ddpm	0.813 (0.781, 0.844)	0.508 (0.446, 0.573)	0.635 (0.546, 0.718)	0.846 (0.668, 0.972)
nflow	0.737 (0.713, 0.761)	0.354 (0.288, 0.427)	0.415 (0.34, 0.485)	0.975 (0.968, 0.981)
tvae	0.792 (0.764, 0.818)	0.506 (0.436, 0.565)	0.675 (0.63, 0.722)	0.966 (0.953, 0.978)
We assess our framework on a filtered version of the Tabular Classification from Numerical features benchmark suite from [40]. To reduce computational costs, we filter the benchmark suite to only include datasets with less than 100.000 samples and less than 50 features which reduced the number of datasets from 16 to 11. The datasets span several domains and contain a highly varied number of samples and features (see B for more details.). Notably, the datasets have been preprocessed to meet a series of criteria to ensure their suitability for benchmarking tasks. For instance, the datasets have at least 5 features and 3000 samples, are not too easy to classify, have missing values removed, have balanced classes, and only contain low cardinality features.
4.2 Generative models
To cover a representative sample of the space of generative models, we evaluate 5 different models with different architectures as reviewed in 2: bayesian networks (bayesian_network), conditional tabular generative adversarial network (ctgan), tabular variational autoencoder (tvae), normalizing flow (nflow), diffusion model for tabular data (ddpm).
4.3 Supervised classification model training
The variety of models employed in our study includes: extreme gradient boosting (xgboost), random forest, logistic regression, decision tree, k-nearest neighbors, support vector classifier, gaussian naive bayes, and multi-layer perception. It is the ranking of these models that is evaluated for the model selection task. Given the large number of models, we restrict the classification results in the main paper to be from the xgboost model. Feature selection results are reported for xgboost models. Classification and feature selection results for the other classifiers can be found in Appendix C.
4.4 Experimental procedure
Main study The experimental process followed the structure outlined in Sec. 3 and Figure 2, repeated for each of the 11 datasets, 5 generative models, 10 random seeds, and all permutations of pre- and postprocessing methods for each of the three data-centric methods (Cleanlab, Data-IQ, and Data Maps). We comprehensively evaluate the results across classification performance, model selection, feature selection, and statistical fidelity. In total, we fit more than 8000 generative models.
Impact of label noise To assess the impact of label noise on the effect of data-centric pre- and postprocessing, we carried out an analogous experiment to the main study, on the Covid mortality dataset [41]. Here, we introduce label noise to  $\mathbb{D}_{\mathrm{train}}$  before applying any processing. We study the impact of adding  $[0, 2, 4, 6, 8, 10]$  percent label noise.
All results reported in the main paper use Cleanlab as the data-centric method for both pre- and postprocessing. This decision was made to ensure clarity in the reported results and because Cleanlab was found to outperform Data-IQ and Data Maps in a simulated benchmark. For the benchmark of the data-centric methods as well as results using Data-IQ and Data Maps, we refer to Appendix C.
Table 2: Percentage increase in performance from baseline, i.e., no data-centric pre- or postprocessing (as seen in Table 1), per generative model for each pre- and postprocessing strategy, averaged across all datasets and seeds.
Generative Model	Preprocessing Strategy	Postprocessing Strategy	Classification	Model Selection	Feature Selection	Statistical Fidelity
bayesian_network	baseline	no-hard	0.31 (-4.89, 5.98) ↑	-27.73 (-86.26, 32.7) ↓	-52.26 (-166.02, 47.61) ↓	-0.007 (-0.054, 0.046) ↓
easy-hard	baseline	0.35 (-4.98, 6.03) ↑	92.18 (31.99, 151.42) ↑	9.79 (-110.47, 129.68) ↑	-0.013 (-0.054, 0.027) ↓
no-hard	0.8 (-4.4, 6.34) ↑	27.25 (-31.99, 90.28) ↑	-9.39 (-132.22, 107.39) ↓	-0.023 (-0.067, 0.021) ↓
ctgan	baseline	no-hard	1.23 (-2.03, 4.44) ↑	11.47 (-1.92, 24.28) ↑	3.66 (-6.48, 12.61) ↑	-0.054 (-1.423, 0.805) ↓
easy-hard	baseline	-0.78 (-4.14, 2.88) ↓	-1.12 (-13.34, 11.47) ↓	0.15 (-10.66, 10.13) ↑	-0.006 (-1.061, 0.804) ↓
no-hard	0.37 (-2.96, 3.8) ↑	14.73 (2.13, 26.68) ↑	0.18 (-10.04, 9.43) ↑	-0.119 (-1.218, 0.701) ↓
ddpm	baseline	no-hard	0.63 (-3.55, 4.24) ↑	-6.35 (-20.41, 7.86) ↓	1.7 (-10.67, 13.37) ↑	-0.166 (-19.424, 15.964) ↓
easy-hard	baseline	0.65 (-2.84, 4.06) ↑	7.0 (-5.87, 20.19) ↑	1.58 (-10.92, 13.92) ↑	0.356 (-16.583, 13.893) ↑
no-hard	1.32 (-2.06, 4.68) ↑	5.98 (-7.27, 18.63) ↑	4.19 (-6.95, 16.16) ↑	0.284 (-16.721, 14.567) ↑
nflow	baseline	no-hard	1.05 (-2.45, 3.97) ↑	1.22 (-16.88, 18.41) ↑	5.82 (-11.28, 21.68) ↑	0.035 (-0.743, 0.688) ↑
easy-hard	baseline	0.7 (-2.64, 3.81) ↑	2.37 (-15.28, 20.63) ↑	9.63 (-8.01, 25.65) ↑	0.022 (-0.752, 0.625) ↑
no-hard	1.64 (-1.76, 4.81) ↑	4.66 (-13.67, 22.84) ↑	7.28 (-11.54, 25.04) ↑	0.052 (-0.705, 0.654) ↑
tvae	baseline	no-hard	1.1 (-2.29, 4.38) ↑	-3.58 (-18.44, 11.01) ↓	-0.13 (-6.84, 6.38) ↓	-0.053 (-1.418, 1.113) ↓
easy-hard	baseline	-0.25 (-3.46, 3.16) ↓	-5.83 (-19.45, 6.73) ↓	1.67 (-6.03, 8.17) ↑	0.24 (-0.941, 1.296) ↑
no-hard	0.71 (-2.59, 3.95) ↑	0.11 (-13.79, 14.38) ↑	3.41 (-3.38, 9.89) ↑	0.199 (-0.929, 1.285) ↑
4.4.1 Evaluation metrics
The classification performance is evaluated in terms of the area under the receiver operating characteristic curve (AUROC), model selection performance as Spearman's Rank Correlation between the ranking of the supervised classification models trained on the original data and the supervised classification models trained on the synthetic data, and feature selection performance as Spearman's Rank Correlation between the ranking of features in an xgboost model trained on the original data and an xgboost model trained on the synthetic data.
Statistical fidelity is insufficient for evaluating generative models. Measures of statistical fidelity fail to capture variability in performance on downstream tasks, as shown in Table 1. Surprisingly, the worst performing model across all tasks (bayesian network), has the highest inverse KL-divergence of all synthetic datasets, which should indicate a strong resemblance to the original data. Conversely, the lowest inverse KL-divergence is found for ddpm which is one of the consistently best performing models.
Practical guidance: The benchmarking results illustrate that when selecting a generative model, even if the statistical fidelity appears similar, different generative models may perform differently on the 3 downstream tasks (classification, model selection, feature selection). Hence, beyond statistical fidelity, practitioners should understand which aspect is most crucial for their purpose to guide selection of the generative model.
Different generative models for different tasks. As shown in Table 1, training on synthetic data leads to a marked decline in classification performance compared to real data, as well as highly differing model and feature rankings. The effect differs largely by generative model, where CTGAN, TabDDPM, and TVAE most closely retain the characteristics of the real data. No one model is superior across all tasks. Specifically, TabDPPM achieves the highest classification performance, CTGAN performs best in model selection, and TVAE excels in feature selection. These findings indicate that one should test a range of generative models and consider the trade-offs in data utility before publishing synthetic data. Additionally, Appendix C reveals that although there are slight differences in performance based on the supervised model type, the overall pattern of results remains consistent across generative models.
Practical guidance: No generative model reigns supreme (highlighting the inherent challenge of synthetic tabular data). However, over tabular data sets, we show that CTGAN and TVAE offer the best trade-off between high statistical fidelity and strong performance on the three downstream tasks.
Data-centric methods can improve the utility of synthetic data. The addition of data-centric pre- and postprocessing strategies has a generally positive effect across all tasks as seen in Table 2 and Figure 3, despite resulting in lower statistical fidelity. In terms of classification performance, 13 out of 15 evaluations showed a net improvement, with gains up to  $1.64\%$  better classification
performance compared to not processing the data. Model selection exhibited more pronounced effects, particularly for bayesian networks, which demonstrated the greatest variability overall. While the model selection results for TVAE decreased following data-centric processing, the other generative models saw positive effects, with performance improvements ranging from  $4.66\%$  to  $92\%$ . Regarding feature selection, 12 out of 15 evaluations demonstrated a net benefit of data-centric processing, resulting in improvements of  $3.41\%$  to  $9.79\%$  in Spearman's rank correlation. The benefit of data-centric processing was found to be statistically significant for classification and feature selection (see Appendix D for details).
Practical guidance: Before releasing a synthetic dataset, practitioners are advised to apply the data-centric methods studied in this paper as an add-on. This will ensure enhanced utility of the synthetic data in terms of classification performance, model selection, and feature selection.
Data-centric processing provides benefits across levels of label noise. Data-centric pre- and postprocessing lead to consistently higher performance across tasks for all augmented datasets. As shown in Figure 4, the magnitude of the effect of data-centric processing decreases with higher levels of label noise, particularly above  $8\%$ , although this effect is not statistically significant. Even though the level of statistical fidelity decreased marginally by applying data-centric processing, data-centric processing led to statistically significant increases in performance on all three tasks.
Practical guidance: Fitting generative models on noisy "real-world" data can lead to sub-optimal downstream performance despite seemingly high statistical fidelity. Data-centric methods are especially useful at reasonable levels of label noise, typically below  $8\%$ . Therefore, we recommend their application when fitting generative models on real-world datasets.
5.1 Limitations and future work
Our work delves into the performance-driven aspects of synthetic data generation, focusing primarily on data utility and statistical fidelity, particularly within the realm of tabular data. While tabular data is highly diverse and contains many intricacies, we also recognize several directions for further exploration. Our current framework, while rooted in tabular data, hints at the broader applicability to other data types such as text and images. Accommodating our framework to these modalities would require further work on modality-specific tasks. For instance, images or text do not possess a direct analog to feature selection. Such disparities underscore the need for a bespoke benchmarking methodology tailored to each specific data type.
Figure 3: Average performance across all datasets for each generative model by pre- and postprocessing method.
Figure 4: Performance of a single generative model (TabDDPM) on the Covid mortality dataset with varying levels of label noise across the pre- and postprocessing conditions.
This research provides novel insights into integrating data-centric AI techniques into synthetic tabular data generation. First, we introduce a framework to evaluate the integration of data profiles for creating more representative synthetic data. Second, we confirm that statistical fidelity alone is insufficient for assessing synthetic data's utility, as it may overlook important nuances impacting downstream tasks. Third, the choice of generative model significantly influences synthetic data quality and utility. Last, incorporating data-centric methods consistently improves the utility of synthetic data across varying levels of label noise. Our study demonstrates the potential of data-centric AI techniques to enhance synthetic data's representation of real-world complexities, opening avenues for further exploration at their intersection.
This work was partially supported by DeiC National HPC (g.a. 2022-H2-10). NS is supported by the Cystic Fibrosis Trust. LH was supported by a travel grant from A.P. Møller Fonden til Lægevidenskabens Fremme and is supported by grants from the Lundbeck Foundation (grant number: R344-2020-1073), the Central Denmark Region Fund for Strengthening of Health Science (grant number: 1-36-72-4-20), and The Danish Agency for Digitisation Investment Fund for New Technologies (grant number: 2020-6720).
[35] Weixin Liang, Girmaw Abebe Tadesse, Daniel Ho, L Fei-Fei, Matei Zaharia, Ce Zhang, and James Zou. Advances, challenges and opportunities in creating data for trustworthy ai. Nature Machine Intelligence, 4(8):669-677, 2022.
[36] Nabeel Seedat, Fergus Imrie, and Mihaela van der Schaar. Dc-check: A data-centric ai checklist to guide the development of reliable machine learning systems. arXiv preprint arXiv:2211.05764, 2022.
[37] Nabeel Seedat, Jonathan Crabbé, and Mihaela van der Schaar. Data-suite: Data-centric identification of in-distribution incongruous examples. In International Conference on Machine Learning, pages 19467-19496. PMLR, 2022.
[38] Nabeel Seedat, Jonathan Crabbé, Zhaozhi Qian, and Mihaela van der Schaar. Triage: Characterizing and auditing training data for improved regression. In Advances in Neural Information Processing Systems, 2023.
[39] Arthur Gretton, Karsten M Borgwardt, Malte J Rasch, Bernhard Schölkopf, and Alexander Smola. A kernel two-sample test. The Journal of Machine Learning Research, 13(1):723-773, 2012.
[40] Léo Grinsztajn, Edouard Oyallon, and Gael Varoquaux. Why do tree-based models still outperform deep learning on tabular data?
[41] Pedro Baqui, Ioana Bica, Valerio Marra, Ari Ercole, and Mihaela van Der Schaar. Ethnic and regional variations in hospital mortality from covid-19 in brazil: a cross-sectional observational study. The Lancet Global Health, 8(8):e1018-e1026, 2020.
[42] Takuya Akiba, Shotaro Sano, Toshihiko Yanase, Takeru Ohta, and Masanori Koyama. Optuna: A next-generation hyperparameter optimization framework.
[43] Kevin Bello, Bryon Aragam, and Pradeep Ravikumar. DAGMA: Learning DAGs via m-matrices and a log-determinant acyclicity characterization.
This appendix provides additional details on the data-centric methods used in our study, as well as outlining limitations and future work.
A.1 Data-centric AI methods
Our paper considers different data-centric AI methods to perform data characterization, which forms the basis of our data profiles. As discussed in the main manuscript, the primary difference between these methods is their scoring mechanism  $S$ . The three approaches considered in this paper next, i.e. Cleanlab, Data-IQ and Data Maps will be described next. Importantly, these methods are applicable to tabular data models like XGBoost, unlike several other data-centric methods which are only applicable to differentiable models. Recall that the data characterization methods apply a threshold  $\tau$  to  $S$ . The implementation of  $\tau$  in this work is described in Appendix C.3.
Cleanlab [34]. Cleanlab estimates the joint distribution of noisy and true labels, thereby characterizing data into profiles easy and hard, where hard data points might indicate mislabeling. It operates on the output of classification models and can be applied to any modality. We focus on Cleanlab in the main manuscript, however, our framework could be applied with any of the following data-centric tools.
Data-IQ [18]. Data-IQ is a training dynamics-based method, which characterizes data based on the aleatoric uncertainty (inherent data uncertainty), i.e.  $\mathbb{E}[\mathcal{P}(x,\vartheta)(1 - \mathcal{P}(x,\vartheta))]$ . We are able to extract three data profiles: easy, ambiguous and hard. Typically, these "ambiguous" and "hard" examples are harmful to model performance and might be mislabeled or "dirty".
Data Maps [33]. Data Maps is a training dynamics-based method, which characterizes data based on the variability/epistemic uncertainty (model uncertainty), i.e.  $\mathbb{V}[\mathcal{P}(x,\vartheta)]$ . We are able to extract three data profiles: easy, ambiguous and hard. Typically, these "ambiguous" and "hard" examples are harmful to model performance and might be mislabeled or "dirty".
By accident, the results reported in the main paper were based on data from 8 random seeds instead of the intended 10, as stated in the paper. To maintain consistency in the results, the results reported in the Appendix are also presented with 8 seeds.
B.1 Computational resources
All experiments were conducted using NVIDIA T4 16GB GPUs. The total number of GPU hours spent across all experiments is approximately 1000. All experiments were performed on the UCloud platform.
B.2 Hyperparameter tuning of generative models
In order to reduce the computational costs of running the main experiment, we conducted a hyperparameter search on the Adult dataset instead of tuning the parameters for each dataset in our benchmark. For each model (bayesian_network, ctgan, ddpm, nflow, tvae), we conducted a search over the hyperparameter space defined for each model in the synthcity [5] implementation. We used Optuna [42] to conduct 20 trials for each model. The best hyperparameters for each model are listed in Table 3.
B.3 Classification model set
Throughout the experiments, we fit a set of models from the scikit-learn library, namely, XGBClassifier, RandomForestClassifier, LogisticRegression,
Table 3: Hyperparameters used for the generative models.
Model	Parameters
bayesian_network	struct_learning_search_method: hillclimb, struct_learning_score: bic
ctgan	generator_n_layers Hidden: 2, generator_n.units Hidden: 50, generator_nonlin: tanh, n_iter: 1000, generator_dropout: 0.0575, discriminator_n_layers Hidden: 4, discriminator_n.units Hidden: 150, discriminator_nonlin: relu
ddpm	lr: 0.0009375080542687667, batch_size: 2929, num_timesteps: 998, n_iter: 1051, is_classification: True
nflow	n_iter: 1000, n_layers Hidden: 10, n.units Hidden: 98, dropout: 0.11496088236749386, batch_norm: True, lr: 0.0001, linear_transform_type: permutation, base_transform_type: rq-autoregressive, batch_size: 512,
tvae	n_iter: 300, lr: 0.0002, decoder_n_layers Hidden: 4, weight Decay: 0.001, batch_size: 256, n.units_embedding: 200, decoder_n.units Hidden: 300, decoder_nonlin: elu, decoder_dropout: 0.194325119117226, encoder_n_layers Hidden: 1, encoder_n.units Hidden: 450, encoder_nonlin: leaky_relu, encoder_dropout: 0.04288563703094718,
DecisionTreeClassifier, KNeighborsClassifier, SVC, GaussianNB, and MLPClassifier. All hyperparameters were kept at their default value.
B.4 Datasets
Main experiments As described in Sec 4.1, we used the "Tabular benchmark numerical classification" downloaded from OpenML (suite id 337), filtered to only include datasets with less than 100.000 samples and less than 50 features for our main experiments. The number of samples, features, and links to the datasets are provided in Table 4.
Noise dataset The dataset used to investigate the impact of label noise was the Covid mortality dataset from [41]. The data includes data on patients with Covid with a label for whether the patient will die within 14 days. The dataset contains 6882 samples and 21 features.
Adult dataset The adult dataset is a classic machine learning dataset, containing 48842 and 14 samples, with the task of predicting whether an individual's income exceeds $50k per year based on census data.
Table 4: Datasets used in the benchmark.
Task id	Dataset name	n_features	n_samples	URL
361055	credit	10	16714	https://openml.org/search?type=task&collections.id=337&sort=runs&id=361055
361060	electricity	7	38474	https://openml.org/search?type=task&collections.id=337&sort=runs&id=361060
361062	pol	26	10082	https://openml.org/search?type=task&collections.id=337&sort=runs&id=361062
361063	house_16H	16	13488	https://openml.org/search?type=task&collections.id=337&sort=runs&id=361063
361065	MagicTelescope	10	13376	https://openml.org/search?type=task&collections.id=337&sort=runs&id=361065
361066	bank-marketing	7	10578	https://openml.org/search?type=task&collections.id=337&sort=runs&id=361066
361070	eye movements	20	7608	https://openml.org/search?type=task&collections.id=337&sort=runs&id=361070
361273	Diabetes130US	7	71090	https://openml.org/search?type=task&collections.id=337&sort=runs&id=361273
361275	default-of-credit-card-clients	20	13272	https://openml.org/search?type=task&collections.id=337&sort=runs&id=361275
361277	california	8	20634	https://openml.org/search?type=task&collections.id=337&sort=runs&id=361277
361278	heloc	22	10000	https://openml.org/search?type=task&collections.id=337&sort=runs&id=361278
Figure 5: Average classification performance across all datasets and generative models by pre- and postprocessing strategy and classification model type, using Cleanlab as the data-centric method.
C.1 Performance on other models
Table 5 shows classification performance across all classification models, averaged over all datasets. Random Forest and XGBoost performs the best across all combinations of pre- and postprocessing, hence, we chose to focus our results in the main manuscript on the XGBoost model. As evident from Table 5 and Figure 5, the benefit of data-centric pre- and postprocessing is consistent across model types.
Table 5: Average classification performance across all datasets and generative models by pre- and postprocessing strategy and classification model type, using Cleanlab as the data-centric method. Numbers in parentheses indicate  $95\%$  bootstrapped CI.
Preprocessing Strategy	Postprocessing Strategy	Classification model	AUROC
baseline	baseline	DecisionTreeClassifier	0.64 (0.63, 0.64)
GaussianNB	0.7 (0.69, 0.71)
KNeighborsClassifier	0.65 (0.64, 0.66)
LogisticRegression	0.72 (0.7, 0.73)
MLPClassifier	0.7 (0.68, 0.71)
RandomForestClassifier	0.76 (0.75, 0.78)
SVC	0.69 (0.67, 0.7)
XGBClassifier	0.76 (0.75, 0.77)
no-hard	DecisionTreeClassifier	0.66 (0.65, 0.67)
GaussianNB	0.71 (0.69, 0.72)
KNeighborsClassifier	0.66 (0.65, 0.67)
LogisticRegression	0.72 (0.7, 0.73)
MLPClassifier	0.72 (0.7, 0.73)
RandomForestClassifier	0.77 (0.75, 0.78)
SVC	0.69 (0.68, 0.71)
XGBClassifier	0.77 (0.75, 0.78)
easy-hard	baseline	DecisionTreeClassifier	0.64 (0.63, 0.65)
GaussianNB	0.7 (0.69, 0.72)
KNeighborsClassifier	0.65 (0.64, 0.66)
LogisticRegression	0.72 (0.7, 0.73)
MLPClassifier	0.71 (0.69, 0.72)
RandomForestClassifier	0.77 (0.75, 0.78)
SVC	0.69 (0.68, 0.71)
XGBClassifier	0.76 (0.75, 0.77)
no-hard	DecisionTreeClassifier	0.67 (0.66, 0.68)
GaussianNB	0.71 (0.69, 0.72)
KNeighborsClassifier	0.66 (0.65, 0.67)
LogisticRegression	0.72 (0.71, 0.73)
MLPClassifier	0.72 (0.71, 0.73)
RandomForestClassifier	0.77 (0.76, 0.78)
SVC	0.7 (0.68, 0.71)
XGBClassifier	0.77 (0.75, 0.78)
Real data	DecisionTreeClassifier	0.74 (0.73, 0.75)
GaussianNB	0.73 (0.72, 0.74)
KNeighborsClassifier	0.73 (0.71, 0.74)
LogisticRegression	0.78 (0.77, 0.79)
MLPClassifier	0.77 (0.76, 0.79)
RandomForestClassifier	0.86 (0.85, 0.87)
SVC	0.74 (0.72, 0.75)
XGBClassifier	0.87 (0.86, 0.88)