SlowGuess/NeurIPS_2023_OCR huggingface dataset:
A composite dataset involves multiple sub datasets. Synthcity can handle the benchmarking of different classes of composite datasets.
Currently, it supports (1) static datasets with the
same features, collected from different domains, (2) a static and a time series dataset. The latter setting is common in applications. For example, a patient's medical record may contain both static demographic information and longitudinal follow up data.
Figure 2: Supported tabular data modalities.
3 Comparison with existing libraries
In this section, we compare synthcity with other popular open source libraries for synthetic data generation to demonstrate its suitability as a comprehensive benchmark framework. Here we only consider the libraries that can generate synthetic data while preserving the statistical properties of real data, which includes YData Synthetic, Gretel Synthetics, SDV, DataSynthesizer, SmartNoise and nbsynthetic. Libraries that generate "fake" data for software testing are not considered because they do not attempt to learn the distribution of real data.
220   
221   
222   
223   
224   
225   
226   
227   
228   
229   
230   
231   
232   
233   
234   
235   
236   
237   
238   
239   
240   
241   
242   
243   
244   
245   
246   
247   
248   
249   
250   
251   
252   
253   
254   
255   
256   
257   
258   
259   
260   
261   
262   
263   
264
Setting\Software	Synthcity	YData	Gretel	SDV	DataSynthesizer	SmartNoise	nbsynthetic
Data modalities
Static data	✓	✓	✓	✓	✓	✓	✓
Regular time series	✓	✓	✓	✓
Irregular time series	✓
Censored features	✓
Composite data	✓	✓
Use cases
Generation	✓	✓	✓	✓	✓	✓	✓
Fairness (balance)	✓	✓	✓	✓	✓
Fairness (causal)	✓
Privacy (DP)	✓	✓	✓
Privacy (TM)	✓
Cross domain aug.	✓
Table 3: The data modalities and use cases supported by synthcity and other open source synthetic data libraries. Comparisons are based on the software versions available at the time of writing.
Table 3 shows that synthcity supports much broader use cases and data modalities than the alternatives. The existing libraries often focus on a single data modality or use case because they are intended as a solution to a specific problem rather than a benchmark framework. Furthermore, Synthcity includes many more data generators, including all major flavors of deep generative models as well as traditional generative models. It also contains a built-in evaluation module that assesses various aspects of the generator. A more detailed comparison of the supported data generators and evaluation metrics are available in the Appendix. The broad coverage of data modalities, use cases, data generators and evaluation metrics make Synthcity unique in its capacity for model evaluation and comparison.
4 Illustrative case studies
In this section, we present two illustrative use cases to show the type of benchmark studies that Synthcity can facilitate. We stress that these examples do not cover the full capability of Synthcity and they are used as illustrations.
4.1 Static tabular data generative model benchmark
We study which generative model has the strongest performance in generating synthetic tabular data. Synthcity allows us to compare a variety of state-of-the-art algorithms in this study, including ARF, GOGGLE, TabDDPM, CTGAN and TVAE. These algorithms are representative of broader families of generative models such as GANs, VAEs, Diffusion models, and forest-based generative models.
Similar to prior tabular data benchmarks [Grinsztajn et al., 2022], we have selected 18 datasets from the OpenML benchmark, which cover common regression and classification datasets encountered in data science projects [Vanschoren et al., 2014]. The datasets cover a range of sample sizes (4,209 to 1,025,010) and feature counts (5 to 771).
Synthcity can automatically calculate more than 25 supported evaluation metrics in a benchmark. In this study, we focus on evaluating the fidelity of synthetic data. Similar to Liu et al. [2023], we report the average of the three-dimensional metrics ( $\alpha$ -precision,  $\beta$ -recall, and authenticity), as proposed in Alaa et al. [2022], as a measure of data quality—whether the synthetic data are realistic, cover the true data distribution, and are generalized. Furthermore, we report the detection score, which reflects how often the synthetic data can be distinguished from the real data. To reduce the variability from the classifiers, we report the average AUROC scores from three different post-hoc data classifiers, as in Liu et al. [2023].
Table 4 shows the experimental results averaged across all the datasets. We observe that the ARF model achieves strong performance in the quality score and stands out in terms of the detection score.
265   
266   
267   
268   
269   
270   
271   
272   
273   
274   
275   
276   
277   
278   
279   
280   
281   
282   
283   
284   
285   
286   
287   
288   
289   
290   
291   
292   
293   
294   
295
This suggests that the tree-based generative models are strong competitors to deep generative models for static tabular data. And this area is a promising avenue for further research.
Quality	Detection
ARF	0.5475	0.6721
GOGGLE	0.4054	0.9261
TabDDPM	0.5436	0.7074
CTGAN	0.5475	0.7758
TVAE	0.5487	0.7389
4.2 Tabular data fairness and augmentation benchmark
We consider a benchmark study on cross-domain data augmentation for improving predictive performance on minority groups. We use the SIVEP-Gripe public dataset as an illustrative example, which contains anonymized records of COVID-19 patients in Brazil [Baqui et al., 2021]. In this dataset, 'Mixed' and 'White' are the majority ethnicity groups while 'Black', 'East Asian' and 'Indigenous' are the minority groups (accounting for less than  $10\%$  of the total population). The dataset is used for training a downstream model to predict COVID-19 mortality. Due to the distributional imbalance, the downstream predictor is likely to under-perform on the minority groups, which may raise fairness issues (Section 2.2.3). This study aims to benchmark the utility of different generative models for data augmentation by measuring the AUROC of mortality prediction on the minority groups.
Synthcity allows us to easily compare RadialGAN, which was designed for cross-domain data augmentation, and the conditional generative models (TabDDPM, CTGAN, and TVAE). We use Synthcity's pre-configured pipeline for data augmentation benchmark, which reduces the amount of code and prevents data leakage. Synthcity also allows us to evaluate the performance gain for different downstream models, and we have selected a multi-layer perceptron classifier and a xgBoost classifier. The results are listed in Table 5. We observe that data augmentation consistently improves the accuracy of mortality prediction for minority groups. TabDDPM, a novel diffusion model, achieves the best overall performance, followed by RadialGAN.
Table 4: Benchmark results for static tabular data generation. Quality: the higher the better; Detection: the lower the better.
Neural net	XgBoost
TabDDPM	0.7241	0.7786
RadialGAN	0.7137	0.7627
CTGAN	0.6477	0.7507
TVAE	0.3623	0.7794
Baseline	0.3244	0.7327
Table 5: Benchmark results for cross-domain data augmentation. The metric reported is the AUROC of mortality prediction on the minority groups
Synthetic data is an emerging field where many novel algorithms have been proposed; yet there lacks an easy way to benchmark generative models across different desired or off label use cases, compare them with diverse baselines, and explain their performance gain. In this work, we present the open source Synthcity library as a solution to the benchmark challenge. Synthcity contains many built-in generative models, architectures and evaluation metrics, which are easily accessible through end-to-end evaluation pipelines. It can help researchers to perform in-depth and comprehensive benchmark studies with minimal programming effort.
296   
297   
298   
299   
300   
301   
302   
303   
304   
305   
306   
307   
308   
309   
310   
311   
312   
313   
314   
315   
316   
317   
318   
319   
320   
321   
322   
323
6 Funding Statement
This work is supported by Cancer Research UK. The computational experiments in this work was supported by Azure sponsorship credits granted by Microsoft's AI for Good Research Lab.
We thank all the open source contributors to the Synthcity library. The full list of contributors is available on GitHub. Special thanks goes to Bogdan Cebere, who made major contributions to Synthcity while he was working as a Research Engineer at the van der Schaar Lab.
Martín Abadi, Andy Chu, Ian Goodfellow, H Brendan Mcmahan, Ilya Mironov, Kunal Talwar, and Li Zhang. Deep Learning with Differential Privacy. Proceedings of the 2016 ACM SIGSAC Conference on Computer and Communications Security, 2016. doi: 10.1145/2976749. URL http://dx.doi.org/10.1145/2976749.2978318.
Takuya Akiba, Shotaro Sano, Toshihiko Yanase, Takeru Ohta, and Masanori Koyama. Optuna: A next-generation hyperparameter optimization framework. In Proceedings of the 25th ACM SIGKDD international conference on knowledge discovery & data mining, pages 2623-2631, 2019.
Haleh Akrami, Sergul Aydore, Richard M Leahy, and Anand A Joshi. Robust variational autoencoder for tabular data with beta divergence. arXiv preprint arXiv:2006.08204, 2020.
Ahmed Alaa, Alex James Chan, and Mihaela van der Schaar. Generative time-series modeling with fourier flows. In International Conference on Learning Representations, 2021.
Ahmed Alaa, Boris Van Breugel, Evgeny S Saveliev, and Mihaela van der Schaar. How faithful is your synthetic data? sample-level metrics for evaluating and auditing generative models. In International Conference on Machine Learning, pages 290-306. PMLR, 2022.
Antreas Antoniou, Amos Storkey, and Harrison Edwards. Data Augmentation Generative Adversarial Networks. 11 2017. URL http://arxiv.org/abs/1711.04340.
Pedro Baqui, Valerio Marra, Ahmed M Alaa, Ioana Bica, Ari Ercole, and Mihaela van der Schaar. Comparing Covid-19 risk factors in brazil using machine learning: the importance of socioeconomic, demographic and structural factors. Scientific reports, 11(1):1-10, 2021.
Brett K Beaulieu-Jones, Zhiwei Steven Wu, Chris Williams, Ran Lee, Sanjeev P Bhavnani, James Brian Byrd, and Casey S Greene. Privacy-preserving generative deep neural networks support clinical data sharing. Circulation: Cardiovascular Quality and Outcomes, 12(7):e005122, 2019.
Simon Bing, Andrea Dittadi, Stefan Bauer, and Patrick Schwab. Conditional Generation of Medical Time Series for Extrapolation to Underrepresented Populations. 2022.
Sam Bond-Taylor, Adam Leach, Yang Long, and Chris G. Willcocks. Deep Generative Modelling: A Comparative Review of VAEs, GANs, Normalizing Flows, Energy-Based and Autoregressive Models. IEEE Transactions on Pattern Analysis and Machine Intelligence, 3 2021. doi: 10.1109/ TPAMI.2021.3116668. URL http://arxiv.org/abs/2103.04922http://dx.doi.org/10. 1109/TPAMI.2021.3116668.
Vadim Borisov, Kathrin Seßler, Tobias Leemann, Martin Pawelczyk, and Gjergji Kasneci. Language models are realistic tabular data generators. arXiv preprint arXiv:2210.06280, 2022.
Hari Prasanna Das, Ryan Tran, Japjot Singh, Xiangyu Yue, Geoffrey Tison, Alberto Sangiovanni-Vincentelli, and Costas J Spanos. Conditional synthetic data generation for robust machine learning applications with limited pandemic data. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 36, pages 11792-11800, 2022.
324   
325   
326   
327   
328   
329   
330   
331   
332   
333   
334   
335   
336   
337   
338   
339   
340   
341   
342   
343   
344   
345   
346   
347   
348   
349   
350   
351   
352   
353   
354   
355   
356   
357   
358   
359   
360   
361   
362   
363   
364   
365   
366   
367   
368
369   
370   
371   
372   
373   
374   
375   
376   
377   
378   
379   
380   
381   
382   
383   
384   
385   
386   
387   
388   
389   
390   
391   
392   
393   
394   
395   
396   
397   
398   
399   
400   
401   
402   
403   
404   
405   
406   
407   
408   
409   
410   
411   
412   
413   
414   
415   
416   
417
418   
419   
420   
421   
422   
423   
424   
425   
426   
427   
428   
429   
430   
431   
432   
433   
434   
435   
436   
437   
438   
439   
440   
441   
442   
443   
444   
445   
446   
447   
448   
449   
450   
451   
452   
453   
454   
455   
456   
457   
458   
459   
460   
461   
462   
463   
464   
465
466   
467   
468   
469   
470   
471   
472   
473   
474   
475   
476   
477   
478   
479   
480   
481   
482   
483   
484   
485   
486   
487   
488   
489   
490   
491   
492   
493   
494   
495   
496   
497   
498   
499   
500   
501   
502   
503   
504   
505   
506   
507   
508   
509   
510   
511   
512   
513
Zhenchen Wang, Puja Myles, and Allan Tucker. Generating and evaluating synthetic uk primary care data: preserving data utility & patient privacy. In 2019 IEEE 32nd International Symposium on Computer-Based Medical Systems (CBMS), pages 126-131. IEEE, 2019.
Zhenchen Wang, Puja Myles, and Allan Tucker. Generating and evaluating cross-sectional synthetic electronic healthcare data: Preserving data utility and patient privacy. Computational Intelligence, 37(2):819-851, 2021.
David S Watson, Kristin Blesch, Jan Kapar, and Marvin N Wright. Adversarial random forests for density estimation and generative modeling. In International Conference on Artificial Intelligence and Statistics, pages 5357-5375. PMLR, 2023.
Liyang Xie, Kaixiang Lin, Shu Wang, Fei Wang, and Jiayu Zhou. Differentially private generative adversarial network. arXiv preprint arXiv:1802.06739, 2018.
Depeng Xu, Shuhan Yuan, Lu Zhang, and Xintao Wu. Fairgan: Fairness-aware generative adversarial networks. In 2018 IEEE International Conference on Big Data (Big Data), pages 570-575, 2018.
Depeng Xu, Yongkai Wu, Shuhan Yuan, Lu Zhang, and Xintao Wu. Achieving causal fairness through generative adversarial networks. In *IJCAI International Joint Conference on Artificial Intelligence*, volume 2019-August, pages 1452-1458. International Joint Conferences on Artificial Intelligence, 2019a. ISBN 9780999241141. doi: 10.24963/ijcai.2019/201.
Lei Xu, Maria Skoularidou, Alfredo Cuesta-Infante, and Kalyan Veeramachaneni. Modeling tabular data using conditional gan. Advances in Neural Information Processing Systems, 32, 2019b.
Jinsung Yoon, James Jordan, and Mihaela Schaar. Radialgan: Leveraging multiple datasets to improve target-specific predictive models using generative adversarial networks. In International Conference on Machine Learning, pages 5699-5707. PMLR, 2018.
Jinsung Yoon, Daniel Jarrett, and Mihaela Van der Schaar. Time-series generative adversarial networks. Advances in neural information processing systems, 32, 2019.
Jinsung Yoon, Lydia N Drumright, and Mihaela Van Der Schaar. Anonymization through data synthesis using generative adversarial networks (ads-gan). IEEE journal of biomedical and health informatics, 24(8):2378-2388, 2020.
Rich Zemel, Yu Wu, Kevin Swersky, Toni Pitassi, and Cynthia Dwork. Learning fair representations. In International conference on machine learning, pages 325-333, 2013.
Jun-Yan Zhu, Taesung Park, Phillip Isola, and Alexei A Efros. Unpaired image-to-image translation using cycle-consistent adversarial networks. In Proceedings of the IEEE international conference on computer vision, pages 2223-2232, 2017.
514   
515   
516   
517   
518   
519   
520   
521   
522   
523   
524   
525   
526   
527   
528   
529   
530   
531   
532   
533   
534   
535   
536   
537   
538   
539   
540   
541   
542   
543   
544   
545
A.1 Code availability
The code for the illustrative use cases are available on GitHub https://github.com/vanderschaarlab/synthcity-benchmarking. The synthcity library is available on pip and GitHub. The tutorials folder contains additional illustrative examples.
A.2 Supported algorithms and metrics
Aspect	Evaluation Metric \Software	Synthcity	YData	Gretel	SDV	DataSynthesizer	SmartNoise	nbsynthetic
Fedely	Jensen-Shannon distance	✓
Wasserstein distance	✓
Total variation distance	✓
KL divergence	✓
Skewness	✓
Max-mean discrepancy	✓	✓
KS test	✓	✓	✓
PRDC	✓
Alpha-precision	✓
Survival Kaplan-Meier dist.	✓
Detection: linear	✓	✓
Detection: NN	✓
Detection: XGB	✓
Detection: Linear	✓
Utility	Linear model	✓	✓
MLP	✓	✓
XGBoost	✓	✓
Static survival	✓
Time-series	✓
Survival time-series	✓
Privacy	Correct attribution prob.	✓	✓
K-anonymity	✓
K-map	✓
Delta-presence	✓
L-diversity	✓
DOMIAS	✓
Identifiability score	✓
Table 6: The evaluation metrics supported by synthcity and other open source synthetic data libraries. Comparisons are based on the software versions available at the time of writing.
Static	Censored	Time Series
Fully connected	Weibull AFT	LSTM
Residual network	Cox PH	GRU
TabNet	Random Survival Forest	RNN
Survival Xgboost	Transformer
Deephit	MLSTM.FCN
Tenn	TCN
Date	InceptionTime
InceptionTimePlus
XceptionTime
ResCNN
OmniScaleCNN
XCM
Table 7: Available network architectures and survival models in synthcity for different data modalities. These components are compatible with multiple algorithms.
546   
547   
548   
549   
550   
551
Algorithm\Software	Synthcity	YData	Gretel	SDV	DataSynthesizer	SmartNoise	nbsynthetic
AIM	✓
GREAT	✓
TabDDPM	✓
ARF	✓
GOGGLE	✓
CTGAN	✓	✓	✓	✓
ACTGAN	✓
TVAE	✓	✓
Bayesian Network	✓
Normalizing Flows	✓
Survial GAN	✓
Survival VAE	✓
DoppelGANger	✓
TimeGAN	✓	✓
FourierFlows	✓
Probabilistic AR	✓	✓
DECAF	✓
RadialGAN	✓
ADSGAN	✓
DPGAN	✓	✓	✓
PATEGAN	✓	✓
PrivBayes	✓	✓
Table 8: The data generating algorithms supported by synthcity and other open source synthetic data libraries. Comparisons are based on the software versions available at the time of writing.