Mixed-Type Tabular Data Synthesis with Score-based Diffusion in Latent Space
Authors: Hengrui Zhang
Jiani Zhang
Zhengyuan Shen
Balasubramaniam Srinivasan
Xiao Qin
Christos Faloutsos
Huzefa Rangwala
George Karypis
Venue: ICLR 2024 oral (2024)
Keywords: Tabular data
tabular generation
diffusion models
Primary area: generative models

Abstract: Recent advances in tabular data generation have greatly enhanced synthetic data quality. However, extending diffusion models to tabular data is challenging due to the intricately varied distributions and a blend of data types of tabular data. This paper introduces TabSyn, a methodology that synthesizes tabular data by leveraging a diffusion model within a variational autoencoder (VAE) crafted latent space. The key advantages of the proposed Tabsyn include (1) Generality: the ability to handle a broad spectrum of data types by converting them into a single unified space and explicitly capturing inter-column relations; (2) Quality: optimizing the distribution of latent embeddings to enhance the subsequent training of diffusion models, which helps generate high-quality synthetic data; (3) Speed: much fewer number of reverse steps and faster synthesis speed than existing diffusion-based methods. Extensive experiments on six datasets with five metrics demonstrate that Tabsyn outperforms existing methods. Specifically, it reduces the error rates by 86% and 67% for column-wise distribution and pair-wise column correlation estimations compared with the most competitive baselines. The code has been made available at https://github.com/amazon-science/tabsyn.

Published as a conference paper at ICLR 2024
MIXED-TYPE TABULAR DATA SYNTHESIS WITH SCORE-BASED DIFFUSION IN LATENT SPACE
Hengrui Zhang! Jiani Zhang*
Balasubramaniam Srinivasan Zhengyuan Shen?
Xiao Qin Christos Faloutso
Huzefa Rangwala
George Karypis
'Computer Science Department, University of Illinois at Chicago AAmazon Web Services 'Computer Science, George Mason University
hzhan55@uic. (zhajiang s srbalasu, donshen] @ @amazon. com (drxqin, falout sut rhuzefa, gka skarypis s ama zon, com
ABSTRACT
Recent advances in tabular data generation have greatly enhanced synthetic data quality. However, extending diffusion models tabular data is challenging due to the intricately varied distributions and blend of data types of tabular data. This paper introduces TABSYN, a methodology that synthesizes tabular data by leveraging diffusion model within a variational autoencoder (VAE) crafted latent space. The key advantages of the proposed TABSYN include (1) Generality: the ability to handle broad spectrum of data types by converting them into single unified space and explicitly capture inter-column relations, (2) Quality: optimizing the distribution of latent embeddings to enhance the subsequent training of diffusion models, which helps generate high-quality synthetic data, (3) Speed: much fewer number of reverse steps and faster synthesis speed than existing diffusion-based methods. Extensive experiments on six datasets with five metrics demonstrate that TABSYN outperforms existing methods. Specifically, it reduces the error rates by 86% and 67% for column-wise distribution and pair-wise column correlation estimations compared with the most competitive baselines. Code has been made available at https: / giithub. com/ amazon www-scciiieeeeeee wwweeeneeaaaaaee tabsyn.
INTRODUCTION
The propossed TabSyn
Single Density
TabSyn
TaDDDPM
Tabular data synthesis has wide range of appli- cations, such as augmenting training data (Fon- seca Bacao, 2023), protecting private data in- stances (Assefa et al., 2021; Hermandez et al., 2022), and imputing missing values (Zheng & Charoenphakdee, 2022). Recent developments in tabular data generation have notably enhanced the quality of synthetic data (Xu et al., 2019; Borisov et al., 2023; Liu et al., 2023b), while the syn- thetic data is still far from the real one. To fur- ther improve the generation quality, researchers have explored adapting diffusion models, which have shown strong performance in image synthe- sis tasks (Ho et al., 2020; Rombach et al., 2022), for tabular data generation (Kim et al., 2022; Kotelnikov et al., 2023; Kim et al., 2023; Lee et al., progress made by these methods, tailoring diffusion model for tabular data leads to several challenges. Unlike image data, which comprises pure continuous pixel values with local spatial correlations, tabular data features have complex and varied distributions (Xu et al.,
CoDi
a-Precision
STaSy
GReaT
Pair Correlation
26
p-Recall
MLE
Figure 1: Our TABSYN consistently outperforms SOTA tabular data generation methods across five data quality metrics.
2023). Despite
'Wok conducted during an internship at Amazon Web Services.
Corresponding author.
Huzefa Rangwala is on LOA as Professor of Computer Science at George Mason University. This paper describes work performed Amazon.

---

Published as a conference paper at ICLR 2024
2019), making it hard to learn joint probabilities across multiple columns. Moreover, typical tabular data often contains mixed data types, i.c., continuous (c.g., numerical features) and discrete (e.g.. categorical features) variables. The standard diffusion process assumes continuous imput space with Gaussian noise perturbation, which leads to additional challenges with categorical features. Existing solutions either transform categorical features into numerical ones using techniques like one-hot encoding (Kim et al., 2023; Liu et al., 2023b) and analog bit encoding (Zheng & Charoenphaklee, 2022) or resort to two separate diffusion processes for numerical and categorical features (Kotelnikov et al., 2023; Lee et al., 2023). However, it has been proven that simple encoding methods lead to suboptimal performance (Lee et al., 2023), and learning separate models for different data types makes it challenging for the model to capture the co-occurrence patterns of different types of data. Therefore, we seek to develop a diffusion model in a joint space of numerical and categorical features that preserves the inter-column correlations.
This paper presents TABSYN, principled approach for tabular data synthesis. To handle mixed- typed inputs. TABSYN first transforms raw tabular data into continuous embedding space, where well-developed diffusion models with Gaussian noises become feasible. Subsequently, we learn a score-based diffusion model in the embedding space to capture the distribution of latent embeddings. To learn an informative, smoothed latent space while maintaining the decoder's 's reconstruction ability, specifically designed Variational AutoEncoder (VAE (Kingma & Welling, 2013)) model for tabular- structre data. Our proposed VAE model includes 1) Transforrer--rchhiecttrr encoders and decoders for modeling inter-column relationships and obtaining token-level representations. facilitating token-level tasks. 2) Adaptive loss weighting to dynamically adjust the reconstruction loss weights and KL-divergence weights, allowing the model to improve reconstruction performance gradually while maintaining regularized embedding space. 3) Finally, when applying diffusion models in the latent space, we adopt a simplified forward diffusion process. which adds Gaussian noises of linear standard deviation with respect to time. We demonstrate through theoretical analysis and empirical justifications that this approach can reduce the errors in the reverse process, thus improving sampling speed.
The advantages of TABSYN are fhree-fold: (1) Generality: Mixed-type Feature Handling TABSYN transforms diverse input features, encompassing numerical, categorical, etc., into unified embedding space. (2) Quality: High Generation Quality- with tailored designs of the VAE model, the tabular data is mapped into regularized latent space of good shape, e.g., standard normal distribution. This will greatly simplify training the subsequent diffusion model (Vahdat et al., 2021), making TABSyN more expressive and enabling to generate high-quality synthetic data. (3) Speed: With the proposed linear noise schedule, our TABSYN can generate high-quality synthetic data with fewer than 20 reverse steps, which is significantly fewer than existing methods.
Recognizing the absence of unified and comprehensive evaluations for synthetic tabular data (Du & Li, 2024), we perform extensive experiments, which involve comparing TABSYN with seven state-of- art methods on six mixed-type tabular datasets using over five distinct evaluation metrics. The experimental results demonstrate that TABSYN consistently outperforms previous methods (see Figure 1). Specifically, TABSYN reduces the average errors in column-wise wise distribution shape estimation (i.e., single density) and pair-wise column correlation estimation (i.e., pair correlation) tasks by 86% and 67% than the most competitive baselines. Furthermore, we demonstrate that TABSYN achieves competitive performance across two downstream tabular data tasks, machine learning efficiency and missing value imputation. Specifically, the well-learned unconditional YN is able to be applied to missing value imputation without retraining. Moreover, thorough ablation studies and visualization case studies substantiate the rationale and effectiveness of our developed approach.
2 RELATED WORKS
Deep Generative Models for Tabular Data Generation. Generative models for tabular data have become increasingly important and have widespread applications Assefa et al. (2021); Zheng & Charoenphakdee (2022); Hernandez et al. (2022). To deal with the imbalanced categorical features, Xu et al. (2019) proposes CTGAN and TVAE based on the popular Generative Adversarial Net- works (Goodfellow al., 2014) and VAE (Kingma & Welling, 2013), respectively. Multiple advanced methods have been proposed for synthetic tabular data generation in the past year. Specifically, GOG- GLE (Liu et al., 2023b) became the first explicitly model the dependency relationship between
2

---

Published as a conference paper at ICLR 2024
p(50) p())
2x 20 the
p(ar) s p(nr)
Age Edusction Erioea
Diffusion Process
Transformers
Forward Process
Docrcati
(e))
2g
berossin MCP
Reverse
Lite Deumatte
1908
Transformers
Flatten
Process
Detokenizer
Decoder
Latent
Noise Distribution
Distribution 2g
Zt
TT
Space(VAE)
Latent Space (Diffusion)
Figure 2: An overview of the proposed TABSYN. Each row data r is mapped to latent space via column-wise tokenizer an encoder. diffusion process En + 2r applied in the latent space. Synthesis - 20 starts from the base distribution p(zr) and generates samples in latent space through reverse process. These samples are then mapped from latent z to data space ă using decoder and detokenizer.
columns, proposing a VAE-based model using graph neural networks as the encoder and decoder models. Inspired by the success of large language models in modeling the distribution of natural languages, GReaTT transformed each row in a table into a natural sentence and learned sentence-level distributions using auto-regressive GPT2. recent years, the physical diffusion process has inspired lot of advanced research in deep learning. For example, DIFFormer (Wu et al., 2023) develops scalable Transformer model for geometric data via a constrained diffusion process, and the Denoising Diffusion models have achieved great success in image generation (Ho et al., 2020). STaSy (Kim et al., 2023), TabDDPM (Kotelnikov et al., 2023), and CoDi (Lee et al., 2023) concurrently applied the popular diffusion-based generative models for synthetic tabular data generation.
Generative Modeling in the Latent Space. While generative models the data space have achieved significant success, latent generative models have demonstrated several advantages, includ- ing more compact and disentangled representations, robustness to noise, and greater flexibility in controlling generated styles (van den Oord et al., 2017; Razavi et al., 2019; Esser et al., 2021). For example, the recent GAN literature (Li et al., 2022) has demonstrated superior controllability via adversarial learning in the latent space. Recently, the Latent Diffusion Models (LDM) (Rombach et al., 2022; Vahdat et al., 2021) have achieved great success in image generation as they exhibit better scaling properties and expressivity than the vanilla diffusion models in the data space (Ho et al., 2020; Song et al., 2021b; bb; Karras et al., 2022). The success LDMs in image generation has also inspired their applications best of our knowledge, the proposed work is the first to explore the application of latent diffusion models for general tabular data generation tasks.
video (Blatimann al., 2023) and audio data (Liu al., 2023a). To the
3 SYNTHETIC TABULAR DATA GENERATION WITH TABSYN
Figure 2 gives an overview of TABSYN. In Section 1. we first formally define the tabular data generation task. Then, we introduce the design details of TABSYN's autoencoding and diffusion process in Section 3.2 and 3.3. We summarize the training and sampling algorithms in Appendix A.
3.1 PROBLEM DEFINITION OF TABULAR DATA GENERATION
Let Maum and Meat be the number of numerical columns and categorical columns, respectively. Each row represented as a vector of numerical features and categorical features num cat] where num с Mam and cat E R Meat, Specifically, the i-th categorical attribute has C; finite candidate values, therefore we have cat E (1, ......... C.), Vi. This paper focuses on the unconditional generation task. With tabular dataset T = {æ), we aim to learn parameterized generative model (T))), with which realistic and diverse synthetic tabular data & T can be generated.

---

Published as a conference paper at ICLR 2024
3.2 AUTOENCODING FOR TABULAR DATA
Tabular data is highly structured of mixed-type column features, with different columns having distinct meanings and being highly dependent on each other. These characteristics make it challenging to design an approximate encoder to model and effectively utilize the rich relationships between columns. Motivated by the successes of Transformers in classfication/regresson of tabular data (Gorishniy et al., 2021), we first learn a unique tokenizer for each column, and then the voken(column)-wi) representations are fed into a Transformer for capturing the intricate relationships among columns.
Feature Tokenizer. The feature tokenizer converts each column (both numerical and categorical) into a d-dimensional vector. First, we use one-hot encoding to prepprocess categorical features, i.e., 221 = 26th E II Each record is represented as [-- - 2h E RR Mg-M + Mr C, Then, we apply linear transformation for numerical columns and create an embedding lookup table for categorical columns, where each category is assigned a learnable d-dimensional vector, i.e.,
e, equm
zi pum w + bpum,
efst = =
W. + bfat
where w mum boum, bfat I W RCxxd are learnable parameters of the tokenizer, e
eum ? E Rxxd Now, each record is expressed as the stack of the embeddings of all columns
E Le
pum Maun'
ei
cat Mons
Rdxd
Transformer Encoding and Decoding. As with typical VAEs, we use the encoder to obtain the mean and log variance of the latent variable. Then, we acquire the latent embeddings with the reparameterization tricks. The latent embeddings are then passed through the decoder to obtain the reconstructed token matrix É Mxd The detailed architectures are in Appendix D.
Detokenizer, Finally, we apply a detokenizer to the recovered token representation of each column to reconstruct the column values. The design of the detokenizer is symmetrical to that of the tokenizer:
num i
W
oh a
Softmax(é,
Weat
(cat).
I
Aum Muum
L1
Meall
where uouum
dx jqum
RIXI
W E Rdx. iest RIxx, are detokenizer's parameters.
Training with adaptive weight coefficient. The VAE model is usually learned with the classical ELBO loss function, but here we use 3-VAE (Higgins et al., 2016), where coefficient balances the importance of the reconstruction loss and KL-divergence loss
C Grocon (2x, a)) + 3la1-
(4)
Crecon is the reconstruction loss between the input data and the reconstructed one, and Eu is the KL divergence loss that regularizes the mean and variance of the latent space. In the vanilla VAE model, is set to be 1 because the two loss terms are equally important to generate high-quality synthetic data from Gaussian noises. However, in our model, 3 is expected to be smaller, as we do not require the distribution of the embeddings to precisely follow standard Gaussian distribution because have an additional diffusion model. Therefore, we propose to adaptively schedule the scale of /3 in the training process, encouraging the model achieve lower reconstruction error while maintaining an appropriate embedding shape.
With an initial (maximum) = 9max, we monitor the epoch-wise reconstruction loss Locon- When frecon fails to decrease for a predefined number of epochs (which indicates that the KL-divergence dominates the overall loss), the weight is scheduled by B AB, À < This process continues until 3 approaches predefined minimum value 3min- This strategy is simple yet very effective, and we empirically justify the effectiveness the design in Section
3.3 SCORE-BASED GENERATIVE MODELING IN THE LATENT SPACE
Training and sampling via denoising. After the VAE model is well-learned, extract the latent embeddings through the encoder and flatten the encoder's output as x = Flatten (x)) e Rixdd such that the embedding of record is vector rather than matrix. To learn the underlying
4

---

Published as a conference paper at ICLR 2024
distribution of embeddings p(z), we consider the following forward diffusion process and reverse sampling process (Song et al., 2021b; Karras et al., 2022):
Zt 20 + (t)e, E 1),
(Forward Process)
(5)
dz 2 logr/t do + (t)
(Reverse Process)
(6)
where 20 = x is the initial embedding from the encoder, 2, is the diffused embedding at time t, and (t) is the noise level. In the reverse process, Vz, log pt (2,) is the score function of zt, and wi is the standard Wiener process. The training of the diffusion model is achieved via denoising score matching (Karras et al., 2022):
L = (ppx)( Bio Be Berpopeeeeeroooollooe..... - =) where 2, + (I)e,
where 6g is a neural network (named denoising function) to approximate the Gaussian noise using the perturbed data æ, and the time t. Then v., log p(z)) Ey (Zt, (t) After the model is trained, synthetic data can be obtained via the reverse process in Eq. 6. The detailed algorithm description of TABSYN is provided in Appendix Detailed derivations are in Appendix B.
Schedule of noise level (f). The noise level (t) defines the scale of noises for perturbing the data at different time steps and significantly affects the final Differential Equation solution trajectories (Song et al., 2021b; Karras et al., 2022). Following the recommendations in Karras et al. (2022), we set the noise level ((t) = that is linear w.r.t. the time. We show in Proposition that the linear noise level schedule leads to the smallest approximation errors in the reverse process:
Proposition 1. Consider the reverse diffusion process in Equation (6) from 2t, to Zt., (ta > ta), the numerical solution 2,, has the smallest approximation error to 2t., when o (1) =
See proof in Appendix C. A natural corollary of Proposition is that small approximation error allows us to increase the interval between two timesteps, thereby reducing the overall number of sampling steps and accelerating the sampling. In Section we demonstrate that with this design. TABSYN can generate synthetic tabular data of high quality within less than 20 NFEs (number of function evaluations), which is much smaller than other tabular-data synthesis methods based on diffusion (Kim et al., 2023; Kotelnikov et al., 2023).
4 BENCHMARKING SYNTHETIC TABULAR DATA GENERATION ALGORIHTM
4.1 EXPERIMENTAL SETUPS
Datasets. We select six real-world tabular datasets consisting of both numerical and categorical attributes: Adult, Default, Shoppers, Magic, Faults, Beijing, and News. Table 6 provides the overall statistics of these datasets, and the detailed descriptions can be found in Appendix E.
Baselines. We compare the proposed TABSYN with seven existing synthetic tabular data generation methods. The first two are classical GAN and VAE models: CTGAN (Xu et al., 2019) and TVAE (Xu al., 2019). Additionally, we evaluate five SOTA methods introduced recently: GOGGLE (Liu et al., 2023b), a VAE-based method; GRea'T (Borisov et al., 2023), a language model variant; and three diffusion-bassee CoDi (Lee et al., 2023). Notably, these approaches were nearly simultaneously introduced, limiting opportunities for extensive comparison. For reference, we also compare with the representative interpolation- based method SMOTE (Chawla et al., 2002). Our paper fills this gap by providing the first comprehensive evaluation of their performance in a standardized setting.
methods: STaSy (Kim al., 2023). TabDDPM (Kotelnikov et al., 2023), and
Evaluation Methods. We evaluate the quality of the synthetic data from three aspects: 1) Low-order statistics column-wise density estimation and pair-wise column correlation, estimating the density of every single column and the correlation between every column pair (Section 4.2). We also perform Classifier Two Sample Test (C2ST) to evaluate if the synthetic data can be detected from the real data via a machine learning model (Appendix F.3); 2) High-order metrics a-precision and 3-recall scores (Alaa et al., 2022) that measure the overall fidelity and diversity of synthetic data (the results deferred to Appendix F.2); 3) Privacy Protection: we also evaluate if the synthetic data is randomly sampled according to the distribution density rather than copied from the training data via Distance to Closest Records (DCR) in Appendix F.6; 4) Performance downstream tasks
5

---

Published as a conference paper at ICLR 2024
Table Error rate (%) of column-wise density estimation. Bold Face represents the best score on each dataset. Lower values indicate more accurate estimation (superior results). TABS outperforms the best generative baseline model by 86.0% on average.
Method
Adult
Default
Shoppers
Magic
Beijing
News
Average
SMOTE
1.60
1.48+0.16
2.68+40.19
0.91=0.05
a.
4.31
2.30
CTGAN
16.84 0.03
16.83+0.04
21.15=0.10
9.80=
21.39 20.39
16.0920.02
17.02
TVAE
14.2240.0s
10.1740.05
24.51:0.06
8.25=0.06
19.1640.06
16.62
15.49
GOGGLE
16.97
17.02
22.33
1.90
16.93
25.32
16.74
GReaT
12.1220.00
19.94100.8
14.11
16. 16.1620.00
8.25
00M
14.20
STaSy
11.29=00.0
5.77+0.06
9.37=+0.09
6.29=0018
6.71+0.08
6.89+0.08
7.72
CoDi
21.38=0.06
0.07
31.84+0.05
11.56=0.26
16.9440.02
32.27=0.04
21.63
TabDDPM
1.75.10.3
1.57
2.72.0.13
1.0110.09
1.3010.03
78.75.1..0
14.52
TABSYN
0.58
0.85
1.43
0.88+0.08
1.1240.05
1,64:0.04
1.08
Improv.
66.9% 4
45.9%
47.4% 4
12.9% 4.
13.8% 4
76.2% 4
86.0% 4
GOGGLE fixes the random seed during sampling in the official codes, and we follow it for consisteney. GReaT cannot be applied News because of the maximum length limit.
TabDDDMM fails to generate meaningful content on the News dataset.
Table 2: Error rate (%) of pair-wise column correlation score. Bold Face represents the best score on each dataset. TABSYN outperforms the best baseline model by 67.6% on average.
Method
Adult
Default
Shoppers
Magic
Beijing
News
Average
SMOTE
3.2810.20
8.41
3.5610.22
3. 3.1610.41
2.30
1.
4.36
CTGAN
20.23
26.951a.93
13.08
7.0010.19
22.95
5.37
15.93
TVAE
14.15:1.88
1..5
18.67
0.82+..49
18.01
6.17
13.72
GOGGLE
45.29
21.94
23.90
9.47
45.94
23.19
28.28
GReaT
17.5910:22
70.0220.12
45.1610.18
10.2310.a
59.60
0OM
44.24
STaSy
14.5110.25
5.9610.26
8.4910.15
6.61
8.00
3.07
7.77
CoDi
22.49.10.0s
68.41
17.78
6.
7.07
11.1010.00
22.23
TabDDPM
3.01+0.25
4.80
6.6140.16
1.70
2.71:0.00
13.16:0.11
5.34
TABSYN
1.54=0.27
2.05++.112
2.07+0.21
1.06+0.a1
2.24+0.28
1.44-0.03
1.73
Improve.
48.8% L
58.1% 4
68.7% $
37.6% J.
17.3% $
53.1%
67.6%
machine learning efficiency (MLE) and missing value imputation. MLE is to compare the testing accuracy on real data when trained on synthetically generated tabular datasets. The performance on privacy protection is measured by MLE tasks that have been widely adopted in previous literature (Section 4.3). We also extend TABSYN for the missing value imputation task, which aims fill in missing features/labels given partial column values (Appendix F.4). The reported results are averaged over randomly sampled synthetic data. The implementation details are in Appendix E.
4.2 ESTIMATING LOW-ORDER STATISTICS OF DATA DENSITY
Metrics. We employ the Kolmogorov-Sirnov Test (KST) for numerical columns and the Total Variation Distance (TVD) for categorical columns to quantify column-wise density estimation. For pair-wise column correlation, use Pearson correlation for numerical columns and contingency similarity for categorical columns. The performance is measured by the difference between the correlations computed from real data and synthetic data. For the correlation between numerical and categorical columns, we first group numerical values into categorical ones via bucketing, then calcu- late the corresponding contingency similarity. Further details on these metries in Appendix E.3.
Column-wist distribution density estimation. In Table 1, we note that TABSYN consistently outperforms baseline methods in the column-wise distribution density estimation task. On average, TABSYN surpasses the most competitive baselines by 86.0%. While STaSy and TabDDPM perform well, STaSy is sub-optimal because treats one-hot embeddings of categorical columns continuous features. Additionally, TabDDPM exhibits unstable performance across datasets, failing to generate meaningful content the News dataset despite standard training process.

---

Published as a conference paper at ICLR 2024
Table 3: AUC (classification task) and RMSE (regression task) scores of Machine Learning Er- ficiency. 1 (L) indicates that the higher (lower) the score, the better the performance. TABSYN consistently outperforms all others across all datasets.
Adult
Default
Shoppers
Magic
Beijing
News'
Average Gap
Methods
AUC
AUC
AUC
A
RMSE 4
RMSE 4
%
Real
.27...00
.7701.005
.266..00
.9461.0m
4231
8421.
0%
SMOTE
899 007
4..1
.911+.012
.044.008
1533
8874.036
9.39%
CTGAN
8862
6962.000
8855
8552.000
.9022.
8802.018
24.5%
TVAE
878..
.7241.005
.8711.00
.8871.003
.770..011
1.011.018
20.9%
GOGGLE
.778=.012
.584+.005
.658
.654..02
1.09#.025
.877.0002
43.6%
GReaT
.13=
.755=-000
.902
.8882.008
.6532.013
OOM
13.3%
STaSy
.906+.001 001
.752+.006
.914 914++003 005
.934+.008
.656+.014
+.01
10.9%
CoDi
.871#.000 006
.525+.006
.865+.006 D0G
.932+.008
.818+.021
1.21
30.5%
T
.907..001 001
.7581.004
.9181.o0s
.9351.003
.5921.011
4.8613.04
9.14%
TABSYN
9151..00 0002
.04
.9201.000 .00
.38...02
.5821.00 1.08
861 0207
7.23%
Following CoDi (Lee al., 2023), continuous targets are standardized prevent large values.
TahDDPM collapses on News, leading to an extremely high error on this dataset. We exclude this dataset when computing the average gap of TahDDPM.
Pair-wise column correlations. Table 2 displays the results of pair-wise column correlations. TABSYN outperforms the best baselines by an average of 67.6%. Notably, the performance of GReaT is significantly poorer in this task than in the column-wise task. This indicates the limitations of autoregressive language models density estimation, particularly in capturing the joint probability distributions between columns.
4.3 PERFORMANCE ON DOWNSTREAM TASKS
Machine Learning Efficiency. We then evaluate the quality of synthetic data by evaluating their performance in Machine Learning Efficiency tasks. Following established settings (Kotelnikov et al., 2023; Kim et al., 2023; Lee et al., 2023), we first split real table into real training and a real testing set. The generative models are learned on the real training set, from which synthetic set of equivalent size is sampled. This synthetic data is then used to train a classification/regesssoo model (XGBoost Classifier and XGBoost Regressor (Chen & Guestrin, 2016)), which will be evaluated using the real testing set. The performance of MIE measured by the AUC score for classification tasks and RMSE for regression tasks. The detailed settings of the MLE evaluations are in Appendix E.4.
In Table 3, we demonstrate that TABSYN consistently outperforms all the baseline methods. The performance gap between methods is smaller compared to column-wis density and pair-wise column correlation estimation tasks (Tables and 2). This suggests that some columns may not significantly impact the classificatinn/egressso tasks, allowing methods with lower performance in previous tasks to show competitive results in MLE (e.g.. GReaT on Default dataset). This underscores the need for comprehensive evaluation approach beyond just MLE metries. As shown above, we have incorporated low-order and high-order statistics for a more robust assessment.
Missing Value Imputation. One advantage of the diffusion model is that well-trained uncon- ditional model can be directly used for data imputation (e.g., image inpainting (Song et al., 2021b; Lugmayr et al., 2022)) without additional training. This paper explores adapting TABSYN Y for missing value imputation, a crucial task in real-world tabular data. Due to space limitation, the detailed algorithms for missing value imputation and the results are deferred to Appendix F.4.
4.4 ABLATION STUDIES
The effect of adaptive 3-VAE. We assess the effectiveness of scheduling the weighting coefficient 3 in the VAE model. Figure 3 presents the trends of the reconstruction loss and the KL-divergence loss with the scheduled 3 and constant 3 values (from 10-1 to 10 5) across 4,000 training epochs. Notably, large 3 value leads to subpar reconstruction, while small 3 value results in a large divergence
7

---

Published as a conference paper at ICLR 2024
Reconstruction Loss
KL Divergence Loss
2-11
p- 00001
2-0.0002 Sched
Table 4: The results of single- column density and pair-wise column correlation estimation with different values on the Adult dataset.
5:00
15
Loos
10--
10
10-1
1000
4000
B
Single
Pair
Training Epochs
Training Epochs
10 1
1.24
3.05
10-2
0.87
2.79
Figure 3: The trends of the validation reconstruction (left) and KL-divergence (right) losses on the Adult dataset, with varying constant 3, and our proposed scheduled / (8uax 0.01, 9mi
10-3 3
0.72
2.25
10-4
0.69
2.01
10-3
41.96
69.17
10-5, A = 0.7). The proposed scheduled / obtains the lowest reconstruction loss with fairly low KL-divergence loss.
Scheduled /
0.58
1.54
Adult
Default
Shoppers
Table 5:
Performance of
STady 1a0200
Fndy Seliie
Fldyy baacien
TABSYN's variants on Adult dataset, on low-order statis- tics estimation tasks.
200rage ener
Appe Sagaaea wnter
21 12
112 102
236 X12107
Variants
Single
Pair
NFEs
TabDDPM
1.75
3.01
Figure 4: Quality of synthetic data as function of NFEs on STaSy, TabDDPM, and TABSYN. TABSYN generate synthetic data of the best quality with fewer NFEs (indicating faster sampling speed).
TABSYN-Onete
5.59
6.92
TABSYN DDPM
1.02
2.15
TABSYN
0.58
1.54
between the embedding distribution and the standard Gaussian, making the balance hard to achieve. In contrast, by dynamically scheduling / during training (8max 0.01, Suni 10-5 A 0.7), we not only prevent excessive KL divergence but also enhance quality. Table 4 further evaluates the learned embeddings from various values of the VAE model via synthetic data quality (single- column density and pair- wise column correlation estimation tasks). This demonstrates the superior performance of our proposed scheduled approach to train the VAE model.
The effect of linear noise levels. We evaluate the effectiveness of using linear noise levels, (tt) = t, in the diffusion process. As Section 3.3 outlines, linear noises lead to linear trajectories and faster sampling speed. Consequently, we compare TABS YN and two other diffusion models (STaSy and TabDDPM) in terms of the single-column density and pair-wise column correlation estimation errors relative to the number of function evaluations (NFEs), i.c.. denoising steps to generate the real data. As continuous- time diffusion models, the proposed TABSYN and STaSy are flexible in choosing NFEs. For TabDDPM, we use the DDIM sampler (Song et al., 2021a) to adjust NFEs. Figure 4 shows that TABSYN not only significantly improves the sampling speed but also consistently yields better performance (with fewer than 20 NFEs for optimal results). In contrast, STaSy requires 50-200 NFEs, varying by datasets, and achieves sub-optimal performance. TabDDPM achieves competitive performance with 1,000 NFEs but significantly drops in performance when reducing NFEs.
Comparing different encoding/diffluson methods. We assess the effectiveness of learning the diffusion model in the latent space learned by VAE by creating two TABSyN variants: 1) TABSYN- OneHot: replacing VAE with one-hot encodings of categorical variables and 2) TABSYN-DDPM: substituting the diffusion process in Equation (5) with DDPM as used in TabDDPM. Results in Table 5 demonstrate: 1) One-hot encodings for categorical variables plus continuous diffusion models lead to the worst performance, indicating that it is not appropriate to treat categorical columns simply as continuous features; 2) TABS BIO-SN-DDDPP DDPM the latent space outperforms TabDDPM in the data space, highlighting the benefit of learning high-quality latent embeddings for improved diffusion modeling: 3) TABSYN surpasses TABSyN-Da DDPM,, indicating the advantage of employing tailored diffusion models in the continuous latent space for better data distribution learning.
8

---

Published as a conference paper at ICLR 2024
Default
<hrle Ballgn
0.65
Dentyy
Balge
Boage
0.43
sedeo sececo 156000 203012
0.00
0:55
0:30
6.55
0.20
26
AML
ExtRates
TEMP
20838
27000
32530
32008-
10000
13000
Nall
21000
3000
5780
1000
2300
Tum
wan.oeeLLaa Work
-4
Nctan
Vistor Type
Figure 5: Visualization of synthetic data's single column distribution density (from STaSy, MabDDPM, and TABSYN) v.s. the real data. Upper: numerical columns; Lower: Categorical columns. Note that numerical columns show competitive performance with baselines, while TABSYN excels in estimating categorical column distributions.
CTGAAN
TVAE
GReaT
GOOGLE
CoDi
STaSy
TabDDPM
TabSyn
0.8
0.7
0.6
Defnult
0.5
Shoppers
0.1
0.0
Figure 6: Heatmaps of the pair-wise column correlation of synthetic data v.s. the real data. The value represents the absolute divergence between the real and estimated correlations (the lighter, the better). TABSYN gives the most accurate column correlation estimation.
4.5 VISUALIZATION
In Figure 5, we compare column density across eight columns from four datasets (one numerical and one categorical per dataset). TabDDPM matches TABSYN's accuracy on numerical columns but falls short categorical ones. Figure 6 displays the divergence heatmap between estimated pair-wise column correlations and real correlations. TABSyN gives the most accurate correlation estimation, while other methods exhibit suboptimal performance. These results justify that employing generative modeling in latent space enhances the learning categorical features and joint column distributions.
CONCLUSIONS
In this paper, we have proposed TABSyN for synthetic tabular data generation. The TABSyN framework leverages VAE to map tabular data into a latent space, followed by utilizing a diffusion- based generative model to learn the latent distribution. This approach presents the dual advantages of accommodating numerical and categorical features within unified latent space, thus facilitating a more comprehensive understanding advanced generative models in continuous embedding space. To address potential challenges, TABSyN proposes a model design and training methods, resulting in a highly stable generative model. In addition, TABSyN rectifies the deficiency in prior research employing diverse set of evaluation metrics to comprehensively compare the proposed method with existing approaches, showcasing the remarkable quality and fidelity of the generated samples in capturing the original data distribution.
their interrelationships and enabling the utilization of

---

Published as a conference paper at ICLR 2024
REFERENCES
Ahmed Alaa, Boris Van Breugel, Evgeny S Saveliev, and Mihaela van der Schaar. How faithful is your synthetic data? sample-level metries for evaluating and auditing generative models. In International Conference Machine Learning, pp. 290-306. PMLR, 2022.
Samuel A. Assefa, Danial Dervovic, Mahmoud Mahfouz, Robert E. Tillman, Prashant Reddy, and Manuela Veloso. Generating synthetic data in finance: Opportunities, challenges and pitfalls. In Proceedings of the First ACM International Conference on Al in Finance, ICAIF '20. Association for Computing Machinery, 2021. ISBN 978 9781450375
Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hinton. Layer normalization. arXiv preprint arXiv:1607.064500 2016.
Andreas Blattmann, Robin Rombach, Huan Ling, Tim Dockhorn, Seung Wook Kim, Sanja Fidler, and Karsten Kreis. Align your latents: High-resolution video synthesis with latent diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pp. 22563-22573 2023.
Vadim Borisov, Kathrin Sessler, Tobias Leemann, Martin Pawelczyk, and Gjergji Kasneci. Language models are realistic tabular data generators. In The Eleventh International Conference on Learning Representations, 2023.
Nitesh V Chawla, Kevin W Bowyer, Lawrence Hall, and W Philip Kegelmeyer. Smote: synthetic minority over-sampling technique. Journal of artificial intelligence research, 16:321-357, 2002.
Tianqi Chen and Carlos Guestrin. Xghoost: A scalable tree boosting system. In Proceedings of the
22nd ACM SIGKDD Conference on Knowledge Discovery and Data Mining, pp- 785-794, 2016.
Yuntao Du and Ninghui Li. Towards principled assessment of tabular data synthesis algorithms. arXiv preprint arXhv:2402.06866, 2024.
Patrick Esser, Robin Rombach, and Bjorn Ommer. Taming transformers for high- reshurccolution image synthesis. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 12873-12883, 2021.
Joao Fonseca and Fernando Bacao. Tabular and latent space synthetic data generation: a literature review. Journal of Big Data, 10(1):115, 2023.
Ian J Goodfellow, Jean Pouget-Ahadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. In Proceedings of the 27th International Conference on Neural Information Processing Systems, pp. 2672-2680, 2014.
Yury Gorishniy, Ivan Rubachev, Valentin Khrulkov, and Artem Babenko. Revisiting deep learning models for tabular data. In Proceedings of the 35th International Conference on Neural Information Processing Systems, pp. 1 (19332-18944 2021.
Mikel Hernandez, Gorka Epelde, Ane Alberdi, Rodrigo Cilla, and Debbie Rankin. Synthetic data generation for tabular health records: A systematic review. Neurocomputinng 493:28 45, 2022.
Irina Higgins, Loic Matthey, Arka Pal, Christopher Burgess, Xavier Glorot, Matthew Botvinick, Shakir Mohamed, and Alexander Lerchner. beta-vae: Learning basic visual concepts with constrained variational framework. In The Forth International Conference on Learning Represen- tations, 2016.
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Proceedings of the 34th International Conference Neural Information Processing Systems, pp. 6840-6851, 2020.
Emiel Hoogeboom, Didrik Nielsen, Priyank Jaini, Patrick Forré, and Max Welling. Argmax flows and multinomial diffusion: Learning categorical distributions. In Proceedings of the 35th International Conference Neural Information Processing Systems, pp. 22454-12465 2021.
10

---

Published as a conference paper at ICLR 2024
Tero Karras, Miika Aittala, Timo Aila, and Samuli Laine. Elucidating the design space of diffusion- based generative models. In Proceedings of the 36th International Conference on Neural Informa- fion Processing Systems, pp.
2022.
Jayoung Kim, Chaejeong Lee, Yehjin Shin, Sewon Park, Minjung Kim, Noseong Park, and Jihoon Cho. Sos: Score-based oversampling for tabular data. In Proceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, pp. 762-772, 2022.
Jayoung Kim, Chaejeong Lee, and Noseong Park. Stasy: Score-based tabular data synthesis. In The Eleventh International Conference on Learning Representations, 2023.
Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In International
Conference on Learning Representations, 2015.
Diederik P Kingma and Max Welling. Auto-encoding variational bayes. arXiv:1312.6114, 2013.
arXiv preprint
Akim Kotelnikov, Dmitry Baranchuk, Ivan Rubachev, and Artem Babenko. Tabddpm: Modelling tabular data with diffusion models. International Conference on Machine Learning, pp. 17564- 17579. PMLR, 2023.
Chaejeong Lee, Jayoung Kim, and Noseong Park. Codi: Co-evolving contrastive diffusion models for mixed-type tabular synthesis. In International Conference on Machine Learning, pp. 18940-18956. PMLR, 2023.
Yang Li, Yichuan Mo, Liangliang Shi, and Junchi Yan. Improving generative adversarial networks adversarial learning in latent space, Advances in Neural Information Processing Systems, 35: 8868-8881, 2022.
Haohe Liu, Zehua Chen, Yi Yuan, Xinhao Mei, Xubo Liu, Danilo Mandic, Wenwu Wang, and Mark D Plumbley. Audioldm: Text-to-audio generation with latent diffusion models. arXiv preprint arXiv:2001.12500 2023a.
Tennison Liu, Zhaozhi Qian, Jeroen Berrevoets, and Mihaela van der Schaar. Goggle: Generative modelling for tabular data by learning relational structure. In The Eleventh International Conference on Learning Representations, 2023b.
Andreas Lugmayr, Martin Danelljan, Andres Romero, Fisher Yu, Radu Timofte, and Lue Van Gool. Repaint: Inpainting using denoising diffusion probabilistic models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 11461-11471, 2022.
Ali Razavi, Aäron den Oord, and Oriol Vinyals. Generating diverse high- images with vq-vae-2. In Proceedings of the 33rd International Conference on Neural Information Processing Systems, pp. 14866-14876, 2019.
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High- resolution image synthesis with latent diffusion models. In Proceedings of the IEEF/CVF confer- ence on computer vision and pattern recognition, pp. 10695-10695, 2022.
Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. In The Ninth International Conference Learning Representations, 2021a.
Yang Song, Jascha Sohl-Dickstein, Diederik Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. In The Ninth International Conference on Learning Representations, 2021b
Arash Vahdat, Karsten Kreis, and Jan Kautz. Score-based generative modeling in latent space. In Proceedings of the 35th International Conference on Neural Information Processing Systems, pp. 11287-11302, 2021.
Aaron den Oord, Oriol Vinyals, and Koray Kavukcuoglu. Neural discrete representation learning. In Proceedings of the 31st International Conference on Neural Information Processing Systems, pp. 6309-6318, 2017.
11

---

Published as a conference paper at ICLR 2024
Qitian Wu, Chenxiao Yang, Wentao Zhao, Yixuan He, David Wipf, and Junchi Yan. Difformer: Scal- able (graph) transformers induced by energy constrained diffusion. In The Eleventh International Conference Learning Representations, 2023.
Lei Xu, Maria Skoularidou, Alfredo Cuesta-Infante, and Kalyan Veeramachaneni. Modeling tabular data using conditional gan. In Proceedings of the 33rd International Conference on Neural Information Processing Systems, pp. 7335-7345, 2019.
Shuhan Zheng and Nontawat Charoenphakdee, Diffusion models for missing value imputation in tabular data, XrXiv preprint arXiv:2210.171288 2022.
12

---

Published as a conference paper at ICLR 2024
A ALGORITHMS
In this section, provide algorithmic illustration of the proposed TABSYN. Algorithm 1 and Algorithm 2 present the algorithms of the VAE and Diffusion phases of the training process of TABSYN, respectively. Algorithm 3 presents TABSYN'S sampling algorithm.
Algorithm 1: TABSYN: Training of VAE
Algorithm 2: TABSYN: Training of Diffu-
Sample (2 z0a6) p(T)
sion
2: Get tokenized feature via Eq. 1
1: Sample the embedding 2o from
= =(z))
3: Get μs and log a via VAE's Transformer encoder
2: Sample time steps / from p(t) then get (t)
3: Sample noise vectors E - (0,
4: Reparameterization: & = a E o, where
~ I",
4: Get perturbed data
20 +
5: Get via VAE's Transformer decoder
5: Calculate loss P(0) = ||ca(z,, t) - e
6: Update the network parameter 0 via Adam optimizer
6: Get detokenized feature via Eq. 3
7: Calculate loss
C Lecon 2z, 3lal(, or)
8: Update the network parameter via Adam optimizer
9: if frecon fails to decrease for S steps then
B AB
11: end if
Algorithm 3: TABSYN: Sampling
1: Sample 2y ~ N/0, (T) Imax =
2: for max,
do
(z, 4: get - solving the SDE in Eq. 6.
3: Vx,, log (ogp(zz;) = E , (t)/((t))
5: end for
6: Put 2g as imput of the VAE's Transformer decoder, then acquire
7: Get detokenized feature 2 via Eq. 3
8: 2 the sampled synthetic data
B DIFFUSION MODELS BASICS
Diffusion models are often presented as a pair of two processes.
A fixed forward process governs the training of the model, which adds Gaussian noises of increasing scales to the original data.
A corresponding backward process involves utilizing the trained model iteratively to denoise the samples starting from fully noisy prior distribution.
B.1 FORWARD PROCESS
Although there are different mathematical formulations (discrete or continuous) of the diffusion model, Song et al. (2021b) provides unified formulation via the Stochastic Differential Equation (SDE) and defines the forward process of Diffusion as (note that in this paper, the independent variable is denoted as 2)
dz f(z t) + g(t dwi.
(8)
where f(-) and g(-) are the drift and diffusion coefficients and are selected differently for different diffusion processes, e.g., the variance preserving (VP) and variance exploding (VE) formulations. w the standard Wiener process. Usually, f(-) is of the form f(z. t) = f(t) z. Thus, the SDE can be equivalently written
dz f(t) di + g() dur--
13

---

Published as a conference paper at ICLR 2024
Let be function of the time t, i.e., (t), then the conditional distribution of given 20 (named as the perturbation kernel of the SDE) could be formulated as:
p(z1(z0) (2 st)) (t))
(10)
where
s(t) exp (L. S(E)dE): and o(t)
V. gP(E) de. SP(E)
(11)
Therefore, the forward diffusion process could be equivalently formulated by defining the perturbation kernels (via defining appropriate st)) and (()).
Variance Preserving (VP) implements the perturbation kernel Eq. 10 by setting s(t) = / 3()), and (t)
Btr) (()) (s²(t) + (t) = 1). Denoising Diffusion Probabilistic Models
(DDPM, Ho al. (2020)) belong VP-SDE by using discrete finite time steps and giving specific
functional definitions of (t)).
Variance Exploding (VE) implements the perturbation kemnel Eq. 10 by setting s(t) indicating that the noise is directly added to the data rather than weighted mixing. Therefore, The noise variance (the noise level) is totally decided by a (t). The diffusion model used TABSYN belongs to VE-SDE, but we use linear noise level (i.e., (t) = t) rather than (t)) vt in the vanilla VE-SDE (Song et al., 2021b). When st)) = 1, the perturbation kernels become:
p(z)(z0)= Nz 0, (
21 20 (t)E,
(12)
which aligns with the forward diffusion process in Eq. 5.
B.2 REVERSE PROCESS
The sampling process of diffusion models is defined by corresponding reverse SDE:
dz = [f(z,)) -0 (1) > lo)d: + g())dw,
(13)
For VE-SDE, s(t) = 1 + f(z.t) t) = f(t) 2 = O, and
(11) V S. (P())EEE = S ((())EE= o2(),
(2(1)= = do²(t) 20 (1))(1),
(14)
dt
g()) 20(())(1))
Plugging g() into Eq. 13, the reverse process in Eq. is recovered:
dz (2) (ogp)((()) (1)())(())d..)
(15)
B.3 TRAINING
As f(z, t), (t), wi are all known, if V= log pz()) (named the score function) is also available,
can sample synthetic data via the reverse process from random noises. Diffusion models train neural network (named the denoising function) Dø(z,, t) to approximate V. log (z)).. However, V. log P. (z) itself is intractable, as the marginal distribution p. (=) = p(z,) ) is intractable. Fortunately, the conditional distribution p(z (2)) tractable. Therefore, we can train the denoising function to approximate the conditional score function instead Vz, log pl z 22), and the training process is called denoising score matching:
min E Exo Ey, Peopppp Do(2,, D Vz, 1og)(z+(20))
(16)
14

---

Published as a conference paper at ICLR 2024
where v log logp(zi(zu) has analytical solution according to Eq. 10:
1
V (ogp(zz(220)
V Vx,P(ze(zo)
p(2)(20)
(2(()2211) (21- =(()2o))) p(2z(2))
p(zi|20)
(2) s(t)=o)
(17)
(2())22(t)
(s(t) + s(t)o(t)e =(()=
(t)))
s())(11)
Therefore, Eq. 16 becomes
min Ezg up(z)) B Berp () + (())a(1)
(18)
+ min Egg up Be (le =
where Do(z,,t) t) c0(
((()(())) After the training ends, sampling is enabled by solving Eq. 13 (replacing Vx log with Dz(z,, I)).
C PROOFS
C.I PROOF FOR PROPOSITION
We first introduce Lemma (from (Karras et al., 2022)), which introduces a family of SDEs sharing the same solution trajectories with different forwarding processes:
Lemma 1. Let g(t) be a free parameter functional of t, and the following family of (forward) SDEs have the same marginal distributions of the solution trajectories with noise levels a (t) for any choice of (t):
dz =
V= log g(t)dwr.
(19)
The reverse family of SDEs of Eq. 19 is given by changing the sign of the first term:
dz = 1oV
- (t)o log p(>; (t)) + g())d
(20)
Lemma indicates that for specific (forward) SDE following Eq. 19, we can obtain its solution trajectory by solving Eq. 20 any gl)).
Since our forwarding diffusion process (Eq, 5) lets g(l) = (1) (see derivations Ap- pendix B), its solution trajectory can be solved via letting y(() 0 in Eq. 20:
dz (t) logp(z;
dz
(21)
log
dt
Eq. 21 is usually named the Probability-Flow ODE, since it depicts deterministic reverse process without noise terms. Based on Lemma 1, we can study the solution of Eq. 6 using Eq. 21.
To prove Proposition 1, we only have to study the absolute enror between the ground-truth E the approximated one by solving Eq. 6 from zz, where At + 0:
1 At
(22)
Since 2, 20 a()),, there is 2 - (a(1) all - -()))-
15

---

Published as a conference paper at ICLR 2024
The solution of 2i-Ar from 2 can be obtained using 1st-order Euler method:
At dz
Euler 2 T- De.
= Zt
dt
2t At (1) log((zz)
(23)
2t Al ( 1)/o(1)
Zt ()) t)
2t - -()ee1),
2 Ar. - Fuler
= E (o(1)= - o(l - Al) =(1)AA
(24)
= (o(t) - o(t At) - ((t)tt)
Specifically, if o(t) = t, = 1, there is
=|
Fuler 2t- Arll 0.
(25)
Comparably, setting (t) = V (as in VE-SDE Song et al. (2021b)) leads to
At
- -t- = Ie(VE Vt At
Fuler
2 0.
(26)
2/
Therefore, the proof is complete.
D NETWORK ARCHITECTURES
D.1 ARCHITECTURES OF VAE
In Sec. 3.2, we introduce the general architecture our VAE module, which consists of a tokenizer, Transformer encoder, a Transformer decoder, and detokenizer. Figure is detailed illustration of the architectures of the Transformer encoder and decoder.
Reparameterization
u E
VAE's Tronsformer Encoder
VAE's Transformer Decader
Encoder
log Encoder
Decoder
Add á Narm
Add & Norm
Add Nerm
Feed Forward
Feed Forword
Feed Forword
2x
2x
2x
Self-Attention
Self-Attertion
Self-Attertion
B c Mid
Z RMa
Figure 7: Architectures VAE's Encoder and Decoder. Both the encoder and decoder are imple- mented as two-layer Transformer of identical architectures.
The VAE's encoder takes the tokenizer's output E R xdd as input. As we are using the Variational AutoEnco,er, the encoder module consists of y encoder and a log encoder of the same architecture.
16

---

Published as a conference paper at ICLR 2024
Each encoder is implemented two-layer Transformer, each with Self-Attention (without multi- head) module and a Feed Forward Neural Network (FFNN). The FFNN used in TABSYN is simple two-layer MLP with ReLU activation(the input of the FFNN is denoted by Ho):
H1 = ReLU(FC(HH)) E RRD
(27)
H2 = FC(H)). RMxxd
where FC denotes the fully-connected layer, and D is FFNN's hidden dimension. In this paper, set = and D 128 for all datasets. "Add & Norm" Figure denotes residual connection and Layer Normalization (Ba et al., 2016), respectively.
The VAE encoder outputs two matrixes: mean matrix μ E R. Mxd and log standard deviation matrix log RRxx Then, the latent variables are obtained via the parameterization trick:
Z μ E, ~ (00, I).
(28)
The VAE's decoder is another two-layer Transformer of the same architecture the encoder, and takes Z imput. The decoder is expected output É IR Mxd for the detokenizer.
D.2 ARCHITECTURES OF DENOISING MLP
In Figure 7, we present the architecture of the denoising neural networks sa(2,, 1) in Eq. 7, which is a simple MLP of the similar architecture as used in TabDDPM (Kotelnikov et al., 2023).
Input
(the denoising function)
Ouput
hidden
hidden
hidden layer
loyer
output layer
Zt
EØ(2i, t)
temb
Figure 8: Architectures of denoising function Ez f). The denoising function is a 5-layer MLP with SiLU activations. temb is the sinusoidal timestep embeddings.
The denoising MLP takes the current time step and the corresponding latent vector 2, E R1xMd as input. First, 2, is fed into linear projection layer that converts the vector dimension to be dpadden:
hg = FCn((z)) E IR 1g1x ddadden.
(29)
where hg the transformed vector, and dhidden the output dimension of the input layer.
Then, following the practice in TabDDPM (Kotelnikov et al., 2023), the sinusoidal timestep embed- dings bemb E IR xddaidden is added to ho to obtain the input vector Ippidden:
hm ho + temb.
(30)
The hidden layers are three fully connected layers of the size duidden - 22dhidden -2=dhidden -hiddens with SiLU activation functions (in consistency with TabDDPM (Kotelnikov al., 2023)):
h S SiLU(FC;(h))
R 1 x2edniaden)
h (iL)(FCy(h)) R xx2edhidmo
(31)
h S SiLU(FCs (22) R 1xduidde
The estimated score is obtained via the last linear layer:
EØ t) hout Coot (hs) E R 1xd.. Finally, (o(2,, t) is applied Eq. for model training.
(32)
17

---

Published as a conference paper at ICLR 2024
E DETAILS OF EXPERIMENTAL SETUPS
We implement TABSYN and all the baseline methods with PyTorch. All the methods optimized with Adam (Kingma & Ba, 2015) optimizer. All the experiments are conducted on an Nvidia RTX 4090 GPU with 24G memory.
E. DATASETS
We use 6 tabular datasets from UCI Machine Learning Repository': Adult, Default, Shoppers, Magic, Beijing, and News, where each tabular dataset associated with machine-learning task. Classification: Adult, Default, Magic, and Shoppers. Regression: Beijing and News. The statistics of the datasets are presented in Table
Table 6: Statistics of datasets. # Num stands for the number of numerical columns, and Cat stands for the number of categorical columns.
Dataset
Rows
# Num
# Cat
Train
Validation
Test
Task
Adult
48,
28, 943
618
16,
Classification
Default
30, 000
11
24, 000
3, 0000
3, 0.000
Classification
Shoppers
12, 330
10
8
9, 864
1,233
1, 233
Classification
Magic
19, 019
10
15, 215
1,902
1, 902
Classification
Beijing
43, 824
35, 058
4,383
4, 383
Regression
News 39, 641
31,714
3,965
3, 3,965
Regression
In Table 6, # Rows denote the number of rows (records) in the table. # Num and Cat denote the number of numerical features and categorical features, respectively. Note that the target column counted as either a numerical or categorical feature, depending on the task type. Specifically, the target column belongs to the categorical column if the task is classification; otherwise, it is numerical column. Each dataset is split into training, validation, and testing sets for the Machine Learning Efficiency experiments. As Adult has its official testing set, we directly set. The original training set of Adult is further split into training and validation split with the ratio 8 1. The remaining datasets are split into training/ waapiing/viiiation/esstinn sets with the ratio 1 with a fixed seed.
it as the testing
Below detailed introduction each dataset:
Adult:: The "Adult Census Income" dataset contains the demographic and employment- related features people. The task is to predict whether an individual's 's income exceeds 50, 000.
The "Default of Credit Card Clients Dataset" dataset contains information on default payments, demographic factors, credit data, history of payment, and bill statements of credit card clients in Taiwan from April 2005 to September 2005. The task is to predict whether the client will default payment next month.
Default
Shoppers':
The "Online Shoppers Purchasing Intention Dataset" contains information of user's webpage visiting sessions. The task predict the user's session ends with shopping behavior.
Magic The "Magic Gamma Telescope" dataset simulate registration of high energy gamma particles in ground-based atmospheric Cherenkov gamma telescope using the imaging technique. The task to classify high-energy Gamma particles the atmosphere.
hhttps://archive..
18

---

Published as a conference paper at ICLR 2024
The "Beijing PM2.5 Data" dataset contains the hourly PM2.5 data of US Embassy in Beijing and the meteorological data from Beijing Capital International Airport. The task is to predict the PM2.5 value.
Beijing":
News": The "Online News Popularity" dataset contains a heterogeneous set of features about articles published by Mashable in two years. The goal is to predict the number of shares in social networks (popularity).
E.2 BASELINES
In this section, we introduce and compare the properties of the baseline methods used in this paper.
CTGAN and TVAE are two methods for synthetic tabular data generation proposed in one paper (Xu et al., 2019), using the same techniques proposed but based on different basic generative models GAN for CTGAN while VAE for TVAE. The two methods contain two important designs: 1) Mode-specific Normalization to deal with numerical columns with complicated distributions. Conditional Generation of numerical columns based on categorical columns to deal with class imbalance problems.
GOGGLE (Liu et al., 2023b) is a recently proposed synthetic tabular data generation model based on VAE. The primary motivation of GOGGLE is that the complicated relationships between different columns are hardly exploited by previous literature. Therefore, it proposes to learn graph adjacency matrix to model the dependency relationships between different columns. The encoder and decoder of the VAE model are both implemented as Graph Neural Networks (GNNs), and the graph adjacent matrix is jointly optimized with the GNNs parameters.
GReaT (Borisov et al., 2023) treats a row of tabular data as a sentence and applies the Auto-regressive GPT model learn the sentence-level row distributions. GReaT involves well-designed serialization process to transform row into a natural language sentence of a specific format and a corresponding deserialization process to transform sentence back to the table format. To ensure the permutation invariant property of tabular data, GReaT shuffles each row several times before serialization.
STaSy (Kim al., 2023) is recent diffusion-basee model for synthetic tabular data generation. STaSy treats the one-hot encoding of categorical columns as continuous features, which are then processed together with the numerical columns. STaSy adopts the VP/VE SDEs from Song et al. (2021b) as the diffusion process to learn the distribution of tabular data. STaSy further proposes several training strategies, including self- celf-paced learning and fine-tuning, to stabilize the training process, increasing sampling quality and diversity.
CoDi (Lee et al., 2023) proposes to utilize two diffusion models for numerical and cate- gorical columns, respectively. For numerical columns, it uses the DDPM (Ho et al., 2020) model with Gaussian noises. For categorical columns, it uses the multinominal diffusion model (Hoogeboom et al., 2021) with categorical noises. The two diffusion processes are inter-conditioned each other to model the joint distribution of numerical and categorical columns. In addition, CoDi adopts contrastive learning methods to further bind the two diffusion methods.
TabDDPM (Kotelnikov et al., 2023). Like CoDi, TabDDPM introduces two diffusion processes: DDPM with Gaussian noises for numerical columns and multinominal diffusion with categorical noises for categorical columns. Unlike CoDi, which introduces many additional techniques, such as co-evolved learning via inter-conditioning and contrastive learning, TabDDPM concatenates the numerical and categorical features input and output of the denoising function (an MLP). Despite its simplicity, our experiments have shown that TabDDPM performs even better than CoDi.
We further compare the properties of these baseline methods and the proposed TABSYN Table The compared properties include: 1) Compatibility: if the method can deal with mixed-type data columns, e.g., numerical and categorical. 2) Robustness: the method has stable performance across
hhttps://archi 'https://archivr iss usi wwtpe//aaaeeee//////// http:////...oonnneeeeeeeeeeeeeee inennewsrpgpllarity
19

---

Published as a conference paper at ICLR 2024
different datasets (measured by the standard deviation of the scores < 10% not) different datasets (from Table and Table 2). 3) Quality: Whether the synthetic data can pass column- wise Chi- Chi-Squared Test (p 0.95). 4) Efficiency: Each method can generate synthetic tabular data of satisfying quality within less than 20 steps.
Table 7: A comparison the properties of different generative models for tabular data. Base model denotes the base generative model type: Generative Adversarial Networks (GAN), Variational AutoEncoders (VAE), Auto- Regresseeeeevv Language Models (AR), and Diffusion.
Method
Base Model
Compatibility
Robustness
Quality
Efficiency
CTGAN
GAN
TVAE
VAE
x
GOGGLE
VAE
X
X
GReaT
AR
V
X
X
STaSy
Diffusion
X
V
X
X
CoDi
Diffusion
X
TabDDPM
Diffusion
TABSYN
Diffusion
E.3 METRICS OF LOW-ORDER STATISTICS
In this section, we give detailed introduction of the metrics used in Sec. 4.2.
E.3.1 COLUMN-WISE DENSITY ESTIMATION
Kolmogorov-Sirnov Test (KST): Given two (continuous) distributions (x) and P. (x) (r denotes real and s denotes synthetic), KST quantifies distance between the two distributions using the upper bound of the discrepancy between two corresponding Cumulative Distribution Functions (CDFs);
KST = =sup Sup|F.CC (z) Fs(x)),
(33)
where (x) and F2(x) are the CDFs of pr (x) and ps (x). respectively:
F(x) = I p(z)dz
(34)
Total Variation Distance (TVD): TVD computes the frequency cach category value and expresses it probability. Then, the TVD score is the average difference between the probabilities of the categories:
TVD
Σ R(w) S(w)). wEn
(35)
2
where describes all possible categories in column Ω. R(-) and S(-) denotes the real and synthetic frequencies these categories.
E.3.2 PAIR-WIST WISE COLUMN CORRELATION
Pearson Correlation Coefficient: The Pearson correlation coefficient measures whether two continu- ous distributions are linearly correlated and is computed as:
Cov (2)
Px.y
(36)
T2Øy
where x and y are two continuous columns. Cov is the covariance, and is the standard deviation.
Then, the performance of correlation estimation is measured by the average differences between the real data's correlations and the synthetic data's corrections:
Pearson Score
= Ex (xc v) - -p8 v).
(37)
2
20

---

Published as a conference paper at ICLR 2024
where (z, 3) and p² (z. y)) denotes the Pearson correlation coefficient between column and column of the real data and synthetic data, respectively. As E [-1, 1]. the average score is divided by 2 to ensure that it falls in the range of [0. 1], then the smaller the score, the better the estimation.
Contingency similarity: For a pair of categorical columns A and B, the contingency similarity score computes the difference between the contingency tables using the Total Variation Distance. The process is summarized by the formula below:
Contingency Score
ΣΣ GEABEB
RR. Sa,9ls
(38)
2
where a and describe all the possible categories in column A and column B, respectively. Ra.s and Sa,3 are the joint frequency of a and [3 in the real data and synthetic data, respectively.
E.4 DETAILS OF MACHINE LEARNING EFFICIENCY EXPERIMENTS
As preliminarily illustrated in Sec. 4.3 and Appendix E. each dataset is first split into the real training and testing set. The generative models are learned on the real training set. After the models are learned, a synthetic set of equivalent size is sampled.
The performance synthetic data on MLE tasks is evaluated based on the divergence of test scores using the real and synthetic training data. Therefore, we first train the machine learning model on the real training set, split into training and validation sets with 8 ratio. The classifier/regressor is trained on the training mance on the validation set. After the optimal hyperparameter setting is obtained, the corresponding classifier/rerressor is retrained on the training and evaluated the real testing We create 20 random splits for training and validation sets, and the performance reported in Table 3 the mean and std of the AUC/RMSE score over the 20 random trails. The performance of synthetic data is obtained in the same way.
and the optimal hyperparameter setting is selected according the perfor-
Below is the hyperparameter search space of the XGBoost Clasifferr/Regressor used in MLE tasks, and we select the optimal hyperparameters via grid search.
Number of estimators: [10, 50, 100]
Minimum child weight: [5, 10, 20).
Maximum tree depth: [1,10].
Gamma: [0.0, 1.0].
We the implementations of these metrics from SDMetric
F ADDITION EXPERIMENTAL RESULTS
In this section, we compare the training and sampling time of different methods, taking the Adult dataset example.
F.1 TRAINING SAMPLING TIME
As shown in Fig. though having additional VAE training process, the proposed TABSYN has a similar training time to most of the baseline methods. In regard to the sampling time, TABSYN requires merely 1.784s generate synthetic data of the same size Adult's training data, which is close to the one-step sampling methods CTGAN and TVAE. Other diffusion- methods take much longer time for sampling. E.g., the most competitive method TabDDPM (Kotelnikov et al.. 2023) takes 28.92s for sampling. The proposed TABSyN reduces the sampling time by 93%, and achieves even better synthesis quality.
"https://docs sdv. adv.dev//eeeaaee...d scmetrics
21

---

Published as a conference paper at ICLR 2024
Table 8: Comparison of training and sampling time of different methods, on Adult dataset. TABSYN'S training time is the summation of VAE's and Diffusion's 's training time.
Method
Training
Sampling
CTGAN
1029.8s
0.8621s
TVAE
352.6s
0,5118s
GOGGLE
1h 34min
5.342s
GReaT
2h 27min
2min 19s
STaSy
2283s
8.941s
CoDi
2h 56min
4.616s
TabDDPM
1031s
28.92s
TABSYN
1758s 664s
1.784s
F.2 SAMPLE-WIS QUALITY SCORE OF SYNTHETIC DATA (a-PRECISON AND B-RECALL)
Experiments in Sec. 4 have evaluated the performance of synthetic data generated from different models using low-order statistics, including the column-wise density estimation (Table 1) and pair- wise column correlation estimation (Table 2). However, these results are insufficient to evaluate synthetic data's ' overall density estimation performance, the generative model may simply learn to estimate the density of each single column individually rather than the joint probability of all columns. Furthermore, the MLE tasks also cannot reflect the overall density estimation performance since some unimportant columns might be overlooked. Therefore, in this section, we adopt higher-order metries that focus more on the entire data distribution, i.e., the joint distribution of all the columns.
Following Liu et al. (2023b) and Alaa et al. (2022), we adopt the a-Precision and 8-Recall proposed in Alaa al. (2022), two sample-level metric quantifying how faithful the synthetic data is. In general, a-Precision evaluates the fidelity of synthetic data whether each synthetic example comes from the real-data distribution, 3-Recall evaluates the coverage of the synthetic data, e.g., whether the synthetic data can cover the entire distribution of the real data (In other words, whether a real data sample close to the synthetic data). Liu al. (2023b) also adopts the third metric, Authenticity - whether the synthetic sample is generated randomly or simply copied from the real data. However, we found that authenticity score and beta-recall exhibit a predominantly negative correlation - their sum is nearly constant, and improvement in beta-recall is typically accompanied by decrease in authenticity score (we believe this is the reason for the relatively small differences in quality scores among the various models in GOGGLE (Liu et al., 2023b)). Therefore, we believe that beta-recall and authenticity are not suitable for simultaneous use.
In Table 9 and Table 10 report the scores of a-Precision and 3-Recall, respectively. TABSYN obtains the best a-Precision scores on all the datasets, demonstrating the superior capacity of TABSYN in generating synthetic data that close to the real ones. In Table 10, we observe that TabSyn consistently achieves high B-Recall scores across six datasets. Although some baseline methods obtain higher 3-recall scores on specific datasets, it can hardly be concluded that the synthetic data generated by these methods are of better quality since 1) their synthetic data has poor a- a-Precision scores (e.g., GReaT on Adult, and STaSy on Magic), indicating that the synthetic data far from the real data's 's manifold; 2) they fail to demonstrate stably competitive performance on other datasets (e.g., GReaT has high (e-ecall scores on Adult but poor scores on Magic). We helieve that to assess the quality of generation, the first consideration is whether the generated data is sufficiently authentic (a- (a-Precision), and the second is whether the generated data can cover all the modes the real dataset (B-Recall). According to this criterion, the quality of data generated by TabSyn is the highest. It not only possesses the highest fidelity score but also consistently demonstrates remarkably high coverage on every dataset.
F.3 DETECTION: CLASSIFIER Two SAMPLE TESTS (C2ST)
We further study how difficult it is to tell apart the real data from the synthetic data, therefore evaluating if the synthetic data can recover the real data distribution. To this end, we employ the
22

---

Published as a conference paper at ICLR 2024
Table 9: Comparison of a-Preeision scores. Bold Face represents the best score each dataset. Higher values indicate superior results. TABSYN outperforms all other baseline methods on all datasets.
Methods
Adult
Default
Shoppers
Magic
Beijing
News
Average
Ranking
CTGAN
7
62 08
76.97=0.30
86, 20.990000..
96. 20.14
96.90 06.96=0.17
82 82
TVAE
98.17a0.17
85.57
58.1920 26
19 48
97.20go.10
86.41
GOGGLE
50.68
68.89
86.95
90.88
88.81
86.41
78.77
GReaT
55. 36.79+0.03
85, 16.90=0.17
78.88+0.15
86, 06.554
98. 2..2222
80.5 80.87
STaSy
=2.87=0.26
90.48
89.65 as
86.56=0.19
89.1620.11
94.76
CoDi
17.58
82.3810.15
94.95.10
85.01
18.1311..
15.00.
87.03
TabDDPM
90.36++0.20
97.59
86.55+0.55
98.59
97.93
0.00
79.83
7
TABSYN
99.52.0.10
99.26.10.25
99.16.1..2
99. 1..27
98.47.1011
96.80.0.
98.67
Table 10: Comparison of 3-Recall scores. Bold Face represents the best score on each dataset. Higher values indicate superior results. TABSYN gives consistently high 3-recall scores, indicating that the synthetic data covers wide range of the real distribution. Although some baseline methods obtain higher scores on specific datasets, they fail to demonstrate stably competitive performance on other datasets.
Methods
Adult
Default
Shoppers
Magic
Beijing
News
Average
Ranking
CTGAN
30. a0.820
18
31.80=0.500
11. 2.
24.97
25.39
TVAE
23.13=a..
19.78.0.10
32.44=0.a
28.45=2.08
29.66.2.2
28.72
GOGGLE
8.80
14.38
9.79
9.88
19.87
2.03
10.79
8
GReaT
49.12
19.04.t..1
44.90s0.17
34.91
43.34
43.34
STaSy
20.21+0.84
39.3
19.33+...0
37.24+0.45
53. 03.977
54. 71.79.
39. 4242222.. 32
42.32
CoDi
9.20go.15
19.94
20.8210 as
50.5619931
52.1920.19
34.40go.3
31.19
TabDDPM
47.05:20.25
27.83
47.79 25
48.4620.4
56. 46.92:10.1
0.00
41.34
TABSYN
47.562n.99
48.00 as
48.95.1..s
48.03
35.8420.18
45.0410 a4
48.90
detection metric provided by sdmetrics 9, In Table 11, present the results obtained using logistic regression as the detection method. As indicated in the table, the Detection score exhibits superior
Table 11: Detection score (C2ST) using logistic regression classifier. performance.
Higher scores indicate better
Method
Adult
Default
Shoppers
Magic
Beijing
News
SMOTE
0.9710
0.9274
0.9086
0.9961
0.9888
0.9344
CTGAN
0.5949
0.4875
0.7488
0.6728
0.7531
0.6947
TVAE
0.6315
0.6547
0,2962
0.7706
0.8659
0.4076
GOGGLE
0.1114
0.5163
0,1418
0.9526
0.4779
0.0745
GReaT
0.5376
0.4710
0.4285
0.4326
0.6893
STaSy
0.4054
0.6814
0.5482
0.6939
0.7922
0.5287
CoDi
0.2077
0.4595
0.2784
0.7206
0.7177
0.0201
TabDDPM
0.9755
0.9712
0.8349
0.9998
0.9513
0.0002
TABSYN
0.9986
0.9870
0.9740
0.9732
0.9603
0.9749
discriminative power compared to other metrics, such as single-column density estimation, pair-wise column shape estimation, and MLE. The detection score shows significant variations across different models for synthetic data generation. As indicated in the table, the Detection score exhibits superior discriminative power compared to other metrics, such as single-column density estimation, pair-wise column shape estimation, and MLE. The detection score shows significant variations across different models for synthetic data generation. The proposed TABSYN consistently achieves notably high scores across all datasets (SMOTE (Chawla al., 2002) directly interpolates within the training set, so it is not surprising that it achieves high scores in the detection metric.).
httpss: pettps://et ://occs sdv. adv.dev/ http://mmerriii cs /wt.r/eriiirr. centrrics/eee ctc.c/////iseee in-bett/ detect on- onn-innle--tabl
23

---

Published as a conference paper at ICLR 2024
F.4 MISSING VALUE IMPUTATION
Adapting TABSYN for missing value imputation An essential advantage of the Diffusion Model is that unconditional model can be directly used for missing data imputation tasks (e.g., image inpainting (Song et al., 2021b; Lugmayr et al., 2022) and missing value imputation) without retraining. Following the inpainting methods proposed in Lugmayr al. (2022), we apply the proposed TABSYN in Missing Value Imputation Tasks.
For row of tabular data z; - [ 2h i.M. mmd E RIxMMam 2 E Ix Assume the index set of masked numerical columns is mmum- and of masked categorical columns is Meats We first preprocess the masked columns as follows:
The value of a masked numerical column is set as the averaged value of this column, i.e., mumm mean(z.. (m Vj 7mmmm bg
The masked categorical column is set 2 lo,,.. ".,.. T RRII, vj Meat-
The updated z; (we omit the subscript in the remaining parts) fed to TABSYN's frozen VAE's encoder to obtain the embedding E IR IxMM architecture, there deterministic mapping from the masked indexes in the data space Muum and Meat to the masked indexes in the latent space. For example, the first dimension of the numerical column is mapped to dimension
Mdd As TABSYN's VAE employs the Transformer
in z. Therefore, we can create a masking vector m indicating whether each dimension is masked. Then, the known and unknown part of could be expressed as m x and (1 m) O x, respectively.
Following Lugmayr et al. (2022), the reverse step is modified as mixture of the known part's forwarding and the unknown part's denoising:
known 44-1
otti-t)) NOO,I),
1/--
2/,-
unknown
21: + ct
dž
(39)
known + (1 m.) ( uuknown
21,11 m 2
2
where dz; -((()((1)V logp(z,)dt / (f(l)
The reverse imputation from time t; f;-1 also involves resampling in order to reduce the error brought by each step (Lugmayr et al., 2022). Resampling indicates that Eq. 39 will be repeated for U steps from 2,, to ,,,-,- After completing the reverse steps, obtain the imputed latent vector 2o. which could be put into TABSYN's VAE decoder to recover the original input data.
The algorithm for missing value imputation is presented in Algorithm 4.
24

---

Published as a conference paper at ICLR 2024
Algorithm 4: TABSYN for Missing Value Imputation
1. VAE encoding
2: z [ _21" .. Maas is data sample having missing values.
3: Mmum denotes the missing numerical columns.
4: Meat denotes the missing categorical columns.
5: mum
num) i.j = mean 2.
Vj E Moum-
6: iJ
= C.
d,) EleBi RRRi Vj E Meat.
7: 2 = Flatten (Encoder()) E RIXMd
8:
9: 2. Obtaining the masking vector
Rixasd
11: for j =
do
if ([j/d]) E Mumm or (lj/d] mummm Meat then
13: m, =
14: else
15: m =1
16: end if
17: end for
18:
19: 3. Missing Value Imputation via Denoising
20: 2tmax = x + 0 (fmax) (O,
for max,
do
22: for u u.
do
known 2 E
24:
21:
uunknown 2r, + +
+ dzı,
25:
dz is defined in Eq. 6
26:
_t._, m 3 2 known + (1 - m) 3 t
unknown
27:
if < and >> then
28: ### Resampling
21 - + --
30: end if
end for
32: end for
20 2to
34:
35: 4. VAE decoding
36: & Eerodee(Unflaaten//
37: & is the imputed data
Dassification/Regresssio missing value imputation. With Algorithm 4, able to TABSYN for imputation with any missing columns. this section, we consider a more interesting ap- plication treating clasiffcation/reeressoon as missing value imputation tasks directly. As illustrated in Section 1, each dataset is assigned a machine learning task, either a classification or regression on the target column in the dataset. Therefore, we can mask the values of the target columns, then apply TABSYN to impute the masked values, completing the classification or regression tasks.
In Table 12, we present TABSYN's performance in missing value imputation tasks on the target column of each dataset. The performance is compared with directly training a classifier/regressor using the remaining columns to predict the target column (the 'Real' row Machine Learning Efficiency tasks, Table 3). Surprisingly, imputing with TabSyn shows highly competitive results on all datasets. On four six datasets, TabSyn outperforms training a discriminate ML classifier/regressorr on real data. We suspect that the reason for this phenomenon may be that discriminative ML models are more prone to overfitting on the training set. In contrast, by learning the smooth distribution of the entire data, generative models significantly reduce the risk of overfitting. The excellent results on the missing value imputation task further highlight the significant importance of our proposed TabSyn for real-world applications.
25

---

Published as a conference paper at ICLR 2024
Our TABSYN is not trained conditionally other columns for the missing value imputation tasks, and the performance can be further improved by training a separate conditional model specifically for this task. We leave it for future work.
Table 12:
Performance of YN in Missing Value Imputation tasks, compared with training an XGBoost classifier/regressorr using the real data.
Adult
Default
Shoppers
Magic
Beijing
News
Methods
1
AUC
1
AUC
RMSE :
RMSE +
Real with XGBoost
92.7
77.0
92.6
94.6
0.423
0.842
Impute with TABSYN
93.2
87.2
96.6
88.8
0.258
1.253
F.5 IMPACTS OF THE QUALITY OF VAES
Intuitively, the performance of TabSyn appears to be highly dependent on the quality of the pre- trained VAE model. Therefore, we conduct further research to investigate how a less trained VAE model would impact the quality of synthetic data generated by TabSyn. To this end, we investigate the quality of synthetic data generated by TabSyn using the embeddings of the VAE obtained at different epochs as the latent space. Figure 9 plots the results of single-column density estimation and pair-wise column correlation estimation on the Adult and Default datasets, with intervals set at 400 epochs. We can observe that increasing the training epochs of the VAE does indeed improve the quality of TabSyn's 's generated data. Additionally, even when the VAE is sub-optimal (e.g., training epochs around 2000), TabSyn's 's performance is already very close to the optimal ones. Furthermore, even with a relatively low number of VAE training epochs (e.g.. 800-1 (00),, TabSyn's performance approaches even surpasses the most competitive baseline, TabDDPM. Based on this observation, recommend thoroughly training the VAE to achieve superior data generation quality when resources are abundant. However, when resources are limited, reducing the VAE training duration still yields decent performance.
Adult
Default
Error rate (%)
Error rate (column) Error rate (pair)
Error rate (%)
6
Error rate (column)
Error rate (pair)
1
1000
2000
3000
4000
1000
2000
3000
4000
Training epochs
Training epochs
Figure 9: Performance of TABSYN with VAEs trained for different epochs. By default, TABSYN's VAE is trained with 4000 epochs
F.6 PRIVACY PROTECTION: DISTANCE TO CLOSEST RECORD (DCR)
To study if the synthetic data will cause privacy information leakage issues, compute the DCRs of the synthetic data. Specifically, we follow the 'synthetic vs. holdout' setting 10. We initially divide the dataset into two equal parts: the first part served as the training set for training our generative model, while the second part was designated as the holdout set, which is not used for training. After completing model training, we sample synthetic set of the same size as the training set (and the holdout set).
1tttps: h/ww:////wwww///// clwcclaaaaaeoc htt.://ww..aaaa.aa///////0-----e preservation-pate
26

---

Published as a conference paper at ICLR 2024
We then calculate the DCR scores for each sample in the synthetic set concerning both the training set and the holdout set. We can create histograms to visualize the distribution of DCR scores for the synthetic set in comparison to both the training and holdout sets. Intuitively, if there is privacy issue (e.g. if the synthetic set is directly copied from the training set), then the DCR scores for the training set should be closer 0 than those for the testing set. Conversely, if there is no privacy issue, the distributions of DCR scores of the training and holdout sets should largely overlap. In Figure 10, we plot the distributions of synthetic sets' DCRs concerning the training set and holdout set on Default and Shoppers. We can observe that deep generative models such as CoDi, STaSy, TabDDPM, and TabSyn do not suffer from privacy issues, while the interpolation- based method SMOTE might not be able to protect privacy information.
defoat bileye
8000
$0.00
4000
40:00
20:0
4000
1000
DCR
DCR
3CR
ovsggers
decppess
Ovsgners
ducipers
5500
6.00
15:00
2005
100
Figure 10: Distributions of the DCR scores between the synthetic dataset and the training/holdout datasets. Deep generative models have similar DCR distributions concerning the training set and holdout set, while in the interpolation- based method SMOTE, DCRs concerning the training set are much smaller than DCRs concerning the holdout set.
Additionally, we calculate the probability that a synthetic sample is closer to the training set (rather than the holdout set). When this probability is close to 50% (i.e., 0.5), it indicates that the distribution of distances between synthetic and training instances is very similar (or at least not systematically smaller) than the distribution of distances between synthetic and holdout instances, which is a positive indicator in terms of privacy risk. Table 13 displays the results obtained by different models on Default and Shoppers datasets.
Table 13: DCR score (the probability that a synthetic example is closer to the training set rather than the holdout set (%. a score closer to 50% is better).
Method
Default
Shoppers
SMOTE
91.41 % 42
96.40%
STaSy 30.23% 09
51.53%:20.1o
CoDi
51.82%c2..26
51.06%%=+o.18
TabDDPM
52.15%.20.20
63. 63.23%6-0..26
TABSYN
51.20%22.11
52.90% 1:0.2
G DETAILS FOR REPRODUCTION
In this section, we introduce the details of TABSYN, such as the data preprocessing, training, and hyperparameter details. We also present details of our reproduction for the baseline methods.
G.1 DETAILS OF IMPLEMENTING TABSYN
Data Preprocessing. Real-world tabular data often contain missing values, and each column's 's data may have distinct scales. Therefore, we need to preprocess the data. Following TabDDPM (Kotel- nikov et al., 2023), missing cells are filled with the column's average for numerical columns. For
27

---

Published as a conference paper at ICLR 2024
categorical columns, missing cells are treated one additional category. Then, each numerical/cate- gorical column transformed using the Quantile Transformer" Transforme""/OneHotttie /OOneHotEnodder" from respectively.
Hyperparameter settings. TABSYN uses the same set of parameters for different datasets. (except Smax for Shoppers). The detailed architectures VAE and Diffusion model of TABSYN have been presented in Appendix D.1 and Appendix D.2, respectively. Below is the detailed hyperparameter setting.
Hyperparameters of VAE:
Token dimension d 4,
Number of Layers of VAEs' encoder/decoder: 2,
Hidden dimension of Transformer's FFN: 4 32 128,
mmax 0.01 (8max 0.001 for Shoppers),
Amin 10-5,
A 0.7.
Hyperparameters of Diffusion:
MLP's hidden dimension dhidden = 1024.
Unlike the cumbersome hyperparameter search process in some current methods (Kotelnikov et al., 2023; Kim et al., 2023; Lee et al., 2023) to obtain the optimal hyperparameters, TABSyN consistently generates high-quality data without the need for meticulous hyperparameter tuning.
G.2 DETAILS OF IMPLEMENTING BASELINES
Since different methods have adopted distinct neural network architectures, is inappropriate to compare the performance of different methods using identical networks. For fair comparison, adjust the hidden dimensions different methods, ensuring that the numbers trainable parameters are close (around 10 million). Note that enlarging the model size does lead better performance for the baseline methods. Under these conditions, we reproduced the baseline methods based on the official codes, and our reproduced codes are provided in the supplementary. Below are the detailed implementations of the baselines.
CTGAN and TVAE (Xu al., 2019): For CTGAN, we follow the implementations in the official codes"! 13 where the hyperparameters are well-given. Since the default discriminnator/geeeaato MLPs are small, we enlarge them to the same size as TABSYN for fair comparison. The interface for TVAE is not provided, so we simply use the default hyperparameters defined in the TVAE module. The sizes TVAE's encoder/decoder are enlarged as well.
GOGGLE (Liu et al., 2023b): We follow the official implementations In GOGGLE's official implementation, each node a column, and the node feature is the 1-dimensional numerical value of this column. However, GOGGLE did not illustrate and failed to explain how handle categorical columns iS To extend GOGGLE to mixed-type tabular data, we first transform each categorical column into its C-dimensional one-hot encoding. Then, each single dimension of 0/11 binary values becomes the graph node. Consequently, for a mixed-type tabular data Muam numerical columns and Meat categorical columns and the i-th categorical column of C; categories, GOGGLE's graph has M + > C; nodes.
appprceessiggguuuaanieeerrrrnnoo html
"httee://attubb.comm/ddddddaaaaa
28

---

Published as a conference paper at ICLR 2024
We follow the official implementations" 16 During reproduction, found that the training of GReaT memory and mmeeccosumiig (because it is fine-tuning a large language model). Typically, the batch size is limited to 32 on the Adult dataset, and training for 200 epochs takes over 2 hours. In addition, since GRea'T is textual-based, the generated content is not guaranteed to follow the format of the given table. Therefore, additional post-processing has to be applied.
GReaT:
STaSy (Kim et al., 2023): In STaSy, the categorical columns are encoded with one-hot encoding and then put into the continuous diffusion model together with the numerical columns. We follow the default hyperparameters given by the official codes 17 except for the denoising function's size, which enlarged for fair comparison.
CoDi (Lee et al., 2023): We follow the default hyperparameters given by the official codes" 18 Similarly, the denoising U-Nets used by CoDi also enlarged to ensure similar model parameters.
TabDDPM (Kotelnikov et al., 2023): The official code of TabDDPM 19 is for conditional generation tasks, where the non-target columns are generated conditioned on the target column(s). We slightly modify the code to be applied to unconditional generation.
/main
29


Reviews:
[{"review_id": "cBCAr9WBf2", "reviewer": "Reviewer_q7Xg", "summary": "This work introduces a latent diffusion model for generating tabular data, and presents a benchmark consisting of six datasets and five quality metrics to evaluate the performance. The comparison in this unified testing environment demonstrates the superiority of the proposed method.", "questions": "1.\tAre the results shown in Figure 3 derived from the training set or the validation set?\n\n2.\tDoes replacing the MLP in the diffusion model with a more powerful architecture, such as a Transformer, have any impact on the performance?", "limitations": "", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "•\tThis paper presents a benchmark that is beneficial to the community.\n\n•\tThe better performance over previous work across five quality metrics showcases its effectiveness in generating high-quality tabular data.", "weaknesses": "1.\tThe motivation behind using latent diffusion for tabular data generation is not thoroughly discussed in the paper, and the model design does not effectively exploit the characteristics of tabular data.\n\n2.\tThe VAE decoder design is tailored specifically for either numerical or categorical features, which limits its applicability in a wider range of tabular data scenarios, such as datasets containing a mixture of both numerical and categorical features.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "QHmeOXRpN8", "reviewer": "Reviewer_Fs1y", "summary": "Extending diffusion models to handle tabular data presents challenges due to the complex distributions and diverse data types inherent to such data. To address this, the authors introduce the use of a Variational Autoencoder (VAE) to learn a regularized latent embedding representation of the data, which is subsequently processed by a diffusion network for synthesis. Notably, the study employs a comprehensive set of multi-dimensional evaluation metrics for the generated data, filling a gap often observed in previous research. The proposed method excels across these metrics, underscoring its efficacy in generating synthetic data that closely mirrors the original data distribution.", "questions": "Thank you for sharing your code with the community; it's a valuable resource. While exploring it, I encountered a few queries and points of feedback:\n\n1. **Device Attribute Error**: When executing the command `python main.py —dataname adult —method vae —mode train`, I came across the “AttributeError: ’Namespace’ object has no attribute ‘device’”. I was able to address this by introducing an else statement post line 7 in `main.py` to default to 'cpu'. Consider incorporating this for broader compatibility.\n   ```python\n   if …:\n       args.device = …\n   else:\n       args.device = 'cpu'\n   ```\n\n2. **Sample Size Limitation**: I attempted the VAE training phase with 40 samples and encountered an `IndexError: index out of range in self`. This wasn't an issue with the full sample size of 32561. Is the model designed to accommodate only larger samples, or is there a potential to adapt it to smaller sample sizes?\n\n3. **Epoch Setting for VAE Model**: The default epoch for the VAE model in the code is set to 4000. Based on my prior experiences with the TVAE model using CTGAN's code, training for around 300 epochs usually suffices. Is the transformer architecture inherently more demanding in terms of training duration? Additionally, what criteria do you rely on to determine the termination of training? Introducing an early stopping mechanism might be beneficial, especially considering the subsequent training phase for the diffusion model. It's also noteworthy that a Train/Val accuracy of 100% seems achievable by the 1000th epoch.", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 4, "strengths": "- The method effectively manages mixed-type data by transforming them into a single cohesive space to ensure capturing or inter-column relationships.\n\n- Compared to existing diffusion-based methods, this method requires fewer reverse steps and offers faster data synthesis.\n\n- The authors have provided a unified comparison environment for their proposed tabular data synthesis, as well as all the compared baseline methods, and made their code base publicly available.\n\n- The study employs a diverse set of multi-dimensional evaluation metrics for a holistic assessment of the generated data, addressing a common shortcoming in previous research.\n\n- The method has been rigorously tested on six datasets using five metrics, and it consistently outperforming other existing methods, indicating its prowess in generating synthetic data that closely reflects the original data distribution.", "weaknesses": "- The method's efficacy is contingent upon a well-trained VAE. It would be beneficial to compare the outcomes between optimally and sub-optimally trained VAEs, providing insights into worst-case vs. best-case scenarios.\n\n- Given the generative capability of VAEs, it would be insightful to see results from data generated solely by the VAE used in this study. The distinction between the paper's transformer-based VAE and TVAE warrants further exploration to determine the independent efficacy of the former.\n\n- While adjusting default hyperparameters for a fair comparison is commendable, understanding performance under default settings across consistent training epochs would give a fuller picture. This would ascertain whether hyperparameter enlargement (as done for CTGAN and CoDi) equally benefits the models or favors the presented method disproportionately.\n\n- The discrepancy observed where TabDDPM struggles with the News dataset (poor performance in Table 1), yet exhibits a low error rate in Table 2 seems unintuitive. Additionally, given TabDDPM's consistent second-place ranking, except for the News dataset, its fourth-place average rank seems unfair. An alternative could be per-dataset ranking or reporting modal / averaged ranks. \n\n- Given that the News dataset is primarily of numeric nature, it seems counterintuitive that TabDDPM, a diffusion based model would underperform on this dataset. It would be beneficial to understand the authors' rationale behind the model's inability to generate meaningful content for this dataset.\n\n- The deployment of MLE as a metric for privacy is unconventional. Traditionally, MLE assesses the synthetic data's task-performance equivalence to real data, not privacy leakage. It would be enriching if the authors could shed light on this choice.\n\n\nOverall, this paper stands out for its meticulous code, articulate presentation, and thorough analysis. I commend the authors for their contribution.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Gl3Ts1thfD", "reviewer": "Reviewer_aVor", "summary": "Authors propose a generative model for mixed type tabular data. The proposed model first tokenizes the mixed type columns, feeds it to a one transformer layer, which then forms as the encoder in the VAE model. Finally the latent space is fixed by diffusion.", "questions": "- what would be the accuracy in the downstream task if latent code would be used directly (and no synthetic data). I understand that this is not possible for all models. But for the models that it is possible it would be interesting to see how much benefit there is (i.e. can you win real)\n- Are all classification tasks in downstream binary tasks? If no, then AUC is not a correct metric.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 2, "strengths": "Very reasonable model for the mixed type tabular data. Definitely something that I would use in my day to day work. Results are also convincing.", "weaknesses": "- Model itself seems to be pretty much the same as Vahdat 2021, except that in that paper authors used only images, whereas now tokenization is needed to use the same model. I would like authors to comment on this, and it would really help the paper to be very clear in the Introduction that where the technical novelty lies.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "FgHlqLDK1D", "reviewer": "Reviewer_dVGm", "summary": "This paper present a new latent diffusion model/code for tabular data generation.\nTransposing to tabular data the recent ideas of (Rombach et al. CVPR 2022 and Karras et al. NeuIPS 2022), their model architecture is two-folds:\n- a transformer-based \\beta-VAE to embedd tabular data into a latent space\n- a score-based generator based on (Song et al. ICLR 2021)'s architecture\n\nAs a slight algorithmic contributions, the authors propose to use an \"adaptive VAE loss weighing\".\n\nIn the experiment section, the method is benchmarked against 6 state of the art tabular data generation models on 6 datasets. A few ablation tests are provided (one to justify the adaptive weighing).\nThis paper is only focused on unconditional generation, some experiments on missing-values imputation are also provided.\n\nIt is worth noting that both the code and a rich appendix are provided as supplementary material.\nThe code is clear and well commented.\nThe appendix provides a clear background on recent score-based generation best-practices and several supplementary experiments. Several implementation and methodology details allows the readers to retrieve what they needs to reproduce and understand this work.", "questions": "- Could you use the \"sdmetrics\" library to provide some privacy (like DCR) and detection (like C2ST) metrics in your benchmark ?\n- A few more datasets common with previous papers like (Kotelnikov et al. 2022, see Table 2) could improve the benchmark.\n- Could you add SMOTE (with unconditional sampling) as a baseline in your results ?\n- Are the confidence intervals on result tables computed through cross validation or only through multiple-sampling ?\n\n- Will you publish your code ?\n- Did you try your code on 2d synthetic sklearn examples ?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "I suggest acceptance:\n\n- I really liked reading this paper. It is well written with several clear illustrations. \n- The contribution is mostly incremental but solid and well driven.\n- The provided code is clear and will be useful for the community (if it is published)\n- The supplementary material provides a detailed and clear background summary.\n- The experiment section could be improved but seems solid: the method seems efficient and fast when compared against other SOTA methods.\n- The model is tested with a single hyper-parameter configuration on all datasets", "weaknesses": "- The scientific contribution is mostly incremental and expected\n- The authors claim that no \"unified and comprehensive evaluation\" exists for tabular data synthesis. To my opinion, one weakness of this paper is indeed that it feeds this lack of a unified benchmark by proposing another new benchmark with new metrics that are not used in other papers. Sticking a bit more to previous paper's metrics and datasets could improve that point.\n- given the size of the appendix, one is surprised to see that no simple baselines like Bayesian Networks or SMOTE are provided in the experiments. SMOTE is known to be a competitive baseline for \"target-conditional\" data generation.\n- No privacy preservation metrics (like DCR) are provided. No detection test metric (like C2ST) is provided\n- The absence of hyper-parameters tuning in the benchmark is both laudable and questionable as it may hinder some of the other models (a fair option could be to report the total training time with a fixed budget).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (oral)