TabDDPM: Modelling Tabular Data with Diffusion Models
Authors: Akim Kotelnikov
Dmitry Baranchuk
Ivan Rubachev
Artem Babenko
Venue: Submitted to ICLR 2023 (2023)
Keywords: tabular data
diffusion models
generative modelling

Abstract: Denoising diffusion probabilistic models are currently becoming the leading paradigm of generative modeling for many important data modalities. Being the most prevalent in the computer vision community, diffusion models have also recently gained some attention for other domains, including speech, NLP, and graph-like data. In this work, we investigate if the framework of diffusion models can be advantageous for general tabular problems, where datapoints are typically represented by vectors of heterogeneous features. The inherent heterogeneity of tabular data makes it quite challenging for accurate modeling, since the individual features can be of completely different nature, i.e., some of them can be continuous and some of them can be discrete. To address such data types, we introduce TabDDPM --- a diffusion model that can be universally applied to any tabular dataset and handles any types of features. We extensively evaluate TabDDPM on a wide set of benchmarks and demonstrate its superiority over existing GAN/VAE alternatives, which is consistent with the advantage of diffusion models in other fields. Additionally, we show that TabDDPM can be successfully used in privacy-oriented setups, where the original datapoints cannot be shared.

Under review as a conference paper at ICLR 2023
TABDDPM: MODELLING TABULAR DATA WITH DIFFUSION MODELS
A Annnymous authors Paper under double-blind review
ABSTRACT
Denoising diffusion probabilistic models are currently becoming the leading paradigm generative modeling for many important data modalities. Being the most prevalent in the computer vision community, diffusion models have also recently gained some attention in other domains, including speech, NLP, and graph-like data. In this work, we investigate if the framework of diffusion mod- els can advantagcous for general tabular problems, where datapoints are typi- cally represented by vectors of heterogeneous features. The inherent heterogene- ity of tabular data makes it quite challenging for accurate modeling, since the individual features be of completely different nature, i.e., some of them be continuous and some of them can be discrete. To address such data types, diffusion model that can be universally applied to any tabular dataset and handles any type of feature. We extensively eval- uate TabDDPM on wide set of benchmarks and demonstrate its superiority over existing GAN/VAE alternatives, which is consistent with the advantage of diffusion models in other fields. Additionally, we show that TabDDPM is eli- gible for privacy-oriented setups, where the original datapoints cannot be pub- licly shared. The source code of TabDDPM and our experiments is available pettps://anoyynen 4oper. sc www.sscceeeeeeeeeeeeeeeee http:r/ttabbbbddbboee 2483
we introduce TabDDPM
INTRODUCTION
Denoising diffusion probabilistic models (DDPM) (Sohl-Dickstein et al. 2015 Ho et al. 2020)
have recently become an object of great research interest in the generative modelling community since they often outperform the alternative approaches both in terms of the realism of individual samples and their diversity (Dhariwal & Nichol 2021). The most impressive successes of DDPM were demonstrated in the domain of natural images (Dhariwal & Nichol, 2021; Saharia al. 2022 Rombach et al., 2022), where the advantages diffusion models are successfully exploited in ap- plications, such colorization (Song et al.] (2021). inpainting (Song al.] 22021) segmentation Baranchuk et al. (202T), super-resolution (Saharia et al. [021 Li et al.) (021), semantic editing (Meng et al. 2021 and others. Beyond computer vision, the DDPM framework is also investigated in other fields, such as NLP (Austin et al., [222]] Li et al. (002)). waveform signal processing (Kong al. 2020 Chen al. 2020). molecular graphs (Jing al., [2022] Hoogeboom al. 2022). time scries (Tashiro et al. 2021), testifying the universality of diflusion models across wide range of problems.
The aim of our work is to understand if the universality of DDPM can be extended to the case of general tabular problems, which are ubiquitous in various industrial applications that include data described by a set of heterogeneous features. For many such applications, the demand for high- quality generative models is especially acute because of the modern privacy regulations, like GDPR, which prevent publishing real user data, while the synthetic data produced by generative models can be shared. Training a high-quality model of tabular data, however, can be more challenging compared computer vision NLP due to the heterogeneity of individual features and relatively small sizes of typical tabular datasets. In our paper, we show that despite these two intricacies, the diffusion models successfully approximate typical distributions of tabular data, leading to state-of-the-aa performance on most of the benchmarks.
In more detail, the main contributions of work are the following:

---

Under review as a conference paper at ICLR 2023
1. We introduce TabDDPM- the simplest design of DDPM for tabular problems that can be applied to any tabular task and can work with mixed data, which includes both numerical and categorical features.
2. We demonstrate that TabDDPM outperforms the alternative approaches designed for tab- ular data, including GAN-based and VAE-based models from the literature, and illustrate the sources of this advantage for several datasets.
3. We show that data produced by TabDDPM appears to be a "sweet spot" for privacy- concerned scenarios when synthetics used to substitute the real user data that cannot be shared.
The souree code of TabDDPM is publicly available
RELATED WORK
Diffusion models (Sohl-Dickstein et al. 2013 Ho et al. 2020) are a paradigm of generative mod- elling that aims approximate the target distribution by the endpoint of the Markov chain, which starts from a given parametric distribution, typically a standard Gaussian. Each Markov step is per- formed by deep neural network that effectively learns to invert the diffusion process with known Gaussian kernel. Ho et al. demonstrated the equivalence of diffusion models and score matching (Song Ermon 2019] 2020). showing them to be two different perspectives on the gradual con- version of a simple known distribution into a target distribution via the iterative denoising process. Several recent works (Nichol 2021| Dhariwal
Nichol 202I) have developed more powerful model architectures as well as different advanced learning protocols, which led to the "victory" of DDPM over GANs terms of generative quality and diversity in the computer vision field. In our work, demonstrate that one can also successfully use diffusion models for tabular problems.
Generative models for tabular problems are currently an active research direction in the machine learning community (Xu et al.] 2019 Engelmann & Lessmann 2021 Jordon et al. 2018 Fan al. 2020 Torfi et al., 2022) Zhao et al. 2021] Kim et al. 2021: Zhang et al. 2021 Nock & Guillame- Bert 2022 Wen et al. 2022) since high-quality synthetic data is of large demand for many tabular tasks. First, the tabular datasets are often limited in size, unlike in vision or NLP problems, for which huge "extra" data is available on the Internet. Second, proper synthetic datasets do not con- tain actual user data, therefore they are not subject to GDPR-like regulations and can be publicly shared without violation of anonymity. The recent works have developed large number of models, including tabular VAEs (Xu et al. (2019) and GAN- approaches (Xu et al.) [2019] EEneelman & Lessmann 2021 Jordon al. [018]] Fan et al. 2020 Tort al. 2022: Zhao al. [202]] Kim et al.
2021 Zhang et al. 2021 Nock & Gui Guillame-Berl 2022 Wen et al. 2022). By extensive evaluations
on a large number of public benchmarks, we show that our TPbDDPM model surpasses the existing alternatives, often by large margin.
"Shallow" syntheties generation. Unlike unstructured images or natural texts, tabular data is typ- ically structured, i.c., the individual features often interpretable and it is not clear if their mod- elling requires several layers of "deep" architectures. Therefore, the simple interpolation techniques, like SMOTE (Chawla al. 2002) (originally proposed address class-imbalance) simple and powerful solutions as demonstrated in (Camino et al. (020), where SMOTE is shown to outperform tabular GANs for minor class oversampling. In the experiments, demonstrate the ad- vantage of synthetics produced by TabDDPM over synthetics produced by interpolation techniques from the privacy-preservie perspective.
serve as
3 BACKGROUND
Diffusion models (Sohl-Dickstein et al. 2015 Ho et al. 2020) are likelihood-based generative models that handle the data through forward and reverse Markov processes. The forward process q (x = II.-, ( gradually adds noise to an initial sample in from the data distribu- tion y (Ir) sampling noise from the predefined distributions y (II| with variances (Bi,... 3r)
URI.
2

---

Under review as a conference paper at ICLR 2023
The reverse diffusion process p (Zu:r) = (
(24-1(z1) gradually denoises latent variable Drr~g (xT) and allows generating new data samples from g(zo). Distributions p (xt-1 are usu- ally unknown and approximated by neural network with parameters 0. These parameters learned from the data by optimizing a variational lower bound:
T
q (zo) 2 Bg(x)) log pa (zo mi) KL (q (#T(20) (2T))
Σ
KL (2 wpe (at- (2x))
Le
LT
(1)
Gaussian diffusion models operate in continuous spaces (xt (R)) where forward and reverse processes are characterized by Gaussian distributions:
(xg| == N (z1: V1 1,11-1,8111 g (IT) == N (rT;
pe (-=)1 = N (mt-1; 140 (Ie, 1), Do (xr,())
Ho al. (2020) suggest using diagonal Eg (Iee t) with constant or and computing μá (rr, t) as function of I; and co(xr, t):
Be VI at
pi (Ia, 1) vo
It
co (12,1)
where ag := := - 3r ã₁ == II a and cg t) predicts a "groundtruth" noise component e for
the noisy data sample vt. In practice, the objective can be simplified to the sum of mean-squared errors between c0 t) and over all timesteps t:
I. simple E.g.,,, (x(, 1)
Multinomial diffusion models (Hoogeboom et al. 2021) are designed to generate categorical data where I₁ E {0, 1) is one-hot encoded categorical variable with K values. The multinomial forward diffusion process defines (Ir (xte) as categorical distribution that corrupts the data by uniform noise over K classes:
< (1+-1) 1) = Cat (Ir;; (1 Bx) xt-1 + BB/K))
q (IT) = Cal (vT; /K)
(Ir| Cat (Zei 0, + (1 ā.) /K)
From the equations above, the posterior g( [2r,, 20) can be derived:
q ( 12,,, Cat 11-1;
k=1
where π = [ag + (1 - K]] 3 1ā- (1 K)-.)K
The reverse distribution po (x,- 1 bxy) is parameterized as y (Ir-1 [ar, fo(x,, ()), where do is pre- dicted by neural network. Then, the model is trained to maximize the variational lower bound (I).
TABDDPM
In this section, we describe the design of TabDDPM well as its main hyperparameters, which affect the model's effectiveness.
TabDDPM uses the multinomial diffusion to model the categorical and binary features, and the Gaussian diffusion to model the numerical ones. In more detail, for tabular data sample
I = [ Teati ..., Pc that consists of Nnum numerical features 2pam E N.um and C categorical features Feat, with K; categories each, our model takes one-hot encoded versions of categorical features as an input (i.e. zoke E {0, 13 Ki) and normalized numerical features. There- fore, the input zo has dimensionality of (Naum K,). For preprocessing. the gaussian

---

Under review as a conference paper at ICLR 2023
0.8
Quartile
-3.0
Transfo rmerr
1.2
T7
morm
Feat
ohe Tt cxt
MLP
Lait1
5: ohe cxf1
Softmax
One-hot Uncoder
ohe 2' cat2
Loatz
5: ghe cat2
T. catg
Figure 1: TabDDPM scheme for classification problems: I, y and l denote diffusion timestep, class label, and logits, respectively.
quantile transformation from the seikit-learn library (Pedregosa al. 2011). Each categorical fea- ture is handled by separate forward diffusion process, i.c., the noise components for all features are sampled independently. The reverse diffusion step in TabDDPM is modelled by a multi-layer neural network that has output of the same dimensionality as zo, where the first Naum coordi- nates are the predictions of e for the Gaussian diffusion and the rest are the predictions of ohe for the multinomial diffusions.
The TabDDPM model for the classifica- tion problems is schematically presented on Figure T The model is trained by minimizing a sum of mean-squared error L, Lammle (Equation (2)) for the Gaussian diffusion term and the KI. divergences TA for each multinomial diffusion term (Equation (I)) The total loss of multino- mial diffusions is additionally divided by the number of categorical features.
Hyperparameter
Search space
Learning rate Batch size Diffusion timesteps Training iterations MLP layers
LogUniform [0. (0.00001, 0.003)
Cai (256, 4096)
Cai 1000]
C5t 10000, 20000}
Int 4, 6, 8}
MLP width of layers Proportion of samples
Int [128 256, 512, 1024) Float (0.25, 0.5, 1, 2, 4,8)
Dropout
0.0
Scheduler
cosine Nichol 2021 MSE
L TabDDDM = L. simple Eisc T
Gaussian diffusion loss
Number of tuning trials 50
For classification datasets, we use a class- conditional model, i.e. po p0( y) is learned. For regression datasets, we con- sider target value additional numer- ical feature, and the joint distribution is learned.
Table 1: The main hyperparameters of TabDDPM.
To model the reverse process, we use a simple MLP architecture adapted from GGorishniy et al. 2021
MZP(2) Linear (MLPBlock (... (MLPB1ock())))
MLPB1ock(2) = (rrpo//k)))___pppooeeaaaaaaaaaar
As in (Nichol 2021 Dhariwal & Nichol 202 are processed as follows.
tabular input Zmn timestep t, and class label
Lemb wtw.earssS10LL..eeerrrttttteneee
y_emb = Embedding(y)
(5)
Linear( t_emb + y_emb
where SinTime refers to sinusoidal time embedding in Nichol 2021 Dhariwal & Nichol 2021 with a dimension of 128. All Linear layers in Eqquaion have fixed projection dimension 128.
4

---

Under review as a conference paper at ICLR 2023
Abbr
Name
# Train # Validation
Test
# Num # Cat
Task type
AB
Abalone
2672
669
836
7
Regression Binclass
AD
Adult ROC
26048
6513
16281
6
BU
Buddy
12053
3014
3767
4
Multiclass
CA
California Housing
13209
3303
4128
8
Regression
CAR
Cardio
44800
11200
14000
5
Binclass
CH
Chum Modelling
6400
1600
2000
7
Binclass
DE
Default
19200
4800
6000
20
Binclass
DI
Diabetes
491
123
151
8
Binclass
FB
Facebook Comments Volume
157638
19722
19720
50
Regression
GE
Gesture Phase
6318
1580
1975
32
Multiclass
Higgs Small
62751
15688
19610
28
Binclass
HO
House 16H
14581
3646
4557
16
Regression
IN
Insurance
856
214
268
3
Regression
KI
King
13832
3458
4323
17
Regression
MI
MiniBooNE
83240
20811
26013
50
Binclass
WI
Wilt
3096
775
968
5
Binclass
Table 2: List of datasets used for the evaluation and their descriptions.
Hyperparameters in TabDDPM are essential since in the experiments we observed them having strong influence on the model effectiveness. Table [ lists the main hyperparameters as well as the search spaces for each of them, which we recommend to use. The process of tuning is described in detail in the experimental section.
5 EXPERIMENTS
In this section, we extensively evaluate TabDDPM against existing alternatives.
Datasets. For systematic investigation of the performance of tabular generative models, we consider diverse set of 15 real-world public datasets. These datasets have various sizes, nature, number of features, and their distributions. Most datasets were previously used for tabular model evaluation in Zhao et al. 2021 Gorishniy al. 2021). The full list of datasets and their properties presented Table
Baselines.
Since the number of generative models proposed for tabular data is enormous, eval- uate TabDDPM only against the leading approaches from each paradigm of generative modelling. Also, we consider only the baselines with the published source code.
TVAE (Xu et al. 2019 the state-of-the-art variational auto-encoder for tabular data generation. To the best of our knowledge, there are no alternative VAE-like models that outperform TVAE and have public source code.
CTABGAN (Zhao et al. 2021) - a recent GAN-based model that shown to outperform the existing tabular GANs on diverse set of benchmarks. This approach cannot handle regression tasks.
2022 an extension of the CTABGAN model that was pub- lished in the very recent preprint. We not aware of the GAN-based model for tabular data that is proposed after CTABGAN+ and has a public source code.
CTABGAN+ (Zhao et al.
method that "generates" synthetic point convex combination of real data point and its k-th nearest neighbor from the dataset. This method was originally proposed for minor class oversampling. Here, we generalize and apply synthetic data generation as simple sanity check.
SMOTE (Chawla et al. 2002 a "shallow" interpolation-basee
Evaluation measure. Our primary evaluation measure is machine learning (ML) efficiency (or utility) (Xu et al.] 2019). In more detail, ML efficiency quantifies the performance of classification or regression models that are trained on synthetic data and evaluated on the real test set. Intuitively, models trained high-quality synthetics should be competitive (or even superior) models trained
5

---

Under review as a conference paper at ICLR 2023
run
cal, Seature
CH. HUP. Inture
rurn Instrre
AD. crt 2
num, JustuIn
HD, rum Jeaken
FR: mam. Stiture 26
Paol
DORM
Peol
CTARGAN
Resl
Top
Ruol
D0PM
CTARDAN
Real
TWN
Figure 2: The individual feature distributions for the real data and the data generated by TabDDPM, CTABGAN+, and TVAE. TabDDPM produces more realistic feature distributions than alternatives in most cases.
on real data. In our experiments, we use two evaluation protocols to compute ML efficiency. In the first protocol, which more common the literature (Xu al. 2019 Zhao al., [02T; Kim et al. 2022), we compute average efficiency with respect to set of diverse ML models (logistic regression, decision tree, and others). the second protocol, we evaluate ML efficiency only with respect to the CatBoost model (Prokhorenkova et al. [018)), which is arguably the leading GBDT implementation providing state-of-the-art performance on tabular tasks Gorishniy et al. (2021). In experiments in subsecttoo 5.2] we show that it is crucial to the second protocol, while the first one can often be misleading.
Tuning process. To tune the hyperparameters of TabDDPM and the baselines, we use the Optuna library (Akiba et al. 2019). The tuning process is guided by the values of the ML efficiency (with respect Catboost) of the generated synthetic data hold-out validation dataset (the score is aver- aged over five different sampling seeds). The search spaces for all hyperparameters of TabDDPM are reported in Table (for baselines in Apppeddix C). Additionally,
demonstrate that tuning the hyperparameters using the CatBoost guidance does not introduce any sort of "aaaboost-biasddesss" and the Catboost-tuned TabDDPM produces synthetics that are also superior for other models, like MLP. These results are reported in Appendix A
5.1
QUALITATIVE COMPARISON
Here, we qualitatively investigate the ability of TabDDPM to model the individual and joint feature distributions compared to the TVAE and CTABGAN+ baselines. particular, for each dataset, we sample a synthetic dataset from TabDDPM, TVAE, and CTABGAN+ of the same size real train set in a particular dataset. For classification datasets, each class sampled according to its propor- tion in the real dataset. Then, we visualize the typical individual feature distributions for real and synthetic data in Figure 2 For completeness, the features of different types and distributions are pre- sented. In most cases, TabDDPM produces more realistic feature distributions compared to TVAE and CTABGAN+. The advantage is more pronounced (1) for numerical features, which are uni- formly distributed, (2) for categorical features with high cardinality, and (3) for mixed type features

---

Under review as a conference paper at ICLR 2023
AB
AD
BU
CA
CH
HO
DDPM
TVAE CTABGAN+
The absolute difference between correlation matrices computed real and synthetic datasets. A more intensive red colour indicates a higher difference between the real and synthetic correlation values.
Figure 3:
that combine continuous and discrete distributions. Then, we also visualize the differences between the correlation matrices computed real and synthetic data for different datasets, see Feiuue To compute the correlation matrices, we use the Pearson correlation coefficient for numerical-numerical correlations, the correlation Ratio for categorical-numeriic cases, and Theil's U statistic between categorical features. Compared to CTABGAN+ and TVAE, TabDDPM generates synthetic datasets with more realistic pairwise correlations. These illustrations indicate that our TabDDPM model is more flexible compared to alternatives and produces superior synthetic data.
5.2 MACHINE LEARNING EFFICIENCY
In this section, we compare TabDDPM to alternative generative models in terms machine learning efficiency. From each generative model, we sample synthetic dataset with the size of a real train set in proportion from Table This synthetic data is then used to train a classification/reerrssiio model, which is then evaluated using the real test set. In experiments, classification performance evaluated by the F1 score, and regression performance is evaluated by the R2 score. We use two protocols:
1. First, compute average ML efficiency for diverse set of ML models, performed in previous works (Xu et al.) 2019 Zhao et al., 2021 Kim et al. 2022). This set includes Decision Tree, Random Forest, Logistic Regression
Ridge Regression) and MLP mod-
els from the scikit-learn library (Pedregosa et al.) 2011) with the default hyperparameters except for "max-depth" equals 28 for Decision Tree and Random Forest, "maximum iter- ations" equals 500 for Logistic and Ridge regressions, and "maximum iterations" equals 100 for MLPs.
2. Second, we compute MI. efficiency with respect to the current state-of-the-arr model for tabular data. Specifically, we consider CatBoost (Prokhorenkova et al. 2018) and MLP architecture from (Gorishniy et al. 2021) for evaluation. CatBoost and MLP hyperparam- eters are thoroughly tuned on each dataset using the search spaces from (Gorrshhiy et al.) 2021]. We argue that this evaluation protocol demonstrates the practical value of synthetic data more reliably since in most real scenarios practitioners are not interested in using weak and suboptimal classifiersreegressoss.
Main results. The ML efficiency values computed by both protocols presented in Table 3 and in Table 4 The ML efficiency for the tuned MLP is reported in/ppenddix A] To compute each value, average the results over five random seeds for synthetics generation, and for each generated

---

Under review as a conference paper at ICLR 2023
AB (R2)
AD (F1)
BU (PI)
CA (R2)
CAR (F1)
(F1)
(F1)
(P1)
TYAE
1.23
0.742+.001
0.779+.004
- -13 13.0++151
0.693=.00 002
0.681
003
0.613+.00 008
0.712+ 010
CTABGAN
0.737+.007
0.786+.00s
0.684=.003 000
0.636
0.614+.007
0.655++.015
CTABGAN+
0.3161.021
0.7301 .077
0.8374.006 006
-7.592.615
0.7084 .00
0.650=
008
0.648 po8
0.227
SMOTE
0,400+ 009
U.750+-004
0.842+.003
0.667+.000
0.693+.00
000
0.690-
003
0.649+
000
0,6774 013
TahDDPM
0.392 009
0.758
00s
8511 003
0.695 002
0.6962.00 001
0.693:
003
0.659:
003
0.675 011
Real
0.023+.009
0. 0.730+.006
0.845+.004
0.663+.002
0.683+.002
0.679=.00 003
0.618+.009
0.699+.012
FB (R2)
(F1)
HT (F1)
HO (R2)
IN (R2)
KT (R2)
MI (F1)
WI (P1)
TYAE
<
0.372+
0.590+..04
0.174
0.470=.024
0.666 008
0.880 0002
0.497: 001
CTABGAN
0.3391 .099
0.5391.000 006
0.8561 0o3
0.6561 011
CTABGAN
<0
0.3734 000
0.598 004
0.222:
042
0.669
018
0.197 051
0.807 002
0.653 027
SMOTE
0.651+
0.178 006
0.6611.008
0.391+.008
0.7099.00
008
0.751#.005
0.860 001
0.793=0004
TahDDPM
0.5274.005 .05
0.4624
00.
0.670
002
0.426
007
0.7344 007
0.61 12 013
0.850 0o4
0.792 004
Real
0.6452.005
0.431 1 .005
0.6632.002 002
0.415 007
0.708=.00 007
0.768 013
0.850 oo1
0.6844 000
Table 3: The values of machine learning efficiency computed with regards to five weak classifica- tion/regression models. Negative scores denote negative R2, which means that performance is worse than an optimal constant prediction.
AB (R2)
AD (F1)
(F1))
CA (R2)
CAR (F1)
CH (FF)
DE (F1)
(P1)
TVAE
0.4331.008
0.781 002
0.8641.00 000
0.7521 001
0.717 0m1
0.732 006
0.656 007
0.714 o39
CTABGAN
0.783+.o02
0.855
0.717=.001
0.688
0.644#.011
0.731=.u22
CTABGAN+
0.4871.001
0.7721 003
0.8841.000
o0s
0.525
001
0.7332.001 001
0.702 012
0.686 not
0.7342 020
SMOTE
0.5494
0.791
0.891
0.840#
0.732=
0.743=
0.693:
0.6834
TabDDPM
0.550 010
0. 0.795=.00 001
0.90
0.906+.00m
000
0.836*.00
002
7.7374000
0.755+.000
0.691
.04
0,7410..02 020
Real
0.5561.004 004 0.8154 .02
0.906 002
0.857 001
0.738 001
0.740 oo8
0,088 0oa
0.785a 013
FB (R2)
GE (F1)
HI (F1)
HO (R2)
IN (R2)
KI (R2)
(P1)
W1 (P11
TVAE
0.685+.008
0.4344 006
0.6382.008
0.493 .00
0.784=.010
0.824= 003
0.912#
0.5014
CTABGAN
0.392+.000 .006
0,575+.004
0.889+.002 0002
0.906-.019
CTABGAN+
0.5094.011
0.4064 000
0.6644..00 002
0.504
005
0.797= 005
0.444 014
0,892 002
0.798 021
SMOTE
0.803+
0.658+.00
0. 0.722+.001
0.662+.004
0.812+.002
0.8 0.842+.004
0.932+.00
001
0.913=.007
TaDDPM
0.7 0.713+.002 .002
0 0.397 .006
0. 0.722 +: 001
.6677..010
0.809 = 002
0.938 001
0.904: 009
Real
0. 0.837+.001
0. 0.636+.007
0.721+.001
0.662+.003 003
0.814 p01
0.907 002 0.931+.000 000
0. 0.898+.00 000
Table 4: The values of machine learning efficiency computed with regards to the state-of-the-an tuned CatBoost model.
dataset, we average over ten random seeds for training classiffers/rggressoss. The key observations described below:
In both evaluation protocols, TabDDPM significantly outperforms TVAE and CTABGAN+ on most datasets, which highlights the advantage of diffusion models for tabular data as well as demonstrated for other domains in prior works.
The interpolation-basee SMOTE method demonstrates the performance competitive to TabDDPM and often significantly outperforms the GAN/VAE approaches. Interestingly, most of the prior works on generative models for tabular data do not compare against SMOTE, while it appears to be simple baseline, which is challenging to beat.
While many prior works use the first evaluation protocol to compute the ML efficienc, we argue that the second one (which uses the state-of-the- the-arte model, like CatBoost) more appropriate. Table 3) and Table 4 show that the absolute values of classification/rerrssso performance are much lower for the first protocol, i.e., weak classifeersreerrssoors are sub- stantially inferior to CatBoost on the considered benchmarks. Therefore, one can hardly use these suboptimal models instead of CatBoost and their performance values are uninfor- mative for practitioners. Moreover, in the first protocol, training synthetic data is often advantageous compared to training on real data. This creates an impression that the data produced by generative models are more valuable than the real ones. However, it is not the case when one uses the tuned ML model, as in most practical scenarios. Appendix A confirms this observation for the properly tuned ML. model.
8

---

Under review as a conference paper at ICLR 2023
Overall, TabDDPM provides state-of-the-an generative performance and can be used as source of high-quality synthetic data. Interestingly, in terms of ML efficiency, a simple "shallow" SMOTE method is competitive to TabDDPM, which raises the question if sophisticated deep generative mod- els are needed. In the section below, we provide an affirmative answer to this question.
5.3 PRIVACY
Here, we demonstrate that TabDDPM is preferable SMOTE in setups with privacy concerns, e.g., sharing the data without disclosure of personal or sensitive information. In these setups, one is interested in high-quality synthetics that do not reveal the datapoints from the original real dataset. To quantify the privacy of synthetic, we use median Distance to Closest Record (DCR) (Zhao jet al. 2021) between synthetic and real datapoints. Specifically, for each synthetic sample, we find the minimum distance real datapoints and take the median of these distances. Low DCR values indicate that all synthetic samples are essentially copies some real datapoints, which violates the privacy requirements. In contrast, larger DCR values indicate that the generative model can produce something "new" rather than just copies of real data. [Table 5| 5|compares the DCR values for SMOTE and TabDDPM and demonstrates the advantage of TTbDDDM consistently for all datasets. We also visualize histograms of the minimal synthetic-to-reat distances on Figure 4 For SMOTE, most distance values are concentrated around zero, while TabDDPM samples are better separated from real datapoints. This experiment confirms that TabDDPM synthetics while providing high ML efficiency, are also more appropriate for privacy-concemee scenarios.
AD
BU
CAR
500re
DCR
500re
DCR
Scote
DCR
DCR
Score
DCR
score
DCR
52000
DCR
seor
DCR
SMOTE
0.549 0.014
0.791 0.024
0.891
0.054 0.840
0.014
0.732 0.007 0.743 0.077 0. 693 0.027
0.683
0.068
TaNDDPM 0.550
0.050 0.795 0. 104
0. 906
143
0.836
0.041
0.737
0.012 0.755 0. 157 0.691 0.112 0.740 0. 204
FB
GE
III
110
DCR
DCR
DCR
SMOTE
0.503 0.027
0.658
0.023
0.066 0.932
0.016 0.913 0.007
TANDDPM 0.713 0.112 0.597 0.059 0.722 0.449
0.677
0.086 0.809
0.041 0.833 0.189 0.936 0.022 0.904 0.016
Table 5: ML efficiency CatBoost scores and privacy scores for SMOTE and TabDDPM models.
AB
BU
CH
SMOTT
Figure 4: Histograms of minimal synthetic-to-real distances for TabDDM and SMOTE.
CONCLUSION
In this paper, we have investigated the prospect of the diffusion modelling framework in the field of tabular data. In particular, we have described the design of DDPM that can handle mixed data consisting of numerical, ordinal, and categorical features. We also demonstrate the importance of the model's hyperparameters and explain the protocol of their tuning. For the most considered benchmarks, the synthetics produced by model has consistently higher quality compared to

---

Under review as a conference paper at ICLR 2023
produced by the GAN/VAE-basee rivals and interpolation techniques, especially for the setups, where the privacy of the data must be ensured.
REFERENCES
Takuya Akiba, Shotaro Sano, Toshihiko Yanase, Takeru Ohta, and Masanori Koyama. Optuna: A next-generation hyperparameter optimization framework. In Proceedings of the 25th ACM SIGKDD international conference on knowledge discovery & data mining, pp. 2623-2631, 2019.
Jacob Austin, Daniel D Johnson, Jonathan Ho, Daniel Tarlow, and Rianne den Berg. Structured denoising diffusion models in discrete state-spaces. Advances in Neural Information Processing Systems, 34:17981-17998 2021.
Baldi, Sadowski, and Whiteson. Searching for exotic particles in high-energy physics with deep learning. Nature Communications, 5, 2014.
Dmitry Baranchuk, Ivan Rubachev, Andrey Voynov, Valentin Khrulkov, and Artem Babenko. Label- efficient semantic segmentation with diffusion models. arXiv preprint arXiv:2112.031 2021.
Ramiro D Camino, Christian A Hammerschmidt, et al. Oversampling tabular data with deep gener- ative models: Is it worth the effort? 2020.
Nitesh V Chawla, Kevin W Bowyer, Lawrence Hall, and W Philip Kegelmeyer. Smote: synthetic minority over-sampling technique. Journal of artificial intelligence research, 16:321-357, 2002.
Nanxin Chen, Yu Zhang, Heiga Zen, Ron Weiss, Mohammad Norouzi, and William Chan. Wave- grad: Estimating gradients for waveform generation. arXiv preprint 2XXi:20009007733, 2020.
Prafulla Dhariwal and Alex Nichol. Diffusion models beat gans on image synthesis. 2021.
Justin Engelmann and Stefan Lessmann. Conditional wasserstein gan-based oversampling tabular data for imbalanced learning. Expert Systems with Applications, 174:1 14582, 2021.
Ju Fan, Tongyu Liu, Guoliang Li, Junyou Chen, Yuwei Shen, and Xiaoyong Du Relational data synthesis using generative adversarial networks: A design space exploration. arXiv preprint arXiv:2008.1 12763, 2020.
Yury Gorishniy, Ivan Rubachev, Valentin Khrulkov, and Artem Babenko. Revisiting deep learning models for tabular data. Advances in Neural Information Processing Systems, 34:18932-18843 2021.
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. 2020.
Emiel Hoogeboom, Didrik Nielsen, Priyank Jaini, Patrick Forré, and Max Welling. Argmax flows and multinomial diffusion: Learning categorical distributions. Advances in Neural Information Processing Systems, 34:12454-12465, 2021.
Emiel Hoogeboom, Victor Garcia Satorras, Clement Vignac, and Max Welling. Equivariant dif- fusion for molecule generation in 3d. In International Conference Machine Learning, pp. 8867-8887, PMLR, 2022.
Bowen Jing, Gabriele Corso, Jeffrey Chang, Regina Barzilay, and Tommi Jaakkola. Torsional diffu- sion for molecular conformer generation. arXiv preprint arXiv:2206.01722 2022.
James Jordon, Jinsung Yoon, and Mihaela Van Der Schaar. Pate-gan: Generating synthetic data with differential privacy guarantees. In International conference learning representations, 2018.
R. Kelley Pace and Ronald Barry. Sparse spatial autoregressions. Statistics & Probability Letters, 33(3):291-297, 1997.
Jayoung Kim, Jinsung Jeon, Jaehoon Lee, Jihyeon Hyeong, and Noseong Park. Oct-gan: Neural ode-based conditional tabular gans. Proceedings of the Web Conference 2021, pp. 1506-1515, 2021.
10

---

Under review as a conference paper at ICLR 2023
Jayoung Kim, Chaejeong Lee, Yehjin Shin, Sewon Park, Minjung Kim, Noseong Park, and Jihoon Cho. Sos: Score-basee oversampling for tabular data. Proceedings of the 28th ACM SIGKDD Conference Knowledge Discovery and Data Mining, pp. 1622-77. 2022.
Ron Kohavi. Scaling up the accuracy of naive-bayes classifiers: decision-tree hybrid. In KDD, 1996.
Zhifeng Kong, Wei Ping, Jiaji Huang, Kexin Zhao, and Bryan Catanzaro. Diffwave: A versatile diffusion model for audio synthesis. arXiv preprint (rXiv:2009.09661 2020.
Haoying Li, Yifan Yang, Meng Chang, Huajun Feng. Zhihai Xu, Qi Li, and Yueting Chen. Srdiff: Single image super-resolution with diffusion probabilistic models. 2021.
Xiang Lisa Li. John Thickstun, Ishaan Guirajani. Percy Liang, and Tatsunori B Hashimoto. Diffusion-Im improves controllable text generation. arXiv preprint arXiv:2205.14217, 2022.
Renata C. B. Madeo, Clodoaldo Ap. M. Lima, and Sarajane Marques Peres. Gesture unit segmen- tation using support vector machines: segmenting gestures from rest positions. In Proceedings of the 28th Annual ACM Symposium Applied Computing, SAC, 2013.
Chenlin Meng, Yang Song, Jiaming Song, Jiajun Wu, Jun-Yan Zhu, and Stefano Ermon. Sdedit: Image synthesis and editing with stochastic differential equations. 2021.
Muhammad Ferjad Naeem, Seong Joon Oh, Youngjung Uh, Yunjey Choi, and Jaejun Yoo. Reliable fidelity and diversity metrics for generative models. In International Conference on Machine Learning, pp. 7176-7185. PMLR, 2020.
Prafulla Nichol, Alex & Dhariwal. Improved denoising diffusion probabilistic models. ICML, 2021.
Richard Nock and Mathieu Guillame- Generative trees: Adversarial and copycat. ICML, 2022.
Pedregosa, G. Varoquaux, Gramfort, V. Michel, Thirion, O. Grisel, M. Blondel, P. Pretten- hofer, R. Weiss, V. Dubourg. J. Vanderplas, A. Passos, D. Cournapeau, M. Brucher, M. Perrot, and E. Duchesnay. Scikit-learn: Machine learning Python. Journal of Machine Learning Research, 12:2825-2838 2011.
Liudmila Prokhorenkova, Gleb Gusev, Aleksandr Vorobev, Anna Veronika Dorogush, and Andrey Gulin. Catboost: unbiased boosting with categorical features. Advances in neural information processing systems, 31, 2018.
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High- resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF Con- ference on Computer Vision and Patlern Recognition, pp. 10684-10695, 2022.
Chitwan Saharia, Jonathan Ho, William Chan, Tim Salimans, David J Fleet, and Mohammad Norouzi. Image super-resolution via iterative refinement. 2021.
Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily Denton, Seyed Kam- yar Seyed Ghasemipour, Burcu Karagol Ayan, S Sara Mahdavi, Rapha Gontijo Lopes, et al. Photorealistic sxxttoo-imag diffusion models with deep language understanding. arXiv preprint arXiv:2205.11487 2022.
Kamaljot Singh, Ranjeet Kaur Sandhu, and Dinesh Kumar. Comment volume prediction using neural networks and decision trees. In IEEE UKSim-AMSS 17th International Conference on Computer Modelling and Simulation. UKSim, 2015.
Jascha Sohl-Dickstein, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsupervised learning using nonequilibrium thermodynamics. In ICML, 2015.
Yang Song and Stefano Ermon. Generative modeling by estimating gradients of the data distribution. In NeurIPS, 2019.
Yang Song and Stefano Ennon. Improved techniques for training score-based generative models. NewrIPS, 2020.
11

---

Under review as a conference paper at ICLR 2023
Yang Song, Jascha Sohl-Dicksteii Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. 2021.
Yusuke Tashiro, Jiamnig Song, Yang Song, and Stefano Ermon. Csdi: Conditional score-based
diffusion models for probabilistic time series imputation. Advances in Neural Information Pro- cessing Systems, 34:24804-2481 2021.
Amirsina Torfi, Edward A Fox, and Chandan K Reddy. Differentially private synthetic medical data generation using convolutional gans. Information Sciences, 586:485-500, 2022.
Joaquin Vanschoren, Jan N. van Rijn, Bernd Bischl, and Luís Torgo Openml: networked science in machine learning. arXiv,
2014.
Bingyang Wen, Yupeng Cao, Fan Yang. Koduvayur Subbalakshmi, and Rajarathnam Chandramouli. Causal-tgan: Modeling tabular data using causally- gan. In ICLR Workshop on Deep Gen- erative Models for Highly Structured Data, 2022.
Lei Xu, Maria Skoularidou, Alfredo Cuesta-Infante and Kalyan Veeramachaneni. Modeling tabular data using conditional gan. Advances in Neural Information Processing Systems, 32, 2019.
Yishuo Zhang, Nayyar A Zaidi, Jiahui Zhou, and Gang Li. Ganblr: tabular data generation model. In 2021 IEEE International Conference on Data Mining (ICDM). pp- 181-190. IEEE, 2021.
Zilong Zhao, Aditya Kunar, Robert Birke, and Lydia Chen. Ctab-gan: Effective table data syn- thesizing. Asian Conference on Machine Learning, pp. 97-112. PMLR, 2021.
Zilong Zhao, Aditya Kunar, Robert Birke, and Lydia Chen. Ctab-gan+: Enhancing tabular data synthesis. arXiv preprint 00Xiv:2204.0040), 2022.
12

---

Under review as a conference paper at ICLR 2023
APPENDIX
A MLP EVALUATION AND TUNING
Here, we show that tuning the hyperparameters using the CatBoost guidance results in the TabDDPM models that produce synthetics that is also optimal for other classifers/reerresooss. The results for subset of datasets are presented on Table 6 The methods denoted with "-CB" and "-MLP" denote the CatBoost guidance and different types of evaluation (CatBoost and MLP, re- spectively). The "-MLP-tune" suffix stands for the MI. guidance tuning and MLP evaluation.
AB (R2)
AD (F1)
BU (F1)
CA (R2) CAR (F1)
CH (P1)
DE (F1) DI (F1)
TRbDDPM-CB
0,550=.010
0.795+.00
001
0.906+.0n8 0. 0.836+.00 002 0.737 001
0.755+.000 000
0.691+ 004
0.740+.020
Real-CB
0.556=.004 0.815+.00
002 0.906#.002
0.8574 001 0.7384
001
0.740
009 0.6881.
003 0.785
Ta6DDPM-MLP
0.569=.010
0.794+.000
002
0.903+.00
.00 O. 8099 .008 0.737
001
0.750+.00
000
0.679+
.08
0.7544.024
Real-MLP
0.581=.00s
0.795#.001
0.905#.003
0. 0.8084.00
0.739
0.741 008
0.688
0.7544.017
TabDDPM-MLP-H
0.559=.00 0.792+.002
0.901+.008
0.803+.004
0.737
0.749+.00
0.671+
0.741+.018
FB (R22) (E(F))
HI (F1)
HO (R2)
IN (Sta)
(R2)
MI (F1)
WI
TaNDDPM-CB
0.713 002
0.597 0o0
0.7221.001 .00
0.6774 010
0.8092 002
0.8332 oss
0.9362 001
0.9044
Real-CB
0.837=.00 001
0.6364.
007
0.7244.003
0.0624
000 003 0.814 001 0.907 002 0.934
0.8984
0.5951 0o0
0.7171.00
.022
0.6431 010
0.7944 0os
0.804 015
0.9382 001
0.921 i
Real-MLP
0.607
0.717
0.6144
0.800
0.8822
0.936
0.905
TahDDPM M-nu-uune
0.6264 009 0.800 o00 0.7992 oss
0.9141.000
Table 6: ML utility score with MLP evaluation and MLP tuning compared with CatBoost evaluation and CatBoost tuning.
B ADDITIONAL RESULTS
Here, we provide results for CTGAN Xu et al. (2019) model (Talle 7}. We also follow Zhao et al. (021) and provide additional quantitative comparison that shows how well individual feature distributions are modelled (Tale Table 9 Table 10). Finally, we include density and coverage
metrics from Naeem et al. 2020) that improved alternatives of precision and recall, respectively
Table 11
Table 12p.
AB (R2)
AD (F1)
BU (F1)
CA (R2)
CAR (F1)
CH (F1)
DE (F1)
DI (F1)
CTGAN
0.420++.on4
0.867+t00s 0.789+.001
0.686+.00 0o3
0.730=.00 001
0.723 cos
0.699 009
0.459++009
TVAE
0.4331 .08
17811-002
0.8641 .055
0.7521 001
0.717= 001
0.7321 0o8
0.656 o07
0.714: 039
CTABGAN
0.7831. 002
0.8554. 005
0.717
001
0.6884 ooe
0.6441
011
0.7311 022
CTABGAN+
0.167 .044
1.772+.008
1.884=.005
0.525 804
0.733
001
0.7022 012
0.
0.686= 004
0.734= 020
SMOTE
0.549+.000
0.791 +.002
1.891+.00s
0.840+.000
0.732-
.001
0.7433+.00
0.693+.00
-03
0.6834.037
TaDDDPM
0.550 010
0. 7.75 001
0.
0.906+.000 000
0.836 002
0.737
0. 0.755 000
0.691 004
0.740= 020
Real
0.5561 004
0.815a. 002
0.9061 002
0.8571. 001
0.738 001
0.740 coa
0.688 o0a
0.7854 613
FB (72)
GE (F1)
(FF)
HO (22)
IN (T2)
KI (I2)
(F1)
WI (F11
CTGAN
0.443 .05
0.333
0.575 oo0
0.4334. 0o5
0.745 008
0.7724
0.783
0.7494
TVAE
0.685+.on8
0.43-14.006
0.638+.003
0.493+.005
0.781=.010 010
0.821 coa
0.912 001
0.501+.012
CTABGAN
0.3924.006
0.375++.00.
0.889 0002
0.906= 019
CTABGAN+
0.5091.013
0.40611.099
0.664...0
0.504 005
0.707
005
0.444 014
0.802
002
0.798.1 021
SMOTE
0.803 002
6.88 607
0.7221 000
0.6621 004
0.8121.
002
0.8421 004
0.9324 001
0.9131
007
TabDDPM
0.7131.002 0.5974.006
0.722
0.6771.010
0,809
0.8331..11
0.9361.001
0.904:
Real
0.837
0.724+.001 0.636+.007
0.662+.003
.011
0.907 .000
0.9344.00 0000
0.898+.00
Table 7: ML utility score with CatBoost evaluation.
13

---

Under review as a conference paper at ICLR 2023
CA
CH1
CIGAN
0.008
0.010
0.015
0.004
0.001
0,000
0.001
0.085
TVAE
0.020
0.016
0.039
0.007
0.027
6,040
0.009
0.044
CTABGAN+
0.008
0.011
0.016
0.019
0.003
0.046
0.022
0.016
SMOTE
TabDDPM
0.005
0.002
0.003
0.002
0.000
0.005
0.012
0.008
FB
GE
HI
HO
IN
KI
M1
W1
CTGAN
0.004
0.010
0.003
0.005
0.021
0.022
0.004
0.013
TVAE
0.008
0.009
0.076
0.007
0.025
0.012
0.004
0.016
CIABGAN+
0.078
0,007
0.052
0.008
0.025
0.021
0.006
0.006
SMOTE
0.000
0.004
0.009
0.005
0.011
0.004
0.000
0.002
TabDDPM
0.089
0.011 0.003
0.004 0.006
0.014
0.001 0.002
Table 8: Wasserstein distance between numerical features.
BU
CA
CA
CH
DE
CTGAN
0.085
0.108
nan
0.076
0.039
0.120
0.298
TYAE
0.027
0,095
0.072
0.181
0.019
0.157
0.052
CTABGAN+
0.035
0.052
0.037
nan
0.009
0.018
0.030
0.017
SMOTE
0.005
0.074
0.072
0.069
0.030
0.058
0.004
TaNDDPM
0.007
0.019
0.026
non 0.011
0.017
0.009
0.006
FB
GE
HII
H0 IN
KI
MI
CTGAN
0.017
0.240
0.091 non 0.071
0.296
0.140
0.532
non
CTABGAN+
0.051
0.094
0.009 nan
0.023
0.044
0.075
0.017
SMOTE
0.027
0.000
0.000
nan 0.013
0.102
0.000
0.000
TaNDDPM
0.046
0.001
0.001
0.008 0.060
0.000 0.002
Table 9: Jeesenn-Shanon divergence between categorical features.
AB
AD
BU
CA
CA
CH
DE
CTGAN
0.471
0.390
0.492
0.606
0.712
TVAE
0.517
0.636
0.569
0.753
2.437
0.564
1.965
0.736
CTABGAN+
0,283
0.376
0.164
0.749
0.738
0.727
1.496
0.435
SMOTE
0.185
0,482
0.245
0.127
0.599
0.147
0.642
0.838
TabDDPM
0.333
0.133
0.068
0.090
0.202
0.161
0.934
0.186
FB
GE
HO
IN
KI
M1
CTGAN
5.651
5.301
1.413
0.742
0.196
1.530
43.815
0.538
TVAE
5.960
2.996
2.759
1.001
0.550
CTABGAN+
6.782
1.977
1.241
0.978
0.207
3.898
31.704
0.319
0.354
0.452
0.258
TabDDPM
16.120
1.1
0. 233
0.336
0.077
3.623
185
0.375
Table 10: L2 distance between correlation matrices.
AB
AD
BU
CA
CA
CH
DE
DI
CTGAN
0.224
0.708
0.780
0.586
0.938
0.865
0.098
0.238
TYAE
0.317
1.126
1.032
0.746
0.815
1.043
0.808
1.565
CTABGAN+
0.957
SMOTE
1.389
1.415
1.226
1.329
1.200
1.238
1.282
1.413
TabDDPM
0.904
1.008
1.116
1.027
1.011
1.148
0.810
0.831
KI
CTGAN
0.147
0.035
0.702
0.467
0.927
0.719
0.361
0.763
TVAE
0.005
0.248
0.960
0.604
1.072
0.868
0.747
0.919
CTABGAN+
0.187
0.148
0.730
0.565
1.052
0.186
0.110
0.831
TabDDPM
0.633
1.460
1.152
1.195
1.150
0.884
0.972
1.009
Table 11: Density of synthetic data.
14

---

Under review as a conference paper at ICLR 2023
CIGAN
0.651
0.9418
0.966
0.759
0.920
1,000
0.777
0.572
TVAE
0.769
0.886
0.585
0.922
0.208
0.991
0.672
0.978
CTABGAN+
0.960
0.951
0.999
0.459
0.960
0.830
0.841
1.000
TabDDPM
1.000
0.994
1.000
0.998
0.978
1.000
0.967
0.955
FB
GE
HI
HO
IN
K1
M1
W1
CTGAN
0.238
0.029
0.871
0.839
0.986
0.739
0.576
0.986
TVAE
0.014
0.669
0.255
0.875
0.987
0.874
0.823
0.867
CIABGAN+
0.222
0.640
0.557
0.952
1.000
0.479
0.241
0.994
SMOTE
0.928
1.000
0.999
1.000
0.995
0.945
0.991
1.000
TabDDPM
0.782
0.997
0.980
1.000
1.000
0.969
0.956
1.000
Table 12: Coverage of synthetic data.
C HYPERPARAMETERS SEARCH SPACES
Parameter
Distribution
Max depth
UniformInt(3. 10]
Learning rate
LogUniform [1--5, 1]
Bagging temperature
Uniform [0.
L2 leaf reg
LogUnifornu 1, 10]
Leaf estimation iterations
UniformInt [1, 10]
Number of tuning trials
100
Table 13: CatBoost hyperparameters space from Gorishniy et al.
2021
Parameter
Distribution
Layers Layer size
UniformInt 11,
[nt(66, 128, 256, 512, 1024) {0, Uniform (0, 0.5)
Dropout
Learning rate
Log le-5, 1e-2]
Weight decay
{0, Log Uniform/ 1e-6, 1e-3]}
Number of tuning trials 100
Table 14: MLP hyperparameters space from Gorishniy et al. (0021)
Parameter
Distribution
k_neighbours
[n 20]
Arange
Float (0, 1
Proportion of samples
Float F0.25, 0.5, 1,2,4,8,
Number of tuning trials 50
Table 15: SMOTE hyperparameters search space. Arange denotes the range of interpolation coeffi- cient to sample from
15

---

Under review as a conference paper at ICLR 2023
Parameter
Distribution
claassif. layers
UniformInt[1, 4]
Classif. layer size Training iterations
Int 128, 256}
Cat 5000, 10000) Int[512, 1024, 2048)
Batch Size
random_din
[nt 32, 64, 128)
num .cmannlsslls
[n6 32, 64] Fl.at(0.22,, 0.5, 2,
Proportion of samples
Number of tuning trials 35
Table 16: CTABGAN and CTABGAN+ hyperparameters search space. See an official implementa- tion
Parameter
Distribution
claassif. layers Classif. layer size Training iterations Batch Size embedding.die dim loss factor Proportion of samples
UniformInt[1,
[nt 128, 256, 512)
(5000 30000,30000,
Cat 40963 Int Int(16, 32, 64, 128, 256, 512, 1024] LogUniform (0.01, 10)
Float (aatt00.22 (0.25, 0.5, 1,2,
Number of tuning trials 50
Table 17: TVAE hyperparameters search space. See an official implementation
D DATASETS
We used the following datasets:
Abalone (OpenML)
Adult (income estimation, Kohavi (996))
Buddy (Kaggle)
California Housing (real estate data, Kelley Pace & Barry (1997)
Cardiovascular Disease dataset (Kaglle)
Churn Modeling (Kaggle)
Diabetes (OpenML)
Facebook Comments (Singh al. (2015)
Gesture Phase Prediction Madeo (2013)
Higgs (simulated physical particles, Baldi et al. (2014); we use the version with 98K sam- ples available at the OpenML. repository Vanschoren et al. (2014))
House 16H (OpenLL)
Insurance (Kaggle)
King (Kaggle)
MiniBooNE (OpenLL)
Wilt (OpenML)
"https//gitub..com/Taa TAAAAAATTTTTTTTAAAAAPPA http://gthha.oom//aa//DDDDtAAAAa hhtg:://gthhhh.oo///vvve/CCCCC
16

---

Under review as a conference paper at ICLR 2023
E ENVIRONMENT AND RUNTIME
Experiments were conducted under Ubuntu 20.04 on a machine equipped with GeForce RTX 2080 Ti GPU and Intel(R) Core(TM) i7-7800X CPU @ 3.50GHz. We used Pytorch 10.1, CUDA 11.3, scikit-learn 1.1.2 and imbalanced-learn 0.9.1 (for SMOTE).
As for runtime of the proposed method, it depends the dataset and hyperparameters. We pro- vide 3 examples below. All three examples use T 1000 and batch_size = 4096. Note that hyperparameters tuning contains 50 runs and takes usually 8-10 hours.
Dataset imput dim
model layers
train steps n to sample train time sample time
CH
16
1256,1024,100 1024,
30k
26k
670s
6s
HI
28
[512,1024, 1024, 1024,
30k
502k
502s
430s
FB
146
[512,1024]
30k
1264k
783s
470s
Table 18: Training and sampling time for TabDDPM.
17


Reviews:
[{"review_id": "X-VbS59qZ9", "reviewer": "Reviewer_x66q", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper focuses on tabular problems. It proposes a diffusion model called TabDDPM to model tabular data. The TabDDPM can be universally applied to any tabular dataset and handles any type of feature. The experimental results show that the TabDDPM outperforms several SOTA models, which is evaluated on several datasets.", "summary_of_the_review": "Tabular problem is an important task, and it is quite interesting to use the diffusion model to model tabular data. While the compared models (e.g., SMOTE) are not strong enough. The improvement of TabDDPM is obvious in Figure 3 and Figure 4, but not obvious in Table 3 and Table 4.", "strength_and_weaknesses": "Strength:\n(1) It is interesting to apply the diffusion model to other tasks except for computer vision. \n(2) The experimental results look good. Several widely used datasets are used, and the TabDDPM achieves the best results for most datasets.\n\nWeaknesses:\n(1) I have two main concerns:\n  A. Can those models in Table 3 and Table 4 (the SMOTE, CTABGAN, and CTABGAN+ models) represent the SOTA results on all datasets? Actually, SMOTE is published in 2002.\n  B. The recent task tends to have a large data size (i.e., FB dataset in this paper), while TabDDPM does not perform well on this dataset.\n(2) Minor:\n  A. The bold numbers of the last column in Table 4 are weird (DI and WI). ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "QLwiWyhM79", "reviewer": "Reviewer_d279", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The authors propose to combine multimodal and gaussian diffusion models as a new diffusion model that can handle small datasets with mixed kinds of features. ", "summary_of_the_review": "The paper combines two available diffusion models but the usefulness of the method for tabular data is wrong. ", "strength_and_weaknesses": "The paper is fairly well structured, however, it includes a few wrong claims. For example, it describes tabular data as data that includes different feature types, however, this is not necessarily a definition of tabular data. The features of tabular data can be only from a single family. In another part, the authors stated \"Unlike unstructured images or natural texts, tabular data is typically structured,\" which is totally wrong. We have structure in image and NLP and the CNN is using this feature as a biased into the architecture. On the other hand, as stated in many tabular data papers, including SubTab [NeurIPS 2021] and VIME [NeurIPS 2020], and etc, the main problem of tabular data is that it does not have structure or the structured is not obvious as other domains. \n\nApart from that, the proposed method is not novel and the experiments are not convincing. \n\nThe number of features is too small in all experiments. The experiments are not showing how the generated samples are diverse.\n\nOne can also normalize features and then use diffusion models. This is not included in the baselines. ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "1: The main claims of the paper are incorrect or not at all supported by theory or empirical results.", "technical_novelty_and_significance": "1: The contributions are neither significant nor novel.", "empirical_novelty_and_significance": "1: The contributions are neither significant nor novel."}, "extra_text": {}}, {"review_id": "H1WLj1Wn6sc", "reviewer": "Reviewer_qTDQ", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper proposes TabDDPM, a diffusion model for generic tabular data consisting of both continuous and discrete features. The paper compares TabDDPM to existing VAE and GAN methods, and against a simple SMOTE baseline. ", "summary_of_the_review": "While the overall motivation of the paper is good, I'm not sure the proposed method does enough to justify itself over the SMOTE baseline (although demonstrating the effectiveness of that baseline is worthwhile in and of itself). I find it hard to recommend acceptance as things stand. ", "strength_and_weaknesses": "**Strengths**\nI think the overall motivation for the paper is good. It is often the case that generative models are specialized to particular media, and can be unsuitable out-of-the-box for generic tabular data. \n\nThe experimental results mainly highlight the unreasonable effectiveness of SMOTE as a baseline, beating both the VAE and GAN approaches. This is a useful observation. \n\n**Weaknesses**\nWhile demonstrating the strong SMOTE baseline is useful, it also highlights that the proposed method only matches this baseline. I'm also not sure about the comparison to SMOTE on sensitive data, both in terms of the metric used, and whether fitting generative models on sensitive data is a useful idea. The latter is discussed below, but in terms of the DCR metric, surely this only captures one possible vector of attack for a potential adversary? That is, a comprehensive comparison of privacy preservation in this setting would go beyond a simple proximity check, and instead marginalize over a number of possible attack vectors? \n\nFor a stronger baseline beyond SMOTE, I would have thought the authors would try a model which factorizes the continuous and discrete (non-ordinal) features. That is, train one model on the continuous features (e.g. a standard diffusion model), and then another model for the discrete features conditioned on the continuous features (e.g. a conditional autoregressive model), or vice versa. This would exploit the state-of-the-art generative models for both modalities, and would provide a strong baseline for a model trained on all features jointly to beat. Moreover, the statement 'In contrast, larger DCR values indicate that the generative model can produce something “new” rather than just copies of real data.' seems far too strong in that (i) it argues that the data and model capacity are sufficient to recover something close to the 'true' distribution, and (ii) DCR is a sufficient metric to determine this. \n\nI may be unfamiliar with common practice, but I'm not sure that I buy the arguments regarding sensitive data and privacy. In particular, if data is sensitive enough to not be shared, training classification/regression models etc. on that data is itself a tricky task, potentially introducing issues of bias, fairness, etc., not to mention generic failure cases. To me, adding the extra step of first fitting a generative model to the data, which is then used to create synthetic data for input to the rest of the pipeline, only exacerbates these potential issues. For example, even though diffusion models have seen impressive breakthroughs in image generation, using diffusion models to generate e.g. medical scans for the purpose of enhancing a disease-detection model seems highly questionable. \n\nFinally, when judging the performance of the proposed method, Figure 2 seems to just visualize a selection of the marginal feature distributions. This seems lacking, in that (i) the evaluation is qualitative, and (ii) surely we're primarily interested in judging the model's capacity to model the joint dependencies in the data correctly? ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "5A1QLhL_oq", "reviewer": "Reviewer_8b5K", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The authors propose TabDDPM, a denoising diffusion model for tabular data modeling and synthesis.  Within this framework, categorical and binary features are modeled using multinomial diffusion, and, completely independently, numerical features are modeled using Gaussian diffusion.  The model is compared to several recent deep generative models on the task of \"Train on fake, test on real (TFTR)\" [1] across different datasets considering the average of five classification/regression algorithms and, separately, Catboost.\n\n[1] Jordon, James, Jinsung Yoon, and Mihaela Van Der Schaar. \"PATE-GAN: Generating synthetic data with differential privacy guarantees.\" International conference on learning representations. 2018.", "summary_of_the_review": "The proposed method has critical design flaws.  The evaluation and comparison to SOTA tabular data synthesized also has significant room for improvement.  I recommend rejection.", "strength_and_weaknesses": "Strengths\n------------\n-The use of diffusion models for tabular data modeling is interesting\n\nWeaknesses\n---------------\n-The proposed method suffers from one major flaw: \"TabDDPM uses the multinomial diffusion to model the categorical and binary features, and the Gaussian diffusion to model the numerical ones.\" <- This ignores correlations between categorical and numerical variables.  Consider a categorical feature, \"Name,\" and a numerical feature, \"Name_length.\"  Your model could not learn, or even approximate, this deterministic relationship.  Corresponding synthesized data will also be incorrect.\n\n-\"Each categorical feature is handled by a separate forward diffusion process, i.e., the noise components for all features\nare sampled independently.\" <- Same criticism as above\n\n-The authors state the utility of their method for privacy.  But the two aforementioned deficiencies render synthetic data incorrect, e.g., synthetic data may disrespect even obvious correlations.\n\n-Why is baseline SMOTE used?  Other variants, like SVM SMOTE, are reported to work much better.\n\n-Only TVAE is compared to.  However, CTGAN should also be considered,\nas it's source is open (it is the package as TVAE, the CTGAN package)\nand is the main synthesizer (TVAE was developed as a competitor in the\nCTGAN paper).\n\n-The paper lacks a large amount of important details.  E.g.: 1) \"Smote... we generalize it\" <- How? (2) Details for utilized software, both implementation (SMOTE) and version (sklearn), are lacking from the paper.\n\n-\"we evaluate ML efficiency only with respect\nto the CatBoost model (Prokhorenkova et al., 2018), which is the leading GBDT implementation\nproviding state-of-the-art performance on tabular tasks\" <- Please say\n\"arguably is the leading...,\" as XGBoost is more widely used and also\nprovides SOTA GBDT performance.\n\n-No discussion on the runtime (training and synthesis) of TabDDPM.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "1: The main claims of the paper are incorrect or not at all supported by theory or empirical results.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject