TabDiff: a Mixed-type Diffusion Model for Tabular Data Generation
Authors: Juntong Shi
Minkai Xu
Harper Hua
Hengrui Zhang
Stefano Ermon
Jure Leskovec
Venue: ICLR 2025 Poster (2025)
Keywords: Tabular Representative Learning
Generative Models
Diffusion Models
Primary area: applications to computer vision, audio, language, and other modalities

Abstract: Synthesizing high-quality tabular data is an important topic in many data science tasks, ranging from dataset augmentation to privacy protection. However, developing expressive generative models for tabular data is challenging due to its inherent heterogeneous data types, complex inter-correlations, and intricate column-wise distributions. In this paper, we introduce TabDiff, a joint diffusion framework that models all mixed-type distributions of tabular data in one model. Our key innovation is the development of a joint continuous-time diffusion process for numerical and categorical data, where we propose feature-wise learnable diffusion processes to counter the high disparity of different feature distributions. TabDiff is parameterized by a transformer handling different input types, and the entire framework can be efficiently optimized in an end-to-end fashion. We further introduce a mixed-type stochastic sampler to automatically correct the accumulated decoding error during sampling, and propose classifier-free guidance for conditional missing column value imputation. Comprehensive experiments on seven datasets demonstrate that TabDiff achieves superior average performance over existing competitive baselines across all eight metrics, with up to $22.5\%$ improvement over the state-of-the-art model on pair-wise column correlation estimations. Code is available at https://github.com/MinkaiXu/TabDiff.

Published as a conference paper at ICLR 2025
TABDIFF: A MIXED-TYPE DIFFUSION MODEL FOR TABULAR DATA GENERATION
Juntong ShiPt, Minkai Xu xu Harper Hua't, Hengrui Zhang Stefano Ermon', Jure Leskovec* 1 Stanford University Uniiversity of Southern California University of Illinois Chicago
ABSTRACT
Synthesizing high-quality tabular data an important topic in many data science tasks, ranging from dataset augmentation to privacy protection. However, develop- ing expressive generative models for tabular data is challenging due to its inherent heterogeneous data types, complex inter-correlations, and intricate column-wit distributions. In this paper, we introduce TABDIFF, joint diffusion framework that models all mixed-type distributions of tabular data in one model. Our key innovation is the development of a joint continuous-time diffusion process for numerical and categorical data, where we propose feature-wise learnable diffusion processes to counter the high disparity of different feature distributions. TABDIr is parameterized by transformer handling different input types, and the entire framework can be efficiently optimized in an end-to-end fashion. We further in- troduce a mixed-type stochastic sampler to automatically correct the accumulated decoding error during sampling, and propose classifier-fre free guidance for condi- tional missing column value imputation. Comprehensive experiments on seven datasets demonstrate that TABDIFF achieves superior average performance over existing competitive baselines across all eight metrics, with up to 22.5% improve- ment over the ofate-of-the-art model on pair wise column correlation estimations. ff.
Code is available at https: / thub. com
INTRODUCTION
Tabular data is ubiquitous in various databases, and developing effective generative models for it fundamental problem in many data processing and analysis tasks, ranging from training data augmentation (Fonseca & Bacao, 2023), data privacy protection (Assefa et al., 2021; Hernandez et al., 2022), to missing value imputation (You et al., 2020; Zheng & Charoenphakdee, 2022). With versatile synthetic tabular data that share the same format and statistical properties the existing dataset, are able to completely replace real data in workflow or supplement the data to enhance its utility, which makes it casier share and use. The capability of anonymizing data and enlarging sample size without compromising the overall data quality enables it to revolutionize the field of data science. Unlike image data, which comprises pure continuous pixel values with local spatial correlations, or text data, which comprises tokens that share the same dictionary space, tabular data features have much more complex and varied distributions (Xu al., 2019; Borisov et al., 2023), making it challenging to learn joint probabilities across multiple columns. More specifically, such inherent heterogeneity leads obstacles from two aspects: 1) typical tabular data often contains mixed-tyee -type data types, i.e., continuous (e.g.. numerical features) and discrete (e.g.. categorical features) variables; 2) within the same feature type, features do not share the exact same data property because of the different meaning they represent, resulting in different column-wiss marginal distributions (even after normalizing them into same value ranges).
In recent years, numerous deep generative models have been proposed for tabular data generation with autoregressive models (Borisov et al., 2023), VAES (Liu et al., 2023), and GANs (Xu et al., 2019) in the past few years. Though they have notably improved the generation quality compared to traditional machine learning generation techniques such as resampling (Chawla et al., 2002), the generated data quality is still far from satisfactory due to limited model capacity. Recently, with the rapid progress in diffusion models (Song & Ermon, 2019; Ho et al., 2020; Rombach et al., 2022),
"Corresponding author, mi nnki ecs stanford 'Equal contribution

---

Published as a conference paper at ICLR 2025
Denoshyg Network
Reverse Diff
Age
Ogphrl Goe
Theare
Gorder
Transform
00
60x
76880
x
Xqu-o
8y0
Normalize
(x (. (A4 20
cc NV,
(x=
(xx7"), with P av(t)
m with 2 1 auft)
Forward Diff
One- hot
X0
Append Mask
t= 0.0 (less noisy)
I = 1.0 (more noisy)
Figure 1: A high-level overview of TABDIFF. TABDIFF operates by normalizing numerical columns and converting categorical columns into one-hot vectors with an extra [MASK] class. Joint forward diffusion processes are applied to all modalities with each column's 's noise rate controlled by learnable schedules. New samples are generated via reverse process, with the denoising network gradually denoising X into Xg and then applying the inverse transform to recover the original format.
researchers have been actively exploring extending this powerful framework to tabular data (Kim ct al., 2022; Kotelnikov et al., 2023; Zhang et al., 2024). For example, Zheng & Charoenphakdee (2022); Zhang al. (2024) transform all features into a latent continuous space via various encoding techniques and apply Gaussian diffusion there, while Kotelnikov et al. (2023); Lee et al. (2023) combine discrete-ting continuous and discrete diffusion processes (Austin et al., 2021) deal with numerical and categorical features separately. However, prior methods are trapped sub-optimal performance due to additional encoding overhead imperfect discrete-time diffusion modeling, and none of them consider the feature-wise distribution heterogeneity issue in a mixed-type framework.
In this paper, we present TARDIFF, a novel and principled mixed-type diffusion framework for tabular data generation. TAFDIFF perturbs numerical and categorical features with joint diffusion process, and learns a single model to simultaneously denoising all modalities. Our key innovation is the development of mixed-type feature- wise learnable diffusion processes to counteract the high heterogeneity across different feature distributions. Such ceature-speciff learnable noise schedules enable the model to optimally allocate the model capacity to different features in the training phase. Besides, encourages the model to capture the inherent correlations during sampling since the model can denoise different features in flexible order based on the learned schedule. We parameterize TABDIFF with a transformer operating on different input types and optimize the entire framework efficiently in end-to-end fashion. The framework is trained with continuous- limit of evidence lower bound. To reduce the decoding error during denoising sampling, we design a mixed-ty stochastic sampler that automatically corrects the accumulated decoding error during sampling. In addition, we highlight that TABDIFF can also be applied to conditional generation tasks such as missing column imputation, and we further introduce classifier-free guidance technique improve the conditional generation quality.
TABDIFF enjoys several notable advantages: our model learns the joint distribution in the original data space with an expressive continuous-ting time diffusion framework; 2) the framework is sensitive to varying feature marginal distribution and can adaptively reason about feature-specific information and pair-wise correlations. We conduct comprehensive experiments to evaluate TABDIFF against state-of-te --tt-hh-rttttt methods across seven widely adopted tabular synthesis benchmarks. Results show that TABDIFF consistently outperforms previous methods over eight distinct evaluation metrics, with up to 22.5% improvement over the state-of-the. model on pair-wise column correlation estimations, suggesting our superior generative capacity mixed-type tabular data.
2
METHOD
2.1
OVERVIEW
Notation. For a given mixed-type tabular dataset T. we denote the number of numerical features and categorical features M and Meat, respectively. The dataset is represented as collection of
2

---

Published as a conference paper at ICLR 2025
data entries T= (x) {[x="mm "x"",,, where each entry is a concatenated vector consisting of its numerical features x num and categorical features x We represent the numerical features as a Mnum dimensional vector x=um R Muuuum and denote the i-th feature as (x""m);; ) E R. We represent one-hot column vector (x**)); ) E {0, 1) (C)+1) with an extra dimension dedicated to the [MASK] state. The (Cj + 1)-th category corresponds to the special [MASK] state and we use m E {0, 1 as the one-hot vector for it. In addition, we define (: TT) as the categorical distribution over K classes with probabilities given by E A A where A K is the K K-simplex.
each categorical column with Cj finite categories
Different from common data types such as images and text, developing generative models for tabular data is challenging as the distribution is determined by mixed-type data. We therefore propose TABDIFF, unified generative model for modeling the joint distribution p(x) using continuous-time diffusion framework. DIFD can learn the distribution from finite samples and generate faithful, diverse, and novel samples unconditionally. We provide high-level overview in Figure 1, which includes forward diffusion process and reverse generative process, both defined in continuous time. The diffusion process gradually adds noise to data, and the generative process learns to recover the data from prior noise distribution with neural networks parameterized by 0. In the following sections, we elaborate on how we develop the unified diffusion framework with learnable noise schedules and perform training and sampling in practice.
2.2 MIXED-TYPE DIFFUSION FRAMEWORK
Diffusion models (Sohl-Dickstein et al., 2015; Song & Ermon, 2019; Ho et al., 2020) are likelihood- based generative models that learn the data distribution via forward and reverse Markov processes. Our goal is to develop a principled diffusion model for generating mixed-type tabular data that faithfully mimics the statistical distribution of the real dataset. Our framework DIFF is designed to directly operate on the data space and naturally handle each tabular column in its built-in datatype. TABDIFF is built on a hybrid forward process that gradually injects noise to numerical and categorical
data types separately with different diffusion schedules o mumm and c Let (x, : L ~ [0. 1]} denote
sequence of data in the diffusion process indexed by a continuous time variable t [o, 1]. where
Xo Po are i.i.d. samples from real data distribution and X1 ~ are pure noise from prior distribution. The hybrid forward diffusion process can be then represented as:
q(x xo) q (x:
xx a nu (t)) q (xxa* xxfa cat (t))
(1)
Then the true reverse process can be represented as the joint posterior:
q(x, x,, Xo) = q(x. num Xx, Xo) g(x Xp)
(2)
where s and are two arbitrary timesteps that 0 <<<<st 1. We aim learn a denoising model (x, xx.) to match the true posterior. In the following, we discuss the detailed formulations of diffusion processes for continuous and categorical features in separate manners. To enhance clarity, we omit the superscripts on " and x when the inclusion is unnecessary for understanding.
Gaussian Diffusion for Numerical Features. In this paper, we model the forward diffusion for continuous features bum as a stochastic differential equation (SDE) dx = f(x, t)dt + g(t)dw, with f(.,t) : R Man - R Mamu being the drift coefficient, g(.) R -> being the diffusion coefficient, and w being the standard Wiener process (Song et al., 2021; Karras et al., 2022). The revere generation process solves the probability flow ordinary differential equation (ODE) dx [[(x, t) =g(()²Vx log P. (x)]dt, where Vx log px(())))))))))))))>()) is the score function of (x). In this paper,
we use the Variance Exploding formulation with f(.,t) = 0 and g(t) = 2 d o num o num (t),
which yields the forward process:
x x pum + o num (t)c, e~ NO, I
(3)
And the reversal can then be formulated accordingly as:
d
dx - o num (t))o "u)(,) log Pe (x (dt.
(4)
dt
In TABDIFF, we train the diffusion model μ4 to jointly denoise the numerical and categorical features. We use p to denote numerical part of the denoising model output, and train the model via optimizing the denoising loss:
Lnum (0, p) Exoxpixa) Be BE (|14""
(xx; () El1
3

---

Published as a conference paper at ICLR 2025
Masked Diffusion for Categorical Features, For categorical features, we take inspiration from the recent progress on discrete state-space diffusion for language modeling (Austin et al., 2021; Shi et al., 2024; Sahoo et al., 2024). The forward diffusion process is defined masking (absorbing) process that smoothly interpolates between the data distribution x)) and the target distribution cat(;; m), where all probability mass are assigned the [MASK] state:
(6) Or E [0, 1] is a strictly decreasing function of f, with ag Rs 1 and 0n R 0. It represents the probability for the real data X, be masked at time step t. By the time t 1, all inputs are masked with probability 1. In practice this schedule is parameterized by
((x:|x0) = cat Xx + (1 - (1-a))m) at)m).
exp( sup(-oo"tt (t)), where cat (1) : [0, 1] + Rt a strictly increasing function. Such forward process entails the step transition probabilities g(x,|x,)
cat((x;; xx, + (1 O'ale ) ))) where otls at/a,, Under the hood, this transition means that at each diffusion step, the data will be perturbed to the [MASK] state with a probability of (1 asjss, and remains there until f 1 if perturbed.
xx)
Similar to numerical features, in the reverse denoising process for categorical ones, the diffusion model pg aims to progressively unmask each column from the 'masked' state. The true posterior distribution conditioned on Xy has the close form of:
cat(Xx;)))) cat Xx: (1-a, (a, Ixn
X
q(x, ((x xo)
1-a:
X
We introduce the denoising network ( we can approximate the unknown true posterior as:
1) C x [0, - A
to estimate Xo. through which
(x. x(at) cat, (1-o, Im (a, - 1el (X.iit
f m,
pa( pe(x (at))=
(8)
cat
x
X m,
1-0c
which implied that at each reverse step, we have a probability of (a, aat)/aa/(oon) to recover zo. and once being recovered, 24 stays fixed for the remainder of the process. Extensive works (Kingma et al., 2021; Shi et al., 2024) have shown that increasing discretization resolution can help approximate tighter evidence lower bound (ELBO). Therefore, we resort to optimizing the likelihood bound Leat. under continuous time limit:
a/ 1 Oz
Leat (0, k) By
1 (xx=m) log( (xe,, t), x ( dt,
where of is the first order derivative of C't.
2.3 TRAINING WITH ADAPTIVELY LEARNABLE NOISE SCHEDULES
Tabular data is inherently highly hetero- geneous of mixed numerical and categor- ical data types, and mixed feature distri-
Algorithm 1 Training
butions within each data type. Therefore, 1: repeat
unlike pixels that share similar distribu- tion across three RGB channels and word
2: Sample Xg ~ ~ px())
3: Sample & (0
tokens that share the exact same vocabu-
4: Sample Enum ~ ~N(0 I
lary space, each column (feature) of the table has its own specific marginal distribu-
5:
x num + o (t)
6:
a₁ exp(- (tt(-atttt)))
tions, which requires the model to amortize its capacity adaptively across different fea-
7:
Sample x ~ q(xe Xxo oct)
Eq. (6)
8:
[xpum, xçat]
tures. We propose to adaptively learn a more fine-grained noise schedule for each feature respectively. To balance the trade- off between the learnable noise schedule's s flexibility and robustness, we design two function families: the power mean numerical schedule and the log-linear categorical schedule.
9:
Take gradient descent step on &.
10: until converged
Power-mean schedule for numerical features. For the numerical noise schedule o nu (t) in Eq. (3),
we define (um [a (L), with P; being learnable parameter for individual numerical features. For Vi {1, M ), ). we have a num (L)
4

---

Published as a conference paper at ICLR 2025
mum (1)
min
fiomix
T min
(10)
Log-linear schedule for categorical features. Similarly, for the categorical noise schedule o o (t) in Section 2.2, we define o o(at(t) (t) = [oxg+(t)) with & being a learnable parameter for individual categorical features. For Vj E {1, Meat) ). we have okg (t)(() as:
cat ak3 (t) = 1t)=1-t"
(11)
In practice, we fix the same initial and final noise levels across all numerical features so that (qum(() = min and (1) = max for Vi E (1,... Mmmm }- We similarly bound the initial and final noise levels for the categorical features, as detailed in Appendix B. This enables us to constrain the freedom of schedules and thus stabilize the training.
Joint objective function. We update Mnum + Ment parameters pl, :PMam and ki. KMest backpropagation without the need of modifying the loss function. Consolidating Lmum and Leas, we have the total loss with two weight terms Anom and Aeat as:
TTARDIF (0, k) = Anam (0, + Aeat (0,
Acat oi at
(E.-U(0.1) (...
Anam 114 mum (x,, t) <l12 [ +
(x:=m) m] log( (x,, f), x
(12) With the forward process defined in Eq. (3) and Eq. (6), we present the detailed training procedure in Algorithm 1. Here, we sample a continuous time step / from a uniform distribution (O, 1) and then perturb numerical and categorical features with their respective noise schedules based on this same time index. Then, we input the concatenated x,umm and x.a into the model and take gradient on the joint loss function defined in Eq. (12).
2.4 SAMPLING WITH BACKWARD STOCHASTIC SAMPLER
One notable property of the joint sampling process is that the inter- mediate decoded categorical fea- ture will not be updated anymore during sampling (see Eq. (8)). However, as tabular data are highly structured with complicated inter- column correlations, we expect the model to correct the error during sampling. To this end, we intro- duce a novel stochastic sampler by restarting the backward process with an additional forward process at each denoising step. Related work on continuous diffusions Kar- al. (2022); Xu et al. (2023) has shown that incorporating such stochasticity can yield better gen- eration quality. We extend such intuition to both numerical and cat- egorical features in tabular genera- tion. At each sampling step f, we first add small time increment to the current time step to i+ = t + 7et according to a factor Ye- and then perform the intermediate forward sampling between ++ and t by joint diffusion process Equations (3) and (6). From the increased-noise sample X,+. . we solve the ODE backward for num and at from + to t 1, respectively, with single update. This framework enables celf-correction by randomly perturbing decoded features in the forward step. We summarize the sampling framework in Algorithm 2, and provide the ablation study for the stochastic sampler in Section 4.4. We also provide illustrative example of the sampling process in Appendix C.
Algorithm Sampling
1: Sample X²"'
num ~ NO, MM0m)) eat m
for T to 1 do
3:
+ + t +-t+,t,n 1/T
> Numerical forward perturbation:
Sample euu ~ ((0, M
5:
x +- x + guuu (1-)² n (1) E mum
Categorical forward perturbation:
6:
Sample x (x(a* xxatt 1 a1+/aa1)
Eq. (6)
Concatenate:
7:
X, [xpumm x
Numerical backward ODE;
dum (xmum pg num (x+,,
num
x f- x num + (a num (t
σ numl (1+) Idx
t+
> Categorical backward sampling:
10:
Sample x ~ po (x(x cat cat
Ma (x++.
Eq. (8)
11:
end for
12: return mum cat
5

---

Published as a conference paper at ICLR 2025
2.5 CLASSIFIER-FRES FREE GUIDANCE CONDITIONAL GENERATION
TABDIFF can also be extended as a conditional generative model, which is important in many tasks such missing value imputation. Let
[ y be the collection of provided properties in tabular data, containing both categorical and numerical features, and let x denote the missing interest features in this section. Imputation means we want to predict x = {[x= x="]] conditioned on y. TABDIFF can be freely extended to conditional generation by only conducting denoising sampling for x,, while keeping other given features Ye fixed as y.
Previous works on diffusion models (Dhariwal & Nichol, 2021) show that conditional generation quality can be further improved with a guidance classifier/egressor p(y x). However, training the guidance classifier becomes challenging when x is a high-dimensional discrete object, and existing methods typically handle this by relaxing x as continuous (Vignac et al., 2023). Inspired by the classifier-fire guidance (CFG) framework (Ho & Salimans, 2022) developed for continuous diffusion, we propose a unified CFG framework that climinates the need for classifier and handles mixed-type x and y effectively. The guided conditional sample distribution is given by (x, ly) oc po(xely ly )oo )", where w > 0 controls strength of the guidance. Applying Bayes' Rule, we get
p yy )p(y)
po(x)ly))n p(y)
(x(x) (y) x pe (x:ly)ps (ylxx)" (xel))
(13)
Px(xx)
Px(x,)
We drop ply) for it does no depend on 0. Taking the logarithm of the probabilities, we obtain,
log (x ly) = (1 + w) log po yy) - w po(e).
(14)
which implies the following changes in the sampling steps. For the numerical features, 14 (x;, t) is
replaced by the interpolation of the conditional and unconditional estimates (Ho & Salimans, 2022):
f Xt, V, t) (1 + (x.. y) - wan (xx, t).
(15)
And for the categorical features, instead predict zo with pa (x [2, y), satisfying
log Pe (x. aa y). (1 w) log Pe (x (x,,
log po (x,""(x))
(16)
Under the missing value imputation task, our target columns is x, and the remaining columns constitute y. Implementing CFG becomes very lightweight, as the guided probability utilizes the original unconditional model trained over all table columns as the conditional model and requires only an additional small model for the unconditional probabilities over the missing columns. We provide empirical results for CFG sampling in Section 4.3 and implementation details in Appendix B.2.
RELATED WORK
Recent studies have developed different generative models for tabular data, including VAE-based methods, TVAE (Xu et al., 2019) and GOGGLE (Liu et al., 2023), and GAN (Generative Adversarial Networks)- methods, CTGAN (Xu et al., 2019) and TabelGAN (Park et al., 2018). These methods usually lack sufficient model expressivity for complicated data distribution. Recently, diffusion models have shown powerful generative ability for diverse data types and thus have been adopted by many tabular generation methods. Kotelnikov et al. (2023); Lee et al. (2023) designed separate discrete-time diffusion processes (Austin et al., 2021) for numerical and categorical features separately. However, they built their diffusion processes discrete time steps, which have been proven to yield a looser ELBO estimation and thus lead to sub-optimal generation quality (Song al., 2021; Kingma et al., 2021). To tackle such problem caused by limited discretization of diffusion processes and push it to a continuous time framework, Zheng & Charoenphakdee (2022); Zhang al. (2024) transform features into a latent continuous space various encoding techniques, since advanced diffusion models are mainly designed for continuous random variables with Gaussian perturbation and thus cannot directly handle tabular data. However, it has shown that these solutions either are trapped with sub-optimal performance due to encoding overhead or cannot capture complex co-occurrence patterns of different modalities because of the indirect modeling and low model capacity. Concurrent work Mueller et al. (2024) also proposed feature-wise diffusion schedules, but the model still relies on encoding to continuous latent space with Gaussian diffusion framework. In summary, none of existing methods have explored the powerful mixed-type diffusion framework in the continuous-tima limit and explicitly tackle the feature-wit wise heterogeneity issue in the mixed-type diffusion process.

---

Published as a conference paper at ICLR 2025
4 EXPERIMENTS
We
evaluate TAFDIFF by comparing it to various generative models across multiple datasets and metrics, ranging from data fidelity conduct ablation studies to investigate the effectiveness of each component of TIBDIFF, e.g., the learnable noise schedules.
and privacy downstream task performance. Furthermore,
4.1 EXPERIMENTAL SETUPS
Datasets.
We conduct experiments on seven real-world tabular datasets Adult, Default, Shoppers, Magic, Faults, Beijing, News, and Diabetes each containing both numerical and categorical attributes. In addition, each dataset has inherent machine-learning task, either classification or regression. Detailed profiles of the datasets are presented in Appendix A.
Baselines. We compare the proposed TABDIFF with eight popular synthetic tabular data generation methods that are categorized into four groups: 1) GAN-based method: CTGAN (Xu et al., 2019); 2) VAE-based methods: TVAE (Xu al., 2019) and GOGGLE (Liu al., 2023); 3) Autoregressive Language Model: GReaT (Borisov et al., 2023); 4) Diffusion-basse methods: STaSy (Kim et al., 2023), CoDi (Lee et al., 2023), TabDDPM (Kotelnikov al., 2023) and TabSyn (Zhang et al., 2024).
Evaluation Methods. Following the previous methods (Zhang et al., 2024), we evaluate the quality of the synthetic data using eight distinct metries categorized into three groups 1) Fidelity: Shape, Trend, a-Precision, 8- B-Recall, and Detection assess how well the synthetic data can faithfully recover the ground-truth data distribution; 2) Downstream tasks: Machine learning efficiency and missing value imputation reveal the models' potential to power downstream tasks; 3) Privacy: The Distance to Closest Records (DCR) score evaluates the level privacy protection by measuring how closely the synthetic data resembles the training data. We provide detailed introduction of all these metrics in Appendix A.2.
Implementation Details. All reported experiment results are the average of 20 random sampled synthetic data generated by the best-validated models. Additional implementation details, such as the hardware/software information well hyperparameter settings, are in Appendix D.
4.2
DATA FIDELITY AND PRIVACY
Shape and Trend. We first evaluate the fidelity of synthetic data using the Shape and Trend metrics. Shape measures the synthetic data's 's ability to capture each single column's marginal density, while Trend assesses its capacity to replicate the correlation between different columns in the real data.
The detailed results for Shape and Trend metrics, measured across each dataset separately, are presented in Tables 1 and 2, respectively. On the Shape metric, TAFDIFF outperforms all baselines five out of seven datasets and surpasses the current state-of-the-art method TABSYN by an average of 13.3%. This demonstrates TABDIFF's superior performance in maintaining the marginal distribution of individual attributes across various datasets. Regarding the Trend metric, TABDIFF consistently outperforms all basclines and surpasses TABSYN by 22.6%. This sigmificant improvement suggests that DIFD is substantially better at capturing column- column relationships than previous methods. Notably, TIFDIFF maintains strong performance in Diabetes, a larger, more aategorical-heav dataset, surpassing the most competitive baseline by over 35% on both Shape and Trend. This exceptional performance thus demonstrates our model's capacity to model datasets with higher dimensionality and discrete features.
Additional Fidelity Metrics. We further evaluate the fidelity metrics across a-precision, ß-recall, and CS2T scores. On average, TABDIFF outperforms other methods all these three metrics. We present the results for these three additional fidelity metrics in Appendix E.
Data Privacy. The ability to protect privacy is another important factor when evaluating synthetic data since we wish the synthetic data to be uniformly sampled from the data distribution manifold rather than being copied (or slightly modified) from each individual real data example. In this section, we use the Distance to Closest Records (DCR) score metric (Zhang et al., 2024), which measures the probability that synthetic example's nearest neighbor is from holdout the training set.
7

---

Published as a conference paper at ICLR 2025
Table I: Performance comparison on the error rates (%) of Shape.
Method
Adult
Default
Shoppers
Magic
Beijing
News
Diabetes
Average
CTGAN
16.84 0.03
16.8320.or
21.1520.10
9.81 10.0s
21 19.
16. 16.0920.0 02
9.8210.0s
15.99
TVAE
14.2222.00 os
10.17.0.00
20.51
8.25 os
19.1620.00
16.622o 03
18.86
15.97
GOGGLE
16.97
17.02
22.33
1.90
16.93
25.32
24.92
17.91
GRea'T
12.1220.0
19.9420.0s
14.5120.12
16. 16
8.25520.12
OOM
0OM
14.20
STaSy
11.2940.00 oG
5.77=0.06
9.37=0000
6, 6.29
6.71
6, 6.89
OOM
7.72
CoDi
21.38+0.00 os
15.77+ 0.07
31.84+0.00
11.56.0.26
16.91+0.02
32.27 22.27=00.04
21.13=0.26
21.55
TahDDPM
1.75
0.07
2.72200.1
20.00
1.30
78.751.. o1
31.44.0.06
16.93
TABSYN
0.81
1.01
1.4420.07
1.03g0.14
1.26220.0
2.06.1..0
1.85.20.02
TABDIFF
0.63.20.05
1.24=0.07
1.28
0.78
1.08
2.35
0.89 25
1.17
Improv.
22.2%
0.0%
11.11% 4
14.29%
18.25% 4
0% 4
46.39%
13.3% 4
TABSYN's performance is obtained via our reproduction. The results of other basclines except Diabctes, are taken from Zhang et al. (2024). The OOM entries are explained Appendix D.
Table 2: Performance comparison on the error rates (%) of Trend.
Method
Adult
Default
Shoppers
Magic
Beijing
News
Diabetes
Average
CTGAN
20.23+1.20
26.95+0..8
13.08
7.00
22.95
5.37
18.95
16.36
TVAB
14.1520.88
19.5010.95
18.6710.38
5.8210.49
18.01 10 os
6.1710.08
32.7440 26
16.44
GOGGLE
45.29
21.94
23.90
9.47
45.94
23:19
27.56
28.18
GReaT
17.50+0.22
70.02+0012
15.16+0.18
10.23
59. 00.60+0.00 55
OOM
OOM
44.24
STaSy
14.51
5.9610.00
8.4922.15
6.61 10:53
8.00
3.07
OOM
7.77
CoDi
22.4920.0s
68.41
17.78+0.11
6.53
7.07
11.10
29.21
23.21
TabDDPM
3.01#1.25
4.80
6.61#0.16
1.70+0.22
2.71=0.09
13.16=0.11
51.54+0.00
11.95
TABSYN
1.9320007
2.81.10.49
2.1329.10
0.881..11 18
3.1320.34
1.5240.03
3.9020.04
2.33
TABDIFF
1.4910.18
2.55 75
1.74:20.s
0.76
2.59
1.28=0004
2.20=0.16
1.80
Improve.
22.8% 4
9.3% 4
18.3% 4
13.6% 4
4.4% 4
15.8% 4
37.3% 4
22.6%
Due to space limits, the explanations for the additional fidelity metrics and data privacy metrics, along with the corresponding experiments, are deferred to Appendices A.2 and E.
4.3 PERFORMANCE ON DOWNSTREAM TASKS
Machine Learning Efficiency. A key advantage of high-quality synthetic data is its ability to serve as an anonymized proxy for real datasets and power effective learning on downstream tasks such as classification and regression. We measure the synthetic table's capacity to support downstream task learning via Machine Learning Efficiency (MLE). Following established protocols (Kim et al., 2023; Lee et al., 2023; Xu et al., 2019), we first split the real dataset into training and test sets, then train the given generative model on the real training set. Subsequently, we sample a synthetic dataset of equal size to the real training set from the models and use it to train XGBoost Classifier or XGBoost Regressor (Chen & Guestrin, 2016). 6). Finally, we evaluate these machine learning models against the real test set calculate the AUC score and RMSE for classification and regression tasks, respectively.
According to the MLE results presented in Table 3, TABDIFF consistently achieves the best or second-best performance across all datasets, with the highest average performance outperforming the most competitive baseline TABSYN by 15.0%. This demonstrates our method's ' competitive capacity capture and replicate key features of the real data that most relevant to learning downstream machine learning tasks. However, while TAEDIFF shows strong performance on MLE, we observe that methods with varying performance on data fidelity metrics might have very close MLE scores. This suggests that the MIE score evaluated under the current setting may not be reliable indicator of data quality. Therefore, we complement MLE with additional quality metrics Appendix E, which better highlights the superior performance of TABDUFF.
Missing Value Imputation. We further evaluate TABDIFF's conditional generation capacity through the Missing Value Imputation task. Following the approach in Zhang et al. (2024), we treat the inherent classification/eerrsssio task of each dataset as an imputation task. Specifically, for each table, we train generative models the training set to generate the target column while conditioning on the remaining columns. The imputation performance is measured by the model's accuracy in recovering the target column of the test Implementing classifier-free guidance (CFG) for this
8

---

Published as a conference paper at ICLR 2025
Table 3: Evaluation of MLE (Machine Learning Efficiency): AUC and RMSE are used for classifica- tion and regression tasks, respectively.
Adult
Default
Shoppers
Magic
Beijing
News'
Diabetes
Average Gup
Methods
AUC
AUC
A
AUC 1
RMSE
RMSE
A
%
Real
9277.000
7702.00s
.26
940
4233..00 .003
$422.000
0.0
CTGAN
.866 .02
696+..05
-875+000 000
855+.006
.90=.019 010
880#.016
569+..o 004
23.7
TVAE
.888= 004
.T242
.871
887
.770=.011
1.01
594 0o9
20.2
GOGGLE
.7782.012
58411.00
.6581.oo
.654
1.092.025
.8771.002
.475
42.1
GReaT
.93=.o0a
.755+.000
1002+0005
888+.008
.553+.01 018
OOM
OOM
13.3
STaSy
.90
.752:.000
.914
.844 .00s
.6562.014
$8712.002
OOM
10.9
CoDi
8871=000
.525+.000
.865
.92++
8818=.02
1.21#.005
.505
30.2
TabDDPM
990= 001
174
.9182.00
9356 003
59222.011
4.86
.521 a: 00s
11.95
TaBSyn
.909..0 001
7633 002
.914 00t
.977 .002
.5802.09
1.221.024
..84
6.78
TABDIFF
.12+.002
766++005
.91+ .044
.93++.00
.555+.013
.866+.021
689+ 0000
5.76
Table 4: Performance of TABDIFF in the Missing Value Imputation task. We draw direct comparison the generative approach employed by TABSYN, with the performance of XGBoost classifiers/re- gressors included as a reference.
Methods
Adult
Default
Shoppers
Magic
Beijing
News
Diabetes
Avg. Improv.
AUC T
AUC
AUC
AUC1
RMSE
RMSE
AUCT
Predicted by XGBoost
77.0
0.423
Impute with TABSYN
93.1
86.7
96.5
91.3
0.386
0.818
66.6
4.99
Impute with TASDUE CFG (w = 0.0)
92.5
91.6
95.7
92.5
0,121
0.828
66.0
3.76
Impute with TANDIIt CFG (w 0.6)
93.2
91.7
96.4
93.0
0.414
0.815
66.5 96.9
5.60
task is straightforward. We approximate the conditional model using the unconditioned TABDIFF trained on all columns from the previous unconditional generation tasks. For the unconditional model, train TABDIFF on the target column with significantly smaller denoising network. Detailed implementation is provided in Appendix D, and results are presented in Table 4.
As demonstrated, TABDIFF achieves higher imputation accuracy than TABSYN five of seven datasets, with an average improvement of 5.60% over the non-gencrative XGBoost classifier. This indicates TABDITF's superior capacity for conditional tabular data generation. Moreover, we empirically demonstrate the efficacy of our CFG framework by showing that the model consistently performs better with 0.6 compared
0.0 (which is equivalent to TABDIFF without CFG).
4.4 ABLATION STUDIES
Stochastic Sampler. We conduct ablation studies to assess the effectiveness of the stochastic sampler, discussed in Section 2.4. The results presented in Table 5. We use 'Det.' and 'Sto.' as abbreviations for deterministic and stochastic samplers. The deterministic sampler refers to the conventional diffusion backward process described in Song al. (2021); Karras et al. (2022), consisting of a series deterministic ODE steps. According to Table 5, under both fixed and learnable noise schedules, TABDIFF with the stochastic sampler consistently outperforms the deterministic sampler on the fidelity metrics Shape and Trend, regardless of whether learnable noise schedules are enabled. These confirm the efficacy of additional stochasticity in reducing decoding errors during backward diffusion sampling.
Adaptively Learnable Noise Schedule. Next, we perform an ablation study to evaluate the effective- ness of our adaptively learnable noise schedules. We compare the model with Iearnable schedules against the model with non-learnable noise schedules, where the noise parameters for numerical features are fixed to A = 7, Vi in Eq. (10) and, for numerical features, fixed to k 1, Vjj in Eq. (11). We refer to these models as 'Learn.' and 'Fix.', respectively. According to the results in Table 5, the learnable noise schedules substantially improve performance, particularly in Trend and regardless of whether the stochastic sampler is enabled. Furthermore, we closely examine the training process of both models the Adult dataset by plotting their changes of training loss in Figure 2. According to the figure, the learnable schedules (orange curves) significantly reduce both numerical and categorical losses in Eq. (12).

---

Published as a conference paper at ICLR 2025
Method
Shape
Trend
(A) Categorical Loss
(8) Numerical Loss
TABSYN
1.35
2.33
Learnale
Learaaile
Dee.
1.39
2.29
1.0
1.0
TABDIFF-Fix+4to.
1.20
0.8
0.8
TABDiH--Leam.4De.
1.24
1.92
TABDiaF--cam.+tto.
1.17
1.80
0.6
1000
0.6
4000
Epoch
Table 5: Ablation Studies on the stochastic sampler and learnable noise schedules.
Figure The adaptively learnable noise schedules reduce training loss.
Denalty
0.02
Data
0,050
Buth
Donstty
0.02
Pesl Data
Density
9.02
Roel Data
0.025
Beijing:
Beijing: DEWP
ng: DEWP
Beijing: DEWP
Density
Orts
Donaity
Donsity
Density
0.0
6.1
02
0.0
0.1
0.2
0.0
01
0.1
0.2
Shoppers: BounceRates
Shoppers: BounceRates
Shoppers: BounceRates
Shoppers: BounceRates
Count
22000
12866-
8025
4030
1000
6538
1066
2010
4200
2538
1026
1020
Adult: mtntta.status
Beling: month
Magic: class
Shoppers: Operatin pyystemn
Figure 3: Visualization of the marginal densities of the generated data in comparison to the real data. Top and Middle: individual numerical column; Bottom: individual categorical column.
4.5 VISUALIZATIONS SYNTHETIC DATA
We present a comprehensive set of visualiza- tions to compare single-column marginal dis- tributioos and pairwise correlations across four models-CoDi, TabDDPM, TABSYN, and our TABDIFF-and and four distinct datasets: Adult, Beijing, Magic, and Shoppers. Figure 3, we provide I-dimensional kernel density estimation (KDE) curves for chosen numerical feature, alongside histograms for chosen categorical feature. According to the figures, the density of TABDIFF's samples matches most closely with that the real data, highlighting TABD- IFF's ability to capture the original distribution patterns. Furthermore, in Figure 4, we include correlation heatmaps that show the correlation error rate for each pair of columns. These pic- tures consistently demonstrate that the TABDIFF archives the closest match to real correlation scores, highlighting its superior ability to capture the column-wise correlation of the real data.
CoDi
TabDDPM
TabSyn
TabDiff
Huum
Aduit
Cat
Beijing
Num
Gat
Magic
Num
Cat Num Col
Shoppers
Figure 4: Pair-wise correlation heatmaps. Values represent the eиror rate (the lighter, the better).
5 CONCLUSION
In this paper, we have introduced TABDIFF, mixed-type diffusion framework for generating high- quality synthetic data. TABDIFF combines hybrid diffusion process to handle numerical and categorical features in their native formats. To address the disparate distributions features and their interrelationships,
further introduced several key innovations, including learnable column-wise noise schedules and the stochastic sampler. We conducted extensive experiments using a diverse set of datasets and metrics, comprehensively comparing TABDIFF with existing approaches. The results demonstrate TABDIFF's superior capacity learning the original data distribution and generating faithful and diverse synthetic data to power downstream tasks.
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGMENT
We gratefully acknowledge the support of NSF under Nos. OAC-1835598 (CINES), CCF-1918940 (Expeditions), AMS-2327709 (IHBEM), IIS-2403318 (III): Stanford Data Applications Initiative, Wu Tsai Neurosciences Institute, Stanford Institute for Human-Centered AI, Chan Zuckerberg Initiative, Amazon, Genentech, GSK, Hitachi, SAP, and UCB. We also gratefully acknowledge the support of ARO INF-11111-) ONR (N00014-23-1-2559)) and Chan Zuckerberg Biohub. Minkai Xu thanks the generous support of Sequoia Capital Stanford Graduate Fellowship.
REFERENCES
Ahmed Alaa, Boris Van Breugel, Evgeny S Saveliev, and Mihaela van der Schaar. How faithful is your synthetic data? sample-level metries for evaluating and auditing generative models. In International Conference on Machine Learning, pp. 290-306. PMLR, 2022.
Samuel Assefa, Danial Dervovic, Mahmoud Mahfouz, Robert Tillman, Prashant Reddy, and Manuela Veloso. Generating synthetic data in finance: Opportunities, challenges and pitfalls. In Proceedings of the First ACM International Conference on Al in Finance, ICAIF '20. Association for Computing Machinery, 2021. ISBN 978 9781450375849.
Jacob Austin, Daniel D Johnson, Jonathan Ho, Daniel Tarlow, and Rianne Van Den Berg. Structured denoising diffusion models in discrete state-spaces. Advances in Neural Information Processing Systems, 34: 1 19981-17993 2021.
Vadim Borisov, Kathrin Sessler, Tobias Leemann, Martin Pawelczyk, and Gjergji Kasneci. Language models are realistic tabular data generators. In The Eleventh International Conference on Learning Representations, 2023.
Nitesh V Chawla, Kevin W Bowyer, Lawrence Hall, and Philip Kegelmeyer. Smote: synthetic minority over-sampling technique. Journal of artificial intelligence research, 16:321-351 2002.
Tianqi Chen and Carlos Guestrin. Xgboost: A scalable tree boosting system. In Proceedings of the
22nd ACM SIGKDD Conference on Knowledge Discovery and Data Mining, pp. 785 -944,999555 2016.
Prafulla Dhariwal and Alexander Quinn Nichol. Diffusion models beat GANs on image synthesis. In A. Beygelzimer, Y. Dauphin, P. Liang, and J. Wortman Vaughan (eds.), Advances in Neural Information Processing Systems, 2021.
Joao Fonseca and Fernando Bacao. Tabular and latent space synthetic data generation: a literature review. Journal of Big Data, 10(1):115, 2023.
Mikel Hernandez, Gorka Epelde, Ane Alberdi, Rodrigo Cilla, and Debbie Rankin. Synthetic data generation for tabular health records: systematic review. Neurocomputting, 493:28 45, 2022.
Jonathan Ho and Tim Salimans. Classifier-free diffusion guidance. arXiv preprint arXiv:2207.12598. 2022.
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. In Proceedings of the 34th International Conference on Neural Information Processing Systems, pp. 8840-6851, 2020.
Tero Karras, Miika Aittala, Timo Aila, and Samuli Laine. Elucidating the design space of diffusion- based generative models. In Proceedings of the 36th International Conference Neural Informa- fion Processing Systems, pp. 26565-2651 2022.
Jayoung Kim, Chaejeong Lee, Yehjin Shin, Sewon Park, Minjung Kim, Noseong Park, and Jihoon Cho. Sos: Score-based oversampling for tabular data. In Proceedings of the 28th ACM SIGKDD Conference Knowledge Discovery and Data Mining, pp. 762-772, 2022. 2022.
Jayoung Kim, Chaejeong Lee, and Noseong Park. Stasy: Score-based tabular data synthesis. The Eleventh International Conference Learning Representations, 2023.
11

---

Published as a conference paper at ICLR 2025
Diederik Kingma, Tim Salimans, Ben Poole, and Jonathan Ho. Variational diffusion models, Advances in neural information processing systems, 34:21696-21107 2021.
Akim Kotelnikov, Dmitry Baranchuk, Ivan Rubachev, and Artem Babenko. Tabddpm: Modelling tabular data with diffusion models. In International Conference on Machine Learning, pp. 17564- 17579. PMLR, 2023.
Chaejeong Lee, Jayoung Kim, and Noseong Park. Codi: Co-evolving contrastive diffusion models for mixed-type tabular synthesis. In International Conference on Machine Learning, pp. 18940-18956. PMLR, 2023.
Tennison Liu, Zhaozhi Qian, Jeroen Berrevoets, and Mihaela van der Schaar. Goggle: Generative modelling for tabular data by learning relational structure. In The Eleventh International Conference on Learning Representations, 2023.
Markus Mueller, Kathrin Gruber, and Dennis Fok. Continuous diffusion for mixed-type tabular data, 2024. URL https: / /arxivv.. wwwrriii.or//bbsaeeeeeeeee prgg/bs........oo..eee..oee/ htt////bb//2...../////a 2312. 1431.
Noseong Park, Mahmoud Mohammadi, Kshitij Gorde, Sushil Jajodia, Hongkyu Park, and Young- min Kim. Data synthesis based on generative adversarial networks. Proceedings of the VLDB Endowment, 11(10), 2018.
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High- resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF confer- ence on computer vision and pattern recognition, pp. 10684-10695, 2022.
Subham Sekhar Sahoo, Marianne Arriola, Yair Schiff, Aaron Gokaslan, Edgar Marroquin, Justin Chiu, Alexander Rush, and Volodymyr Kuleshov. Simple and effective masked diffusion language models. arXiv preprint arXiv:2406.075:: 2024.
Jiaxin Shi, Kehang Han, Zhe Wang, Arnaud Doucet, and Michalis K Titsias. Simplified and generalized masked diffusion for discrete data. arXiv preprint arXiv:2406 04329, 2024.
Jascha Sohl-Dickstein. Eric Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsupervised learning using nonequilibrium thermodynamics. In International conference on machine learning, pp. 2265-2265. PMLR, 2015.
Yang Song and Stefano Ermon. Generative modeling by estimating gradients of the data distribution. Advances in neural information processing systems, 32, 2019.
Yang Song. Jascha Sohl-Dickstein, Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-basee based generative modeling through stochastic differential equations. In The Ninth International Conference Learning Representations, 2021.
Clement Vignac, Igor Krawczuk, Antoine Siraudin, Bohan Wang, Volkan Cevher, and Pascal Frossard. Digress: Discrete denoising diffusion for graph generation. In The Eleventh International Confer- ence on Learning Representations, 2023.
Lei Xu, Maria Skoularidou, Alfredo Infente, and Kalyan Veeramachaneni. Modeling tabular data using conditional gan. In Proceedings of the 33rd International Conference on Neural Information Processing Systems, pp. 7335-7345, 2019.
Yilun Xu, Mingyang Deng, Xiang Cheng, Yonglong Tian, Ziming Liu, and Tommi Jaakkola. Restart sampling for improving generative processes. Advances in Neural Information Processing Systems, 36.76806-76838, 2023.
Jiaxuan You, Xiaobai Ma, Yi Ding, Mykel Kochenderfer, and Jure Leskovec. Handling missing data with graph representation learning. Advances in Neural Information Processing Systems, 33: 19075-19087, 2020.
Hengrui Zhang, Jiani Zhang, Zhengyuan Shen, Balasubramaniam Srinivasan, Xiao Qin, Chris- tos Faloutsos, Huzefa Rangwala, and George Karypis. Mixed-type tabular data synthesis with score- based diffusion in latent space. The Twelfth International Conference on Learning Representations, 2024.
12

---

Published as a conference paper at ICLR 2025
Shuhan Zheng and Nontawat Charoenphakdee. Diffusion models for missing value imputation in tabular data, arXiv preprint 1et::2220..1118,, 2022.
13

---

Published as a conference paper at ICLR 2025
A DETAILED EXPERIMENT SETUPS
A.1 DATASETS
We seven tabular datasets from UCI Machine Learning Repository': Adult, Default, Shoppers, Magic, Beijing, News, and Diabetes, where each tabular dataset is associated with a machine-learning task. Classification: Adult, Default, Magic, Shoppers, and Diabetes. Regression: Beijing and News. The statistics of the datasets are presented in Table 6.
Table 6: Statistics of datasets. Num stands for the number numerical columns, and # Cat
stands for the number of categorical columns. # Max Cat stands for the number of categories of the categorical column with the most categories.
Dataset
# Rows
Num
Cat
Max Cat
Train
# Validation
Test
Task
Adult
48, 842
28, 943
6.688
16, 281
Classification
Default
30, 000
24, 000
3,000
3, 000
Classification
Shoppers
12, 330
10
20
9,864
1,233
1,333
Classification
Magic
19. 019
10
15,215
1,902
1,902
Classification
Beijing
43, 824
31
35, 058
4,383
4,383
Regression
News
39, 644
31, 714
3, 3,965
3, 965
Regression
Diabetes
101, 766
27
716
61,
2,0353
20,354
Classification
A.2
METRICS
A.2.1
SHAPE AND TREND
Shape and Trend proposed by SDMetrics². They used to measure the column-wiss density estimation performance and pair-wise column correlation estimation performance, respectively. Shape uses Kolmogorov-Siman Test (KST) for numerical columns and the Total Variation Distance (TVD) for categorical columns to quantify column- density estimation. Trend uses Pearson correlation for numerical columns and contingency similarity for categorical columns to quantify pair-wise correlation.
Shape. Kolmogorov-Sirnov Test (KST): Given two (continuous) distributions (*) and (*(2) (r denotes real and denotes synthetic). KST quantifies the distance between the two distributions using the upper bound of the discrepancy between two corresponding Cumulative Distribution Functions (CDFs):
KST sup F (x) Fx(x)),
(17)
where Fx(r) and Fx(x) are the CDFs of (x) and p. (z), respectively:
| plz)dr.
F(x)
(18)
Total Variation Distance: TVD computes the frequency of each category value and expresses it as probability. Then, the TVD score is the average difference between the probabilities the categories:
TVD
Σ R(w)) S(w))
WEn
where w describes all possible categories in a column Ω. R(-) and SI-) denotes the real and synthetic frequencies these categories.
Trend. Pearson Correlation Coefficient: The Pearson correlation coefficient measures whether two continuous distributions are linearly correlated and is computed as:
Cov (2, w)
Px.y
(20)
'nttps://archive.. ics uci het..eddddddeeeeeeee datasets htttps://dccs.. devi sdmetrica
14

---

Published as a conference paper at ICLR 2025
where r and y are two continuous columns. Cov is the covariance, and ø is the standard deviation.
Then, the performance of correlation estimation measured by the average differences between the real data's 's correlations and the synthetic data's 's corrections:
Pearson Score = 1
v) p
(21)
where p=( y) and (0x, y)) denotes the Pearson correlation coefficient between column æ and column of the real data and synthetic data, respectively. As E [-1, 1]. the average score is divided by to ensure that it falls in the range of [0, 1]. then the smaller the score, the better the estimation.
Contingency similarity: For pair of categorical columns A and B, the contingency similarity score computes the difference between the contingency tables using the Total Variation Distance. The process is summarized by the formula below:
Contingency Score
Σ GEA BEB
Σ
RR..3 - Sa,99,
(22)
2
where and & describe all the possible categories in column A and column B, respectively. Ra.p and Sa,3 are the joint frequency of a and the real data and synthetic data, respectively.
A.2.2 a-PRECISION AND 8-RECALL
Following Liu et al. (2023) and Alaa et al. (2022), we adopt the a-Precision and 3-Recall proposed in Alaa al. (2022), two sample-level metric quantifying how faithful the synthetic data is. In general, a-Precision evaluates the fidelity of synthetic data whether each synthetic example comes from the real-data distribution, 3-Recall evaluates the coverage of the synthetic data, e.g., whether the synthetic data can cover the entire distribution of the real data (In other words, whether a real data sample close to the synthetic data.)
A.2.3 DETECTION
The detection measures the difficulty of detecting the synthetic data from the real data when they mixed. We use the classifer-twe ceaffer-two-sappee two-sample-tes test (C2ST) implemented by SDMetries, where a logistic regression model plays the role of detector.
A.2.4 MACHINE LEARNING EFFICIENCY
In MLE, each dataset first split into the real training and testing set. The generative models are leamed on the real training set. After training, synthetic set of equivalent size is sampled.
The performance of synthetic data on MLE tasks is evaluated based on the divergence of test scores using the real and synthetic training data. Therefore, we first train the machine learning model on the real training set, split into training and validation sets with is trained on the training set, and the optimal hyperparameter setting is selected according to the performance on the validation set. After the optimal hyperparameter setting is obtained, the corre- sponding classifier/regressor is retrained on the training set and evaluated on the real testing set. The performance of synthetic data is obtained in the same way.
ratio. The classifier/regressor
B METHOD DETAILS
B. ADAPTIVELY LEARNABLE NOISE SCHEDULES
For numerical stability, we need to bound Fmin and Fmax within (0, 1). As shown in Eq. (10), our formulation of the power-mean noise schedule boundaries the noise level in between amin and dmax- To make sure that the noise level for numerical features also bounded, we linearly map to the interval [8, 5), thus recasting Eq. (11) into
cat (t) log (1 ((1 5) - + +8)) kg
(23)
15

---

Published as a conference paper at ICLR 2025
B.2 CLASSIFIER GUIDANCE
In this section, we elaborate on how to implement our classifier-free guided conditional generation.
Simple way to compute (x5* y) We first show that, under simple masked diffusion framework, the guided posterior probability for categorical columns, pe Xx,y) y) can be simply computed by directly interpolating the model's raw estimates of Xo. I.C./48 cat (X y, t) and pag 1gth (Xt, t).
If X is already unmasked (i.e., x m), we remain at the current state as before. Otherwise, we compute the posterior according to Eq. (16). Note that all operations below are performed element-wise.
lo Xx, y) (1 w) log po (xx"""" Xx. y) log (x (x x1).
po (xx5" Xx. y
po (x,,))
po (xxaa )x)
(1 (1-a, m (a, (a.-a)) (xx (x).8,0) w+1
(a, (a))) (xc,.)
I-ax
((1 - 0g m + (a, a4 44 (X y, w
((1 a, ) + (a, (1 (xt,
Since fat and m must have zero probability mass in each other's dimension that can have a positive mass, the exponent of summations into the summation of exponents:
((1 - 0r.,) 0m))))))tt w+1 + ((a, - ( Xt. y. ()))) 1))
((1 0r. ) + ((a, (v,) (xe: ())
1-
By the same property, can perform division for and paga separately:
(1 ar m - (a, - az)) cat (x,, y, t))+t
1
peg (xe, 1)w0
1- a4
(1 02 m + (a, an) exp (1 + w) log 14 (X,, - w log cat (x,, 1))
az
Therefore, formulate po (x.:at [xe, y)
cat((x xca)))
x; cat * m,
pa (x y)
xaa, (1-oa)
2 (xc.t)
cat
1-ac
X2 = m,
where fzgat (xt,1) 1) can be simply computed as the interpolation of gagat (xx, y f) and pa (x1,
log jight (xx; () (1 w) log 1ig cat
(x,, () log 11 cat (xx;
C DETAILED ILLUSTRATIONS OF TRAINING AND SAMPLING PROCESSES
Training details. Algorithm outlines the training procedure for our hybrid diffusion model that jointly processes numerical and categorical variables. At each training iteration, we begin by sampling an initial data point Xy from the training data distribution p(xo) and a timestep uniformly from [o, 1]. Etcp.Each
Then, we perform the forward diffusion step. Each numerical feature is perturbed with Gaussian noise whose intensity determined by the o mt); each categorical feature flipped into the mask token with probability a; (i.e., eq. (6)). The noised numerical and categorical components concatenated together to form a noisy version x, of the table row. Lastly, we compute the training objective LTARDIFE and perform gradient descent on the model parameters 0, p, and k.
Sampling details. Here, we present a vivid visual example in Figure 5 to illustrate the backward sampling process described in Algorithm 2. Our example demonstrates generating table with
16

---

Published as a conference paper at ICLR 2025
Durstion (min)
zwod Bating
Durstion (mir)
IM00 Batting
Durchen (eaq)
IMG0 Batiig
6db
-4.
[MA]]]
232
-5.2
Actin
[MAbK]
Actinn
[MAXK]
-1898
990
32.0
[MASK]
-37
4.8
Horrer
None
199
1.5
Horrer
Nore
1.0l (more noiay)
0.0 (less neiay)
A- 1.0 (mone nolay)
1 0.0 (less neldy)
1.0 (mone nolay)
0.0 (less nolay)
Duration
Iwoo Ratine
Gurre
Avord
Durstton (eir))
ZWOO Rating
Durchex (ear)
IMOS Roting
(MA6))
161
19.9
[MASK]
-19
13.4
Romantic
[MASK]
91
12.0
Romantic
[MASK]
1002
0.0 (less nalay)
0.0 (less noiey)
(mors nokey)
(mors novay)
(less noiey)
(mone eaily)
Duration (win))
zwod Battnng
Avord
Durction (elr))
Iwod Batting
Aword
Dunehen (ear)
IMBO Bating
Aword
362
-3.0
[M54
[MASK]
409
-5.9
Action
[MASK]
150
6.5
Action
(MAR))
(MAR))
1246
21.9
Horrer
Nose
999
6
Homar
None
120
5.2
Horrer
Nore
= 1.0
1 0.0 (less noity)
1= 1.0 (more novay)
== 0.0 (less noidy)
1= 1.0 (more noiey)
I 0.0 (less noity)
(more novey)
Figure 5: A vivid visualization of TABDIFF'S generation process.
two numerical columns (movie duration and IMDB rating) and two categorical columns (genre and awards status). Each row represents an independent sample.
First, at t = 1.0 (the first frame). the numerical features are initialized with Gaussian noise, and all categorical components are masked. The algorithm then iterates backward through time steps from t = 1.0 to 0.0.
At each timestep 4, we first perform the forward stochastic perturbation step (the yellow section of Algorithm 2), the core of our stochastic sampler. All features are first perturbed forward to tt, slightly noisier state, following the same process as the forward step during training. While this step is not explicitly depicted in visualization in Figure it implies that, for instance, during the transition at the third frame, the unmasked "None" entry could be stochastically flipped back to the [MASK] state. This would then allow the model to re-predict the value, potentially yielding different result (e.g "Won"") than "None" in the subsequent frame.
After the stochastic perturbation, perform the denoising/unmaakkn Algorithm 2). For numerical features, we denoise to x by solving an ODE The update delta is determined by the normalized difference, dx="um, between the current state and the model's prediction, scaled by the decrease in noise levels. For categorical variables, we perform the unmasking step. Intuitively, if the column already unmasked, we stay in the current state. This is demonstrated in Figure 5, where the "None" entry persists once it has been flipped. If it is still masked, we flip the mask token to some valid value of that column with a certain probability (---1-9)) that increases as sampling proceeds. We choose which unmasked token to move to based on the model's 's predicted probability 14) gat tz) over all possible categories of the column.
step (the blue section of
D IMPLEMENTATION DETAILS
We perform our experiment on an Nvidia RTX A4000 GPU with 16G memory and implement TARDIFF with PyTorch.
17

---

Published as a conference paper at ICLR 2025
Data preprocessing. The raw tabular datasets usually contain missing entries. Thus in the first phase of preprocessing, we make up these missing values in the same way as existing works (Kotelnikov al., 2023; Zhang et al., 2024), with numerical missing values being replaced by the column average and categorical missing values being treated as a new category. Moreover, the diverse range of numerical features typically leads to more diffieult and unstable training. To counter this, we then transform the numerical values with the Quantile Transformer', and recover the original values using its inverse during sampling.
Data splits. For datasets other than Diabetes, we follow the exact same split as Zhang et al. (2024). Each dataset is split into the "real" and "test" sets. For the unconditional generation data fidelity and the imputation task, the models are trained on the "real" set and evaluated on the "real" set. For the MILE task, the "real" set further split into a training and validation set, and the "test" set is used for testing. Finally, for the data privacy measure DCR, the original dataset is equally split into two halves, with one being treated as the training set and the other as the holdout set.
genonitiooaalgnneattinn task on which
For Diabetes, split it into train, validation, and test sets with a ratio of 0.6/0.2/00 2.0.2/ For the MLE task. The training and test sets are regarded as the "real" and "test" sets for the unconditional generation and imputation tasks. For DCR, we apply equal split.
Architecture of the denoising network. In our implementation, we project each column individually to d dimensional vector using linear layer, ensuring that all columns treated with the same importance. We set the embedding size as 4, matching the size used in Zhang et al. (2024). We then process these projected vectors with two-layer transformer, appending positional encodings the end. The transformed vectors are then concatenated and passed through a five-layer MLP conditioned on the time embedding. Finally, the output is obtained by sequentially applying another transformer followed by projection layer that recovers the original dimensions. Our denoising network has comparable number of parameters as those experimented in TabDDPM (Kotelnikov et al.. 2023) and TABSyN (Zhang et al., 2024), as our shared MLP model accounts for most of the parameters.
Hyperparameters Setting. TABDIFF employs the same hyperparameter setting for all datasets. We train our models for 8000 epochs with the Adam optimizer. The training and sampling batch sizes are set to 4,096 and 10,000, respectively. Regarding the hyperparameters in TABDIFF, the values mii and d max are set to 0.002 and 80.0, referencing the optimal setting in Karras et al. (2022), and are set to 1e-3. For the loss weightings, we fix Arat. to 1.0 and linear decay Anum from 1.0 to 0.0 as training proceeds.
During inference, we select the checkpoint with the lowest training loss. We observe that our model achieves the superior performance reported paper with as few as 50 discretization steps (I' 50).
Details on OOMs experiment result tables:
1. GOOGLE set fixed random seed during sampling in the official codes, and we follow it for consistency.
2. GReaT cannot be applied on News for maximum length limit.
3. STaSy runs out of memory on Diabetes that has hight cardinality categorical columns
4. TabDDPM cannot produce meaningful content on the News and Diabetes datasets.
Imputation. As mentioned in Section 4.3, obtain the unconditional model of the target column by trannigg TAFDIFF on it with a smaller denoising network. For this network, we keep the same architecture but reduce the number of MLP layers to one.
E DETAILED EXPERIMENTS RESULTS
In the following sections, we discuss the result on a-precision, 3-recall, detection score (C2ST), and DCR in detail.
httrtp/:///ikiiiiiamm.ooo/eeeee emoduu http:///tttiimoooooeee/eeeeeeeee searr.. preprocessi
ig..eeseennnnn.... ng. Quant Quantt wettttieeeraaarrmmmmaa html
18

---

Published as a conference paper at ICLR 2025
E. ADDITIONAL FIDELITY METRICS
a-precision. We first evaluate TABDIFF on a-Precision score, metric that measures the quality of synthetic data. Higher scores indicate the synthetic data more faithful the real. We present the results across all seven datasets in Table 7. TABDIFF achieves the best or second-best performance on all datasets. Specifically, TABDIFF ranks first with an average a- P-Precision score of 98. 98,22, surpassing all other baseline methods.
3-recall. Next, we compare TABDIFF to the baselines on the 3-Recall scores, which evaluates the extent to which synthetic data covers the real data distribution. The results are presented in Table 8, with a higher score reflecting a more comprehensive coverage of the real data's 's feature space. TABDIFF consistently outperforms or matches the top- sop-performing baselines, achieving the highest average 3-Recall score of 49.40. This indicates that the generated data spans broad range of the real distribution. Though some baseline methods attained higher scores specific datasets, they fail to demonstrate competitive performance on a- a-Prreision, as models have to trade off fine-grained details in order capture broader range of features.
Overall, TABDIFF maintains a balance between broad data coverage and preserving fine-grained details. This balance highlights TABDIFF 's capability in generating synthetic data that faithfully captures both the breadth and depth of the original data distribution.
Detection Score (C2ST). Lastly, we assess the fidelity of synthetic data by using the C2ST test, which evaluates how difficult it is to distinguish the synthetic data from the real data. The results are shown in Table 9, where a higher score indicates better fidelity. TABDIFF achieves the best performance on five of seven datasets, outperforming the most competitive baseline model by 6.89% on average. Notably, TABDIFF excels on Diabetes, which contains many numerous high-cardinality categorical features (as indicated by # Max Cal in Table 6), showcasing its ability to generate high-quality categorical data. These results, therefore, demonstrate TABDIFF's capacity to generate synthetic data that closely resembles the real data.
Table 7: Comparison of a-Prreisioo scores. Bold Face highlights the best score for each dataset.
Higher scores reflect better performance.
Methods
Adult
Default
Shoppers
Mugic
Beijing
News
Diubetes
Averuge
Runking
CTGAN
77.74-0.1
62 0820 os
76.97.0.50
86 90 za
96 (6.27a0.. 14
96 16.9622.11 IV
79.89ko.11 to
82.
TVAE
98.17=0.17
35.57
58.19
86. 1990
97.200+0.00
86 =0
19.21+c.15
75.85
GOGGLE
50.68
68.89
90.88
23.09
70.81
GReaT
55.79-0.00
85. 90
78.88-00.1
85. 46
98. 32.022
0DM
0OM
80.87
STuSy
42.87_...0
90 10.48110.1
80.65
86. 26.56.10.0
16.1610.13
94.76_0.30
QOM
88.91
CoDi
77.58-0.4n
38..8.8
94.95-00.3
85. 33.01+o.as
98. 13
87. 87.15mm..12
64.80
84.29
TaPDDPM
96.36_n.30
97. 36
68.551n0.a
98. 18.59.1.11 17
07.9010.00
0.0010.00
28.35166:1
72.48
TABSYN
99.39:0.18
98.36.0.5a
90.42 as
97.5110.54
95 95.06:2030
06.61:0024
97.86
TARDINE
99.02200.0
08.4911009
99.11
99.47..0
98.06130:1
97. 07.30.1.11
00.69a0.19
98.22
Table 8: Comparison of R-Recall scores. Bold Face highlights the best score for each dataset. Higher scores reflects better results.
Methods
Adult
Default
Shoppers
Magic
Beijing
News
Diabetes
Average
Ranking
CTGAN
30.50_...0
18.2210.17
31.80
11.75
34.80.0.30
24.9720.29
9.42:0.26
23.11
TVAE
38.87_0.a1
23.1311
19.78.0.10
32 35.44
28.4 28.45511.00
29 661001
4.02
25.
GOGGLB
9.88
3.74
9.78
GReaT
19.122001
42.04 1o
44.902..17
34.9120
43.34g0 at
0OM
GOM
43:34
STaSy
29.21=0.54
39.31+o.3
37. 37.24=0.44
53. 57
51.7990.11
39. 42e0.22
OOM
42.32
Col
9:20120.1
19.9410 za
20.8220000
a1
52. 19..0.0..12 12
34.4020.11
2.7020.0
27.12
R&DDDMM
47.05-0.28
17. 17.834o.ss
17.79-0.25
48. 18.16++0.42
56.92+0.15
0.00+e.00
0.03-o.a
35.41
TABSYN
47.92_...0
46.4510.a5
19.10.0.00
48.031...0
59.1510.03
43.01100.s
33.7216.15
46.77
TAUDUN
51.4 $1.6420.2
51. 090 as
19.76au.8
48 48.0120aa1
59. 63:00
42 10.102
41.74 41.74=0.17
49.40
E.2 DATA PRIVACY.
Table 10 shows the DCR scores across all datasets. The DCR score represents the probability that generated data sample is more similar to the training set than to the test set, with a score closer to 50% being ideal, as it indicates balance between the similarity training and test distributions.
19

---

Published as a conference paper at ICLR 2025
Table Detection score (C2ST) using logistic regression classifier. Higher scores reflect superior
performance.
Method
Adult
Default
Shoppers
Magic
Beijing
News
Diabetes
Average
CTGAN
0.5949
0.4875
0.7488
0.6728
0.7531
0.6947
0.5593
0.6444
TVAE
0.6315
0.6547
0.2962
0.7706
0.8659
0.4076
0.0487
0.5250
GOGGLE
0.1114
0.5163
0.1418
0.9526
0.4779
0.0745
0.0912
0.3380
GReaT
0.5376
0.4710
0,4285
0,4326
0.6893
OOM
OOM
0,5118
STaSy
0.4054
0.6814
0.5482
0.6939
0.7922
0.5287
QOM
0.6083
CoDi
0.2077
0.4595
0.2784
0.7206
0.7177
0.0201
0.0008
0.3435
TabDDPM
0.9755
0,9712
0.8349
0.9998
0.9513
0.0002
0.1980
0.7044
TABSYN
0.9910
0.9826
0.9662
0.9960
0.9528
0.9255
0.5953
0.9156
TABDiFF 0.9950
0.9774
0.9843
0.9989
0.9781
0.9308
0.9865
0.9787
Improv.
0.40%
0.0% L
1.87%
0.0% 4
2.66% 4
0.57% +
65.71% 4
6.89% 4
Table 10: DCR score, which represents the probability that a generated data sample is more similar to the training set than to the test set. A score closer 50% more preferable. Bold Face highlights the best score for each dataset.
Method
Adult
Default
Shoppers
Beijing
News
Diabetes
STaSy
50.33% 10.19
50.23%
51.53%.0..
50.59% 10.29
50.59 10.114
OOM
CoDi
19.92%10.1
51.82% 10.26
51.06%
50.87% 10.11
50.79%
51.12% 10.19
TabDDPM
51.14%
52.15%
63.23%
80.11%
79.31%
+0.29
37.76%
TABSYN
50.14%
51.20%
52.90% +0.22
50.37 % =0.13
50.85%
+0.33
50.62%
TABDUFF
50.10 % +0.32
51.11 %0.0.30
50.24% =0.62
50.50% +0.36
51.04% #.92
50.43 % #0.18
Across the datasets, TABDIFF consistently achieves DCR scores near 50%, highlighting its ability to generalize well while maintaining fidelity to the original data distribution.
F STUDY OF MODEL EFFICIENCY AND ROBUSTNESS
In this section, we present a thorough analysis of TahDDPM's efficiency and robustness. For efficiency, we compare the training and sampling speed of TABDIFF against other competitive baseline methods (TabDDPM, TABSYN) using four different metries. For robustness, we first explore the tradeoff between discretization steps (i.e., efficiency) and sample quality in diffusion- based models. We then dig into detailed eттоr rates Shape and Trend to see whether models are biased towards learning some particular columns of datasets. Lastly, we discussed the robustness issues we found with the competitive baseline. We use the representative Adult dataset, which contains balanced number of numerical and categorical columns, throughout the experiment. Our results show that, among the competitive methods, TABDIFF is not only the most effective but also the most robust and highly efficient. Below, we present a detailed analysis.
F.1 EFFICIENCY
Training Time. We first measure the training time of each method. The epoch lengths are set to the default configuration of each method, and the validation frequencies are set the same value once every 200 epochs. Our measurements are presented in the first column of the Table 11, with the entry of TABSYN being split into VAE training time + Diffusion training time.
We that all three methods take comparable time to complete training run, with TABSyN being 10% faster than TAFDIFF and TABDIFF being 20% faster than TabDDPM. The current gap between TAEDIFF and TABSYN is likely due to TABDIFF 's slightly deeper network architecture compared to TABSYN. We believe that the architecture TABDIFF can be further optimized to improve efficiency, and we leave this as future work.
Nevertheless, is also important to consider model robustness when assessing efficiency. As highlighted in Appendix F.2, the training process for TABSYN is notably unstable due the difficulty
20

---

Published as a conference paper at ICLR 2025
2.5
(A) Shape
5.0
(8) Trend
TabODPM
Method
Train.T (min)
Sample.T (see)
NFE
2.0
TabSyn
4.0
TabSyn
TabDiff
TabiDfff
TabDDPM
112
120.1
TABSYN
45 + 40 85
8.840.005
2
TARDIFF
94
15.2200007
2
1.0
Table 11: Model Efficiency.
200
1000
2000
3000
4000
200
1000
2000
3000
4000
Epoch
Epoch
Figure Model Convergence speed.
in training VAEs, often requiring you to retrain many times in order produce model capable of generating samples comparable in quality to TABDIFF. On the other hand, TabDDPM fails to scale to more complicated datasets as shown by its poor performance News and Diabetes. Thus, when taking into account training robustness, DIFD is the most robust and efficient model among all competitive methods.
Training Convergence. Next, we assess training convergence by evaluating the quality of samples generated from intermediate checkpoints during the training process. Figure 6 plots result. The curves show that DIFF converges faster than the other methods, as TIBDIFF produces more high-quality samples when shown the datasets for the same number of times (i.e., at a same epoch).
Number of Function Evaluation (NFE). For sampling, we first theoretically analyze the number of network calls involved in a single diffusion step (i.e. the denoising step from 24 Tz-1)) for each method. The numbers are shown in the third column of Table 11. TABDIFF and TABSYN involve an extra network call because DIFF employs the second-order correction trick introduced in Karras et al. (2022).
Sampling Time. We empirically measure the time to generate the same number of samples as the test set (32561 samples for Adult). The numbers are presented in the second column of Table 11. According them, TABDIFF and TABSYN samples 10x faster than TabDDPM. This result is expected, both TABDIFF and TABSYN, by default, use 20 times fewer sampling steps than TabDDPM, while making twice as many network calls per step compared to TabDDPM. TABDIFF slightly longer sampling time is also attributed to the evaluation of its deeper network. We believe this can be optimized in future work.
F.2 ROBUSTNESS
Controling quality-efficiency tradeoff through discretization steps. One advantage of continuous- time diffusion models (which currently include TABDIFF and TABSYN) is the ability to sample with arbitrary discretization steps, allowing them to flexibly tradeoff sample quality with sample efficiency. We conduct an experiment that compares how TABDIFF and TABSyn perform when sampled with different discretization steps. The results and their visualizations are presented in Table 12 and Figure 7.
Our results show that TABDIFF consistently achieves higher sample quality across all levels of discretization steps. Notably, when the number of steps is reduced to just 5 (requiring only one second for sampling), TABSYN fails generate meaningful content, as indicated by an error rate approaching 50%. In contrast, TABDIFF continues to produce valid and coherent content even under these highly constrained conditions.
Evaluting volumn-wise learning bias To address what it means by "optimal allocation of model capacity across heterogeneous features," we analyze how generation quality varies between different features. The Shape and Trend errors presented in Table are averaged over all columns and columns pars. Now, using the representative Adult dataset, we visualize the errors at each column and column pair in Figures 8 and 9, along with the normalized standard deviations in Table 13 to quantitatively measure the variations of errors. All results show that TABDIFF with learnable schedules not only achieves lower average error but also exhibits more consistent errors across columns. This balance indicates that learnable schedules help the model balance its capacity on different dimensions of the data, improving the model's is ability handle heterogeneous distributions.
21

---

Published as a conference paper at ICLR 2025
TABSYN
TABDIFF
(A) Shope
(8) Trend
50
Tabsyn TabDiff
TabSyn
Steps
Shape
Trend
Shape
Trend
25
TabDir
5
34.09
49.30
12.51
22.15
Shape
Trend
10
1.99
3.92
1.55
3.36
20
0.84
1.96
0,62
1.50
0.81
1.95
0.63
1.49
100
0.82
1.94
0.64
1.53
25
50
100
3
10
25
100
Somple Steps (in log scalle
Sample Steps únn log scole)
Table 12: Ablate sample steps.
Figure 7: Visualize the sample step ablation
Shape Erro Distribution
1.4
Models
Fioed
Method
Shape
Trend
Shape Std.
'reend Std.
Shape
TABSYN
0.81
1.93
1.01
0.88
TAB TABDIFF-Fixed
0.74
1.73
0.42
0.75
TAB Leaa-
0.63
1.49
0.29
0.64
0.0
Column
Table 13: Evaluating column wise learning bias.
Figure 8: TABDIFF with learnable schedules has more balanced Shape performance.
Fixed
Learnable
Adult
2.0
0.0
Figure 9: TABDIFF with learnable schedules has more balanced Trend performance
Robustness issues of baselines. We identify several robustness issues with the competitive bascline methods. Specifically, TabDDPM struggles to scale larger datasets, and TABSyN 's performance is highly dependent on the training quality of VAEs, which can vary significantly across different runs.
TabDDPM: As shown by the results in Tables and 2, TabDDPM achieves poor performance larger datasets like News and Diabetes. This is because it failed to generate meaningful samples, as can see in Figure 11 that the numerical columns of TabDDPM's Diabetes samples all collapsed to the minimal and maximal values of the domains. After examining the training logs, we discovered that this poor generation performance might be due the explosion of training loss, as shown in Figure 10.
TABSYN: TABSYN is another competitive tabular generation model whose performance, our best knowledge, is closest to TABDIFF. However, this method has limitation: as mentioned in Zhang al. (2024), the quality of the VAE has significant impact on TABSYN 's performance, as it's the only component that recovers the original data shape. When reproducing TABSYN's result, we observed that across different training runs, the sample quality varies significantly. For poorly performing
attempted to retrain the diffusion model and even increased the number of training epochs, but these efforts did not improve the results. This confirms that the issue lies with the VAE.
To further illustrate this, we present the results of Shape and Trend that are averaged across 10 random training runs in Table 14 (Note: in the paper, follow the convention of previous works and reported results based on 20 different runs of the same checkpoint, and we put the result of the best reproduction for TABSYN). These additional results demonstrate that TAR achieves
22

---

Published as a conference paper at ICLR 2025
TabDDPM on Diabetes
3
Continuous Loss
Method
Adult
Default
Beijing
Discrete Loss
2
TABSYN (Shape)
1.31
1.17
2.69
Loss
TABSYN (Trend)
2.73
4..2
5.05 1.888
TARDIFF (Shape) TARDIFF (Trend)
0.65220.8
1.19=0.12
1.07=0.6 2.61..10.20
1.47:0.18
2.4620.62
2500
5000
7500
10000
Table 14: Models' consistency across different training runs.
Epoch
Figure 10: TabDDPM failed con- verge on Diabetes
Density
0.02
TabDDPM
0.05
2,00
0.00
0.00
0.00
100
Diabetes: #labProcedures
Diabetes: #medications
Diabetes: #outPatient
Real
0.50
Density
2.00
TabDDPM
0.00
0.00
0.00
10
Diabetes: #emergency
Diabetes: #inpatient
Diabetes: age
1.00
Real TabDDPM
Density
0.20
0.00
15
0.00
0.00
5
10
15
Diabetes: timeinHospital
Diabetes: #procedures
Diabetes: #diagnoses
Figure 11: TabDDPM failed to produce meaningful samples Diabetes
significantly more consistent performance across different training runs, as shown by the smaller average and standard deviation.
23


Reviews:
[{"review_id": "FOMNq4opT9", "reviewer": "Reviewer_gKXC", "summary": "While extensive work has attempted to solve the task of tabular data generation, they often face different problems, such as the failure of cross-modal modeling and the additional cost of training auto-encoders. To this end, this paper proposes TabDiff, a diffusion-based model that unifies modal heterogeneity within one single approach. The proposed method, built upon several existing techniques, models numerical and categorical attributes separately and introduces a learnable noise scheduler that further facilitates the sampling process. Extensive experiments verify the effectiveness of TabDiff. Ablation studies also provide additional insights.", "questions": "1. Do numerical and categorical value modeling help each other, or are they just modeled separately?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "1. The paper is well-written and well-structured. It is self-content and easy to follow.\n2. The problem investigated in this paper is well-motivated and practical.\n3. While most of the techniques have been investigated in prior work, this work seamlessly integrates them and adapts to the tabular data generation task.\n4. Extensive experiments sufficiently verify the effectiveness of the proposed approach, and ablation studies further outline the design choices and the benefits of the learnable noise schedulers.", "weaknesses": "1. Despite the claim of cross-modal modeling, the proposed method currently still models numerical and categorical values separately, which might make it difficult to synthesize more complicated tabular datasets. \n2. While I generally appreciate the effort and the performance, I am not fully convinced by the claim of the multi-modal diffusion framework. The experiments in Tables 1, 2, and 3 all suggest comparable performance compared to TabSyn. As TabSyn has been published in ICLR 2024, the authors should clearly state the main difference and closely compare the two works.\n3. Lack of motivation for schedulers. What is the motivation for choosing power-mean and log-linear schedulers? The choice looks random to me.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "PF6Ttw0tjq", "reviewer": "Reviewer_zTT5", "summary": "This work proposed a multi-modal diffusion framework for generating high-quality synthetic data. A key innovation is the development of a joint continuous-time diffusion process for numerical and categorical data, where authors propose learnable column-wise noise schedules to counter the high disparity of different feature distributions. Extensive experiments are conducted to demonstrate the superior capacity of proposed approach.", "questions": "See weakness.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "This manuscript demonstrates advantages in multiple aspects. It provides a clear and comprehensive introduction of the research problem, and conducts extensive experiments on multiple data sets and evaluation criteria.\n\nAdditionally, the paper elaborates on relevant algorithms and techniques. The algorithmic sections present some innovative and original findings, contributing to advancements in the field of machine learning and offering new insights to the academic community.", "weaknesses": "1. Some necessary explanations are lacking in the description of algorithm provided in the Method and Appendix, requiring more detailed descriptions to enhance the readability of the paper.\n\n2. The description of categorical column needs an example to illustrate the method more clearly.\n\n3. Formula (10) is missing a parenthesis.\n\n4. Why are formula (10) and formula (11) constructed this way? Is there a source? If not, can you explain the construction idea?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "WkV3VQR1Cs", "reviewer": "Reviewer_UD32", "summary": "This paper introduces an approach called TABDIFF, which develops a joint continuous-time diffusion process for mixed-type data in tabular datasets. The authors address the challenge of high disparity between different feature distributions by proposing feature-wise learnable diffusion processes. The key contribution is a hybrid diffusion process to handle numerical and categorical features.", "questions": "**Q1** The title of this paper is somewhat confusing. Before reading it, I assumed it referred to a diffusion model capable of generating various types of data structures, such as tabular data and images. After reading, I believe the term \"modality\" may refer to either of two scenarios: (1) multiple modalities within a specific variable (a column in a tabular dataset); (2) columns of a tabular dataset that are mixed-type. If the latter is the intended meaning, the paper should demonstrate how the proposed method differs from TabDDPM, as TabDDPM also handles mixed-type data generation in tabular datasets. It is inappropriate to use such an ambitious title.\n\n\n**Q2** The proposed method utilizes the masked diffusion for categorical features. What is the point of using masked diffusion?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "**1** The experiments are extensive, demonstrating the efficiency of the proposed method.\n\n**2** Hybrid diffusion models are employed to handle mixed-type data in tabular datasets.", "weaknesses": "**1** The authors claim that they explicitly tackle the feature-wise heterogeneity issue in the multi-modal diffusion process. However, the details about how to handle this problem is missing.\n\n**2** Even though experiments show the efficiency of the proposed method, it lacks of the motivation of considering a masked diffusion for categorical features.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "oybj3h13un", "reviewer": "Reviewer_kzHM", "summary": "The paper introduces TabDiff, an innovative multi-modal diffusion model designed to generate high-quality synthetic tabular data. It addresses the challenges of heterogeneous tabular data, which includes both numerical and categorical features with complex interdependencies. TabDiff employs a joint continuous-time diffusion process with feature-wise learnable components, allowing it to handle different data modalities while adapting to the varying distributions of individual features.", "questions": "See weaknesses, I will raise the score if all the concerns are addressed.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. Tabdiff introduces a unified joint continuous-time diffusion process with feature-wise learnable components for handling both numerical and categorical data is innovative and allows the model to adapt to the specific characteristics of different data distributions.\n2. The paper provides a thorough evaluation of TabDiff on a wide range of metrics, offering a deep and detailed assessment of the model performance from multiple dimensions, which is highly commendable.\n3. The paper emphasizes the importance of TabDiff in generating high-quality synthetic data that enables effective model training while ensuring privacy preservation. The authors validate this effectiveness through the use of various metrics, demonstrating that the model not only maintains data fidelity but also safeguards against privacy risks. It's a practical trial for addressing the growing need for privacy-conscious data generation in sensitive domains.", "weaknesses": "1. Although the paper does not explicitly highlight it, the complexity of TabDiff, particularly in handling high-dimensional features through diffusion, could result in increased computational costs compared to simpler models. It would be good to provide a comparison of generation or training speed metrics with other baselines to better illustrate this trade-off.\n\n2. The performance drop of TabDDPM on the News and Diabetes datasets is surprising, especially given that other diffusion-based methods maintain stable results. This discrepancy is beyond expected boundaries, with precision and recall even dropping to 0 in Table 7 and Table 8, and similarly poor performance across other tables for these datasets. \nThus, while I appreciate the authors' comprehensive comparison across various metrics with baselines, the paper lacks sufficient analysis and discussion of the experimental results, particularly in explaining the performance gaps between TabDiff and the baselines. It would strengthen the paper if the authors provided more detailed discussion, and offered insights into the experimental setup of the baselines.\n\n3. The paper lacks sufficient explanation of the experimental setup for these baselines, which makes it harder to interpret the results. Providing more clarity on these aspects would strengthen the case for TabDiff's performance and offer a more comprehensive understanding of the comparative results.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "WyuaiMPUtg", "reviewer": "Reviewer_Buu2", "summary": "This paper introduces TABDIFF, a joint diffusion framework which aims to handle data with both discrete and continuous data types. The authors validated the effectiveness of this framework through comprehensive experiments.", "questions": "NA", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1.The method introduces a new approach by jointly learning the diffusion processes for discrete and continuous features, which is not explored in the previous work.\n\n2. The adaptively learnable noise schedules look novel.\n\n3.The paper provides extensive experimental evaluations across different benchmark datasets, thoroughly validating the method’s performance.", "weaknesses": "In the title of this paper, it is claimed that this paper addresses multi-modal distributions. However, the paper does not clearly specify what multi-modal distribution it handles. Does it only refer to distributions with both discrete and continuous variables?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)