Continuous Diffusion for Mixed-Type Tabular Data
Authors: Markus Mueller
Kathrin Gruber
Dennis Fok
Venue: ICLR 2025 Poster (2025)
Keywords: synthetic data generation
diffusion model
generative model
tabular data
mixed-type data
Primary area: generative models
TL;DR: We propose a diffusion model for mixed-type tabular data that carefully balances generation of continuous and categorical features and show that accounting for feature heterogeneity in the design of adaptive noise schedules increases sample quality.

Abstract: Score-based generative models, commonly referred to as diffusion models, have proven to be successful at generating text and image data.
However, their adaptation to mixed-type tabular data remains underexplored. In this work, we propose CDTD, a Continuous Diffusion model for mixed-type Tabular Data. CDTD is based on a novel combination of score matching and score interpolation to enforce a unified continuous noise distribution for both continuous and categorical features. We explicitly acknowledge the necessity of homogenizing distinct data types by relying on model-specific loss calibration and initialization schemes. To further address the high heterogeneity in mixed-type tabular data, we introduce adaptive feature- or type-specific noise schedules. These ensure balanced generative performance across features and optimize the allocation of model capacity across features and diffusion time. Our experimental results show that CDTD consistently outperforms state-of-the-art benchmark models, captures feature correlations exceptionally well, and that heterogeneity in the noise schedule design boosts sample quality. Replication code is available at https://github.com/muellermarkus/cdtd.

Published as a conference paper at ICLR 2025
CONTINUOUS DIFFUSION FOR MIXED-TYPE TABULAR DATA
Markus Mueller! mueller@ese.eur. eur.n1
Kathrin Gruber gruber@ese eur .n1
Dennis Fok! dfokêese eur.nl
Econometric Institute, Erasmus University Rotterdam
ABSTRACT
score-based generative models, commonly referred to as diffusion models, have proven be successful at generating text and image data. However, their adaptation to mixed-type tabular data remains underexplored. Continuous Diffusion model for mixed-type Tabular Data. CDTD is based on a novel combination of score matching and score interpolation to enforce unified continuous noise distribution for both continuous and categorical features. We explicitly acknowledge the necessity of homogenizing distinct data types by relying on model-specific loss calibration and initialization schemes. To further address the high heterogeneity in mixed-type tabular data, we introduce adaptive feature- or type-specific noise schedules. These ensure balanced generative performance across features and optimize the allocation of model capacity across features and diffusion time. Our experimental results show that CDTD consistently outperforms state-of- the-art benchmark models, captures feature correlations exceptionally well, and that heterogeneity in the noise schedule design boosts sample quality. Replication code is available https: / thub. cwm/ ermarke rkur ccttd.
this work, we propose CDTD,
INTRODUCTION
Score-based generative models (Song et al., 2021), also termed diffusion models (Sohl-Dickstein al., 2015; Ho et al., 2020), have shown remarkable potential for the generation of images (Dhariwal & Nichol, 2021; Rombach et al., 2022), videos (Ho et al., 2022), text (Li et al., 2022; Dieleman et al., 2022: Wu et al., 2023), molecules (Hoogeboom et al., 2022), and other highly complex data structures with continuous features. The framework has since been adapted to categorical data in various ways, including discrete diffusion processes (Austin et al., 2021; Hoogeboom et al., 2021), diffusion in continuous embedding space (Dieleman et al., 2022: Li et al., 2022; Regol & Coates, 2023; Strudel et al., 2022), and others (Camphell et al., 2022; Meng et al., 2022; Sun et al., 2023). However, their adaptation mixed-type tabular data, which includes both continuous and categorical features, remains under explored. Existing models build directly on advances from the image domain (Kim al., 2023; Kotelnikov al., 2023; Lee al., 2023; Dolicoeur-Martinenr are not designed to deal with challenges specific to mixed-type tabular data: The type-specific diffusion processes and their losses are neither aligned nor balanced. naive combination of different losses can cause the generative model to favor the sample quality of some features or data types over others (Ma et al., 2020). Furthermore, tabular data often includes categorical features high cardinality, However, existing diffusion models for tabular data typically rely discrete diffusion framework to model one-hot-encoded categorical features (c.g., Kotelnikov et al., 2023; Lee et al., 2023). As consequence, these models do not scale well and fail to capture the full uncertainty during the denoising process, as a data sample can never be 'in-between' categories.
al., 2024) and, therefore,
Noise schedules determine the amount of noise each diffusion timestep and are typically defined manually (Nichol Dhariwal, 2021; Karras et al., 2022), but can also be learned (Dieleman et al., 2022; Kingma al., 2021). They crucial component in score-based generative models (Kingma

---

Published as a conference paper at ICLR 2025
et al., 2021; Chen al., 2023; Chen, 2023; Jabri et al., 2023; Wu et al., 2023) they aim to focus the model capacity on the noise levels most important to sample quality. However, despite their importance, previous work on diffusion models for tabular data simply adopted noise schedules from the image domain, which is not optimal. First, diffusion models for mixed-type tabular data inherently rely on combining type-specific diffusion processes. This makes it important, but difficult, to balance noise schedules across feature types. Unbalanced noise schedules negatively affect the allocation of model capacity. For instance, both TabDDPM (Kotelnikov al., 2023) and CoDi (Lee et al., 2023) use the discrete multinomial diffusion framework (Hoogeboom et al., 2021) to model categorical features. This induces different types of noise for continuous and categorical features, making alignment or comparison of noise schedules impossible. Second, the domain, nature, and marginal distribution can vary significantly across features (Xu et al., 2019). For instance, any two continuous features may be subject to different levels of discretization or skewness, even after applying common data preprocessing techniques; and any two categorical features may differ in the number of categories, or the degree of imbalance. The high heterogencity and lack of balance warrant a rethinking of fundamental parts of the diffusion framework, including the noise schedule and the effective combination of diffusion processes for different data types, rather than simply copying what has been working for images.
In this paper, we introduce Continuous Diffusion for mixed-fype Tabular Data (CDTD) address the aforementioned shortcomings. We combine score matching (Hyvärinen, 2005) with score inter- polation (Dieleman al., 2022) to derive score-based model that pushes the diffusion process for categorical data into embedding space, and thus enables a Gaussian diffusion process for both contin- uous and categorical features. This way, the different noise processes become directly comparable, easier to balance, and enable the application of, for instance, classifier free guidance (Ho & Salimans, 2022), accelerated sampling (Lu et al., 2022), and other advances to mixed type tabular data.
We counteract the high feature heterogeneity inherent to mixed-tyre -ype tabular data with distinct feature- or type-specific adaptive noise schedules. The learnable noise schedules allow the model to directly take feature or type heterogeneity into account during both training and generation, and thus avoid the reliance on image-specific noise schedule designs. Likewise, Shi et al. (2024) also propose the use of feature-specific noise schedules for tabular data but do not account for the necessity of homogenizing different data types. In contrast, we propose a diffusion-specific loss calibration and initialization scheme for effective data type homogenization. These improvements ensure a better allocation of our model's 's capacity across features, feature types and timesteps, and yield high quality samples. CDTD outperforms state-of-the-art baseline models across diverse set of sample quality metrics and datasets as well as computation time. Our experiments show that CDTD captures feature correlations exceptionally well, and that explicitly allowing for data-type heterogeneity in the noise schedules benefits sample quality.
In sum, we make several contributions specific score-based modeling of tabular data:
We propose unified continuous diffusion model for both continuous and categorical features such that all noise distributions are Gaussian.
We balance model capacity across continuous and categorical features with novel loss calibration, an adjusted score model initialization and adaptive type- or feature-specs speciffc noise schedules.
We suggest novel functional form to efficiently learn adaptive noise schedules, and to allow for an exact evaluation and incorporation of prior information on the relative importance of noise levels.
We drastically improve the scalability of tabular data diffusion models to high-cardinality features.
We are the first enable the use of advanced techniques, like classifier free guidance, for mixed-type tabular data directly in data space as opposed to a latent space.
2 SCORE-BASED GENERATIVE FRAMEWORK
We start with outlining the score- score-based frameworks for continuous and categorical features. Next, we combine these into single, unified model to learn the joint distribution of mixed-type tabular data.
2

---

Published as a conference paper at ICLR 2025
Retrieve noise levels
Nolslfy real data
Generate new data
joint score model
Continuous features
Cuse
Matching
(9) (/ (0.
Adastvo
Noie
Faek
Score
Categorical features
10 Mi0.
Cem
P -d
Score Interpoletion
Figure 1: CDTD framework. Adaptive noise schedules are trained to fit the (possibly aggregated) MSE and CE losses and transform the uniform timestep / to potentially feature-specific noise level to diffuse ("noisify") the scalar values (for continuous features) or the embeddings (for categorical features). Associated sampling processes are highlighted orange. The approximated score functions concatenated and passed to ODE solver for sample generation.
2.1 CONTINUOUS FEATURES
We denote (6) E R the i-th continuous feature and Xy Xeom R Kou the stacked feature vector. Further, let {x, t==1 be diffusion process that gradually adds noise in continuous time t [0, 1] Xo. and let (x(x) denote the density function of the data time t. Then, this process transforms the real data distribution (x(x) into terminal distribution of pure noise pi (x) from which we can sample. Our goal learn the reverse process that allows us go from noise X1 Pı (x) to a new data sample x (x).
The forward- pass this continuous-ting time diffusion process is formulated the solution to stochastic differential equation (SDE):
dx f(x, t)dt + g())dw,
where f(-,t): R - R Kom is the drift coefficient, g(-) R R is the diffusion coefficient, and a Brownian motion (Song et al., 2021). The reversion yields the trajectory of x as t goes backwards in time from 1 to O, and is formulated as probability flow ordinary differential equation (ODE):
1 [f(x - (1) log (x)]ddt
dx
We approximate the score function Vx log lo the only unknown in Equation (2), by training time-dependent score-based model se (x, t) via score matching (Hyvärinen, 2005). The parameters trained to minimize the denoising score matching objective:
E, Mx, Ex, xxo ||so(x,, Vx, log Pou (x, xx))
(3)
where A : [0, + R4 is positive weighting function for timesteps ~ Mj..1. and (x, (xo) is the density of the noisy x, given the ground-truth data Xa (Vincent, 2011).
In this paper, we use the EDM formulation (Karras et al., 2022), that is, f(: = 0 and zero
g(t) = 21 such that por(x:|x0) = ()) We standardize mean and unit variance. Then, for sufficiently large d²(1), we can start the reverse process with sampling X1 ~ (x()) = A/0 I We then gradually guide x1 towards high density regions in the data space with so(x replacing the unknown, true score function in Equation (2).
2.2 CATEGORICAL FEATURES
Let a E [I,....,C,, C2} be the j-th categorical feature with C2 distinct classes. We learn a feature- (f) specific encoder represent each category d-dimensional vector e Ene, (x(g) Further,
3

---

Published as a conference paper at ICLR 2025
let X
(
{(e ec be the noiseless embedding at = 0. To unify the diffusion frameworks for
categorical and continuous data as much as possible. we base both on the same Gaussian-tys coussian-type noise. Thus, instead of adding noise to the categorical variable directly, we add noise to the embedding
x(") Poe (5) () = -xxx xx)) (1)Taa such that x(() pn (x (0 (1)Ta) analo-
gous to score matching.
For categorical data, denoising score matching (Equation (3)) is not directly applicable learn V x log Pos (x( |x,"). since the score can only take on Cy distinct values. To proceed, we transform the score matching approach into discrete choice problem. Note that for given t and x it is sufficient to find E Ixxx
[V
x(2) logPoe Poe (x, (1())))))) )x as it minimizes Equation (3). Presuming
Gaussian noise, have
log por (x( (x) 1 o²(() P px4 x -
x
We can thus approximate the score by computing the probability-weighted aver- age of the Cj possible embedding vectors, that is, x! -p(x pxf2 .g1xx" [x Since P(x = e
exi (x() t) = P( = , tt,, we can estimate P(x xx t) via a classifier that predicts the C class probabilities and is trained to minimize the cross-entropy (CE). This
procedure effectively interpolates between the C2 ground-truth embeddings (5) and ís therefore known as score interpolation (Dieleman et al., 2022).
This framework casily extends to multiple features. Most importantly, Enc, is trained alongside the model such that X is directly optimized for denoising the data. During sampling, the model only has to commit a category at the final step of generation, i.e., we allow for smooth, continuous transition between states. This is unlike multinomial diffusion (Hoogeboom et al., 2021), which imposes discrete transitioning steps and is the framework used by several existing diffusion models for tabular data (Kotelnikov et al., 2023; Lee et al., 2023). By defining diffusion for categorical data in embedding space, we allow our model take uncertainty at intermediate timesteps fully into account, which improves the consistency of the generated samples (Dieleman et al., 2022). Thus, CDTD can more accurately capture subtile dependencies both within and across data types.
METHOD
To model the joint distribution of mixed-type data, we combine score matching (Equation (3)) with score interpolation (Equation (4)). Next, discuss the important components of our method. In par- ticular, the combination of the different losses for score matching and score interpolation, initialization and loss weighting concerns, and the adaptive type- or feature-specific noise schedule designs.
3.1 GENERAL FRAMEWORK
Figure 1 gives an overview of our Continuous Diffusion for mixed-rype Tabular Data (CDTD) framework. The score model is conditioned on (1) the noisy continuous features, (2) the noisy embeddings of categorical features, and (3) the timestep f. It predicts the ground-truth value z for continuous features and the class-specific probabilities P(x:(a) c) for categorical features. Additional conditioning information is straightforward to add. Note that while the Gaussian noise process acts directly on continuous features, it acts the embedded categorical features x5") This way, we ensure a unified continuous noise process for both data types. Further details on the implementation are provided in Appendix K. During generation,
concatenate the score estimates,
g00 and (55) for all features and j before passing them to an ODE solver. Our model the first to utilize feature- or type-specific noise schedules also during the generation of tabular data, which allows the sampler to take feature- or type-specift steps. Details on the sampling process and the algorithm given in Appendix I..
4

---

Published as a conference paper at ICLR 2025
3.2 HOMOGENIZATION OF DATA TYPES
Let LMSE (accome 1) be the time-weighted MSE (i.e., score matching) loss of the i-th continuous feature single timestep and LCE(=cat ()) f) the CE (i.e., score interpolation) loss of the j-th categorical feature. Naturally, the two losses are defined on different scales. This leads to an unintended importance weighting of features in the generative process (Ma et al., 2020). To solve this, we observe that an unconditional generative model should a priori, i.e., without having any information, be indifferent between all features. For diffusion models this reflects the state of the model the terminal timestep 1.
Definition (Calibrated losses), Scaled losses LissE and Cev are called calibrated if
ElCMsE (Feomt 1)] = El[Ce( 1)]
for all continuous features i and categorical features j.
Assumption (No information at t t= =), The noise level (1) is high enough such that the input to the score model cannot be distinguished from noise, i.e., the model has information.
Proposition 1 (Homogenization of ceature-specffie losses). Under Assumption 1, if each Z has unit variance and Z is the feature-specitee entropy of categorical feature j. then the losses LiMSE LLSS and fez = -cr(xvd 1) are calibrated for all continuous features i and categorical features j- See Appendix B for a detailed proof.
Given Assumption and Proposition we can derive joint loss function without unintended importance weighting by averaging the K Keont Keat calibrated losses at given t:
((1) Kan
Kun Σ
LMSE (J ()
Cee
Implications for the score model initialization. The loss calibration and the multi- collii-modality of the data have implications for the optimal initialization of the score model. To match the loss calibration, we aim to initialize the model such that all feature-specific losses are one. We therefore initialize the output layer weights and biases for continuous features to zero and rely on the timestep weights of the EDM parameterization (Karras et al., 2022) to achieve unit loss for all t. For the categorical features, we initialize the biases to match each category's 's empirical log probability the training set (see Appendix C for details).
Weighting across time. The initial equal importance across all will change over the course of training. To allow for changes in the relative importance among features but ensure equal importance of all timesteps throughout training, we employ a normalization scheme for the average diffusion loss (Karras et al., 2024; Kingma & Gao, 2023). Specifically, learn the time-dependent normalization Z (t) such that (t) = 1. This ensures consistent gradient signal and can be implemented by training a neural network predict C(t) alongside our diffusion model (see Appendix for details).
3.3 NOISE SCHEDULES
Evidently, the noise schedule of one feature impacts the optimality of noise schedules for other features, and different data types have different sensitivities to additive noise. For instance, given the same embedding dimension, more noise may be needed to remove the same amount signal from embeddings of features with fewer classes (see Appendix A). Likewise, delayed noise schedule for one feature might improve sample quality as the model can rely on other correlated features that have been (partially) generated first. Therefore, introduce feature-speciffe type-specific noise schedules. We make the noise schedules Icarnable, and therewith adaptive to avoid the reliance on designs for other data modalities.
We investigate the following noise schedule variants: (1) a single adaptive noise schedule, (2) adaptive noise schedules differentiated per data type and (3) ceature-specific adaptive noise schedules. We only introduce the feature-specine noise schedules explicitly. The other noise schedule types are easily derived from argument by appropriately aggregating terms across features.
5

---

Published as a conference paper at ICLR 2025
0.1
02
-0
0.4
unform
eveneeigt low noss loels (er, < (0.0) by factor three
Fa. lop.
Sn. log
Fa. Kos.k
1.00
0.00
0.0
0.9
0.5
1.0
1.0
62. scaled [o.1]
23 scaled [0.1]
2% scaled [0.1]
2g scaled [0.1]
Figure 2: (Left) pdf (fa. 2) and edf (Fd.a.log.k) ,.) of the domain-adapted Logistic distribution for five different values of the location parameter μa and for a given curve steepness Vi:- = 3. (Right) impact of uniform vs. adjusted timewarping initialization on the pdf (fá.a.log.k) and the cdf (oa...
Feature-specifiic noise schedules.
Following Equation (1), we define the diffusion process of xf as
d = V2 dt Ncam,(t) (t)dur
dx coo
(
and likewise the trajectory of Xeat as
V d heats heat (f)dw,
(7)
( Rd is the embedding of ()) in Euclidean space. The noise schedules haom, where Xeat and ( (t) represent the feature-specific standard deviations ctm and (ta), (t) of the added
Gaussian noise, respectively. Thus, each feature is affected by distinct adaptive noise schedule. On the other hand, type-specific noise schedules involve only two functions, heat (t) and heom such that all features of the same type are affected by the same noise schedule.
Adaptive noise schedules. We aim to learn a noise schedule h t ++ a for each feature k. Inspired by Dieleman al. (2022), we learn function F& that predicts the feature-specific (not explicitly weighted) loss & given the noise level dg. By normalizing and inverting FR. we achieve Fx This encourages the relation between t and Eg to be linear.
the mapping of interest h4
Higher noise levels imply a lower signal-to-noise ratio and therefore a larger incurred loss for the score model. Accordingly, F3 must be a monotonically increasing and S-shaped function. We let FR = 7t F...a..o,,,..g where "/ > 0 a scaling factor that enables fitting loss l > / near the start of training, and a loss & < 1 in case conditioning information is included. Further, we use the cdf the domain-adapted Logistic distribution Fd...log..(or, where the input is pre-processed via a Logit function, with parameters 0 < μa < (the location of the inflection point) and ex 2 1 (the steepness of the curve). Note that with pre-specified minimum and maximum noise levels, we can always scale de to lie in j0, 1]. Figure 2 illustrates the effect of the location parameter. The implicit importance of the noise levels is conveniently represented by the pdf fa.a.ogg To normalize and invert FR. we set = 1 and directly use the closed form quantile function F log, The detailed derivation of all relevant functions is given in Appendix E. To avoid biasing the noise schedule to frequently sampled timesteps during training, we derive importance weights from fa.a.log,. when fitting /i- We use the adaptive noise schedules during both training and generation, and give examples of learned noise schedules in Appendix P.
Our functional choice has several advantages. First, each noise schedule can be evaluated exactly without the need for approximations and only requires three parameters. Second, these parameters are well interpretable in the diffusion context and provide information on the inner workings of the model. For instance, for μl < H2g the model starts generating feature before feature 1 in the reverse process. Third, the proposed functional form is less flexible than the original piece-wise linear function (Dieleman et al., 2022) such that an EMA on the parameters is not necessary, and the fit is more robust to "outliers" encountered during training. This is crucial when using feature-specific specification as opposed to a single noise schedule.
3.4 ADDITIONAL CUSTOMIZATION TO TABULAR DATA
In the diffusion process, add noise directly to the continuous features but to the embeddings of categorical features. We generally need more noise to remove all the signal from the categorical rep- resentations. We therefore define !ype-specific minimum and maximum noise levels: For categorical

---

Published as a conference paper at ICLR 2025
features, let min mont = 80.
0 and mat, 100; for continuous features, set min 0 and
Lastly, uninformative initialization of the adaptive noise schedules requires μ4 0.5, My x 1 and Th: = 1 such that lo...log.k corresponds approximately to the edf of a uniform distribution. We can improve upon this with more informative prior: As opposed images, in tabular data the location of features in the data matrix, and therefore the high-level structure, is fixed. Instead, we are interested in generating details as accurately as possible. Note that the inflection point, Pkx corresponds the proportion of high noise levels (i.e.. ok 2 0.5 (amax + min)) ) in the distribution. Therefore, we empirically choose μı = 1/4 such that low noise levels (i.e., 0k < 0.5 (o + mmin)) are initially presumed to be three times more important than high noise levels (see Figure 2). We let vy z 1 for dispersed initial probability mass and initialize the scaling factor to 7/k = 1.
EXPERIMENTS
We benchmark our model against several generative models across multiple datasets, Additionally, we investigate three different noise schedule specifications: (1) single adaptive noise schedule for both data types (single), (2) continuous and categorical data type-specific adaptive noise schedules (per type), and (3) feature-specific adaptive noise schedules (per feature).
Baseline models. We use a diverse benchmark set of state-of-ther generative models for mixed-type tabular data. This includes SMOTE (Chawla et al., 2002), ARF (Watson al., 2023), CTGAN (Xu et al., 2019), TVAE (Xu et al., 2019), TabDDPM (Kotelnikov et al., 2023), CoDi (Lee al., 2023) and TabSyn (Zhang et al., 2024). Each model follows different design and/or modeling philosophy. Note that CoDi is an extension of STaSy (Kim et al., 2023, the same group of authors) that has shown to be superior in performance. For scaling reasons, ForestDiffusion (Jolicoeur- Martineau Molicoeur-Martineau et al., 2024) is not an applicable benchmark.' Further details on the respective benchmark models and their implementations are provided in Appendix G and Appendix H. respectively. We provide an in-depth comparison of CDTD to the diffusion-based baselines in Appendix O. To keep the comparison fair, we use the same internal architecture for CDTD as TahDDPM (which was also adopted by TabSyn), with minor changes to accommodate the different inputs (see Appendix K).
Datasets. We systematically investigate our model on 10 publicly available datasets (see Appendix for details). The datasets vary in size, prediction task (regression vs. binary classification"), number of continuous and categorical features and their distributions. The number of categories for categorical features varies significantly across datasets. We remove observations with missings in the target or any of the continuous features and encode missings in the categorical features as separate category. All datasets are split in train (60%), validation (20%) and test (20%) partitions, hereinafter denoted Durain D and Diest, respectively. For classification tasks, we use stratification with respect to the outcome. We round the integer-valued continuous features after generation.
4.1 EVALUATION METRICS
In our experiments, we follow conventions from previous papers and use four sample quality criteria, which we assess using a comprehensive set of measures. All metrics are averaged over five random seeds that affect the sampling process of synthetic data D gen of size min([Pmmin)) 50 000).
Machine learning efficiency. We follow the conventional train-synthetic-test-real strategy (see, Borisov et al., 2023; Liu al., 2023; Kotelnikov et al., 2023; Kim et al., 2023; Xu et al., 2019; Watson et al., 2023). We train logistic/ridge regression, a random forest and catboost model, on the data-specific prediction task (see Appendix for details) and then compare the model-averaged
Vatiiceur-Martineau et al. (2024) report that they used 10-20 CPUs with 64-256 GB of memory for datasets with a median number of 540 observations. With the suggested hyperparameters (for improved efficiency) and the small nmes data. Note that the model estimates
64 CPUs. the model took approx. 500 min of training K" separate models, with K heing the number of features and T the noise levels. Therefore, we consider FifestDiffusion be prohibitively expensive for higher-dimensional data generation.
For ease of presentation, only analyze hinary targets. However, CDTD trivially extends to targets with multiple classes.
7

---

Published as a conference paper at ICLR 2025
real test performance, Perf(Duuin uain Dua), to the performance when trained the synthetic data, Perf(( gen: Dust). The results are averaged over ten different model seeds (in addition to the five seeds for the sampling). For regression tasks, consider the RMSE and for classification tasks, the macro-averaged F1 and AUC scores. We report [Perf( gen; Dres) Perf(D Diea)l- An absolute difference close
is preferable since then synthetic and real data induce the same performance.
Detection score. For each generative model, we report the accuracy of a catboost model that is trained to distinguish between real and generated (fake) samples (Borisov al., 2023; Liu et al., 2023; Zhang et al., 2024). First, we subsample the real data subsets, Dusin, Deali and Duss. to a maximum of 25 000 data samples limit evaluation time. Then, we construct Dieeect Dektect
train valid and Ddetcer with equal proportions of real and fake samples. We tune each catboost model on valid and report the accuracy of the best-fitting model on Dilskes (see Appendix I for details). A (perfect) detection score of 0.5 indicates that the model is unable to distinguish fake from real samples.
Statistical similarity. Similar to Zhang et al. (2024), we assess the statistical similarity between real and generated data at both the feature and sample levels. We largely follow Zhao et al. (2021) and compare: (1) the -hasnn-Shnnoon divergence (ISD; Lin, 1991) to quantify the difference in categorical distributions, (2) the Wasserstein distance (WD: Ramdas et al., 2017) to quantify the difference in continuous distributions, and (3) the L2 distance between pairwise correlation matrices. We use the Pearson correlation coefficient for continuous features, the Theil uncertainty coefficient for two categorical features, and the correlation ratio for mixed types.
Privacy. We compute the distance to closest record (DCR) as the minimum Euclidean distance of generated data point to any observation in Dunin (Borisov et al., 2023; Zhao et al., 2021). We one-hot encode categorical features and standardize all features to zero mean and unit variance ensure each feature contributes equally to the distance. We compute the average DCR as a robust estimate. For brevity, we report the absolute difference of the DCR of the synthetic data and the DCR of the real test set. A good DCR value, indicating both realistic and sufficiently private data, should be close to zero.
4.2 RESULTS
Table I: Average performance rank of each generative model across eleven datasets. Per metric, bold indicates the best, underline the second best result. We assigned the rank 10 for CoDi on lending and aiabtes TabDDPM on acs i ncome and di abetes, SMOTE on acsi n come and covertype. RMSE, FI, AUC and DCR are measured in abs. differences to the real test set.
CDTD (single)
CDTD (per type)
CDTD (per feature)
SMOTE
ARF
CTGAN
TVAE
TabDDEM
CoDi
TabSyn
RMSE
3.64+3
3.4+2.1
8.0ee1
7,8+2.2
8.4e10
7.0e1.
7.2+La
4.0e1.7
2.6-1.0
3.215
1.2429
6.242.5
8.0214
8.0g1.
4.3
6.5aa.0
8.2
1.0
2.3
3.5,14
AUC
4.3 4.3+3.
5.7+1.
8.3e12
7.8e1.1
4.3s.
6.8ea.1
8.2+1.
3.3
2.7=1.a
3.7+12
dist. con.
5.0
5.742.a
8.3219
7.9214
6.045.
7.0a2.
7.1a12
3.1211
2.0.o.s
28,11
Detection score
5.7
6.1.1.7
8.8.1.5
7.6g
4.813
7.929.4
6.221
3.1
1.6_10
3.3g11
JSD
7.2+222
1.2404
7:6424
8.821.0
6.942.1
7.2218
6.8
2.5.a7
2.8
4.1009
WD
3.2.3.
6.8114
7.4g94
7.9.1.
6.613s
8.4218
5.8190
4.b114
3:221
3.421
DCR
5.812.7
6.51a.
8.413.7
6.1139
4.5134
6.618.4
6.31sm
4.1
3.0.2a
3.0120
Table 1 shows the average rank of each generative model across all datasets for the considered metrics. Detailed results (including standard errors) for each model and dataset are reported in Appendix S. The ranks in terms of the Fl and AUC scores are averaged over the classification task datasets. Likewise, the RMSE rank averages include the regression task datasets. We assign the maximum possible rank when a model could not be trained on a given dataset or could not be evaluated in reasonable time. This includes TabDDPM, which outputs NaNs for acsincome and d abetees and CoDi, which we consider to be prohibitively expensive to train on di abetes (estimated 14.5 hours) and lending (estimated 60 hours). Similarly, SMOTE is very inefficient in sampling for large datasets (78 min for 1000 samples on acsincom and 182 min on covertype and does not finish the evaluation within 12 hours. We provide visualizations of the captured correlations in the synthetic sample compared to the real training set in Appendix R and distribution plots for a qualitative comparison in Appendix Q
8

---

Published as a conference paper at ICLR 2025
Table 2:
Ablation study for five CDTD configurations. We report the median performance. The grey column depicts the impact our data type homogenization.
CDTD
Config.
B
C
D
(per type)
RMSE (abs. diff.; 4)
0.090
0.069
0.060
0.058
0.055
Fl (abs. diff.: J)
0.015
0.011
0.010
0.006
0.009
AUC (abs. diff.;
0.008
0.005
0.005
0.005
0.006
La distance of corr. (4)
0.483
0.421
0.457
0.450
0.444
Detection score (4)
0.762
0.739
0.774
0.662
0.701
JSD (1)
0.010
0.013
0.011
0.011
0,011
WD (1)
0.008
0.006
0.007
0.005
0.006
DCR (ahs. diff. test; .)
0.639
0.552
0.574
0.350
0.544
Sample quality. CDTD consistently outperforms the considered benchmark models most sample major performance edge in terms of the detection score, the L2
quality metrics. Specifically, distance of the correlation matrices and the ML efficiency metrics. Using score interpolation, CDTD able model the intricate correlation structure more accurately than other frameworks. Due their inductive biases, SMOTE and ARF perform very well in terms of univariate distribution for con- tinuous features (as measured by WD) and categorical (as measured by JSD), respectively. However, in both of those cases, CDTD performs competitively, in particular when compared to the diffusion- based models. Interestingly, TabSyn, latent space diffusion model, performs considerably worse than CDTD and often TabDDPM, which define diffusion in data space. In Appendix N, we further compare CDTD and TabSyn and investigate the benefits of defining a diffusion model in data space.
Most importantly, type-specific noise schedules mostly outperform the feature-specife and single noise schedule variants. This illustrates the necessity to account for the high heterogeneity tabular data on the feature-type level. Having distinct noise schedules many constraints on the model and, thus, decreases sample quality. Per-feature noise schedules would also require more training steps to fully converge, can be seen in Appendix P. We investigate the sensitivity of CDTD to important hyperparameters Appendix M.
feature instead appears to force too
Training and sampling time. Figure shows the average training and sampling wall-clock times (for all feasible models) over datasets (see Appendix for details sample quality
CTBAN
TWME
function of sampling time). We exclude SMOTE due to its considerably longer sampling times with average of 1377 seconds for 1000 samples. CDTD's use of embeddings (instead of one-hot encoding) for categorical features improves its scaling to increasing number of categories and thus, drastically reduces training times. The sampling speed of CDTD is
Takyyn
20
sample 1000 samples
Figure 3: Average training and sam- pling wall-clock times (excl. SMOTE, acsincome, diabetes, lending). competitive, in particular compared to the diffusion- based benchmarks CoDi, TabDDPM and TabSyn.
Ablation study. We investigate the separate components of our CDTD framework. The summarized results are given in Table 2 and the detailed results Appendix T. The baseline model Config. includes a single noise schedule with the original piece-wise linear functional form (Dieleman et al., 2022) and the CE and MSE losses naively averaged. Note that this configuration still novel con- tribution. Config. B adds our data type homogenization (i.e., loss calibration, improved initialization and dimendependn normalization schemes), Config. adds our proposed functional form for single noise schedule with uniform initialization, and Config. D imposes per-type noise schedules. Lastly, we increase the importance of low noise levels at initialization arrive the full CDTD (per type) model.
The results show that data type homogenization benefits sample quality significantly. Metrics associated with continuous features, i.c., RMSE and WD, as well as those relying on all features being generated well, i.e., the detection score, L2 distance of corr. matrices and DCR, improve dramatically. Switching from the piecewise linear noise schedule to our more robust functional form slightly harms sample quality. However, the per-type variant and the improved initialization more than compensate for this. The latter appears to trade-off some sample quality for faster convergence during training.

---

Published as a conference paper at ICLR 2025
5 CONCLUSION AND DISCUSSION
In this paper, we introduce a Continuous Diffusion model for mixed-type Tabular Data (CDTD) that combines score matching and score interpolation and imposes Gaussian diffusion processes on both continuous and embedded categorical features. Our results indicate that addressing the high feature heterogeneity in tabular data on the feature type level by aligning type-specific diffusion elements, such as the noise schedules or losses, substantially benefits sample quality. Moreover, CDTD shows vastly improved scalability and can accommodate an arbitrary number of categories.
Our paper serves as an important step to customizing score-based models to tabular data. The common type of noise schedules allows for an easy to extend framework. Crucially, CDTD allows the direct application of diffusion-relann relatte advances from the image domain, like classífier- free guidance, to tabular data without the need for a latent encoding. We leave further extensions to the tabular data domain, e.g., the exploration of accelerated sampling, efficient score model architectures, the adaptation to the data imputation task for future work.
Finally, want to Any generated data should not be blindly trusted, and synthetic data based inferences should always be compared to results from the real data. However, the correct of generative models for tabular data enables better privacy preservation and facilitates data sharing and open science practices.
against the potential misuse of synthetic data to support unwarranted claims.
LIMITATIONS
The main limitation of CDTD is the addition of hyperparameters, and tuning hyperparameters of generative model can be a costly endeavor. However, our results also show that (1) per type schedule is most often optimal and (2) our default hyperparameters perform well across diverse of datasets. Dieleman al. (2022) show that the results of score interpolation for text data can be sensitive to the initialization of the embeddings. We have not encountered similar problems on tabular datasets (see Table 7). While the DCR indicates no privacy issues for the benchmark datasets used, additional caution must be taken when generating synthetic data from privacy sensitive sources. Lastly, for specific types of tabular data, such as series, our model may be outperformed by other generative models specialized for that type. While CDTD could be directly used for imputation using RePaint (Lugmayr et al., 2022), a separate training process required to achieve the best results (Liu et al., 2024). Therefore, we leave the adaptation of CDTD to the imputation task for future work.
ACKNOWLEDGEMENTS
This work used the Dutch national e-infrastructure with the support of the SURF Cooperative using grant no. EINF-7437. We also thank Sander Dieleman for helpful discussions.
10

---

Published as a conference paper at ICLR 2025
REFERENCES
Jacob Austin, Daniel D. Johnson, Jonathan Ho, Daniel Tarlow, and Rianne van den Berg. Struc- tured Denoising Diffusion Models in Discrete SpacesS In Advances in Neural Information Processing Systems, volume 34, pp. 17981-17995 2021.
Barry Becker and Ronny Kohavi. Adult, 1996.
Jock Blackard. Covertype, 1998.
Vadim Borisov, Kathrin Seßler, Tobias Leemann, Martin Pawelczyk, and Gjergji Kasneci. Lan- guage Models are Realistic Tabular Data Generators. In International Conference Learning Representations, 2023.
Andrew Campbell, Joe Benton, Valentin De Bortoli, Tom Rainforth, George Deligiannidis, and Amaud Doucet. Continuous Time Framework for Discrete Denoising Models. Advances in 2022.
Neural Information Processing Systems, volume 35, pp.
Nitesh V. Chawla, Kevin W. Bowyer, Lawrence O. Hall, and W. Philip Kegelmeyer. SMOTE: Synthetic Minority Over-sampling Technique. Journal of Artificial Intelligence Research, 16: 321-357, 2002.
Song Chen. Beijing PM2.5, 2015.
Ting Chen. On the Importance of Noise Scheduling for Diffusion Models. arXiv preprint vrXiv:2301.10972 2023.
Ting Chen, Ruixiang Zhang, and Geoffrey Hinton. Analog Bits: Generating Discrete Data using Dif- fusion Models with self-Conditioning. In International Conference Learning Representations, 2023.
John Clore, Krzysztof Cios, Jon DeShazo, and Beata Strack. Diabetes 130-US Hospitals for Years 1999-2008, 2014.
Partha Deb and Pravin K. Trivedi. Demand for Medical Care by the Elderly: A Finite Mixture Approach. Journal of Applied Econometrics, 12(3):313-336 1997.
Prafulla Dhariwal and Alex Nichol. Diffusion Models Beat GANs Image Synthesis, In Advances in Neural Information Processing Systems, volume 34, pp. 8780-8794, 2021.
Sander Dieleman, Laurent Sartran, Arman Roshannai, Nikolay Savinov, Yaroslav Ganin, Pierre H. Richemond, Arnaud Doucet, Robin Strudel, Chris Dyer, Conor Durkan, Curtis Hawthorne, Rémi Leblond, Will Grathwohl, and Jonas Adler. Continuous diffusion for categorical data. arXiv preprint arXiv:2211. 2022.
Frances Ding, Moritz Hardt, John Miller, and Ludwig Schmidt. Retiring Adult: New Datasets for Fair Machine Learning. In Advances in Neural Information Processing Systems, volume 34, pp. 6478-6499 2021.
Kelwin Fernandes, Pedro Vinagre, Paulo Cortez, and Pedro Sernadela. Online News Popularity, 2015.
Jonathan Ho and Tim Salimans. Classifier-Free Diffusion Guidance. arXiv preprint arXiv:2207.12598 2022.
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising Diffusson Probabilistic Models. In Advances in Neural Information Processing Systems, volume 33, pp. 6840-6851, 2020.
Jonathan Ho. Tim Salimans, Alexey Gritsenko, William Chan, Mohammad Norouzi, and David J. Fleet. Video Diffusion Models. arXiv preprint 2eee:2224400445.. 2022.
Emiel Hoogeboom, Didrik Nielsen, Priyank Jaini, Patrick Forré, and Max Welling. Argmax Flows and Multinomial Diffusion: Learning Categorical Distributions. In Advances in Neural Information Processing Systems, volume 34, pp. 12454-12466 12465, 2021.

---

Published as a conference paper at ICLR 2025
Emiel Hoogeboom, Victor Garcia Satoraas, Clément Vignac, and Max Welling. Equivariant Diffusion for Molecule Generation in 3D. In International Conference on Machine Learning, pp. 8867-8887, 2022.
Aapo Hyvärinen. Estimation of Non-Normalized Statistical Models by Score Matching. Journal of Machine Learning Research, 6(24):695-709, 2005.
Allan Jabri, David Fleet, and Ting Chen. Scalable Adaptive Computation for Iterative Generation. In International Conference on Machine Learning, pp. 14569- 14589, 2023.
Alexia Jolicceur-Martineuu, Kilian Fatras, and Tal Kachman. Generating and Imputing Tabular Data via Diffusion and Flow-based Gradient-Booster Trees. In International Conference Artificial Intelligence and Statistics, pp. 1288-1296, 2024.
Tero Karras, Miika Aittala, Timo Aila, and Samuli Laine. Elucidating the Design Space of Diffusion- Based Generative Models. In Advances in Neural Information Processing Systems, volume 35, pp. 26565-26577 2022.
Tero Karras, Mika Aittala, Jaakko Lehtinen, Janne Hellsten, Timo Aila, and Samuli Laine. Analyzing and Improving the Training Dynamics of Diffusion Models. /EEE/CVF Conference on Computer Vision and Puttern Recognition, pp. 24174 24174-24184 2024.
Alex Kendall, Yarin Gal, and Roberto Cipolla. Multi-Task Learning Using Uncertainty to Weigh Losses for Scene Geometry and Semanties. In /EEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 7482-7491, April 2018.
A. Keramati, R. Jafari-Marandi, M. Aliannejadi, I. Ahmadian, M. Mozaffari, and U. Abbasi. Improved churn prediction in telecommunication industry using data mining techniques. Applied Soft Computing, 24:994-1012, 2014.
Jayoung Kim, Chaejeong Lee, and Noseong Park. STaSy: Score-based Tabular data Synthesis. In International Conference on Learning Representations, 2023.
Diederik P. Kingma and Ruiqi Gao. Understanding Diffusion Objectives the ELBO with Simple Data Augmentation. In Advances Neural Information Processing Systems, volume 36, pp. 65484-65516, 2023.
Diederik P. Kingma, Tim Salimans, Ben Poole, and Jonathan Ho. Variational Diffusion Models. In Advances in Neural Information Processing Systems, volume 34, pp. 21696-21707, 2021.
Akim Kotelnikov, Dmitry Baranchuk, Ivan Rubachev, and Artem Babenko. TabDDPM: Modelling Tabular Data with Diffusion Models. In International Conference on Machine Learning, pp.
17564-17579, 2023.
Chaejeong Lee, Jayoung Kim, and Noseong Park. CoDi: Co-evolving Contrastive Diffusion Models for Mixed-rype Tabular Synthesis. In International Conference on Muchine Learning, pp. 18940- 18956, 2023.
Lending Club. Loan data from Lending Club, 2015.
Xiang Lisa Li, John Thickstun, Ishaan Gulrajani, Percy Liang, and Tatsunori B. Hashimoto. Diffusion- LM Improves Controllable Text Generation. In Advances in Neural Information Processing Systems, volume 35, pp. 4328 4348,4433,44 2022.
Jianhua Lin. Divergence Measures Based on the Shannon Entropy. IEEE Transactions on Information Theory, 37(1):155-15,, 1991.
Tennison Liu, Zhaozhi Qian, Jeroen Berrevoets, and Mihaela van der Schaar. GOGGLE: Generative Modelling for Tabular Data by Learning Relational Structure. International Conference on Learning Representations, 2023.
Yixin Liu, Ajanthan Thalaiyasingam, Hisham Husain, and Vu Nguyen. Self-supervision improves diffusion models for tabular data imputation. In ACM International Conference on Information and Knowledge Management, pp. 1513-1522, 2024.
12

---

Published as a conference paper at ICLR 2025
Cheng Lu, Yuhao Zhou, Fan Bao, Jianfei Chen, Chongxuan Li, and Jun Zhu. DPM-Solver: A Fast ODE Solver for Diffusion Probabilistic Model Sampling in Around 10 Steps. In Advances in Neural Information Processing Systems, volume 35, pp. 5775-5787, 2022.
Andreas Lugmayr, Martin Danelljan, Andres Romero, Fisher Yu, Radu Timofte, and Luc Van Gool. RePaint: Inpainting using Denoising Diffusion Probabilistic Models. In IEEE/CVF Conference
Computer Vision and Puttern Recognition, pp. 1451-11161, 2022.
Chao Ma. Sebastian Tschiatschek, José Miguel Hermández-Lobato Richard Turner, and Cheng Zhang. VAEM: Deep Generative Model for Heterogemeous Mixed Type Data. Advances in Neural Information Processing Systems, volume 33, pp. 11237-11247, 2020.
Chenlin Meng, Kristy Choi, Jiaming Song, and Stefano Ermon. Concrete Score Matching: General- ized Score Matching for Discrete Data. In Advances in Neural Information Processing Systems, volume 35, pp. 34532-34545, 2022.
Sérgio Moro, Paulo Rita, and Paulo Cortez. Bank Marketing, 2014.
Alex Nichol and Prafulla Dhariwal. Improved Denoising Diffusion Probabilistic Models. In Interna- tional Conference on Machine Learning, pp. 8162 8171, 2021.
Neha Patki, Roy Wedge, and Kalyan Veeramachaneni, The Synthetic Data Vault. In JEEE Interna- tional Conference on Data Science and Advanced Analytics, pp. 399-410, 2016.
Liudmila Prokhorenkova, Gleb Gusev, Alcksandr Vorobev, Anna Veronika Dorogush, and Andrey Gulin. CatBoost: Unbiased boosting with categorical features. In Advances in Neural Information Processing Systems, volume 31, pp. 6638-6648, 2018.
Zhaozhi Qian, Bogdan-Cnnstantin Cebere, and Mihaela van der Schaar. Synthcity: Facilitating inno- vative use cases of synthetic data in different data modalities. In Advances Neural Information Processing Systems, volume 36, pp. 3173-3188, 2023.
Aaditya Ramdas, Nicolas Garcia, and Marco Cuturi. On Wasserstein Two Sample Testing and Related Families of Nonparametric Tests. Entropy, 19(2), 2017.
Florence Regol and Mark Coates. Diffussig Gaussian Mixtures for Generating Categorical Data. In AAAI Conference on Artificial Intelligence, volume 37, pp. 9570-9578, 2023.
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High- Resolution Image Synthesis with Latent Diffusion Models. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 10674-10685, 2022.
Juntong Shi, Minkai Xu, Harper Hua, Hengrui Zhang. Stefano Ermon, and Jure Leskovee. TabDifff A Multi-Moda Diffusion Model for Tabular Data Generation. arXiv preprint 2rX1v:2410.20626. 2024.
Jascha Sohl-Dickstein, Eric A. Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep Unsupervised Learning using Nonequilibrium Thermodynamics. In International Conference on Machine Learning, pp. 2256-2265, 2015.
Yang Song, Jascha Sohl-Dickstein. Diederik P. Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-Based Generative Modeling through Stochastic Differential Equations. In International Conference Learning Representations, 2021.
Robin Strudel, Corentin Tallec, Florent Altché, Yilun Du, Yaroslav Ganin, Arthur Mensch, Will Grathwohl, Nikolay Savinov, Sander Dieleman, Laurent Sifre, and Rémi Leblond. Self-conditioned Embedding Diffusion for Text Generation. arXiv preprint arXiv:221..042:3 2022.
Haoran Sun, Lijun Yu, Bo Dai. Dale Schuurmans, and Hanjun Dai. Score- based Continuous-time time Discrete Diffusion Models. In International Conference on Learning Representations, 2023.
Pascal Vincent. A Connection Between Score Matching and Denoising Autoencoders. Neural Computation, 23(7):1661-1674 2011.
13

---

Published as a conference paper at ICLR 2025
David S. Watson, Kristin Blesch, Jan Kapar, and Marvin N. Wright. Adversarial random forests for density estimation and generative modeling. In International Conference on Artificial Intelligence and Statistics, pp. 5357-5375, 2023.
Tong Wu, Zhihao Fan, Xiao Liu, Yeyun Gong, Yelong Shen, Jian Jiao, Hai-Tao Zheng, Juntao Li, Zhongyu Wei, Jian Guo, Nan Duan, and Weizhu Chen. AR-Diffusion: Auto-Regressive Diffusion Model for Text Generation, In Advances in Neural Information Processing Systems, volume 36, pp. 39957-39974, 2023.
Lei Xu, Maria Skoularidou, Alfredo Cuesta-Infante, and Kalyan Veeramachaneni, Modeling Tabular Data using Conditional GAN. In Advances Neural Information Processing Systems, volume 32, pp. 7335-7345 2019.
I-Cheng Yeh. Default of Credit Card Clients, 2009.
Hengrui Zhang. Jiani Zhang, Balasubramaniam Srinivasan, Zhengyuan Shen, Xiao Qin, Christos Faloutsos, Huzefa Rangwala, and George Karypis. Mixed-Type Tabular Data Synthesis with Score-based Diffusion in Latent Space. In International Conference on Learning Representations, 2024.
Zilong Zhao, Aditya Kunar, Hick Van der Scheer, Robert Birke, and Lydia Y. Chen. CTAB-GAN: Effective Table Data Synthesizing. Asian Conference on Machine Learning, pp. 97-112, 2021.
14

---

Published as a conference paper at ICLR 2025
A NOISE IMPLICATIONS OF HETEROGENEOUS CARDINALITIES
One distinct characteristic mixed-type tabular data is that each categorical feature may have different cardinality, i.c., different number of categories. Below, we briefly empirically investigate how this data characteristic warrants a rethinking of diffusion noise schedules. We show that, under some assumptions, categorical features of higher cardinality require more added noise than lower cardinality features to remove the same amount of signal.
(i) Let caat be a single categorical feature with Cj categories. We assume that each category has equal probability P(c) = 1/C; and train a CDTD model with a fixed, linear noise schedule to
learn the distribution of single = For each class {1, [1.-... C2}, extract the associated learned embedding ec For 500 timesteps t, linearly spaced on [0, 1], we derive the associated noise levels (t) from the learned noise schedules and for each o (t), we sample 1000 noisy embeddings
e, (t) - e) + (t) such that ~ NN, I). We input all ex (t) into the score model to compute the average (calibrated) cross-entropy (CE) loss. We define the remaining signal of the feature at given t as S. = 1 (t), where CE2- reflects the CE loss when using the noisy embedding for class c to predict the true class. Lastly, we compute the overall remaining signal over all classes as max. S.. Figure 4 shows the result of repeating this procedure with varying Cj and four-dimensional embeddings. As we can see, lower cardinality features, i.e., those with a low Cj. attain the same signal only at higher noise levels ot). This example of the effect of feature heterogeneity in tabular data encourages us to investigate Seature-specity noise schedules.
1.0
2 classes
3 classes
0.8
4 classes
5 classes
6 classes
0.6
7 classes
8 classes
0.4
signal
0.2
0.0
10a
noise level o(t)
Figure 4: Comparison of signal remaining in embedded categorical features given by max, Sa. where S. = (t.(t) and CE2 reflects the CE loss when using the noisy embedding for class E {l,..., C₁} to predict the true class. Lower cardinality features tend to require higher noise level (t) to achieve the same amount remaining signal as higher cardinality features.
B LOss CALIBRATION
Under Assumption 1, the signal-to-noise ratio the terminal timestep is sufficiently low to approximate situation which the model has no information about the data. We want to let the model be indifferent between features, that we scale the loss of each feature such that at the terminal timestep the same expected loss is attained. Therefore, we are looking for scaled losses Cisse ( 1) and Cer CCr(" (x 1) which at = achieve unit loss in expectation.
(j)
For a single scalar feature and a given timestep t. we can write the empirical denoising score matching loss (Equation (3)) when using the EDM parameterization (Karras et al., 2022)
EMSE (2 conts t) = X(t) Cakip (t) + Cour
(t (t)
-I -oct
20(x..0)
where (r(i) denotes the neural network output for feature that parameterizes the denoiser Sq. The score model is then given by so (ar, 1) 2 1) x²))
15

---

Published as a conference paper at ICLR 2025
The parameters Cski((t) a d and Cou(t) = o() Odwa/ (v o2(() + dam depend on (t) (and data) and therefore on timestep t. For - 1, a() approaches the maximum noise level man and Ctip(t) - 0 and Con (t) + such that the score model directly predicts the data at high noise levels. For - O, the model shifts increasingly towards predicting the error that has been added to the true data. In the EDM parameterization, the explicit timestep weight (used achieve unit loss across timesteps at initialization, see Appendix C) is A(t) = 1/Cour (t)² ~ 1 for t = 1.
At the terminal timestep
1, we then have:
E p(
(AMSE (AASE(=ccm)) (f)
1)] X(1) p(= p(=um)
(skip(1)az + Cour (1)E
Foont
(2)
= E (2))
(0-z1 +
FFg
(6) cont
(10(0)
(5) -Icont
E p(x )
Without information, it is optimal to always predict the average value E minimum expected loss becomes:
p(a [ooml and thus, the
B (LMS (3 1)] E p(
p(as(a)) (B. (Bpcosu) [1rcom]
Tcom
(5)
Var[arcom]
Therefore, have LUSE (4 1) LMSE (#eam: 1) as long as standardize Feamt unit variance.
For single categorical feature, Weat is distributed according to the proportions Pe (for categories 1,.... Cj). The denoising model for score interpolation is trained with the CE loss:
(C
(1) - Σ 1 (
c) log (
where Fl. denotes the score model's prediction of the class probability at timestep 1. Without information, it is optimal to assign the e-th category the same proportion as in the training set.
At / = 1, we thus let ((j)= = Pe such that the loss equals:
(Cre(red (i) 1)] -E
( (1) c) log p())
E p(u
(8)
p(ac
p( [1 = c) logne),
Σ Pe log Pe
(10)
Note that this is the feature-specific entropy and we can use the training set proportions to compute this normalization constant Z, = - - Po. log to scale the loss for categorical features. Then,
E p(a [Cen (5) 1)] = E
CCE( () 1)/2,] 1.
p(
We have thus achieved calibrated losses with respect to the terminal timestep 1, that is, Me [Cèe (1) 1)] = for all continuous features and categori-
E
( [Cusse (2:c) 1)]
E
cal features j.
C OUTPUT LAYER INITIALIZATION
At initialization, we want the neural network to reflect the state no information (see Appendix B). I Likewise, our goal is loss of one across all features and timesteps.
16

---

Published as a conference paper at ICLR 2025
For continuous features i, initialize the output layer weights (and biases) to zero such that the output of the score model for a single continuous feature, F also zero. Since we use the EDM parameterization (Karras et al., 2022), we apply the associated explicit timestep weight o o X(t) = (a(())aa))² This is explicitly designed to achieve a unit loss across timesteps at initialization
and we show this analytically below. We denote the variances of the data zcont and of the Gaussian noise e at time as data and o² (1), respectively, At initialization we have:
"p(= p(() (LMSEE ( (Arum t)] X(t) E ((al)))p(c)) (C) + c) + Com Com((1) (1)
(5) Iecunt
X(t) E p(aci))p(ee))
Cakip (t) + c)
cont
(e)
E (o ddaaa)
(²()) + a
pla pp(c)
(2 daan
Iaann
(2 c)
Fcont
(8)
2
(²()) aan
data C 0² (t)
(1)
E
(l) data
pla p(e)
0 (2(1) + data
datta
a(1) x
2
E
(²()) + data
p(a p(())
(())
data
(t)))
data
(x )
2ex
(i) cont
0² (t) + 2
data
p(2i p(c))
(2())
o data
Pasta (t
Var(e)
Var Var(=comt)
- 2 Cov(e, Fcont
(1) + Iana
daaa
x²(1)
data + to o(()))= =1
(t) + Fata
For categorical features j. we initialize the output layer such that the model achieves the respective losses under no information. Using the loss normalization constant Z, (see Appendix B) and dropping the expectation over p(c), we have
E (((c(2)) (CCE((E (1) t)] E p(all) [CCE(ZEa) ( , = E Z, p( |Cce [Cee(ala) (2la) t)].
Hence, for E P(oa)) Cor( t)] = Z,. we obtain an expected loss of one irrespective of t. The neural network outputs vector of logits F(" that transformed into probabilities with softmax function for each categorical feature. We denote the e-th element of that vector softmax (-)c. Since 7, is derived in Equation (10) by imposing probabilities equal to the training set proportions for that category, Pe, we have
exp( (P((F))))
(()))
log Pc
log (ogsollmax((1)) log
loog Σ exp(F()) R
Ck=1 exp exp (F())
We initialize the neural network such that p()) log pe for all c. This is achieved by initializing the output layer weights zero and the output layer biases the relevant training set log-proportions of the corresponding class, Hence, this initialization gives us
C
F()) log
exp (()) = log Pc log Σ Pc
log Pe:
k=1
k=1
which in turn leads to an initial loss of Zj for all t and therefore achieves uniform, calibrated loss of one at initialization similar to the continuous feature case.
17

---

Published as a conference paper at ICLR 2025
D ADAPTIVE NORMALIZATION OF THE AVERAGE DIFFUSION LOSS
Both the loss calibration (sce Appendix B) and output layer initialization (sce Appendix C) ensure that the losses across timesteps (and features) equal at initialization. During training, the adaptive noise schedules allow the model to focus automatically on the noise levels that matter most, i.e., where the loss increase is steepest. However, the better the model becomes at given timestep t, the lower the loss at the respective timestep, and the lower the gradient signal relative to the signal for timesteps f > t. We counteract this with adaptive normalization of the average diffusion loss (averaged over the features) across timesteps. Specifically, we want to weight the average diffusion loss at timestep t, C(t) given in Equation (5), such that the normalized loss is the same (equal to one) for all t. Similar methods have been used by Karras et al. (2024) and Kingma & Gao (2023), we follow the latter in the setup of the corresponding network.
We train neural network alongside our diffusion model to predict [t)) based on t and use the MSE loss to learn this weighting. First, we compute Graise (t) - log(t)/4 following the EDM parameterization (Karras et al., 2022). Then, we embed Cooise in frequency space (1024-dimensional) using Fourier features. The result is passed through a single linear layer to output a scalar value and through an exponential function to ensure that the prediction [(t) 2 0. We initialize the weights and biases to zero, to ensure that at model initialization we have unit normalization.
E DERIVATION OF THE FUNCTIONAL TIMEWARPING FORM
Since higher noise levels, a, imply a lower lignal-t-noiise ratio, and in turn a larger loss, L, we know that the loss must be a monotonically increasing and S-shaped function of the noise level. Additionally, the function has to be easy to invert and differentiate. We incorporate this prior information in the functional timewarping form of F: a + f. convenient choice is the edf of the logistic distribution:
Flog (v) = [1 + exp(-v(y-
(11)
where '² describes the location of the inflection point of the S-shaped function and v 2 2 indicates the steepness of the curve.
We let logit(a) ()= = log(o (1 - ))) to change the domain of Flog from (-x0, oo) to (0. 1). The latter covers all possible values of the noise level scaled [0, with the pre-specified minimum and maximum noise levels min and a maxx. To define the parameter μ in the same space and ensure that 0 < f < 1, we also let μ logit(/u). Accordingly, we derive the edf of the homain-adapter Logistic distribution:
1 1-1- 10111 1
Fd.
1
(12)
Since E is not bounded, we introduce a multiplicative scale parameter, 7 > 0, such that for timewarp- ing we predict the potentially feature-specific loss as f = Faa) = FFa.a. (g). Fa.a. can also be initialized to the edf of the uniform distribution with μ 0.5, v/ 1 and 7 such that all noise levels are initially equally weighted. However, an initial overweighting of lower noise levels is beneficial for tabular data (see also Section 3.4).
Likewise, we can derive the inverse edf Fi1l (f), that is our mapping of interest from timestep to noise level a, in closed form:
1 In
= == d.a.log (t) sigmoid(c), with e = In
+
(13)
When training the diffusion model, we learn the parameters of Fa.a.log well as y by predicting the diffusion loss using F(a) and the noise levels scaled to [0, 1]. At the beginning each training step, we then use the current state of the parameters and F-1 log with sampled timestep ~ Uj..1 as input, to derive o. To allow for feature-specifel adaptive noise schedules, we separately introduce (a(a)) for each feature k. to predict the feature-specific loss l, based on the feature-specifie scaled noise level d
Note that with timewarping we create a feedback loop in which we generate more and more ds from the region of interest, decreasing the number of observations available to learn the parameters in
18

---

Published as a conference paper at ICLR 2025
different noise level regions. We thus weight the timewarping loss, - when fitting F(a) the data by the reciprocal of the pdf fd.a log (a) to mitigate this adverse effect (see Dieleman et al., 2022). Again, this function is available to in closed form. With Flog and flog denoting the respective edf and pdf of the Logistic distribution, we have
8 foa.og(()) = dy Flog(y)
In
do
1
y=logi((o)
(logit()))))))
0 - σ)
2(0,
a σ) (1
where we defined ((,,H,2) (1- 1=4) and used the definitions of flog and the parameter p².
F BENCHMARK DATASETS
Our selected benchmark datasets are highly diverse, particularly in the number of categories for categorical features (see Table 3). For the diabeees and covertype datasets, transform the original multi-class classification problem into a binary classification task for ease of presentation. For the diabetes data, convert the task to predicting whether a patient was readmitted hospital. For the covertypo ype data, the task is converted into predicting whether a forest of type 2 is present given licensed under creative commons.
30 meter area. All datasets are publicly accessible and (except nmes)
Table Overview the selected experimental datasets. We count the target towards the respective features that remain after removing continuous features with excessive number of missings. The minimum and maximum number of categories are taken over all categorical features.
Total no. observations
No. of features categorical
No, of categories min
Dataset
License
Prediction task
continuous
(Ding al., 2021)
CCD
regression
664 500
8
$29
adull (Becker & Kohavi, 1996)
CC 4.0
binary class,
48 842
9
42
(Moro al., 2014)
CC
binary class.
188
11
12
bei beijing (Chen, 2015)
CCBY 4.0
regression
41 757
1
10
4
charn (Keramati
2014)
binary class.
5
covertype (Blackard, 1998)
CC BY B.0 4.0
binary class. binary class.
581 012
44
10
2
default (Ych, 2009)
10
11
diebetes (Clore 2014)
CCBY 4.0
binary class.
101 766
28
716
landing (Lending Club, 2015)
DbCL. 1.0
regression
9182
10
34
3151
(Fermandes al., 2015)
CC 4.0
regression
39644
14
2
nmea (Deb & Trivedi, 1997)
unknown
regression 4406
8
11
4
G BASELINE MODELS
Below, we give a brief description of our selected generative baseline models (including code sources).
SMOTE (Chawla al., 2002) technique (not generative model) typically used oversample minority classes based on interpolation between ground-trui observations. We use SMOTENC for mixed-type data from the scikit-learn package and mostly adapt the code from the TabDDPM repository (Kotelnikov et al., 2023). For sampling, we utilize 16 CPU cores.
ARF (Watson et al., 2023) density estimation. The implementation is available https: / /i thub. com/bbps hb/ hhr/bbbbyypy arrpy and licensed under the MIT license. We use package version 0.1.1. For training, utilize 16 CPU cores.
recent generative approach that is based on a random forest for
CTGAN (Xu et al., 2019) - one the most popular Generative Aaaartvvv-Advverrra----eeeee for tabular data. The implementation is available part of the Synthetic Data Vault (Patki et al., 2016) at https: / /itttb.. com/ sdv-dev/CTGaN and licensed under the Business Source License 1.1. We use package version 0.9.0.
models
19

---

Published as a conference paper at ICLR 2025
TVAE (Xu et al., 2019)
model for tabular data. Similar to CTGAN. The implementation available as part of the Synthetic Data Vault (Patki et al., 2016) at https: /tt::/gittti com/ con/sdv-des sdv-ddv/ CCTAAN and licensed under the Business Source License 1.1. We use package version 0.9.0. Note that since we only use TVAE (and CTGAN) as benchmark, and do provide synthetic data creation service, the license permits the free usage.
TabDDPM (Kotelnikov et al., 2023) a diffusion-based generative model for tabular data that combines multinomial diffusion (Hoogeboom et al., 2021) and diffusion in continuous space. An implementation available as part of the synthcity package (Qian et al., 2023) at https: //github. coon/vadersscaal
synthcity/ and licensed under the Apache 2.0 license. We use package version 0.2.7 with slightly adjusted code to allow for the manual specification of categorical features.
CoDi (Lee et al., 2023) a diffusion model trained with an additional contrastive loss, and which factorizes the joint distribution of mixed-type tabular data into distribution for continuous data condi- tional on categorical features and distribution for categorical data conditional on continuous features. Similarly, the authors utilize the multinomial diffusion framework (Hoogeboom et al., 2021) to model categorical data. An implementation is available at https: /tt////ii /github. com/ Chac under an unknown license.
TabSyn (Zhang et al., 2024) a diffusion-based model that first learns transformer-based VAE to map mixed-type data to a continuous latent space. Then, the diffusion model is trained on that latent space. Note that despite TabSyn utilizing separately trained encoder, this does not result in lower-dimensional latent space and therefore, does not speed sampling, We use the official code available at https: / / gi thub. com/ ama zon- sci ence/ tabsyn under the Apache 2.0 license.
H IMPLEMENTATION DETAILS
Each of the selected benchmark models requires a rather different, more specialized neural network architecture. Imposing the same architecture across models is therefore not possible. The same inability holds for the comparison of CDTD to other diffusion-based models: Our model is the first to use a continuous noise distribution on both continuous and categorical features, and therefore the alignment of important design choices, like the noise schedule, across models is not possible. In par- ticular, the forward process of the multinomial diffusion framework (Hoogeboom et al., 2021) used in TabDDPM and CoDi, which based on Markov transition matrices, does not translate our setting,
To ensure a fair comparison in terms of sampling steps, we set the steps for CDTD, TabDDPM, CoDi and TabSyn to max (200, default). We therefore increase the default number of sampling steps for CoDi and TabSyn (from 50 steps) and TabDDPM (from 100 steps for classification datasets). For TabDDPM and regression datasets, we use the suggested default of 1000 sampling steps.
We adjust each architecture to a total of ~ 3 million trainable parameters on the adult dataset to improve the comparability further (see Table 4) and use the same architectures for all considered datasets. Note that the total number of parameters may vary slightly across datasets due to different number of features and categories affecting the one-hot encoding but is still comparable across models. We also align the embetding/bottleneck dimensions for CTGAN, TVAE, TabDDPM, TabSyn and CDTD to 256. To align TabDDPM, TabSyn and CDTD further, we use the TabDDPM architecture for all models, with appropriate adjustments for different input types and dimensions. If applicable, all models are trained for 30k steps on a single RTX 4090 instance, using Py PoToch version 2.2.2.
Below, we briefly discuss our model-specific hyperparameter choices.
Table 4: Total number of trainable parameters per model on the adult dataset.
Model CTGAN
Trainable parameters
3 000 397
TVAE
2 2996 408
TabDDPM
3 001 786
CoDi
2998 043
TabSyn
2997 765
CDTD (per type)
3 002 969
20

---

Published as a conference paper at ICLR 2025
SMOTE (Chawla et al., 2002): We use the default hyperparameters suggested for the SMOTENC scikit-lear implementation.
ARF (Watson et al., 2023): We use the authors's suggested default hyperparameters. In particular, we use 20 trees, 5 = 0 and a minimum node size of 5. We follow the official package implementation and set the maximum number of iterations to 10 (see https: githtb.thhuu..// www.co/////...... http//lpsshaaargggga
CTGAN (Xu et al., 2019): We follow the popular implementation in the Synthetic Data Vault package (see https: //ggtthbb..... com/ sdv- dev/CTGAN). For this model to work, the batch size must be divisible by 10. Therefore, we adjust the batch size necessary. We use a 256-dimensional embedding (instead of the default embedding dimension of 128) to better align the CTGAN architecture with TVAE, TPMDDPM, TabSyn and CDTD.
thub.
TVAE (Xu et al., 2019): We again follow the implementation in the Synthetic Data Vault. We use a 256-dimensional embedding to better align the architecture with CTGAN, TabDDPM, TabSyn and CDTD.
TabDDPM (Kotelnikov et al., 2023): There are no general default hyperparameters provided. Hence, we mostly adapt the papers' tuned hyperparameters for the adult dataset (one of the few used datasets that includes both continuous and categorical features). However, we decrease the learning rate from 0.002 to 0.001, since most of the tuned models in the paper used learning rates around 0.001. For regression task datasets, we use 1000 sampling steps in accordance with the author's settings. For classification task datasets, we use 200 sampling steps (instead of the default 100 steps), to better align the model with CoDi and CDCD. Note also that for classification task datasets, TabDDPM models the conditional distribution p(xly), instead of the unconditional distribution p(x) which is modeled for regression tasks. We adjust the dimension of the bottleneck to 256 (instead of the default 128) to also accommodate also larger datasets and align the model with CTGAN, TVdE,and CDTD.
CoDi (Lee et al., 2023); We use the default hyperparameters from the official code (see https: //github. com/ Chae jaangLee/CCDDDD
TabSyn (Zhang et al., 2024): We use the default hyperparameters as suggested by the authors. The training steps that go towards training the VAE and the denoising network follow the proportions given in the official code (see httpp::/ / /github. com/ amazon- sena-ssiie science/ wwweeeeeeeeaeeeeeeeeeeeeeeeeeeee abbyyn). To improve comparability to TabDDPM, CoDi and CDTD, use the same neural network architecture as TabDDPM, which only differs slightly from the original architecture. We leave the VAE untouched.
CDTD (ours): To ensure comparability in particular to TabDDPM, CoDi and TabSyn, we use the same neural network architecture as TabDDPM. We only change the input layers to accommodate our embedding- and concatenate them with the scalar valued continuous features. The adjusted output layer ensures that we predict a single value for each continuous features and set of class-specific probabilities for each categorical feature. Since hidden layers slightly down relative to the TabDDPM to ensure approximately 3 million trainable parameters (instead of 798 neurons per layer we use 796) the adult dataset. More details on the CDTD implementation are given in Appendix K.
framework. In the input layer, we vectorize all embedded categorical features
of embeddings introduces additional parameters, we scale the
I TUNING OF THE DETECTION MODEL
We use catboost model (Prokhorenkova et al., 2018) to test whether real and generated samples can be distinguished. We generate the same number of fake observations for each of the real train, validation and test sets. We cap the maximum size of the real data subsets to 25 000, and subsample them if necessary, to limit the computational load. Per set, combine real and fake observations to train valid and Deleseet, respectively. The catboost model is trained on Deletee with the task of predicting whether an observation is real or fake. We tune the catboost model with optuna and for 50 trials to maximize the accuracy on valid The catboost hyperparameter search space is given in Table 5. Afterwards, we repeat the sampling process and the creation of Deletect Dileter and Deletect for five different seeds. Each time, the model is trained on Deleuce with the previously tuned hyperparameters, and evaluated on Delttct The average test set accuracy over the five seeds yields the estimated detection score.
21

---

Published as a conference paper at ICLR 2025
Table 5: Catboost hyperparameter space settings. The model is tuned for 50 trials.
Parameter
Distribution
no. iterations
1000
learning rate depth
Log Uniform [0.001, 1.0]
Cat((3445.66.788
L2 regularization bagging temperature leaf estimation iters
Uniform [0.1, 10]
Uniform [0. 1]
Integer Uniform [1, 10]
J MACHINE LEARNING EFFICIENCY MODELS
For the group of machine learning efficiency models, we use the scikit-learn and catboost package implementations including the default parameter settings, if not specified otherwise below:
Logistic or Ridge Regression: max. iterations 1000
Random Forest: max. depth 12, no. estimators 100
Catboost: no. iterations = 2000, early stopping rounds 50, overfitting detector pval 0.001
We subsample Drain in case of more than 50000 observations to upper-bound the computational load.
K CDTD IMPLEMENTATION DETAILS
To enable fair comparison to the other methods, and to TabDDPM and TabSyn in particular, the CDTD score model utilizes the exact same architecture as Kotelnikov al. (2023), which was also adapted by TabSyn (Zhang et al., 2024). We use the GeantileTrassfommes to pre-process continuous features, followed by a standardization to zero mean and unit variance. An overview of the score model is provided in Figure 5: First, the noisy data, i.e., the noisy scalars for continuous features and the noisy embeddings for categorical features, and the timestep t, are projected onto a 256- dimensional space. Then, all 256-dimensional vectors are added and the result is processed by
P Pi
c) Yo
Linear
x
ReLU (Lineaz)
Linear
SiLU([innerr)
Linear
TimeEmb
286
Positional Embedding
250
Xesl
Figure 5: Overview of the CDTD architecture adapted from TabDDPM. The dimensions of the inputs and layer outputs are stated in the lower-left hand corner for continuous features Zcont and categorical features Teat- Note that each categorical features can have a different number of categories [C], impacting the output dimension of the final layer. Scalars are colored orange, embeddings red and linear layers blue. The positional embedding highlighted in green refers to the positional sinusoidal embedding. CDTD only conditions on y, i.e., the target feature, for classification task datasets.
22

---

Published as a conference paper at ICLR 2025
set of five dully-connected linear layers with ReLU activation functions. Lastly, linear projection maps the output of the fully-connected layers to the required output dimensions, which depend on the number of features and number of categories per feature.
The only major difference to the TabDDPM setup are the inputs, as we need to embed the categorical need to predict single scalar for each Zeont and C₁ values for each ()) with Cy being the number of categories of feature j. We change the initialization of the output layer described in Appendix C. To handle our inputs, embed the categorical features in 16-dimensional space and add a fcature-specific bias of the same dimension, which captures feature-specifie information common to all categories and is initialized to zero. We Lz-normalize each embedding to prevent a degenerate embedding space in which embeddings are pushed further and further apart (see Dieleman et al., 2022). Also, Dieleman et al. (2022) argue that the standard deviation of the Normal distribution used to initialize the embeddings, denoted by miit an important hyperparameter. In this paper, we set mit = 0.001 for all datasets and have not seen detrimental effects. Table 7 indicates that CDTD is not sensitive to the choice of init
features in Euclidean space. The output dimensions are the same, (8)
Since we utilize embeddings, we have to scale the neurons per layer slightly down in the stack of the five fully-connected layers (from 798 for TabDDPM to 796). Also, since TabDDPM samples discrete steps from [0, T). with T > 1, we scale our timesteps E [0. 1] up by 1000. We use the same optimizer (Adam), learning rate (0.001), learning rate decay (linear), EMA decay (0.999), and training steps (30000). However, since we work with embeddings we add a linear warmup schedule over the first 1000 steps. Lastly, instead of using uniform (time)step sampling as TabDDPM, the CDTD model uses antithetic sampling (Dieleman et al., 2022: Kingma et al., 2021). The timesteps still uniformly distributed but spread out more evenly over the domain, which benefits the training of the adaptive noise schedules. For generation, we use deterministic (Fuler) sampler with 200 steps to minimize the discretization error (see Appendix I. for details).
L CDTD SAMPLING
Algorithm 1 shows our deterministic sampling approach, where Do and De represent the score model output for the i-th continuous and j-th categorical feature, respectively. We found results similar to Karras et al. (2022), i.e., that adding stochasticity to the sampling process does not benefit sample quality. We also experimented with second-order samplers (e.g.. Heun) but found them to add little benefit over our first-order method while requiring double the NFEs. As default, we use 200 sampling steps. Table shows that the gains sample quality are marginal to non-existent after more than 500 sampling steps.
There is another subtlety in our sampler: Unlike in EDM (Karras et al., 2022), in the final step the sampler steps into t = 0, which is not associated with ao = 0 but on = min- This is consequence of utilizing more than one noise schedule. We condition the score model not on but on which indicates the "global" time common to all noise schedules. Moving from d min to a 0 in the final step, would imply < Therefore, we let a 0 for all feature types.
To sample from the learned distribution, we need to run the reverse process of the probability flow ODE (Equation (2)). For example, for two different features in and x2, we deconstruct the ODE as:
dx=
G()G())) Vx
T (1)
dt
(t(
(t)
カー
(1)
dt
0z (1)
-==ZZ o2(1)
In practice, we use an Fuler sampler with 200 discrete timesteps Al = 1++1 - < 0. The timesteps
are generated as a linearly spaced grid on [0, 1] and transformed afterwards into noise levels (t) via the described timewarping procedure. For the discretized and simplified ODE at step i, this yields
(t)
A (t) Aoz (t)
X x;--
A02(t) A
(1
At -x=x++
+
(2))
(2)
23

---

Published as a conference paper at ICLR 2025
Algorithm 1 Deterministic Sampling
Input: N sampling steps
Sample fo (O, o I Kam) Vi E {l...., Koott
Sample xeat,1o N0, o cat max I Kan) Vj E ( (1,...; Keat
(3)
1: for {0,..... N 1] do
2: l = 1 8/N
3:
16-1 = (8 11/N
(Kai)
(Kou))
4: X. - (Xcalt," ..... Xeat,f, L.oomt,, tont,
5:
for all E (I..... Keon do
6: daccn (Fcomt, Dooai (xsi (lss)
(i)
7:
comtt.uu1 cont,,, + co (to+1) Joun, (tx)] dr end for
(4)
(i)
8:
9:
for all j E [1..... K do
10:
P(x (5)
cat.t. Do (xx, tx)
dx(²)
(j)
Pr (1) ix
(x / / / (t.) (t.) from Equation (4) sa (ts+1) Gen, (tx)) dx
(
(i)
calit
end for 14: end for
Recover classes from embeddings with additional pass score model (K.)
15: XN + (Xea. I CaL fo connt
(Kouu)
16: for all j E (1....... Ken)) do
P (i) xealer) Deat. (xN, (XN,LN-1 1N11)
18: Feat +- arg max,ec; P(x ec Xeatttx) (pick most likely category)
(i)
(3)
(5) (5)
19: end for
20: return (Fea Feal Fontt,ty Fcont,
(Koa) (1)
(Koon)
where @ denotes the element-wing wise product. Hence, we are effectively featuree feature-speciffe steps of length Aru (t). The adaptive noise schedules (timewarping) therefore not only affect the training process, but also focus most work in the reverse process on the noise levels that matter most for sample quality (i.e., where (tx(t) (t) is small).
We use finite differences to approximate o,, instead of the available, analytical variant, since drx(t) + oo as t + 1. The step At would therefore be required to decrease as t + 1 to en- sure At dt holds. For large number of steps, this assumption does hold in practice, and for do)(t)
dr the update of x overshoots the target drastically. Intuitively, a (t) becomes too steep near the terminal timestep
1 such that the step size can not sufficiently compensate for the slope increase to turn (e)(t) into a good approximation of the actual change o (L). Moreover, the analytical solution would approximate dtx(t) o& (t)dt, i.c.. the change in the noise level caused by change in t. Since we know exactly where (t) will end up when changing t, we are better off using that exact value and let do (t)
di
Table 6: Performance sensitivity of CDTD (per type) to increasing number of sampling steps. Each metric averaged over five seeds. As robust measure, we report the median over the ablation study datasets acs i income, adult, bei jing and churn.
Steps
RMSE
F1
AUC
L distance of corr.
Detection score
JSD
WD
DCR
100
0.038
0.011
0.007
0.131
0.574
0.012
0.003
0.315
200 (default)
0.030
0.009
0.006
0.130
0.565
0.012
0.003
0,316
500
0.027
0.009
0.006
0.130
0.557
0.013
0.002
0.313
1000
0.028
0.008
0.006
0.130
0.562
0.013
0.002
0.311
24

---

Published as a conference paper at ICLR 2025
M SENSITIVITY TO IMPORTANT HYPERPARAMETERS
The training and sampling processes of CDTD are affected by various novel hyperparameters. Generally, per-type noise schedule works best, as show in main results in Table for diverse set of benchmark datasets. Here, we examine the sensitivity of CDTD to two additional important hyperparameters: (I) the standard deviation of the noise used to initialize the embeddings (and therefore specific to score interpolation), mit and (2) the weight of the low noise levels used to initialize the μik in the adaptive noise schedule parameterization.
The experiments in Dieleman et al. (2022) show that Jnit is a crucial hyperparameter for score interpolation on text data. Table shows that this sensitivity does not translate to the tabular data domain. This may be explained by the much smaller embedding dimension (16 vs. 256) or by our usage of Seaaurre-ppcctte et al., 2022), only face a maximum of 3151 categories in the lendinng ng dataset (see Table 3). Thus, unlike other generative (diffusion) models for tabular data, CDTD scales to a practically arbitrary number of categories.
embeddings. Compared to a vocabulary size of 32000 for text data (Dieleman
Our proposed functional form for the adaptive noise schedules (see Appendix E) is the first to allow for the incorporation of prior information about the importance of low high (normalized) noise levels. For this, we adjust the weight of low noise levels which directly determines the location of the inflection point fl (see Section 3.3). The results in Table indicate low sensitivity of sample quality to weight changes for a per-type noise schedule. The initialization only impacts the time to convergence but not (much) the location of the optimum. In our experiments, the number of training steps (30000) appears to be high enough for all model variants to converge to similar states.
Table 7: Performance sensitivity of CDTD (per type) to changes in the standard deviation ait in the initialization of the embeddings of categorical features. Each metric is averaged over five seeds. As a robust measure, we report the median over the ablation study datasets acs i ncome, a adul 1 t, bei jeijing and churn.
Init
RMSE
F1
AUC
12 distance of corr.
Detection score
JSD
WD
DCR
0.031
0.012
0.004
0.123
0.555
0.013
0.003
0.294
0.1
0.027
0.012
0.004
0.125
0.556
0.013
0.003
0.323
0.01
0.031
0.007
0.005
0.129
0.575
0.013
0.003
0.320
0.001 (default)
0.030
0.009
0.006
0.130
0.565
0.012
0.003
0.316
Table 8: Performance sensitivity of CDTD (per type) to changes in the prior weight of low noise
levels in the initialization of the adaptive noise schedules. Each metric is averaged over five seeds. As robust measure, we report the median over the ablation study datasets acs income, a adul t, bei jijingg and churn.
Weight
RMSE
F1
AUC
L2 distance of corr.
Detection score
JSD
WD
DCR
0.031
0.010
0.005
0.126
0.570
0.012
0.002
0.238
0.030
0.009
0.004
0.125
0.570
0,013
0.002
0.289
3 (default)
0.030
0.009
0.006
0.130
0.565
0.012
0.003
0.316
0.028
0.011
0.005
0.134
0.574
0.011
0.003
0.350
N ADVANTAGES OF DIFFUSION IN DATA SPACE
These days, inspired from diffusion models in the image and video domains, much work relies on the idea of latent diffusion. Here, we want to briefly discuss and emphasize that for tabular data, diffusion in latent space (represented by TabSyn) has important drawbacks and how CDTD, diffusion model defined in data space alleviates those.
Latent diffusion models first encode the data in latent space. The diffusion model itself is then trained in that latent space instead of directly on the features. Hence, the performance of the diffusion model directly depends on second model, with separate training procedure. TabSyn uses VAE model to encode mixed-type data into a common continuous space that is not nower-dimensional so to minimize reconstruction errors. Any reconstruction errors caused by the VAE reduce the
25

---

Published as a conference paper at ICLR 2025
sample quality of the eventually generated samples. matter the capacity of the diffusion model. This suggests that we would want to train a highly capable encoder/decoder, which adds additional training costs. Figure shows that latent diffusion is not necessarily more efficient in the tabular data domain. In particular, if the latent space is not lower-dimensional or the encoder/decoder is very complex, then sampling speed is not improved.
We further hypothesize that much tabular data, due to the lack of redundancy and spatial or sequential correlation, difficult to summarize efficiently in a joint latent space. Hence, compared other domains, larger VAEs and higher-dimensional latent spaces are required, increasing the training time. Also, there the risk of the VAE not picking up on subtle correlations within the data or distorting existing correlations by mapping into the latent space. Any correlations not properly encoded cannot be learned by the diffusion model. Since we optimize the VAE on an average loss, its reconstruction and encoding performance of, for instance, minority classes or extreme values in long-tailed distributions is likely lacking. This makes the job of the diffusion model more difficult, not sometimes impossible.
Lastly, we take great care in homogenizing categorical and continuous features throughout the training process (see Appendix B and C). This is crucial part of modeling mixed-type data. Using VAE to define a diffusion process in latent space only shifts the necessity for homogenization to the VAE training process. Not balancing different feature- data-types and their losses induces implicit importance weights. Thus, the VAE may sacrifice the reconstruction quality of some features in favor of others (Kendall et al., 2018; Ma et al., 2020).
To empirically investigate the difference of diffusion in data space (CDTD) and latent diffusion (TabSyn), we examine feature--euurttettte sample quality metrics and those that directly benefit from all features being generated well. Our results in Table 9 show that latent diffusion comes with a considerable decrease in sample quality (while imposing similar architecture and number of parameters as well as sampling steps, see Appendix H). In particular, the attained maximum univariate metrics as well as the detection score and the L2 distance of the correlation matrices indicate that TabSyn has issues modeling all features and their correlations sufficiently well. This supports our argument that a homogenization of data types is important and that an encoding in latent space may complicate the learning of joint data characteristics.
Table 9: A comparison of the CDTD model to latent diffusion (TabSyn). We average each metric over five sampling seeds and as robust measure report the median over the 11 datasets. Abs. diff. in corr. matrices refers to the absolute differences in the correlation matrices between ground truth and synthetic data. The max, min and mean are taken across features.
Abs. diff. in matrices
Detection score
La dist. of corr.
JSD
WD
min
mean
max
min
mean
max
min
max
TabSyn CDTD (per type)
0.859
0.919
0.004
0.044
0.141
0.002
0.012
0.025
0.000
0.261
0.701
0.444
0.002
0.011
0.025
0.001
0.006
0.010
0.000
0.088
Improv, over TabSyn
18.4%
51.7%
50.0%
75.0%
82.3%
50.0%
50.0%
60.0%
66.3%
O COMPARISON TO RELATED WORK
Table 10 summarizes our comparison of CDTD to the diffusion-bassee benchmark models, that is. TabSyn, TabDDPM and CoDi. Of those models, only TabSyn applies diffusion in latent space, which comes with both advantages and costs (as discussed in Appendix N). TabSyn is the only other model besides CDTD that avoids one-hot encoding categorical features by using embeddings. This improves the scalability to a higher number of categories without blowing up the input dimensions. Although both models utilize embeddings, TabSyn's 's generative capabilities are more constrained by jointly encoding all features in a latent space. It should also be noted that TabSyn makes use of a Transformer architecture in its VAE, which means that scales quadratically in the number of features and therefore may not be easily scaled to high- dinensiinni data.
CDTD is the first model to utilize adaptive and type- or feature-specife noise schedules to model tabular data. Further, we take great care in homogenizing categorical and continuous features
26

---

Published as a conference paper at ICLR 2025
throughout the training process (see Appendix B and C). No other model attempts balancing the different features types. This is problematic as it suggests that other models may suffer from feature- specific implicit importance weights that impact both training and generation. Hence, the sample quality of some features may be unintentionally sacrificed favor of increasing the sample quality of other features (Kendall et al., 2018; Ma al., 2020). Note that this also applies TabSyn: Even though their diffusion model avoids this issue by relying on single type of loss due to the continuous latent space, the VAE training process does not account for any balancing issues between the two data types. Hence, the balancing issue is not eliminated but got only shifted to the VAE.
Lastly, CDTD and TabSyn are the only models that define the diffusion process in continuous space. As such, other advanced techniques, like classifier-fre freee guidance or ODE/SDE samplers, can be directly applied. To accommodate categorical data, CoDi and TabDDPM make use of multinomial diffusion (Hoogeboom et al., 2021), which is an inherently discrete process and therefore prohibits such applications.
Table 10: Comparison CDTD to the diffusion-based generative models CoDi, TabDDPM and TabSyn. (x) Note that the VAE trained as part of the TabSyn model does not balance type-specific losses, which induces an implicit weighting among features. This can worsen the sample quality of some features in favor of others.
defined in feature space
avoids one-hot encoding
aduptive noise schedule
type- specific noise schedules
diffusion
feature types
continuous space
CoDi
TabDDPM TabSyn
CDTD (ours)
27

---

Published as a conference paper at ICLR 2025
P EXAMPLES OF LEARNED NOISE SCHEDULES
Next, we show the learned noise schedules for the largest (acs in ncome) and the smallest (churn) datasets, Additionally, we illustrate the fit of single, per type and per feature schedules to the respective diffusion losses they were trained to fit.
noise level ex
scaled to (0.1)
Single
15
0.4
0.0
0
calogorica
callgorical uous
neise level dr
scaled to [0.1]
0.8
continous
Type
0.4
0.0
Feature noise level dz
scaaedd to [0,1]
0.8
(00)))
0.4
timestep
scaled to [0.1]
Figure 6: (Left): Learned noise schedules for acsincome. This reflects F Fd.. (Right): Implicit weighting of noise levels / timesteps. This visualizes fo...log.k
fred kunstonn
categorical
fited functon difuusion
diflusion
continuus
0.8
1lted luncion
Tue diftusion loss
0.6
10
10
10
10
10a
10
10
10
10
10"
10
10
di scaled [0.1]
di scaled [0.1]
dr scaled
[0.1]
Figure 7: Illustration the goodness of fit of the timewarping function F₃ for single (left), per type (middle) and per feature noise schedules (right) on the acsincome data.
28

---

Published as a conference paper at ICLR 2025
noise lovel dr
scaled to (0.1)
Single
0.8
plmn)
0.2
1.0
calogorica
calogorical cordinuou
continuuus
noise level 0r
scaled to (0,1)
Type
0.6
15
0.0
1.0
Per Feature
noise loval dt
scaled to [0.1]
plo))
0.4
0.0
timestep
scaled [0.1]
Figure 8: (Left): Learned noise schedules for churn. This reflects F-1 do (Right); Implicit weighting noise levels timesteps. This visualizes fa..log..
1.0
fned funaion
catagorical
fited tunction
Tue
loss
1itad lunatinn
diflusion
0.4
0.2
10
10
10
107
10
10
10
107
10
10
10
10
(7 scaled to [0.1]
d7 scaled to [0.1]
scaled to [0.1]
Figure 9: Illustration of the goodness of fit of the timewarping function Fj for single (left), per type (middle) and per feature noise schedules (right) on the churn data.
29

---

Published as a conference paper at ICLR 2025
Q QUALITATIVE COMPARISONS
Real Data
SMOTE
ARF
CTGAN
TVAE
10.5
7.5
2.5
TANDPM
Cofi
Tok8yn
CDT (per tpe)
CDTD festure)
7.5
2.5
ago
aggn
ago
ago
ago
Figure 10: Bivariate density for age and educational num from the adult data.
CTGAN
Cafk
CTTDD qMe sypt CDTD Sularo
Tand
Bank
NMES
0.005
0..5
Dessity
Dessity
0.00
0.008
Dessity
0.1
0.02
0.00
0.004
0.03
0.002
colccationol-mmm
Call Failane
13000
125000
3100
1250
Chual
100000
10000
Conill
1000)
7500
750
5000
2000
maatital
education
Age Group
region
Figure 11: Comparison of some univariate distributions for adult, bank, churn, n.e
Real SMITTS:
ARP
CTGAN TaintP
Calk Tyd
(0000 Pypt CDTD Sndwer
Beijing
Lending
0.085
0,006
Dassiyy
Dessity
Dasiity
Dessity
0.0006
0,004
0,2004
0.2
0.0003
0.003
0.0000
0.000
0,200
0.0
200
1000
20
50
100
0.0
0.5
1.0
AGB
2yg.posithve ppolatireepplaiicy
10000
200000
90000
6000
Chunl
15000
4000
9000
10000
2000
2000
12000
1011
cbud
EDUCCATION
purpoce
monday
Figure 12: Comparison of some univariate distributions for jing default. lend ng, news. (Note that CoDi is prohibitively expensive to train on lendi ng and therefore excluded.)
30

---

Published as a conference paper at ICLR 2025
R
VISUALIZATIONS OF CAPTURED CORRELATIONS
Cort
Col
Cir
Cont
CDTD iper lype)
(single)
lealure)
COrL
Cont Cort
Coe
Ce
Cool
Core:
0.02
0.04
6.6
0.66
2.0
Figure 13: Element-wise absolute differences of the correlation matrices between the real training set and the synthetic data for the acsincome dataset. TabDDPM generates NaNs for this dataset and is therefore excluded. SMOTE takes too long for sampling. Continuous (cont.) and categorical (cat.) features are indicated the axes.
SMOTE
ARF
TVAE
CTGAN
TabDOPM
CoDi
Oa
Cant.
Cat
Cont.
Cot.
Cont
Cal.
Cont.
Cat
Cont.
Cot.
Cont
Czl.
Cont.
CDTD
CDTD
CDTD
TabSyn
(singlo)
(per
(per feature)
Roal Tost Set
Cal.
Card.
0.02
0.04
0.06
0.08
0.1
Figure 14: Element-wise absolute differences of the correlation matrices between the real training set and the synthetic data for the adult dataset. Continuous (cont.) and categorical (cat.) features are indicated on the axes.
SMOTE
ARF
CTGAN
TabDDPM
Cork
Cxtl
Cont
Cont
Gat
Cont
Cat
Cont
Cont
Gal
Cont.
CDTD (single)
CDTD (per type)
CDTD
TabSSyn
(per feature)
Real Test Set
Dani
Qat
Cont.
Cot.
Cont
Cat.
Cont.
Oat.
Cont.
Cat.
Cont.
0.94
818
0.1
Figure 15: Element-wiss absolute differences of the correlation matrices between the real training set and the synthetic data for the bank dataset. Continuous (cont.) and categorical (cat.) features are indicated on the axes.
31

---

Published as a conference paper at ICLR 2025
CDTD (singlo)
CDTD (per type)
CDTD pper feature)
TabSyn
Roal Test Set
Ca:
Cont.
Cont.
Cont.
Cont.
Cont.
0.02
0.04
0.05
0.08
0.
Figure 16: Element-wise absolute differences of the correlation matrices between the real training set and the synthetic data for the be jing dataset. Continuous (cont.) and categorical (cat.) features are indicated on the axes.
SMOTE
CTGAN
TabDDPM
CoDi
Cork.
Gm
Cont
Cat.
Cont:
Coc
Cont
GM
Come:
Cat.
Cont.
Dor.
Dont
CDTD (single)
CDTD (per type)
CDTD
TabSyn
(per feature)
Roal Test Set
Cant:
Cal.
Cat.
B12
0.94
0.08
0.1
Figure 17: Element-wise absolute differences of the correlation matrices between the real training set and the synthetic data for the churn dataset. Continuous (cont.) and categorical (cat.) features indicated on the axes.
ARF
TVAE
CTGAN
TabODPM
CoDi
Cat
Cort
Oor
Cort.
Ccl
Cort
Cont.
Cot
Octt
CDTD (single)
CDTD pper type)
CDTD
TabSyy
(per fealure)
Real Test Sel
Cuc.
Cont.
Cal
Cart
Oac.
Cont.
Cal.
Cont
Cat.
Cont.
0.00
C.4
6.0t
6:68
2.01
Figure 18: Element-wise absolute differences of the correlation matrices between the real training set and the synthetic data for the covertype dataset. SMOTE takes too long for sampling. Continuous (cont.) and categorical (cat.) features are indicated on the axes.
32

---

Published as a conference paper at ICLR 2025
SMOTE
ARF
TIKE
CTGAN
TabDDPM
CoDi
Qal
Dant.
Cat.
CDTD
CDTD (per type)
CDTD (per feature)
TaoSyn
(single)
Real Test Set
Cat
Cont.
Coc.
Contt
Cdl.
Cont
Cat
Cont.
Col.
Cont.
Figure 19: Element-wiss absolute differences of the correlation matrices between the real training set and the synthetic data for the de f Fuul dataset. Continuous (cont.) and categorical (cat.) features indicated the axes.
SMOTE
ARF
TVAE
CTGAN
OOFL
Car.
Cart
CaL
Cont.
Cal.
Cont.
Cut.
Cunt
CDTD (single)
CDTD [per bype)
CDTD
TabSSy
pper feelurre)
Real Test Set
OZIL
Car.
Cart
O21
Cont.
Cal
Cont.
Cur.
Ount
Cat
Cord.
0.00
0.88
Figure 20: Element-wise absolute differences of the correlation matrices between the real training set and the synthetic data for the di abetes dataset. TabDDPM generates NaNs for this dataset and is therefore excluded. CoDi is prohibitively expensive to train and therefore excluded. Continuous (cont.) and categorical (cat.) features are indicated on the axes.
SMOTE
ARF
TVAE
CTGAN
TaDDDPM
CDTD (single)
CDTD jper type)
CDTD pper teature)
TabSyn
Poal Test Set
606
Figure 21: Element-wiss absolute differences of the correlation matrices between the real training set and the synthetic data for the lending dataset. CoDi is prohibitively expensive to train and therefore excluded. Continuous (cont.) and categorical (cat.) features indicated the axes.
33

---

Published as a conference paper at ICLR 2025
SMOTE
ARF
TWAE
CTGAN
TaaDDPM
CoDi
Gat
Casi
Cont
Corr
CDTD (single)
CDTD (per type)
CDTD (per feature)
TaiSyn
Call
Cont:
Con
Cont.
Cat
Cont
Call
Cont.
Cor
Cont
Figure 22: Element wise absolute differences of the correlation matrices between the real training set and the synthetic data for the news dataset. Continuous (cont.) and categorical (cat.) features are indicated on the axes.
SMOTE
ARF
TVAE
CTGAN
TabDDPM
CoDi
Ca1
Cont.
Cot.
Coit.
Cat.
Cont
Cal
Cont.
Cot.
Cort.
Cat.
Cont.
CDTD (single)
CDTD
CDTD
TaiSyn
(per
(per teature)
Real Test Set
Df:
Oat
0.22
0.94
0.88
0.1
Figure 23: Element-wise absolute differences of the correlation matrices between the real training set and the synthetic data for the nmes dataset. Continuous (cont.) and categorical (cat.) features are indicated on the axes.
34

---

Published as a conference paper at ICLR 2025
S DETAILED RESULTS
CoDi is prohibitively expensive to train on lending and diabetes and TabDDPM produces NaNs for acsiiccone and diabetes. SMOTE takes too long to sample datasets of sufficient size for acs and covertype (see Table 29). For those models, the performance metrics on these datasets are therefore not reported. They are assigned a rank of 10 in Table and the worst metric-specific performance across all models before computing the average metrics reported in Table 11.
Table 1I: Model evaluation results averaged over 11 datasets (if a model was not trainable on a given dataset, we assign the maximum, i.e., worst, value over all models for that dataset to this model) for seven benchmark models and for CDTD with three different noise schedules. Per performance metric, bold indicates the best, underline the second best result.
SMOTE
ARF
CTGAN
TVAE
TabDDPM
CoDi
TabSyn
CDTD (single)
CDTID (per type)
(per feature)
CDTD
RMSE (abs. ()
0,366
0.091
0.635
0.868
0.674
0.278
0.309
0,086
0.091
0.102
F1 (abs, diff.: L)
0.068
0.053
0.074
0.020
0.048
0.111
0.020
0.015
0.014
AUC (abs. dill.; J)
0.043
0.020
0.980
0.065
0,023
0.039
0.066
0.015
0.014
0.015
La distance corr.
1.287
1.321
2.190
2.707
3.001
2.383
2.097
0.792
0.684
0.920
Dotection score (1)
0.869
0.933
0,986
0.977
0.790
0.947
0.858
0.761
0.739
0.770
JSD (L)
0.077
0
0,101
0.135
0,086
0.073
0.052
0.015
0.016
0.018
WD
0.011
0.011
0.024
0.023
0.050
0.059
0.017
0.010
0.008
0.009
DCR (abs, diff. to test; L)
1.813
1,588
3.317
1.602
2,061
2,838
2.648
0.927
0.867
0.760
Table 12: L2 norm (incl. standard errors in subscripts) of the correlation matrix differences of real and synthetic train sets for seven benchmark models and for CDTD with three different noise schedules.
SMOTE
ARF
CTGAN
TYAE
TabDDPM
ColN
TabSya
(stagle)
CDTD
(per type)
CDTD
CDTD (per feature)
aosincome
1 20.004
a.517
0.500
0.136anecs
0.135
0.123
adult.
0.570ggnmo
2000
0.
0.175aneer
0.125aa0
1.120_aam
bene
me
0.947
0.529 wms
0.400
0.750gn
0.335
0.281 am
0.258_
loc bci1ing
0.061anxer
1.470
0.368 2.228
0.373
0.086
0.075an.cc
0.073
aluun
0.266420.06
1.301
8.746-33.002
0.746
0.613
0.269
0.26faaaee
B.271 m0000
covertype
1022
5
6.668
I. 1.13
1.020
3.760anae
L.70
1:357an
1.07220001
defaullt
0. 1.709
1:22810121
2,607
2.60
1.564
0.685 s0.
1.072
1.125
0.724
0.641
2611
0.85
diabctas
2 200
5,351
2.79640a44
L381
1.215l.n.0
1,35 1.35
lending
1.321
34773
2.420
6.7022
1.148
1.230
1.351
1.84
4.338
4.611an0s
4.61227m
12:556_g1
4.874
5.155an
2.502
1.81
5. 3.646
0.66510047
0.717am004
1.660.0ms
(1.08220200
11600
0.91910.077
0,45410045
0.444pam
0.943
Table 13: Jensen-Shannon divergence (incl. standard errors in subscripts) for seven benchmark models and for CDTD with three different noise schedules.
SMOTE
CTGAN
TVAE
TaNDDPM
CaDi
TabSyn
CDTD (simgle)
CDTD (per type)
(per feature)
CDTT
acsin.come
0.2 0.250
0.309
2.078
0.082
0.021
0.922:3:00
adult
a.064
0.007
8.132
0.118anmm
0.04610000
0.0222...00
0.011ammt
0.005
00015.0.00
bank
0.039
0.004
0:086-2400
0:101a000
0.021
0.03822 oo
0.063-ge.0m
0.01
2001ao
0.01 23:00
0.015-0000
bwi nn
0.006_5.002
0.00420000
0.006-a.002
0.074
0.03422.922
0.011 2ooss
0.01 20000
0.00520022
0.006.3.00
0.07
churr.
0.012g4..01
0.011
0.005_...03
0.048an.0.
0.015
a.048an
0.031
0.011aome
0.010,3:000
0.012.0000
coveztyow
200
0.314422000
0.04820000
0.001.3:03
0.0088
0.044220000
0.0042000
0007220000
detaull
0.042ac.001
1.00810000
0.194_
0.177amom
0.076
0.075
0.060.2..01
0.0121a002
0.016
0.17200001
diabctes
22.00
2000
0.050_20000
0.187an000
0.058__0.00
0.0254000
0.025,43000
lending
0.143_r.0m
200
0.002-0000:
0.188
0-267
0.1:9-e.mm:
0.060guun
23007
news
0.063.c.001
0.00210001
0.022.0.00:
1.128
0.01Fac.035
a.012
0.017ueoo0:
0.005
0.000
0.000.0.00
nnea
0.000gr.001
g.00
2.117g2000
0,022
0.027 40 nee
0.029gge00
0.009gg
0,007
00012-1001
Table 14: Wasserstein distance (inel. standard errors in subscripts) for seven benchmark models and for CDTD with three different noise schedules.
SMOTE
ARF
CTGAN
TYAE
TANDDPM
Colli
TobSyn
(slagle)
CDTD
CDTD (per type)
CDTD (per feature)
accin.ome
0.007
0.021an.0 55005
0.017
0.002.0.00 0.005_
a.000
0,001
edult
00000_2..00
0.012.000
0.316-3000
0.221a000
0.002231000
0.01820
0.007
0.00
0L002
benk
0.002
0.01210000
0.021
0.021
0.040aamm
0.001,5.000
0.050....o
0.006.s.000
0.00010.000
0.008
0.007
beining
2.008
0.000_
0.00854000
0.007
0.019.n
0.001_.0000
0.003440
chuen:
0.006g5.001
0.0132000
=0000:
0.032
0.007
2.048
0.013-20.01
0.006gguu
0.006-3-00
zovestype
2.00822
0.04122200
0:022
0.002222493
0.0:92e:000
0:018gmu
0.000:3:000
0.13
sofacic
0.0022...00
8.011.08.00
1.005
0.0022:2400
1.013
0.000uu0.00
0000410.00
0.003
dlabetes we
0.001-2:000
0.01240000
0.030=a000
0:0384000
0.01:_20.00
0.0224000
0.031
2011_-00
londing
0.00605.000
0.015gmmmn
0.01129009
0.0152um
0.410a2:021
0.05333.000
0.012amm
0.013:3:000
0.011-0.003
newa
0.007 20.003
0.02240..00
8.000...000
0.018anneo
0.03342:00
a00 no0
0.1229...200
1.00%
0,006,3.000
0.008 1008
nnees
0.006
0.012gm
0:036-2--99
0.008an.e
0.007
0.016
0.032:g..0m
0.008gmmm
0:006
3.006_00.m 00
35

---

Published as a conference paper at ICLR 2025
Table 15: Detection score (incl, standard errors in subscripts) for seven benchmark models and for
CDTD with three different noise schedules.
SMOTE
ARF
CTGAN
TVAE
TANDDPM
Cabii
TobSSy
(single)
CDTD
(per type)
CDTT
CDTD (per feature)
inicuute adult
0.389-
0.58
0.825
0.629 0.088
0.527 233002
0.528 201
0,687
0.907 0.
0.967
0.594
0.902.0.00
0.621 a..4
0
0.500u2:004
0
benk
0.18884000
1.000..0
0.853_...01
jinn
0.338-2.002
0.50
0.23842.054
2.989
0.723
0.020123000
chutr
a4.57
0,853
0.945_...03
0.842
0.356
0,730
u..59
0.615
0.541
0.639...011
covestype
2.900l our
defac lt
05.024
0.981
20
0. 8.008-00000
0:997 smom
0.085go.c0 mo
2.825
0.783.3.001
0.877.0.011
siabetes
0.726 24.001
0 0.935_.8-00
0.997 s0o0t
a3.37
0.862 mm
lunding
0.9K6 25.06
0.997
0.900-20022
1.000
0.008 20.003
0.965:2:091
0.560
news
0.998
0.905
1.000_..003
1.00
0.974
I. 1.000 an
0.059
0.567aame
0.950as.092
0.972.u0.m
0..726.5.007
0,18770
0.9922_8.001
0,988200022
0.83225.011
0,98819
0.829
0.6482aux
0.636.00.00
Table 16: Distance closest record of the generated data (incl. standard errors in subscripts) for seven benchmark models and for CDTD with three different noise schedules.
TabNNPMM
TahSyn
(singlo
CDTD
CDTD
type)
(per featurs)
CDTD
weainceee
7.878 ov02
10.758 8.77
6,002
10.877
10.750
5.357 Loocs
5. 5.207
5,364
Adtali
1.577
2.573aaa7
3.012aune
2:277
7:750
2 2.
1.327
351
bane
5 5:130gs om
3.0022m 313
2
1.013g0 oa
bouiling Ling
0.73324000
0.500.0we
11734 exc
0.588
0.47
0.473aa and
0.347
0,502
1.804
314
0.150
0.520
1.2 1.44aaacs
5.173a 5.788
0.877auee
1.50
182h
1.50
Jalh
dedessie
1.412
1,002
5.0
5.218 21.00 eas
1-43 1-43
2,50
1.305
1352
diaberea
20 esc
20000
12.199ts on
6.214g eoo
25.194
14.58 cno
14.458
18886 ons
Bendling
18
17
7822 143
17.770
10:00820ors
20
14:05
14.9022000
3,700
1.951_000
1.504
2.205
2,971
1.71018815
0,880.o077
1.581
2 2.06
0,801
0.78019.533
Table 17: Machine learning efficiency F1 score for seven benchmark models, the real training data and for CDTD with three different noise schedules. The standard deviation takes into account five different sampling seeds and uses the average results of the four machine learning efficiency models computed across ten model seeds.
Real Data
SMOTE
ARF
CTGAN
TYAE
TabDiPM
Cale
Tubiiyn
bingle)
CDTD
(per typs)
CDTD
CDTD (per fizature)
eduiit
8.287 Loaio
1078
8.788
1.7651
0,788 sos
0.789
0.789 ooo
bank:
0.7.5 Loaee
2000
0:080 100e
ess
0.622 995
0.761
1008
6.782 es3
0.796
0.717 .aa
charn
0.761 ses
0.502amerr
0855
0.65
4.
0.850 oa:
0.505
0.500
covextyge
0.713
0.709
0.707 aeom
0.700 es
0.707 am
0.705
default
80772
20027
eaz
0.80220
20038
6.85 mm
0,673am ess
8.670anmm
0.62)anam
0.672
0,607
0.565
0.660ga
0,614gr ess
0.610gnmm
0.514
Table 18: Machine learning efficiency AUC score for seven benchmark models, the real training data and for CDTD with three different noise schedules. The standard deviation takes into account five different sampling seeds and uses the average results of the four machine learning efficiency models computed across ten model seeds.
Real
SMOTE
CTGAN
TYAE
TahDDPM
Cale
TukSyn
(diggge)
CDID
CDID
CDTD
aduute
20000
0.838 2
0.880
0880
0.900aa
0,000
bank
0.9172mmm
0 1e
0.50
0.506ansa
0 0.9112a0e
chazz
0
0.964_nomm
0.96110ow
0,882
0.54850
11507.000
0.0111000
0,000
0.507
n 0.500_n00
covertypee
0 0.80013000
0.677
0.277
0.67529 01862200
G.840aa
0.sa0-nam
de detiult
0.768 sua
a.700
0.744 0.754
0.781 aon0l
0.765anaa
0.738 0.739
6.732
0.705 eaz
0.764
0. 0.705 caae
di aberes
poam
11677
0.660
0.502 0.626.0
0.612
0.671 202
0.673
Table 19: Machine learning efficiency RMSE for seven benchmark models, the real training data and for CDTD with three different noise schedules. The standard deviation takes into account five different sampling seeds and uses the average results of the four machine learning efficiency models computed across ten model seeds.
Real Data
SMOTE
ARF
CTGAN
TYAB
TabDIPM
Cotl
Totöyn
bengle)
CDTD
CDID (per typs)
CDTD (per fictturd)
acsincoms
1.802_00122
1.202
1004
0.857 Leors
0.05519000
0.507
0.512 coo.
bei ping
0.71L.c0m Loaot
0.742 Loon
0.77
1,050.a403
1.205.08ts
0.849 cooo
6.772uxx63
0.7051883
0.2784
0.137441e:
0.101amem
0.789
0.9
0.00 20001ammm
0.002
4.30
0.782ana
0.717 -uso
1,00
1.1322nmm4
1,3311
1.327
0.5635g
1,344a
1.306-mmm
36

---

Published as a conference paper at ICLR 2025
T ABLATION STUDY DETAILS
Table 20: L2 norm (inel. standard errors in subscripts) of the correlation matrix differences of real and synthetic train sets for five CDTD configurations with progressive addition of model components.
CDTD (per type)
Configuration A
B
acsincome
0.13820.004
0.135 00.08
0.152+0.001
0.135
adult
0.135
0,113
0.159
0.100
0.125
bank
0.478g0.019
0.243
0.269 012
0.19640..012
0.231 =0.009
bei beijing
0.076:2.0009 0009
0.070.10.007
0.067 0.0.00 0.004
0.068 008
0.073:0..m 00.007
churn
0.294
0. 0..00
0.262
0.239 4:0
0. 0.255 021
covertype
1.178
2.099
1.974 136
1 1.808
1.357 0.55
default
0.905g0..10
0.799
0.727 20.110 132
0.52 124
0.641 130
diabetes
0.719=0.049
1.435
1.39720.00 005
1.230
1.213_0.029
lending
1.450=0.046
1,127 10:083
1.178 059
1.295+00004
1.239 =0
news
2.
2.136
1.973
2.01640.4s2
1.811 1.811 +0.295
0..0s
0.421 +0.022
0.457
0.45040.044
0.444
Table 21: Gensen-Shannon divergence (incl. standard errors in subscripts) for five CDTD configura- tions with progressive addition of model components.
CDTD
Configuration
A
(per type)
acsincome
0.016g0.000
0.022
0.026=0.001
0.032:1.000
0.022
adult
0.010=0.001
0.013
0.01320.00 0000
0.015+0.001
0.015
bank
0.008 000
0.017:0.000
0.011
0.011
0.011
beijing
0.005=20.001
0.005++0.004
0.004
0.004+0.001
0.006
churn
0.015
0.011
0.009 003
0.01044.003
0.010 20.02
covertype
0.002-0..00
0.003
0,004 poa
0.0044:..00
0.003_00000
default
0.014:0.00 001
0.015
0.01220.000 002
0.017:10.00
0.016:0.000
diabetes
0.024 000
0.03040.000
0.022
0.0244 000
0.025 000
lending
0.055
0.055
0.055 001
0.037
0.057 0.001
0.003
0.003
0.003 001
0.00440.001
0.003
mmes
0.008go.002
0.008
0.008 002
0.009
0.007=0.002
Table 22: Wasserstein distance (incl. standard errors in subscripts) for five CDTD configurations with progressive addition of model components.
CDTD
Configuration
A
B
D
(per type)
acsincome
0.005-20.0m
0.002+0.000
0. 0.0022n..00
4..000
20.000
adult
0.008+0..om
0.005
0.006+0.000
0.002
0.003
bank
0.007
0.004440.000
0.005 001
0.003
0.003 0.000
beijing
0.006 000
0.004++0.000
0.002 oou
0.0004+0.000
0.002
churn
0.008 001
0.007 200000
0.007 -0077=0.000 001
0.0074+.00 002
0.006
covertype
0.003
0.020+40.000
0.019
0.012 000
0.009 1009==0.00
default
0.003
0.0044 0.0044+..000
0.003 000
0.003
diabetes
0.021 000
0.0022++0000
0.041 DOU
0.033
0.03-1+0.0o
lending
0.012=2.0om 000
0.011+0.000
0.012 DOO
0.013
0.013=2.000
news
0.009
0.006+4.0000
0.007 +77 poa
0.005+0.000
0.006
nnes
0.008 001
0.007
0.008
0.00811.000 001
0.006
37

---

Published as a conference paper at ICLR 2025
Table 23: Detection score (incl. standard errors in subscripts) for five CDTD configurations with progressive addition of model components.
CDTD (per type)
Configuration
A
acsincome
0.0001
0.533
0.540 002
0.546 003
0.527 0.002
adult
0.640 0..000
0.595 0..022
0.621 10.00 002
0.581 40.002
0.590=0.000
bank
0.880
0.739 2.0006
0.798 00.001
0.662 4.000
0.701 0.0005
beijing
0.69920.003
0.65840.004
0.61522.00
0.6264:.002
0.620=n.00
churn
0.710=a.00
0.610+0.008
0.580
0.560
0.541
covertype
0.887 =0.002 002
0.991 100001
0.989 =0. 001
0.984410.001
0.981 = 000
default
0.925 002
0.816=0.003
0.774 003
0.759=0.000
0.793 005
diabetes
0.762=0..00
0.895+0.000
0.870
+0.51
0.837 0.837 =0.001
lcnding
0.989
0.938
0.958
0.944
0.955
0.993
0.95644..002
0.965 0012
0.946+o.ozz
0.950 =0.000 002
0.663-20.014
0.651 10.007
0.667 0000
0.633 0.648110.015
Table 24: Distance closest record of the generated data (incl. standard errors in subscripts) for five CDTD configurations with progressive addition of model components.
CDTD
Real Test Set
A
B
c
D
(per type)
acs nncome
7.673=0.01r
8.550 055
8.34222.ost
8.28440.030
8.31 1:000
8.397
adult
1.870
1.231 016
1.318 o08
1.296
1.520
1.327
bank
2.369
1.583+0
2.016=0.007
20.014
2.18T+0.000
1.913=0.000
beijing
0.385=0.c0m
0.502 001
0.537
0.505+0.001
0.51 1.511+0..01
0.47320.00
churn
0.347
0.526
0.383=0.014
0.327 1o
0.333
0.278
covertype
0,529 =
0.87 0.871+00.000 +
1.919
1.872+0.00
1.698
1.594
default
1.812
1.321 1:0
1.333
1.306
1.48410.012
1,355
diabetes
15.608 L.
13.299 10.020
14.958 Lo
15.007
14.755 10.045
14.468 1..00s
lending
11.18 11.18420.00x
15.13040.50
15. 12
14.957 20.12..44
14.91620.
14.96220.050
news
3.615 20.000
3.894420.01
3.737
3.69220-014
3.74629.000
3.676
nmes
1.931 =0.00
1.202+0.019
1.014=0.018
0.958+00.00
0.952+0.022
0.780=0..03
Table 25: Machine learning efficiency Fl score for five CDTD configurations with progressive addition of model components. The standard deviation accounts for five different sampling seeds and uses the average results of the four machine learning efficiency models across ten model seeds.
CDTD
Real Data
A
B
C
D
(per type)
adult bank
0.797
0.7802o.002
0.788
0.78620..01
0.790
20.8920.002
0.745
0.759-2..00
0. 0.758
0. 0.760+o.605
0.751
0.766
churn
0.873
0.850 006
0.861 1111:00.00 000008
0.863 56.3+0.004 004
0.860 008
0.863 20.00
covertype
0.817+0..uD1
0.791 +00 D01
4745+a..001
0.761 +0.001
0.768
0.767 +0.001
default
0.67444.o0m
0.672=u..00z
0.674
0.67 ( +0.000
0.67340.000
0.675 d..75
diabetes
0.621
0.616=0.002
0.612+a..um
0. 0.612
0.617
0.619
Table 27: Machine learning efficiency RMSE for five CDTD configurations with progressive addition of model components. The standard deviation accounts for five different sampling seeds and uses the average results of the four machine learning efficiency models across ten model seeds.
CDTD (per type)
Real Data
A
B
D
acs income
0.804
366820.011
0.80840.014
0.82010.012
0.800=1.00 011
10.000
bei beijing jing
0.71 100001
0.801 10.006 006
0.78044.00m
0.771 10.005
0.76944.om 0008
0.76611..03
iendi
0.030
0.124 006
0.059
0.07210.000
0.067
0.066
1.00140.om
0.772
0.835
0.805
0.763
0.752
nmes g
1.001+0.003
0.967 20.064
1.128
1.195
1.225
40.05
38

---

Published as a conference paper at ICLR 2025
Table 26: Machine learning efficiency AUC score for five CDTD configurations with progressive addition of model components. The standard deviation accounts for five different sampling seeds and uses the average results of the four machine learning efficiency models across ten model seeds.
CDTD (per type)
Real Data
A
B
D
aqult
0.915:40.000
0.906g0.001
0.90944..00
0.909
0.909=0.00m
0..09
bank
0.947
0.945
0.946440.000
0.945+0.c00
0.94640.000
0.946
churn
0.964
0.959 0.0003
0.961 000002
0.962 0.0002
0.960 0002
0.957 0.000
covertype
0.892 2000000
0.870 0.000.000000 000
0.826
0.838 10.381000011
0.844
0.844
default
0.768 1.0.0..000 0.0000
0.763 0..8.000002
0.7644a..00 0010
10.002
0.764
0.764 0.0002
diabetes
0.69310.000
0.675g0.001
0.664
20.001
0.67320.00
0.67320.002
U TRAINING AND SAMPLING TIMES DETAILS
Table 28: Training times minutes. TabDDPM produces NaNs during training on income and di abeete,, and is therefore excluded for these data. CoDi is considered prohibitively expensive to train on diabetes and lending and we report estimated (est.) training times.
CDTD (per feature)
SMOTE
ARF
CTGAN
TVAE
TabDDPM
CoDi
TabSyn
acsincome adult
80.3
59.9
26.0
231.9
13.4
5.8
7.4
36.2
23.7
38.3
48.3
32.7
6,9
bank
11.0
37.6
24.6
40.5
42.7
48.5
26.3
beijing
3.7
34.3
23.9
36.1
24.9
25.8
23.4
churn
0.3
27.1
13.7
18.2
25.7
21.5
6.1
covertype
130.2
58.0
36.5
44.9
69.2
30.7
28.2
default
12.0
38.3
24.8
38.9
45,9
40.1
26.4
diabetes
58.5
90.1
25.3
870 (est.)
34.6
26.9
lending
5.2
157.9
36.6
48.7
3000 (est.)
42.1
25.3
news
23.0
48.8
33.3
37.2
41.5
57.9
25.2
nmes
0.4
32.8
17.2
24.9
30.2
31.0
6.3
Table 29: Sample times in seconds per 1000 samples. TabDDPM produces NaNs during training on acsincome and diabetes, and is therefore excluded for these data. CoDi is considered prohibitively expensive train on di abetes and lending.
CDTD
SMOTE
ARF
CTGAN
TYAE
TabDDPM
CoDi
TabSyn
(per feature)
Income
4674.45
4.20
0.23
0.07
10.26
3.53
0.59
adult
10.71
1.78
0.31
0.16
0.82
3.65
0.88
0.56
bank
16.19
2.24
0.44
0.44
0.87
3.38
0.80
0.64
beijing
3.98
0.34
0.41
0.32
2.09
2.45
0.99
0.26
churn
0.52
1.00
0.40
0.24
0.95
2.78
0.80
0.39
covertype
10913.34
9.74
0.28
0.25
2.45
4.35
0.85
1.97
default
2.07
0.27
0.25
0.86
3.48
0.82
0.60
diabetes
166.75
5.87
0.53
0.15
0.83
1.33
lending
4.06
2.49
0.45
0,54
4.33
0.85
0.69
news
66.49
3.89
0.43
0.30
5.13
2.93
0.86
0.85
nmes
0.69
1.54
0.31
0.17
4.17
2.91
0.82
0.55
39

---

Published as a conference paper at ICLR 2025
Figures 24 and 25 show the benefit of deep generative models over SMOTE. Even though SMOTE often praised as simple, casy-to-use oversampling tool for tabular data, it relies on identifying nearest neighbors, making sampling very inefficient for larger datasets. As consequence, deem SMOTE to infeasible to use for the a as ncome and covertyp datasets. The figures also illustrate the performance edge of CDTD, in particular compared other diffusion-based models.
(sec / /00000000000 samples)
15
SMOTE
SMOTE
Sampling Time
1DDDDPM
ARF CoDi
CTGAN
ARF
CoDi
TVAE
TabDDPM
cCDTD
TabSyne
TVAE
CDTD
Tabsyne
CTGAN
0.70
0.75
0.80
0.85
0.90
0.95
1.00
1.0
1.2
1.6
1.8
2.0
Detection Score
Distance Correlation Matrices
(sec 11000 samples)
15
SMOTE
SMOTE
Sampling Time
CDRD
ARF
TabSyn
TabDDPM
CoDI
CTGAN
TVAE
CDRD
TabDDPM
TabSyne
ARF
CTGAN
IVAE
CoDi
0.02
0.04
0.08
0.005
0,010
0.015
0.025
JSD
WD
Figure 24: Average sample quality metries function of sampling time. Diffusion- prasue models are indicated orange. Only datasets for which we could retrieve results from models are included, this excludes acsinces icsme, covertype, diabetes, lending.
(sec 1000 samples)
15
SMOTE
SMOTE
Sameling Time
10
CTGAN
ARF
CoDi
TabDDPM
CTGAN
TVAE
CoDi
AR
TabDDPM
TabSyn
TVAE
CDTD
TabSyf
CDTD
0.68
0.70
0.72
0.74
0.76
0.85
0.86
0.87
0.88
0.89
0.90
F1
AUC
(sec/ 1000 samples)
15
SMOTE
SMOTE
Sampling Time
TabDDPM
5-
ARF
TaUDDPM
CoDi
0-
TaeSy
CDTD
CoDi
TVAE
TabSyn og TVAE
0.6
0.8
1.0
1.2
1.4
1.6
2.0
2.5
3.0
Figure 25: Average MI. efficiency metrics and DCR as function sampling time. Diffusion- based models are indicated in orange. The dotted line indicates the test set performance of the real data. Only datasets for which we could retrieve results from all models are included, this excludes acs acsincome, covertype, diabetes, lending.
40


Reviews:
[{"review_id": "wd0weu8sIe", "reviewer": "Reviewer_eakR", "summary": "This paper proposes a continuous diffusion model for mixed-type tabular data, which\ncombines score matching and score interpolation techniques and imposes Gaussian diffusion\nprocesses on both continuous and embedded categorical features. \n\nIt proposes several strategies to balance model capacity between continuous and categorical features, including\nweighted loss calibration, adjusted model initialization, and the adaptive noise schedule\ndesigns.", "questions": "The section that introduces learnable noise schedules is unclear. For example, its\nmotivation is confusing.   What does it mean that “given the same embedding dimension,\nmore noise is needed to remove the same amount of signal from embedding of features\nwith fewer classes”? \n\nCan you explain the difference between Feature-specific Noise Schedules and Adaptive Noise Schedules in more detail ?\n\n[post rebuttal comment]. Some of the questions above are addressed by the author. I am happy to upgrade my rate for the paper.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1. The problem is well motivated, and the intuition of this research is quite clear, which is to propose a unified pipeline to deal\nwith mixed type tabular data, which contains both continuous and categorical features.  In addition, the proposed strategies effectively deal with the imbalance and calibration between these two types of data.\n\n2. Mathematical derivations are quite solid and appear to be valid, though having not checked completely.  \n\n3. Implement comprehensive experiments to demonstrate the effectiveness of the overall pipeline as well as each design component.", "weaknesses": "Weaknesses:   \n\n\n1. The overall pipeline has limited novelty, as the key idea to push categorical data into\nembedding space and use a Gaussian diffusion process to deal with it is a common\npractice.\n\n\n2. The preliminary section 2.2 that introduces diffusion for categorical features is not as clear\nas that in 2.1. I wish the author to elaborate this part a bit more since this should be the\nbasic foundation of CDTD.\n\n\n3. The customization on tabular data in section 3.4 is quite heuristic, are they supported by\ntheoretical proof or are they simply determined from empirical results?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "SPBOtRqFg6", "reviewer": "Reviewer_2dwE", "summary": "The paper proposes a joint continuous diffusion model for mixed type tabular data. Due to the mixed type of data format, the paper designs a score matching method and a score interpolation method for both continuous and categorical data. It presents a loss calibration method to balance the losses between the continuous and categorical data, and proposes an adaptive noise schedule.", "questions": "See the weakness section", "limitations": "", "rating": 5, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "+ the paper is relatively easy to follow. The presentation of the paper is good\n+ comparing with baseline methods, the proposed method seems to be working relatively well, although I have not directly worked on diffusion model for tabular data. There could be related work I am missing\n+ contributions and limitations of the paper are properly discussed", "weaknesses": "- the scope of the method is relatively limited on just mixed type tabular data. Diffusion model for tabular data had already been studied (e.g. Kotelnikov, 2023), and the contribution of this paper seems to be rather incremental in the sense that it just makes the diffusion model work a bit better in mixed type tabular data\n- the method, although makes sense, is not novel in the whole landscape of diffusion models. Modeling continuous and categorical features has been studied in the past and many had been cited by the authors. This seems to be mostly applying these to this particular type of data", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "5be09G3Mo1", "reviewer": "Reviewer_cZJV", "summary": "This paper combines score matching and score interpolation to ensure a common continuous noise distribution for both continuous and categorical features in tabular data generation. The proposed method focuses on adjusting noise schedules to enhance performance in generating synthetic tabular data.", "questions": "Please see the weakness part", "limitations": "", "rating": 5, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1. Superior Performance: The results reported in the paper demonstrate strong performance on existing datasets, showing significant improvements compared to baseline methods.\n\n2. Simplicity and Clarity: The methodology centers on adjusting noise schedules, which is straightforward and easy to understand, making the approach accessible and replicable.", "weaknesses": "1. Limited Novelty: The primary method introduced is Adaptive Noise Schedules, an area that has been extensively explored in prior research. Additionally, the paper addresses the homogenization of different data types—a problem that has been widely studied in previous tabular data generation works [1],[2]. Consequently, the novelty of this paper is significantly diminished, and the authors need to clearly articulate the innovative aspects of their approach.\n\n2. Insufficient Experimental Reporting: The experimental results are not thoroughly reported. The authors compare their method with baseline approaches using only an average metric across various datasets, which leads to the loss of detailed information. To provide a more comprehensive evaluation, the authors should conduct detailed comparisons on each individual dataset, referencing methodologies from previous studies [1].\n\n[1] Zhang, Hengrui, et al. \"Mixed-type tabular data synthesis with score-based diffusion in latent space.\" arXiv preprint arXiv:2310.09656 (2023).\n\n[2] Kotelnikov, Akim, et al. \"Tabddpm: Modelling tabular data with diffusion models.\" International Conference on Machine Learning. PMLR, 2023.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ncpdqmQ0XC", "reviewer": "Reviewer_697T", "summary": "This paper trains a diffusion model for mixed-type (continuous and categorical) tabular data. The authors combine score matching with score interpolation to derive a diffusion model that add noise of categorical data in the embedding space, which makes it able to apply denoising process to mixed-type tabular data. To deal with the heterogenity of different types and features of data, the author further use noise calibration to balance the scale of continuous and categorical data. They also experimented on different types of noise schedules for better model training. The authors performed experiments and ablation studies on multiple datasets and used different metrics, the results showed that their model was able to achieve great performance on mixed-type tabular data.", "questions": "Please see weaknesses part. I would like to discuss with the authors on those points.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- The authors proposed the first model to allow diffusion models to train on mixed-type tabular data, and allows the use of advanced techniques such as classifier-free guidance in training.\n- To allow training on mixed-type tabular data, the authors have introduced several customizations to diffusion models: such as loss balancing, and noise schedules.\n- The experiments and comparisons look quite complete for me: the authors compared with seven methods on eleven datasets, using different kinds of metrics. The analysis of the results are also quite good. Results have shown that their method can achieve good results and higher efficiency.\n- The authors have provided details of all the implementation and experiments, they also provided their code in the supplementary. I believe this is quite useful for further researchers.", "weaknesses": "- The paper is more like a combination of existing models: diffusion models have already been applied to both categorical data and continuous data before. The authors combine them to deal with mixed-type data.\n- The introduced changes: reweighting losses and changing noise schedules, although useful, but I think they are a bit minor to be counted as big technical contributions. The main reason is that they are a bit straightforward and also limited to current data setting. I am also curious that whether the noise schedules are also useful for data beyond mixed-type tabular data.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)