TabNAT: A Continuous-Discrete Joint Generative Framework for Tabular Data
Authors: Hengrui Zhang
Liancheng Fang
Qitian Wu
Philip S. Yu
Venue: ICML 2025 poster (2025)
Keywords: Tabular data generation
missing value imputation
autoregressive models
Primary area: deep_learning->generative_models_and_autoencoders

Abstract: While autoregressive models dominate natural language generation, their application to tabular data remains limited due to two challenges: 1) tabular data contains heterogeneous types, whereas autoregressive next-token (distribution) prediction is designed for discrete data, and 2) tabular data is column permutation-invariant, requiring flexible generation orders. Traditional autoregressive models, with their fixed generation order, struggle with tasks like missing data imputation, where the target and conditioning columns vary. To address these issues, we propose Diffusion-nested Non-autoregressive Transformer (TabNAT), a hybrid model combining diffusion processes and masked generative modeling. For continuous columns, TabNAT uses a diffusion model to parameterize their conditional distributions, while for discrete columns, it employs next-token prediction with KL divergence minimization. A masked Transformer with bi-directional attention enables order-agnostic generation, allowing it to learn the distribution of target columns conditioned on arbitrary observed columns. Extensive experiments on ten datasets with diverse properties demonstrate TabNAT's superiority in both unconditional tabular data generation and conditional missing data imputation tasks.

TabNAT: A Continuous-Discret Joint Generative Framework for Tabular Data
Hengrui Zhang" Liancheng Fang
Qitian Wu? 2 Philip S. Yu'
Abstract
including VAES (Liu et al., 2023), GANs (Xu et al., 2019), diffusion models (Zhang et al., 2024b), and LLMs (Borisov et al., 2023) have been explored, the potential of autoregres- sive modeling the dominant paradigm in language gener- ation (Mann et al., 2020; Achiam et al., 2023)-remaine underexplored for tabular data. This oversight persists de- spite tabular data's 's natural compatibility with sequential decomposition: p(x) = II p(x* (x5t where each x' represents the i-th column of the table, and x denotes the set of "x',...,'''"" x. We identify two fundamental limitations hindering the adoption of autoregressive models: 1) Type-specific modeling mismatch: Traditional autoregres- sive frameworks excel at handling discrete tokens through categorical distributions but struggle with continuous values unless using restrictive parametric assumptions or discretiza- tions (Gulati & Roysdon, 2023). 2) Order-agnostic genera- tion: Unlike language, which follows an inherent sequence, tabular columns exhibit permutation invariance'. Existing solutions that enforce fixed generation orders (Castellon et al., 2023) limit flexibility and introduce information loss. Furthermore, traditional autoregressive models that assume fixed generation sequence struggle to adapt flexibly to tasks such as conditional sampling or missing data imputa- tion, where the target columns and their conditioning context may vary.
While autoregressive models dominate natural lan- guage generation, their application to tabular data remains limited due to two challenges: 1) tabular data contains heterogeneous types, whereas au- toregressive next-token (distribution) prediction is designed for discrete data, and 2) tabular data is column permutation-invariann requiring flexi- ble generation orders. Traditional autoregressive models, with their fixed generation order, struggle with tasks like missing data imputation, where the target and conditioning columns vary. To address these issues, we propose Diffusion-nested Non- autoregressive Transformer (TabNAT), hybrid model combining diffussio processes and masked generative modeling. For continuous columns, TabNAT uses a diffusion model to parameterize their conditional distributions, while for discrete columns, it employs next-token prediction with KL divergence minimization. A masked Trans- former with bi-directional attention enables order- agnostic generation, allowing it to learn the distri- bution of target columns conditioned on arbitrary observed columns. Extensive experiments on ten datasets with diverse properties demonstrate Tab- NAT's superiority in both unconditional tabular data generation and conditional missing data im- putation tasks.
On the other hand, Diffusion models (Ho al., 2020; Song et al., 2021; Karras et al., 2022; Austin al., 2021; Lou et al., 2024) are inherently well-suited for continuous data, as they iteratively denoise Gaussian noise to match the target distribution without imposing restrictive parametric assump- tions. However, they struggle with discrete data, requiring inefficient and less effective adaptations such as embedding or categorical diffusion (Zhang et al., 2024b). Additionally, their parallel denoising process eliminates the need for predefined generation order, naturally aligning with the per- mutation invariance of tabular data. While this parallelism enhances flexibility, it also limits the explicit modeling of column dependencies - a key strength of autoregressive approaches (see the comparison in Fig. 1).
1. Introduction
The growing demand for synthetic tabular data in critical applications such as
data sharing and augmented machine learning pipelines (Fonseca & Bacao, 2023; Hernandez et al., 2022) has spurred significant inter- est in deep generative models. While multiple architectures,
'Equal contribution 'Computer Science Department, Univer- sity of Illinois at Chicago, Chicago, United States "Eric and Wendy Schmidt Center, Broad Institute of MIT and Harvard, Cam- bridge, United States. Correspondence to: Qitian Wu <wuqi- tian @ broadinstitute. porg., Philip S. Yu eosyu@uic.cdie
To address these challenges, we propose Diffusion-nested Non-A Autoregressive Transformer (TabNAT), novel frame- work that seamlessly integrates diffusion and autoregres-
Proceedings of the 4nnd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 20255 by the author(s).
CColmmn order carries no semantic meaning

---

TabNAT: A Contenuous-Disertt Joint Generative Framcwork for Tabular Data
dataffoo
ctS observation
cts latent
cts prediction
mask
dataflow
bidirectional dataflow
- compute loss
dst observation
dst latent
dst prediction
Training
given generation order
Training
Sampling
Sampling
TabNAT supports both continuous and discrete variables through masked generative modeling. During train- ing (left), the model learns to predict conditional distribu- tions for masked positions in parallel. During sampling (right), the model generates values for masked positions sequentially following a given generation order. The bidi- rectional dataflow enables flexible conditioning on both past and future contexts, combining the benefits of parallel train- ing with controlled sequential generation.
Figure 2:
Diffusion
Augo-regressive
next token prediction
Figure Comparison of Diffusion and Auto-regressive models: (Left) Diffusion models suited for continuous (cts) data with parallel generation capabilities, allowing simulta- neous sampling across timesteps; (Right) An auto-regressive next-token prediction model designed for discrete (dst) data, requiring sequential generation one token at time.
contributions of this paper as follows:
1) We propose TabNAT, a novel framework that integrates diffusion models with Bi-directional masked Transformers for heterogeneous tabular data generation.
sive approaches for heterogeneous Tabular data generation (Fig. 2). The core of TabNAT's dependency modeling lies in its masked generative architecture, where bi-directional masked Transformer learns to generate conditional embed- dings for target positions, which subsequently guide the modeling of their conditional distributions. For continuous columns, TabNAT employs a conditional diffusion model to capture their distributions based on the generated embed- dings. For discrete columns, the model directly optimizes the categorical distributions by minimizing the KL diver- gence. This unified architecture enables TabNAT to handle mixed data types naturally and effectively. Furthermore, its -oo-euoorreressvvve training paradigm, combined with flexible autoregressive sampling capabilities, facilitates both efficient parallel generation and order-aware inference, mak- ing versatile for various inference tasks.
2) We develop a unified architecture that seamlessly handles both continuous and discrete variables through conditional diffusion modeling and categorical distribution learning, respectively.
3) Extensive experiments on diverse real-world datasets demonstrate that TabNAT achieves state-of-the-art perfor- mance in both unconditional synthetic data generation and conditional missing data imputation tasks. Our ablation studies further validate the effectiveness of each component in our proposed framework.
2. Related Works
We conduct comprehensive experiments datasets of various data types and scales to verify the efficacy of the proposed TabNAT. Experimental results comprehen- sively demonstrate TabNAT 's superior performance in three key areas: 1) Statistical Fidelity: The ability of synthetic data to faithfully recover the ground-trui data distribution; 2) Data Utility: The effectiveness synthetic data in down- stream Machine Learning tasks, i.e., Machine Learning Efficiency and 3) Privacy Protection: Ensuring that the synthetic data is sampled from the underlying distribution of the training data rather than being direct replication. In missing data imputation tasks, TabNAT achieves remark- able performance, even surpassing ofatc-of-the-a art methods specifically designed for such tasks. We summarize the
tabular
Synthetic Tabular Data Generation Generative models for tabular data have become increasingly important and have widespread applications (Assefa et al., 2021: Zheng & Charoenphakdee, 2022: Hernandez et al.. 2022). For exam- ple, CTGAN and TAVE (Xu et al., 2019) deal with mixed- type tabular data generation using the basic GAN (Goodfel- low al., 2014) and VAE (Kingma & Welling, 2013) frame- work. DP-TBART (Castellon al., 2023) and TabMT (Gu- lati & Roysdon, 2023) apply discretization techniques to numerical columns and then model the data distribution using Transformers (Vaswani et al., 2017), treating con- tinuous features as discrete ones. Recently, inspired by the success of Diffusion models image generation, lot of diffusion- based methods have been proposed, such as
2

---

TabNAT: A Continuous-Diseree Joint Generative Framcwork for Tabular Data
TabDDPM (Kotelnikov et al., 2023), STaSy (Kim et al., 2023), CoDi (Lee et al., 2023), and TabSyn (Zhang al., 2024b), which have achieved SOTA synthesis quality, The proposed TabNAT combines Diffusion models and Trans- formers. It utilizes Transformers to generate conditional inputs for target columns, employs a conditional diffusion model to model continuous columns, and uses a categorical distribution model to handle discrete columns.
Then, correspondingly, we use Xets RD to denote the continuous columns of x, xast E N to denote the i-th discrete column of x, where = 1. Dast. then
x = [eets X ... .Xatt ]. Dost
4. TabNAT
4.1. Overview
In this section, we introduce the detailed architecture and implementations of TabNAT for modeling the distribution of heterogeneous tabular data.
Autorrrsureeeeeeesssvv Models for Continuous Data Au- toregressive next-token generation is undoubtedly dominant in text generation (Mann et al., 2020; Achiam al., 2023). However, in image generation, although autoregressive mod- els (Van den Oord et al., 2016; Salimans et al., 2017) were proposed early on, their pixel-level characteristics limited their further development. Subsequently, diffusion models, which are naturally suited to modeling continuous distribu- tions, have become the most popular method in the field of image generation. In recent years, some studies have attempted to use discrete-value image tokens (van den Oord et al., 2017; Razavi et al., 2019) and employ autoregressive transformers for image-generation tasks (Kolesnikov et al., 2022; Tian et al., 2024). However, discrete tokenizers are both difficult to train and inevitably cause information loss. To this end, recent work has attempted to combine contin- uous space diffusion models with autoregressive methods. For example, Li et al. (2024b) employs autoregressive diffusion loss in causal Transformer for learning image representations; Li et al. (2024a) proposes using a diffusion model to model the conditional distribution of the next con- tinuous image and employs masked bidirectional attention mechanism to enable the non-eutoreeressvve generation of any number of tokens in arbitrary order.
Recall that our objective is to learn the joint distribution p(x) via distribution decomposition, e.g.,
p(x) = p(Xets: X
Dasc ast
Dena
(I)
= =p(Xets)
II
p(x Xetss xdst)
and the corresponding log- logllikeihhod is:
log p(x) Logp((xuu )
Nogp(xass (eates x (2)
Eq. 2 demonstrates that maximizing the likelihood of the data distribution can be achieved by maximizing the log- likelihood of each (conditional) distribution under the fac- torization. Given the permutation invariance property of tabular data, its joint distribution can be factorized along any arbitrary ordering. Consequently, this essentially re- quires us to learn every possible ((Xxergee\Xxeee where S represents an arbitrary subset of the remaining columns. TabNAT employs a conditional generative model based on a bi-directional masked transformer achieve the aforemen- tioned objectives. Fig. presents the overall framework of the proposed TabNAT, which takes table of mixed con- tinuous and discrete features as input and aims to model the distribution of masked positions conditioned on unmasked positions.
3. Problem Formulation and Notations
In this paper, we always use uppercase boldface (e.g.. X) letters to represent matrices, lowercase boldface letters (c.g., x) to represent vectors. Tabular data refers to data orga- nized in a tabular format consisting of rows and columns. Each row represents instance observation, while each column represents feature variable. In this work, we consider heterogeneous tabular data that may contain both numerical and categorical columns or only one of these types. Let D = (x) denote a tabular dataset, where each instance x (x1 ..,,..,,........................ DD is D-dimensional vector representing the values of D columns. We further cate- gorize the columns into two types: continuous (cts) columns; 2) discrete (dst) columns. Let Dets and Dast (D + Dest D) be the number of continuous/discrete columns, respectively. Without loss of generality and for the sake of convenience, we assume that the first Dete columns of the table represent continuous variables, while the remaining Dast columns represent discrete variables.
4.2. Model Architecture
Preprocessing and Embedding Layer. Raw input tables are typically unordered and irregularly structured (e.g., con- taining continuous features with varying scales and text- based discrete features). Therefore, in addition to the previ- ously mentioned assumption that the first Dets columns are continuous features and the remaining Dast columns discrete features, we first perform separate preprocessing steps for the continuous and discrete features, respectively.
For continuous columns, we standardize them to have zero mean and unit variance (i.c., a standard deviation of 1). For each discrete column i, we apply label encoding to transform
3

---

TabNAT: A Continuous-Diserde Joint Generative Framcwork for Tabular Data
M U(1,1 Dast). Given M, randomly sample masking vector mmask E {0, 1) s.t., E, E,mas M. " = indicates that position i is masked and = 0 vice versa. Then, the embeddings of masked positions are replaced with position-specific mask embeddings, e.g..
Conditional gerenative neceiing
Difilssion loss
Croes Entrapy lass
Iill
p(x:(z;)
Fets
Adas
Bi-directional Transformer
H mask (1 mask) H mask ① M,
(4)
position
Place
encoding
f
+
dP
+
Riazz
where M = [mets. mast dst
Dast is the masking em- bedding matrix, and each mets or m is a learnable mask embedding for the corresponding position.
mesking
[MASK)
(ND MASK]
[MASK]
[NO MASK!
R
umbeodding byer
Ercoden
Encader
Eneader
Eneeder
Position encodings. Similar conventional Trans- former architectures, TabNAT incorporates position encoding for each individual position, i.e., P Peta. ... ,.. Panl (D++)x))xd In our implemen- tation, we employ learnable positional encodings that are continuously updated during training. This approach en- sures permutation invariance for table columns, as these learnable positional encodings are column-speciin and inde- pendent of the relative positions between different columns.
Aetr
Pas
Ades
prepracess
0-1 Standardizatiin
Label Encodier
agc
tol gain
hour
gender
educst
work cless
input
row
3B
7,688
50
H5 ynal
Private
coneeeee peneonennnneneeee. discrete features
continuous features
The overall framework of TabNAT. After prepro- cessing, an embedding layer first encodes each column into vector. The masks are then added to the target columns ('age' and 'education"). With Bi-direction Transformers' decoding, the output vectors z are used as conditions for predicting the distribution of current columns. TabNAT uses the conditional diffusion loss for continuous columns and cross-entropy loss for discrete columns. Losses are com- puted only the masked positions
Figure 3:
Bi-directional Transformer. With position encodings P added to the masked embeddings Hmask we em- ploy a Bi-directional Transformer to capture the inter- relationships among different positions in the sequence, which subsequently generates an output embedding ma- trix of identical dimensionality the input. We let Z Ietta "est:... .uust R(1+Da.)xd be the output of the last layer, where Zets and zlase are the output embeddings at corresponding positions.
them into index codes ranging from 0 |C,| 1, where [C,] represents the number of possible categories of the i- th discrete feature. Thus, we can represent the continuous features as Xots E IR and the i-th discrete feature as
4.3. Loss function and Training
Recall that aim to learn p(Xearget (xyEs) Based on the aforementioned model design, Xtarget corresponds to a masked position, while Xjes represents the unmasked positions. Given that each masked position's 's correspond- ing 2 Larges inherently a function of the unmasked in- puts, TabNAT learns p(Xearget/ /taaget) as a remedy for p(Xtarget Xjes).
x E {0, 1., ,e,,1,,.,,.--11 13.
Subsequently, we employ distinct continuous and discrete encoders transform each type of variable into embeddings of identical dimensionality. For continuous columns, we utilize simple two-layer MLP model Pete IRd) to map the Det:- -dimensiona continuous features into a d- dimensional embedding. For the i-th discrete column, we allocate a d-dimensional (learnable) embedding vector to each distinct category. Finally, for each row of data x in the table, we obtain an embedding matrix as follows:
Discrete columns. For discrete column with input xLat and output ziar. the target distribution follows a categor- ical distribution of dimension |C,]. Therefore, we employ a shallow MLP model (Rd + RR to map Z to a (Cal- dimensional space, followed by the application of the stan- dard cross-entropy loss, i.e..
H [hetsr M b ..... Dae (IdDD
C CE(MPP dat (Z) )
(5)
where hets _d is the embedding vector of Xetss and hast is the embedding vector of x respectively. H as sequence of length 1 Dast is subsequently taken as the input of a Bi-directional Masked Transformer.
Continuous columns. For the continuous columns, Tab- NAT utilizes conditional diffusion model (Karras et al., 2022; Song et al., 2021) to learn p(Xots (ets), which has the following forward process:
Position masking. TabNAT utilizes mask embeddings to mark the target positions to predict. We first uni- formly sample the number of positions to mask/predict
x x +o(l)e N. x -Xats- (6)

---

TabNAT: A Continuous-Disertte Joint Generative Framcwork for Tabular Data
Algorithm Loss for discrete columns
Algorithm 3 TabNAT: Training Process
1: Input: Condition vector zast: target discrete value x',
1: Input: data x (Xato, Xasu.""
x dat
predictor MLP
2: Output: Model parameters
dst
2: Output: Loss Lasst
3: heta = embedding(xxtt)
3: Get Kast MLP dat (z')
4: for i E 1, 2. ... , do
4: Compute loss: Cist CE(Xgat xgst)
haat = embedding(x,; (x
end for
Algorithm 2 Loss for continuous columns
7: Let H [hets: hast. dat
8: Generate masking vector mmaax
1: Input: Condition vector Zets Rd. target continuous
9: Hmask = mmaak) ) @ H maak ① M
vector Xets E ID" .x-encoder, z-encoder, denoising
10: H' = -Hmase +P
network dL
11: = Transformers(HH)
2: Output: Lets
12: Compute Lets according to Algorithm 2
3: Sample t p(t)
13: for i E 1, Dast do
4: Sample E ~ N0,
14: Compute flast according to Algorithm 1
5: Get x Xcts + (tt) E
15: end for
6: x I- encoder(x (x (Le)
16: L Lets Mmask.ets + E Par Cast maaak
7: Zaa
encoder(zeta
f=1
17: Back-propagation to optimize model parameters
8: tan sinusoidal(t)
9: co Zets. t) = MLP d: (x + Zan. + tan))
10: Compute loss: Lets = || (x (x Rota, 1)
ineatcked
satched diserss
x-encoder
X
Then, the corresponding (conditional) reverse process is:
Zets
2-encoder
Zdn
MLP
+
Eg (xctsr Acts. t)
dx = 26 (t) log d(x'(xx + (f)())(())d)))
Trec erbeddigs
Cun
where is the standard Wiener process. (t) is the noise schedule, and (t) is the derivative of (t) w.r.t. t. In this paper, we set o (t) = t. Then, a sample can be obtained via integrating the reverse process from the maximum timestep We present the algorithms for com- puting the losses for discrete and continuous columns in Algorithm and Algorithm 2, respectively.
Figure 4: The architecture of the denoising function es (x Zets. Zetsst t). Since x'_., Zeis, and have distinct di- mensions, they are first mapped to the same dimension, then added as the input of an MLP. The output of the MLP eg (x Zeta, 1) is utilized to predict the noise
t T' until
A diffusion model is learned by using denoising/score net- work 6a (x Zeta, 1) to approximate the conditional score function Vx log p(x't Bets) (named score- score-matching matching). The final loss function could reduced to a simple formu- lation where the denoising network is optimized to approxi- mate the added noise i.e.,
4.4. Inference
After TabNAT is well-trained, we can easily perform vari- ety of inference tasks.
Unconditional data generation. To generate uncondi- tional data examples x ~ p(x), we can first randomly sam- ple a generation order, c.g. Xets + x ,, x Starting with all columns masked, we can first generate the values of all continuous columns, then generate the value of each discrete column one by one. It is worth emphasizing that TabNAT supports both traditional autoregressive next- token prediction and parallel generation of multiple tokens simultaneously, offering significant advantages in flexibility and efficiency.
Lote = Ee- | (x Zeta. t) - =|l2. (8)
In Fig. 4, we illustrate the implementation of our denoising function. Given that x E TR Das, Zets E Rd and t E IR possess different dimensionalities, we first project all of them into a shared space with dimension daire- For x and Zers, we use x-encoder and z-encoder, which essentially two learnable linear transformations, while for 1, we utilize the dimeessiieaio sinusoidal timestep embeddings. After that, they are added and serve as the input of a denoising MLP, which will subsequently output eo Ex t). We present the entire training algorithm of TabNAT in Algorithm 3.
Missing data imputation. Owing to TabNAT 's inherent capability to model various possible conditional distribu- tions, naturally supports conditional inference tasks, such as missing value imputation. The process is analogous to
5

---

TabNAT: A Contenuous-Disertt Joint Generative Framework for Tabular Data
Table 1: Comparison of different methods regarding the Statistical Fidelity and Machine Learning Efficiency of the synthetic data. All metrics have been normalized so that lower numbers indicate better performance, to facilitate better numerical comparison.
Method
Statistical Hidelity
MLE
Marginal] %
Joint %
Precision
Recall.
C2ST)
JSDL 10-2
Gapl
RMSE Gap! %
Interpofation
SMOTE (Chowla et al., 2002)
1.72
2.95
3.78
16.7
2.26
0.11
3.00aa.oc
0.15
VAE based
TYAE (Ku al. 2019)
16.8.1
17.4g1sa
18-2a00.
70.4g
43.5 43.0gmm
1.01
6-43aa
0.36
GOGGLE (Liu et 2023)
17.2
29.1
21.8
90.8ea..4
25. 25.01
0.16
GAN-buned
CTGAN eXu at..
17.9
18.4sa1
17.7axx.1
69.1esn.s
53.0=22.5
1.18
8.40
0.35
LLM-bused
GReaT (Berisow ol., 2023)
12.926.0
44.3
17.2
$0.2
42.4g15.a
1.43g1.18
1.90
0.23
Diffusion-bared
StaSy (Kim al., 2023)
14.327.40
13.0a
21 8224.7
35 Gart
1.25211
2.3422.94
0.17
CoDi al., 2023)
1744+1.
15.2+19u
10.0=n.co
51.7-su.1
41.0ess.
0.70epu
7.066112.1
0.21
T2&DDPM (Kotelnikrv et al., 2023)
15.0
7.92
20.6.
49.6g.545
24.6.aa.a
1.03
6.80
2.23
TahSyn (Zhang et al., 2024h)
17
1:00
2.06g
43.9gn4.
2.10
0.12g
1.19g)
0.13
Mong)-aattmgeestuubbooo DP-TBART (Castellon et al., 2023)
3.2
2 2.71+a.e
2.11=2.3
48 18.0ee
5.36ia4
0.16
2.96
0.15
Tab-MT MT (Gulati& & Roysdon, 2023)
14.9-15.1
8.11+104
23.2:31.9
63.5e3s.3
48.6
0.80
3.88
0.92
TaNNAT (ours)
1.00
1.71
2.01-1.10
48.4+m0
2.03-1.42
0.09
0.7 0.72+o..
0.09
Improvement
22. 8%
10.5%
4.7%
10.2%
18. 2%
39.5%
30.8%
unconditional generation, with the key difference being that now only requires autoregressive generation at the miss- ing positions in sequential order. It is noteworthy that when missing values are present in any continuous column, we mask the entire Xets/bett /hats vector. With bete, TabNAT uti- lizes the RePaint technique (Lugmayr et al., 2022; Zhang et al., 2024a), which iteratively refines the sampled val- ues during the reverse denoising process based on the ob- served continuous columns. The detailed illustration is in Appendix B.5. We present the experiments of unconditional data generation and missing data imputation in Sec. 5 and Sec. 6, respectively.
et al., 2023), TabDDPM (Kotelnikov et al., 2023), and TabSyn (Zhang et al., 2024b). 6) Autoregressive methods DP-TBART (Castellon et al., 2023) and Tab-MT (Gulati & Roysdon, 2023). The proposed TabNAT is close to both Diffusion and autoregressive methods, since it nests a diffu- sion model into a mnoo)-uutoregressvve framework in order to handle continuous features.
Evaluation. Following previous works (Zhang et al., 2024b), we evaluate the quality of synthetic tabular data across three key dimensions: 1) Statistical Fidelity: Whether the synthetic data accurately recovers the under- lying distribution of the ground-truth data. 2) Utility: The performance of synthetic data in downstream tasks, mea- sured by Machine Learning Efficiency (MLE). 3) Privacy Protection: Whether the synthetic data avoids copying real records, ensuring data privacy. Detailed descriptions of the evaluation metrics can be found in Appendix B.7. We each model and dataset, we sample 20 synthetic dataset of the same size as the training set. More implementation details are provided in Appendix B.4.
5. Experiments: Synthetic Tabular Data Generation
5.1. Experimental Setups
Datasets. We select ten real-world tabular datasets of vary- ing data types and sizes: 1) two containing only continuous features california and letter; 2) two containing only cat- egorical features - car and nursery; 3) six heterogeneous datasets of mixed continuous and discrete features adult, default, shoppers, magic, news, and beijing. The detailed introduction of these datasets can be found in Appendix B.2.
5.2. Main Results
Statistical Fidelity. We first investigate whether synthetic data can faithfully reproduce the distribution of the original data. We use various metrics from different aspects, includ- ing Marginal, Joint, a-Precision, B-Recall, C2ST, and JSD. Due space limitations, in this section, we only present the average performance with standard deviation each metric across all ten datasets. The detailed performance on each individual dataset is in Appendix C.I.
We compare the proposed TabNAT with ten powerful synthetic tabular data generation methods belong- ing to six categories. 1) The nor-parametric interpolation method SMOTE (Chawla et al., 2002). 2) VAE-based meth- ods TVAE (Xu et al., 2019) and GOGGLE (Liu et al., 2023). 3) GAN-based method CTGAN (Xu al., 2019). 4) LLM- based method GReaT (Borisov et al., 2023). 5) Diffusion- based methods: STaSy (Kim et al., 2023), CoDi (Lee
Baselines.
In Table 1, we present the performance comparison on these
6

---

TabNAT: A Continouss-iserree. Joint Generative Framework for Tabular Data
Table 2: Impacts of the key components of TabNAT.
Table 4: Comparison of training, sampling time, and fidelity metrics of different methods on Adult dataset.
Variants
Marginal
Joint
with MSE loss
12.31%
18.27%
Method
Training
Sampling
Marginal
Joint
with Gaussian
5.44%
7.59%
CTGAN
1030s
0.8621s
17.9%
18.4%
with fixed order
1.86%
2.74%
TVAE
353s
0.5118s
15.8%
17. 17.4%
GRea'
2h 27min
2min 19%
12.9%
44.3%
TabNAT 1.05%
1.71%
STaSy
2283s
8,941s
14.3%
13.5%
CoDi
2h 56min
4.616s
17.4%
15.2%
TabDDPM
1031s
28.92s
15.0%
7.925%
Table 3: Impacts of the model depth (number of Transformer layers) and width (embedding dimension d) on the fidelity of synthetic data.
TahSyn
2422s
11.84s
1.36%
1.91%
TR-TBART
1855s
2.11s
8.24%
2.71%
Tab-MT
1882s
2.18s
14.9%
8.11%
TabNAT
2754s
15.69s
1.05%
1.71%
Depth
Marginal
Joint
Dim
Marginal
Joint
and the proposed TabNAT. As demonstrated, SMOTE per- forms poorly in privacy protection, as its synthetic sample tends to be closer to the training set rather than the holdout set, since its synthetic examples are obtained via interpola- tion between training examples. By contrast, the remaining deep generative methods are all effective at preserving the privacy of the training data, resulting in almost completely overlapped DCR distributions between the training set and the holdout set.
2
1.79%
2.85%
8
2.88%
3.79%
4
1.44%
2.37%
16
1.28%
2.09%
1.05%
1.71%
32
1.05%
1.71%
1.1%
2.13%
64
1. 1.22%
1.93%
fidelity metries. As demonstrated, TabNAT achieves perfor- mance far surpassing the second-best method in five out of six fidelity metrics, with advantages ranging from 10.5% to 39.5%. Considering that these metrics have already reached a considerably high level due to the rapid development of recent deep generative models for tabular data, our improve- ment is substantial. The only exception is 3-Recall, on which the simple interpolation method SMOTE achieves the best performance. This is because they perform linear or nonlinear interpolation directly between existing real sam- ples, so the generated samples will definitely fall within the support set of the real data distribution, making them more likely to have similar feature distributions to the real minor- ity class samples. However, the limitation of interpolation methods lies in the fact that the synthetic data is too close to the real data, making it resemble copy from the training set rather than a sample from the underlying distribution, which may cause privacy issues. Detailed experiments are in the Privacy Protection section.
5.3. Ablation Studies
Effects of the key components of TabNAT. We first study if TabNAT benefits from 1) the diffusion loss and 2) arbi- trary generation order in the synthetic data generation task. To validate the importance of the diffusion loss, we con- sider two alternative approaches: (1) directly predicting the values of continuous columns using an MSE loss, and (2) parameterizing the conditional probability of each contin- uous column as a Gaussian distribution, with the model predicting its parameters. To investigate the benefits of arbi- trary generation order, we employed fixed causal masking during training based on the default column ordering of the table and generated synthetic tables following this prede- fined sequence. In Table 2, we compare TabNAT with the three variants on the average synthetic data's 's fidelity w.r.t. Marginal and Joint across all the datasets. We can observe that both the diffusion loss and random training/gennraaionn order important to TabNAT 's success. Specifically, the diffusion loss targeting the modeling of the conditional dis- tribution of the continuous columns contributes the most, and the random ordering further improves TabNAT 's perfor- mance. Furthermore, random ordering enables TabNAT to perform flexible conditional inference tasks like imputation.
Utility on Downstream Tasks. We then evaluate the quality of synthetic data by assessing its performance in the Ma- chine Learning Efficiency (MLE) task. The performance dif- ference is measured by the difference between AUC/RMSE obtained from synthetic data and AUC/RMSE obtained from real training data. As demonstrated in Table 1, the proposed TabNAT achieves satisfactory performance on MLE tasks, yielding over 30% improvement over the most competitive baselines. These results demonstrate that the synthetic data generated by TabNAT is comparable to real training data for training machine learning dlassfficttoonrregrsssoo models.
Sensitivity to hyperparameters. We then study TabNAT sensitivity its hyperparameters that specify its transformer architecture: the model depth (number of Transformer lay- ers) and the embedding dimension d. From Table 3, we can observe that although there is an optimal hyperparameter set- ting (i.e.. depth = 6, d= 32), changes in these two hyperpa- rameters do not significantly harm the model's 's performance. These results demonstrate that our model is relatively robust
Privacy Protection. Finally, we use the Distance to Closest Record (DCR) score for measuring the capacity of TabNAT in privacy protection. In Table 5 and Figure 5, we present the numerical comparison and the visualizations of the DCR distributions of SMOTE, TabDDPM, TabSyn, DP-TBART,
7

---

TabNAT: A Contenoous-Dieerte Joint Generative Framcwork for Tabular Data
SMOTE
TabDDPM
TabSyn
12.5-
Holdout 9ith
Hold Tan
Hold
Table 5: Probability that syn- thetic example's s DCR to the train- ing set is smaller than that of the holdout set, value closer to 50% better.
10.9
Wole
Yein
7.5
5.0
2.5
0.0
0.0
Method
Default
Shoppers
Hed--r Vhin
0.8
Hil our
Hold out Vall
HG5 our
SMOTE
91.41%
96.40%
Shoppis
0.5
Ta
51.30%
51.74%
0.4
0.4
0.4
0.4
TabSyn
50.88%
51.50%
0.2
DP-TBART
50.83%
51.27%
0.0
0.0
0.0
0.0
Distribution of DCR
Distribution of DCR
Distribution of DCR
Distribution of DCR
Distribution of DCR
TabNAT
51.13%
50.97%
Figure 5: Distribution of DCR scores from training and holdout set
TabNAT
Remasker
Hyperimpute
MOT
GRAPE
KNN
0.6
0.6
MAE
0.4
0.4
0.2
0.2
0.8
Accuracy
0.6
0.6
0.4
0.5
Adult
Default
Shoppers
News
Beijing
Average
Figure 6: (30% MCAR) Comparison of missing value imputation performance with SOTA imputation methods on five heterogeneous datasets. Top: Average MAE on continuous features. Bottom: Average Accuracy on discrete features.
to these hyperparameters. Therefore, good results be obtained for different datasets without the need for specific hyperparameter tuning.
6. Experiments: Missing Data Imputation
6.1. Experimental Setups
Comparison of training/sampling time. In Table 4, we compare the training and sampling time of TabNAT with other methods on the Adult dataset, which contains 6 con- tinuous columns, 8 discrete columns and 22, 792 samples.
Datasets. We select five datasets in Sec. 5.1 that contain heterogeneous data types: Adult, Default, Shoppers, News, and Beijing. Since TabNAT is not primarily designed for this task, only consider an out-of-sample imputation setting, where the complete training data is observable, and we aim to predict the missing entries in the testing set. The missing data mechanism follows Missing Completely at Random (MCAR), with a missing rate 30% applied to the test set. Specifically, each column has independent probability of 0.3 of containing missing values.
As shown in the table results, despite employing dual architecture combining diffusion and autoregressive mod- eling, TabNAT ' training speed does not significantly lag behind other competitive models such as TabSyn and DP- TBART. In terms of sampling efficiency, since the primary computational overhead lies the diffusion process for continuous features, and TabNAT requires only a single dif- fusion pass to sample all continuous columns, its sampling speed is comparable to TabSyn. Considering the substantial improvement in synthetic data quality achieved by TabNAT, this trade-off is highly justified.
Baselines. We compare the proposed TabNAT with the state- of-the-art (SOTA) methods for missing data imputation, including KNN (Pujianto et al., 2019), GRAPE (You et al., 2020), MOT (Muzellec et al., 2020), HyperImpute (Jarrett et al., 2022) and Remakser (Du et al., 2024).
8

---

TabNAT: A Contnnouus-iierrte Joint Generative Framcwork for Tabular Data
CoDi
STaSy
TabDDPM
TabSyn
DF-TBART
TabNAT
Ground Truth
cducation num
12
8
ag
age
age
age
age
age
age
Figure 2D joint distribution density of 'age' and 'education num' of Adult dataset.
STaSy
TabDDPM
TabSyn
DR-TBART
TabNAT
Ground Truth
40
lattude
38
36
-120 Longitude
-115
-120 Longitude
-115
-120 Longitude
-115
-120 Longitude
-115
-120 Longitude
-115
-120 Longitude
-115
-120 Longitude
-115
Figure 2D joint distribution density of 'Longitude" and 'Latitude' of California dataset.
Coll
7. Visualizations
Mi
In this section, we visualize the synthetic data to demon- strate that the proposed TabNAT can generate synthetic data that closely resembles the ground-truth distribution. Fig- ure and Figure 8, we plot the 2D joint distribution of two columns of the Adult and California datasets investigate if the ground-truth joint distribution density can be learned by the synthetic data. In Figure 9, we further plot the heatmaps of the estimation error column pair correlations. These results visually demonstrate that TabNAT can generate syn- thetic samples very close the distribution of real data and faithfully reflect the correlations between different columns of the data.
bofout
009
Lotto
Majic
2005
1.00
Figure 9: Heatmaps of the joint column correlation estima- tion error. Lighter areas indicate a lower error in estimating the correlation between two columns.
8. Conclusion
We present TabNAT, a generative model that nests a diffu- sion model within bidirectional transformer framework used for heterogeneous tabular data synthesis. TabNAT uses conditional diffusion loss and traditional cross-entropy loss learn the conditional distributions of continuous columns and discrete columns, respectively, enabling sin- gle model to generate both continuous and discrete features simultancously. Furthermore, TabNAT employs masked bidirectional attention to simulate arbitrary autoregressive orders, allowing the model to generate in any direction. This not only enhances the accuracy of joint probability modeling but also enables more flexible conditional generation.
6.2. Main Results
In Figure 6, present the performance comparison on each of the five datasets as well as the average imputation perfor- mance. Since these datasets contain both continuous and dis- crete features, we employ the Mean Absolute Error (MAE) metric to evaluate imputed continuous features, while using accuracy as the evaluation metric for discrete features. As demonstrated, the proposed model exhibits superior per- formance across all five datasets. achieving comparable better results than current state-of-the-ae methods for each data category within each dataset. These results confirm that TabNAT not only suitable for unconditional generation but also applicable to other conditional generation tasks, demonstrating wide range of application values.

---

TabNAT: A Continuous-Disercte Joint Generative Framcwork for Tabular Data
Impact Statement
Dosovitskiy, A. An image is worth 16x16 words: Trans- formers for image recognition scale. In International Conference on Learning Representations, 2021.
This paper presents work whose goal to advance the field of Machine Learning. There are many potential societal consequences of our work, none which feel must be specifically highlighted here.
Du, T., Melis, L., and Wang. T. Remasker: Imputing tab- ular data with masked autoencoding. In International Conference on Learning Representations, 2024.
Acknowledgement
Fonseca, and Bacao, Tabular and latent space synthetic data generation: literature review. Journal of Big Data, 10(1):115, 2023.
This work is supported in part by NSF under grants III- 2106758, and POSF-23461.88
Goodfellow, I. J.. Pouget-Abadie, J., Mirza, M.. Xu, B., Warde-Farley, D., Ozair, S., Courville, A., and Bengio, Y. Generative adversarial nets. In Advances in Neural Information Processing Systems, pp. 2672-2680, 2014.
References
Abadi, M., Chu, A., Goodfellow, I., McMahan, H. B., Mironov, I., Talwar, K., and Zhang, L. Deep learning with differential privacy. In Proceedings of the 2016 ACM SIGSAC conference computer and communications security, pp. 308-318,200 2016.
Gulati, M. S. and Roysdon, F. Tabmt: generating tabular data with masked transformers. In Advances Neural Information Processing Systems, pp. 46245-46254, 2023,
Achiam, J., Adler, S., Agarwal, S., Ahmad, L.., Akkaya, I., Aleman, F. L., Almeida, D., Altenschmidt, J., Altman, S.,
Hernandez, M., Epelde, G., Alberdi, A., Cilla, R., and Rankin, D. Synthetic data generation for tabular health records: A systematic review. Neurocomputingg 493: 28-45, 2022.
Anadkat, S., al. Gpt-4 technical report. arXiv preprint arXiv:2303.088774, 2023.
Alaa, A.. Van Breugel, B., Saveliev, E. S., and van der Schaar, M. How faithful is your synthetic data? sample- level metries for evaluating and auditing generative mod- els. In International Conference on Machine Learning, pp. 290-306. PMLR, 2022.
Ho, J., Jain, A., and Abbeel, P. Denoising diffusion prob- abilistic models. In Advances in Neural Information Processing Systems, pp. 6840-6855 2020.
Jarrett, D., Cebere, B. C., Liu, T. Curth, A., and van der Schaar, M. Hyperimpute: Generalized iterative imputa- tion with automatic model selection. In International Con- ference on Machine Learning, pp. 9916-9937. PMLR, 2022.
Assefa, S. A., Dervovic, D., Mahfouz, M., Tillman, R. E., Reddy, P., and Veloso, M. Generating synthetic data in finance: Opportunities, challenges and pitfalls. In Proceedings of the First ACM International Conference on Al in Finance, ICAIF '20. Association for Computing Machinery, 2021. ISBN 9781450375844
Karras, T., Aittala, M., Aila, T., and Laine, S. Elucidating the design space of diffusion-based generative models. In Advances in Neural Information Processing Systems, pp. 2022.
Austin, J., Johnson, D. D., Ho, J., Tarlow, D., and Van Den Berg, Structured denoising diffusion models in discrete state-spaces. Advances in neural information processing systems, 1t:11791117777 2021.
Kim, J., Lee, C.,, and Park, N. Stasy: Score-based tabular data synthesis. In International Conference on Learning Representations, 2023.
Borisov, V., Sessler, K., Leemann, T., Pawelczyk, M., and Kasneci, Language models are realistic tabular data generators. In International Conference on Learning Representations, 2023.
Kingma, D. P. and Ba, J. Adam: A method for stochastic optimization. In International Conference on Learning Representations, 2015.
Castellon, R., Gopal, A., Bloniarz, B., and Rosenberg, D. Dp-tbart: A transformer- based autoregressive model for differentially private tabular data generation. arXiv preprint XrXiv:2307.11 10430, 2023.
Kingma, D. P. and Welling, M. Auto-encoding variational bayes. arXiv preprint arXiv:1312.6114, 2013.
Kolesnikov, A., Susano Pinto, A., Beyer, L.,, Zhai, X., Harm- sen, J., and Houlsby, N. Uvim: A unified modeling approach for vision with learned guiding codes. Ad- vances in Neural Information Processing Systems, 35: 26295-26308, 2022.
Chawla, N. V., Bowyer, K. W., Hall, L. O., and Kegelmeyer, W. P. Smote: synthetic minority over-sampling technique. Journal of artificial intelligence research, 16:321-357, 2002.
10

---

TabNAT: A Continuous-Disertt Joint Generative Framcwork for Tabular Data
Kotelnikov, A., Baranchuk, D., Rubachev, I., and Babenko, A. Tabddpm: Modelling tabular data with diffusion mod- els. In International Conference on Machine Learning, pp. 17564- 17579. PMLR, 2023.
Song, Y., Sohl-Dickstein, J., Kingma, D. P., Kumar, A., Er- mon, S., and Poole, B. Score-based generative modeling through stochastic differential equations. In International Conference on Learning Representations, 2021.
Lee, C... Kim, J., and Park, N. Codi: Co-evolving con- trastive diffusion models for mixed-type tabular synthesis. In International Conference on Machine Learning, pp- 18940-18956. PMLR, 2023.
Tian, K., Jiang, Y., Yuan, Z., Peng, B., and Wang, L. Visual autoregressive modeling: Scalable image generation via next-scale prediction. Advances in neural information processing systems, 37:84839 84865, 2024.
Li, T., Tian, Y., Li, H., Deng, M., and He, K. Autoregressive image generation without vector quantization. Advances in Neural Information Processing Systems, 37:56424- 56445, 2024a.
Van den Oord, A., Kalchbrenner, N., Espeholt, L.,, Vinyals, O., Graves, A., al. Conditional image generation with pixelenn decoders. Advances in neural information pro- cessing systems, 29, 2016.
Li, Y., Bornschein, J., and Chen, T. Denoising autoregres- sive representation learning. In International Conference on Machine Learning. PMLR, 2024b.
van den Oord, A., Vinyals, O., and Kavukcuoglu, K. Neural discrete representation learning. In Advances in Neural Information Processing Systems, pp. 6309-63 2017.
Liu, T., Qian, Z., Berrevoets, J., and der Schaar, M. Goggle: Generative modelling for tabular data by learn- ing relational structure. In International Conference on Learning Representations, 2023.
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., and Polosukhin, Atten- tion is all you need. In Advances in Neural Information Processing Systems, pp. 5998-6008, 2017.
Lou, A., Meng, C., and Ermon, S. Discrete diffusion mod- eling by estimating the ratios of the data distribution. In International Conference on Machine Learning, pp. 92819-32844 PMLR, 2024.
Xu, L., Skoularidou, M., Cuesta-Infante, A., and Veera- machaneni, K. Modeling tabular data using conditional gan. In Advances in Neural Information Processing Sys- tems, pp. 7335-7345 2019.
Lugmayr, A., Danelljan, M., Romero, A., Yu, F., Timofte, R., and Van Gool, L. Repaint: Inpainting using denoising diffusion probabilistic models. In JEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 11461- 1 11471, 2022.
You, J., Ma. X., Ding, Y.. Kochenderfer, M. J., and Leskovec, Handling missing data with graph repre- sentation learning. Advances in Neural Information Pro- cessing Systems, 33:1 13:119755-19000 2020.
Zhang, H., Fang. L., and Yu, S. Unleashing the potential of diffusion models for incomplete data imputation. arXiv preprint arXiv:2405.20660 2024a.
Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., Agarwal, S., et al. Language models are leanerss arXiv preprint arXiv: /4165, 2020.
Zhang, H., Zhang, J., Shen, Z., Srinivasan, B. Qin, X., Faloutsos, C., Rangwala, H., and Karypis, G. Mixed- type tabular data synthesis with score-based diffusion in latent space. In International Conference on Learning Representations, 2024b.
Muzellec, B., Josse, J., Boyer, C., and Cuturi, M. Miss- ing data imputation using optimal transport. In Interna- tional Conference on Machine Learning, pp. 7130-7140. PMLR, 2020.
Pujianto, U., Wibawa, A. P., Akbar, M. I., et K-nearest neighbor (k-nn) based missing data imputation. In 2019 5th International Conference on Science in Information Technology (ICSITech), pp. 83 88. IEEE, 2019.
Zheng, S. and Charoenphakdee, N. Diffusion models for missing value imputation in tabular data, arXiv preprint arXiv:2210.11128, 2022.
Razavi, A., van den Oord, A., and Vinyals, O. Generating diverse high-fidelity images with vq-vae-2. In Advances in Neural Information Processing Systems, pp. 14866- 14876, 2019.
Salimans, T.. Karpathy, A., Chen, X., and Kingma, D. P. Pixelenn++: Improving the pixelenn with discretized lo- gistic mixture likelihood and other modifications. arXiv preprint arXiv: /70.0557.. 2017.

---

TabNAT: A Continuous-Disertte Joint Generative Framcwork for Tabular Data
A. Diffusion SDEs
code is written in Python 3.10.14 and we use PyTorch 2.2.2
CUDA 12.2 train the model the GPU.
This paper adopts the simplified version of the Variance- Exploding SDE in (Song et al., 2021). Song al. (2021) has proposed the following general-form forward SDE:
B.2. Datasets
The dataset used in this paper could be automatically down- loaded using the script in the provided code. We use 10 tabular datasets from Kaggle² or UCI Machine Learning Repository": Adult', Default', Shoppers Magic", Beijing" and News", California Letter'l Carl2, and Nursery which contains varies number of numerical and categori- cal features. The statistics the datasets are presented in Table 6.
dx f(x, t)dt + g(t) dw, dx f(t) dt + g() dw
(9)
Then the conditional distribution of x, given X (named as the perturbation kernel of the SDE) could be formulated as:
p(x:| xo0 = (x;;;; x0, s² (t)),
(10)
where
s(t) exp (L f(E)dE). and (t) V. (5) s²(E) de.
In Table 6, # Rows denote the number of rows (records) in the table. # Continuous and # Discrete denote the number of continuous features and discrete features, respectively. Note that there is an additional # Target column. The target columns are either continuous or discrete, depending on the task type. All datasets (except Adult) are split into training and testing sets with the ratio seed. As Adult has its official testing set, directly use it as the testing set. For Machine Learning Efficiency (MLE) evaluation, the training set will be further split into training and validation split with the ratio 8
(11)
Therefore, the forward diffusion process could be equiva- lently formulated by defining the perturbation kernels (via defining appropriate s()) and (t)).
with a fixed random
Variance Exploding (VE) implements the perturbation ker- indicating that the noise
Eq. 10 by setting stt) directly added the data rather than weighted mixing. Therefore, the noise variance (the noise level) is totally de- cided by (t)) When s(t) = 1, the perturbation kernels become:
B.3. Detailed Model Architectures
p(x, xo) = (x,;; 0, o (()T) = x = xp + (f)e, (12)
In this section, we introduce the detailed architecture of TabNAT.
which aligns with the forward diffusion process in Eq. 9.
The sampling process of diffusion SDE is given by:
dx = (xx V log p. (x)]dt + (()dw,, (13)
Data Preprocessing. Then continuous columns are trans-
For VE-SDE, s() + f(x,1) f(t) 0, and + 1 (2(()d5 =t)
formed follow a normal distribution by QuantileTrans,
(tt) = V V. (())dE +
httpps://wwwkagggle
www.iiss.....ee.eeeooeo htw.e//aaaceeeeeettttttttt. Phtcps://archiie..cs..u
(t) = dt²(t) = (t)
(14)
adult
dt
de
(t) (t)
468/ 4 http:m/iiieesspppppppprrrreeennn
Plugging gt) into Eq. 13, the reverse process is recovered
dataset. httpp://aecaiee...s....ddad/aaaa
as:
dx, (1) log + ())
maglcctgammatteeescce
auttpa://archivee.ccs. http:://aaraaiee......uaaaeaaaaa http::/ww.....c.ccctaatttttttat. wdw.ed/aaaattt//11111
(15)
beijjlng+pm2+5+ddaaa
The conditional version of the reverse process shown in Eq.
"nttpe::/archhvve.ess. uci
obtained via replacing Vx, dt with V (x,
ouLLinetaesstpppuaarrty
"htres//www.kaggeeccc//aaaaaaaee
B. Detailed Experimental Setups
"ettee+*rcognnition
B.I. Hardware Specification and Environment
cartevaluatinn http://rrhhiieeeee.c.ccceeeaaaae
We run our experiments on a single machine with Intel i9- 14900K, Nvidia RTX 4090 GPU with 24 GB memory. The
Pntsps://archive.
nursery
12

---

TabNAT: A Continuous-Diserce Joint Generative Framcwork for Tabular Data
Table 6: Dataset statistics.
Dataset
# Rows
# Continuous
# Discrete
# Target
# Train
# Test
MLE Task
California
20, 6410
9
1
18, 390
2, 520
Classification Classification
Letter
20, 000
16
1
18, 000
2, 000
Car
728
1
1, 555
173
Classification
Nursery
12, 960
1
11, 664
1, 2,6
Classification
Adult
32,561 561
6
8
1
22, 792
16, 281
Classification
Default
300
14
10
1
27,000
3,000
Classification
Shoppers
12,
10
7
1
11,098
1,232
Classification
Magic
19,02 021
10
1
17,118
1,903
Classification
Beijing 43, 824
7
5
1 1
39,
4,383
Regression
News 39, 644
46
35, 679
3, 3,965
Regression
former and discrete columns are encoded as integers by OrdinalEncoder"'. Finally, we normalize the continuous features to have zero mean and unit variance.
Given the output embeddings from the Trans- former layers, i.c. Zets and za we predict the distribution of the corresponding column(s) conditioned on z
Predictors.
For each discrete column, we apply simple MLP predictor MLP (+), each being two-layer MLP with hidden dimen- sions d - d - Cassel with SiLU activation. |Case is the number of possible categories of the i-th discrete columns.
B.3.1. TRANSFORMER PART
Embedding layer We an embedding layer to trans- form the continuous features and each discrete feature into the same embedding dimension d.
For the continuous columns, Zets will used as the con- dition of the diffusion model, and we defer this part to Appendix B.3.2.
For the continuous features Nets E IR Dats, we use a simple two-layer MLP to map Xets E IR Dere to
The MLP has hidden dimensions Dato + Zets E <dd.
B.3.2. DIFFUSION PART
Dots + with SiLU activations.
In this section, we introduce the architecture of the diffusion model. In nutshell, we use simple MLPs as our denoising neural network, which is similar to the design in Zhang et al. (2024h) and (Kotelnikov et al., 2023), the only difference is our denoising network takes an additional input z' as the conditional information.
For each discrete column x we assign learnable embedding vector E Rdd for every possible category.
Transformer layers After embedding and masking, add column-wise positional encoding to each token embed- ding. Furthermore, we append the [pad] token embedding at the beginning of the obtained data sequence. The pro- posed data will be further processed by a series of Trans- former blocks.
Encoders. Given x E R.Dr. Zets E Rdd and t R, we use several encoders to map them into the same dimension dairt for diffusion.
For xf ot from R to Rdarr x! encoder(x{,
R.ss --encoder is linear transformation
We use ViT (Dosovitskiy, 2021) the backbone of the Transformer layers, which consists of multiple Transformer blocks (Vaswani et al., 2017). Each Transformer block contains multi-head
For Zets E IRd z-encoder is linear transformation from Rd to Rddir Zán
encoder(zees)
mechanism and feed- forward network. Specifically, we use a stack of six Trans- former blocks with four attention heads.
For t E I, we use sinusoidal timestep embedding of dimension daree directly. tän sinusoidal
Mattesr/seckkt--lam..cr///a mecell/e/eennaeee////lee.ppppsee Dhantslleerranfffforrr...tt
Denoising MLP. The denoising neural network takes the summation of x Zan, and tan as input. The neural net- work a four-layer MLP with SiLU activations, and has the following hidden dimensions daiss - x daiss + da:te - dasst.
"/ttpes//ecckttt-eaam.oo////
medellle/ggnnratee///taaamereeee Brdinallkncodrr.htm
13

---

TabNAT: A Continuous-Disertte Joint Generative Framcwork for Tabular Data
B.4. Hyperparameters.
and m is the mask indicator vector, whose 1-entries indicate the unknown locations while Olentries indicate the known locations.
TabNAT uses a fixed set of hyperparameters for all datasets. Table shows the hyperparameters. Our experi- ments show that TabNAT is robust to the choice of hyper- parameters, saving the time of meticulous hyperparameter tuning for each dataset.
The values of unknown continuous columns can be gradu- ally sampled via the reverse process in Eq. 17, conditioned on the known continuous columns.
Type
Parameter
Value
B.6. Baseline Implementations
optimizer
Adam
Tab-MT and DP-TBART. Tab-MT (Gulati & Roysdon, 2023) and DP-TBART (Castellon et al., 2023) are two re- cently proposed tabular data generation models based on MAE. To handle numerical features (with continuous dis- tribution), Tab-MT quantizes the numerical features into 100 uniform bins, and DP-TBART quantizes the numerical features into 100 bins where each bin has the same nearest center determined by K-means. Additionally, DP-TBART employs DP-SGD (Abadi et al., 2016) to enhance the differ- ential privacy performance. Since the focus of this paper not on differential privacy, in our implementation, we use Adam (Kingma & Ba, 2015) optimizer.
initial learning rate
le-3
weight decay
le-6
Training
I.R scheduler
training epochs
5000
batch size
1024
# #Transformer blocks
embedding dim d
32
Transformers
diffusion dimension duier
512
#heads
4
Table 7: Default hyperparameter setting of TabNAT.
B.5. Missing Data Imputation
Other
Baselines. The implementations of SMOTE (Chawla et al., 2002), CTGAN (Xu et al., 2019), TVAE (Xu et al., 2019), GOOGLE' (Liu et al.,
We formulate missing data imputation as a conditional gen- eration task. For each data example with missing entries, we use TabNAT to sample all the missing entries for 20 times, then use the average of the 20 samples as the predicted imputations.
2023), GReaTT (Borisov et al., 2023), CoDi (Leee et al.,
2023), STaSy (Kim et al., 2023), TabDDPM (Kotelnikov et al., 2023), TabSyn (Zhang et al., 2024b) follows the codebase of Zhang et al. (2024b)
For a data sample of missing entries, we generate its con- ditional example following a random order. At each step, the target position is a discrete distribution, we directly sample from the categorical distribution given by zazci
B.7. Metries
Most of the metrics (including Marginal, Joint, a-Precision, 3-Recall, C2ST, MLE, and DCR) used in this paper di- rectly follow the setups in Zhang al. (2024b). Here is reference:
Softmax(MLPI.. (z)st)
(16)
If a continuous column is missing, mask the entire tinuous columns to generate Sets, and sample the miss- ing continuous columns using RePaint technique (Lugmayr et al., 2022; Zhang et al., 2024b).
Marginal: Appendix E.3.1 in Zhang al. (2024b).
Joint: Appendix E.3.2 in Zhang et al. (2024b).
a-Precision and 3-Recall: Appendix F.2 in Zhang et al. (2024b).
Let's denote X known continuous columns as Xknowns and the missing ones as Xunknown- the reverse step is mod- ified mixture of the known part's forwarding and the unknown part's denoising:
C2ST: Appendix F3 in Zhang et al. (2024b).
MLE: Appendix E,4 in Zhang et al. (2024b).
Knnwwn = Xots + (0
DCR: Appendix F.6 in Zhang et al. (2024b).
x/1 + frr dx!
unknown
Below is summary of how these metries work.
f--1 = (1 m) @ Xknown + m @ x 4-1 Xeta
f--1
Xunknwnn'
'éWe find the result of GOOGLE is hard to reproduce due to memory issues, so we directly use the results Zhang et al. (2024b)
where dxf gts -1(()o(1)Vxz xt. log 1 dt
+ (((d))
17 Pnttps://githus com/ amazon science/ tabsyn/tree/maii htwpe//aannnnnnn//eeeaaeeeaaaaee ines
(17)
14

---

TabNAT: A Continuous-Diserete Joint Generative Framework for Tabular Data
B.7.1. MARGINAL DISTRIBUTION
the average score is divided by 2 to ensure that falls in the range of [0, 1], then the smaller the score, the better the estimation.
The Marginal metric evaluates if each column's marginal distribution is faithfully recovered by the synthetic data. We use Kolmogorov-Sirmov Test for continuous data and Total Variation Distance for discrete data.
Contingency similarity For pair of categorical columns A and B, the contingency similarity score computes the difference between the contingeney tables using the Total Variation Distance. The process summarized by the for- mula below:
Kolmogorov-Sirno Test (KST) Given two (continuous) distributions (x) and (x) (r denotes real and s denotes synthetic), KST quantifies the distance between the two distributions using the upper bound of the discrepancy be- tween two corresponding Cumulative Distribution Functions (CDFs):
Contingency Score
Σ
Σ BEB
RR.e - Sa, (23)
KST sup (x) - F,(x)),
(18)
where a and | describe all the possible categories in column A and column B, respectively. Ra,8 and Sa.s are the joint frequency of a and 3 in the real data and synthetic data, respectively.
where (x) and (2) are the CDFs of (x) and (2(),, respectively:
F(x)= = _ p(x)dx.
F(x)
(19)
B.7.3. PRECISION AND 3-RECALL
Total Variation Distance (TVD) TVD computes the fre- quency of each category value and expresses it as probabil- ity. Then, the TVD score the average difference between the probabilities of the categories:
a- c--Precssoon and 3-Recall are two sample-level metrics quan- tifying how faithful the synthetic data is proposed in Alaa et al. (2022). In general, a-Precision evaluates the fidelity of synthetic data - whether each synthetic example comes from the real-data distribution, 33-Recall evaluates the cov- erage of the synthetic data, e.g., whether the synthetic data can cover the entire distribution of the real data (In other words, whether real data sample is close to the synthetic data).
TVD = Σ [R(w) - S(w)),
(20)
2
WE
where describes all possible categories in column Ω R(-) and 5-) denotes the real and synthetic frequencies of these categories.
B.7.4. CLASSIFIER- PPWO-SAMPLE. (C2ST)
C2ST studies how difficult is to distinguish real data from synthetic data, therefore evaluating whether synthetic data can recover real data distribution. The C2ST metric used in this paper is implemented by the SDMetrics package.
B.7.2. JOINT DISTRIBUTION
The Joint metric evaluates if the correlation of every two columns in the real data is captured by the synthetic data.
Pearson Correlation Coefficient The Pearson correlation coefficient measures whether two continuous distributions are linearly correlated and is computed as:
B.7.5. MACHINE LEARNING EFFICIENCY (MLE)
In MLE, each dataset is first split into the real training and testing set. The generative models are learned on the real training set. After the models are learned, a synthetic set of equivalent size is sampled.
Cov(x. v) JgTy
Pz.y
(21)
The performance of synthetic data on MLE tasks is evalu- ated based the divergence of test scores using the real and synthetic training data. Therefore, we first train the machine learning model on the real training set, split into training and validation sets with a 8 : ratio. The classifier/regressor is trained on the training set, and the optimal hyperparame- ter setting is selected according to the performance on the validation set. After the optimal hyperparameter setting obtained, the corresponding classifier/regressor is retrained the training set and evaluated on the real testing set. We
where x and y are two continuous columns. Cov is the covariance, and is the standard deviation.
Then, the performance of correlation estimation is measured by the average differences between the real data's 's correla- tions and the synthetic data's corrections:
1 Pearson Score [ Ex, Ex (ol -0000~((x)) (2,1)),
(22)
where ( y) and م (x,z)) 8)) denotes the Pearson corre- lation coefficient between column & and column of the real data and synthetic data, respectively. As p [-,,1],
18 #ntppa:/ /ttpppa:////....ppeeeppsss sdv www.:sddwwdv.......e.eeee ttt.://deeeriic///eeriiieieeeeee netrica cstrica--oo in--ii--eeee beta hwtp///eeeeoeeoeeoeooeoo
15

---

TabNAT: A Continuous-Disertt Joint Generative Framework for Tabular Data
create 20 random splits for training and validation sets, and the performance reported is the mean of the AUC/RMSE score over the 20 random trails. The performance of syn- thetic data is obtained in the same way.
C.2. Detailed Results on Machine Learning Efficiency
Similarly, Table 14, we present the detailed comparison table Machine Learning Efficiency task.
B.7.6. DISTANCE TO CLOSEST RECORD
We follow the 'synthetic vs. holdout" setting 19, We initially divide the dataset into two equal parts: the first part served the training for training our generative model, while the second part designated the holdout set, which is not used for training. After completing model training, we sample synthetic set of the same size as the training set (and the holdout set).
We then calculate the DCR scores for each sample in the synthetic set concerning both the training set and the hold- We further calculate the probability that synthetic sample is closer to the training set (rather than the holdout set). When this probability is close 50% (i.e., 0.5), it indicates that the distribution of distances between synthetic and training instances is very similar (or at least not system- atically smaller) than the distribution distances between synthetic and holdout instances, which a positive indicator terms of privacy risk.
C. Additional Experimental Results
In this section, we provide more detailed empirical com- parison between the proposed TabNAT and other baseline methods.
C.1. Detailed Results on the Fidelity Metrics
Note that in Table 1, we only present the average perfor- mance of each method on the six fidelity metrics across the datasets. In this section, we present detailed perfor- mance comparison of each individual dataset:
Marginal Distribution: Table
Joint Correlation: Table
a- P-Prcissio: Table 10
8-Recall: Table 11
Clasiffe--Too-Samppe--sss Table 12
Jensen-Shannon Divergence: Table 13
2022-06- 2022-05-07 1an--00-----yctheii--tttttt for-prvacy--
preservation-partt>
16

---

TabNAT:
A Continuous-Diserte Joint Generative Framcwork for Tabular Data
Table 8: Performance comparison on the Marginal Distribution Density metric. Numbers represent the error rate in %, the
lower the better.
Continuous only
Discrete only
Hctcrogeous
Method
California
Letter
Car
Nursery
Adult
Beijing
Default
Magic
News
Shoppers
interpolation SMOTE
0.99
0.97
1.00
0.57
1.59
1.78
1.49
1.07
5.28
2.48
VAE-based TVAE
5.37
16.70
24.12
9.81
24.32
25.13
9.94
4.39
18.48
23.93
GAN-based CTGAN 12.84
18.79
16.46
12.33
19.32
21.98
18.25
5.69
13.90
25.71
LLM-based
GReaT
14.93
4.88
2.22
5.08
12.12
8.25
19.94
16.16
14.51
Diffusion-based
STaSy
10.82
11.93
24.38
10.93
10.41
6.38
11.34
13.02
8.54
16.14
CoDi
18.98
22.62
1.53
0.65
24.84
12.54
16.54
11.64
28.13
36.48
TahDDPM
57.34
61.43
1.53
0.65
1.32
1.20
7.59
1.09
2.86
TabSyn
1.00
2.53
2.48
1.04
0.58
1.12
0.85
0.88
1.64
1.45
Autoregressive
DP-TBART
3.30
4.46
1.98
0.53
1.17
2.68
5.03
3.90
6.28
3.05
Tab-MT
5.87
3.29
0.96
0.70
17.20
25.10
25.17
21.88
46.54
2.20
TabNAT
0.99
1.79
1.31
0.73
0.59
0.80
0.65
0.80
1.47
1.32
C.3. Visualizations
We present the 2D visualizations of the synthetic data gen- erated by all baseline methods in Figure 10 and Figure 11, respectively. We also present the heat maps of all methods on the four datasets in Figure 12, Figure 13, Figure 14, and Figure 15, respectively.
17

---

TabNAT: A Continuous-Disertt Joint Generative Framcwork for Tabular Data
Table 9: Performance comparison on the Joint Column Correlation metric. Numbers represent the error rate in %, the low
the better.
Continuous only
Discrete only
Heterogeous
Method
California
Letter
Car
Nursery
Adult
Beijing
Default
Magic
News
Shoppers
interpolation SMOTE
2.70
1.19
3.16
1.21
3.56
1.53
6.93
2.84
2.87
3.53
VAE-based TVAE
5.85
5.28
38.66
18.34
36.65
31.12
19.37
4.46
6.45
20.12
GAN-based
CTGAN
14.49
11.40
25.63
18.14
27.35
27.08
30.52
5.04
5.22
24.24
1IM-based
GReaT
9.66
3.46
4.72
8.38
17.59
59.60
70.02
59.96
45.16
Diffusion-bused
STaSy
3.59
5.34
36.40
15.02
13.50
8.71
10.65
5.58
3.06
15.29
CoDi
6.89
5.25
3.52
1.31
22.72
6.42
67.88
6.93
10.81
20.18
TabDDPM
19.83
22.35
3.52
1.31
2.50
3.31
11.55
0.67
6.23
TabSyn
0.78
1.78
4.28
1.85
1.54
2.24
2.05
1.06
1.44
2.07
Autoregressive
DP-TBART
2.52
1.94
3.54
1.19
2,50
2.55
6.70
1.73
1.60
2.83
Tab-MT
5.87
3.29
0.96
0.70
17.20
25.10
25.17
21.88
46.54
2.20
TabNAT
0.61
1.45
2.99
1.36
1.36
2.27
2.83
1.86
1.50
1.89
Table 10: Performance comparison the a-Preciision metric. Numbers represent 1 a- a-Precision. The lower the better. Note that the numbers in Table are in % while numbers in this table are in raw scale.
Continuous only
Discrete only
Heterogeous
Method
California
Nursery
Adult
Beijing
Default
Magic
News
Shoppers
interpolation SMOTE
0.0173
0.0222
0.0103
0.0040
0.0729
0.0118
0.0228
0.0186
0.1256
0.0725
VAE-based TVAE
0.0191
0.0937
0.2322
0.0972
0.4124
0.1100
0.1610
0.0338
0.1530
0.5655
GAN-based CTGAN
0.2933
0.0522
0.1023
0.0677
0.2528
0.0723
0.3595
0.1106
0.0177
0.1292
LLM-based
GReaT
0.1665
0.0891
0.0262
0.0836
0.4421
0.0168
0.1410
0.1454
1.0000
0.21 12
Diffusion-basel
STaSy
0.8385
0.0158
0.3347
0.1385
0.2250
0.0350
0.1320
0.1853
0.0809
0.1656
CoDi
1333
0.0963
0.0323
0.0037
0.1805
0.0471
0.1722
0.1434
0.1041
0.0825
TabDDPM
0.8385
1.0000
0.0323
0.0037
0.0444
0.0130
0.0924
0.0146
1.0000
0.0807
TabSyn
0.0062
0.0998
0.0206
0.0056
0.0048
0.0153
0.0074
0.0062
0.0320
0.0084
Autoregnessive
DPATAT
0.0256
0.0427
0.0093
0.0054
0.0054
0.0049
0.0727
0.0160
0.0149
0.0145
Tab-MT
0.0239
0.0415
0.0140
0.0146
0.1776
0.4839
0.0581
0.5507
0.9369
0.0158
TabNAT
0.0080
0.0383
0.0190
0.0052
0.0133
0.0032
0.0073
0.0050
0.0784
0.0233
18

---

TabNAT: A Continuous-Diserte Joint Generative Framcwork for Tabular Data
Table 11: Performance comparison the 3-Recall metric. Numbers represent 1 8-Recall. The lower the better. Note that the numbers in Table are in % while numbers in this table in raw scale.
Continuous only
Discrete only
Heterogeous
Method
California
Letter
Car
Nursery
Adult
Beijing
Default
Magic
News
Shoppers
Interpolation SMOTE
0.2157
0.1338
0.0045
0.0012
0.2325
0. 0.2080
0.2390
0.193
0.2011
0.2386
VAE-based TVAE
0.6512
0.8324
0.4369
0.2625
0.8969
0.9392
0.7078
0.6243
0.7413
0.7622
GAN-based
CTGAN 0.8412
0.9903
0.9879
0.0530
0.8246
0.6230
0.8889
0.8472
0.7732
0.7441
LLM-based
GReaT
0.5515
0.6643
0.0034
0.0024
0.5088
0.5666
0.5796
0.6509
1.0000
0.5510
Diffusion-basdd
STaSy
0.9288
0.7332
0.1075
0,0029
0.6812
0.5061
0.6421
0.5686
0.6033
0.7174
CoDi
0,5998
0,4551
0.0040
0.0009
0.9032
0.4472
0.7811
0.5139
0.6505
0.8187
TabDDPM
0.9288
1.0000
0.0040
0.0009
0.5152
0.4335
0.6150
0.5206
1.0000
0.4492
TabSyn
0.5706
0.7486
0.0031
0.0007
0.5244
0.4416
0.5200
0.5
0.5496
0.5105
Autoregressive
DP-TBART
0.6138
0.8859
0.0038
0.0010
0.5033
0.4536
0.5562
0.6044
0.6532
0.5285
Tab
0.6272
0.8385
0.0067
0.0008
0.9681
0.4930
0.9179
0.9844
1.0000
0.5105
TabNAT
0.5038
0.4562
0.0040
0.0008
0.4928
0.4249
0.5162
0.4590
0.5277
0.4561
Table 12: Performance comparison on the C2ST metric. Numbers represent 100 x (1 C2ST) (i.e. in base of 10-2). The
lower the better.
Continuous only
Discrete only
Heterogeous
Method
California
Letter
Car
Nursery
Adult
Beijing
Default
Magic
News
Shoppers
interpolation SMOTE
0.61
0.00
0.00
0.00
3.05
0.44
7.69
1.93
6.49
9.80
VAF-based TVAE
12.48
22.27
70.30
52.73
72.39
45.53
41.65
12.07
60.27
70.04
GAN-based CTGAN
50.11
82.40
59.59
48.63
36.79
56.82
64.60
14.15
27.64
48.86
LLM-bused
GReaT
28.38
16.14
6.05
18.41
46.24
31.07
52.90
56.74
57.15
Diffasion- based
STaSy
54.61
47.75
78.52
40.43
54.02
23.48
49.29
53.97
50.21
62.20
CoDi
47.48
42.85
0.00
0.22
80.02
15.70
52.37
27.70
91.62
81.84
TahDDPM
88.01
96.86
0.00
0.22
3.95
3.29
11.75
0.95
16.37
TabSyn
0.71
3.18
2.90
2.88
8.05
3.60
1.33
0.08
1.77
3.05
Autoregressive
DP-TBART
3.89
11.07
0.07
0.00
0.81
3.63
7.41
5.01
12.96
8.73
Tab-MT
8.71
9.34
0.37
0.00
99.86
99.97
91.62
74.47
100.00
1.47
TabNAT
1.27
0.55
0.00
0.74
1.30
1.55
3.43
0.1100
7.70
3.74
19

---

TabNAT: A Continuous-Disertt Joint Generative Framcwork for Tabular Data
Table 13: Performance comparison on the Jensen- Shannon Divergence (JSD) metric. The lower the better. Note that numbers Table are in % while numbers this table are in raw scale.
Continuous only
Discrete only
Heterogeous
Method
California
Letter
Car
Nursery
Adult
Beijing
Default
Magic
News
Shoppers
interpolation SMOTE
0.0006
0.0006
0.0013
0.0008
0.0008
0.0003
0.0006
0.0008
0.0040
0.0015
VAE-based TVAE
0.0041
0.0072
0.0323
0.0132
0.0078
0.0077
0.0052
0.0036
0.0109
0.0107
GAN-based CTGAN 0.0182
0.0135
0.0220
0.0165
0.0038
0.0033
0.0114
0.0056
0.0078
0.0066
LLM-based
GReaT
0.0111
0.0022
0.0030
0.0068
0.0182
0.0023
0.0076
0.0107
0.0056
Diflasion-based
STaSy
0.0380
0.0057
0.0326
0.0146
0.0041
0.0030
0.0055
0.0107
0.0070
0.0086
CoDi
0.0152
0.0085
0.0020
0.0009
0.0073
0.0017
0.0067
0.0142
0.0092
0.0103
TabDDPM
0.0380
0.0382
0.0020
0.0009
0,0004
0.0092
0.0008
0.0013
0.0019
TabSyn
0.0006
0.0017
0.0033
0.0014
0.0004
0.0012
0.0003
0.0007
0.0016
0.0007
Antoregressive
DP-TBART
0.0023
0.0010
0.0023
0.0008
0.0004
0.0006
0.0025
0.0018
0.0036
0.0008
Tab-MT
0.0038
0.0019
0.0013
0.0009
0.0026
0.0024
0.0034
0.0211
0.0221
0.0007
TabNAT
0.0004
0.0010
0.0018
0.0010
0.0003
0.0004
0.0007
0.0011
0.0015
0.0008
Table 14: AUC (classification task) and RMSE (regression task) scores of Machine Learning Efficiency. 1 (L) indicates that the higher (lower) the score, the better the performance. MLE metrics measure the relative performance difference between synthetic data and training data.
Continuous only
Discrete only
Heterogcous
Method
California AUCt
Letter
Car AUC1
Nursery AUCt
Adult AUC
Default AUC1
Shoppers AUC1
Magic AUC
News RMSE
Beijing RMSE
AUCT
Real data
0.999
0.989
0.999
1.000
0.927
0.770
0.926
0.946
0.842
0.423
VAE-based
TVAE
0.986
0.989
0.746
0.939
0.846
0.744 0.584
0.898 0.658
0.912
0.979
1.010
GOGGLE
0.778
0.654
1.09
0.877
GAN-based CTGAN
0.925
0.729
0.899
1.000
0.874
0.736
0.868
0.874
0.895
1.065
LLM-based
GReaT
0.996
0.983
0.979
0.999
0.913
0.755
0.902
0.888
0.653
Diffusion-basee bassd
STaSy
0.997
0.990
0.927
0.982
0.903
0.749
0.909
0.923
0.933
0.672
CoDi
0.981
0.998
0.995
1.000
0.829
0.497
0.855
0.930
0.999
0.750
TahDDPM
0.992
0.513
0.995
1,000
0.911
0.763
0.915
0.933
2.665
TabSyn
0.993
0.990
0.971
0.997
0.904
0.764
0.913
0.934
0.862
0.669
Autoregressive
DP-TBART
0.993
0.985
0.990
0.917
0.918
0.717
0.896
0.924
0.896
0.676
Tab-MT
0.988
0.985
0.981
1.000
0.873
0.714
0.912
0.822
1.002
2.098
TabNAT
0.994
0.994
0.996
0.900
0.904
0.764
0.916
0.935
0.856
0.579
20

---

TabNAT: A Continuous-Disertt Joint Generative Framcwork for Tabular Data
15.0
15.0
15
15.0
12.5
125
30.0
10.0
255
10.0
7.5
10.0
75
0.0
25.-
0.0
0.
40
ago
ago
age
ugo
(a) CTGAN
(b) TVAE
(c) GOGGLE
(d) GReaT
17.5
15
15.0
15.0
12.5
12.5
12.5
10.0
20.0
10.0
73.
7.5
75
5.0
5.0
40
age
age
Ege
age
(e) CoDi
(f) STaSy
(g) TabDDPM
(h) TabSyn
15.0
15.0
15.0
15.0
30.0
20.0
28.0
30.0
75-
7.5
75
2.5
0.0
0.
00
10e
age
1gc
uge
(i) Tab-MT
DP-TBART
(k) TabNAT (Ours)
(I) Ground Truth
Figure 10: Kernel density estimation (KDE) plot of the 2D joint density of "education.num" and 'age" features in the Adult dataset. The results from GOGGLE and Tab-MT are not plotted since they either fail to generate generate singleton synthetic data on one feature (e.g. always generate eduucttio.num"" equals one).
21

---

TabNAT: A Contenoous-Disertt Joint Generative Framcwork for Tabular Data
-214 -122
-018
-014
-55 -022 -008 -130 H0TE
-014
253
-00 -234 -10 -01 -000 -014
(a) CTGAN
(b) TVAE
(c) GOGGLE
(d) GReaT
-2M4 -22 -139 -018
-131
-12
+20 +12
-18
-12
-224
-222 -12
-014 -131
-207 -234
-222
-220
-138
-130
(e) CoDi
(I) STaSy
TabDDPM
(h) TabSyn
200
+244
20
+23
+244 -122
( Tab-MT
DP-TBART
(k) TabNAT (Ours)
(I) Ground Truth
Figure 11: Scatter plots of the 2D joint density of the Longitude and Latitude features in the California Housing dataset. Blue lines represent the geographical border of California.
22

---

TabNAT: A Continuous-Disertt Joint Generative Framcwork for Tabular Data
CTGAN
TVAE
GREAT
CoDi
StaSy
025
920
TabDDPM
TabSyn
DP-TBART
Tab-MT
TabNAT
810
9.25
Figure 12: Heat map of synthetic data of Letter dataset.
CTGAN
TVAE
GREAT
CoDi
StaSy
020
TabDDPM
TabSyn
DP-TBART
Tab-MT
TabNAT
0.5
0.55
Figure 13: Heat map synthetic data Adult dataset.
23

---

TabNAT: A Continuous-Disertt Joint Generative Framcwork for Tabular Data
CTGAN
TVAE
GREAT
CoDi
StaSy
025
920
TabDDPM
TabSyn
DP-TBART
Tab-MT
TabNAT
8.0
3.25
Figure 14: Heat map of synthetic data of Default datasct.
CTGAN
TVAE
GREAT
CoDi
StaSy
020
TabDDPM
TabSyn
DP-TBART
Tab-MT
TabNAT
0.5
0.55
Figure 15: Heat map of synthetic data of Magic dataset.
24


Reviews:
[{"review_id": "RI1zufDUwA", "reviewer": "Reviewer_YG4L", "summary": "The paper tackles the task of generating tabular data. It claims that the application of current autoregressive models for generating tabular data is limited due to two challenges:\n1) tabular data contains heterogeneous types, whereas autoregressive next-token (distribution) prediction is designed for discrete data\n2) tabular data is column permutation-invariant, requiring flexible generation orders.\n\nIn order to alleviate these issues, it proposes using a transformer which encodes the continuous data in each row (each row is a single data sample) into a vector at position 1 of the sequence, and the rest of the data (discrete), in that row is seen as the rest of the sequence, where each entry (corresponding to one column) is represented as a vector with learnable entries. As such, each row can be seen as a sequence of (L+1) vectors, where each vector has dimension $d$. Uniform random masking is applied before feeding the sequence to the transformer, and the outputs are then used to predict the masked (missing) discrete data, and also used to condition to generate the continuous data. \n\nThe proposed method is then compared to other methods in multiple datasets, and assessed using multiple metrics. In addition ablation studies are performed to see how the main two claimed weaknesses above influence the performance.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank the reviewer for their thoughtful and constructive feedback on our manuscript. We appreciate the time and effort spent reviewing our work, and we believe the suggestions will significantly improve the quality of our paper. Below, we address each point raised by the reviewer.\n\n### Modeling of Continuous Data\n\nYour observation about our approach to modeling continuous data is insightful. We agree with your assessment that our method doesn't directly model the full continuous distribution $p(X)$, but rather learns a conditional distribution $p(X|c)$, where the constant vector $c$ represents the case where all discrete columns are masked.\n\nWhile this is conceptually different from modeling the complete joint distribution, in practice, the two approaches yield equivalent results for generation purposes. This is because when all discrete columns are masked, the model effectively sees only a generic placeholder with no specific information, forcing it to generate continuous values based solely on the learned marginal distribution of those values. The mask tokens in this case serve merely as positional indicators without conveying any actual discrete data information.\n\nYour suggested approach of encoding perturbed continuous dimensions into vectors at positions 1 to K is elegant and theoretically sound. Following your suggestion, we implemented and tested this approach. Our experiments confirmed your intuition—the results were indeed comparable to our original method in terms of generation quality.\n\nHowever, we identified a practical limitation: this approach significantly extends the sequence length, approximately tripling the training computational cost. Given the similar performance outcomes but increased resource requirements, we opted for our original approach as a more efficient solution. Nevertheless, we appreciate this valuable suggestion and will discuss this alternative formulation and its trade-offs in our revised paper.\n\n### Discrete Diffusion Models\n\nThank you for highlighting the recent advances in discrete diffusion models that we overlooked. While we did include basic approaches like multinomial diffusion in our baselines (which showed limited effectiveness), we missed the opportunity to explore more recent developments in this area.\n\nThe papers you referenced indeed offer promising directions for modeling discrete data without left-to-right bias. In our revised paper, we'll discuss these advancements and explore how they might complement our approach for mixed-type tabular data generation. We're particularly intrigued by masked diffusion approaches and their potential in heterogenous tabular data generation.\n\n\n### Choice of Stochastic Generation Approach\n\nWe thank the reviewer for this insightful suggestion. Our primary focus was to effectively integrate diffusion models with autoregressive approaches for tabular data generation. Regarding the choice of sampling method, we acknowledge that different approaches offer various trade-offs between quality and efficiency. Following the reviewer's suggestion, we conducted additional experiments with flow matching. While this approach did improve sampling speed (requiring only 20 steps), we observed a slight decrease in generation quality. It's worth noting that our current sampling method requires only 50 sampling steps—significantly fewer than the 1000 steps typically needed in traditional DDPM approaches—thus already providing a good balance between quality and efficiency.\n\n### Position-Specific Masking Encoding\n\nIntuitively, the position-specific mask embedding serves a different purpose than the position embeddings. While position embeddings encode the absolute position of each token in the sequence, the position-specific mask embedding is designed to capture information about which specific attributes are masked during the any-order training process. This allows the model to better understand the relationship between masked and unmasked attributes in different positions, enhancing its ability to handle partial information scenarios. In fact, we found that the empirical performance difference between the two designs is not substantial (with position-specific mask embedding performing slightly better). Considering that the additional parameters introduced by this design are negligible, we adopted this approach in our final model.\n\n### Notation and Figure Clarity\n\nWe thank the reviewer for pointing out the notation issues and concerns about figure clarity. We will correct all notation errors and improve the clarity of Figures 1 and 2 in the revised version of our manuscript.", "claims_and_evidence": "The paper does not make any theoretical claims. Regarding performance claims, based on the provided results, the proposed method appears to outperform the other methods in most of the cases. However, it should be noted that my main area of expertise are diffusion models, and autoregressive models (mostly applied to language), therefore I cannot fully evaluate the significance of these results. My review, will mostly be focused on the merits of the design of modelling and application of diffusion and other discrete modelling approaches, rather than performance.", "methods_and_evaluation_criteria": "It should be noted that my main area of expertise are diffusion models, and autoregressive models (mostly applied to language), therefore I cannot fully evaluate the significance of these results. My review, will mostly be focused on the merits of the design of modelling and application of diffusion and other discrete modelling approaches, rather than performance. \n\nThat being said the criteria appear to be sound and cover multiple aspects.", "theoretical_claims": "The paper makes no novel theoretical claims.", "experimental_designs_or_analyses": "The experiment designs appear to be sound and cover multiple aspects.", "relation_to_broader_scientific_literature": "I am not familiar with other tabular data generation methods.\n\nThe paper makes use of continuous diffusion models, and Bert like masking architecture. However, it does not include categorical diffusion models (which are broadly known as discrete diffusion models).", "essential_references_not_discussed": "The main essential references which are missing in my opinion are discrete diffusion model papers (some references for example [1,2,3,4,5]). The authors use a masking schedule similar to Bert, but do not attempt to model discrete data with discrete diffusion, which has been shown to perform well in modelling discrete data (such as language). In the context of this paper, it could be even more useful as the data lacks the inherent left to right bias. The choice not to include them due to not modelling explicitly (I believe this means analytically the pdf) dependencies between dimensions is not properly supported considering that the goal of the paper is data generation, not density/probability modelling.\n\n[1] Structured Denoising Diffusion Models in Discrete State-Spaces, Austin et al, 2021.\n[2] A Continuous Time Framework for Discrete Denoising Models, Campbell et al 2022.\n[3] Discrete Diffusion Modeling by Estimating the Ratios of the Data Distribution, Lou et al 2024.\n[4] Simple and Effective Masked Diffusion Language Models, Sahoo et al, 2024\n[5] Discrete Flow Matching, Gat et al, 2024.", "other_strengths_and_weaknesses": "**Strengths**\n\n1. The claims made in this paper are quite interesting, in particular the attempt to not model discrete tabular data with autoregressive models, but utilize full attention. Results show the latter is superior.\n2. The section which explains the method is very clearly written and illustrated.\n3. Based on the provided results, the proposed method improves upon the existing ones in the vast majority of cases.\n\n**Main weaknesses**\n\n1. The main weakness in this paper is the modelling of continuous data. The paper suggest in Equations (1) and (2), to model the continuous data separately, and then learn discrete data conditioning on continuous data. I believe this is the right approach. The inference explanation in lines 270-274 reflects this approach. On the other hand during the actual training, the model does not learn the whole continuous distribution, but learns a family of conditional ones, conditioned on a masked sequence. As such a natural order is missing (rule of choosing unmasking positions) which in discrete diffusion is mitigated. The main missing part in this paper is comparing this approach against the following:\n\nEncode the perturbed continuous dimensions into a higher dimensionality vector and then split it into multiple vectors, which will be placed into positions 1 to K in the sequence. If L is the number of discrete columns, and C the number of continuous columns, then then K/L should be roughly C/L. The discrete tokens come after, that is so far the sequence has length K+L. Finally the last K positions are reserved to the continuous data so that the discrete data can attend to. That is, the first K tokens attend only to themselves, in other words, (K, , ) attends to (K, , ) in the (K,L,K) sequence. The L tokens ( ,L, ) attend to ( ,L,K), and finally ( , ,K) attends to ( , ,K). The last K tokens do not provide an output at the end, they are there just so that the L tokens can condition on them. In this way the transformer could model both the continuous and the (discrete|continuous) distribution simultaneously. This would fit the objective proposed in Equation (1) and (2) in the paper. Otherwise the section containing these equations should be modified in order to more properly set the tone for the proposed method.\n\n2. The work does not include advances in modelling discrete distributions, which have been developing since at least 2021. More recently, discrete diffusion modelling with absorbing (masked) dynamics have shown very competitive results in text modelling. It is likely that they would perform very well on data without left to right bias. In addition, they offer a natural unmasking schedule. Applying this approach to (_,L,$\\cdot$) discrete data is straightforward.\n\n**Minor Weaknesses**\n\n1. There are some small issues with notation and grammar:\n\na. Line 110, right column, I believe it should be $\\mathbb{R}^{D_{cts}}$.\n\nb. Line 111, right column, I believe it should be $\\in \\mathbb{N}$\n\nc.  Line 200, left column, I believe it should be $\\in \\\\{0,...,C_i-1\\\\}$\n\nd. Line 303, left column. *For each model*.\n\n2. Figures 1 and 2 are unclear. The rectangles have varying sizes and no labeling , which makes it difficult to interpret the figure.\n\n3. While not a weakness, the results of comparing MSE against diffusion are not surprising.", "other_comments_or_suggestions": "The main 2 weaknesses are simultaneously suggestions, which I believe would improve the paper. Overall, I am on the borderline regarding acceptance. I am looking forward to reading the responses of authors, and am willing to reevalute the paper based on their replies, as well as based on other reviews which could highlight something I might have missed.", "questions_for_authors": "1. Why was a stochastic generating approach used instead of the ODE formulation, or flow matching, which could speed up generation?\n2. Why are masked encoded differently at each position, when we already add positional encoding?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "hznM2BpMW6", "reviewer": "Reviewer_t2DD", "summary": "The paper focuses on unconditional tabular data generation and conditional missing data imputation tasks. Considering the heterogeneous nature of tabular data, which includes both discrete and continuous variables, the proposed method utilizes a diffusion model to parameterize the conditional distributions for continuous columns. For discrete columns, it employs next-token prediction with KL divergence minimization. Additionally, a masked Transformer with bi-directional attention is used to support order-invariant generation. Experiments conducted on ten datasets with diverse attributes demonstrate the effectiveness of the proposed TabNAT method.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank the reviewer for the thorough review and constructive feedback on our manuscript. We have addressed the raised concerns as follows:\n\n### The suitability of diffusion modeling for continuous columns in tabular data\n\nWe agree that this is an important consideration that deserves clarification. Our choice of diffusion modeling for continuous columns is supported by empirical evidence from recent SOTA work:\n\n- TabDDPM was the first model to successfully apply diffusion models to tabular data synthesis, using traditional DDPM for continuous columns and discrete diffusion with multinomial distribution for discrete columns. Its strong performance validated the effectiveness of diffusion models for capturing the complex distributions in continuous (multi-column) tabular data.\n- TabSyn further advanced this approach by mapping both discrete and continuous columns to a continuous embedding space, enabling the application of standard diffusion models to the entire table. TabSyn has achieved state-of-the-art results in tabular data generation, demonstrating the exceptional capability of diffusion models to capture the distributions of continuous tabular data.\n- Our empirical experiments align with these findings. It's important to note that direct autoregressive generative modeling of continuous values is not feasible without discretization. Baseline methods such as Tab-MT and DP-TBART both employ this discretization-plus-autoregressive approach, and our experiments demonstrate that they consistently underperform compared to our method across all metrics, particularly in Statistical Fidelity scores. \n\n### On the differences between continuous columns in tabular data and images\n\nWe agree with the reviewer that a deeper analysis of the differences between continuous values in tabular data and images would strengthen our paper. These differences are fundamental to our methodological choices:\n\n- Tabular data columns are fundamentally more independent than image pixels, with each column typically representing a distinct feature with its own semantic meaning. In contrast, image data pixels have strong spatial continuity, with neighboring pixels highly correlated and collectively representing coherent visual elements.\n- This structural difference necessitates different denoising neural network architectures. For images, CNN-based UNet models have become the standard architecture for diffusion models because they effectively leverage the local spatial correlations through convolutional operations and capture multi-scale features through their encoder-decoder structure with skip connections.\n- For tabular data, where columns have distinct meanings without inherent spatial relationships, traditional MLP architectures are more appropriate as denoising neural networks. In our approach, we use an MLP for denoising continuous data, while the Transformer is employed solely to generate conditional vectors. This design choice aligns with other successful tabular diffusion models, such as TabDDPM and TabSyn, which also utilize MLPs as their denoising neural networks.\n\n### On the missing data imputation experiment\n\nWe would like to clarify that the baselines we compared against (Remasker, HyperImpute, and GRAPE) are indeed SOTA methods for tabular missing data imputation. These same methods were used as the top-performing baselines in a recent ICLR 2025 Spotlight paper on tabular data imputation [1]. Our selection of baselines is therefore aligned with the current standards in the field. We will aslo include the new paper as an additional baseline in our revised manuscript to ensure our comparisons remain comprehensive and up-to-date.\n\nDue to page limitations, we were unable to include ablation studies for the missing data imputation task in the original submission. We have now conducted these additional experiments, and the results are presented in Figure 15 and Figure 16 in rebuttal.pdf in the anonymous link：https://anonymous.4open.science/r/ICML-TabNAT. The findings are summarized as follows:\n\n- Simple MSE loss is as effective as the Diffusion loss in missing data imputation since we care more about the expectation of the missing entry rather than its distribution. The imputation of discrete columns is sensitive to the order, and the random-order sampling used in our paper is beneficial.\n\n- The proposed TabNAT is very robust to the model depth and width in the missing data imputation task.\n\nThese results further validate our design choices and demonstrate that each component of our model contributes significantly to its superior performance in the missing data imputation task.\n\n### Missing references\n\nWe thank the reviewer for pointing out the related works. We will cite these works and add a discussion about the application of diffusion losses in autoregressive image modeling.\n\nReferences:\n\n[1] Zhang et al.  DiffPuter: Empowering Diffusion Models for Missing Data Imputation. In ICLR 2025.", "claims_and_evidence": "The paper provides a detailed analysis of diffusion modeling and autoregressive modeling, employing a diffusion model to model continuous columns using Diffusion Loss and using next-token prediction for discrete columns with Cross Entropy Loss. While this claim appears reasonable at first glance, the continuous data in the table consists of individual continuous values, where the dependencies between data points are not as strong as in images, where contextual tokens exhibit high continuity. From this perspective, diffusion modeling may not necessarily be more suitable than autoregressive prediction for modeling continuous columns in tables. Therefore, the paper’s proposition of \"modeling continuous columns in tables using the diffusion process\" warrants further scrutiny.", "methods_and_evaluation_criteria": "The paper evaluates the effectiveness of the proposed model on multiple tabular datasets, demonstrating that the proposed methods outperform other approaches. For the task of Synthetic Tabular Data Generation, the results are assessed using metrics such as Statistical Fidelity and Machine Learning Efficiency (MLE). In the Missing Data Imputation task, evaluation is conducted using Average MAE for continuous features and Average Accuracy for discrete features. The chosen evaluation metrics are appropriate.", "theoretical_claims": "The paper's approach to modeling and model utilization is well-founded. Continuous data is normalized to the [0,1] range and encoded using an MLP-based encoder, while discrete data is label-encoded and embedded. To account for the column invariance of tabular data, position encoding is applied at the column level. A Bi-directional Transformer is employed to simultaneously predict both continuous and discrete data, utilizing Diffusion Loss and Cross Entropy Loss, respectively.", "experimental_designs_or_analyses": "•\tFor the task of Synthetic Tabular Data Generation, the model is evaluated on ten datasets, including two with only continuous features, two with only discrete features, and six heterogeneous datasets. It is compared against six major categories of methods, including VAE, GAN, LLM, and Diffusion-based approaches. The results are assessed using metrics such as Statistical Fidelity and Machine Learning Efficiency (MLE), which are appropriate for this task. Additionally, corresponding ablation studies are conducted.\n\n•\tFor the Missing Data Imputation task, experiments are performed on five heterogeneous datasets. However, no ablation study is conducted for this task, and the compared baselines appear to be relatively weak.", "relation_to_broader_scientific_literature": "•\tThis paper explores modeling tabular data by applying the diffusion process to continuous columns and using masked generative modeling for discrete columns. However, the applicability of diffusion loss to continuous tabular data requires further in-depth analysis.\n\n•\tThe paper provides an overview of various existing approaches for tabular data generation, including VAE, GAN, and autoregressive modeling. However, its analysis of diffusion loss remains insufficient. In image modeling, diffusion loss involves multiple iterative predictions of the overall image distribution, effectively capturing correlations between tokens. In contrast, discrete tabular data differs from images in that image data exhibits strong global continuity beyond individual tokens. In comparison, individual values in tabular data do not exhibit the same level of continuity. Therefore, the paper should further analyze and discuss related work on diffusion models in this context.", "essential_references_not_discussed": "•\tThe paper lacks sufficient citations on Diffusion Loss, including its application to autoregressive modeling for continuous data.[Tian K et al.'24], [Li T et al.'24]\n\n•\tTian K, Jiang Y, Yuan Z, et al. Visual autoregressive modeling: Scalable image generation via next-scale prediction[J]. Advances in neural information processing systems, 2024, 37: 84839-84865.\n\n•\tLi T, Tian Y, Li H, et al. Autoregressive image generation without vector quantization[J]. Advances in Neural Information Processing Systems, 2024, 37: 56424-56445.", "other_strengths_and_weaknesses": "Strengths:\n\n•\tThe method and model architecture are clearly presented.\n\n•\tThe analysis of diffusion modeling and autoregressive modeling is reasonable.\n\n•\tA substantial number of experiments have been conducted, yielding promising results.\n\nWeaknesses:\n\n•\tThe differences between continuous columns in tabular data and images are not thoroughly analyzed; diffusion loss is simply applied to continuous columns without deeper justification.\n\n•\tThe second experiment lacks ablation studies, and the chosen baselines appear to be relatively weak.", "other_comments_or_suggestions": "The overall work is complete, and the method is explained clearly.", "questions_for_authors": "Although both continuous columns in tabular data and images consist of continuous values, their overall continuity differs, which may warrant a more in-depth analysis.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {"ethical_review_concerns": "n/a"}}, {"review_id": "wAbBS4eO8w", "reviewer": "Reviewer_EVi7", "summary": "The paper proposes a Non-Autoregressive transformer-based generative model for tabular data (TabNAT). It can handle both continuous and discrete columns. It uses a (non-causal) transformer to encode the input and masks to perform any-order training. \nThe overall modeling paradigm is quite similar to TabDDPM (Kotelnikov et al. (2023)) and TabMT (Gulati et al. (2023)), with the key distinctions being how the continuous values are modeled (using continuous diffusion) and the use of the transformer encoder. \nThe proposed approach shows promising results for the three standard evaluation criteria: statistical fidelity, data utility, and privacy protection. Ablations are also performed to demonstrate the impact of using the diffusion model for continuous values and the impact of any-order training.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank the reviewer for their thoughtful assessment of our manuscript. We appreciate the recognition of our paper's strengths, including its clear writing, promising results, and well-supported claims regarding statistical fidelity, utility, and privacy preservation. Below, we address each of the reviewer's questions and concerns.\n\n### Response to Comments on Figures\nWe thank the reviewer for the suggestion regarding Figures 1 and 2. We agree that Figure 3 provides better context and will consider restructuring the presentation of figures in the revised manuscript to improve clarity.\n\n### Position-specific mask embedding and position embeddings\nIntuitively, the position-specific mask embedding serves a different purpose than the position embeddings. While position embeddings encode the absolute position of each token in the sequence, the position-specific mask embedding is designed to capture information about which specific attributes are masked during the any-order training process. This allows the model to better understand the relationship between masked and unmasked attributes in different positions, enhancing its ability to handle partial information scenarios. In fact, we found that the empirical performance difference between the two designs is not substantial (with position-specific mask embedding performing slightly better). Considering that the additional parameters introduced by this design are negligible, we adopted this approach in our final model.\n\n### MLE task performance clarification\nThank you for raising this important point. We apologize for not including these results in Table 1. The complete results for classifiers and regressors trained on the original data can be found in the Appendix, page 20, Table 14. We commit to revising Table 1 to include this information in the main text for better clarity.\n\nRegarding the number of synthetic samples used, the results reported in the last two columns of Table 1 were obtained using the same number of synthetic examples as in the training set, which varies according to the size of each dataset. This 1:1 ratio ensures a fair comparison across different methods. This information is provided in Appendix C.7.5. \n\n### DCR metric definition\n\nWe sincerely apologize for only providing the experimental setup for DCR without explaining how this metric is calculated. \n\nDCR (Distance to Closest Record) is a commonly used metric for evaluating the privacy protection performance of synthetic data. For each synthetic example, we calculate its minimum distance to any example in the training set. \n\nFormally, let's assume we have $N$ training examples {$r_1$, $r_2$, ..., $r_N$} and $n$ synthetic examples {$s_1$, $s_2$, ..., $s_n$}. For each synthetic example $s_i$, we compute:\n\n$$\\text{DCR}^{train}(s_i) = \\min_{j \\in \\{1,2,...,N\\}} d(s_i, r_j)$$\n\nwhere $d(\\cdot,\\cdot)$ represents the normalized Euclidean distance between two records after appropriate feature normalization.\n\nSimilarly, for $N$ holdout examples {$h_1$, $h_2$, ..., $h_N$}, we can compute their DCRs:\n\n$$\\text{DCR}^{holdout}(s_i) = \\min_{j \\in \\{1,2,...,N\\}} d(s_i, h_j)$$\n\nSince the training set and holdout set are i.i.d. samples from the same distribution, if the synthetic samples learn the underlying distribution from the training set correctly, $\\text{DCR}^{train}$ and $\\text{DCR}^{holdout}$ should be similar. Otherwise, if  the synthetic examples are copied from the training data, the $\\text{DCR}^{train}$ of each synthetic example $s_i$ should be close to zero.\n\n\n### Setting of missing data imputation\n\nThank you for raising this point about our imputation task setup. We would like to clarify that in real-world scenarios, we often face in-sample missing data imputation tasks, such as during the data preprocessing phase of large data science projects.\n\nBoth in-sample and out-of-sample imputation approaches have been explored in previous research [1,2]. Methods like HyperImpute adopt out-of-sample evaluation primarily because, as a hybrid approach, it requires supervised labels for model selection. Our method does not have this requirement. We also note that works like Remasker seem to adopt the in-sample setting as well.\n\nThat said, we agree that out-of-sample imputation is equally important. Our model is naturally suitable for out-of-sample settings, and we will conduct additional experiments to demonstrate its effectiveness in this context. We have included these results in rebuttal.pdf in the anonymous link：https://anonymous.4open.science/r/ICML-TabNAT to provide a more comprehensive evaluation of our method's imputation capabilities across both settings.", "claims_and_evidence": "While the claims of superior statistical fidelity, utility, and privacy preservation are well supported. The imputation experiments look a bit weak. Please see the \"Questions section\" for the details.", "methods_and_evaluation_criteria": "The evaluation criteria used in the paper follow the established metrics in the literature.", "theoretical_claims": "NA", "experimental_designs_or_analyses": "Please see the \"Questions section\".", "relation_to_broader_scientific_literature": "The paper generally follows a line of works on neural generative models for tabular data (Assefa et al., 2021; Zheng &\nCharoenphakdee, 2022; Hernandez et al., 2022), and is most closely related to the most recent works that use discrete and continuous diffusion as well as masking based training (Castellon et al., 2023; Gulati & Roysdon, 2023; Kotelnikov et al., 2023; Kim et al.,\n2023; Lee et al., 2023; Zhang et al., 2024b). The overall modeling paradigm is quite similar to TabDDPM (Kotelnikov et al. (2023)) and TabMT (Gulati et al. (2023)), with the key distinctions being how the continuous values are modeled (using continuous diffusion) and the use of the transformer encoder.", "essential_references_not_discussed": "None", "other_strengths_and_weaknesses": "The paper is well written and easy to follow, especially the model description and Figure 3. The reported results also look promising, except for the imputation task (see the questions section below).", "other_comments_or_suggestions": "I don't know if Figures 1 and 2 add much value to the paper. In fact, looking at these figures before Figure 3 caused some confusion for me. You might consider moving these figures to the appendix or placing them after Figure 3, which sets up the context much better.", "questions_for_authors": "1. Why do you need position-specific mask embedding when  separate position embeddings are applied?\n\n2. **MLE task**: What is the performance of the classifiers and regressors trained on the original data? Can you describe, quantitatively, how many synthetic samples were used to get the results reported in the last two columns of Table 1? \n\n3. What is the exact definition of the DCR metric? I could not find an expression for it in the paper or in the reference provided (Zhang et. al. (2024b)). \n\n4. Why is the missing data imputation task set to predict missing values in the training set? Previous works (Du et. al. (2024), Jarret et. al. (2022)) seem to use the standard held-out set for imputation.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Cb4FHlUVR2", "reviewer": "Reviewer_xsBt", "summary": "This paper proposes a model named TabNAT for tabular data synthesis (and imputation). The core idea is to use a diffusion model to generate all columns (of a row) with continuous values together, and a transformer to generate the categorical columns. Experimental results on multiple benchmark datasets show the effectiveness of the proposed model.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank the reviewer for the thoughtful and constructive feedback on our manuscript. We appreciate the time and effort invested in providing these valuable comments, which will help improve the quality of our paper. Below, we address the points raised:\n\n### Choice of Experimental Datasets\n\nThe reviewer raised concerns about our dataset selection, particularly noting differences from those used in DP-TBART/Tab-MT papers. We would like to clarify that our experimental design primarily followed TabSyn's pipeline, as this was our initial baseline for comparison. Accordingly, we incorporated the six heterogeneous datasets used in the TabSyn paper to enable direct performance benchmarking.\n\nTo demonstrate TabNAT's versatility across different data types, we deliberately expanded our evaluation by adding two datasets containing only continuous columns and two datasets with only discrete columns. This comprehensive approach allows us to assess TabNAT's generalizability across diverse tabular data structures.\n\nWe acknowledge the reviewer's valid point regarding datasets used in TabMT and DP-Tbart. We have conducted additional experiments using five additional datasets used in these papers, with results presented in rebuttal.pdf in the anonymous link: https://anonymous.4open.science/r/ICML-TabNAT. These supplementary experiments further validate TabNAT's effectiveness across a broader range of benchmark datasets.\n\n### Performance Comparison with TabSyn\n\nWe appreciate the reviewer's insightful observation regarding TabNAT's performance relative to TabSyn across different datasets. Our detailed analysis reveals several important patterns in performance differences:\n\nFirst, TabNAT consistently outperforms TabSyn on both continuous-only and discrete-only datasets. This advantage stems from TabNAT's architecture, which employs specialized generation models optimally suited for each data type. Unlike TabSyn's approach of using a VAE encoder to create a latent space, TabNAT processes different feature types more directly and naturally, minimizing information loss that typically occurs during VAE encoding.\n\nFor mixed-type datasets, our analysis reveals that TabNAT demonstrates clear advantages when the proportion of discrete features is higher, as evidenced by superior performance on the Adult and Beijing datasets. This performance differential can be attributed to TabNAT's transformer-based categorical generation component, which better captures the complex dependencies between categorical variables through its self-attention mechanism. Additionally, TabNAT's architecture enables more effective modeling of interactions between continuous and discrete features, particularly when discrete features play a dominant role in the underlying data distribution. \n\nImportantly, even in datasets where TabNAT and TabSyn demonstrate comparable statistical performance, TabNAT offers significant practical advantages that TabSyn cannot match. Specifically, TabSyn's VAE-based architecture fundamentally limits its ability to perform flexible conditional generation. In contrast, TabNAT's arbitrary-order autoregressive generation capability supports any form of conditional generation, including missing data imputation. This flexibility stems from TabNAT's bidirectional masking strategy and specialized components for different data types, allowing users to specify any subset of features as conditions while generating the remaining features—a crucial capability for many real-world applications.\n\n### Theoretical Analysis\n\nWe thank the reviewer for suggesting a theoretical analysis to support TabNAT's empirical performance. Our approach is grounded in fundamental probability theory, specifically the decomposition of joint distributions as expressed in Equations (1) and (2). TabNAT's key theoretical contribution lies in transforming the tabular data generation problem into a series of conditional distribution modeling tasks through our bidirectional Masked Transformer architecture.\n\nThis decomposition allows us to leverage the most appropriate generative modeling techniques for different variable types: diffusion models for continuous variables and autoregressive models for discrete variables. Both approaches have strong theoretical foundations in distribution modeling. Diffusion models offer provable convergence to the target distribution through a gradual denoising process, while autoregressive \n\n### Minor Corrections\n\nWe have corrected the noted typographical errors:\n\n- \"After TabNAT is well-trained\" → \"After TabNAT is well trained\"\n- Removed the extra whitespace in \"TabNAT 's training speed\"\n\nAdditionally, we have thoroughly reviewed the manuscript to ensure consistency in terminology and formatting throughout.", "claims_and_evidence": "Yes. The claimed advantages of the proposed model in terms of statistical fidelity, data utility, and privacy protection are shown with experimental results on 10 benchmark datasets.", "methods_and_evaluation_criteria": "The proposed model is intuitive given the setup of the problem. \n\nThe chosen evaluation metrics have been used in the literature and seem suitable for the claims.", "theoretical_claims": "There are no theorem of proofs in the paper.", "experimental_designs_or_analyses": "An issue with the experiments is the choice of datasets. While the proposed model is quite relevant to DP-TBART/Tab-MT, many of the datasets used in the DP-TBART/Tab-MT papers are not used in the experiments of this paper. It is unclear why this has been the case.\n\nFor the other closely related work TabSyn, the datasets used in the TabSyn paper are used in this one which is good for comparison. However, the proposed model TabNAT is often outperformed by (or very close to) TabSyn over (some of) these datasets, as shown in Tables 8 to 14. \n\nInstead of claiming an overall better model, perhaps the paper could focus on the datasets where TabNAT shows better performance and analyzes why TabNAT achieved strong results on those datasets.", "relation_to_broader_scientific_literature": "This paper builds upon the idea of tabular data generation models using transfomer (auto-regressive; DP-TBART/Tab-MT) and diffusion models (TabDDPM/TabSyn). Instead of using either model, it combines the two to achieve a different design while showing that it is also as effective.", "essential_references_not_discussed": "None that I'm aware of.", "other_strengths_and_weaknesses": "Overall this paper proposes a new heuristic structure of tabular data synthesis model based on diffusion models and transformer. Both the technical novelty and depth are somewhat limited (there is no theoretical analysis on the effectiveness of the model design). On the other hand, the proposed model has shown strong performance on some of the experimental datasets. It would make a more interesting paper if the strong performance could be supported by a theoretical analysis. \n\nTabNAT can use a fixed set of hyperparameters for all datasets is an advantage.\n\n\n=========================\n\nUpdate after rebuttal: I'm happy with the additional experimental results and am raising my score from \"Weak reject\" to \"Weak accept\".", "other_comments_or_suggestions": "Typo: \"After TabNAT is well-trained\" => \"After TabNAT is well trained\"; \"TabNAT ’s training speed\" (extra whitespace)", "questions_for_authors": "The author(s) may want to comment on the choice of experimental datasets.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: There is consensus among reviewers that this paper should be accepted. Concerns about experimental setup appear to have been adequately addressed by the authors. This paper will represent a solid contribution to the ICML program.
Decision: Accept (poster)