Diffusion-Nested Auto-Regressive Synthesis of Heterogeneous Tabular Data

## Abstract

Abstract Autoregressive models are predominant in natural language generation, while their application in tabular data remains underexplored. We posit that this can be attributed to two factors: 1) tabular data contains heterogeneous data type, while the autoregressive model is primarily designed to model discrete-valued data; 2) tabular data is column permutation-invariant, requiring a generation model to generate columns in arbitrary order. This paper proposes a Diffusion-nested Autoregressive model ( TabDAR ) to address these issues. To enable autoregressive methods for continuous columns, TabDAR employs a diffusion model to parameterize the conditional distribution of continuous features. To ensure arbitrary generation order, TabDAR resorts to masked transformers with bi-directional attention, which simulate various permutations of column order, hence enabling it to learn the conditional distribution of a target column given an arbitrary combination of other columns. These designs enable TabDAR to not only freely handle heterogeneous tabular data but also support convenient and flexible unconditional/conditional sampling. We conduct extensive experiments on ten datasets with distinct properties, and the proposed TabDAR outperforms previous state-of-the-art methods by 18% to 45% on eight metrics across three distinct aspects. The code is available at https://github.com/fangliancheng/TabDAR .

## 1 Introduction

Figure: Figure 1: Challenges in Auto-Regressive tabular data generation. (a) The conditional distribution of continuous columns is hard to express. (b) Tabular data is column-permutation-invariant.
Refer to caption: x1.png

Due to the widespread application of synthetic tabular data in real-world scenarios, such as data augmentation, privacy protection, and missing value prediction (Fonseca & Bacao, ; Assefa et al., ; Hernandez et al., ), an increasing number of studies have begun to focus on deep generative models for synthetic tabular data generation. In this domain, various approaches, including Variational Autoencoders (VAEs)(Liu et al., ), Generative Adversarial Networks (GANs)(Xu et al., ), Diffusion Models (Zhang et al., ), and even Large Language Models (LLMs)(Borisov et al., ), have demonstrated significant progress. However, Auto-Regressive models, a crucial category of generative models, have been largely overlooked in this process. In language modeling, autoregressive models have become the de facto solution (e.g., GPTs (Mann et al., ; Achiam et al., )). Tabular data shares similarities with natural language in its discrete structure, making an autoregressive decomposition of its distribution a natural approach, i.e.,

$$ $p(\mathbf{x})=p(x_{1})\prod_{i=2}^{D}p(x_{i}|x_{1},x_{2},\cdots,x_{i-1})=p(x_{ 1})\prod_{i=2}^{D}p(x_{i}|\mathbf{x}^{<i})$ (1) $$

where each $x_{i}$ represents the value at the $i$-th column. However, research on autoregressive models for tabular data generation has not received adequate attention.

We posit that this is primarily due to two key challenges (see Fig. 1): 1) Modeling continuous distribution: Autoregressive models aim to learn the per-token (conditional) probability distribution, which is convenient for discrete tokens (such as words in Natural Languages) because their probability can be represented as a categorical distribution. However, the same approach is challenging to apply to continuous tokens unless there are strong prior assumptions about their distribution, such as assuming they follow Gaussian distributions. 2) No fixed order: Unlike natural language, which possesses an inherent causal order from left to right, tabular data exhibits column permutation invariance (^1^11Columns of a table are permutation invariant since we can randomly shuffle the columns of a table without changing its semantic meaning.). To reflect this property in the autoregressive model, we need to ensure a sequence of tokens can be generated in arbitrary order. Existing autoregressive models for tabular data generation adopt simplistic approaches to address these challenges. For instance, they discretize continuous columns, allowing them to learn corresponding categorical distributions (Castellon et al., ; Gulati & Roysdon, ). However, this method inevitably leads to information loss. Regarding the generation order, these models typically default to a left-to-right column sequence (Castellon et al., ), failing to reflect the column permutation invariant property.

To address these challenges, this paper proposes Diffusion-nested AutoRegressive Tabular Data Generation (TabDAR in short). TabDAR addresses the aforementioned issues through two ingenious design features:
1) Nested diffusion models for modeling the conditional probability distributions of the next continuous-valued tokens. TabDAR nests a small diffusion model (Ho et al., ; Karras et al., ) into the autoregressive framework for learning the conditional distribution of a continuous-valued column. Specifically, we employ two distinct loss functions for learning the distribution of the next continuous/discrete-valued tokens, respectively. For discrete columns, their conditional distribution is learned by directly minimizing the KL divergence between the prediction vector and the ground-truth one-hot category embedding. For continuous columns, we learn their distribution through a diffusion model conditioned on the output of the current location. In this way, TabDAR can flexibly learn the distribution of tabular data containing arbitrary data types.
2) Masked Transformers with bi-directional attention. TabDAR simulates arbitrary sequence orders via a transformer-architectured model with bi-directional attention mechanisms and masked inputs. The masked/unmasked locations indicate the columns that are missing/observed, ensuring that when predicting a new token, the model has only access to the known tokens.

Compared to other synthetic tabular data generation models, TabDAR offers several unparalleled advantages. 1) TabDAR employs the most appropriate generative models for different data types, i.e., diffusion models for continuous columns and categorical prediction for discrete columns, seamlessly integrating them into a unified framework. This avoids contrived processing methods such as discrete diffusion models (Lee et al., ; Kotelnikov et al., ), continuous tokenization of categorical columns (Zhang et al., ), and ineffective discretization (Castellon et al., ). 2) Through an autoregressive approach, TabDAR models the conditional distribution between different columns, thereby better capturing the dependency relationships among various columns and achieving superior density estimation. 3) Combining autoregression and masked Transformers enables a trained TabDAR to compute and sample from the conditional distribution of target columns given an arbitrary set of observable columns, enabling generation in arbitrary order. This capability facilitates precise and convenient posterior inference (e.g., conditional sampling and missing data imputation).

We conduct comprehensive experiments on ten tabular datasets of various data types and scales to verify the efficacy of the proposed TabDAR. Experimental results comprehensively demonstrate TabDAR’s superior performance in:
1) Statistical Fidelity: The ability of synthetic data to faithfully recover the ground-truth data distribution;
2) Data Utility: The performance of synthetic data in downstream Machine Learning tasks, such as Machine Learning Efficiency; and
3) Privacy Protection: Whether the synthetic data is sampled from the underlying distribution of the training data rather than being a simple copy.
In missing value imputation tasks,
TabDAR exhibits remarkable performance, even surpassing state-of-the-art methods that are specially designed for missing data imputation tasks.

## 2 Related Works

##### Synthetic Tabular Data Generation

Generative models for tabular data have become increasingly important and have widespread applications Assefa et al. (); Zheng & Charoenphakdee (); Hernandez et al. (). For example, CTGAN and TAVE (Xu et al., ) deal with mixed-type tabular data generation using the basic GAN (Goodfellow et al., ) and VAE (Kingma & Welling, ) framework. GOGGLE (Liu et al., ) incorporates Graph Attention Networks in a VAE framework such that the correlation between different data columns can be explicitly learned. DP-TBART (Castellon et al., ) and TabMT (Gulati & Roysdon, ) use discretization techniques to numerical columns and then apply autoregressive transformers for a generation. Recently, inspired by the success of Diffusion models in image generation, a lot of diffusion-based methods have been proposed, such as TabDDPM (Kotelnikov et al., ), STaSy (Kim et al., ), CoDi (Lee et al., ), and TabSyn (Zhang et al., ), which have achieved SOTA synthesis quality.

##### Autoregressive Models for Continuous Space Data

In text generation, autoregressive next-token generation is undoubtedly the dominant approach (Mann et al., ; Achiam et al., ). However, in image generation, although autoregressive models (Van den Oord et al., ; Salimans et al., ) were proposed early on, their pixel-level characteristics limited their further development. Subsequently, diffusion models, which are naturally suited to modeling continuous distributions, have become the most popular method in the field of image generation. In recent years, some studies have attempted to use discrete-value image tokens (van den Oord et al., ; Razavi et al., ) and employ autoregressive transformers for image-generation tasks (Kolesnikov et al., ). However, discrete tokenizers are both difficult to train and inevitably cause information loss. To this end, recent work has attempted to combine continuous space diffusion models with autoregressive methods. For example, Li et al. () employs an autoregressive diffusion loss in a causal Transformer for learning image representations; Li et al. () proposes using a diffusion model to model the conditional distribution of the next continuous image and employs a masked bidirectional attention mechanism to enable the generation of any number of tokens in arbitrary order.

## 3 Preliminaries

### 3.1 Definitions and Notations

In this paper, we always use uppercase boldface (e.g., $\mathbf{X}$) letters to represent matrices, lowercase boldface letters (e.g., $\mathbf{x}$) to represent vectors, and regular italics (e.g., $x$) to denote scalar entries in matrices or vectors. Tabular data refers to data organized in a tabular format consisting of rows and columns. Each row represents an instance or observation, while each column represents a feature or variable.
In this work, we consider heterogeneous tabular data that may contain both numerical and categorical columns or only one of these types. Let $\mathcal{D}=\{\mathbf{x}_{i}\}_{i=1}^{N}$ denote a tabular dataset comprising $N$ instances, where each instance $\mathbf{x}=(x^{1},x^{2},\ldots,x^{D})$ is a $D$-dimensional vector representing the values of $D$ features or variables. We further categorize the features into two types:
1) Numerical/continuous features: $\mathcal{N}=\{i\mid x^{i}\in\mathbb{R}\}$ is the set of indices corresponding to numerical features.
2) Categorical/discrete features: $\mathcal{C}=\{i\mid x^{i}\in\mathcal{C}_{i}\}$ is the set of indices corresponding to categorical features, where $\mathcal{C}_{i}$ is the set of possible categories for the $i$-th feature. Note that $\mathcal{N}\cup\mathcal{C}={1,2,\ldots,D}$ and $\mathcal{N}\cap\mathcal{C}=\emptyset$.

### 3.2 Diffusion Models

Diffusion models (Ho et al., ; Song et al., ; Karras et al., ) learn the data distribution $p(\mathbf{x})$ through a diffusion SDE, which consists of a forward process that gradually adds Gaussian noises of increasing scales to $\mathbf{x}$ (which are pre-normalized to have zero-mean and unit-variance), and a reverse process that recovers the clean data from the noisy one(^2^22This formulation is a simplified version of VE-SDE (Song et al., ). See Appendix B for details.):

$$ $\displaystyle\mathbf{x}_{t}$ $\displaystyle=\mathbf{x}_{0}+\sigma(t)\bm{\varepsilon},\;\bm{\varepsilon}\sim{ \mathcal{N}}(\bm{0},\mathbf{I}),$ $\displaystyle(\text{Forward Process})$ (2) $\displaystyle{\rm d}\mathbf{x}_{t}$ $\displaystyle=-2\dot{\sigma}(t)\sigma(t)\nabla_{\mathbf{x}_{t}}\log p(\mathbf{ x}_{t}){\rm d}t+\sqrt{2\dot{\sigma}(t)\sigma(t)}{\rm d}\bm{\omega}_{t},$ $\displaystyle(\text{Reverse Process})$ (3) $$

where $\mathbf{\omega}_{t}$ is the standard Wiener process. $\sigma(t)$ is the noise schedule, and $\dot{\sigma}(t)$ is the derivative of $\sigma(t)$ w.r.t. $t$ A diffusion model is learned by using a denoising/score network $\bm{\epsilon}_{\theta}(\mathbf{x}_{t},t)$ to approximate the conditional score function $\nabla_{\mathbf{x}_{t}}\log p(\mathbf{x}_{t}|\mathbf{x}_{0})$ (named score-matching). The final loss function could be reduced to a simple formulation where the denoising network is optimized to approximate the added noise $\bm{\varepsilon}$, i.e.,

$$ ${\mathcal{L}}(\mathbf{x})=\mathbb{E}_{t\sim p(t)}\mathbb{E}_{\bm{\varepsilon} \sim{\mathcal{N}}(\bm{0},\mathbf{I})}\|\bm{\epsilon}_{\theta}(\mathbf{x}_{t},t )-\bm{\varepsilon}\|^{2}.$ (4) $$

The sampling process starts from a large timestep $T$ such that ${\bm{x}}_{T}\approx{\mathcal{N}}(\bm{0},{\sigma^{2}(T)}\mathbf{I})$ recovers $\mathbf{x}_{0}$ via solving the reverse SDE in Eq. 3, using mature numerical solution tools.

## 4 Methods

In this section, we introduce the proposed TabDAR. We first introduce the key ingredients of TabDAR for solving the challenges in applying autoregressive methods for tabular data synthesis: 1) Masked transformers with bidirectional attention for simulating arbitrary generation orders. 2) Modality-specific losses for handling discrete/continuous columns. Then, we introduce the detailed layer-by-layer architecture of TabDAR, including the training and inference algorithms.

### 4.1 Key Ingredients of TabDAR

##### Modeling tabular data using autoregression.

TabDAR follows the autoregressive criteria for modeling the distribution of tabular data. Autoregressive modeling decomposes the joint data distribution into the product of a series of conditional distributions in raster order, and the optimization is achieved by minimizing the standard negative log-likelihood:

$$ $p(\mathbf{x})=\prod\limits_{i=1}^{D}p(x^{i}|\mathbf{x}^{<i}),\;{\mathcal{L}}=- \log p(\mathbf{x})=-\sum\limits_{i=1}^{D}\log p(x^{i}|\mathbf{x}^{<i})$ (5) $$

Therefore, instead of directly modeling the complicated joint distribution, one can seek to model each conditional distribution, i.e., $p(x^{i}|\mathbf{x}^{<i})$, separately, which is intuitively much simpler.

Autoregression is natural for language modeling, as language inherently possesses a left-to-right (L2R in short) order, and we can expect that the generation of a new word depends entirely on the observed words preceding it. Therefore, Transformers (Vaswani et al., ) with causal attention (where only the previous tokens are observable to predict the following tokens) are widely used for text generation. Unlike text data, tabular data has long been considered column permutation invariant. In this case, one has to randomly shuffle the columns many times and then apply causal attention to each generated order, which is conceptually complicated.

Figure: Figure 2: With appropriate masking, bidirectional attention is equivalent to causal attention in arbitrary order.
Refer to caption: x2.png

##### Simulate arbitrary order using masked Bi-directional Attention.

To deal with this, an interesting observation is that even if given the default order of data, we can simulate arbitrary-order causal attention using masked bi-directional attention. As illustrated in Fig. 2, giving the tabular data arranged in the default order [‘age’, ‘capital gain’, ‘work’, ‘education’, ‘income’] and a shuffled generation order [‘education’ $\rightarrow$ ‘income’ $\rightarrow$ ‘capital gain’, $\rightarrow$ ‘age’, $\rightarrow$ ‘work’], the prediction of ‘capital gain’ can be equivalently achieved by 1) causal attention based on the previous columns ‘education’ and ‘income’; 2) bidirectional attention where all other columns (‘age, ‘capital gain’, and ‘work’) are masked.

##### Column-specific losses for discrete/continuous columns

Autoregressive models aim at learning the conditional distribution of the target column given the observations of previous columns, i.e., $p(x^{i}|\mathbf{x}^{<i})$. Take $\mathbf{x}^{<i}$ as input, Transformers are able to generate column-specific output vectors, i.e., $\mathbf{z}^{i}$ for column $i$. Then we only need to model the conditional distribution $p(x^{i}|\mathbf{z}^{i})$.

Discrete Columns. For a discrete column $x^{i}$, the target distribution is a categorical distribution, which could be represented by a $|\mathcal{C}_{i}|$-dimensional vector. Consequently, we can directly project $\mathbf{z}^{i}$ to a $|\mathcal{C}_{i}|$-way classifier using a prediction head $f_{i}(\cdot)$ (e.g., a shallow MLP: $\mathbb{R}^{d}\rightarrow\mathbb{R}^{|\mathcal{C}_{i}|}$), and then minimize the KL-divergence between the predicted distribution and the one-hot encoding of a sample:

$$ ${\mathcal{L}}({x}^{i},\mathbf{z}^{i})=-\log p(p({x}^{i}|\mathbf{z}^{i}))=\text {Cross-Entropy}({x}^{i},\text{softmax}(f_{i}(\mathbf{z}^{i})),\;\;i\in{ \mathcal{C}}$ (6) $$

Continuous Columns. Unlike discrete columns, a continuous column might have infinite value states and, therefore, cannot be modeled by a categorical distribution(^3^33MSE loss is also not acceptable since it is a deterministic mapping rather than a conditional distribution). Inspired by the prominent capacity of Diffusion models in modeling arbitrary continuous distributions, we hereby adopt a conditional diffusion model to learn the conditional continuous distribution (Li et al., ) $p(x^{i}|\mathbf{z}^{i})$. Compared with the unconditional diffusion loss in Eq. 4, the denoising function $\mathbf{\epsilon}_{\theta}$ takes the condition vector $\mathbf{z}^{i}$ as an additional input. The final loss function is expressed as follows:

$$ ${\mathcal{L}}({x}^{i},\mathbf{z}^{i})=-\log p(p({x}^{i}|\mathbf{z}^{i}))=-\log \mathbb{E}_{t\sim p(t)}\mathbb{E}_{\bm{\varepsilon}\sim{\mathcal{N}}(\bm{0}, \mathbf{I})}\|\bm{\epsilon}_{\theta}({x}_{t}^{i},t,\mathbf{z}^{i})-\bm{ \varepsilon}\|^{2},\;\;i\in{\mathcal{N}}$ (7) $$

### 4.2 Detailed Architecture of TabDAR

In this section, we introduce the detailed architecture and implementations of TabDAR. The overall framework of TabDAR is presented in Fig. 4.

Figure: Figure 3: The overall framework of TabDAR. An embedding layer first encodes each column into a vector. The masks are then added to the target columns (‘age’ and ‘education’). With Bi-direction Transformers’ decoding, the output vectors ${\bm{z}}$ are used as conditions for predicting the distribution of current columns. TabDAR uses the conditional diffusion loss for continuous columns and cross-entropy loss for discrete columns. Losses are computed only on the masked tokens
Refer to caption: x3.png

Tokenization layer. Given a row of tabular data $\mathbf{x}=(x^{1},x^{2},\cdots,x^{D})$, the tokenization layer embeds each feature to a $d$-dimensional vector using column-specific learnable linear transformation. For a continuous column, the linear transformation is $\mathbf{W}^{i}\in\mathbb{R}^{1\times d}$, while for a discrete column, the transformation matrix is $\mathbf{W}^{i}\in\mathbb{R}^{|\mathcal{C}_{i}|\times d}$. The output token of column $i$ is denoted by $\mathbf{h}^{i}$, and we obtain an embedding matrix of $D$ tokens $\mathbf{H}\in\mathbb{R}^{D\times d}$. We also add column-specific learnable positional encoding so that the token embeddings at different locations can be discriminated.

Input masking. As illustrated in Section 4.1, we can use masked bidirectional attention to simulate arbitrary causal attention. Therefore, we first uniformly sample the number of tokens to mask/predict $M\sim\mathcal{U}(1,D)$. Given $M$, we randomly sample a masking vector $\mathbf{m}\in\{0,1\}^{D}$, s.t., $\sum_{i}m^{i}=M$. $m^{i}=1$ indicates that column $i$ is masked and vice versa. Then, the input embeddings of masked columns are set as zero, making them unobservable when predicting the target columns.

$$ $\mathbf{H}^{\prime}=(1-\mathbf{m})\odot\mathbf{H}.$ (8) $$

Bi-directional Transformers. Given the masked token sequence, we then follow the standard pipeline of Transformer implementations. We first pad the embedding of the class token [pad] at the beginning of the sequence. The [pad] tokens not only enable class-conditional generation but also help stabilize the training process (Li et al., ). By default, we use one unique [pad] token, equivalent to unconditional generation (we may use class-specific [pad] tokens for class-conditional distribution). Afterward, these tokens are processed by a series of standard Transformer blocks and will finally output column-wise latent vectors $\{\mathbf{z}^{i}\}_{i=1}^{D}$.

Loss function and model training. After obtaining the latent vector $\mathbf{z}^{i}$ for a target/masked column $i$, we use it as the condition for predicting the distribution of $\mathbf{x}^{i}$, i,e., $p({x}^{i}|\mathbf{z}^{i})$.

Discrete columns. As presented in Eq. 6, we use a column-specific prediction head $f_{i}(\cdot)$ to obtain the $|{\mathcal{C}}_{i}|$-dimensional prediction then compute the cross-entropy loss.

Continuous columns. For predicting the distribution of the continuous column ${x}^{i}$, we use the conditional diffusion loss described in Eq. 7. The training of the conditional diffusion model follows the standard process of diffusion models (Karras et al., ). To be specific, we first randomly sample a timestep $t\sim p(t)$, and standard Gaussian noise ${\varepsilon}\sim{\mathcal{N}}({0},1)$, obtaining ${x}_{t}^{i}={x}^{i}+\sigma(t)\cdot{\varepsilon}$. Then, a denoising neural network $\mathbf{\epsilon}_{\theta}(x^{i}_{t},t,\mathbf{z}^{i})$ takes $x^{i}_{t},t$, and $\mathbf{z}^{i}$ as input to predict the added noise $\varepsilon$ (via the MSE loss in Eq. 7). The denoising neural network is implemented as a shallow MLP, and its detailed architecture is presented in Appendix C.

Model Training. The training of TabDAR is end-to-end: all the model parameters, including the Embedding layer, transformers, prediction heads for discrete columns, and diffusion models for the continuous columns, are jointly trained via gradient descent on the summation of the losses of the target/masked columns in the current batch.

Model inference. After TabDAR is trained, we can easily perform a variety of inference tasks, e.g., unconditional and conditional generation. An illustration of TabDAR’s auto-regressive sampling process is presented in Fig. 4.

Unconditional Sampling. To generate unconditional data examples $\mathbf{x}\sim p(\mathbf{x})$, we can first randomly sample a generation order. Then, starting with all columns masked (except the [pad] token), we can generate the value of each column one by one according to the sampled order.

Conditional Sampling. One important advantage of TabDAR’s autoregressive generation manner is that one can perform flexible conditional sampling, such as simple class-conditional generation. This can be achieved by directly setting the corresponding column values to the desired ones and then randomly sampling the generation order of other columns. Missing value imputation can be regarded as a special case of conditional sampling by resorting to the conditional distribution of missing columns given observed values. More implementation details can be found in Appendix D.4.

## 5 Experiments

### 5.1 Experimental Setups

Datasets. We select ten real-world tabular datasets of varying data types and sizes: 1) two contain only continuous features – California and Letter; 2) two contain only categorical features – Car and Nursery; 3) six datasets of mixed continuous and discrete features – Adult, Default, Shoppers, Magic, News, and Beijing. The detailed introduction of these datasets can be found Section D.2.

Baselines.
We compare the proposed TabDAR with ten powerful synthetic tabular data generation methods belonging to six categories. 1) The non-parametric interpolation method SMOTE (Chawla et al., ). 2) VAE-based methods TVAE (Xu et al., ) and GOGGLE (Liu et al., ). 3) GAN-based method CTGAN (Xu et al., ). 4) LLM-based method GReaT (Borisov et al., ). 5) Diffusion-based methods: STaSy (Kim et al., ), CoDi (Lee et al., ), TabDDPM (Kotelnikov et al., ), and TabSyn (Zhang et al., ). 6) Autoregressive methods DP-TBART (Castellon et al., ) and Tab-MT (Gulati & Roysdon, ). The proposed TabDAR belongs to the autoregressive family, yet it utilizes a diffusion model for modeling continuous columns.

Evaluation Methods. We evaluate the quality of synthetic tabular data from three distinct dimensions: 1) Fidelity - if the synthetic data faithfully recovers the ground-truth data distribution.
2) Utility - the performance when applied to downstream tasks, and we focus on Machine Learning Efficiency (MLE). 3) Privacy - if the synthetic data is not copied from the real records. More introduction of these metrics is in Appendix D.6.

### 5.2 Main Results

##### Statistical Fidelity

We first investigate whether synthetic data can faithfully reproduce the distribution of the original data. We use various statistical metrics to reflect the degree to which synthetic data estimates the distribution density of the original data. These metrics include univariate density estimation (i.e., the marginal distribution of a single column), the correlation between any two variables (which reflects the joint probability distribution), $\alpha$-Precision (reflecting whether a synthetic data example is close to the true distribution), $\beta$-Recall (reflecting the coverage of synthetic data over the original data), Classifier Two Sample Test (C2ST, reflecting the difficulty in distinguishing between synthetic data and real data), and Jensen-Shannon divergence (JSD, estimating the distance between the distributions of real data and synthetic data). Due to space limitations, in this section, we only present the average performance with standard deviation on each metric across all ten datasets. The detailed performance on each individual dataset is in Appendix E.

In Table 1, we present the performance comparison on these fidelity metrics. As demonstrated, our model achieved performance far surpassing the second-best method in five out of six fidelity metrics, with advantages ranging from $18.18\%$ to $45.65\%$. Considering that these metrics have already been elevated to a considerably high level due to the explosive development of recent deep generative models for tabular data, our improvement is very significant.
The only exception is $\beta$-Recall, which measures the degree of coverage of the synthetic data over the entire data distribution. On this metric, the simple classical interpolation method SMOTE achieved the best performance, far surpassing other generative models. On the other hand, SMOTE also achieves very good performance on other metrics, surpassing many generative methods. These phenomena indicate that simple interpolation methods can indeed obtain synthetic data with a distribution close to that of real data. However, the limitation of interpolation methods lies in the fact that the synthetic data is too close to the real data, making it resemble a copy from the training set rather than a sample from the underlying distribution, which may cause privacy issues. Detailed experiments are in the Privacy Protection section.

**Table 1: Comparison of different methods regarding the statistical fidelity of the synthetic data. All metrics have been scaled so that lower numbers indicate better performance, to facilitate better numerical comparison.**
|  | Method | Marginal$\downarrow\%$ | Joint$\downarrow\%$ | $\alpha$-Precision$\downarrow\%$ | $\beta$-Recall$\downarrow\%$ | C2ST$\downarrow\%$ | JSD$\downarrow 10^{-2}$ |
| --- | --- | --- | --- | --- | --- | --- | --- |
| Interpolation |  |  |  |  |  |  |  |
|  | SMOTE (Chawla et al., ) | $1.72$$\pm 1.36$ | $2.95$$\pm 1.66$ | $3.78$$\pm 3.94$ | 16.7$\pm\textbf{9.16}$ | $3.00$$\pm 3.66$ | $0.11$$\pm 0.10$ |
| VAE-based |  |  |  |  |  |  |  |
|  | TVAE (Xu et al., ) | $15.8$$\pm 17.1$ | $17.4$$\pm 18.3$ | $18.2$$\pm 20.1$ | $70.9$$\pm 26.3$ | $43.9$$\pm 22.7$ | $1.01$$\pm 0.70$ |
|  | GOGGLE (Liu et al., ) | $17.2$$\pm 6.28$ | $29.1$$\pm 11.8$ | $21.8$$\pm 17.3$ | $90.8$$\pm 5.64$ | - | - |
| GAN-based |  |  |  |  |  |  |  |
|  | CTGAN (Xu et al., ) | $17.9$$\pm 6.99$ | $18.4$$\pm 9.11$ | $17.7$$\pm 15.1$ | $69.1$$\pm 33.8$ | $53.0$$\pm 22.5$ | $1.18$$\pm 0.69$ |
| LLM-based |  |  |  |  |  |  |  |
|  | GReaT (Borisov et al., ) | $12.9$$\pm 6.05$ | $44.3$$\pm 27.3$ | $17.2$$\pm 12.8$ | $53.2$$\pm 26.0$ | $42.4$$\pm 19.2$ | $1.43$$\pm 1.18$ |
| Diffusion-based |  |  |  |  |  |  |  |
|  | STaSy (Kim et al., ) | $14.3$$\pm 7.40$ | $13.5$$\pm 9.76$ | $21.8$$\pm 24.7$ | $55.6$$\pm 29.6$ | $53.9$$\pm 16.6$ | $1.25$$\pm 1.13$ |
|  | CoDi (Lee et al., ) | $17.4$$\pm 11.3$ | $15.2$$\pm 19.8$ | $10.0$$\pm 5.93$ | $51.7$$\pm 31.1`$ | $44.0$$\pm 33.3$ | $0.76$$\pm 0.50$ |
|  | TabDDPM (Kotelnikov et al., ) | $15.0$$\pm 25.3$ | $7.92$$\pm 8.16$ | $23.6$$\pm 2.93$ | $49.6$$\pm 34.5$ | $24.6$$\pm 38.9$ | $1.03$$\pm 1.60$ |
|  | TabSyn (Zhang et al., ) | $1.73$$\pm 0.76$ | $2.53$$\pm 1.45$ | $2.52$$\pm 2.93$ | $44.1$$\pm 24.5$ | $2.76$$\pm 2.19$ | $.12$$\pm 0.09$ |
| Autoregressive-based |  |  |  |  |  |  |  |
|  | DP-TBART (Castellon et al., ) | $3.24$$\pm 1.76$ | $2.71$$\pm 1.56$ | $2.11$$\pm 2.15$ | $48.0$$\pm 27.8$ | $5.36$$\pm 4.58$ | $0.16$$\pm 0.11$ |
|  | Tab-MT (Gulati & Roysdon, ) | $14.9$$\pm 15.1$ | $8.11$$\pm 10.8$ | $23.2$$\pm 31.9$ | $63.5$$\pm 38.3$ | $48.6$$\pm 47.7$ | $0.60$$\pm 0.83$ |
|  | TabDAR (ours) | $\bf{{\color[rgb]{0.8,0.25,0.1}1.21}}$$\bf{{\color[rgb]{0.8,0.25,0.1}\pm 0.51}}$ | $\bf{{\color[rgb]{0.8,0.25,0.1}1.80}}$$\bf{{\color[rgb]{0.8,0.25,0.1}\pm 0.72}}$ | $\bf{{\color[rgb]{0.8,0.25,0.1}1.33}}$$\bf{{\color[rgb]{0.8,0.25,0.1}\pm 1.10}}$ | $37.2$${\pm 20.1}$ | $\bf{{\color[rgb]{0.8,0.25,0.1}1.50}}$$\bf{{\color[rgb]{0.8,0.25,0.1}\pm 1.42}}$ | $\bf{{\color[rgb]{0.8,0.25,0.1}0.09}}$$\bf{{\color[rgb]{0.8,0.25,0.1}\pm 0.048}}$ |
|  | Improvement | $\bf{{\color[rgb]{0.8,0.25,0.1}29.65\%}}$ | $\bf{{\color[rgb]{0.8,0.25,0.1}28.85\%}}$ | $\bf{{\color[rgb]{0.8,0.25,0.1}36.97\%}}$ | - | $\bf{{\color[rgb]{0.8,0.25,0.1}45.65\%}}$ | $\bf{{\color[rgb]{0.8,0.25,0.1}18.18\%}}$ |

##### Utility on Downstream Tasks

**Table 2: AUC (classification task) and RMSE (regression task) scores of Machine Learning Efficiency. ↑ (↓) indicates that the higher (lower) the score, the better the performance.**
|  | Method | Continuous only |  | Discrete only |  | Heterogeous |  |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
|  | California | Letter |  | Car | Nursery |  | Adult | Default | Shoppers | Magic | News | Beijing |  |  |
|  |  | AUC$\uparrow$ | AUC$\uparrow$ |  | AUC$\uparrow$ | AUC$\uparrow$ |  | AUC$\uparrow$ | AUC$\uparrow$ | AUC$\uparrow$ | AUC$\uparrow$ | RMSE$\downarrow$ | RMSE $\downarrow$ |  |
|  | Real data | $0.999$ | $0.989$ |  | $0.999$ | $1.000$ |  | $0.927$ | $0.770$ | $0.926$ | $0.946$ | $0.842$ | $0.423$ |  |
| VAE-based |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
|  | TVAE | $0.986$ | $0.989$ |  | $0.746$ | $0.939$ |  | $0.846$ | $0.744$ | $0.898$ | $0.912$ | $0.979$ | $1.010$ |  |
|  | GOGGLE | - | - |  | - | - |  | $0.778$ | $0.584$ | $0.658$ | $0.654$ | $1.09$ | $0.877$ |  |
| GAN-based |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
|  | CTGAN | $0.925$ | $0.729$ |  | $0.899$ | $1.000$ |  | $0.874$ | $0.736$ | $0.868$ | $0.874$ | $0.845$ | $1.065$ |  |
| LLM-based |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
|  | GReaT | $0.996$ | $0.983$ |  | $0.979$ | $0.999$ |  | $0.913$ | $0.755$ | $0.902$ | $0.888$ | - | $0.653$ |  |
| Diffusion-based |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
|  | STaSy | 0.997 | $0.990$ |  | $0.927$ | $0.982$ |  | $0.903$ | $0.749$ | $0.909$ | $0.923$ | $0.933$ | $0.672$ |  |
|  | CoDi | $0.981$ | $0.998$ |  | $0.995$ | $1.000$ |  | $0.829$ | $0.497$ | $0.855$ | $0.930$ | $0.999$ | $0.750$ |  |
|  | TabDDPM | $0.992$ | $0.513$ |  | $0.995$ | $1.000$ |  | $0.911$ | $0.763$ | $0.915$ | $0.933$ | - | $2.665$ |  |
|  | TabSyn | $0.993$ | $0.990$ |  | $0.971$ | $0.997$ |  | $0.904$ | $0.764$ | $0.913$ | $0.934$ | $0.862$ | $0.669$ |  |
| Autoregressive |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
|  | DP-TBART | $0.993$ | $0.985$ |  | $0.990$ | $0.917$ |  | $\bf{{\color[rgb]{0.8,0.25,0.1}0.918}}$ | $0.717$ | $0.896$ | $0.924$ | $0.896$ | $0.676$ |  |
|  | Tab-MT | $0.988$ | $0.985$ |  | $0.981$ | $1.000$ |  | $0.873$ | $0.714$ | $0.912$ | $0.822$ | $1.002$ | $2.098$ |  |
|  | TabDAR | $0.994$ | $\bf{{\color[rgb]{0.8,0.25,0.1}0.994}}$ |  | $\bf{{\color[rgb]{0.8,0.25,0.1}0.996}}$ | $\bf{{\color[rgb]{0.8,0.25,0.1}1.000}}$ |  | $0.904$ | $\bf{{\color[rgb]{0.8,0.25,0.1}0.764}}$ | $\bf{{\color[rgb]{0.8,0.25,0.1}0.916}}$ | $\bf{{\color[rgb]{0.8,0.25,0.1}0.935}}$ | $\bf{{\color[rgb]{0.8,0.25,0.1}0.856}}$ | $\bf{{\color[rgb]{0.8,0.25,0.1}0.579}}$ |  |

We then evaluate the quality of synthetic data by assessing their performance in Machine Learning Efficiency (MLE) tasks. Following previous settings (Zhang et al., ), we first split a real table into a real training and a real testing set. The generative models are trained on the real training set, from which a synthetic set of equivalent size is sampled. This synthetic data is then used to train a classification/regression model (XGBoost Classifier and XGBoost Regressor (Chen & Guestrin, )), which will be evaluated using the real testing set. The performance of MLE is measured by the AUC score for classification tasks and RMSE for regression tasks.
As demonstrated in Table 2, the proposed TabDAR gives a fairly satisfying performance on the MLE tasks, and the classification/regression performance obtained via training on the synthetic set is rather close to that on the original training set. Also, similar to the observations in (Zhang et al., ), the differences between various methods on MLE tasks are very small, even though some methods may not correctly learn the distribution of the original data. Therefore, it is important to combine this with the fidelity metrics in Table 1 to obtain a more comprehensive evaluation of synthetic data.

##### Privacy Protection

Finally, a good synthetic dataset should not only faithfully reproduce the original data distribution but also ensure that it is sampled from the underlying distribution of the real data rather than being a copy. To this end, we use the Distance to Closest Record (DCR) score for measurement. Specifically, we divide the real data equally into two parts of the same size, namely the training set and the holdout set. We train the model based on the training set and obtain the sampled synthetic dataset. Afterward, we calculate the distribution of distances between each synthetic data example and its closest sample in both the training set and the holdout set. Intuitively, if the training set and holdout set are both drawn uniformly from the same distribution and if the synthetic data has learned the true distribution, then on average, the proportion of synthetic data samples that are closer to the training set should be the same as those closer to the holdout set (both 50%). Conversely, if the synthetic data is copied from the training set, the probability of it being closer to samples in the training set would far exceed 50%.

In Table 3 and Figure 5, we present the probability comparison and the distributions of the DCR scores of SMOTE, TabDDPM, TabSyn, and the proposed TabDAR. As demonstrated, SMOTE is poor at privacy protection, as its synthetic sample tends to be closer to the training set rather than the holdout set, as its synthetic examples are obtained via interpolation between training examples. By contrast, the remaining deep generative methods are all good at preserving the privacy of training data, leading to almost completely overlapped DCR distributions.

Figure: Table 3: Probability that a synthetic example’s DCR to the training set rather than that of the holdout set), a score closer to $50\%$ is better.
Refer to caption: x5.png

### 5.3 Missing Data Imputation

Since TabDAR directly models the factorized conditional probability of the target column(s) given the observation of other columns, it has a strong potential for missing data imputation. In this section, we compare the proposed TabDAR with the current state-of-the-art (SOTA) methods for missing data imputation, including KNN (Pujianto et al., ), GRAPE (You et al., ), MOT (Muzellec et al., ), and Remakser (Du et al., ). We consider the task of training the model on the complete training data and then imputing the missing values of testing data. The missing mechanism is Missing Completely at Random (MCAR), and the missing ratio of testing data is set at 30%.
In Figure 6, we present the performance comparison on four datasets: Letter, Adult, Default, and Shoppers (as well as the average imputation performance). The proposed TabDAR demonstrates superior performance on these four datasets, significantly outperforming the current best methods on three out of four datasets. On the Adult dataset, it was slightly inferior. These results confirm that TabDAR is not only suitable for unconditional generation but also applicable to other conditional generation tasks, demonstrating a wide range of application values.

### 5.4 Ablation Studies

Effects of diffusion loss and random ordering.
We first study if the two key ingredients of TabDAR– 1) the diffusion loss and 2) random ordering – are indeed beneficial. We consider three variants of TabDAR: ‘w/o. Diff. Loss’, which indicates that we discretize the continuous columns into $100$ uniform bins, treating them as discrete ones, such that we can apply the cross-entropy loss; ‘w/o rand. order,’ which indicates that we follow the default left-to-right order in both the training and generation phases using a causal attention Transformer model; ‘w/o. both’, which indicates the version that combines the two variants.
In Table 4, we compare TabDAR with the three variants on the average synthetic data’s fidelity across all the datasets. We can observe that both the diffusion loss and random training/generation order are important to TabDAR’s success. Specifically, the diffusion loss targeting the modeling of the conditional distribution of the continuous columns contributes the most, and the random ordering further improves TabDAR’s performance. Furthermore, random ordering enables TabDAR to perform flexible conditional inference tasks like imputation.

Figure: Figure 6: (30% MCAR) Comparison of missing value imputation performance with 4 competitive baselines.
Refer to caption: x6.png

Sensitivity to hyperparameters.
We then study TabDAR’s sensitivity to its hyperparameters that specify its transformer architecture: the model depth (number of Transformer layers) and the embedding dimension $d$. From Table 5 and Table 6 (grey cells indicate the default setting), we can observe that although there exists an optimal hyperparameter setting (i.e., depth = 6, $d$ = 32), the change of the two hyperparameters has little impact on the model performance. These results demonstrate that our model is relatively robust to these hyperparameters. Therefore, good results can be obtained for different datasets without the need for specific hyperparameter tuning.

Figure: Table 5: Impacts of the model depth (number of Transformer layers) on Adult and Beijing.

## 6 Visualizations

Figure: Figure 7: 2D joint distribution density of ‘age’ and ‘education num’ of Adult dataset.
Refer to caption: x7.png

Figure: Figure 8: 2D joint distribution density of ‘Longitude’ and ‘Latitude’ of California dataset.
Refer to caption: x8.png

Figure: Figure 9: Heatmaps of the joint column correlation estimation error. Lighter areas indicate a lower error in estimating the correlation between two columns.
Refer to caption: x9.png

In this section, we visualize the synthetic data to demonstrate that the proposed TabDAR can generate synthetic data that closely resembles the ground-truth distribution. In Figure 7 and Figure 8, we plot the 2D joint distribution of two columns of the Adult and California datasets to investigate if the ground-truth joint distribution density can be learned by the synthetic data. In Figure 9, we further plot the heatmaps of the estimation error of column pair correlations. These results visually demonstrate that TabDAR can generate synthetic samples very close to the distribution of real data and faithfully reflect the correlations between different columns of the data.

## 7 Conclusion

This paper has presented TabDAR, a generative model that embeds a diffusion model within an autoregressive transformer framework used for multi-modal tabular data synthesis. TabDAR uses a novel diffusion loss and traditional cross-entropy loss to learn the conditional distributions of continuous columns and discrete columns, respectively, enabling a single autoregressive model to generate both continuous and discrete features simultaneously. Furthermore, TabDAR employs masked bidirectional attention to simulate arbitrary autoregressive orders, allowing the model to generate in any direction. This not only enhances the accuracy of joint probability modeling but also enables more flexible conditional generation. Extensive experimental results demonstrate the effectiveness of the proposed method.

## Appendix A Algorithms

In this section, we provide detailed algorithms of TabDAR. Algorithm 3 provides the training algorithm of TabDAR. Algorithm 4 and 5 provide the unconditional and conditional generation algorithm of TabDAR, respectively. For simplicity, the algorithms are presented with a single data as input, and it is straightforward to extend to a batch setting.

Figure: Algorithm 1 Loss for continuous columns

Figure: Algorithm 2 Loss for discrete columns

Figure: Algorithm 3 TabDAR: Training

Figure: Algorithm 4 TabDAR: Unconditional Generation

Figure: Algorithm 5 TabDAR: Conditional Generation (Imputation)

## Appendix B Diffusion SDEs

This paper adopts the simplified version of the Variance-Exploding SDE in  (Song et al., ).  Song et al. () has proposed the following general-form forward SDE:

$$ ${\rm d}\mathbf{x}={\bm{f}}(\mathbf{x},t){\rm d}t+g(t)\;{\rm d}{\bm{w}}_{t}={ \rm d}\mathbf{x}=f(t)\;\mathbf{x}\;{\rm d}t+g(t)\;{\rm d}{\bm{w}}_{t}.$ (9) $$

Then the conditional distribution of $\mathbf{x}_{t}$ given $\mathbf{x}_{0}$ (named as the perturbation kernel of the SDE) could be formulated as:

$$ $p(\mathbf{x}_{t}|\mathbf{x}_{0})={\mathcal{N}}(\mathbf{x}_{t};s(t)\mathbf{x}_{ 0},s^{2}(t)\sigma^{2}(t){\bm{I}}),$ (10) $$

where

$$ $s(t)=\exp\left(\int_{0}^{t}f(\xi){\rm d}\xi\right),\;\mbox{and}\;\sigma(t)= \sqrt{\int_{0}^{t}\frac{g^{2}(\xi)}{s^{2}(\xi)}{\rm d}\xi}.$ (11) $$

Therefore, the forward diffusion process could be equivalently formulated by defining the perturbation kernels (via defining appropriate $s(t)$ and $\sigma(t)$).

Variance Exploding (VE) implements the perturbation kernel Eq. 10 by setting $s(t)=1$, indicating that the noise is directly added to the data rather than weighted mixing. Therefore, The noise variance (the noise level) is totally decided by $\sigma(t)$. When $s(t)=1$, the perturbation kernels become:

$$ $p(\mathbf{x}_{t}|\mathbf{x}_{0})={\mathcal{N}}(\mathbf{x}_{t};\bm{0},\sigma^{2 }(t)\mathbf{I})\;\Rightarrow\;\mathbf{x}_{t}=\mathbf{x}_{0}+\sigma(t)\bm{ \varepsilon},$ (12) $$

which aligns with the forward diffusion process in Eq. 2.

The sampling process of diffusion SDE is given by:

$$ ${\rm d}\mathbf{x}=[{\bm{f}}(\mathbf{x},t)-g^{2}(t)\nabla_{\mathbf{x}}\log p_{t }(\mathbf{x})]{\rm d}t+g(t){\rm d}{\bm{w}}_{t}.$ (13) $$

For VE-SDE, $s(t)=1\Leftrightarrow{\bm{f}}(\mathbf{x},t)=f(t)\cdot\mathbf{x}=\bm{0}$, and

$$ $\begin{split}&\sigma(t)=\sqrt{\int_{0}^{t}g^{2}(\xi){\rm d}\xi}\Rightarrow\int _{0}^{t}g^{2}(\xi){\rm d}\xi=\sigma^{2}(t),\\ &g^{2}(t)=\frac{{\rm d}\sigma^{2}(t)}{{\rm d}t}=2\sigma(t)\dot{\sigma}(t),\\ &g(t)=\sqrt{2\sigma(t)\dot{\sigma}(t)}.\end{split}$ (14) $$

Plugging $g(t)$ into Eq. 13, the reverse process in Eq. 3 is recovered:

$$ ${\rm d}\mathbf{x}_{t}=-2\sigma(t)\dot{\sigma}(t)\nabla_{\mathbf{x}_{t}}\log p( \mathbf{x}_{t}){\rm d}t+\sqrt{2\sigma(t)\dot{\sigma}(t)}{\rm d}\bm{\omega}_{t}.$ (15) $$

## Appendix C Detailed Model Architectures

In this section, we introduce the detailed architecture of TabDAR, which consists of a tokenization layer, several transformer blocks, and two (group of) predictors for numerical and categorical features, respectively.

### C.1 Tokenizers and Transformers

##### Tokenization.

We first apply one-hot encoding to the categorical features and then project both numerical and categorical features into the embedding space. We employ column-wise tokenizers for the feature of every single column, respectively, following the setup in Zhang et al. ().

- •
For a numerical column, we use a simple linear projection to map the scalar into a $d$-dimensional vector.
$\mathbf{h}^{i}=\mathbf{W}^{i}x^{i},\;\mbox{where}\;\mathbf{W}^{i}\in{\mathbb{R
}}^{1\times d}$
(16)
- •
For a categorical column, we use a simple linear projection to map the one-hot encoding of $x^{i}$ into a $d$-dimensional vector.
$\mathbf{h}^{i}=\mathbf{W}^{i}x^{i},\;\mbox{where}\;\mathbf{W}^{i}\in{\mathbb{R
}}^{{\mathcal{C}}_{i}\times d}$
(17)

##### Transformer layers

After tokenization, we add column-wise positional encoding to each token embedding, and then we apply a]the zero mask to the predefined masked/target tokens. Furthermore, we append the [pad] token embedding at the beginning of the obtained data sequence. The proposed data will be further processed by a series of Transformer blocks.

We use ViT (Dosovitskiy, ) as the backbone of the Transformer layers, which consists of multiple Transformer blocks (Vaswani et al., ). Each Transformer block contains a multi-head self-attention mechanism and a feed-forward network. Specifically, we use a stack of six Transformer blocks with four attention heads.

##### Predictors.

Given the output token embeddings from the Transformer layers, i.e., $\mathbf{z}^{i}$, we further use additional predictors such that it is tailored for learning the conditional distribution $p(x^{i}|\mathbf{z}^{i})$

We apply a simple MLP predictor $f_{i}(\cdot)$ for each token. For each discrete column, $f_{i}(\cdot)$ is a 4-layer MLP with ReLU activation.

For each continuous column, the output token embedding $\mathbf{z}^{i}$ be further fed into the denoising neural network to predict the noise. We defer this part to Section C.2.

### C.2 Diffusion Model

In this section, we introduce the architecture of the diffusion model. In a nutshell, we use simple MLPs as our denoising neural network, which is similar to the design in Zhang et al. () and  (Kotelnikov et al., ), the only difference is our denoising network takes an additional input $\mathbf{z}^{i}$ as the conditional information.

##### Denoising neural network.

The denoising neural network takes three inputs: the noisy data $x^{i}_{t}=x^{i}+\sigma(t)\cdot\varepsilon$ (note that we let $\sigma(t)=t$), the current timestep $t$, and the conditional information $\mathbf{z}^{i}$. Following the practice in (Kotelnikov et al., ), the current timestep $t$ is first embedded with sinusoidal positional embedding, then further embedded with a 2-layer fully connected MLP with SiLU activation;

$$ $e_{pe}=\texttt{PE}(t),e_{t}=\texttt{MLP}_{t}(e_{pe})$ $$

where PE denotes the sinusoidal positional embedding (Vaswani et al., ).

Similarly, the conditional information $\mathbf{z}^{i}$ is embedded with a similar 2-layer MLP with SiLU activation.

$$ $e_{z}=\texttt{MLP}_{z}(\mathbf{z}^{i})$ $$

Now we have the embeddings of current timestep and conditional information, we add them together as the final condition embedding $e^{\ast}$.

$$ $e^{\ast}=e_{t}+e_{c}$ $$

For the noisy data $x^{i}_{t}$, we first project it with a single linear layer to align the dimension with the conditional embedding, then add it with $e^{\ast}$ and feed into a 4-layer MLP with SiLU activation for the final prediction of the denoised data.

$$ $\bm{\epsilon}_{\theta}(x^{i}_{t},t,\mathbf{z}^{i})=\texttt{MLP}(\texttt{Linear }(x_{t}^{i})+e^{\ast})$ $$

Finally, we minimize the MSE loss between the output of the denoising neural network and the added noise:

$$ $\min\|\bm{\epsilon}_{\theta}(x^{i}_{t},t,\mathbf{z}^{i})-\varepsilon\|_{2}^{2}$ $$

## Appendix D Detailed Experimental Setups

### D.1 Hardware Specification and Environment

We run our experiments on a single machine with Intel i9-14900K, Nvidia RTX 4090 GPU with 24 GB memory. The code is written in Python 3.10.14 and we use PyTorch 2.2.2 on CUDA 12.2 to train the model on the GPU.

### D.2 Datasets

The dataset used in this paper could be automatically downloaded using the script in the provided code.
We use $10$ tabular datasets from Kaggle(^4^44[https://www.kaggle.com](https://www.kaggle.com)) or UCI Machine Learning Repository(^5^55[https://archive.ics.uci.edu/datasets](https://archive.ics.uci.edu/datasets)): Adult(^6^66[https://archive.ics.uci.edu/dataset/2/adult](https://archive.ics.uci.edu/dataset/2/adult)), Default(^7^77[https://archive.ics.uci.edu/dataset/350/default+of+credit+card+clients](https://archive.ics.uci.edu/dataset/350/default+of+credit+card+clients)), Shoppers(^8^88[https://archive.ics.uci.edu/dataset/468/online+shoppers+purchasing+intention+dataset](https://archive.ics.uci.edu/dataset/468/online+shoppers+purchasing+intention+dataset)), Magic(^9^99[https://archive.ics.uci.edu/dataset/159/magic+gamma+telescope](https://archive.ics.uci.edu/dataset/159/magic+gamma+telescope)), Beijing(^10^1010[https://archive.ics.uci.edu/dataset/381/beijing+pm2+5+data](https://archive.ics.uci.edu/dataset/381/beijing+pm2+5+data)), and News(^11^1111[https://archive.ics.uci.edu/dataset/332/online+news+popularity](https://archive.ics.uci.edu/dataset/332/online+news+popularity)), California(^12^1212[https://www.kaggle.com/datasets/camnugent/california-housing-prices](https://www.kaggle.com/datasets/camnugent/california-housing-prices)), Letter(^13^1313[https://archive.ics.uci.edu/dataset/59/letter+recognition](https://archive.ics.uci.edu/dataset/59/letter+recognition)), Car(^14^1414[https://archive.ics.uci.edu/dataset/19/car+evaluation](https://archive.ics.uci.edu/dataset/19/car+evaluation)), and Nursery(^15^1515[https://archive.ics.uci.edu/dataset/76/nursery](https://archive.ics.uci.edu/dataset/76/nursery)), which contains varies number of numerical and categorical features. The statistics of the datasets are presented in Table 7.

**Table 7: Dataset statistics.**
| Dataset | # Rows | # Continuous | # Discrete | # Target | # Train | # Test | Task |
| --- | --- | --- | --- | --- | --- | --- | --- |
| California | $20,640$ | $9$ | - | $1$ | $18,390$ | $2,520$ | Classification |
| Letter | $20,000$ | $16$ | - | $1$ | $18,000$ | $2,000$ | Classification |
| Car | $1,728$ | - | $7$ | $1$ | $1,555$ | $173$ | Classification |
| Nursery | $12,960$ | - | $9$ | $1$ | $11,664$ | $1,296$ | Classification |
| Adult | $32,561$ | $6$ | $8$ | $1$ | $22,792$ | $16,281$ | Classification |
| Default | $30,000$ | $14$ | $10$ | $1$ | $27,000$ | $3,000$ | Classification |
| Shoppers | $12,330$ | $10$ | $7$ | $1$ | $11,098$ | $1,232$ | Classification |
| Magic | $19,021$ | $10$ | $1$ | $1$ | $17,118$ | $1,903$ | Classification |
| Beijing | $43,824$ | $7$ | $5$ | $1$ | $39,441$ | $4,383$ | Regression |
| News | $39,644$ | $46$ | $2$ | $1$ | $35,679$ | $3,965$ | Regression |

In Table 7, # Rows denote the number of rows (records) in the table. # Continuous and # Discrete denote the number of continuous features and discrete features, respectively. Note that there is an additional # Target column. The target columns are either continuous or discrete, depending on the task type. All datasets (except Adult) are split into training and testing sets with the ratio $9:1$ with a fixed random seed. As Adult has its official testing set, we directly use it as the testing set. For Machine Learning Efficiency (MLE) evaluation, the training set will be further split into training and validation split with the ratio $8:1$.

### D.3 TabDAR Implementation Details

##### Data Preprocessing.

We first fill the missing values with the columns’s average for numerical columns. For categorical columns, missing cells are treated as an additional category. Then numerical columns are transformed to follow a normal distribution by QuantileTransformer(^16^1616[https://scikit-learn.org/stable/modules/generated/sklearn.preprocessing.QuantileTransformer.html](https://scikit-learn.org/stable/modules/generated/sklearn.preprocessing.QuantileTransformer.html)) and categorical columns are encoded as integers by OrdinalEncoder(^17^1717[https://scikit-learn.org/stable/modules/generated/sklearn.preprocessing.OrdinalEncoder.html](https://scikit-learn.org/stable/modules/generated/sklearn.preprocessing.OrdinalEncoder.html)). Finally, we normalize the numerical features to have 0 mean and 0.5 variance, following (Karras et al., ).

##### Hyperparameters.

TabDAR uses a fixed set of hyperparameters for all datasets. Table 8 shows the hyperparameters. Our experiments show that TabDAR is robust to the choice of hyperparameters, saving the time of meticulous hyperparameter tuning for each dataset.

**Table 8: Default hyperparameter setting of TabDAR.**
| Type | Parameter | Value |
| --- | --- | --- |
| Training | optimizer | Adam |
| initial learning rate | 1e-3 |  |
| weight decay | 1e-6 |  |
| LR scheduler | ReduceLROnPlateau |  |
| training epochs | 5000 |  |
| batch size | 4096 |  |
| Transformers | #Transformer blocks | 6 |
| embedding dim | 32 |  |
| #heads | 4 |  |

### D.4 Missing Value Imputation

Following (Zhang et al., ), we use the Expected A Posteriori (EAP) estimator to impute the missing values. Specifically, denote a sample $\mathbf{x}$ with missing values as $\mathbf{x}=(\mathbf{x}_{\text{obs}},\mathbf{x}_{\text{mis}})$ where $\mathbf{x}_{\text{obs}}$ and $\mathbf{x}_{\text{mis}}$ are the observed and missing values, respectively. Our goal is to compute the expectation of the missing values conditioned on the observed values:

$$ $\mathbb{E}_{\mathbf{x}_{\text{mis}}\sim p(\mathbf{x}_{\text{mis}}|\mathbf{x}_{ \text{obs}})}[\mathbf{x}_{\text{mis}}]$ $$

To estimate the expectation, we sample $k$ times from the posterior distribution $p(\mathbf{x}_{\text{mis}}|\mathbf{x}_{\text{obs}})$ (with Algorithm 5) and obtain a set of samples $\{\hat{\mathbf{x}}^{i}_{\text{mis}}\}_{i=1}^{k}$.

For numerical features, the imputation $\hat{\mathbf{x}}_{\text{mis}}$ is set to the mean of the samples:

$$ $\hat{\mathbf{x}}_{\text{mis}}=\frac{1}{k}\sum_{i=1}^{k}\hat{\mathbf{x}}^{i}_{ \text{mis}}$ $$

For categorical features, we apply a majority vote on every column to approximate the expectation. Suppose the column has categories $\mathcal{C}=\{c_{1},c_{2},\cdots,c_{m}\}$, the imputation $\hat{\mathbf{x}}_{\text{mis}}$ is set to the most frequent category:

$$ $\hat{\mathbf{x}}_{\text{mis}}=\operatorname{arg\,max}_{c_{j}\in\mathcal{C}} \frac{1}{k}\sum_{i=1}^{k}\delta(\hat{\mathbf{x}}^{i}_{\text{mis}}=c_{j})$ $$

where $\delta(\cdot)$ is an indicator function that equals to $1$ if the input is true and 0 0 otherwise. The majority vote can be understood as a mean estimator that is more robust to outliers.

Finally, we concatenate the observed values with the imputed missing values to obtain the final imputed sample:

$$ $\mathbf{x}_{\text{imputed}}=(\mathbf{x}_{\text{obs}},\hat{\mathbf{x}}_{\text{ mis}})$ $$

In our experiments, we find that $k=10$ is a good trade-off between performance and efficiency.

### D.5 Baseline Implementations

##### Tab-MT and DP-TBART.

Tab-MT (Gulati & Roysdon, ) and DP-TBART (Castellon et al., ) are two recently proposed tabular data generation models based on MAE. To handle numerical features (with continuous distribution), Tab-MT quantizes the numerical features into 100 uniform bins, and DP-TBART quantizes the numerical features into 100 bins where each bin has the same nearest center determined by K-means. Additionally, DP-TBART employs DP-SGD (Abadi et al., ) to enhance the differential privacy performance. Since the focus of this paper is not on differential privacy, in our implementation, we use Adam (Kingma & Ba, ) optimizer.

##### Other Baselines.

The implementations of SMOTE (Chawla et al., ), CTGAN (Xu et al., ), TVAE (Xu et al., ), GOOGLE(^18^1818We find the result of GOOGLE is hard to reproduce due to memory issues, so we directly use the results in Zhang et al. ()) (Liu et al., ), GReaT (Borisov et al., ), CoDi (Lee et al., ), STaSy (Kim et al., ), TabDDPM (Kotelnikov et al., ), TabSyn (Zhang et al., ) follows the codebase of Zhang et al. ()(^19^1919[https://github.com/amazon-science/tabsyn/tree/main/baselines](https://github.com/amazon-science/tabsyn/tree/main/baselines)).

### D.6 Metrics

Most of the metrics (including Marginal, Joint, $\alpha$-Precision, $\beta$-Recall, C2ST, MLE, and DCR) used in this paper directly follow the setups in Zhang et al. (). Here is a reference:

- •
Marginal: Appendix E.3.1 in Zhang et al. ().
- •
Joint: Appendix E.3.2 in Zhang et al. ().
- •
$\alpha$-Precision and $\beta$-Recall: Appendix F.2 in Zhang et al. ().
- •
C2ST: Appendix F.3 in Zhang et al. ().
- •
MLE: Appendix E.4 in Zhang et al. ().
- •
DCR: Appendix F.6 in Zhang et al. ().

Below is a summary of how these metrics work.

#### D.6.1 Marginal Distribution

The Marginal metric evaluates if each column’s marginal distribution is faithfully recovered by the synthetic data. We use Kolmogorov-Sirnov Test for continuous data and Total Variation Distance for discrete data.

##### Kolmogorov-Sirnov Test (KST)

Given two (continuous) distributions $p_{r}(x)$ and $p_{s}(x)$ ($r$ denotes real and $s$ denotes synthetic), KST quantifies the distance between the two distributions using the upper bound of the discrepancy between two corresponding Cumulative Distribution Functions (CDFs):

$$ ${\rm KST}=\sup\limits_{x}|F_{r}(x)-F_{s}(x)|,$ (18) $$

where $F_{r}(x)$ and $F_{s}(x)$ are the CDFs of $p_{r}(x)$ and $p_{s}(x)$, respectively:

$$ $F(x)=\int_{-\infty}^{x}p(x){\rm d}x.$ (19) $$

##### Total Variation Distance (TVD)

TVD computes the frequency of each category value and expresses it as a probability. Then, the TVD score is the average difference between the probabilities of the categories:

$$ ${\rm TVD}=\frac{1}{2}\sum\limits_{\omega\in\Omega}|R(\omega)-S(\omega)|,$ (20) $$

where $\omega$ describes all possible categories in a column $\Omega$. $R(\cdot)$ and $S(\cdot)$ denotes the real and synthetic frequencies of these categories.

#### D.6.2 Joint Distribution

The Joint metric evaluates if the correlation of every two columns in the real data is captured by the synthetic data.

##### Pearson Correlation Coefficient

The Pearson correlation coefficient measures whether two continuous distributions are linearly correlated and is computed as:

$$ $\rho_{x,y}=\frac{{\rm Cov}(x,y)}{\sigma_{x}\sigma_{y}},$ (21) $$

where $x$ and $y$ are two continuous columns. Cov is the covariance, and $\sigma$ is the standard deviation.

Then, the performance of correlation estimation is measured by the average differences between the real data’s correlations and the synthetic data’s corrections:

$$ ${\text{Pearson Score}}=\frac{1}{2}\mathbb{E}_{x,y}|\rho^{R}(x,y)-\rho^{S}(x,y)|,$ (22) $$

where $\rho^{R}(x,y)$ and $\rho^{S}(x,y))$ denotes the Pearson correlation coefficient between column $x$ and column $y$ of the real data and synthetic data, respectively. As $\rho\in[-1,1]$, the average score is divided by $2$ to ensure that it falls in the range of $[0,1]$, then the smaller the score, the better the estimation.

##### Contingency similarity

For a pair of categorical columns $A$ and $B$, the contingency similarity score computes the difference between the contingency tables using the Total Variation Distance. The process is summarized by the formula below:

$$ $\text{Contingency Score}=\frac{1}{2}\sum\limits_{\alpha\in A}\sum\limits_{ \beta\in B}|R_{\alpha,\beta}-S_{\alpha,\beta}|,$ (23) $$

where $\alpha$ and $\beta$ describe all the possible categories in column $A$ and column $B$, respectively. $R_{\alpha,\beta}$ and $S_{\alpha,\beta}$ are the joint frequency of $\alpha$ and $\beta$ in the real data and synthetic data, respectively.

#### D.6.3 α 𝛼 \alpha italic_α -Precision and β 𝛽 \beta italic_β -Recall

$\alpha$-Precision and $\beta$-Recall are two sample-level metrics quantifying how faithful the synthetic data is proposed in Alaa et al. (). In general, $\alpha$-Precision evaluates the fidelity of synthetic data – whether each synthetic example comes from the real-data distribution, $\beta$-Recall evaluates the coverage of the synthetic data, e.g., whether the synthetic data can cover the entire distribution of the real data (In other words, whether a real data sample is close to the synthetic data).

#### D.6.4 Classifier-Two-Sample-Test (C2ST)

C2ST studies how difficult it is to distinguish real data from synthetic data, therefore evaluating whether synthetic data can recover real data distribution. The C2ST metric used in this paper is implemented by the SDMetrics(^20^2020[https://docs.sdv.dev/sdmetrics/metrics/metrics-in-beta/detection-single-table](https://docs.sdv.dev/sdmetrics/metrics/metrics-in-beta/detection-single-table)) package.

#### D.6.5 Machine Learning Efficiency (MLE)

In MLE, each dataset is first split into the real training and testing set. The generative models are learned on the real training set. After the models are learned, a synthetic set of equivalent size is sampled.

The performance of synthetic data on MLE tasks is evaluated based on the divergence of test scores using the real and synthetic training data. Therefore, we first train the machine learning model on the real training set, split into training and validation sets with a $8:1$ ratio. The classifier/regressor is trained on the training set, and the optimal hyperparameter setting is selected according to the performance on the validation set. After the optimal hyperparameter setting is obtained, the corresponding classifier/regressor is retrained on the training set and evaluated on the real testing set. We create 20 random splits for training and validation sets, and the performance reported is the mean of the AUC/RMSE score over the $20$ random trails. The performance of synthetic data is obtained in the same way.

#### D.6.6 Distance to Closest Record

We follow the ‘synthetic vs. holdout’ setting (^21^2121[https://www.clearbox.ai/blog/2022-06-07-synthetic-data-for-privacy- preservation-part-2](https://www.clearbox.ai/blog/2022-06-07-synthetic-data-for-privacy-%5C%5C%0Apreservation-part-2)). We initially divide the dataset into two equal parts: the first part served as the training set for training our generative model, while the second part was designated as the holdout set, which is not used for training. After completing model training, we sample a synthetic set of the same size as the training set (and the holdout set).

We then calculate the DCR scores for each sample in the synthetic set concerning both the training set and the holdout set. We further calculate the probability that a synthetic sample is closer to the training set (rather than the holdout set). When this probability is close to 50% (i.e., 0.5), it indicates that the distribution of distances between synthetic and training instances is very similar (or at least not systematically smaller) than the distribution of distances between synthetic and holdout instances, which is a positive indicator in terms of privacy risk.

## Appendix E Additional Experimental Results

In this section, we provide a more detailed empirical comparison between the proposed TabDAR and other baseline methods.

### E.1 Detailed Results on the Fidelity Metrics

Note that in Table 1, we only present the average performance of each method on the six fidelity metrics across the ten datasets. In this section, we present a detailed performance comparison of each individual dataset:

- •
Marginal Distribution: Table 9
- •
Joint Correlation: Table 10
- •
$\alpha$-Precision: Table 11
- •
$\beta$-Recall: Table 12
- •
Classifier-Two-Sample-Test: Table 13
- •
Jensen-Shannon Divergence: Table 14

**Table 9: Performance comparison on the Marginal Distribution Density metric. Numbers represent the error rate in $\%$, the lower the better.**
|  | Method | Continuous only |  | Discrete only |  | Heterogeous |  |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
|  | California | Letter |  | Car | Nursery |  | Adult | Beijing | Default | Magic | News | Shoppers |  |  |
| interpolation |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
|  | SMOTE | $0.99$ | $0.97$ |  | $1.00$ | $0.57$ |  | $1.59$ | $1.78$ | $1.49$ | $1.07$ | $5.28$ | $2.48$ |  |
| VAE-based |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
|  | TVAE | $5.37$ | $16.70$ |  | $24.12$ | $9.81$ |  | $24.32$ | $25.13$ | $9.94$ | $4.39$ | $18.48$ | $23.93$ |  |
| GAN-based |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
|  | CTGAN | $12.84$ | $18.79$ |  | $16.46$ | $12.33$ |  | $19.32$ | $21.98$ | $18.25$ | $5.69$ | $13.90$ | $25.71$ |  |
| LLM-based |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
|  | GReaT | $14.93$ | $4.88$ |  | $2.22$ | $5.08$ |  | $12.12$ | $8.25$ | $19.94$ | $16.16$ | $-$ | $14.51$ |  |
| Diffusion-based |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
|  | STaSy | $10.82$ | $11.93$ |  | $24.38$ | $10.93$ |  | $10.41$ | $6.38$ | $11.34$ | $13.02$ | $8.54$ | $16.14$ |  |
|  | CoDi | $18.98$ | $22.62$ |  | $1.53$ | $0.65$ |  | $24.84$ | $12.54$ | $16.54$ | $11.64$ | $28.13$ | $36.48$ |  |
|  | TabDDPM | $57.34$ | $61.43$ |  | $1.53$ | $0.65$ |  | $1.32$ | $1.20$ | $7.59$ | $1.09$ | $-$ | $2.86$ |  |
|  | TabSyn | $1.00$ | $2.53$ |  | $2.48$ | $1.04$ |  | $2.75$ | $2.43$ | $0.95$ | $0.79$ | $1.77$ | $1.52$ |  |
| Autoregressive |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
|  | DP-TBART | $3.30$ | $4.46$ |  | $1.98$ | $0.53$ |  | $1.17$ | $2.68$ | $5.03$ | $3.90$ | $6.28$ | $3.05$ |  |
|  | Tab-MT | $5.87$ | $3.29$ |  | $0.96$ | $0.70$ |  | $17.20$ | $25.10$ | $25.17$ | $21.88$ | $46.54$ | $2.20$ |  |
|  | TabDAR | $0.99$ | $1.79$ |  | $1.31$ | $0.73$ |  | $0.59$ | $0.80$ | $1.74$ | $0.80$ | $2.03$ | $1.32$ |  |

**Table 10: Performance comparison on the Joint Column Correlation metric. Numbers represent the error rate in $\%$, the low the better.**
|  | Method | Continuous only |  | Discrete only |  | Heterogeous |  |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
|  | California | Letter |  | Car | Nursery |  | Adult | Beijing | Default | Magic | News | Shoppers |  |  |
| interpolation |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
|  | SMOTE | $2.70$ | $1.19$ |  | $3.16$ | $1.21$ |  | $3.56$ | $1.53$ | $6.93$ | $2.84$ | $2.87$ | $3.53$ |  |
| VAE-based |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
|  | TVAE | $5.85$ | $5.28$ |  | $38.66$ | $18.34$ |  | $36.65$ | $31.12$ | $19.37$ | $4.46$ | $6.45$ | $20.12$ |  |
| GAN-based |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
|  | CTGAN | $14.49$ | $11.40$ |  | $25.63$ | $18.14$ |  | $27.35$ | $27.08$ | $30.52$ | $5.04$ | $5.22$ | $24.24$ |  |
| LLM-based |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
|  | GReaT | $9.66$ | $3.46$ |  | $4.72$ | $8.38$ |  | $17.59$ | $59.60$ | $70.02$ | $59.96$ | $-$ | $45.16$ |  |
| Diffusion-based |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
|  | STaSy | $3.59$ | $5.34$ |  | $36.40$ | $15.02$ |  | $13.50$ | $8.71$ | $10.65$ | $5.58$ | $3.06$ | $15.29$ |  |
|  | CoDi | $6.89$ | $5.25$ |  | $3.52$ | $1.31$ |  | $22.72$ | $6.42$ | $67.88$ | $6.93$ | $10.81$ | $20.18$ |  |
|  | TabDDPM | $19.83$ | $22.35$ |  | $3.52$ | $1.31$ |  | $2.50$ | $3.31$ | $11.55$ | $0.67$ | $-$ | $6.23$ |  |
|  | TabSyn | $0.78$ | $1.78$ |  | $4.28$ | $1.85$ |  | $4.64$ | $4.16$ | $3.30$ | $0.91$ | $1.43$ | $2.18$ |  |
| Autoregressive |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
|  | DP-TBART | $2.52$ | $1.94$ |  | $3.54$ | $1.19$ |  | $2.50$ | $2.55$ | $6.70$ | $1.73$ | $1.60$ | $2.83$ |  |
|  | Tab-MT | $5.87$ | $3.29$ |  | $0.96$ | $0.70$ |  | $17.20$ | $25.10$ | $25.17$ | $21.88$ | $46.54$ | $2.20$ |  |
|  | TabDAR | $0.61$ | $1.45$ |  | $2.99$ | $1.36$ |  | $1.36$ | $2.27$ | $2.83$ | $1.86$ | $1.50$ | $1.89$ |  |

**Table 11: Performance comparison on the $\alpha$-Precision metric. Numbers represent $1-\alpha\text{-}\textbf{Precision}$. The lower the better. Note that the numbers in Table 1 are in $\%$ while numbers in this table are in raw scale.**
|  | Method | Continuous only |  | Discrete only |  | Heterogeous |  |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
|  | California | Letter |  | Car | Nursery |  | Adult | Beijing | Default | Magic | News | Shoppers |  |  |
| interpolation |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
|  | SMOTE | $0.0173$ | $0.0222$ |  | $0.0103$ | $0.0040$ |  | $0.0729$ | $0.0118$ | $0.0228$ | $0.0186$ | $0.1256$ | $0.0725$ |  |
| VAE-based |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
|  | TVAE | $0.0191$ | $0.0937$ |  | $0.2322$ | $0.0972$ |  | $0.4124$ | $0.1100$ | $0.1610$ | $0.0338$ | $0.1530$ | $0.5655$ |  |
| GAN-based |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
|  | CTGAN | $0.2933$ | $0.0522$ |  | $0.1023$ | $0.0677$ |  | $0.2528$ | $0.0723$ | $0.3595$ | $0.1106$ | $0.0177$ | $0.1292$ |  |
| LLM-based |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
|  | GReaT | $0.1665$ | $0.0891$ |  | $0.0262$ | $0.0836$ |  | $0.4421$ | $0.0168$ | $0.1410$ | $0.1454$ | $1.0000$ | $0.2112$ |  |
| Diffusion-based |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
|  | STaSy | $0.8385$ | $0.0158$ |  | $0.3347$ | $0.1385$ |  | $0.2250$ | $0.0350$ | $0.1320$ | $0.1853$ | $0.0809$ | $0.1656$ |  |
|  | CoDi | $0.1333$ | $0.0963$ |  | $0.0323$ | $0.0037$ |  | $0.1805$ | $0.0471$ | $0.1722$ | $0.1434$ | $0.1041$ | $0.0825$ |  |
|  | TabDDPM | $0.8385$ | $1.0000$ |  | $0.0323$ | $0.0037$ |  | $0.0444$ | $0.0130$ | $0.0924$ | $0.0146$ | $1.0000$ | $0.0807$ |  |
|  | TabSyn | $0.0062$ | $0.0998$ |  | $0.0206$ | $0.0056$ |  | $0.0239$ | $0.0049$ | $0.0123$ | $0.0053$ | $0.0457$ | $0.0279$ |  |
| Autoregressive |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
|  | DP-TBART | $0.0256$ | $0.0427$ |  | $0.0093$ | $0.0054$ |  | $0.0054$ | $0.0049$ | $0.0727$ | $0.0160$ | $0.0149$ | $0.0145$ |  |
|  | Tab-MT | $0.0239$ | $0.0415$ |  | $0.0140$ | $0.0146$ |  | $0.1776$ | $0.4839$ | $0.0581$ | $0.5507$ | $0.9369$ | $0.0158$ |  |
|  | TabDAR | $0.0080$ | $0.0383$ |  | $0.0190$ | $0.0052$ |  | $0.0029$ | $0.0070$ | $0.0133$ | $0.0147$ | $0.0218$ | $0.0025$ |  |

**Table 12: Performance comparison on the $\beta$-Recall metric. Numbers represent $1-\beta\text{-}\textbf{Recall}$. The lower the better. Note that the numbers in Table 1 are in $\%$ while numbers in this table are in raw scale.**
|  | Method | Continuous only |  | Discrete only |  | Heterogeous |  |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
|  | California | Letter |  | Car | Nursery |  | Adult | Beijing | Default | Magic | News | Shoppers |  |  |
| Interpolation |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
|  | SMOTE | $0.2157$ | $0.1338$ |  | $0.0045$ | $0.0012$ |  | $0.2325$ | $0.2080$ | $0.2390$ | $0.1934$ | $0.2011$ | $0.2386$ |  |
| VAE-based |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
|  | TVAE | $0.6512$ | $0.8324$ |  | $0.4369$ | $0.2625$ |  | $0.8969$ | $0.9392$ | $0.7078$ | $0.6243$ | $0.7413$ | $0.7672$ |  |
| GAN-based |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
|  | CTGAN | $0.8412$ | $0.9903$ |  | $0.9879$ | $0.0530$ |  | $0.8246$ | $0.6230$ | $0.8889$ | $0.8472$ | $0.7732$ | $0.7441$ |  |
| LLM-based |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
|  | GReaT | $0.5515$ | $0.6643$ |  | $0.0034$ | $0.0024$ |  | $0.5088$ | $0.5666$ | $0.5796$ | $0.6509$ | $1.0000$ | $0.5510$ |  |
| Diffusion-based |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
|  | STaSy | $0.9288$ | $0.7332$ |  | $0.1075$ | $0.0029$ |  | $0.6812$ | $0.5061$ | $0.6421$ | $0.5686$ | $0.6033$ | $0.7174$ |  |
|  | CoDi | $0.5998$ | $0.4551$ |  | $0.0040$ | $0.0009$ |  | $0.9032$ | $0.4472$ | $0.7811$ | $0.5139$ | $0.6505$ | $0.8187$ |  |
|  | TabDDPM | $0.9288$ | $1.0000$ |  | $0.0040$ | $0.0009$ |  | $0.5152$ | $0.4335$ | $0.6150$ | $0.5206$ | $1.0000$ | $0.4492$ |  |
|  | TabSyn | $0.5706$ | $0.7486$ |  | $0.0031$ | $0.0007$ |  | $0.5484$ | $0.4847$ | $0.5365$ | $0.5146$ | $0.5602$ | $0.4399$ |  |
| Autoregressive |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
|  | DP-TBART | $0.6138$ | $0.8859$ |  | $0.0038$ | $0.0010$ |  | $0.5033$ | $0.4536$ | $0.5562$ | $0.6044$ | $0.6532$ | $0.5285$ |  |
|  | Tab-MT | $0.6272$ | $0.8385$ |  | $0.0067$ | $0.0008$ |  | $0.9681$ | $0.4930$ | $0.9179$ | $0.9844$ | $1.0000$ | $0.5105$ |  |
|  | TabDAR | $0.5038$ | $0.4562$ |  | $0.0040$ | $0.0008$ |  | $0.4928$ | $0.4249$ | $0.5162$ | $0.3887$ | $0.5277$ | $0.4078$ |  |

**Table 13: Performance comparison on the C2ST metric. Numbers represent $100\times(1-\textbf{C2ST})$ (i.e. in base of $10^{-2}$). The lower the better.**
|  | Method | Continuous only |  | Discrete only |  | Heterogeous |  |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
|  | California | Letter |  | Car | Nursery |  | Adult | Beijing | Default | Magic | News | Shoppers |  |  |
| interpolation |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
|  | SMOTE | $0.61$ | $0.00$ |  | $0.00$ | $0.00$ |  | $3.05$ | $0.44$ | $7.69$ | $1.93$ | $6.49$ | $9.80$ |  |
| VAE-based |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
|  | TVAE | $12.48$ | $22.27$ |  | $70.30$ | $52.73$ |  | $72.39$ | $45.53$ | $41.65$ | $12.07$ | $60.27$ | $70.04$ |  |
| GAN-based |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
|  | CTGAN | $50.11$ | $82.40$ |  | $59.59$ | $48.63$ |  | $36.79$ | $56.82$ | $64.60$ | $14.15$ | $27.64$ | $48.86$ |  |
| LLM-based |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
|  | GReaT | $28.38$ | $16.14$ |  | $6.05$ | $18.41$ |  | $46.24$ | $31.07$ | $52.90$ | $56.74$ | $-$ | $57.15$ |  |
| Diffusion-based |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
|  | STaSy | $54.61$ | $47.75$ |  | $78.52$ | $40.43$ |  | $54.02$ | $23.48$ | $49.29$ | $53.97$ | $50.21$ | $62.20$ |  |
|  | CoDi | $47.48$ | $42.85$ |  | $0.00$ | $0.22$ |  | $80.02$ | $15.70$ | $52.37$ | $27.70$ | $91.62$ | $81.84$ |  |
|  | TabDDPM | $88.01$ | $96.86$ |  | $0.00$ | $0.22$ |  | $3.95$ | $3.29$ | $11.75$ | $0.95$ | $-$ | $16.37$ |  |
|  | TabSyn | $0.71$ | $3.18$ |  | $2.90$ | $2.88$ |  | $8.05$ | $3.60$ | $1.33$ | $0.08$ | $1.77$ | $3.05$ |  |
| Autoregressive |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
|  | DP-TBART | $3.89$ | $11.07$ |  | $0.07$ | $0.00$ |  | $0.81$ | $3.63$ | $7.41$ | $5.01$ | $12.96$ | $8.73$ |  |
|  | Tab-MT | $8.71$ | $9.34$ |  | $0.37$ | $0.00$ |  | $99.86$ | $99.97$ | $91.62$ | $74.47$ | $100.00$ | $1.47$ |  |
|  | TabDAR | $1.27$ | $0.55$ |  | $0.00$ | $0.74$ |  | $0.47$ | $1.55$ | $3.43$ | $0.1100$ | $3.12$ | $3.74$ |  |

**Table 14: Performance comparison on the Jensen-Shannon Divergence (JSD) metric. The lower the better. Note that the numbers in Table 1 are in $\%$ while numbers in this table are in raw scale.**
|  | Method | Continuous only |  | Discrete only |  | Heterogeous |  |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
|  | California | Letter |  | Car | Nursery |  | Adult | Beijing | Default | Magic | News | Shoppers |  |  |
| interpolation |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
|  | SMOTE | $0.0006$ | $0.0006$ |  | $0.0013$ | $0.0008$ |  | $0.0008$ | $0.0003$ | $0.0006$ | $0.0008$ | $0.0040$ | $0.0015$ |  |
| VAE-based |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
|  | TVAE | $0.0041$ | $0.0072$ |  | $0.0323$ | $0.0132$ |  | $0.0078$ | $0.0077$ | $0.0052$ | $0.0036$ | $0.0109$ | $0.0107$ |  |
| GAN-based |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
|  | CTGAN | $0.0182$ | $0.0135$ |  | $0.0220$ | $0.0165$ |  | $0.0038$ | $0.0033$ | $0.0114$ | $0.0056$ | $0.0078$ | $0.0066$ |  |
| LLM-based |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
|  | GReaT | $0.0111$ | $0.0022$ |  | $0.0030$ | $0.0068$ |  | $0.0182$ | $0.0023$ | $0.0076$ | $0.0107$ | $-$ | $0.0056$ |  |
| Diffusion-based |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
|  | STaSy | $0.0380$ | $0.0057$ |  | $0.0326$ | $0.0146$ |  | $0.0041$ | $0.0030$ | $0.0055$ | $0.0107$ | $0.0070$ | $0.0086$ |  |
|  | CoDi | $0.0152$ | $0.0085$ |  | $0.0020$ | $0.0009$ |  | $0.0073$ | $0.0017$ | $0.0067$ | $0.0142$ | $0.0092$ | $0.0103$ |  |
|  | TabDDPM | $0.0380$ | $0.0382$ |  | $0.0020$ | $0.0009$ |  | $0.0004$ | $0.0092$ | $0.0008$ | $0.0013$ | $-$ | $0.0019$ |  |
|  | TabSyn | $0.0006$ | $0.0017$ |  | $0.0033$ | $0.0014$ |  | $0.0004$ | $0.0012$ | $0.0003$ | $0.0007$ | $0.0016$ | $0.0007$ |  |
| Autoregressive |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
|  | DP-TBART | $0.0023$ | $0.0010$ |  | $0.0023$ | $0.0008$ |  | $0.0004$ | $0.0006$ | $0.0025$ | $0.0018$ | $0.0036$ | $0.0008$ |  |
|  | Tab-MT | $0.0038$ | $0.0019$ |  | $0.0013$ | $0.0009$ |  | $0.0026$ | $0.0024$ | $0.0034$ | $0.0211$ | $0.0221$ | $0.0007$ |  |
|  | TabDAR | $0.0004$ | $0.0010$ |  | $0.0018$ | $0.0010$ |  | $0.0003$ | $0.0004$ | $0.0007$ | $0.0011$ | $0.0015$ | $0.0008$ |  |

### E.2 Training / Sampling Time

In Table 15, we compare the training and sampling time of TabDAR with other methods on the Adult dataset.

**Table 15: Comparison of training and sampling time of different methods, on Adult dataset.**
| Method | Training | Sampling |
| --- | --- | --- |
| CTGAN | 1029.8s | 0.8621s |
| TVAE | 352.6s | 0.5118s |
| GReaT | 2h 27min | 2min 19s |
| STaSy | 2283s | 8.941s |
| CoDi | 2h 56min | 4.616s |
| TabDDPM | 1031s | 28.92s |
| TabSyn | 2422s | 11.84s |
| DP-TBART | 2355.8s | 2.11s |
| Tab-MT | 2352.23s | 2.18s |
| TabDAR | 2701.5s | 21.66s |

### E.3 Additional Visualizations

We present the 2D visualizations (Figure 7 and Figure 8) of the synthetic data generated by all baseline methods in Figure 10 and Figure 11, respectively. We also present the heat maps of all methods on the four datasets (Figure 9) in Figure 12, Figure 13, Figure 14, and Figure 15, respectively.

Figure: (a) CTGAN
Refer to caption: x10.png

Figure: (a) CTGAN
Refer to caption: x21.png

Figure: Figure 12: Heat map of synthetic data of Letter dataset.
Refer to caption: x33.png

Figure: Figure 13: Heat map of synthetic data of Adult dataset.
Refer to caption: x34.png

Figure: Figure 14: Heat map of synthetic data of Default dataset.
Refer to caption: x35.png

Figure: Figure 15: Heat map of synthetic data of Magic dataset.
Refer to caption: x36.png

## Peer Reviews

confidence: 4
contribution: 3
initial_rating: 6
presentation: 2
questions: - For continuous columns, why use a diffusion loss?  You could also use z to parameterize any simple continuous distribution, e.g. a Gaussian.  Given that the diffusion process can be computationally expensive, do you think that it is a necessary component of the method?  I understand that you do an ablation in which you compare against discretizing the continuous variable, but this also seems like an unnatural approach compared to simply passing it through a continuous distribution. 
- In your embedding/masking, how do you distinguish between a continuous value being equal to 0 and you masking out that value?  To me, it seems that both would lead to the same result after embedding the data.  Is there a particular reason you chose to use zero for missing values instead of an embedding for a [mask] token, as typical in masked language models? 
- Details on the diffusion aspect are generally missing.  Is the denoising network the same for different continuous columns or is the denoising network column-specific?  How many steps of diffusion are generally used?  
- Table 2 — since some of these values are quite close, could you provide some standard deviations?  Also, if there are ties with your method in the table, you should probably also bold these for consistency and fairness to other approaches.  
- Figure 6 — It would be interesting to see standard deviations for these bars.  I wonder how sensitive the results are to different initializations / different MCAR masks.  The authors mention that they are “significantly outperforming” other methods, but I think standard deviations are needed to justify this somewhat bold claim.  It would also be interesting to see how the percent missing from the data affects the results (currently it is fixed at 30%)? 
- For conditional sampling (Algorithm 5), is there a reason why you need to generate step-by-step?  During training, it seems the model learns to predict all masked columns from unmasked ones in one step during training.  Do you get better performance by generating step-by-step at test time?
soundness: 3
strengths: - This paper tackles an interesting and important problem -- how to learn the generative distribution of tabular data.  Tabular data is indeed challenging because of the potentially unstructured and diverse nature of the columns at hand.  As the authors mention, there has been a lot of recent interest in developing deep generative models for this domain.
- The authors' proposed solution is an interesting one.  They provide comparisons in the experiments against several other methods in the literature and generally show superior performance on the statistical fidelity metrics (Table 1).  
- I appreciate the extensive set of experiments, such as machine learning efficiency, the various visualizations, and the ablation studies. These help give a holistic view of the relative performance of their method compared to the literature.  
- The code is available and generally looks well-organized and a useful resource for the community.
summary: The authors develop a generative modeling method for tabular data.  Their method learns a conditional generative model that takes masked input and predicts the missing columns.  At inference time, they use this model autoregressively to generate data one column at a time -- for both unconditional data generation and missing values imputation. The model comprises of a linear embedder, a transformer encoder, and either a diffusion decoder or categorical classifier -- depending on whether the particular column is continuous or discrete in nature.  The authors perform experiments on various tabular datasets and use various metrics to compare the samples generated by their method against those generated by other methods in the literature.  They also perform ablation studies to understand the relative importance of various components of their approach.
weaknesses: - Nitpick: The authors sometimes use overly promotional language, e.g. line 71 — “through two ingenious design features”, line 84 —  “TABDAR offers several unparalleled advantages” — I recommend against the use of overly promotional words such as “ingenious”, “unparalleled advantages” in the paper.
 - Nitpick: Eq. (5) — Define $x^{<i}$.  For a reader familiar with autoregressive modeling notation, it is obvious what this means, but for others it may not be.  E.g. it would be good to clarify what this means for $i = 1$.
- One point that was confusing to me at first was that the model is not trained as an autoregressive model -- it is trained as a masked language model.  If I understand Eq. (8) / Algorithm 3 correctly, you are only masking out a particular set of columns at each training step. 
 Thus, you are not really learning an autoregressive distribution, this is more like a conditional distribution like masked language modeling in which you predict p(masked | unmasked).  I understand that you are using the learned conditional distribution in an autoregressive way at test time / generation time -- it would be good to clarify this point in the text.  
- There are also some questions about the method I have, listed below.  I am happy to give a positive score for this paper, but it is contingent on the questions below being addressed.

---

confidence: 4
contribution: 2
initial_rating: 5
presentation: 3
questions: See previous section.
soundness: 3
strengths: - Good results, and extensive experimental results.
summary: The authors propose a Diffusion-nested Autoregressive model (TABDAR) that can handle both discrete and continuous tabular data. Continuous features are modeled by nested diffusion models, and based on the assumption that table columns are permutation invariant, bi-directional attention is utilized to support generation of the columns in any order. TABDAR is compared to several recent methods for hetrogeneous table generation, and is shown to perform comparably or better in terms of statistical fidelity (Table 1), generation efficiency for machine learning tasks (Table 2), and Distance to Closest Record (DCR) scores (Table 3, Figure 5). Ablations suggest that both the use of diffusion losses and random sampling of order improve performance (Table 4), and that the results are quite stable wrt depth and embedding dimension (Table 5 and 6). Imputation results are also competitive (Figure 6).
weaknesses: - The TabSyn methods results stated in the paper are different (worse) that those reported in the TabSyn paper. These differences need to be fully explained and accounted for before the paper can be considered for publication.
- In addition, since TabSyn and TABDAR perform similarly and address the same problem, the advantages and differences between the two approaches should be discussed. 
- As a method that simply integrates diffusion modeling into bidirectional transformers to handle continuous tabular entries, the novelty of the paper seems on the lower side.
- The use of bidirectional attention is based on the assumption that columns are permutation invariant, while in practice, one expects that columns will sometimes have dependencies, and these would always be based on reading order (e.g. for English, left to right). Also, bi-directional attention is generally not equivalent to arbitrarily ordered causal attention, as suggested in Figure 2, due to the use of position embeddings. Based on these observations, It's not clear that bi-directional attention necessarily the best approach.

---

confidence: 3
contribution: 2
initial_rating: 3
presentation: 2
questions: See above.
soundness: 2
strengths: The authors present state-of-the-art empirical results (by some margin). The proposed method is based on interesting observations and to the best of my knowledge, the way they combine multi-modal signals is novel.
summary: This work proposes to combine autoregressive sequence modeling for discrete (token) generation and diffusion for continuous signal.
weaknesses: - The authors claim that columns are permutation invariant, which I tend to agree with. However, combining this observation with autoregressive generation is impossible. From the manuscript, I couldn't figure out if the method combines lower triangular mask and permutations of the tokens. Can the authors clarify?
- Superscript and subscripts - it seems like the authors used mixed notations in different sections (I think?). E.g., in equation (1), items in a sequence are subscript and then $<i$ is in superscript; later, the authors use the subscript for diffusion time and superscript for items (section 3.1, equation 2, equation 3, equation 5, and more). It feels like a mess. Please let me know if I didn't understand correctly.
- The authors claim that it is straightforward to do conditional sampling using their method, but I find it hard to understand how it is actually done. I.e., condition on continuous representation in discrete phase and vice versa.
- Learning a permutation invariant representation is well known in transformers, and to the best of my knowledge, the classical way to do it is by removing the positional embeddings. Can the authors explain why they chose a different path (which seems less effective)?
- Equation 6 - is it correct?
- Results - I am not familiar with the benchmarks used, but I am not sure what I can learn about the proposed method compared to prior works when the improvements are so small (as it seems that the tasks are roughly solved).

---

confidence: 3
contribution: 3
initial_rating: 5
presentation: 3
questions: A.	I think the data generation time will take long because of the autoregressive nature of the model. I think it would be great if authors can compare the generation times with other baseline methods.

B.	In line 85, the authors claim they use two types of generative models: one is diffusion, and another one is categorical prediction. But categorical prediction is not a generative model. 

C.	Authors should mention they use one-hot-encoding for preprocessing categorical data. I noticed they mentioned it in Appendix, but they should be mentioned in main paper to avoid confusion.

D.	In Figure 3, the role of diffusion model is not emphasized. 

E.	Are multiple diffusion models used for each of the continuous variable? If it is true, for each 

F.	In Tables 5 & 6, are the values in the tables averaged results over the two datasets? 

G.	Is it safe to say the model is robust to the hyperparameters? The results are based only on two datasets. I think authors need to work on more datasets for validating this claim. 

H.	Are the P(P(x|z))s in Eq. (6) and (7) typos?
soundness: 3
strengths: Strong experimental results.
summary: This paper suggests a new tabular data synthesizer by combining bi-directional transformer and diffusion model.
weaknesses: Please see the questions below.

Meta-review: The paper proposes a generative model for tabular data called Diffusion-nested Autoregressive model (TabDAR), which combines masked bi-directional attention transformers with diffusion models to handle mixed numerical and categorical data and supports arbitrary generation orders. The reviewers acknowledge that the paper tackles an important problem, proposes an interesting solution, and has extensive experiments. However, reviewers raised concerns regarding presentation issues, such as inconsistent notations and limited novelty compared to prior methods like TabSyn, which also exhibit similar empirical performance. Some reviewers also questioned the empirical discrepancies between reported baseline results and prior literature and found the experimental gains marginal on certain tasks, suggesting that benchmarks may not sufficiently differentiate methods. In the rebuttal, the authors addressed many technical and presentation issues. However, some conceptual critiques, such as novelty concerns, remain partially addressed. While the paper presents extensive experimental results, the marginal empirical gains, conceptual weaknesses, and mixed reviews on novelty mean that I recommend the paper be rejected.

Decision: Reject