Diffusion-Nested Auto-Regressive Synthesis of Heterogeneous Tabular Data

Authors: Hengrui Zhang, Liancheng Fang, Qitian Wu, Philip S. Yu
Primary area: generative models
Keywords: Tabular data synthesis, autoregressive models, diffusion models

Abstract:
Autoregressive models are predominant in natural language generation, while their application in tabular data remains underexplored. We posit that this can be attributed to two factors: 1) tabular data contains heterogeneous data type, while the autoregressive model is primarily designed to model discrete-valued data; 2) tabular data is column permutation-invariant, requiring a generation model to generate columns in arbitrary order. This paper proposes a Diffusion-nested Autoregressive model (TabDAR) to address these issues. To enable autoregressive methods for continuous columns, TabDAR employs a diffusion model to parameterize the conditional distribution of continuous features. To ensure arbitrary generation order, TabDAR resorts to masked transformers with bi-directional attention, which simulate various permutations of column order,  hence enabling it to learn the conditional distribution of a target column given an arbitrary combination of other columns. These designs enable TabDAR to not only freely handle heterogeneous tabular data but also support convenient and flexible unconditional/conditional sampling. We conduct extensive experiments on ten datasets with distinct properties, and the proposed TabDAR outperforms previous state-of-the-art methods by 18\% to 45\% on eight metrics across three distinct aspects.

Decision: reject
Reviewer scores: 6, 5, 3, 5

Reviews and discussion:
<thread>
creation date:
1730572973195

thread_type:
official_review

<thread_object>
creator: reviewer_dbtr

<content>
summary:
This paper suggests a new tabular data synthesizer by combining bi-directional transformer and diffusion model.

strengths:
Strong experimental results.

weaknesses:
Please see the questions below.

questions:
A.	I think the data generation time will take long because of the autoregressive nature of the model. I think it would be great if authors can compare the generation times with other baseline methods.

B.	In line 85, the authors claim they use two types of generative models: one is diffusion, and another one is categorical prediction. But categorical prediction is not a generative model. 

C.	Authors should mention they use one-hot-encoding for preprocessing categorical data. I noticed they mentioned it in Appendix, but they should be mentioned in main paper to avoid confusion.

D.	In Figure 3, the role of diffusion model is not emphasized. 

E.	Are multiple diffusion models used for each of the continuous variable? If it is true, for each 

F.	In Tables 5 & 6, are the values in the tables averaged results over the two datasets? 

G.	Is it safe to say the model is robust to the hyperparameters? The results are based only on two datasets. I think authors need to work on more datasets for validating this claim. 

H.	Are the P(P(x|z))s in Eq. (6) and (7) typos?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730657943131

thread_type:
official_review

<thread_object>
creator: reviewer_au72

<content>
summary:
This work proposes to combine autoregressive sequence modeling for discrete (token) generation and diffusion for continuous signal.

strengths:
The authors present state-of-the-art empirical results (by some margin). The proposed method is based on interesting observations and to the best of my knowledge, the way they combine multi-modal signals is novel.

weaknesses:
- The authors claim that columns are permutation invariant, which I tend to agree with. However, combining this observation with autoregressive generation is impossible. From the manuscript, I couldn't figure out if the method combines lower triangular mask and permutations of the tokens. Can the authors clarify?
- Superscript and subscripts - it seems like the authors used mixed notations in different sections (I think?). E.g., in equation (1), items in a sequence are subscript and then $<i$ is in superscript; later, the authors use the subscript for diffusion time and superscript for items (section 3.1, equation 2, equation 3, equation 5, and more). It feels like a mess. Please let me know if I didn't understand correctly.
- The authors claim that it is straightforward to do conditional sampling using their method, but I find it hard to understand how it is actually done. I.e., condition on continuous representation in discrete phase and vice versa.
- Learning a permutation invariant representation is well known in transformers, and to the best of my knowledge, the classical way to do it is by removing the positional embeddings. Can the authors explain why they chose a different path (which seems less effective)?
- Equation 6 - is it correct?
- Results - I am not familiar with the benchmarks used, but I am not sure what I can learn about the proposed method compared to prior works when the improvements are so small (as it seems that the tasks are roughly solved).

questions:
See above.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730676536117

thread_type:
official_review

<thread_object>
creator: reviewer_hueq

<content>
summary:
The authors propose a Diffusion-nested Autoregressive model (TABDAR) that can handle both discrete and continuous tabular data. Continuous features are modeled by nested diffusion models, and based on the assumption that table columns are permutation invariant, bi-directional attention is utilized to support generation of the columns in any order. TABDAR is compared to several recent methods for hetrogeneous table generation, and is shown to perform comparably or better in terms of statistical fidelity (Table 1), generation efficiency for machine learning tasks (Table 2), and Distance to Closest Record (DCR) scores (Table 3, Figure 5). Ablations suggest that both the use of diffusion losses and random sampling of order improve performance (Table 4), and that the results are quite stable wrt depth and embedding dimension (Table 5 and 6). Imputation results are also competitive (Figure 6).

strengths:
- Good results, and extensive experimental results.

weaknesses:
- The TabSyn methods results stated in the paper are different (worse) that those reported in the TabSyn paper. These differences need to be fully explained and accounted for before the paper can be considered for publication.
- In addition, since TabSyn and TABDAR perform similarly and address the same problem, the advantages and differences between the two approaches should be discussed. 
- As a method that simply integrates diffusion modeling into bidirectional transformers to handle continuous tabular entries, the novelty of the paper seems on the lower side.
- The use of bidirectional attention is based on the assumption that columns are permutation invariant, while in practice, one expects that columns will sometimes have dependencies, and these would always be based on reading order (e.g. for English, left to right). Also, bi-directional attention is generally not equivalent to arbitrarily ordered causal attention, as suggested in Figure 2, due to the use of position embeddings. Based on these observations, It's not clear that bi-directional attention necessarily the best approach.

questions:
See previous section.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730707502186

thread_type:
official_review

<thread_object>
creator: reviewer_qpgs

<content>
summary:
The authors develop a generative modeling method for tabular data.  Their method learns a conditional generative model that takes masked input and predicts the missing columns.  At inference time, they use this model autoregressively to generate data one column at a time -- for both unconditional data generation and missing values imputation. The model comprises of a linear embedder, a transformer encoder, and either a diffusion decoder or categorical classifier -- depending on whether the particular column is continuous or discrete in nature.  The authors perform experiments on various tabular datasets and use various metrics to compare the samples generated by their method against those generated by other methods in the literature.  They also perform ablation studies to understand the relative importance of various components of their approach.

strengths:
- This paper tackles an interesting and important problem -- how to learn the generative distribution of tabular data.  Tabular data is indeed challenging because of the potentially unstructured and diverse nature of the columns at hand.  As the authors mention, there has been a lot of recent interest in developing deep generative models for this domain.
- The authors' proposed solution is an interesting one.  They provide comparisons in the experiments against several other methods in the literature and generally show superior performance on the statistical fidelity metrics (Table 1).  
- I appreciate the extensive set of experiments, such as machine learning efficiency, the various visualizations, and the ablation studies. These help give a holistic view of the relative performance of their method compared to the literature.  
- The code is available and generally looks well-organized and a useful resource for the community.

weaknesses:
- Nitpick: The authors sometimes use overly promotional language, e.g. line 71 — “through two ingenious design features”, line 84 —  “TABDAR offers several unparalleled advantages” — I recommend against the use of overly promotional words such as “ingenious”, “unparalleled advantages” in the paper.
 - Nitpick: Eq. (5) — Define $x^{<i}$.  For a reader familiar with autoregressive modeling notation, it is obvious what this means, but for others it may not be.  E.g. it would be good to clarify what this means for $i = 1$.
- One point that was confusing to me at first was that the model is not trained as an autoregressive model -- it is trained as a masked language model.  If I understand Eq. (8) / Algorithm 3 correctly, you are only masking out a particular set of columns at each training step. 
 Thus, you are not really learning an autoregressive distribution, this is more like a conditional distribution like masked language modeling in which you predict p(masked | unmasked).  I understand that you are using the learned conditional distribution in an autoregressive way at test time / generation time -- it would be good to clarify this point in the text.  
- There are also some questions about the method I have, listed below.  I am happy to give a positive score for this paper, but it is contingent on the questions below being addressed.

questions:
- For continuous columns, why use a diffusion loss?  You could also use z to parameterize any simple continuous distribution, e.g. a Gaussian.  Given that the diffusion process can be computationally expensive, do you think that it is a necessary component of the method?  I understand that you do an ablation in which you compare against discretizing the continuous variable, but this also seems like an unnatural approach compared to simply passing it through a continuous distribution. 
- In your embedding/masking, how do you distinguish between a continuous value being equal to 0 and you masking out that value?  To me, it seems that both would lead to the same result after embedding the data.  Is there a particular reason you chose to use zero for missing values instead of an embedding for a [mask] token, as typical in masked language models? 
- Details on the diffusion aspect are generally missing.  Is the denoising network the same for different continuous columns or is the denoising network column-specific?  How many steps of diffusion are generally used?  
- Table 2 — since some of these values are quite close, could you provide some standard deviations?  Also, if there are ties with your method in the table, you should probably also bold these for consistency and fairness to other approaches.  
- Figure 6 — It would be interesting to see standard deviations for these bars.  I wonder how sensitive the results are to different initializations / different MCAR masks.  The authors mention that they are “significantly outperforming” other methods, but I think standard deviations are needed to justify this somewhat bold claim.  It would also be interesting to see how the percent missing from the data affects the results (currently it is fixed at 30%)? 
- For conditional sampling (Algorithm 5), is there a reason why you need to generate step-by-step?  During training, it seems the model learns to predict all masked columns from unmasked ones in one step during training.  Do you get better performance by generating step-by-step at test time?
</content>
</thread_object>

</thread>
<thread>
creation date:
1731815605392

thread_type:
public_comment

<thread_object>
creator: ~calvin_mccarter1

<content>
title:
Related work for autoregressive tabular modeling

comment:
It's worth noting that gradient-boosted decision tree-based methods have previously shown performance exceeding neural network-based methods on this problem. ForestFlow [1] showed state-of-the-art performance for tabular generation, and UnmaskingTrees [2] showed state-of-the-art performance for tabular imputation. The latter work is of particular relevance, because it trains per-feature conditional models to autoregressively generate data, and avoids naive quantization via recursive hierarchical partitioning. It's also worth noting that while the paper claims "these models typically default to a left-to-right column sequence" (line 068), both TabMT [3] and UnmaskingTrees randomize over all possible column orderings.

[1] Jolicoeur-Martineau, Alexia, Kilian Fatras, and Tal Kachman. "Generating and imputing tabular data via diffusion and flow-based gradient-boosted trees." International Conference on Artificial Intelligence and Statistics. PMLR, 2024.

[2] McCarter, Calvin. "Unmasking Trees for Tabular Data." arXiv preprint arXiv:2407.05593 (2024).

[3] Gulati, Manbir, and Paul Roysdon. "TabMT: Generating tabular data with masked transformers." Advances in Neural Information Processing Systems 36 (2024).
</content>
</thread_object>

</thread>


Metareview:
metareview:
The paper proposes a generative model for tabular data called Diffusion-nested Autoregressive model (TabDAR), which combines masked bi-directional attention transformers with diffusion models to handle mixed numerical and categorical data and supports arbitrary generation orders. The reviewers acknowledge that the paper tackles an important problem, proposes an interesting solution, and has extensive experiments. However, reviewers raised concerns regarding presentation issues, such as inconsistent notations and limited novelty compared to prior methods like TabSyn, which also exhibit similar empirical performance. Some reviewers also questioned the empirical discrepancies between reported baseline results and prior literature and found the experimental gains marginal on certain tasks, suggesting that benchmarks may not sufficiently differentiate methods. In the rebuttal, the authors addressed many technical and presentation issues. However, some conceptual critiques, such as novelty concerns, remain partially addressed. While the paper presents extensive experimental results, the marginal empirical gains, conceptual weaknesses, and mixed reviews on novelty mean that I recommend the paper be rejected.