Continuous Diffusion for Mixed-Type Tabular Data

Authors: Markus Mueller, Kathrin Gruber, Dennis Fok
Primary area: generative models
Keywords: synthetic data generation, diffusion model, generative model, tabular data, mixed-type data

Abstract:
Score-based generative models, commonly referred to as diffusion models, have proven to be successful at generating text and image data.
However, their adaptation to mixed-type tabular data remains underexplored. In this work, we propose CDTD, a Continuous Diffusion model for mixed-type Tabular Data. CDTD is based on a novel combination of score matching and score interpolation to enforce a unified continuous noise distribution for both continuous and categorical features. We explicitly acknowledge the necessity of homogenizing distinct data types by relying on model-specific loss calibration and initialization schemes. To further address the high heterogeneity in mixed-type tabular data, we introduce adaptive feature- or type-specific noise schedules. These ensure balanced generative performance across features and optimize the allocation of model capacity across features and diffusion time. Our experimental results show that CDTD consistently outperforms state-of-the-art benchmark models, captures feature correlations exceptionally well, and that heterogeneity in the noise schedule design boosts sample quality. Replication code is available at https://github.com/muellermarkus/cdtd.

Decision: accept (poster)
Reviewer scores: 6, 5, 5, 6

Reviews and discussion:
<thread>
creation date:
1730148756670

thread_type:
official_review

<thread_object>
creator: reviewer_697t

<content>
summary:
This paper trains a diffusion model for mixed-type (continuous and categorical) tabular data. The authors combine score matching with score interpolation to derive a diffusion model that add noise of categorical data in the embedding space, which makes it able to apply denoising process to mixed-type tabular data. To deal with the heterogenity of different types and features of data, the author further use noise calibration to balance the scale of continuous and categorical data. They also experimented on different types of noise schedules for better model training. The authors performed experiments and ablation studies on multiple datasets and used different metrics, the results showed that their model was able to achieve great performance on mixed-type tabular data.

strengths:
- The authors proposed the first model to allow diffusion models to train on mixed-type tabular data, and allows the use of advanced techniques such as classifier-free guidance in training.
- To allow training on mixed-type tabular data, the authors have introduced several customizations to diffusion models: such as loss balancing, and noise schedules.
- The experiments and comparisons look quite complete for me: the authors compared with seven methods on eleven datasets, using different kinds of metrics. The analysis of the results are also quite good. Results have shown that their method can achieve good results and higher efficiency.
- The authors have provided details of all the implementation and experiments, they also provided their code in the supplementary. I believe this is quite useful for further researchers.

weaknesses:
- The paper is more like a combination of existing models: diffusion models have already been applied to both categorical data and continuous data before. The authors combine them to deal with mixed-type data.
- The introduced changes: reweighting losses and changing noise schedules, although useful, but I think they are a bit minor to be counted as big technical contributions. The main reason is that they are a bit straightforward and also limited to current data setting. I am also curious that whether the noise schedules are also useful for data beyond mixed-type tabular data.

questions:
Please see weaknesses part. I would like to discuss with the authors on those points.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730368375135

thread_type:
official_review

<thread_object>
creator: reviewer_czjv

<content>
summary:
This paper combines score matching and score interpolation to ensure a common continuous noise distribution for both continuous and categorical features in tabular data generation. The proposed method focuses on adjusting noise schedules to enhance performance in generating synthetic tabular data.

strengths:
1. Superior Performance: The results reported in the paper demonstrate strong performance on existing datasets, showing significant improvements compared to baseline methods.

2. Simplicity and Clarity: The methodology centers on adjusting noise schedules, which is straightforward and easy to understand, making the approach accessible and replicable.

weaknesses:
1. Limited Novelty: The primary method introduced is Adaptive Noise Schedules, an area that has been extensively explored in prior research. Additionally, the paper addresses the homogenization of different data types—a problem that has been widely studied in previous tabular data generation works [1],[2]. Consequently, the novelty of this paper is significantly diminished, and the authors need to clearly articulate the innovative aspects of their approach.

2. Insufficient Experimental Reporting: The experimental results are not thoroughly reported. The authors compare their method with baseline approaches using only an average metric across various datasets, which leads to the loss of detailed information. To provide a more comprehensive evaluation, the authors should conduct detailed comparisons on each individual dataset, referencing methodologies from previous studies [1].

[1] Zhang, Hengrui, et al. "Mixed-type tabular data synthesis with score-based diffusion in latent space." arXiv preprint arXiv:2310.09656 (2023).

[2] Kotelnikov, Akim, et al. "Tabddpm: Modelling tabular data with diffusion models." International Conference on Machine Learning. PMLR, 2023.

questions:
Please see the weakness part
</content>
</thread_object>

</thread>
<thread>
creation date:
1730573806223

thread_type:
official_review

<thread_object>
creator: reviewer_2dwe

<content>
summary:
The paper proposes a joint continuous diffusion model for mixed type tabular data. Due to the mixed type of data format, the paper designs a score matching method and a score interpolation method for both continuous and categorical data. It presents a loss calibration method to balance the losses between the continuous and categorical data, and proposes an adaptive noise schedule.

strengths:
+ the paper is relatively easy to follow. The presentation of the paper is good
+ comparing with baseline methods, the proposed method seems to be working relatively well, although I have not directly worked on diffusion model for tabular data. There could be related work I am missing
+ contributions and limitations of the paper are properly discussed

weaknesses:
- the scope of the method is relatively limited on just mixed type tabular data. Diffusion model for tabular data had already been studied (e.g. Kotelnikov, 2023), and the contribution of this paper seems to be rather incremental in the sense that it just makes the diffusion model work a bit better in mixed type tabular data
- the method, although makes sense, is not novel in the whole landscape of diffusion models. Modeling continuous and categorical features has been studied in the past and many had been cited by the authors. This seems to be mostly applying these to this particular type of data

questions:
See the weakness section
</content>
</thread_object>

</thread>
<thread>
creation date:
1730682677546

thread_type:
official_review

<thread_object>
creator: reviewer_eakr

<content>
summary:
This paper proposes a continuous diffusion model for mixed-type tabular data, which
combines score matching and score interpolation techniques and imposes Gaussian diffusion
processes on both continuous and embedded categorical features. 

It proposes several strategies to balance model capacity between continuous and categorical features, including
weighted loss calibration, adjusted model initialization, and the adaptive noise schedule
designs.

strengths:
1. The problem is well motivated, and the intuition of this research is quite clear, which is to propose a unified pipeline to deal
with mixed type tabular data, which contains both continuous and categorical features.  In addition, the proposed strategies effectively deal with the imbalance and calibration between these two types of data.

2. Mathematical derivations are quite solid and appear to be valid, though having not checked completely.  

3. Implement comprehensive experiments to demonstrate the effectiveness of the overall pipeline as well as each design component.

weaknesses:
Weaknesses:   


1. The overall pipeline has limited novelty, as the key idea to push categorical data into
embedding space and use a Gaussian diffusion process to deal with it is a common
practice.


2. The preliminary section 2.2 that introduces diffusion for categorical features is not as clear
as that in 2.1. I wish the author to elaborate this part a bit more since this should be the
basic foundation of CDTD.


3. The customization on tabular data in section 3.4 is quite heuristic, are they supported by
theoretical proof or are they simply determined from empirical results?

questions:
The section that introduces learnable noise schedules is unclear. For example, its
motivation is confusing.   What does it mean that “given the same embedding dimension,
more noise is needed to remove the same amount of signal from embedding of features
with fewer classes”? 

Can you explain the difference between Feature-specific Noise Schedules and Adaptive Noise Schedules in more detail ?

[post rebuttal comment]. Some of the questions above are addressed by the author. I am happy to upgrade my rate for the paper.
</content>
</thread_object>

</thread>
<thread>
creation date:
1731602375730

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Overall Response

comment:
Dear Reviewers,

We would like to thank all reviewers for providing constructive feedback. We are encouraged by the positive statements made. Specifically, that reviewers think our paper:

- proposes strategies that "**effectively deal** with the imbalance and calibration between" (Reviewer eakR) continuous and categorical features;
- implements "**comprehensive experiments** to demonstrate the effectiveness" (Reviewer eakR) and demonstrates "**strong performance** on existing datasets, showing significant improvements compared to baseline methods." (Reviewer cZJV);
- provides **experiments and comparisons** which "look **quite complete** for me: the authors compared with seven methods on eleven datasets, using different kinds of metrics. The analysis of the results are also quite good. Results have shown that their method can achieve **good results and higher efficiency**." (Reviewer 697T);
- and "centers on adjusting noise schedules, which is straightforward and easy to understand, making the **approach accessible and replicable**." (Reviewer cZJV)
- is "easy to follow. The presentation of the paper is good" (Reviewer 2dwE).

Lastly, we want to emphasize that our goal in this paper is, as our title conveys, to improve/develop diffusion models for mixed-type tabular data. We believe that this is an understudied (and perhaps undervalued) type of data. However this type of data is extremely important in many sciences (medicine, economics, social science in general). We are careful to not make claims about the generalization of our framework to any other data types.

Our approach is therefore focused on adapting the diffusion generative framework to tabular data and all of our design choices and model elements reflect this goal, which inherently comes with the challenge of homogenizing different feature types (continuous and categorical in our case).

We **updated the pdf with additional clarifications** (highlighted in blue) based on your comments. Please see our reviewer-specific feedback for more information.
</content>
</thread_object>

</thread>


Metareview:
metareview:
The paper proposes a continuous diffusion model for mixed-type tabular data that combines score matching and score interpolation to handle both continuous and categorical features. The key innovation is ensuring a common continuous noise distribution between feature types and using adaptive noise schedules per feature/data type to manage heterogeneity. The authors claim their method achieves state-of-the-art performance across benchmark datasets and captures feature correlations well. Strengths include its principled approach to balancing continuous and categorical features, comprehensive experiments, and clear problem statement. However, reviewers questioned the technical novelty since similar approaches exist for combining categorical and continuous data diffusion. Additional weaknesses include limited discussion of prior work and lingering questions about the necessity of some model components.

Nevertheless, reviewers failed to address or inadequately addressed many of the author's rebuttal comments in the discussion phase. In reviewing these responses, many of the initial reviewer complaints seem to be reasonably covered in the discussion, aside from claims of novelty. In a final review of the paper, I'm inclined to agree with the borderline positive reviews. The studies presented in the paper seem like they could benefit future research.