Improving Tabular Generative Models: Loss Functions, Benchmarks, and Iterative Objective Bayesian Approaches

Authors: Minh Hoang Vu, Daniel Edler, Carl Wibom, Tommy Löfstedt, Beatrice Melin, Martin Rosvall
Primary area: generative models
Keywords: generative adversarial network, synthetic data, correlation- and distribution-aware loss function, iterative objective refinement Bayesian optimization, benchmarking framework

Abstract:
Access to extensive data is essential for improving model performance and generalization in deep learning (DL). When dealing with sparse datasets, a promising solution is to generate synthetic data using deep generative models (DGMs). However, these models often struggle to capture the complexities of real-world tabular data, including diverse variable types, imbalances, and intricate dependencies.
Additionally, standard Bayesian optimization (SBO), commonly used for hyper-parameter tuning, struggles with aggregating metrics of different units, leading to unreliable averaging and suboptimal decisions.

To address these gaps, we introduce a novel correlation- and distribution-aware loss function that regularizes DGMs, enhancing their ability to generate synthetic tabular data that faithfully represents actual distributions. To aid in evaluating this loss function, we also propose a new multi-objective aggregation method using iterative objective refinement Bayesian optimization (IORBO) and a comprehensive statistical testing framework. While the focus of this paper is on improving the loss function, each contribution stands on its own and can be applied to other DGMs, applications, and hyperparameter optimization techniques.

We validate our approach using a benchmarking framework with twenty real-world datasets and ten established tabular DGM baselines. Results demonstrate that the proposed loss function significantly improves the fidelity of the synthetic data generated with DGMs, leading to better performance in downstream machine learning (ML) tasks. Furthermore, the IORBO consistently outperformed SBO, yielding superior optimization results. This work advances synthetic data generation and optimization techniques, enabling more robust applications in DL.

Decision: reject
Reviewer scores: 5, 5, 5, 6

Reviews and discussion:
<thread>
creation date:
1730137824950

thread_type:
official_review

<thread_object>
creator: reviewer_hdf4

<content>
summary:
This work introduces correlation and moment-matching loss functions to regularize the loss function of different deep generative models for tabular data. Its results show that with proper selection of hyperparameters, its approach consistently improves the baselines. A Bayesian optimization procedure is introduced for hyperparameter tuning.

strengths:
The added regularizers are described clearly and intuitively, with a well-defined methodology and comprehensive benchmark design. This approach encompasses various generative models and employs Bayesian optimization to identify optimal hyperparameter configurations. Consistent improvements over baseline models are demonstrated.

weaknesses:
What I miss from the paper is a discussion on how to tune the method in the case of data heterogeneity and its performance and robustness in missing data scenarios.  How do the regularizers formulate in the case of counting distributions (e.g., Poisson likelihood) or ordinal variables? Do they consistently improve the results in the case of large fractions of missing entries in the database? I set my score to 6 since I feel that without a proper discussion on these aspects, the impact of the paper is limited.

questions:
See above
</content>
</thread_object>

</thread>
<thread>
creation date:
1730357358537

thread_type:
official_review

<thread_object>
creator: reviewer_mc61

<content>
summary:
**Review of "Improving Tabular Generative Models: Loss Functions, Benchmarks, and Iterative Objective Bayesian Approaches"**

This paper proposes several methods to enhance deep generative models (DGMs) for synthetic data generation with a particular focus on tabular data. While the work presents promising results in experiment, certain aspects need further clarification and further improvement.

strengths:
The paper presents an approach to enhancing Deep Generative Models (DGMs) for synthetic data generation on tabular data. Introduction of a correlation- and distribution-aware loss function, iterative objective refinement Bayesian optimization, and a detailed benchmarking framework are presented.

weaknesses:
1. **"Moment Generating Function (MGF)":**
The term "Moment Generating Function (MGF)" appears to be misused. The paper discusses empirical moments themselves rather than the empirical MGF $\hat{M_X}(t)$ from which the $n$-th moments can be obtained by taking $n$-th derivatives wrt $t$ at $t=0$. [See *Casella, Statistical Inference, 1990* (pp61)]

2. **Biased Estimator in Synthetic Data:**
 A biased estimator is used to calculate the standard deviation. This includes the estimator on synthetic data sampled at size $B$, which is not enough for the biased estimator to converge to the unbiased one. It would be beneficial for the paper to address or justify this choice. 

3. **Hyperparameter $\lambda$:**
Hyperparameter $\lambda$ in Eq.6 scales the $L_{\text{distribution}}$ in a manner the same as $\beta$ in custom losses, since $\lambda$ is 
 proportional to  $L_{\text{distribution}}$ in Eq.6.  Simultaneous inclusion of $\lambda$ and $\beta$ in the hyperparameter search may lead to issues such as multi-collinearity for Bayesian optimization.

questions:
1. **Significance Levels and Decision-Making:**
   In Table 1, the column for significance levels presents $p$-value ranges. A more detailed description of the decision based on the test statistic (or $p$-value obtained) may be helpful in understanding the experiment since a two-sided test is concerned.

2. **Distribution matching loss:**
It is possible for non-converging distributions to have similar moments, especially in lower orders. And, moment estimators of higher order moments introduce instability in the finite sample sense, and this instability goes up when the moment order goes up. It would be helpful if the author could justify using moments for distribution rather than the usual distance/score-based metrics for distribution similarity.



**Some Suggestions:**
 ***Reordering Loss Components:***
  For clearer presentation, consider swapping the order of the two proposed loss components to explain what $\mu$ and $\sigma$ are before presenting them in Eq.2.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730393238984

thread_type:
official_review

<thread_object>
creator: reviewer_xqkd

<content>
summary:
- Introduced a correlation- and distribution-aware loss function designed as a regularizer for DGMs in tabular data synthesis that displays promising results
- Introduced a hyperparameter tuning approach, IORBO, that leverages rank-based aggregation. (concerns of units
- They introduce a benchmarking system evaluating statistical similarity, ML TSTR performance, and ML augmentation performance, with robust statistical tests.

strengths:
**Originality and Quality**

The correlation- and distribution-aware loss function is new and interesting to me. I have not encountered works that display the effectiveness of enforcing correlation and high-order moments in the loss function to improve generative models. It is nice to see an improvement in existing hyperparameter tuning algorithms such as Standard Bayesian Optimization by adding an iterative refinement process.

**Clarity**

Individual sections of the paper are well written.

**Significance**

Tabular data generation is gaining traction in real-world applications such as electronic health records. This work helps bring progress to tabular data generation.

weaknesses:
- I am struggling to find a central theme/research question the paper is trying to answer. It provides solutions from three different perspectives: 1) Loss Function Regularization: Improving generative model outputs by enforcing statistical properties (e.g., correlation, distribution); 2) Hyperparameter Tuning: Using methods like IORBO for iterative optimization; 3) Statistical Tests: Providing a framework for assessing model performance across metrics. I am unable to determine a flow to link the three ideas together/how one idea enforces the other.
- L486: How does IORBA perform against other hyperparameter tuning methods such as [Randomised Optimization, GridSearch etc.](https://scikit-learn.org/1.5/modules/grid_search.html#tuning-the-hyper-parameters-of-an-estimator) in terms of performance? What about the computational cost for IORBA vs. SBO and other mentioned baselines, what is this tradeoff? Additionally, what are the optimized hyperparameters that you obtain from your method? Ablation studies of the aforementioned would make your case stronger.
- In [TabSyn](https://arxiv.org/abs/2310.09656), the authors provided a comprehensive evaluation of synthetic tabular data using over five distinct evaluation metrics. Their metrics are straightforward and easy to comprehend. It will be nice to compare and justify why your metrics are more convincing and better than their proposed benchmark so that users should use your metrics instead of/in addition to TabSyn’s.
- Privacy is also crucial in synthetic tabular generation. How does your proposed loss function affect privacy-preserving metrics such as DCR and C2ST?

questions:
The individual contributions of the paper are good. However, my main concern is the overall theme of the paper. I am unable to determine the overall research question the paper is trying to address. Please see weaknesses.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730501937605

thread_type:
official_review

<thread_object>
creator: reviewer_1icq

<content>
summary:
This paper introduces two regularization terms for improving the performance of the tabular generative model. The authors further propose to use ranking-based Bayesian Optimization to choose the hyperparameter. They finally evaluate the proposed method in Twenty tabular datasets on 10 base generative models by using TSTR, augmentation.

strengths:
The experiments are comprehensive. Hyperparameters are chosen reasonably.

weaknesses:
The proposed method is heuristic. The paper does not provide an optimality or convergence guarantee of the proposed loss. These two proposed losses are reasonable for tabular data but not general enough for other types of data. The hyperparameters are chosen by the new proposed Bayesian Optimization without theoretical guarantees.

questions:
What will the performance if using Standard Bayesian optimization rather than IORBO proposed by this paper?
</content>
</thread_object>

</thread>
<thread>
creation date:
1731952407260

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
To all reviewers

comment:
Thank you to all reviewers for your valuable comments. We greatly appreciate your feedback. A revised version of our manuscript has been uploaded, with all changes and updates highlighted in blue. We look forward to your response.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732532355228

thread_type:
official_comment

<thread_object>
creator: area_chair_fgk5

<content>
title:
Discussions between reviewers and authors

comment:
Time for discussions as author feedback is in. I encourage all the reviewers to reply. You should treat the paper that you're reviewing in the same way as you'd like your submission to be treated :)
</content>
</thread_object>

</thread>
<thread>
creation date:
1733159878024

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
To all Reviewers

comment:
We would like to express our gratitude to all the reviewers for their constructive feedback and insightful comments. Below, we provide a summary of the key changes and additions made in response to your suggestions.


---

**Reviewer 1iCq:**  

- *Query on Standard Bayesian Optimization (SBO):* To address your query, we conducted additional experiments comparing our proposed IORBO with SBO. The results, included in the revised manuscript, demonstrate that IORBO consistently outperforms SBO.
  
---

**Reviewer XQkD:**  

- *Central Theme and Research Flow:* We acknowledge your concern regarding the interconnectedness of our three contributions (loss function regularization, hyperparameter optimization via IORBO, and statistical tests). In response, we have clarified in both the abstract and introduction that these components can function independently or as a unified framework, with each addressing specific challenges in synthetic tabular data generation.
- *Comparison with TabSyn Datasets, Metrics, and Models:*   While TabSyn is an important contribution to the field, we have highlighted the gaps our method addresses in benchmarking metrics and its broader applicability. 
- *Privacy Metrics:*   While privacy-preserving metrics (e.g., DCR, C2ST) are critical in synthetic data research, our study focuses on publicly available datasets where privacy concerns are less relevant. As such, we emphasize evaluating statistical fidelity and downstream ML performance rather than privacy-specific metrics.
- *Ablation Studies:* Additional ablation studies were conducted to evaluate the contributions of the loss function and IORBO. Included in the supplementary material, these studies highlight the significant impact of both components on model performance. While each contributes positively, the combination consistently delivers the best results, emphasizing their complementary roles in optimization.



---

**Reviewer mC61:**  

- *Biased Estimator:* Regarding the use of biased estimators for standard deviation (and moments), we have clarified that mini-batch training is essential for efficient deep learning. While it introduces bias, this trade-off allows for handling large datasets, reducing memory usage, and enabling frequent model updates. 
- *Higher-Order Moments:*   We acknowledge the potential numerical instability when incorporating higher-order moments. As a solution, we propose using an exponential moving average of moments over iterations to ensure stability. This suggestion has been added as a potential future improvement in the conclusion.
- *Validation of Lower-Order Moments:*   Our approach is rooted in the method of moments, a well-established statistical technique, we have offered some intuitive reasoning: The loss can be thought of as "pinning down" the low-frequency components of the distribution (i.e., the lower-order moments), which stabilizes the GM's learning process by ensuring that it does not deviate significantly from the observed distribution in these aspects.
- *Multimodal Distributions:* The distribution-aware loss doesn’t need to fully represent the entire data distribution. Instead, the generative model learns additional characteristics, like the positions and weights of modes in multimodal distributions. Empirical results show that aligning lower-order moments (up to the 4th) captures key characteristics of tabular data.

---

**Reviewer HdF4:**  

- *Data Heterogeneity and Missing Data:* We have expanded the discussion on handling data heterogeneity and missing data. This includes addressing challenges such as the dominance of one data type over others and proposing strategies to mitigate this risk. Additionally, we emphasize how our approach balances regularization terms across diverse variable types.
- *Evaluation of Metrics by Data Type:*   While we currently apply a unified distribution-aware loss function to all data types (e.g., continuous, non-ordered discrete, ordinal, and Poisson-like variables), our results in Tables 2–4 demonstrate significant improvements in fidelity and downstream ML performance. This indicates the effectiveness of our general approach despite the lack of separate regularizers for different data types.
- *Behavior of Optimization Across Variable Types:*  We analyzed the impact of variable types (discrete vs. continuous) on optimization, considering factors such as: the ratio of discrete to continuous variables, normalization methods, and encoding schemes.

---

Once again, we deeply appreciate the time and effort invested by the reviewers in providing such detailed feedback. We believe these revisions have significantly strengthened the manuscript and addressed the key concerns raised. Thank you!
</content>
</thread_object>

</thread>


Metareview:
metareview:
This paper introduces a correlation- and distribution-aware loss function for tabular data generative model training. In addition, the authors improved Bayesian optimisation for tuning the hyper-parameters of the model and training procedure. They also introduced an evaluation benchmark to compare the performance for various number of tabular generative models.

Reviewers have various concerns in their initial reviews, regarding both the technical details and the confusion on the central contribution of the paper. Author rebuttal addressed some, but not all, of the concerns, see below "additional comments on reviewer discussion".

After a brief read, I also have the feeling that, 3 rather independent ideas are presented in the paper. For the significance of the contribution, I checked whether

(1) the paper is stellar in arguing each of the 3 independent ideas thoroughly; or

(2) the paper combines the 3 ideas in a clever way to make an overall significant contribution.

Unfortunately the answer for both options are NO. 

- First the benchmarking framework needs to be justified independently -- why this evaluation method is better than some other frameworks? 

- Second, if the authors were to claim that the new training loss + new BO procedure give the best generative model, then they need to provide ablation study regarding both components. E.g., would training the model without the new training loss + standard BO perform similarly?

I encourage the authors to revise their manuscript by incorporating the suggestions from this reviewing process.