A Unified Framework for Tabular Generative Modeling: Loss Functions, Benchmarks, and Improved Multi-objective Bayesian Optimization Approaches
Authors: Minh Hoang Vu
Daniel Edler
Carl Wibom
Tommy Löfstedt
Beatrice Melin
Martin Rosvall
Venue: Accepted by TMLR ()

Abstract: Deep learning (DL) models require extensive data to achieve strong performance and generalization. Deep generative models (DGMs) offer a solution by synthesizing data. Yet current approaches for tabular data often fail to preserve feature correlations and distributions during training, struggle with multi-metric hyperparameter selection, and lack comprehensive evaluation protocols. We address this gap with a unified framework that integrates training, hyperparameter tuning, and evaluation. First, we introduce a novel correlation- and distribution-aware loss function that regularizes DGMs, enhancing their ability to generate synthetic tabular data that faithfully represents the underlying data distributions. Theoretical analysis establishes stability and consistency guarantees. To enable principled hyper-parameter search via Bayesian optimization (BO), we also propose a new multi-objective aggregation strategy based on iterative objective refinement Bayesian optimization (IORBO), along with a comprehensive statistical testing framework. We validate the proposed approach using a benchmarking framework with twenty real-world datasets and ten established tabular DGM baselines. The correlation-aware loss function significantly improves the synthetic data fidelity and downstream machine learning (ML) performance, while IORBO consistently outperforms standard Bayesian optimization (SBO) in hyper-parameter selection. The unified framework advances tabular generative modeling beyond isolated method improvements. Code is available at: https://github.com/vuhoangminh/TabGen-Framework.

Published in Transactions on Machine Learning Research (12/2025)
A Unified Framework for Tabular Generative Modeling: Loss Functions, Benchmarks, and Improved Multi-objective Bayesian Optimization Approaches
Minh H. Vu Deportment of Diagnostics and Intervention
minh. www.wouwwa.....
Umed University
Daniel Edler
daniel. edler @ummus
Department of Physics
Umed University
Carl Wibom
carl, www.ommmmmu.....
Department of Diagnostics and Indervention
Umed Universily
Tommy Löfstedt
tommmy..fsstedtt@nmu...
Department of Computing Science
Umed University
Beatrice Mclin
beatrice. Cralricee.mcllin@uumu.....
Deportment of Diagnostics and Intervention
Umeò University
Martin Rosvall
mortin.rosval@@umee
Department of Physics
Umed University
Reviewed on OpenReview: https: // openreview. net/ forum? id- RPZOEW01
Abstract
Deep learning (DL) models require extensive data to achieve strong performance and gener- alization. Deep generative models (DGMs) offer solution by syuthesizing data. Yet current approaches for tabular data often fail to preserve feature correlations and distributions during training, struggle with multi-metric hyperparameter selection, and lack comprehen- sive evaluation protocols. We address this gap with a unified framework that integrates training, hyperparameter tuning, and evaluation. First, introduce novel correlation- and distribution-aware loss function that regularizes DGMs, enhancing their ability to gen- erate synthetic tabular data that faithfully represents the underlying data distributions. Theoretical analysis establishes stability and consistency guarantees. To enable principled snper-parameted search via Bayesian optimization (BO), we also propose a new multi- objective aggregation strategy based on iterative objective refinement Bayesian optimization (IORBO), along with a comprehensive statistical testing framework. We validate the proposed approach using benehmarking framework with twenty real-world datasets and ten estab- lished tabular DGM baselines. The correlation-awan aware loss function significantly improves the synthetic data fidclity and downstream machine learning (ML) performance, while IORBO consistently outperforms standard Bayesian optimization (SBO) in pareer-arraettr selec- tion. The unified framework advances tabular generative modeling beyond isolated method improvements. Code available at: https:// http:://iithbb..oo. com wtt.aau...oo///oaagmnllllll salhoangminhhTabGeen. TabbGe--Fraaewoo Framework.

---

Published in Transactions on Machine Learning Research (12/2025)
1 Introduction
For a wide range of deep learning (DL) applications, large amounts of data are crucial to improve both model performance and generalization. The fast-paced advancements in deep generative modeling have opened exciting possibilitics for data synthesis. Models trained on images and text effectively learn probability distributions over complex data and generate high-quality, realistic samples (Karras et al., 2021; Team et al., 2023). This success on structured data has fueled surge in deep generative model (DGM)-based methods (Goodfellow et al., 2014) for tabular data generation in recent years.
Yet tabular data poses unique challenges that resist direct transfer of techniques from other domains. Unlike images or text, tabular data lacks clear structure and contains mixed contimnous and discrete variables with complex interactions, imbalances, and non-linear relationships. Recent hybrid approaches have explored combining diffusion processes with flow-based models and wradient-booster trees to boost synthesis fidelity on tabular benchmarks (lolicoeur-Martinne et al., 2024; Zein & Urvoy, 2022). However, these methods still roly on unguided likelihood or adversarial objectives and do not explicitly enforce key statistics such feature correlations higher-order moments.
This mismatch between tabular data complexity and current generative approaches cascades across the entire modcling lifecycle. Existing deep neural network (DNN)-based generative models often struggle to reliably capture correlations and other statistical dependencies in tabular data sometimes failing to approximale even basic statistics such as the mean and variance particularly in limited-data settings (Xu et al., 2019). Current approaches to improve downstream machine learning (ML) analyses focus primarily on addressing data imbalance (Xu et al., 2019; Sun et al., 2023; Zhao et al., 2021) while neglecting the equally crucial role of feature distributions and correlations.
The cascade extends to hyper-parameter optimization. While Bayesian optimization (BO) is widely used, standard approaches like standard Bayesian optimization (SBO) are ill-suited to aggregating the multiple heterogeneous metries required for synthetic data evaluation. Combining metries with different ranges and units, such as classification accuracy and regression error, via simple averaging can overweight individual objectives and yield suboptimal parameter selections.
Finally, the cascade undermines evaluation itself, where rigorous assessment remains fragmented. Existing methods often suffer from limited evaluation scopes that focus on narrow metric subsets, making it difficult to assess model performance across the complexities of diverse datasets. This evaluation gap obscures whether apparent improvements reflect genuine advances or artifacts of selective testing.
The central problem is the absence of a unified framework that addresses tabular generative modeling across its full lifecycle: training, hyper-parameter luming, and evaluation. Current approaches tackle these stages independently, missing opportunities for integrated solutions that could amplify improvements at each step.
To address this gap, we propose a comprehensive unified framework that tackles training, perer-parameter tuning, and evaluation as interconnected challenges. First, we introduce a novel correlation- and distribution- aware loss function for DGMs designed to enforce statistical properties that cxisting generative models fail to capture reliably. Second, we develop iterative objective refinement Bayesian optimization (IORBO), which aggregates multiple evaluation metrics through ranking resolve inconsistencies caused by metrics with different units scales. Third, we establish a comprehensive benchmarking framework that evaluates synthetic data across twenty datasets using statistical, regression, and classification metrics. By integrating these components, we create unified pipeline where training improvements and robust. tuning work in concert with rigorous evaluation. The tight coupling between training, tuning, and evaluation improves statistical fidelity, robust optimization, and benchmarking rigor across diverse datascts.
In summary, we provide:
A Correlation- and Distribution-Aware Loss Function: We propose a custom correlation- and distribationnawarre loss function that emphasizes the importance of fcature correlations and distributions in tabular data. This custom loss function is used as an auxiliary regularization term, complementing the primary training objective. It significantly enhances the performance of DGMs,
2

---

Published in Transactions on Machine Learning Research (12/2025)
including generative adversarial network (GAN), variational auto-encoder (VAE), and denoising diffusson probabilistic model (DDPM), as demonstrated through extensive benchmark evaluations.
2. Iterative Objective Refinement Bayesian Optimization: We propose IORBO to aggregate multiple objectives through ranking, resolving inconsistencics caused by metrics with different units or scales.
3. Benchmarking Framework for Synthetic Data Generation Algorithms: We establish a comprehensive open-source benchmarking framework that includes twenty tabular datasets and various evaluation metrics based on statistical tests. This framework implements ten state-of-the-art tabular DGMs and supports extensions with additional methods and datasets.
The rest of the manuscript is organized as follows: and highlights recent developments motivating our approach. Section 3 presents the proposed correlation- and distribution-aware loss function (Section 3.1), the IORBO (Section 3.2), and the benchmarking framework (Section 3.3 -36)..). Section provides theoretical guarantees for the proposed loss functions, including stability analysis, error bounds, and generalized approximation guarantees. In Section 5, we describe the datasets and implementation details and training. Section 6 reports and discusses the results, including performance improvements and ablation studies. Finally, Section 7 concludes the paper and outlines future directions.
Section 2 reviews related work on DGMs for tabular data
2 Related Work
Most existing methods to generate synthetic tabular data developed in the past docade model measurements in a table as a joint parametric density and then sample from that parametric model. Different models have been employed based data characteristics: multivariate Gaussian (Prühwirth--cchatiee et al., 2018), Bayesian networks (Aviñó et al., 2018; Zhang et al., 2017), and copulas (Patki et al., 2016) for non-linearly correlated contimuous variables. However, these methods rely on strong modeling assumptions and are limited in their ability to generalize beyond those assumptions. As a result, they often fail to capture the complex relationships present in real-world tabular data.
To overcome these limitations, recent approaches have turned to more expressive and flexible DGMs, such as VAEs (Kingma & Welling, 2013), diffusion models "Sohl-Dickstein et al., 2015; Kotelnikov al., 2023), and GANs with their numerous extensions (Arjovsky et al., 2017; Gulrajani et al., 2017; Zhu et al., 2017; Yu et al., 2017), have made them very appealing for data representation. This appeal extends to generating tabular data, especially in the healthcare domain. For example, Yahi et al. (2017) leveraged GANs to create synthetic continuous time-series medical records, and Camino et al. (2018) proposed to generate discrete tabular healthcare data using GANs. CTGAN (Xu et al., 2019), DP-CGANS (Sun et al., 2023), and CTAB-GAN (Zhao et al., 2021) were proposed to address the complexities of mixed-type tabular data and to address challenges when generating realistic syuthetic data, particularly for imbalanced datasets. More recently, TabDDPM (Kotelnikov et al., 2023), diffusion model designed specifically for tabular data, offers the flexibility to incorporate various backbone architectures to model the reverse process. While these DGMs improve flexibility, they still struggle to capture the diverse variable types, imbalances, and intricate dependencies inherent in real-world tabular data.
Prior works in correlation and statistical regularization most prominently CORAL by Sun et al. (2017)-focur on aligning socond-order statistics (covariance) between source and target domains for unsupervised domain adaptation in computer vision. While effective for visual tasks, these methods typically assume continuous features and perform full covariance alignment, which can be costly both computationally and statistically in digh-dimensional settings. By contrast, our correlation- and distribution-aware loss is tailored to tabular data with mixed continuous and discrete variables: it leverages variance- (diagonal covariance) based statistics for scalability, explicitly allows matching arbitrary higher-order moments (not just the second moment as CORAL), and thus avoids the expense of estimating full covariance matrices while retaining the ability to capture richer distributional structure. Furthermore, our loss is integrated across multiple DGMs and evaluated on extensive downstream ML tasks and statistical metrics, providing theoretical guarantees alongside empirical improvements over standard baseline losses.
3

---

Published in Transactions on Machine Learning Research (12/2025)
Beyond DGMs, several recent works have explicitly tackled the challenge of generating imputing heteroge- neous tabular data with mixed data types and missingness. Nazabal et al. (2020) proposed variational autoencoder framework tailored for incomplete heterogeneous data, demonstrating strong performance in both imputation and generative tasks. Building on this direction, Ma et al. (2020) introduced VAEM, a DGM capable of capturing complex dependencies in mixed-type data, providing a principled approach for handling heterogencous feature spaces. More recently, Peis et al. (2022) extended these ideas with deep hierarchical models combined with Hamiltonian Monte Carlo, achieving state-of-the-art results missing data imputation and acquisition for structured heterogeneous datasets. These methods highlight the importance of explicitly modeling heterogeneous variable types, direction complementary to our proposed correlation- and distribution-aware losses.
In parallel, BO has become an essential tool for methods such as the tree-structured parzen estimator approach (TPE; Bergstra al., 2011) have been widely used for black-box optimization but often struggle with multi-metric objectives, treating metries independently aggregaling them naively, which can bias hyper-parameter selection. More recent approaches improve multi-objective optimization and robustness, yet they still face challenges with heterogeneous, sometimes conflicting evaluation metrics common in tabular generative modeling. Our proposed IORBO addresses these issues by introducing a rank-based aggregation scheme that preserves metric relationships and supports fair, robust hyper-parameter tuning, directly tackling the sensitivity and correlation-ignorance of prior methods while improving model training and evaluation.
tuning in generative modeling. Classic
3 Methods
DGMs learn to map a random noise vector, denoted by z, to an output sample. This allows them to generate data instances that resemble the training data. DGMs have found varions applications, e.g., to generate images (Goodfellow et al., 2014; Karras et al., 2020), multi-modal medical images (Zhu et al., 2017), vectors of tabular data (Xu et al., 2019; Sun et al., 2023). In this work, the focus was to generate tabular data with continuous and discrete variables.
3.1 A Correlation- and Distribution-Aware Loss Function
Let the training dataset be X = (a, = (o. x. (d)) Vi E NN)....N)), where N is the number of training samples. The I; I denotes the i-th training sample from X, and x and ( are continuous and discrete features, respectively. Let p& be the learned probability density over the synthetic data, ã, such that ã E Rm is a sample from the DGM, G. Here, G is a learned mapping from a prior distribution p(z) to the data space p( z).
Corvelalion-awarr loss function. The correlation-awwre loss function defined as
m
m
Ccorrelation m(m
(g
gg.1)²
(1)
k=j+1
where g is the sample correlation over the real data and ğ is the sample correlation the generated data, such that
N
1
Fi.j - 11
9j.k =
(2)
N
σ +
i=1
1
14j
0ikk j4
0j.k
(3)
B
0j +
0 + c
with B the size of the mini-batch used when training the DGM, and elements æ, and क.j belonging vectors 2; X and 2, X, respectively. A small positive value, 1- 10-5, was added the denominators of the correlation terms to avoid division by zero. The mean and standard deviation of the j-th column in
4

---

Published in Transactions on Machine Learning Research (12/2025)
tabular data set, X, were estimated
N Dij and oj
N Σ 1
Σ Mj N
[i. - p4))
(4)
N
Similarly, μú and o, were estimated as the mean and standard deviation the generated data, {ě, e (l,..., B)). We discuss the theoretical guarantees for the correlation-aware loss function, including the stability analysis, in Section 4.1.
Distribation-awarr loss function. The distribution-aware loss function integrates the strengths of the method of moments and maximum likelihood estimation (MLE) to align with the true distribution by capturing both statistical moments and likelihood properties order to enhance the model's ability to learn accurate data representations (Pearson, 1936; Rice, 2007). Additionally, the choice of moments over distance- based metrics, such as Wasserstein, motivated by their computational efficiency and stability, as lower-order moments provide robust approximation of the distribution while avoiding the high computational cost associated with distance-based methods. To characterize the training data distribution, employed the first and central second moments,
5/D)
1
Fig = pij
(5)
N
N
5
1
(ii.j 111)² = =o
(6)
N
and for h 2 3 the standardized higher moments,
sgx (h)
Dij M4j =
(7)
Similarly, the empirical moments were computed for the synthetic data, denoted as 5/ 5/2) and S again for h 2 3. In this case, B was used place of N. Finally, the distribution loss was defined as
m
(h) +
Edistribution
1
(8)
h
S +
where the number of moments, H, treated hyper-parameter. Instead of making the moments equal, their quotient was made to be equal to one way to handle scale differences. By using unified distribution-awware loss, we handle continuous and discrete variables in the same manner, simplifying the implementation and preventing imbalances that could arise from separate regularization terms for different data types. We discuss the thcoretical guarantces for the distribution-awaro loss function, including the numerical stability and consistency, in Section 4.2.
Custom loss function for DGMs. The correlation- and distribution-aware loss function was integrated into three DGMs: GAN, VAE, and DDPM. For GANs, the proposed loss function incorporated into the generator's loss
LG -xxppx()
[oo((1 D(G(=))) Ceorrcattioon + &Ldistribution
Co
where Lo the original GAN's generator loss, and G and D the generator and discriminator of the GAN, respectively. The hyper-parameters, a and 3, controlled the influence of the correlation and distribution terms.
5

---

Published in Transactions on Machine Learning Research (12/2025)
We extended the TVAE model (Xu et al., 2019) (a VAE designed for tabular data) with the proposed loss function
TVAE Creconstruction + CKLD +oCorrretation + @
(10)
CTVAE
where LTVAE is the original TVAE's loss, and Preconstruction and LKLD are the reconstruction loss and the Kullback Leibler (KL) regularization term, respectively.
For the diffusion model, TabDDPM (Kotelnikov et al., 2023), the proposed loss function was integrated into the total loss of the multinomial diffusions
Eiso 1i
È ITDDDPM Li
csimple
C
tal
correlattiooon
(d) correlationn + (L
distribution + (C
disteibution
(11)
LTADDDPM
where CrabDDPM denotes the original TaDDDPM loss, comprising the mean-squared error for the Gaussian diffusion term, Csimple, and the KL divergence for all multinomial diffusion terms, Eise Li/c (Kullback & Leibler, 1951).
Unlike other DGMs, TabDDPM handles continuous and discrete features separately. For continuous features, TabDDPM predicts the Gaussian noise added through a forward Markov process. For discrete features, it predicts their one-hot encoded representation. To align our proposed loss functions with this characteristic, we adapted the correlation and distribution loss functions, Correlation and Cidd distribution to focus exclusively on discrete features. For continuous features, the Gaussian input noise is treated as the real data and the TabDDPM's prodicted noise component as the synthetic data. To encourage the model to capture the distribution of the predicted noise, we introduce a distributional loss term distribution for continuous features. A weighting parameter Ç is applied to this term to control its influence in the overall loss function, allowing to balance the importance of distributional aligument between real and predicted noise against other objectives.
((c)
Tt important to note that the proposed correlation- and distribution-aware loss functions are incorporated as auxiliary regularization terms added the primary training objective (e.g., likelihood or adversarial loss), ensuring that the DGM remains guided by its standard optimization criterion while explicitly preserving correlations and distributional properties.
Scope and applicability. The proposed correlation and distribution-awaree loss functions are specifically designed for tabular data, where feature-level correlations and marginal distributions carry the most informative signals. While the losses themselves are general, applying them directly to raw pixels or text is less meaningful due strong local structure and sequential dependencies. Nonetheless, the framework could be adapted to work on learned embeddings from other modalities, such image text encoders, which provide structured representation suitable for correlation- and distribution-base regularization.
3.2 Iterative Objective Refinement Bayesian Optimization
Previous research on DNN often relied on tuning byper-parameters based on a single metric or aggregating multiple metrics with varying units in SBO. For example, the objective function guiding the BO process could be the Dice score for medical segmentation (Vu et al., 2021), mean macro-accuracy for visual question answering (Vu et al., 2020), or metrics like F-score (classification) and R-squared (regression) evaluated with Catboost (Dorogush et al., 2018) on synthetie tabular data (Kotelnikov et al., 2023). A significant. challenge in SBO arises from managing diverse metrics, such as those used in statistical evaluations and ML performance, that differ in units, complicating direct aggregation. This limitation can hinder the ability to fully capture trade-offs between different objectives. To overcome the issues associated with aggregating metries with varying units in multi-objective SBO, we propose ranking- approach, named TORBO, to enhance BÖ performance.
To illustrate, consider optimizing DGM. We define ya as the vector comprising all evaluated metries where E U), with U representing the number of samples used in the optimization. In the SBO, the

---

Published in Transactions on Machine Learning Research (12/2025)
Algorithm SBO
Algorithm 2 IORBO
Initialize surrogate model
Initialize surrogate model
Initialize generative model Gi
Initialize generative model G:
Suggest imitial hyper-parameters 9
Suggest initial hyper-parameters C1
Build and train Gi Perform evaluation to obtain yı
Build and train G
Perform evaluation to obtain yı
Compute f(y)
Compute r.
g(1n y1)
Fit surrogate model with (91.11)
Fit surrogate model with (01,r)") (1)
u + 2 to U do
for 1 + 2 to U/ do
Suggest O
Suggest θμ
Build and train G.
Build and train Gi
Perform evaluation obtain and Update surrugate model with Tu)
Perform evaluation obtain
Update ranks {r( (w) .....?u (n) based on (y), (y1.va,..... va....,yu
Fit surrogate model with revised samples
)..,........ (01, ri (. (4)) J.... (Qu.r(.)
end for
end for return Optimal syper-parameters O*
return Optimal hyper-parameters e*
Comparison between SBO and IORBO algorithms.
objective function of sample u is defined as r. (yu) where f is aggregation function. As outlined in Algorithm 1, the SBO holds Tu constant throughout the optimization.
In contrast, IORBO defines the objective function (p) where S p and p E {1,...,U] U) (sce Algorithm 2). Here, represents the iteration where the objective is first generated, while p denotes when it is updated, introducing iterative refinement into the process. In the IORBO, the objective function of sample u is defined r( g yi ........20. up)) where g is rank-based function. For example, the second iteration, y22 is evaluated, then both (2) and r2 are computed. In the third iteration, ys is added, allowing for the computation of (3) (.(), and (3) and so on. The objective functions are recalculated as the mean ranks of all generated samples, yielding r(") .72 (u) ..... based on Vı, U2...., Vu- To compute the mean ranks, all data points that are generated by the TORBO for each evaluated metric are first ranked and then the average rank across metrics is caleulated.
ri.
The objective function for the first of syper-parameters, O1, is iteratively updated: 1 - r
(()) For the 22, we update: r (2) + 12 - .... r2) (U) and so on. The surrogate model is simultancously (()),,
(3)
(w)), (02, (0..(())) (w)) )....,, (O,, r'")). IORBO incurs a slight additional cost. refitted with the revised samples, (0 10
for refitting the surrogate model with revised samples during the itcrative refinement. However, this overhead
is negligible compared to the overall computational cost. Apart from this refinement step, the process is essentially the same as SBO. For numerical illustration, see Section 3.2.1.
3.2.1 Illustrative Example: SBO and IORBO in Practice Comparison
Table 1 and Table 2 illustrate the key difference between SBO and IORBÖ across three optimization iterations using a toy example with four evaluation metrics: ş {a, b, d}. Each row in the tables corresponds to the metric values obtained by evaluating set of hyper-perarneters at given iteration.
In Table 1, the objective function is computed as the mean of all metric values for each evaluated sample. As new samples are added, the objective values for previous samples remain fixed, meaning r1 = T2 = r3 = across all iterations. This is because the SBO applies a static aggregation each sample independently, without revisiting or comparing across itcrations.
In contrast, Table 2 shows how IORBO iteratively refines the objective function. At each iteration, metric values ranked across all evaluated samples compute the average rank per sample. This process results in dynamic objective values: (3) # r2 # ra
(3) and r((1 # (2) # + The key idea is that as more samples are evaluated, the ranking context changes, and thus the relative standing of earlier samples is updated to
(3)

---

Published in Transactions on Machine Learning Research (12/2025)
Table Example of SBO where the objective function is computed as the mean of all evaluated metrics.
Iteration
Iteration
Iteration 3
Metric/ Sample
2
3
a
0.5
1
0.5
2.4
b
1
2.5
1
2,5
0.2
c
0.5
0.5
0.8
d
0.5
0.5
0.6
Objective function
F1 1
F1
12
F1 =
12
73
Table 2: Example of IORBO.
Iteration
Iteration 2
Iteration 3
Metric/ / Sample
2
2
0.5
0.5
2.1
2.5
2.5
0.2
1
0.5
0.5
0.8
0.5
0.5
0.6
Metric ranking Sample
2
2
1
1
3
2
2
3
1
1
1
3
1
2
Objective function
(1)
ri
1.75
7'2
(2)
1.25
r1
(3)
2.5
r2
(3)
1.5
13
(3)
reflect the expanded information. For instance, the first sample has objective values r 1, r 1.75, and y( = 2.5, demonstrating the refinement process.
This illustrative example highlights how TORBO uses cross-sample comparison to improve the fidelity of the optimization signal, allowing it to better differentiate between competing parameer configurations. In contrast, SBO may lead to flat or misleading optimization signals when metric values vary scale importance. This rank- based refinement in IORBO enables more informed and robust surrogate model updates throughout the optimization.
3.3 Evaluation
Statistical similarity. The statistical similarity evaluation focuses how well the statistical properties of the real training data are preserved in the synthetic data. Inspired by previous review study (Goncalves et al., 2020), we compared two aspects: (1) Individual variable distributions assess how closely the distributions of each variable in the real and synthetic data sets resemble each other; and (2) pairwise correlations reveal the differences pairwise correlations between variables across the real and synthetic data (Step Figure 2).
We employed four key metrics to quantify how closely the real and synthetic data distributions resemble each other. (1) The KL divergence (Hershey & Olsen, 2007): This method quantifies the information loss incurred when approximating true probability distribution with another one. (2) The Pearson's Chi-Square (CS) test (Pearson, 1992): This test focuses on categorical variables and assesses whether the distribution of categories in the synthetic data matches the distribution in the real data. (3) The Kolmogorov--Sigg Smirnov (KS) test (Massey Jr, 1951): This test is designed for contimuous variables and measures the distance between the cumulative distribution functions (CDFs) of the real and synthetic data. (4) The dimension-
probability (DWP): We leveraged the DWP (Armanious al., 2020) to quantitatively assess the quality of the generated data. This metric evaluates how well the model captures the distribution of each individual class variable.
8

---

Published in Transactions on Machine Learning Research (12/2025)
To calculate the DWP metric, we compute the average distance between scatter points and perfect diagonal line (y = z). Each scatter point represents either a class within a categorical variable or the mean value of a continuous variable.
To assess how effectively the synthetic data captures the inherent relationships between variables observed in the real data, we compare correlation coefficients between variable pairs. For continuous variables, we employ the widely-used Pearson correlation coefficient, calculated from both the real and synthetic data matrices. In the case of categorical variables, leverage Cramer's V coefficient to quantify the association strength between each pair in both datasets (Frey, 2018).
While Pearson and Cramer's V handle contintous-continuoor and creegorical-catgorien correlations, continuous-caalegooicaa dependencies are not explicitly measured. These can be partially captured by one-hot encoding categorical variables or indirectly via the ittribution-awarr loss. Addressing this explicitly remains a direction for future work.
div. CS test Cramer's ' V Pearson DWP
ML methods
Classificttin
ML methods
Chasification
Training
Regression
Bal. accuracy Precision:
Regression
Bal. Precision
Rocall
Rocall
Synthetic
Training
SVM
G m-mem
Training
SVM
G m-maa
F-ature AUC
F-scoce
Systhotic
Dagging
Training
Bagging
Step 1: Statistical evaluation
XCBovet
Regression
Synthetic
XGHoowt
Regression
MAE MSE
MAE MSE
Optinal ML parameters
Tost
RP
Test
RF
Train prodiction models
R
Tel prediction models
Step 2: ML TSTR evaluation
Step 3: ML augmentation evalnation
Figure 2: Evaluation pipeline. For dataset Dj and ML method Mp, the optimal prper-parameters 0 M, were determined using five-fold cross-validation based on ML evaluation metrics (see Figure 4 in the Appendix).
MT. performance. The ML. performance evaluation is meant to enable researchers to leverage synthetic data when developing ML methods two key areas: augmentation (sce Figure and Steps and 3). In the TSTR task (Step 2 in Figure 2), the goal for ML methods trained on synthetic data was to achieve performance comparable or identical those trained on real data.
Seyythheii-Test--BEeee (TSTR) (Lu et al., 2023) and
In addition, we introduce the concept of an ML augmentation benchmark task (Step 3 in Figure 2), which, to our knowledge, represents the first systematic application of augmentation for evaluating tabular synthetic data. Here, models are trained on a combination of real and synthotic data, and the objective is to outporform models trained solely on real data. While data augmentation with synthetic samples is a well-established practice machine learning, our contribution lies formalizing it as a reproducible and domain-agnostic benchmark for assessing DGMs across diverse tabular datasets. By incorporating synthetic data in this standardized evaluation, models can learn from richer distributions, providing practical measure of generative model utility.
To comprehensively evaluate the performance of trained ML models on imbalanced classification datasets, we employed a suite of metrics inchuding balanced accuracy, precision, recall, geometric mean (G-mean), F-score, and area under the ROC curve (AUC). For regression, we used metrics focused on capturing regression error: mean absolute error (MAE), mcan squared error (MSE), and the cocfficient of determination, R-squared (RP). This combined evaluation approach provides a nuanced understanding of model performance across both classification and regression tasks.
To assess the ML performance in both the TSTR and augmentation tasks, we split the experimental datasets into 80% training and 20% lesting sets. First, we trained the DGMs on the real training data to produce synthetic data. The real testing set served a critical role in assessing the generalizability of trained ML models on unscen data. Subsequently, for TSTR, we trained various ML methods inchuding logistic regression (LG), support vector machine (SVM), random forest (RF), bagging (bootstrap aggregating) on top of LG, and XGBoost independently on both the real and synthetic training sets. In the augmentation task, we
9

---

Published in Transactions on Machine Learning Research (12/2025)
trained the same ML models independently both the real training set and combined set consisting of real training and synthetic data.
3.4 Syper-parameter Search
Hypper-paraneters play a pivotal role in tailoring MI. methods and DGMs to specific datasets and achieving optimal performance. To systematically optimize the prpee-parameter, we employed BO, which was introduced in Section 3.2. Specifically, we considered both SBO (baselinc) and the proposed IORBO variant for multi-ojjective aggregation. In practice, unless otherwise stated, all experiments are with IORBO, while SBO is included only for comparison.
We implemented BO using the TPE algorithm (Bergstra et al., 2011) within the Hyperopt' library, which efficienlyy explores black- box functions to identify optimal pappe--rr configurations. This approach enabled us effectively navigate the complex hyper-parameter space and select suitable settings for each experiment.
We conducted two distinct tuning processes. First, each ML method used in ML TSTR and augmentation evaluation (Figure 2 and Step 2 and 3), was fine-tuned for each dataset using five-fold cross-validation on the ML evaluation metrics (Figure 4 in the Appendix). Second, we optimized the hyper-parameters for each combination DGM, dataset, and loss function (Figure 3 in the Appendix).
9r, Dj and Lx
θ O DiiD
Sufficient # triels
Suggest hyper- peper-parameters
Update Ganssian model
Bayesian optimization
Eyyer-parametenn
Iterative objective refinement
Pg.
Evaluation Stats./TSTPp AAug
Build
Train Ci
Generative model
Input generative model: 9;, dataset: Dj, loss function: Lx
Output optimal hyper-parameters for G. Dj and Cs
Figure 3: Hyper-parameter search for a single generative model.
3.5 Statistical Tests
In Table 3, the specifications are based on the commonly accepted interpretation of p-values hypothesis testing. A p-value less than or equal to 0.01 (p < 0.01) indicates that the result is highly significant, mcaning that the null hypothesis (0.01 < 0.05) indicates significant. results, where there is still reasonable level of evidence against the mull hypothesis, though not strong as for the highly significant results. For p-values greater than 0.05, we consider the result not to be significant, indicating insufficient evidence to reject the null hypothesis.
be rejected with high confidence. A p-value between 0.01 and 0.05
hhttpe://hyprropt.ger withppppppt.tt.tt.oo.eeo.cc..bet wwt.hiit........o.a.......
10

---

Published in Transactions on Machine Learning Research (12/2025)
Mp and Dj
Sufficient # trials
Suggest lyper-parametert
Update Gaussian model
Bayesian optimization
Hyper- prpee-parameters M,
Iterative objective refinement
Train M, on 5-fold CV
Evalnation
Build M₁
5-fold CV
ML algorithm
Input- ML algorithm: My. dataset: D,
Output optimal lypem-parameters for MM and DD
Figure 4: Hyper-parameter search for an ML algorithm.
Regarding the two-sided test, the Nemenyi post-hoc test used in our analysis is based on the Friedman test, which is a non-parametric test for repeated measures. The Nemenyi test performs pairwise comparisons between the groups following the Friedman test and a two-sided test. This means that the test evaluates whether the differences between the groups are statistically significant both directions, i.e. it considers whether one group is significantly better or worse than another group.
To compare loss functions across DGMs and datasets, used the Friodman test (Friedman, 1937; 1940) to rank the loss functions independently. For non-parametric analysis of repeatedl-eassres data, the Friedman test offers an alternative to the widely used sppeated-measure ANOVA (Fisher, 1919). We used the Friedman test with equivalence on two ML efficacy problems for test set predictions and statistical similarity between training and syuthetic data (detailed Section 3.6). Following Demšar (2006), we further explored significant. differences between methods using the Nemenyi post-hoc test (Nemenyi, 1963). Table 3 shows the p-values divided into three positive and three negative differences.
Table 3: Ranges of p-values and specification obtained from statistical tests.
Notation
Rank
Range of p-value
Specification
Better
S 0.01
Highly significantly better
+
Better
0.01 < S 0.05
Siguificantly better
Better
P > 0.05
Not significantly better
0
Worse
> 0.05
Not significantly worse
Worse
0.01 < p S 0.05
Significantly worse
Wonse
S 0.01
Highly significantly worse
3.6 Benchmarking Framework
Figure provides an overview of the proposed benchmarking framework consisting the following core components:
Generative models. DGMs are used to generate synthetic data. We evaluated models. Three models that leverage conditional GANs for data synthesis: CTGAN (Xu al., 2019), CTAB-GAN (Zhao al., 2021), and DP-CGANS (Sun et al., 2023). A model that combines Gaussian Copula with the CTGAN architecture:
11

---

Published in Transactions on Machine Learning Research (12/2025)
Vanilla loss function
Statistical tests
+ D..D
Train &i on Dj using C
Syuthetic data o,.
Evaluate X5.
Concatenate
Friedian test
Mr
Proposed loss function
G..D
Train G. on Dj using
Syuthetic data Xo.o
Evaluate XguD,
Concatenate
Nemenyi test
Figure Proposed benchmarking framework. The Gj, Dj. Ls and Mp denote a DGM, a dataset, a loss function, and an ML method, respectively. @* denotes the optimal set eyper-parameters. See Figure 3 and Figure 4 in the Appendix see how we determined D,,, and M,,D,
CopulaGAN. A model that utilizes VAËs (Kingma & Welling, 2013) for data generation: TVAE (Xu et al., 2019). Finally, model that employs DDPM: TabDDPM (Kotelnikov et al., 2023). To explore the impact of the conditional element, we additionally evaluated versions of CTGAN, CopulaGAN, and DP-CGANS with conditioning disabled. We also used two backbones for TabDDPM: simple multilayer perceptron (MLP) and a ResNet.
Custom loss function. During training, each evaluated DGM utilized either the custom loss function defined in Equation (for GAN models), the presented in Equation 10 (for TVAE model) or the one in Equation 11 (for TabDDPM model). We subsequently fixed B, and Ç to specific values of 0 positive value, resulting in two different experiments: vanilla loss function (C with 3 = c= 0) and the proposed loss function, L, with at least one non-zero
Statistical tests. We used the Friedman test on all evaluated metrics, followed by the Nemenyi post-hoc test detailed in Section 3.5 for comparative analyses. These analyses can be divided into three categories: (1) General-purpose loss function assesses which loss function- in the evaluated DGM) and the proposed propssd--erformss performs better for general applications; (2) Dataset-specifie
the vanilla (original loss function used
loss determines which loss function is more effective for each evaluated dataset; and (3) Method-specifie loss identifies the superior loss function for each evaluated DGM architecture. For each calegory, we based
the evaluations either statistical similarity, ML TSTR performance, ML augmentation performance, or combination of evaluated metries.
4 Theoretical guarantees
4.1 Correlation-aware loss function
We now analyze the thcoretical properties of the proposed correlation-awart loss function, including its stability during optimization ion. This analysis provides justification for its robustness practice. The following remark clarifies how we implement the correlation loss in a stochastic optimization setting.
Remark 4.1 (Optimization details). During the stochastic optimization of Coorrelation: the running estimates μJ (raw first moment) and o, (central second moment) are treated as fixed constants within each mini- batch. To ensure numerical stability and avoid division by zero, a small constant i.e., we use o, +e.
added the denominator,
To quantify how well matching the first H moments controls the overall density approximation error, we now state our main result.
Proposition 4.2 (Stability). Assume the slandardized synthetic data points
Ti.k j4
Zi, =
σ +
12

---

Published in Transactions on Machine Learning Research (12/2025)
have tails that decay at least rub-Gaussian; that is, there erist constants K, v > 0 such that
Pr(1zaa| 1) S Me -ff
for all > 0. Fir 8 10 and set
M
to
In
Then, with probubility at least 1 6, the gradient of Lcorrelation with respect to any Big satisfies
Ceorrelation
8 m(m - 1) Be
ts
where B is the batch size and 6 > is the smoothing constant.
Proof. To analyze the gradient of Ceorelation with respect to Bigy, consider a single pair (J.b). The gradient can be expressed
b e
4
0ğj Dæs.
m(m (gj.A gj.k)
dā.jj
Recall that the synthetic correlation term is given by:
1
Tz.j p1.
Lisk jAx 0
B
σ +
Since the statistics ūú and o, are treated constants during stochastic optimization (Remark 4.1), the derivative simplifies to:
Øgj,k
1 Lisk jAA
B
o, +e
0k +
Taking the absolute value and using the property lab| =|a|bb, we obtain:
IL eccorelattion
Li.k j4g
m(m 1) 19 9j.n
B(o, +
σk +
1
Ti.k 14
1) 19 Jghl
m(m
B c)
0% + C
Here, the inequality follows because the absolute value of a product the product of the absolute values.
Since the correlations bounded (i.e., lrrul 1 and l & 1, implying lgj. -5.el 2) and by the bounded data assumption we have
b ecorrelation
2
θz
m(m 1) B(ō, +
By the tail bound,
Pr ( (s) & Me -vth = 6.
Thus with probability at least 1 - we have |Z I to, and noting that o, + e 2 e 0, it follows that
i Lecrrclation
8 ts
m(m- 1) B
This completes the proof.
13

---

Then for each feature index j and moment order h, the partial derivative of Published in Transactions on Machine Learning Research (12/2025)
4.2 Distribution-aware loss function
We now establish theoretical guarantees for the proposed distribution-aware loss function.
Assumption 4.3 (Higl-Probability Moment Matching). For any confidence level 6, there exist constants (0) 2 0 such that, with probability least 1,
Pa ma| < Ba(d) for all = 0, 1........ H.
Proposition 4.4 (Numerical stability). Under Assumption 4.3, for any fized confidence level, 1 8, there erist constants (6) > such that, with probability at least 1 5, the empirical moments satisfy
| S (0) and | (a( (j m], I,... H.
m j=1k=1
((h) +
Edistribution
5 +
with respect the synthetic moment ) obeys, with probability at least 8,
Ö disitribbttion
4 (8)
as! (h)
he
Note: We do not allow e to go to zero. Instead, we treat as a small constant that maintains 5 + 22c That way, the bound remains finite and numerically reasonable.
Proof. Recall that the distribution-awwrr loss function defined as:
m
+c) S +
(1)
2
Edistribution
1
where smoothing term that ensures numerical stability by preventing division by zero.
To analyze the stability Cdistributions we compute its gradient with respect to the synthetic moment S(h);
@diistrbaatinn asyh
5KM +e
(-S())
5 +
+e
The smoothing term e 0 ensures that the denominator absolute value of this partial derivative, obtain:
s/xh) +e is bounded away from zero. Taking the
giN 5th
( - (1)) -(()
@disttibbttion as
2
5/x0 +e
h
(s) +
Under the high-probability bound in the Proposition 4.4, |s < (8) and (8)11 (d),), we have
-50 < Ba(),, 5/T + +e26. e.
14

---

2 BB(5) Published in Transactions on Machine Learning Research (12/2025)
Therefore, with probability at least 1 - 6,
@aiittibbution
2
4 (x
h 2²
he
Since the bound depends only h, e, and (x all of which are independent of the feature index t-hhis implies that the partial derivatives are uniformly bounded across all and h. This completes the proof.
Remark 4.5 (Empirical moment consistency). After training on N real samples, we can draw B synthetic samples from the learned model to estimate any feature moment. By the Law of Large Numbers:
N (ai - m3)
B
P ++ E [(x, - -1)"]]
aas - RB)) B--00
E - -m
N
N-
B
Since the model's moments cannot typically be computed in closed form, we estimate them by averaging over B synthetic samples. Larger values of B lead more accurate moment estimates. In practice, sctting B comparable to larger than N (e.g-, B E [N, 10N]) lypically yields sufficiently accurale moment estimates.
5
Experiments
5.1
Datasets
Two datasets come from the UCI Machine Learning Repository (Dua & Graff, 2017) (Adult and News) and feature tabular structures with separate columns for attributes and labels. Thirteen additional datasots were preprocessed and shared by Kotelnikov et al. (2023) including Abalone, Buddy, California, Cardio, Churn2, Diabetes-ML, Gesture, Higgs-Small, House-16h, Insurance, King, Miniboone, and Wilt. We sourced the remaining datasets from Kaggle² (Credit, Diabetes, Balanced Diabetes, and House). To investigate the proposed method's behavior on high-dimensional binary data in (Xu et al., 2019), we transformed the Modified National Institute of Standards and Technology dalabase (MNIST) dalaset (LeCun & Cortes, 2010). Specifically, we binarized the original 28 28 images, converted each sample into 784-dimensional vector, and added a label column. The images were then resized to 12 x 12, reducing them to 144-dimensional vectors. We refer to this dataset as MNIST12.
Table 4 provides a comprchensive overview of the datasots evaluated in this study. It includes a diverse set of datasets, encompassing various data types and tasks to thoroughly test the proposed methods. The datasets range from small, specialized datasets like Diabetes-ML with 768 rows and 8 continuous variables, to large, extensive datasets such Credit with 277 640 rows and 29 continuous variables. Tasks represented include regression, binary classification, and multiclass classification, showcasing the breadth of application scenarios covered. For instance, Abalone and California are used for regression lasks, while Adult, Cardio, and Churn2 are employed for binary classification tasks. Multiclass classification tasks are represented by datasets such as Buddy and MNIST12.
Additionally, the datasets exhibit range of characteristics in terms of the number of continuous and discrete variables. For example, Gesture has high number of continuous variables (32) with discrete variables, whereas Diabetes features a substantial number of discrete variables (21) with no continuous variables. The varied nature of these datasets allows for robust evaluation of the proposed methods across different types of data and tasks, providing insights into their generalizability and effectiveness. The inclusion of datasets with different characteristies, such as Higgs-Small with 28 continuous variables and MNIST12 with 144 discrete variables, ensures a comprehensive assessment of performance and applicability.
Our benchmarking framework evaluates the proposed methods across twenty diverse real-world tabular datasets, spanning range of scales to assess performance under varying conditions. These include small- scale dalasets with thousands of rows, as well as mid- to large-scale ones, such as the Credit dataset with
2https://wwx
http://wwwwagggleee.///aaaaaaaa cateegle..omddaaaa.e com/datasets
15

---

Published in Transactions on Machine Learning Research (12/2025)
Table Description of experimented datasets.
Dataset
# Rows
#Continuous
# #Discrete
Task
Abalone
177
Regression
Adult
48 88381
6
8
Binclass
Buddy
18834
4
5
Multiclass
California
20
8
0
Regression
Cardio
700000
5
6
Binclass
Churn2
10000
7
4
Binclass
Credit
277 640
29
0
Binclass
Diabetes
234 245
0
21
Binclass
Diabetes-ML
768
Binclass
Diabetes Bal.
69515
Binclass
Gesture
9873
Multiclass
Higgs-Small
049
28
0
Binclass
House
21 613
10
8
Regression
House-16h
22 784
16
0
Regression
Insurance
1 1338
3
3
Regression
King
21 613
17
3
Regression
Miniboone
060
50
0
Binclass
MNIST12
70.000
0
Multiclass
News
39644
Regression
Wilt
4839
5
0
Binclass
approximately 277 000 rows. This selection allows us test the methods on datasets representative of practical scenarios, though note that even larger datasets (e.g., millions of rows) common in industrial applications and warrant future exploration.
5.2 Implementation Details and Training
Implementation Detuils and Training. We implemented all DGMs (CTGAN, CTAB-GAN, DP-CGANS, CopulaGAN, TVAE, and TabDDPM) and the proposed losses using PyTorch 1.13. To ensure replicability, we maintained the DGMs' original framework structures and adopted the model parameters specified in their publications. We disabled conditional elements within evaluated DGMs by reimplementing their data samplers. This modification removed the conditional vector from the training process, effectively transforming them into unconditional DGMs. For all DGMs, we employed the Adam optimizer (Kingma & Ba, 2015). We used the proposed IORBO approach introduced in Section 3.2 to fine-tune the hyper-parameterss in two tuning processes (Section 3.4).
The experiments ran on a high-performance computing cluster equipped with NVIDIA A100 Tensor Core graphical processing units (GPUs) (40GB RAM each) and Intel(R) Xeon(R) Gold 6338 CPUs (256GB DDR4 RAM). Training time per model varied significantly by dataset and DGM, ranging from one hour to two weeks.
To accelerate the ML performance evaluation, we used the cuML library (Raschka et al., 2020). This library provides a Python API largely compatible with scikit-learn (Pedregosa et al., 2011) and allows seamless execution of traditional tabular MI. tasks on GPUs. We used scikit-learn for classification and regression metrics, scipy for statistical evaluation metrics, and scikit-posthocs for the statistical tests, ensuring consistency throughout the evaluation process.
16

---

Published in Transactions on Machine Learning Research (12/2025)
6 Results and Discussion
Loss function. To analyze the performance of the proposed loss function against the vanilla version, we employed the proposed benchmarking framework (Section 3.6) across four key tasks: statistical evaluation (Stat.), TSTR evaluation, augmentation evaluation (Aug.), and comprehensive evaluation (Comp.) combining all three. The statistical tests evaluated the performance of the proposed loss function compared to the vanilla loss. In addition, we define the win rate as the proportion of evaluated metrics where the proposed loss function exceeds the vanilla loss function, relative to the total number of metrics assessed. A win rate of indicates that the proposed loss function performed better than the vanilla version across all evaluated metrics, while a value of 0 signifies that performed worse in every metric. win rate greater than 0.5 indicates that the proposed loss function was "better" more often than was "worse." We also report standard errors for each metric, estimated from 1 000 bootstrap rounds.
It is important to note that throughout the reported experiments, the "proposed loss function" corresponds to the vanilla training objective augmented with the auxiliary correlation- and distribution-aware regularizer. This ensures that the results reffect improvements attributable to the added regularization term rather than replacing the underlying optimization criterion (see Section 3.1). Furthermore, unless otherwise stated, all experiments are conducted using the proposed IORBO for ppper-prramete search, with SBO included solely as baseline for comparison.
Table 5: Results of the Nemenyi post-hoe test and win rate (with standard error in parentheses) comparing the proposed against the vanilla loss function on all DGMs and datascts. Loss functions were evaluated for statistical similarity (Stat.), TSTR, augmentation (Aug.), and comprehensive evaluation (Comp.) combining all metrics. For details on value ranges, refer to Table 3.
Statistical Tests
Win Rate
Comparison
Stat.
TSTR
Aug.
Comp.
Stat.
TSTR
Aug.
Comp.
Propeeed vs. Vanilla
+-
++
0.481 (0.012)
0.611 (0.007)
0.551 (0.007)
0.567 (0.001)
Generul-purpose loss function. Table 5 presents the results of a comprehensive analysis comparing the performance of the proposed loss function against the vanilla loss function across all DGMs and datasets. The table highlights the influence of loss function selection for general purposes. First, the two loss functions performed statistically similarly (zero (0) in the "Stat." column in Table 5). However, this metric does not fully capture performance in downstream tasks. In contrast, in the ML TSTR evaluation, the proposed loss function significantly outperformed the vanilla version, with a win rate of 0.611 and a standard eгror of 0.007, suggesting that the proposed loss function better captures the complexities of real-world tabular data during synthetic data generation. Similarly, the augmentation evaluation consistently favored the proposed loss function (win rate 0.551), demonstrating its ability to enhance the performance of predictive models trained on mix of real and synthetic data. Finally, the comprchensive evaluation (win rate 0.567), which combined all prior evaluations, continues this trend, indicating the proposed loss function's potential to improve model generalizability. A possible reason for this superiority is that the proposed loss function provides a regularizing effect, which reduces overfitting on unseen data and positions it as a strong candidate for general-purpose use in generative modeling tasks.
Method-specific loss function. Table compares the performance of the proposed loss function against the vanilla loss functions across all datasets and different DGM selections. Models denoted with an asterisk (*) have disabled conditioning. For most models, the proposed loss function demonstrates significant improvements in ML TSTR performance and augmentation effectiveness. For instance, CTGAN, CTGAN* CopulaGAN, and DP-CGANS consistently show highly significant gains (++) in TSTR, augmentation, and comprehensive evaluation. For example, DP-CGAMS* achieved the highest win rate across the TSTR metric, 0.798, indicating that the proposed loss function significantly enhanced its ability to generate synthetic data that boosts downstream ML performance. Interestingly, the statistical similarity (Stat.) evaluation reveals no significant differences between the proposed and vanilla loss functions for most models, suggesting that both loss functions perform similarly in terms of generating senth data that statistically match the
17

---

Published in Transactions on Machine Learning Research (12/2025)
Table 6: Results of the Nemenyi post-hoe test and win rate (with standard error in parentheses) comparing the proposed against the vanilla loss function across various DGMs on all datasets. Evaluations include TSTR, augmentation (Aug.), statistical similarity (Stat.), and a comprehensive measure (Comp.) combining all evaluated metrics. Models denoted with asterisk (*) have disabled conditioning. For details p-value ranges, refer to Table 3.
Statistical Tests
Win Rate
Method
Stat.
TSTR
Aug.
Comp.
Stat.
TSTR
Aug.
Comp,
CTCAN
0
+
0.478 (0.034)
0.639 (0.020)
0.583 (0.021) 0.593 (0.014)
CTGAN*
0
4-1
4
0.459 (0.036)
0.726 (0.018)
0.611 (0.020)
0.640 (0.013)
TVAE
0
0.519 (0.034)
0.501 (0.021)
0.593 (0.020)
0.543 (0.013)
CopulaCAN
+-
4
0.491 (0.033)
0.633 (0.020)
0.541 (0.022)
0.577 (0.013)
CopulaGAX*
0
+-
4
++
0.447 (0.034)
0.684 (0.019)
0.554 (0.020)
0.595 (0.013)
DP-CGANS
0
+
0.500 (0.051)
0.669 (0.028)
0.683 (0.028)
0.651 (0.018)
DP-CGANS
0.587 (0.054)
0.798 (0.023)
0.538 (0.080)
0.656 (0.019)
CTAB-GAN
0
0.391 (0.033)
0.418 (0.020)
0.497 (0.020)
0.448 (0.014)
TARDDPM-NLP
0
0
++
0.516 (0.035)
0.617 (0.020)
0.482 (0.021)
0.545 (0.014)
MBBDDK--ReNNes
0.512 (0.033)
0.547 (0.021)
0.487 (0.021)
0.517 (0.013)
real data distributions. However, the CTAB-GAN model stands out as an exception, showing statistically significant decrease (--) in performance across most evaluations when using the proposed loss function. This result suggests that the CTAB-GAN may require a more specialized loss function or optimization strategy to fully benefit from the proposed approach. The comprchensive evaluation (Comp.), which combines all three metries, underscores the effectiveness of the proposed loss function on eight out of ten evaluated DGMs. Models including CTGAN, CopulaGAN, DP-CGANS, and their non-conditioned variants, consistently outperform the vanilla loss function with win rates exceeding 0.5. These results imply that the proposed loss function offers a well-rounded improvement across various aspects of synthetic data generation, specifically in terms of enhancing MI. utility and model augmentation performance.
Dataset-speciff loss function. Table 7 compares the proposed loss function to the vanilla loss function across various datasets on all DGMs. The results demonstrate the effectiveness of the proposed loss function across diverse datasets. The statistical tests reveal that the proposed loss function achieves statistically significant improvements in TSTR performance for 14 out of 20 datasets, as indicated by the total count of (+) and (++). Additionally, the proposed loss function exhibits a consistent advantage in augmentation (Aug.). Specifically, datasets such as Insurance and MNIST12 show marked improvements in win rates (0.7). Conversely, the proposed loss function shows variable performance in statistical similarity (Stat.) across datasets. While it significantly improves TSTR and augmentation tasks for many datascts, its impact on statistical similarity is less consistent, with some datasets like Cardio, Higgs-Small, and Miniboone exhibiting inferior results compared to the vanilla loss function. From Table we see that the proposed loss function demonstrates significant improvement over the vanilla, loss function in 15 out of 20 datascts, indicated by the comprehensive evaluation (Comp.) in the statistical tests column. Among the remaining datasets, four show no significant difference (0) and only one shows a scatist significant disadvantage (-).
Bayesian optimization method. The performance of the IORBO was compared to that of the SBO using two aggregation methods: mean and median aggregation. For each dataset, we fine-tuned each generative model (DGM) across two loss functions and employed three distinct BÒ approaches. To assess the performance of these methods, statistical tests were conducted, focusing comparing the effectiveness of each approach. The results are summarized in Table 8, which presents the win rates and standard errors, as well as the outcomes of the Nemenyi post-hoc test that compares the methods shown in the rows against those in the columns. The findings from the Nemenyy post-hoe test clearly show that the IORBO significantly outperforms both SBO-Mean and SBO-Median. Specifically, the win rates for the IORBO compared to SBO-Mean and SBO-Median were 0.591 and 0.561, respectively. These results demonstrate that the IORBO not only more effective but also more robust in handling datasets with metrics that have different units, which often pose challenges in opt tasks. The consistent superiority of TORBO highlights its potential a reliable
18

---

Published in Transactions on Machine Learning Research (12/2025)
Table 7: Results of the Nemenyi post-hoe test and win rate (with standard error in parentheses) comparing the proposed against the vanilla loss function across various datasets on all evaluated DGMs. Evaluations
include TSTR, augmentation (Aug.), statistical similarity (Stat.), and a comprehensive measure (Comp.) combining all three. For details on p-value ranges, refer to Table 3.
Statistical
Tests
Win Rate
Dataset
Stat.
TSTR
Aug.
Comp.
Stat.
TSTR
Aug.
Comp.
Abalane
0
+
0.594 (0.059)
0.633 (0.043)
0.375 (0.045) 0.523 (0.027)
Adult
0
0.538 (0.052)
0.622 (0.025)
0.553 (0.025)
0.582 (0.017)
Buddy
++
++
0.500 (0.051)
0.607 (0.022)
0.552 (0.022)
0.570 (0.014)
California
0
0
0.500 (0.045)
0.583 (0.044)
0.583 (0.043)
0.566 (0.027)
Cardio
++
+
++
0.387 (0.054)
0.577 (0.027)
0.590 (0.027)
0.560 (0.019)
Churn2
0
0.500 (0.059)
0.637 (0.025)
0.460 (0.026)
0.543 (0.018)
Cradit
0.500 (0.U57)
0.544 (0.027)
0.554 (0.030)
0.543 (0.018)
Diabates
0
++
++
0.413 (0.031)
0.620 (0.027)
0.533 (0.027)
0.557 (0.018)
Diabetes-NL
++
0.469 (0.057)
0.719 (0.029)
0.479 (0.033)
0.584 (0.020)
Diabetes Bal.
++
0.438 (0.032)
0.717 (0.022)
0.538 (0.025)
0.605 (0.016)
Gesture
0.609 (0.056)
0.562 (0.032)
0,450 (0.029)
0.518 (0.021)
Higga-Snall
0.359 (0.055)
0.575 (0.031)
0.635 (0.032)
0.576 (0.021)
House
0.438 (0.049)
0.667 (0.039)
0.667 (0.038)
0.618 (0.025)
House-16h
0.500 (0.044)
0.442 (0.044)
0.500 (0.046)
0.477 (0.027)
Insurance
0.494 (0.052)
0.693 (0.037)
0.700 (0.037)
0.654 (0.025)
King
0.519 (0.055)
0.673 (0.039)
0.567 (0.039)
0.599 (0.026)
Miniboone
0.359 (0.054)
0.402 (0.031)
0.512 (0.026)
0.446 (0.020)
MNIS112
+
0.484 (0.040)
0.756 (0.023)
0.700 (0.024)
0.699 (0.016)
News
0.612 (0.052)
0.607 (0.040)
0.553 (0.042)
0.587 (0.024)
Wilt
0
0.469 (0.056)
0.538 (0.025)
0.552 (0.026)
0.536 (0.016)
and broadly applicable BÒ method, suggesting it could be a valuable tool in a wide range of optimization tasks involving diverse types of data and models.
Table 8: Results of the Nemenyi post-hoc test and win rate (with standard error in parentheses) comparing
the row to column method. For details on valuee ranges, refer to Table 3.
Statistical Tests
Win Rate
BO method
SBO-Mean
SBO- 3BO-Median
IORBO
SBO-Mean
SBO Median
IORBO
SBO-Mean
0.461 (0.004)
0.409 (0.001)
SBO-Median
0.539 (0.004)
0.439 (0.004)
IORBO
0.591 (0.004)
0.561 (0.004)
Bayesian optimization computational cost. Figure compares the per-iteration computational cost of IORBO and SBO across different numbers of evaluated metrics. Each iteration measures the time required to update the surrogate model (see Algorithms 1 and 2). The analysis shows that both TORBO and SBO have the same asymptotic computational complexity, scaling linearly with the number of metrics. As shown in the timing comparison, execution times for both methods largely overlap across metric counts of 1, 10, 30, and 100, with observed differences remaining negligible. Thus, IORBO achieves its improved optimization performance without incurring additional computational cost practice.
Ablation studies. The ablation study results presented in Table 9 evaluate the impact of different loss function variants combined with two optimization strategies: SBO and IORBO (denoted IOR). In our experiments, the loss function variants are defined as follows: V represents the vanilla loss; C denotes the addition of the correlation loss; D indicates the addition of the distribution loss; and CD corresponds to the combination of both correlation and distribution losses (i.., the proposed loss function). The table is organized with comparisons vs. Mean and vs. Median to succinctly present the performance metrics.
In the vs. Mean comparison, adding the correlation loss (C) to the vanilla setting (SBO-V) increases the win rate from 0.500 to 0.563, making it the most impactful individual component. In contrast, adding only
19

---

Published in Transactions on Machine Learning Research (12/2025)
# Metrics 1
#Metrics 10
0.010
IORBO
IORBO
Time (seconds)
SBO
Time (seconds)
0.06
0.008
0.006
0.04
0.004
0.002
0.00
Metrics 30
# *Metrics 100
0.20
IORBO
IORBO
Time (seconds)
SBO
Time (seconds)
SBO
0.15
0.4
0.10
0.2
0.05
10
15
20
25
30
35
40
45
5
10
15
20
25
30
35
40
45
Iteration
Iteration
Figure Per-iteration computational time of IORBO versus standard SBO across different numbers of evaluated metrics. Both methods show nearly identical timings, confirming that IORBO introduces significant computational overhead.
the distribution loss (D) results in smaller improvement from 0.500 to 0.508. However, the distribution loss should not be underestimated- targets the overall data distribution, ensuring that the synthetic data not only capture pairwise correlations but also match higher-order statistical properties. When both loss components are combined in SBO-CD, the win rate reaches 0.542, indicating that the distribution loss provides complementary information that can enhance robustness and generalization, even though its impact is most pronounced when used in tandem with the correlation loss.
The effect of IOR-based optimization is more substantial. Replacing SBO with IOR (IOR-V) boosts the win rate from 0.500 to 0.572, already surpassing most SBO variants except SBO-C. Further adding correlation loss (IOR-C) significantly improves performance, increasing the win rate from 0.572 to 0.636. Although adding only the distribution loss (IOR-D) results in a modest increase to 0.581, its contribution becomes crucial when combined with the correlation loss: the full combination (IOR-CD) achieves win rate of 0.623, which, while slightly lower than TOR-C alone, offers more consistent performance across both vs. Mean and vs. Median evaluations. Notably, 1OR-CD maintains stable win of 0.522 in both comparisons, suggesting that the distribution loss enhances the overall robustness and consistency of the optimization process.
A similar pattern emerges in the vs. Median results. The addition of C to SBO-V improves performance from 0.500 to 0.558, while adding D results in a smaller gain to 0.514. The combined loss SBO-CD achieves 0.534, again slightly lower than SBO-C, which underscores that while the correlation loss the dominant. factor in boosting win rates, the distribution loss contributes to a more reliable performance. When switching to IOR, IOR-V increases the win rale from 0.500 to 0.546, and adding C further boosts performance to 0.599, making it the mest-performing individual term. Although adding D alone results in slight drop to 0.543, the full TOR-CD configuration achieves 0.5ii,, slightly lower than IOR-C alone but with the added benefit. of improved generalization across evaluation criteria.
Overall, these results highlight three key findings. First, among the two loss components, the correlation loss (C) has the strongest impact on performance. Sccond, while the distribution loss (D) provides only modest gains when used alone, plays crucial complementary role when combined with C, ensuring that both pairwise relationships and higher-order statistical properties are effectively captured. Third, TOR-based
20

---

Published in Transactions on Machine Learning Research (12/2025)
Table 9: Ablation study results of the Nemenyi post-hoc test and win rate (with standard error in parentheses) comparing row to column methods. The table presents performance across different configurations using baseline methods with SBO and IORBO (denoted IOR). The loss function variations are as follows: V represents the vanilla loss, C adds the correlation loss, D adds the distribution loss, and CD adds both distribution and correlation losses, forming the proposed loss function. Both Mean and Median aggregation strategies are evaluated. For details on p-value ranges, refer to Table 3.
Statistical Tests
vs. Mean
SBO-V
JOR-V
SBO-C
JOR-C
SBO-D
1OR-D
SBO-CD
JOR-CD
SBO-V
IQR-V
44
4-
1-
0
SRO-C
++
0
IOR-C
SBO-D
IOR-D
SBO-CD
JOR-CD
Win Rate
vs. Mean
SIO-V
IOR-V
SBO-C
IOR-C
SIO-D
1ORLD
SBO-CD
IOR-CD
SBO-V
0,128 (0.005)
0.437 (0.005)
0.364 (0.005)
0.492 (0.005)
0.119 (0.005)
0.458 (0.005)
0.377 (0.005)
TOR-V
0.572 (0.005)
0.538 (0.005)
0.451 (0.005)
0.579 (0.005)
0.526 (0.005)
0.547 (0.005)
0.427 (0.005)
SBO-C
0.563 (0.004)
0.162 (0.0055
0,400 (0.005)
0.555 (0.005)
0.157 (0.005)
0.520 (0.005)
0.309 (0.005)
IOR.C
0.636 (0.005)
0.549 (0.005)
0.600 (0.005)
0.639 (0.005)
0.564 (0.005)
0.614 (0.005)
0.478 (0.005)
SBO-D
0.508 (0.005)
0.121 (0.005)
0.145 (0.005)
0.361 (0.005)
0.125 (0.005)
0.135 (0.005)
0.375 (0.005)
IOR-D
0.581 (0.005)
0.474 (0.005)
0.543 (0.005)
0.438 (0.005)
0.575 (0.005)
0.544 (0.005)
0.429 (0.005)
SBO-CD
0.542 (0.005)
0.153 (0.005)
0.180 (0.005)
0,386 (0.005)
0.545 (0.005)
0.156 (0.005)
0.390 (0.005)
1OR-CD
0.623 (0.005)
0.573 (0.005)
0.601 (0.005)
0.522 (0.005)
0.625 (0.005)
0.571 (0.005)
0.610 (0.005)
Statistical Tests
va. Median
SBO-V
JOR-V
SBO-C
IOR-C
SBO-D
1OR-D
SBO-CD
IOR-CD
SBO-V
IOR-V
0
0
SBO-C
IOR-C
SBO-D
JOR-D
++
0
D
SBO-CD
++
TOR-CD
Win Rate
Median
SBO-V
10R-V
SBO-C
IOR-C
SBO-D
1OR-D
SBO-CD
JOR-CD
SBO-V
0.154 (0.005)
0.112 (0.005)
0.101 (0.005)
0.186 (0.005)
0.157 (0.005)
0.166 (0.005)
0.114 (0.005)
1OR-V
0.546 (0.005)
0.195 (0.005)
0.151 (0.005)
0.541 (0.005)
0.526 (0.005)
0.517 (0.005)
0.127 (0.005)
SBO-C
0.558 (0.005)
0.505 (0.005)
0.442 (0.005)
0.558 (0.005)
0.514 (0.005)
0.524 (0.005)
0.450 (0.005)
JOR-C
0.599 (0.005)
0.549 (0.005)
0.558 (0.005)
0.593 (0.005)
0.564 (0.005)
0.562 (0.005)
0.478 (0.005)
SBO-D
0.514 (0.005)
0.459 (0.005)
0.442 (0.005)
0.407 (0.005)
0.458 (0.005)
0.474 (0.005)
0.409 (0.005)
1OR.D
0.543 (0.005)
0.474 (0.005)
0.486 (0.005)
0.436 (0.005)
0.542 (0.005)
0.506 (0.005)
0.429 (0.005)
SBO-CD
0.534 (0.005)
0.480 (0.005)
0.476 (0.005)
0.438 (0.005)
0.526 (0.005)
0.494 (0.005)
0.426 (0.005)
IOR-CD
0,586 (0.005)
0.573 (0.005)
0.550 (0.005)
0.522 (0.005)
0.591 (0.005)
0,571 (0.005)
0.574 (0.005)
optimization consistently enhances performance across all variants, with the best overall results achieved using the full combination (IOR-CD), which balances high win rates with robust generalization.
While our work focuses on tabular datasets, note that the proposed losses could potentially be applied to embeddings derived from images or text. Such embeddings capture feature-level structure in way that preserves meaningful correlations and distributions, making them suitable targets for correlation- and distribution-awwree regularization. Exploring this direction represents a promising avenue for future research.
A key limitation of our current evaluation that it focuses on datasets up to approximately 277,000 rows, whereas real-wor world tabular applications often involve millions of samples. This limitation is primarily due to computational time constraints rather than the scalability of the methods themselves. Specifically, training
21

---

Published in Transactions on Machine Learning Research (12/2025)
DGMs and performing extensive hyper-parameter optimization across multiple baselines becomes prohibitively time-consuming on very large datasets. In our super-computing environment, we impose 7-day maximum training time per experiment, which effectively caps the dataset sizes we can explore. Theoretically, our correlation- and distribution-aware loss functions scale lincarly with the number features (m) and batch size (B); with time complexities of O(m² B) for the correlation term (due lo pairwise computations) and B) for the distribution term (where H the number of moments). Since these losses operate on mini-batches rather than the full dataset, they remain efficient as dataset size grows. Similarly, IORBO adds negligible overhead, with its ranking and surrogate refitting steps scaling, which is minor compared to DGM training costs. We expect our approach to behave robustly on larger scales: the losses should continue to enforce statistical fidelity effectively and may offer even greater benefits in high-volume scenarios where capturing complex dependencies is more challenging. Nonetheless, practical bottlenekks particularly computational time, memory requirements for large batches, and the need for distributed training may limit experiments on extremely large datascts.
7 Conclusion
We presented a unified framework for tabular generative modeling that integrates training. tuning, and evaluation. While tightly integrated, each component is also designed to be applied independently, making the framework modular. First, we introduced a novel correlation- and distribbution-aware loss function designed regularizer for DGMs in tabular data synthesis, which outperforms the vanilla loss function across most DGMs. To ensure its robustness, we provided thcoretical guarantoes, including stability and consistency. The results suggest that the proposed loss function effectively captures the complexities of arbitrary DGMs. Future research could focus on developing a tailored loss function for the CTAB-GAN family to match the strong performance seen with other DGMs. Second, we introduced a novel IORBO approach that leverages rank-based aggregation to ensure more meaningful comparisons between multiple objectives with varying units, providing more robust optimization process. Last, we developed a comprehensive benchmarking system evaluating statistical similarity, ML TSTR performance, and ML augmentation performance, with robust statistical tests, offering valuable tool for future rescarch.
Acknowledgements
The computations and data handling were enabled by resources provided by the National Academic Infras- tructure for Supercomputing in Sweden (NAISS) and the Swedish National Infrastructure for Computing (SNIC) at the Uppsala Multidisciplinary Center for Advanced Computational Science (UPPMAX), partially funded by the Swedish Research Council under grant agreements no. 2022-06725 and no. 2018-05973. This work was further supported by the WASP-DDLE postdoctoral grant Visualization and de- do-eetntiiinnoon of Biobank Data Propel Precision Medicine Research (KAW 2023-03705), by the Swedish Cancer Foundation (24 3406 Pj 01 H; 21 1384 Pj 01 H), and by Umeå University Infrastructure funding (FS 2.1.6-189--24).
References
Martin Arjovsky, Soumith Chintala, and Léon Bottou. Wasserstein generative adversarial networks. In International Conference on Machine Learning, 2017.
Karim Armanious, Chenming Jiang, Mare Fischer, Thomas Küstner, Tobias Hepp, Konstantin Nikolaou, Sergios Gatidis, and Bin Yang. MedGAN: Medical image translation using GANs. Computerized Medical Imaging and Graphics, 79:101684, 2020.
Laura Aviñó, Matteo Rulfini, and Ricard Gavaldà. Generating Synthetic but Plausible Healthcare Record Datasets. In KDD workshop on Machine Learning for Medicine and Healthcare, 2018.
James Bergstra, Rémi Bardenet, Yoshua Bengio, and Balázs Kégl. Algorithms for Hyper-Paranenn Opti- mization. In Advances in Neural Information Processing Systems, pp. 2546-2554, 2011.
22

---

Published in Transactions on Machine Learning Research (12/2025)
Ramiro Camino, Christian Hammerschmidt, and Radu State. Generating Multi-Cant Categorical Samples with Generative Adversarial Networks. In ICML workshop on Theorelical Foundalions and Applications of Deep Generative Models, 2018.
Janez Demšar. Statistical comparisons of classifiers over multiple data sets. Journal of Machine Learning Research, 7:1-30, 2006. ISSN 15337928.
Anna Veronika Dorogush, Vasily Ershov, and Andrey Gulin. Botst gradient boosting with categorical features support. arXiv preprint arXi::1810.1183 2018.
Dheeru Dua and Casey Graff. UCI Machine Learning Repository, 2017. URL http://archiie. ics uci edu/ml.
RA Fisher. -he Correlation between Relatives the Supposition of Mendelian Inheritance. Transactions of the Royal Society of Edinburgh, 52(2):399 433, 1919.
Bruce Frey. The SAGE Encyclopedia of Educational Research, Measurement, and Evaluation. SAGE Publications, 2018.
M Friedman. The use of ranks to avoid the assumption of normality implicit in the analysis of variance. Journal of the American Statistical Association, 1937.
Milton Friedman. Comparison of Alternative Tests of Significance for the Problem of m Rankings. The Annals of Mathematical Statistics, 11(1):86 92, 1940.
Sylvia Frühwwrth-Schnatter Gilles Celeux, and Christian P Robert. Handbook of Mixture Analysis, 2018.
A Goncalves, P Ray, B Soper, J Stevens, L. Coyle, and AP Sales. Generation and evaluation of synthetic patient data. BMC Medical Research Melhodology, 20(1):108 108, 2020.
Ian Goodfellow, Jean Pouget-Abadie, Mchdi Mirza, Bing Xu, David Warde-Farley Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative Adversarial Nets. Z. Ghahramani, M. Welling, C. Cortes, N. Lawrence, and K.Q. Weinberger (eds.), Advances in Neural Information Processing Systems, volume 2, pp. 2672 2680. Curran Associates, Inc., 2014.
Ishaan Gulrajani, Faruk Ahmed, Martin Arjovsky, Vincent Dumoulin, and Aaron C Courville. Improved training of wasserstein gans. In Advances in Neural Information Processing Systems, 2017.
John R Hershey and Peder A Olsen. Approximating the Kullback Leibler divergence wet Gaussian mixture models. In 2007 IEEE International Conference Acoustics, Speech and Signal Processing-CCASSPP00 volume 4, pp. IV-317. IEFE, 2007.
Alexia Jolicoeur-Martincann gradient-boosted trees. In International Conference on Artificial Intelligence and Statistics, pp. 1288 1296. PMLR, 2024.
diffusion and Martincau, Kilian Fatras, and Tal Kachman. Generating and imputing tabular data via
Tero Karras, Samuli Laine, Mika Aittala, Janne Hellsten, Jaakko Lehtinen, and Timo Aila. Analyzing and improving the image quality of stylegan. In IEEE Conference Computer Vision and Pattern Recognition (CVPR), pp. 8107 8116, 2020.
Tero Karras, Mika Aittala, Samuli Laine, Erik Härkönen, Janne Hellsten, Jaakko Lehtinen, and Timo Aila. Alias-Free Generative Adversarial Networks. Advances in Neural Information Processing Systems, 34: 852-863, 2021.
Diederik P Kingma and Jimmy Ba. Adam: A Method for Stochastic Optimization. In 7th International Conference on Learning Representations (ICLR), 2015.
Diederik P Kingma and Max Welling. Auto-encoding variational bayes. In International Conference on Learning Representations, 2013.
23

---

Published in Transactions on Machine Learning Research (12/2025)
Akim Kotelnikov, Dmitry Baranchuk, Ivan Rubachev, and Artem Babenko. Tabddpm: Modelling tabular data with diffusion models. In International Conference on Machine Learning, pp. 17564-17579) PMLR, 2023.
S Kullback and RA Leibler. On Information and Sufficieney. The Annals of Mathematical Statistics, 22(1): 79-86, 1951.
Yann LeCun and Corinna Cortes. MNIST handwritten digit database, 2010. URL http://yamm.eecun.co exdb/mnist/.
Yingzhou Lu, Minjie Shen, Huazheng Wang, Xiao Wang, Capucine van Rechem, and Wenqi Wei. Machine learning for synthetic data generation: a review. arXiv preprint (xXi:2332204002 2023.
Chao Ma, Sebastian Tschiatschek, Richard Turne,, José Miguel Hernández-Lobato, and Cheng Zhang, VAEM: a deep generative model for heterogencous mixed type data. In Advances in Neural Information Processing Systems, volume 33, pp. 11237-11247, 2020.
Frank J Massey Jr. The Kolmogorov-Snirrov Test for goodness of fit. Journal of the American Statislical Association, 1951.
Alfredo Nazabal, Pablo M Olmos, Zoubin Ghahramani, and Tsabel Valera. Handling incomplete heterogeneous data using VAEs. Pattern Recognition, 107:107501, 2020.
Peter Bjorn Nemenyi. Distribution--fee multtiple comporisons. Princeton University, 1963.
Neha Patki, Roy Wedge, and Kalyan Vecramachaneni. The Synthetic Data Vault. In International Conference Data Science and Advanced Analytics. IEEE, 2016.
Karl Pearson. Method of moments and method of maximum likelihood. Biometrika, 28(1/2):34 59, 1936.
Karl Pearson. On the Criterion that a Given System of Deviations from the Probable in the Case of a Correlated System of Variables is Such that it Can be Reasonably Supposed to have Arisen from Random Sampling. Breakthroughs in Statistics: Methodology and Distribution, pp. 11-28, 1992.
F. Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel, P. Prettenhofer, R. Weiss, V. Dubourg, J. Vanderplas, A. Passos, D. Cournapeau, M. Brucher, M. Perrot, and E. Duchosnay. Scikit-learn: Machine Learning in Python. Journal of Machine Learning Research, 12:2825 2830, 2011.
Tker Peis, Chao Ma, and José Miguel Hernández-LLoatto Missing data imputation and acquisition with deep hierarchical models and Hamiltonian Monte Carlo. In Advances in Neural Information Processing Systems, volume 35, pp. 15839935851. 2022.
Scbastian Raschka, Joshua Patterson, and Corey Nolct. Machine Learning in Python: Main developments and technology trends in data science, machine learning, and artificial intelligence. arXiv preprinl arXiv:2002.0,58 04800, 2020.
John A Rice. Mathematical statistics and data analysis, volume 371. Thomson/Brrooks/CGe Coole Belmont, CA, 2007.
Jascha Sohl-Dickstein, Eric Weiss, Niru Maheswaranathann, and Surya Ganguli. Deep Unsupervised Learning using Nonequilibrium Thermodynamics. In International Conference on Machine Learning, pp. 2256-2265, PMLR, 2015.
Baochen Sun, Jiashi Feng, and Kate Saenko. Correlation Alignment for Unsupervised Domain Adaptation. Domain Adoptation in Computer Vision Applications, pp. 153, 2017.
Chang Sun, Johan van Soest, and Michel Dumontier. Generating synthetic personal health data using conditional generative adversarial networks combining with differential privacy. Journal of Biomedical Informatics, pp. 104404, 2023.
24

---

Published in Transactions on Machine Learning Research (12/2025)
Gemini Team, Rohan Anil, Sebastian Borgeaud, Yonghui Wu, Jean- -rann-aptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, et al. Gemini: family of highly capable multimodal models. arXiv preprint arXiv:2312.11800 2023.
Minh H Vu, Tommy Löfstedt, Tufve Nyholm, and Raphael Sznitman. A Question-Centric Model for Visual Question Answering in Medical Imaging. IEEE Transactions Medical Imaging, (9(9):2856-286) 2020.
Minh H Vu, Gabriella Norman, Tufve Nyholm, and Tommy Löfstedt. A Data-Adaptive Loss Function for Incomplete Data and Incremental Learning in Semantic Image Segmentation. IEEE Transactions on Medical Imaging, 41(6):1320 1330, 2021.
Lei Xu, Maria Skoularidou, Alfredo Cuesta-Infante, and Kalyan Veeramachaneni. Modeling tabular data using conditional GAN. In Proceedings of the 33rd International Conference on Neural Information Processing Systems, pp. 7335-7345, 2019.
Alexandre Yahi, Rami Vanguri, Noémie Elhadad, and Nicholas P Tatonetti. Generative Adversarial Networks for Electronic Health Records: A Framework for Exploring and Evaluating Methods for Predicting Druu-Inducedd Laboratory Test Trajectories. NIPS workshop on machine learning for health care, 2017.
Lantao Yu, Weinan Zhang, Jun Wang, and Yong Yu. Seqgan:
Sequence generative adversarial nets with
policy gradient. In AAAI Conference on Artificial Intelligence, 2017.
EL Hacen Zein and Tanguy Urvoy, Tabular data generation: Can we fool xgboost? In NewrIPS 2022 First Table Representation Workshop, 2022.
Jun Zhang, Graham Cormode, Cecilia M Procopiue, Divesh Srivastava, and Xiaokui Xiao. Privbayes: Private data release via bayesian networks. ACM Transactions on Database Systems, 42(4):25, 2017.
Zilong Zhao, Aditya Kunar, Robert Birke, and Lydia Y Chen. CTAB-GAN: Effective Table Data Synthesizing. In Asian Conference on Machine Learning, pp. 97 112. PMLR, 2021.
Jun-Yan Zhu, Tacsung Park, Phillip Isola, and Alexii A. Efros. Unpaired Imago-to-Image Translation Using Cycle- Consistent Adversarial Networks. In 2017 IEEE International Conference Computer Vision (ICCV), pp. 2017.
25

---

Published in Transactions on Machine Learning Research (12/2025)
A Hyper-Pane Param-err Search Spaces
Table Logistic Regression search space for classification dataset.
Parameter
Distribution
LogUniform (-4,4) 4)
max iter
IntUniform (50, 200)
11 rrtaoo
Uniform (0, 1)
algorithm
("svd", "eig", "qr", "svd-qr", "svd-jacobi") ("newton-cg" "Ibfgs", "liblinear", "sag", "saga") ("balanced", None)
solver
class haas__weight
number of tuning iterations
Table 11: FlasticNet search space for regression dataset.
Parameter
Distribution
alpha max iter
Uniform (1, 10)
IntUniform (100, 2000)
11 ratio tol
Uniform (0,1)
LogUniform (10-5, 10-1)
fit intercept
(True, False)
normalize
(True, False]
number of tuning iterations
30
Table 12: Bagging for Logistic Regression search space for classification dataset.
Parameter
Distribution
LogUniform (-4,4)
max iter
Uninfirr (50,200)
ratio algorithm
Uniform (0,1) {svv"" "eig", "qr", "svd-qr", "svd-jacobi") ["qn"} ("balanced", None}
solver
class_ _eight
number of tuning iterations
30
For hyper-parameter search related to TabDDPM, please refer to the work by Kotelnikov et al. (2023).
26

---

Published in Transactions on Machine Learning Research (12/2025)
Table 13: Bagging for ElasticNet search space for regression dataset.
Parameter
Distribution
alpha max iter
Uniform (1, 10)
IntUniform (100, 2000)
11 ratio tol
Uniform (0, 1)
LogUniform (10-5, 10=²)
fit intercept
('True, False]
normalize
(True, False]
number tuning iterations
30
Table 14: SVM search space for classification dataset (LincarSVC).
Parameter
Distribution
C
LogUniform (0.1, 10)
max iter
IntUniform (100, 1500)
tol
LogUniform (-5,-1) -1)
penalty
("hinge", "squared hinge")
loss
(True, False)
fit. intercept
(True, False]
penalized intercept
(True, False)
blass_weight
{ ("balanced", None}
number of tuning iterations
30
Table 15: SVM search space for regression dataset (LinearSVR).
Parameter
Distribution
C
LogUniform (0.1, 10)
max iter tol epsilon
IntUniform (100, 1500)
LogUniform (-5,
Uniform (0, 1)
fit. intercept
(True, False]
penalized intercept
(True, False)
number of tuning iterations
30
Table 16: RF search space for classification and regression dataset (RandomForestClassfferr and RandomFort Fore- st.Rerrssor)..
Parameter
Distribution
estimators
IntUniform (50, 500)
max__deple samplessmsess____ split min min jin__sampes__eed leaf max features
Int Uniform (10, 100)
Int Uniforrm (2, 20)
Uniforr (1, (0)
["sqrt", "log2")
number tuning iterations
27

---

Published in Transactions on Machine Learning Research (12/2025)
Table 17: XGBoost search space for classification and regression dataset (XGBClassifier and XGBRegressor)
Parameter
Distribution
n estimators
Intunifrrm (50, 500)
max max_depth
IntUniform (3, 15)
learning rate
Uniform (0.01,0.3)
subsample
Uniform (0.5, 1)
colsample__yye bytree
Uniform (0.5, 1)
gamma
Uniform (0,5)
reg alpha
Uniform (0, 1)
reg lambda
Uniform (0,
saale_pos_wwettee weigght
Uniform 10)
number of tuning iterations
30
Table 18: CTGAN, CopulaGAN and DP-CGANS search space.
Parameter
Distribution
epochs batch_ size embedding dim generator_ _im discriminator dim generator Leaarnior_learning rate
UntUniform (100, 100)
IntUniform (500, 30000,00000 100)
(32, 64, 128, 256}
{32, 64, 128, 256}
(32, 64, 128, 256)
Uniform (10-5, 1000""00000000000 10-33
generator_decat
Uniform (10 -7 10100
discriminator tetttmiinnttr__eannnnnngoooo rate
Uniform (10 (10-5 50000
discriminator _eciratttrreeayyyeeeeee
Uniform (10 (10-7,10**
a
Uniform (10-2, 2,0))
8
Uniform (10-10; 110)
number of moments
(1,2,3,4)
number of tuning iterations
30
Table 19: TVAE search space.
Parameter
Distribution
epochs batch_ size embedding dim
IntUniform (100, 2 2000, 100) IntUniform (500, 300, 100)
{32, 64, 128, 256)
compress dims
(32, 64, 128, 256]
decompress dim loss factor 12scale
(32, 64, 128, 256}
(0.25, 0.5, 1, 2, 4}
Uniform (10 10-4)
Uniform (10-2, 10
Ø number of moments
Uniform (10-10 10²)
(1,2,3,4)
number of tuning iterations
30
28

---

Published in Transactions on Machine Learning Research (12/2025)
Table 20: CTAB-GAN search space.
Parameter
Distribution
epochs
IntUniffrmm (100, 2 000, 100) Unffrio (500, 400,4000 000, 100)
batch size
test ratio
{0.1, 0.2, 0.3, 0.4, 0.53
c_class_ _ayer class dim
21,2,3.4) (32, 64, 128, 256)
random_ dim mum channels
32, 64, 128]
316,32, 64)
Uniform (10-2, 10*)
a
Uniform (10-10,10))
number of moments
(1,2,344
number of tuming iterations
30
29


Reviews:
[{"review_id": "1V8OMJar7S", "reviewer": "Reviewer_zMUy", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "### Strengths\n1. Paper is written clearly and thoroughly.\n2. Proposed approach is easy to apply for different DGMs\n3. Deep evaluation: authors evaluated their approach on various datasets, demonstrating that IORBO provides advantages over the SBO\n4. Theory for both correlation-aware and distribution-aware losses, providing bounds that could be used for improving training behaviour\n5. Work demonstrates practical relevance for the tabular workflows and synthetic data generation for tabular data\n\n### Weaknesses/Questions\n\n1. Correlation loss ignores mixed-type dependencies. In the evaluation authors use Pearson correlation for continuous-continuous variables, and Cramer V for categorical-categorical variables, but what about continuous-categorical variables? It would be beneficial to clarify that in the manuscript.\n2. Correlation-aware and distribution-aware loss prior works: I believe it would be very beneficial to the paper to discuss prior works in the area of statistical/correlation regularisers (for example [1]). Ideally, even using some of the similar approaches for the comparison, as currently comparisons are drawn with the standard baselines.\n\n[1] Sun, Baochen, Jiashi Feng, and Kate Saenko. \"Correlation alignment for unsupervised domain adaptation.\" Domain adaptation in computer vision applications. Cham: Springer International Publishing, 2017. 153-171.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper targets the practical bottleneck in tabular synthetic data generation with deep generative models (DGMs) -- inability to capture the complexities of real-world tabular data. In order to mitigate this, authors propose novel additions to the loss function of DGMs: a correlation and distribution-aware regularizers for DGMs. Authors propose multi-objective aggregation strategy: Iterative Objective Refinement Bayesian Optimization (IORBO) and provide algorithm as well as the demonstration of its application to the toy example. Authors provide a benchmarking of different methods, illustrating the utility of the proposed approach, demonstrating the significant improvement of the fidelity of the synthetically generated data.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "The overall state of the paper is good, and I believe it meets all the requirements for acceptance. It would be beneficial to address the points raised in the Weaknesses/Questions section.", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "Ebvdnalt5M", "reviewer": "Reviewer_kBgG", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths**\n\n* As generative models for tabular data gain traction, addressing their robustness is both timely and highly relevant.\n* The proposed method is conceptually simple, with broad applicability across various tabular data models.\n* The paper is clear and easy to follow.\n\n**Weaknesses**\n\n* The paper lacks a clear unifying theme. Its three contributions — loss regularization, hyperparameter tuning, and evaluation — feel somewhat disconnected, with limited explanation of how they relate to each other. For example, the Bayesian optimization component could apply to any model involving multiple metrics, without any specific link to tabular data.\n* In theory, the proposed correlation- and distribution-based losses could be applied to data types beyond tabular data. The main reason this might not work well for more structured data — such as images — is that the underlying distributions in pixel space are too complex, making correlations less meaningful. I suggest the authors clarify this point in the paper to better justify why their method is specifically suited to tabular data. Additionally, these losses could potentially be applied to image or text embeddings generated by (pre)trained encoders, which might extend the applicability of the method to other data modalities. A broader discussion would be appreciated. \n* The experiments are conducted on relatively small tabular datasets, whereas real-world applications often involve millions of samples. This is a crucial limitation, as the size of the training data can significantly impact the behavior and performance of generative models. It also raises the question of how the proposed approaches scale and perform as the dataset size increases — an aspect not addressed in the current evaluation.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "In this paper, the authors propose a new method to leverage generative models in the context of tabular data. They introduce additional loss regularizations — correlation- and distribution-based — that improve the quality of the generated synthetic data samples compared to other approaches used for tabular data. Namely, in addition to the generative loss function, they match correlations and distribution moments between real and generated data.\n\nThey also propose a convenient method to perform Bayesian optimization in the context of multi-dimensional targets, namely when one tries to optimize several metrics at once. The core idea is that instead of considering absolute metric values for a set of parameters, we can compare them in terms of relative rank value among each other, which makes the optimization targets adapt during each optimization step.\n\nFinally, they propose a comprehensive evaluation setup for synthetic data-based tabular models, such as statistical similarity of generated data to real data, and ML performance of the model trained on synthetic data.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "* Clarify the unifying motivation behind the three contributions. Explain how loss regularization, IORBO-based optimization, and the evaluation framework work together.\n* Discuss the limitations and scope of the correlation- and distribution-based losses beyond tabular data. Address why they are particularly suited for tabular data, and under what conditions they could generalize (e.g., to embeddings of other modalities).\n* Provide experimental insights on scalability by including at least one experiment on a larger-scale tabular dataset (or synthetic proxy). Alternatively, discuss how the approach is expected to behave as the data size grows.", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "zigPNeAAN5", "reviewer": "Reviewer_QYER", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "## Strengths\n- The paper addresses an important problem in synthetic tabular data generation, where deep generative models often struggle.\n- The proposed correlation- and distribution-aware loss function can be integrated into multiple DGM families (GANs, VAEs, DDPMs).\n- The proposed rank-based Bayesian optimization method (IORBO) is an interesting contribution that addresses challenges in aggregating multi-metric objectives.\n\n## Weaknesses\n- The three main contributions (loss function, benchmarking framework, and IORBO) feel only loosely connected. The overall flow of the paper suffers from this disjointedness.\n- Section 3.2 introduces IORBO as a new method, but Section 3.4 returns to standard BO as if introduced for the first time. In the experiments, IORBO is used exclusively, which adds to the confusion.\n- Section 6 describing the experiment setup is repetitive and unclear — several paragraphs (e.g., the first and third) appear to describe the same content.\n- There are theoretical inaccuracies, such as the suggestion that \"neural networks cannot capture correlations\" (Introduction), which is misleading.\n- The proposed “loss function” is auxiliary and not an optimization objective in itself, though it is sometimes described as such.\n- The claim of novelty for the “ML augmentation task” is questionable; using synthetic data alongside real data to improve model performance is a well-established technique.\n- The related work section is vague and lacks coverage of key areas, especially existing methods for handling heterogeneous tabular data (e. g. [1-3]) and BO literature.\n- In general, unless I misunderstood, the obtained win rates are not highly significant and should be interpreted with care. Only Table 7 shows significant results, which seem due to augmentations (well-known results).\n- Unless I misinterpreted, Table 9 suggests that SBO-V (using standard losses with stochastic Bayesian optimization) often performs better than the proposed variant, which undermines the claimed benefits.\n\n## References\n[1] Nazabal, A., Olmos, P. M., Ghahramani, Z., & Valera, I. (2020). Handling incomplete heterogeneous data using vaes. Pattern Recognition, 107, 107501.\n\n[2] Ma, C., Tschiatschek, S., Turner, R., Hernández-Lobato, J. M., & Zhang, C. (2020). Vaem: a deep generative model for heterogeneous mixed type data. Advances in Neural Information Processing Systems, 33, 11237-11247.\n\n[3] Peis, I., Ma, C., & Hernández-Lobato, J. M. (2022). Missing data imputation and acquisition with deep hierarchical models and hamiltonian monte carlo. Advances in Neural Information Processing Systems, 35, 35839-35851.\n\n[4] Kullback, Solomon, and Richard A. Leibler. \"On information and sufficiency.\" The annals of mathematical statistics 22.1 (1951): 79-86.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper presents three contributions aimed at improving tabular data generation with deep generative models (DGMs): (1) a correlation- and distribution-aware loss function to enhance the statistical fidelity of synthetic data, (2) a comprehensive benchmarking framework for evaluating synthetic tabular data using statistical and downstream ML metrics, and (3) a rank-based Bayesian optimization method (IORBO) for multi-metric hyperparameter tuning. While each component is potentially valuable, the connections between them are loosely integrated.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "- **Clarify better the connections between contributions**: Improve the flow of the paper so that the three contributions (loss, benchmarking, IORBO) are better integrated and motivated rather than appearing as isolated components.\n- **Correct theoretical statements**: Rephrase or clarify inaccurate claims such as \"neural networks cannot capture correlations.\"\n- **Reframe the loss function**: Clearly describe the proposed loss as an auxiliary term, not the main optimization objective.\n- **Revise the related work section**: Expand coverage to include prior work on:\n  - Handling heterogeneous tabular data.\n  - Bayesian optimization methods in generative modeling.\n- **Clarify the novelty of the ML augmentation task**: Either justify its novelty or acknowledge prior work that uses real + synthetic data to improve ML performance.\n- **Fix citation for KL Divergence**: Use the original reference [4] above.\n- **Address clarity issues in methodology**: Avoid redundancy in Sections 3.2, 3.4, and 6. Clarify when IORBO is introduced and used.\n- **Evaluate and report IORBO cost**: The paper claims the cost is negligible, but this should be justified with empirical or computational analysis.", "broader_impact_concerns": "While including an impact statement could be improved by briefly addressing risks of bias amplification in synthetic data, I don’t see major ethical concerns requiring changes.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Decision: Accept (poster)