STaSy: Score-based Tabular data Synthesis
Authors: Jayoung Kim
Chaejeong Lee
Noseong Park
Venue: ICLR 2023 notable top 25% (2023)
Keywords: Score-based generative model
Tabular data
Self-paced learning

Abstract: Tabular data synthesis is a long-standing research topic in machine learning. Many different methods have been proposed over the past decades, ranging from statistical methods to deep generative methods. However, it has not always been successful due to the complicated nature of real-world tabular data. In this paper, we present a new model named $\textbf{S}$core-based $\textbf{Ta}$bular data $\textbf{Sy}$nthesis ($\texttt{STaSy}$) and its training strategy based on the paradigm of score-based generative modeling. Despite the fact that score-based generative models have resolved many issues in generative models, there still exists room for improvement in tabular data synthesis. Our proposed training strategy includes a self-paced learning technique and a fine-tuning strategy, which further increases the sampling quality and diversity by stabilizing the denoising score matching training. Furthermore, we also conduct rigorous experimental studies in terms of the generative task trilemma: sampling quality, diversity, and time. In our experiments with 15 benchmark tabular datasets and 7 baselines, our method outperforms existing methods in terms of task-dependant evaluations and diversity.


Published as a conference paper at ICLR 2023
STASY: SCORE-BASED TABULAR DATA SYNTHESIS
Jayoung Kim, Chaejeong Lee, and Noseong Park
Department of Artificial Intelligence
Yonsei University
Seoul, South Korea [jayoung. .imm chaejeong lee, noseong) (areonn)@yoosette ac.kr
ABSTRACT
Tabular data synthesis is long-standing research topic in machine learning. Many different methods have been proposed over the past decades, ranging from statistical methods to deep generative methods. However, it has not always been successful due to the complicated nature of real-world tabular data. In this paper, we present a new model named Score-based Tabular data Synthesis (STaSy) and its training strategy based on the paradigm of score-based generative modeling. Despite the fact that score-based generative models have resolved many issues in generative models, there still exists room for improvement in tabular data synthesis. Our proposed training strategy includes a self-paced learning technique and fine-tuning strategy, which further increases the sampling quality and diversity by stabilizing the denoising score matching training. Furthermore, we also conduct rigorous experimental studies in terms of the generative task trilemma: sampling quality, diversity, and time. In our experiments with 15 benchmark tabular datasets and 7 baselines, our method outperforms existing methods in terms of cusk-dependant evaluations and diversity.
INTRODUCTION
Tabular data synthesis is of non-trivial im- portance in real-world applications for var- ious reasons: protecting the privacy of orig- inal tabular data by releasing fake tabular data (Park et al. 2018 Lee et al.) 2021), augmenting the original tabular data with fake data for better training machine learn- ing models (Chawla al. 2002; Han et al., 2005, et 2008]] Kim et al. 2022), and However, it is well-known that tabular data frequently has such peculiar characteris- ties that deep generative models to synthesize all possible details of the origi- nal tabular data (Park al. 2018 Xu et al.) 2019)- - given a set of columns in tabular data, columns typically follow unpredictable (multi-modal) distributions and therefore, it is hard to model their joint probability.
Table 1: Summary of experimental results. We report the average sampling quality, diversity, and time.
Quality (FI R²)
Diversity (coverage)
Runtime (second)
Methods
MedGAN VEEGAN
-0.717
0.037
0.246
-0.368
0.038
0.109
CTGAN
0.560
0.352
0.704
TVAE
0.474
0.494
0.100
TableGAN
0.386
0.434
0.046
OCT-GAN
0.552
0.381
26.926
able
RNODE
0,366
0,328
13.392
NaIve-STaSy
0.708
0.637
8,855
STaSy
0.727
0.658
10.663
couple of recent methods, however, showed remarkable successes (with some failure cases) in synthesizing fake tabular data, such as CTGAN (Xu et al. 2019), TVAE (Xu et al. 2019)) IT-GAN (Lee) ett al. 2021), ), and OCT-GAN (Kim et al. 2021). addition, a recent generative model paradigm, called score-based generative modeling (SGMs), successfully resolves the two problems of the generative learning trilemma (Xiao et al. 2021). i.c., score-based generative models provide high sampling quality and diversity, although their training/sampling time relatively longer than other deep generative models. In this paper, we adopt a score-bar ccsse- generative modeling paradigm and design Score-based Tabular data Synthesis (STaSy) method.
Our model designs significantly outperform all existing baselines in terms of the sampling quality and diversity (ef. Na SS-SSaSy and STaSy in Table Na V STaSy y is
naive conversion

---

Published as a conference paper at ICLR 2023
of SGMs toward tabular data, and STaSy additionally uses our proposed self-paced learning and fine-tuning methods. Figure shows the uneven and long-taii long-tailed loss distribution of Haive-STaSy at the end of its training process. The figure implies the training of Naïve- Saive-STaSy by the denoising score matching failed to learn the score values of some records. This may let the model be (partially) underfitted to training data. In contrast, STaSy with our two proposed training methods yields many loss values around the left corner (i.e., close to 0).
In order to alleviate the training difficulty of Naive-STaSy, STaSy, we design i) self-paced learning method, and ii) a fine-tuning approach. Our proposed self- learning technique trains our model from easy to hard records based on their loss values by modifying the objective function. The technique makes the model learn records selectively and eventually. During this pro- cess, the model can be better trained. addition, our proposed fine-tuning method, which modestly adjusts the model param- eters, can further improve the sampling quality and diversity.
Naive-STa5y
STaSy
Frequency
In Table we summarize our experimental results, where we compare our sTaSy with other existing tabular data synthesis methods in terms of the sampling quality, diversity, and time. As shown, our basic model even without our proposed self- paced learning and fine-tuning, denoted Halves-SaSy, significantly outperforms all baselines except for runtime.
Figure 1: Distributions of denoising score matching loss in Shoppers
In summary, our contributions are follows: i) We design score-based generative model for tabular data synthesis. ii) We alleviate the training difficulty of the denoising score matching loss by designing self-paced learning strategy and further enhance the sampling quality and diversity using a proposed fine-tuning method. STaSy, thus, clearly balances among the generative learning trilenma: sampling quality, diversity, and time. iii) Our proposed method outperforms other deep learning methods in all cases by large margins, which we consider a significant advance in the field of tabular data synthesis. iv) We evaluate various methods in terms the generative learning trilemma in rigorous manner.
RELATED WORK
2.1 SCORE-BASED GENERATIVE MODELS
Score-based generative models (SGMs) use a diffusion process defined by the following Itô stochastic differential equation (SDE);
dx (((x()dd + g())dw
(1)
where f(x, t) f())x, f and g are drift and diffusion coefficients of x(t), and w is the standard
Wiener process. Depending on the types of f and g. SGMs can be divided into variance exploding (VE), variance preserving (VP), and sub-variance preserving (sub-VP) models (Song et al. 2021).
The definitions of f and g are in Appendix A The reverse of the diffusion process is a denoising
process as follows:
dx = (F(x V lo + g(()dw;
(2)
where this reverse SDE is process of generating samples. The score function Vx p. is approximated by a time-dependent score-based model So (x, l), called score nelwork.
In general, following the diffusion process in Equation we derive x(!) time E [o, 7). where x()) and x(T) means a real and noisy sample, respectively. The transition probability (t) x())) time t is easily approximated by this process, and it always follows Gaussian distribution. allows us to collect the gradient of the log transition probability, Vx(e) log((x(()) (0)), during the diffusion process. Therefore, we can train a score network Sx(x, 1) as follows:
arg min E,E Ex(0) (())))((x(())1) 0
Vx(e) log logg(x()))(00)))
2

---

Published as a conference paper at ICLR 2023
where X(t is to control the trade-off between the sampling quality and likelihood. This is called denoising score matching, and 0 solving Equation 3 can accurately solve the reverse SDE in Equa-
tion 2 Vincent 2011).
After the training process, we can synthesize fake data records with i) the predictor framework ii) the probability flow method, deterministic method based the ordinary differential equation (ODE) whose marginal distribution is equal to that of Equation ] (Song et al. 2021). In particular, the latter enables fast sampling and exact log-probability computation.
2.2 TABULAR DATA SYNTHESIS
Many distinct methods exist for tabular data synthesis, which creates realistic synthetic tables depending on the data types. For example, a recursive table modeling utilizing a Gaussian copula is used synthesize continuous variables (Patki et al. 20166). Discrete variables can be generated by Bayesian networks (Zhang et al. 2017; Aviñó et al. 2018 and decision trees (Reiter) (205). Several data synthesis methods based on GANs have been presented to generate tabular data in recent years. RGAN (Esteban et al. (017) creates continuous time-series healthcare records, whereas
MedGAN (Choi al. 2017) and corrGAN (Patel et al. 2018) generate discrete records. EhrGAN (Chel
iet al. 2017) utilizes semi-supervised learning to generate plausible labeled records to supplement limited training data. PATE-GAN (Jordon et al. 2019) generates synthetic data without jeopardizing the privacy of real data. TableGAN (Park et al. 2018)) employs convolutional neural networks to enhance tabular data synthesis and maximize label column prediction accuracy. CTGAN and
TVAE (Xu et al. (2019) adopt slllumn-typ--ppccffe preprocessing steps to deal with multi-modality in the original dataset distribution. OCT-GAN (Kim et al., 202T) is generative model design based on neural ODEs. SOS (Kim et al. 2022) proposed style-transfer-based oversampling method for imbalanced tabular data using SGMs, whose main strategy is converting a major sample to a minor sample. Since its task is not compatible our task to generate from the scratch, direct comparisons are not possible. However, we convert our method to an oversampling method following their design guidance and compare with SOS in Appendix B
2.3 SELF-PACED LEARNING
Self-paced learning (SPL) is training strategy related to curriculum learning to select training
records in meaningful order, inspired by the learning process of humans (Kumar et al., 2010b Ting al. 2014). It refers to training model only with subset of data that has low training losses and gradually expanding to the entire training data. We denote the training set as D {x, [ where x;
the i-th record. The model M with parameters has a loss 4 [M,, 0)), where L is the loss
function. A vector v = [ U/ E (0,1) indicates whether x; is casy or not for all i. SPL aims to learn the model parameter 0 and the selection importance by minimizing:
N
N
min E(O, 8.v
Σ
U. LMM 0))
U
(4)
K
where a parameter to control the learning pace. general, the second term Equation called self-paced regularizer, can be customized for a downstream task.
The alternative convex search (ACS) (Bazaraa et al., 1993) is typically used to solve Equation Ku-
mar et al.) 2010a|| Tang et al.) 2012). By alternately optimizing variables while fixing others, we can
optimize Equation 4 i.e., update v after fixing 0, and vice versa. With fixed 0, the global optimum
v" | is detined as follows:
11
4 K 0,
2 K
When updating v with fixed 0, a record x; with 1 < chosen for training. Only easy records are used to train the model. Otherwise, x; is regarded as hard record and will be unselected. To involve more records in the training process, K is gradually decreased.
K is regarded as an easy record and will be
3

---

Published as a conference paper at ICLR 2023
3 PROPOSED METHOD
STa is an SGM-based method for tabular data synthesis. STaSy uses SPI. to ensure its training stability. The suggested fine-tuning method takes advantage of favorable property of SGMs, which is that we can measure the log-proba log-pobbabilities records.
3.1 SCORE NETWORK ARCHITECTURE & MISCELLANEOUS DESIGNS
It is known that each column in tabular data typically has complicated distributions, whereas pixel values image datasets typically follow Gaussian distributions (Xu et al. 2019). Moreover, tabular synthesis models should learn the joint probability of multiple columns generate record, which is one main reason why tabular data synthesis is difficult. However, one good design point is that the dimensionality of tabular data is typically far less than that of image data, e.g., 784 pixels even in MNIST, one of the simplest image datasets, vs. 30 columns in Credit.
We found through our preliminary experiments that the SDE in Equations [ [and can well model the joint probability iff its score network, which approximates Vx log (x), is well trained. We carefully design our score network for tabular data synthesis considering these points. Our proposed score network architecture is in Appendix C
The network consists of residual blocks of FC layers.
Since SGMs were theoretically designed from the idea of perturbing data with an infinite number of noise scales, SGMs typically require large-scale computation, e.g., T = 1, 000 for images in (Song al. 2021). as an approximation to the infinite number. With large number of steps, the denoising process requires long time to complete, which is one part of the generative learning trilemma. However, we found that = 50 steps Equation are enough to train a network to approximate the gradient of the log-likelitee ogglikkelhood, which means that our STa naturally has less sampling time than SGMs for images with = 1, 000 steps.
of tabular data To handle mixed types of data, which is challenge in tabular data generation, we pre/post-proces columns. We use the min-max scaler to pre-process numerical columns, and its reverse scaler is used for posc-processing after generation. We also apply one-hot encoding to pre-process categorical columns, and use the softmax function, followed by the rounding function, when generating.
How to generate After sampling a noisy vector z ~ Nu, I2),, the reverse SDE can convert z into fake record. The prior distribution ~ of) varies depending the type of SDEs: N((O, I for VE, and (O, for VP and sub-VP. Tmar is a hyperparameter. In particular, we adopt the probability flow method solve the reverse SDE, which will be shortly described in Equation T0
3.2 SELF-PACED LEARNING APPROACH
In order alleviate the training difficulty, we apply a curriculum learning technique for STaSy, more specifically, self-paced learning. Instead of letting v; E {0, 1j. we use a "soft" record sampling method, i.e., U, E [o, 1]. If 4, which the denoising score matching objective on i-th record, less than threshold, we set to 1 to ensure that the record is fully involved in training. At the end of the training, o; must be set to for all to train the model with the entire data. The denoising score matching loss for i-th training record x; is defined as follows:
4 = Ex,
(xx
Vx((t) pox p(x)
(6)
Then, have the following STaSy objective:
N
min
1gls + a(
where 0 0 w; < for all i, r-)) is a self-paced regularizer. a E [0, 1] and 33 E [0, 1] are variables to define thresholds, which are monotonically increasing as training goes on (see Appendix D for their exact controlling mechanism).
Definition 1. Let Q()) be quantile function defined as inf R : F(I),, where F is cumulative distribution function of the denoising score matching objective. That is, Q(p) is the minimum value for which the CDF is greater than or equal the given probability p.
4

---

Published as a conference paper at ICLR 2023
Algorithm 1: How to train STaSy
Initialize
x* Train SGM based on SPL training strategy
for each mini-batch of records do Update with Equation
Update after fixing with Equation]
Update and 3 with the control method in Appendix D
Fine-tune the trained model using log tog-probability
+ (x:)
F {x, log p(x.) where x; D, smaller than the average (or median) Nog-probability..
for each fine-tune epoch do for each x; E do
Update with Equation 6 Ti)
11
{x,
return 0
Theorem 1. Let the self-paced regularizer a(v; be defined as follows:
Q(())
N
N
a(
Σ
v - Q(3)
(8)
2
i=1
t=1
where the closed-form optimal solution for v" [u U2...., , uir given fixed 0, is defined as follows its proof is in Appendix
if4 < Q(a).
242)
(9)
Q(3) Q(a) - Q(B)* otherwise.
Specifically, records with 4, < Q(a) are considered easy records and will be selected for training. whereas records with 4 2 Q(3) are considered complicated (or potentially noisy) and will not be selected. If not both cases, records will be partially selected during training, i.e., E [0, 1]. and B from the initial values Ogg and fo. proportionally to training progress to are gradually increased ensure that all data records are involved in training. As a and /3 increase, the difficult records are gradually involved training, and the model also becomes more robust to those difficult cases. We set oo and 3 in such way that more than 80% of the training records are included in the learning process from the beginning.
3.3 FINE-TUNING APPROACII
For solving the reverse SDE process, score-based generative models rely various numerical approaches. One of the techniques is the probability flow method in Equation T0 Song et al. [202]] which uses deterministic process whose marginal probability the same the SDE, With the approximated score function Se(-), the probability flow method uses the following neural ordinary differential equation (NODE) based model (Chen et 2018):
1 ())()) ( log (x)
dx (f(x,
(10)
In our experiments, the probability flow shows better quality than other methods to solve the original reverse SDE, and our default solver is the probability flow (see Section 4.3). In addition, NODEs facilitate computing the probabiiyyl defined in Equation 10 through the instantancous change of variables theorem. Consequently, unbiased Hutchinson's 's estimator (Hutchinson 1989 Grathwohl et al. 2018). Thus, we propose to fine-tune based the exact log-pro probability.
calculate the exact log- probability efficiently with the
After learning the model parameter as described in Section we set the sample-wise threshold T; log p(x;)) (ef. of Algorithm []. We then prepare the fine-tuning candidate set F (cf. Line
5

---

Published as a conference paper at ICLR 2023
of Algorithm Tp. After fine-tuning for the samples in F, update the candidate set (ef. Line [1 of Algorithm I. Our goal is for achieving a better log- log-probabillty than the initial one T; before the fine-tuning process.
3.4 TRAINING ALGORITHM
Algorithm [ shows the overall training process for our STaSy. Firstly, we initialize the parameters of the score network O. Utilizing ACS, we then train STaSy with the SPL training strategy. At this step, we iteratively optimize and v. We can obtain an optimal 0 by optimizing Equation] 7 with fixed v. and the global optimum v is calculated by Equation 9 We also update a and & in proportion to training progress, After finishing the main SPL training step. we can generate fake records from the model. To further improve the trained score network, we retrain our model for every record x; whose oog-robbabilities are less than its threshold тą. At Line T0 of Algorithm [ one can use the log-pre pog--bbabilityy instead of Equation 6 as a fine-tuning objective. However, we found that Equation more effective (see Appendix
EXPERIMENTS
We analyze methods in terms of the generative learning trilemma. We list only aggregated results over datasets in the main paper - details are in Appendix G We repeat experiments 5 times.
4.1 EXPERIMENTAL ENVIRONMENTS
brief description of our experimental environments is follows: i) We use 15 real-world tabular datasets for classification and regression and 7 baseline methods. To be specific about our models, SSTTSy a naive conversion of SGMs without our proposed training strategies, and STaSy is trained with self-pacce learning and the fine-tuning method. îï) In general, we follow the "train 2017,Jordon
on synthetic, test on real (TSTR)" framework (Esteban et al. 2017 Jordon et al. 2019), which is a
widely used evaluation method for tabular data (Xu et al., 2019 Kim et al. 2021 Lee al. 2021), to evaluate the quality of sampling - in other words, we train various models, including DecisionTree. AdaBoost, Logistic/lLinear Regression, MLP classifien/regressor, RandomForest, and XGBoost, with fake data, validate with original training data, and test them with real test data. For Ident train with real training data, choose the best performing model using the cross-validation, and test with real test data, whose score can be criterion to evaluate the sampling quality of various generative methods. iii) We use various metries to evaluate in various aspects. For the sampling quality, we mainly use average F1 for classification, and also report AUROC and Weighted-Fi We use R² and RMSE for regression. For the sampling diversity, we use coverage (Naeem et al. 2020 which was proposed to measure the diversity of generated records. Full results are in Appendix Detailed environments and hyperparameter settings are Appendix H and respectively.
4.2
EXPERIMENTAL RESULTS
4.2.1 SAMPLING QUALITY
Table summarizes the key results on the sampling quality. We use task-oriented metrics, such as F1, R² and on, under the TSTR evaluation framework. MedGAN and VEEGAN, two early GaN-based methods, show relatively lower test scores than other GAN-base methods, i.c., CTGAN, TableGAN, and OCT-GAN. In general, CTGAN and OCT-GAN show reliable quality among the baselines. However, our two score-basee based models always mark the best and the second best quality.
Our methods, Naive-STašy and STaSy, significantly outperform all the baselines by large margins. In particular, our methods perform well in small datasets, e.g., Crowdsource, Obes ity, and Robot, while other methods show poor quality, shown in Table 14 Appendix G. These multi-class classification datasets have small number of per class record, e.g., the smallest class in Crowdsource has 79 records in the training set, which means that our methods are able to capture fine- grained modes from the original data. Moreover, in Credi t, which has severe class imbalance ratio of 99.7% for class 0 and 0.3% for class 1, more than half of the baselines failed to generate the minority class, showing an F1 score close to 0 in Table [1 of Appendix CTGAN and Tab leGAN also achieve good Fl score close to that of Identi ty, but STaSy takes the first place again.

---

Published as a conference paper at ICLR 2023
Table 2: Classification//eeressioo with real data. We report the average F1 (resp. macro F1), AUROC, and Weighted-F1 for binary (resp. multi-class) classification, and R and RMSE for regression. The best (resp. the second best) results are highlighted in bold face (resp. with underline).
Classification
Regression
Methods
F1
AUROC
Weighted-FI
R²
RMSE
Identity
0.796
0.932
0.801
0.452
0.772
MedCAN
0.232
0.608
0.221
-6,885
2.589
VEEGAN
0.329
0.661
0.316
-4.901
2.500
CTGAN
0.627
0.838
0.635
0.122
0,946
TVAE
0.539
0.795
0.542
0.056
0.959
TableGAN
0.567
0.826
0.572
-0.788
1.227
OCT-GAN
0.622
0.860
0.626
0.104
0.946
RNODE
0.496
0.796
0,502
-0.483
1.112
Naive-STe STaSy
0.768
0,917
0.776
0.321
0.789
STaSy
0.783
0.922
0.790
0.361
0.766
As flow-based generative models and SGMs with the probability flow method calculate the exact log- probability of records, we present the log-probal log-probability as another metric for the sampling quality. Table shows the median of the log-probabilities of testing records, averaged over all datasets. Since the log- probbabbbiity is not bounded, we take a median of them to handle case of outliers. Our methods, even without the fine-tuning, show a much better log-probability than RNODE, which optimizes the log-probability as its objective. Moreover, the median log-proc peebbbbbbbiltyy of testing records even improves after the proposed fine-tuning method.
Table 3: The median of the log-probabilities of testing records, averaged over all datasets
Methods
Log Log-probable probability
RNODE
59.327
STaSy w/o fine-tuning STaSy
129. 129.293
131.734
Putting it all together, our proposed score-based generative models, i.e. Naīve STaSy and STaSy, show reasonable performance in all cases regarding the machine learning efficacy and log- prebbabbbbllty. Furthermore, as shown Tables ( 3 the sampling quality always improves with the proposed training strategies, i.e., the self- self-paced learning and the fine-tuning, which justifies their efficacy.
4.2.2 SAMPLING DIVERSITY
Figure 2: t-SNE visualizations of fake and the original records in Robot.
Table 4: Sampling diversity in terms of coverage averaged over all datasets
For the quantitative evaluation of the sampling diversity be- tween existing methods and our proposed method, we use the coverage score (Naeem al. (2020). which bounded between 0 and 1. Coverage is the ratio of real records that have least one fake record in its manifold. A manifold sphere around the sample with radius r, where r is the dis- tance between the sample and the k-th nearest neighborhood. Table Sommarizsss the averaged coverage of each method. MedGAN and VEEGAN show poor coverage scores, close to 0. This trend is also shown in the t-SNE visualizations in Appendix J.2] Among the baseline methods, CTGAN per- forms the best in terms of the sampling quality, whereas it
Methods
Coverage
MedGAN
0,037
VEECAN
0.038
CTGAN
0,352
TVAE
0.494
TableGAN
0.434
OCT-GAN
0,381
RNODE
0,328
Naïve -Tree
0.637
STaSy w/o fine-tuning
0.655
STaSy
0.658
7

---

Published as a conference paper at ICLR 2023
Real
Real
OCT-GAN
OCT-GAI GAN
CTGAN
STaSy
STaSy
STaSy
Figure 3: (Left and Middle) Histograms of values in Roundness and Compactness columns of Bean, respectively. (Right) t-SNE (van der Maaten & Hinton 2008 visualizations of the fake and original records in Obesity. More visualizations are in Appendix
shows relatively inferior coverage performance than others. In specific, in Robot, STasy shows coverage of 0.94, while other three nop-performing baselines, CTGAN, TableGAM, and OCT-GAM show coverage scores less than 0.26 Table 20 of Appendix [G.2] Figure also presents the di- versity of each fake data by each method qualitatively, which reflects the results of coverage. In general, STaSy shows stable performance across the sampling quality and the sampling diversity, outperforming others by large margins.
In Figure (Left and Middle), the fake data by STaSy shows an almost identical distribution to that of real data. In contrast, OCT-GAN, which was proposed to address the multi-modality issue of tabular data, fails to do This means STaSy is able to capture every mode the columns, while OCT-GAN is not. In Figure (Right), CTGAN generates some out-of-distribation records, highlighted in red.
4.2.3 SAMPLING TIME
Table 5: Runtime evaluation results, averaged over all datasets
Methods MedGAN VEEGAN CTGAN TVAE TableGAN OCT-GAN RNODE Naive-STaSy STaSy
Runtime
0.246
0.109
0.704
0.100
0.046
26.926
13.392
8.855
10.663
summarize runtime Table In order to compare the runtime all methods, we measure the wall-clock time taken sample N records, where N is training size, 5 times, and average them. In general, simple GAN-based methods, especially TableGAN and TVAE, show faster runtime. On the other hand, SGMs, OCT-GAN, and RNODE take relatively long time for sampling. Our proposed methods, Na i' ve--Saay and STaSy, take a long sampling time compared to simple GAN-based methods but faster than OCT-GAN and RNODE, which means well-balanced trade-off between the sampling quality, diversity, and time.
4.3 ABLATION & SENSITIVITY STUDIES
Table 6: Ablation study. We report Fl (resp. R²) for classification (resp. regression).
Datasets
Naive-STaSy
w/o fine-tuning
w/o SPL
STaSy
Credit Default Shoppers
0.737+0.083
0.752+0.053
0.775=0.050
0.795+0.034
0.506=0.013
0.515+00008
0.509=0.013
0.519+0.009
0.630=0.015
0.639
0.63420.018
0.640=0.008
Contraceptive
0.39320.043
0.424+0.014
0.40620.016
0.425+0.003
Crowdsource
0.702+0. 113
0.712+0. 140
0.714+0.112
0.717*0..26
Shurtle
0.754+0.044
0.763+0.074
0.77420.036
0.80020.043
Bei fing
0.648+0.088
0.672+0.108
0.648+0.088
0.679+0.115
We define three ablation models: NaIve-STaSy' without SPL and fine-tuning, "w/o fine-tuning'
without fine-tuning but with SPI., and 'w/o SPL! without SPL but with fine-tuning. All ablation
8

---

Published as a conference paper at ICLR 2023
Table 7: Sensitivity analyses. We report Fl (resp. R²) for classification (resp. regression).
Datasets
SDE Type
Metric
Ag
Metric
Bo
Metric
VE
0.875=0.023
0.05
0.875+0.02
0.70
0.907=0.027
Spanbase
VP
0.880=0.029
0.10
0.909=0.022 028
0.80
0.909=0.023
sub- VP
0.912:0.024
0.30
0.911=0.030
0.90
0.90620.035
VE
0.704=0.141
0.05
0.686+0. 116
0.75
0.662+0.129
Crowdsource
VP
0.697=0.111
0.10
0.692#0.1 112
0.80
0.675=0.136
sub- VP
0.630=0.115
0.30
0.669
0.95
0.683+0.112
VE
0.679+0.115
0.05
0.667+0.107
0.70
0.667=0.100
Beijing
VP
0.645=0.086
0.10
0.673
0.75
0.663#0.100
sub VP
0.56320.036
0.30
0.66420.106
0.95
0.67520.111
models are inferior to STaSy in Table 6 showing the effectiveness of SPL and fine-tuning. In
particular, SPI. improves the sampling diversity as in Figure Na Nav--STaSy suffers from mild mode collapses, as highlighted in red.
Table 7 shows sensitivity analyses w.r.t. some important hyperparameters. In general, all settings show reasonable results, outperforming the baselines. We recommend 0.2 and 0.25 for og and 0.9 and 0.95 for 3g-
We can adopt a variety of methods to solve the reverse SDE process in Equation 2 generate fake records with the predictor-correcto framework (Pred. Corr.) the probability flow (PF) method (Song et al. 2021). The former uses the ancestral sampling (AS), reverse diffusion (RD), or Fuler-Maruyane (EM) method for solving the reverse SDE, and for the correction process, the Langevin corrector. In Table g the probability flow method in Equation 10 mostly leads to successful results, and other datasets also show similar results.
Our method can
Real
Naive STaSy
Table 8: Results of NaIve-STaSy by various reverse SDE solvers in Magic, c, where 'Pred.' means the predictor-only method. We report Fl.
fine-tuning
Predictor
Pred.
Pred. Corr.
AS
0.754 028 0.780&0 054
RD
0.777+0.059 0.77940 9066
EM
0.77720.058
0.772+0.056
Figure 4: t-SNE visualizations of the fake and original
PF
0.781=0.058
No Corr.
records in Beij Biiging
CONCLUSIONS AND DISCUSSIONS
Synthesizing tabular data is an important yet non-trivial task, it requires modeling a joint probability of multi-modal columns. To this end, we presented our detailed designs and experimental results with thorough analyses. Our proposed method, STaSy, is score-based model equipped with our proposed self-paced learning and fine-tuning methods. In our experiments with 15 benchmark datasets and baselines, STaSy outperforms other deep learning methods in terms of the sampling quality and diversity (and with an acceptable sampling time). Based on these considerations, we believe that STaSy shows significant advancements tabular data synthesis. We expect much follow-up work in utilizing SGMs for tabular data synthesis.
Limitations. Although our model shows the best balance for the deep generative task trilemma, we think that there exists room to improve runtime further - existing simple GAN-based methods are faster than our method for sampling fake records. In addition. SGMs are known to be sometimes unstable for nigh-dimensional data, e.g., high-resolution images, but in general, stable for low- dimensional data, e.g., tabular data. Therefore, we think that SGMs have much potential for tabular data synthesis in the future.

---

Published as a conference paper at ICLR 2023
ETHICS STATEMENT
Indeed, people do not always use artificial intelligence technology for righteous purposes. One can use our method achieve his/her wrongful goals, e.g., selling high-quality fake data generated by our method, and retrieving private original data records from synthetic data. However, we believe that our research has much more beneficial points. One can use our method to generate fake data and share (after hiding the original data) to prevent potential privacy leakages. We, of course, need more studies achieve the privacy protection goal based on our model. However, research trend exists where researchers try to use deep generative model to protect privacy (Park et al. 2018 2021).
Lee et al.
7 REPRODUCIBILITY STATEMENT
To reproduce the experimental results, we have made the following efforts: 1) Source codes used in the experiments are available in the supplementary material. By following the README guidance, the main results are easily reproducible. 2) All the experiments are repeated five times, and their mean and standard deviation values are reported in Appendix. We provide extensive experimental details in Appendix H
ACKNOWLEDGMENTS
Jayoung Kim and Chaejeong Lee equally contributed. Noscong Park is the corresponding author. This work was supported by the Institute of Information & communications Technology Planning Evaluation (ITTP) grant funded by the Korea government (MSIT) (90% from No. 2021-0-00231, Development Approximate DBMS Query Technology to Facilitate Fast Query Processing for Exploratory Data Analysis and 10% from No. 2020-0-01361, Artificial Intelligence Graduate School Program (Yonsei University)).
REFERENCES
Statlog (Shuttle). UCI Machine Learning Repository.
Laura Aviñóó, Matteo Ruffini, and Ricard Gavaldà. Generating synthetic but plausible healthcare record datasets, 2018.
M.S. Bazaraa, H.D. Sherali, and C.M. Shetty, Nonlinear Programming Theory and Algorithms. John Wiley, New York, 1993.
R. Bock. MAGIC Gamma Telescope. UCI Machine Learning Repository, 2007.
Mete Katircioglu Yomi Kastro C Okan Sakar, Olcay Polat. Real-time prediction of online shoppers' purchasing intention using multilayer perceptron and Istm recurrent neural networks. Neural Computing and Applications, 31:6893-6666 0606866998888--8-111888888888888 10 2019.
Nitesh V Chawla, Kevin W Bowyer, Lawrence Hall, and W Philip Kegelmeyer. Smote: synthetic minority over-sampling technique. Journal of artificial intelligence research, 16:321-357, 2002.
Zhengping Che, Yu Cheng, Shuangfei Zhai, Zhaonan Sun, and Yan Liu. Boosting deep learning risk prediction with generative adversarial networks for electronic health records. 2017.
Tian Chen, Yulia Rubanova, Jesse Bettencourt, and David Duvenaud. Neural ordinary differential equations, 06 2018. 8.
Edward Choi, Siddharth Biswal, A. Bradley Maline, Jon Duke, F. Walter Stewart, and Jimeng Sun. Generating multi-label discrete electronic health records using generative adversarial networks. 2017.
Cristóbal Esteban, L. Stephanie Hyland, and Gunnar Rätsch. Real-valued (medical) time series generation with recurrent conditional gans, 2017.
10

---

Published as a conference paper at ICLR 2023
Kelwin Fernandes. A proactive intelligent decision support system for predicting the popularity of online news. In EPIA, 2015.
Chris Finlay, JömnHenriik Jacobsen, Levon Nurbekyan, and Adam M. Oberman. How to train your neural ode: the world jacobian and kinetic regularization. In ICML, 2020.
Veloso Marcus Barreto Guilherme Freire, Ananda. Wall-Following Robot Navigation Data, UCI Machine Learning Repository, 2010.
Will Grathwohl, Ricky TQ Chen, Jesse Bettencourt, Ilya Sutskever, and David Duvenaud. Ffjord: Free-form continuous dynamics for scalable reversible generative models. arXiv preprint arXiv:1810.01367, 2018.
Hui Han, Wen- Wen-Yuan Wang, and Bing-Huan Mao. Borderline-smote: A new over-sampling method in imbalanced data sets learning. In Proceedings of the 2005 International Conference on Advances in Intelligent Computing- Volume Purt I, ICIC"05, pp. 878-887, Berlin, Heidelberg, 2005. Springer- Verlag. ISBN 3540282262. doi: 10. 10.1007/11538058 91.
11538059_99
Haibo He, Yang Bai, Edwardo Garcia, and Shutao Li. Adasyn: Adaptive synthetic sampling approach for imbalanced learning. pp. 1322 1328, 07 2008. doi: 10.1 1010///CNN..000.6663.3
Reeber Erik Forman George Suermondt Jaap Hopkins, Mark. Spambase. UCI Machine Learning Repository, 1999.
M.F. Hutchinson. A stochastic estimator of the trace of the influence matrix for laplacian smoothing splines. Communication in Statistics- Simulation and Computation, 18:99-110 01 1989. doi: 00.1080003610011008888
Lu Jiang, Deyu Meng, Shoou-I Yu, Zhenzhong Lan, Shiguang Shan, and Alexander Hauptmann. Self-paced learning with diversity. Advances in neural information processing systems, 27, 2014.
Brian Johnson and Kotaro lizuka. Integrating openstreetmap crowdsourced data and landsat time- series imagery for rapid land use/land cover (lule) mapping: Case study of the laguna de bay area of the philippines. Applied Geography, 67:140-149 02 2016. doi: 10. 21101/6.apgeog. 2015..2.00.0000
James Jordon, Jinsung Yoon, and V. D. Mihaela Schaar. Pate-gan Pate-gan: Generating synthetic data with differential privacy guarantees. In International Conference on Learning Representations, 2019.
Jayoung Kim, Jinsung Jeon, Jachoon Lee, Jihyeon Hyeong, and Noseong Park. Oct-gan: Neural ode-based conditional tabular gans. In The Web 2021.
Jayoung Kim, Chaejeong Lee, Yehjin Shin, Sewon Park, Minjung Kim, Noseong Park, and Jihoon Cho. Sos: Score-based oversampling for tabular data. arXiv preprint arXiv:2206.0888 08555, 2022.
Murat Koklu and Ilker Ali Ozkan. Multiclass classification of dry beans using computer vision and machine learning techniques. Comput. Electron. Agric., 174: 174:105507, 2020.
M. Kumar, Ben Packer, and Daphne Koller. Self-paced learning for latent variable models. pp. 1189-1199 1197, 01 2010a.
M Kumar, Benjamin Packer, and Daphne Koller. Self-paced learning for latent variable models. Advances in neural information processing systems, 23, 2010b.
Jaehoon Lee, Jihyeon Hyeong, Jinsung Jeon, Noseong Park, and Jihoon Cho. Invertible tabular GANs: Killing two birds with one stone for tabular data synthesis. In NeurIPS, 2021.
Xuan Liang, Tao Zou, Bin Guo, Shuo Li, Haozhe Zhang, Shuyi Zhang, Hui Huang, and Song Xi Chen. Assessing beijing's 's pm2. pollution: severity, weather impact, apec and winter heating. Proceedings of the Royal Society A: Mathematical, Physical and Engineering Sciences, 471(2182): 20150257, 2015.
M. Lichman. UCI machine learning repository. 2013. URL http://archive. ics. uci eduu
11

---

Published as a conference paper at ICLR 2023
Tjen-Sien Lim. Contraceptive Method Choice. UCI Machine Learning Repository, 1997.
Robert Lyon. HTRU2. UCI Machine Learning Repository, 2017.
Lee Mohammad, Rami McCluskey. Phishing Websites. UCI Machine Learning Repository, 2015.
Muhammad Ferjad Naeem, Seong Joon Oh, Youngjung Uh Yunjey Choi, and Jaejun Yoo. Reliable fidelity and diversity metrics for generative models. In International Conference Machine Learning, pp. 7176-7185. PMLR, 2020.
Fabio Mendoza Palechor and Alexis de la Hoz Manotas. Dataset for estimation of obesity levels based eating habits and physical condition in individuals from colombia, peru and mexico. Data in brief, 25: 104344, 2019.
Noseong Park, Mahmoud Mohammadi, Kshitij Gorde, Sushil Jajodia, Hongkyu Park, and Youngmin Kim. Data synthesis based on generative adversarial networks. arXiv preprint 1 03384, 2018.
Shreyas Patel, Ashutosh Kakadiya, Maitrey Mehta, Raj Derasari, Rahul Patel, and Ratnik Gandhi. Correlated discrete data generation using adversarial training. 2018.
Neha Patki, Roy Wedge, and Kalyan Veeramachaneni. The synthetic data vault. In DSAA, 2016.
Jerome Reiter. Using cart to generate partially synthetic, public use microdata. Journal of Official Statistics, 21:441, 01 2005.
Yang Song, Jascha Sohl-Dickstein, Diederik Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. In ICLR, 2021.
Akash Srivastava, Lazar Valkov, Chris Russell, Michael U Gutmann, and Charles Sutton. Veegan: Reducing mode collapse in gans using implicit variational learning. Advances in neural information processing systems, 30, 2017.
Kevin Tang, Vignesh Ramanathan, Li Fei-Fei, and Daphne Koller. Shifting weights: Adapting object detectors from image to video. In Proceedings of the 25th International Conference on Neural Information Processing Systems Volume 1, NIPS' 12, pp. 638 646, Red Hook, NY, USA, 2012. Curran Associates Inc.
Laurens van der Maaten and Geoffrey Hinton. Visualizing data using t-sne. Journal of Machine Learning Research, 9(86), 2008
Pascal Vincent. A connection between score matching and denoising autoencoders. Neural Comput., 23(7):166111674 2011
Zhisheng Xiao, Karsten Kreis, and Arash Vahdat. Tackling the generative learning trilemma with denoising diffusion gans, 12 2021.
Lei Xu, Maria Skoularidou, Alfredo Cuesta-Infante and Kalyan Veeramachaneni. Modeling tabular data using conditional gan. In NeurIPS. 2019.
Jun Zhang, Graham Cormode, Cecilia M. Procopiue, Divesh Srivastava, and Xiaokui Xiao. Privbayes: Private data release via bayesian networks. ACM Transactions Database Systems, 2017.
12

---

Published as a conference paper at ICLR 2023
A VE, VP, AND SUB-VE VP SDES
We introduce the definitions of f and g as follows:
0,
if VE, if VP, if sub-VP,
S(1) =
(t)xx (()xx,
(II)
if VE, if VP, if sub-VP,
de (t),
s(0) =
(12)
(1(((1 - -2.66 )()
for t E [0, 1]. where Omin and o mar are hyperparameters, and we use Omin = {0.01, 0.1] and maar = (5.0, 10.0). (()) = Tmin + (Ymar 7mmn) for E [0, 1], where Tmain and Ymaz are hyperparameters, and we use Ymin {0.01, 0.1} and "maz {5.0, 10.03.
where (t) and 7t) are noise functions w.r.t time t. o(t) min Omin
B COMPARISON BETWEEN SOS AND STASY FOR THE OVERSAMPLING TASK
Table 9: Comparison between SOS and STaSy y in terms of Weighted-Fi
Methods
Default
HTRU
Magic
Shoppers
Robot
Identity 0.514620.0072
0.8657#0.0163
0.7807+0.0348
0.5673+0.0055
0.9031 #0,048
SOS
0.5618#0.0016
0.8767#0.000 0.7949#0.0011
0.6568+0.005
0.9197=0.0040
STaSy
0.5683+0.0026
0.8803=0.0027
0.7960=0.0015
0.6595+0.0044
0.9270=0.0029
In this section, we discuss the difference between SOS and STaSy. They are both based on SGMs, but they are optimized towards different goals by using different objective functions and training strategies. SOS has many design points specialized to augment minor classes only (rather than synthesizing entire tabular data) - for instance, SOS adopts style-transfer-base idea to convert major class sample to a minor one via their own SGM model without any considcration on the training difficulty of the denoising score matching. However, our STaSy more focuses synthesizing entire tabular data by proposing special self- eeff-paced training and fine-tuning methods.
We use five imbalanced datasets to oversample many datasets in our main experiments not imbalanced. We conduct the oversampling experiments with STaSy to compare with SOS (Kim) et al., 2022). In this experiment, STaSy is converted design guidance of sos, i.e., each minority class has its own score network and is separately trained. We train, for fair comparison, STaSy w/o fine-tuning for each minor class and generate minority samples to be the same size of the majority class. We compare the two models in terms of the sampling quality using Weighted-Fl which specialized in evaluating imbalanced data. We note that Ident i yy means that we do not use any oversampling methods, which is, therefore, a minimum quality requirement upon oversampling.
oversampling method following the
As shown in Table g STaSy w/o fine-tuning outperforms SOS. The result shows that our proposed training strategy, i.e., the self-paced learning, improves the model training regardless of tasks.
C NETWORK ARCHITECTURE
We propose the following score network Sø (x(((t),
ho x(t), h, WHH t) + 1 dy
(x() FC(haw),
13

---

Published as a conference paper at ICLR 2023
where x(t) is record (or row) time t in tabular data, h, is the i-th hidden vector, and w is an activation function. dy is the number of hidden layers. For various layer types of H2 (h;- t), we provide the following options:
(C;((; o ((FC((())
Squash,
H (); = FC, @ h,-1),
if Concat,
(h,(h,,)) O (bFC (1) FC if Concatsquash,
where we can choose one of the three possible layer types hyperparameter, means the element- wise multiplication, @ means the concatenation operator, v is the Sigmoid function, and FC is a fully connected layer. We modify the architecture of (Song et 2021 by using the layer types, being inspired by (Grathwohl et al., 2018).
D THRESHOLD CONTROLLING MECHANISM
Our threshold controlling mechanism is designed to meet the following 3 requirements: i) it can control when the entire dataset is used for training, starting from a subset, ii) should gradually increase the size of used training records while logarithmically decreasing the number of hard records (that are not involved in training), and iii) it should be a monotonically increasing/decreasing function guarantee that the training difficulty gets more challenging the training process goes on.
The threshold controlling variables a and 3, where 0 < a S B < 1, are gradually increased to to involve the entire data records for training. We increase them proportionally training steps, where
((5))) 10---0 8))) -80)- the natural logarithm, any and By are initial values of a and 3, c the current training step, and S determines when to utilize the entire data records. We use 10,000 for S. We set Bn at least 0.8 to ensure 80% of the data records are involved in training at the start the training. We set a = oo and / 3o at the beginning of the training, since is O. We note that training sample xy whose quantile of loss value is greater than 3 is regarded as a hard sample and is set to be O. If we set So be 0.8, the top 20% of difficult samples will not used the first training step. In this way, one can control the proportion of training samples involved in the training from the beginning.
+ log 11
(1-oa) on) and 3 30 + log
is the base of
E PROOF OF THEOREM
As defined in Section 3.2 the STaSy objective is follows:
Q(a) - ((3)
N
min 8.V
Σ o
Q(3)
Σ
(13)
2
where (; the score matching loss for i-th training record in Equation 6 We can rewrite the optimal solution for each training record U; with respect to fixed 0 in the veriex form. Let L(v;) be the objective with fixed O, which quadratic function with respect to V, Then,
[(o:) v,l, Q(a) (
u Q(3)
2
Q(a) - Q(3)
op (4 Q(3)o,
2
Q(a)= - Q(3)
(12 2(4 Q(9))
Q(a) Q(8) v1
2
Q(a) - Q(3)
( 2(4 Q(3)) " +
4 Q(3)
Q(a) ((8)) ) }
4 Q(3)
- (Q(a))9(3))
2
Q(a) - Q(3)
Q(a) - Q(3)
1; Q(3) 2 +
Q(a) Q(3)
4 4-Q(3)
2
1/
Q(a:) - Q(3)
Q(aa) Q(3))
(14)
14

---

Published as a conference paper at ICLR 2023
Because Q(a)-Q(3) is less than equal to 0 and Q(a)-Q(())
4 ((8 Q(a) QQ()) ) is constant, the solution
which minimizes Equation 14 is U/ Q(a)- Q(ay 2y follows:
4=Q(9))
Considering 2 [0, 1], we can get the optimal
1,
4 - Q(a),
0,
if 2 Q(3).
1
(15)
4 ((3)
otherwise.
Q(a) - ((3)
F THE HUTCHINSON'S S ESTIMATION AS A FINE-TUNING OBJECTIVE
We use the denoising score matching loss in Line of Algorithm T In this section, we describe the results of an additional experiment in which the Hutchinson's 's log-probability estimation is used for the fine-tuning objective.
TO summarizzs the F1 score and the median of fog-probabilitess when we use the denoising score matching loss and the Hutchinson's 's estimation as the tine-tuning objective. In Cont rateetaceptiii there does not exist clear winner between the two fine-tuning objectives, and similar results are also shown other datasets. However, in some datasets, e.g., Shoppers and Crowdsou ree, the former shows better F1 scores and better medians of the log-probabilities than the latter by large margins. In addition, when we update using the Hutchinson's 's estimation, in Defaul t, the sampling quality is lower than before fine-tuning. Considering these results, we use the denoising score matching loss, which shows the generalizability, as our default fine-tuning objective.
Table 10: We report the score and the median the log-probabilities of testing records according to the fine-tuning objective.
STaSy w/o fine- mne-tnnin
Fine-tine with
Fine-tune with
Datasets
the denoising matching loss
the Hutchinson' estimation
F1
Log-probability
F1
Log-probability
F1
Log-probability
Default
0.51,50.0008
131.011
0.51920.009
131.122
0.51240.008
106.184
Shoppers
0.639#0.008 000
189.422
0.640=0.008
220.589
0.636#0.00m 009
206.924
Contraceptiv
0.42410.014
225.334
0.425=0.003
225.638
0.425as.0011
225.392
Crowdsource
0.712+0.140
47.567
0.717=0.126
48.905
0.710=0.133
48.683
G ADDITIONAL EXPERIMENTAL RESULTS
G.1 SAMPLING QUALITY
We mainly use Fl (resp. It²) for the classfication (resp. regression) TSTR evaluation, and also report AUROC and Weighted-F1 (resp. RMSE) results. Full results for all datasets are in Tables IT 12 and
13 for binary classification. Tables 14 15 and [6 for multi-class classification, and Table 17 for regression. We train and test various base tassifiersrrerresoors and report their mean and standard the log-probability
deviation. Moreover, results are in Table [8] The best results are highlighted in bold face and the second best results with underline. As shown, Sesive--TaaS and STaSy show the best and the second best performances in almost all cases.
another metric for the sampling quality. Full
15

---

Published as a conference paper at ICLR 2023
Table 11: Classification with real data. We report F1 for binary classification.
Binary classification
Methods
Credit
Default
HIRU
Magic
Phishing
Shoppers
Spambase
Identity
0.76520.071
0.442a0.099
0.88020.009
0.77820.064
0.94820.024
0,934 0.601
MedGAN
0.000=00000
0.00000.000
0.000=0:000
0.553+0.068
0+0.500.000
0.198 1322
0.438=0.217
VEEGAN
0.000=0:000
0.36720.002
0.403=0.266
0.546x0.063
0.795+0. 108
0.322 082
0.522z0.091
CIGAN
0,743=0
0.48920.024
0.85
0.72620.032
0.89420.008
0.509 052
0.773+0.030
TVAE
0.026
0.43220.038
0.84920.008
0.69320.016
0.91 120.000
0.75220.036 0.573
TableGAN
0.729c0:00
0.425
0.73620.048 2084
0.90320.007
0,586200.06
0.710
OCT-GAN
0.113=0.171
0.484=0.016
0.863=0.011
0.725+0.018
0,904+0.006
0.619+0.03 037
0.858+0.023
RNODE
0.077=0.082
0.39520.013
0.58220.035
0.746x0.032
0,89420.013
0.554 026
0.82220.048
Naive-STaSy
0.737=0.083
0.50620.013
0.881
0.78 +0.008
0.925:0.010
0.630e0.015
0.897=0.031
STaSy
0.795=0.034
0.51920.009
0.882:0.005
0.783+0.058
0.930:0.012
0.640=0.008
0.912:0.024
Table 12: Classification with real data. We report AUROC for binary classification.
Binary classification
Methods
Credit
Default
HTRU
Magic Phishing Shoppers Spambase
Identity
0.968=0021
0.764=0016
0.969=0.00
0.907+0.039
0.988+0.009
0.920+0014 0.983+0.018
MedGAN
0.500=00000
0.50020.000
0.470
0.68820.070
0.6602a.174
0.76420.067
0.80920.067
VEEGAN
0.660=0.10v
0.498=0.061
0.818=0.192
0.773+0.036
0.869+0.121
0.782+00075
0.707=0.066
CTGAN
0.957=00033
0.749=0.006
0,950=0.020
0.869+0.025
0.969+0.007
0.847=0003
0.904=0.027
TVAE
0.702=0.12
0.743+0.012
0,954=00012
0,843+0.029
0.979+0.004 0004
=0.0200020
0.901=0.031
TableGAN
0.928=0.051
0.66020.028
0.957=0.013
0.887+0.027
0.972+0.006
0.85620.038
0.92920.041
OCT-GAN
0.803=0.114
0.72620.012
0.957=0.015
0.86520.018
0.97320.005
0.88820.017
0.94920.020
RNODE
0.88220.093
0.718200013
0.928
0.874x0.032
0.96620.007
0.89020.021
1.
Maive
0.958=0.038
0.74920.018
0.967=0.011
0.899:0.039
0.984+0.005
0. 0,908=0.009
0, 0.96920.025
STaSy
0.959=0.039
0.744=0.019
0.966=0.009
0,899+0.038
0.986:0.005
0.915:0.008
0.97620.019
Table 13: Classification with real data. We report Weighted-F1, which is inversely weighted to its class size, for binary classification.
Binary classification
Methods
Credit
Do faulu
HTRU
Magic
Phishing Shoppers Spambase
Identity
0.770=0003
1
0.890=n.099
0.818+0.051
0.953++.022
0.650x0.051
0.944+0.021
MedGAN
0.002-0000
0.19770.000
0.087+0000
0.482+0. 166
0.343+0.003
0.288+0.098
0.527+0 121
VEEGAN
0.002=00000
0.285=0.000
0.419=02288
0.520=0
0.799+0.11
0.424=0.039
0.570=0.083
CIGAN
0.746=0 069
0.57120.019
0.86420.007
0.77520.025
0.90620.007
0.56020.048
0.80220.028
TVAE
0.041
0.531
0.86220.007
0.73510.009
0.91920.006
0.62410.028
0.770
TableGAN
0.737=0.037
0,506#0.012
0.855=0.005
0.786+0.038
0.90.3+0.006
0.642+0.057
1...190.093
OCT-GAN
0.103=0.160
0.560=0.019
0.874=0.010
0.770+0.015
2+0.05
0.66440.032
0.877=0.021
RNODE
0.089=00080
0.503
0.620=0.037
0.78920.027
0.90420.011
0.610200023
0.85
Na MaIve-STaSy STaSy
0.744=0.085 0.590=n.010
0.89224.006
0.932+0.009 0.820+0.047
0.673
0.91 1
STaSy
0.795=9.036
0.596:0.008
+0.91
0.820
0.937
0.681:4.006
0.926=0.020
16

---

Published as a conference paper at ICLR 2023
Table 14: Classification with real data. We report macro F1 for multi-class classification.
Multi-class classification
Methods
Bean
Contraceptive Crowdsource
Opesity
Robot
Shuttie
Identity
0.932a0.009
0.470a0.023
0.760a0.100
0.96420.010
0.973
0.90420.11
MedGAN
0.05820.000
0.412+0.012
0.153+0.013
0.187=0.082
0.28 =0.06
0. 0.126e0.00
VEEGAN
0.270
0.31320.032
0.13720.000
0.176
0.297=0.008
0.126:00.00
CTGAN
0.881
0.34020.016
0.525
0.116=00018
0.655a0.14 0.64620.037
TVAE
0.892
0.387+0.027
0.137=0.000
0.456=0.031
00.719
0.1 26=0.000
TableGAN
0.606=+0.000
0.327+0.021
0.406+0.088
0.294=0.024
0.400=0.032
0.396=0.010
OCT-GAN
0.912=0.018
0.376+0.00 007
0,579+0.1 111
0.253=0.030
0.768+0.035
0.626=0.00
RNODE
0.805=0.114
0.350s0.012
0.13720.000
0.472
0.494s0.112
0.126:0.000
Na Navve-S rayyy
0.930=0.010
0.393+0.043
0.702+0.
0.902=0:026
0.940=0.033
0.754=0.044
STaSy
0.932+0.009
0.425+0.003
0.717#0.126
0.902+0.025
0.940=0.028
0.800s0.043
Table 15: Classification with real data. We report AUROC for multi-class classification.
Multi-class classification
Methods
Bean
Contraceptive Crowdsource Obesity
Rebot
Shuttle
Identity
0.992-0.006
0.692a0.015
0.94620.053
0,99720.008
0.99520.006
0,99920.002
MedGAN
0.500=0.000
0.622+0.022
0.599
0.694+0. 110 0.600+0.068
0.500+0.000
VEEGAN
0.720=0.115
0.602=0.045
0,500=0.00 0.000
0.574+0.020
0.585+0.054
0.500+0.000
CTGAN
0.972=0.036
0.515#0.024
0.88 #0.045
0.49620.008
0.82010.037
0.963+0.044
TVAE
0.980=0.016
0.60020.048
0.500+0.000
0.83320.007
0.93920.045
0.5002a.000
TableGAN
0.898=00000
0.563a00013
0.872#0.058
0.737+0.034
0.701
0.784+4.076
OCT-GAN
0.982=0.021
+0.018
0.913=0.053
0.663+0.033
0.923+0.026
0.960+a.046
RNODE
0.973=0.026
0.532+0.020
0.500+0.000
0.8 1
0.842+0.087
0.500+0.000
NaIve-STaSy
0,986=0.01
0.631=0.016
0.931+0.072
0,981+00025
0,988:0.012
0,967+0.058
STaSy
0.991=0.008
0.651#0.004
0.950=0.046
0.987:0.012
0.987+0.014
0.980+0.036
Table 16: Classification with real data. We report Weighted-FF which inversely weighted to its class size, for multi-class classification.
Multi-class classification
Methods
Bean
Cont Conertctiivve Crowdsource
Obesity
Robot
Shut te
Identity
0.935200099
0.448aau17
0.630M0.276
0.96240015
0.96720.04
0.902a0.109
McdGAN
0.050=0.000 000
0.385#0.012
0.066:0.007
0.180x0.074
0.230+0.052
0.032+0.000
VEEGAN
0.272=0.095
0.29420.036
0.050=0.000
0.181
0.257+0.018
0.032+x.000
CTGAN
0.882=0.015
0.331
2..81
0.116220017
0.642a0.047
0.583a0.1 166
TVAE
0.893
0.369a0.021
0.050#0.000
0.453+9.021
0.76840.040
0.032+4.000
TableGAN
0.606=0.031
0.325=0.022
0.337
0,292+0.020
0.363+4.033
0.308
OCT-GAN
0.915=00016
0.364++0.09
0.533+0.12 120
0.251+
0.764+4.036
0.558+0. 108
RNODE
0,804=0.122
0.333+0.019
0.050+0.000
0,472+0.062
0.467+0.124
0.032+0.000
SeTiv--TTaSS
0.933=00010
0.37740.048
0.674#0.116
0.899+0.028
0.939+4.030
O. 0.7044a.035
STaSy
0.935=a.009
0.400+0.004
0.678+0.153
0.903+0.025
0.939+0.026
0.769+a.079
17

---

Published as a conference paper at ICLR 2023
Table 17: Regression with real data. We report R and RMSE for regression.
R
RMSE
Methods
Hei jinng
News
Hei jing
News
Icentity
0.741
0.162a0.022
0.691a0.104
0.85320:043
MedGAN
-4.268+2.100
9.50322.145
2.560 655
2.61920.024
VEEGAN
-6.44020.566
-3.36220.702
3.12
1.88010.1e 162
CTGAN
0.210s0.217
0.035
1.012a0.131
0.880=40012
TVAE
0.385
-0.273=0.067
0.902+0.014
1.015+0.027
Tabl eGAN
0.219+0.054
-1.795=0.988
1.018+0.037
1.437+0. 148
OCT-GAN
0.379 087
-0.172=0.1 167
0.908+0.00
0.984+00084
RNODE
0.282+0.117
-1.248=1.237
0.974+0.078
1.25 1+0.270
NaTve-STaßy
0.648+0.088
-0.006=0.088
0.678+0.082
0.900=0.039
STasy
0.679a0.115
0.042+0.072
0.647=0.111
0.885+0.042
Table 18: The median of the log-probabilities of testing records are reported.
Datasets
RNODE
STaSy w/o fine-tuning
STaSy
Credit Default
71.248
117.001
117.404
55.284
131.011
131.122
Binary
HTRU
27.852
5,435
5.326
Magic
17.413
30.247
30.216
Phishing
53.478
228.912
228.914
Shoppers
62.843
189.422
220.589
Spambase
140.686
253.935
253.798
Bean
46.764
92.555
92.635
Contraceptive
26.797
225.334
225,638
Multi.
Crowdsource
23.323
47.567
48.905
Obesity
130.018
330.003
333.017
Robot
14.238
79.233
79.355
Shuttle
48.875
114.247
114.294
Reg.
Beijing
42.614
17.720
18.192
News
128.472
76.770
76.600
18

---

Published as a conference paper at ICLR 2023
G.2 SAMPLING DIVERSITY
We use the coverage score as a metric for the sampling diversity. Full results for all datasets are in Tables 19 20 and 21 We measure the coverage score 5 times with different fake records and report their mean and standard deviation.
Coverage is bounded between O and I, and higher coverage means more diverse samples. This k-NN-based measurement is expected to achieve 100% performance when the real and fake records are identical, but in practice, this is not always the case. For the dataset whose coverage score does not show for two same data records, we choose the hyperparameter & to achieve at least greater than 0.95. In our experiments, k for Phi Phishing is 7, and for others, k is 5. As shown in Tables 19 20 and 21 in 12 out of 15 datasets, our methods outperform others by large margins.
Table 19: Sampling diversity in terms of coverage for binary classification datasets
Binary classification
Methods
Credit
Default
ETRU
Magic
Phishing
Shoppers Sparbase
MedGAN
0.000+0.000
0.000=0:00
0.000+0.000
+000000000
0.002+0.001
0.002+0000 0.000+D
VEBCAN
0.000s0.000
0.00010.000 0000000000000000000000000.0...00
0.00010:000
0.003:10000
0.03520.000
0.141
0.201
CTGAN
0.174
0.190aa.001
0.461
0.65510.003
0.41620.001
0.72310.006
0.471
TVAE
0.373+0.001
0.272=0.002
0.741+0.001
0.650+0.003
0.623+0.002
0.737+0.001
0.698+0.002
Tabl @GAN
0.458+0.002
0.21640.00
0.299+0.002
0.748+0.002
0.512+0.007
0.698+0.008
0..11
OCT-GAN
0.17 0.000s0.000
0.375a0.004
0.63420.001
0,450:0.003
0.714x0:000
0.470a0016
RNODE
0.025+0.000
0.293
+0751
0.61 240.200003
0,272+0.008
0.447+0.004
0.326H0.099
Naive-STaSy
0.01420.000
0.921 0.149
0.919a0.002
0.65520.006
0.832x0.005
0.68420011
smaSy w/o fine-tuning
0.013+0.000
0.083+0.001
0.907+0.02
0.943+0.003
0.780=n.005
0.796+0.006
0.727=0.006
STaSy
0.014x0.000
00000
0.91 1+0000
0.94420.001
0.779:0.005
0.798+0.004
0.727=0013
Table 20: Sampling diversity in terms of coverage for multi-class classification datasets
Multi-class classification
Methods
Bean
Contraceptive Crowdsource Obesity
Robot
Shut t
MedGAN
0.000+0.000
0.538+0.007
0.0000=p.000
0.007+0.001
0.000++0000
0.000+0.000
VEEGAN
0.004 001
0.082+0.002
0.000=0.000
0.10020.003
0.005a0.001
0.001
CTGAN
0.053+0.001
0.753+0.011
0.06444..000
0.252+0.015
0.14004.004
0.031+0.000
TVAE
0.118
0.68020.011
0.254g0.003
0.29720.003
0.472a0.014
0.111
TableCAN
0.11 003
0.751
0.33920.001
0.440:0.006
0.25920.003
0.00510.000
OCT-GAN
0.106420000
0.76420.011
0.13320.000
0.345
0.250sa0004
0.02320.000
RNODE
0.292:9.006
0.547=0.011
0.10420.003
0.375
0.110220004
0.005=0.000
Naive- STaSyy
0.092a0:003
0.839a0.012
0.920=0.004
0.82510.006
0.93520.007
0.13320.003
STaSy w/o fine-tuning
0.095
0.879+0.007
0.970=0.002
0.777+0.013
0.937=0.004
0.207=0.001
STaSy
0.10010.005
0.89410.010
0.971=0.005
0.77840.008
0.937=0.004
0.209
Table 21: Sampling diversity in terms of coverage for regression datasets
Regression
Methods
Bei Beijing
News
MedGAN
0.000+0.000
0.000*0.000
VEEGAN
0,000s0.00%
0.002*0.000
CTGAN
0.532+0.00 0002
0.366+0.001
TVAA
0.720 0002
0.665=00.01
TableGAN
0.803 0003
0.154
OCT-GAN
0.693+0.000
0.582+0.000
RNODE
0..00
0.255+0.002
Na Naive-STal
0.876+0.00 0003
0.755+0.003
STaSy w/o tine-tuning
0.943
0.762+0.000 0004
STraSy
0.900
0.762+0.002
19

---

Published as a conference paper at ICLR 2023
G.3 SAMPLING TIME
Tables 22 23 and 24 show runtime evaluation results of each method. We measure the wall-clock time taken to sample fake records times, and report their mean and standard deviation. In almost all datasets, Na ve ve -Syyy and STaSy show faster runtime than OCT-GAN and RNODE. Tabl 1 eGAN and TVAE take a short sampling time, but considering their inferior sampling quality and diversity, only our proposed model resolves the problems of the generative learning trilemma.
Table 22: Wall-clock runtime for binary classification datasets
Binary classification
Methods
Credit
Default
HTRU
Magic
Phishing
Shoppers
Sparbase
MedGAN
0.756G0.328
0.223+0.534
0.201
0.205=0.333
0.
0.187s0.328 0.203
VEBGAN
0.80020.031
0.112a0.0m
0.05120.005
0.061=0.007
0.041
0.04110.006
0.01920.002
CTGAN
6.342+0.300
0.546+0.054
0.299+0.085
0.284=0.002
0.163=20.05
0.107=00008 0.20440.005
TVAE
0.93740.185
0.052+0000
0.028+0.001
0.049=0.004
0.022-0.004
0.039+0.00
0.014+0.005
TableGAN
0.381s0.025
0.043+0.007
0.01322.001
0.027=0.007
0.01920.007
0.02020.007
0.011:0.005
OCT-GAN GAN
241.104
28.206+33599
10.312+3.688
11.08343.701
8.622+3.704
7.74 +3.714
4.224 +3.680
RNODE
10.61 3eb. 139
=2.701
4.346+0007
2.576=0.622
11.385+0.067
16.461=1.326
21.270+0.12 121
Na NaV'v STaBy
79.202+0.350
7.192+0.320
3.27550.245
3.466=0.254
0.733+0,239
3.774+4.327
2.234+0.229
STaSy
79.085+0.449
9.073+0.119
11.903+0201
3.788-0.284
3.287=0.267
50
2.17340
Table 23: Wall-clock runtime for multi-class classification datasets
Multi-class classification
Methods
Bean
Cont racept i ve Crowdsource Obesity
Robot
Shuttle
MedGAN
0.218
0.174.0.135
=0.366
0.196=a.334
0.181=0.338
0.27 +00.300
VEEGAN
0.029=0.006
0.006=0.000
0.033=0.006
0.009+0.001
0.024=0.010
0.102:0.004
CTIGAN
0,227+0.009
0.025=0.00 002
0,252+0.027
0.036+0.002
0.113=0.014
0.712+0.009
TVAE
0.023=n.002
0.004=0.001
0.038200005
0. 0.039#0.040
0.009=00000
0. 0.138=0.008
TableGAN
0,026:0.00 008
0,00220.000
0.021=0.007
0.004:0.001
0.00620.000
0.041=0.007
OCT-GAN
6.595x3.725
2. 2.84993.678
6.917s3.743
2.885a3.613
5,01323.670
29 46993.67777
RNODE
5.210s0.234
12.452=0.270
7.985
23.675a0.432
10.86120.800
6,8 6.81920.257
-SaSy
2.797
1.34620.220
3.090s0.237
1.387=0.232
1.692=0.215
12.935+0.85
STaSy
2.750
1.045202247
3.12620.229
1.023a0.237
1.37
13.35220.382
Table 24: Wall-clock runtime for regression datasets
Regression
Methods
Bei Beijing
News
MedGAN
0.212+0.334
0.284=0.344
VEEGAN
0.067=0.005
0.234=0.037
CTGAN
0.346+0.023
0.902+0.013
TVAE
0.036+0.009
0.075=0.003
TableGAN
0.023+0.008
0.052+0.005
CCT-GAN
18.008+3.566
20. 20.86643.648
RNODE
19.380+0.500
34.955=0.276
Na Naive-STaSy
7.338+0. 146
2.362+0.241
STaSy
7.818+1.074
1..13241.241
H EXPERIMENTAL ENVIRONMENTS
Our software and hardware environments are as follows: UBUNTU 18.04 LTS, PYTHON 3.8.2, PY- TORCH 1.8.1, CUDA 11.4, and NVIDIA Driver 470.42.01, i9 CPU, and NVIDIA RTX 3090. Our code for the experiments is mainly based on https: /ttp://///thtbbbt com/ wtt.:/cw.ooooooooee score ssde_pytorch (Apache License 2.0).
20

---

Published as a conference paper at ICLR 2023
H.1 BASELINES
We utilize a set of baselines that includes various generative models.
Identity is a case where do not synthesize but use original data.
2017 is GAN that incorporates non-adversarial losses generate MedGAN Choi et al.
discrete medical records.
VEEGAN' Srivastava et 2017) is GAN for tabular data that avoids mode collapse by adding reconstructor network.
CTGAN' (Xu et al. 2019) and TVAE' (Xu et al. 2019) are a conditional GAN and a VAE for tabular data with mixed types of variables.
TableGAN' (Park et al. 2018) is GAN for tabular data using convolutional neural networks.
OCT-GAN (Kim et al. 2021 GAN that has generator and discriminator based neural ordinary differential equations.
RNODE Finlay al., 2020) is advanced flow-based model with two regularization terms added to the training objective of FFJORD (Grathwohl et al. 2018).
H.2 DATASETS
In this section, we describe real-world tabular datasets for our experiments. We select the datasets for experiments with metrics: 1) how many times dataset has been cited/used in previous papers, and 2) how many times a dataset has been viewed/downloaded Machine Learning Repository and Kaggle. Among them, choose the datasets that can be used for classification and regression tasks, with more than 5 columns and 1,000 rows.
famous repositories, such as UCI
Credit a binary classification dataset collected from European cardholders for credit card fraud detection.
Default (Lichman 2013) is binary classification dataset describing the information on credit card clients in Taiwan regarding default payments.
HTRU (Loon 2017) is binary classification dataset that describes sample of pulsar candidates collected during the High Time Resolution Universe Survey.
Magic (Bock) 2007) binary classification dataset that simulates the registration of high-energy gamma particles in the atmospheric telescope.
Phishing (Mohammad] 2015) is a binary classification dataset used to distinguish between phishing and legitimate web pages.
Shoppers
(C Okan Sakar 2019 9
binary classification dataset about online shoppers'
intention.
binary classification dataset that indicates whether an email is spam or non-spam.
Spambase (Hopkins)
1999)
Bean (Koklu & Ozkan 2020) a multi-class classification dataset that includes types of
beans with their characteristics.
Contraceptive [Lim 1997) is multi-class classification dataset about Indonesia con-
traceptive prevalence.
Crowdsource (Johnson & lizuka 2016) is multi-class classification dataset used to classify satellite images into different land cover classes.
Obesity (Palechor de la Hoz Manotas 2019 is a multi-class classification dataset describing obesity levels based on eating habits and physical condition.
Robot (Freire 2010) is a multi-class classification dataset collected as the robot moves around the room, following the wall using ultrasound sensors.
htteas///attbbbb.ccm///dddd////a (MTT (MTT License)
https://qithub.. htteas//attbbb..oooo/beoooysssss /oop://ooyyyosee//////// /OC
(MIT License)
21

---

Published as a conference paper at ICLR 2023
Table 25: Datasets used for our experiments
Datasets
#train
#test
#continuous
#categorical
task (ftcaas)
Credit
264.8K
20K
29
Binary classification
Default
24K
6K
13
11
Binary classification
HTRU
14.3K
3.6K
1
Binary classification
Magic
15.2K
3.8K
10
1
Binary classification
Phishing
8.8K
2.2K
0
31
Binary classification
Shoppers
9.8K
2.4K
10
8
Binary classification
Spambase
3.7K
0.9K
57
Binary classification
Bean
10.8K
2.7K
16
Multi-class classification
Contraceptive
1.2K
0.3K
Multi-class classification
Crowdsource
8.6K
2.1K
Multi-class classification
Obesity
1.6K
0.4K
10
Multi-class classification
Robot
4.4K
1.IK
Multi-class classification
Shuttle
46.4K
11.6K
Multi-class classification
Beijing
15.2K
3,8K
6
Regression
News
31.6K
8K
45
14
Regression
Shuttle (shu) is a multi-class classification dataset for extracting conditions in which automatic landing is preferred over manual control of the spacecraft.
Beijing (Liang et 2015 is regression dataset about PM2.5 air quality in the city of
Beijing.
News (Fernandes) 2015) is regression dataset about online news articles to predict the number of shares in social networks.
The statistical information of datasets used in our experiments is in Table 25 ous, #categorical, and #class mean the number of training data, testing data, continuous columns, categorical columns, and class, respectively.
#train, #test, #continu-
The raw data of 15 datasets are available online:
Credit: 1.0)
httppp//www..kgggg.crror/g////tr (DbCL
Default: http://///cooii../.......//////r
Tttp:///aaas///ollo............r 4.0)
Magic:
http:://eeeileea.ss.u.e////1//12 websites (CC BY 4.0)
Phishing:
Shoppers:
Sthpapashtssssoo/occ....ccooeoer (CC
BY 4.0)
Bean:
Dataset (CC BY
Contracept
Crowdsource:
http:////peeeaaa..aaaaaaa/////aa conditions (CC BC
22

---

Published as a conference paper at ICLR 2023
Robot:
https://archive. uci wct..cdd..mm//aaasassss
Shuttle:
(Shuttle) (CC BY
Beijing:
5+Data (CC BY 4.0)
News: popularity (CC B.0.0
H.3 EVALUATION METHONS
The reported scores for TSTR results in the paper are calculated as follows:
1. We download dataset. If used previously, we perform a new train-test split. The train-test split ratio is 80% and 20%, respectively.
their train-test split. If not used before,
2. Generate fake records which has same number records as the original training set for other fake data generation methods.
3. Using the training records from Step2, we train base classiffers/regresorrs to predict. We search the best hyperparameter set each classiffer/regressorr For validation, we use original training data's 's task scores, e.g., Fl or R2. In Table 26 considered hyperparameters and their candidate settings are summarized. We use DecisionTree AdaBoost, Logistic Regression, MLP classifiers, RamdomForest. and XGBoost for binary classification tasks; Decision Tree, MLP classifiers, RandomForest, and XGBoost for multi-class classification tasks; MLP regressor, RandomForest, XGBoost, and Linear Regression for regression tasks.
4. Test the best performing classifiers/regressor with testing data. We use various evaluation metrics for rigorous evaluations as reported earlier.
We repeat the 4th step five times for all datasets. We then calculate the average score for each method and for each evaluation metric. Detailed metrics for our experiment are as follows:
1. Binary F1 for binary classification datasets: fl _l_score from delearn.metricn after setting the 'average' option 'binary'.
2. Macro Fl for multi-class classification datasets: fl [l_score from the "average" option to 'macro'.
after setting
3. Weighted-F1 for classification datasets: Weighted-F1 E Wist, where N is the number of classes, the weight of i-th class w; is I-p; N-1 P; is the proportion of i-th class's ' cardinality in total dataset, and score is a per-class Fl of i-th class (in a One-vs-Rest manner). This formula allows us to evaluate synthesized tables with more focus on mode collapse by giving higher weight to smaller class, which is more likely to be forgotten by the model.
4. AUROC: toc_auc_score from sklearn.metrics.
5. Coverage: compute_prde from
https://github.. http:///iihubbcoo///e cow.oovvaaaccoooaaaaaaaaaoa
renerative-evallatioon
I HYPERPARAMETERS
Hyperparameter settings for the best models are in Table 27 We have three SDE types, which are VE, VP, and sub-VE and three layer types as shown in Appendix C Concat, Squash, and Concatsquash. We use a learning rate in {2e - 03, 2r 04). We search for Ogg and 3o. in total, with 9 combinations using og (0.20, 0.25, 0.30) and fo (0.80, 0.90, 0.95).
We also consider the hyperparameters for the fine-tuni fine-tuning process. To compute the exact log- bogoobbaayylity with the Hutchinson's 's estimation (Hutchinson 1989 Grathwohl et al. (0018), we use Gaussian or Rademacher distribution for p(e), where e is noise vector. The fine-tuning epoch is (1,...,20) 20). and the fine-tuning learning rate {2 10-1/; {4,5,66 6)).
23

---

Published as a conference paper at ICLR 2023
Table 26: Hyperparameters of the base classifietsrreerrssooss
Models
Hyperparameters
Values
max _ee_h
8, 16, 32 2, 8
DecisionTree
min _a___ssmmllppp_p
min min_ saissssmllessssee leaf
1,2, 4, 8
AdaBoost
n esimators
10, 50, 100, 200
solver n n_jobs
Ibfgs
-1
Logistic Regression
max_iter iter
10, 50, 100, 200 0.1, 0.0001, 0.01, 0.1
C
tol
hidden layer_sizee
(100, ) (200,), (100, 100) 50. 100
MLP
max_iter iiter
alpha
0.0001, 0.001
max depth min_ bin_ssmples_aae split min sii__samples_eer n_jobs
8, 16, Inf 2,
RandomForest
1,
-1
esttimattors min_child weight
10, 50, 100
1, 10
XGBoost
max depth
5, 10, 20
gamma
0.0, 1.0 -1
nthread
Linear Regression
Table 27: The best hyperparameters used in Table 2
Hyperparameters for SPL of STaSy
Hyperparameters for fine tuning
Datasets
SDE Type
Layer Type
Activation
Leam. Rate Og
Hutchinson Type
Leam. Rate
Credit
VP
concatsquash
LeakyReLU
2e-03
0.25 0.90
Rademacher
2e-05
Default
VP
concatsquash ReLU
2e-03
0.30 0.90
Rademacher
2e-04
HTRU
VE
concatsquash LeakyReLU
2e-04
0.25 0.95
Rademacher
2e-05
Mag' c
sub-
squash
ReLU
2e-03
0.95
Rademacher
2c-07
Phishing
VP
squash
LeakyReLU
2e-03
0.2 0.95
Rademacher
2e-07
Shoppers
VE
concatsquash
ELU
2e-03
0.30 0.95
Rademacher
2c-07
Spanbase
VP
concat
LeakyReLU
2e-04
0.95
Rademacher
2e-07
Bean
squash concatsquash
ReLU
2e-03
0.80
Rademacher
2e-05
iontraceptive
VP
LeakyReLU
2c-03
0.2 0.95
Rademacher
2c-07
Crowdsource
VE
squash
LeakyReLU
2e-03
0.25 0.90
Gaussian
2e-07
Obesity
VP
squash
ELU
2c-03
0.25 0.90
Gaussian
2c-05
Robot sub VP
concat
ELU
2e-03
0.3 0.95
Rademacher
2e-07
Shuttle
sub-
squash
ReLU
2c-03
0.95
Rademacher
2c-07
Beijing
concatsquash LeakyReLU
2e-03
0.90
Rademacher
2c-07
News
VE
concatsquash LeakyReLU
2c-03
0.25 0.90
Gaussian
2c-07
24

---

Published as a conference paper at ICLR 2023
J ADDITIONAL VISUALIZATIONS
We show several visualizations that are missing in the main paper. In each subsection, we show column-wise histograms and I-SNE visualizations on HTRU, Robot, and News, respectively,
J.1 ADDITIONAL VISUALIZATIONS IN HTRU
As shown in Figure 5 the fake data distributions of TVAE, TableGAN, and RNODE are dissimilar to the real data distributions, and these baselines fail to sample high-quality fake records. In Figure 6 CIGAN, TableGAN, OCT-GAI and RNODE suffer from mode collapses as highlighted in red. In sTasy, however, the mode collapse problem is clearly alleviated, which means that STaSy is effective in enhancing the diversity.
Real
Raaal VEEGAN
Real
Real
RodSaM
CTGM
TIAE
STa5y
Figure 5: Histograms of values in the excess kurtosis column of HTRU
Redl
TabladNi
SCT-40 G4B
Figure 6: t-SNE visualizations of fake and original records in HTRU
25

---

Published as a conference paper at ICLR 2023
J.2 ADDITIONAL VISUALIZATIONS IN ROBOT
In Robot, TVAE, OCT - GAN, and STaSy generate relatively similar distribution to that of real data shown in Figure In Figure 8 CTGAN, TableGAN and RNODE generate out-of-distribution records as highlighted in red, and TVAE and OCT-GAN suffer from mode collapses. Our proposed methods generate the most diverse records among various methods.
Dual RedGAM
Rual VERGAN
Ital
Real
CTON
TVAE
Real TobloGA9
Real OCT-GA8
Real RMODE
Rewl 5T25y
Figure 7: Histograms of values in the ultrasound sensor column of Robot
Rold
Real
Sail
Peol
Figure 8: t-SNE visualizations of fake and original records in Robot
26

---

Published as a conference paper at ICLR 2023
J.3 ADDITIONAL VISUALIZATIONS IN NENS
In News, the fake data by STaSy shows more reliable column-wise histogram than others in Figure 9 As shown in Figure TO all methods except for TableGAN and RNODE well generate fake records.
Roal
Roal
Real
Real
HudEaM
VEEGAM
CTAAN
TWAE
TableGoo eGos
OCT-GA6
RIOOT
STASy
Figure 9: Histograms of values in the min of best keyword column of News
Knal
Real
Real
Fax
Figure 10: I-SNE visualizations of fake and original records in News
27


Reviews:
[{"review_id": "z27OP1uU1Us", "reviewer": "Reviewer_A3vU", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The authors propose the use of score-based generative modeling (i.e., diffusion models) for general tabular synthesis, and demonstrate the effectiveness this approach across a large number of tabular datasets and synthesizers (e.g., CTGAN, VEEGAN, TableGAN).  Building on this approach, the authors introduce a novel self-paced learning algorithm following by fine-tuning (the combination of which is called STaSy), which further improves upon the direct application of score-based generative modeling.  Extensive studies and results are shown, demonstrating the the strengths of the additional components of STaSy, which achieves SOTA in all performance evaluations (but is an order of magnitude slower than other GAN-based approaches during test time).", "summary_of_the_review": "High quality paper with minor comments and suggestions for improvement.  Only major concern is with the runtime of the method, suggest authors address this in the main paper.  I recommend acceptance.", "strength_and_weaknesses": "Strengths\n------------\n-The paper's contributions are significant: 1) Demonstration of score-based generative modeling as a superior means of tabular synthesis (compared to direct GAN approaches), (2) A new novel SPE algorithm, (3) A new tabular synthesizer, STaSy, which demonstrates impressive performance on the evaluated tasks\n\n-Extensive results and ablation studies are compelling and convincing to show the utility of the SPE and fine-tuning algorithms\n-The paper is well written\n\n-The use of simple feature pre/post-processors is a big win, as mixture-model based pre-processing (i.e., those used in CTGAN/TVAE) are extremely expensive for large-scale data\n\nWeaknesses\n---------------\n-The runtime of STaSy is a major concern.  E.g., in Table 1, STaSy runtime is more than an order of magnitude slower than CTGAN, which itself struggles on high-dimensional data (e.g., it is not scalable for feature dimensions >150).  Could the authors comment on how STaSy would fare in such high-dimensional regimes?  Since runtime is such an important practical metric, it would greatly strengthen the paper if the authors could further discuss how STaSy's runtime could be decreased.  E.g., could you speak to the possibility of using [1] to avoid Gaussian sampling in the reverse process and speed up sampling time?\n\n-While well-written, there is a lot of content in the main text, as well as the appendix.\n\n-\"we follow the \"train on synthetic, test on real (TSTR)\" framework\": this has become a widely adapted framework in tabular synthesizer papers.  However, it is impractical, and does not actually show how such synthesizers may be realistically leveraged in practice.  E.g., while privacy arguments are commonly made, the most important categorical features cannot be synthesized directly (e.g., a new email would not be generated by STaSy, CTGAN, VEEGAN, etc.).  Yet, it is impressive that STaSy is the first method which achieves identical testing performance given purely synthetic training data (as well as being a rare example where performance under purely real training data is even reported).  A much better demonstration of utility is data augmentation.  I think including more discussion of the oversampling result in the appendix would further show how impressive this method is.\n\n[1] Xiao, Zhisheng, Karsten Kreis, and Arash Vahdat. \"Tackling the\ngenerative learning trilemma with denoising diffusion gans.\" arXiv\npreprint arXiv:2112.07804 (2021).", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "iitLZ5lsEHP", "reviewer": "Reviewer_AiyZ", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper studies a score-based generative model (SGM) for tabular data. They found there's a difficulty in adopting SGM to tabular modeling that the training process is unstable. Hence they introduced a self-paced learning algorithm, a variation of curriculum learning, to train SGM. In experiments, the proposed method outperforms other generative models, including GAN-based models, with several real datasets. ", "summary_of_the_review": "A solid SGM is proposed for tabular data synthesis. Although there's room to improve the description of the method/algorithm derivation, the paper's quality is sufficient for publication. ", "strength_and_weaknesses": "Strengths\n- A reasonable adaptation of SGM for tabular data.\n- The performance is quite nice. \n- The performance is evaluated in both quantitative and qualitative ways. \n- The paper provides rich information (comparison with another SGM model, details of experiments, etc) in Appendix. \n\nWeaknesses\n- The motivation for introducing self-paced learning is not clearly explained. \n- There are additional hyperparameters to tune in the proposed method.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "hP-LFlzPIJ", "reviewer": "Reviewer_dEhg", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This work proposes to generate tabular datasets with score-based generative models. For improved performance, authors apply self-paced learning to create a curriculum where harder data points are trained later than easier ones. A final fine-tuning process is proposed to further improve the performance on hard data points. Experiments demonstrate that the proposed Stasy method outperforms existing generative models in creating realistic tabular data samples that can fool classifiers. Ablation study corroborates the efficacy of self-paced learning and fine-tuning.", "summary_of_the_review": "Nice written paper with simple but effective approaches and strong experimental results.", "strength_and_weaknesses": "## Strength\n\n1. Tabular data is a very important data modality. Improving the performance on tabular data generation is very valuable for privacy protection, data augmentation, etc.\n\n2. Paper is clearly written. The proposed method is straightforward, properly motivated, and easy to implement.\n\n3. Experiments demonstrate the efficacy of StaSy at generating tabular data from multiple directions. Ablation study clearly shows the advantages of self-paced learning and fine-tuning.\n\n## Weaknesses\n\n1. The finetuning process as given in Algorithm 1 uses log likelihood to compute the threshold, but it seems that authors actually used equation (6) instead of log-likelihoods. Why not change Algorithm 1 to use equation (6) instead of log-likelihood? You can mention in text that log-likelihood is also a natural choice but turns out to perform worse in practice.\n\n2. Some minor writing issues need to be fixed:\n\n  * The bottom line on page 1: \"However\" is an awkward word to be there.\n\n  * Right below equation (3): remove \"the\" before \"denoising score matching\"\n\n  * First line in Theorem 1: replace \"is\" with \"be.\n\n  * Also in Theorem 1: the optimal solution is not defined, but proved in equation (9)\n\n  * Section 3.4: \"However\" is again a bad choice of word.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "EVaYTKkIiT", "reviewer": "Reviewer_xhm1", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper considers the problem of generating tabular data with a similar distribution to an existing dataset. It provides a technique for generating such artificial data, using SGM (score-based generative models). The results are further improved by using self-paced learning and fine-tuning of the model per dataset. The new method is compared with 5 previously proposed methods, showing that the new method generates data that is harder to distinguish from the real data and covers a larger portion of the distribution (about 1.5x better), but runs significantly slower than most previous methods (two were slower, three were ~100x faster). The paper compares modeling results on the artificial data with 7 previous methods, as well as with real data (the test data is the same real data in all cases). The evaluation is done on 15 datasets, including binary classification, multiclass classification and regression. The training is done using several models (e.g., XGBoost, MLP, Decision tree, Adaboost, Logistic Regreesion, Linear Regression), a hyperparameter search is done for each of them and the best result is used (average of 5 runs). The paper shows that training the models on the new artificial data yields better predictions than on the data generated using previous methods. The results are close to the results obtained when training on real data. The paper additionally presents illustrations that show examples where previous methods generated out of distribution samples, partial coverage or differences in distribution, while the new method performed well.\n", "summary_of_the_review": "This paper presents an interesting technique for an important problem. It could be useful, for example, when the actual data is private and cannot be shared. However, I have several concerns about the evaluation and about presentation, as described in the weaknesses paragraph above. Based on the authors response I will reconsider my score.", "strength_and_weaknesses": "The paper shows an interesting new approach to an important problem, and it is clearly written. However, I have several concerns:\n- The first comparison of quality, diversity and run-time was only done with 5 out of the 7 previous methods. Why weren't all of them checked?\n- It is unclear which data was used for the hyper-parameter tuning on each dataset (no validation set was defined and no cross-validation was mentioned).\n- The hyper-parameter search for the classification/regression models is very limited. What are the state-of-the-art results obtained for these datasets? Are they close to the presented results?\n- It would be interesting to see how the results of one state-of-the-art model, such as XGBoost, are impacted by using different types of artificial data (or real data), while trying either just the default hyperparameters, or a large set of possible hyperparameters.\n- In the Appendix, the results on the artificial data are sometimes better than the results on the real data - what could explain this?\n- Why the comparison with SOS was only done using three datasets? (in the Appendix)\n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: notable-top-25%