Distributional Learning of Variational AutoEncoder: Application to Synthetic Data Generation
Authors: Seunghwan An
Jong-June Jeon
Venue: NeurIPS 2023 poster (2023)
Keywords: Variational AutoEncoder
distributional learning
synthetic data generation
CRPS
asymmetric Laplace distribution

Abstract: The Gaussianity assumption has been consistently criticized as a main limitation of the Variational Autoencoder (VAE) despite its efficiency in computational modeling. In this paper, we propose a new approach that expands the model capacity (i.e., expressive power of distributional family) without sacrificing the computational advantages of the VAE framework. Our VAE model's decoder is composed of an infinite mixture of asymmetric Laplace distribution, which possesses general distribution fitting capabilities for continuous variables. Our model is represented by a special form of a nonparametric M-estimator for estimating general quantile functions, and we theoretically establish the relevance between the proposed model and quantile estimation. We apply the proposed model to synthetic data generation, and particularly, our model demonstrates superiority in easily adjusting the level of data privacy.

Distributional Learning of Variational AutoEncoder: Application to Synthetic Data Generation
Seunghwan An and Jong-June Jeon" Department of Statistical Data Science, University of Seoul, S. Korea {dkstmdghks79, jj jeoneenn ac.kr kr
Abstract
The Gaussianity assumption has been consistently criticized as main limitation of the Variational Autoencoder (VAE) despite its efficiency in computational modeling. In this paper, we propose a new approach that expands the model capacity (i.e., expressive power of distributional family) without sacrificing the computational advantages of the VAE framework. Our VAE model's decoder composed infinite mixture of asymmetric Laplace distribution, which possesses general distribution fitting capabilities for continuous variables. Our model is represented by special form of nonparametric M-estimator for estimating general quantile functions, and we theoretically establish the relevance between the proposed model and quantile estimation. We apply the proposed model to synthetic data generation, and particularly, our model demonstrates superiority in easily adjusting the level data privacy.
1 Introduction
Variational Autoencoder (VAE) 31 51
and Generative Adversarial Networks (GAN) 22 are
generative models that used estimate the underlying distribution of given dataset. To avoid the curse of dimensionality, VAE and GAN commonly introduce a low-dimensional latent space on which conditional generative model is defined. By minimizing information divergence between the original data and its generated data, the generative models are learned to produce synthetic data similar the original one. Accordingly, VAE and GAN have been applied in various applications, such as generating realistic images, texts, and synthetic tabular data for privacy preservation purposes
30 60 63 65 34).
However, the difference in the strength of the assumption about the generative distribution brings significant contrasts in the VAE and GAN generation performances [7]] In the GAN frame- work, the adversarial loss enables direct minimization of the Jensen-Shannow divergence between the ground-truth density function and the generative distribution under distributional assumption B 59]. Roughly speaking, the GAN employs nonparametrie model as its conditional generative model defined on the latent space.
On the contrary, in the VAE framework, the Gaussianity assumption has been favored [3 39. It is because Gaussianity gives us three advantages: 1) the reconstruction loss can be interpreted as the mean squared error that one of the most popular losses in optimization theory, 2) generating sample is computationally straightforward, and 3) KL-divergence is computed in a simple closed form. However, these benefits have led us to pay the price for the distributional capacity of the generative model, in that the generative model of the VAE is constrained in the form of marginalization of the product the two Gaussian distributions. Here, the distributional capacity means the expressive power of the distributional family. This restricted distributional capacity has
"Corresoondigg author.
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

been the critical limitation 8 33 and leads to heavy parameterization of the decoder mean vector to approximate complex underlying distributions.
To increase the distributional capacity in synthetic data generation, 66 65
introduce the multi-
modality in the distributional assumption of the decoder, which is known the mode-specific normalization technique. Although the mixture Gaussian decoder modeling of [63 651 allows handling more complex distributions of the observed dataset while preserving all of the advantages of Gaussianity, we numerically find that the mixture Gaussian is not enough to capture the complex underlying distribution.
Our main contribution is that, beyond Gaussianity, we propose a novel VAE learning method that directly estimates the conditional cumulative distribution function (CDF) while maintaining the objective of maximizing the Evidence Lower Bound (ELBO) of the observed dataset. It implies that we have nonparametric distribution assumption the generative model. We call this approach distributional learning of the VAE, which is enabled by estimating an infinite number of conditional
quantiles 4 [20]. By adopting the continuous ranked probability score (CRPS) loss, the objective function of our proposed distribution learning method is computationally tractable (21 43 20
In our proposed distributional learning framework, 1) the reconstruction loss is equivalent to the CRPS loss, which is a proper scoring rule [3). 2) generating a new sample is still computationally straightforward due to the inverse transform sampling, and 3) KL-divergence is still computed in simple closed form. To show the effectiveness our proposed model in capturing the underlying distribution of the dataset, we evaluate our model for synthetic data generation with real tabular datasets.
2 Related Work
Modeling of the decoder and reconstruction loss. To increase the distributional capacity, many papers have focused on decoder modeling while not losing the mathematical link to maximize the ELBO. 5 assume their decoder distributions as Student-t and asymmetric Laplace distributions, respectively, to mitigate the vero-variance problem that the model training becomes unstable if the estimated variance of the decoder shrinks to zero Gaussian VAE 41 55 15
proposes a general distribution of the decoder, which allows improved robustness by optimizing the shape of the loss function during training. Recently, (S] proposes reconstruction loss that directly minimizes the blur error of the VAE by modeling the covariance matrix of multivariate Gaussian decoder.
On the other hand, there exists research direction that focuses on replacing the reconstruction loss without concern for losing the mathematical derivation of the lower bound. [38 52 46) replace the reconstruction loss with adversarial loss of the GAN framework. (27) introduces reature-based loss that is calculated with a pre-trained convolutional neural network (CNN). Another approach by [1] adopts Watson's 's perceptual model, and (28) directly optimizes the generative model in the frequency domain by focal frequency reconstruction loss. Most of the above-mentioned methods aim to capture the properties of human perception by replacing the element- loss (I, or L2-norm), which hinders the reconstruction of images 38
Synthetic data generation. The GAN framework is widely adopted in the synthetic data generation
task since it enables synthetic data generation in nonparametric approach [12 [7] 63]6633636 65). [63 65
assume that continuous columns of tabular datasets can be approximated by the Gaussian mixture distribution and model their decoder using Gaussian mixture distribution. Additionally, [63 65
preprocess the continuous variables using the variational Gaussian mixture model [3). which known as the mode-specific normalization technique. However, the preprocessing step requires additional computational resources and hyperparameter tuning of the number of modes. Other approaches by [47 65| regularize the discrepancy between the first and second-order statistics of the observed and synthetic dataset. [12] proposes the GAN-based synthesizer, which focuses on generating high-dimensional discrete variables with the assistance of the pre-trained AutoEncoder.
Proposal
Let x E R8+4 be an observation consisting of continuous and discrete variables and 7= Ic U = {1.*.,, (p + 4)} be an index set of the variables, where Ic and Ip correspond to index sets of

---

continuous and q discrete variables. T₃ denotes the number of levels for the discrete variables x,.J In. We denote the ground -tuth underlying distribution (probability density function, PDF) as p(x) and the ground-truth CDF as F(x).
Let z be latent variable, where z E Rd and d < < q. The prior and posterior distribution of z are assumed to be p(z) = (z I) and q( ②) = N (z) Ø), diag(o (x; o))). respectively. Here, I is d x identity matrix, 14 RP-4 + Rd _² R2*9 - R neural networks parameterized with 6, and diag(a), a E IR denotes diagonal matrix with diagonal elements a. Moreover, we consider [0, 1] as a random variable having density p(a).
3.1 Distributional Learning
Our proposed model assumes that p(x) is parametrized by infinite mixture of asymmetric Laplace distribution (ALD) 4 The ALD is characterized by two parameters: o, representing the asymmetry, and / > 0, representing the scale. By considering these parameters, along with the model parameter 8, we can define the probability model of x as follows:
= J p(xlz, a; (1)p2,,2
p(x; 0,
Assumption 1. (1) {x}),e jer are conditionally independent given (2) The discrete random variables KyLDEI are independent of a. (3) a and z are independent.
By Assumption (1),) we model the dependency between x,s solely through the latent variable
49). Assumption (2) implies that a is related only to the continuous variables. Then, the decoder of our VAE model denoted as p(x|z, 0, B) is specified by equation Q
p(x|z,a;;,1)
II jElc
p(x a: az (3,
II JELD
(1)
a(1
(-pa exp -Pa
X, Dj(a, z; 2;0,)
II JETc
II II jEID 1=1
(z; 8 ((x,
3
B
where 0 = (01.-.. 0ptg). ). 8 is non-trainable constant, (v) = u(v - I(u < 0)) (check function), and ((() denotes the indicator function. D)(--;0,)= 1] x Rd + R is the location parameter of ALD, which is parameterized with 0j []. For discrete variables, (=;0)); Rdd + A -1 is a neural network parameterized with f,. where A T; is the standard (T, - 1)-simplex for all z E Rdd and the subscript / referes the (th element of the output n.
Assumption (3) leads our objective function,
T3
min 0.6
Ep(x) Bg(e)xo))))
EL Pa px(xx - 2;)) da Σ jEID 1=1
Σ
>
I(x, = 1) log (z; 0,)
jETc
B [XC]gg/) })(l(())]],
where constant terms omitted. In order achieve balanced learning of the reconstruction losses in (2), we have removed the weight associated with the second reconstruction loss. We refer our model as 'DistVAE.' The first term in (2) corresponds the CRPS loss, which measures the accuracy of the proposed CDF approximation with respect to the ground-truth CDF of the underlying
distribution 21 43 20
Interestingly, 2 is the limit of the negative ELBO derived from a finite mixture of ALD. We introduce a, discrete uniform random variable taking values ag k/K for k 1,... K. Then, the negative ELBO p(x; 0, ,3) scaled with 3 is written by
K
Pon X; - Di(as, 0;))) D)(assz200)))
K
Ey(as (x:b)
Σ jELc
-3
Σ
log a4 ax) + Bp log 3
K
K
k=
1=1
T,
- 3 Eg((l)oc)))
Σ
I(x; = log (zg
+ 3 (x)(((//:)))lp(2)
(3)
JElo 1=1
3

---

(see Appendix A.1 for detailed derivation). The reconstruction loss, which corresponds to the first
term of (3). composite quantile loss for estimating the target quantiles ags 64 35 45 62
This entails adopting a Bayesian perspective for a prior (the Bayesian modeling for estimating multiple quantiles). Furthermore, throughout the derivation of the reconstruction loss, the role of is pivotal in ensuring the representation of the reconstruction loss. To prevent the observation from influencing the distribution a, a is only assigned with a prior distribution, and the resulting reconstruction loss becomes the CRPS loss, proper scoring rule.
However, for distributional learning of VAE, it necessary to estimate conditional quantiles for an infinite number of quantile levels. denoted by K - do [50]. The subsequent Theorem | establishes the convergence the negative ELBO ( to our objective function (2) as K
Theorem 1. For all j E Tc, suppose that Jo Pottscaa
D2(a, 0;)) < co, and Ep(x) (pcetxss) - (D) (a z; 02)) is continuous over a E (0, Then,
K
Pan (x, Dy(ak, (1))
Bp(x) By(elxxd) [/ Pa
Pa(x (xx -Dj(a, z; 01) 0;))
lim Ry(llxd)) K0000
R=1
and limg-xe K
Ek= log aex ox) -ft log a(1 a)da -2
3.2 Theoretical Results
In this section, we aim to provide theoretical insights into the ability of DistVAE, utilizing the objective function (2). recover the ground-truth distribution p(x). To simplify the analysis without loss of generality, we consider the scenario where comprises only continuous random variables. Hence, we have = Ic {I.... .). and p(x) is defined over x R2 with p(x) >> for all x E I.. First, define function g(x| Φ) by
oxx|z;)) = p(x)q(z(x:c))
q(z
where ((2; Φ) f [z|xx; ())(()dx is the aggregated posterior x for given z. q(x 2; Φ)) a conditional PDF parametrized by Φ and is an approximated PDF of lo p(x/z, a; 0, d)... Since we assume that x,S are conditionally independent in Assumption q(x/2; Φ) 11 Oj (x) /2: Φ)
58
Clearly, ((x|2; σ) is PDF of
and the conditional CDF is written as
P F(x|z; Φ) II (x,,2;0)) where (x(x)1z;))== 1 (z(z) Ø)dr. j=1
(4)
For notational simplicity, we let
Σ 1=1
(O()) E arg min (x(x)(((xx)))
Pa(x) (,(a);;0)))da,
where 8" (0) (.. ()))
Assumption 2. (1) Given arbitrary 6, F5( zz Φ) [0, is absolutely continuous and strictly monotone increasing for all j 1, 1,... P. and Rd. (2) Given arbitary 6, Dy( z; f2) is invertible and differentiable for all j = 1.... ,p and E RR. (3) The aggregated posterior q(: o) is absolutely continuous to the prior distribution of
Theorem 2. Under Assumption for an arbitrary 4b
/ dx,
P
d (7
KL
p(x)
0.
Theorem 2 shows that DistVAE is capable of recovering the ground-truth distribution p(x), indicating its ability to facilitate distributional learning rather than data reconstruction. Nevertheless, relying on the aggregated posterior distribution may lead to overfitting [25] 42, and sampling from the

---

aggregated posterior can introduce computational challenges due to the absence of a straightforward closed-form representation for q(; 6). To address these concerns, we propose an alternative approach that leverages the prior distribution p(z) instead of g(2; Φ),, thereby enabling computationally efficient synthetic generation process. This substantiated by Theorem
We define the estimated PDF p(x; 0 (+)) and CDF F(; 0* (6)) as
x(x;0(()))
/
II j=1
P
d
Dj (xj. 2; ()()))((z)))
(5)
dxj
0(x;0(())
(1)))))
/
III (x,,
(6)
==1
Theorem 3. Suppose that & is given such that K.((q(z; [[z)) for any O. Then, under Assumption
2x <c. / j=1
KL p(x)
dx,
Theorem shows that even if use the prior distribution p(z) instead of the aggregated posterior ((2; Φ), it is feasible to minimize the KL-divergence between the ground-truth PDF p(x) and our
KL--ivergence term in (2) is the upper bound of K.C(g(z; Φ) [p(z)) and is minimized during the training process. Since each
estimated PDF p(x; 0* (6)) of (5) This is achievable because
conditional distribution of the estimated PDF depends on the same latent variable, can be seen that the correlation structure between covariates is modeled implicitly.
24 37 have highlighted the role of the KL-divergence weight parameter / in controlling the precision of reconstruction. In our case, where the reconstruction loss is based on the CRPS loss, an increase in leads a less accurate estimation of the ground-truth CDF. implies that large corresponds to lower-quality synthetic data, but it also enhances privacy level. Thus, 3 introduces trade-off between the quality synthetic data and the risk of privacy leakage. The privacy level can be adjusted by manipulating 3 [7], as demonstrated in the experimental results presented in Section
4
3.2.1 Synthetic Data Generation
By estimating the conditional quantile functions. transform the synthetic data generation process into inverse transform sampling. This conversion offers a notable advantage as it provides a straightforward and efficient approach generating synthetic data. We denote synthetic sample of Xj as ē for j E I, and the synthetic data generation process can be summarized as follows:
1. Sampling from the prior distribution: p(z).
2. Inverse transform sampling: For j E Ic. f. (2
03), where u, U(, 1). 3. Gumbel-Max trick [23:: For j E ID. 3t arg max:=1,- Ty (log (2; 0,) Gt), where
G ~ Gumbel(O, 1), and 1, ...
Note that both continuous and discrete variables share the same latent variable z. This shared latent variable allows for capturing the dependencies between variables. We numerically observe that the sampling, implemented using the Gumbel-Maz trick, maintains the imbalanced ratio of the labels in the discrete variable.
3.2.2 Parameterization of ALD
As introduced 20 for j E Ic, we parameterize the function Dj. the location parameter of ALD,
by a linear isotonic spline follows:
M
Dj(a, z, 0,) = (2) +
( (z)(( a dm)+ s.t.
Σ m=0
6 (z) 2 0, = 1,... .,,,, (7)
M=D
5

---

where (2) E IR, (z) 1b0g) (z). (z)) E RM++,, d =
E [0, 1j
0 = do < < dM = 1, and (u)+ = max 05 is a neural network parameterized mapping such that 8) Rd + R x R RM+1, which takes z as input and outputs (z) and 800) (2). The
constraint introduced to ensure monotonicity. As demonstrated in (20), the reconstruction loss can be computed in a closed form by utilizing the parameterization of 7 (refer to Appendix A.5 for detailed description of the loss function). This implies that our objective function (2) is computationally tractable. Note that the linear isotonic spline is not differentiable for finite points where the measure has no point mass.
4 Experiments
4.1 Overview
Dataset. For evaluation, we consider following real tabular datasets: covertype, credit, loan, adult, cabs, and kings (see Appendix [A88 for detailed data descriptions). We treat the ordinal variables as continuous variables and discretize the estimated CDF (see Appendix [A.6] for the discretization algorithm). Synthetic samples of ordinal variables are rounded to the first decimal
place.
Compared models. We compare Dist with the state-of-the-art synthesizers; CTGAN (S)
TVAE 663, and CTAB-GAN 63). All models have the same size of the latent dimension (d = 2). The chosen latent space indeed limits the capacity of decoders for all models. However, we maintain small and consistent number of parameters across all models during the experiment isolate the performance differences in synthetic data generation to the methodologies of each synthesizer, specifically emphasizing the contribution of the decoder model's flexibility in estimating underlying distributions (see Table in Appendix(A.99) comprehensive comparison of the model parameters).
4.2 Evaluation Metrics
To assess the quality of the synthetic data, we employ three types of assessment criteria: 1) machine learning utility, 2) statistical similarity, and 3) privacy preservability. Each criterion is evaluated using multiple metries, and the performance of synthesizers is reported by averaged metries over the real tabular datasets. The synthetic dataset is generated to have an equal number of samples as the real training dataset.
Machine learning utility. The machine learning utility (MLu) is measured by the predictive performance of the trained model over the synthetic data. We consider three popular machine learning algorithms: linear (logistic) regression, Random Forest [6], and Gradient Boosting [19]. We measure the performance by utilizing the mean absolute relative error (MARE) for regression tasks 47 and
the F1 score for classification tasks [63 65
61
29
18.
Statistical similarity. The marginal distributional similarity between the real training and synthetic datasets is evaluated using two metries: the Kolmogorov statistic and the 1-Wasserstein distance 18] These metrics measure the distance between the empirical marginal CDFs [40] The joint
distributional similarity assessed by comparing the correlation matrices (65]. To compute the
correlation matrix and measure the T-2 distance between the correlation matrices of the real training
and synthetic datasets, we employ the dython library | These enable a comprehensive evaluation of both marginal and joint distributional similarities between the real training and synthetic datasets.
Privacy preservability. The privacy-preserving capacity is measured using three metrics: the distance to closest record (DCR) [4 651,, membership inference attack [54] 12, 47). and attribute As in (65) the DCR is defined as the 5th percentile of the L2 distances between all real training samples and synthetic samples. Since the DCR is a L2 distance- based metric,
disclosure 12 44
computed using only continuous variables. higher DCR value indicates more effective preservation of privacy, indicating a lack of overlap between the real training data and the synthetic samples. Conversely, an excessively large DCR score suggests lower quality of the generated synthetic dataset. Therefore, the DCR metric provides insights into both the mrivacy-preserving capability and the quality of the synthetic dataset.
We release the code at http://hw//ggggh...o..aaaaaaaaaa http::/hhhhtttttyyyy///ssss////s

---

The membership inference attack evaluation follows the steps detailed in Appendix A.7 The procedure is customized to be applied a VAE-based synthesizer, such as DistVAE and TVAE. By transforming the problem into a binary classification task, we aim to identify the intricate relationship between the real training data and the synthetic samples. Higher binary classification scores indicate higher vulnerability of the target synthesizer to membership inference attacks.
Attribute disclosure refers to the situation where attackers can uncover additional covariates of record by leveraging a subset of covariates they already possess, along with similar records from the synthetic dataset. To quantify the extent to which attackers can accurately identify these additional covariates, we employ classification metrics. Higher attribute disclosure metrics indicate an increased risk of privacy leakage, implying that attackers can precisely infer unknown variables. In terms of privacy concerns, attribute disclosure can be considered a more significant issue than membership inference attacks, as attackers are assumed to have access to only a subset of covariates for a given record [2].
4.3 Results
Machine learning utilityy We expect a high-quality synthesizer to generate synthetic data with comparable predictive performance to the real training dataset, denoted as the 'Baseline' in Table I The results in [de that DistVAE achieves a competitive MARE score and outperforms other methods in terms of the F1 score. Furthermore, the performance of MIu improves as the value of B decreases, indicating that the quality of the generated synthetic data controlled by (3. For a comprehensive overview of the MLu scores for all tabular datasets, please refer to Appendix A.10
Table 1: Averaged MLu metries (MARE, Fi). Mean and standard deviation values are obtained from 10 repeated experiments. 1 denotes higher is better and denotes lower is better.
Table 2: Averaged correlation structural similarity. 'CorrDist' represents Lg dis- tance between the correlation matrices of synthetic and real training datasets. Mean and standard deviation values are obtained from 10 repeated experiments. Lower is better.
Model
MARE L
Fi t
Baseline
0.150
0.814=0.101
CTGAN
0.321
10.27 10.271
0.672 0.67220.234
Model
CorrDist
TVAE
0.225 20.215
0.594-0.295
20.992
0. 0..12
CTGAN
2.180
TVAE
2.739 20.796
DistVAE(8 = 0.5)
20.492
0.769 =01128
CTAB-GAN
2.575
DistVAE(8 = 1)
0.344
0.762_0.133
DistVAE(3 = 5)
0.392
0.679 1.190
DistVAE(3 = 0.5)
1.473 20.398
DistVAE(3 1)
1.730 10.548
DistVAE(3 = 5) 3.113
Statistical similarity. The evaluation results for joint and marginal distributional similarities are presented in Table 3 In Table DistVAE achieves the lowest CorrDist score, indicating its ability to accurately preserve the correlation structure of the real training dataset the generated synthetic dataset. Furthermore, DistVAE surpasses other methods in Table 3 when it comes to marginal distributional similarity, suggesting that it successfully captures the underlying distribution of the observed dataset. Notably, reducing the value of leads to an enhancement in the quality of the synthetic dataset, as evidenced by improvements the correlation structure and similarity of the marginal distributions. Figure provides visualizations of the estimated CDFs (6) for each continuous (or ordinal) variable in the cabs dataset. For detailed statistical similarity scores and additional visualizations of estimated CDFs for all tabular datasets, please refer to Appendix A.
Privacy preservability. The privacy preservability performances of synthesizers, measured by the DCR, are presented in Table 4 DistVAE performs best in preserving privacy. with the highest DCR values compared to other methods. Notably, as the value of 3 increases in DistVAE, the DCR between the real training and synthetic datasets (R&S) also increases. This indicates that the risk of privacy leakage can be controlled by adjusting 3, where higher values of /3 correspond to higher level of privacy protection. Moreover, DistVAE consistently achieves large DCR values for the synthetic dataset (S) across all values, indicating ability to generate diverse synthetic samples. On the other hand, CTAB-GAR generates duplicated records in the synthetic dataset, resulting in
7

---

Table Averaged marginal distributional similarity. K-S denotes the Ballooorro--mmineee statistic.
and I-WD represents the I-Wasserstein distance. Mean and standard deviation values are obtained from 10 repeated experiments. Lower is better.
Continuous
Discrete
Model
K-S
1-WD
K-S
1-WD
CTGAN
0..3000000
0.08740.002
0. 19.155
0.521 20.32
TVAE
0.196=20.13
0.220
0.385
1.681=1.668
CTAB-GAN
0.157=0.089
0.130=0.037
0.106=0.083
0.412=0.378
DistVAE(3 = 0.5)
0.090=0.000
0.075=0.020
0.030
0.118_0.100
Dist 1)
0.081 030
0.083=0.019
0.027 #0.021
0.116 00116
Dis(VAE(3 = 5)
0.092=0.037
0.121
0.059+0.034
0.241
6.75
0.75
0.75
0.75
6.50
0.50
0.50
0.50
0.50
0.50
0.00
0.00
0.00
0.00
0.00
0.00
0
10
0
Figure 1: cabs dataset. Empirical (solid orange) and estimated (dashed blue) CDFs of continuous and ordinal variables (Monte Carlo approximated with 5000 samples). We standardize covariates and remove observations outside the 1% and 99% percentile range.
relatively lower DCR scores for the synthetic dataset (S). For detailed DCR scores for all tabular datasets, please refer to Appendix(A.10 A.10
Table 4: Privacy preservability: Averaged distance to closest record (DCR) between real training and synthetic datasets (R&S), between the same real training datasets (R), and between the same synthetic datasets (S). Mcan and standard deviation values are obtained from 10 repeated experiments. The DCR (R) score represents the baseline diversity of datasets. Higher better.
Model
R&S
R
S
CTGAN
0.426+0.222 22.
0.237=+0113
0.35644..22 20.20
TVAE
0.470 0.47040.188 10.181
0.237
0.278 19.195
CTAB-GAN
0.508 20
0.237
0.0394u.073
DistVAE(3 = 0.5)
0..444u..250
0.23741.153
0.463
DistVAB(8 = 1)
0.463 +0.33-0.200 282
0.237
0.4794u.310
DistVAE(3 = 5) 0.517 +0.272
0.237+0.153
0.511
To evaluate the membership inference attack, we prepare one attack model per class. The attack testing records comprise equal number of real training and test records, distinguished by the labels in and oul, respectively. Note that the test records are not employed in constructing the attack models. We employ pradient-boosting classifiers the attack models, and for computational feasibility, we limit the number of attack models to one (i.c., = 1).
For the membership inference attack evaluation, we utilize accuracy and AUC (Area Under Curve) the evaluation metrics. Since the target labels (in/out) are balanced, and the task binary classification problem, these metrics are appropriate. The results presented in Table 5 reveal that both VAs and TVAE achieve nearly identical accuracy and AUC scores of 0.5. This indicates that the attack models can nor distinguish between members of the real training and test datasets. Consequently, the membership inference attack is unsuccessful for both models. Therefore, DistVAE effectively generates synthetic datasets while ensuring privacy against membership inference attacks. comprehensive assessment membership inference attack performances for all tabular datasets can be found in Appendix A.10

---

Table 5: Privacy preservability: Averaged membership inference attack performance. Mean and standard deviation values are obtained from 10 repeated experiments.
Model
Accuracy
AUC
TVAE
0.495=0.019 0.500=0.003
0.495 0.495=0.018 20.019
DistVAE(3 0.5)
0.500
Table 6: Privacy preservability: Averaged attribute disclosure performance with the Fi score. Mcan and standard deviation values are obtained from 10 repeated experiments. Lower better.
Number of neighbors (k)
Model
10
100
CTGAN
0.26210.099
1..087
0.27510.088
TVAE
0,437 162
0,43844.166
0.432+0. 162
CTAB-GAN
0.257 123
0.258:+0.14
0.26140.111
DistVAB(B = 0.5)
0.328
0.328
0.310
Dís(VAB(8 = 1)
0.307 +0.073
0.313
0.297+10.00
DistVAB($ = 5)
0.265
0.253
0.232
We present the attribute disclosure performance results Table For each value of k, we observe that as 3 increases, the Fi score of DistVAE decreases. Also, DistVAE achieves the smallest Fi score when equals 10 and 100. Based on these results, we can conclude that DistVAE can generate synthetic datasets with low risk of attribute disclosure, and the level of privacy preservation is controlled by B. Please refer to Appendix A. for detailed evaluation of attribute disclosure performance for all tabular datasets.
Quantile estimation. To investigate the quantile estimation performance, we also evaluate DistVAE using the Vrate(a) metric [TT]. The Vrate(a) is defined as [Tesel Eiglure I(x; Qa), where E (0, 1), Irsst is the set of indices for the test dataset, I; is the i-th sample in the test dataset, and Q. is the empirical q--quanele of the synthetic data. Since Vrate(a:) indicates the proportion of compliance samples the test dataset, the Vrate(ox) score should be close a.
Table 7: Averaged Vrate(a:) and la Vrate(a)).
0.1
0.3
0.5
0.7
0.9
Vrate(a)
0.204
0.373
0.533
0.725
0.908
a Vrate(a))
0.104
0.083
0.04
0.032
0.008
The Vrate(a) evaluation results are presented in Table Table 7 shows that the ratio of violated test samples (/a Vrate(a)]) decreases as increases, indicating better performance for estimating larger quantiles. However, the ratio of violated test samples is relatively large for smaller a values, which may be due extremely skewed continuous variables, such as capital-gan -gain and capital- loss from adult dataset, that make the quantile estimation unstable.
5 Conclusion and Limitations
This paper introduces a novel distributional learning method of VAE, which aims to effectively capture the underlying distribution of the observed dataset using nonparametric approach. Our proposed method involves directly estimating CDFs using the CRPS loss while maintaining the mathematical derivation of the lower bound.
In our study, we confirm that the proposed decoder enhances the performance of generative models for tabular data. However, this conclusion relies on the assumption of conditional independence among the observed variables given the latent variable. When the dimension of the latent variable is small, this assumption is prone to violation. Therefore, in cases where the size of the latent

---

space is limited, the proposed nonparametric fitting of the decoder might not accurately represent the underlying distribution. Particularly in the image domain. where adjacent pixel values exhibit significant dependence, it remains uncertain whether our proposed model would lead to notable improvements in image data generation with a low-dimensional latent space.
Nevertheless, classical image datasets, such as CIFAR-I0 [36], often exhibit pixel value distributions that deviate considerably from Gaussian, with the frequencies of edge values (0 and 255) dominating
more than other pixel values [16] [3]. Other image datasets, as presented in 48 demonstrate
multi-modality in pixel value distributions. These experimental findings suggest that leveraging the capacity of distributional learning could be advantageous in approximating the ground-truie distribution of image data when the latent variable effectively captures conditional independence among the image pixels. Consequently, we expect that compromising biases arising from violating conditional independence and marginally misspecified distributions may further enhance our results, and we leave it for future research.
On the other hand, we consider two approaches to enhance the performance of quantile estimation. Firstly, we plan to extend the parameterization of conditional quantile functions to more flexible monotonic regression model. Secondly, we intend incorporate the Uniform Pessimistic Risk (UPR) (26] into the VAE framework to handle lower quantile levels better. Furthermore, we are exploring the expansion of Vis into a time-series distributional forecasting model by adopting the conditional VAE framework [56]. This extension will enable the application of our method to time-series data, opening new avenues for distributional forecasting.
Acknowledgments and Disclosure of Funding
This work was supported by the National Research Foundation of Korea (NRF) grant funded by the Korea government (MSIT) (No. NRF-2022R A4433033874 and No. 1F--2022IFFIA00740
This work was also supported by Korea Environmental Industry & Technology Institute (KEITI) through 'Core Technology Development Project for Environmental Diseases Prevention and Man- agement', funded by Korea Ministry of Environment (MOE) (202 1003310005) The authors ac- knowledge the Urban Big data and AI Institute of the University of Seoul supercomputing resources (http://ubal) uos. ac kr) made available for conducting the research reported this paper.
References
[1] Haleh Akrami, Anand Joshi, Sergul Aydore, and Richard Leahy. Deep quantile regression for uncertainty estimation in unsupervised and supervised lesion detection. Machine Learning for Biomedical Imaging, 1:1-23, 2022.
[2] Jonathan T. Barron. A general and adaptive robust loss function. 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 4326-4334, 2017.
[3] David M. Blei, Alp Kucukelbir, and Jon D. McAuliffe. Variational inference: A review for statisticians. Journal of the American Statistical Association, 1 112:859 877, 2016.
[4] Axel Brando, Jose A Rodriguez, Jordi Vitria, and Alberto Rubio Muñoz. Modelling heteroge- neous distributions with an uncountable mixture of asymmetric laplacians. Advances in neural information processing systems, 32, 2019.
[5] Gustav Bredell, Kyriakos Flouris, Krishna Chaitanya, Ertune Erdil, and Ender Konukoglu. Explicitly minimizing the blur error of variational autoencoders. In The Eleventh International Conference Learning Representations, 2023.
[6] Leo Breiman. Random forests. Machine learning, 45:5-32, 2001.
[7] Andrew Brock, Jeff Donahue, and Karen Simonyan. Large scale GAN training for high fidelity natural image synthesis. In International Conference on Learning Representations, 2019.
[8] Yuri Burda, Roger Baker Grosse, and Ruslan Salakhutdinov. Importance weighted autoencoders. CoRR, 509.0599,,, 2015.
[9] Alex J. Cannon. Non-crossing nonlinear regression quantiles by monotone composite quantile regression neural network, with application to rainfall extremes. Stochastic Environmental Research and Risk Assessment, 32:3207-3225, 2018.
10

---

[10] Lluis Castrejón. Nicolas Ballas, and Aaron C. Courville. Improved conditional vrnns for video prediction. 2019 IEEE/CVF International Conference on Computer Vision (ICCV). pages 7607-7616, 2019.
[11] Cathy W.S. Chen, Richard Gerlach, Bruce B.K. Hwang, and Michael McAleer. Forecasting Value-at-Risk using nonlinear regression quantiles and the intra-day range. International Journal of Forecasting, 28(3):557-574, 2012.
[12] E. Choi, Siddharth Biswal, Bradley A. Malin, Jon D. Duke, Walter F. Stewart, and Jimeng Sun. Generating multi-label discrete patient records using generative adversarial networks, In Machine Learning in Health Care, 2017.
[13] Antonia Creswell, Tom White, Vincent Dumoulin, Kai Arulkumaran, Biswa Sengupta, and Anil A Bharath. Generative adversarial networks: An overview. IEEE signal processing magazine, 35(1):53-65, 2018.
[14] Steffen Czolbe, Oswin Krause, Ingemar Cox, and Christian lgel. A loss function for generative neural networks based on watson's 's perceptual model. In Proceedings of the 34th International Conference on Neural Information Processing Systems, NIPS'20, 20, Red Hook, NY, USA, 2020.
Curran Associates Inc.
[15] Bin Dai and David Wipf. Diagnosing and enhancing VAE models. In International Conference on Learning Representations, 2019.
[16] Luke N Darlow, Elliot J Crowley, Antreas Antoniou, and Amos J Storkey. Cinic-10 10 is not imagenet or 10. arXiv preprint arxiv:/810.03500, 2018.
[17] Andrés Diaz-Pinto, Adrián Colomer, Valery Naranjo. Sandra Morales, Yanwu Xu, and Alejan- dro F Frangi. Retinal image synthesis for glaucoma assessment using degan and vae models. In Ideal, 2018.
[18] Kevin Fang, Vaikkunth Mugunthan, Vayd Ramkumar, and Lalana Kagal. Overcoming challenges of synthetic data generation. 2022 IEEE International Conference on Big Data (Big Data), pages 262-270, 2022.
[19] Jerome H Friedman. Greedy function approximation: gradient boosting machine. Annals of statistics, pages 1189-1232, 2001.
[20] Jan Gasthaus, Konstantinos Benidis, Yuyang Wang, Syama Sundar Rangapuram, David Salinas, Valentin Flunkert, and Tim Januschowski. Probabilistic forecasting with spline quantile function mns. In The 22nd international conference on artificial intelligence and statistics, pages 1901-1910. PMLR, 2019.
[21] Tilmann Gneiting and Adrian E. Raftery. Strictly proper scoring rules, prediction, and estimation. Journal of the American Statistical Association, 102:359 378, 2007.
[22] Ian J. Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron C. Courville, and Yoshua Bengio. Generative adversarial nets. NIPS, 2014.
[23] Emil Julius Gumbel. Statistical theory of extreme values and some practical applications: series of lectures, volume 33. US Government Printing Office, 1954.
[24] Irina Higgins, Loic Matthey, Arka Pal, Christopher Burgess, Xavier Glorot, Matthew Botvinick, Shakir Mohamed, and Alexander Lerchner. beta-VAES VAE: Learning basic visual concepts with constrained variational framework. In International Conference on Learning Representations, 2017.
[25] Mathew D Hoffman and Matthew J Johnson. Elbo surgery: yet another way to carve up the variational evidence lower bound. Workshop in Advances in Approximate Bayesian Inference, NIPS, volume 1, 2016.
[26] Sungchul Hong and Jong-June Jeon. Uniform pessimistic risk and optimal portfolio. ArXiv, abs/2303.0715 0588 2023.
[27] Xianxu Hou, L. Shen, Ke Sun, and Guoping Qiu. Deep feature consistent variational autoencoder. 2017 IEEE Winter Conference on Applications of Computer Vision (WACV), pages 1133-1141, 2016.
[28] Liming Jiang, Bo Dai, Wayne Wu, and Chen Change Loy. Focal frequency loss for image reconstruction and synthesis. 2021 IEFF/CVF International Conference on Computer Vision (ICCV), pages 13899-13909, 2020.
11

---

[29]
Sanket Kamthe, Samuel A. Assefa, and Mare Peter Deisenroth. Copula flows for synthetic data generation. ArXiv, abs/2101. 1000090,,0.0,,,, 2021.
[30] Tero Karras, Samuli Laine, and Timo Aila. A style-based generator architecture for generative adversarial networks. 2019 JEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 4396-4405, 2018
[31] Diederik Kingma and Max Welling. Auto-Encoding Variational Bayes. In 2nd International Conference on Learning Representatiion, ICLR 2014, Banff. AB, Canada, April 14-16, 2014, Conference Track Proceedings, 2014.
[32] Durk P Kingma, Shakir Mohamed, Danilo Jimenez Rezende, and Max Welling. Semi-supervised learning with deep generative models. In Advances in neural information processing systems, pages 3581-3589, 2014.
[33] Durk P Kingma, Tim Salimans, Rafal Jozefowicz, Xi Chen, Ilya Sutskever, and Max Welling. Improved variational inference with inverse autoregressive flow. In D. Lee, M. Sugiyama, U. Luxburg, Guyon, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 29, Curran Associates, Inc., 2016.
[34] A Kiran and S Saravana Kumar. A comparative analysis of gan and vae based synthetic data generators for high dimensional, imbalanced tabular data. In 2023 2nd International Conference for Innovation in Technology ( (INOCON), pages 1-6. IEEE, 2023.
[35] Roger W. Koenker, Limin Peng, Victor Chernozhukov, and Xuming He. Handbook of quantile regression. 2017.
[36] Alex Krizhevsky. Learning multiple layers of features from tiny images. 2009.
[37] Abhishek Kumar and Ben Poole. On implicit regularization in ß-vaes. In International Conference on Muchine Leurning, pages 5480-5490. PMLR, 2020.
[38] Anders Boesen Lindbo Larsen, Søren Kaae Sønderby, Hugo Larochelle, and Ole Winther. Autoencoding beyond pixels using a learned similarity metric. In Maria Florina Balcan and Kilian Q. Weinberger, editors, Proceedings of The 33rd International Conference on Machine Learning, volume 48 of Proceedings of Machine Learning Research, pages 1558-1566, New York, New York, USA, 20-22 Jun 2016. PMLR.
[39] Alex X. Lee, Anusha Nagahandi, Pieter Abbeel, and Sergey Levine Stochastic latent actor- critic: Deep reinforcement learning with latent variable model. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems, volume 33, pages 741-752. Curran Associates, Inc., 2020.
[40] Erich Leo Lehmann. Elements of large- theory. 1998.
[41] James Lucas, G. Tucker, Roger Baker Grosse, and Mohammad Norouzi. Don't blame the elbo! a linear vae perspective on posterior collapse. In Neural Information Processing Systems, 2019.
[42] Alireza Makhzani, Jonathon Shlens, Navdeep Jaitly, Ian Goodfellow, and Brendan Frey. Adver- sarial autoencoders. arXiv preprint arXiv: 1.05644, 2015.
[43] James E. Matheson and Robert L. Winkler. Scoring rules for continuous probability distributions. Management Science, 22: 1086-1096, 1976.
[44] Stan Matwin, Jordi Nin, Morvarid Sehatkar, and Tomasz Szapiro. A review of attribute disclosure control. In Advanced Research in Data Privacy, 2015.
[45] Sangjun Moon, Jong-June Jeon, Jason Eng Hun Lee, and Yongdai Kim. Learning multiple quantiles with neural networks. Journal of Computational and Graphical Statistics, 30:1238
1248, 2021.
[46] Prateek Munjal, Akanksha Paul, and N. C. Krishnan. Implicit discriminator in variational autoencoder. 2020 International Joint Conference on Neural Networks (IJCNN), pages 1-8, 2019.
[47] Noseong Park, Mahmoud Mohammadi, Kshitij Gorde, Sushil Jajodia, Hongkyu Park, and Youngmin Kim. Data synthesis based on generative adversarial networks. Proc. VLDB Endow., 11:1071-1083, 2018.
[48] V. Parmar, Bogdan Penkovsky, Damien Querlioz, and Manan Suri. Hardware-efficient stochastic binary enn architectures for near-sensor computing. Frontiers in Neuroscience, 15, 2022.
12

---

[49] J. Peters, Dominik Janzing, and Bernhard Schölkopf. Elements of causal inference: Foundations and learning algorithms. 2017.
[50] Kostantinos N Plataniotis and Dimitris Hatzinakos. Gaussian mixtures and their applications to signal processing. Advanced signal processing handbook, pages 89-124, 2017.
[51] Danilo Jimenez Rezende, Shakir Mohamed, and Daan Wierstra. Stochastic backpropagation and approximate inference in deep generative models. In International Conference on Machine Learning. 2014.
[52] Mihaela Rosca, Balaji Lakshminarayanan, David Warde-Farley, and Shakir Mohamed. Varia- tional approaches for auto-encoding generative adversarial networks. ArXiv, 2017.
[53] Tim Salimans, Andrej Karpathy, Xi Chen, and Diederik P. Kingma. PixeICNN++: Improving the pixcICNN with discretized logistic mixture likelihood and other modifications. In International Conference on Learning Representations, 2017.
[54] R. Shokri, Marco Stronati, Congzheng Song, and Vitaly Shmatikov. Membership inference attacks against machine learning models. 20/7 IEEE Symposium on Security and Privacy (SP), pages 3-18, 2016.
(55] Nicki Skafte, Martin Jø rgensen, and Sø ren Hauberg. Reliable training and estimation of variance networks. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d' d'Alché-Buc Buc, E. Fox, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 32, Curran Associates, Inc., 2019.
[56]
Kihyuk Sohn, Honglak Lee, and Xinchen Yan. Learning structured output representation using deep conditional generative models. In NIPS, 2015.
[57] Hiroshi Takahashi, Tomoharu Iwata, Yuki Yamanaka, Masanori Yamada, and Satoshi Yagi. Student-t variational autoencoder for robust density estimation. In International Joint Confer- ence on Artificial Intelligence, 2018.
[58] Jakub Tomezak and Max Welling. Vae with a vampprior. In Amos Storkey and Fernando Perez-Cruz, editors, Proceedings of the Twenty- First International Conference on Artificial Intelligence and Statistics, volume 84 of Proceedings of Machine Learning Research, pages 1214 1223. PMLR, 09-11 Apr 2018.
[59] Kunfeng Wang, Chao Gou, Yanjie Duan, Yilun Lin, Xinhu Zheng, and Fei-Yue Wang. Genera- tive adversarial networks: introduction and outlook. IEEE/CAA Journal of Automatica Sinica, 4(4):588-598, 2017.
[60] Wenlin Wang. Zhe Gan, Hongteng Xu, Ruiyi Zhang, Guoyin Wang, Dingham Shen, Changyou Chen, and Lawrence Carin. Topic-guided variational auto-encoder for text generation. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume (Long and Short Papers), pages 167-177, Minneapolis, Minnesota, June 2019. Association for Computational Linguistics.
[61] Bingyang Wen, Yupeng Cao, Fan Yang, Koduvayur Subbalakshmi, and Rajarathnam Chan- dramouli. Causal-TGAN: Causally-aware synthetic tabular data generative adversarial network, 2022.
[62] Ruofeng Wen, Kari Torkkola, Balakrishnan Narayanaswamy, and Dhruv Madeka. A multi- horizon quantile recurrent forecaster. arXiv: Machine Learning, 2017.
[63] Lei Xu, Maria Skoularidou, Alfredo Cuesta-Infante, and Kalyan Veeramachaneni. Modeling tabular data using conditional gan. In H. Wallach, Larochelle, A. Beygelzimer, F. d'Alché- Buc, E. Fox, and R. Gamett, editors, Advances in Neural Information Processing Systems, volume 32. Curran Associates, Inc., 2019.
[64] Keming Yu and Rana Moyeed. Bayesian quantile regression. Statistics Probability Letters, 54:437-447, 2001.
[65] Zilong Zhao, Aditya Kunar, Robert Birke, and Lydia Y. Chen. Ctab-gan: Effective table data synthesizing. In Vineeth N. Balasubramanian and Ivor Tsang, editors, Proceedings of The 13th Asian Conference on Machine Learning, volume 157 of Proceedings of Machine Learning Research, pages 97-112. PMLR, 17-19 Nov 2021.
13

---

A Appendix
A.1 Derivation of ELBO
logp(x;; 3)
Σ plax) / axi0,
log
k=1
p(z) Σ p((x)) / p(x12, g(z|x: Φ) ( Φ)dz
log
k=1
Σ p(ax)) / q(2)xx))) (p(xiz, on; 0, p(z)
K
2
dz
((11x;; Φ))
k=1
K
1
(x) logp(x/z, 8)] ())
K
K
B (x(()x
Σ
log 1ogp(x;12, ous 0) +
Σ
logp(x; 03;
K
jElC
jElD
Σ
a1+(1 - -ox)
x
Dj(a& 1 kz)
(x(x) xc))
log
Pax
K
B
jETc
km
T3
+Byge xio))
Σ
T(x; =)) log (z)
- (<()(());;))lxxxz))
JEID 1=1
K
K
= (x)))))))
Pax (x) X,
-Dy(allz,0))
+
P
log as ax) pp log B
K
K
JElc
k=1
T2
+ +By(s(x:d)
Σ
Σ
[(x; = log m((2
- (C[(z()/x())lpte))
jEID 1=1
K
K
3 By(elooce))
Σ
Pax (x, (x-- -Di(aal200))
3 P
Σ
log ar (1 - an) + Bp log
K
K
jEle
k=1
k=1
T
-3 (x(()
Σ
T(x; 1) log (2;
+ +3. (C (x
JEID 1=1
by Jensen's inequality.
14

---

A.2
Proof of Theorem |
Proof.
Pesa Xj Dy(as 12, 0;))
E Ep(x) By xx:)
M(ax))
K k=1
K
K
Σ h(ax)) (ax - ak-1);
Am
where og == O and Bp(x) B (e(e)x0) Pax. (x, - Dy(as 2, 0j)) is denoted as h(ax).
Since ag E [ag-1, axl and h()) [0. 1] + a continuous function, for j E Ic.
lim Ep(x) (g(e) -00
Pax Xj - D)(aulz,0)))
lim K-000
h(as) . (ax ax-1)
L h(a)da
/ E B Pa (aa(x] (xx (D,,a(((),,,)))) da
1 Pa (x,- Dy(alz, (x(xx)-D((aaz0))) da
pp(x (
by the definition of the Riemann integral and the Fubini-Tonelli theorem. The proof is complete. 17
A.3
Proof of Theorem
Proof.
min (x) (b)))
Σ
Pa(x, - DD(a, 2; 2;8;))do
= Egtrs))oxx E/ Palxx - -D(a,, z: 2;8,))
8
3=1
B Be(z Fotx)1a0) / Palxx Dy(a, 2: 2d
So. for all j =- 11,.....
(7) E arg min B) By(x) / Pa(x; Dj(a, z; 0,))da,
9,
and it is proper scoring rule relative F Φ) for all E Rd. It implies that Dj (xx, Fj (x,122; Φ). and ax; (x, = / (x, 2: Φ), for all x, E TR, by Assumption
2; (7())
follows that
= A (x)1x
dx,
= / ((x|z; 0)q(; Φ)dz
/
p(xx)((z)x;)) の)
q(z d)
q(z;0)
/ p(x))(22);;)) d)22
p(x).
The proof is complete.
15

---

A.4
Proof of Theorem 3
Proof.
l' 2=1
min E (z
Σ
Pa(x; Dj(a, 2;0;
Bo(zn))o(x Σ Pa(x; D2(a, 2; da
5=1
P
2) Egt, (e(x S Palxx - D, zi (,
So, for all =
()) E arg min (x)))))) Ex(x) / Pa(x; 10---y (a, 2,0,,2;00 2;8,))da.
",
and it is proper scoring rule relative to F. [2; ()) for all E IRdd. It implies that Dj (x,
Pr'(x,,) (x,)))())))) =
Fg (x,12; Φ). and ax, 1 (x,, 07 (b)) = dj (x, z; Φ). for all xj E IR, by Assumption 2
follows that
j=1 dx, Dj (xx / II
/ II
KL
p(x)
= KL
p(x)
og 1x(2x) Φ)p(z)dz
j=1
p(x) / [(x|2; (2(2)))((z)dt
KC
p(x) / ((x))(Z/)x:) Φ)
KL
p(2)dz
8(2; x))
p(x) /
p p(2))(e(x: 2)
KCC
p(x)
p(z)g(z)x:() dz
Ep(x)
-log
q(z; Φ)
/ [2))xx))))) log
(2z dz
I pxx))
p(z)
// p(x)g(z)x; の) log
O(z: Φ)
dzdx
p(z)
|/ p(x))(z(xx()) log
Ozz Φ)
dxdz
p(z)
/ ((zz Φ) log ((z dz
p(z)
<C(g(z;)))lpla
< c,
by Jensen's inequality, Fubini-Tonelli theorem, and the assumptions. The proof complete.
A.5 Closed Form Loss
2 Jo Pa Xx D,(a, 2,)) (2ā, 1)x, + (1 (0)))
bld) (z)
1 - d3
dm max(āj, dmi) + max(āz dm)dm )
where D,(āg,z;8) 2;0,) = X,, āj x,-7 (1) (z)+ 5
m0
(0 (z)dm. and D, (dmo 2;0;) S x, s
> 6.2 (z)
D, (;
16

---

A.6 Discretization of Estimated CDF
To ensure appropriate discretization of the estimated CDF for ordinal variables, we propose a post-ad- hoc discretization step [53]. We focus on the case where p = 1 and
for brevity. We denote the set of observed possible values for the ordinal variable as x:(1), x(²) x.(m), The discretization algorithm for the estimated CDF is presented in Algorithm | and we provide an example of the
discretization algorithm's 's outcome in Figure 2
Algorithm Discretization of Estimated CDF
Input (x(), (0()) ,..,, (.()) Estimated CDF (::0)
Output Discretized CDF F* (::
(I) Compute (z 0.5; 0) and F(x!1) + +..5; 0) for i = ,, m.
(2) Discretization: For é 1, m.
;);)();))
+ P(x(1) + +0.5;0) p(x 0.5;0).
where F)(((();)) 0)) =.
(3) Ensure monotonicity: For i 1,... , 1, if F (x(0): 0) 1=x(++1));
f0(x((+);00 = (1())
1.0
empirical
estimate
calibration
0.6
0.4
0.0
2
4
6
10
12
14
16
educational-num
Figure 2: Discretized CDF for ordinal variable educational-mum of adult dataset. 'estimate' indicates F): 'calibration' indicates F (:; and 'empirical" indicates the empirical CDF of the observed dataset.
17

---

A.7 Membership Inference Attack
propose the customized membership inference attack method of [54] attack the GAN- -ased synthesizer. Similarly, we propose the customized membership inference attack method of [54] to attack the VAE-based synthesizer.
Assumption 3 (154)) In the membership inference atlack, the allacker altacks the target model under the following assumplions:
(A1) The attacker only allowed for black-box access, where the attacker can only supply inputs to the model and receive the model's output(s).
(A2) The attacker can obtain as many outputs as they want from target model to attack.
(A3) The real and synthetic datasets should not have common records.
(A4) The attacker knows the algorithm and architecture of the target model.
Denote Dirasn and Diest the real training and test datasets. Under Assumption the overall steps of the membership inference attack are outlined below:
1. Generate shadow training and test datasets D Prain D. 1,.... C from M by (A1), where M the model attacker wants to attack. By (A2), the attacker is allowed to obtain
shadow datasets such [D(() Draain!
D frain and ( braán n D(d Ø, for i 1, ,..
Under (A3), (1) n Dirain = 2, for 1,.." C.
2. Train shadow models Mi,... MMc under (A4), i.e., each shadow model is trained similarly to the target model M
3. For i = L,.-,,
(a) Obtain representation vectors z from the encoder of M, with the input of D train attacking training records are (y, z, in).
Then,
(b) Obtain representation vectors from the encoder of M, with the input of D Then, attacking training records are (y, z, out).
where y is the labels of shadow dataset records. And we assume that y consists of the MLu classification target.
4. Merge all attack training records, (y, z, in/oul).
5. For each class of y, train attack model which is binary classification model which classifies in / out based on the representation vectors
6. Now it is ready to attack.
Note that we use the representation vector of VAE instead the output of the GAN discriminator
47
18

---

A.8 Dataset Descriptions
Websites
covertype:
credit:
loan: http:///wwwwauaaoooo.//////////a
adult: http://:www......e/////aaaaaaaa0
http://hhh///oogggooo/o/////aaae Table http://wwwww.aaa.........ssasasr denotes the number of discrete variables.
cabs:
Description of datasets. #C represents the number of continuous and ordinal variables. #D
Dataset
Train/Test Split
Regression Target
Classification Target
#C
#D
covertype credit
45k/5k
Elevation SMT_CREDIT
Gover_Type 10
1
45k/5k
TARGET
9
loan
4k/lk
Age
Personal Loan
5
6
adult
40k/5k
age
income
5
9
cabs
40k/1k
Trip_ Distance
Surge Surge_Pricingg
wwwciiinnnnn... Type
6
7
kings
20k/1k
long
condition
7
19

---

A.9
Experimental Settings
We rum all experiments using Geforee RTX 3090 GPU, and our experimental codes are all available with pytorch.
Table 9: Hyper-parameter settings for tabular dataset experiments.
Model
epochs
batch size
learning rate
33 (or decoder std range)
d
M
CTGAN
300
500
0.0002
2
TVAE
200
256
0.005
[0.1,
1]
2
CTAB-GAN
150
500
0.0002
2
DistVAE
100
256
0.001
0.5
2
10
Table 10: The number of model parameters for tabular dataset experiments.
Model
covtype
credit
adult
loan
cabs
kings
CTGAN
20k
32k
52k
13k
30k
51k
TVAE
10k
12k
13k
6k
10k
10k
CTAB-GAN
12k
13k
14k
5k
12k
15k
Dis(VAE
10k
12k
13k
6k
10k
16k
Table 11: Classifier and regressor used to evaluate synthetic data quality. The names of all parameters used in the description consistent with those defined in corresponding packages.
Tasks
Model
Description
Package: statemodels app api.a. OLS, setting: without intercept, defaulted values Puckage: sklearn. ensemble RandomFore wewtoorrrrrreeeerrrrrrrro.o selting: random state=0, defaulted values Package: aklearn. ensemble eklearr..nseemlee Gradi antBooatingtegreaooo.
Linear Regression
Regression
Random Forest
Gradient Boosting
setting: random state=0), defaulted values
Package: sklearn linear htttaalaaeaaaamellaaaaeeaa wwt.mmmelloooliiii...eem creiieasieeg
Logistic Regression
setting: muutt_class=='vve i ficjintercept=Faale. defaulted values
Package: sklearn. ensemble anssebbee.Rammddooooeeecedd
Classification
Random Forest
selting: random state=0, defaulted values Package: sklearn. ensemble delearr.ensemmle....m GradientBoostinnlllasse
Gradient Boosting
setting: random random_state-le state=)), defaulted values
20

---

A.10
Detailed Experimental Results
1.00
1.00
1.00
1.00
1.00
0.75
0.30
025
0.25
0.2
0.25
0.25
-2.5
0.0
2.5
0 Aspert
2.5
5.0
Elevation
Stape
Horizontal Distarce, To, Hydrology
Vartical Vernicl_paaaacee Datance, R. byyycolygy
1.00
1.00
1.00
1.00
1.00
0.75
0.75
0.50
0.25
0.25
0.25
0.25
0.25
-2.5
0.0
2,5
Horizontal, Distance To Reathays
Hilshade 5um
HB Hishodo Noor
Hilshade_3pm 3pm
Ha taontal Distance,, To Poin
(a) covtype
1.00
1.00
1.00
1.00
1.00
0.75
0.75
0.75
0.75
0.75
0.25
0.25
0.25
0.25
0.25
0.00
0.00
0.00
0.00
0.00
AMT_ERMIIT
1.00
1.00
1.00
1.00
0.75
0.75
0.50
0.25
0.25
0.2
0.25
0.25
0.00
0.00
WITH
CYR
DAYS AEGISTRAATIOONN
(b) credit
1.02
0.75
0.75
0.50
0.50
0.50
0.50
0.25
0.25
0.25
enotrial
-2
0
0.0
2.5
0.0
2.5
5.0
Age
Biperlance
Ircame
CCAN)
Mortgage
(c) loan
Figure 3: Empirical and estimated CDFs of continuous and count variables (Monte Carlo approxi- mated with 5000 samples). We standardize covariates and remove observations outside the 1% and 99% percentile range.
21

---

1.00
1.00
1.0
1.00
0.75
0.75
0.75
0.3
0.5
0.00
0.00
0.4
0.00
0.0
2.5
-2.5
0.0
25
10
lalgan
iapial-ms
(a) adult t
0.75
0.75
0.75
0.75
0.75
0.75-
0.25
0.25
0.25
0.25
3.00
0.00
0.00
(b) cabs
0.75
0.75
0.75
0.75
2.8
3.50
0.50
0.5C
2.50
0.50
3.25
0.25
0.25
0.25
0.25
0.4
0.00
0.00
0.00
0.00
0.00
10
20
-2
C
prke
wpt. Prig
HG7: abone
wF2. Deserentt
74.541
1.00
1.00
1.00
1.00
0.8
8.75
0.25
0.25
0.25
0.00
0.30
2.00
0.00
30
632
030
V./orevated
10t
bng
kings
Figure 4: Empirical and estimated CDFs of continuous and count variables (Monte Carlo approxi- mated with 5000 samples). We standardize covariates and remove observations outside the 1% and 99% percentile range.
22

---

Table 12: MLu metrics (MARE, Fi). Mean and standard deviation values are obtained from 10 repeated experiments. denotes higher is better and & denotes lower is better.
(a)
Dataset
covertype
credit
loan
Model
MARE
Fi t
MARE
Fi 1
MARE І
Fi f
Bascline
0.035
0.718
0.064
0.927
0.020
0.948
CTGAN TVAE
0.058=0.007
20.00
0.593+0.150
0.914=0.006
0.258++0.020
0.8422+0.109
0.079=0.007
0.50140.032
0.260
0.091 20.286
0.12440.033
0.785
STAB-GAN
0.065=0.004
0.493+4.027
0..35
0.913g0.00s
0.247:000019
0.887
DisVAE(3 0.5)
0.044
0.60540.006
0.763++0.06
0.926=0.001
0.2449+0.007
0.914400000
Dis(VAE(3 1)
0.045
10.007
0.774
0.020
0.249 20.000 0.0000
0.897100.00 0.00
DistVAE(8 = 5)
0.063
0.443
0.870
0.904=0.010
0.249
0.893 1.000
(b)
Dataset
adult
cabs
kings
Model
MARE
F t
MARE 4
Fi 1
MARE 4
Fj f
Baseline
0.216
0.854
0.565
0.743
0.001
0.695
CTGAN
00007
(). 0.796240.02
0.721 40.046
0.674_0.024
4..00
1.579+0.038
TVAE
20.00
0.809 10.016
0.642.
0.689 =689000031
0.010
0.687
CTAB-GAN
0.321 20.036
0. 0..00
0.894 16
0.582_00047
0.001
0.608
DisVVAE( 0.5)
0.232200004
0.825
0.803:20.128
0.707_20.010
0.001
0.640
DistVAE(8
0.204
0..02
0.760
0.725=0.004
0.001
0.64440.008
Dis = 5)
0.327 00.08
0.751=0.010
0.839:20.0g
0.447=0.00e
0.002
0.637
DistVAE
CTGAN
0.8
TVAE
CTAB-GAN
0.6
covtype
6
credit
0.4
loan
adult
0.2
cabs
kings
0.0
0.2
0.4
0.6
0.8
MARE
Figure Machine learning utilities for compared models and real tabular datasets.
For detailed comparison of the models and their performance, we present the paired (MARE, Fi) scores for all tabular datasets in Figure better score, indicating superior performance in terms of MLu, is represented by dot located in the upper left corner. Notably, Figure 5 consistently demonstrates that VAE achieves the best or least competitive MLu across all tabular datasets. TVAE exhibits notably low Fi score in the credit dataset because it fails to handle the highly imbalanced discrete target variable. This comparative analysis highlights the strong MLu performance of DistVAE and the specific limitations of TVAE in certain scenarios.
23

---

Table 13: Correlation structure similarity. 'CorrDist' represents L, distance between the correlation matrix of synthetic and real datasets. Mean and standard deviation values are obtained from 10 repeated experiments. Lower is better.
Dataset
covertype
credit
loan
adult
cabs
kings
Model
CorrDist
ComDist
ComDDit
CorrDist
CorDDitt
CorrDist
CTGAN
2.16720.419
2.323
2. 2.28220.17
1.788_2.211
1.679
2.839 246
TVAE
10969
4.021 +0.451
2. 2.40140.408
2.231 20.209
3.136
2.66540.290
CTAB-GR GAA
20.51
2.696
2.073
2.387
2.532
3.411
DistVAE(3 = 0.5)
2.079
2.072 2.072 10.162
1.6544a.050
0.830=2.079
1.481 20.071
1.55940.135
DistVAE(3 = 2.359
2.229 10010
1.910
0.746220.022
1.495
1.621 (62110.140
DistVAE(S = 5)
2.946
3.161 10.006
2.11340.015
3.186_p0.07
1.930
5.339
24

---

Table 14: Marginal statistical similarity. K-S denotes the KolmogorovSmirnov statistic and 1-WD represents the I-Wasserstein distance. Mean and standard deviation values are obtained from 10 repeated experiments. Lower is better.
(a) Continuous
Dataset
covertype
credit
coan
Model
K-S
I-WD
K-S
1-WD
K-S
I-WD
CTGAN TVAE
0.080
0.108
0.132
0.088_0.013
0.11244.065
0.061 +0.016
0.088=200008
0.156:0.0118
0.156 036
0.200
0.162 0.402
0.201 024
CTAB-GAN
0.073
0.09644.014
0.140
0.111
0.181 0.181
0.153
Dis(VAE(3 0.5)
0.002
0.041
0. 00.014
0.089=0.004
0.060++0.010
0.048
Dis(VAE(8 1)
0.049 20,001
4.009
0.084
0.093
0.056
0.054 0.002
DistVAE(8 = 5)
0.064
0.112+0.000
0. 141466 40.00
0.113g0.001
0.057+0.000
0.056 0022
(b) Continuous
Dataset
adult
cabs
kings
Model
K-S
I-WD
K-S
1-WD
K-S
1-WD
CTGAN
0.323
0.086
0.045
0.060_o.011
0.109
0.1 0.116+00.01
TVAE
0.053
0.414
1..988 0.008
0.139 =.029
0.195 0.195 10.018
0.213 046
CTAB-GAN
0.275a0.122
0.17840.053
0.08614..00
0.118
0.18840.008 10.055
0.128
DistVAE(8 0.5)
0.209
0.11440.00
0.044 000
0.067
0.110 008
0.089
DistVAE(3
0.138
0.11140.000
0.046
0.068
0.115 0.115
0.003
DistVAE(3 5)
0.11520.005
0.23440.002 000
0..55+0.001 000001
0.074=0.002
0.120 20.002
0.137:0.002
(c) Discrete
Dataset
covertype
credit
loan
Model
K-S
I-WD
K-S
1-WD
K-S
I-WD
CTGAN TVAE
20.0ns
1.62940.011
0.061 +0.00s
0.14720.024
0.070
0.076 013
0.238=0.042
0.606 033
0.583
1.566 116
0.193 028
0.221 043
CTAB-GAR
0.052=#0.028
0.180 126
0.034++.0006
0.076
0.039440.011
0.046+t0.013
Dis(VAE(8 = 0.5)
0.023
0.073 046
0. 0.020
0,046
0.019 005
0.027 008
Di 1)
0.011 20.005
0.036
0.01 8
0.042_0.003
0.01 +0.002
01.15
Dis(VAE(3 5)
0.
0.37940..11
0.027
0.068
0.009 10009..000 002
0.013 005
(d) Discrete
Dataset
adult
cabs
kings
Model
K-S
1-WD
K-S
1-WD
K-S
1-WD
CTGAN TVAE
0.065
0.463 051
0.069
0,238
0.140
0.529 064
0.17940.018
5.228 273
0.41
1:000000000
1.202
0.405 034
1. 1.262 099
CTAB-GA
0.169
0.745
0.0864.0.013
0.443_20.074
0.255
0.979
Dis(VAE(3 - 0.5)
0.037 20.004
0.24884.022
0.060
0.241
0.022
0.07 0.07110.011
8 = 1)
0.031
0.23940.019
0.070
0.292
0.023 003
0.073 0os
Dis - 5)
0.070
0.457
0.083
0.326
0.059
0.204
25

---

Table 15: Privacy preservability: Distance to closest record (DCR) between real training and synthetic datasets (R&S), between the same real training datasets (R), and between the same synthetic datasets (S). Mean and standard deviation values are obtained from 10 repeated experiments. Higher is better.
(a)
(b)
covertype
credit
Model
R&S
R
Model
R&S
CTGAN
0.715=0.026
0.329=0.000
0.51 420.044
CTGAN
0.624
0.452=0.000
0.592
TVAE
0.670-..001
=0.000
n.452
TVAE
0.627
0 0.452=0.000
0.423
CTAB-GAN
0.892_0.081
0.329:.0000
1.011
CTAB-GAN
0.715
0. 0..52.0.00
0.014.00.06
DistVAE(3 = 0.5)
0.765-0.00s
0.329
0.819 0.819
D = 0.5)
0 0.692
0. 0.13220.000
0.742
Dis(VAE 0.878_0.00s
000 0.329
2.900
DistVAE(3
0. go
0.4322..00
0.750 o0e
DistVAE 5)
0.907
0.930
DisIVAE(3 5)
0.718
0.452
0.757
(c)
(d)
Daraset
loan
Dataset
adult
Model
R&S
R
Model
R&S
CTGAN
0.249-0.011
0.100=0..0m
0.243
CTGAN
0.063
0.000 0.000=0.000
TVAE
0.298 0.298_0.057
20.000
0.151g0.0as
TVAE
0.277 gou0de
0.000go.0m0
0.000..0.00
CTAB BANN
0.272_0.084
0,109 20.00000
0.076g0.051
CTAB BAAN--ANN
0.13220000:
0.000
0.000
Dis 0.5)
0.244=0.025
0. 1.10000.000
0.245gn.020
DistVAF(3 0.5)
0.060
0.0 0.000=0.000
0.005=0.000
DistVAE
0.238
0.109 =0.000
20.22
DisIVAEL
0.0:8
0.000ga.om
0.003
DisIVAF(3 5)
0.243_00011
0. 100000000
0.240zn.025
DistVAFL
0.177 20.002
0. 0.000=0..00
0.001 =0,000
(e)
ff)
Dataset
cabs
Dalaset
kings
Model
R&S
R
Model
R&S
R
CTGAN
0.353_0.000 006
0.33220.0m
0.34120.0m
CTGAN
0.550
0.199 200000
0.447
TVAE
0.330_o.00s
0.332-2.000
0.105
TVAE
0.603
0. 0.13920.000
0.414_a.000
CTAB-GAN
0.123_0.025
0.332:0.000
0.012
CTAB-GAN
0. 0.59640.080
0. 199
0. 0.
DistVAE(3 0.5)
0.364_0.004
a0.00000.000
20.005
DistVAELe 0.5)
0.5-002o.oo
0. 20.00
0.600
Dis
0.361_o.004
0,332
0.367
DistVAE(3
0.552
0. 0.1999_..mm
0.605
DisIVAE(3
0.368_0.00 00.
0.332 20.000
0.365
DisIVAE(3
0.69220.013
0.199 1..9922..00
0.763_..000
Table 16: Privacy preservability: Membership inference attack performance. Mean and standard deviation values are obtained from repeated experiments.
Dataset
covertype
Dataset
credit
Model
Accuracy
AUC
Model
Accuracy
AUC
TVAE
0.499 4..4944..077 40.007
0.499 $0.007
TVAE
0.500
4..00
DistVAB( 0.5)
4a.0m3
+0.003
DiStVAEE 0.5)
0.500
0.500
(c)
Dataset
loan
Dataset
adult
Model
Accuracy
AUC
Model
Accuracy
AUC
TVAE
0.497 0.49711..15 10.015
0.497 10.015
TVAE
0.493
0.493 10.017
DistVAE(S 0.5)
00.00
0..0240..m6
DistVAB(3 0.5)
0.500
0.500 20.000
(e)
Dataset
Dataset
kings
Model
Accuracy
AUC
Model
Accuracy
AUC
TVAE
0.480 033
0. 10.000.033 033
TVAE
0.507 0.507
10.025
DistVARCS 0.5)
0.498 1. 003
0.498 10.8..0000
DisiVAR(2 0.5)
0.502
0.502 004
26

---

Table 17: Privacy preservability; Attribute disclosure performance with F1 score. Mean and standard deviation values are obtained from 10 repeated experiments. Lower is better.
covtype
credit
Number of neighbors (A)
Number of neighbors (k)
Model
10
100
Model
10
100
CTGAN
0.161 00.000
0.175.0.02
CTGAN
2.009
0.330
0.323.0.011
TVAE
0.356
0.357
0.349
TVAE
0.615
0.618_o.18
0.614_0.11
CTAB-GAN
0.200
0.225g0.042
0.23820.053
CTAB-GAN GAN
0.32220.007
0.33220.012
0.331 0.331
DisIVAE(3
0.005
0.338_0.037
2.013
DisiVAEL 0.5)
20009
0.315 0.337
DisVAEF(3
0.264_0.024
0.294=0.036
0.28220.029
DisIVAFL 1)
0.339
0.324_0.005
0.301 =0.0040.00
DistVAE(8
0.141
0.115gn.011 g0.015
DistVAE(2
0.332:x.007
0..17.0.007
0.293
(c) loan
(d) adult
Number of neighbors (k)
Number of neighbors (k)
Model
10
100
Model
CTGAN
0.439_0.097
0.447 uza
0.426 usa
CTGAN
0.234
0.255 021
0.261
TVAE
0.611_0.153
0.60220.152
0.596
TVAE
0.318
0.318
0.307
CTAB-GAN
0.475=0.045
0.443ga.027
0.4
CTAB-GAN
0.190
0.20220.052
0.205
DisVVAF(8 0.5)
0.505_0.045
0.465
0.439
DisIVAFL
0..70
0. 28022.021
0. 00266_20.00
DistVAE(8
0.449
0.440
0.423
DistVAE( 1)
0. 0,264 so.ces
0. 0.276=o.00s
0. 0.260=a..00
Dis(VAE(3
0.458_o.01
0.441 go.0s
0.416go.ust
DisIVAE(3
g0.00
0.187 20.00s
0.166
(0) cabs
( kings
Number of neighbors (k)
Number of neighbors (k)
Model
10
Model
CTGAN
0.238-p.0m
0..46
0.201
CTGAN
g0200
0.257 20.052
0.269
TVAE
0.385_0.079
0.38320.0m
0,381
TVAE
0.338 gouoes
0.349_2..0s
0.315
GAN
0.235_0.010
0.236go.00
20.01
CTAB GAN
g0.00
0.111go.oss
0.1233_0.114
Dis 0.5)
0.251
0.241
0.225
DistVAEL
0.200
0.310=a.04z
0.301 =0.000
DisEVAE(8
0.246_0.010
0.240=0.000
0.227=20000
DistVAE(3 1)
0.281 20.016
0.306
0.290=a.044
D
0.238_o.010
0.220
00.00
D
0.215
0.221 20.000
0.205
27


Reviews:
[{"review_id": "UuubNCqdwo", "reviewer": "Reviewer_a4Hg", "summary": "In this paper the authors propose to define the decoder of a VAE using an infinite mixture of asymmetric Laplacian distribution, which results in distributional learning. Specifically, they design a non-parametric generative model in the VAEs. This class of VAEs estimates the conditional CDF while trained with ELBO objective as standard VAEs. They evaluated their approach on multiple datasets and show that their method outperforms baselines.", "questions": "Please see my questions in the previous section", "limitations": "I don't think there is any potential negative impact in their work.", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "This idea of increasing the model capacity of VAEs is well motivated, because Gaussian assumption has limited the applications of VAEs in the past. So I think this paper brings up some interesting topic and is novel as well.\nIn general this paper is clearly structured. The authors provide detailed derivations of the methodology. Also they perform comprehensive evaluations on multiple datasets.", "weaknesses": "My main confusion is about the 2 dimensional latent space in the evaluations. If I understand correctly, all the models are defined with 2D latent space. I wonder if this puts limits on the decoder capacity by itself, and I wonder how the model performs when we do higher latent dimensions in the experiments.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> Q1 (Question). My main confusion is about the 2 dimensional latent space in the evaluations. If I understand correctly, all the models are defined with 2D latent space. I wonder if this puts limits on the decoder capacity by itself, and I wonder how the model performs when we do higher latent dimensions in the experiments.\n\nWe sincerely appreciate your valuable feedback regarding the dimension of the latent space in our evaluation processes. As you pointed out, the chosen latent space indeed limits the capacity of decoders for all models. However, our primary goal was maintaining a small and consistent number of parameters across all models during the experiment. This deliberate choice allowed us to isolate the performance differences in synthetic data generation to the methodologies of each synthesizer, specifically emphasizing the contribution of the decoder model's flexibility in estimating underlying distributions. Additionally, we anticipate that using a low-dimensional latent space can enhance the explainability of the generative models, and having a smaller number of parameters is advantageous for the model's utility. This decision was crucial as the size of the model directly influences the quality of synthetic data generation. Accordingly, we carefully designed a simulation setting, ensuring that the latent dimension, a decisive factor in determining the model size, consistently remained low for all models. A comprehensive comparison of the model parameters can be found in Appendix A.9 Table 10.\n\nFurthermore, our proposed model's performance, as stated in Theorem 2 and 3, is heavily reliant on the accuracy of the conditional quantile function $D$ in approximating the ground truth CDF (cumulative distribution function). If we significantly increase the capacity of the decoder or utilize a higher-dimensional latent space, the neural network parameterized $D$, taking the latent variable as input, would achieve sufficient complexity to provide a better approximation of the ground truth CDF. Given that we have already demonstrated the superiority of our DistVAE in low-dimensional latent spaces with a similar number of model parameters in the evaluations, we firmly believe that even with a substantial increase in decoder capacity or higher-dimensional latent space, our DistVAE will continue to outperform other models in synthetic data generation."}, {"review_id": "5Sf670u4c4", "reviewer": "Reviewer_B22e", "summary": "This paper introduces a novel distributional learning method of VAE, which aims to effectively capture the underlying distribution of the observed dataset using a nonparametric approach. The proposed  decoder is composed of an infinite mixture of asymmetric Laplacian distribution, which possesses general distribution fitting capabilities for continuous variables.  The authors provide empirical results and demonstrate superiority in easily adjusting the level of data privacy.", "questions": "Why $\\alpha$ is assigned with a prior? I don’t get any intuition from your analysis.", "limitations": "Too many assumptions in the theoretical analysis which make the generality to real data suspicious.", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "This work provides concrete theoretical analysis and empirical results with the proposed distributional decoder. The proposed decoder has more capacity to capture the data distribution than the standard Gaussian. ", "weaknesses": "As [15] indicated, “the model training becomes unstable if the estimated variance of the decoder shrinks to zero in Gaussian VAE” is not theoretically correct.\n\nI don't find major weakness in this work.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> Q1 (Weakness). As [15] indicated, “the model training becomes unstable if the estimated variance of the decoder shrinks to zero in Gaussian VAE” is not theoretically correct.\n\nThank you for bringing up the issue regarding the citation of related papers in our work. We sincerely appreciate your valuable feedback and acknowledge that we should have been more careful in citing and describing the relevant papers in Section 2 Related Work. We understand that [1, 2] have empirically demonstrated the instability of training with moderate sample size, while [5] has provided theoretical evidence that the ground truth data distribution can be estimated with infinite samples, even if the variance of the decoder is trainable. To address this concern, we will revise lines 60-62 in Section 2 Related Work to ensure that the content accurately reflects the contributions of each paper:\n\n- Old version: \n\n*[1, 2] assume their decoder distributions as Student-t and asymmetric Laplace distributions, respectively, to mitigate the zero-variance problem that the model training becomes unstable if the estimated variance of the decoder shrinks to zero in Gaussian VAE.*\n\n- Revised version: \n\n*[1, 2] address the zero-variance problem in Gaussian VAE when the estimated variance of the decoder becomes too small, leading to unstable model training with a moderate sample size. To mitigate this problem, [1] adopts the Student-t distribution as the decoder distribution, while [2] uses the asymmetric Laplace distribution. However, theoretical evidence presented in [3] demonstrates that the ground-truth data distribution can be accurately estimated with an infinite number of samples, even if the variance of the decoder is trainable.*\n\n---\n\n> Q2 (Question). Why $\\alpha$ is assigned with a prior? I don’t get any intuition from your analysis.\n\nWe sincerely appreciate your valuable question regarding the relationship between $\\alpha$ and the prior distribution. As you pointed out, our reconstruction loss can be defined without the prior distribution of $\\alpha$. But we think the quantile loss provides a natural explanation of the reconstruction loss as an extension of the estimation for a single quantile. In our paper, we established the generative model by introducing an infinite mixture of asymmetric Laplace distribution (as mentioned in line 99). This entailed adopting a Bayesian perspective for $\\alpha$ as a prior (this is the Bayesian modeling for estimation of multiple quantiles), and the infinite mixture model served as a theoretical link between a proper scoring rule and our proposed reconstruction loss in the objective function (please see Equation (2) in our paper).\n\nThroughout the derivation of the reconstruction loss, the role of $\\alpha$ was pivotal in ensuring the representation of the reconstruction loss. Moreover, it was of utmost importance to prevent the observation $x$ from influencing the distribution of $\\alpha$; thus, the resulting reconstruction loss becomes the CRPS loss (a proper scoring rule). This meticulous design provided the integrity of our approach and its adherence to the desired properties. Additionally, we want to emphasize that by changing the prior distribution of $\\alpha$, we can derive more general models than the proposed one.\n\n---\n\n> Q3 (Limitation). Too many assumptions in the theoretical analysis which make the generality to real data suspicious.\n\nWe are sincerely grateful for your insightful comments regarding our theoretical analysis. Our assumptions in the theoretical analysis can be categorized into two main aspects: 1) assumptions about the data and 2) assumptions about the model.\n\nRegarding data assumptions, our focus on continuous random variables in the observation $x$ was intended to facilitate the theoretical analysis of our crucial contribution, which is incorporating the CRPS loss into the ELBO of the VAE framework. However, we expect that even datasets containing discrete variables can benefit from the same theoretical development by adopting the proper scoring rule introduced in [4]. Furthermore, although we assumed that the support of continuous random variables spans the entire real space, we believe this assumption is mild and can potentially be relaxed through additional constraints on the model. Additionally, a critical data assumption we made involves the conditional independence of $x_1, \\cdots, x_p$ given $z$ in the definition of $q(x|z;\\phi)$. It is possible to test this assumption in a trained model [5, 6], and such verification would serve as validation of the effectiveness of our proposed model.\n\nRegarding our model assumptions, particularly Assumption 1, we would like to emphasize that they primarily pertain to the model's complexity, computational cost, and asymptotic approximation performance rather than directly affecting its applicability to real-world datasets. Therefore, we firmly believe that the assumptions made in the theoretical analysis do not significantly impede the model's generalization to real data.\n\n---\n\n- References: \n\n[1] Takahashi, H., Iwata, T., Yamanaka, Y., Yamada, M., & Yagi, S. (2018). Student-t Variational Autoencoder for Robust Density Estimation. International Joint Conference on Artificial Intelligence.\n\n[2] Akrami, H., Joshi, A.A., Aydöre, S., & Leahy, R.M. (2020). Addressing Variance Shrinkage in Variational Autoencoders using Quantile Regression. ArXiv, abs/2010.09042.\n\n[3] Dai, B., & Wipf, D. (2019). Diagnosing and enhancing VAE models. arXiv preprint arXiv:1903.0578.\n\n[4] Carvalho, A. (2016). An Overview of Applications of Proper Scoring Rules. Decis. Anal., 13, 223-242.\n\n[5] Zhang, K., Peters, J., Janzing, D., & Schölkopf, B. (2012). Kernel-based conditional independence test and application in causal discovery. arXiv preprint arXiv:1202.3775.\n\n[6] Li, C., & Fan, X. (2020). On nonparametric conditional independence tests for continuous variables. Wiley Interdisciplinary Reviews: Computational Statistics, 12(3), e1489."}, {"review_id": "O676dOFlDy", "reviewer": "Reviewer_4gp6", "summary": "In this paper, the authors present a novel nonparametric distributional learning method for VAE. They provide a comprehensive theoretical analysis of the proposed solution (DistVAE), as well as prove its superiority over CTGAN, TVAE, and CTAB-GAN when applied to synthetic data generation on real tabular datasets.", "questions": "1. The authors claim that their approach \"extends model capacity [...] without sacrificing the computational advantages of the VAE framework\". Perhaps the paper would benefit from an experimental comparison (in terms of capturing data distribution and computational efficiency) of the proposed model with different VAEs trained on classical image datasets. Why was this not supplied? My final opinion will depend on the authors' response to this question.\n\n2. Minor comments:\n\nl. 14: I believe the authors would like to (additionally) cite another original paper on VAE: Rezende et al., ICML 2014.\n\nl. 104: '.' --> ':'.\n\nl. 133: $q(x|z; \\phi)dx$ --> $q(x|z; \\phi)$.\n\nl. 136: $q_j$ was already used to denote the number of levels for $x_j$.\n\nl. 156: \"hessian\" --> \"Hessian\".\n\nl. 160: \"bounded above\" --> \"bounded from above\".", "limitations": "A more detailed discussion of limitations would be fruitful.", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. The theory of the paper seems to be original and significant.\n\n2. The experimental results obtained for DistVAE are superior to those of competitors.\n\n3. The paper is well written.", "weaknesses": "1. The experimental setting is limited to tabular datasets and few synthesizers.\n\n2. A more detailed discussion of limitations would be fruitful.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> Q1 (Question). The authors claim that their approach \"extends model capacity [...] without sacrificing the computational advantages of the VAE framework\". Perhaps the paper would benefit from an experimental comparison (in terms of capturing data distribution and computational efficiency) of the proposed model with different VAEs trained on classical image datasets. Why was this not supplied? \n\nWe genuinely appreciate the reviewer's valuable feedback and thoughtful question regarding the types of datasets used in the evaluations. The performance of the generative model depends on both the dimension of the latent variables and the flexibility of the decoder model. First, let me explain the relationship between the size of the latent space and the flexibility of the decoder model within the scope of our understanding, as well as its implications in image data generation.\n\nIn our paper, we made an assumption that, given the latent variables, the observed variables are conditionally independent, as represented by Equation (1):\n\n$X_1 \\perp X_2 \\vert Z_1, \\cdots, Z_{d-1} \\Rightarrow X_1 \\perp X_2 \\vert Z_1, \\cdots, Z_d. \\mbox{\\quad (1)}$\n\nWhen the dimension of the latent variables is smaller, this conditional independence assumption may be violated. In such cases, the proposed non-parametric fitting of the decoder might not adequately represent the ground truth distribution. Particularly in image data, where significantly intricate dependency structures exist among pixel values, a low-dimensional latent space may not lead to substantial improvement in image data generation performance. Therefore, to effectively compare how the flexibility of the decoder model affects the fit of the probability distribution, we chose tabular datasets for our evaluation, as it allows relatively easy control of probabilistic conditional independence.\n\nHowever, it does not imply that our model is not inapplicable to image data. Representative classical image datasets like CIFAR-10 often exhibit pixel value distributions far from the Gaussian since the frequencies of edge values (0 and 255) dominate more than other pixel values [1, 2]. [3] also presents other image dataset examples where pixel value distributions exhibit multi-modality. Therefore, we expect that our DistVAE, capable of capturing multi-modal underlying distributions, would be effective compared to other VAE models in the image domain.\n\nConsidering the aforementioned limitations, we found it more appropriate to compare the synthetic data generation performance between our DistVAE and other synthesizers using real tabular datasets. However, in Section 5 Conclusion and Limitations, we will actively discuss the limitations of our proposed model in the image domain.\n\n---\n\n> Q2 (Limitation). A more detailed discussion of limitations would be fruitful.\n\nWe sincerely appreciate your valuable feedback. We agree that providing a more comprehensive discussion of limitations will enhance the overall quality of our paper. In response to the previous question, we intend to include the following content in Section 5 Conclusion and Limitations:\n\n*In our study, we have confirmed that the proposed decoder enhances the performance of generative models for tabular data. However, this conclusion relies on the assumption of conditional independence among the observed variables given the latent variable. When the dimension of the latent variable is small, this assumption is prone to violation. Therefore, in cases where the size of the latent space is limited, the proposed non-parametric fitting of the decoder might not accurately represent the underlying distribution. Particularly in the image domain, where adjacent pixel values exhibit significant dependence, it remains uncertain whether our proposed model would lead to notable improvements in image data generation with a low-dimensional latent space.*\n\n*Nevertheless, classical image datasets, such as CIFAR-10, often exhibit pixel value distributions that deviate considerably from Gaussian, with the frequencies of edge values (0 and 255) dominating more than other pixel values [1, 2]. Additionally, other image datasets, as presented in [3], demonstrate multi-modality in pixel value distributions. These experimental findings suggest that leveraging the capacity of distributional learning could be advantageous in approximating the ground truth distribution of image data when the latent variable effectively captures conditional independence among the observed variables. Consequently, we expect that compromising biases arising from the violation of conditional independence and marginally misspecified distributions may further enhance our results, and we leave it for our future research.*\n\n---\n\n> Q3 (Question). Minor comments.\n\nWe appreciate the reviewer's attention to detail and valuable feedback. We have thoroughly addressed all minor comments and made the necessary corrections. Specifically, we rectified the typos on lines 104, 133, 156, and 160. Furthermore, we have included the citation of the original VAE paper [4], and we now denote the number of levels for $x_j, j \\in I_D$ as $T_j$ instead of $q_j$. \n\n---\n\n- References:\n\n[1] Darlow, L.N., Crowley, E.J., Antoniou, A., & Storkey, A.J. (2018). CINIC-10 is not ImageNet or CIFAR-10. ArXiv, abs/1810.03505.\n\n[2] Salimans, T., Karpathy, A., Chen, X., & Kingma, D. P. (2017). Pixelcnn++: Improving the pixelcnn with discretized logistic mixture likelihood and other modifications. arXiv preprint arXiv:1701.05517.\n\n[3] Parmar, V., Penkovsky, B., Querlioz, D., & Suri, M. (2022). Hardware-efficient stochastic binary CNN architectures for near-sensor computing. Frontiers in Neuroscience, 15, 781786.\n\n[4] Rezende, D. J., Mohamed, S., & Wierstra, D. (2014, June). Stochastic backpropagation and approximate inference in deep generative models. In International conference on machine learning (pp. 1278-1286). PMLR."}]

Author rebuttal:

Meta-review: This paper proposes a novel non-parametric approach applied to the decoder of VAE, and showed better performance than the baselines.

Reviewers originally reported 

- Novel and interesting idea
- Well-written
- Comprehensive evaluations on multiple datasets

as pros and 

- Experimental setting is limited to tabular datasets and synthesized datasets

and some minor issues as cons.  The cons have been well addressed in the rebuttal, and the reviewers agree to accept the paper.
Decision: Accept (poster)