Title: Distributional Learning of Variational AutoEncoder: Application to Synthetic Data Generation
Venue: NeurIPS 2023
Accepted: yes
Keywords: Variational AutoEncoder

distributional learning

synthetic data generation

CRPS

asymmetric Laplace distribution
Abstract: The Gaussianity assumption has been consistently criticized as a main limitation of the Variational Autoencoder (VAE) despite its efficiency in computational modeling. In this paper, we propose a new approach that expands the model capacity (i.e., expressive power of distributional family) without sacrificing the computational advantages of the VAE framework. Our VAE model's decoder is composed of an infinite mixture of asymmetric Laplace distribution, which possesses general distribution fitting capabilities for continuous variables. Our model is represented by a special form of a nonparametric M-estimator for estimating general quantile functions, and we theoretically establish the relevance between the proposed model and quantile estimation. We apply the proposed model to synthetic data generation, and particularly, our model demonstrates superiority in easily adjusting the level of data privacy.

Distributional Learning of Variational AutoEncoder: Application to Synthetic Data Generation

Abstract

The Gaussianity assumption has been consistently criticized as a main limitation of the Variational Autoencoder (V AE) despite its efficiency in computational modeling. In this paper, we propose a new approach that expands the model capacity (i.e., expressive power of distributional family) without sacrificing the computational advantages of the VAE framework. Our VAE model's decoder is composed of an infinite mixture of asymmetric Laplace distribution, which possesses general distribution fitting capabilities for continuous variables. Our model is represented by a special form of a nonparametric M-estimator for estimating general quantile functions, and we theoretically establish the relevance between the proposed model and quantile estimation. We apply the proposed model to synthetic data generation, and particularly, our model demonstrates superiority in easily adjusting the level of data privacy.

1 Introduction

Variational Autoencoder (VAE) [31, 51] and Generative Adversarial Networks (GAN) [22] are generative models that are used to estimate the underlying distribution of a given dataset. To avoid the curse of dimensionality, V AE and GAN commonly introduce a low-dimensional latent space on which a conditional generative model is defined. By minimizing an information divergence between the original data and its generated data, the generative models are learned to produce synthetic data similar to the original one. Accordingly, V AE and GAN have been applied in various applications, such as generating realistic images, texts, and synthetic tabular data for privacy preservation purposes [30, 60, 63, 65, 34].

However, the difference in the strength of the assumption about the generative distribution brings significant contrasts in the V AE and GAN generation performances [30, 7, 17]. In the GAN framework, the adversarial loss enables direct minimization of the Jensen-Shannon divergence between the ground-truth density function and the generative distribution under no distributional assumption [13, 59]. Roughly speaking, the GAN employs a nonparametric model as its conditional generative model defined on the latent space.

On the contrary, in the V AE framework, the Gaussianity assumption has been favored [31, 32, 15, 10, 39]. It is because Gaussianity gives us three advantages: 1) the reconstruction loss can be interpreted as the mean squared error that is one of the most popular losses in optimization theory, 2) generating a new sample is computationally straightforward, and 3) KL-divergence is computed in a simple closed form. However, these benefits have led us to pay the price for the distributional capacity of the generative model, in that the generative model of the VAE is constrained in the form of marginalization of the product of the two Gaussian distributions. Here, the distributional capacity means the expressive power of the distributional family. This restricted distributional capacity has

been the critical limitation [8, 33] and leads to a heavy parameterization of the decoder mean vector to approximate complex underlying distributions.

To increase the distributional capacity in synthetic data generation, [63, 65] introduce the multimodality in the distributional assumption of the decoder, which is known as the mode-specific normalization technique . Although the mixture Gaussian decoder modeling of [63, 65] allows handling more complex distributions of the observed dataset while preserving all of the advantages of Gaussianity, we numerically find that the mixture Gaussian is not enough to capture the complex underlying distribution.

Our main contribution is that, beyond Gaussianity, we propose a novel VAE learning method that directly estimates the conditional cumulative distribution function (CDF) while maintaining the objective of maximizing the Evidence Lower Bound (ELBO) of the observed dataset. It implies that we have a nonparametric distribution assumption on the generative model. We call this approach distributional learning of the VAE , which is enabled by estimating an infinite number of conditional quantiles [4, 20]. By adopting the continuous ranked probability score (CRPS) loss, the objective function of our proposed distribution learning method is computationally tractable [21, 43, 20].

In our proposed distributional learning framework, 1) the reconstruction loss is equivalent to the CRPS loss, which is a proper scoring rule [21, 43], 2) generating a new sample is still computationally straightforward due to the inverse transform sampling, and 3) KL-divergence is still computed in a simple closed form. To show the effectiveness of our proposed model in capturing the underlying distribution of the dataset, we evaluate our model for synthetic data generation with real tabular datasets.

2 Related Work

Modeling of the decoder and reconstruction loss. To increase the distributional capacity, many papers have focused on decoder modeling while not losing the mathematical link to maximize the ELBO. [57, 1] assume their decoder distributions as Studentt and asymmetric Laplace distributions, respectively, to mitigate the zero-variance problem that the model training becomes unstable if the estimated variance of the decoder shrinks to zero in Gaussian VAE [41, 55, 15]. [2] proposes a general distribution of the decoder, which allows improved robustness by optimizing the shape of the loss function during training. Recently, [5] proposes a reconstruction loss that directly minimizes the blur error of the VAE by modeling the covariance matrix of multivariate Gaussian decoder.

On the other hand, there exists a research direction that focuses on replacing the reconstruction loss without concern for losing the mathematical derivation of the lower bound. [38, 52, 46] replace the reconstruction loss with an adversarial loss of the GAN framework. [27] introduces a feature-based loss that is calculated with a pre-trained convolutional neural network (CNN). Another approach by [14] adopts Watson's perceptual model, and [28] directly optimizes the generative model in the frequency domain by a focal frequency reconstruction loss. Most of the above-mentioned methods aim to capture the properties of human perception by replacing the element-wise loss ( L 1 or L 2 -norm), which hinders the reconstruction of images [38].

Synthetic data generation. The GAN framework is widely adopted in the synthetic data generation task since it enables synthetic data generation in a nonparametric approach [12, 47, 63, 65]. [63, 65] assume that continuous columns of tabular datasets can be approximated by the Gaussian mixture distribution and model their decoder using Gaussian mixture distribution. Additionally, [63, 65] preprocess the continuous variables using the variational Gaussian mixture model [3], which is known as the mode-specific normalization technique. However, the preprocessing step requires additional computational resources and hyperparameter tuning of the number of modes. Other approaches by [47, 65] regularize the discrepancy between the first and second-order statistics of the observed and synthetic dataset. [12] proposes the GAN-based synthesizer, which focuses on generating high-dimensional discrete variables with the assistance of the pre-trained AutoEncoder.

3 Proposal

Let x ∈ R p + q be an observation consisting of continuous and discrete variables and I = I C ∪ I D = { 1 , · · · , ( p + q ) } be an index set of the variables, where I C and I D correspond to index sets of

p continuous and q discrete variables. T j denotes the number of levels for the discrete variables x j , j ∈ I D . We denote the ground-truth underlying distribution (probability density function, PDF) as p ( x ) and the ground-truth CDF as F ( x ) .

Let z be a latent variable, where z ∈ R d and d < p + q . The prior and posterior distribution of z are assumed to be p ( z ) = N ( z | 0 , I ) and q ( z | x ; φ ) = N ( z | µ ( x ; φ ) , diag ( σ 2 ( x ; φ )) ) , respectively. Here, I is d × d identity matrix, µ : R p + q ↦→ R d , σ 2 : R p + q ↦→ R d + are neural networks parameterized with φ , and diag ( a ) , a ∈ R d denotes a diagonal matrix with diagonal elements a . Moreover, we consider α ∈ [0 , 1] as a random variable having density p ( α ) .

3.1 Distributional Learning

Our proposed model assumes that p ( x ) is parametrized by an infinite mixture of asymmetric Laplace distribution (ALD) [4]. The ALD is characterized by two parameters: α , representing the asymmetry, and β > 0 , representing the scale. By considering these parameters, along with the model parameter θ , we can define the probability model of x as follows:

Assumption 1. (1) { x j } j ∈ I are conditionally independent given z . (2) The discrete random variables { x j } j ∈ I D are independent of α . (3) α and z are independent.

By Assumption 1-(1), we model the dependency between x j s solely through the latent variable z [49]. Assumption 1-(2) implies that α is related only to the continuous variables. Then, the decoder of our VAE model denoted as p ( x | z , α ; θ, β ) is specified by equation (1):

where θ = ( θ 1 , · · · , θ p + q ) , β is a non-trainable constant, ρ v ( u ) = u ( v -I ( u < 0)) (check function), and I ( · ) denotes the indicator function. D j ( · , · ; θ j ) : [0 , 1] × R d ↦→ R is the location parameter of ALD, which is parameterized with θ j [1]. For discrete variables, π ( · ; θ j ) : R d ↦→ ∆ T j -1 is a neural network parameterized with θ j , where ∆ T j -1 is the standard ( T j -1) -simplex for all z ∈ R d , and the subscript l referes to the l th element of the output π .

Assumption 1-(3) leads our objective function,

where constant terms are omitted. In order to achieve balanced learning of the two reconstruction losses in (2), we have removed the weight β associated with the second reconstruction loss. We refer to our model as 'DistVAE.' The first term in (2) corresponds to the CRPS loss, which measures the accuracy of the proposed CDF approximation with respect to the ground-truth CDF of the underlying distribution [21, 43, 20].

Interestingly, (2) is the limit of the negative ELBO derived from a finite mixture of ALD. We introduce α , a discrete uniform random variable taking values on α k = k/K for k = 1 , · · · , K . Then, the negative ELBO of p ( x ; θ, β ) scaled with β is written by

(see Appendix A.1 for detailed derivation). The reconstruction loss, which corresponds to the first term of (3), is a composite quantile loss for estimating the target quantiles α k s [64, 35, 45, 62, 9]. This entails adopting a Bayesian perspective for α as a prior (the Bayesian modeling for estimating multiple quantiles). Furthermore, throughout the derivation of the reconstruction loss, the role of α is pivotal in ensuring the representation of the reconstruction loss. To prevent the observation x from influencing the distribution of α , α is only assigned with a prior distribution, and the resulting reconstruction loss becomes the CRPS loss, a proper scoring rule.

However, for distributional learning of V AE, it is necessary to estimate conditional quantiles for an infinite number of quantile levels, denoted by K →∞ [50]. The subsequent Theorem 1 establishes the convergence of the negative ELBO (3) to our objective function (2) as K →∞ [4].

Theorem 1. For all j ∈ I C , suppose that ∫ 1 0 E p ( x ) E q ( z | x ; φ ) ρ α ( x j -D j ( α, z ; θ j ) ) dα < ∞ , and E p ( x ) E q ( z | x ; φ ) [ ρ α ( x j -D j ( α, z ; θ j ))] is continuous over α ∈ (0 , 1) . Then,

and lim K →∞ 1 K ∑ K k =1 log α k (1 -α k ) = ∫ 1 0 log α (1 -α ) dα = -2 .

3.2 Theoretical Results

In this section, we aim to provide theoretical insights into the ability of DistV AE, utilizing the objective function (2), to recover the ground-truth distribution p ( x ) . To simplify the analysis without loss of generality, we consider the scenario where x comprises only p continuous random variables. Hence, we have I = I C = { 1 , · · · , p } , and p ( x ) is defined over x ∈ R p with p ( x ) > 0 for all x ∈ R p . First, define a function q ( x | z ; φ ) by

where q ( z ; φ ) := ∫ q ( z | x ; φ ) p ( x ) d x is the aggregated posterior [58]. Clearly, q ( x | z ; φ ) is a PDF of x for a given z . q ( x | z ; φ ) is a conditional PDF of x parametrized by φ and it is an approximated PDF of ∫ 1 0 p ( x | z , α ; θ, β ) dα . Since we assume that x j s are conditionally independent in Assumption 1, q ( x | z ; φ ) = ∏ p j =1 q j ( x j | z ; φ ) and the conditional CDF is written as

For notational simplicity, we let

where θ ∗ ( φ ) = ( θ ∗ 1 ( φ ) , · · · , θ ∗ p ( φ )) .

Assumption 2. (1) Given an arbitrary φ , F j ( ·| z ; φ ) : R ↦→ [0 , 1] is absolutely continuous and strictly monotone increasing for all j = 1 , · · · , p , and z ∈ R d . (2) Given an arbitrary θ , D j ( · , z ; θ j ) is invertible and differentiable for all j = 1 , · · · , p and z ∈ R d . (3) The aggregated posterior q ( · ; φ ) is absolutely continuous to the prior distribution of z .

Theorem 2. Under Assumption 2, for an arbitrary φ ,

Theorem 2 shows that DistVAE is capable of recovering the ground-truth distribution p ( x ) , indicating its ability to facilitate distributional learning rather than data reconstruction. Nevertheless, relying on the aggregated posterior distribution may lead to overfitting [25, 42], and sampling from the

aggregated posterior can introduce computational challenges due to the absence of a straightforward closed-form representation for q ( z ; φ ) . To address these concerns, we propose an alternative approach that leverages the prior distribution p ( z ) instead of q ( z ; φ ) , thereby enabling a computationally efficient synthetic generation process. This is substantiated by Theorem 3.

We define the estimated PDF ˆ p ( x ; θ ∗ ( φ )) and CDF ˆ F ( x ; θ ∗ ( φ )) as

Theorem 3. Suppose that φ is given such that KL ( q ( z ; φ ) ‖ p ( z )) < glyph[epsilon1] for any glyph[epsilon1] > 0 . Then, under Assumption 2,

Theorem 3 shows that even if we use the prior distribution p ( z ) instead of the aggregated posterior q ( z ; φ ) , it is feasible to minimize the KL-divergence between the ground-truth PDF p ( x ) and our estimated PDF ˆ p ( x ; θ ∗ ( φ )) of (5). This is achievable because the KL-divergence term in (2) is the upper bound of KL ( q ( z ; φ ) ‖ p ( z )) and is minimized during the training process. Since each conditional distribution of the estimated PDF depends on the same latent variable, it can be seen that the correlation structure between covariates is modeled implicitly.

[24, 41, 37] have highlighted the role of the KL-divergence weight parameter β in controlling the precision of reconstruction. In our case, where the reconstruction loss is based on the CRPS loss, an increase in β leads to a less accurate estimation of the ground-truth CDF. It implies that a large β corresponds to lower-quality synthetic data, but it also enhances privacy level. Thus, β introduces a trade-off between the quality of synthetic data and the risk of privacy leakage. The privacy level can be adjusted by manipulating β [47], as demonstrated in the experimental results presented in Section 4.

3.2.1 Synthetic Data Generation

By estimating the conditional quantile functions, we can transform the synthetic data generation process into inverse transform sampling. This conversion offers a notable advantage as it provides a straightforward and efficient approach to generating synthetic data. We denote a synthetic sample of x j as ˆ x j for j ∈ I , and the synthetic data generation process can be summarized as follows:

Sampling from the prior distribution: z ∼ p ( z ) .

Inverse transform sampling: For j ∈ I C , ˆ x j = D j ( u j | z ; θ j ) , where u j ∼ U (0 , 1) .

Gumbel-Max trick [23]: For j ∈ I D , ˆ x j = arg max l =1 , ··· ,T j { log π l ( z ; θ j ) + G l } , where G l ∼ Gumbel (0 , 1) , and l = 1 , · · · , T j .

Note that both continuous and discrete variables share the same latent variable z . This shared latent variable allows for capturing the dependencies between variables. We numerically observe that the sampling, implemented using the Gumbel-Max trick, maintains the imbalanced ratio of the labels in the discrete variable.

3.2.2 Parameterization of ALD

As introduced in [20], for j ∈ I C , we parameterize the function D j , the location parameter of ALD, by a linear isotonic spline as follows:

where γ ( j ) ( z ) ∈ R , b ( j ) ( z ) = ( b ( j ) 0 ( z ) , · · · , b ( j ) M ( z )) ∈ R M +1 , d = ( d 0 , · · · , d M ) ∈ [0 , 1] M +1 , 0 = d 0 < · · · < d M = 1 , and ( u ) + := max(0 , u ) . θ j is a neural network parameterized mapping such that θ j : R d ↦→ R × R M +1 , which takes z as input and outputs γ ( j ) ( z ) and b ( j ) ( z ) . The constraint is introduced to ensure monotonicity. As demonstrated in [20], the reconstruction loss can be computed in a closed form by utilizing the parameterization of (7) (refer to Appendix A.5 for a detailed description of the loss function). This implies that our objective function (2) is computationally tractable. Note that the linear isotonic spline is not differentiable for finite points where the measure has no point mass.

4 Experiments

4.1 Overview

Dataset. For evaluation, we consider following real tabular datasets: covertype , credit , loan , adult , cabs , and kings (see Appendix A.8 for detailed data descriptions). We treat the ordinal variables as continuous variables and discretize the estimated CDF (see Appendix A.6 for the discretization algorithm). Synthetic samples of ordinal variables are rounded to the first decimal place.

Compared models. We compare DistVAE † with the state-of-the-art synthesizers; CTGAN [63], TVAE [63], and CTAB-GAN [65]. All models have the same size of the latent dimension ( d = 2 ). The chosen latent space indeed limits the capacity of decoders for all models. However, we maintain a small and consistent number of parameters across all models during the experiment to isolate the performance differences in synthetic data generation to the methodologies of each synthesizer, specifically emphasizing the contribution of the decoder model's flexibility in estimating underlying distributions (see Table 10 in Appendix A.9 for a comprehensive comparison of the model parameters).

4.2 Evaluation Metrics

To assess the quality of the synthetic data, we employ three types of assessment criteria: 1) machine learning utility, 2) statistical similarity, and 3) privacy preservability. Each criterion is evaluated using multiple metrics, and the performance of synthesizers is reported by averaged metrics over the real tabular datasets. The synthetic dataset is generated to have an equal number of samples as the real training dataset.

Machine learning utility. The machine learning utility (MLu) is measured by the predictive performance of the trained model over the synthetic data. We consider three popular machine learning algorithms: linear (logistic) regression, Random Forest [6], and Gradient Boosting [19]. We measure the performance by utilizing the mean absolute relative error (MARE) for regression tasks [47] and the F 1 score for classification tasks [63, 65, 61, 29, 47, 12, 18].

Statistical similarity. The marginal distributional similarity between the real training and synthetic datasets is evaluated using two metrics: the Kolmogorov statistic and the 1-Wasserstein distance [18]. These metrics measure the distance between the empirical marginal CDFs [40]. The joint distributional similarity is assessed by comparing the correlation matrices [65]. To compute the correlation matrix and measure the L 2 distance between the correlation matrices of the real training and synthetic datasets, we employ the dython library ‡ . These enable a comprehensive evaluation of both marginal and joint distributional similarities between the real training and synthetic datasets.

Privacy preservability. The privacy-preserving capacity is measured using three metrics: the distance to closest record (DCR) [47, 65], membership inference attack [54, 12, 47], and attribute disclosure [12, 44]. As in [65], the DCR is defined as the 5 th percentile of the L 2 distances between all real training samples and synthetic samples. Since the DCR is a L 2 distance-based metric, it is computed using only continuous variables. A higher DCR value indicates a more effective preservation of privacy, indicating a lack of overlap between the real training data and the synthetic samples. Conversely, an excessively large DCR score suggests a lower quality of the generated synthetic dataset. Therefore, the DCR metric provides insights into both the privacy-preserving capability and the quality of the synthetic dataset.

The membership inference attack evaluation follows the steps detailed in Appendix A.7. The procedure is customized to be applied to a V AE-based synthesizer, such as DistV AE and TV AE. By transforming the problem into a binary classification task, we aim to identify the intricate relationship between the real training data and the synthetic samples. Higher binary classification scores indicate a higher vulnerability of the target synthesizer to membership inference attacks.

Attribute disclosure refers to the situation where attackers can uncover additional covariates of a record by leveraging a subset of covariates they already possess, along with similar records from the synthetic dataset. To quantify the extent to which attackers can accurately identify these additional covariates, we employ classification metrics. Higher attribute disclosure metrics indicate an increased risk of privacy leakage, implying that attackers can precisely infer unknown variables. In terms of privacy concerns, attribute disclosure can be considered a more significant issue than membership inference attacks, as attackers are assumed to have access to only a subset of covariates for a given record [12].

4.3 Results

Machine learning utility. We expect a high-quality synthesizer to generate synthetic data with comparable predictive performance to the real training dataset, denoted as the 'Baseline' in Table 1. The results in Table 1 demonstrate that DistV AE achieves a competitive MARE score and outperforms other methods in terms of the F 1 score. Furthermore, the performance of MLu improves as the value of β decreases, indicating that the quality of the generated synthetic data is controlled by β . For a comprehensive overview of the MLu scores for all tabular datasets, please refer to Appendix A.10.

Statistical similarity. The evaluation results for joint and marginal distributional similarities are presented in Table 2 and 3. In Table 2, DistV AE achieves the lowest CorrDist score, indicating its ability to accurately preserve the correlation structure of the real training dataset in the generated synthetic dataset. Furthermore, DistVAE surpasses other methods in Table 3 when it comes to marginal distributional similarity, suggesting that it successfully captures the underlying distribution of the observed dataset. Notably, reducing the value of β leads to an enhancement in the quality of the synthetic dataset, as evidenced by improvements in the correlation structure and similarity of the marginal distributions. Figure 1 provides visualizations of the estimated CDFs (6) for each continuous (or ordinal) variable in the cabs dataset. For detailed statistical similarity scores and additional visualizations of estimated CDFs for all tabular datasets, please refer to Appendix A.10.

Privacy preservability. The privacy preservability performances of synthesizers, as measured by the DCR, are presented in Table 4. DistVAE performs best in preserving privacy, with the highest DCR values compared to other methods. Notably, as the value of β increases in DistV AE, the DCR between the real training and synthetic datasets (R&S) also increases. This indicates that the risk of privacy leakage can be controlled by adjusting β , where higher values of β correspond to a higher level of privacy protection. Moreover, DistVAE consistently achieves large DCR values for the synthetic dataset (S) across all β values, indicating its ability to generate diverse synthetic samples. On the other hand, CTAB-GAN generates duplicated records in the synthetic dataset, resulting in

1.00

1.00

1.00

1.00

1.00

1.00

0.75

0.75

0.75

0.75

0.75

0.75

0.50

0.50

0.50

0.50

0.50

0.50

0.25

0.25

0.25

0.25

0.25

0.25

empirical

0.00

0.00

0.00

0.00

0.00

0.00

estimate

0

2

0

10

2

0

2

0

5

0

10

0

5

10

Trip_Distance

Life_Style_Index

Customer_Rating

Var1

Var2

Var3

relatively lower DCR scores for the synthetic dataset (S). For detailed DCR scores for all tabular datasets, please refer to Appendix A.10.

To evaluate the membership inference attack, we prepare one attack model per class. The attack testing records comprise an equal number of real training and test records, distinguished by the labels in and out , respectively. Note that the test records are not employed in constructing the attack models. We employ gradient-boosting classifiers as the attack models, and for computational feasibility, we limit the number of attack models to one (i.e., C = 1 ).

For the membership inference attack evaluation, we utilize accuracy and AUC (Area Under Curve) as the evaluation metrics. Since the target labels ( in/out ) are balanced, and the task is a binary classification problem, these metrics are appropriate. The results presented in Table 5 reveal that both DistVAE and TVAE achieve nearly identical accuracy and AUC scores of 0.5. This indicates that the attack models can not distinguish between members of the real training and test datasets. Consequently, the membership inference attack is unsuccessful for both models. Therefore, DistV AE effectively generates synthetic datasets while ensuring privacy against membership inference attacks. A comprehensive assessment of membership inference attack performances for all tabular datasets can be found in Appendix A.10.

We present the attribute disclosure performance results in Table 6. For each value of k , we observe that as β increases, the F 1 score of DistVAE decreases. Also, DistVAE achieves the smallest F 1 score when k equals 10 and 100. Based on these results, we can conclude that DistV AE can generate synthetic datasets with a low risk of attribute disclosure, and the level of privacy preservation is controlled by β . Please refer to Appendix A.10 for a detailed evaluation of attribute disclosure performance for all tabular datasets.

Quantile estimation. To investigate the quantile estimation performance, we also evaluate DistV AE using the Vrate( α ) metric [11]. The Vrate( α ) is defined as 1 | I test | ∑ i ∈ I test I ( x i < ˆ Q α ) , where α ∈ (0 , 1) , I test is the set of indices for the test dataset, x i is the i -th sample in the test dataset, and ˆ Q α is the empirical α -quantile of the synthetic data. Since Vrate( α ) indicates the proportion of compliance samples in the test dataset, the Vrate( α ) score should be close to α .

The Vrate( α ) evaluation results are presented in Table 7. Table 7 shows that the ratio of violated test samples ( | α -Vrate ( α ) | ) decreases as α increases, indicating a better performance for estimating larger quantiles. However, the ratio of violated test samples is relatively large for smaller α values, which may be due to extremely skewed continuous variables, such as capital-gain and capitalloss from adult dataset, that make the quantile estimation unstable.

5 Conclusion and Limitations

This paper introduces a novel distributional learning method of VAE, which aims to effectively capture the underlying distribution of the observed dataset using a nonparametric approach. Our proposed method involves directly estimating CDFs using the CRPS loss while maintaining the mathematical derivation of the lower bound.

In our study, we confirm that the proposed decoder enhances the performance of generative models for tabular data. However, this conclusion relies on the assumption of conditional independence among the observed variables given the latent variable. When the dimension of the latent variable is small, this assumption is prone to violation. Therefore, in cases where the size of the latent

space is limited, the proposed nonparametric fitting of the decoder might not accurately represent the underlying distribution. Particularly in the image domain, where adjacent pixel values exhibit significant dependence, it remains uncertain whether our proposed model would lead to notable improvements in image data generation with a low-dimensional latent space.

Nevertheless, classical image datasets, such as CIFAR-10 [36], often exhibit pixel value distributions that deviate considerably from Gaussian, with the frequencies of edge values (0 and 255) dominating more than other pixel values [16, 53]. Other image datasets, as presented in [48], demonstrate multi-modality in pixel value distributions. These experimental findings suggest that leveraging the capacity of distributional learning could be advantageous in approximating the ground-truth distribution of image data when the latent variable effectively captures conditional independence among the image pixels. Consequently, we expect that compromising biases arising from violating conditional independence and marginally misspecified distributions may further enhance our results, and we leave it for future research.

On the other hand, we consider two approaches to enhance the performance of quantile estimation. Firstly, we plan to extend the parameterization of conditional quantile functions to a more flexible monotonic regression model. Secondly, we intend to incorporate the Uniform Pessimistic Risk (UPR) [26] into the V AE framework to handle lower quantile levels better. Furthermore, we are exploring the expansion of DistVAE into a time-series distributional forecasting model by adopting the conditional VAE framework [56]. This extension will enable the application of our method to time-series data, opening new avenues for distributional forecasting.

Acknowledgments and Disclosure of Funding

This work was supported by the National Research Foundation of Korea (NRF) grant funded by the Korea government (MSIT) (No. NRF-2022R1A4A3033874 and No. NRF-2022R1F1A1074758). This work was also supported by Korea Environmental Industry & Technology Institute (KEITI) through 'Core Technology Development Project for Environmental Diseases Prevention and Management', funded by Korea Ministry of Environment (MOE) (2021003310005). The authors acknowledge the Urban Big data and AI Institute of the University of Seoul supercomputing resources ( http://ubai.uos.ac.kr ) made available for conducting the research reported in this paper.

References

Haleh Akrami, Anand Joshi, Sergul Aydore, and Richard Leahy. Deep quantile regression for uncertainty estimation in unsupervised and supervised lesion detection. Machine Learning for Biomedical Imaging , 1:1-23, 2022.

Jonathan T. Barron. A general and adaptive robust loss function. 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , pages 4326-4334, 2017.

David M. Blei, Alp Kucukelbir, and Jon D. McAuliffe. Variational inference: A review for statisticians. Journal of the American Statistical Association , 112:859 - 877, 2016.

Axel Brando, Jose A Rodriguez, Jordi Vitria, and Alberto Rubio Muñoz. Modelling heterogeneous distributions with an uncountable mixture of asymmetric laplacians. Advances in neural information processing systems , 32, 2019.

Gustav Bredell, Kyriakos Flouris, Krishna Chaitanya, Ertunc Erdil, and Ender Konukoglu. Explicitly minimizing the blur error of variational autoencoders. In The Eleventh International Conference on Learning Representations , 2023.

Leo Breiman. Random forests. Machine learning , 45:5-32, 2001.

Andrew Brock, Jeff Donahue, and Karen Simonyan. Large scale GAN training for high fidelity natural image synthesis. In International Conference on Learning Representations , 2019.

Yuri Burda, Roger Baker Grosse, and Ruslan Salakhutdinov. Importance weighted autoencoders. CoRR , abs/1509.00519, 2015.

Alex J. Cannon. Non-crossing nonlinear regression quantiles by monotone composite quantile regression neural network, with application to rainfall extremes. Stochastic Environmental Research and Risk Assessment , 32:3207-3225, 2018.

Lluís Castrejón, Nicolas Ballas, and Aaron C. Courville. Improved conditional vrnns for video prediction. 2019 IEEE/CVF International Conference on Computer Vision (ICCV) , pages 7607-7616, 2019.

Cathy W.S. Chen, Richard Gerlach, Bruce B.K. Hwang, and Michael McAleer. Forecasting Value-at-Risk using nonlinear regression quantiles and the intra-day range. International Journal of Forecasting , 28(3):557-574, 2012.

E. Choi, Siddharth Biswal, Bradley A. Malin, Jon D. Duke, Walter F. Stewart, and Jimeng Sun. Generating multi-label discrete patient records using generative adversarial networks. In Machine Learning in Health Care , 2017.

Antonia Creswell, Tom White, Vincent Dumoulin, Kai Arulkumaran, Biswa Sengupta, and Anil A Bharath. Generative adversarial networks: An overview. IEEE signal processing magazine , 35(1):53-65, 2018.

Steffen Czolbe, Oswin Krause, Ingemar Cox, and Christian Igel. A loss function for generative neural networks based on watson's perceptual model. In Proceedings of the 34th International Conference on Neural Information Processing Systems , NIPS'20, Red Hook, NY, USA, 2020. Curran Associates Inc.

Bin Dai and David Wipf. Diagnosing and enhancing VAE models. In International Conference on Learning Representations , 2019.

Luke N Darlow, Elliot J Crowley, Antreas Antoniou, and Amos J Storkey. Cinic-10 is not imagenet or cifar-10. arXiv preprint arXiv:1810.03505 , 2018.

Andrés Diaz-Pinto, Adrián Colomer, Valery Naranjo, Sandra Morales, Yanwu Xu, and Alejandro F Frangi. Retinal image synthesis for glaucoma assessment using dcgan and vae models. In Ideal , 2018.

Kevin Fang, Vaikkunth Mugunthan, Vayd Ramkumar, and Lalana Kagal. Overcoming challenges of synthetic data generation. 2022 IEEE International Conference on Big Data (Big Data) , pages 262-270, 2022.

Jerome H Friedman. Greedy function approximation: a gradient boosting machine. Annals of statistics , pages 1189-1232, 2001.

Jan Gasthaus, Konstantinos Benidis, Yuyang Wang, Syama Sundar Rangapuram, David Salinas, Valentin Flunkert, and Tim Januschowski. Probabilistic forecasting with spline quantile function rnns. In The 22nd international conference on artificial intelligence and statistics , pages 1901-1910. PMLR, 2019.

Tilmann Gneiting and Adrian E. Raftery. Strictly proper scoring rules, prediction, and estimation. Journal of the American Statistical Association , 102:359 - 378, 2007.

Ian J. Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron C. Courville, and Yoshua Bengio. Generative adversarial nets. In NIPS , 2014.

Emil Julius Gumbel. Statistical theory of extreme values and some practical applications: a series of lectures , volume 33. US Government Printing Office, 1954.

Irina Higgins, Loic Matthey, Arka Pal, Christopher Burgess, Xavier Glorot, Matthew Botvinick, Shakir Mohamed, and Alexander Lerchner. beta-VAE: Learning basic visual concepts with a constrained variational framework. In International Conference on Learning Representations , 2017.

Matthew D Hoffman and Matthew J Johnson. Elbo surgery: yet another way to carve up the variational evidence lower bound. In Workshop in Advances in Approximate Bayesian Inference, NIPS , volume 1, 2016.

Sungchul Hong and Jong-June Jeon. Uniform pessimistic risk and optimal portfolio. ArXiv , abs/2303.07158, 2023.

Xianxu Hou, L. Shen, Ke Sun, and Guoping Qiu. Deep feature consistent variational autoencoder. 2017 IEEE Winter Conference on Applications of Computer Vision (WACV) , pages 1133-1141, 2016.

Liming Jiang, Bo Dai, Wayne Wu, and Chen Change Loy. Focal frequency loss for image reconstruction and synthesis. 2021 IEEE/CVF International Conference on Computer Vision (ICCV) , pages 13899-13909, 2020.

Sanket Kamthe, Samuel A. Assefa, and Marc Peter Deisenroth. Copula flows for synthetic data generation. ArXiv , abs/2101.00598, 2021.

Tero Karras, Samuli Laine, and Timo Aila. A style-based generator architecture for generative adversarial networks. 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , pages 4396-4405, 2018.

Diederik P. Kingma and Max Welling. Auto-Encoding Variational Bayes. In 2nd International Conference on Learning Representations, ICLR 2014, Banff, AB, Canada, April 14-16, 2014, Conference Track Proceedings , 2014.

Durk P Kingma, Shakir Mohamed, Danilo Jimenez Rezende, and Max Welling. Semi-supervised learning with deep generative models. In Advances in neural information processing systems , pages 3581-3589, 2014.

Durk P Kingma, Tim Salimans, Rafal Jozefowicz, Xi Chen, Ilya Sutskever, and Max Welling. Improved variational inference with inverse autoregressive flow. In D. Lee, M. Sugiyama, U. Luxburg, I. Guyon, and R. Garnett, editors, Advances in Neural Information Processing Systems , volume 29. Curran Associates, Inc., 2016.

A Kiran and S Saravana Kumar. A comparative analysis of gan and vae based synthetic data generators for high dimensional, imbalanced tabular data. In 2023 2nd International Conference for Innovation in Technology (INOCON) , pages 1-6. IEEE, 2023.

Roger W. Koenker, Limin Peng, Victor Chernozhukov, and Xuming He. Handbook of quantile regression. 2017.

Alex Krizhevsky. Learning multiple layers of features from tiny images. 2009.

Abhishek Kumar and Ben Poole. On implicit regularization in β -vaes. In International Conference on Machine Learning , pages 5480-5490. PMLR, 2020.

Anders Boesen Lindbo Larsen, Søren Kaae Sønderby, Hugo Larochelle, and Ole Winther. Autoencoding beyond pixels using a learned similarity metric. In Maria Florina Balcan and Kilian Q. Weinberger, editors, Proceedings of The 33rd International Conference on Machine Learning , volume 48 of Proceedings of Machine Learning Research , pages 1558-1566, New York, New York, USA, 20-22 Jun 2016. PMLR.

Alex X. Lee, Anusha Nagabandi, Pieter Abbeel, and Sergey Levine. Stochastic latent actorcritic: Deep reinforcement learning with a latent variable model. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems , volume 33, pages 741-752. Curran Associates, Inc., 2020.

Erich Leo Lehmann. Elements of large-sample theory. 1998.

James Lucas, G. Tucker, Roger Baker Grosse, and Mohammad Norouzi. Don't blame the elbo! a linear vae perspective on posterior collapse. In Neural Information Processing Systems , 2019.

Alireza Makhzani, Jonathon Shlens, Navdeep Jaitly, Ian Goodfellow, and Brendan Frey. Adversarial autoencoders. arXiv preprint arXiv:1511.05644 , 2015.

James E. Matheson and Robert L. Winkler. Scoring rules for continuous probability distributions. Management Science , 22:1087-1096, 1976.

Stan Matwin, Jordi Nin, Morvarid Sehatkar, and Tomasz Szapiro. A review of attribute disclosure control. In Advanced Research in Data Privacy , 2015.

Sangjun Moon, Jong-June Jeon, Jason Eng Hun Lee, and Yongdai Kim. Learning multiple quantiles with neural networks. Journal of Computational and Graphical Statistics , 30:1238 1248, 2021.

Prateek Munjal, Akanksha Paul, and N. C. Krishnan. Implicit discriminator in variational autoencoder. 2020 International Joint Conference on Neural Networks (IJCNN) , pages 1-8, 2019.

Noseong Park, Mahmoud Mohammadi, Kshitij Gorde, Sushil Jajodia, Hongkyu Park, and Youngmin Kim. Data synthesis based on generative adversarial networks. Proc. VLDB Endow. , 11:1071-1083, 2018.

V. Parmar, Bogdan Penkovsky, Damien Querlioz, and Manan Suri. Hardware-efficient stochastic binary cnn architectures for near-sensor computing. Frontiers in Neuroscience , 15, 2022.

J. Peters, Dominik Janzing, and Bernhard Schölkopf. Elements of causal inference: Foundations and learning algorithms. 2017.

Kostantinos N Plataniotis and Dimitris Hatzinakos. Gaussian mixtures and their applications to signal processing. Advanced signal processing handbook , pages 89-124, 2017.

Danilo Jimenez Rezende, Shakir Mohamed, and Daan Wierstra. Stochastic backpropagation and approximate inference in deep generative models. In International Conference on Machine Learning , 2014.

Mihaela Rosca, Balaji Lakshminarayanan, David Warde-Farley, and Shakir Mohamed. Variational approaches for auto-encoding generative adversarial networks. ArXiv , abs/1706.04987, 2017.

Tim Salimans, Andrej Karpathy, Xi Chen, and Diederik P. Kingma. PixelCNN++: Improving the pixelCNN with discretized logistic mixture likelihood and other modifications. In International Conference on Learning Representations , 2017.

R. Shokri, Marco Stronati, Congzheng Song, and Vitaly Shmatikov. Membership inference attacks against machine learning models. 2017 IEEE Symposium on Security and Privacy (SP) , pages 3-18, 2016.

Nicki Skafte, Martin Jø rgensen, and Sø ren Hauberg. Reliable training and estimation of variance networks. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buc, E. Fox, and R. Garnett, editors, Advances in Neural Information Processing Systems , volume 32. Curran Associates, Inc., 2019.

Kihyuk Sohn, Honglak Lee, and Xinchen Yan. Learning structured output representation using deep conditional generative models. In NIPS , 2015.

Hiroshi Takahashi, Tomoharu Iwata, Yuki Yamanaka, Masanori Yamada, and Satoshi Yagi. Student-t variational autoencoder for robust density estimation. In International Joint Conference on Artificial Intelligence , 2018.

Jakub Tomczak and Max Welling. Vae with a vampprior. In Amos Storkey and Fernando Perez-Cruz, editors, Proceedings of the Twenty-First International Conference on Artificial Intelligence and Statistics , volume 84 of Proceedings of Machine Learning Research , pages 1214-1223. PMLR, 09-11 Apr 2018.

Kunfeng Wang, Chao Gou, Yanjie Duan, Yilun Lin, Xinhu Zheng, and Fei-Yue Wang. Generative adversarial networks: introduction and outlook. IEEE/CAA Journal of Automatica Sinica , 4(4):588-598, 2017.

Wenlin Wang, Zhe Gan, Hongteng Xu, Ruiyi Zhang, Guoyin Wang, Dinghan Shen, Changyou Chen, and Lawrence Carin. Topic-guided variational auto-encoder for text generation. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) , pages 166-177, Minneapolis, Minnesota, June 2019. Association for Computational Linguistics.

Bingyang Wen, Yupeng Cao, Fan Yang, Koduvayur Subbalakshmi, and Rajarathnam Chandramouli. Causal-TGAN: Causally-aware synthetic tabular data generative adversarial network, 2022.

Ruofeng Wen, Kari Torkkola, Balakrishnan Narayanaswamy, and Dhruv Madeka. A multihorizon quantile recurrent forecaster. arXiv: Machine Learning , 2017.

Lei Xu, Maria Skoularidou, Alfredo Cuesta-Infante, and Kalyan Veeramachaneni. Modeling tabular data using conditional gan. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'AlchéBuc, E. Fox, and R. Garnett, editors, Advances in Neural Information Processing Systems , volume 32. Curran Associates, Inc., 2019.

Keming Yu and Rana Moyeed. Bayesian quantile regression. Statistics & Probability Letters , 54:437-447, 2001.

Zilong Zhao, Aditya Kunar, Robert Birke, and Lydia Y. Chen. Ctab-gan: Effective table data synthesizing. In Vineeth N. Balasubramanian and Ivor Tsang, editors, Proceedings of The 13th Asian Conference on Machine Learning , volume 157 of Proceedings of Machine Learning Research , pages 97-112. PMLR, 17-19 Nov 2021.

A Appendix

A.1 Derivation of ELBO

by Jensen's inequality.

A.2 Proof of Theorem 1

Proof.

where α 0 := 0 and E p ( x ) E q ( z | x ; φ ) [ ρ α k ( x j -D j ( α k | z , θ j ) )] is denoted as h ( α k ) .

Since α k ∈ [ α k -1 , α k ] and h ( · ) : [0 , 1] ↦→ R is a continuous function, for j ∈ I C ,

by the definition of the Riemann integral and the Fubini-Tonelli theorem. The proof is complete.

A.3 Proof of Theorem 2

Proof.

So, for all j = 1 , · · · , p ,

and it is proper scoring rule relative to F j ( ·| z ; φ ) for all z ∈ R d . It implies that D -1 j ( x j , z ; θ ∗ j ( φ )) = F j ( x j | z ; φ ) , and d d x j D -1 j ( x j , z ; θ ∗ j ( φ )) = q j ( x j | z ; φ ) , for all x j ∈ R , by Assumption 2. It follows that

The proof is complete.

A.4 Proof of Theorem 3

Proof.

So, for all j = 1 , · · · , p ,

and it is proper scoring rule relative to F j ( ·| z ; φ ) for all z ∈ R d . It implies that D -1 j ( x j , z ; θ ∗ j ( φ )) = F j ( x j | z ; φ ) , and d d x j D -1 j ( x j , z ; θ ∗ j ( φ )) = q j ( x j | z ; φ ) , for all x j ∈ R , by Assumption 2.

It follows that

by Jensen's inequality, Fubini-Tonelli theorem, and the assumptions. The proof is complete.

A.5 Closed Form Loss

A.6 Discretization of Estimated CDF

To ensure appropriate discretization of the estimated CDF for ordinal variables, we propose a post-adhoc discretization step [53]. We focus on the case where p = 1 and q = 0 for brevity. We denote the set of observed possible values for the ordinal variable as x (1) , x (2) , · · · , x ( m ) . The discretization algorithm for the estimated CDF is presented in Algorithm 1, and we provide an example of the discretization algorithm's outcome in Figure 2.

Algorithm 1 Discretization of Estimated CDF

Input { x (1) , x (2) , · · · , x ( m ) } , Estimated CDF ˆ F ( · ; θ )

Output Discretized CDF ˆ F ∗ ( · ; θ )

Compute ˆ F ( x ( i ) -0 . 5; θ ) and ˆ F ( x ( i ) +0 . 5; θ ) for i = 1 , · · · , m .

(2) Discretization: For i = 1 , · · · , m ,

where ˆ F ∗ ( x (0) ; θ ) := 0 .

(3) Ensure monotonicity: For i = 1 , · · · , m -1 , if ˆ F ∗ ( x ( i ) ; θ ) > ˆ F ∗ ( x ( i +1) ; θ ) ,

1.0

empirical

estimate

0.8

calibration

0.6

0.4

0.2

0.0

2

4

6

8

10

12

14

16

educational-num

A.7 Membership Inference Attack

[47] propose the customized membership inference attack method of [54] to attack the GAN-based synthesizer. Similarly, we propose the customized membership inference attack method of [54] to attack the V AE-based synthesizer.

Assumption 3 ([54]) . In the membership inference attack, the attacker attacks the target model under the following assumptions:

(A1) The attacker is only allowed for black-box access, where the attacker can only supply inputs to the model and receive the model's output(s).

(A2) The attacker can obtain as many outputs as they want from a target model to attack.

(A3) The real and synthetic datasets should not have common records.

(A4) The attacker knows the algorithm and architecture of the target model.

Denote D ∗ train and D ∗ test as the real training and test datasets. Under Assumption 3, the overall steps of the membership inference attack are outlined below:

Generate shadow training and test datasets D ( i ) train , D ( i ) test , i = 1 , · · · , C from M ∗ by (A1), where M ∗ is the model attacker wants to attack. By (A2), the attacker is allowed to obtain shadow datasets such as | D ( i ) train | = | D ∗ train | and D ( i ) train ∩ D ( j ) test = ∅ , for i = 1 , · · · , C . Under (A3), D ( i ) train ∩ D ∗ train = ∅ , for i = 1 , · · · , C .

Train shadow models M 1 , · · · , M C under (A4), i.e., each shadow model is trained similarly to the target model M ∗ .

For i = 1 , · · · , C ,

(a) Obtain representation vectors z from the encoder of M i with the input of D ( i ) train . Then, attacking training records are ( y , z , in ) .

(b) Obtain representation vectors z from the encoder of M i with the input of D ( i ) test . Then, attacking training records are ( y , z , out ) .

where y is the labels of shadow dataset records. And we assume that y consists of the MLu classification target.

Merge all attack training records, ( y , z , in/out ) .

For each class of y , train attack model which is a binary classification model which classifies in/out based on the representation vectors z .

Now it is ready to attack.

Note that we use the representation vector of V AE instead of the output of the GAN discriminator [47].

A.8 Dataset Descriptions

Websites

covertype : https://www.kaggle.com/datasets/uciml/ forest-cover-type-dataset

credit : https://www.kaggle.com/c/home-credit-default-risk

loan : https://www.kaggle.com/datasets/teertha/personal-loan-modeling

adult : https://www.kaggle.com/datasets/uciml/adult-census-income

cabs : https://www.kaggle.com/datasets/arashnic/ taxi-pricing-with-mobility-analytics?select=test.csv

kings : https://www.kaggle.com/datasets/harlfoxem/housesalesprediction

A.9 Experimental Settings

We run all experiments using Geforce RTX 3090 GPU, and our experimental codes are all available with pytorch .

A.10 Detailed Experimental Results

(a)

covtype

(b)

credit

(c)

loan

1.00

1.00

1.00

1.00

1.00

0.75

0.75

0.75

0.75

0.75

0.50

0.50

0.50

0.50

0.50

0.25

0.25

0.25

0.25

0.25

0.00

0.00

0.00

0.00

0.00

-2.5

0.0

2.5

-1

0

1

0

5

0.0

2.5

5.0

0

5

Elevation

Aspect

Slope

Horizontal_Distance_To_Hydrology

Vertical_Distance_To_Hydrology

1.00

1.00

1.00

1.00

1.00

0.75

0.75

0.75

0.75

0.75

0.50

0.50

0.50

0.50

0.50

0.25

0.25

0.25

0.25

0.25

empirical

0.00

0.00

0.00

0.00

0.00

estimate

0

2

5

0

5

0

2.5

0.0

2.5

0

2

4

Horizontal_Distance_To_Roadways

Hillshade_9am

Hillshade_Noon

Hillshade_3pm

Horizontal_Distance_To_Fire_Poin

1.00

1.00

1.00

1.00

1.00

0.75

0.75

0.75

0.75

0.75

0.50

0.50

0.50

0.50

0.50

0.25

0.25

0.25

0.25

0.25

0.00

0.00

0.00

0.00

0.00

0

50

100

0

5

0

10

0

5

0

2

AMT_INCOME_TOTAL

AMT_CREDIT

AMT_ANNUITY

REGION_POPULATION_RELATIVE

1.00

1.00

1.00

1.00

1.00

0.75

0.75

0.75

0.75

0.75

0.50

0.50

0.50

0.50

0.50

0.25

0.25

0.25

0.25

0.25

empirical

0.00

0.00

0.00

0.00

0.00

estimate

-2

0

2

0

2

5.0

2.5

0.0

2

0

2

0.0

2.5

DAYS_BIRTH

DAYS_EMPLOYED

DAYS_REGISTRATION

DAYS_ID_PUBLISH

OWN_CAR_AGE

1.00

1.00

1.00

1.00

1.0

0.75

0.75

0.75

0.75

0.8

0.50

0.50

0.50

0.50

0.6

0.25

0.25

0.25

0.25

0.4

empirical

0.00

0.00

0.00

0.00

estimate

-2

0

2

-2

0

2

0

2

0.0

2.5

0.0

2.5

5.0

Age

Experience

Income

Mortgage

(a)

adult

(b)

cabs

(c)

kings

1.00

1.00

1.0

1.0

1.00

0.75

0.75

0.75

0.8

0.8

0.50

0.50

0.50

0.6

0.25

0.25

0.25

0.6

empirical

0.00

0.00

0.4

0.00

estimate

0.0

2.5

2.5

0.0

2.5

0

5

10

0

5

10

0

5

age

educational-num

capital-gain

capital-loss

hours-per-week

1.00

1.00

1.00

1.00

1.00

1.00

0.75

0.75

0.75

0.75

0.75

0.75

0.50

0.50

0.50

0.50

0.50

0.50

0.25

0.25

0.25

0.25

0.25

0.25

empirical

0.00

0.00

0.00

0.00

0.00

0.00

estimate

0

2

0

10

2

0

2

0

5

0

10

0

5

10

Trip_Distance

Life_Style_Index

Customer_Rating

Var1

Var2

Var3

1.00

1.00

1.00

1.00

1.0

1.00

0.75

0.75

0.75

0.75

0.8

0.75

0.50

0.50

0.50

0.50

0.50

0.6

0.25

0.25

0.25

0.25

0.25

0.4

0.00

0.00

0.00

0.00

0.00

0

10

20

0

10

0

20

40

0

5

0

5

2

0

price

sqft_iving

sqft_lot

sqft_above

sqft_basement

yr_built

1.0

1.00

1.00

1.00

1.00

1.0

0.8

0.75

0.75

0.75

0.75

8'0

0.50

0.50

0.50

0.50

0.6 -

0.6

0.4

0.25

0.25

0.25

0.25

0.4

-z'0

0

2

4

0.00

2

0

0.00

0

5

0.00

0

5

0.00

0

20

0.0

0.00

0.25

0.50

0.75

1.00

yr_renovated

lat

long

sqft_living15

sqft_lot15

(a)

(b)

DistVAE

CTGAN

0.8

TVAE

CTAB-GAN

0.6

covtype

credit

0.4

loan

adult

★

0.2

+

cabs

kings

0.0

0.2

0.4

0.6

0.8

MARE

For a detailed comparison of the models and their performance, we present the paired (MARE, F 1 ) scores for all tabular datasets in Figure 5. A better score, indicating a superior performance in terms of MLu, is represented by a dot located in the upper left corner. Notably, Figure 5 consistently demonstrates that DistVAE achieves the best or at least competitive MLu across all tabular datasets. TVAE exhibits a notably low F 1 score in the credit dataset because it fails to handle the highly imbalanced discrete target variable. This comparative analysis highlights the strong MLu performance of DistVAE and the specific limitations of TVAE in certain scenarios.

(b) Continuous cabs

1-WD

±

0

.

017

±

±

±

±

±

0

0

0

0

0

.

.

.

.

.

070

053

009

007

002

±

±

±

±

±

0

0

0

0

0

.

045

.

098

.

086

.

044

.

046

K-S

0

.

007

0

0

0

0

0

0

±

.

052

(c) Discrete credit

K-S

1-WD

±

0

.

011

±

±

±

±

±

0

0

0

0

0

.

.

.

.

.

033

126

046

025

0

.

008

0

0

0

0

±

±

±

±

±

0

0

0

0

0

.

061

.

583

.

034

.

020

.

018

014

0

±

0

.

027

(d) Discrete

1-WD

±

0

.

051

±

±

±

±

±

0

0

0

0

0

.

.

.

.

.

273

294

026

019

010

0

0

0

0

0

0

.

069

.

411

.

086

.

060

.

070

.

083

±

±

±

±

±

±

045

.

.

006

.

003

.

001

.

.

0

1

.

0

.

0

.

0

.

001

0

.

cabs

K-S

0

.

016

0

0

0

0

0

.

088

.

013

.

017

.

003

.

003

0

.

1

.

0

.

0

.

0

.

0

.

147

566

076

046

042

068

238

202

443

241

292

326

.

008

.

008

.

003

.

002

.

0

.

0

.

0

.

0

.

0

.

060

139

118

067

068

001

0

.

074

Dataset

Model

CTGAN

TVAE

CTAB-GAN

DistVAE(

= 0

β

DistVAE(

DistVAE(

Dataset

Model

CTGAN

TVAE

CTAB-GAN

DistVAE(

= 0

β

DistVAE(

DistVAE(

Dataset

Model

CTGAN

TVAE

CTAB-GAN

DistVAE(

= 0

β

DistVAE(

DistVAE(

β

β

= 1

= 5

β

β

= 1

= 5

β

β

= 1

= 5

.

5

)

)

.

5

)

)

.

5

)

)

)

)

)

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

.

323

.

477

.

275

.

209

.

138

.

115

.

591

.

238

.

052

.

023

.

011

.

109

.

065

.

479

.

169

.

037

.

031

.

070

±

±

±

±

±

±

±

±

±

±

±

±

±

±

±

±

±

±

adult

K-S

0

.

126

0

0

0

0

.

053

.

122

.

064

.

041

0

0

0

0

0

.

.

.

.

.

086

414

178

114

111

0

.

005

0

.

234

covertype

K-S

0

.

003

0

0

0

0

0

042

.

.

028

.

010

.

005

.

1

0

0

0

0

.

.

.

.

.

003

0

.

adult

K-S

0

.

008

0

0

0

0

0

.

048

.

018

.

004

.

004

.

002

0

5

0

0

0

0

.

.

.

.

.

.

629

606

180

073

036

379

463

228

745

248

239

457

1-WD

±

0

.

011

±

0

±

0

±

0

±

0

.

.

.

.

023

015

003

003

±

0

.

002

1-WD

±

0

.

024

±

0

±

0

±

0

±

0

.

.

.

.

116

018

006

003

±

0

.

004

1-WD

±

0

.

036

±

0

±

0

±

0

±

0

±

0

.

.

.

.

.

212

074

084

010

011

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

109

195

188

110

115

120

070

193

039

019

011

009

140

405

255

022

023

059

kings

K-S

±

0

±

.

012

0

±

0

±

0

±

0

±

0

.

018

.

035

.

008

.

007

.

0

.

116

0

.

213

0

.

128

0

.

089

0

.

002

093

0

.

loan

K-S

±

0

.

010

±

0

±

0

±

0

±

0

±

0

028

.

.

011

.

005

.

002

.

0

.

137

076

0

.

221

0

.

046

0

.

027

0

.

015

002

0

.

013

kings

K-S

±

0

.

018

±

0

±

0

±

0

±

0

±

0

.

034

.

027

.

004

.

003

.

002

0

.

529

1

.

262

0

.

979

0

.

071

0

.

073

0

.

204

1-WD

±

0

.

016

±

±

±

±

0

0

0

0

.

.

.

.

046

019

003

002

±

0

.

002

1-WD

±

0

.

013

±

±

±

±

0

0

0

0

.

.

.

.

043

013

008

005

±

0

.

005

1-WD

±

0

.

064

±

±

±

±

±

0

0

0

0

0

.

.

.

.

.

099

107

012

008

011

(a)

(b)

Peer reviews:
comments: SUMMARY:
In this paper, the authors present a novel nonparametric distributional learning method for VAE. They provide a comprehensive theoretical analysis of the proposed solution (DistVAE), as well as prove its superiority over CTGAN, TVAE, and CTAB-GAN when applied to synthetic data generation on real tabular datasets.

STRENGTHS:
1. The theory of the paper seems to be original and significant.

2. The experimental results obtained for DistVAE are superior to those of competitors.

3. The paper is well written.

WEAKNESSES:
1. The experimental setting is limited to tabular datasets and few synthesizers.

2. A more detailed discussion of limitations would be fruitful.

QUESTIONS:
1. The authors claim that their approach "extends model capacity [...] without sacrificing the computational advantages of the VAE framework". Perhaps the paper would benefit from an experimental comparison (in terms of capturing data distribution and computational efficiency) of the proposed model with different VAEs trained on classical image datasets. Why was this not supplied? My final opinion will depend on the authors' response to this question.

2. Minor comments:

l. 14: I believe the authors would like to (additionally) cite another original paper on VAE: Rezende et al., ICML 2014.

l. 104: '.' --> ':'.

l. 133: $q(x|z; \phi)dx$ --> $q(x|z; \phi)$.

l. 136: $q_j$ was already used to denote the number of levels for $x_j$.

l. 156: "hessian" --> "Hessian".

l. 160: "bounded above" --> "bounded from above".

LIMITATIONS:
A more detailed discussion of limitations would be fruitful.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 2
DATE: 2023-07-03

comments: SUMMARY:
This paper introduces a novel distributional learning method of VAE, which aims to effectively capture the underlying distribution of the observed dataset using a nonparametric approach. The proposed  decoder is composed of an infinite mixture of asymmetric Laplacian distribution, which possesses general distribution fitting capabilities for continuous variables.  The authors provide empirical results and demonstrate superiority in easily adjusting the level of data privacy.

STRENGTHS:
This work provides concrete theoretical analysis and empirical results with the proposed distributional decoder. The proposed decoder has more capacity to capture the data distribution than the standard Gaussian.

WEAKNESSES:
As [15] indicated, “the model training becomes unstable if the estimated variance of the decoder shrinks to zero in Gaussian VAE” is not theoretically correct.

I don't find major weakness in this work.

QUESTIONS:
Why $\alpha$ is assigned with a prior? I don’t get any intuition from your analysis.

LIMITATIONS:
Too many assumptions in the theoretical analysis which make the generality to real data suspicious.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-04

comments: SUMMARY:
In this paper the authors propose to define the decoder of a VAE using an infinite mixture of asymmetric Laplacian distribution, which results in distributional learning. Specifically, they design a non-parametric generative model in the VAEs. This class of VAEs estimates the conditional CDF while trained with ELBO objective as standard VAEs. They evaluated their approach on multiple datasets and show that their method outperforms baselines.

STRENGTHS:
This idea of increasing the model capacity of VAEs is well motivated, because Gaussian assumption has limited the applications of VAEs in the past. So I think this paper brings up some interesting topic and is novel as well.
In general this paper is clearly structured. The authors provide detailed derivations of the methodology. Also they perform comprehensive evaluations on multiple datasets.

WEAKNESSES:
My main confusion is about the 2 dimensional latent space in the evaluations. If I understand correctly, all the models are defined with 2D latent space. I wonder if this puts limits on the decoder capacity by itself, and I wonder how the model performs when we do higher latent dimensions in the experiments.

QUESTIONS:
Please see my questions in the previous section

LIMITATIONS:
I don't think there is any potential negative impact in their work.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-06