Title: value: Distributional Learning of Variational AutoEncoder: Application to Synthetic Data Generation
Authors: value: Seunghwan An
Jong-June Jeon
Venue: NeurIPS 2023
Keywords: value: Variational AutoEncoder
distributional learning
synthetic data generation
CRPS
asymmetric Laplace distribution

Abstract: value: The Gaussianity assumption has been consistently criticized as a main limitation of the Variational Autoencoder (VAE) despite its efficiency in computational modeling. In this paper, we propose a new approach that expands the model capacity (i.e., expressive power of distributional family) without sacrificing the computational advantages of the VAE framework. Our VAE model's decoder is composed of an infinite mixture of asymmetric Laplace distribution, which possesses general distribution fitting capabilities for continuous variables. Our model is represented by a special form of a nonparametric M-estimator for estimating general quantile functions, and we theoretically establish the relevance between the proposed model and quantile estimation. We apply the proposed model to synthetic data generation, and particularly, our model demonstrates superiority in easily adjusting the level of data privacy.


Review discussion:
1. The theory of the paper seems to be original and significant.
2. The experimental results obtained for DistVAE are superior to those of competitors.
3. The paper is well written.
1. The experimental setting is limited to tabular datasets and few synthesizers.
2. A more detailed discussion of limitations would be fruitful.
1. The authors claim that their approach "extends model capacity [...] without sacrificing the computational advantages of the VAE framework". Perhaps the paper would benefit from an experimental comparison (in terms of capturing data distribution and computational efficiency) of the proposed model with different VAEs trained on classical image datasets. Why was this not supplied? My final opinion will depend on the authors' response to this question.
2. Minor comments:
l. 14: I believe the authors would like to (additionally) cite another original paper on VAE: Rezende et al., ICML 2014.
l. 104: '.' --> ':'.
l. 133: $q(x|z; \phi)dx$ --> $q(x|z; \phi)$.
l. 136: $q_j$ was already used to denote the number of levels for $x_j$.
l. 156: "hessian" --> "Hessian".
l. 160: "bounded above" --> "bounded from above".
A more detailed discussion of limitations would be fruitful.
No ethics review needed.
> Q1 (Question). The authors claim that their approach "extends model capacity [...] without sacrificing the computational advantages of the VAE framework". Perhaps the paper would benefit from an experimental comparison (in terms of capturing data distribution and computational efficiency) of the proposed model with different VAEs trained on classical image datasets. Why was this not supplied?
We genuinely appreciate the reviewer's valuable feedback and thoughtful question regarding the types of datasets used in the evaluations. The performance of the generative model depends on both the dimension of the latent variables and the flexibility of the decoder model.
First, let me explain the relationship between the size of the latent space and the flexibility of the decoder model within the scope of our understanding, as well as its implications in image data generation.
In our paper, we made an assumption that, given the latent variables, the observed variables are conditionally independent, as represented by Equation (1):
$X_1 \perp X_2 \vert Z_1, \cdots, Z_{d-1} \Rightarrow X_1 \perp X_2 \vert Z_1, \cdots, Z_d. \mbox{\quad (1)}$
When the dimension of the latent variables is smaller, this conditional independence assumption may be violated. In such cases, the proposed non-parametric fitting of the decoder might not adequately represent the ground truth distribution. Particularly in image data, where significantly intricate dependency structures exist among pixel values, a low-dimensional latent space may not lead to substantial improvement in image data generation performance.
Therefore, to effectively compare how the flexibility of the decoder model affects the fit of the probability distribution, we chose tabular datasets for our evaluation, as it allows relatively easy control of probabilistic conditional independence.
However, it does not imply that our model is not inapplicable to image data. Representative classical image datasets like CIFAR-10 often exhibit pixel value distributions far from the Gaussian since the frequencies of edge values (0 and 255) dominate more than other pixel values [1, 2]. [3] also presents other image dataset examples where pixel value distributions exhibit multi-modality.
Therefore, we expect that our DistVAE, capable of capturing multi-modal underlying distributions, would be effective compared to other VAE models in the image domain.
Considering the aforementioned limitations, we found it more appropriate to compare the synthetic data generation performance between our DistVAE and other synthesizers using real tabular datasets.
However, in Section 5 Conclusion and Limitations, we will actively discuss the limitations of our proposed model in the image domain.
> Q2 (Limitation). A more detailed discussion of limitations would be fruitful.
We sincerely appreciate your valuable feedback. We agree that providing a more comprehensive discussion of limitations will enhance the overall quality of our paper. In response to the previous question, we intend to include the following content in Section 5 Conclusion and Limitations:
*In our study, we have confirmed that the proposed decoder enhances the performance of generative models for tabular data.
However, this conclusion relies on the assumption of conditional independence among the observed variables given the latent variable.
When the dimension of the latent variable is small, this assumption is prone to violation.
Therefore, in cases where the size of the latent space is limited, the proposed non-parametric fitting of the decoder might not accurately represent the underlying distribution. Particularly in the image domain, where adjacent pixel values exhibit significant dependence, it remains uncertain whether our proposed model would lead to notable improvements in image data generation with a low-dimensional latent space.*
*Nevertheless, classical image datasets, such as CIFAR-10, often exhibit pixel value distributions that deviate considerably from Gaussian, with the frequencies of edge values (0 and 255) dominating more than other pixel values [1, 2].
Additionally, other image datasets, as presented in [3], demonstrate multi-modality in pixel value distributions. These experimental findings suggest that leveraging the capacity of distributional learning could be advantageous in approximating the ground truth distribution of image data when the latent variable effectively captures conditional independence among the observed variables.
Consequently, we expect that compromising biases arising from the violation of conditional independence and marginally misspecified distributions may further enhance our results, and we leave it for our future research.*
> Q3 (Question). Minor comments.
We appreciate the reviewer's attention to detail and valuable feedback. We have thoroughly addressed all minor comments and made the necessary corrections.
Specifically, we rectified the typos on lines 104, 133, 156, and 160.
Furthermore, we have included the citation of the original VAE paper [4], and we now denote the number of levels for $x_j, j \in I_D$ as $T_j$ instead of $q_j$.
- References:
[1] Darlow, L.N., Crowley, E.J., Antoniou, A., & Storkey, A.J. (2018). CINIC-10 is not ImageNet or CIFAR-10. ArXiv, abs/1810.03505.
[2] Salimans, T., Karpathy, A., Chen, X., & Kingma, D. P. (2017). Pixelcnn++: Improving the pixelcnn with discretized logistic mixture likelihood and other modifications. arXiv preprint arXiv:1701.05517.
[3] Parmar, V., Penkovsky, B., Querlioz, D., & Suri, M. (2022). Hardware-efficient stochastic binary CNN architectures for near-sensor computing. Frontiers in Neuroscience, 15, 781786.
[4] Rezende, D. J., Mohamed, S., & Wierstra, D. (2014, June). Stochastic backpropagation and approximate inference in deep generative models. In International conference on machine learning (pp. 1278-1286). PMLR.
I am satisfied with the authors' rebuttal. I have raised my rating accordingly.
This work provides concrete theoretical analysis and empirical results with the proposed distributional decoder. The proposed decoder has more capacity to capture the data distribution than the standard Gaussian.
As [15] indicated, “the model training becomes unstable if the estimated variance of the decoder shrinks to zero in Gaussian VAE” is not theoretically correct.
I don't find major weakness in this work.
Why $\alpha$ is assigned with a prior? I don’t get any intuition from your analysis.
Too many assumptions in the theoretical analysis which make the generality to real data suspicious.
No ethics review needed.
> Q1 (Weakness). As [15] indicated, “the model training becomes unstable if the estimated variance of the decoder shrinks to zero in Gaussian VAE” is not theoretically correct.
Thank you for bringing up the issue regarding the citation of related papers in our work. We sincerely appreciate your valuable feedback and acknowledge that we should have been more careful in citing and describing the relevant papers in Section 2 Related Work. We understand that [1, 2] have empirically demonstrated the instability of training with moderate sample size, while [5] has provided theoretical evidence that the ground truth data distribution can be estimated with infinite samples, even if the variance of the decoder is trainable. To address this concern, we will revise lines 60-62 in Section 2 Related Work to ensure that the content accurately reflects the contributions of each paper:
- Old version:
*[1, 2] assume their decoder distributions as Student-t and asymmetric Laplace distributions, respectively, to mitigate the zero-variance problem that the model training becomes unstable if the estimated variance of the decoder shrinks to zero in Gaussian VAE.*
- Revised version:
*[1, 2] address the zero-variance problem in Gaussian VAE when the estimated variance of the decoder becomes too small, leading to unstable model training with a moderate sample size. To mitigate this problem, [1] adopts the Student-t distribution as the decoder distribution, while [2] uses the asymmetric Laplace distribution.
However, theoretical evidence presented in [3] demonstrates that the ground-truth data distribution can be accurately estimated with an infinite number of samples, even if the variance of the decoder is trainable.*
> Q2 (Question). Why $\alpha$ is assigned with a prior? I don’t get any intuition from your analysis.
We sincerely appreciate your valuable question regarding the relationship between $\alpha$ and the prior distribution. As you pointed out, our reconstruction loss can be defined without the prior distribution of $\alpha$.
But we think the quantile loss provides a natural explanation of the reconstruction loss as an extension of the estimation for a single quantile. In our paper, we established the generative model by introducing an infinite mixture of asymmetric Laplace distribution (as mentioned in line 99). This entailed adopting a Bayesian perspective for $\alpha$ as a prior (this is the Bayesian modeling for estimation of multiple quantiles), and the infinite mixture model served as a theoretical link between a proper scoring rule and our proposed reconstruction loss in the objective function (please see Equation (2) in our paper).
Throughout the derivation of the reconstruction loss, the role of $\alpha$ was pivotal in ensuring the representation of the reconstruction loss.
Moreover, it was of utmost importance to prevent the observation $x$ from influencing the distribution of $\alpha$;
thus, the resulting reconstruction loss becomes the CRPS loss (a proper scoring rule). This meticulous design provided the integrity of our approach and its adherence to the desired properties.
Additionally, we want to emphasize that by changing the prior distribution of $\alpha$, we can derive more general models than the proposed one.
> Q3 (Limitation).
Too many assumptions in the theoretical analysis which make the generality to real data suspicious.
We are sincerely grateful for your insightful comments regarding our theoretical analysis. Our assumptions in the theoretical analysis can be categorized into two main aspects: 1) assumptions about the data and 2) assumptions about the model.
Regarding data assumptions, our focus on continuous random variables in the observation $x$ was intended to facilitate the theoretical analysis of our crucial contribution, which is incorporating the CRPS loss into the ELBO of the VAE framework.
However, we expect that even datasets containing discrete variables can benefit from the same theoretical development by adopting the proper scoring rule introduced in [4].
Furthermore, although we assumed that the support of continuous random variables spans the entire real space, we believe this assumption is mild and can potentially be relaxed through additional constraints on the model.
Additionally, a critical data assumption we made involves the conditional independence of $x_1, \cdots, x_p$ given $z$ in the definition of $q(x|z;\phi)$. It is possible to test this assumption in a trained model [5, 6], and such verification would serve as validation of the effectiveness of our proposed model.
Regarding our model assumptions, particularly Assumption 1, we would like to emphasize that they primarily pertain to the model's complexity, computational cost, and asymptotic approximation performance rather than directly affecting its applicability to real-world datasets.
Therefore, we firmly believe that the assumptions made in the theoretical analysis do not significantly impede the model's generalization to real data.
- References:
[1] Takahashi, H., Iwata, T., Yamanaka, Y., Yamada, M., & Yagi, S. (2018). Student-t Variational Autoencoder for Robust Density Estimation. International Joint Conference on Artificial Intelligence.
[2] Akrami, H., Joshi, A.A., Aydöre, S., & Leahy, R.M. (2020). Addressing Variance Shrinkage in Variational Autoencoders using Quantile Regression. ArXiv, abs/2010.09042.
[3] Dai, B., & Wipf, D. (2019). Diagnosing and enhancing VAE models. arXiv preprint arXiv:1903.0578.
[4] Carvalho, A. (2016). An Overview of Applications of Proper Scoring Rules. Decis. Anal., 13, 223-242.
[5] Zhang, K., Peters, J., Janzing, D., & Schölkopf, B. (2012). Kernel-based conditional independence test and application in causal discovery. arXiv preprint arXiv:1202.3775.
[6] Li, C., & Fan, X. (2020). On nonparametric conditional independence tests for continuous variables. Wiley Interdisciplinary Reviews: Computational Statistics, 12(3), e1489.
This idea of increasing the model capacity of VAEs is well motivated, because Gaussian assumption has limited the applications of VAEs in the past.
So I think this paper brings up some interesting topic and is novel as well.
In general this paper is clearly structured. The authors provide detailed derivations of the methodology.
Also they perform comprehensive evaluations on multiple datasets.
My main confusion is about the 2 dimensional latent space in the evaluations. If I understand correctly, all the models are defined with 2D latent space. I wonder if this puts limits on the decoder capacity by itself, and I wonder how the model performs when we do higher latent dimensions in the experiments.
Please see my questions in the previous section
I don't think there is any potential negative impact in their work.
No ethics review needed.
> Q1 (Question). My main confusion is about the 2 dimensional latent space in the evaluations. If I understand correctly, all the models are defined with 2D latent space. I wonder if this puts limits on the decoder capacity by itself, and I wonder how the model performs when we do higher latent dimensions in the experiments.
We sincerely appreciate your valuable feedback regarding the dimension of the latent space in our evaluation processes. As you pointed out, the chosen latent space indeed limits the capacity of decoders for all models.
However, our primary goal was maintaining a small and consistent number of parameters across all models during the experiment. This deliberate choice allowed us to isolate the performance differences in synthetic data generation to the methodologies of each synthesizer, specifically emphasizing the contribution of the decoder model's flexibility in estimating underlying distributions.
Additionally, we anticipate that using a low-dimensional latent space can enhance the explainability of the generative models, and having a smaller number of parameters is advantageous for the model's utility. This decision was crucial as the size of the model directly influences the quality of synthetic data generation.
Accordingly, we carefully designed a simulation setting, ensuring that the latent dimension, a decisive factor in determining the model size, consistently remained low for all models. A comprehensive comparison of the model parameters can be found in Appendix A.9 Table 10.
Furthermore, our proposed model's performance, as stated in Theorem 2 and 3, is heavily reliant on the accuracy of the conditional quantile function $D$ in approximating the ground truth CDF (cumulative distribution function). If we significantly increase the capacity of the decoder or utilize a higher-dimensional latent space, the neural network parameterized $D$, taking the latent variable as input, would achieve sufficient complexity to provide a better approximation of the ground truth CDF. Given that we have already demonstrated the superiority of our DistVAE in low-dimensional latent spaces with a similar number of model parameters in the evaluations, we firmly believe that even with a substantial increase in decoder capacity or higher-dimensional latent space, our DistVAE will continue to outperform other models in synthetic data generation.

Meta-review:
role: PC
data: comment: value: This paper proposes a novel non-parametric approach applied to the decoder of VAE, and showed better performance than the baselines.

Reviewers originally reported 

- Novel and interesting idea
- Well-written
- Comprehensive evaluations on multiple datasets

as pros and 

- Experimental setting is limited to tabular datasets and synthesized datasets

and some minor issues as cons.  The cons have been well addressed in the rebuttal, and the reviewers agree to accept the paper.

Decision: Accept (poster)