SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Seunghwan An and Jong-June Jeon*
Department of Statistical Data Science, University of Seoul, S. Korea
{dkstmdghks79, jj.jeon}@uos.ac.kr
The Gaussianity assumption has been consistently criticized as a main limitation of the Variational Autoencoder (VAE) despite its efficiency in computational modeling. In this paper, we propose a new approach that expands the model capacity (i.e., expressive power of distributional family) without sacrificing the computational advantages of the VAE framework. Our VAE model's decoder is composed of an infinite mixture of asymmetric Laplace distribution, which possesses general distribution fitting capabilities for continuous variables. Our model is represented by a special form of a nonparametric M-estimator for estimating general quantile functions, and we theoretically establish the relevance between the proposed model and quantile estimation. We apply the proposed model to synthetic data generation, and particularly, our model demonstrates superiority in easily adjusting the level of data privacy.
Variational Autoencoder (VAE) [31, 51] and Generative Adversarial Networks (GAN) [22] are generative models that are used to estimate the underlying distribution of a given dataset. To avoid the curse of dimensionality, VAE and GAN commonly introduce a low-dimensional latent space on which a conditional generative model is defined. By minimizing an information divergence between the original data and its generated data, the generative models are learned to produce synthetic data similar to the original one. Accordingly, VAE and GAN have been applied in various applications, such as generating realistic images, texts, and synthetic tabular data for privacy preservation purposes [30, 60, 63, 65, 34].
However, the difference in the strength of the assumption about the generative distribution brings significant contrasts in the VAE and GAN generation performances [30, 7, 17]. In the GAN framework, the adversarial loss enables direct minimization of the Jensen-Shannon divergence between the ground-truth density function and the generative distribution under no distributional assumption [13, 59]. Roughly speaking, the GAN employs a nonparametric model as its conditional generative model defined on the latent space.
On the contrary, in the VAE framework, the Gaussianity assumption has been favored [31, 32, 15, 10, 39]. It is because Gaussianity gives us three advantages: 1) the reconstruction loss can be interpreted as the mean squared error that is one of the most popular losses in optimization theory, 2) generating a new sample is computationally straightforward, and 3) KL-divergence is computed in a simple closed form. However, these benefits have led us to pay the price for the distributional capacity of the generative model, in that the generative model of the VAE is constrained in the form of marginalization of the product of the two Gaussian distributions. Here, the distributional capacity means the expressive power of the distributional family. This restricted distributional capacity has
been the critical limitation [8, 33] and leads to a heavy parameterization of the decoder mean vector to approximate complex underlying distributions.
To increase the distributional capacity in synthetic data generation, [63, 65] introduce the multimodality in the distributional assumption of the decoder, which is known as the mode-specific normalization technique. Although the mixture Gaussian decoder modeling of [63, 65] allows handling more complex distributions of the observed dataset while preserving all of the advantages of Gaussianity, we numerically find that the mixture Gaussian is not enough to capture the complex underlying distribution.
Our main contribution is that, beyond Gaussianity, we propose a novel VAE learning method that directly estimates the conditional cumulative distribution function (CDF) while maintaining the objective of maximizing the Evidence Lower Bound (ELBO) of the observed dataset. It implies that we have a nonparametric distribution assumption on the generative model. We call this approach distributional learning of the VAE, which is enabled by estimating an infinite number of conditional quantiles [4, 20]. By adopting the continuous ranked probability score (CRPS) loss, the objective function of our proposed distribution learning method is computationally tractable [21, 43, 20].
In our proposed distributional learning framework, 1) the reconstruction loss is equivalent to the CRPS loss, which is a proper scoring rule [21, 43], 2) generating a new sample is still computationally straightforward due to the inverse transform sampling, and 3) KL-divergence is still computed in a simple closed form. To show the effectiveness of our proposed model in capturing the underlying distribution of the dataset, we evaluate our model for synthetic data generation with real tabular datasets.
Modeling of the decoder and reconstruction loss. To increase the distributional capacity, many papers have focused on decoder modeling while not losing the mathematical link to maximize the ELBO. [57, 1] assume their decoder distributions as Student- $t$  and asymmetric Laplace distributions, respectively, to mitigate the zero-variance problem that the model training becomes unstable if the estimated variance of the decoder shrinks to zero in Gaussian VAE [41, 55, 15]. [2] proposes a general distribution of the decoder, which allows improved robustness by optimizing the shape of the loss function during training. Recently, [5] proposes a reconstruction loss that directly minimizes the blur error of the VAE by modeling the covariance matrix of multivariate Gaussian decoder.
On the other hand, there exists a research direction that focuses on replacing the reconstruction loss without concern for losing the mathematical derivation of the lower bound. [38, 52, 46] replace the reconstruction loss with an adversarial loss of the GAN framework. [27] introduces a feature-based loss that is calculated with a pre-trained convolutional neural network (CNN). Another approach by [14] adopts Watson's perceptual model, and [28] directly optimizes the generative model in the frequency domain by a focal frequency reconstruction loss. Most of the above-mentioned methods aim to capture the properties of human perception by replacing the element-wise loss ( $L_{1}$  or  $L_{2}$ -norm), which hinders the reconstruction of images [38].
Synthetic data generation. The GAN framework is widely adopted in the synthetic data generation task since it enables synthetic data generation in a nonparametric approach [12, 47, 63, 65]. [63, 65] assume that continuous columns of tabular datasets can be approximated by the Gaussian mixture distribution and model their decoder using Gaussian mixture distribution. Additionally, [63, 65] preprocess the continuous variables using the variational Gaussian mixture model [3], which is known as the mode-specific normalization technique. However, the preprocessing step requires additional computational resources and hyperparameter tuning of the number of modes. Other approaches by [47, 65] regularize the discrepancy between the first and second-order statistics of the observed and synthetic dataset. [12] proposes the GAN-based synthesizer, which focuses on generating high-dimensional discrete variables with the assistance of the pre-trained AutoEncoder.
3 Proposal
Let  $\mathbf{x} \in \mathbb{R}^{p + q}$  be an observation consisting of continuous and discrete variables and  $I = I_{C} \cup I_{D} = \{1, \dots, (p + q)\}$  be an index set of the variables, where  $I_{C}$  and  $I_{D}$  correspond to index sets of
$p$  continuous and  $q$  discrete variables.  $T_{j}$  denotes the number of levels for the discrete variables  $\mathbf{x}_j, j \in I_D$ . We denote the ground-truth underlying distribution (probability density function, PDF) as  $p(\mathbf{x})$  and the ground-truth CDF as  $F(\mathbf{x})$ .
Let  $\mathbf{z}$  be a latent variable, where  $\mathbf{z} \in \mathbb{R}^d$  and  $d < p + q$ . The prior and posterior distribution of  $\mathbf{z}$  are assumed to be  $p(\mathbf{z}) = \mathcal{N}(\mathbf{z}|\mathbf{0},\mathbf{I})$  and  $q(\mathbf{z}|\mathbf{x};\phi) = \mathcal{N}\big(\mathbf{z}|\mu (\mathbf{x};\phi),diag(\sigma^2 (\mathbf{x};\phi))\big)$ , respectively. Here,  $\mathbf{I}$  is  $d \times d$  identity matrix,  $\mu :\mathbb{R}^{p + q}\mapsto \mathbb{R}^d$ ,  $\sigma^2:\mathbb{R}^{p + q}\mapsto \mathbb{R}_+^d$  are neural networks parameterized with  $\phi$ , and  $diag(a)$ ,  $a \in \mathbb{R}^d$  denotes a diagonal matrix with diagonal elements  $a$ . Moreover, we consider  $\alpha \in [0,1]$  as a random variable having density  $p(\alpha)$ .
3.1 Distributional Learning
Our proposed model assumes that  $p(\mathbf{x})$  is parametrized by an infinite mixture of asymmetric Laplace distribution (ALD) [4]. The ALD is characterized by two parameters:  $\alpha$ , representing the asymmetry, and  $\beta > 0$ , representing the scale. By considering these parameters, along with the model parameter  $\theta$ , we can define the probability model of  $\mathbf{x}$  as follows:
$$
p (\mathbf {x}; \theta , \beta) = \int \int_ {0} ^ {1} p (\mathbf {x} | \mathbf {z}, \alpha ; \theta , \beta) p (\mathbf {z}, \alpha) d \alpha d \mathbf {z}.
$$
Assumption 1. (1)  $\{\mathbf{x}_j\}_{j\in I}$  are conditionally independent given  $\mathbf{z}$ . (2) The discrete random variables  $\{\mathbf{x}_j\}_{j\in I_D}$  are independent of  $\alpha$ . (3)  $\alpha$  and  $\mathbf{z}$  are independent.
By Assumption 1-(1), we model the dependency between  $\mathbf{x}_j$ s solely through the latent variable  $\mathbf{z}$  [49]. Assumption 1-(2) implies that  $\alpha$  is related only to the continuous variables. Then, the decoder of our VAE model denoted as  $p(\mathbf{x}|\mathbf{z},\alpha ;\theta ,\beta)$  is specified by equation (1):
$$
\begin{array}{l} p (\mathbf {x} | \mathbf {z}, \alpha ; \theta , \beta) = \prod_ {j \in I _ {C}} p \left(\mathbf {x} _ {j} | \mathbf {z}, \alpha ; \theta_ {j}, \beta\right) \cdot \prod_ {j \in I _ {D}} p \left(\mathbf {x} _ {j} | \mathbf {z}; \theta_ {j}\right) \tag {1} \\ = \prod_ {j \in I _ {C}} \frac {\alpha (1 - \alpha)}{\beta} \exp \left(- \rho_ {\alpha} \left(\frac {\mathbf {x} _ {j} - D _ {j} (\alpha , \mathbf {z} ; \theta_ {j})}{\beta}\right)\right) \cdot \prod_ {j \in I _ {D}} \prod_ {l = 1} ^ {T _ {j}} \pi_ {l} (\mathbf {z}; \theta_ {j}) ^ {\mathbb {I} (\mathbf {x} _ {j} = l)}, \\ \end{array}
$$
where  $\theta = (\theta_{1},\dots ,\theta_{p + q}),\beta$  is a non-trainable constant,  $\rho_v(u) = u(v - \mathbb{I}(u < 0))$  (check function), and  $\mathbb{I}(\cdot)$  denotes the indicator function.  $D_{j}(\cdot ,\cdot ;\theta_{j}):[0,1]\times \mathbb{R}^{d}\mapsto \mathbb{R}$  is the location parameter of ALD, which is parameterized with  $\theta_{j}$  [1]. For discrete variables,  $\pi (\cdot ;\theta_j):\mathbb{R}^d\mapsto \Delta^{T_j - 1}$  is a neural network parameterized with  $\theta_{j}$ , where  $\Delta^{T_j - 1}$  is the standard  $(T_{j} - 1)$ -simplex for all  $\mathbf{z}\in \mathbb{R}^d$ , and the subscript  $l$  refers to the  $l$ th element of the output  $\pi$ .
Assumption 1-(3) leads our objective function,
$$
\begin{array}{l} \min  _ {\theta , \phi} \qquad \mathbb {E} _ {p (\mathbf {x})} \mathbb {E} _ {q (\mathbf {z} | \mathbf {x}; \phi)} \left[ \sum_ {j \in I _ {C}} \int_ {0} ^ {1} \rho_ {\alpha} \Big (\mathbf {x} _ {j} - D _ {j} (\alpha , \mathbf {z}; \theta_ {j}) \Big) d \alpha - \sum_ {j \in I _ {D}} \sum_ {l = 1} ^ {T _ {j}} \mathbb {I} (\mathbf {x} _ {j} = l) \cdot \log \pi_ {l} (\mathbf {z}; \theta_ {j}) \right] \\ + \quad \beta \cdot \mathbb {E} _ {p (\mathbf {x})} [ \mathcal {K L} (q (\mathbf {z} | \mathbf {x}; \phi) \| p (\mathbf {z})) ], \tag {2} \\ \end{array}
$$
where constant terms are omitted. In order to achieve balanced learning of the two reconstruction losses in (2), we have removed the weight  $\beta$  associated with the second reconstruction loss. We refer to our model as 'DistVAE.' The first term in (2) corresponds to the CRPS loss, which measures the accuracy of the proposed CDF approximation with respect to the ground-truth CDF of the underlying distribution [21, 43, 20].
Interestingly, (2) is the limit of the negative ELBO derived from a finite mixture of ALD. We introduce  $\alpha$ , a discrete uniform random variable taking values on  $\alpha_{k} = k / K$  for  $k = 1,\dots ,K$ . Then, the negative ELBO of  $p(\mathbf{x};\theta ,\beta)$  scaled with  $\beta$  is written by
$$
\begin{array}{l} \mathbb {E} _ {q (\mathbf {z} | \mathbf {x}; \phi)} \left[ \sum_ {j \in I _ {C}} \frac {1}{K} \sum_ {k = 1} ^ {K} \rho_ {\alpha_ {k}} \left(\mathbf {x} _ {j} - D _ {j} \left(\alpha_ {k}, \mathbf {z}; \theta_ {j}\right)\right) \right] - \beta \frac {p}{K} \sum_ {k = 1} ^ {K} \log \alpha_ {k} (1 - \alpha_ {k}) + \beta p \log \beta \\ - \quad \beta \cdot \mathbb {E} _ {q (\mathbf {z} | \mathbf {x}; \phi)} \left[ \sum_ {j \in I _ {D}} \sum_ {l = 1} ^ {T _ {j}} \mathbb {I} (\mathbf {x} _ {j} = l) \cdot \log \pi_ {l} (\mathbf {z}; \theta_ {j}) \right] + \beta \cdot \mathcal {K L} (q (\mathbf {z} | \mathbf {x}; \phi) \| p (\mathbf {z})) \tag {3} \\ \end{array}
$$
(see Appendix A.1 for detailed derivation). The reconstruction loss, which corresponds to the first term of (3), is a composite quantile loss for estimating the target quantiles  $\alpha_{k}$  s [64, 35, 45, 62, 9]. This entails adopting a Bayesian perspective for  $\alpha$  as a prior (the Bayesian modeling for estimating multiple quantiles). Furthermore, throughout the derivation of the reconstruction loss, the role of  $\alpha$  is pivotal in ensuring the representation of the reconstruction loss. To prevent the observation  $\mathbf{x}$  from influencing the distribution of  $\alpha$ ,  $\alpha$  is only assigned with a prior distribution, and the resulting reconstruction loss becomes the CRPS loss, a proper scoring rule.
However, for distributional learning of VAE, it is necessary to estimate conditional quantiles for an infinite number of quantile levels, denoted by  $K \to \infty$  [50]. The subsequent Theorem 1 establishes the convergence of the negative ELBO (3) to our objective function (2) as  $K \to \infty$  [4].
Theorem 1. For all  $j \in I_C$ , suppose that  $\int_0^1 \mathbb{E}_{p(\mathbf{x})} \mathbb{E}_{q(\mathbf{z}|\mathbf{x};\phi)} \rho_\alpha (\mathbf{x}_j - D_j(\alpha, \mathbf{z}; \theta_j)) d\alpha < \infty$ , and  $\mathbb{E}_{p(\mathbf{x})} \mathbb{E}_{q(\mathbf{z}|\mathbf{x};\phi)} [\rho_\alpha (\mathbf{x}_j - D_j(\alpha, \mathbf{z}; \theta_j))]$  is continuous over  $\alpha \in (0,1)$ . Then,
$$
\lim _ {K \to \infty} \mathbb {E} _ {p (\mathbf {x})} \mathbb {E} _ {q (\mathbf {z} | \mathbf {x}; \phi)} \left[ \frac {1}{K} \sum_ {k = 1} ^ {K} \rho_ {\alpha_ {k}} \Big (\mathbf {x} _ {j} - D _ {j} (\alpha_ {k}, \mathbf {z}; \theta_ {j}) \Big) \right] = \mathbb {E} _ {p (\mathbf {x})} \mathbb {E} _ {q (\mathbf {z} | \mathbf {x}; \phi)} \left[ \int_ {0} ^ {1} \rho_ {\alpha} \Big (\mathbf {x} _ {j} - D _ {j} (\alpha , \mathbf {z}; \theta_ {j}) \Big) d \alpha \right],
$$
and  $\lim_{K\to \infty}\frac{1}{K}\sum_{k = 1}^{K}\log \alpha_k(1 - \alpha_k) = \int_0^1\log \alpha (1 - \alpha)d\alpha = -2.$
3.2 Theoretical Results
In this section, we aim to provide theoretical insights into the ability of DistVAE, utilizing the objective function (2), to recover the ground-truth distribution  $p(\mathbf{x})$ . To simplify the analysis without loss of generality, we consider the scenario where  $\mathbf{x}$  comprises only  $p$  continuous random variables. Hence, we have  $I = I_{C} = \{1,\dots ,p\}$ , and  $p(\mathbf{x})$  is defined over  $\mathbf{x}\in \mathbb{R}^p$  with  $p(\mathbf{x}) > 0$  for all  $\mathbf{x}\in \mathbb{R}^p$ . First, define a function  $q(\mathbf{x}|\mathbf{z};\phi)$  by
$$
q (\mathbf {x} | \mathbf {z}; \phi) := \frac {p (\mathbf {x}) q (\mathbf {z} | \mathbf {x} ; \phi)}{q (\mathbf {z} ; \phi)},
$$
where  $q(\mathbf{z};\phi) \coloneqq \int q(\mathbf{z}|\mathbf{x};\phi)p(\mathbf{x})d\mathbf{x}$  is the aggregated posterior [58]. Clearly,  $q(\mathbf{x}|\mathbf{z};\phi)$  is a PDF of  $\mathbf{x}$  for a given  $\mathbf{z}$ .  $q(\mathbf{x}|\mathbf{z};\phi)$  is a conditional PDF of  $\mathbf{x}$  parametrized by  $\phi$  and it is an approximated PDF of  $\int_0^1 p(\mathbf{x}|\mathbf{z},\alpha ;\theta ,\beta)d\alpha$ . Since we assume that  $\mathbf{x}_j$ s are conditionally independent in Assumption 1,  $q(\mathbf{x}|\mathbf{z};\phi) = \prod_{j = 1}^{p}q_{j}(\mathbf{x}_{j}|\mathbf{z};\phi)$  and the conditional CDF is written as
$$
F (\mathbf {x} | \mathbf {z}; \phi) = \prod_ {j = 1} ^ {p} F _ {j} \left(\mathbf {x} _ {j} | \mathbf {z}; \phi\right), \text {w h e r e} F _ {j} \left(\mathbf {x} _ {j} | \mathbf {z}; \phi\right) := \int_ {- \infty} ^ {\mathbf {x} _ {j}} q _ {j} (x | \mathbf {z}; \phi) d x. \tag {4}
$$
For notational simplicity, we let
$$
\theta^ {*} (\phi) \in \arg \min _ {\theta} \mathbb {E} _ {p (\mathbf {x}) q (\mathbf {z} | \mathbf {x}; \phi)} \sum_ {j = 1} ^ {p} \int_ {0} ^ {1} \rho_ {\alpha} (\mathbf {x} _ {j} - D _ {j} (\alpha , \mathbf {z}; \theta_ {j})) d \alpha ,
$$
where  $\theta^{*}(\phi) = (\theta_{1}^{*}(\phi),\dots ,\theta_{p}^{*}(\phi))$
Assumption 2. (1) Given an arbitrary  $\phi$ ,  $F_{j}(\cdot|\mathbf{z};\phi): \mathbb{R} \mapsto [0,1]$  is absolutely continuous and strictly monotone increasing for all  $j = 1,\dots,p$ , and  $\mathbf{z} \in \mathbb{R}^{d}$ . (2) Given an arbitrary  $\theta$ ,  $D_{j}(\cdot,\mathbf{z};\theta_{j})$  is invertible and differentiable for all  $j = 1,\dots,p$  and  $\mathbf{z} \in \mathbb{R}^{d}$ . (3) The aggregated posterior  $q(\cdot;\phi)$  is absolutely continuous to the prior distribution of  $\mathbf{z}$ .
Theorem 2. Under Assumption 2, for an arbitrary  $\phi$
$$
\mathcal {K L} \left( \right.p (\mathbf {x}) \left\| \right. \int \prod_ {j = 1} ^ {p} \frac {d}{d \mathbf {x} _ {j}} D _ {j} ^ {- 1} \left(\mathbf {x} _ {j}, \mathbf {z}; \theta_ {j} ^ {*} (\phi)\right) q (\mathbf {z}; \phi) d \mathbf {z}\left. \right) = 0.
$$
Theorem 2 shows that DistVAE is capable of recovering the ground-truth distribution  $p(\mathbf{x})$ , indicating its ability to facilitate distributional learning rather than data reconstruction. Nevertheless, relying on the aggregated posterior distribution may lead to overfitting [25, 42], and sampling from the
aggregated posterior can introduce computational challenges due to the absence of a straightforward closed-form representation for  $q(\mathbf{z};\phi)$ . To address these concerns, we propose an alternative approach that leverages the prior distribution  $p(\mathbf{z})$  instead of  $q(\mathbf{z};\phi)$ , thereby enabling a computationally efficient synthetic generation process. This is substantiated by Theorem 3.
We define the estimated PDF  $\hat{p} (\mathbf{x};\theta^{*}(\phi))$  and CDF  $\hat{F} (\mathbf{x};\theta^{*}(\phi))$  as
$$
\hat {p} (\mathbf {x}; \theta^ {*} (\phi)) := \int \prod_ {j = 1} ^ {p} \frac {d}{d \mathbf {x} _ {j}} D _ {j} ^ {- 1} \left(\mathbf {x} _ {j}, \mathbf {z}; \theta_ {j} ^ {*} (\phi)\right) p (\mathbf {z}) d \mathbf {z} \tag {5}
$$
$$
\hat {F} (\mathbf {x}; \theta^ {*} (\phi)) := \int \prod_ {j = 1} ^ {p} D _ {j} ^ {- 1} \left(\mathbf {x} _ {j}, \mathbf {z}; \theta_ {j} ^ {*} (\phi)\right) p (\mathbf {z}) d \mathbf {z}. \tag {6}
$$
Theorem 3. Suppose that  $\phi$  is given such that  $\mathcal{KL}(q(\mathbf{z};\phi)\| p(\mathbf{z})) < \epsilon$  for any  $\epsilon >0$ . Then, under Assumption 2,
$$
\mathcal {K L} \left( \right.p (\mathbf {x}) \left\| \right. \int \prod_ {j = 1} ^ {p} \frac {d}{d \mathbf {x} _ {j}} D _ {j} ^ {- 1} \left(\mathbf {x} _ {j}, \mathbf {z}; \theta_ {j} ^ {*} (\phi)\right) p (\mathbf {z}) d \mathbf {z}\left. \right) <   \epsilon .
$$
Theorem 3 shows that even if we use the prior distribution  $p(\mathbf{z})$  instead of the aggregated posterior  $q(\mathbf{z};\phi)$ , it is feasible to minimize the KL-divergence between the ground-truth PDF  $p(\mathbf{x})$  and our estimated PDF  $\hat{p} (\mathbf{x};\theta^{*}(\phi))$  of (5). This is achievable because the KL-divergence term in (2) is the upper bound of  $\mathcal{KL}(q(\mathbf{z};\phi)\| p(\mathbf{z}))$  and is minimized during the training process. Since each conditional distribution of the estimated PDF depends on the same latent variable, it can be seen that the correlation structure between covariates is modeled implicitly.
[24, 41, 37] have highlighted the role of the KL-divergence weight parameter  $\beta$  in controlling the precision of reconstruction. In our case, where the reconstruction loss is based on the CRPS loss, an increase in  $\beta$  leads to a less accurate estimation of the ground-truth CDF. It implies that a large  $\beta$  corresponds to lower-quality synthetic data, but it also enhances privacy level. Thus,  $\beta$  introduces a trade-off between the quality of synthetic data and the risk of privacy leakage. The privacy level can be adjusted by manipulating  $\beta$  [47], as demonstrated in the experimental results presented in Section 4.
3.2.1 Synthetic Data Generation
By estimating the conditional quantile functions, we can transform the synthetic data generation process into inverse transform sampling. This conversion offers a notable advantage as it provides a straightforward and efficient approach to generating synthetic data. We denote a synthetic sample of  $\mathbf{x}_j$  as  $\hat{x}_j$  for  $j\in I$ , and the synthetic data generation process can be summarized as follows:
1. Sampling from the prior distribution:  $z\sim p(\mathbf{z})$
2. Inverse transform sampling: For  $j \in I_C$ ,  $\hat{x}_j = D_j(u_j | z; \theta_j)$ , where  $u_j \sim U(0, 1)$ .
3. Gumbel-Max trick [23]: For  $j \in I_D$ ,  $\hat{x}_j = \arg \max_{l=1,\dots,T_j} \{\log \pi_l(z; \theta_j) + G_l\}$ , where  $G_l \sim G_{\text{Gumbel}}(0,1)$ , and  $l = 1,\dots,T_j$ .
Note that both continuous and discrete variables share the same latent variable  $z$ . This shared latent variable allows for capturing the dependencies between variables. We numerically observe that the sampling, implemented using the Gumbel-Max trick, maintains the imbalanced ratio of the labels in the discrete variable.
3.2.2 Parameterization of ALD
As introduced in [20], for  $j \in I_C$ , we parameterize the function  $D_{j}$ , the location parameter of ALD, by a linear isotonic spline as follows:
$$
D _ {j} (\alpha , \mathbf {z}; \theta_ {j}) = \gamma^ {(j)} (\mathbf {z}) + \sum_ {m = 0} ^ {M} b _ {m} ^ {(j)} (\mathbf {z}) \left(\alpha - d _ {m}\right) _ {+} \quad \text {s . t .} \quad \sum_ {m = 0} ^ {k} b _ {m} ^ {(j)} (\mathbf {z}) \geq 0, k = 1, \dots , M, \tag {7}
$$
where  $\gamma^{(j)}(\mathbf{z})\in \mathbb{R},b^{(j)}(\mathbf{z}) = (b_0^{(j)}(\mathbf{z}),\dots ,b_M^{(j)}(\mathbf{z}))\in \mathbb{R}^{M + 1},d = (d_0,\dots ,d_M)\in [0,1]^{M + 1},$ $0 = d_{0} < \dots < d_{M} = 1$ , and  $(u)_{+}\coloneqq \max (0,u)$ .  $\theta_{j}$  is a neural network parameterized mapping such that  $\theta_{j}:\mathbb{R}^{d}\mapsto \mathbb{R}\times \mathbb{R}^{M + 1}$ , which takes  $\mathbf{z}$  as input and outputs  $\gamma^{(j)}(\mathbf{z})$  and  $b^{(j)}(\mathbf{z})$ . The constraint is introduced to ensure monotonicity. As demonstrated in [20], the reconstruction loss can be computed in a closed form by utilizing the parameterization of (7) (refer to Appendix A.5 for a detailed description of the loss function). This implies that our objective function (2) is computationally tractable. Note that the linear isotonic spline is not differentiable for finite points where the measure has no point mass.
Dataset. For evaluation, we consider following real tabular datasets: covertype, credit, loan, adult, cabs, and kings (see Appendix A.8 for detailed data descriptions). We treat the ordinal variables as continuous variables and discretize the estimated CDF (see Appendix A.6 for the discretization algorithm). Synthetic samples of ordinal variables are rounded to the first decimal place.