SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Compared models. We compare  $\mathrm{DistVAE}^{\dagger}$  with the state-of-the-art synthesizers; CTGAN [63], TVAE [63], and CTAB-GAN [65]. All models have the same size of the latent dimension ( $d = 2$ ). The chosen latent space indeed limits the capacity of decoders for all models. However, we maintain a small and consistent number of parameters across all models during the experiment to isolate the performance differences in synthetic data generation to the methodologies of each synthesizer, specifically emphasizing the contribution of the decoder model's flexibility in estimating underlying distributions (see Table 10 in Appendix A.9 for a comprehensive comparison of the model parameters).
To assess the quality of the synthetic data, we employ three types of assessment criteria: 1) machine learning utility, 2) statistical similarity, and 3) privacy preservingability. Each criterion is evaluated using multiple metrics, and the performance of synthesizers is reported by averaged metrics over the real tabular datasets. The synthetic dataset is generated to have an equal number of samples as the real training dataset.
Machine learning utility. The machine learning utility (MLu) is measured by the predictive performance of the trained model over the synthetic data. We consider three popular machine learning algorithms: linear (logistic) regression, Random Forest [6], and Gradient Boosting [19]. We measure the performance by utilizing the mean absolute relative error (MARE) for regression tasks [47] and the  $F_{1}$  score for classification tasks [63, 65, 61, 29, 47, 12, 18].
Statistical similarity. The marginal distributional similarity between the real training and synthetic datasets is evaluated using two metrics: the Kolmogorov statistic and the 1-Wasserstein distance [18]. These metrics measure the distance between the empirical marginal CDFs [40]. The joint distributional similarity is assessed by comparing the correlation matrices [65]. To compute the correlation matrix and measure the  $L_{2}$  distance between the correlation matrices of the real training and synthetic datasets, we employ the dychon library. These enable a comprehensive evaluation of both marginal and joint distributional similarities between the real training and synthetic datasets.
Privacy preserving capacity. The privacy-preserving capacity is measured using three metrics: the distance to closest record (DCR) [47, 65], membership inference attack [54, 12, 47], and attribute disclosure [12, 44]. As in [65], the DCR is defined as the  $5^{th}$  percentile of the  $L_{2}$  distances between all real training samples and synthetic samples. Since the DCR is a  $L_{2}$  distance-based metric, it is computed using only continuous variables. A higher DCR value indicates a more effective preservation of privacy, indicating a lack of overlap between the real training data and the synthetic samples. Conversely, an excessively large DCR score suggests a lower quality of the generated synthetic dataset. Therefore, the DCR metric provides insights into both the privacy-preserving capability and the quality of the synthetic dataset.
We release the code at https://github.com/an-seunghwan/DistVAE.
http://shakedzy.xyz/dython/modules/nominal/#associations
The membership inference attack evaluation follows the steps detailed in Appendix A.7. The procedure is customized to be applied to a VAE-based synthesizer, such as DistVAE and TVAE. By transforming the problem into a binary classification task, we aim to identify the intricate relationship between the real training data and the synthetic samples. Higher binary classification scores indicate a higher vulnerability of the target synthesizer to membership inference attacks.
Attribute disclosure refers to the situation where attackers can uncover additional covariates of a record by leveraging a subset of covariates they already possess, along with similar records from the synthetic dataset. To quantify the extent to which attackers can accurately identify these additional covariates, we employ classification metrics. Higher attribute disclosure metrics indicate an increased risk of privacy leakage, implying that attackers can precisely infer unknown variables. In terms of privacy concerns, attribute disclosure can be considered a more significant issue than membership inference attacks, as attackers are assumed to have access to only a subset of covariates for a given record [12].
Machine learning utility. We expect a high-quality synthesizer to generate synthetic data with comparable predictive performance to the real training dataset, denoted as the 'Baseline' in Table 1. The results in Table 1 demonstrate that DistVAE achieves a competitive MARE score and outperforms other methods in terms of the  $F_{1}$  score. Furthermore, the performance of MLu improves as the value of  $\beta$  decreases, indicating that the quality of the generated synthetic data is controlled by  $\beta$ . For a comprehensive overview of the MLu scores for all tabular datasets, please refer to Appendix A.10.
Table 1: Averaged MLu metrics (MARE,  $F_{1}$ ). Mean and standard deviation values are obtained from 10 repeated experiments.  $\uparrow$  denotes higher is better and  $\downarrow$  denotes lower is better.
Model	MARE ↓	F1 ↑
Baseline	0.150±0.200	0.814±0.101
CTGAN	0.321±0.271	0.672±0.234
TVAE	0.225±0.215	0.594±0.295
CTAB-GAN	0.403±0.392	0.702±0.162
DistVAE(β = 0.5)	0.349±0.328	0.769±0.128
DistVAE(β = 1)	0.344±0.316	0.762±0.134
DistVAE(β = 5)	0.392±0.348	0.679±0.190
Table 2: Averaged correlation structural similarity. 'CorrDist' represents  $L_{2}$  distance between the correlation matrices of synthetic and real training datasets. Mean and standard deviation values are obtained from 10 repeated experiments. Lower is better.
Model	CorrDist
CTGAN	2.180±0.467
TVAE	2.739±0.796
CTAB-GAN	2.575±0.513
DistVAE(β = 0.5)	1.473±0.398
DistVAE(β = 1)	1.730±0.548
DistVAE(β = 5)	3.113±1.119
Statistical similarity. The evaluation results for joint and marginal distributional similarities are presented in Table 2 and 3. In Table 2, DistVAE achieves the lowest CorrDist score, indicating its ability to accurately preserve the correlation structure of the real training dataset in the generated synthetic dataset. Furthermore, DistVAE surpasses other methods in Table 3 when it comes to marginal distributional similarity, suggesting that it successfully captures the underlying distribution of the observed dataset. Notably, reducing the value of  $\beta$  leads to an enhancement in the quality of the synthetic dataset, as evidenced by improvements in the correlation structure and similarity of the marginal distributions. Figure 1 provides visualizations of the estimated CDFs (6) for each continuous (or ordinal) variable in the cabs dataset. For detailed statistical similarity scores and additional visualizations of estimated CDFs for all tabular datasets, please refer to Appendix A.10.
Privacy preservingness. The privacy preservingness performances of synthesizers, as measured by the DCR, are presented in Table 4. DistVAE performs best in preserving privacy, with the highest DCR values compared to other methods. Notably, as the value of  $\beta$  increases in DistVAE, the DCR between the real training and synthetic datasets (R&S) also increases. This indicates that the risk of privacy leakage can be controlled by adjusting  $\beta$ , where higher values of  $\beta$  correspond to a higher level of privacy protection. Moreover, DistVAE consistently achieves large DCR values for the synthetic dataset (S) across all  $\beta$  values, indicating its ability to generate diverse synthetic samples. On the other hand, CTAB-GAN generates duplicated records in the synthetic dataset, resulting in
Table 3: Averaged marginal distributional similarity. K-S denotes the Kolmogorov-Smirnov statistic, and 1-WD represents the 1-Wasserstein distance. Mean and standard deviation values are obtained from 10 repeated experiments. Lower is better.
Model	Continuous	Discrete
K-S	1-WD	K-S	1-WD
CTGAN	0.133±0.106	0.087±0.025	0.168±0.195	0.521±0.532
TVAE	0.196±0.135	0.220±0.099	0.385±0.144	1.681±1.668
CTAB-GAN	0.157±0.089	0.130±0.037	0.106±0.083	0.412±0.378
DistVAE(β=0.5)	0.090±0.065	0.075±0.026	0.030±0.017	0.118±0.100
DistVAE(β=1)	0.081±0.039	0.083±0.019	0.027±0.021	0.116±0.110
DistVAE(β=5)	0.092±0.037	0.121±0.058	0.059±0.034	0.241±0.163
Figure 1: cabs dataset. Empirical (solid orange) and estimated (dashed blue) CDFs of continuous and ordinal variables (Monte Carlo approximated with 5000 samples). We standardize covariates and remove observations outside the  $1\%$  and  $99\%$  percentile range.
relatively lower DCR scores for the synthetic dataset (S). For detailed DCR scores for all tabular datasets, please refer to Appendix A.10.
Table 4: Privacy preservingability: Averaged distance to closest record (DCR) between real training and synthetic datasets (R&S), between the same real training datasets (R), and between the same synthetic datasets (S). Mean and standard deviation values are obtained from 10 repeated experiments. The DCR (R) score represents the baseline diversity of datasets. Higher is better.
Model	R&S	R	S
CTGAN	0.426±0.229	0.237±0.153	0.356±0.202
TVAE	0.470±0.181	0.237±0.153	0.278±0.195
CTAB-GAN	0.508±0.259	0.237±0.153	0.039±0.073
DistVAE(β=0.5)	0.444±0.250	0.237±0.153	0.463±0.288
DistVAE(β=1)	0.463±0.282	0.237±0.153	0.479±0.310
DistVAE(β=5)	0.517±0.272	0.237±0.153	0.511±0.335
To evaluate the membership inference attack, we prepare one attack model per class. The attack testing records comprise an equal number of real training and test records, distinguished by the labels in and out, respectively. Note that the test records are not employed in constructing the attack models. We employ gradient-boosting classifiers as the attack models, and for computational feasibility, we limit the number of attack models to one (i.e.,  $C = 1$ ).
For the membership inference attack evaluation, we utilize accuracy and AUC (Area Under Curve) as the evaluation metrics. Since the target labels  $(in/out)$  are balanced, and the task is a binary classification problem, these metrics are appropriate. The results presented in Table 5 reveal that both DistVAE and TVAE achieve nearly identical accuracy and AUC scores of 0.5. This indicates that the attack models can not distinguish between members of the real training and test datasets. Consequently, the membership inference attack is unsuccessful for both models. Therefore, DistVAE effectively generates synthetic datasets while ensuring privacy against membership inference attacks. A comprehensive assessment of membership inference attack performances for all tabular datasets can be found in Appendix A.10.
Table 5: Privacy conservability: Averaged membership inference attack performance. Mean and standard deviation values are obtained from 10 repeated experiments.
Model	Accuracy	AUC
TVAE	0.495±0.019	0.495±0.019
DistVAE(β=0.5)	0.500±0.003	0.500±0.003
Table 6: Privacy preservingability: Averaged attribute disclosure performance with the  $F_{1}$  score. Mean and standard deviation values are obtained from 10 repeated experiments. Lower is better.
Model	Number of neighbors (k)
1	10	100
CTGAN	0.262±0.091	0.282±0.087	0.275±0.087
TVAE	0.437±0.162	0.438±0.160	0.432±0.162
CTAB-GAN	0.257±0.123	0.258±0.114	0.261±0.111
DistVAE(β = 0.5)	0.328±0.088	0.328±0.076	0.310±0.072
DistVAE(β = 1)	0.307±0.073	0.313±0.068	0.297±0.066
DistVAE(β = 5)	0.265±0.105	0.253±0.103	0.232±0.101
We present the attribute disclosure performance results in Table 6. For each value of  $k$ , we observe that as  $\beta$  increases, the  $F_{1}$  score of DistVAE decreases. Also, DistVAE achieves the smallest  $F_{1}$  score when  $k$  equals 10 and 100. Based on these results, we can conclude that DistVAE can generate synthetic datasets with a low risk of attribute disclosure, and the level of privacy preservation is controlled by  $\beta$ . Please refer to Appendix A.10 for a detailed evaluation of attribute disclosure performance for all tabular datasets.
Quantile estimation. To investigate the quantile estimation performance, we also evaluate DistVAE using the Vrate  $(\alpha)$  metric [11]. The Vrate  $(\alpha)$  is defined as  $\frac{1}{|I_{test}|}\sum_{i\in I_{test}}I(x_i < \hat{Q}_\alpha)$ , where  $\alpha \in (0,1)$ ,  $I_{test}$  is the set of indices for the test dataset,  $x_i$  is the  $i$ -th sample in the test dataset, and  $\hat{Q}_\alpha$  is the empirical  $\alpha$ -quantile of the synthetic data. Since Vrate  $(\alpha)$  indicates the proportion of compliance samples in the test dataset, the Vrate  $(\alpha)$  score should be close to  $\alpha$ .
Table 7: Averaged Vrate(α) and  $\left| {\alpha  - \operatorname{Vrate}\left( \alpha \right) }\right|$  .
α	0.1	0.3	0.5	0.7	0.9
Vrate(α)	0.204	0.373	0.533	0.725	0.908
|α - Vrate(α)|	0.104	0.083	0.04	0.032	0.008
The  $\mathrm{Vrate}(\alpha)$  evaluation results are presented in Table 7. Table 7 shows that the ratio of violated test samples  $(|\alpha - \mathrm{Vrate}(\alpha)|)$  decreases as  $\alpha$  increases, indicating a better performance for estimating larger quantiles. However, the ratio of violated test samples is relatively large for smaller  $\alpha$  values, which may be due to extremely skewed continuous variables, such as capital-gain and capital-loss from adult dataset, that make the quantile estimation unstable.
This paper introduces a novel distributional learning method of VAE, which aims to effectively capture the underlying distribution of the observed dataset using a nonparametric approach. Our proposed method involves directly estimating CDFs using the CRPS loss while maintaining the mathematical derivation of the lower bound.
In our study, we confirm that the proposed decoder enhances the performance of generative models for tabular data. However, this conclusion relies on the assumption of conditional independence among the observed variables given the latent variable. When the dimension of the latent variable is small, this assumption is prone to violation. Therefore, in cases where the size of the latent
space is limited, the proposed nonparametric fitting of the decoder might not accurately represent the underlying distribution. Particularly in the image domain, where adjacent pixel values exhibit significant dependence, it remains uncertain whether our proposed model would lead to notable improvements in image data generation with a low-dimensional latent space.
Nevertheless, classical image datasets, such as CIFAR-10 [36], often exhibit pixel value distributions that deviate considerably from Gaussian, with the frequencies of edge values (0 and 255) dominating more than other pixel values [16, 53]. Other image datasets, as presented in [48], demonstrate multi-modality in pixel value distributions. These experimental findings suggest that leveraging the capacity of distributional learning could be advantageous in approximating the ground-truth distribution of image data when the latent variable effectively captures conditional independence among the image pixels. Consequently, we expect that compromising biases arising from violating conditional independence and marginally misspecified distributions may further enhance our results, and we leave it for future research.
On the other hand, we consider two approaches to enhance the performance of quantile estimation. Firstly, we plan to extend the parameterization of conditional quantile functions to a more flexible monotonic regression model. Secondly, we intend to incorporate the Uniform Pessimistic Risk (UPR) [26] into the VAE framework to handle lower quantile levels better. Furthermore, we are exploring the expansion of DistVAE into a time-series distributional forecasting model by adopting the conditional VAE framework [56]. This extension will enable the application of our method to time-series data, opening new avenues for distributional forecasting.
This work was supported by the National Research Foundation of Korea (NRF) grant funded by the Korea government (MSIT) (No. NRF-2022R1A4A3033874 and No. NRF-2022R1F1A1074758). This work was also supported by Korea Environmental Industry & Technology Institute (KEITI) through 'Core Technology Development Project for Environmental Diseases Prevention and Management', funded by Korea Ministry of Environment (MOE) (2021003310005). The authors acknowledge the Urban Big data and AI Institute of the University of Seoul supercomputing resources (http://ubai.uos.ac.kr) made available for conducting the research reported in this paper.
[1] Haleh Akrami, Anand Joshi, Sergul Aydore, and Richard Leahy. Deep quantile regression for uncertainty estimation in unsupervised and supervised lesion detection. Machine Learning for Biomedical Imaging, 1:1-23, 2022.
[2] Jonathan T. Barron. A general and adaptive robust loss function. 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 4326-4334, 2017.
[3] David M. Blei, Alp Kucukelbir, and Jon D. McAuliffe. Variational inference: A review for statisticians. Journal of the American Statistical Association, 112:859 - 877, 2016.
[4] Axel Brando, Jose A Rodriguez, Jordi Vitria, and Alberto Rubio Muñoz. Modelling heterogeneous distributions with an uncountable mixture of asymmetric laplacians. Advances in neural information processing systems, 32, 2019.
[5] Gustav Bredell, Kyriakos Flouris, Krishna Chaitanya, Ertunc Erdil, and Ender Konukoglu. Explicitly minimizing the blur error of variational autoencoders. In The Eleventh International Conference on Learning Representations, 2023.
[6] Leo Breiman. Random forests. Machine learning, 45:5-32, 2001.
[7] Andrew Brock, Jeff Donahue, and Karen Simonyan. Large scale GAN training for high fidelity natural image synthesis. In International Conference on Learning Representations, 2019.
[8] Yuri Burda, Roger Baker Grosse, and Ruslan Salakhutdinov. Importance weighted autoencoders. CoRR, abs/1509.00519, 2015.
[9] Alex J. Cannon. Non-crossing nonlinear regression quantiles by monotone composite quantile regression neural network, with application to rainfall extremes. Stochastic Environmental Research and Risk Assessment, 32:3207-3225, 2018.
A.1 Derivation of ELBO
by Jensen's inequality.
$$
\begin{array}{l} \mathbb {E} _ {p (\mathbf {x})} \mathbb {E} _ {q (\mathbf {z} | \mathbf {x}; \phi)} \left[ \frac {1}{K} \sum_ {k = 1} ^ {K} \rho_ {\alpha_ {k}} \Big (\mathbf {x} _ {j} - D _ {j} (\alpha_ {k} | \mathbf {z}, \theta_ {j}) \Big) \right] = \frac {1}{K} \sum_ {k = 1} ^ {K} h (\alpha_ {k}) \\ = \sum_ {k = 1} ^ {K} h (\alpha_ {k}) \cdot (\alpha_ {k} - \alpha_ {k - 1}), \\ \end{array}
$$
where  $\alpha_0\coloneqq 0$  and  $\mathbb{E}_{p(\mathbf{x})}\mathbb{E}_{q(\mathbf{z}|\mathbf{x};\phi)}\left[\rho_{\alpha_k}\Bigl (\mathbf{x}_j - D_j(\alpha_k|\mathbf{z},\theta_j)\Bigr)\right]$  is denoted as  $h(\alpha_{k})$
Since  $\alpha_{k}\in [\alpha_{k - 1},\alpha_{k}]$  and  $h(\cdot):[0,1]\mapsto \mathbb{R}$  is a continuous function, for  $j\in I_C$
$$
\begin{array}{l} \lim _ {K \to \infty} \mathbb {E} _ {p (\mathbf {x})} \mathbb {E} _ {q (\mathbf {z} | \mathbf {x}; \phi)} \left[ \frac {1}{K} \sum_ {k = 1} ^ {K} \rho_ {\alpha_ {k}} \Big (\mathbf {x} _ {j} - D _ {j} (\alpha_ {k} | \mathbf {z}, \theta_ {j}) \Big) \right] = \lim _ {K \to \infty} \sum_ {k = 1} ^ {K} h (\alpha_ {k}) \cdot (\alpha_ {k} - \alpha_ {k - 1}) \\ = \int_ {0} ^ {1} h (\alpha) d \alpha \\ = \int_ {0} ^ {1} \mathbb {E} _ {p (\mathbf {x})} \mathbb {E} _ {q (\mathbf {z} | \mathbf {x}; \phi)} \left[ \rho_ {\alpha} \left(\mathbf {x} _ {j} - D _ {j} (\alpha | \mathbf {z}, \theta_ {j})\right) \right] d \alpha \\ { = } { \mathbb { E } _ { p ( \mathbf { x } ) } \mathbb { E } _ { q ( \mathbf { z } | \mathbf { x } ; \phi ) } \left[ \int _ { 0 } ^ { 1 } \rho _ { \alpha } \Bigl ( \mathbf { x } _ { j } - D _ { j } ( \alpha | \mathbf { z } , \theta _ { j } ) \Bigr ) d \alpha \right] , } \\ \end{array}
$$
by the definition of the Riemann integral and the Fubini-Tonelli theorem. The proof is complete.
$$
\begin{array}{l} \min  _ {\theta} \mathbb {E} _ {p (\mathbf {x}) q (\mathbf {z} | \mathbf {x}; \phi)} \sum_ {j = 1} ^ {p} \int_ {0} ^ {1} \rho_ {\alpha} (\mathbf {x} _ {j} - D _ {j} (\alpha , \mathbf {z}; \theta_ {j})) d \alpha = \mathbb {E} _ {q (\mathbf {z}; \phi) q (\mathbf {x} | \mathbf {z}; \phi)} \sum_ {j = 1} ^ {p} \int_ {0} ^ {1} \rho_ {\alpha} (\mathbf {x} _ {j} - D _ {j} (\alpha , \mathbf {z}; \theta_ {j})) d \alpha \\ = \sum_ {j = 1} ^ {p} \mathbb {E} _ {q (\mathbf {z}; \phi)} \mathbb {E} _ {q (\mathbf {x} _ {j} | \mathbf {z}; \phi)} \int_ {0} ^ {1} \rho_ {\alpha} (\mathbf {x} _ {j} - D _ {j} (\alpha , \mathbf {z}; \theta_ {j})) d \alpha . \\ \end{array}
$$
So, for all  $j = 1,\dots ,p$
$$
\theta_ {j} ^ {*} (\phi) \in \arg \min _ {\theta_ {j}} \mathbb {E} _ {q (\mathbf {z}; \phi)} \mathbb {E} _ {q (\mathbf {x} _ {j} | \mathbf {z}; \phi)} \int_ {0} ^ {1} \rho_ {\alpha} (\mathbf {x} _ {j} - D _ {j} (\alpha , \mathbf {z}; \theta_ {j})) d \alpha ,
$$
and it is proper scoring rule relative to  $F_{j}(\cdot |\mathbf{z};\phi)$  for all  $\mathbf{z}\in \mathbb{R}^d$ . It implies that  $D_{j}^{-1}(\mathbf{x}_{j},\mathbf{z};\theta_{j}^{*}(\phi)) = F_{j}(\mathbf{x}_{j}|\mathbf{z};\phi)$ , and  $\frac{d}{d\mathbf{x}_j} D_j^{-1}(\mathbf{x}_j,\mathbf{z};\theta_j^* (\phi)) = q_j(\mathbf{x}_j|\mathbf{z};\phi)$ , for all  $\mathbf{x}_j\in \mathbb{R}$ , by Assumption 2.
$$
\begin{array}{l} \int \prod_ {j = 1} ^ {p} \frac {d}{d \mathbf {x} _ {j}} D _ {j} ^ {- 1} (\mathbf {x} _ {j}, \mathbf {z}; \theta_ {j} ^ {*} (\phi)) q (\mathbf {z}; \phi) d \mathbf {z} = \int \prod_ {j = 1} ^ {p} q _ {j} (\mathbf {x} _ {j} | \mathbf {z}; \phi) q (\mathbf {z}; \phi) d \mathbf {z} \\ = \int q (\mathbf {x} | \mathbf {z}; \phi) q (\mathbf {z}; \phi) d \mathbf {z} \\ = \int \frac {p (\mathbf {x}) q (\mathbf {z} | \mathbf {x} ; \phi)}{q (\mathbf {z} ; \phi)} q (\mathbf {z}; \phi) d \mathbf {z} \\ = \int p (\mathbf {x}) q (\mathbf {z} | \mathbf {x}; \phi) d \mathbf {z} \\ = \quad p (\mathbf {x}). \\ \end{array}
$$
$$
\begin{array}{l} \min _ {\theta} \mathbb {E} _ {p (\mathbf {x}) q (\mathbf {z} | \mathbf {x}; \phi)} \sum_ {j = 1} ^ {p} \int_ {0} ^ {1} \rho_ {\alpha} (\mathbf {x} _ {j} - D _ {j} (\alpha , \mathbf {z}; \theta_ {j})) d \alpha = \mathbb {E} _ {q (\mathbf {z}; \phi) q (\mathbf {x} | \mathbf {z}; \phi)} \sum_ {j = 1} ^ {p} \int_ {0} ^ {1} \rho_ {\alpha} (\mathbf {x} _ {j} - D _ {j} (\alpha , \mathbf {z}; \theta_ {j})) d \alpha \\ = \sum_ {j = 1} ^ {p} \mathbb {E} _ {q (\mathbf {z}; \phi)} \mathbb {E} _ {q (\mathbf {x} _ {j} | \mathbf {z}; \phi)} \int_ {0} ^ {1} \rho_ {\alpha} (\mathbf {x} _ {j} - D _ {j} (\alpha , \mathbf {z}; \theta_ {j})) d \alpha . \\ \end{array}
$$
So, for all  $j = 1, \dots, p$ ,
$$
\theta_ {j} ^ {*} (\phi) \in \arg \min  _ {\theta_ {j}} \mathbb {E} _ {q (\mathbf {z}; \phi)} \mathbb {E} _ {q (\mathbf {x} _ {j} | \mathbf {z}; \phi)} \int_ {0} ^ {1} \rho_ {\alpha} (\mathbf {x} _ {j} - D _ {j} (\alpha , \mathbf {z}; \theta_ {j})) d \alpha ,
$$