Missing Data Imputation and Acquisition with Deep Hierarchical Models and Hamiltonian Monte Carlo
Authors: Ignacio Peis
Chao Ma
José Miguel Hernández-Lobato
Venue: NeurIPS 2022 Accept (2022)
Keywords: Variational Autoencoders
Hamiltonian Monte Carlo

Abstract: Variational Autoencoders (VAEs) have recently been highly successful at imputing and acquiring heterogeneous missing data. However, within this specific application domain, existing VAE methods are restricted by using only one layer of latent variables and strictly Gaussian posterior approximations. To address these limitations, we present HH-VAEM, a Hierarchical VAE model for mixed-type incomplete data that uses Hamiltonian Monte Carlo with automatic hyper-parameter tuning for improved approximate inference. Our experiments show that HH-VAEM outperforms existing baselines in the tasks of missing data imputation and supervised learning with missing features. Finally, we also present a sampling-based approach for efficiently computing the information gain when missing features are to be acquired with HH-VAEM. Our experiments show that this sampling-based approach is superior to alternatives based on Gaussian approximations.

Missing Data Imputation and Acquisition with Deep Hierarchical Models and Hamiltonian Monte Carlo
Ignacio Peis
Chao Ma Microsoft Research University of Cambridge
Universidad Carlos III de Madrid Madrid, Spain ipeisêtsc. uc3m.es
Cambridge, UK cm905@cam. ac
José Miguel Hernández-Loband University of Cambridge
Cambridge, UK jmh2338cam ac uk
Abstract
Variational Autoencoders (VAEs) have recently been highly successful at imputing and acquiring heterogeneous missing data. However, within this specific appli- cation domain, existing VAE methods restricted by using only one layer of latent variables and strictly Gaussian posterior approximations. To address these limitations, present HH- VAEM, Hierarchical VAE model for mixed-type in- complete data that uses Hamiltonian Monte Carlo with automatic tuning for improved approximate inference. Our experiments show that outperforms existing baselines in the tasks missing data imputation and super- vised learning with missing features. Finally, also present campling-basee approach for efficiently computing the information gain when missing features are be acquired with HH-VAEM. Our experiments show that this sampling- approach is superior to alternatives based on Gaussian approximations.
Introduction
Many real-world unsupervised learning tasks require dealing with complicated datasets with mixed types (real, positive-valued, continuous, or discrete) and missing values. For this purpose, variational autoencoders 24 25 [1 stand out in the recent literature as robust generative models that efficiently handle high-dimensional data. However, in their naive configuration, every data dimension is assumed to have similar statistical properties (i.e., homogeneity), and all dimensions considered to be completely observed. Both assumptions won't Itn't hold in many real-world scenarios. Recent works have adapted VAES to handle incomplete demonstrated improved performance in downstream tasks such as missing data imputation and active information acquisition. Despite these advances, existing approaches are far from optimal as they are based on restrictive design choices: 1), only one layer of latent variables are considered; 2), Gaussian posterior approximations are usually adopted. These will lead to limited flexibility and additional bias, especially under real-world settings with complex mixed-type incomplete data.
36) and mixed-type data [18
33 38), and
In the literature, the issue of model flexibility and inference bias often addressed separately. For example, approximate inference bias can be reduced by using Monte Carlo sampling [46] [50] More specifically, Hamiltonian Monte Carlo (HMC) [2] stands out among MCMC methods machine learning due to its superior efficiency for exploring the target density. In the context of VAE, HMC has also been combined with stochastic variational inference for improving the training of VAES.
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

On the other hand, the flexibility of VAES can be improved by considering hierarchical VAEs with multiple layers of hidden variables [8] 35 48 52 By using a hierarchical structure in the latent space, they enforce the information to flow from high-level representations to more specific observable factors, imitating the way information is often organized in the real world.
However, the issue of modeling flexibility and approximate inference bias are often heavily inter- twined, and addressing them simultaneously in a joint manner highly non-trivial. The hierarchical organization of the latent variables creates complicated posterior dependencies that are not straight- forward to deal with and require special consideration. To improve Gaussian approximate inference, most works opt by defining shared paths between the recognition and generative networks. While this makes hierarchical VAEs practical, the bias introduced by the Gaussian approximations is still present. To the best of our knowledge, none of the aforementioned hierarchical VAEs has been previously combined with Monte Carlo algorithms for improving over standard Gaussian approximate inference.
To overcome such limitations, we focus on training new hierarchical VAE models for heterogeneous mixed-type data with HMC. Our models can be used for missing data imputation and for supervised learning with missing data. We also present a sampling-based framework that allows our models to perform accurate sequential active information acquisition.
Our main contributions are as follows:
We present HH- VAEM, deep hierarchical model for handling mixed-type incomplete data that uses HMC with automatic hyper-pant arper-parameter tuning for outperforming amortized variational inference by generating low bias samples from the true posterior.
We propose sampling- campling-based based strategy for missing feature acquisition that benefits from the improved inference of HH-MA By using histograms estimate the mutual information, this strategy achieves lower bias than other Gaussian -based alternatives.
We exhaustively evaluate HH- VAEM in the tasks of 1) missing data imputation, 2) supervised learning with missing data and 3) information acquisition with our compling-based strategy, In all cases we report significant gains with respect to baselines.
2 Related work
2.1 VAEs for mixed-type incomplete data
Variational Autoencoders 44
are deep generative models that make use of encoder and decoder networks for mapping data into a latent Gaussian distribution, and reconstructing the latent codes into the original observational space, respectively. The parameters of these networks are trained using amortized Variational Inference [10, (55 optimizing a lower bound (ELBO) on the log evidence:
(x) (=le)) log m)(= where the generative model po (ar,z) can be expressed in terms of the likelihood
(=)2)) and the prior p(z). In mixed-type data, the vector x is composed by data from different types: real, positive real, categorical, binary, etc. A naive approach is consider factorized decoder using different likelihood contributions (0: ()) = IIa Po [ral =) [ Nonetheless, the problem of handling unbalanced likelihoods leads to the domination of some dimensions during the optimization process. In 333, authors propose a solution using set of marginal VAEs that encode each feature into a Gaussian uni-dimensional space, and dependency VAE that captures the inter-dimensional dependencies more effectively using balanced Gaussian likelihoods.
By marginalizing each dimension of the decoder, incomplete data can be easily handled by dividing the vector æ into the observed xo and unobserved xy parts. This methodology is completely valid when using the missing-at-randon (MAR) assumption [30], i.e. assuming the missing mechanism is independent of the missing values. In this our work, we use the same assumption. As proposed in
38 and (36), the ELBO objective is transformed into a lower bound on the observed data, and the unobserved data is replaced with zeros.
2.2 Hierarchical VAFS
Hierarchical models have been successfully employed in deep generative modeling, 2 44 45
In
VABs, defining hierarchical latent space for VAEs can be straightforward. Nevertheless, potential pitfalls require special attention. Concretely, if the decoder is powerful enough, the model tends

---

to uniquely use the shallowest layers, ignoring the deepest ones and falling into the well-known 533. In the last few years, several works have investigated
problem of posterior collapse [35 40
possible hierarchical structures for VA In 48
bottom-up deterministic path is used along with
top-down inference path that shares parameters with the top-down structure of the generative model.
In [55], the authors use a bidirectional stochastic inference path. More recently, [52) or [8] have adapted these architectures to complex datasets and high quality images. Possibly motivated by the residual connections in [23], all these works use shared path between recognition and generative models that helps in tying the divergences between approximations and priors the ELBO.
2.3 Hamiltonian Monte Carlo
HMC [4 12, (3] is particularly effective MCMC algorithm for sampling from a target distribution p(z) (z) where Z the normalization constant and z is d-dimensional vector. By augmenting this model to p(r, 2) = (N; 0, p2z), and denoting r as the momentum variable with diagonal covariance matrix M, with the same dimensionality as z, HMC samples are obtained from the distribution by simulating the time-evolution of a fictitious physical system.
The algorithm starts by firstly sampling z and from an initial proposal and the momentum dis- tribution, respectively. Chains with length 7 are built by recurrently proposing and accepting new states. To propose new state, the Hamiltonian dynamics are simulated using a symplectic integrator, Leapfrog being the most common choice. The following updates are repeated for = =1 LF steps:
7 +
O V., logp" log)"(z1),
1
*4+1 = 21 +
M
T1+1 = 4
V 244 log p" (z1-1),
where O refers to the Hadamard product, and Φ is the step size hyperparameter. Although it is typically defined as a scalar for simplicity, d-dimensional vector can be considered to apply different step sizes per dimension, or further, as considered in this work, a T x d matrix to apply different steps per each proposal of the chain. The new proposal (z', ,') is accepted with probability min exp( H r') (zz, r]]], where
Hzz, r) log (z) + 2r
(2)
For the consecutive T proposals, new momentum r is resampled and the updates of eq. repeated for LF steps to update the state if (z', r') is accepted.
① are
2.4 Active Feature Acquisition
is of special interest in cost-sensitive applications for modeling trade-off between the improvement of predictions and the cost of acquiring new data at the feature level. Several works in the recent literature have studied methods for performing the task of sequentially acquiring high-value information by selecting features that maximize an information thcoretical reward function and enhance the accuracy of the predictions. This task is denoted by SAIA (Sequential Active Information Acquisition). In M an efficient method is proposed for approximating non-tractale -nttractable reward by using the encoder of a VAE that handles missing data. In [3] they extend this method for handling mixed-type data. Both works estimate the reward by relying on Gaussian approximations given by the encoder networks.
Among all the Active Learning techniques, Active Feature Acquisition 20
37
43 49
3 Hamiltonian hierarchical VAE for mixed-type incomplete data (HH-VAEM)
The HHEM model (Figure I (a)) is a Hierarchical VAE for mixed-type, incomplete data that optimization for sampling from the posterior
incorporates HMC with automatic of the latent variables. In a first stage, the mixed-type data is encoded into marginal Gaussian posterior distributions as given by univariate VAEs fitted to each data dimension. In a second stage, hierarchical structure captures the dependencies among the standarized, homogeneous dimensions with well-balanced Gaussian likelihoods. The model is trained using samples from the posterior of the
3

---

EL
ht
Zd
28
yo
yu
(a) The HH-VAEM model
(b) HMC sampling
Figure 1: The HH-VAEM model (a). Illustrative example (b): samples e obtained with HMC (orange) following the true posterior p(e xe yo) (green) using the Gaussian distribution given by the encoder q (0)) (e)zo, yo) (blue) as the initial proposal, with latent dimension M = 2.
hierarchical latent variables by means of HMC. whilst the HMC byper-parameters are automatically tuned. more detailed description is provided in the following subsections.
3.1 Notation
The model generates both data RDD and output y E R.P, where each of these variables is divided into observed parts 2o, yo and unobserved parts zv. yv- Each dimension of x is denoted by v.- A training set is composed of N observations as tuples (x/ (n) Yo (a)). For ease of notation, omit the observation index n, and the objectives are presented for single observation. The dependency latent space is composed of L latent variables (c,...., Ey.] with E Rm The dimension of the joint latent
distribution is > my = M. In the marginal VAEs, the latent variables zd are unidimensional.
3.2 Handling heterogeneous incomplete data
Following the strategy proposed by 333, we perform two-staged approach for handling hetero- geneous data. The marginal distribution of each feature Pe, (xta) is modeled by one-dimensional VAE. First, the marginal VAEs are trained independently by maximizing the marginal ELBO over observed points
Pa zd) Ira (ha(Zalle (za) (ra
La(ara; (0a, ra)) (ara E xo ( log
where 1 ( E xo) an indicator function that activates the ELBO when the feature is observed. Under the missing-at-randee at-raado-roommm assumption, which is the considered in this work, Equation (3)
leads to a lower bound on observed data likelihoods. Second, a dependency VAE encodes a vector z
with the concatenated samples from the marginal posteriors Ira (zalxa) into the global latent variable h. using zero-filling for the unobserved variables. By using this approach, z is now homogeneous and can then be easily modeled using standard Gaussian decoder. By contrast, other works [13 38
directly operate with different decoding likelihoods per data type. This approach often leads to having very different magnitudes in the ELBO and may reduce learning efficiency. The ELBO for the second stage dependency VAE is
Po yo, c) qu [E)
C(xo. yo: (0, ()) Equ log
(4)
2o yo)
where (E1,.., { is set reparameterized hierarchical latent variables. Further details on the design of the hierarchical dependency VAE are provided below.

---

3.3 Predictive enhancement
The combination of generative and discriminative models is an effective well-studied strategy for dealing with predictive models under missing data 51]. In [13], they model p(z) for imputing missing data using Gaussian Mixture Model. In a deep learning context, recent supervised VAE models have revisited this combination [21 47 or used factorisations of type p(=)p(az]=) (p)())))
22 to learn meaningful representations. In the authors propose a deep generative model with factorisation p(z)p(a () a. z) for detecting adversarial attacks.
With the aim at reinforcing the prediction of the variable of interest, we turn into supervised model by including a separate predictor for Pe,, (y (ē, h), apart from the decoder Pe. (z|/). The vector f = (z, E Io, 2, E xτ) includes the observed part and imputation of the missing variables ž by decoding the latent h into using p(= (h1), ), and each dimension 2j into 2, using p(zzl2z). The predictor parameters Og are optimized along with the decoder parameters 0z.
3.4 Hierarchical reparameterized latent space
A hierarchical structure over the latent space h = (hi..... h enriches the prior assumptions and permits flexible generation of data in a more natural fashion. Nevertheless, as stated 5] HMC can be pathological when used for sampling from hierarchical densities, where the magnitude of autoregressive variations increase with the depth. For approximating the Hamiltonian dynamies, inside each Leapfrog integrator step, gradients VA.. log p" (h1-L) are required. Due to the strong curvature regions, huge norm of high-order derivatives are backpropagated and might eventually explode, ending in overflow issues (Figure 35 in (5)).
If we were to run our HMC method over the hierarchical variables without any reparameterization (Figure/4h) 4h), by the time the states reached the aforementioned problematic regions, the integrator would diverge and we would experience the aforementioned overflow problems. By rejecting these problematic states, chains would get stuck close to the proposal and the hierarchical density would not be properly explored (concluding that HMC would not improve the Gaussian proposal). To give an example, in [52], the AR path p(h; (he))) would provoke instabilities inside the HMC integrator due to huge gradients in VALL log p" (h1.L).
We successfully solved this issue by introducing hierarchical reparameterization technique. The representation each layer reparameterized from variable E; with standard Gaussian prior p(E))
h4 = fu, (h(41) + fa, (hi41) EI,
(5)
where the functions fme (h1+ 1) and for 1)) are applied by NNs with parameters 6 = (0/m On,
The result is equivalent as learning the mean and covariance of autoregressive variables (see Figure 4 for illustrative details). However, thanks to this trick, we relax the dependencies among the latent variables, resulting a smoother joint posterior density p( o. yo). Performing the inference over
{EI...., EL) Icads to a better posed basis for running
HMC optimization, detailed in section
3.5 and avoids the necessity of employing more advanced HMC samplers like [6). We include
Turther details the pathological behavior and demonstration of the effectiveness of our solution in
Section B.3 of the Supplementary. Provided the promising results we obtain in Section we propose reparameterizztion trick as novel contribution for solving the pathological behavior of HMC combined with hierarchical VAEs.
Or.). The For the sake of simplicity, we name the generative parameters as 0 (02,0y, 01.,,,0 dependency ELBO under this hierarchical reparameterized model becomes
po (z yo: E) qo qe xo. yo)
Lvi yo: (+) = Egg log
(6)
Egg [logpo(zolhn) h log po hi)l
Σ DKL. (40(e))oo.
We name the hidden representation at each layer, and defining To {æo, yol }. we employ NNs with parameters vir for computing vi = (r(rr- These vectors are mapped into the parameters of the variational posterior dv, (E1 (0, (o). using NNs for computing the mean as grg (ri) and the
covariance as 9o, (r:), with parameters vin and vo, With compactness purposes, we will denote the encoder parameters as v {11,... ..,, }1 with 1 (wre Du }.
5

---

3.5 HMC over the hierarchical density
In recent works, HMC has been combined with deep generative models for improving the inference of the latent variables by obtaining better samples from the posterior [9]. In this work, we propose to transcend these previous approaches and build a generalized method for sampling from complicated, hierarchical latent structures composed by several layers. Inspired by [6] and their method for sampling from the posterior within a vanilla VAE framework while tuning the HMC hyperparameters, we follow a procedure for training the dependency model where i) during a pre-training stage, the encoder and decoder are optimized using standard VI and the ELBO from equation (6). and ii) using the pre-trained encoder for starting from good proposal [9]. HMC samples are obtained to follow the true posterior and jointly optimize the generative model and the HMC hyperparameters. In Figure (b) we include illustrative example.
We denote by 2 hyper-parameter of HMC are named o, a T x d matrix containing the step sizes for each dimension at each step of the chain. Within this perspective, the syper-parameters can be optimized using variational inference by maximizing the ELBO
(T)) (c|zo, To yo) the implicit distribution for the posterior after T HMC steps. The
E %
(r) [lg =0 yo, c)] + Hl (T) (E xo, yo)l:
where the first part is the HMC objective, and can be easily estimated via Monte Carlo
CHMOO yo: 10, 0)) E (c) Poo (11) + log po (o h1) + Σ p(c (T) (8)
Nevertheless, the entropy term H (l (el yo)l in Equation is not tractable since
not able to explicitly evaluate the distribution q. (elxo. yo). Although optimizing the first term might result in a well-posed algorithm, this would bring consequences that must be considered. Namely, without a proper regularization term, and in case the initial proposal g
(T) (e|zo. yo) is concentrated in high density regions, the chains would barely move from the initial state and only these regions with high density would be explored (see Section B.1 of the Supplementary for illustrative details). To cope with this problem, we define an inflation parameter s to increase the variance of the proposal (e(e)z Do, yo) given by the Gaussian encoder, ending in the proposal 11 (gg,, (ri), 81 Go, (ri)). Whilst in (6] the authors define this parameter as a scalar factor applied to all the latent dimensions, in our work, we extend this to apply different inflation factor at each latent level s = {si,.... .......... } In order to tune these inflations we ensure a wider coverage of the density by minimizing the Sliced Kemelized Stein Discrepancy (SKSD) [17]
(T) (E|zO, To, yo: 8), p(E zo xo, yol). LSKSD(wo, yo:s) 8) = SKSD to
which fits perfectly our requirements, since only requires samples from HMC and gradients V. log p(zo, yo, e) for measuring discrepancy between the implicit and the true posterior. Further, the SKSD performs better than other discrepancies like [1 in high dimensional latent spaces.
We provide in Section B.1 of the Supplementary a toy demonstration on the efficacy of the HMC optimization, and in Section B.2 a demonstration of the optimization convergence.
3.6 HH-VAEM optimization
The optimization of the HH-VAEM is divided into three stages. In a first stage, we train one independent marginal VAE per dimension. In second stage, the dependency VAE is trained, using as inputs the concatenation of the encoded dimensions 2 = (=1,-..,2p) and the target y. Finally, in third stage, the HMC hyperparameters, the decoder and the predictor are tuned using the HMC objective, the inflation parameter is trained using the SKSD discrepancy, and the encoder is trained used the ELBO. The pseudocode for HH-VAEM, training is shown in Algorithm
3.7 Computational cost
In VAE, the data fed the encoder with aim at obtaining the variational parameters for sampling from the Gaussian approximated posterior. In our method, the data is similarly encoded to obtain the

---

initial Gaussian proposal 9 and the samples from this distribution are updated for T cycles to obtain the implicit g()) using HMC. Within each of these iterations, L leapfrog steps (T) are executed. For each of these steps, the computation of the gradients Ve: log p(z, y, Ei) and Vel.. log p(z, 2 y, Et+1) is required. To obtain these gradients, we need to i) compute the parameters of the likelihood p(z, yle) that are given by the decoder (p(z h2))) and predictor (p(y) (ē, h1)). ii) evaluate the likelihood and iii) perform the automatic differentiation. Thus, for running our method, an additional cost from both decoding and performing differentiation total of 27 L times is introduced.
By jointly optimizing the HMC hyperparameters we are able to achieve faster convergence with smaller lengths. To reduce the com- putational cost, we optimize the by- perparameters in a final training stage, since convergence is rapidly achieved (as demonstrated empirically in Sec- tion B.2 of the Supplementary). At test, samples from the Gaussian ((0) (faster and cheaper), or from HMC ((T) (slower and better) can be used to fit computational constraints.
Algorithm 1 Training algorithm for HAEM
Input: data 2 30
(1:N) (1:N))
steps: Ta, TvI, THME
Parameters: y, 0, th, o,
STAGE 1: MARGINAL VAES for d 1 do
Initialize marginal VAE 70) for to Ta do 066 + Adam 04 (Ca)
Rd
end for end for STAGE 2: DEPENDENCY VAE
for t = 1 to Av do
01+1 +/+ + Ad (LvT)
Sampling-based
end for
Active Learning
STAGE 3: JOINTLY OPTIMIZING VAE + HMC
for t = 1 to THMC do
wtt &- Adamms (Cv1)
Considering that the input data are tuples of observed and missing fea- tures (zo. Zy J. our Active Learning
a++1 16,5511 + Adamo (o (CHMr)
s11 + Adam, (CsKSD)
framework follows 33
34 and de-
end for
termines which feature K; E zy will
enhance the prediction of the target y the most for a particular To. Concretely, in Sequential Active Information Acquisition (SAIA) scenario, this decision is taken sequentially to optimally increase knowledge and accurately predict y. From information theoretical perspective, this task can be performed recurrently by maximizing a reward function R at each step d. This reward might represent abstract quantities of interest like the cost or benefit of acquiring æ; (depending on the sign). In Bayesian experimental analysis, R is the expected gain of information [29]. Following [3), we can define it as
R(i, xo) = p(( DKL (p(yla,, xo (o)p()(=0))
(10)
where i is the index of each unobserved feature. Intuitively, this quantity can be interpreted as the expected change in the predictive distribution when æ;; is observed. The reward needs to be estimated via Monte Carlo by sampling from p(r) With robust generative model that handles missing data like HH-VAEM, these samples are easily obtained: first, using HMC, we sample e!T') from p(e (.).. Second, we decode these samples to obtain 2; from p( ((z(hi) and p(zz|za). Nonetheless, the reward defined in (TO) is intractable since both p(y (ı.)) and p(y (r; zo) are intractable: computing them requires to integrate out the latent variables. This motivates the authors of [3] 34 to present
transformation of the reward for being computed in the latent space using the encoder network. Although they prove that this transformation effectively provides a good estimation in several datasets, we demonstrate that for low dimensional targets (commonly one or two dimensions), an approximation using histograms is more effective. Concretely, the reward in (10) can be rewritten as
DxL. [p xz xx( lPp(y) ( xo )xx; [=o]] = I(y; (æ=).
(11)
While set of advanced non- parameerreiic estimators of the mutual information are available [26] many are not easilly adapted for parallelization. We demonstrate that the simplest one,
42).
p(i.))
i x; (xO)= = Σ p(t,j) log (1),
(12)
12
based on binning the x; and y domains, effective and easy to parallelize. the equation, (x) The (i)/NN () = nn/NN and j) n(i, J)NN are the relative frequencies that approximate
7

---

Bank
Insurancs
Avocado
Naval
Yasch
Diabetes
Concrele
Wine
Energy
Boston
VARM
1.81
1.80
+.00000.00
0.23
MIWAEM
1,92
2
0.51
2.76
11-VAEM
2.82.1 0.06
180 0.04
1.80 1
3.06 0:31
2.74
0.09
1.221.02.
2.85 1: 0.56
1.221:0111
1.8.1:00 0.24
BMC-VALM
2,09 4: 0.05
1.77 2: 0.00
1 = 0002
0.49:0.00
2.21 0.24
2.72
2
0.20
2 0:29
2,83 0.46
1.73
1 0.16
HH-YARM
2.63
0.04
1.76 0.03
1,88
0.05
= 0.05
2.47 0.33
2.54
0.18
0.13
2.28 0.08
1,00
1.71 0.04
1.83
Table 1: Test NLL of the unobserved features for model and baselines.
Bank
Incerance
Avocado
Naval
Yinch
Diabetes
Conerete
Wine
Energy
Boston
VAEM
0.56 L 0.06
1:20 1 0.00
1.18 L 0.02
2.69 0.01
0.61 1 1.002
0.19
1.07 1.009
0.25 1 0.09
0.61 0.144
0.85 0.21
MIWAEM H-VATM
0.00
1.15
0.01
0.60 0.00
1.36 0.10
0.25 0.13
0.54 0.112
0.80 0.21
1.18
0.00
0:00
0.66+:0.14
0.00 0:22
HMC
1.12-
0.52
1:20
0.28
2.10
HII-VAM
0.49 1: 0.03
0.93 0.08
1.10 1: 0.01
2.62 0.001
0.56
1.002
1.38 0.18
0.95
0.008
0.20 1 0.04
0.32 = 0.05
0.53 0.004
Table 2: Test NLL of the predicted target for model and basclines.
p(xr), ply) and p(æ, y) for each bin. Thus, (i). n,(() and nii, the number of samples inside each interval. The number of bins defines the width of uniformly distributed intervals over and supports. Since this estimator is sampling-basee based, under certain conditions (namely, if all densities exist as proper functions), Equation (13) indeed converges to [yy; 24 (xo) if we first let the number of samples N oc 26
Experiments
The evaluation of the HH-VAEM model is organized into three quantitative experiments and one qualitative experiment. The ablation study includes the validation of our proposed HMC-based, hierarchical model with respect to the Gaussian and onc-layered alternatives. Namely, the comparison is performed with the following baseline models:
(without including the Partial VAB). MIWAEM: layer, Gaussian- based, importance weighted IWAEM (VAEM + IWAE (3)
VAEM: 1 layer, Gaussian- VAEM 33
H- H-VAEM: 2 layers, Gaussian-based VAEM.
HMC-VAEM: layer, HMC-based (with our optimization method) VAEM.
VAE
MIWAE
H. VAB
HMC
HH-VAE
MNIST 0.124 0.001
0.121 1 0.001
0.119 1 0.001
0.101 i 0.004
0.094 + 0.003
hn
F-MNIST 0.162 = 0.002
0.160 = 0.002
0,156 + 0.002
0.150 = 0.002
0.111 = 0.002
Table 3: Test NLL of the unobserved features for the MNIST datasets.
(a) Autoregressive
VAB
MIWAB
H-VAB
HMC-VAE
HH-VAE
MNIST 0.153 + 0.009
0,151 # 0.007
0,146 + 0.006
0,067 = 0,007
0.056 + 0.019
F-NNITT 0.501 0.012
0.496 1 0.008
0.494
0.007
0.357
0.060
0.337 + 0.069
Table 4: Test NLL of the predicted target for the MNIST datasets.
VAB
MIWAE
VAB
HMC VAB
HH-VAE
(b) Reparameterization
MNIST 0.953 0.004
0.953
0.003
0.953
0.003
0.978
0.003
0.981 + 0.005
F-NNIST
0.821 0.005
0.821
0.001
0.821 # 0.001
0.869 = 0.015
0.876 + 0.017
Figure 4: AR (a) vs repa- Table Test accuracy the predicted digits for the MNIST datasets. rameterized (b).
For all the models, manually introduce missing data in the training set by randomly setting per data point a feature as missing with a probability sampled uniformly in the interval [0.01, 0.99] within each batch. Both the input data and the target y can be missing. For the test set, a fixed probability of 0.5 leads to about half of the input data being observed, whilst the target is completely unobserved.
For the quantitative experiments, total of 10 UCI datasets [1 that include mixed data are employed for the evaluation. We include both MNIST (27) and MaSHIO (54) datasets for evaluating our model in higher dimensional observational and latent spaces and bigger architectures (3 layered convolutional nets for encoder/decoder), For the qualitative results, we evaluate our model
in the image inpainting task MNIST and CelebA [22. For the three image datasets, the marginal
8

---

VAEM KL
VAEM
HELVAEM
0.40
15
0.35
Last gose
bost mise
6-
2
Steps
Stepe
Steps
Steps
(a) Avocado
(b) Yatch
(c) Boston
(d) Energy
0.8
0.6
0.0116
70-
1e43 mm
9.4
0.0014
10
0.0112
50-
8.0
0.0110
10
20
0
0.0
2.5
7.
15.0
Staps
Staps
Stupa
Stape
(e) Wine
(f) Naval
(g) Concrete
(h) Diabetes
Figure 5: SAIA curves. Horizontal axis shows number of discovered features. Vertical axis is RMSE.
VAES are not included and the dependency VAE is fed directly with the Bernoulliddistriuuee pixels. We name this model HH-VAE, and similarly, the baselines renamed VAE, MIWAE, HMC-VAE and H- VAE. Extended experiments and validations are provided in the Supplementary. The source code for reproducing our work is available at lttps: btttss///githut www.wpeesmeai..ccccccccc. HHEMM
5.1 Mixed type conditional data imputation
order to evaluate the performance the model in terms data imputation, we opt by computing the negative log likelihood of the unobserved features. We make use of the Monte Carlo approximation
1 T: En(arlee)) log p(ay aro) log (elro) [p( log
(13)
which is averaged over features in order to compare the imputation performance with the baselines. Additionally, we include in Section C.3 similar results averaging each of the considered likelihoods. Results on the 10 UCI datasets and the MNIST datasets are included in Tables and 3 showing that
for most of the datasets, incremental improvement is obtained: VAEM < H-VAEM < HMC-VAEM HH- Extended results with the imputation error, included in Section C.I, corroborate this.
5.2 Target prediction
For this experiment, we compute the negative log likelihood of the target under the predictive distribution using the same procedure as in the previous section. Results included in Tables and show the same incremental improvement in the prediction task.
5.3 Sequential active information acquisition (SATA)
In this experiment, our HH-VARM model and our acquisition method are evaluated in a SAIA task. Starting by predicting from completely unobserved inputs, each step, the missing feature that maximizes the reward is acquired. Figure the error curves for the UCI datasets. Blue lines correspond to the Gaussian-based reward proposed by (33). Orange lines are our sampling-based reward in a VAEM framework. Green lines correspond to HH-VAEM with our reward. In most of the cases, our model and acquisition method obtains lower errors and faster discovery of information.
5.4 Conditional image inpainting
We include in this experiment qualitative results when comparing our model with the baselines on the image conditional inpainting task. We include results for MNIST and CelebA in Figure 6 (a)

---

Original
8 6 7 9
9
5
0
8
4
2
6
9
5
0
9007
Observed
2
1
7
7
2
VAR
8 5 67799
0
0
6
7
0
0
H-VAE
8 6 7
5
0
6
4
0
6
9
4 '
0 0
107
HMC-VAE
8 6 7
5
0
3
1
6
9 9
9
007
HH-VAE
8
6 7
9
5
0
4
6
5
0
9
0
MNIST
Original
Observed
VAB
H-VAE
HAC
HH-VAB VAE
(b) CelebA
Figure 6: Image conditional inpainting on MNIST (a) and CelebA (b). First row: original images from the set. Second row: input to the model, Do. with black square missing mask æv manually introduced. Third, fourth, fifth and sixth rows: imputed dy for VAE, H-VAE, HMC-VAD and HH-VAE, our model, where रу is decoded from samples of the approximate posterior (Gaussian for VAE and H- VAE, or HMC-based for HMC-VAE VAE and HH-VAE).
and (b), respectively, that show the superiority our method. First, the hierarchical Gaussian model (fourth row) considerably improves the one-layered Gaussian alternative. Second, the HMC-based methods (two last rows) vastly improve the Gaussian methods. Third, in some specific cases, an extra improvement is added by HH-VAE with respect to the one-layered HMC-based model (columns 2, 4, 9, 11, and 13 Figure 6 (a) or columns 1, 2, 5 and 12 in (b)).
Conclusion
We presented HH-VAEM, to our knowledge, the first hierarchical VAE for mixed-type incomplete data that uses HMC with automatic hyper-parameter tuning for improved inference. We provide both quantitative and qualitative experiments that demonstrate its superiority with respect the baselines in the tasks of missing data imputation and supervised learning, placing HH- VAEM as a robust model for real-world datasets. Further, we have developed novel sampling based technique for dynamic feature selection that outperforms Gaussian-based alternatives and results an efficient method for active learning in deep generative models.
Acknowledgments and Disclosure of Funding
Ignacio Peis acknowledges support from Spanish government Ministerio Ciencia, Innovación y Universidades under grants FPU18/005 16, RTT2018 1000008--9955---11000 EST21/00467 and PID2021- i231820B-100 and from Comunidad de Madrid under grant Y2018/TCS-4705 PRACTICO-CM. José Miguel Hrnnández-Lobalte acknowledges support from Boltzbit.
10

---

References
[1] D. Barrejón, P. M. Olmos, and A. Ants-RRodriguz. Medical data wrangling with sequential Variational Autoencoders. arXiv preprint arXiv:2103.07206, 2021.
[2] Y. Bengio. Learning deep architectures for AI. Now Publishers Inc, 2009.
[3] J. M. Bernardo. Expected information as expected utility. the Annals of Statisticx, pages 686-690, 1979.
[4] M. Betancourt. conceptual introduction to Hamiltonian Monte Carlo. arXiv preprint arXiv:1701.02434 2017.
[5] M. Betancourt and M. Girolami. Hamiltonian Monte Carlo for hierarchical models. Current trends in Bayesian methodology with applications, 79(30):2 2015.
[6] A. Campbell, W. Chen, V. Stimper, J. M. Hernandez-Lobato, and Y. Zhang. A gradient Based Strategy for Hamiltonian Monte Carlo Hyperparameter optimization. In International Conference Machine Learning, pages 1238 1238-1248. PMLR, 2021.
[7] A. L. Caterini, A. Doucet, and Sejdinovic. Hamiltonian Variational Auto-Encoder. arXiv preprint arXiv:1805.. 11328, 2018.
[8] R. Child. Very deep VAEs generalize autoregressive models and can outperform them on images. arXiv preprint 1.0.x:20011.166530 2020.
[9] M. Collier, A. Nazabal, and C. K. Williams. VAEs in the presence of missing data. arXiv preprint wwXiv22006.05300 2020.
[10] C. Cremer, X. Li, and D. Duvenaud. Inference Suboptimality in Variational Autoencoders, In International Conference on Machine Learning, pages 1078-1086. PMLR, 2018
[11] D. Dua, C. Graff, et al. UCI machine learning repository. 2017.
[12] S. Duane, A. D. Kennedy, B. J. Pendleton, and D. Roweth. Hybrid Monte Carlo. Physics letters B, 195(2):216-222, 1987.
[13] S. Eduardo, A. Nazábal, C. K. Williams, and C. Sutton. Robust Variational Autoencoders for outlier detection and repair of mixed-type data. In International Conference on Artificial Intelligence and Statistics, pages 4056 4066. PMLR, 2020.
[14] M. Gamelo, D. Rosenbaum, C. Maddison, T. Ramalho, D. Saxton, M. Shanahan, Y. W. Teh, D. Rezende, and S. A. Eslami. Conditional Neural Processes. International Conference on Machine Learning, pages 1704-1713. PMLR, 2018.
[15] Z. Ghahramani and M. Jordan. Learning from incomplete data. 1995.
[16] M. Girolami and B. Calderhead, Riemann manifold langevin and hamiltonian monte carlo methods. Journal of the Royal Statistical Society: Series B (Statistical Methodology), 73(2):123- 214, 2011.
[17] W. Gong, Y. Li, and J. M. Hernández-Lobato. Sliced Kernelized Stein Discrepancy. arXiv preprint arXiv:2006. 16531, 2020.
[18] Y. Gong, H. Hajimirsadeghi, J. He. T. Durand, and G. Mori. Variational Selective Autoencoder: Learning from partially-observe heterogeneous data. In International Conference on Artificial Intelligence and Statistics, pages 2377-2385. PMLR, 2021.
[19] M. D. Hoffman. Learning deep latent Gaussian models with Markov Chain Monte Carlo. In International conference on machine learning, pages 1510-1519. PMLR, 2017.
[20] S.-J Huang. M. Xu, M.-K. Xie, M. Sugiyama, G. Niu, and S. Chen. Active feature acquisition with supervised matrix completion. Proceedings of the 24th ACM SIGKDD International Conference Knowledge Discovery & Data Mining, pages 1571-1579, 2018.
11

---

[21] N. Ipsen, P.-.. Mattei, and J. Frellsen. How to deal with missing data in supervised deep learning? In Artemiss-ICML Workshop on the Art of Learning with Missing Values, 2020.
[22] T. Joy, S. Schmon, P. Torr, S. Narayanaswamy, and T. Rainforth. Capturing label characteristics in vaes. In Proceedings of the ICLR Conference 2021. OpenReview, 2021.
[23] D. P. Kingma, T. Salimans, R. Jozefowicz, X. Chen, I. Sutskever, and M. Welling, Improved Variational Inference with Inverse Autoregressive Flow. Advances in neural information processing systems, 29:4743-4751,
[24] D. P. Kingma and M. Welling. Auto-encoding variational bayes. arXiv preprint arXiv: 2013.
[25] D. P. Kingma and M. Welling. An introduction Variational Autoencoders. arXiv preprint arXiv: 02691, 2019.
[26] A. Kraskov, H. Stögbauer, and P. Grassberger. Estimating mutual information. Physical review E, 69(6):066138, 2004.
[27] Y. LeCun. The MNIST database of handwritten digits. http://yann. lecun. com/exdb/nmistl/ 1998.
[28] Y. Li, J. Bradshaw, and Y. Sharma. Are generative classifiers more robust to adversarial attacks? In International Conference on Machine Learning, pages 3804 3814. PMLR, 2019.
[29] D. V. Lindley. On a measure of the information provided by an experiment. The Annals of Mathematical Statistics, pages 986-1005, 1956.
[30] R. J. Little and D. B. Rubin. Statistical analysis with missing data, volume 793. John Wiley & Sons, 2019.
[31] Q. Liu, Lee, and M. Jordan. A Kernelized Stein Discrepancy for goodness-of-fe tests. In International conference machine learning, pages 276-284 PMLR, 2016.
[32] Z. Liu, P. Luo, X. Wang, and X. Tang. Deep learning face attributes in the wild. In Proceedings of International Conference Computer Vision (ICCV), December 2015.
[33] C. Ma, S. Tschiatschek, M. Hernández-LLoatto, R. Turner, and C. Zhang. VAEM: a Deep Generative Model for Heterogeneous Mixed Type Data. arXiv preprint arXiv:2006. 1941, 2020.
[34] C. Ma, S. Tschiatschek, K. Palla, J. M. Hernández-Lobato, S. Nowozin, and C. Zhang. EdDI: Efficient Dynamic Discovery of High- Value Information with Partial VAE. arXiv preprint arXiv: / 142, 2018.
[35] L. Maaløe, M. Fraccaro, V. Liévin, and O. Winther. BIVA: A very deep hicrarchy of latent variables for generative modeling. arXiv preprint arXiv: 190../9022222.2 2019.
[36] P.-A Mattei and J. Frellsen. MIWAE:
Deep generative modelling and imputation of incomplete data sets. In International Conference on Machine Learning, pages 4413-4423, PMLR, 2019,
[37] P. Melville, M. Saar-Tsechansky, F. Provost, and R. Mooney. Active feature-value acquisition for classifier induction. In Fourth IEEE International Conference on Data Mining (ICDM '04), pages 483-486. IEEE, 2004.
[38] A. Nazabal, P. M. Olmos, Z. Ghahramani, and I. Valera. Handling Incomplete Heterogeneous Data using VAEs. Pattern Recognition, 107:107501, 2020.
[39] R. M. Neal et al. MCMC using Hamiltonian dynamics. Handbook of markov chain monte carlo, 2(11):2, 1.
[40] A. Razavi, A. v. Oord, Poole, and O. Vinyals. Preventing posterior collapse with delta- VAEs. arXiv preprint 191.:4190. 2019.
12

---

[41] D. J. Rezende, S. Mobamed, and D. Wierstra. Stochastic backpropagation and approximate inference deep generative models. International conference on machine learning, pages 1278-1286. PMLR, 2014.
[42] B. C. Ross. Mutual information between discrete and continuous data sets. PloS one, 92)xc87357, 2014.
[43] M. Saar-Tsechansky, P. Melville, and F. Provost. Active feature-value acquisition. Management Science, 55(4))666 2009.
[44] R. Salakhutdinov. Learning deep generative models. Annual Review of Statistics and Its Application, 2:361-385, 2015,
[45] R. Salakhutdinov and G. Hinton. Deep Boltzmann machines. In Artificial intelligence and statistics, pages 448 455. PMLR, 2009.
[46] Salimans, D. Kingma, and M. Welling. Markov Chain Monte Carlo and Variational Inference: Bridging the gap. In International Conference Machine Learning, pages 2015.
PMLR,
[47] M. Smieja, L. Struski, J. Tabor, B. Zieliñski, and P. Spurek. Processing of missing data by neural networks. Advances neural information processing systems, 31, 2018.
[48] C. K. Sønderby, T. Raiko, L. Maalge, S. K. Sønderby, and O. Winther. Ladder Variational Autoencoders. Advances in neural information processing systems, 29:3738-3746, 2016.
[49] M. Thahir, T. Sharma, and M. K. Ganapathiraju. An efficient heuristic method for active feature acquisition and application to protein- protein interaction prediction. In BMC proceedings, volume 6, pages 1-9. BioMed Central, 2012.
[50] A. Thin, N. Kotelevskii, A. Doucet, A. Durmus, E. Moulines, and M. Panov. Monte carlo variational auto-eneoders. In International Conference on Machine Learning, pages 10247- 10257. PMLR, 2021.
[51] V. Tresp, S. Ahmad, and R. Neuneier. Training neural networks with deficient data. Advances in neural information processing systems, 6, 1993.
[52] A. Vahdat and J. Kautz. NVAE: A Deep Hierarchical Variational Autoencoder. arXiv preprint arXiv:2007.00 038998 2020.
[53] Y. Wang and J. Cunningham. Posterior collapse and latent variable non-identifiabiiity. In Third Symposium on Advances in Approximate Bayesian Inference, 2020.
[54] H. Xiao, K. Rasul, and R. Vollgraf. Fushion-MNIST: novel image dataset for benchmarking machine learning algorithms. arXiv preprint arXiv:1708.077444 2017.
[55] C. Zhang, J. Bitepage, H. Kjellström, and S. Mandt. Advances in Variational Inference. IEEE Iransactions on pattern analysis and machine intelligence, (1)):2008-2026 2018.
13


Reviews:
[{"review_id": "pNoEBryaPKM", "reviewer": "Reviewer_Uewz", "summary": "The authors combine hierarchical VAEs and MCMC-within-VI, while emphasizing that the hierarchical aspect of VAEs is useful for dealing with missing data problems.", "questions": "1.  In lines 73-74, \"unobserved data is replaced with zeros\" and line 139, \"using zero-filling for the unobserved variables\"\n\nWhether or not zero imputation leads to lower bounds on observed data likelihoods  depends on assumptions on the missing data mechanism. For example, [31] make the missing at random (MAR) assumption. However, there is not a discussion of assumptions on misingness in your text. Since it is impossible to work with missing data without assumptions [see e.g. Tsiatis Semiparametrics and Missing Data], which assumptions do you make on missingness in your text?\n\nIn the experiments you describe :\n\n\"For all the models, we manually introduce missing data in the training set by randomly setting per data point a feature as missing with a probability sampled uniformly in the interval [0.01, 0.99] within each batch. Both the input data x and the target y can be missing. \n\nThis means the x dimensions are \"missing completely at random\". Which modifications to your method are necessary if some dimensions of x are missing conditional on y or on other dimensions of x? If you don't handle this case in your work, you should state that.\n\n\n2. For the predictive model ptheta(y|xhat, h), could you give a precise equation or sampling statement for xhat? \n\nI get roughly that you encode x (with some features possibly missing) into the per-dimension z's, and from those to h, then decode from h to z's, and finally \"each marginal z_d into xhat_u\".\n\nHow exactly does the last step \"each marginal z_d into xhat_u\" work? Does each z_d generate dimension d of xhat_u? For those x features that are observed, do you set xhat_d to x_d?\n\n3. (lines 219-223) \"Nonetheless, by using our reparameterized version, we are able to relax these dependencies and maintain the standard HMC method for obtaining samples that approximately follow the true posterior.\"\n\nHow was this evaluated/measured? Test NLLs of the resulting models (for x and for y) are a good goal in general for modeling but these NLLs do not necessarily show that inference worked in the way that is claimed.\n\n\n4.  You report test NLLs for missing features and for y, but how about held-out test NLL of the observed features? Whether or not it is also necessary to report NLLs on observed features again depends on the missingness assumptions. Also, possibly discrepancies between likelihoods on observed and missing data dimensions may depend on how hard the modeling tasks for those two sets of features are.\nSince you only implicitly described the missingness assumptions by describing the algorithm, it would be helpful for you to state what about your missingness assumptions and general data generation assumptions justifies the choice of reported metrics. Without further assumptions stated, I would wonder about held-out likelihood on observed data dimensions.\n\n5. What do you do with target prediction NLLs for data that has missing y? Exclude?\n                                                                                                                            \n\n6. The technical part of this paper seems to be about training (a variant of) hierarchical VAEs with MCMC and with a novel 2-part ELBO, rather than a specific paper on missingness (since missingness assumptions are not really discussed). On the other hand, the experiments highlight missingness. I would possibly suggest changing the title as well as revise the text to mention more missingness.\n\n\n7. A writing suggestion: never end sub-sections with an equation (e.g. sections 3.2 and 3.4). It helps to make a concluding remark.\n\nI am tentatively scoring \"weak accept\" due to good results but confusing discussion of assumptions and metrics. Glad to improve my score if authors sufficiently engage with these questions and revise the text.\n\n", "limitations": "Yes.", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strength:\n\nNovel combination of hierarchical VAEs (modeling flexibility) and MCMC-within-VI (inference quality) along with a new objective that addresses two distinct ELBOs of the model.\n\nWeaknesses: \n\nsome assumptions/scope not clear. See Questions.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "h1atDa_06Ic", "reviewer": "Reviewer_xZXW", "summary": "This work introduces HH-VAEM, a method for missing-feature imputation, learning and acquisition with heterogeneous data types.\nHH-VAEM is designed to improve the performance of the previous methods on the above tasks by combining the flexibility of hierarchical latent space model and the accuracy of the posterior inference with Hamiltonian MC (HMC).\n\nIn particular, HH-VAEM is a variant of VAEM, whose second stage is replaced with a hierarchical generative model trained with HMC decorated with minor tweaks previously proven to be effective.\nThe originality of the paper is found in i) the combination of VAEM, hierarchical model and HMC, and ii) the specific structure of the hierarchical model: the reparametrized autoregressive latent-space model.\n\nAfter introducing HH-VAEM, results of numerical experiments are presented to demonstrate quantifiable performance improvement brought by HH-VAEM.", "questions": "- What is the fundamental difficulty or non-triviality in the combination of hierarchical model and HMC, which is referred to in the last line of the first page?\n- To optimize the HMC hyperparameter, you need to differentiate Eq 9 with respect to $\\phi$, which requires higher-order derivatives (up to the L-th) of the generative density function since perturbation on the step size $\\phi$ propagates through the entire HMC chain. How do you implement such computation? If there is any approximation, how do you justify it?\n", "limitations": "The limitation is discussed only in terms of computational cost.\nI would suggest to add discussion on the extra hyperparameters introduced with the richer hierarchical model in terms of, e.g., the sensitivity.\n", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The strength of this work is that it achieves quantifiable and consistent performance improvements in the empirical experiments.\nSpecifically, the consistency of the results indicates the non-superficial effectiveness of the combination of the technical elements in HH-VAEM.\n\nThe weakness is the insufficiency in discussion and validation of the design choice of HH-VAEM, which reduces the significance of this work.\nMore specifically, performance improvement is rather trivial if one can properly combine richer model (i.e., hierarchical model) and more accurate inference method (i.e., HMC) provided sufficient amount of data.\nThus, the contribution of this paper should lie in the subtlety of the way they are combined such as the reparametrization trick for the hierarchical model.\nAlthough the authors briefly justify their design choices (e.g., reparametrization in terms of density landscape) upon the introduction of each technical element, none of them is formally validated in the experiment except stating that HH-VAEM is superior than rather random baselines.\nMore concretely, it would be nice if the baseline methods are organized more carefully for ablation study and the differences in the numbers are discussed in terms of the significance of each of the design choices of HH-VAEM.\n\nMinor issues:\n- Experimental detail is insufficient (or difficult to find) to reproduce the results. For example, what is the depth of hierarchical model for HH-VAEM?\n- Notational collision: L is used both for the length of HMC and the depth of the hierarchical model.\n- I did not get the point of Eq 5 when we have more informative expression Eq 7.\n- After the first equality of Eq 7, $y_O$ is missing in the numerator.\n- Eq 11 is broken.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "I_D05aNQ85o", "reviewer": "Reviewer_Mwck", "summary": "Uses Hierarchical VAE to impute missing heterogeneous data using HMC. Allows users to leverage a model with hierarchy and non Gaussians, which should lead to better modelling of the likelihood and generation.", "questions": "- The authors should add experiments focusing on high fidelity generation. Otherwise there doesn't seem to be much point in focussing on hierarchical VAEs if the results are presented on simple datasets. I'm aware that obtaining sota is often infeasible unless you work for FANG. But there should still be at least some effort to obtain results that match the motivation of the paper.\n- If the authors can add additional datasets (I certainly don't expect these in two weeks) then I'm of the opinion that this paper should be accepted.", "limitations": "Doesn't seem to be.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n- The idea is novel and the goal is relevant to the literature.\n- The approach is sound and suitable for the problem.\n- The paper is well written and contains the relevant literature. \n\nWeaknesses:\n- The main weakness is that the results are not conclusive as the datasets used are limited.\n- In the evaluation there doesn't seem to be a conclusion about why imputing data in hierarchal models is worth while? Hierarchical VAEs are renowned for high generation fidelity, so the authors should report results to highlight this.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "7rnrD0h7ZJj", "reviewer": "Reviewer_gfdz", "summary": "After rebuttal from authors:\nThanks for your replies, as well as the new updates/experiments introduced in the new version of the paper.\nConsidering these, I think this paper is now worth accepting to the conference, and I have updated my score accordingly.\n\n--------------------\n\n\nThis paper introduces the HH-VAEM, a VAE-based model for missing data imputation of heterogeneous data.\nThe HH-VAEM uses a hierarchical latent space to better model the data, and an HMC sampler to improve approximate inference. The authors show the good performances of the model in several different datasets and tasks.", "questions": "See questions in the \"Weaknesses\" section above.\n", "limitations": "As stated above, there should be more focus on the limitations of the HMC sampler.", "rating": 7, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "STRENGHTS\n1. While in many real-world applications data is missing and heterogeneous, much of the literature considers datasets containing fully observed data points of the same type. This paper, on the other hand, provides a novel solution to the problem of imputing and acquiring mixed data. \n\n2.  Experiments show that the model outperforms competing methods on a wide number of datasets.\n\n3. The paper is overall well written.\n\n4. While the presented method is quite complex in its usage of the HMC sampler and the SKSD discrepancy measure, the authors make this model more accessible by focusing on automatic tuning of the step size parameters, and providing code.\n\n\nWEAKNESSES\n1. As opposed to standard VAE training this method is much more complex to understand and implement, and this might limit the impact of this work in the Neurips community.\nGiven that the authors shared the code this would not be a problem, as long as the method was \"plug and play\". However, even conidering the effort made by the authors to make it simple to use, I am not convinced it is.\nHMC is notoriously hard to use in highly correlated high dimensional spaces, since it is hard to tune the multiple important parameters, and it is also hard to reliably diagnose when the sampler is not mixing properly.\nThe paper is missing a more detailed discussion on practical considerations on the HMS sampler, for example:\n* What are the pitfalls of the HMC sampler? How can one detect a poorly mixing sampler?\n* What is the rejection rate of the sampler over the different training epochs? A too low rejection rate might show that the learned step sizes are too small and the sampler is therefore not exploring the space as it should.\n* How robust is the method to the choice of the HMC hyperparameters (L, T, initialization for step sizes)\n\n2. Due to the HMC sampler, the training time of the model is much higher than competing methods. The better performances might still justify the overhead in practical applications. However one doubt I have is whether the baselines for non-HMC methods are fair. Have you tried for example using for the standard VAEs models a much higher number of samples to match the computational times? And also for example the IWAE objective from Burda et al, 2015?\n\n\nMinor comments.\n* Typo in line 189: no->not\n* line 301 -> what is HH-VAE?\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "40ujMTDhcAl", "reviewer": "Reviewer_jW2a", "summary": "The authors propose a new way of doing generative modeling  with an incomplete (potentially labelled) heterogeneous data set. Dealing generatively with heterogeneous and/or incomplete data is an important open problem, for which few solutions exist today.\n\nTheir approach is to construct a generative model based on a new kind of hierarchical VAE (HVAE) architecture. The model is trained via a blend of variational inference and Hamiltonian Monte Carlo (HMC).\n\nAfter training, they mainly look at two specific applications: imputation and active learning of the missing values. Regarding active learning, they propose a new approach to estimate the mutual information using HMC samples, improving on the previous approach of Ma et al. (2019) to using VAEs for active learning of the missing values.\n\nThey perform a series of experiments that indicate that their HVAE outperforms other VAE-type models for heterogeneous data, for imputation, prediction of the labels, active learning, and anomaly detection.\n\n\nMa et al., EDDI: Efficient Dynamic Discovery of High-Value Information with Partial VAE, ICML 2019", "questions": "My questions are contained in the \"Weaknesses\" section of my review.", "limitations": "I do not see potential negative societal impact that this work could create.\n\nAbout the limitations, I think the authors should discuss the assumptions necessary for their method to work. In particular, the fact that they consider the likelihood of the observed data suggests that their method requires a missing at random assumption. This should be discussed. On the other hand, I really like that the authors include computing times.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "STRENGTHS\n\nThe paper has many clever contributions. I list and praise them below.\n\n1) The idea called \"predictive enhancement\" by the authors is a quite smart blend of generative and discriminative ideas. Indeed $p(x,y)$ is modeled as $\\int p(z)p(x|z)p(y|z,x)dz$, where $p(x|z)$ is parametrised by a generative network, and $p(y|z,x)$ by a discriminative one.\n\n2) The HVAE architure proposed by the authors is quite different from standard ones, as it models directly the \"noise\" variables implied in the reparametrisation trick. This in particular allows the authors to avoid complex inference network architectures used in other HVAE papers.\n\n3) The authors propose a new way to perform HMC in a VAE context, by focusing on the \"noise\" variables $\\epsilon$, which do not depend on any parameters, and then doing variational inference where the proposal for $\\epsilon$ is obtained through HMC. Because of HMC, the likelihood lower bound can't be estimated unbiasedly, so they use a few tricks to transform it into something that can be optimised.\n\n4) I like the extended scope of the experiments, even if I have a few issues with some experimental details.\n\n\nWEAKNESSES\n\nThe main weakness of the paper is that these individual contributions are described too hastily, and without much discussing of the motivations, what is new, what is not, and what is inspired by other work. More importantly, even though several interesting ablation studies are done, it remains unclear what contributions bring the most to the table. \n\na) It is not clear if there is anything new in Section 3.2, compared to Ma et al. (2020, cited as [28]). If there isn't, why include this here rather than in the \"related work\" section?\n\nb) As I mentioned above, the \"predictive enhancement\" idea is quite clever, and likely to improve the quality of the predictions substantially. I find it however quite odd not discussing more its motivations or related works. In this missing-data context, using a generative model combined with a discriminative one goes back at least to the work of Tresp et al. (1994) and Ghahramani and Jordan (1995), who used a GMM for $p(x)$ instead of a VAE. In a deep learning context, this idea was also explored by Smieja (2018) and by and Ipsen et al. (2022). Supervised VAE models have also used similar factorisations, for instance Joy et al. (2021) essentially uses a factorisation of type $p(z)p(x|z)p(y|z)$, and the DFZ model of Li et al. (2019) uses the same factorisation as you, i.e. $p(z)p(x|z)p(y|x,z)$.\n\nc) The small change in HVAE architecture is poorly motivated (Section 3.4). Many claims here are vague and not really supported by theory or evidence, e.g. \"However, by choosing this reparameterization, we relax the dependencies among the latent variables, resulting in a smoother joint posterior density p(ϵ|xO, yO)\" (what does smoother means here ? Maybe doing a toy experiment would help ?). Is it only useful for HMC or does it also change something for variational inference? How does it relate to usual tricks for HVAEs?\n\nd) I must admit I do not understand the rationale behind the modification of the ELBO proposed in Section 3.5. The problematic term $H(q_\\phi)$ is simply removed while the variance of $q_\\phi$ is artificially inflated (the authors do not specify exactly how) by a hyperparameter $\\mathbf{s}$. This $\\mathbf{s}$ is fit by minimising a divergence (the Sliced Kernelized Stein Discrepancy) between the true posterior and $q_\\phi$. This trick needs to be more properly explained, and maybe illustrated on a simple example. How does this trick relates to other methods that have been used to do HMC in a VAE context, for instance Caterini et al. (2018 [8]) or Hoffman (2017 [19]) ?\n\ne) While nicely motivated, I think that the description of the mutual information estimate of Equation (13) is a bit unclear: which samples are used ? how do you create the bins? What are the properties of the estimate?\n\nI also have a few issues with the imputation experiments:\n\nf) To evaluate the quality of the imputations, you approximate the test log likelihood of the missing values. Equation (14) reads $ \\log p(x_U | x_O) = \\log E_{\\epsilon \\sim q} p(x_U | \\epsilon) \\approx log \\frac{1}{k} \\sum_{i=1}^k p(x_U | \\epsilon_i)$,\nbut the first part of the equation is false because $q$ is not equal to the true posterior. I think this evaluation protocol is sensible, but requires a lot of approximations that can be hard to grasp (for instance, it is possible that the models trained with HMC are not really \"better\", but leads to posteriors that are better approximated by HMC, so a HMC-based evaluation of non-HMC-based models may be seen as a bit unfair). I think it would be nice to keep this evaluation method, but maybe complement it with something simpler, for instance the imputation error (MSE or accuracy). Another nice thing with the imputation error is that it allows you to easily add non-VAE baselines, which would add a lot of information. For instance, for heterogeneous data, the missforest R package (based on Stekhoven and Buehlmann, 2012) is still, in my opinion, easy to use and very difficult to beat. It would be interesting to see how it compares to VAE-like models.\n\ng) Moreover, as you rightfully claim in the beginning of the paper, in an heterogeneous context, \"the problem of handling unbalanced likelihoods leads to the domination of some dimensions during the optimization process\". This means that the full likelihood can be a bit misleading, and showing feature-specific scores (maybe in the appendix) may be interesting.\n\nMINOR ISSUES\n\n\n\n- There are a few issues in the references. Most of them are in a numeric format, but some are not, e.g. (Betancourt et al., 2015) line 165. In multiple cases of published papers, the arxived versions are cited instead of the published ones (e.g. Hendrycks and Dietterich was published at ICLR 2019, the two papers of Ma et al. were published at ICML 2019 and NeurIPS 2020, Maaløe et al. was published at NeurIPS 2019, Caterini et al. was published at NeurIPS 2019).\n\n- sksd is never properly defined\n\n\n\nADDITIONAL REFERENCES\n\nGhahramani and Jordan. Learning from incomplete data. MIT Center for Biological and Computational Learning Technical Report 108, 1995.\nJoy et al., Capturing Label Characteristics in VAEs, ICLR 2021\nLi et al. Are Generative Classifiers More Robust to Adversarial Attacks?, ICML 2019\nTresp et al. Training neural networks with de\fcient data. NeurIPS 1994\nSmieja et al., Processing of missing data by neural networks, NeurIPS 2018\nIpsen et al., How to Deal with Missing Data in Supervised Deep Learning, ICLR 2022\nStekhoven and Buehlmann, MissForest - nonparametric missing value imputation\nfor mixed-type data, Bioinformatics, 2012\n\n\n\n\n---------------- POST-REBUTTAL EDIT ------------------\n\nMany thanks for this thorough revision. A lot of these updates are significant, in particular the additions of new experiments (more proper metrics, adding missforest, doing impaining), and the clarifications of the active learning strategies.\n\nMy last remaining significant concern is the lack of strong justification for the HMC tricks that the authors use (the variance inflation, the SKSD). I appreciate the additional details given by the authors though, and I understand the general rationale.\n\nOverall, I am updating my score to 6, since I think this paper would be a valuable fit for NeurIPS.\n\n\n\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept