Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: The authors propose a new way of doing generative modeling  with an incomplete (potentially labelled) heterogeneous data set. Dealing generatively with heterogeneous and/or incomplete data is an important open problem, for which few solutions exist today.

Their approach is to construct a generative model based on a new kind of hierarchical VAE (HVAE) architecture. The model is trained via a blend of variational inference and Hamiltonian Monte Carlo (HMC).

After training, they mainly look at two specific applications: imputation and active learning of the missing values. Regarding active learning, they propose a new approach to estimate the mutual information using HMC samples, improving on the previous approach of Ma et al. (2019) to using VAEs for active learning of the missing values.

They perform a series of experiments that indicate that their HVAE outperforms other VAE-type models for heterogeneous data, for imputation, prediction of the labels, active learning, and anomaly detection.


Ma et al., EDDI: Efficient Dynamic Discovery of High-Value Information with Partial VAE, ICML 2019
strengths_and_weaknesses: STRENGTHS

The paper has many clever contributions. I list and praise them below.

1) The idea called "predictive enhancement" by the authors is a quite smart blend of generative and discriminative ideas. Indeed $p(x,y)$ is modeled as $\int p(z)p(x|z)p(y|z,x)dz$, where $p(x|z)$ is parametrised by a generative network, and $p(y|z,x)$ by a discriminative one.

2) The HVAE architure proposed by the authors is quite different from standard ones, as it models directly the "noise" variables implied in the reparametrisation trick. This in particular allows the authors to avoid complex inference network architectures used in other HVAE papers.

3) The authors propose a new way to perform HMC in a VAE context, by focusing on the "noise" variables $\epsilon$, which do not depend on any parameters, and then doing variational inference where the proposal for $\epsilon$ is obtained through HMC. Because of HMC, the likelihood lower bound can't be estimated unbiasedly, so they use a few tricks to transform it into something that can be optimised.

4) I like the extended scope of the experiments, even if I have a few issues with some experimental details.


WEAKNESSES

The main weakness of the paper is that these individual contributions are described too hastily, and without much discussing of the motivations, what is new, what is not, and what is inspired by other work. More importantly, even though several interesting ablation studies are done, it remains unclear what contributions bring the most to the table. 

a) It is not clear if there is anything new in Section 3.2, compared to Ma et al. (2020, cited as [28]). If there isn't, why include this here rather than in the "related work" section?

b) As I mentioned above, the "predictive enhancement" idea is quite clever, and likely to improve the quality of the predictions substantially. I find it however quite odd not discussing more its motivations or related works. In this missing-data context, using a generative model combined with a discriminative one goes back at least to the work of Tresp et al. (1994) and Ghahramani and Jordan (1995), who used a GMM for $p(x)$ instead of a VAE. In a deep learning context, this idea was also explored by Smieja (2018) and by and Ipsen et al. (2022). Supervised VAE models have also used similar factorisations, for instance Joy et al. (2021) essentially uses a factorisation of type $p(z)p(x|z)p(y|z)$, and the DFZ model of Li et al. (2019) uses the same factorisation as you, i.e. $p(z)p(x|z)p(y|x,z)$.

c) The small change in HVAE architecture is poorly motivated (Section 3.4). Many claims here are vague and not really supported by theory or evidence, e.g. "However, by choosing this reparameterization, we relax the dependencies among the latent variables, resulting in a smoother joint posterior density p(ϵ|xO, yO)" (what does smoother means here ? Maybe doing a toy experiment would help ?). Is it only useful for HMC or does it also change something for variational inference? How does it relate to usual tricks for HVAEs?

d) I must admit I do not understand the rationale behind the modification of the ELBO proposed in Section 3.5. The problematic term $H(q_\phi)$ is simply removed while the variance of $q_\phi$ is artificially inflated (the authors do not specify exactly how) by a hyperparameter $\mathbf{s}$. This $\mathbf{s}$ is fit by minimising a divergence (the Sliced Kernelized Stein Discrepancy) between the true posterior and $q_\phi$. This trick needs to be more properly explained, and maybe illustrated on a simple example. How does this trick relates to other methods that have been used to do HMC in a VAE context, for instance Caterini et al. (2018 [8]) or Hoffman (2017 [19]) ?

e) While nicely motivated, I think that the description of the mutual information estimate of Equation (13) is a bit unclear: which samples are used ? how do you create the bins? What are the properties of the estimate?

I also have a few issues with the imputation experiments:

f) To evaluate the quality of the imputations, you approximate the test log likelihood of the missing values. Equation (14) reads $ \log p(x_U | x_O) = \log E_{\epsilon \sim q} p(x_U | \epsilon) \approx log \frac{1}{k} \sum_{i=1}^k p(x_U | \epsilon_i)$,
but the first part of the equation is false because $q$ is not equal to the true posterior. I think this evaluation protocol is sensible, but requires a lot of approximations that can be hard to grasp (for instance, it is possible that the models trained with HMC are not really "better", but leads to posteriors that are better approximated by HMC, so a HMC-based evaluation of non-HMC-based models may be seen as a bit unfair). I think it would be nice to keep this evaluation method, but maybe complement it with something simpler, for instance the imputation error (MSE or accuracy). Another nice thing with the imputation error is that it allows you to easily add non-VAE baselines, which would add a lot of information. For instance, for heterogeneous data, the missforest R package (based on Stekhoven and Buehlmann, 2012) is still, in my opinion, easy to use and very difficult to beat. It would be interesting to see how it compares to VAE-like models.

g) Moreover, as you rightfully claim in the beginning of the paper, in an heterogeneous context, "the problem of handling unbalanced likelihoods leads to the domination of some dimensions during the optimization process". This means that the full likelihood can be a bit misleading, and showing feature-specific scores (maybe in the appendix) may be interesting.

MINOR ISSUES



- There are a few issues in the references. Most of them are in a numeric format, but some are not, e.g. (Betancourt et al., 2015) line 165. In multiple cases of published papers, the arxived versions are cited instead of the published ones (e.g. Hendrycks and Dietterich was published at ICLR 2019, the two papers of Ma et al. were published at ICML 2019 and NeurIPS 2020, Maaløe et al. was published at NeurIPS 2019, Caterini et al. was published at NeurIPS 2019).

- sksd is never properly defined



ADDITIONAL REFERENCES

Ghahramani and Jordan. Learning from incomplete data. MIT Center for Biological and Computational Learning Technical Report 108, 1995.
Joy et al., Capturing Label Characteristics in VAEs, ICLR 2021
Li et al. Are Generative Classifiers More Robust to Adversarial Attacks?, ICML 2019
Tresp et al. Training neural networks with decient data. NeurIPS 1994
Smieja et al., Processing of missing data by neural networks, NeurIPS 2018
Ipsen et al., How to Deal with Missing Data in Supervised Deep Learning, ICLR 2022
Stekhoven and Buehlmann, MissForest - nonparametric missing value imputation
for mixed-type data, Bioinformatics, 2012




---------------- POST-REBUTTAL EDIT ------------------

Many thanks for this thorough revision. A lot of these updates are significant, in particular the additions of new experiments (more proper metrics, adding missforest, doing impaining), and the clarifications of the active learning strategies.

My last remaining significant concern is the lack of strong justification for the HMC tricks that the authors use (the variance inflation, the SKSD). I appreciate the additional details given by the authors though, and I understand the general rationale.

Overall, I am updating my score to 6, since I think this paper would be a valuable fit for NeurIPS.




questions: My questions are contained in the "Weaknesses" section of my review.
limitations: I do not see potential negative societal impact that this work could create.

About the limitations, I think the authors should discuss the assumptions necessary for their method to work. In particular, the fact that they consider the likelihood of the observed data suggests that their method requires a missing at random assumption. This should be discussed. On the other hand, I really like that the authors include computing times.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: The authors combine hierarchical VAEs and MCMC-within-VI, while emphasizing that the hierarchical aspect of VAEs is useful for dealing with missing data problems.
strengths_and_weaknesses: Strength:

Novel combination of hierarchical VAEs (modeling flexibility) and MCMC-within-VI (inference quality) along with a new objective that addresses two distinct ELBOs of the model.

Weaknesses: 

some assumptions/scope not clear. See Questions.
questions: 1.  In lines 73-74, "unobserved data is replaced with zeros" and line 139, "using zero-filling for the unobserved variables"

Whether or not zero imputation leads to lower bounds on observed data likelihoods  depends on assumptions on the missing data mechanism. For example, [31] make the missing at random (MAR) assumption. However, there is not a discussion of assumptions on misingness in your text. Since it is impossible to work with missing data without assumptions [see e.g. Tsiatis Semiparametrics and Missing Data], which assumptions do you make on missingness in your text?

In the experiments you describe :

"For all the models, we manually introduce missing data in the training set by randomly setting per data point a feature as missing with a probability sampled uniformly in the interval [0.01, 0.99] within each batch. Both the input data x and the target y can be missing. 

This means the x dimensions are "missing completely at random". Which modifications to your method are necessary if some dimensions of x are missing conditional on y or on other dimensions of x? If you don't handle this case in your work, you should state that.


2. For the predictive model ptheta(y|xhat, h), could you give a precise equation or sampling statement for xhat? 

I get roughly that you encode x (with some features possibly missing) into the per-dimension z's, and from those to h, then decode from h to z's, and finally "each marginal z_d into xhat_u".

How exactly does the last step "each marginal z_d into xhat_u" work? Does each z_d generate dimension d of xhat_u? For those x features that are observed, do you set xhat_d to x_d?

3. (lines 219-223) "Nonetheless, by using our reparameterized version, we are able to relax these dependencies and maintain the standard HMC method for obtaining samples that approximately follow the true posterior."

How was this evaluated/measured? Test NLLs of the resulting models (for x and for y) are a good goal in general for modeling but these NLLs do not necessarily show that inference worked in the way that is claimed.


4.  You report test NLLs for missing features and for y, but how about held-out test NLL of the observed features? Whether or not it is also necessary to report NLLs on observed features again depends on the missingness assumptions. Also, possibly discrepancies between likelihoods on observed and missing data dimensions may depend on how hard the modeling tasks for those two sets of features are.
Since you only implicitly described the missingness assumptions by describing the algorithm, it would be helpful for you to state what about your missingness assumptions and general data generation assumptions justifies the choice of reported metrics. Without further assumptions stated, I would wonder about held-out likelihood on observed data dimensions.

5. What do you do with target prediction NLLs for data that has missing y? Exclude?
                                                                                                                            

6. The technical part of this paper seems to be about training (a variant of) hierarchical VAEs with MCMC and with a novel 2-part ELBO, rather than a specific paper on missingness (since missingness assumptions are not really discussed). On the other hand, the experiments highlight missingness. I would possibly suggest changing the title as well as revise the text to mention more missingness.


7. A writing suggestion: never end sub-sections with an equation (e.g. sections 3.2 and 3.4). It helps to make a concluding remark.

I am tentatively scoring "weak accept" due to good results but confusing discussion of assumptions and metrics. Glad to improve my score if authors sufficiently engage with these questions and revise the text.


limitations: Yes.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: Uses Hierarchical VAE to impute missing heterogeneous data using HMC. Allows users to leverage a model with hierarchy and non Gaussians, which should lead to better modelling of the likelihood and generation.
strengths_and_weaknesses: Strengths:
- The idea is novel and the goal is relevant to the literature.
- The approach is sound and suitable for the problem.
- The paper is well written and contains the relevant literature. 

Weaknesses:
- The main weakness is that the results are not conclusive as the datasets used are limited.
- In the evaluation there doesn't seem to be a conclusion about why imputing data in hierarchal models is worth while? Hierarchical VAEs are renowned for high generation fidelity, so the authors should report results to highlight this.
questions: - The authors should add experiments focusing on high fidelity generation. Otherwise there doesn't seem to be much point in focussing on hierarchical VAEs if the results are presented on simple datasets. I'm aware that obtaining sota is often infeasible unless you work for FANG. But there should still be at least some effort to obtain results that match the motivation of the paper.
- If the authors can add additional datasets (I certainly don't expect these in two weeks) then I'm of the opinion that this paper should be accepted.
limitations: Doesn't seem to be.
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: This work introduces HH-VAEM, a method for missing-feature imputation, learning and acquisition with heterogeneous data types.
HH-VAEM is designed to improve the performance of the previous methods on the above tasks by combining the flexibility of hierarchical latent space model and the accuracy of the posterior inference with Hamiltonian MC (HMC).

In particular, HH-VAEM is a variant of VAEM, whose second stage is replaced with a hierarchical generative model trained with HMC decorated with minor tweaks previously proven to be effective.
The originality of the paper is found in i) the combination of VAEM, hierarchical model and HMC, and ii) the specific structure of the hierarchical model: the reparametrized autoregressive latent-space model.

After introducing HH-VAEM, results of numerical experiments are presented to demonstrate quantifiable performance improvement brought by HH-VAEM.
strengths_and_weaknesses: The strength of this work is that it achieves quantifiable and consistent performance improvements in the empirical experiments.
Specifically, the consistency of the results indicates the non-superficial effectiveness of the combination of the technical elements in HH-VAEM.

The weakness is the insufficiency in discussion and validation of the design choice of HH-VAEM, which reduces the significance of this work.
More specifically, performance improvement is rather trivial if one can properly combine richer model (i.e., hierarchical model) and more accurate inference method (i.e., HMC) provided sufficient amount of data.
Thus, the contribution of this paper should lie in the subtlety of the way they are combined such as the reparametrization trick for the hierarchical model.
Although the authors briefly justify their design choices (e.g., reparametrization in terms of density landscape) upon the introduction of each technical element, none of them is formally validated in the experiment except stating that HH-VAEM is superior than rather random baselines.
More concretely, it would be nice if the baseline methods are organized more carefully for ablation study and the differences in the numbers are discussed in terms of the significance of each of the design choices of HH-VAEM.

Minor issues:
- Experimental detail is insufficient (or difficult to find) to reproduce the results. For example, what is the depth of hierarchical model for HH-VAEM?
- Notational collision: L is used both for the length of HMC and the depth of the hierarchical model.
- I did not get the point of Eq 5 when we have more informative expression Eq 7.
- After the first equality of Eq 7, $y_O$ is missing in the numerator.
- Eq 11 is broken.

questions: - What is the fundamental difficulty or non-triviality in the combination of hierarchical model and HMC, which is referred to in the last line of the first page?
- To optimize the HMC hyperparameter, you need to differentiate Eq 9 with respect to $\phi$, which requires higher-order derivatives (up to the L-th) of the generative density function since perturbation on the step size $\phi$ propagates through the entire HMC chain. How do you implement such computation? If there is any approximation, how do you justify it?

limitations: The limitation is discussed only in terms of computational cost.
I would suggest to add discussion on the extra hyperparameters introduced with the richer hierarchical model in terms of, e.g., the sensitivity.
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: After rebuttal from authors:
Thanks for your replies, as well as the new updates/experiments introduced in the new version of the paper.
Considering these, I think this paper is now worth accepting to the conference, and I have updated my score accordingly.

--------------------


This paper introduces the HH-VAEM, a VAE-based model for missing data imputation of heterogeneous data.
The HH-VAEM uses a hierarchical latent space to better model the data, and an HMC sampler to improve approximate inference. The authors show the good performances of the model in several different datasets and tasks.
strengths_and_weaknesses: STRENGHTS
1. While in many real-world applications data is missing and heterogeneous, much of the literature considers datasets containing fully observed data points of the same type. This paper, on the other hand, provides a novel solution to the problem of imputing and acquiring mixed data. 

2.  Experiments show that the model outperforms competing methods on a wide number of datasets.

3. The paper is overall well written.

4. While the presented method is quite complex in its usage of the HMC sampler and the SKSD discrepancy measure, the authors make this model more accessible by focusing on automatic tuning of the step size parameters, and providing code.


WEAKNESSES
1. As opposed to standard VAE training this method is much more complex to understand and implement, and this might limit the impact of this work in the Neurips community.
Given that the authors shared the code this would not be a problem, as long as the method was "plug and play". However, even conidering the effort made by the authors to make it simple to use, I am not convinced it is.
HMC is notoriously hard to use in highly correlated high dimensional spaces, since it is hard to tune the multiple important parameters, and it is also hard to reliably diagnose when the sampler is not mixing properly.
The paper is missing a more detailed discussion on practical considerations on the HMS sampler, for example:
* What are the pitfalls of the HMC sampler? How can one detect a poorly mixing sampler?
* What is the rejection rate of the sampler over the different training epochs? A too low rejection rate might show that the learned step sizes are too small and the sampler is therefore not exploring the space as it should.
* How robust is the method to the choice of the HMC hyperparameters (L, T, initialization for step sizes)

2. Due to the HMC sampler, the training time of the model is much higher than competing methods. The better performances might still justify the overhead in practical applications. However one doubt I have is whether the baselines for non-HMC methods are fair. Have you tried for example using for the standard VAEs models a much higher number of samples to match the computational times? And also for example the IWAE objective from Burda et al, 2015?


Minor comments.
* Typo in line 189: no->not
* line 301 -> what is HH-VAE?

questions: See questions in the "Weaknesses" section above.

limitations: As stated above, there should be more focus on the limitations of the HMC sampler.
review_title: null

Meta-review: metareview: Thanks to the authors for this submission.  The reviewers agreed that this work presented an interesting and novel combination of techniques to achieve good imputation results.  The reviewers also agree that the author-response and subsequent revisions have improved the submission and addressed the vast majority of reviewer concerns — in fact most reviewers increased their scores throughout the process.  I believe this work is technically sound and of interest to the broader community.  
confidence: Less certain
Decision: Accept