Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The paper addresses the problem of predictive modelling with missing input features. The authors formulate the problem as a latent variable model, and in addition to the standard variational lower bound (ELBO) propose to use a variational upper bound based on CUBO (Dieng at al 2017) modified by an exponential divergence to solve the MC estimation of CUBO. They further propose surrogate parametrization to reduce the variance in the gradients. The experimental evaluation over standard regression UCI datasets with randomly dropped features shows marginal improvements over existing baselines.
main_review: (+) pros / (-) cons
-------------------

(+) Predictive modelling over inputs with missing features is an important problem arising often in many application domains. This paper contributes this somewhat underexplored field.

(-) The rather low documented performance benefits over simpler baselines do not justify the use of the complex model (combining 5? networks) proposed here as opposed to simpler VAE or CVAE variants.

(+) The method and the various bounds introduced are mathematically intriguing, well motivated and potentially useful in follow-up research, however, ... 
(-) the paper is difficult to  follow and at places the reader is left guessing what the authors meant. This should be improved. Concretely
1. last para of section 2 - the optimization of negative L "is relatively difficult". Why? What makes it difficult?
2. last para of section 2 - "... have been no equivalent of VI ... " What about CUBO and its variants pick up from in your work? These have some specific flaws for which they do not qualify here?
3. before equation 3 - "exponential divergence". You mean the Bregman exponential divergence? A citation to help the reader?
4. $p(u, z | \theta)$ in equations (11) and (12) seem to use the same parameters $\theta$ though for (11) $u = (y, \tilde{x})$ and for (12) it is $u = \tilde{x}$. Is this in practice the same network with two outputs?
5. But then in equation (15) these use different $z_{\theta}$ and $z_{\psi}$ samples. How is this designed and trained in practice?
6. page 5 - clarify notation for and explain the gain function; what is the intuition / purpose for it?
7. Def 1 - effective parameters are those with gradient zero. ".. i.e. the set of parameters inducing tight variational approximation." How zero gradient achieves this in a complex non-convex problem, i.e. can't this be a local non-tight extremum?
8. page 7 DVAE/ DVAE* - you say these are MNAR and MCAR model variants as in Collier at al. 2020. Can you clarify how these translate into your rather more complex model formulation and what specifically changes in the loss (especially the EUBO part)?

Further questions for clarification/discussion
-----------------------------------------------

1) Why do you condition $y$ and $x$ on $m$ in equation (1). These are the complete $x$ data so they should not depend on the masking so that $p(y, x | m) = p(y, x)$. Or is this not true? Or is it the $y$ that depends on $m$? Or is it rather the $m$ which depends on $x$? (As in some values being more likely to be masked?)

2) You introduce two latent variable models in equations (7) and (8). My understanding is that the latent $z$ is shared as (8) is just a marginalization of (7) over $y$. You then formulate to approximate posteriors $q(z| y, \tilde{x})$ and $q(z| \tilde{x})$ the first learned through ELBO maximization, the 2nd through EUBO minimization. There is currently no link between the two (approximate) posteriors. Would it make sense to somehow link them? (Sorry, I don't know how and may not be possible, or not easily.)

3) You used the Bayes rule to decompose the predictive conditional log likelihood into two terms in equation (4) of which one you are bounding from the bottom (ELBO) and the oher from top (EUBO). What is the effect on the predictive conditional $p(y | x)$? Is it somehow sandwiched or not really due to splitting and modelling the two non-conditional log likelihoods separately? 


Minor text problems / typos
---------------------------

1. This first proposition in in page 6 is numbered 2 (not 1) - confusing
summary_of_the_review: The paper contributes to practically very important yet relatively little explored area of research - that of predictive modelling with missing data. The proposed method is rather complex, composed of multiple steps adding onto each other to solve a problem arising in the previous steps. These are all well motivated, however, overall the current presentation of the method is difficult to follow and should be improved to help the reader (see main review). Moreover, the documented performance benefits seem to be rather little to justify the use of such a complex method over simpler baseline. These two (lack of clarity, low performance given the complexity of method) are for me the reasons not to consider the paper for this conference.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The paper considers the problem of prediction with missing (incomplete) features. The authors propose a class of generative models that includes missingness of features, and develop a discriminative learning algorithm that maximises the conditional (posterior) log-likelihood of the training data approximately. Experiments show that the method is competitive compared with existing approaches, and in particular with approaches based on VAEs. 
main_review: The ab-initio generative model class proposed by the authors for handling predictions with missing features is convincing. It has the advantage that the involved distributions are simple (factorising), however at the price of introducing latent variables. To learn it discriminatively requires to maximise a difference of concave functions. The first term is lower bounded by ELBO as in VAEs. The second term requires a tractable upper bound. The authors develop a  novel upper bound (starting from alpha-Rényi divergence) that admits a stochastic gradient estimator.  They further introduce a data dependent surrogate reparametrisation in order to achieve an estimator with low variance. The technical part of the paper is concisely written and correct. 

The authors prove that the transformation used for the reparametrisation preserves the effective parameter subset, i.e. the subset of parameter combinations for which the overall bound is tight. This is indeed a desirable property, but is in my view not sufficient. The reason is, that this effective subset can be very small and cover a subset of simple models only. Moreover, there is no guarantee that the respective gap will become small during learning.

The experimental section first analyses the learning properties of the method in an ablation study. The authors then show competitiveness of their method by comparing it with existing approaches on a subset of tasks taken from the  UCI Machine Learning Repository. The description of the experiments is clear and reproducible. The experiments are however not fully convincing w.r.t. the scalability of the approach. All networks used for the model and bound construction have only one fully connected hidden layer. This seems to be sufficient for the considered tasks from the  UCI repository. However, this would be not sufficient e.g. for image classification tasks where the involved networks are usually deep CNNs.

Further comments:
- You mention earlier works  (Ghahramani & Jordan, 1994; Smola et al., 2005), noting that their applicability is restricted to exponential families. Please explain whether these approaches are / are not applicable for the model class analysed in your work. As I understand it, the models p(y,x,z|m) considered by you are exponential families, but of course after marginalising over z, the resulting mixture model p(y,x|m) is not any more. It remains however unclear to me, whether a DCA (difference of convex functions algorithm) for learning p(y|x,z,m) can be somehow generalised for learning p(y|x,m).

- I would suggest to drop the data instance superscript earlier in the text, e.g. starting from subsection 2.2. the latest. This would in my view improve readability and reduce clutter.
summary_of_the_review: The conceptual part, i.e. the model and the proposed learning approach are in my view concise and sufficiently novel. This outweighs the missing scalability analysis in the experimental part. I would however expect the authors to clearly address the raised conceptual questions.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The authors propose a new method, DIG, for discriminative tasks with missing input features. It uses latent variable models to marginalize out the label and the missing part of the features given the latent variable, in order to compute the objective, the conditional log likelihood of the label given the corrupted features. As this objective is intractable, the paper builds a conditional evidence lower bound (CELBO) that can be unbiasedly approximated using Monte Carlo samples. CELBO consists of the regular ELBO as the lower bound for the log joint probability of the label and the observed features, and an evidence upper bound (EUBO) that bounds the log marginal probability of the observed features. The derivation of EUBO involves the alpha-renyi divergence and the exponential divergence. The stochastic CELBO contains a density ratio that can lead to large variance in the stochastic gradients during optimization, so the authors propose a surrogate parameterization to bound the gradient norm. Experiments on real datasets justify the effectiveness of the variational approximations to stabilize the optimization. When compared with VAE, CVAE, and MICE, the DIG algorithm shows better or comparable predictive performance and robustness against feature corruption.
main_review: **Strengths**
The paper is overall clearly written. The issue it tries to solve, discriminative tasks with missing input features, has great impact for a wide range of practical machine learning problems in real life. Technically, the paper has quite some novelty including the creation of a rigorous lower bound to the true objective using recent advances in the variational inference area, and designed an effective surrogate parameterization to stabilize the optimization.

**Weaknesses and Questions**
1. Sec. 3.1: More detailed explanation of the exponential divergence would be beneficial. Is there a reference for it? What role does $f(\boldsymbol{u}; \xi)$ play? If it can be any real-valued function, why was it chosen to be a Gaussian pdf, as shown in the appendix?
2. Sec. 3.2: Which standard automatic differentiation library was used? The submission mentioned both the reparameterization trick and the REINFORCE trick - which one was actually used in the experiments?
3. Sec. 3.3: I don't quite understand how this part works. All I can see that in Eq(17) the problematic ratio term is multiplied by $G$, which is always smaller than 1 and non-increasing based on Figure 3. But why $G$ was defined in that math format? What does $\vee$ mean? Why does $G$ represent the ratio before and after the transform (equation between proposition 2 and 3)?
4. Sec. 4.1: Missing value processes (MCAR and MNAR). What do they mean? Are they different ways to decide what values are missing in the feature, and thus leading to different versions of a dataset? If so, shouldn't we also add CVAE*, Simple* and MICE*? Could you give more explanation for the last sentence of Section 4 (saying DVAE* is more robust than DVAE)?
5. Size of the test datasets. Based on Table 3, the datasets are all quite small, ranging from 353 to 10k data points. And we further split these points into training and test, which makes the training sets even smaller. In the appendix it's said the minibatch size is 521 -- what if the entire training set is smaller than 512? How long did the algorithm take to run on YearPred? Is the algorithm able to be easily extended to larger datasets?
6. How does DIG works compared with other more recent imputation baselines such as MIWAE (Mattei & Frellsen, 2019) and GAIN (Yoon et al., 2018)?
summary_of_the_review: Given the strengths of the paper listed above, I would recommend acceptance for this paper, if the authors can figure out a clear feedback for the questions I summarized when reading the paper.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper proposes a new method for learning with missing data. Compared with previous approaches, the authors choose to perform discriminative learning with generative modeling so as to borrow the benefits from these two types of methods. To optimize with the underlying intractable loss function, the authors start from the traditional variational lower bound ELBO and one upper bound CUBO from a previous work (the $\chi$-divergence lower bound [1]) and derives a lower bound for the original loss function. To solve the issue with the estimation bias as well as the potential huge variance, the authors change the divergence function in CUBO as well as add the surrogate parameterization so that the Monte Carlo estimation of the loss can be unbiased and (potentially) with smaller variances. Experiment results show the proposed methods run stably and perform comparably or better compared to baseline methods.

References:

[1] Adji Bousso Dieng, Dustin Tran, Rajesh Ranganath, John Paisley, and David Blei. Variational inference via χ upper bound minimization. In Advances in Neural Information Processing Systems, pp. 2732–2741. 2017.
main_review: Strengths:

1. The idea of learning missing data using discriminative learning together with generative modeling is interesting. As mentioned in the paper, performing such kind of learning will resulting a loss function as a subtraction on two integrals with respect to the latent variables, which makes it harder to derive a lower bound compared to the traditional variational inference cases. To solve the term being subtracted, the authors found a upper bound that could be estimated in an unbiased way with Monte Carlo methods.

2. The exponential function in the first version of CELBO will potentially has a bigger variance when estimated using Monte Carlo. To solve this issue, the authors propose adding a regularization to the loss function while remain the optimal solution unchanged under the zero gap case. This also helps a lot in making training more stable, as shown in the experiments.

Weakness:

1. The topic of this paper is to focus on missing data. However, this paper does not put enough efforts on learning various cases of data missing patterns. As in the experiments, the authors only test the case where the data are missing completely at random (MCAR), which may not be the most common case in reality. MNAR case might be a more interesting situation to study with. The proposed method is mostly focusing on solving the variational upper bound, while overlooks modeling the missing patterns. Suggest the authors could add some experiments with MNAR data. Also it would be better if the authors could add some modeling part on the missing pattern into the loss. For example, let the mask $m$ to depends on $(x, y, z)$. This will make the proposed model more useful in practice.

2. From Proposition 2, we know that the surrogate parameterization can make the optimal solution of CELBO remains under the zero gap case. However, it is nearly impossible to reach the zero gap case in reality since it is unlikely to select variational distributions (i.e. $q(\cdot)$'s) to perfectly estimate the model posterior distributions. What about the "sub-optimal" cases? Is the CELBO-SP optimal solution close to the CELBO optimal solution in a small gap (but not zero gap) case? I understand this might not be easy, but it will be better if the authors could add some theoretical analysis on this.

3. The performance metrics in Table 1 is not showing the proposed methods can outperform the baselines with big gaps, meaning that the proposed methods is not much better compared to previous approaches empirically. However, I think there are many ways that the authors could try to improve the performances. For example, the authors could try a different divergence function, a better way to add the regularization, etc.
summary_of_the_review: The author propose an interesting discriminative learning approach with generative modeling to solve the missing data modeling problem, by extending the traditional variational lower bound (ELBO), with a novel and stable upper bound that can be estimated without bias with Monte Carlo estimation. It is better if the author can study more on the missing data pattern (both empirically and theoretically) and the optimal solution preservation (under sub-optimal cases). Also, the empirical performance still has some space to improve.
review_title: null

Meta-review: While generative model can be used to input data, this work propose to a novel discriminative learning approach to optimize this data imputation phase by deriving a discriminative version of the traditional variational lower bound (ELBO). The resulting bound can be estimated without bias with Monte Carlo estimation leads to a practical approach, leading to encouraging experimental performances.

The reviewers recognised the novelty and suggest that this approach, given its novelty and wide applicability, could be considered for an oral presentation.
Decision: Accept (Oral)