How to deal with missing data in supervised deep learning?
Authors: Niels Bruun Ipsen
Pierre-Alexandre Mattei
Jes Frellsen
Venue: ICLR 2022 Poster (2022)

Abstract: The issue of missing data in supervised learning has been largely overlooked, especially in the deep learning community. We investigate strategies to adapt neural architectures for handling missing values. Here, we focus on regression and classification problems where the features are assumed to be missing at random. Of particular interest are schemes that allow reusing as-is a neural discriminative architecture. To address supervised deep learning with missing values, we propose to marginalize over missing values in a joint model of covariates and outcomes. Thereby, we leverage both the flexibility of deep generative models to describe the distribution of the covariates and the power of purely discriminative models to make predictions. More precisely, a deep latent variable model can be learned jointly with the discriminative model, using importance-weighted variational inference, essentially using importance sampling to mimick averaging over multiple imputations. In low-capacity regimes, or when the discriminative model has a strong inductive bias, we find that our hybrid generative/discriminative approach generally outperforms single imputations methods.

Published as a conference paper at ICLR 2022
HOW TO DEAL WITH MISSING DATA IN SUPERVISED DEEP LEARNING?
Niels Bruun Ipsen nbip@dtu. dk
Pierre-Alexandre Mattei pierre-alexande htt.::/aacc.cee.aatttttiieiiaaaa matteieinnia.........iai fr
Jes Frellsen" jefr@dru. dk
ABSTRACT
The issue of missing data in supervised learning has been largely overlooked, es- pecially in the deep learning community. We investigate strategies to adapt neural architectures for handling missing values. Here, we focus on regression and clas- sification problems where the features are assumed to be missing at random. Of particular interest are schemes that allow reusing as-is neural discriminative ar- chitecture. address supervised deep learning with missing values, we propose to marginalize over missing values in joint model of covariates and outcomes. Thereby, we leverage both the flexibility of deep generative models to describe the distribution of the covariates and the power of purely discriminative models to make predictions. More precisely, a deep latent variable model can be learned jointly with the discriminative model, using importance-weigheee ference, essentially using importance sampling to mimick averaging over multi- ple imputations. In low-capacity regimes, when the discriminative model has strong inductive bias, we find that our hybrid generative/discriinattyy approach generally outperforms single imputations methods.
variational in-
INTRODUCTION
Missing data affect data analysis across wide range of domains and the sources missing values span equally wide range. Recently, deep latent variable models (DLVMs, Kingma & Welling 2014: Rezende et al.) 2014) have been applied to missing data problems in an un-
By (xcrhn)
supervised setting (e.g. Rezende et al. 2014
Nazabal et al.) 2020 Ma et al.] [2018]
2019
Ke nu My (m)
Ivanov al. 2019 Mattei & Frellsen
2018
2019 Yoon et al. 2018 Li et al. 2019 Ipsen
Aa(a))
al. 2021 Ghalebikesabl al.] 2021). while
the supervised setting has not seen the same at- tention. The progress in the unsupervised set- ting is focused on inference and imputation and can therefore be useful as an imputation step before learning discriminative model. Tradi- tionally the focus has also been on inference and imputation either a goal in itself or be- fore supervised learning on the (possibly mul- tiple) imputations and observed data (Rubin) 1976 1996
locte sepiing
poly xitey
Figure 1: Left: SupMIWAE computational struc- ture: encoding network gy, decoding network he and discriminative network fet- Right: Graphi- cal model: supMIWAE seen VAE for missing values concatenated with neural discriminator.
Little & Rubin 2019)
Supervised learning in the presence of miss- ing values has different goals and pose different challenges than inference and imputation (Josse
"Department of Applied Mathematics and Computer Science, Technical University of Denmark, Denmark
"Université Côte d'Azur, Inria (Maasai team). Laboratoire J.A. Dieudonné, CNRS. France
Equua contribution

---

Published as a conference paper at ICLR 2022
Here the overall aim is minimize expected pre- diction error. However, optimal single imputation does not necessarily lead to optimal prediction in terms of minimizing prediction error (Josse et al. 2019). One challenge is that predictions based on inputs with missing values can be ambiguous, that is, the conditional distribution of the miss- ing values given the observed may be multimodal and the optimal prediction may change with the mode. With single imputation, the conditional distribution over the missing data is discarded, and the optimal single imputation can no longer reflect this ambiguity, see figure 2 Even the optimal single imputation leads to biased parameter estimates compared to the fully observed case (Bertsi- mas al., 2021). and may in some cases lie outside the distribution of the data. Another challenge that the number of missing value patterns grows combinatorially with the number of features p, requiring 2 submodels fit Bayes predictor in the linear case (Le Morvan et al. [020a).. In multiple imputation (Rubin 1996), several imputations are drawn from the posterior predictive dis- tribution of the missing values, reflecting the full conditional distribution of the missing values and thus the uncertainty about what is missing. This allows for uncertainty estimates in downstream tasks such as prediction. This in turn requires fitting as many discriminative models as the number of imputations.
et al. 2019 Le Morvan al., 2020a 2021
1.1 CONTRIBUTIONS
In order to address supervised deep learning with missing values we develop the supervised missing data imporfance-weighted autoencoder (supMIWAH) bound, based the approximate maximum likehood techniques used by Burda et al. (0166) and Mattei & Frellsen (2019). This is a scalable approach marginalizing over missing values in a joint model of covariates and outcomes
(1) The covariate model (x > p(x*t x p2)((z dz is DLVM in this work, but can be any probabilistic model imposing a joint density over covariates. The outcome model misss) is any neural discriminative architecture that would have been used in the complete-data case, parameterizing a density over the outcomes. The graphical model is shown in figure [ Jlong with its computational structure.
Po.o x miss) Po (y|x"ss xx misss) )po (x " miss).
P (
x (y|x
Once the joint model has been trained using the SupMIWAE bound, the conditional distribution of the outcome given the observed parts of the input can be found
Ps (y (y (x)))) = f po (y (x
x Ie
miss)
(e miss (x miss,
(2)
This is approximated using importance sampling techniques, averaging predictions over multiple importance samples (akin multiple imputations, see appendix C generative model.
for deeper discussion) from the
The model can trained end-to-end or pretrained generative model can be coupled with a dis- criminative model. Having a joint model allows for supervised imputations, where available labels can help guide the imputations from the generative model by adjusting the importance weights ac- cordingly.
Joint models over covariates and labels have previously been used to marginalize over missing values in the covariates, using Gaussian mixture models either the model over the covariates directly as the joint model over covariates and outcomes (Ghahramani & Jordan| [995] Ghahramani & Hinton
1996 Ahmad Tresp 1993. Tresp et al. 1994 1995| Smieja et al. 2018). Our approach shows how use more flexible generative models, such as DLVMs, provides efficient optimization procedure and allows for keeping any deep discriminative architecture unchanged.
BACKGROUND AND NOTATION
We define the random variable (I1, (I1,... 2p) X' which takes values in a p-dimensional feature space X' = Xi X ... x X,. There is a corresponding (possibly vector valued) response variable y E .. A missing process obscures parts of resulting in the mask random variable {0, 1)
where
if 2y observed,
8, 1
(3)
if xy missing.
2

---

Published as a conference paper at ICLR 2022
Dysimdd insulation
0.4
0.2
0.0
Imputation
supMIWAR
Figure 2: Left: Kernel density of two-class dataset (the burger dataset), where and y = 0 is blue and y 1 is brown/orange. Observations with missing values are shown as sticks in the bottom margin of the plots and colored according to their class label. For observations with missing 2nd coordinate there is an inherent ambiguity about the class with p(y = 1|x = 2/3. The optimal single imputation is shown as the dashed black line. Middle: Decision surface for neural classifier trained on drtimally-imputed data. The decision surface has to reflect that for the imputed data p(y 1|4o(x**)) = 2/3, thus the increased class y = probability around the optimal imputations (here 40 is the imputation function, see appendix [A.I]. Right: Decision surface as learnt by the SupMIWAE. is similar to the decision surface that would be found in the absence of missing data.
We let obs(s) denote the indices of the non-zero entries of s and miss(s) denote the indices of the
zero-entries of s, such that x=)(()) is the set of observed elements of x, and miss(() is the set of missing elements of x. For simplicity we will omit the and write obs miss respectively, whenever the context is clear.
Following (Yoon et al. 2018 Le Morvan et al. 2020b Ghalebikesabi et al., 2021) we define the .......... E X which takes values in = (X (na)) x ... (Xp U (ma}) where missing values are represented by the symbol na, such that na Ij na and na 0. We typically only have access to X, where
incomplete random variable x = (..,...,, Fp)
= x ② s O (1 s)
(4)
and @ is the Hadamard product. Finally, we are given n i.i.d. copies of the random variables x and y which we collect in a dataset yi or alternatively D { (xt y:)
We make the additional assumption that the data are missing at random (MAR. see e.g. Scaman al., 2013 Little & Rubin 2019). Specifically, this means that we assume that and x miss are independent given 0 This assumption allows to avoid to explicitly model the missing mechanism. Our approach could be extended beyond MAR by using a generative model fit for this purpose, like the not-MIWAE of Ipsen et al. I2021), or the deep pattern-st mixture of Ghalebikeesabi et al. 202
2.1 CHALLENGES WHEN TRAINING DISRIMINATIVE MODELS WITH MISSING DATA
Our predictive model will defined through a mapping fs : X -> H used to parameterize a conditional distribution
( = (x)
(5)
Here (V (m) H is a parametric family of distributions over the outcome space V, such as a Gaus- sian distribution in regression categorical distribution in classification. Under the maximum- likelihood framework (or equivalently the logarithmic scoring rule), an optimal mapping S within
a class F = (fø)øE parameterized by & E Φ is defined as
fāz E arg max Epp [log (y)))( J&EF
(6)
where p" (xx y) is the true data generating distribution. When the data complete, this is typically approximated by maximizing the log likelihood of the parameters &f given the data D Prain
{(Φ)=
Σ
log (y(x),
(x.y)
3

---

Published as a conference paper at ICLR 2022
When the covariates have missing values the log-likelihood cannot be maximized directly as the likelihood is not defined since Po depends on the full input vector. The simplest approach is instead to learn separate mappings for each missing pattern. This reduces the amount of data available for the training of each mapping and does not scale well with the dimensionality of the input as in general 2 networks need to be trained. An often used approach is instead impute the missing values using a single imputation (for instance, using po (2enis]xes)) [x or an approximation it), and then map the concatenation of observed and imputed values approximate the conditional distribution
Pp (y 0bs) RJ Pyplyx obs E (x bennc) (x [) = 1 (y
(xemln (xeil ( (xx ))
Instead of Ep (xtiia [xm which is the optimal imputation under the mean squared error, one could use any kind of imputation, for instance using a constant, or the mean.
This general approach is adequately called impute- impute-then-regnee then-regress by Berttsias al. 2021 and Le Morvan et al. (202T). While it will be Bayes-consistent given a powerful enough classifier (Le Mor- van et al. (222)) it leads to biased parameter estimates compared to having complete data (Bertsimas) et al. 2021). This is illustrated in figure 2 where the optimal (under the mean squared error) sin- gle imputations are placed entirely within one of the classes, requiring a biased decision surface to properly reflect the class label probabilities for the records with missing values. From prob- abilistic perspective, computing P (x requires marginalizing over the missing features as in equation (T. that is
Ps p (y|x = Ep (xet)) IP (y x obs x mmiss]]
(9)
Notice the difference to single imputation since, in general
E (xemi bi [P /x m # P = (x po (xciis [x (x)mss)))
obs
(10)
except in some pathological cases (e.g. when P& is linear or pe is a Dirac distribution). Therefore, if the discriminative model is nonlinear and the generative model is complex enough, then using single imputation will give very different result than marginalising over the missing features. This exemplified in figure 2 where a classifier trained using the optimal imputation finds a solution that is very different to what would be found by one trained without missing data. Our technique, marginalizing the missing values, allows to recover the same decision surface as classifier trained without missing data.
2.2 OUR JOINT MODEL AND ITS SUPMIWAE VARIATIONAL BOUND
As mentioned in the introduction, we posit a joint model over covariates and outcome, that will make use of latent variable
Ps. x obs x z) Pe Py(yxx xx misss) (x (x miss z)po
(11)
Under the MAR assumption, the relevant substitute of the likelihood is the likelihood of the observed data
P (y, x = Po Po(y) x x po (x misss z))o (z) dz dx
(12)
The integral in equation 12) is in all but the simplest cases analytically intractable and direct max- imum likelihood methods for learning the parameters (0, Φ) cannot be used. In order to learn the parameters of the joint model, we turn to amortized importance weighted variational infer- ence, where a lower bound is maximized instead of the log likelihood itself (Burda et al., 2016 Domke & Sheldon 2018). It is based on the fact that unbiased estimates of likelihood can be turned into lower bounds of the log- loglliiei Indeed, R(xxbs, .y) is random variable such that [R(x*hh .y]] = pe. (x*",, .y), then E [log R(x*" [y)] I log E [R(x .y)] = log Pe. (xx"h, .y), which means that IE [log [xxx"*,,)) is a lower bound of the log likelihood. A simple way to find a suitable R(x .y) is using importance sampling from variational distribution over z. More specifically, let
K
Po (y) (x/ miss) po (xobs (2 (20))) gy 2x )
Rx Rx(xxto ,y) = K
(13)
4

---

Published as a conference paper at ICLR 2022
where (2(2xx)))) is the variational distribution (learnale proposal) and (xx ZE)RE (1....... are i.idd samples from pe px 2)qy(2/xx0h) Then we have that Exxgin 22) [Rx (x .y] Po.e (x"xx""" .) and (xxain =x) [log Rx (x y)] is a lower bound on the log likelihood. The vari- ational distribution can be parameterized via neural network with weights y I, similarly to Mattei & Frellsen (2019), Nazabal et al. (020), and [[psen et al. (202I).
Now that we have way to bound each per-datum log-likelihood, we can define our supMIWAE variational bound L (0, Φ 7) that a lower-bound of the log-likelihood of the observed data
Cx 6. Σ B (epr ze)) [log Rx (x; v.) S log Po xh
(14)
The lower bound Lx (0 7) a Monte Carlo objective that can be maximized instead of the log likelihood itself, using techniques from stochastic optimization (see e.g. MMhamed jct al. 2020). As detailed in appendix D] benefits from the usual thcoretical advantages of importance-weighted variational bounds: Lx (0, Φ, 7) converges monotonically to the log- likelihood Σi= log P (y., x) at speed 1/K. This means that, the larger the K, the closer objective will be to the true likelihood.
Note that the contribution of any fully observed data point (x,y) will be simply
po (x(x (pe)))))p(()))
log Po (y(x) + ((x) log
(15)
47 (Z,
which is just the sum of the discriminative likelihood and the standard IWAE bound of Burda et al. (016). An interesting property of our bound is that the discriminative and generative parts can be trained jointly with single and statistically sound objective. However, if we were to encounter data set without missing data, equation (I5) means that the training of the two parts of the models would be decoupled: the discriminative model would be the same as a standardly trained discrimi- native model, and the generative model would be the same as a standard IWAE-trained DLVM.
2.3 PREDICTION
Once the model has been jointly trained using the supMIWAE bound, is possible to use it to do pre- diction with missing data. Indeed normalerd importance sampling can be used to approximate the conditional
K
Pep (y|
P (yl obs x miss).
(16)
where
pe Px Zk Ipo (zk)
Tk
W6
and FR
(17)
+
TK
47 (7 [ xob
and (xitiss ke K} are i.i.d. samples from pe (x |z) 4z (z) (b Again, this technique mim- icks multiple imputation: several imputations are generated using the generative model, are then fed the classifier, and the predictions of each imputation are averaged.
RELATED WORK
Marginalizing over missing covariates Different approaches have previously been taken to marginalize over missing values in joint models over features and labels. series of publica- tions (Ahmad & Tresp 1993 Tresp et al., [194 1993), closed-form solution using Gaussian Basis Function networks was used to approximate this marginalization, while keeping the neural discrim- inative model fixed. Ghahramani & Jordan (1995) took different approach, modeling the joint distribution p(y, x) directly as a mixture model, obtaining p(y tha) by conditioning on observed quantities in the joint distribution, thus removing the explicit discriminative neural architecture. In the context of kernel methods, Dick et al. (2008) proposed to learn a distribution over missing val- ues by minimising the regularized empirical risk. (Smieja et al. (2018) trained a Gaussian mixture model (GMM) and discriminative model jointly, and in place of any missing values the activation of the corresponding input neuron was set to the average activation over the GMM conditioned on observed values.
5

---

Published as a conference paper at ICLR 2022
Consistency of single imputation A review of approaches to handling missing data in (non-deep) supervised learning was given byJJosse et al. (2019). Here was shown that under some assumptions about the capacity of the learner, constant imputation is asymptotically consistent in the supervised setting. Le Morvan et al. (2000b) investigated the case of a linear predictor on covariates with missing data, showing that in the presence of missing, the optimal predictor may not be linear and how constant imputation of each feature can be optimized with regards to the model loss. The linear case was further investigated in (Le Morvan et al. 2020aa), deriving the analytical form of the optimal predictor and proposing NeuMiss networks to approximate this. Bertsimas et al. (202T) analyzed the impute-then-regeee then-regress situation and noted that while this paradigm is widely used in practice incurs a bias in the parameter estimates, but mean imputation is still asymptotically consistent. Le Morvan et al. (202T) showed that the impute-then-regress approach is asymptotically Bayes optimal given a learner with large capacity and that this holds for all missing mechanisms, even missing not at random (Rubin 1976). However, the choice of imputation function affects the difficulty of the subsequent regression task, when not in the asymptotic regime. As a consequence they proposed to learn imputation and regression jointly, where the missing values are handled by a NeuMiss network.
Other approaches for missing covariates al (2019) tackled the issue of sparsity, and specif- ically large variations in sparsity, by introducing sparsity normalization. This addresses the issue of model output covarying with the sparsity level the input. Ma et al. (2018) 2019) used permu- tation invariant setup to avoid imputing missing data in the input of a variational autoencoder. This approach can be readily extended to the supervised setting, using the permutation invariant setup as a modified input layer as we show in appendix A.. Some tree approaches to regression/classiiees can use observations with missing values directly (see e.g. [Twala et al., 20088] Chen & Guestrin] 2016 Josse et al., 2019; Gómez-Méndez & Joly (021). Sparse coding was used by Caiafa et al. 2020 to train a dictionary and a classifier simultaneously.
DLVMs and missing values Deep latent variable models have been applied missing data prob- lems in the unsupervised setting, focused on inference, single and multiple imputations. Early works assumed that a DLVM had previously been trained on complete data (Rezende et al.) 20144 Mattei
& Frellsen 2018), but a variety of techniques to handle incomplete training sets was then proposed.
both in M(C)AR (Yoon et al. 2018 Ma et al. 2018 2019 Ivaoov et al. 2019; Mattei & Frellsen
2019 Li et al. 2019) and MNAR (lpsen et al. 2021 Ghalebikesabi et al. (021) cases. In (Li &
Marlin 2020) irregularly sampled time-series are approached as a missing data problem and they propose VAE and GAN based models for handling this. The proposed models can also be used for supervised learning by training a classifier on the latent representations, either by training the model jointly or using pretrained encoder. Without missing data, the idea of jointly training a VAE and discriminative model, has been explored by Ji et al. 2020) or Joy et al. (2021).
4 EXPERIMENTS
We now evaluate discriminative models trained using the supMIWAE bound on a range of super- vised learning tasks. Throughout the experiments the generative part of the model is pretrained using the MIWAE bound of Mattei & Frellsen 2019). The pretrained DLVM can be used to draw single imputations, referred to as MIWAE single imputation, the SupMIWAF computational structure. Here, the generative part of the model is kept fixed while updating the discriminative part of the model according to the SupMIWAE bound. As the DLVM in MIWAE single imputations and the SupMIWAE are similar, any difference in performance between the two is attributed their different strategies for handling missing values. Other strategies for han- dling missing data, used for comparison in this work, are briefly described in appendix A Missing values are introduced in the training, validation and test sets according to the missing mechanism used in the given experiment. Predictive performance is measured on a test-set with missing values. In all experiments, there is computational overhead compared to single imputation of training the supMIWAE that scales linear with the number of importance samples K, c.f. equation (1)
it can be used as the generative part in
4.1 2D CLASSIFICATION
A qualitative analysis of the bias in the learnt parameters of the discriminative model, due to single imputation, is set up on simple 2D datasets, figure 8 of appendix B The simplicity allows us

---

Published as a conference paper at ICLR 2022
supMIWAE
MIWAE
O-impute
Leamable Impulation
MICE
MissForest
Figure 3: Top row: (except top left) kernel density of the half-moons dataset together with im- putations from the given model. The supMIWAE does not rely on single imputations, instead it draws multiple importance samples which are passed through the discriminator to give an impor- tance weighted prediction. For the SupMIWAE multiple imputations at three different values of the horizontal coordinate are shown and kernel density of the multiple imputations are shown to left. For the rest of the methods single imputations are shown in red. Bottom row: Decision sur- faces learnt, depending on the strategy for handling missing values. The methods based on single imputation need to warp the decision boundaries.
to directly inspect the impact of different imputation strategies the learnt decision surface and compare this to the marginalization procedure used supMIWAE.
All datasets consist of training set with 3k records and validation and test sets with 1K records. An MCAR missing process is introduced in the horizontal coordinate, where each element be- comes missing with probability are such that there will be inherent ambiguity about the class label for some the records with missing values. For example in the Burger dataset, all records with missing value will have p(y bbs = 2/3.
0.5. Further training details are in appendix E The datasets
In figure imputations from different imputation models are shown along with the corresponding learnt decision surface. Partially observed examples are shown in the margins of the plots as sticks, colored by their class label. All methods use the same discriminative architecture, only the im- putation strategy differs. For the supMIWAE. multiple imputations are shown for three partially observed records, alongside their kernel density. These multiple imputations reflect the appearance of importance samples used during training, for these three partially observed records.
Decision surface artifacts The single imputation methods introduce artifacts into the learnt de- cision surface. As the discriminative model tries reflect the proper conditional probabilities p(y xob)) at the location of the imputed values, the ambiguity about the class label needs to be reflected in the predictions. Notice how the MIWAE produces nearly optimal imputations in terms of minimizing a mean Sean--quare-erro and how this translates into decision surface artifacts. These artifacts are related the notion of imputation manifolds used by Le Morvan al.] (202I) to prove their main theorem on the Bayes consitency of impute-then-regree procedures. The remaining setup (ap-
datasets are shown in figures 13 appendix EE where also the permutation-invarian pendix A.3) and histogram- based gradient boosting included. Like the SupMIWAE, these two approaches do not rely on explicit imputations.
As the supMIWAE can draw multiple importance samples, instead of one single imputation, the conditional probabilities p(y) (bs)) can be properly reflected, when averaging over the importance samples. This avoids the single imputation artefacts.
Predictive performance Do the artifacts introduced by single imputation methods, compared to marginalizing over the missing values, translate into differences predictive performance? The predictive performance corresponding the decision surfaces in figure Jre shown in table i ap- pendix E. Both the test-set accuracies and log likelihoods are more or less similar across the different strategies for handling missing values. This aligns with the results by Josse et al. (2019); Bertsimas eet al. 2021); Le Morvan et al. (202T): the impute-then-regress procedures with powerful learners consistent and Bayes optimal, i.e. when the learner has high capacity any single imputation can
7

---

Published as a conference paper at ICLR 2022
essentially be undone. While single-imputatin methods can lead to Bayes-optimal learners, Le Morvan et al. 202I) address that the difficulty of the learning problem depends on the choice of imputation function. This is seen in figure 4 while the supMIWAE and 0-imputation eventually ends up with similar performance the learning problem appears easier for the SupMIWAF. For 0- imputation the learner has to introduce high-frequency artifacts the decision surface in order to get the conditional probabilities p(y "b")) calibrated, but in the case of a high capacity learner this can eventually be done.
Limited capacity learner We now turn to a situation where the capacity of the learner limited in order to assess how single imputation methods fare against marginalizing over missing values. We start by not-
half-moons
0.80
0.86
ing that the decision surface for the Burger dataset, in the complete-data case, can be learnt using two sig- moids. When using MSE-optimal single imputations in the missing-values case this changes, as now 4 sigmoids needed to make sure the imputations brated. This illustrates that the capacity of the learner determines whether single imputations can be undone. In figure [5] we explore how the capacity of the learner, in terms of the number of hidden units, affects the predic- tive performance, depending the imputation strategy. More capacity experiments are found figure
0.84
0.82
0.80
O O-mputationn supMIWAE
0.70
50
100
150
200
250
300
well cali-
Epuch
Figure 4: Learning curves the half- moons dataset: train (full lines) and test (dashed lines) set accuracies, when the classifier has full capacity.
4.2 IMAGE CLASSIFICATION
When the learning problems become harder, strong inductive biases are sometimes used help learning. This is typically the case with image classification where convolutional neural networks (CNNs) are often used introduce translational equivariance. The previous section showed that the strategy for handling missing data affects predictive performance when the learner is not in the high capacity regime. This is further investigated in terms of classification accuracy in image experiments with different MCAR missing processes, such as observed squares, missing squares and random dropout, see figures [7] The discriminator is a convolutional neural network, see appendix [F.]] thus introducing an inductive bias in the learner that has proved useful in image classification in the complete-data case. While CNNs are fit for images, it should be noted that they are not universal approximators, and may consequently have hard time undoing single imputations.
We apply the SupMIWAE and single imputations methods to the MNIST dataset (LeCun et al. 1998) and the fashion MNIST dataset (Xiao al. 2017) with three different MCAR missing mechanisms: 1) observed squares, where 12 x 12 randomly placed square observed, 2) missing squares, where 12 12 randomly placed square missing, and 3) dropout, where each pixel is missing with some constant probability m, over a range of missing rates.
In practice, we that this setting where there is strong inductive bias is similar to previous low-capacity experiments: for both MNIST and Fashion MNIST, the SupMIWAE is significantly
burger
burger
0.78
0.82
0.80
uast logply/x)
-0.50
-0.52
NIKAE
SINATWAE
8 0.76
-0.54
2 0.77
0.74
-0.56
Insrnaate Impotattinn
MICE
-0.58
0.70
0.88
-066
capacity
capacity
Figure 5: Predictive performance when varying the capacity of the learner in terms of number of hid- den units. The discriminative model has the same architecture across methods, only the imputation strategy differs.
8

---

Published as a conference paper at ICLR 2022
squares
missing squaces
dropout
0.82
0.95
0.95
a0c
0.79
0.78
0.80
0.77
Gimpete
expMinin
bumalle-inpittiion
0.82
0.94
0.80
0.85
0.90
SIJEENAE
Grrpgtt
MIКAР
0.78
0.91
0.1
0.4
0.7
0.75
missing rale
Inchton: observed
fastson:
Eashsom:
0.88
0.875
0.050
0,74
test acc
0.73
0.72
0.85
0.750
0.725
0.700
0,70
0.675
0.69
0.2 0.3 0.4 0.5 0.7 0.9
miissing Toto
Figure 6: Test set accuracies on the MNIST and Fashion MNIST datasets, with different missing mechanisms. Left column: observed squares. Middle column: missing squares. Right column: random dropout over range of missing rates.
protein
superconduuctivit
0.7
0.7
MIMIIWAE
Lest russe
test rusee
0.5
0.4
0.6
Euputation
0.3
0.5
munkForest
0.2
0.4
RPCA
0.1
0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9
0.3
0.1
0.2
0.3 0.4 0.5 0.6 0.7 0.8 0.9
gettitiatin invatinao
missing rate
missing rate
Figure 7: Test-set root mean square error on UCI datasets at varying missing rates.
more accurate than single-imputation competitors, as seen figure 6 Additional experiments, including experiments natural images (SVHN) and visualizations of multiple imputations, available in appendix F
4.3 REGRESSION
We now turn regression in smaller and lower dimensional datasets from the UCI database Dua & Graff, 2017). An experimental setup modified from Hernandez-Lobato & Adams (2015) and Skafte et al. (2019) is used here. Predictive performance in terms of root-mean--suuar--eroo presented over range of missing rates. Results are seen in figure 7 and more results can be found in figure 20 appendix G There is no clear winner here, although it appears that gradient boosting generally performs quite well, and that accurate single imputations (such MIWAE's or MissForest's s) generally outperform inaccurate ones (such as zero imputation). In general, our joint model performs as well as feeding accurate single imputations.
5 CONCLUSION
In the context of supervised deep learning with missing values, single imputation methods are an often used tool. While this approach is asymptotically consistent it incurs bias in the learnt pa- rameters. have introduced a scalable approach to marginalizing over missing values using deep generative models, as probabilistic alternative to single imputation. Besides limiting single impu- tation artifacts our results indicate that there are two cases where using a generative model can be quite valuable for supervised learning with missing values: when the classifier is not very flexible, or has a strong inductive bias. Moreover, leveraging such a generative model allows us to able to untouched any discriminative architecture (for instance, any pretrained one).

---

Published as a conference paper at ICLR 2022
REPRODUCIBILITY STATEMENT
Code for reproducing paper experiments is available at SupMIWAE
ACKNOWLEDGEMENTS
This work has been supported by the French government, through the 3IA Côte d' Azur Invest- ments in the Future project managed by the National Research Agency (ANR) with the refer- ence number
Furthermore, it was supported by the Novo Nordisk Foundation (NNF200C0062606 and NNF200C0065611 and the Independent Research Fund Denmark (9131- 00082B).
REFERENCES
Subutai Ahmad and Volker Tresp. Some solutions to the missing feature problem in vision. In Advances in Neural Information Processing Systems, pp. 393-400, 1993.
Dimitris Bertsimas, Arthur Delarue, and Jean Pauphilet. Prediction with missing data. arXiv preprint arXiv:2104.03158, 2021.
Yuri Burda, Roger B. Grosse, and Ruslan Salakhutdinov. Importance weighted autoencoders. In International Conference Learning Representations, 2016.
S van Buuren and Karin Proothuis-Oudshoor... mice: Multivariate imputation by chained equations in Journal of statistical seftware, pp. 1-68, 2010.
Cesar Caiafa, Ziyao Wang, Jordi Solé-Casals, and Qibin Zhao. Learning from incomplete data by simultancous training of neural networks and sparse coding. arXiv preprint arXiv:2011.14047, 2020.
Tianqi Chen and Carlos Guestrin. XGBoost: A scalable tree boosting system. In Proceedings of the 22nd acm SIGKDD International Conference on Knowledge Discovery and Data Mining, pp- 785-794, 2016.
Yann N Dauphin, Angela Fan, Michael Auli, and David Grangier. Language modeling with gated convolutional networks. In International conference on machine learning, pp. 933-941. PMLR, 2017.
Uwe Dick, Peter Haider, and Tobias Scheffer. Learning from incomplete data with infinite impu- tations. In Proceedings of the 25th international conference on Machine learning, pp. 232-239, 2008.
Justin Domke and Daniel R Sheldon. Importance weighting and variational inference. In Advances in Neural Information Processing Systems, volume 31. Curran Associates, Inc., 2018
Dheeru Dua and Casey Graff. UCI machine learning repository, 2017. URL/tttp: httphhttp///cciiee
ics. www.ccciee. eduuml ml
Jerome H Friedman. Greedy function approximation: a gradient boosting machine. Annals of statistics, pp. 1189-1232 2001.
Andrew Gelman, John B Carlin, Hal S Stem,, David B Dunson, Aki Vehtari, and Donald B Rubin. Bayesian data analysis. CRC press, 2013.
Zoubin Ghahramani and Geoffrey E Hinton. The EM algorithm for mixtures of factor analyzers. Technical report, CRG-TR BRG-TR-96-L University of Toronto, 1996.
Zoubin Ghahramani and Michael Jordan. Learning from incomplete data. MIT Center for Biolog- ical and Computational Learning Technical Report 108, 1995.
Sahra Ghalebikesabi, Rob Cornish, Chris Holmes, and Luke Kelly. Deep generative pattern-set mixture models. In Proceedings of The 24th International Conference on Artificial Intelligence and Statistics, pp. 3727-3735, 2021.
10

---

Published as a conference paper at ICLR 2022
Irving Gómez-Méndez and Emilien Joly. Regression with missing data, comparison study of techniques based on random forests. Preprint, available at battps://irrinnggomee com/ publication, 2021.
José Miguel Herbández-Lobato and Ryan Adams. Probabilistic backpropagation for scalable learn- ing of bayesian neural networks. In International Conference on Machine Learning, pp. 1861- 1869, 2015.
Niels Bruun Ipsen, AlexanAee Mattei, and Jes Frellsen. not-MIWAE: Deep generative mod- elling with missing not at random data. International Conference on Learning Representattons, 2021.
Oleg Ivanov, Michael Figurnov, and Dmitry Vetrov. Variational autoencoder with arbitrary condi- tioning. In International Conference on Learning Representations, 2019.
Tianchen Ji, Sri Theja Vuppala, Girish Chowdhary, and Katherine Driggs-Campbell Multi-modal anomaly detection for unstructured and uncertain environments. In Conference on Robot Learn ing, 2020.
Matthew Johnson, David K Duvenaud, Alex Wiltschko, Ryan Adams, and Sandeep Datta. Composing graphical models with neural networks for structured representations and fast infer- ence. Advances in Neural Information Processing Systems, 29:2946-2955 2016.
Julie Josse, Nicolas Prost, Erwan Scornet, and Gaël Varoquaux. On the consistency of supervised learning with missing values. arXiv preprint arXiv:1902.0669 069331, 2019.
Tom Joy, Sebastian Schmon, Philip Torr, N Siddharth, and Tom Rainforth. Capturing label charac- teristies in VAEs. In International Conference Learning Representations, 2021.
Diederik Kingma and Max Welling. Auto-encoding variational Bayes. In International Conference on Learning Representations, 2014.
Marine Le Morvan, Julie Josse, Thomas Moreau, Erwan Scornet, and Gaël Varoquaux. Neumiss networks: differentiable programming for supervised learning with missing values. arXiv preprint vrXiv:2007.01627 2020a.
Marine Le Morvan, Nicolas Prost, Julie Josse, Erwan Scornet, and Gaël Varoquaux. Linear predictor linearly- gennrrtte data with missing values: consistency and solutions. In International Conference on Artificial Intelligence and Statistics, pp. 3165-3174, 2020b.
Marine Le Morvan, Julie Josse, Erwan Scomet, and Gaël Varoquaux. What's 's good imputation to predict with missing values? arXiv preprint arXiv:2106.004 00311, 2021.
Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient-based learning applied to document recognition. Proceedings of the IEFE, 86(11):2278-233) 1998.
Steven Cheng-Xian Li and Benjamin Marlin. Learning from irregularl-sampple time series: A missing data perspective. In International Conference Machine Learning, pp. 5937-5946, 2020.
Steven Cheng Xian Li, Bo Jiang, and Benjamin Marlin. MisGAN: Learning from incomplete data with generative adversarial networks. International Conference on Learning Representations, 2019.
Roderick JA Little and Donald Rubin. Statistical analysis with missing data, volume 793. John Wiley & Sons, 2019.
Gabriel Loaiza-Ganem and John P Cunningham. The continuous Bernoulli: fixing pervasive error in variational autoencoders. Advances in Neural Information Processing Systems, 2019.
Chao Ma, Wenbo Gong, José Miguel Hermández-Lobato, Noam Koenigstein, Sebastian Nowozin, and Cheng Zhang. Partial VAE for hybrid recommender system. In 3rd NeurIPS Workshop on Bayesian Deep Learning, 2018.
11

---

Published as a conference paper at ICLR 2022
Chao Ma, Sebastian Tschiatschek. Konstantina Palla, Jose Miguel Hernandez-Lobato. Sebastian Nowozin, and Cheng Zhang. EDDI: Efficient dynamic discovery of high-value information with partial VAE. In Internationaa Conference Machine Learning. pp. 4234-4243, 2019.
Plexad-Aleandre Mattei and Jes Frellsen. Leveraging the exact likelihood of deep latent variable models. In Advances in Neural Information Processing Systems, 2018.
Pierre-Alexanrre Mattei and Jes Frellsen. MIWAE: Deep generative modelling and imputation of incomplete data sets. In International Conference on Machine Learning, pp. 4413-444444 2019.
Shakir Mohamed, Mihaela Rosca, Michael Figurnov, and Andriy Mnih. Monte Carlo gradient estimation in machine learning. Journal of Machine Learning Research, 21(132):1-62, 2020.
Alfredo Nazabal, Pablo M Olmos, Zoubin Ghahramani, and Isabel Valera. Handling incomplete heterogeneous data using VAEs. Puttern Recognition, 107: 107500,,,, 2020.
Yuval Netzer, Tao Wang, Adam Coates, Alessandro Bissacco, Bo Wu, and Andrew Y Ng. Reading digits in natural images with unsupervised feature learning. NIPS Workshop on Deep Learning and Unsupervised Feature Learning, 2011.
Pedregosa, G. Varoquaux, Gramfort, V. Michel, Thirion, O. Grisel, M. Blondel, P. Pretten- hofer, R. Weiss, V. Dubourg, J. Vanderplas, A. Passos, D. Cournapeau, M. Brucher, M. Perrot, and E. Duchesnay. Scikit-learn: Machine learning in Python. Journal of Machine Learning Research, 12:2825-2830, 2011.
Charles R Qi, Hao Su, Kaichun Mo, and Leonidas J Guibas. Pointnet: Deep learning on point sets for 3D classification and segmentation. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 652-660, 2017.
Alec Radford, Luke Metz, and Soumith Chintala. Unsupervised representation learning with deep convolutional generative adversarial networks. arXiv preprint arXiv:1511.0644 2015.
Danilo Jimenez Rezende, Shakir Mohamed, and Daan Wierstra. Stochastic backpropagation and ap- proximate inference in deep generative models. In International Conference on Machine Learn- ing, 2014.
Sam Roweis. EM algorithms for PCA and SPCA. Advances in Neural Information Processing Systems, pp. 626 626-632, 1998.
Donald B Rubin. Inference and missing data. Biometrika, 63(3):581-592. 1976.
Donald B Rubin. Multiple imputation after 18+ years. Journal of the American Statistical Associa- tion, 91(434):473-489 1996.
Tim Salimans, Andrej Karpathy, Xi Chen, and Diederik P Kingma. PixeICN++:: Improving the PixelCNN with discretized logistic mixture likelihood and other modifications. In International Conference on Learning Representations, 2017.
Shaun Seaman, John Galati, Dan Jackson, and John Carlin. What is meant by "missing at random""? Statistical Science, pp. 257-268, 2013.
Nicki Skafte, Martin Jergensen, and Søren Hauberg. Reliable training and estimation of variance networks. In Advances in Neural Information Processing Systems, pp. 6326-6336, 2019.
Marek Smieja, Fukasz Struski, Jacek Tabor, Bartosz Zieliński, and Przemyslaw Spurek. Processing of missing data by neural networks. In Advances in Neural Information Processing Systems, pp- 2719-2729, 2018.
Daniel Stekhoven and Peter Bühlmann. Missforest- non normmmeretttiic missing value imputation for mixed-type data. Bioinformatics, 18-1): 2012.
Michael E Tipping and Christopher M Bishop. Probabilistic principal component analysis. Journal of the Royal Statistical Society: Series B (Statistical Methodology), 61(3):61 1-622, 1999.
12

---

Published as a conference paper at ICLR 2022
Volker Tresp, Subutai Ahmad, and Ralph Neuneier. Training neural networks with deficient data. In Advances in Neural Information Processing Systems, pp. 128-135, 1994.
Volker Tresp, Ralph Neuneier, and Subutai Ahmad, Efficient methods for dealing with missing data in supervised learning. In Advances Neural Information Processing Systems, pp. 689-696, 1995.
Bheki E T H Twala, M C Jones, and David Hand. Good methods for coping with missing data in decision trees. Pattern Recognition Letters, 29(7):950 956, 2008.
Han Xiao, Kashif Rasul, and Roland Vollgraf. Fashion-MNIST: a novel image dataset for bench- marking machine learning algorithms. arXiv preprint arXiv: (70.:00770007747 2017.
Joonyoung Yi, Juhyuk Lee, Kwang Joon Kim, Sung Ju Hwang, and Eunho Yang. Why to use zero imputation? correcting sparsity bias in training neural networks. arXiv preprint 1eXiv:1966.00150 2019.
Jinsung Yoon, James Jordon, and Mihaela Van Der Schaar. GAIN: missing data imputation using generative adversarial nets. In International Conference on Machine Learning, 2018.
Manzil Zaheer, Satwik Kottur, Siamak Ravanbakhsh, Barnabas Poczos, Russ R Salakhutdinov, and Alexander J Smola. Deep sets. In Advances Neural Information Processing Systems, pp- 3391-3401,2011
13

---

Published as a conference paper at ICLR 2022
A TRAINING DEEP SUPERVISED MODELS WITH MISSING DATA
In the following we group different strategies for handling missing data in supervised deep learning.
A.1 CONSTANT IMPUTATION
We denote imputation methods where there is a single constant for each feature which replaces missing data as constant imputation. We define imputation function in general such that (() E X and ((x)abs = x A simple version of constant imputation is O-imputation, which has the intuitive appeal that the activation from the input node is zeroed out (absent) and no gradient updates
are made for the weights for that feature.
to <x(x) = x x S (1-s).
(18)
In the unsupervised setting, constant imputation biases marginal and joint distributions, but Josse jet al. (2019); Le Morvan et al. (2021); Bertsimas et al. (2021) have shown that mean imputation can be consistent in the supervised setting. Furthermore, Le Morvan et al. (2020b); Tresp et al. (1994) noted that the constants can be optimized with respect to the prediction loss.
Learnable parameters À X' can be inserted in place of the missing data, that
(x = =00
(19)
We denote this learnable imputation.
A.2 REGRESSIVE/MODEL BASED SINGLE IMPUTATION
In contrast to constant imputation. methods such as MICE (Buuren & Sebothuis-Oudshoon 2010) MissForest (Stekhoven & Bühlmann 2012), PPCA (Tipping Bishop 1999; Roweis 1998) flex-
ible generative models (Mattei & Frellsen 2019 Yoon el al. 2018) provides imputation of missing
features conditional on observed features p
(x) x G Epxxii pett)
[x]
④ (1 - s)
(20)
In MICE, this conditional distribution comes from an implicit joint model over the covariates by regressing each covariate with missing values on all the others, round-robin fashion. MissForest is somewhat similar to MICE, using random forests as the regressors. Probabilistic PCA defines an explicit joint model over all covariates and single imputations of missing data can obtained by conditioning on observed covariates. Here, MICE is proper Bayesian multiple imputation method, while PPCA and MissForest draws multiple imputations without regard to parameter uncertaintics. The recent advances in imputation of missing data using deep generative models can also be used here to the extent that they impose a joint distribution over covariates.
A.3 PERMUTATION INVARIANCE
Ma et al. 2018 2019) used the permutation invariant setup (Zaheer et al. 2017 Qi et al. 2017) handle missing data in the context of variational autoencoders. For each feature j obs (), there is an embedding e, E RR", which in general can hold any auxiliary information, but in our case will be learnable embedding. Each element of the embedding is multiplied with the feature value t; Cj- Iy and fed to neural network ht-)) mapping inputs from R to RM where U the embedding dimension and M is the dimension of the code-space being mapped into. This
permutation invariant setup finally aggregates all the neural network outputs by summation to get one fixed-length vector, that can be fed into a discriminative network,
(x
Σ
h(tz))
(21)
Note that in this case (xb()) M while in the other imputation functions ((x) X.
14

---

Published as a conference paper at ICLR 2022
A.4 TREE METHODS
Decision trees can in some cases handle missing data directly, without the need for imputation. There are different approaches to handle missing values in tree based methods, see (Josse al. 2019) for a review. In our experiments we use histogram-basee gradient boosting (Friedman (001) as implemented in scikit-learn (Pedregosa et al. 201T). This approach uses the Block Propagation method for the missing data. While discriminative model using MLPs has fixed capacity, gradient boosting methods can adapt its capacity terms of tree depth based on validation set error.
B GRADIENTS
The log likelihood, when integrating over missing input features, is specified as log Pos (y)xx + log po (xx00))
log Pog.o.o x
(22)
where
log P ( = log
Pe (v/ x
x
miss
pe (x
miss (xs miss
(23)
Both Tresp et al. 1994) and Ghahramani Jordan with respect to the discriminative network parameters.
1995) analyzed the gradient of the log likelihood
i log Po.0 (y. x i log (b
a Olog (x:
(24)
dΦ
dΦ
OΦ
@plxx
(25)
Pe P x0bs
OΦb
/ Pe P (3/ (bl
x miss
) (x /x
dis
(26)
deb
/
Opolylxoos miss)
po (xii (x dx
(27)
Pe (y)xx)))))
O6
Here we see that the gradient of the discriminative network parameters, due to the classifica- tion/regression loss, a weighted sum over all possible missing values, where the missing values are weighted according their density in the data model, conditional on the observed data.
The gradient with respect to the 0 parameters depends to some extend on the & parameters.
i log Po.o (y, xbbs log P (u/xx
log po (x)s))
(28)
00
00
00
1
@po
log pe (x
+
(29)
Pe (
00
00
1
a
/
a log po (x
Po (0/x
miss po (x miss
nho cx miss
Po (1((xxuuby
00
00
(30)
1
0
E
px bott) (poluixoon xmlsss)
Ô log pe pop (x
(31)
Po (y) xbb))
00
00
(32)
There are contributions to the gradient from both the generative part of the model and the discrim- inative part. This means that, if needed, the discriminative model can affect the generative model, as is the case in (Ipsen al.] 12021). The gradient of the expectation be approached using the reparameterization trick (Kingma & Welling 2014 Rezende et al. 2014).
C RELATION TO MULTIPLE IMPUTATION
The gold standard for imputation multiple imputation Rubii 1996). This requires learning a model of the observed covariates obs with parameters 0. K sets of imputations then drawn from
15

---

Also note that we have no need for several fixed sets of imputations, at each gradient step a new of samples are drawn from the imputation model, miss Pe (xi xh))). The number of imputations can be set dynamically, using 5-50 during training and 10k for testing. Our approach, using Published as a conference paper at ICLR 2022
the posterior predictive distribution
X
miss p(x miss /xx"s////000000 ) == I= K
(33)
where p(x x J p(x 0 p( x d0 if the model is Bayesian (see e.g. Gelman et al,, 2013] chapter 18.2). Several frequentist multiple imputation methods also exist, where the uncer- tainty with respect to the imputation model parameters is disregarded. In that case, the Bayesian posterior predictive is replaced by conditional distribution p(x xxbs) = Pa P (xmsss (xbs). where
is any point estimate of 0.
In the case of regresssion/classficttinn if we roughly follow the notations of our paper, the analysis proceeds as follows
For each set of imputed values (xxtk " lisn,kK. use a learning algorithm to estimate the discriminative parameters 11,-.... PK:
i = arg max
log (uil ,x11 Σ
(34)
ΦK = arg max
log P&b Po (x] obs miss
(35)
&
Obtain an average estimate of & using
K
ΦK
Sx
(36)
K
and variance estimate can similarly be obtained as described in (Gelman et al. 2013
Rubin 1996).
In the limit of infinite imputations, we can write this as the conditional expectation
Φ
Rigiii
arg max Σ log Po [x;
x miss)
x1
(37)
Instead of fitting K conditional models to K imputed datasets, our approach (when 0 is fixed) is, in comparison, to directly maximize the expectation the conditional distribution with respect to the missing data
Φ = arg max Σ log E X Po [U (x
.x,iix |x
(38)
DLVMS as imputation models, is not Bayesian, but any available proper Bayesian imputation model could be used in place of the DLVMS.
In practice, an important difference between traditional multiple imputation and our approach, is that multiple imputation uses a uniform weight 1/K for each imputation, while we use importance weights, that will treat each imputation based on its likelihood.
D THEORETICAL PROPERTIES OF THE SUPMIWAE BOUND
We will briefly see how the Mup. bound has the same theoretical properties as IWAE bounds.
We follow Ipsen et al. (2021 appendix D), who were following Burda al. (2016) and Domke & Sheldon (2018). Recall the definition of the bound: we have
K
( xb
x ) (xobs (ax (2o:))
RK Rx ob ,y) =
(39)
K
16

---

Published as a conference paper at ICLR 2022
Half-moons
Circles
Burger
Pin-wheels
Figure 8: 2D datasets with two or more classes.
and
LK(0, 0.
Σ
E (xtii mn) log Rx (xc xi)
(40)
The ith term of the sum can be seen regular IWAE bound with observation (x .y). latent
variable (z, x with prior pe (x miss z pe (z), and variational distribution pe( 2) (z) xxxx
Therefore, Theorem 1 from Burda al. (016) can be applied, and leads to the monotonicity of the bound: (9 Φ,, 7) v ... < Lx 0,7)
To show that the bound additionally converges the log- likelihood when K - co, we Theo- rem 3 of Domke & Sheldon (2018) for all n terms of the sum that constitutes the bound.
Theorem. Assuming that, for all E (I....,..
there exists a such that [ [xt" y.) (x, 2t) < 00,
limsupp E [1/Rx] < <0,
the supMIWAE bound converges to the true joint likelihood at rate 1/K:
Var Rı (xcebr ys)] 20o (x, y:) y
log Po.,oly,,, Cx(0. 10000
(41)
E 2D CLASSIFICATION
The datasets used in the 2D classification experiments are shown in figure The Half-moons and Circles datasets are from scikit-learn (Pedregosa et al. 201T). The Pin-wheel dataset is modified from Johnson et al. [ The Burger dataset, we have designed to illustrate the case where optimal single imputation requires severe changes in the decision surface.
The discriminative model is an MLP with 3 hidden layers with 50 hidden units each. The final layer parameterizes categorical distribution over the dataset classes. The generative model used for the MIWAE single imputations and the supMIWAE consists of an encoder and decoder with 3 hidden layers containing 50 hidden units each. The encoder parameterizes an isotropic Gaussian distribution over a latent space with dimension same the feature space. The decoder parameterizes an isotropic Gaussian distribution over the feature space. In figure 9 samples from the generative model are shown, that is, samples from the MIWAE trained on each of the datasets.
In (Smieja et al. 2018) covariates while jointly training a discriminator. In case of missing values, the discriminator's first input neuron's 's activations over the missing values in the GMM. We adapt the official code to this experiment by adjusting the number of hidden layers and number of mixture components while keeping the original training setup as is.
Gaussian Mixture Model, GMM, is used to model the density of the
set to the average activations found
conditional distribution
17

---

Published as a conference paper at ICLR 2022
In figures 10 13 decision surfaces and imputations are shown. The corresponding quantitative sults are in tables
Figure 9: Samples from the MIWAE, fitted to the data with missing values on the four 2D datasets. The class color is determined by afterwards sampling from the discriminative distribution.
18

---

Published as a conference paper at ICLR 2022
Table 1: Half-moons test-set results
model
missking
on
test log p(y(a)
missing
supMIWAE
-0.52
-0.32
MIWAL
0.88 + 0.00
0.77
0.01
0.99 + 0.00
-0.42 + 0.01
-0.52 = 0.01
-0.33
0.01
0 impute
0.88 1 0.01
0.77
2 0.02
1.00 1: 0.00
-0.42 4: 0.00
-0.52 0.01
-0.32
0.00
iimputttion
0.89 1: 0.01
0.75
0.001
1.00 0.00
-0.42 2 0.00
-0.52 -002
-0.32 0.000
missFurest
0.87 + 0.00
0.75
0.01
0.99 0.001
-0.45 4: 0.00
-0.33 0.01
-0.37 0.00
PPCA
0.88 + 0.01
0.77 + 0.02
1.00 0.00
-0.42 = 0.00
-0.02 0.01 0.01
-0.32 0.00
0.00
0.00
Pormatation
Smigaa al. 2018)
0.83 +: 0.04
Table 2: Burger test-set results
model
observed
log p(v(z)
observed
SUOMIWAE
0.81 + 0.01
01.4
0.03
0.97 + 0.01
-0.51 + 0.00
-0.06 0.01
-0.35
0.01
MIWAE
+
-0.67
-0.38
Osingute
0.80
0.03
-0.51
-0.66
-0.35 0.000
0.81 0:01
0.64 = 0.03
0.97 + 0.01
-0.51
= 6.00
-0.66 0.01
-0.55 -0.55 0.00
MICE
0.80 + 0.01
0.64
0.962:.00
-0.51 - 0.00
-0.66
-0.35
missForest
0.64
-0.58
-0.47
PECA
0.36
GR
0.80 + 0.01
0.00
0.97 0.97 0.01
-0.51 : 0.01
-0.66 0.01
-0.35
arrmuration-immiiae
0.81 + 0.01
0.64 0.03
0.97 0.00
0.01
-0.51 = 0.00
-0.06 0.01
-0.35
Siigja
2013)
Table 3: Circles test-set results
model
12SE ace
on missing
on observed
test log p(yla))
on missing
-on observed
SupMIWAE
+ 0.01
0.77
0.01
0.99 0.01
-0.44
-0.56 0.01
-0.32 0.01
MIWAF
0.87 0.01
0.76
= 0.02
0.99
0.01
-0.45 = 0.01
-0.56
0.00
-0.33 2:33:000 0.00
impule
0.88 0.01
0.77
: 0.02
0.99:1:000
-0.44
-0.56 0.00
-0.33
leumabl- iimmmalt-mmmatiiionnn
MICE
0.88 0.01
0.77
0.02
0.99
-0.44
-0.56-000
-0.33
missFurest
0.86 1.6.1000 0.01
0.75 0.01
0.901
-0.48 0.01
0.56-0001
-0.39 -0.39 0.001
PPCA
0.88 0.01
0.77 0.00
0.99
-0.44 0.00
-0.56
-0.33
permutation imariance
0.88 1 0.01
0.77 0.002
0.99110.00
- 0.00
-0.56 0.01
-0.32 0.00
Smigaa al 2018)
0.86 1 0.02
Table 4: Pin-wheel test-set results
model
observed
log p(ylat
missing
observed
SipMIWAE
u.77
1.00
0.01
-0.75
0.00
O O-impute
0.58
0.01
0.77
1 0.011
1.00 1: 0.00
-0.88
0.01
-1.02
0.00
-0.75
0.0.
Jeamable- imputation
0.88 + 0.00
u.77 + 0.01
1.00
0.00
-0.88 4 0.01
-1.22
0.01
-0.70
0.00
MICE
0.77
1.00
-1.02
-0.75
0.000
PPCA
0.88 + 0.03
0.77 + 0.01
1.00 0.00
-0.88 + 6.01
-1.02 0.01
-0.75 0.00
GB
0.87 + 0.01
0.74
0.02
1.00 0.00
-0.88 4 0.01
-1.02 0.01
-0.75 0.00
permatation-innen innieinne
0.00
-1.02
19

---

Published as a conference paper at ICLR 2022
supMIWAE
MIWAE
0-impute
Perm. Inv.
(Smieja al 2018
Leamablee Imputation
Gradient Boosting
Figure 10: First and third rows: (except top left) kernel density of the half-moons dataset together with imputations from the given model. The supMIWAE does rely single imputations, in- stead it draws multiple importance samples which are passed through the discriminator to give an importance weighted prediction. For the SupMIWAE multiple imputations at three different values of the horizontal coordinate are shown and kernel density of the multiple imputations are shown to the left. Permutation invariance, the joint GMM and discriminator approach of Smieja et al. (2018) and gradient boosting does not rely on explicit imputations, for the of the methods single im- putations are shown in red. Second and fourth rows: Decision surfaces learnt, depending on the strategy for handling missing values.
20

---

Published as a conference paper at ICLR 2022
SUpMIWAE
MIWAE
0-impute
Perm. Inv.
Smieja al. 2018
Learablee Imputation
MissForest
Gradient Boosting
Figure 11: First and third rows: (except top left) kernel density of the burger dataset together with imputations from the given model. The supMIWAE does not rely single imputations, in- stead it draws multiple importance samples which are passed through the discriminator to give an importance weighted prediction. For the SupMIWAE multiple imputations at three different values of the horizontal coordinate are shown and kernel density of the multiple imputations are shown to the left. Permutation invariance, the joint GMM and discriminator approach of Smieja et al. (2018) and gradient boosting does not rely on explicit imputations, for the of the methods single im- putations are shown in red. Second and fourth rows: Decision surfaces learnt, depending on the strategy for handling missing values.
21

---

Published as a conference paper at ICLR 2022
supMIWAE
MIWAE
O-impute
Perm. Inv.
Smieja al 2018
Imputation
Gradient Boosting
Figure 12: First and third rows: (except top left) kernel density of the circles dataset together with imputations from the given model. The supMIWAE does not rely single imputations, in- stead it draws multiple importance samples which are passed through the discriminator to give an importance weighted prediction. For the SupMIWAE multiple imputations at three different values of the horizontal coordinate are shown and kernel density of the multiple imputations are shown to the left. Permutation invariance, the joint GMM and discriminator approach of Smieja et al. (2018) and gradient boosting does not rely on explicit imputations, for the of the methods single im- putations are shown in red. Second and fourth rows: Decision surfaces learnt, depending on the strategy for handling missing values.
22

---

Published as a conference paper at ICLR 2022
supMIWAE
MIWAE
0-impate Permutation Invariance
Leamable Imputation
MICE
PPCA
Gradient Boosting
Figure 13: First and third rows: (except top left) kernel density the pin-wheel dataset together with imputations from the given model. The supMIWAE does not rely single imputations, in- stead it draws multiple importance samples which are passed through the discriminator to give an importance weighted prediction. For the SupMIWAE multiple imputations at three different values of the horizontal coordinate are shown and a kernel density of the multiple imputations are shown the left. Permutation invariance and gradient boosting does not rely explicit imputations, for the rest of the methods single imputations are shown in red. Second and fourth rows: Decision surfaces learnt, depending on the strategy for handling missing values.
23

---

Published as a conference paper at ICLR 2022
E.1 CAPACITY
The predictive performance of the discriminator at different capacities, terms of number of hidden units, is shown in figure [4
burger
burger
0.82
-0.50
0.80
tast logplylx)
-0.5
00.78
0.76
-0.44
0.74
-0.56
0.72
-0588
0.70
0000
7
10
8
10
capacity
capacity
circles
circles
-0.45
0.85
test logp(y(x)
test 300
0.80
-0.50
0.75
-0.55
0.70
-0.60
0.65
-0.65
4
7
10
7
8
9
10
capacity
capacity
half-moons
half-moons
0.90
-0.425
0.88
-0.450
test age
0.86
0.84
best Jeol -.500
0.82
-0.525
-05550
0.80
-05755
0.78
-0.600
4
7
8
10
2 3
8
9
10
capacity
capacity
pin-wheel
pin-wheel
-0.875
0.88
X
-0.000
test, a0c
0.86
1 -0.950
-0.225
Joo0 -0.975
0.84
1.000
0.82
RES -1.025
-1.050
0.80
-1.075
2
3
4
5
7
8
9
10
2
3
8
9
10
capacity
capacity
MOSE
misoTors PPCA
Figure 14: Predictive performance when varying the capacity the learner, in terms of hidden units.
24

---

Published as a conference paper at ICLR 2022
F IMAGE CLASSIFICATION
F.I TRAINING DETAILS FOR MNIST/EMNIST
For all MNIST/MMNIST experiments, the discriminative model convolutional neural network with four layers and categorical distribution over classes, see table 5 The generative model uses a convolutional architecture similar to the DCGAN (Radford et al. 2015) and a latent space of dimension 128. A Continuous Bernoulli (Loaiza-Ganem & Cunningham (2019) observation model used for MNIST and the discretized logistic distribution (Salimans et (0017) is used as the observation model for Fashion MNIST. During training, K" =5 importance samples and batch size of 128 are used.
Table 5: Discriminative network for MNIST//MNIST
classfficaioooo classification.
Action (resulting layer size)
Input æ (28 x 28 1)
Conv2D(14 x 14 x 16)
Conv2D(7 x 7 x 32)
Conv2D(3 3 x 64)
Reshape(576) Class probabilities: Dense(10)
F.2 IMPUTATIONS
Figures 15 show imputations on the image datasets for the three different MCAR missing processes. The first column contains data with missing values. The second column is single im- putation by the MIWAE, and following columns are MIWAE multiple imputations using sampling- peportance-resamplin
F.3 NATURAL IMAGES (SVHN)
In order to assess classification of natural images we now turn to the Street View House Numbers dataset (SVHN, Netzer et al. 2011), with different missing mechanisms. In the "observed squares" missing mechanism 20 x 20 randomly placed squares are observed while in the "missing squares" missing mechanism 16 16 randomly placed squares missing. In the "random dropout" missing mechanism pixels are missing with probability m 0.5.
In the "observed squares" experiment we apply both a low capacity layer discriminator (1 repeti- tion) and higher capacity 8 layer discriminator (3 repetitions) using the convolutional gated blocks from Dauphin et al. (2117). see architectures in tables 7 and g In table 6 test set accuracies are shown and for the 8 layer discriminator, the accuracy on the fully observed test set is also shown. In the low capacity regime, the supMIWAE outperforms the two single imputation methods and performs on par with MIWAE. With the higher capacity discriminator supMIWAE has performance similar to 0-imputation and learnable imputation, while the MIWAE imputation is slightly worse. This displays the same phenomenon in the 2D classification experiments, where powerful learn- are able to undo single imputations. While the discriminative models learnt based on constant imputation can achieve the same performance as the SupMIWAE, the kinds of discriminative models that are learnt are different. In this case, when applied fully observed test-set the discriminator learnt by the supMIWAE outperforms the other models.
The high capacity discriminator is also applied to the "missing squares" and "random dropout" missing mechanisms, see figure for the test-set accuracies (3 repetitions). In figure 19 models trained on data with missing values applied to fully observed test-set.
25

---

Published as a conference paper at ICLR 2022
7
7
7
7
7
7
7
7
7
7
7
7
2
2
2
2
2
2
2
2
2
2
2
1
d
d
7
7
7
d
J
7
7
7
0
0
0
O
G
O
G
O
O
O
9
9
4 1 7
9 1
7
4
4
4
9
1
1
4
7
9
7
7
7
7
7 4
7 R
R
A
R
4
5
7
9
9
7
9
9
Q
O
O 6
6
4
4 9
4 DOO
9
4
4
0
1
1
1
1
5
5
3 3
3
0
14
4
4
4
a
8
4
4
4
7
7
7
7
7
7
7
7 6
7 6 4
7 6
8
6
8
8
6
6
6
8
4
4
4
H
(a) MNIST, Continuous Bernoull..
(b) Fashion, discretized logistic.
Figure 15: Observed squares. Left column contains data with missing values represented as red pixels. Second column contains single imputations from the MWIAE using self-normalized impor- tance sampling. Following columns contain multiple from the MIWAE using sampling-importanee resampling.
Table 6: SVHN observed squares classification accuracies
acc (incomplete test set)
acc (complete test set)
model
layers
8 layers
layers
supMIWAE
0.8264
0.8795 + 0.0027
0.9268 + 0.0015
MIWAE
0.8233
0.8745 0.0024
0.9209 0.0010
0-impute
0,8155
0.8819 + 0.0024
0.9059 + 0.0064
learnable-imputation
0.8139
0.8798 0.0042
0.8953 0.0234
G REGRESSION
All discriminative models are neural networks with one hidden layer and 50 hidden units, where the final layer of the neural networks parameterizes Gaussian distribution. The datasets are split randomly 20 times with 90% of the data in training set and 10% in a test set. A validation set with 10% the training data used for early stopping and batch size of 256 is used. For the permutation invariant setup, an embedding of dimension U 20 and fixed length vectors (xb)) of dimension M 10 are used. Missing values are introduced completely at random in the training, validation and test sets removing each feature with probability m, where is the missing rate. In order to avoid reducing the sample size, due to completely missing covariates, whenever all the covariates of an observation go missing,
of them observed, selected at random.
26

---

Published as a conference paper at ICLR 2022
7
7
7
7
7
7
71
7
7 2
7 2
7 2
2
2
2
2
2
2
2
2
d
7
7
7
T
4
9
9
Q
5
5
5
5 5
7
9
9
7
7
7
1
4
(a) MNIST, Continuous Bernoulli.
(b) Fashion, discretized logistic.
Figure 16: Missing squares. Left column contains data with missing values represented as red pixels. Second column contains single imputations from the MWIAE using self-normalized impor- tance sampling. Following columns contain multiple from the MIWAE using compling-importancee resampling.
Table 7: Low capacity (4-layer) discriminative network for SVHN classification.
Action (resulting layer size)
Input (32 32 3)
Conv2D(16 x 16 x 16)
Conv2D(8 8 x 32)
Conv2D(4 (44x)) x 4 x 64)
Reshape (1024)
Class probabilities: Dense(10)
For the generative models, 2 layered neural networks with 128 hidden units used for the encoder and the decoder and a latent space of dimension 10. A Gaussian observation model used and K 25 importance samples are used during training.
We compare different imputation techniques, modelling approaches and a non-deep competitor. Zero-imputation. MissForest (Stekhoven Bühlmann, (012)) and MICE (Buuren & Groothuis- Oudshoorn 2010) are different imputation techniques applied, before training the discriminative model. Learnable imputation using learnable constants as imputation, while permutation invari- ance is utilizing a learnable embedding. Finally. histogram-basee based gradient boosting (Friedman) 2001), as implemented in sklearn (Pedregosa al.) (201T). 1), non-deep baseline, using the valida- tion set for early stopping. Results are seen in figure 20
27

---

Published as a conference paper at ICLR 2022
7
7
7
7
7
7
7
7
7
7
7
2
2- 2-
2-
2
2
2
2
2
2
2
2
/
/
/
1
/
/ 0 4
/ 0 4
0 O
3
4 4
4
4
4
4
4
4
E
1 I
1
4
Y 4 4 4 4
4
4 4 9
9 9
O 00000000000000000000000000000000
6
6 666666666
9 99999999999999
0
O O 0000000000000
J /
55555555555
5
9
9
9
9
9 9
7
7
8
8
8
4
4
4
4
4
4
4
4
4
(a) MNIST, Continuous Bernouli..
(b) Fashion, discretized logistic.
Figure 17: Missing squares. Left column contains data with missing values represented as red pixels. Second column contains single imputations from the MWIAE using self-normalizee impor- tance sampling. Following columns contain multiple from the MIWAE using resampling.
Table 8: High capacity (8-layer) discriminative network for SVHN classification.
Action (resulting layer size)
Input æ (32 x 32 x 3)
Conv2D(16 16 16)
Conv2D(8 x 8 x 32)
Gated block (Dauphin et 2017
(8 32)
Gated block (Dauphin et al.
2017
(8 x 32)
Gated block (Dauphin et al.
2017
(8 x x 32)
Gated block (Dauphin et al.
2017
(8 x 32)
Conv2D(4 x x 64)
Reshape(102))
Class probabilities: Dense(10)
28

---

Published as a conference paper at ICLR 2022
squares
missing squares
dropout
0.884
0.930
test acc
0.882
0.80
MIWAR
0.925
0.79
0,878
0.876
0.70
0.915
0.872
0.76
0.910
Test set accuracies the SVHN dataset, with different missing mechanisms. Left column: observed squares. Middle column: missing squares. Right column: random dropout with missing rate 0.5.
Figure 18:
0.93
squares
missing squares
0.93d
dropout
0.92
test 80C
0.930
0.89
0.928
0.88
Olmpett
0.905
0.86
Figure 19: Fully observed test set accuracies the SVHN dataset, with different missing mecha- nisms. Left column: observed squares. Middle column: missing squares. Right column: random dropout with missing rate 0.5.
29

---

Published as a conference paper at ICLR 2022
boston
carbon
ConCEete
1.0
1 0.7
0.9
0.0
109 R8
996 183
0.8
0.7
0.9
0.6
8.5
0.6
0.5
0.4
0.4
0.3
0.1
0.2
0.3 0.5 0.6
0.8 0.9
0.1
0.2
0.3
0.4
0.5 0.6 0.7 0.0 0.9
0.2 0.3
0.4 0.5
0.7
0.0 0.9
missing
missing
missing rate
0.8
energy
1.0
naval
0.8
power-plant
380 0.6
0.7
0.B
3888 7894
0.8
0.7
388 0.4
0.5
0.5
0.4
0.4
0.2
0.3
maising
missinn
missing
protein
superconduurtwity
0.7
0.6
0.7
0.95
0.
0.5
0.6
0.90
0.5
0.85
0.8
0.80
0.3
missing
missing
SUpMIWAL
Wine-White
yncht
MEWAE
1.0
O-impute
0.95
Jearnable lebbll--ppuuation
0.00
MICE
D.6
0.80
0.4
PPCA
0.1 0.2 0.3 0.4 0.5 D.6 0.7 0.8 0.9
0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 missing rate
permutation invariance
missing rate
Figure 20: Test-set root mean square error on UCI datasets at varying missing rates.
30


Reviews:
[{"review_id": "dRybc9SzXzj", "reviewer": "Reviewer_w981", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strenghts:\n- Strong theoretical foundations\n- Clearly written with nice illustrations of the used principles.\n- Good review of previous approaches to the supervise learning with missing data problem\n\nWeakness:\n- While the authors claim that their method is for deep learning, the provided examples are given on small datasets and used neural networks are not very deep. For example, a 4-layer CNN is used for MNIST datasets. It is not clear if the proposed method could be used on bigger datasets and deeper architectures.\n- A comparison of computational time with respect to other methods is missing, which makes to think that maybe this method is too expensive to implement for big datasets and deep learning architectures.\n- Experiments do not contain comparison with previous methods other than imputation based ones. There are recently published methods cited by the authors that provided implementation code for learning classifiers and missing data at the same time, i.e. avoiding the imputation followed by classification approach.\n", "summary_of_the_paper": "This paper approaches the problem of supervised learning with missing data. The authors propose a probabilistic approach by jointly modeling the observed data, missing data and outcomes. The main contribution of this model is that they rely on deep generative models which seems to be an improvement from previous methods based on simpler generative models like mixture of Gaussians, for example. With the proposed model, the authors derive an optimization problem that consists in optimizing the discriminative model (classifier) and the generative model simultaneously. For the training and testing phases, the proposed algorithm considers multiple imputations of missing data, which is demonstrated to be superior to single imputation methods. Experimental results on small and simple datasets (2D datasets, MNIST digits-Fashion and regression) are nicely presented and analyzed.", "summary_of_the_review": "A good paper including very clear and technically sound principles for solving the problem of supervised learning with missing data. Experiments are illustrative although it would be needed to implement them on bigger datasets and deeper neural network architectures as well as to compare to other methods in the literature.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "cLqiZ3bif1Q", "reviewer": "Reviewer_FaAu", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Pros:\n1. The authors do a good job of explaining the challenges of training discriminative models with missing data as well as going over the related works.\n2. Their approach is modular and extendable to many different choices of neural network architectures. \n3. The paper is well written and easy to follow. Good work there!\n\nConcerns and questions:\n1. Are eq. 13, 14 novel contributions of this work? Or these were previously derived and new to the proposed missing value handling settings?\n2. The novelty of the work is marginal. The multiple imputation idea, viewing the imputation problem as joint distributions (eq. 1,2) and also going through a few of the related works mentioned in this paper, I feel that most of the ideas proposed in this work are already out there. This work finds a combination of them which works well but the experiments are not extensive. \n3. The experiments are restricted to very small datasets. Would definitely like to see how this method performs on scale. As, I can foresee some issues with the training as it uses sampling. \n", "summary_of_the_paper": "The paper handles the issue of missing values in supervised deep learning settings. The fig.1 describes their method aptly. Their method (supMIWAE) is a combination of a VAE with a neural network classifier. Given the task to predict p(Y|x_{obs}, x_{miss}), the authors view it as a joint model of covariates and outcomes. Their model takes in x_{obs} and first fits a distribution to get x_{miss} and then models them as a joint distribution to predict outcome Y|(x_{obs}, x_{miss}). They use importance sampling technique to get multiple samples from the generative model. ", "summary_of_the_review": "The work is good but not novel enough. Need to show results on larger datasets. Please also highlight the fail cases in the paper. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "TS-kVuewlKQ", "reviewer": "Reviewer_zqzK", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper is written clearly and there are no obvious errors I can see in the theory or the experiments presented in this paper. However, I have not checked some of the derivations in detail. The method presented in the paper is largely based on previous work on variational inference, but it is otherwise novel and noteworthy theoretical contribution. Their application to train a joint model of covariates and outcomes with DLVMs and neural classifier in the presence of missing data is a fairly novel approach to my knowledge. The claims made in the paper are well backed by experiments and therefore the results are quite solid. In terms of impact, the problem of missing values is a very ubiquitous one and therefore the potential impact of the paper is significant. Moreover, the results presented by the authors indicate that their method outperforms other relevant approaches (single imputations methods) in a restricted number of scenarios: low-capacity regimes, or when the discriminative model has a strong inductive bias.  ", "summary_of_the_paper": "The paper presents a method to more optimally learn deep learning classifiers in the presence of missing values. More specifically, the authors present the approach called supervised missing data importance-weighted autoencoder (supMIWAE) bound which allows them to train a joint model of covariates and outcomes by marginalizing over the missing values. ", "summary_of_the_review": "Noteworthy application of variational auto-encoders to improve classification in the presence of missing data. Overall a solid paper, I recommend accepting the paper in the conference.  ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "S6iMzJ0XlM", "reviewer": "Reviewer_U6yy", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n+\tThe proposed method offers end to end training, which is useful for scalability and transferability. They claim that such a method can be applied to discriminators. \n+\tauthors have both visually and empirically shown improvements in data imputation for MNIST and Fashion-MNIST\n+\tIt is shown to perform better than MIWAE, (Mattei & Frellsen, 2019), the main baseline for this works\n\nWeaknesses:\n-\tWhile there are some gains in performance, it feels incremental both in % increase and scale. MNIST and FMNIST are very similar small, almost binary valued datasets, and weakly supports the claim of scalability.\n-\tIn UCI datasets, where features are real numbers, and probably contain more information per feature, supMIWAE performs at par with MIWAE.\n-\tMost aspects of the technique are leveraged from the prior works. \n\nGeneral comments:\n●\tHow does the supervision and joint training impact the training complexity of the system?\n●\tInterestingly, supMIWAE performs definitely better than MIWAE for MNIST, and FMNIST, however such gains are no longer visible for UCI datasets. The gains of joint training seem to be limited to a certain type of data. \n●\tIn terms of practical applications, where do the authors see their method bring the most gains? Image, language, or statistical datasets. \n\n", "summary_of_the_paper": "The authors propose supMIWAE, a supervised missing data importance-weighted autoencoder, to address the challenges of training with missing data. They claim that the new approach, a VAE combined with a discriminator, jointly trained, is more scalable, and performs better than existing data imputation methods, in particular MIWAE (Mattei & Frellsen, 2019) ", "summary_of_the_review": "The addition of supervision and joint training to MIWAE seems novel, and while the work improves upon the prior methods of using DLVM for missing data, it falls short in the claim of scalability and overall gains in performance across different types of datasets. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper addresses features that are missing at random in deep supervised learning, especially regression and classification. A deep latent generative model is trained in conjunction with a discriminative model, so that the distribution of the covariates is properly modeled and allows efficient variational inference for imputation.  Superior performance is achieved in low-capacity domain or when strong inductive bias is present in the discriminative model.

The paper is well written, with solid empirical support.  The approach is also generic.  Although there is some concern on novelty, overall this paper appears a solid contribution and is a good addition to the proceedings.
Decision: Accept (Poster)