not-MIWAE: Deep Generative Modelling with Missing not at Random Data
Authors: Niels Bruun Ipsen
Pierre-Alexandre Mattei
Jes Frellsen
Venue: ICLR 2021 Poster (2021)

Abstract: When a missing process depends on the missing values themselves, it needs to be explicitly modelled and taken into account while doing likelihood-based inference. We present an approach for building and fitting deep latent variable models (DLVMs) in cases where the missing process is dependent on the missing data. Specifically, a deep neural network enables us to flexibly model the conditional distribution of the missingness pattern given the data. This allows for incorporating prior information about the type of missingness (e.g.~self-censoring) into the model. Our inference technique, based on importance-weighted variational inference, involves maximising a lower bound of the joint likelihood. Stochastic gradients of the bound are obtained by using the reparameterisation trick both in latent space and data space. We show on various kinds of data sets and missingness patterns that explicitly modelling the missing process can be invaluable.

Published as a conference paper at ICLR 2021
NOT-MIWAE: DEEP GENERATIVE MODELLING WITH MISSING NOT AT RANDOM DATA
Niels Bruun Ipsen bb i tbiipddu.com dk
Pierre-Alexandre Mattei 1
Jes Frellsen*1 jefrêdtu.dk dk
ABSTRACT
When a missing process depends on the missing values themselves, it needs be explicitly modelled and taken into account while doing likelihood-basee infer- ence. We present an approach for building and fitting deep latent variable models (DLVMs) in cases where the missing process is dependent on the missing data. Specifically, deep neural network enables us to flexibly model the conditional distribution of the missingness pattern given the data. This allows for incorpo- rating prior information about the type missingness (c.g. self-censoring) into variational
the model. Our inference technique, based on importance-weighhae inference, involves maximising a lower bound of the joint likelihood. Stochastic gradients of the bound are obtained by using the reparameterisation trick both in latent space and data space. We show on various kinds of data sets and missing- ness patterns that explicitly modelling the missing process can be invaluable.
INTRODUCTION
Missing data often constitute systemic issues in real-world data analysis, and can be an integral part of some fields, e.g. recommender systems. This requires the analyst to take action by ei- ther using methods and models that are appli- cable to incomplete data or by performing im- putations of the missing data before applying models requiring complete data. The expected model performance (often measured in terms of imputation error or innocuity of missingness on the inference results) depends on the as- sumptions made about the missing mechanism and how well those assumptions match the true missing mechanism. In seminal paper, Rubin (176)) introduced a formal probabilistic frame- work to assess missing mechanism assumptions and their consequences. The most commonly used assumption, either implicitly or explicitly, is that a part of the data is missing at random (MAR). Essentially, the MAR assumption means that the missing pattern does not depend on the missing values. This makes it possible to ignore the missing data mechanism in likelihood-based inference by marginalizing over the missing data. The often implicit assumption made in non- probabilistic models and ad-hoc methods is that the data are missing completely at random (MCAR). MCAR is a stronger assumption than MAR, and informally it means that both observed and missing data do not depend on the missing pattern. More details on these assumptions can be found in the monograph of Little & Rubin (2002); of particular interest are also the recent revisits of [Seaman et al. (2013) and Doretti et al. (2018). In this paper, our goal is to posit statistical models that lever- age deep learning in order to break away from these assumptions. Specifically, we propose general
not-MIWAE PPCA
(a)
(b)
Figure 1: (a) Graphical model of the not-MIWAE. (b) Gaussian data with MNAR values. Dots are fully observed, partially observed data are dis- played as black crosses. A contour of the true dis- tribution shown together with directions found by PPCA and not-MIWAE with PPCA decoder.
DDepartment of Applied Mathematics and Computer Science, Technical University of Denmark. Denmark I Université Côte d'Azur, Inria (Maasai team), Laboratoire J.A. Dieudonné, UMR CNRS 7351, France I Equal contribution

---

Published as a conference paper at ICLR 2021
recipe for dealing with cases where there is prior information about the distribution of the missing pattern given the data (e.g. self-censoring).
The MAR and MCAR assumptions violated when the missing data mechanism is dependent on the missing data themselves. This setting is called missing not at random (MNAR). Here the missing mechanism cannot be ignored, doing will lead to biased parameter estimates. estimatte. This setting generally requires a joint model for data and missing mechanism.
Deep latent variable models (DLVMs., Kingma& & Welling 2013; Rezende et al. (014) have recently been used for inference and imputation in missing data problems (Nazabal et al. 2020 Ma et al., 2018 2019; Ivanov et al. 2019 Mattei & Frellsen] 2019). This led to impressive empinical results in the MAR and MCAR case, in particular for high-dimensional data.
1.1 CONTRIBUTIONS
We introduce the not-missing-ate att-riidoo importance-weighttr autoencoder (not-M/WAE) which allows for the application of DLVMs to missing data problems where the missing mechanism is MNAR. This is inspired by the missing data importance-weightn autoencoder (MIWAE, Mattei & Frellsen 2019), a framework to train DLVMs in MAR scenarios, based itself on the importance- weighted autoencoder (IWAE) of Burda et al. (016). The general graphical model for the not- MIWAE is shown in figure Ta The first part the model is simply latent variable model: there is stochastic mapping parameterized by from latent variable and the data may be partially observed. The second part of the model, which we call the missing model, is a stochastic mapping from the data the missing mask & ~ (s]x)) Explicit specifica- tion of the missing model Po (s(() makes it possible to address MNAR issues.
p(z) the data po (arl z)),
The model can be trained efficiently by maximising a lower bound of the joint likelihood (of the ob- served features and missing pattern) obtained via importance weighted variational inference (Burda al. 2016). A key difference with the MIWAE is that we use the reparameterization trick in the data space, as well as in the code space, in order to get stochastic gradients of the lower bound.
Missing processes affect data analysis in a wide range of domains and often the MAR assumption does not hold. We apply our method to censoring in datasets from the UCI database, clipping in images and the issue of selection bias in recommender systems.
2 BACKGROUND
Assume that the complete data are stored within a data matrix X = (xI,.... TT.) E 2 that contain n i.i.d. copies the random variable æ E X, where = X1 x ... X, is p-dimensional feature space. For simplicity, Zij refers to the j'th feature of x,, and æ; refers to the i'th sample in the data matrix. Throughout the text, we will make statements about the random variable æ. and only consider samples æ; when necessary. In a missing data context, each sample can be split into an observed part and a missing part, x; (xx x.)). The pattern of missingness is individual to each copy of x and described by corresponding mask random variable
{0, 1 1)." This leads to a mask matrix S = (81,..., TT. T E {0, 1 verifying 8ij 1 if zij is observed and sij 0 if Taj is missing.
We wish to construct a parametric model po, 8) for the joint distribution of a single data point and its mask which can be factored as
po xz, 8) po (s)
Here (s(x)) = (s|== x>") is the conditional distribution of the mask, which may depend on both the observed and missing data, through its own parameters o. The three assumptions from the framework of Little& & Rubin (2002) (see also Ghahramani & Jordan 1995) pertain to the specific form of this conditional distribution:
MCAR: (s(ar) = (s).
MAR: po (s(x) (s(x")
MNAR: (s(2)) may depend on both x and m
2

---

Published as a conference paper at ICLR 2021
To maximize the likelihood of the parameters (0, Φ), based only on observed quantities, the missing
data is integrated out from the joint distribution
(a.o(a". s) = po(a", x= "p"))' (s)x", ,x") dx"
(2)
In both the MCAR and MAR cases, inference for using the full likelihood becomes proportional to po. (x= s) x po and the missing mechanism can be ignored while focusing only on po (a²). In the MNAR case, the missing mechanism can depend on both observed and missing data, offering no factorization of the likelihood in equation 2 The parameters of the data generating process and the parameters of the missing data mechanism are tied together by the missing data.
2.1 PPCA EXAMPLE
A linear DLVM with isotropic noise variance be used recover model similar probabilis- tic principal component analysis (PPCA, [Roweis| 1998] Tipping & Bishop, 1999). In figure [b] dataset affected by an MNAR missing process is shown together with two fitted PPCA models, regular PPCA and the not-MIWAE formulated as a PPCA-like model. Data is generated from a multivariate normal distribution and an MNAR missing process is imposed by setting the horizontal coordinate to missing when it is larger than its mean, i.c. it becomes missing because of the value it would have had, had it been observed. Regular PPCA for missing data assumes that the miss- ing mechanism is MAR so that the missing process is ignorable. This introduces a bias, both in the estimated mean and in the estimated principal signal direction of the data. The not-MIWAE PPCA assumes the missing mechanism is MNAR so the data generating process and missing data mechanism are modelled jointly as described equation
2.2 PREVIOUS WORK
In (Rubin 1976 the appropriateness of ignoring the missing process when doing likelihood based or Bayesian inference was introduced and formalized. The introduction of the EM algorithm (Dempster
et al. 1977) made it feasible to obtain maximum likelihood estimates in many missing data settings, see e.g. Ghahramani & Jordan (1994] 1995): Little & Rubin (2002). Sampling methods such as Markov chain Monte Carlo have made possible to sample a target posterior in Bayesian models, including the missing data, so that parameter marginal distributions and missing data marginal dis- tributions are available directly (Gelman et al. 2013). This is also the starting point of the multiple imputations framework of Rubin (1977 1996.. Here the samples of the missing data are used to provide several realisations of complete datasets where complete-data methods can be applied get combined mean and variability estimates.
The framework of Little& & Rubin (2002) is instructive in how to handle MNAR problems and a recent review of MNAR methods can be found in (Tang & Ju 2018). Low rank models were used for estimation and imputation in MNAR settings by SSportisse et al. (2020a). Two approaches were taken fitting models, 1) maximising the joint distribution of data and missing mask using an EM algorithm, and 2) implicitly modelling the joint distribution by concatenating the data matrix and the missing mask and working with this new matrix. This implies latent representation both giving rise to the data and the mask. An overview of estimation methods for PCA and PPCA with missing data was given by Mii & Raiko (2010), while PPCA in the presence of an MNAR missing mechanism has been addressed by (Sportisse et al. (2020b). There has been some focus on MNAR issues in the form of selection bias within the recommender system community (Marlin al.) 2007 Marlin & Zemel 2009 Steck 2013: Hernández- Lobatto et al. 2014; Schnabel et al. 2016] Wang et al.) 2019) where methods applied range from joint modelling of data and missing model using multinomial mixtures and matrix factorization to debiasing existing methods using propensity based techniques from causality.
Deep latent variable models are intuitively appealing in missing context: the generative part of the model can be used to sample the missing part of an observation. This was already utilized by Rezende et al.] (2014) to do imputation and denoising by sampling from a Markov chain whose stationary distribution is approximately the conditional distribution of the missing data given the observed. This procedure has been enhanced by Mattei & Frellsen (2018a) using Metropolis- within- -iithi- Gibbs. In both cases the experiments were assuming MAR and a fitted model, based on complete data, was already available.
3

---

Published as a conference paper at ICLR 2021
Approaches to fitting DLVMs in the presence of missing have recently been suggested, such the HI- by Nazabal et al. (2020) using an extension of the variational autoencoder (VAE) lower bound, the p- VAE by Ma et al. (2018: 2019 using the VAE lower bound and permutation invariant encoder, the MIWAE by Mattei & Frellsen (2019), extending the IWAE lower bound (Burda et al.) 2016), and GAIN YYon et al. 2018) using GANs for missing data imputation. All approaches assuming that the missing process is MAR or MCAR. In (Gong et al. 2020), the data and missing mask are modelled together, both being generated by mapping from the same latent space, thereby tying the data model and missing process together. This gives more flexibility in terms of missing process assumptions, akin to the matrix factorization approach by |Sportisse et al. (2020a).
In concurrent work. Collier et al. (2020) have developed a deep generative model of the observed data conditioned on the mask random variable, and Lim et al.) (202T) apply model similar to the not-MIWAR to electronic health records data. In forthcoming work, (ihalebikesahi et al. (2021) propose deep generative model for non-ignorable missingness building on ideas from VAES and pattern-set mixture models.
3 INFERENCE IN DLVMS AFFECTED BY MNAR
In an MNAR setting, the parameters for the data generating process and the missing data mechanism need to be optimized jointly using all observed quantities. The relevant quantity to maximize is therefore the log-(joint) likelihood
{(0, ④)
Σ log pe. (x), s,),
(3)
where we can rewrite the general contribution of data points log 8) as
log / (s(e" xx" (xpo(x"l))pple (="">=>((()))))))))1()1)))())))1 dz dæ."
(4)
using the assumption that the observation model is fully factorized po z) = II (o(zz12), which The integrals over missing and latent variables make direct maximum likelihood intractable. However, the approach of Burda et al. (2016), using an inference network and importance sampling to derive a more tractable lower bound of E(O, Φ). can be used here well. The key idea is to posit conditional distribution 9y (z|=²) called the variational distribution that will play the role of a learnable proposal in an importance sampling scheme.
implies pa(ar) 12) = (=))p(=;"))))) >>)====)pp(""""
As in VAEs (Kingma & Welling 2013 Rezende et al. 2014) and IWAEs (Burda et al., 2016). the distribution (2|xx)) ()) comes from a simple family (e.g. the Gaussian or Student's t family) and its
parameters given by the output of neural network (called inference network or encoder) that takes x as input. The issue is that neural net cannot readily deal with variable length inputs (which the case of x²) This was tackled by several works: Nazabal et al. (2020) and Mattei & Frellsen (2019) advocated simply zero-imputing x" to get inputs with constant length, Mnd|Ma et al. (2018)
2019) used a ietiuuttttoniivvviiaan network able to deal with inputs with variable length.
Introducing the variational distribution, the contribution of a single observation is equal to
log (æ., 8) log / P (=" x (="")oo"")))))p))))))) a, (2 (alla)")p (x=") [2) da." dz 2 (2(x0)
(5)
Pos 20 x (x ( p))(2)
log E Ez-g, (=|=") (xx"|
(6)
x
The main idea of importance weighed variational inference and of the IWAE to replace the ex- pectation inside the logarithm by a Monte Carlo estimate of it (Burda et al. 2016). This leads to the objective function
K
E
log
Wki
K
where, for all A < K, S
Pa (8; p am (o (xe)l 2200 (((z)))
Wki
(8)
4, Zbi xx))
4

---

Published as a conference paper at ICLR 2021
Ki are K i..dd. samples from (2) over which the expec- tation in equation 7 is taken. The unbiasedness of the Monte Carlo estimates ensures (via Jensen's ss inequality) that the objective is indeed a lower-bound of the likelihood. Actually, under the moment conditions of (Domke & Sheldon 2018 Theorem 3), which we detail in Appendix D, it is possible to show that the sequence (Lx 9, K21 converges monotonically (Burda et al. 2016 Theorem 1) to the likelihood:
and (Z16, 21: ...)........ (2
(0 0, 7) S Lxx(0,)
E(0,0).
K0000
Properties of the not-MIWAE objective The bound LK (0, Φ.7) has essentially the same prop- erties as the (M)IWAE bounds, see Mattei & Frellsen 2019 Section 2.4 for more details. The key difference is that we are integrating over both the latent space and part of the data space. This means that, to obtain unbiased estimates of gradients of the bound, we will need to backpropagate through samples from q- (z) (e (x= (z) A simple way do this is use the reparameterization trick both 9- [2 (2f)) and (=<x=1))) This is the approach that we chose in our experiments. The main limitation is that pa(=]z) has to belong to a reparameterizable family, like Gaussians or Student's distributions (see Figurnov et al.) 2018 for list of available distributions). If the distribution is not readily reparametrisable (e.g. the data are discrete), several other options are available, see e.g. the review of Mohamed et al. (2020), and, in the discrete case, the continuous relaxations of Jang et al. (017)) and Maddison et al. (2017).
Imputation When the model has been trained, it can be used to impute missing values. If our performance metric is loss function L(x", )"", optimal imputations st minimise (x:", (x" ; s]. When L is the squared error, the optimal imputation is the conditional mean that can be estimated via self-normalised importance sampling (Mattei & Frellsen 2019), see ap- pendix B for more details.
3.1 USING PRIOR INFORMATION VIA THE MISSING DATA MODEL
The missing data mechanism can both be known/decided upon in advance (so that the full relation- ship (s(s)))) is fixed and no parameters need be learned) the type of missing mechanism can be known (but the parameters need to be learnt) or it can be unknown both in terms of parameters and model. The more we know about the nature of the missing mechanism, the more information we can put into designing the missing model. This in turn helps inform the data model how its param- eters should be modified so as accommodate the missing model. This is in line with the findings of Molenberhhs et al. (2008), who showed that, for MNAR modelling to work, one has to leverage prior knowledge about the missing process. A crucial issue is under what model assumptions the full data distribution can be recovered from incomplete sample. Indeed, some general missing models may lead to inconsistent statistical estimation (see e.g. Mohan Pearl 2021 Nabi et al. 2020).
The missing model is essentially solving classification problem; based on the observed data and the output from the data model filling in the missing data, it needs to improve its "accuracy" in predicting the mask. A Bernoulli distribution is used for the probability of the mask given both observed and missing data
(=|x x=) = (s(() [)) = Bern(s) (er((s)T((z)) = I[ (z)( T (x)(
(10)
Here Ty is the estimated probability of being observed for that particular observation for feature j. The mapping (x) from the data to the probability of being observed for the j'th feature can be as general or specific as needed. A simple example could be that of self-masking or self-censoring, where the probability of the j'th feature being observed is only dependent on the feature value, T-j- Here the mapping can be sigmoid on linear mapping of the feature value, πT. (ar, + b). The missing model can also be based on a group theoretic approach, see appendix C
4 EXPERIMENTS
In this section apply the not-MIWAE to problems with values MNAR: censoring in multivariate datasets, clipping in images and selection bias in recommender systems. Implementation details and link to source code can be found in appendix A
5

---

Published as a conference paper at ICLR 2021
Banknotte
Concrete
Red
White
Yegst
Breast
PPCA
1.39 + 0.00
1.61 + 0.00
1.61 +: 0.00
1.57 + 0.00
1.67 + 0.00
0.90 + 0,00
not-MIWAE- -PCCA
agnostic
1.25 + 0.15
1.47 + 0.01
1.32 + 0.00
1.27 + 0.01
1.20 + 0.05
0.78 + 0.00
self self-nuskig
0.57 +: 0.00
1.31
0.00
1.13 0,00
0.99
0.00
0.78 + 0.00
0.72 + 0,00
self masking known
0.57 1: 0.00
1.31
1: 0.00
1.13 1 0.00
0.99
1 0.00
0.77 1 0.00
0.72 1 0.00
MIWAE
1.19 0.01
1.66 +: 0.01
1.62 + 0.01
1.55 + 0.01
1.72 + 0.01
1,20 + 0.01
not MIWAE
agnostic
0.80 0.08
2.63 + 0.12
1.30 + 0.01
1.37 + 0.00
1.43 + 0.02
1.10 + 0.01
self-masking
1.88 +
1.26 + 0.02
1.08 = 0.02
1.01 0.01
1.48 + 0.03
0.74 + 0.01
melf-masking known
0.74
1.12 + 0.04
1.07 + 0.00
1.04 + 0.00
1.38 + 0.02
0.76 + 0.01
low rank joint model
0.79 1: 0.02
1.57 1 0.01
1.42 1 0.01
1.39 1 0.01
1.19 1 0.00
1.22 1 0.01
Forsst
1.28 + 0.00
1.76 0.01
1.64 0.00
1.63 0.00
1.66 + 0.00
1.57 + 0.00
MICE
1.41 + 0.00
+.000
1.68 = 0.00
1.1
1.72 0.00
1.17 = 0.00
mean
1.73 0.00
1.85 0.00
1.83 0.00
1.71 0.00
1.69 0.00
1.82 0.00
Table I: Imputation RMSE on UCI datasets affecfed by MNAR.
4.1 EVALUATION METRICS
Model performance can be assessed using different metrics. A first metric would be to look at how well the marginal distribution of the data has been inferred. This can be assessed, if we happen to have fully observed test-set available. Indeed, observed test-set as a measure of how close po (x) and the true distribution of x are. In the case of DLVM, performance can be estimated using importance sampling with the variational distribution as proposal (Rezende et al. 2014). Since the encoder is tuned to observations with missing data, it should be retrained (while keeping the decoder fixed) as suggested by Mattei & Frellsen (2018b).
look the test log-liieldhood of this fully
Another metric of interest is the imputation error. In experimental settings where the missing mecha- nism under our control, we have access the actual values of the missing data and the imputation eлror can be found directly as an error measure between these and the reconstructions from the model. In real-worke datasets affected by MNAR processes, we cannot use the usual approach of doing train-test split of the observed data. As the test-set is biased by the same missing mechanism as the training-set it is not representative of the full population. Here we need a MAR data sample to evaluate model performance (Marlin et al. 2007).
4.2 SINGLE IMPUTATION IN UCI DATA SETS AFFECTED BY MNAR
We compare different imputation techniques on datasets from the UCI database (Dua & Graff, (211)) where in an MCAR setting the MIWAE has shown state of the art performance (Mattei & Frellsen 2019). An MNAR missing process is introduced by self-masking in half of the features: when the feature value higher than the feature mean it is set to missing. The MIWAE and MOW-M, as well as their linear PPCA-like versions, are fitted to the data with missing values. For the not- MIWAE three different approaches to the missing model are used: 1) agnostic where the data model output is mapped logits for the missing process via a single dense linear layer, 2) self-masking where logistic regression is used for each feature and 3) self-masking known where the sign of the weights the logistic regression known.
We compare to the low-rank approximation of the concatenation of data and mask by Sportisse al. (2020a) that is implicitly modelling the data and mask jointly. Furthermore we compare to mean imputation, missForest (Stekhoven & Bühlmann 2012) and MICE (Buuren & Groothuis- Oudshoorn 2010) using Bayesian Ridge regression. Similar settings are used for the MIWAE and Mot-MIWAF, see appendix A Results over 5 runs are seen in table Results for varying missing rates are in appendix
The low-rank joint model is almost always better than PPCA, missForest, MICE and mean, i.e. all M(C)AR approaches, which can be attributed to the implicit modelling of data and mask together. At the same time the not-MIWAE PPCA is always better than the corresponding low-rank joint model, except for the agnostic missing model on the Yeast dataset. Supplying the missing model with more knowledge of the missing process (that it self-masking and the direction of the missing mechanism) improves performance. The not-MIWAE performance is also improved with more knowledge in the missing model. The agnostic missing process can give good performance, but is

---

Published as a conference paper at ICLR 2021
60003
33000
35000
22003
23000
15000
12003
33000
10030
3000
3000
244 Ieprrssionn val
1.
02
04
0.
22
0.6
0.
pilr 90.
(a) MIWAE
(b) not- Mot-MIWAA
(c) missing data
Figure 2: SVHN: Histograms over imputed values for (a) the MIWAE and (b) the not-MIWAE, and (c) the pixel values of the missing data.
100 256
5
3
8
1
6
2
Model
RMSE
Clest
T 13
100
256
5
3
8
Σ
10000
MIWAE
0.17298
1867.66
256
3 R
&
I
6
2
not-MIWAE
0.07294
1894.36
256
7
MIWAE missing
1908.11
Figure 3: Rows from top: original im- ages, images with missing, not-MIWAE imputations, MIWAE imputations
Table 2: SVHN:
Imputation RMSE and test- set log-likelihood estimate. Constant imputation with I's has RMSE of 0.1757.
often led astray by an incorrectly learned missing model. This speaks to the trade-off between data model flexibility and missing model flexibility. The not-MIWAE PPCA has huge inductive bias in the data model and so we can employ a more flexible missing model and still get good results. For the Mot-MIWAE having both a flexible data model and flexible missing model can be detrimental to performance. One way to asses the learnt missing processes is the mask classification accuracy on fully observed data. These are reported in table A] and show that the accuracy increases as more information is put into the missing model.
4.3 CLIPPING IN SVHN IMAGES
We emulate the clipping phenomenon in images on the street view house numbers dataset (SVHN, Netzer et al., 2011). Here we introduce self-masking missing mechanism that is identical for all pixels. The missing data is Bernoulli sampled with probability
Pr(si: 1| Ite logiss
logits W b).
(11)
where W -50 and 0.75. This mimmicks a clipping process where 0.75 is the clipping point (the data is converted to gray scale in the J0, 1] range). For this experiment use the true missing process as the missing model in the not-MIWAE.
Table 2 shows model performance in terms of imputation RMSE and test-set log likelihood timated with 10k importance samples. The not-MIWAE outperforms the MIWAE both terms of test-set log likelihood and imputation RMSE. This is further illustrated in the imputations shown in figure Since the MIWAE is only fitting the observed data, the range pixel values in the imputations limited compared the true range. The not-MIWAE is forced push some of the data-distribution towards higher pixel values, in order to get a higher likelihood in the logistic re- gression in the missing model. In figures 2a 2c] histograms over the imputation values are shown together with the true pixel values of the missing data. Here we see that the not-MIWAE puts considerable amount probability mass above the clipping value.
4.4 SELECTION BIAS IN THE YAHOO! R3 DATASET
The Yahoo! R3 dataset twebscope.sandbos. www.ssaandooo.o...oooomm comm) contains ratings scale from 1-5 of songs in the database of the Yahoo! LaunchCast internet radio service and was first presented in (Marlin et al. 2007). It consists of two datasets with the same 1,000 songs selected randomly from
7

---

Published as a conference paper at ICLR 2021
MNAR train samples (b) MCAR test samples
(c) MIWAE impute
(d) not-MIWAE impute
Figure 4: Histograms over rating values for the Yahoo! R3 dataset from (a) the MNAR training set and (b) the MCAR test set. (c) and (d) show histograms over imputations of missing values in the test set, when encoding the corresponding training set. The not-MIWAE imputations (d) are much more faithful to the shape of the test set (b) than the MIWAE imputations (c).
the LaunchCast database. The first dataset is considered an MNAR training set and contains self- selected ratings from 15,400 users. In the second dataset, considered an MCAR test-set, 5,400 of these users were asked to rate exactly 10 randomly selected songs. This gives a unique opportunity to train a model on real-world -affeeffected dataset while being able to get an unbiased estimate of the imputation error, due to the availability of MCAR ratings. The plausibility that the set of self- selected ratings was subject to an MNAR missing process was explored and substantiated by Marlin et al. (2007). The marginal distributions of samples from the self-selected dataset and the randomly selected dataset can be seen in figures 4a and 4b
We train the MIWAE and the not-MIWAE on the MNAR ratings and evaluate the imputation error on the MCAR ratings. Both a gaussian and a categorical observation model is explored. In order to get reparameterized samples in the data space for the categorical observation model, we use the Gumbel-Softmax trick (Jang et al.] 2017) with temperature of 0.5. The missing model is logistic regression for each item/feature, with shared weight across features and individual biases. A description of competitors can be found in appendix [A.3] and follows the setup in (Wang et al.) 2019). The results are grouped in table from top bottom, according models not including the missing process (MAR approaches), models using propensity scoring techniques to debias training losses, and finally models learning a data model and a missing model jointly, without the use of propensity estimates.
The not-MIWAE shows state of the art performance, also compared to models based on propensity scores. The propen- sity based techniques need access to a small sample of MCAR data, i.c. a part of the test-set, to estimate the propensities ing Naive Bayes, though they can estimated using logistic regression if covariates using a nuclear-non muclear-norm-ooo norm-consttrined matrix factorization of the missing mask itself (Ma & Chen 2019). We stress that the not-M/WAE does not need access to similar unbiased data in order learn the missing model. However, the missing model in the not-MIWAE can take available information into account, e.g. we could fit continuous mapping to the propen- sities and use this as the missing model, if propensities were available. Histograms over imputations for the missing data in the MCAR test-set can be seen for the MIWAE and not- MIWAE in (cland/// The marginal distribution MOt-MIWAE imputations test-set better than the marginal distribution of the MIWAE imputations.
Model
MSE
MF
1.891
PMF
1.709
AutoRec
1.438
Gaussian- VAE
1.38
available (Schnabel et al. (2166) or
MIWAE categorical
2.067 0.004
MIWAF Gaussian
2.055
0.001
CPT
1.115
MEJIPS
0.989
ME-DR-JL
0.966
NFM-DR-JL
0.957
MF-MNAR
2.199
Logit-vd
1.301
not MIWAE calegorical
1.293 0.006
not-MIWAE ganssian
0.939 - 0.007
Table 3: Imputation MSEs for the
Yahoo! MCAR test-set. Models are trained on the MNAR training set.
match that of the MCAR
5 CONCLUSION
The proposed not- not-MIWAEE is versatile both in terms of defining missing mechanisms and in terms of application area. There is a trade-off between data model complexity and missing In a parsimonious data model very general missing process be used while in flexible data
model complexity.
8

---

Published as a conference paper at ICLR 2021
model the missing model needs to be more informative. Specifically, any knowledge about the missing process should be incorporated in the missing model to improve model performance. Doing so using recent advances in equivariant/invarian neural networks is interesting avenue for future research (see appendix C). Recent developments on the subject of secoverabiityyiddefffabbii of
MNAR models (Sadinle & Reiter 2018 Mohan & Pearl [02T Nabi et al., 2020 [Sportisse al.
2020b) could also be leveraged to design provably idenfiable not- models.
Several extensions of the graphical models used here could be explored. For example, one could break off the conditional independence assumptions, in particular the one of the mask given the data. This could, for example, be done by using an additional latent variable pointing directly to the mask. Combined with a discriminative classifier, the not-MIWAE model could also be used in supervised learning with input values missing not at random following the techniques by [psen et al. 2020
ACKNOWLEDGMENTS
The Danish Innovation Foundation supported this work through Danish Center for Big Data Analytics driven Innovation (DABAI). JF acknowledge funding from the Independent Research Fund Denmark (grant number 913 9131-00082B) and the Novo Nordisk Foundation (grant numbers NNF200C0062606 and NNF200C0065611).
REFERENCES
Alberto Bietti and Julien Mairal. Invariance and stability of deep convolutional representations. In Advances in Neural Information Processing Systems, pp. 6210-6220, 2017.
Benjamin Bloem-Reddy and Yee Whye Teh. Probabilistic symmetries and invariant neural networks.
Journal of Machine Learning Research, 21(90);1-61, 2020.
Yuri Burda, Roger Grosse, and Ruslan Salakhutdinov. Importance weighted autoencoders. In Inter- national Conference Learning Representations, 2016.
Stef van Buuren and Karin Groothhis-Ouddshoore mice: Multivariate imputation by chained equa- tions in R. Journal of Statistical Software, pp. 1-68, 2010.
Taco S. Cohen, Mario Geiger, and Maurice Weiler. A general theory of equivariant CNNs on homo- geneous spaces. In Advances in Neural Information Processing Systems, volume 32, 2019.
Mark Collier, Alfredo Nazabal, and Chris Williams. VAEs in the presence of missing data. In the
First ICML Workshop on The Art of Learning with Missing Values Artemiss (ARTEMISS), 2020.
Arthur P. Dempster, Nan M. Laird, and Donald B. Rubin. Maximum likelihood from incomplete data via the EM algorithm. Journal of the Royal Statistical Society: Series B (Methodological), 39(1):1-22, 1977.
Joshua V Dillon, Ian Langmore, Dustin Tran, Eugene Brevdo, Srinivas Vasudevan, Dave Moore, Brian Patton, Alex Alemi, Matt Hoffman, and Rif A Saurous. Tensorflow distributions. arXiv preprint 1711 2017.
Justin Domke and Daniel Sheldon. Importance weighting and varational inference. In Advances in
Neural Information Processing Signals, volume 31, 2018.
Marco Doretti, Sara Geneletti, and Elena Stanghellini. Missing data: unified taxonomy guided by conditional independence. International Statistical Review, 86(2):189-25) 2299204, 2018.
Dheeru Dua and Casey Graff. UCI machine learning repository, 2017. UtLpptt:://accii ics uci edu/ml
Michael Figurnov, Shakir Mohamed, and Andriy Mnih. Implicit reparameterization gradients. Ad- vances Neural Information Processing Signals, pp. 439-450, 2018
Andrew Gelman, John B. Carlin, Hal S. Stern. David B. Dunson, Aki Vehtari, and Donald B. Rubin. Bayesian data analysis. Chapman and Hall/CRC, 2013.

---

Published as a conference paper at ICLR 2021
Zoubin Ghahramani and Michael Jordan. Supervised learning from incomplete data via an EM approach. In Advances in Neural Information Processing Systems, pp. 120-127, 1994.
Zoubin Ghahramani and Michael I. Jordan. Learning from incomplete data. Technical Report AIM- 1509CBCL-108, Massachusetts Institute of Technology, 1995.
Sahra Ghalebikesabi, Rob Corishh, Luke J. Kelly, and Chris Holmes. Deep generative pattern-set mixture models for nonignorable missingness. arXiv preprint arXiv:2103.03532, 2021.
Peter W. Glynn. Importance sampling for Monte Carlo estimation of quantiles. In Mathematical Methods in Stochastic Simulation and Experimental Design: Proceedings of the 2nd St. Peters- burg Workshop Simulation, pp. 180-185. Publishing House of St. Petersburg University, 1996.
Yu Gong, Hossein Hajimirsadeghi, Jiawei He, Megha Nawhal, Thibaut Durand, and Greg Mori. Variational selective autoencoder. In Proceedings of The 2nd Symposium on Advances in Approx- imate Bayesian Inference, volume 118 of Proceedings of Machine Learning Research, pp. 1-17. PMLR, 2020.
Xiangnan He and Tat-Seng Chua. Neural factorization machines for sparse predictive analytics. In Proceedings of the 40th International ACM SIGIR conference on Research and Development in Information Retrieval, pp. 355-364, 2017.
José Miguel Hernández-LLoato, Neil Houlsby, and Zoubin Ghahramani. Probabilistic matrix fac- torization with non-random missing data. In International Conference Machine Learning, pp. 1512-1520, 2014.
Alexander Ilin and Tapani Raiko. Practical approaches to principal component analysis in the pres- ence of missing values. Journal of Machine Learning Research, 1((uu):1957-200 2010.
Niels Bruun Ipsen, Plerre-Alexanrre Mattei, and Jes Frellsen. How to deal with missing data in supervised deep learning? In the First ICML Workshop on The Art of Learning with Missing Values Artemiss (ARTEMISS), 2020.
Oleg Ivanov, Michael Figurnov, and Dmitry Vetrov. Variational autoencoder with arbitrary condi- tioning. In International Conference on Learning Representations, 2019.
Eric Jang, Shixiang Gu, and Ben Poole. Categorical reparameterization with Gumbel-softmax. In International Conference Learning Representations, 2017.
Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In International Conference on Learning Representations, 2014.
Diederik P. Kingma and Max Welling, Auto-encoding variational Bayes. In International Confer- ence on Learning Representations, 2013.
Yehuda Koren, Robert Bell, and Chris Volinsky. Matrix factorization techniques for recommender systems. Computer, 42(8):30-37, 2009.
Dawen Liang, Rahul G. Krishnan, Matthew D. Hoffman, and Tony Jebara. Variational autoencoders for collaborative filtering. In Proceedings of the 2018 World Wide Web Conference, pp. 689-698. International World Wide Web Conferences Steering Committee, 2018.
David K. Lim, Naim U. Rashid, Junier B. Oliva, and Joseph G. Ibrahim. Handling non-ignorably missing features in electronic health records data using importance-weighted autoencoders. arXiv preprint arXiv:2101.073577 2021.
Roderick J. A. Little and Donald B. Rubin. Statistical analysis with missing data. John Wiley & Sons, 2002.
Chao Ma, Wenbo Gong, José Miguel Hernández-Lbbtto, Noam Koenigstein, Sebastian Nowozin, and Cheng Zhang. Partial VAE for hybrid recommender system. In NIPS Workshop on Bayesian Deep Learning, 2018.
10

---

Published as a conference paper at ICLR 2021
Chao Ma, Sebastian Tschiatschek. Konstantina Palla, Jose Miguel Hernandez-Lobato. Sebastian Nowozin, and Cheng Zhang. EDDI: Efficient dynamic discovery of high-value information with partial VAE. In International Conference Machine Learning. pp. 4234 4243, 2019.
Wei Ma and George Chen. Missing not at random matrix completion: The effectiveness of estimating missingness probabilities under low nuclear norm assumption. In Advances in Neural Information Processing Systems, pp. 14871-14808 2019.
Chris J. Maddison, Andriy Mnih, and Yee Whye Teh. The concrete distribution: A continuous re- laxation of discrete random variables. In International Conference on Learning Representations, 2017.
Benjamin M. Marlin and Richard S. Zemel. Collaborative prediction and ranking with non-random missing data. In Proceedings of the third ACM conference on Recommender systems, pp. 5-12. ACM, 2009.
Benjamin Marlin, Richard Zemel, Sam Roweis, and Malcolm Slaney. Collaborative filter- ing and the missing at random assumption. In Proceedings of the Twenty-Third Conference Uncertainty in Artificial Intelligence, pp. 267-275. AUAI Press, 2007.
Alexad-Ae Mattei and Jes Frellsen. Leveraging the exact likelihood of deep latent variable models. Advances in Neural Information Processing Systems, volume 31, pp. 3855-3866, 201 2a.
Pierre-Alexanrre Mattei and Jes Frellsen. Refit your encoder when new data comes by. 3rd NeurlPS workshop on Bayesian Deep Learning, 2018b.
Deep generative modelling and imputation of incomplete data sets. In International Conference on Machine Learning, pp. 4413-4423, 2019.
Alexandee Mattei and Jes Frellsen. MIWAE:
Andriy Mnih and Russ R. Salakhutdinov. Probabilistic matrix factorization. In Advances in Neural Information Processing Systems, volume 20, pp. 1257-1264, 2008.
Shakir Mohamed, Mihaela Rosca, Michael Figurnov, and Andriy Mnih. Monte carlo gradient esti- mation machine learning. Journal of Machine Learning Research, 21(132):1 2020.
Karthika Mohan and Judea Pearl. Graphical models for processing missing data. Journal of Ameri- can Statistical Association (in press), 2021.
Geert Molenberghs, Caroline Beunckens, Cristina Sotto, and Michael Kenward. Every missing- ness not at random model has missingness at random counterpart with equal fit. Journal of the Royal Statistical Society: Series (Statistical Methodology), 70(2):371-388, 2008
Razich Nabi, Rohit Bhattacharya, and Ilya Shpitser. Full law identification in graphical models of missing data: Completeness results. In International Conference on Machine Learning, pp- 7153-7163, 2020.
Alfredo Nazabal, Pablo M. Olmos, Zoubin Ghahramani, and Isabel Valera. Handling incomplete heterogeneous data using VAEs. Pattern Recognition, 107:107501, 2020.
Yuval Netzer, Tao Wang, Adam Coates, Alessandro Bissacco, Bo Wu, and Andrew Ng. Reading digits in natural images with unsupervised feature learning. NIPS 2011 Workshop on Deep Learming and Unsupervised Feature Learning, 2011.
Danilo Jimenez Rezende, Shakir Mohamed, and Daan Wierstra. Stochastic backpropagation and ap- proximate inference in deep generative models. In International Conference on Machine Learn- ing, pp. 1278-1286, 2014.
foundrtttttiicfotttioo Christian Robert. The Bayesian choice: from decision-theoretic foundations to computational im- plementation. Springer Science & Business Media, 2007.
Sam T. Roweis. EM algorithms for PCA and SPCA. In Advances in neural information processing systems, pp. 226-632, 1998.
11

---

Published as a conference paper at ICLR 2021
Donald B. Rubin. Inference and missing data. Biometrika, 03(3):581-592 1976.
Donald B. Rubin. Formalizing subjective notions about the effect of nonrespondents in sample surveys. Journal of the American Statistical Association, 72(359):538-433, 1977.
Donald B. Rubin. Multiple imputation after 18+ years. Journal of the American statistical Associa- tion, 91(434):473-48) 1996.
Mauricio Sadinle and Jerome Reiter. Sequential identification of nonignorable missing data mech- anisms. Statistica Sinica, 28(4):1741-1759 2018.
Tobias Schnabel, Adith Swaminathan, Ashudeep Singh, Navin Chandak, and Thorsten Joachims. Recommendations as treatments: Debiasing learning and evaluation. In International conference on machine learning, pp. 1670-1679, 2016.
Shaun Seaman, John Galati, Dan Jackson, and John Carlin, What is meant by "missing at random"? Statistical Science, 28(2):257-22) 2013.
Suvash Sedhain, Aditya Krishna Menon, Scott Sanner, and Lexing Xie. AutoRec: Autoencoders meet collaborative filtering. In Proceedings of the 24th international conference on World Wide Web, pp. 111-112, 2015.
Aude Sportisse, Claire Boyer, and Julie Josse. Imputation and low-rank estimation with missing at random data. Statistics and Computing, 3006):6229-1643 2020a.
Aude Sportisse, Claire Boyer, and Julie Josse. Estimation and imputation in probabilistic princi- pal component analysis with missing not at random data. In Advances Neural Information Processing Systems, volume 33, pp. 7067-7077, 2020b.
Harald Steck. Evaluation of recommendations: rating- prediction and ranking. In Proceedings of the 7th ACM conference on Recommender systems. pp. 213-220. ACM, 2013.
Daniel J. Stekhoven and Peter Bühlmann. tasaFForst-non-paaammett missing value imputation for mixed-type data. Bioinformatics. 28(1):112-11) 2012.
Niansheng Tang and Yuanyuan Ju. Statistical inference for nonignorable missing data problems: selective review. Statistical Theory and Related Fields, 202):105-133, 2018.
Michael E. Tipping and Christopher M. Bishop. Probabilistic principal component analysis. Journal of the Royal Statistical Society: Series B (Statistical Methodology). 61(3):611-62 1999.
Xiaojie Wang, Rui Zhang, Yu Sun, and Jianzhong Qi. Doubly robust joint learning for recommen- dation data missing not at random. In International Conference Machine Learning, pp- 6638-6647, 2019.
Samuel Wigvist, Pierre-Alexandre Mattei, Umberto Picchini. and Jes Frellsen. Partially exchange- able networks and architectures for learning summary statisties in approximate Bayesian compu- tation. In International Conference on Machine Learning, pp. 6798-6807, 2019.
Jinsung Yoon, James Jordon, and Mihaela Van Der Schaar. GAIN: Missing data imputation using generative adversarial nets. Proceedings of the 25th international conference on Machine learning, pp. 5689-5698, 2018.
Manzil Zaheer, Satwik Kottur, Siamak Ravanbakhsh, Barnabas Poczos, Ruslan Salakhutdinov, and Alexander J Smola. Deep sets. In Advances in Neural Information Processing Systems, vol- ume 30, pp. 3391-3401, 2017.
12

---

Published as a conference paper at ICLR 2021
Banknote
Concrete
Red
White
Yeas!
Breaz!
BOE-IIWAE- PPCA
0.80 + 0.03
0.75 + 0.0G
0.88 0.01
0.83 0.00
0.78 0.02
0.00
self-masking
0.92 + 0.05
0.95 + 0.00
0.96 0.00
0.97 = 0.00
0.99 = 0.00
0.98 = 0.00
sell mesking known
0.98 0.00
0.00
0.00
not-MIWAE
agnostic
0.92 +: 0.01
0.54 + 0.04
0.91 0.00
0.88 + 0.00
0.80
0.00
0.93 + 0.00
self masking
0.99 1: 0.00
0.93 1: 0.02
0.95 0.01
0.90 1: 0.02
0.71
0.02
0.98 1: 0.00
self-maskkin known
0.99 + 0.00
0.97 + 0.00
0.97 0.00
0.95 0.00
0.78
0.00
0.98 0.00
Table Al: Mask prediction accuracies on UCI datasets using fully observed data.
A IMPLEMENTATION DETAILS
In all experiments we used TensorFlow probability (Dillon al., 2017) and the Adam optimizer (Kingma & Ba 2014) with learning rate of 0,001. Gaussian distributions were used both as the variational distribution in latent space and the observation model data space. No regularization was used. Similar settings were used for the MIWAE and the not-MIWAF, except for the missing model which is exclusive to the not-MIWAF.
Source code is available at: http::/ggtttuu...ooooo///oo/// com
A.1 UCI
The encoder and decoder consist of two hidden layers with 128 units and tanh activation functions. In the PPCA-like models, the decoder is linear mapping from latent space data space, with learnt variance shared across features. The size of the latent space is set to p 1, K 20 importance samples were used during training and a batch size Data standardized before missing is introduced. The imputation RMSE is estimated using 10k importance samples and the mean and standard errors are found over runs.
was used 100k iterations.
Since the imputation error in a real-world setting cannot be monitored during training, neither a train or validation set, early stopping cannot be done based on this. Both the MIWAE and not- MIWAE are trained for fixed number of iterations. In the low-rank joint model of Sporttsss et al. 2020a), model selection needs to be done for the penalization parameter Xi In order to do this we add 5% missing values (MCAR) to the concatenated matrix of data and mask and use the imputation error on this added missing data to select the optimal lambda. The model is then trained on the original data using the optimal
get the imputation error.
For evaluating the learnt missing model, we report mask classification accuracies when feeding fully observed data as input to the missing model, see table AT As the missing model contains more prior information, the classification accuracy becomes better and better.
A.2 SVHN
For the encoder and decoder a convolutional structure was used (see tables A2 and A3
together with
ReLU activations and a latent space of dimension 20. 5 importance samples were used during training and batch size of 64 was used for IM iterations. The variance in the observation model was lower bounded at 0.02.
A.3 YAHOO!
The MIWAE and the not-MIWAE were trained on the MNAR ratings and the imputation error was evaluated on the MCAR ratings (when encoding the MNAR ratings). We used the permutation invariant encoder by Ma et al. (2018) with an embedding size of 20 and code size of 50, along with a linear mapping to latent space of size 30. In the Gaussian observation model, the decoder is linear mapping and there is a sigmoid activation of the mean in data space, scaled to match the scale
We used original code from the authors found here:
https://gi tttp://ggthuth.ooo wttp:hwwww.coocm.uu.e.....aaa
stat
13

---

Published as a conference paper at ICLR 2021
Table A2: SVHN encoder
Table A3: SVHN decoder
layer(size)
layer(size)
Input æ (32 x 32 X I)
Latent variable x22))
Conv2D(16 16 64)
Dense(4096)
Conv2D(8 X 8 128)
Reshape x 4 256)
Conv2D(4 x 256)
Conv2Dtranspose(() x 256) Conv2Dtransposee(6 16 x 128)
Reshape(4096)
μ.: D2n)e(20)
Jr:
Conv 2transssss) 32 x 64)
log Dense(20)
Crnr2Drrasspoee 32 1) sigmoid
log a: Conv2Drranspose(22 32 64) Conv2Drransoose22 32 x 1)
of the ratings. The categorical observation model also has a linear mapping to its logits. In both latent space and data space, we learn shared variance parameters each dimension. The missing model is logistic regression for each feature, with shared weight across features and individual biases for each feature. We use K = 20 importance samples during training, ReLU activations, a batch size of 100 and train for 10k iterations.
We follow the setup of Wang et al. (2019) and compare to the following approaches:
CPT-v: Marlin et al. (007) show that multinomial mixture model with Conditional Probability Tables missing model give better performance than the multinomial mixture model without missing model. The approach further expanded by Marlin & Zemel (2009), where logistic model, Logit- vd, is also tried as the missing model. The result for the CPT-y model and the Logit-vd model are taken from the supplementary material of ofHHenáándee--Lohatoo
al. (2014).
MF-MNAR: Hernández- et al. (2014) extended probabilistic matrix factorization to include missing data model for data missing from the supplementary material of the paper.
random collaborative filtering setting. Results
MF-IPS: Schnabel et al. (2016) applied propensity- -based methods from causal inference to IPS. The propensities used to debias the matrix factorization are the probabilities of rating being observed for each (user, item) pair. The propensities used for training are found using 5% of the MCAR test-set. Results are from the paper.
trix factorization, specifically tnveerseproppnnsity-esee
Wang al. 2019) combines the coopensity-scorinne approach from Schnabel et al. (2016) with an error- inoorimmptatiion approach by Stteck (2013) to obtain a doubly ro- bust estimator. This used both with matrix factorization and neural factorization muchines (He & Chua 2017). As for [Schnabel et al. (2016), 5% of the MCAR test-set is used to learn the propensities. Results are from the paper.
ME-DR-JL and NFM-DR-JL:
In addition these debiasing approaches, we compare to the following methods, which do not take the missing process into account: MFF (Koren al. 2009). PMF (Mnih Salakhutdinoy 2008). AutoRec (Sedhain et al. (2015) and Gaussian VAE (Liang et al. 2018). The presented results for these methods are from Wang et al. 2019).
B IMPUTATION
Once the model has been trained, it is possible to use it impute the missing values. our performance metric a loss function L(x".y"" optimal imputations z minimise Eg [L"" [,=))="", [x))s). Many loss functions can be minimized using moments of the conditional distribution the missing values, given the observed. Similarly to Mattei & Frellsen (2019) equa- tions 10-11), these moments be estimated via self-normalised importance sampling. For any
14

---

Published as a conference paper at ICLR 2021
function of the missing data h(x=),
E[h(2")) s) h(==")p(²"" dz.
(12)
Using Bayes's theorem, we get
E [l"("")) / h(2r))
p(8//0 ar )
"o"""""""""" x²)
dx
(13)
p(s, x?))
and now we can introduce the latent variable:
E[h(="") 2.,, s| - // h(=*)) p(s]a", 2 2 (p(a))("aaa)) [=)"(=)))>>))))))))))))))>>>)))1 dz dl.
(14)
p(s, x²)
Using self-normalised importance sampling on this last integral with proposal or (z) (p (xx" leads to the estimate
K
E[h(x") > s)
agh(z.), with ag w1 + +... WK
wk
(15)
where the weights w1.... WK are incidentally identical to the ones used for training:
Vk < K, we (s(s) x xx po (x (xx)))))
(16)
47 (2 20))
and (21, ar."; :.... (zK, xxa are K i.d. samples from 4- ppo""=))))>>)))) If the quantity [[h(x")) is easy to compute, then a kto-Blackwelliized version of equation (15 should be pre- ferred
K
m ([(((=")) Rd
ag [x="((("))
(17)
Squared loss When L corresponds to the squared error, the optimal imputation will be the condi- tional mean that can be estimated using the method above (in that case, h is the identity function):
th E[ ox
anE (x", 8], with on W1 + ... + WK
Wk
(18)
A Absolute loss When I is the absolute error loss, the optimal imputation the conditional median, that can be estimated using the same technique and at little additional cost compared to the mean. Indeed, we can estimate the cumulative distribution function each missing feature j E {[......P p}:
Fx,) B[ wx" 8)
ou Fr 2 lan, (2y;),
(19)
where Fzylar the cumulative distribution function of xry 12", s, which will often be available in closed-form (e.g. in the case of a Gaussian, Bernoulli or Student's 's observation model). We can then use this estimate to approximately solve Fz = 0.5. More generally, if L is a multilinear loss, optimal imputations will be quantiles (see e.g. Robert 2007 section 2.5.2) that be estimated
using equation (19. The consistency of similar quantile estimates was studied by Glynn (1996).
Multiple imputation. It is also possible to perform multiple imputation with the same computa- tions, One can obtain approximate samples from p(="|²") using sampling importance resampling with the same set of weights. This allows us do both single and multiple imputation with the same computations.
C MISSING MODEL, GROUP THEORETIC APPROACH
A more complex form of prior information that can be used to choose the form of (z()) is group- theoretic. For example, we may know priori that p.(s| (x)) is invariant certain group action g on the data space:
Vg. (s(m)) po a.).
(20)
15

---

Published as a conference paper at ICLR 2021
This would for example be the case, if the data sets were made of images whose class is invariant to translations (which is the case most image data sets, like MNIST or SVHN), and with a missing model only dependent on the class. Similarly, may know that the missing process is equivariant:
Vgg Pg s|x: = pals .t. x))
(21)
Again, such setting can appear when there is strong geometric structure in the data (e.g. with im- ages or proteins). Invariance or equivariance can be built the architecture of TØ(x) ) by leveraging the quite large body of work on invariant/cquivarian convolutional neural networks, see c.g. [Bietii & Mairal (2017): Cohen et al. (2019): Zaheer et al. (2017): Wigvist al. (2019); Bloem-Reddy & Teh (2020), and references therein.
D THEORETICAL PROPERTIES OF THE NOT-MIWAE BOUND
The properties of the not-MIWAE bound are directly inherited from the ones of the usual IWAE bound. Indeed, as we will see, the not-MIWAE bound a particular instance of IWAE bound with an extended latent space composed of both the code and the missing values. More specifically, recall definition of the not-MIWAE bound
K
Wki , with Wis = po(2 2k )ps (8, Lef, (pf)p((())))
(x(0, 0, =
E
log
(22)
K
47 (2 [=xs|=²])
Each ith term of the sum can be seen as an IWAE bound with extended latent variable (Zki, x.).) whose prior po pa:g [x:)p< (zxs)) The related importance sampling proposal of the ith term pa (a(a) (Z))) g- (Zkaa ( and the observation model is p& p=( æ )ps (2 zxs).
Since all terms of the the monotonicity property:
IWAE bounds, Theorem from Burda al. (2016) directly gives
(0, s... S (0,00)))))
(23)
Regarding convergence of the bound to the true likelihood, we can use Theorem of Domke & Sheldon (2018) for each term the sum to get the following result.
Theorem. Assuming that, for all E mI,
there exists a; > such that IE [w - (0,
< co.
limsupx=aa E [K((wi + ... wk:) < cc.
the Mot-M bound converges to the true likelihood at rate 1/K:
1 Var[wis]
((0,6) (x(0,0,7)
(24)
K=000 K
2pe, (x s,))
E VARYING MISSING RATE (UCI)
The UCI experiments use self-masking missing process half the features: when the feature value is higher than the feature mean it set missing. In order to investigate varying missing rates we change the cutoff point from the mean to the mean plus an offset. The offsets used are (0, 0.25, 0.5, 0.75, 1.0), that the largest cutoff point will be the mean plus one standard deviation. Increasing the cutoff point further results in mainly imputing outliers. Results for PPCA and not- MIWAE PPCA using the agnostic missing model are seen figure 5 and using the self-masking model with known sign of the weights are seen in figure O Figure 7 shows the results for MIWAE and not-MIWAE using self-mast self-masking with known sign of the weights.
16

---

Published as a conference paper at ICLR 2021
PICA
NIMAB PPGA
16-
POCA HIMAE PPCA
14
15:
53055
1.2
2005
1.0
13
2.0
00
0.
1.0
Offic
0thert
(a) Bank
(b) Concrete
(c) Red
20
2.0
1.3
SIT-MININ
1.0
NIWAE PPCA
1.00
NTWAB PPIAA
0.55
1.2
1.4
1.0
135
0. 01bxk
001ed
(d) White
(c) Yeast
Breast
Figure 5: PPCA agnostic: Imputation RMSE varying missing rates on UCI datasets. The vari- ation in missing rate is obtained by changing the cutoff point using an offset, so that offset = corresponds to using the mean as the cutoff point while an offset = 1 corresponds to using the mean plus one standard deviation the cutoff point. Results averages over 2 runs.
IYCA MIKVE PPCA
122
1.4
10501
FPSA
100000
LVCA
08
2.5
2.0
0.0
6.2
3.0
0.6
16
(a) Bank
(b) Concrete
(c) Red
2.0
aut-NIWAR IPCA
NE--EWAEE PPCA
POCA
SHNXIAA IPCA
1.2
0.6
Offict
Offor
(d) White
(e) Yeast
Breast
Figure 6: PPCA self-masking known: Imputation RMSE at varying missing rates on UCI datasets. The variation in missing rate is obtained by changing the cutoff point using offset, so that an offset = corresponds to using the mean as the cutoff point while an offset = corresponds to using the mean plus one standard deviation as the cutoff point. Results are averages over 2 runs.
17

---

Published as a conference paper at ICLR 2021
2.0
NUWAR
1.0
MIWAE SOA
1.5
344
0.
13
0b
000x4
033el
(a) Bank
(b) Concrete
(c) Red
2.0
NITWN
15
NTWAR
1.3
30811
1.3
1.5
0.5
1.2
MIWAE mo-MWAA
Omve
Offict
Otfeet
(d) White
(e) Yeast
Breast
Figure 7: Self-masking known: Imputation RMSE varying missing rates on UCI datasets. The variation in missing rate is obtained by changing the cutoff point using offset, so that an offset = corresponds to using the mean as cutoff point while an offset = corresponds to using the mean plus one standard deviation the cutoff point. Results averages over 2 runs.
18


Reviews:
[{"review_id": "zNe9pk8uoTH", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper introduces a deep latent variable model (DLVM) for missing data problems where the missing mechanism is missing not at random (MNAR) and therefore cannot be ignored. It presents an approach for fitting the model based on importance-weighted variational inference and reparameterization trick, and demonstrates the application of the proposed method in simulated and real data sets.\n\nPros:\n\n-The paper addresses an important problem of dealing with MNAR data, by introducing a DLVM model that allows for incorporating prior information about the type of missingness (for example, self centoring) into the model. This extends the applicability of DLVM models to a wider class of practical problems.\n\n-The paper is clearly written.\n\n-The experimental results demonstrate the trade-off between data model flexibility and missing model flexibility.\n\n\nCons:\n\n-The proposed approach is a relatively straightforward extension of the existing work (MIWAE, Mattei & Frellsen, 2019), using somewhat standard VAE techniques.\n\n-I’m not sure the experimental results really demonstrated the advantages of the proposed methods against existing ones, particularly in the settings of no prior knowledge about the type of missingness mechanism. Though seeing examples of the trade-off between data model flexibility and missing model flexibility is nice.\n\nOverall, I think the results in the paper should be useful in a number of applications and the paper has enough contributions to merit publication.\n\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "xLp5A3Mxfxe", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Overall I very much enjoyed reading this paper! The manuscript is very well written, the related work section is sound, the motivation is clear, the methodology is well formalized and the experimental validation is strong in some aspects. \n\nThe formalisation of the missingness process is very intuitive. Also the literature referenced provides one of the most comprehensive overviews on the topic in the statistics community, on the deep learning side it seems that there is a lot of research covered on the side of deep learning latent variable models based on variational auto encoders (VAEs), but the complementary work on Generative Adversarial Networks (GANs) appears to be less well covered. I’m myself more familiar with the VAE approach, and the authors correctly mention the implicit assumptions of GAN approaches, but as GANs offer an intuitive parametric and flexible way of modelling the missingness mask, I guess it would be helpful to see them in the comparison. \n\nIn the first paragraph on page 5 the authors mention that the approach with the reparametrization only works if the data is continuous. I might be missing something, but rather than using a sampling based approach as in Mohamed et al, referenced by the authors, maybe it would be an option to use something like the Gumbel Softmax? In the experimental section the authors use that approach for the recommender system data, with limited success, it seems, compared to a plain gaussian likelihood. But conceptually it would be good to comment on why that’s not possible? \n\nMy main concerns with this work is the experimental validation. The experimental settings explored are UCI data sets, image data and recommendation systems. It’s great that the authors provide such a comprehensive and heterogenous experimental validation in terms of data sets. What I found a bit limiting is that the not-missing-at-random process was so simple and restricted, and that the missingness ratio was not explored systematically. Many studies on imputation use experimental validation that explores missingness ratios between 0 and 100% of the values and in particular the MAR and MNAR settings are explored by, e.g. sampling a random quantile of a feature to condition the missingness on. This is very simple to implement but would allow for a much more realistic account of missingness structure. Especially for a study like this, which makes a presumably important and strong contribution to the field of missing data imputation I would recommend to demonstrate the effectiveness of the proposed approach by a more realistic experimental setting. \n\nAnother recommendation would be to highlight the advantage of the proposed approach with more synthetic data experiments as in figure 1b, maybe one linear and one non-linear data manifold. That would allow to control for more parameters like the rank of the data and the noise, their covariance structure (strength and independence of features and noise, respectively). But that’s not really necessary I think, the authors did a great job with figure 1b! ", "extra_scores": {}, "extra_text": {}}, {"review_id": "db7Qm2K2Rim", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Review:\n\nThis paper handles the problem of missing not-at-random (MNAR) by extending the MIWAE model to MNAR scenarios. To do this, they use the reparametrization trick in the data space to get the stochastic gradients of the lower bound.\n\nMinor questions/comments:\n\n- At the beginning of section 2, $\\mathbf{x}_i$ is defined as a row the data matrix. However, I have noticed that the $i$ subscript is dropped several times across the text and left simply as $\\mathbf{x}$. Maybe a small comment indicating this at the beginning could help to not get confused while reading section 2.\n\n- In section 4.4. the authors say that both a categorical and a Gaussian observation model are used. I was under the impression that this paper was only evaluated on real attributes, mainly based on the use of MSE and RMSE as evaluation metrics. Are the datasets in Table 1 also a mixture of discrete and continuous variables? In that case, how do the authors compute the MSE of the discrete variables in Table 3 and, if it applies, in Table 1?\n\nSummary:\n\nThe problem of MNAR is very important in practical scenarios, specially when handling tabular data. The paper was well written and explained, and although it can be seen as a simple extension to the MIWAE model, nonetheless I consider it to be a relevant and interesting contribution.", "extra_scores": {}, "extra_text": {}}, {"review_id": "Hrds5skluOz", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes an approach to training deep latent variable models on data that is missing not at random. To learn the parameters of deep latent variable models, the paper adopts importance-weighted variational inference techniques. Experiments on a variety of datasets show that the proposed approach is effective by explicitly modeling missing not at random data.\n\nP1: The related work is well done. This paper reviews the most related studies in several lines of research, including missing data concepts and theories in statistics, missing not at random data in various applications, deep latent variable models for missing data, etc.\n\nP2: The experimental results are quite extensive. This paper conducts experiments on a wide range of datasets from different domains: censoring on multi-variate datasets, clipping on image datasets, and bias on recommendation datasets. The paper also compares the proposed approach against a representative selection of state-of-the-art approaches.\n\nC1: The main concern for this paper is the lack of technical depth. Using variational distribution to derive a tractable lower bound of of a joint likelihood function, using Monte Carlo estimates to unbiasedly approximate the lower bound, and using a reparameterization trick to obtain unbiased estimates of gradients of the lower bound are well-established techniques. It is important for this paper to highlight what is the novelty of the proposed approach in terms of technical innovation.\n\nC2: This paper argues that the proposed approach allows for incorporating prior information about types of missingness. However, it is not clear to me what is the prior information and how does the proposed approach leverage the prior information. It is highly recommended for this paper to provide a formal formulation of the prior information in missing not at random data. It is also recommended for the paper to elaborate how the proposed approach uses the prior information and underlying motivation.\n\nC3: The last sentence in the 4th page of this paper states that it is possible to show that a sequence of objective functions converges to the joint likelihood function. To make the statement more convincing, it would be better if the paper could include a proof that the sequence of objective functions theoretically converges.", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: All the reviewers highlight that the paper addresses the important issue of extending deep latent variable models to handle missing non at random data, which are known to be very difficult. The authors suggest modeling the mechanism of missing values and perform inference  using amortized importance weighted variational inference and demonstrate the capacities of their approach on many experiments. The paper highlight the trade-off between the complexity of the data model and that of the missing data mechanism. The authors appropriately answer reviewers comments, add new experiments varying the percentage of missing values, and give more details on the methodological part. 
I also think that this is a valuable contribution to the community, that the literature is well covered (the historical statistical litterature and the ML one), and that it provides new insights and methods to tackle this difficult problem. 

Decision: Accept (Poster)