Identifiable Generative models for Missing Not at Random Data Imputation
Authors: Chao Ma
Cheng Zhang
Venue: NeurIPS 2021 Poster (2021)
Keywords: missing data
missing not at random
generative models
identifiability
imputation
variational autoencoder

Abstract: Real-world datasets often have missing values associated with complex generative processes, where the cause of the missingness may not be fully observed. This is known as missing not at random (MNAR) data. However, many imputation methods do not take into account the missingness mechanism, resulting in biased imputation values when MNAR data is present. Although there are a few methods that have considered the MNAR scenario, their model's identifiability under MNAR is generally not guaranteed. That is, model parameters can not be uniquely determined even with infinite data samples, hence the imputation results given by such models can still be biased. This issue is especially overlooked by many modern deep generative models. In this work, we fill in this gap by systematically analyzing the identifiability of generative models under MNAR. Furthermore, we propose a practical deep generative model which can provide identifiability guarantees under mild assumptions, for a wide range of MNAR mechanisms. Our method demonstrates a clear advantage for tasks on both synthetic data and multiple real-world scenarios with MNAR data. 

Identifiable Generative Models for Missing Not at Random Data Imputation
Chao Ma Cheng Zhang*
University of Cambridge Microsoff Research Cambridge
cm9050cam ac. uk
cheng. www.:zaannmmicrcocemmm wwt.g/m..coooooo.ccoommo com
Abstract
Real-world datasets often have missing values associated with complex generative processes, where the cause of the missingness may not be fully observed. This is known as missing not at random (MNAR) data. However, many imputation methods do not take into account the missingness mechanism, resulting biased imputation values when MNAR data is present. Although there are a few methods that have considered the MNAR scenario, their model's identifiability under MNAR is generally not guaranteed. That is, model parameters can not be uniquely deter- mined even with infinite data samples, hence the imputation results given by such models can still be biased. This issue is especially overlooked by many modern deep generative models. this work, we fill in this gap by systematically analyzing the identifiability of generative models under MNAR. Furthermore, we propose a practical deep generative model which can provide identifiability guarantees under mild assumptions, for a wide range of MNAR mechanisms. Our method demon- strates a clear advantage for tasks on both synthetic data and multiple real-world scenarios with MNAR data.
Introduction
Missing data is an obstacle in many data analysis problems, which may seriously compromise the performance of machine learning models, well as downstream tasks based on these models. Being able to successfully recover/impute missing data in an unbiased way is the key to understanding the structure of real-world data. This requires us identify the underlying data-gent aett-generating process, as well as the probabilistic mechanism that decides which data is missing.
In general, there are three types of missing mechanisms [44]. The first type is missing completely random (MCAR), where the probability of data entry being missing is independent of both the observed and unobserved data (Figure (a)). In this case, no statistical bias is introduced by MCAR. The second type missing at random (MAR), which assumes that the missing data mechanism is independent of the value of unobserved data (Figure 1 (b)). Under this assumption, maximum likelihood learning methods without explicit modeling of the missingness mechanism can be applied by marginalizing out the missing variables [3, 25, 28). However, both MCAR and MAR do not hold in many real- wolld applications, such as recommender systems [8, 14], healthcare [13], and surveys [51]. For example, in survey, participants with financial difficulties more likely to refuse to complete the survey about financial incomes. This an example of missing not at random (MNAR), where the cause of the missingness (financial income) can be unobserved. In this case, ignoring the missingness mechanism will result in biased imputation, which will jeopardize down-stream tasks.
"Tii work was performed when the authors were (part-time) associated with Microsoft Research, Cambridge
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

Z
Z
X
X3
x2
R
R3
R
R
R
R
R
R2
(a)
(b)
(c)
(d)
(0)
Z
Z.
Z
X
X1
R
R
R
R2
R
R2
(g)
(h)
Figure 1: Exemplar missing data situations. (a): MCAR: (b): MAR; (c)-(i) MNAR
There are few works considering the MNAR setting in scalable missing value imputation. On the one hand, many practical methods for MNAR does not have identifiability guarantees [12, 8, 24]. That is, the parameters can not be uniquely determined, even with access to infinite samples [33, 43]. As result, missing value imputation based on such parameter estimation could be biased. other hand, there theoretical analyses the identifiability in certain scenarios [33, 34, 35, 38, 53, 55, 57], but without associated practical algorithms for flexible and scalable settings (such as deep generative models). Morcover, MNAR data have many possible cases (Figure 1) based on different independence assumptions [38], making the discussion of identifiability difficult. This motivates fill this gap by extending identifiability results of deep generative models different missing mechanisms, and provide a scalable practical solution. Our contribution are threefold:
We provide a theoretical analysis of identifiability for generative models under different MNAR scenarios (Section 3). More specifically, we provide sufficient conditions, under which the ground truth parameters can be uniquely identified via maximum likelihood (ML) learning using observed information [24]. We also demonstrate how the assumptions can be relaxed in the face of roal-world datasets. This provides foundation for practical solutions using deep generative models.
Based on our analysis, we propose a practical algorithm model based on VAEs (Section 4), named GINA (deep generative imputation model for missing not random). This enables to apply flexible deep generative models a principled way, even in the presence MNAR data.
We demonstrate the effectiveness and validity of our approach by experimental evaluations (Section 6) on both synthetic data modeling, missing data imputation in real-world datasets, as well as downstream tasks such as active feature selection under missing data.
Backgrounds
2.1 Problem Setting
A critical component to develop model impute MNAR data is the model identifiablity [18, 43). We give the definition below:
Definition 2.1 (Model identifiability). Assume (X) is a distribution of some random variable X, 0 is its parameter that takes values in some parameter space Ωg. Then, if Po(X] (X) satisfies pa, (X) # (X) > 0, + 02, v0,, 02 E Ωg, we say that is identifiable w.r.. 0 on Ωa-
In other words, a model po X) identifiable, if different parameter configurations implies a different probabilistic distributions over the observed variables. With identifiability guarantee, if the model assumption correct, the true generation process can be recovered. Next, we introduce necessary notations and of missing data, and set up concrete problem setting.

---

Basic Notation. Similar to the notations introduced by [12, 44], let X be the complete set of variables in the system of interest. We call it observable variables. Let [1,.... be the index set of all observable variables, i.e., X = [X,]i I). Let X, denote the set of actually observed variahles, here E I a index set such that X, C X. We call O the observable pattern. Similarly, X,, denotes the set of mixsing/nnobserved variables, and X = X, U... Additionally, we use R to denote the missing mask indicator variable, such that R; = indicates X; is observed, and R; = indicates otherwise. We call probabilistic distribution p(X) on X the reference distribution, that is, the distribution that we would have observed if no missing mechanism is present; and we call the conditional distribution p( [R|x)
(X) the missing mechanism, which decides the probability of each X; being missing. Then, we can define the marginal distribution of partially observed variables, which is given by log p(Xo: R) log Jx. p(Xo; Xu, R)dX, Finally, we will use lowercase letters denote the realized values of the corresponding random variable. For example. (xo, p(X., R) is the realization/samples of X, and R, and the dimensionality of Io may vary for each realizations.
Problem setting. Suppose that have ground truth data generating process, denoted by PD PD(X,, (X,,, R), from which we can obtain (partially observed) samples (I.. ~ po(X,, R). We also have a model be optimized, denoted by (Xe.))(Xo: X., R).where 0 is the parameter of refer- ence distribution (X)). and À the parameter of missing mechanism px (). Our goal can then be described as follows:
To establish the identifiability of the model (X.,))(Xo.. R). That is, we wish to uniquely and correctly identify 0, such that Pa (X) (x()) given infinite amount of partially observed data samples from ground truth, (For PD (X.. R).
Then, given the identified parameter, we will be able to perform missing data imputation, using pa( X XX)). If our parameter estimate is unbiased, then our imputation is also unbiased, that is, Pa X Xx) = (D( (X.) for all possible configurations of Xo.
2.2 Challenges in MNAR imputation
Recall the three types of missing mechanisms: data MCAR, p( (R p( R); if data is MAR, p( Rt X) = p( Rt X otherwise, we call it MNAR. When missing data is MCAR or MAR, missing mechanism can be ignored when performing maximum likelihood (ML) inference based only on the observed data [44], as:
arg max (( log (X = xo) = arg max max (poo)))XX.)) log (X = To, R = r)
6
where log ((.)(X.)= log P(X.; In practice, ML learning on X., can done by EM algo- rithm [3, 24). However, when missing data is MNAR, the above argument does not hold, and the missing data mechanism cannot be ignored during learning. Consider the representative graphical model example in Figure (d), which has appeared in many context of machine learning. In this graphical model, X is the cause of R, and the connections between X and It are fully connected, i.e.,
each single node in R caused by the entire set X. All nodes in It are conditionally independent from each other given X.
Clearly, this is example of data generating process with MNAR mechanism. In this case. Rubin proposed to jointly optimize both the reference distribution Po (X) and the missing data mechanism Px(R)X), by maximizing:
arg max (xx..))-pp(X...) loggoo(X, xx) + (R = =x)]]
(I)
This factorization is referred as selection modeling [12, 24]. There are multiple challenges if we want to Eq. to obtain practical model that provide unbiased imputation. First, we need model assumption to be consistent with the real-world data generation process, (P,,,, Rt). Given wide range of possible MNAR scenarios, it is a challenge to design a general model. Secondly, the model need to be identifiable to enable the possibility learn the underlying process which leads to unbiased imputation.
3

---

2.3 Variational Autoencoders and its identifiability
Variational auto-encdoers [17, 41, 64] is flexible deep generative model that commonly used for estimating densities of PD(X). It takes the following form:
=log log / d
log pe
(2)
where Z is some latent variable model with prior p(Z), and pe (X] 2) given by po( ((fa(Z), with sa(.) being neural network parameterized by 0. Generally, VAEs do not have identifiability guarantees w.r.t. [16]. Nevertheless, inspired by the identifiablity of nonlinear ICA, (16) shows that the identifiability of VAE can be established up to equivalence permutation under mild assumptions, if the unconditional prior p(Z) of VAE replaced by the following the conditionally factorial exponentially family prior,
K Pr.c(Zu0 & QZZ) expl Σ Te (Z))s, j=1
where U some additional observations (called auxiliary variables), Q(Z,) is some base measure, TU( (T,, ...., Ti,k) .)) the sufficient statistics, and (:(U) (G.. (t, the corresponding natural parameters. Then, the new VAE model given by
log pe (XIU) = log / (xP)(X)))TT,.(2))
(4)
is identifiable (Theorem and 2 of [16), see Appendix G). We call the model (4) the identifiable VAE Unfortunately, this identifiability results for VAE only hold when all variables are fully observed; thus, it cannot be immediately applied to address the challenges of dealing with MNAR data stated in Section 2.2. Next, we will analyze the identifiablity of generative models under general MNAR settings (Section 3), and propose a practical method that can be used in MNAR (Section 4),
3 Establishing model identifiability under MNAR
One key issue training probabilistic models under MNAR missing data is its identifiability. Recall that (Definition 2.1) model identifiability characterize the property that the mapping from parameter 0
to the distribution po (X) is one-to-one. This is often closely related to maximum likelihood learning.
In fact, it is not hard show that Definition 2.1 is equivalent to the following Definition 3.1:
Definition 3.1 (Equivalent definition of identifiability) We say a model pe (X) is identifiable, if:
arg max E En--p- (x) log pe (X- z) = 0", v0" E Ωo BE
(5)
In other words, the "correct" model parameter 8* can be identified via maximum likelihood learning (under complete data), and the MI solution is unbiased. Similarly, when MNAR missing mechanism is present, we perform maximum likelihood learning on both X, and R using Eq. 1. Thus, we need log (X.. R) to be identifiable under MNAR, so that can correctly identify the ground truth data generating process, and achieve unbiased imputation. The identifiability of log Pe.x It) under MNAR usually not guaranteed, even in some simplistic settings [33]. In this section, will give the sufficient conditions for model identifiability under MNAR, and study how these can be relaxed for real-world applications
3.1 Sufficient conditions for identifiability under MNAR
In this section, we give sufficient conditions where the model parameters 0 can be uniquely identified by Rubin's 's objective, Eq. 1. Our aim is to i), find a set of model assumptions, so that it can cover many common scenarios and be flexible for practical interests; and ii), under those conditions, we want to show that its parameters shown in Figure 1, MNAR have many possible difference cases depending on its graphical structures. We want results to cover every situation.
be uniquely determined by the partial ML solution Eq. 1. As
Instead of doing case by case analysis, we will start our identifiability anaylsis with one fairly general case as the example shown in Figure 1 (h) where the missingness can be caused by other partially

---

observed variable, by itself (self- masking) or by latent variables. Then, we will discuss how these analysis can be applied to other MNAR scenarios in Section 3.2.
Data setting D1. Suppose the ground truth data generation process satisfies the following conditions: connections among
all variables X are generated from shared latent confounder Z, and there X; and the missingness indicator I variable can not be the parent of other variables. A typical example of such distribution is depicted in Figure (h). We further assume that pD(Xo (X Xu R) has the following parametric form: PD (Xo: Xus Rt) = Sz II Poy ( (R) Z)dZ, where Px X, Z) = Ila Px; X, 2), for some 0*, X
Then, consider the following model:
Model assumption A1. We assume that our model has the same graphical representation, as well as parametric form as data setting D1, that is, our model can be written as:
(X., = S
dX II (X,l (x)lz)
II PA. (Ra(X,Z)p(2
(6)
x.,2
Here, (0 E Z are learnable parameters that belong to some parameter space f2 -2p x f2x. Each 0 is the parameter that parameterizes the conditional distribution that connects X₂ and Z. Po. (X,a Z). Assume that the ground truth parameter of pp belongs to the model parameter space, (0", X*) E
Given such a model, our goal is to correctly identify the ground truth parameter set- tings given partially observed samples from PD o. Xu, R). That is, let (0.
arg o A)en ((z,,))~pp(X, (X..t)) log (XX, Io. R r). we would like to achieve 0 0". In order to achieve this, we must make additional assumptions.
Assumption A2. Subset identifiability: There exist partition of I, denoted by Ar {O.)s.ss. such that: for all O, E Az, Po (X..)) is identifiable on a subset of parameters {02|d E Os).
This assumption basically formalizes the idea of divide and conquer: we partition the whole index set into several smaller subsets { on which each reference distribution po po(Xo.) is only responsible for the identifiability on a subset of parameters.
Assumption A3. There exists collection of observable patterns, denote by Az := 10thsi such that: 1), Ar is cover of I: 2), (X, Ro; 1, RT,o;) > 0 for all 1 < < L; and 3), for all index O,, there exists O, E Az defined in A2, such that O, C O%--
This assumption is about the strict positivity of the ground truth data generating process. PD (X.. X., R). Instead of assuming that complete case data are available as in [38], here we assumes we should at least have some observations, PD px R. 1, Rs = 0) > 0 for O E Az, on which (X,) is identifiable.
To summarize, A1 ensures that our model has the same graphical represemtation//araaettii forms as the ground truth: A2 (X() = fxx Po(X,, X., should be least identifiable for a collection of observable patterns that forms partition of I; and Assumption A3 ensures that (x. X,,, R) should be positive for certain important patterns (i.e., those on which (X.) is identifiable.. In Appendix C, we will provide a practical example that satisfies those assumptions. Given these assumptions, we have the following proposition (See Appendix C for proof.):
Proposition (Sufficient conditions for identifiability under MNAR). Let Po.x(X,r Xu, Ku,Th R) be model on the observable variables X, and missing pattern R. and pD PD(X,, X,, R) be the ground truth distribution. Assume that they satisfies Data setting D1, Assumptions AI, A2 and A3.
Io, R r) be the set of ML solutions of Equation 1. Then, we huve θ - 10*) Ox That is, the ground truth model parameter 0* can be uniquely identified via (partial) maximum likelihood learning.
Let θ arg ma x)en ((z..)))<po(X..e log log
Missing value imputation as inference. Given a model P(e) problem can be then formularized by the Bayesian inference problem po (xx,,, (X,)X.) x po (X,,, X.). If the assumptions of Proposition are satisfied, it enables to correctly identify the ground truth
the missing data imputation
It can be arbitrary partition in the set theory sense.
5

---

reference model parameter, l* Therefore, the imputed values sampled from the posterior pa- (X (x)) will be unbiased, and can be used for down stream decision making tasks.
Remark: Note that Proposition 1 can be extended to the case where model identifiability is defined by equivalence classes [16, 53]. See Appendix F for details.
3.2 Relaxing "correctness of parametric model" assumption (A1)
In this section, we further extend our previous results to the general MNAR cases including all different examples in Figure 1. In particular, we would like to see the if the same model setting in Section 3.1 can applied to scenarios where pp(Xx: Xu, Rt and Pe.x(X,: Xu, R) might have different parametric forms, even different graphical representations.
To start with, we would like to point that the mismatch between pD (X(, R) and the model Po,a(Xo; o, Xa, R) can be, to certain extend, modeled by the mappings between spaces of parameters.
Let Q C R' denote the parameter domain of our model, (e. R.. R). Suppose we have a mapping 4 : Q C R' + R., such that (0, A) E Ω C Ω is mapped to another parameter space (T,7) = () E R' via (.). Here, Q is subset of 52 on which & is defined. Then, the re-parameterized Pa,o(Xo,Xar R) on parameter space E can be rewritten as:
Pr, (X. X.,, R) == ps ((T,)) (Xo: Xu: R)
Assuming that the inverse mapping 4-1 exists. Then trivially, if (X., R) is identifiable with respect to and X, then (X., Rt) should be also identifiable with respect to 7 and 7:
Proposition 2. Let Z c R be the parameter domain of the model pe, Xus R). Assume that the mapping Tb (0, 2) EQCCK + (r, 7) = c RJ is one-to-one on Q (equivalently, the inverse mapping & 111_ - Q injective, and Q is its image set). Consider the induced distribution with parameter space E, defined as Pr, (Xo, o, R) == p4 P4-111,,p 1(7,7) (Xo: R). Then, p is identifiable w.r.t. (T, 7), if (X. R) is identifuble w.r.l 0 and X.
Proposition 2 basically shows that if two distributions po,x (X.,R) R) and PT, R) are related by mapping & with nice properties, than the identifiability will translate between them. This already covers many scenarios of the data-model mismatch. For example, consider the case where ground truth data generation process satisfies the following assumption:
Data setting D2 Suppose the ground truth PD(Xo, Xu, R) satisfies: X are all generated by shared la- tent confounders Z (as in D1), and Rt cannot be the cause of any other variables as in [38, 56]. Typical examples are given by any of the cases in Fig 1(excluding (j) where R₁ is the cause of R2). Furthermore, the ground truth data generating process is given by the parametric form PD PX, Xx, R) = pr. (Xo. X., R), where E = E, x E, denotes its parameter space.
Then, for such ground truth data generating process, can show that the form of Equation 6, such that there exists some mapping t that can model their relationship:
always find model in
Lemma 1. Suppose the ground truth data generating process p (Xo, Xu. R) satisfies setting D2. Then, there exists a model (X. Xu, R), such that: 1), (X, Xu, R) can be written in the form of Equation 6 (i.e., Assumption A1; and 2), there exists a mapping & as described in Proposition 2, such that Pr Xo, R) (1,)) (X, R), for all (T,7) E
Model identification under mata-model mismatch. Since we showed the identifiability can be preserved under the parameter space mapping (Proposition 2), next we will prove that if the model Pa.x(X,, Xx, R) is trained on partially observed data points sampled from p X,,, R) that satisfies data setting D2, then the MI solution is still unbiased. For this purpose, inspired by Lemma 1, we work with the following additional assumption:
Model Assumption A4 Let ". Xx., R) denote our ground truth data generating process that satisfies data setting D2. Then, we assume our model (X. X,, R) the one that satisfies the description given by Lemma 1. That is, its parametric form is given by Equation 6, and there exists mapping & as described in Proposition 2, such that (x, R) = (p-1(r (X,, R).
Then, we have the following proposition:

---

Proposition (Sufficient conditions for identifiability under MNAR and data-model mismatch). Let Pa.x Po.x(Xo: X,, R) be a model on the observable variables X and missing pattern R. and PD( (X., Xa; R) be the ground truth distribution, Assume that they satisfies Data setting D2, Assump- tion A2, A3, and A4. Let
arg max (8.A)E F((..r))po(X.... log P(8,A) (X, To: R=r) r) be the set of ML solutions of Equation 1. Then, we have θ = { (T*)} X Ox Namely, the ground truth model parameter T" of pD be uniquely identified (as ((f') via ML learning.
Remark: practical implications Proposition 3 allows us to deal with the cases where the parameter- ization of ground truth data generating process and model distribution are related through a set of mappings, (Φ.) J. In general, the graphical structure of PD X., R) can be any cases in Figure 1 excluding (j). Then, in those cases, we are still able to use a model that corresponds to Equation (Fig (h)) to perform MIL learning, provided that our model is flexible enough (Assumption A4). This greatly improves the applicability of our identifiability results, and we can build a practical algorithm based Equation 6 to handle many practical MÑAR cases.
4 GINA: A Practical Imputation Algorithm for MNAR
In the previous section, we have established the identifiability conditions for models in the form of Equation (6). However, in order derive practically useful algorithm, still need to specify a parametric form of the model, that is both flexible and compatible with our assumptions. In this section, by utilizing the results Section we propose GINA, a deep generative imputation model for MNAR data (Figure 2). GINA fulfill identifiability assumptions above, and can handle general MNAR case discussed in section 3.2. The code is released https: /gttbbb....ttt/gg thub. com/
micr wacrsooff/projccttt azua.
The parametric form of GINA We utilize the flexibility of deep generative models to model the data generating process. We assume that the reference model po (X) is parameterized by iden- tifiable VAE (see Section 2.3) to satisfy Assumption A2. That is, Po(X(U) = J2 dZp. (X- f(Z))p(( U where U is some fully ob- served auxiliary inputs. The decoder (X fo (Z)) is parameterized by a neural network, f: RR + RR " For convenience, we will drop the input U to po ( x and simply use (x( to denote Po(X [X] (U). Finally, for the missing model px( 2), we use a Bernoulli likeli- hood model, px R XX Z) == where a X, Z) is parameterized by a neural network.
Ia Fa( X, 2) R2 Tx( 2
In Appendix G, we show that GINA fulfill the required assumptions of Proposition and 3. Thus, we can use GINA to identify the ground truth data generating process, and perform missing value imputation under MNAR. The consistency of estimation result is also given in Appendix H.
Graphical represen- tations of our GINA.
Figure 2:
Learning and imputation In practice, the joint likelihood in Equation is intractable. Similar to the approach proposed in [12]. we introduce variational inference network, g0 (x),),, which enable us to derive a importance weighted lower bound of log Po,a (X., R):
log Po.a R) 2 (K, X., R) Ez,
E......."
1 a pal (21x.) log K
Σ wk
Xx -=x (po
the importance weights. Note that we did not notate the missing pattern R. as additional input to qo. as this information is already contained in Xo. Then, optimize the parameters 0, X, by solving the following optimization problem
where wi.
9o12=>"|x.)
0", X", Φ arg max ((e..r)>pp(X... Lx (0, X, 0, X. = To. R = r) 8.A.
Given 8", A", o" we can impute missing data by solving the approximate inference problem: px Po Po (X = - po p (x)(xx1))x(((1xxx)
Thus, GINA can be used to predict missing data even when the data are MNAR.
7

---

5
Related works
We mainly review recent works for handling MNAR data. In Appendix A, we provide a brief review of traditional methods that deal with MCAR and MAR.
When the missing data is MNAR, general framework to learn a joint model on both observable variables and missing patterns [24], in which a model of missing data is usually assumed [52, 11]. This approach is also widely adopted in imputation tasks. For example, in the field of recommender
systems, different probabilistic models are used within such framework [8, 30, 58, 22, 21J. A similar approach has also been taken in the context of causal approach to imputation [60, 59, 20]. Similar to the use of the missing model, they have used explicit model of exposure and adopted causal view, where MNAR is treated as a confounding bias. Apart from these, inverse probability weighting methods are also used to debias the effect of MNAR [48, 58, 29] for imputation.
One issue that is often ignored by many MNAR methods is the model identifiability. Both parametric and non-parametric identifiability under MNAR has been discussed for certain cases ( [34, 33, 35, 57, 55, 53)). For example, [57] proposed the instrumental variable approach to help the identification of MNAR data. [33] investigated the identifiability of normal and normal mixture models, and showed that identifiability for parametric models is highly non-trivial under MNAR. [34] studied conditions for nonparametric identification using shadow variable technique. Despite the resemblance to the auxiliary variable in our approach, [33, 34] mainly considers the supervised learning (multivariate regression) scenario. [38, 37, 50] also discussed similar topic based on graphical and causal approach in a non-parametric setting. Although the notion of recoverability has been extensively discussed, their methods do not directly lead to practical imputation algorithms in a scalable setting. On the contrary, our work takes a different approach, in which we handle MNAR with a parametric setting, by dealing with learning and inference in latent variable models. We step aside from the computational burden with the help of recent advances in deep generative models for scalable imputation.
There has been a growing interest in applying deep generative models to missing data imputation. In [28, 25, 40], scalable methods for training VAEs under MAR have been proposed. Similar methods have also been advocated in the context of importance weighted VAEs, multiple imputation [32]. and heterogeneous tabular data imputation [40, 26, 27]. Generative adversarial networks (GANs) have also been applied to MCAR data [63, 19). More recently, deep generative models under MNAR have been studied [12, 5, 6), where different approaches such as selection models [44, 7] and pattern-set mixture models [23] has been combined with partial variational inference for training VAEs. However, without additional assumptions, the model identifiability remains unclear in these approaches, and the posterior distribution of missing data conditioned on observed data might be biased.
Experiments
We study the empirical performance of the proposed algorithm of Section 4 with both synthetic data (Section 1. and two real-world datasets with music recommendation (Section 6.2) and personalized education (Section 6.3). The experimental setting details can be found in Appendix B.
6.1 Synthetic MNAR dataset
We first consider 3D synthetic MNAR datasets. We generate three synthetic datasets with nonlinear data generation process (shown in Appendix B. For all datasets, X1, X2, X3 are generated via the latent variables, Z,, Zz. Z3. where X1 are fully observed and X2 and X3 are MNAR. For dataset A, we apply self-masking( (similar to Figure I(c)): X; will be missing if X, > O. For datasets B and C. we apply latent- detend--e self-masking: X, will be missing, if (X,, Zi. Z2, Za) > O, where g is a linear mapping whose coefficients are randomly chosen.
We train GINA and baseline models with partially observed data, Then, we use the trained models to generate random samples. By comparing the generated samples with the ground truth data density, we can evaluate whether pD(X) is correctly identified. Results are visualized in Figure 3. addition, we show the imputation results in Appendix J. Across three datasets, PVAE performs poorly, as it does not account for the MNAR mechanism. Not-MIWAE performs better than PVAE, as is able to generate samples that are closer to the mode. However, it is still biased more towards the observed
8

---

Training set
PVAE fit
NOt-MIWAE fIt
Qurs
Data
A(1)
A(II)
A(ill)
A(lv)
B(I)
B(li)
B(Il))
B(lv)
Data
C())
C(ii)
C(ii)
C(iv)
Figure 3: Visualization of generated X2 and X3 from synthetic experiment. Row-wise (A- plots for dataset A. B, and C, respectively: Column-wise (i-iv): training set (only displays fully observed samples), PVAE samples, Not-MIWAE samples, and GINA samples, respectively. Contour plot: kernel density estimate of ground truth density of complete data;
values. On the other hand, GINA is much aligned ground truth, and is able recover the ground truth from partially observed data. This experiment showed the clear advantage of our method under different MNAR situations.
6.2 Recommender dataset imputation with random test set
Method
Test MSE
Next, we apply our models to recommendation sys- tems on Yahoo! R3 dataset [30, 60) for user-song ratings which is designed to evaluate MNAR impu- tation. contains an MNAR training set of more than 300K self-selected ratings from 15,400 users on 1,000 songs, and an MCAR test set randomly se- lected ratings from 5,400 users 10 random songs. We train all models on the MNAR training set, and evaluate on MCAR test set. This is repeated 10 times with different random seeds. Both the missing model for GINA (p( X Z)) and Not-MIWAE (p( R[X) are parameterized by linear neural nets with Bernoulli likelihood functions. The decoders for GINA, PVAE, and Not-MIWAE uses Gaussian likelihoods with the same network structure. See Appendix B for imple- mentation details and network structures.
Matrix Factorization Methods
PMF
I.401
IPW-PMF
1.375
Deconfounded-PMo
1.329
PME-MNAR
PMF-MAR
1.480
VAE-based models
PVAE
1.25910.003
PVAE w/o IW
1.261 004
Not-MIWAE
1.078:1.0000
GINA
1.052:10.022
Oihers
CPTy-MNAR
1.056
Logitvd-MNAR
1.141
AutoRec
1.199
Oracle-test
1.057
Table I: Test MSE on Yahoo! R3
We compare to the following basclines: 1), probabilistic matrix factorization (PMF) [36]; 2), inverse probability weighted PMF [48]; 3), Deconfounded PMF [60]; 4), PMF with MNAR/MAR data [8]; 5), CPTv and Logitv models for MNAR rating [30]; 6). Oracle [8]: predicts ratings based on their marginal distribution the test set; and 7) AutoRec [49]: Autoencoders that ignores missing data.
Results are shown in Table 1. Our method (GINA) gives the best performance among all methods. Also, VAE-based methods are consistently better than PMF-based methods, and MNAR-bas Rbased models consistently outperform their MAR versions. More importantly, among VA VAE-based models, our GINA outperforms non-identifiable counterpart (Not-MIWAF.), and MAR counterpart (PVAE), where both models can not generate unbiased imputation.

---

6.3 Missing data imputation and active question selection on Eedi education dataset
Finally, we apply our methods to the Eedi education dataset [61], one of the largest real-world education response datasets. We consider the Eedi competition task dataset, which contains over 1 million responses from 4918 students to 948 multiple-choiee diagnostic questions. Each diagnostic question is a multiple-choice question. We consider predicting whether a student answers a question correctly or not. Over 70% of the entries are missing. The dataset also contains student metadata which we use as the auxiliary variables. In this experiment, we randomly split the data in a 90% train/ 10% test/ 10% validation ratio, and train our models on the response outcome data.
We evaluate our model on two tasks. Firstly, we perform missing data imputation, where different methods perform imputation over the test set. As opposed Yahoo! R3 dataset, now the test set is MNAR, thus we use the evaluation method suggested by [60], where we evaluate the average per-question MSE For each question, over all students with non-empty response. Then, the MSEs of all questions averaged. We call this metric the debiased MSE. While regular MSE might be biased toward questions with more responses, the debiased MSE treats all questions equally, and can avoid selection bias to a certain degree. We report results for 10 repeats in the first column in Table 2. We can see that our proposed GINA achieves significantly improved results comparing to the baselines.
Secondly, we evaluate personalized education through active question selection [28] on the test set which is task 4 from this competition dataset. The procedure is as follows: for each student in the test set, at each step, the trained generative models are used to decide which is the most informative missing question to collect next. This is done by maximizing the information reward in [28] (see Appendix for details). Since at each step, different students might collect different questions, there isn't simple way to debias the predictive MSE in the imputation task. Alternatively, we evaluate each method with the help of question meta data (difficulty level, which is scalar). Intuitively, when the student response to the previously collected question is correct, we expect the next diagnostic question which has higher difficulty levels, and vice versa. Thus, we can evaluate the mean level change after correct/incorrect responses, and expect them to have significant differences. We also perform t-test between the level changes after incorrect/correct responses and report the p-value.
Table 2: Performance on Eedi education dataset (with standard errors)
Method
Debiased MSE
Level change (correct)
Level change (incorrect)
p-value
PVAE
0.194.10.001
0.13110.133
-0,101 10,160
0,514
Not-MIWAE
0.192:10.000
0.06211.112
-0.073 -0.353=20.189
0.561
GINA
0.188:10.001
0.945:0.151
1.01 10-7
We can see in Table 2, GINA is the only method that reports significant p-value (<0.05) between the level changes of next collected questions after incorrect/correct responses which are desired. This further indicates that our proposed GINA predicts the unobserved answer with the desired behavior.
7 Conclusion
In this paper, we provide a analysis of identifiability for generative models under MNAR, and studies sufficient conditions of identifiability under different scenarios. We provide sufficient conditions under which the model parameters can be uniquely identified, via joint maximum likelihood learning on XX, and R. Therefore, the learned model can be used to perform unbiased missing data imputation. We proposed a practical algorithm based on VAEs, which enables us to apply flexible generative models that is able to handle missing data in a principled way. The main limitation of our proposed practical algorithm is the need for auxiliary variables U, which is inherited from identifiable VAE models [16]. In practice, they may not be always available. For future work, we will investigate how to address such limitation, and how to extend to more complicated scenarios.
10

---

Acknowledgements and Disclosure of Funding
We thank Martin Kukla, Angus Lamb, Yingzhen Li, Dawen Liang, Chang Liu, Ruibo Tu, Yordan Zaykov (in alphabetical order) for helpful discussions and implementation support.
References
[1] Paul D Allison. Missing data. Sage publications, 2001.
[2] James Arbuckle, George A Marcoulides, and Randall E Schumacker. Full information estimation in the presence of incomplete data. Advanced structural equation modeling: Issues and techniques, 243:277, 1996.
[3] Arthur Dempster, Nan M Laird, and Donald B Rubin. Maximum likelihood from incomplete data via the em algorithm. Journal of the Royal Statistical Society: Series B (Methodological), 39(1):1-22, 1977.
[4] Craig K Enders and Deborah I. Bandalos. The relative performance of full information maximum likelihood estimation for missing data in structural equation models. Structural equation modeling, 6(3):430-457) 2001.
[5] Sahra Ghalcbikesabi, Rob Cornish, Chris Holmes, and Luke Kelly. Deep generative missingness pattern-set mixture models. In International Conference on Artificial Intelligence and Statistics, pages 3727-3735. PMLR, 2021.
[6] Yu Gong, Hossein Hajimirsadeghi, Jiawei He, Thibaut Durand, and Greg Mori. Variational selective autoencoder: Learning from partially-observed heterogeneous data. In International Conference on Artificial Intelligence and Statistics, pages 2377-2385. PMLR, 2021.
[7] James Heckman. Sample selection bias specification error. Econometrica: Journal of the econometric society, pages 153-161, 1979.
[8] José Miguel Hernández-Loban Lobato, Neil Houlsby, and Zoubin Ghahramani. Probabilistic matrix factorization with non-random missing data. In International Conference on Machine Learning, pages 15 1522-1520. PMLR, 2014.
[9] Nicholas J Horton and Stuart Lipsitz. Multiple imputation in practice: comparison of software packages for regression models with missing variables. The American Statistician, 55(3):244-254, 2001.
[10] Daniel G Horvitz and Donovan J Thompson. A generalization of sampling without replacement from finite universe. Journal the American statistical Association, 47(260):663-685,, 1952.
[11] Joseph G Ibrahim, Stuart R Lipsitz, and M-H Chen. Missing covariates in generalized linear models when the missing data mechanism is non-ignorable. Journal of the Royal Statistical Society: Series B (Statistical Methodology), 61(1):173-19. 1999.
[12] Niels Bruun Ipsen, Pierre-Alexande Mattei, and Jes Frellsen. not-miwae: Deep generative modelling with missing not at random data. arXiv preprint arXiv:2006. 12871, 2020.
[13] Janus Christian Jakobsen, Christian Gluud, Jørn Wetterslev, and Per Winkel. When and how should multiple imputation be used for handling missing data in randomised clinical trials practical guide with flowcharts. BMC medical research methodology, 17(1):1-10, 2017.
[14] Dietmar Jannach, Markus Zanker, Alexander Felfernig, and Gerhard Friedrich. Recommender systems: an introduction. Cambridge University Press, 2010.
(15) Phimmarin Keerin, Werasak Kurutach, and Tossapon Boongoen. Cluster-based knn missing value imputation for dna microarray data. In 2012 IEEE International Conference on Systems, Man, and Cybernetics (SMC), pages 445-450. IEEE, 2012.
[16] Ilyes Khemakhem, Diederik Kingma, Ricardo Monti, and Aapo Hyvarinen. Variational au- toencoders and nonlinear iea: A unifying framework. In International Conference on Artificial Intelligence and Statistics, pages 2207-2217. PMLR, 2020.

---

[17] Diederik P Kingma and Max Welling, Auto-encoding variational bayes. arXiv preprint 1rX.v:1312.6114, 2013.
[18] Daphne Koller and Nir Friedman. Probabilistic graphical models: principles and techniques. MIT press, 2009.
[19] Steven Cheng-Xian Xian Li, Bo Jiang, and Benjamin Marlin. Misgan: Learning from incomplete data with generative adversarial networks. arXiv preprint arXiv: 2019.
[20] Dawen Liang, Laurent Charlin, and David M Blei. Causal inference for recommendation.
[21] Dawen Liang. Laurent Charlin, James McInerney, and David M Blei. Modeling user exposure in recommendation. In Proceedings of the 25th international conference on World Wide Web, pages 951-961, 2016.
[22] Guang Ling, Haiqin Yang, Michael R Lyu, and Irwin King. Response aware model-based collaborative filtering. arXiv preprint arXiv:/210.4866 2012.
[23] Roderick JA Little. Pattern-mixture models for multivariate incomplete data. Journal of the American Statistical Association, 88(421):125-13, 1993.
[24] Roderick JA Little and Donald B Rubin. Statistical analysis with missing data, volume 793. John Wiley & Sons, 2019.
[25] Chao Ma, Wenbo Gong, José Miguel Hermández-Lobato, Noam Koenigstein, Sebastian Nowozin, and Cheng Zhang. Partial vae for hybrid recommender system.
[26] Chao Ma, Sebastian Tschiatschek, José Miguel Hernández-Lobato Richard Turner, and Cheng Zhang. Vaem: deep generative model for heterogeneous mixed type data. arXiv preprint arXiv:2006. 11941, 2020.
[27] Chao Ma, Sebastian Tschiatschek, Yingzhen Li, Richard Turner, Jose Miguel Hernandez-Lobato and Cheng Zhang. Hm-vaes: deep generative model for real-valued data with heterogeneous marginals. In Symposium Advances in Approximate Bayesian Inference, pages 1-8. PMLR, 2020.
[28] Chao Ma, Sebastian Tschiatschek, Konstantina Palla, José Miguel Vernández-LLobtte Sehastian Nowozin, and Cheng Zhang. Eddi: Efficient dynamic discovery of high-value information with partial vac. arXiv preprint 11X9v:1809.11 142, 2018.
[29] Wei Ma and George H Chen. Missing not at random in matrix completion: The effectiveness of estimating missingness probabilities under a low nuclear norm assumption. arXiv preprint 1910v: 12774, 2019.
[30] Benjamin Marlin and Richard S Zemel. Collaborative prediction and ranking with non- random missing data. In Proceedings of the third ACM conference on Recommender systems, pages 12, 2009.
[31] Herbert W Marsh. Pairwise deletion for missing data in structural equation models: Nonpositive definite matrices, parameter estimates, goodness of fit, and adjusted sample sizes. Structural Equation Modeling: A Multidisciplinary Journal, 5(1):22-36, 1998.
[32] Pierre-Alexandre Mattei and Jes Frellsen. Miwae: Deep generative modelling and imputation of incomplete data sets. In International Conference on Machine Learning, pages 4413-4423 PMLR, 2019.
[33] Wang Miao, Peng Ding, and Zhi Geng. Identifiability of normal and normal mixture models with nonignorable missing data. Journal of the American Statistical Association, 1116516):1673- 1683, 2016.
[34] Wang Miao, Lan Liu, Eric Tchetgen Tchetgen, and Zhi Geng. Identification, doubly robust estimation, and semiparametric efficiency theory of nonignorable missing data with a shadow variable. arXiv preprint arXiv: (tt.::::50000555555 2015.
12

---

[35] Wang Miao and Eric Tchetgen Tchetgen Identification and inference with nonignorable missing covariate data, Statistica Sinica, 28(4):2049. 2018.
[36] Andriy Mnih and Russ R Salakhutdinov. Probabilistic matrix factorization. Advances in neural information processing systems, 20: 20:1257-1264, 2007.
[37] Karthika Mohan and Judea Pearl. Graphical models for recovering probabilistic and causal queries from missing data. Technical report, CALIFORNIA UNIV LOS ANGELES COGNI- TIVE SYSTEMS LAB, 2014.
[38] Karthika Mohan, Judea Pearl, and Jin Tian. Graphical models for inference with missing data. Advances in neural information processing systems, 26:1277-1285, 2013.
[39] Jared S Murray et al. Multiple imputation: review of practical and theoretical findings. Statistical Science, 3322)144-115,, 2018.
[40] Alfredo Nazabal, Pablo M Olmos, Zoubin Ghahramani, and Isabel Valera. Handling incomplete heterogeneous data using vaes. Pattern Recognition, 107:107501, 2020.
[41] Danilo Jimenez Rezende, Shakir Mohamed, and Daan Wierstra. Stochastic backpropagation and approximate inference deep generative models. In International conference on machine learning, pages 1278-1286. PMLR, 2014.
[42] James M Robins, Andrea Rotnitzky, and Lue Ping Zhao. Estimation of regression coefficients when some regressors are not always observed. Journal of the American statistical Association, 89(427):8468805 1994.
[43] Thomas Rothenberg. Identification in parametric models. Econometrica: Journal of the Econometric Society, pages 577-591, 1971.
[44] Donald B Rubin. Inference and missing data. Biometrika, 63(3):581 1976.
[45] Donald B Rubin. Formalizing subjective notions about the effect of nonrespondents in sample surveys. Journal of the American Statistical Association, 72(359):538-533 1977.
[46] Donald B Rubin. An overview of multiple imputation. In Proceedings of the survey research methods section of the American statistical association, pages 79-84. Citeseer, 1988.
[47] Donald B Rubin. Multiple imputation for nonresponse in surveys, volume 81. John Wiley & Sons, 2004.
[48] Tobias Schnabel, Adith Swaminathan, Ashudeep Singh, Navin Chandak, and Thorsten Joachims. Recommendations as treatments: Debiasing learning and evaluation. arXiv preprint arXiv 2016.
[49] Suvash Sedhain, Aditya Krishna Menon, Scott Sanner, and Lexing Xie. Autorec: meet collaborative filtering. In Proceedings of the 24th international conference on World Wide Web, pages 111-112, 2015.
Autoencoders
[50] Iya Shpitser, Karthika Mohan, and Judea Pearl. Missing data causal and probabilistic problem. Technical report, CALIFORNIA UNIV LOS ANGELES DEPT OF COMPUTER SCIENCE, 2015.
[51] Fiona M Shrive, Heather Stuart, Hude Quan, and William A Ghali. Dealing with missing data in multi-question depression scale: comparison of imputation methods. BMC medical research methodology, 6(1):1-10, 2006.
[52] Aude Sportisse, Claire Boyer, and Julie Josse. Imputation and low-rank estimation with missing not at random data. Statistics and Computing, 1669)113291163,, 2020.
[53] Aude Sportisse, Claire Boyer, and Julie Josses. Estimation and imputation in probabilistic principal component analysis with missing not at random data. Advances in Neural Information Processing Systems, 33, 2020.
13

---

[54] Daniel J Stekhoven and Peter Bühlmann. Missforest non non-maraettric missing value imputation for mixed-type data. Bioinformaties, 28(1): 2012.
[55] Gong Tang, Roderick JA Little, and Trivellore E Raghunathan. Analysis of multivariate missing data with nonignorable nonresponse. Biometrika, 90(4):747-764, 2003.
[56] Ruibo Tu, Cheng Zhang, Paul Ackermann, Karthika Mohan, Hedvig Kjellströmm and Kun Zhang. Causal discovery in the presence of missing data. In The 22nd International Conference on Artificial Intelligence and Statistics, pages 1762-1770. PMLR, 2019.
[57] Sheng Wang, Jun Shao, and Jae Kwang Kim. An instrumental variable approach for identi- fication and estimation with nonignorable nonresponse. Statistica Sinica, pages 1097-1116, 2014.
[58] Xiaojie Wang. Rui Zhang, Yu Sun, and Jianzhong Qi. Doubly robust joint learning for recom- mendation on data missing not at random. In International Conference Machine Learning. pages
PMLR, 2019.
[59] Yixin Wang and David M. Blei. The blessings of multiple causes, 2019.
[60] Yixin Wang, Dawen Liang, Laurent Charlin, and David Blei. The deconfounded recom- mender: A causal inference approach recommendation. arXiv preprint #rX8iv/808.000 06581, 2018.
[61] Zichao Wang, Angus Lamb, Evgeny Saveliev, Pashmina Cameron, Yordan Zaykov, José Miguel Hernández-Lobato Richard E Turner, Richard G Baraniuk, Craig Barton, Simon Peyton Jones, Simon Woodhead, and Cheng Zhang. Diagnostic questions: The neurips 2020 education challenge. arXiv preprint arXiv:2007.1206 2020.
[62] Ian R White, Patrick Royston, and Angela M Wood. Multiple imputation using chained equations: issues and guidance for practice. Statistics in medicine, 3704)377399 2011.
[63] Jinsung Yoon, James Jordon, and Mihaela Schaar. Gain: Missing data imputation using generative adversarial nets. In International Conference on Machine Learning, pages 5689- 5698. PMLR, 2018
[64] Cheng Zhang. Judith Bütepage, Hedvig Kjellström, and Stephan Mandt. Advances in variational inference. IEEE transactions on pattern analysis and machine intelligence, 41(8):2008-20 2026, 2018.
14


Reviews:
[{"review_id": "uCSQ9U41U3v", "reviewer": "Reviewer_REC5", "summary": "The article is concerned with model identifiability in Missing Not At Random (MNAR) data setting, with a particular focus on deep generative models. The authors introduce a unified view of model identifiability under MNAR, and derive sufficient conditions for identifiability in several MNAR scenarios, which highlight new important concepts. They also introduce a new imputation method for MNAR data based on identifiable VAE models, which is submitted as a python code, and extensively tested on synthetic and real data sets against state of the art methods for MNAR imputation. ", "questions": "", "limitations": "", "rating": 8, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Identifiability of MNAR data models is an important topic, as MNAR data is a major problem in applied problem which often impairs analysis, and few methods have been developed to overcome this. Even fewer existing work provide some theoretical understanding in such settings. In addition, existing results on identifiability usually lead to computationally costly imputation methods which are not easily generalized. Thus, I think this article is important and quite remarkable, as it provides original theoretical results, along with an efficient software and exhaustive numerical simulations, as well as an actual application to a large scale education data set; I believe the method could have an important impact in applications. I have a few questions for the authors\n- Concerning Assumptions 2 and 3, is there a dependence somewhere in your results in the number of partitions S and L and in their size ? \n- Do you believe these identifiability results could pave the way for estimation consistency in MNAR settings ?\n- What kind of architecture did you use in practice for the VAE in GINA ? Did you use different ones for each problem ? How robust do you think the procedure is to the choice of architecture ?\n\nI have read the authors' rebuttal, which provide relevant answers to my questions. The authors added simulations to empirically evaluate the robustness of the method to the networks' architecture.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The authors adequately addressed the limitations and potential negative societal impact of their work", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "2"}, "extra_text": {}}, {"review_id": "homtJZtnwa4", "reviewer": "Reviewer_DJpx", "summary": "This paper is about identifiability of the data generating process and the missing value mechanism in a missing-not-at-random setting. The paper provides an analysis of identifiability of generative models and propose a deep generative model with identifilibity guarantees under some assumptions. There are existing methods for handling MNAR without identifiability guarantees and there are theoretical analyses on identifiability without practical scalable algorithms. The paper aims to bridge this gap.  \nA VAE based model for imputation is proposed and compared to two other baselines.  The proposed method models the data and missing mechanism jointly, and in order to ensure identifiability the proposed method needs auxiliary variables that are always observed.", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper is well written and clearly presented. The focus on MNAR is important as it has been much overlooked compared to how prevalent it is in real world data and the focus on identifiability is useful to further the use of models handling MNAR data.  \nIt nicely address the challenges of MNAR and identifiability and builds up its case coherently. Identifiability is first analyzed under the correct graphical model, then relaxed to cover data model mismatch. The model assumptions are clearly stated as well as the assumptions about the data generating process.\n\nIdentifiable models makes sure that the model parameters can be uniquely determined, that the parameter estimates are unbiased.\n- In line 38 and 120 this is taken to also be the case for imputations from such models, that identifiability is needed to achieve unbiased imputations. Is this well established? could you provide any references on this? Of course if you indeed get unbiased estimates of the ground truth parameters, imputations will be unbiased, but couldn't two (non-indentifiable) models with different parameter settings in principle give the same (unbiased) imputations? Or is it that non-identifiability CAN lead to biased imputations? \n\nThe experiments are few, but nice. Figure 3 is informative as a model evalution using another metric than imputation MSE. \n- Is it clear that the difference between GINA and not-MIWAE is due to identifiability and not just plain old model misspecification? that is, the missing mechanism is based on the latent variable, GINA uses the latent variable as input to the missing model while the not-MIWAE does not. Could the problem of model misspecification be separated from the issue of non-identifiability in some way in this experiment?\n\nMinor comments:  \n- Eq (1) is referred to as the selection model factorization, but this is just the joint (which could be factorized as selection model or pattern mixture), maybe update text or equation.   \n- Line 177: should it be proposition 1 instead of proposition 2?  \n- Maybe this work [1] on MNAR is also of interest to you.  \n- Appendix B), I think you mean the the not-MIWAE is described by figure 1 d), not figure 1 a) (and the PVAE is described by figure 1 a)). \n\n[1] Ghalebikesabi, Sahra, et al. \"Deep Generative Missingness Pattern-Set Mixture Models.\" International Conference on Artificial Intelligence and Statistics. PMLR, 2021.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The authors have adequately addressed the limitations and potential negative societal impact of their work.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "2"}, "extra_text": {}}, {"review_id": "HDdB3r4oyx", "reviewer": "Reviewer_eDY7", "summary": "This paper studies identifiability of generative models for Missing Not at Random (MNAR) data. MNAR is the most general missing mechanism, where the the missingness can depend on both observed and unobserved data. Most prior imputation techniques work under MCAR or MAR assumptions, and there are few models to handle missing data under MNAR. In this respect, model identifiably under MNAR assumption has been mostly overlooked. In fact, the identifiability of models is crucial for learning from MNAR data to uniquely determine the model parameters and have unbiased imputations. This paper provides the sufficient conditions and assumptions to satisfy identifiability of generative models under MNAR. In addition, it proposes a practical deep generative model (GINA) based on VAEs which can satisfy the identifiability conditions.\n\nThe contributions are as follows:\n* Theoretical analysis of model identifiably under MNAR, which gives sufficient conditions to have identifiable models (uniquely identify the model parameter, matching the ground-truth data generation process).\n\n* A practical algorithm to satisfy these conditions and learn deep generative models under MNAR.\n\n* Experimental studies showing effectiveness of the proposed model on both synthetic and real datasets.", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "### Strengths\n- I appreciate the author’s great effort to include both theoretical analysis and the practical model.\n\n- Most imputation papers try to blindly propose a model which gives better results. This paper originates from an important limitation in existing imputation techniques and tries to fill the gap in the state-of-the-art models. Imputation under MNAR and specially model identifiability under MNAR to have unbiased imputations have been mostly overlooked in the prior work. So, I believe this work has significant contribution to the literature for learning from incomplete data.\n\n\n### Limitations\n- I appreciate the experimental studies by the authors on new datasets which are more relevant to MNAR problem. I think the experiments are sufficient. But, as a recommendation for future work, the UCI datasets in [10] or the paper below (e.g. Table 4) could be also used to have a more comprehensive comparison.\n    - Yu Gong, et al. \"Variational Selective Autoencoder: Learning from Partially-Observed Heterogeneous Data.\" International Conference on Artificial Intelligence and Statistics, 2021.\n    - BTW, this paper can also work under MNAR. But, it does not analyze the model identifiably. It is good to cite this paper and explain the advantage of your work.\n\n### Minor comments\n- The paper is somehow hard to read. \n    - Sometimes the notation is not conventional and makes it complicated to understand. For example, using $\\mathbf{x}$ for random variables is more conventional compared to $X$. I understand that the authors used $X$, $R$, $O$ as sets. But, defining them as sets makes it hard to understand.\n    - GINA algorithm is hard to understand. It is not based on the original VAE formulation. So, it is hard to understand and requires reading [14] and [10].\n    - The main limitation of the model is discussed in two sentences in Section 7. It was hard for me to understand it. What does the auxiliary variables mean?\n    - There are also some typos:\n         - The sentence in line 104-105 is not understandable. \n         - In line 158, I think $R_{I\\O'_l} = 0$ miss $= 0$\n\n\n### After Rebuttal\nI acknowledged I read the authors' response and considered it in the final discussions with the reviewers.\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "There is no sensitive data in this paper. But, in general, imputation techniques might be used to retrieve the users' information without their consent.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "8 hours"}, "extra_text": {}}, {"review_id": "7-wQq-Q9uM", "reviewer": "Reviewer_NoWR", "summary": "This paper proposed a new approach to identify the generative model under missing not at random scenario.\nThe paper starts with a model on the underlying distribution and then propose assumptions so that the model's parameters are still identifiable even if missing at random is violated.", "questions": "", "limitations": "", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "This paper proposed an interesting theoretical framework on the missing data problem. While I see some novelty in this paper, there are some critical issues that the authors did not address.\n\nQ1. [missing data vs model assumption]\nIn missing data problems, there are two types of assumptions we need to consider: missing data assumption and model assumption. \nThis paper starts with a (parametric) model and then impose missing data assumption on it.\n\nWhile this leads to an identifiable result (under additional assumptions), this identification is not \"nonparametric identification\". \nI think this should be discussed in the paper.\n\n\nQ2. [Strong assumption (A1-3)]\nIt seems to me that the main idea that this paper can avoid problems of MNAR is due to assumption (A1-3), which basically imply the problem is similar to the ignorable scenario (MAR). \nThese assumptions seem to be pretty strong.\n\nCan the authors provide some concrete and simple examples on how these assumptions can be true? \nWill the data follows a multivariate Gaussian with some additional constraints make these assumption true?\n\n\nQ3. [Section 3.2]\nThis relaxation is rather weak--the one-to-one assumption almost says that the problem remains the same. \nWhat generally would happen is that the data is not generated from a parametric model but a nonparametric model.\nIn this case, the model we assume will be a best parametric approximation to the target model. \nCan we argue that even under this case, our fitted model will be close to the nonparametric model with some bounds on the difference? \n\n\nQ4. [About GINA]\nI was not very comfortable about this part since the neural network itself is a model and is a very complex one. \nWhat are the conditions on f? This should bee clarified. \n\nAlso, in appendix G, the authors basically need the generative model to be from the model assumed by GINA.\nBut it is pretty difficult to believe that a neural net is the actual generative model of the data (neural net can be a good approximation but not the actual generative process)...\n\nMoreover, I don't think that the authors proved GINA satisfy assumption A4 in appendix G. \nThere needs to be some additional assumptions on the actual generative process when the neural net is not the actual generative model. \n\nQ5. [Model diagnostic]\nGiven that the authors have assumed a particular model for the generative process, the first thing that we can do is to check if the model assumption makes sense (at least for the observed part).\nThis should be discussed.\nAnd what should we do if we find out that our assume model does not fit to the data?\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "I have described limitations of the paper in my main review.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "6"}, "extra_text": {}}]

Author rebuttal:

Meta-review: 
The paper is on imputing missing variables when they are missing not at random (MNAR). All reviewers agree that the paper makes a valuable contribution on a topic that is important but often overlooked. This is a clear accept. 

In the camera-ready version, please incorporate the reviewers' feedback. Moreover, during discussion, reviewers brought up the idea of illustrating assumptions A1-A3 on a simple example, which I would indeed recommend to do (e.g in the appendix).

Decision: Accept (Poster)