Identifiable Generative models for Missing Not at Random Data Imputation

1 Introduction . Missing data is an obstacle in many data analysis problems , which may seriously compromise the performance of machine learning models , as well as downstream tasks based on these models . Being able to successfully recover/impute missing data in an unbiased way is the key to understanding the structure of real-world data . This requires us to identify the underlying data-generating process , as well as the probabilistic mechanism that decides which data is missing . In general , there are three types of missing mechanisms [ 44 ] . The first type is missing completely at random ( MCAR ) , where the probability of a data entry being missing is independent of both the observed and unobserved data ( Figure 1 ( a ) ) . In this case , no statistical bias is introduced by MCAR . The second type is missing at random ( MAR ) , which assumes that the missing data mechanism is independent of the value of unobserved data ( Figure 1 ( b ) ) . Under this assumption , maximum likelihood learning methods without explicit modeling of the missingness mechanism can be applied by marginalizing out the missing variables [ 3 , 25 , 28 ] . However , both MCAR and MAR do not hold in many real-world applications , such as recommender systems [ 8 , 14 ] , healthcare [ 13 ] , and surveys [ 51 ] . For example , in a survey , participants with financial difficulties are more likely to refuse to complete the survey about financial incomes . This is an example of missing not at random ( MNAR ) , where the cause of the missingness ( financial income ) can be unobserved . In this case , ignoring the missingness mechanism will result in biased imputation , which will jeopardize down-stream tasks . ∗This work was performed when the authors were ( part-time ) associated with Microsoft Research , Cambridge 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . There are few works considering the MNAR setting in scalable missing value imputation . On the one hand , many practical methods for MNAR does not have identifiability guarantees [ 12 , 8 , 24 ] . That is , the parameters can not be uniquely determined , even with access to infinite samples [ 33 , 43 ] . As a result , missing value imputation based on such parameter estimation could be biased . On the other hand , there are theoretical analyses on the identifiability in certain scenarios [ 33 , 34 , 35 , 38 , 53 , 55 , 57 ] , but without associated practical algorithms for flexible and scalable settings ( such as deep generative models ) . Moreover , MNAR data have many possible cases ( Figure 1 ) based on different independence assumptions [ 38 ] , making the discussion of identifiability difficult . This motivates us to fill this gap by extending identifiability results of deep generative models to different missing mechanisms , and provide a scalable practical solution.Our contribution are threefold : • We provide a theoretical analysis of identifiability for generative models under different MNAR scenarios ( Section 3 ) . More specifically , we provide sufficient conditions , under which the ground truth parameters can be uniquely identified via maximum likelihood ( ML ) learning using observed information [ 24 ] . We also demonstrate how the assumptions can be relaxed in the face of real-world datasets . This provides foundation for practical solutions using deep generative models . • Based on our analysis , we propose a practical algorithm model based on VAEs ( Section 4 ) , named GINA ( deep generative imputation model for missing not at random ) . This enables us to apply flexible deep generative models in a principled way , even in the presence of MNAR data . • We demonstrate the effectiveness and validity of our approach by experimental evaluations ( Section 6 ) on both synthetic data modeling , missing data imputation in real-world datasets , as well as downstream tasks such as active feature selection under missing data . 2 Backgrounds . 2.1 Problem Setting . A critical component to develop model to impute MNAR data is the model identifiablity [ 18 , 43 ] . We give the definition below : Definition 2.1 ( Model identifiability ) . Assume pθ ( X ) is a distribution of some random variable X , θ is its parameter that takes values in some parameter space Ωθ . Then , if pθ ( X ) satisfies pθ1 ( X ) ̸= pθ2 ( X ) ⇐⇒ θ1 ̸= θ2 , ∀θ1 , θ2 ∈ Ωθ , we say that pθ is identifiable w.r.t . θ on Ωθ . In other words , a model pθ ( X ) is identifiable , if different parameter configurations implies a different probabilistic distributions over the observed variables . With identifiability guarantee , if the model assumption is correct , the true generation process can be recovered . Next , we introduce necessary notations and of missing data , and set up a concrete problem setting . Basic Notation . Similar to the notations introduced by [ 12 , 44 ] , let X be the complete set of variables in the system of interest . We call it observable variables . Let I = { 1 , ... , D } be the index set of all observable variables , i.e. , X = { Xi|i ∈ I } . Let Xo denote the set of actually observed variables , here O ∈ I is a index set such that Xo ⊂ X . We call O the observable pattern . Similarly , Xu denotes the set of missing/unobserved variables , and X = Xo ⋃ Xu . Additionally , we use R to denote the missing mask indicator variable , such that Ri = 1 indicates Xi is observed , and Ri = 0 indicates otherwise . We call a probabilistic distribution p ( X ) on X the reference distribution , that is , the distribution that we would have observed if no missing mechanism is present ; and we call the conditional distribution p ( R|X ) the missing mechanism , which decides the probability of each Xi being missing . Then , we can define the marginal distribution of partially observed variables , which is given by log p ( Xo , R ) = log ∫ Xu p ( Xo , Xu , R ) dXu . Finally , we will use lowercase letters to denote the realized values of the corresponding random variable . For example , ( xo , r ) ∼ p ( Xo , R ) is the realization/samples of Xo and R , and the dimensionality of xo may vary for each realizations . Problem setting . Suppose that we have a ground truth data generating process , denoted by pD ( Xo , R ) , from which we can obtain ( partially observed ) samples ( xo , r ) ∼ pD ( Xo , R ) . We also have a model to be optimized , denoted by p ( θ , λ ) ( Xo , Xu , R ) , where θ is the parameter of reference distribution pθ ( X ) , and λ the parameter of missing mechanism pλ ( R|X ) . Our goal can then be described as follows : • To establish the identifiability of the model p ( θ , λ ) ( Xo , R ) . That is , we wish to uniquely and correctly identify θ̂ , such that pθ̂ ( X ) = pD ( X ) , given infinite amount of partially observed data samples from ground truth , ( xo , r ) ∼ pD ( Xo , R ) . • Then , given the identified parameter , we will be able to perform missing data imputation , using pθ̂ ( Xu|Xo ) . If our parameter estimate is unbiased , then our imputation is also unbiased , that is , pθ̂ ( Xu|Xo ) = pD ( Xu|Xo ) for all possible configurations of Xo . 2.2 Challenges in MNAR imputation . Recall the three types of missing mechanisms : if data is MCAR , p ( R|X ) = p ( R ) ; if if data is MAR , p ( R|X ) = p ( R|Xo ) ; otherwise , we call it MNAR . When missing data is MCAR or MAR , missing mechanism can be ignored when performing maximum likelihood ( ML ) inference based only on the observed data [ 44 ] , as : argmax θ E ( xo , r ) ∼pD ( Xo , R ) log pθ ( Xo = xo ) = argmax θ E ( xo , r ) ∼pD ( Xo , R ) log pθ ( Xo = xo , R = r ) where log p ( Xo ) = log ∫ Xu p ( Xo , Xu ) dXu.In practice , ML learning on Xo can done by EM algorithm [ 3 , 24 ] . However , when missing data is MNAR , the above argument does not hold , and the missing data mechanism can not be ignored during learning . Consider the representative graphical model example in Figure 1 ( d ) , which has appeared in many context of machine learning . In this graphical model , X is the cause of R , and the connections between X and R are fully connected , i.e. , each single node in R are caused by the entire set X . All nodes in R are conditionally independent from each other given X . Clearly , this is an example of a data generating process with MNAR mechanism . In this case , Rubin proposed to jointly optimize both the reference distribution pθ ( X ) and the missing data mechanism pλ ( R|X ) , by maximizing : argmax θ , λ E ( xo , r ) ∼pD ( Xo , R ) [ log pθ ( Xo = xo ) + log pλ ( R = r|Xo = xo ) ] ( 1 ) This factorization is referred as selection modeling [ 12 , 24 ] . There are multiple challenges if we want to Eq . 1 to obtain a practical model that provide unbiased imputation . First , we need model assumption to be consistent with the real-world data generation process , pD ( Xo , R ) . Given a wide range of possible MNAR scenarios , it is a challenge to design a general model . Secondly , the model need to be identifiable to enable the possibility to learn the underlying process which leads to unbiased imputation . 2.3 Variational Autoencoders and its identifiability . Variational auto-encdoers [ 17 , 41 , 64 ] is a flexible deep generative model that is commonly used for estimating densities of pD ( X ) . It takes the following form : log pθ ( X ) = log ∫ Z dZpθ ( X|Z ) p ( Z ) , ( 2 ) where Z is some latent variable model with prior p ( Z ) , and pθ ( X|Z ) is given by pθ ( X|Z ) = N ( fθ ( Z ) , σ ) , with fθ ( · ) being a neural network parameterized by θ . Generally , VAEs do not have identifiability guarantees w.r.t . θ [ 16 ] . Nevertheless , inspired by the identifiablity of nonlinear ICA , [ 16 ] shows that the identifiability of VAE can be established up to equivalence permutation under mild assumptions , if the unconditional prior p ( Z ) of VAE is replaced by the following the conditionally factorial exponentially family prior , pT , ζ ( Z|U ) ∝ ∏ i=1 Q ( Zi ) exp [ K∑ j=1 Ti , j ( Zi ) ζi , j ( U ) ] , ( 3 ) where U is some additional observations ( called auxiliary variables ) , Q ( Zi ) is some base measure , Ti ( U ) = ( Ti,1 , ... , Ti , K ) the sufficient statistics , and ζi ( U ) = ( ζi,1 , ... , ζi , K ) the corresponding natural parameters . Then , the new VAE model given by log pθ ( X|U ) = log ∫ Z dZpθ ( X|Z ) pT , ζ ( Z|U ) ( 4 ) is identifiable ( Theorem 1 and 2 of [ 16 ] , see Appendix G ) .We call the model ( 4 ) the identifiable VAE . Unfortunately , this identifiability results for VAE only hold when all variables are fully observed ; thus , it can not be immediately applied to address the challenges of dealing with MNAR data stated in Section 2.2 . Next , we will analyze the identifiablity of generative models under general MNAR settings ( Section 3 ) , and propose a practical method that can be used in MNAR ( Section 4 ) .

Summary: This paper studies identifiability of generative models for Missing Not at Random (MNAR) data. MNAR is the most general missing mechanism, where the the missingness can depend on both observed and unobserved data. Most prior imputation techniques work under MCAR or MAR assumptions, and there are few models to handle missing data under MNAR. In this respect, model identifiably under MNAR assumption has been mostly overlooked. In fact, the identifiability of models is crucial for learning from MNAR data to uniquely determine the model parameters and have unbiased imputations. This paper provides the sufficient conditions and assumptions to satisfy identifiability of generative models under MNAR. In addition, it proposes a practical deep generative model (GINA) based on VAEs which can satisfy the identifiability conditions.  The contributions are as follows: * Theoretical analysis of model identifiably under MNAR, which gives sufficient conditions to have identifiable models (uniquely identify the model parameter, matching the ground-truth data generation process).  * A practical algorithm to satisfy these conditions and learn deep generative models under MNAR.  * Experimental studies showing effectiveness of the proposed model on both synthetic and real datasets.