Identifiable Generative models for Missing Not at Random Data Imputation

1 Introduction . Missing data is an obstacle in many data analysis problems , which may seriously compromise the performance of machine learning models , as well as downstream tasks based on these models . Being able to successfully recover/impute missing data in an unbiased way is the key to understanding the structure of real-world data . This requires us to identify the underlying data-generating process , as well as the probabilistic mechanism that decides which data is missing . In general , there are three types of missing mechanisms [ 44 ] . The first type is missing completely at random ( MCAR ) , where the probability of a data entry being missing is independent of both the observed and unobserved data ( Figure 1 ( a ) ) . In this case , no statistical bias is introduced by MCAR . The second type is missing at random ( MAR ) , which assumes that the missing data mechanism is independent of the value of unobserved data ( Figure 1 ( b ) ) . Under this assumption , maximum likelihood learning methods without explicit modeling of the missingness mechanism can be applied by marginalizing out the missing variables [ 3 , 25 , 28 ] . However , both MCAR and MAR do not hold in many real-world applications , such as recommender systems [ 8 , 14 ] , healthcare [ 13 ] , and surveys [ 51 ] . For example , in a survey , participants with financial difficulties are more likely to refuse to complete the survey about financial incomes . This is an example of missing not at random ( MNAR ) , where the cause of the missingness ( financial income ) can be unobserved . In this case , ignoring the missingness mechanism will result in biased imputation , which will jeopardize down-stream tasks . ∗This work was performed when the authors were ( part-time ) associated with Microsoft Research , Cambridge 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . There are few works considering the MNAR setting in scalable missing value imputation . On the one hand , many practical methods for MNAR does not have identifiability guarantees [ 12 , 8 , 24 ] . That is , the parameters can not be uniquely determined , even with access to infinite samples [ 33 , 43 ] . As a result , missing value imputation based on such parameter estimation could be biased . On the other hand , there are theoretical analyses on the identifiability in certain scenarios [ 33 , 34 , 35 , 38 , 53 , 55 , 57 ] , but without associated practical algorithms for flexible and scalable settings ( such as deep generative models ) . Moreover , MNAR data have many possible cases ( Figure 1 ) based on different independence assumptions [ 38 ] , making the discussion of identifiability difficult . This motivates us to fill this gap by extending identifiability results of deep generative models to different missing mechanisms , and provide a scalable practical solution.Our contribution are threefold : • We provide a theoretical analysis of identifiability for generative models under different MNAR scenarios ( Section 3 ) . More specifically , we provide sufficient conditions , under which the ground truth parameters can be uniquely identified via maximum likelihood ( ML ) learning using observed information [ 24 ] . We also demonstrate how the assumptions can be relaxed in the face of real-world datasets . This provides foundation for practical solutions using deep generative models . • Based on our analysis , we propose a practical algorithm model based on VAEs ( Section 4 ) , named GINA ( deep generative imputation model for missing not at random ) . This enables us to apply flexible deep generative models in a principled way , even in the presence of MNAR data . • We demonstrate the effectiveness and validity of our approach by experimental evaluations ( Section 6 ) on both synthetic data modeling , missing data imputation in real-world datasets , as well as downstream tasks such as active feature selection under missing data . 2 Backgrounds . 2.1 Problem Setting . A critical component to develop model to impute MNAR data is the model identifiablity [ 18 , 43 ] . We give the definition below : Definition 2.1 ( Model identifiability ) . Assume pθ ( X ) is a distribution of some random variable X , θ is its parameter that takes values in some parameter space Ωθ . Then , if pθ ( X ) satisfies pθ1 ( X ) ̸= pθ2 ( X ) ⇐⇒ θ1 ̸= θ2 , ∀θ1 , θ2 ∈ Ωθ , we say that pθ is identifiable w.r.t . θ on Ωθ . In other words , a model pθ ( X ) is identifiable , if different parameter configurations implies a different probabilistic distributions over the observed variables . With identifiability guarantee , if the model assumption is correct , the true generation process can be recovered . Next , we introduce necessary notations and of missing data , and set up a concrete problem setting . Basic Notation . Similar to the notations introduced by [ 12 , 44 ] , let X be the complete set of variables in the system of interest . We call it observable variables . Let I = { 1 , ... , D } be the index set of all observable variables , i.e. , X = { Xi|i ∈ I } . Let Xo denote the set of actually observed variables , here O ∈ I is a index set such that Xo ⊂ X . We call O the observable pattern . Similarly , Xu denotes the set of missing/unobserved variables , and X = Xo ⋃ Xu . Additionally , we use R to denote the missing mask indicator variable , such that Ri = 1 indicates Xi is observed , and Ri = 0 indicates otherwise . We call a probabilistic distribution p ( X ) on X the reference distribution , that is , the distribution that we would have observed if no missing mechanism is present ; and we call the conditional distribution p ( R|X ) the missing mechanism , which decides the probability of each Xi being missing . Then , we can define the marginal distribution of partially observed variables , which is given by log p ( Xo , R ) = log ∫ Xu p ( Xo , Xu , R ) dXu . Finally , we will use lowercase letters to denote the realized values of the corresponding random variable . For example , ( xo , r ) ∼ p ( Xo , R ) is the realization/samples of Xo and R , and the dimensionality of xo may vary for each realizations . Problem setting . Suppose that we have a ground truth data generating process , denoted by pD ( Xo , R ) , from which we can obtain ( partially observed ) samples ( xo , r ) ∼ pD ( Xo , R ) . We also have a model to be optimized , denoted by p ( θ , λ ) ( Xo , Xu , R ) , where θ is the parameter of reference distribution pθ ( X ) , and λ the parameter of missing mechanism pλ ( R|X ) . Our goal can then be described as follows : • To establish the identifiability of the model p ( θ , λ ) ( Xo , R ) . That is , we wish to uniquely and correctly identify θ̂ , such that pθ̂ ( X ) = pD ( X ) , given infinite amount of partially observed data samples from ground truth , ( xo , r ) ∼ pD ( Xo , R ) . • Then , given the identified parameter , we will be able to perform missing data imputation , using pθ̂ ( Xu|Xo ) . If our parameter estimate is unbiased , then our imputation is also unbiased , that is , pθ̂ ( Xu|Xo ) = pD ( Xu|Xo ) for all possible configurations of Xo . 2.2 Challenges in MNAR imputation . Recall the three types of missing mechanisms : if data is MCAR , p ( R|X ) = p ( R ) ; if if data is MAR , p ( R|X ) = p ( R|Xo ) ; otherwise , we call it MNAR . When missing data is MCAR or MAR , missing mechanism can be ignored when performing maximum likelihood ( ML ) inference based only on the observed data [ 44 ] , as : argmax θ E ( xo , r ) ∼pD ( Xo , R ) log pθ ( Xo = xo ) = argmax θ E ( xo , r ) ∼pD ( Xo , R ) log pθ ( Xo = xo , R = r ) where log p ( Xo ) = log ∫ Xu p ( Xo , Xu ) dXu.In practice , ML learning on Xo can done by EM algorithm [ 3 , 24 ] . However , when missing data is MNAR , the above argument does not hold , and the missing data mechanism can not be ignored during learning . Consider the representative graphical model example in Figure 1 ( d ) , which has appeared in many context of machine learning . In this graphical model , X is the cause of R , and the connections between X and R are fully connected , i.e. , each single node in R are caused by the entire set X . All nodes in R are conditionally independent from each other given X . Clearly , this is an example of a data generating process with MNAR mechanism . In this case , Rubin proposed to jointly optimize both the reference distribution pθ ( X ) and the missing data mechanism pλ ( R|X ) , by maximizing : argmax θ , λ E ( xo , r ) ∼pD ( Xo , R ) [ log pθ ( Xo = xo ) + log pλ ( R = r|Xo = xo ) ] ( 1 ) This factorization is referred as selection modeling [ 12 , 24 ] . There are multiple challenges if we want to Eq . 1 to obtain a practical model that provide unbiased imputation . First , we need model assumption to be consistent with the real-world data generation process , pD ( Xo , R ) . Given a wide range of possible MNAR scenarios , it is a challenge to design a general model . Secondly , the model need to be identifiable to enable the possibility to learn the underlying process which leads to unbiased imputation . 2.3 Variational Autoencoders and its identifiability . Variational auto-encdoers [ 17 , 41 , 64 ] is a flexible deep generative model that is commonly used for estimating densities of pD ( X ) . It takes the following form : log pθ ( X ) = log ∫ Z dZpθ ( X|Z ) p ( Z ) , ( 2 ) where Z is some latent variable model with prior p ( Z ) , and pθ ( X|Z ) is given by pθ ( X|Z ) = N ( fθ ( Z ) , σ ) , with fθ ( · ) being a neural network parameterized by θ . Generally , VAEs do not have identifiability guarantees w.r.t . θ [ 16 ] . Nevertheless , inspired by the identifiablity of nonlinear ICA , [ 16 ] shows that the identifiability of VAE can be established up to equivalence permutation under mild assumptions , if the unconditional prior p ( Z ) of VAE is replaced by the following the conditionally factorial exponentially family prior , pT , ζ ( Z|U ) ∝ ∏ i=1 Q ( Zi ) exp [ K∑ j=1 Ti , j ( Zi ) ζi , j ( U ) ] , ( 3 ) where U is some additional observations ( called auxiliary variables ) , Q ( Zi ) is some base measure , Ti ( U ) = ( Ti,1 , ... , Ti , K ) the sufficient statistics , and ζi ( U ) = ( ζi,1 , ... , ζi , K ) the corresponding natural parameters . Then , the new VAE model given by log pθ ( X|U ) = log ∫ Z dZpθ ( X|Z ) pT , ζ ( Z|U ) ( 4 ) is identifiable ( Theorem 1 and 2 of [ 16 ] , see Appendix G ) .We call the model ( 4 ) the identifiable VAE . Unfortunately , this identifiability results for VAE only hold when all variables are fully observed ; thus , it can not be immediately applied to address the challenges of dealing with MNAR data stated in Section 2.2 . Next , we will analyze the identifiablity of generative models under general MNAR settings ( Section 3 ) , and propose a practical method that can be used in MNAR ( Section 4 ) .

Summary: The article is concerned with model identifiability in Missing Not At Random (MNAR) data setting, with a particular focus on deep generative models. The authors introduce a unified view of model identifiability under MNAR, and derive sufficient conditions for identifiability in several MNAR scenarios, which highlight new important concepts. They also introduce a new imputation method for MNAR data based on identifiable VAE models, which is submitted as a python code, and extensively tested on synthetic and real data sets against state of the art methods for MNAR imputation.