not-MIWAE: Deep Generative Modelling with Missing not at Random Data

1 INTRODUCTION z x s θ φ γ N ( a ) PPCA not-MIWAE PPCA ( b ) Figure 1 : ( a ) Graphical model of the not-MIWAE . ( b ) Gaussian data with MNAR values . Dots are fully observed , partially observed data are displayed as black crosses . A contour of the true distribution is shown together with directions found by PPCA and not-MIWAE with a PPCA decoder . Missing data often constitute systemic issues in real-world data analysis , and can be an integral part of some fields , e.g . recommender systems . This requires the analyst to take action by either using methods and models that are applicable to incomplete data or by performing imputations of the missing data before applying models requiring complete data . The expected model performance ( often measured in terms of imputation error or innocuity of missingness on the inference results ) depends on the assumptions made about the missing mechanism and how well those assumptions match the true missing mechanism . In a seminal paper , Rubin ( 1976 ) introduced a formal probabilistic framework to assess missing mechanism assumptions and their consequences . The most commonly used assumption , either implicitly or explicitly , is that a part of the data is missing at random ( MAR ) . Essentially , the MAR assumption means that the missing pattern does not depend on the missing values . This makes it possible to ignore the missing data mechanism in likelihood-based inference by marginalizing over the missing data . The often implicit assumption made in nonprobabilistic models and ad-hoc methods is that the data are missing completely at random ( MCAR ) . MCAR is a stronger assumption than MAR , and informally it means that both observed and missing data do not depend on the missing pattern . More details on these assumptions can be found in the monograph of Little & Rubin ( 2002 ) ; of particular interest are also the recent revisits of Seaman et al . ( 2013 ) and Doretti et al . ( 2018 ) . In this paper , our goal is to posit statistical models that leverage deep learning in order to break away from these assumptions . Specifically , we propose a general ∗Department of Applied Mathematics and Computer Science , Technical University of Denmark , Denmark †Université Côte d ’ Azur , Inria ( Maasai team ) , Laboratoire J.A . Dieudonné , UMR CNRS 7351 , France ‡Equal contribution recipe for dealing with cases where there is prior information about the distribution of the missing pattern given the data ( e.g . self-censoring ) . The MAR and MCAR assumptions are violated when the missing data mechanism is dependent on the missing data themselves . This setting is called missing not at random ( MNAR ) . Here the missing mechanism can not be ignored , doing so will lead to biased parameter estimates . This setting generally requires a joint model for data and missing mechanism . Deep latent variable models ( DLVMs , Kingma & Welling , 2013 ; Rezende et al. , 2014 ) have recently been used for inference and imputation in missing data problems ( Nazabal et al. , 2020 ; Ma et al. , 2018 ; 2019 ; Ivanov et al. , 2019 ; Mattei & Frellsen , 2019 ) . This led to impressive empirical results in the MAR and MCAR case , in particular for high-dimensional data . 1.1 CONTRIBUTIONS . We introduce the not-missing-at-random importance-weighted autoencoder ( not-MIWAE ) which allows for the application of DLVMs to missing data problems where the missing mechanism is MNAR . This is inspired by the missing data importance-weighted autoencoder ( MIWAE , Mattei & Frellsen , 2019 ) , a framework to train DLVMs in MAR scenarios , based itself on the importanceweighted autoencoder ( IWAE ) of Burda et al . ( 2016 ) . The general graphical model for the notMIWAE is shown in figure 1a . The first part of the model is simply a latent variable model : there is a stochastic mapping parameterized by θ from a latent variable z ∼ p ( z ) to the data x ∼ pθ ( x|z ) , and the data may be partially observed . The second part of the model , which we call the missing model , is a stochastic mapping from the data to the missing mask s ∼ pφ ( s|x ) . Explicit specification of the missing model pφ ( s|x ) makes it possible to address MNAR issues . The model can be trained efficiently by maximising a lower bound of the joint likelihood ( of the observed features and missing pattern ) obtained via importance weighted variational inference ( Burda et al. , 2016 ) . A key difference with the MIWAE is that we use the reparameterization trick in the data space , as well as in the code space , in order to get stochastic gradients of the lower bound . Missing processes affect data analysis in a wide range of domains and often the MAR assumption does not hold . We apply our method to censoring in datasets from the UCI database , clipping in images and the issue of selection bias in recommender systems . 2 BACKGROUND . Assume that the complete data are stored within a data matrix X = ( x1 , . . . , xn ) ᵀ ∈ Xn that contain n i.i.d . copies of the random variable x ∈ X , where X = X1 × · · · × Xp is a p-dimensional feature space . For simplicity , xij refers to the j ’ th feature of xi , and xi refers to the i ’ th sample in the data matrix . Throughout the text , we will make statements about the random variable x , and only consider samples xi when necessary . In a missing data context , each sample can be split into an observed part and a missing part , xi = ( xoi , x m i ) . The pattern of missingness is individual to each copy of x and described by a corresponding mask random variable s ∈ { 0 , 1 } p. This leads to a mask matrix S = ( s1 , . . . , sn ) ᵀ ∈ { 0 , 1 } n×p verifying sij = 1 if xij is observed and sij = 0 if xij is missing . We wish to construct a parametric model pθ , φ ( x , s ) for the joint distribution of a single data point x and its mask s , which can be factored as pθ , φ ( x , s ) = pθ ( x ) pφ ( s|x ) . ( 1 ) Here pφ ( s|x ) = pφ ( s|xo , xm ) is the conditional distribution of the mask , which may depend on both the observed and missing data , through its own parameters φ . The three assumptions from the framework of Little & Rubin ( 2002 ) ( see also Ghahramani & Jordan , 1995 ) pertain to the specific form of this conditional distribution : • MCAR : pφ ( s|x ) = pφ ( s ) , • MAR : pφ ( s|x ) = pφ ( s|xo ) , • MNAR : pφ ( s|x ) may depend on both xo and xm . To maximize the likelihood of the parameters ( θ , φ ) , based only on observed quantities , the missing data is integrated out from the joint distribution pθ , φ ( x o , s ) = ∫ pθ ( x o , xm ) pφ ( s|xo , xm ) dxm . ( 2 ) In both the MCAR and MAR cases , inference for θ using the full likelihood becomes proportional to pθ , φ ( xo , s ) ∝ pθ ( xo ) , and the missing mechanism can be ignored while focusing only on pθ ( xo ) . In the MNAR case , the missing mechanism can depend on both observed and missing data , offering no factorization of the likelihood in equation ( 2 ) . The parameters of the data generating process and the parameters of the missing data mechanism are tied together by the missing data . 2.1 PPCA EXAMPLE . A linear DLVM with isotropic noise variance can be used to recover a model similar to probabilistic principal component analysis ( PPCA , Roweis , 1998 ; Tipping & Bishop , 1999 ) . In figure 1b , a dataset affected by an MNAR missing process is shown together with two fitted PPCA models , regular PPCA and the not-MIWAE formulated as a PPCA-like model . Data is generated from a multivariate normal distribution and an MNAR missing process is imposed by setting the horizontal coordinate to missing when it is larger than its mean , i.e . it becomes missing because of the value it would have had , had it been observed . Regular PPCA for missing data assumes that the missing mechanism is MAR so that the missing process is ignorable . This introduces a bias , both in the estimated mean and in the estimated principal signal direction of the data . The not-MIWAE PPCA assumes the missing mechanism is MNAR so the data generating process and missing data mechanism are modelled jointly as described in equation ( 2 ) . 2.2 PREVIOUS WORK . In ( Rubin , 1976 ) the appropriateness of ignoring the missing process when doing likelihood based or Bayesian inference was introduced and formalized . The introduction of the EM algorithm ( Dempster et al. , 1977 ) made it feasible to obtain maximum likelihood estimates in many missing data settings , see e.g . Ghahramani & Jordan ( 1994 ; 1995 ) ; Little & Rubin ( 2002 ) . Sampling methods such as Markov chain Monte Carlo have made it possible to sample a target posterior in Bayesian models , including the missing data , so that parameter marginal distributions and missing data marginal distributions are available directly ( Gelman et al. , 2013 ) . This is also the starting point of the multiple imputations framework of Rubin ( 1977 ; 1996 ) . Here the samples of the missing data are used to provide several realisations of complete datasets where complete-data methods can be applied to get combined mean and variability estimates . The framework of Little & Rubin ( 2002 ) is instructive in how to handle MNAR problems and a recent review of MNAR methods can be found in ( Tang & Ju , 2018 ) . Low rank models were used for estimation and imputation in MNAR settings by Sportisse et al . ( 2020a ) . Two approaches were taken to fitting models , 1 ) maximising the joint distribution of data and missing mask using an EM algorithm , and 2 ) implicitly modelling the joint distribution by concatenating the data matrix and the missing mask and working with this new matrix . This implies a latent representation both giving rise to the data and the mask . An overview of estimation methods for PCA and PPCA with missing data was given by Ilin & Raiko ( 2010 ) , while PPCA in the presence of an MNAR missing mechanism has been addressed by Sportisse et al . ( 2020b ) . There has been some focus on MNAR issues in the form of selection bias within the recommender system community ( Marlin et al. , 2007 ; Marlin & Zemel , 2009 ; Steck , 2013 ; Hernández-Lobato et al. , 2014 ; Schnabel et al. , 2016 ; Wang et al. , 2019 ) where methods applied range from joint modelling of data and missing model using multinomial mixtures and matrix factorization to debiasing existing methods using propensity based techniques from causality . Deep latent variable models are intuitively appealing in a missing context : the generative part of the model can be used to sample the missing part of an observation . This was already utilized by Rezende et al . ( 2014 ) to do imputation and denoising by sampling from a Markov chain whose stationary distribution is approximately the conditional distribution of the missing data given the observed . This procedure has been enhanced by Mattei & Frellsen ( 2018a ) using Metropolis-withinGibbs . In both cases the experiments were assuming MAR and a fitted model , based on complete data , was already available . Approaches to fitting DLVMs in the presence of missing have recently been suggested , such as the HI-VAE by Nazabal et al . ( 2020 ) using an extension of the variational autoencoder ( VAE ) lower bound , the p-VAE by Ma et al . ( 2018 ; 2019 ) using the VAE lower bound and a permutation invariant encoder , the MIWAE by Mattei & Frellsen ( 2019 ) , extending the IWAE lower bound ( Burda et al. , 2016 ) , and GAIN ( Yoon et al. , 2018 ) using GANs for missing data imputation . All approaches are assuming that the missing process is MAR or MCAR . In ( Gong et al. , 2020 ) , the data and missing mask are modelled together , as both being generated by a mapping from the same latent space , thereby tying the data model and missing process together . This gives more flexibility in terms of missing process assumptions , akin to the matrix factorization approach by Sportisse et al . ( 2020a ) . In concurrent work , Collier et al . ( 2020 ) have developed a deep generative model of the observed data conditioned on the mask random variable , and Lim et al . ( 2021 ) apply a model similar to the not-MIWAE to electronic health records data . In forthcoming work , Ghalebikesabi et al . ( 2021 ) propose a deep generative model for non-ignorable missingness building on ideas from VAEs and pattern-set mixture models .

Summary: This paper proposes an approach to training deep latent variable models on data that is missing not at random. To learn the parameters of deep latent variable models, the paper adopts importance-weighted variational inference techniques. Experiments on a variety of datasets show that the proposed approach is effective by explicitly modeling missing not at random data.