How to deal with missing data in supervised deep learning?

1 INTRODUCTION Missing data affect data analysis across a wide range of domains and the sources of missing values span an equally wide range . Recently , deep latent variable models ( DLVMs , Kingma & Welling , 2014 ; Rezende et al. , 2014 ) have been applied to missing data problems in an unsupervised setting ( e.g . Rezende et al. , 2014 ; Nazabal et al. , 2020 ; Ma et al. , 2018 ; 2019 ; Ivanov et al. , 2019 ; Mattei & Frellsen , 2018 ; 2019 ; Yoon et al. , 2018 ; Li et al. , 2019 ; Ipsen et al. , 2021 ; Ghalebikesabi et al. , 2021 ) , while the supervised setting has not seen the same attention . The progress in the unsupervised setting is focused on inference and imputation and can therefore be useful as an imputation step before learning a discriminative model . Traditionally the focus has also been on inference and imputation either as a goal in itself or before supervised learning on the ( possibly multiple ) imputations and observed data ( Rubin , 1976 ; 1996 ; Little & Rubin , 2019 ) . Supervised learning in the presence of missing values has different goals and pose different challenges than inference and imputation ( Josse et al. , 2019 ; Le Morvan et al. , 2020a ; 2021 ) . Here the overall aim is to minimize an expected prediction error . However , optimal single imputation does not necessarily lead to optimal prediction in terms of minimizing a prediction error ( Josse et al. , 2019 ) . One challenge is that predictions based on inputs with missing values can be ambiguous , that is , the conditional distribution of the missing values given the observed may be multimodal and the optimal prediction may change with the mode . With single imputation , the conditional distribution over the missing data is discarded , and the optimal single imputation can no longer reflect this ambiguity , see figure 2 . Even the optimal single imputation leads to biased parameter estimates compared to the fully observed case ( Bertsimas et al. , 2021 ) , and may in some cases lie outside the distribution of the data . Another challenge is that the number of missing value patterns grows combinatorially with the number of features p , requiring 2p submodels to fit the Bayes predictor in the linear case ( Le Morvan et al. , 2020a ) . In multiple imputation ( Rubin , 1996 ) , several imputations are drawn from the posterior predictive distribution of the missing values , reflecting the full conditional distribution of the missing values and thus the uncertainty about what is missing . This allows for uncertainty estimates in downstream tasks such as prediction . This in turn requires fitting as many discriminative models as the number of imputations . 1.1 CONTRIBUTIONS . In order to address supervised deep learning with missing values we develop the supervised missing data importance-weighted autoencoder ( supMIWAE ) bound , based on the approximate maximum likehood techniques used by Burda et al . ( 2016 ) and Mattei & Frellsen ( 2019 ) . This is a scalable approach to marginalizing over missing values in a joint model of covariates and outcomes pφ , θ ( y , x obs , xmiss ) = pφ ( y|xobs , xmiss ) pθ ( xobs , xmiss ) . ( 1 ) The covariate model pθ ( xobs , xmiss ) = ∫ p ( xobs , xmiss|z ) p ( z ) dz is a DLVM in this work , but can be any probabilistic model imposing a joint density over covariates . The outcome model pφ ( y|xobs , xmiss ) is any neural discriminative architecture that would have been used in the complete-data case , parameterizing a density over the outcomes . The graphical model is shown in figure 1 along with its computational structure . Once the joint model has been trained using the supMIWAE bound , the conditional distribution of the outcome given the observed parts of the input can be found as pφ ( y|xobs ) = ∫ pφ ( y|xobs , xmiss ) pθ ( xmiss|xobs ) dxmiss . ( 2 ) This is approximated using importance sampling techniques , averaging predictions over multiple importance samples ( akin to multiple imputations , see appendix C for a deeper discussion ) from the generative model . The model can be trained end-to-end or a pretrained generative model can be coupled with a discriminative model . Having a joint model allows for supervised imputations , where available labels can help guide the imputations from the generative model by adjusting the importance weights accordingly . Joint models over covariates and labels have previously been used to marginalize over missing values in the covariates , using Gaussian mixture models either as the model over the covariates or directly as the joint model over covariates and outcomes ( Ghahramani & Jordan , 1995 ; Ghahramani & Hinton , 1996 ; Ahmad & Tresp , 1993 ; Tresp et al. , 1994 ; 1995 ; Śmieja et al. , 2018 ) . Our approach shows how to use more flexible generative models , such as DLVMs , provides an efficient optimization procedure and allows for keeping any deep discriminative architecture unchanged . 2 BACKGROUND AND NOTATION . We define the random variable x = ( x1 , . . . , xp ) ∈ X which takes values in a p-dimensional feature space X = X1 × · · · × Xp . There is a corresponding ( possibly vector valued ) response variable y ∈ Y . A missing process obscures parts of x resulting in the mask random variable s ∈ { 0 , 1 } p where sj = { 1 if xj observed , 0 if xj missing . ( 3 ) We let obs ( s ) denote the indices of the non-zero entries of s and miss ( s ) denote the indices of the zero-entries of s , such that xobs ( s ) is the set of observed elements of x , and xmiss ( s ) is the set of missing elements of x . For simplicity we will omit the s and write xobs , xmiss respectively , whenever the context is clear . Following ( Yoon et al. , 2018 ; Le Morvan et al. , 2020b ; Ghalebikesabi et al. , 2021 ) we define the incomplete random variable x̃ = ( x̃1 , . . . , x̃p ) ∈ X̃ which takes values in X̃ = ( X1∪ { na } ) ×· · ·× ( Xp ∪ { na } ) where missing values are represented by the symbol na , such that na · xj = na and na · 0 = 0 . We typically only have access to x̃ , where x̃ = x s + na ( 1− s ) ( 4 ) and is the Hadamard product . Finally , we are given n i.i.d . copies of the random variables x̃ and y which we collect in a dataset D = { x̃i , yi } ni=1 , or alternatively D = { xobsi , yi } n i=1 . We make the additional assumption that the data are missing at random ( MAR , see e.g . Seaman et al. , 2013 ; Little & Rubin , 2019 ) . Specifically , this means that we assume that s and xmiss are independent given xobs . This assumption allows to avoid to explicitly model the missing mechanism . Our approach could be extended beyond MAR by using a generative model fit for this purpose , like the not-MIWAE of Ipsen et al . ( 2021 ) , or the deep pattern-set mixture of Ghalebikesabi et al . ( 2021 ) . 2.1 CHALLENGES WHEN TRAINING DISRIMINATIVE MODELS WITH MISSING DATA . Our predictive model will be defined through a mapping fφ : X → H used to parameterize a conditional distribution pφ ( y|x ) = Ψ ( y|fφ ( x ) ) . ( 5 ) Here ( Ψ ( ·|η ) ) η∈H is a parametric family of distributions over the outcome space Y , such as a Gaussian distribution in regression or a categorical distribution in classification . Under the maximumlikelihood framework ( or equivalently the logarithmic scoring rule ) , an optimal mapping f∗φ within a class F = ( fφ ) φ∈Φ parameterized by φ ∈ Φ is defined as f∗φ ∈ arg max fφ∈F Ep∗ ( x , y ) [ log Ψ ( y|fφ ( x ) ) ] , ( 6 ) where p∗ ( x , y ) is the true data generating distribution . When the data are complete , this is typically approximated by maximizing the log likelihood of the parameters φ given the data Dtrain , ` ( φ ) = ∑ ( x , y ) ∈Dtrain log pφ ( y|x ) . ( 7 ) When the covariates have missing values the log-likelihood can not be maximized directly as the likelihood is not defined since pφ ( y|x ) depends on the full input vector . The simplest approach is instead to learn separate mappings for each missing pattern . This reduces the amount of data available for the training of each mapping and does not scale well with the dimensionality of the input as in general 2p networks need to be trained . An often used approach is instead to impute the missing values using a single imputation ( for instance , using Epθ ( xmiss|xobs ) [ xmiss ] or an approximation of it ) , and then map the concatenation of observed and imputed values to approximate the conditional distribution pφ ( y|xobs ) ≈ pφ ( y|x = ( xobs , Epθ ( xmiss|xobs ) [ x miss ] ) ) = Ψ ( y|fφ ( xobs , Epθ ( xmiss|xobs ) [ x miss ] ) ) . ( 8 ) Instead of Epθ ( xmiss|xobs ) [ xmiss ] , which is the optimal imputation under the mean squared error , one could use any kind of imputation , for instance using a constant , or the mean . This general approach is adequately called impute-then-regress by Bertsimas et al . ( 2021 ) and Le Morvan et al . ( 2021 ) . While it will be Bayes-consistent given a powerful enough classifier ( Le Morvan et al. , 2021 ) , it leads to biased parameter estimates compared to having complete data ( Bertsimas et al. , 2021 ) . This is illustrated in figure 2 where the optimal ( under the mean squared error ) single imputations are placed entirely within one of the classes , requiring a biased decision surface to properly reflect the class label probabilities for the records with missing values . From a probabilistic perspective , computing pφ ( y|xobs ) requires marginalizing over the missing features as in equation ( 1 ) , that is pφ ( y|xobs ) = Epθ ( xmiss|xobs ) [ pφ ( y|x obs , xmiss ) ] . ( 9 ) Notice the difference to single imputation since , in general Epθ ( xmiss|xobs ) [ pφ ( y|x obs , xmiss ) ] 6= pφ ( y|x = ( xobs , Epθ ( xmiss|xobs ) [ x miss ] ) ) , ( 10 ) except in some pathological cases ( e.g . when pφ is linear or pθ is a Dirac distribution ) . Therefore , if the discriminative model is nonlinear and the generative model is complex enough , then using single imputation will give a very different result than marginalising over the missing features . This is exemplified in figure 2 , where a classifier trained using the optimal imputation finds a solution that is very different to what would be found by one trained without missing data . Our technique , marginalizing the missing values , allows to recover the same decision surface as a classifier trained without missing data .

Summary: This paper approaches the problem of supervised learning with missing data. The authors propose a probabilistic approach by jointly modeling the observed data, missing data and outcomes. The main contribution of this model is that they rely on deep generative models which seems to be an improvement from previous methods based on simpler generative models like mixture of Gaussians, for example. With the proposed model, the authors derive an optimization problem that consists in optimizing the discriminative model (classifier) and the generative model simultaneously. For the training and testing phases, the proposed algorithm considers multiple imputations of missing data, which is demonstrated to be superior to single imputation methods. Experimental results on small and simple datasets (2D datasets, MNIST digits-Fashion and regression) are nicely presented and analyzed.