Variational Inference for Discriminative Learning with Generative Modeling of Feature Incompletion

1 INTRODUCTION . We address the problem of prediction with incomplete data , which often arises in real-life data due to the lack of data collecting resources and/or privacy concerns . For example , consider analyzing electronic health records . The input variables of the prediction are patients ’ demographic characteristics and history of medical measurement data obtained with various instruments and the target variable is the survival time of patients . Thus , some of these features are not available depending on patients due to , e.g. , non-standardized medical equipment , legal regulation and privacy concerns . More formally , each input is an incomplete set of features represented with a tuple u = ( x̃ , m ) such that x̃ ∈ Rdx is the corrupted version of the complete input x ∈ Rdx and m ∈ { 0 , 1 } dx indicates the missing entries of x̃ : For all j ∈ [ dx ] , mj = 1 if and only if the j-th feature is missing . In particular , missing entries are filled with zero and non-missing entries are identical to the one in the complete input , x̃j = ( 1−mj ) xj . Given the incomplete features u , we want to predict the outcome y ∈ Rdy , which is modeled with a predictive distribution y ∼ p ( y|u , θ ) . Here , the parameter θ is unknown and should be learned from data . One straightforward way to deal with such incompletion is to incorporate generative models : The missing values in x̃ are imputed with some generative model and the target y is predicted based on the result of the imputation . Sometimes the imputation is done implicitly or simultaneously with the prediction . An advantage of this method lies in that the meaning of m is inherently incorporated . Without such context , algorithms must learn by themselves that m is actually indicating missingness ( if that is relevant to the prediction task ) , which incurs extra generalization errors . In other words , appropriate generative models make the prediction robust to the feature incompletion . On the other hand , the discriminative models that directly learn the conditional density pθ ( y|u ) sometimes exhibit superior performance ( Kuhn et al. , 2013 ) due to their alignment in terms of the objective . This is especially the case when the feature incompletion is not a dominant factor ( Lasserre et al. , 2006 ) . However , it is not as straightforward as generative models to incorporate the context of missingness into discriminative models and therefore their prediction may be less robust to the feature incompletion . To best of our knowledge , there have been no general solution to this dilemma of the generative and the discriminative approaches , i.e. , generic method of discriminative inference with incomplete features for generative models , which we abbreviate as DIG . In particular , although there have been similar efforts made in the literature , the previous methods are limited in terms of the applicable models . For example , several DIG methods have been proposed for exponential families ( Ghahramani & Jordan , 1994 ; Smola et al. , 2005 ) and Gaussian processes ( Pacheco et al. , 2014 ) , but they can not handle modern machine-learning architectures such as deep neural networks . This motivates us to seek for general and widely applicable frameworks that solve the dilemma . To this end , we propose a new variational approximation and a new variance reduction technique , which enable us to perform DIG within the black-box variational inference ( VI ) framework ( Jordan et al. , 1999 ; Zhang et al. , 2018 ) . We have also empirically confirmed the effectiveness of the proposed method through numerical experiments , employing the variational autoencoders ( VAEs ) ( Kingma & Welling , 2013 ) as the base generative model . The rest of the paper is organized as follows . In Section 2 , we introduce the mathematical notation and the technical challenges in DIG with flexible models . Then , in Section 3 , we present our method adopting VI in the DIG setting . In Section 4 , we show the experimental results that confirm the effectiveness of the proposed method . In Section 5 , we review and discuss the related work . Finally , in Section 6 , we conclude the paper with some remarks and future perspectives . All the proofs for theoretical statements are given in the appendix . 2 PRELIMINARY . In this section , we introduce the mathematical formulation of the DIG method and then review some background of its technical challenges . 2.1 DIG FORMULATION . Let i ∈ [ n ] denote the index of instances , where n is the number of data and [ n ] denotes the set of integers { 1 , 2 , . . . , n } . Let y [ n ] : = { yi ∈ Rdy } i∈ [ n ] and x [ n ] : = { xi ∈ Rdx } i∈ [ n ] , be the sets of the target and the complete feature vectors , where dy and dx are the dimensionalities of vectors . We assume we can only observe the set of incomplete feature vectors u [ n ] : = { ui : = ( x̃i , mi ) } i∈ [ n ] instead of x [ n ] , where x̃i ∈ Rdx is the corrupted version of xi and mi ∈ { 0 , 1 } dx is the indicator of corruption ( ‘ mask vector ’ ) for all i ∈ [ n ] . Each mask vector mi indicates which entry of x̃i is corrupted ; We have x̃ij = ( 1−mij ) xij for all i ∈ [ n ] and all j ∈ [ dx ] . Generative model . Suppose that the generative process of the observables ( y [ n ] , u [ n ] ) is modeled with a parametrized family of instance-wise identical distributions . That is , the joint probability density of ( y [ n ] , u [ n ] ) is factored as p ( y [ n ] , u [ n ] |θ ) = ∏ i∈ [ n ] p ( yi , ui|θ ) , ( 1 ) where the function p ( yi , ui|θ ) on the RHS denotes the common density function across instances i ∈ [ n ] indexed with any parametric models θ ( e.g. , by neural networks ) . Discriminative objective . The goal of DIG is to maximize the conditional likelihood of the corrupted data , L [ n ] y|u ( θ ) : = ln p ( y [ n ] |u [ n ] , θ ) = ∑ i∈ [ n ] ln p ( yi|ui , θ ) . ( 2 ) Now , let Liy|u ( θ ) : = ln p ( y i|ui , θ ) denote the i-th summand . Following Bayes ’ rule , each Liy|u ( θ ) is computed with a difference of log likelihoods Liy|u ( θ ) = L i y , u ( θ ) − Liu ( θ ) , ( 3 ) whereLiy , u ( θ ) : = ln p ( yi , ui|θ ) andLiu ( θ ) : = ln ∫ p ( yi , ui|θ ) dyi . Since we employ gradient-based optimization methods , it suffices to consider ( approximate ) evaluation of the value and the gradient of the instance-wise objectives Liy|u ( θ ) separately . Thus , in the following , we limit our focus to the separate objective and omit the index i if there is no risk of ambiguity . 2.2 DIG WITH LATENT VARIABLE MODELS ( DIG-LVM ) . We give further details of DIG with a specific class of generative models , namely the latent variable models ( LVMs ) . LVM is one of the most common and expressive generative models . In the context of DIG , the generative density function is given by p ( y , u|θ ) = ∫ p ( y , u , z|θ ) dz , ( 4 ) where z is the instance-wise latent variable , which is expected to capture some higher-order information of the observables . For example , the variational autoencoders ( VAEs ) ( Kingma & Welling , 2013 ) , the generative adversarial networks ( GANs ) ( Goodfellow et al. , 2014 ) and the normalizing flow models ( Rezende & Mohamed , 2015 ) are instances of LVM . Modeling feature-corruption process with LVM . One advantage of LVM is that complex generative processes can be modeled with relatively simple joint density p ( y , u , z|θ ) . This is especially beneficial in modeling missing data : The full joint density p ( y , u , z|θ ) can be easily designed to reflect one ’ s belief on the corruption process such as MCAR and MNAR ( Rubin , 1976 ) through the factorization of the density ( e.g. , see Collier et al . ( 2020 ) ) , while maintaining the expressibility of LVM . For example , the MCAR process can be modeled with p ( y , u , z|θ ) = p ( m ) · p ( z ) · p ( y|z , θ ) · dx∏ j=1 { p ( x̃j |z , θ ) } 1−mj , ( 5 ) where p ( m ) is the true marginal density function of m , which is unknown but cancels out in the DIG objective ( Equation 3 ) , and p ( z ) is an arbitrary fixed prior of the latent variable ( such as Gaussian ) . On the other hand , the MNAR process can be modeled with p ( y , u , z|θ ) = p ( m ) · p ( z ) · p ( y|z , m , θ ) · dx∏ j=1 { p ( x̃j |z , m , θ ) } 1−mj . ( 6 ) Note that these are just examples and it is possible to further incorporate domain knowledge on the generative process . An example of such models is the MAR process ( Rubin , 1976 ) and its formulation is relegated to the appendix ( Section G ) . Under these factorizations , the log likelihoods on the RHS of Equation 3 for LVMs are computed as Ly , u ( θ ) = ln ∫ p ( y , u , z|θ ) dz , Lu ( θ ) = ln ∫ p ( u , z|θ ) dz , ( 7 ) where both density functions in the integrals can be analytically computed under either of Equation 5 and Equation 6 . Challenge : Inference with DIG-LVM . To maximize Equation 3 with LVM , we have to evaluate the values and gradients of the likelihoods in Equation 7 . If there are only positive log integrals in the objective , this can be approximately done with the variational inference ( VI ) framework ( Jordan et al. , 1999 ; Zhang et al. , 2018 ) in favor of flexibility and scalability . However , with DIG , we also have a negative log integral , i.e. , −Lu ( θ ) . The optimization of such objective is relatively difficult since there have been no equivalent of VI in terms of the flexibility and scalability ( see Section 5 ) . 3 VARIATIONAL INFERENCE FOR DIG-LVM . Now we present the main result , the method of VI for DIG-LVM . In Section 3.1 , we derive a variational approximation to the conditional likelihood given by Equation 3 . Then , in Section 3.2 and 3.3 , we derive a method for optimizing the variational approximation and its variant for variance reduction , respectively . Finally , in Section 3.4 , we show the prediction procedure based on the optimized parameters .

Summary: The paper considers the problem of prediction with missing (incomplete) features. The authors propose a class of generative models that includes missingness of features, and develop a discriminative learning algorithm that maximises the conditional (posterior) log-likelihood of the training data approximately. Experiments show that the method is competitive compared with existing approaches, and in particular with approaches based on VAEs.