Variational Inference for Discriminative Learning with Generative Modeling of Feature Incompletion

1 INTRODUCTION . We address the problem of prediction with incomplete data , which often arises in real-life data due to the lack of data collecting resources and/or privacy concerns . For example , consider analyzing electronic health records . The input variables of the prediction are patients ’ demographic characteristics and history of medical measurement data obtained with various instruments and the target variable is the survival time of patients . Thus , some of these features are not available depending on patients due to , e.g. , non-standardized medical equipment , legal regulation and privacy concerns . More formally , each input is an incomplete set of features represented with a tuple u = ( x̃ , m ) such that x̃ ∈ Rdx is the corrupted version of the complete input x ∈ Rdx and m ∈ { 0 , 1 } dx indicates the missing entries of x̃ : For all j ∈ [ dx ] , mj = 1 if and only if the j-th feature is missing . In particular , missing entries are filled with zero and non-missing entries are identical to the one in the complete input , x̃j = ( 1−mj ) xj . Given the incomplete features u , we want to predict the outcome y ∈ Rdy , which is modeled with a predictive distribution y ∼ p ( y|u , θ ) . Here , the parameter θ is unknown and should be learned from data . One straightforward way to deal with such incompletion is to incorporate generative models : The missing values in x̃ are imputed with some generative model and the target y is predicted based on the result of the imputation . Sometimes the imputation is done implicitly or simultaneously with the prediction . An advantage of this method lies in that the meaning of m is inherently incorporated . Without such context , algorithms must learn by themselves that m is actually indicating missingness ( if that is relevant to the prediction task ) , which incurs extra generalization errors . In other words , appropriate generative models make the prediction robust to the feature incompletion . On the other hand , the discriminative models that directly learn the conditional density pθ ( y|u ) sometimes exhibit superior performance ( Kuhn et al. , 2013 ) due to their alignment in terms of the objective . This is especially the case when the feature incompletion is not a dominant factor ( Lasserre et al. , 2006 ) . However , it is not as straightforward as generative models to incorporate the context of missingness into discriminative models and therefore their prediction may be less robust to the feature incompletion . To best of our knowledge , there have been no general solution to this dilemma of the generative and the discriminative approaches , i.e. , generic method of discriminative inference with incomplete features for generative models , which we abbreviate as DIG . In particular , although there have been similar efforts made in the literature , the previous methods are limited in terms of the applicable models . For example , several DIG methods have been proposed for exponential families ( Ghahramani & Jordan , 1994 ; Smola et al. , 2005 ) and Gaussian processes ( Pacheco et al. , 2014 ) , but they can not handle modern machine-learning architectures such as deep neural networks . This motivates us to seek for general and widely applicable frameworks that solve the dilemma . To this end , we propose a new variational approximation and a new variance reduction technique , which enable us to perform DIG within the black-box variational inference ( VI ) framework ( Jordan et al. , 1999 ; Zhang et al. , 2018 ) . We have also empirically confirmed the effectiveness of the proposed method through numerical experiments , employing the variational autoencoders ( VAEs ) ( Kingma & Welling , 2013 ) as the base generative model . The rest of the paper is organized as follows . In Section 2 , we introduce the mathematical notation and the technical challenges in DIG with flexible models . Then , in Section 3 , we present our method adopting VI in the DIG setting . In Section 4 , we show the experimental results that confirm the effectiveness of the proposed method . In Section 5 , we review and discuss the related work . Finally , in Section 6 , we conclude the paper with some remarks and future perspectives . All the proofs for theoretical statements are given in the appendix . 2 PRELIMINARY . In this section , we introduce the mathematical formulation of the DIG method and then review some background of its technical challenges . 2.1 DIG FORMULATION . Let i ∈ [ n ] denote the index of instances , where n is the number of data and [ n ] denotes the set of integers { 1 , 2 , . . . , n } . Let y [ n ] : = { yi ∈ Rdy } i∈ [ n ] and x [ n ] : = { xi ∈ Rdx } i∈ [ n ] , be the sets of the target and the complete feature vectors , where dy and dx are the dimensionalities of vectors . We assume we can only observe the set of incomplete feature vectors u [ n ] : = { ui : = ( x̃i , mi ) } i∈ [ n ] instead of x [ n ] , where x̃i ∈ Rdx is the corrupted version of xi and mi ∈ { 0 , 1 } dx is the indicator of corruption ( ‘ mask vector ’ ) for all i ∈ [ n ] . Each mask vector mi indicates which entry of x̃i is corrupted ; We have x̃ij = ( 1−mij ) xij for all i ∈ [ n ] and all j ∈ [ dx ] . Generative model . Suppose that the generative process of the observables ( y [ n ] , u [ n ] ) is modeled with a parametrized family of instance-wise identical distributions . That is , the joint probability density of ( y [ n ] , u [ n ] ) is factored as p ( y [ n ] , u [ n ] |θ ) = ∏ i∈ [ n ] p ( yi , ui|θ ) , ( 1 ) where the function p ( yi , ui|θ ) on the RHS denotes the common density function across instances i ∈ [ n ] indexed with any parametric models θ ( e.g. , by neural networks ) . Discriminative objective . The goal of DIG is to maximize the conditional likelihood of the corrupted data , L [ n ] y|u ( θ ) : = ln p ( y [ n ] |u [ n ] , θ ) = ∑ i∈ [ n ] ln p ( yi|ui , θ ) . ( 2 ) Now , let Liy|u ( θ ) : = ln p ( y i|ui , θ ) denote the i-th summand . Following Bayes ’ rule , each Liy|u ( θ ) is computed with a difference of log likelihoods Liy|u ( θ ) = L i y , u ( θ ) − Liu ( θ ) , ( 3 ) whereLiy , u ( θ ) : = ln p ( yi , ui|θ ) andLiu ( θ ) : = ln ∫ p ( yi , ui|θ ) dyi . Since we employ gradient-based optimization methods , it suffices to consider ( approximate ) evaluation of the value and the gradient of the instance-wise objectives Liy|u ( θ ) separately . Thus , in the following , we limit our focus to the separate objective and omit the index i if there is no risk of ambiguity . 2.2 DIG WITH LATENT VARIABLE MODELS ( DIG-LVM ) . We give further details of DIG with a specific class of generative models , namely the latent variable models ( LVMs ) . LVM is one of the most common and expressive generative models . In the context of DIG , the generative density function is given by p ( y , u|θ ) = ∫ p ( y , u , z|θ ) dz , ( 4 ) where z is the instance-wise latent variable , which is expected to capture some higher-order information of the observables . For example , the variational autoencoders ( VAEs ) ( Kingma & Welling , 2013 ) , the generative adversarial networks ( GANs ) ( Goodfellow et al. , 2014 ) and the normalizing flow models ( Rezende & Mohamed , 2015 ) are instances of LVM . Modeling feature-corruption process with LVM . One advantage of LVM is that complex generative processes can be modeled with relatively simple joint density p ( y , u , z|θ ) . This is especially beneficial in modeling missing data : The full joint density p ( y , u , z|θ ) can be easily designed to reflect one ’ s belief on the corruption process such as MCAR and MNAR ( Rubin , 1976 ) through the factorization of the density ( e.g. , see Collier et al . ( 2020 ) ) , while maintaining the expressibility of LVM . For example , the MCAR process can be modeled with p ( y , u , z|θ ) = p ( m ) · p ( z ) · p ( y|z , θ ) · dx∏ j=1 { p ( x̃j |z , θ ) } 1−mj , ( 5 ) where p ( m ) is the true marginal density function of m , which is unknown but cancels out in the DIG objective ( Equation 3 ) , and p ( z ) is an arbitrary fixed prior of the latent variable ( such as Gaussian ) . On the other hand , the MNAR process can be modeled with p ( y , u , z|θ ) = p ( m ) · p ( z ) · p ( y|z , m , θ ) · dx∏ j=1 { p ( x̃j |z , m , θ ) } 1−mj . ( 6 ) Note that these are just examples and it is possible to further incorporate domain knowledge on the generative process . An example of such models is the MAR process ( Rubin , 1976 ) and its formulation is relegated to the appendix ( Section G ) . Under these factorizations , the log likelihoods on the RHS of Equation 3 for LVMs are computed as Ly , u ( θ ) = ln ∫ p ( y , u , z|θ ) dz , Lu ( θ ) = ln ∫ p ( u , z|θ ) dz , ( 7 ) where both density functions in the integrals can be analytically computed under either of Equation 5 and Equation 6 . Challenge : Inference with DIG-LVM . To maximize Equation 3 with LVM , we have to evaluate the values and gradients of the likelihoods in Equation 7 . If there are only positive log integrals in the objective , this can be approximately done with the variational inference ( VI ) framework ( Jordan et al. , 1999 ; Zhang et al. , 2018 ) in favor of flexibility and scalability . However , with DIG , we also have a negative log integral , i.e. , −Lu ( θ ) . The optimization of such objective is relatively difficult since there have been no equivalent of VI in terms of the flexibility and scalability ( see Section 5 ) . 3 VARIATIONAL INFERENCE FOR DIG-LVM . Now we present the main result , the method of VI for DIG-LVM . In Section 3.1 , we derive a variational approximation to the conditional likelihood given by Equation 3 . Then , in Section 3.2 and 3.3 , we derive a method for optimizing the variational approximation and its variant for variance reduction , respectively . Finally , in Section 3.4 , we show the prediction procedure based on the optimized parameters .

Summary: This paper proposes a new method for learning with missing data. Compared with previous approaches, the authors choose to perform discriminative learning with generative modeling so as to borrow the benefits from these two types of methods. To optimize with the underlying intractable loss function, the authors start from the traditional variational lower bound ELBO and one upper bound CUBO from a previous work (the $\chi$-divergence lower bound [1]) and derives a lower bound for the original loss function. To solve the issue with the estimation bias as well as the potential huge variance, the authors change the divergence function in CUBO as well as add the surrogate parameterization so that the Monte Carlo estimation of the loss can be unbiased and (potentially) with smaller variances. Experiment results show the proposed methods run stably and perform comparably or better compared to baseline methods.  References:  [1] Adji Bousso Dieng, Dustin Tran, Rajesh Ranganath, John Paisley, and David Blei. Variational inference via χ upper bound minimization. In Advances in Neural Information Processing Systems, pp. 2732–2741. 2017.