MisGAN: Learning from Incomplete Data with Generative Adversarial Networks

1 INTRODUCTION . Generative adversarial networks ( GANs ) ( Goodfellow et al. , 2014 ) provide a powerful modeling framework for learning complex high-dimensional distributions . Unlike likelihood-based methods , GANs are referred to as implicit probabilistic models ( Mohamed & Lakshminarayanan , 2016 ) . They represent a probability distribution through a generator that learns to directly produce samples from the desired distribution . The generator is trained adversarially by optimizing a minimax objective together with a discriminator . In practice , GANs have been shown to be very successful in a range of applications including generating photorealistic images ( Karras et al. , 2018 ) . Other than generating samples , many downstream tasks require a good generative model , such as image inpainting ( Pathak et al. , 2016 ; Yeh et al. , 2017 ) . Training GANs normally requires access to a large collection of fully-observed data . However , it is not always possible to obtain a large amount of fully-observed data . Missing data is well-known to be prevalent in many real-world application domains where different data cases might have different missing entries . This arbitrary missingness poses a significant challenge to many existing machine learning models . Following Little & Rubin ( 2014 ) , the generative process for incompletely observed data can be described as shown below where x ∈ Rn is a complete data vector and m ∈ { 0 , 1 } n is a binary mask2 that determines which entries in x to reveal : x ∼ pθ ( x ) , m ∼ pφ ( m|x ) . ( 1 ) Let xobs denote the observed elements of x , and xmis denote the missing elements according to the mask m. In addition , let θ denote the unknown parameters of the data distribution , and φ denote the unknown parameters for the mask distribution , which are usually assumed to be independent of θ . In the standard maximum likelihood setting , the unknown parameters are estimated by maximizing the 1 Our implementation is available at https : //github.com/steveli/misgan 2 The complement m̄ is usually referred to as the missing data indicator in the literature . following marginal likelihood , integrating over the unknown missing data values : p ( xobs , m ) = ∫ pθ ( xobs , xmis ) pφ ( m|xobs , xmis ) dxmis . Little & Rubin ( 2014 ) characterize the missing data mechanism pφ ( m|xobs , xmis ) in terms of independence relations between the complete data x = [ xobs , xmis ] and the masks m : • Missing completely at random ( MCAR ) : pφ ( m|x ) = pφ ( m ) , • Missing at random ( MAR ) : pφ ( m|x ) = pφ ( m|xobs ) , • Not missing at random ( NMAR ) : m depends on xmis and possibly also xobs . Most work on incomplete data assumes MCAR or MAR since under these assumptions p ( xobs , m ) can be factorized into pθ ( xobs ) pφ ( m|xobs ) . With such decoupling , the missing data mechanism can be ignored when learning the data generating model while yielding correct estimates for θ . When pθ ( x ) does not admit efficient marginalization over xmis , estimation of θ is usually performed by maximizing a variational lower bound , as shown below , using the EM algorithm or a more general approach ( Little & Rubin , 2014 ; Ghahramani & Jordan , 1994 ) : log pθ ( xobs ) ≥ Eq ( xmis|xobs ) [ log pθ ( xobs , xmis ) − log q ( xmis|xobs ) ] . ( 2 ) The primary contribution of this paper is the development of a GAN-based framework for learning high-dimensional data distributions in the presence of incomplete observations . Our framework introduces an auxiliary GAN for learning a mask distribution to model the missingness . The masks are used to “ mask ” generated complete data by filling the indicated missing entries with a constant value . The complete data generator is trained so that the resulting masked data are indistinguishable from real incomplete data that are masked similarly . Our framework builds on the ideas of AmbientGAN ( Bora et al. , 2018 ) . AmbientGAN modifies the discriminator of a GAN to distinguish corrupted real samples from corrupted generated samples under a range of corruption processes ( or measurement processes ) . For images , examples of the measurement processes include random dropout , blur , block-patch , and so on . Missing data can be seen as a special type of corruption , except that we have access to the missing pattern in addition to the corrupted measurements . Moreover , AmbientGAN assumes the measurement process is known or parameterized only by a few parameters , which is not the case in general missing data problems . We provide empirical evidence that the proposed framework is able to effectively learn complex , highdimensional data distributions from highly incomplete data when the GAN generator incorporates suitable priors on the data generating process . We further show how the architecture can be used to generate high-quality imputations . 2 MISGAN : A GAN FOR MISSING DATA . In the missing data problem , we know exactly which entries in each data examples are missing . Therefore , we can represent an incomplete data case as a pair of a partially-observed data vector x ∈ Rn and a corresponding mask m ∈ { 0 , 1 } n that indicates which entries in x are observed : xd is observed if md = 1 otherwise xd is missing and might contain an arbitrary value that we should ignore . With this representation , an incomplete dataset is denoted D = { ( xi , mi ) } i=1 , ... , N ( we assume instances are i.i.d . samples ) . We choose this representation instead of xobs because it leads to a cleaner description of the proposed MisGAN framework . It also suggests how MisGAN can be implemented efficiently in practice as both x and m are fixed-length vectors . We begin by defining a masking operator fτ that fills in missing entries with a constant value τ : fτ ( x , m ) = x m + τm̄ , ( 3 ) where m̄ denotes the complement of m and denotes element-wise multiplication . Two key ideas underlie the MisGAN framework . First , in addition to the complete data generator , we explicitly model the missing data process using a mask generator . Since the masks in the incomplete dataset are fully observed , we can estimate their distribution using a standard GAN . Second , we train the complete data generator adversarially by masking its outputs using generated masks and fτ and comparing to real incomplete data that are similarly masked by fτ . Specifically , we use two generator-discriminator pairs ( Gm , Dm ) and ( Gx , Dx ) for the masks and data respectively . In this paper , we focus on the missing completely at random ( MCAR ) case , where the two generators are independent of each other and have their own noise distributions pz and pε . We define the following two loss functions , one for the masks and the other for the data : Lm ( Dm , Gm ) = E ( x , m ) ∼pD [ Dm ( m ) ] − Eε∼pε [ Dm ( Gm ( ε ) ) ] , ( 4 ) Lx ( Dx , Gx , Gm ) = E ( x , m ) ∼pD [ Dx ( fτ ( x , m ) ) ] − Eε∼pε , z∼pz [ Dx ( fτ ( Gx ( z ) , Gm ( ε ) ) ) ] . ( 5 ) The losses above follow the Wasserstein GAN formulation ( Arjovsky et al. , 2017 ) , although the proposed framework is compatible with many GAN variations ( Goodfellow et al. , 2014 ; Berthelot et al. , 2017 ; Gulrajani et al. , 2017 ) . We optimize the generators and the discriminators according to the following objectives : min Gx max Dx∈Fx Lx ( Dx , Gx , Gm ) , ( 6 ) min Gm max Dm∈Fm Lm ( Dm , Gm ) + αLx ( Dx , Gx , Gm ) , ( 7 ) where Fx , Fm are defined such that Dx , Dm are both 1-Lipschitz for Wasserstein GANs ( Arjovsky et al. , 2017 ) . Practically , we follow the common practice of alternating between a few steps of optimizing the discriminators and one step of optimizing the generators ( Goodfellow et al. , 2014 ; Arjovsky et al. , 2017 ; Gulrajani et al. , 2017 ) . The coefficient α is introduced when optimizing the mask generator Gm with the aim of minimizing a combination of Lm and Lx . Although in theory we could choose α = 0 to train Gm and Dm without using the data , we find that choosing a small value such as α = 0.2 improves performance . This encourages the generated masks to match the distribution of the real masks and the masked generated complete samples to match masked real data . The overall structure of MisGAN is illustrated in Figure 1 . Note that the data discriminator Dx takes as input the masked samples as if the data are fullyobserved . This allows us to use any existing architecture designed for complete data to construct the data discriminator . There is no need to develop customized neural network modules for dealing with missing data . For example , Dx can be a standard convolutional network for image applications . Note that the masks are binary-valued . Since discrete data generating processes have zero gradient almost everywhere , to carry out gradient-based training for GANs , we relax the output of the mask generator Gm from { 0 , 1 } n to [ 0 , 1 ] n. We employ a sigmoid activation σλ ( x ) = 1/ ( 1 + exp ( −x/λ ) ) with a low temperature 0 < λ < 1 to encourage saturation and make the output closer to zero or one . Finally , we note that the discriminator Dx in MisGAN is unaware of which entries are missing in the masked input samples , and does not even need to know which value τ is used for masking . In the next section , we present a theoretical analysis providing support for the idea that this type of masking process does not necessarily make it more difficult to recover the complete data distribution . The experiments provide compelling empirical evidence for the effectiveness of the proposed framework . 3 THEORETICAL RESULTS . In Section 2 we described how the discriminator Dx in MisGAN takes as input the masked samples using ( 3 ) without knowing what value τ is used or which entries in the input vector are missing . In this section , we discuss the following two important questions : i ) Does the choice of the filled-in value τ affect the ability to recover the data distribution ? ii ) Does information about the location of missing values affect the ability to recover the data distribution ? We address these questions in a simplified scenario where each dimension of the data vector takes values from a finite set P . For n-dimensional data , letM = { 0 , 1 } n be the set of all possible masks and I = Pn be the set of all possible data vectors . Also let DM and DI be the set of all possible probability distributions onM and I respectively , whose elements are non-negative and sum to one . We first discuss the case where the filled-in value τ is chosen from P . Given τ ∈ P and q ∈ DM , we can construct a left transition matrix Tq , τ ∈ RI×I defined below where the ( t , s ) -th entry specifies the transition probability from a data vector s ∈ I to an outcome t ∈ I masked by fτ , which involves all possible masks under which s is converted into t by filling in the indicated missing entries with τ : Tq , τ ( t , s ) = ∑ m∈M : fτ ( s , m ) =t q ( m ) . Let p∗x ∈ DI be the unknown true data distribution we want to estimate . In the presence of missing data specified by q , the masked samples then follow the distribution py = Tq , τp∗x . Without imposing extra application-specific constraints , MisGAN with a fixed mask generator can be viewed as solving the linear system py = Tq , τpx , where px ∈ DI is the unknown data distribution to solve for . Here we assume that py and Tq , τ are given , as those can be estimated separately from a collection of fully-observed masks and masked samples . Note that a transition matrix preserves the sum of the vectors it is applied to since 1 > Tq , τ = 1 > . For px to be a valid distribution vector , we only need the non-negativity constraint because any solution px automatically sums to one . That is , estimating the data generating process in the presence of missing data based on the masking scheme used in MisGAN is equivalent to solving the linear system Tq , τpx = py subject to px 0 . ( 8 ) In Theorem 1 , we state a key property of the transition matrix Tq , τ that leads to the answer to our questions . The proof of Theorem 1 is in Appendix A. Theorem 1 . Given q ∈ DM , all transition matrices Tq , τ with τ ∈ P have the same null space . Theorem 1 implies that if the solution to the constrained linear system ( 8 ) is not unique for a given τ0 ∈ P , that is , there exists some non-negative px 6= p∗x such that Tq , τ0px = Tq , τ0p∗x , then we must have Tq , τpx = Tq , τp∗x for all τ ∈ P . In other words , we have the following corollary : Corollary 1 . Whether the true data distribution is uniquely recoverable is independent of the choice of the filled-in value τ . Here we only discuss the case when the probability of observing all features q ( 1 ) is zero , where q ( 1 ) denotes the scalar entry of q indexed by 1 ∈M . Otherwise , the linear system is uniquely solvable as the transition matrix Tq , τ0 has full rank . With the non-negativity constraint , it is possible that the solution for the linear system ( 8 ) is unique when the true data distribution p∗x is sparse . Specifically , if there exists two indices s1 , s2 ∈ I such that p∗x ( s1 ) = p∗x ( s2 ) = 0 and also v ( s1 ) > 0 and v ( s2 ) < 0 for all v ∈ Null ( Tq , τ ) \ { 0 } , then the solution to ( 8 ) is unique . Sparsity of the data distribution is a reasonable assumption in many situations . For example , natural images are typically considered to lie on a low dimensional manifold , which means most of the instances in I should have almost zero probability . On the other hand , when the missing rate is high , that is , if the masks inM that have many zeros are more probable , the null space of Tq , τ will be larger and therefore it is more likely that the non-negative solution is not unique . Bruckstein et al . ( 2008 ) proposed a sufficient condition on the sparsity of the non-negative solutions to a general underdetermined linear system that guarantees unique optimality . Next we note that in the case of τ ∈ P , an entry with value τ in a masked sample t ∈ I may come either from an observed entry with value τ in the unmasked sample or from an unobserved entry through the masking operation in ( 3 ) . One might wonder if this prevents an algorithm from recovering the true distribution when it is otherwise possible to do so . In other words , if we take the location of the missing values into account , would that make the missing data problem less ill-posed ? However , this is not the case , as we state in Corollary 2 . The proof is in Appendix B where we discuss the case of τ /∈ P . Corollary 2 . If the linear system Tq , τpx = Tq , τp∗x does not have a unique non-negative solution , then for this missing data problem , we can not uniquely recover the true data distribution even if we take the location of the missing values into account . Note that the analysis in this section characterizes how difficult the missing data problem is , which is independent of the choice of the algorithm that solves it . In practice , it is useful to incorporate application-specific prior knowledge into the model to regularize the problem when it is ill-posed . For example , for modeling natural images , convolutional networks are commonly used to exploit the local structure of the data . In addition , decoder-based deep generative models such as GANs implicitly enforce some sparsity constraints due to the use of low dimensional latent codes in the generator , which also helps to regularize the problem . Finally , the following theorem justifies the training objective ( 6 ) of MisGAN for the missing data problem ( see Appendix A for details ) . Theorem 2 . Given a mask distribution pφ ( m ) , two distributions pθ ( x ) and pθ′ ( x ) induce the same distribution for fτ ( x , m ) if and only if they have the same marginals pθ ( xobs|m ) = pθ′ ( xobs|m ) for all masks m with pφ ( m ) > 0.3

Summary: Building upon the success of AmbientGAN by Bora, Price, and Dimakis, this paper studies one of the major issues that is not resolved in AmbientGAN: the distribution of the data corruption is typically unknown. In general this is an ill-defined problem to solve, as the data corruption distribution is not identifiable from the corrupted data. The major insight of this paper is to identify a plausible setting where such identifiability issues are not present. Namely, the corruption itself is identifiable from the corrupted data. The brilliance of this paper is in identifying this niche application of data imputation/missing data/incomplete data.