Handling Incomplete Heterogeneous Data using VAEs

Variational autoencoders (VAEs), as well as other generative models, have been shown to be efficient and accurate to capture the latent structure of vast amounts of complex high-dimensional data. However, existing VAEs can still not directly handle data that are heterogenous (mixed continuous and discrete) or incomplete (with missing data at random), which is indeed common in real-world applications. In this paper, we propose a general framework to design VAEs, suitable for fitting incomplete heterogenous data. The proposed HI-VAE includes likelihood models for real-valued, positive real valued, interval, categorical, ordinal and count data, and allows to estimate (and potentially impute) missing data accurately. Furthermore, HI-VAE presents competitive predictive performance in supervised tasks, outperforming supervised models when trained on incomplete data. Data are usually organized and stored in databases, which are often large, heterogenous, noisy, and contain missing values. For example, an online shopping platform has access to heterogenous and incomplete information of its users, such as their age, gender, orders, wishing lists, etc. Similarly, Electronic Health Records of hospitals might contain different lab measurements, diagnoses and genomic information about their patients. Learning generative models that accurately capture the distribution, and therefore the underlying latent structure, of such incomplete and heterogeneous datasets may allow us to better understand the data, estimate missing or corrupted values, detect outliers, and make predictions (e.g., on patients’ diagnose) on unseen data (Valera et al., 2017a). Deep generative models have been recently proved to be highly flexible and expressive unsupervised methods, able to capture the latent structure of complex high-dimensional data. They efficiently emulate complex distributions from large high-dimensional datasets, generating new data points similar to the original real-world data, after training is completed (Kingma and Welling, 2014; Rezende and Mohamed, 2015). So far, the main focus in the literature is to enrich the prior or posterior of explicit generative models such as variational autoencoders (VAEs); or to propose alternative training objectives to the log-likelihood, leading to implicit generative models such as, e.g., generative adversarial networks (GANs) (Mescheder et al., 2017; Salimans et al., 2016). Indeed, we are witnessing a race between an ever-growing spectrum of VAE models, e.g. VAE with a VampPrior (Tomczak and Welling, 2018), Variational Lossy Autoencoder (Chen et al., 2016), 1 ar X iv :1 80 7. 03 65 3v 3 [ cs .L G ] 3 0 O ct 2 01 8 DVAE++ (Vahdat et al., 2018), Shape Variational Autoencoder (Nash and Williams, 2017) and GAN-style objective functions (f-GAN (Nowozin et al., 2016) , Wasserstein GANs (Arjovsky et al., 2017), MMD-GAN (Li et al., 2015), AdaGAN (Tolstikhin et al., 2017), feature-matching GAN (Mroueh et al., 2017), etc.). While all these approaches compete to generate the most realistic images or readable text, the deployment of such models to solve practically-relevant problems in arbitrary datasets, which are often incomplete and heterogenous, is being overlooked. In the following, we discuss these problems in more detail and why we believe our paper is relevant to data-scientists interested in exploiting the deep generative model pipeline in the data wrangling process. We provide with practical tools to handle both missing and heterogeneous data with little supervision from the user, who merely has to indicate the data type model of each attribute and the position of the missing data. Currently deep generative models focus on highly-structured homogeneous data collections including, e.g., images (Gulrajani et al., 2017; Mescheder et al., 2017; Salimans et al., 2016), text (Yang et al., 2017) or speech (Bando et al., 2017), which are characterized by strong statistical dependencies between pixels or words. The dominant existing approach to account for heterogenous data follows a deep domain-alignment approach, designed to discover relations between two unpaired unlabelled datasets rather than modelling their joint distribution using a probabilistic generative model (Ganin et al., 2016; Kim et al., 2017; Liu et al., 2017; Zhu et al., 2017; Taigman et al., 2016; Castrejon et al., 2016). Surprisingly, not much attention has been paid to describe how deep generative models can be designed to effectively learn the distribution of, usually less structured, heterogeneous datasets. In these datasets there is no clear notion of correlation among the different attributes (or dimensions) to be exploited by weight sharing using convolutional or recurrent neural networks. As we show in this paper, preventing a few dimensions of the data dominating the training is a crucial aspect to effectively deploy deep generative models suitable for heterogeneous data. Similarly, there is no clear discussion in the current literature on how to incorporate missing data during the training of deep generative models. Existing approaches consider either complete data during training (Kingma and Welling, 2014; Rezende and Mohamed, 2015; Mescheder et al., 2017; Salimans et al., 2016), or assume incomplete information only in one of the dimensions of the data, which corresponds to the one they aim to predict (e.g., the label in a classification task) (Sohn et al., 2015; Kingma et al., 2014). However, both approaches are not realistic enough, since it might be crucial for the performance of an unsupervised model to use all the available information during training. Recently, (Yoon et al., 2018) proposed a GAN approach to input missing data, where the generator completes the missing values given the observed ones, and the discriminator aims to distinguish between true and imputed values. However, this approach can only handle continuous or binary data, and it is not easily generalizable to heterogeneous data. As a consequence, strategies to effectively train deep generative models on incomplete and heterogeneous datasets are still required. In this work, we present a general framework for VAEs that effectively incorporates incomplete data and heterogenous observations. Our design presents the following features: i) a generative model that handles mixed numerical (continuous real-valued and positive real-valued, as well as discrete count data) and nominal (categorical and ordinal data) likelihood models, which we parametrize using deep neural networks (DNNs); ii) a stable recognition model that handles Missing Data Completely at Random (MCAR) without increasing its complexity or promoting overfitting; iii) a data-normalization input/output layer that prevents a few dimensions of the data dominating the training of the VAE, improving also the training convergence; and iv) an ELBO (Evidence Lower Bound), used to optimize the parameters of both the generative and the recognition models, that is computed only on the observed data, regardless of the pattern of missing data. The resulting VAE is a fully unsupervised model which allows us not only to accurately solve unsupervised tasks, such as density estimation or missing data completion, but also supervised tasks (e.g., classification or regression ) with incomplete input data. We provide the reader with specific guidelines to design VAEs for real-world data, which are compatible with modern efforts in the design of VAEs and implicit models (GANs), mainly oriented to prevent the mode-dropping effect (Arjovsky et al., 2017; Arora and Zhang, 2017). Our