Tabular Data Imputation: Choose KNN over Deep Learning

1 INTRODUCTION . Missing values constitute a serious issue in data science . Incomplete datasets result from uncollected , lost or voluntarily deleted observations . Most machine learning algorithms can not handle datasets with missing values , and meticulous data preprocessing is therefore needed . The most common preparatory processing consists of Listwise deletion , where only complete observations are kept for further data analysis . This solution , although straightforward to implement , has two main disadvantages : it significantly reduces the size of the dataset and induces a bias , which may lead to false positive research claims ( Lall ( 2016 ) ) . Data imputation is an alternative preprocessing method which involves estimation and replacement of the missing values . It allows to preserve the whole dataset for subsequent analysis but requires careful handling as it can also introduce a bias in the imputed dataset ( Gelman & Hill ( 2006 ) ) . In particular , only ” well-behaved ” values can be imputed and missing values of outlier observations will get overlooked ( Binder ( 1996 ) ) . Several data imputation algorithms have been proposed over the last 70 years : Hot-deck , Colddeck , Mean-value substitution ( Kalton & Kasprzyk ( 1986 ) ) , C4.5 algorithm ( Salzberg ( 1994 ) ) , CN2 induction algorithm ( Clark & Niblett ( 1989 ) ) , KNN ( originally developed in 1951 - Fix & Hodges ( 1989 ) - and later expanded by Altman ( 1992 ) ) . These traditional machine learning algorithms have been shown useful in the past . Notably , KNN has shown best imputation quality amongst these various data imputation algorithms ( Batista & Monard ( 2002 ) , Bertsimas et al . ( 2018 ) , Poulos & Valle ( 2018 ) , Jadhav et al . ( 2019 ) ) . Over the last decade , Artificial Neural Networks ( ANN ) have transformed the fields of statistics , computer science and data analysis . More precisely , Generative Adversarial Networks ( GANs ) have shown spectacular results at generating new observations from a given data distribution . As such , GAN models have been developed to tackle the problem of missing values in data science . GAIN ( Yoon et al . ( 2018 ) ) and MisGAN ( Li et al . ( 2019 ) ) claim state-of-the-art data imputation results . While GAIN has been specifically developed for tabular data imputation , MisGAN is primarily designed to impute degraded images and can be adapted to work with tabular data . This work intends to compare data imputation performances of state-of-the-art GAN models ( GAIN and MisGAN ) with the standard already established K-nearest neighbours algorithm ( KNN ) . We evaluate the imputation quality in MCAR , MAR and MNAR scenarios ( see Section 2 ) . The present study is restricted to tabular data , that is numerical data we can arrange into rows and columns in the form of a table with cells . Section 2 explains the various missing data scenarios and their properties . Section 3 presents the algorithms used in this work . Section 4 introduces the datasets . Section 5 details the method of this study . Section 6 shows the data imputation results . Section 7 includes a summary , a discussion and final recommendations . 2 MISSING DATA SETTINGS . Let x denote the complete vector of a given observation and m its missing value binary mask , so that the actual available data is the element-wise product x̃ = x m. Data imputation methods seek to estimate the missing values of x̃ by taking advantage of patterns in the dataset and using observed properties of x̃ . The probability distribution of m is referred to as the missing data mechanism . Following the classification established by Little & Rubin ( 2014 ) , missing data scenarios belong to one of the following three settings . 2.1 MISSING COMPLETELY AT RANDOM ( MCAR ) . In MCAR scenarios , the missing data mechanism is assumed to be independent of the intrinsic probability distribution of x , and occurs completely at random , such that p ( m|x ) = p ( m ) . We can generate MCAR missing values in a complete dataset by hiding each cell with a given probability , for example 10 % . There exist statistical tests to assess whether the MCAR assumption holds or not ( Little ( 1988 ) ) . Apart from the obvious information loss , MCAR scenarios are convenient assumptions . Indeed , the subset of complete observations is free of bias , and we may therefore safely proceed with further data analysis techniques . However , MCAR is often an unrealistically strong assumption in practice . 2.2 MISSING AT RANDOM ( MAR ) . MAR scenarios occur when the missing data mechanism can be fully explained by the observed data . We may write p ( m|x ) = p ( m|x̃ ) . For example , males are less likely to complete a depression survey , regardless of their depression level . As long as the gender of every participant is known , the dataset can be treated as MAR by assuming that missingness only depends on maleness . MAR assumption is more general than MCAR , but can not be statistically tested and one must rely on its fairness . Subsequent statistical analyses may include biases which can theoretically be corrected under the MAR assumption . 2.3 MISSING NOT AT RANDOM ( MNAR ) . Finally , MNAR scenarios encompass every other scenario : the reason why data is missing depends on unobserved variables . In the previous example , this is the case if respondents do not fill the depression survey because of their actual depression level , or because of another unrecorded variable like their income for instance . As MCAR and MAR are convenient assumptions to facilitate the treatment of incomplete datasets , most missing data mechanisms are actually distributed in a MNAR setting . While being truthful , MNAR scenarios have no systematic ways to handle missing values . One strategy is to accumulate more data in the hope of understanding the missing data mechanism . 3 DATA IMPUTATION ALGORITHMS . We select , analyse and compare the performances of the best algorithms to date . While some tabular data imputation methods learn a probability distribution on a complete dataset before performing data imputation , we focus on algorithms which use the entire dataset ( including missing values ) to perform in-place imputations . On the one hand , GAIN ( Yoon et al . ( 2018 ) ) and MisGAN ( Li et al . ( 2019 ) ) are new state-of-the-art ANN generative models tailored for data imputation . On the other hand , the KNN algorithm is a non-parametric method which can be used for data imputation . 3.1 PRESENTATION OF GAIN . Generative Adversarial Imputation Nets ( GAIN ) have been proposed in 2018 as a generative model specifically designed for tabular data imputation problems . GAIN generalises the well-established architecture of GAN models by looking at individual cells rather than complete rows . The authors claim state-of-the art imputation quality performances ( Yoon et al . ( 2018 ) ) . Given an incomplete observation , GAIN works as follows : 1 . Cells with missing values are filled by random noise while observed values are kept unchanged . A binary mask telling whether each cell is genuine or comes from the random noise is concatenated to the observation , therefore doubling its length . 2 . This vector is provided to the GAIN generator which outputs a generated vector . The original data vector is imputed by conserving genuine cells and replacing missing cells by their corresponding values in the generated vector . 3 . The imputed vector is presented to the GAIN discriminator which tells apart genuine values from imputed values . The GAIN discriminator works on a cell-by-cell basis instead of considering the row as a whole . Both GAIN generator ’ s and discriminator ’ s parameters are trained following the standard adversarial process used by GAN models . GAIN ’ s parameters are updated to minimise the binary cross-entropy loss function for the discriminator given by LD ( i ) = − ∑ j , bij=0 mij log ( ŷij ) + ( 1−mij ) log ( 1− ŷij ) where mij is 0 if cell ( i , j ) is missing or 1 otherwise , bij = 0 indicates cells that have not been hinted to the discriminator , and ŷij is the discriminator prediction for cell ( i , j ) ; and a combination of binary cross-entropy ( for the generated cells ) and RMSE for the discriminator given by LG ( i ) = LG1 ( i ) + αLG2 ( i ) with LG1 ( i ) = − ∑ j , bij=0 ( 1−mij ) log ( ŷij ) and LG2 ( i ) = − ∑ j mij ( x̃ij − xij ) 2 where x̃ij is the generator output and xij the actual observation for cell ( i , j ) . GAIN has two hyperparameters to manually tune : the hint fraction h of true labels to provide the discriminator to stabilise learning , and α the relative weight between cross-entropy and RMSE for the generator loss . After several trails , we decide to use the recommended default values of Yoon et al . ( 2018 ) . Besides , the most relevant hyperparameter for GAIN is the number of training epochs , which we discuss in Section 5 . 3.2 PRESENTATION OF MISGAN . MisGAN has been introduced in 2019 as another GAN model framework capable of handling complex datasets with missing values . Primarily developed for image completion , it can be adapted to handle tabular data . MisGAN claims state-of-the-art imputation quality for images ( Li et al . ( 2019 ) ) . In its original design , MisGAN is composed of three pairs of generator and discriminator . A first pair ( Gx , Dx ) attempts to model the probability distribution of the data , while another pair ( Gm , Dm ) tries to model the missing data mechanism . Both generators Gx and Gm are used in parallel to produce fake deteriorated observations . Finally , a third pair ( Gi , Di ) is used to perform the imputation of missing values . When adapted to tabular data , the missing data mechanism has a simple distribution and does not require complex neural networks to be modelled . Instead , we choose to draw missing masks directly from the observations in the dataset . Like for GAIN , the performances of MisGAN mostly depends on the number of training epochs . Section 5 explains how we fix the number of training epochs for MisGAN . The artificial neural network architecture of GAIN and MisGAN are both presented in Appendix A . 3.3 PRESENTATION OF KNN . The K-Nearest Neighbours algorithm is a non-parametric method , originally developed for classification tasks in 1951 ( Altman ( 1992 ) ) . For data imputation tasks , the KNN algorithm selects the K nearest neighbours of a given incomplete observation , and uses available data from the selected neighbours to estimate missing values . Despite its simplicity and its age , the KNN algorithm has been shown to outperform traditional data imputation algorithms ( Batista & Monard ( 2002 ) ) . KNN imputes missing values using a weighted average of the selected neighbours . The most two common weighting systems are uniform and distance , respectively defined by 1K ( where K is the number of neighbours ) and 1dij ( where dij is the distance between observations i and j . We respectively refer to these algorithms with the names KNN-uniform and KNN-distance .

Summary: This paper considers the problem of tabular data imputation to handle missing numeric data. It compares the RMSE of recently proposed GAN-based methods (GAIN and mis-GAN) with the classical KNN method (either through uniform or distance-weighted imputation). It considers 9 datasets, two are simulated by a mixture of Gaussians and 7 are real datasets, 5 of them used in the GAIN paper. The datasets have 500-30000 samples and 5-57 features (most are below 20). It also considers three different mechanisms of missing data (MCAR - completely random, MAR - random based on the observed data, MNAR - based on unobserved data), each with one or two rates of missing data (more rates are tested for MCAR with the simulated data). The hyperparameters of all the models are tuned once on the simulated data (1000 samples), and scaled based on the number of samples for each dataset (scaling the number of neighbors for KNN, or epochs for the GANs). The models are applied to each imputation case 20 times, presenting the mean and standard deviation for each dataset in each scenario. The paper shows that mis-GAN, originally developed for filling missing image pixels, usually performs much worse than KNN. Furthermore, KNN generally performs roughly on-par compared to GAIN, depending on the dataset and exact scenario (in many cases KNN is better). Based on the presented figures, the RMSE differences are usually within 0.02, but may reach 0.05. The compute required by KNN is much lower, and the paper calls for using the cheaper option, avoiding waste of resources.