Tabular Data Imputation: Choose KNN over Deep Learning
Authors: Florian Lalande
Kenji Doya
Venue: ICLR 2022 Submitted (2022)
Keywords: data imputation
knn
deep learning
artificial neural networks
digital sobriety

Abstract: As databases are ubiquitous nowadays, missing values constitute a pervasive problem for data analysis. Over the last 70 years, various imputation algorithms for tabular data have been developed and shown useful at estimating missing values. Besides, recent infatuations for Artificial Neural Networks have led to the development of complex and powerful algorithms for data imputation. 
This study is the first to compare state-of-the-art deep-learning models with the well-established KNN algorithm (1951). By using real-world and generated datasets in various missing data scenarios, we claim that the good old KNN algorithm is still competitive (nay better) than powerful deep-learning algorithms for tabular data imputation.
This work advocates for an appropriate and reasonable use of machine learning, in a world where overconsumption, performances and rapidity unfortunately often prevails over sustainability and common sense.

Under review as a conference paper at ICLR 2022
TABULAR DATA IMPUTATION CHOOSE KNN OVER DEEP LEARNING
Anonymous authors Paper under double-blind review
ABSTRACT
Recent infatuation for Artificial Neural Networks have led to the development of complex and powerful algorithms for data imputation. This study is compares the best state-of-the-art deep-learning models with the well-established KNN al- gorithm (1951). By using real-world and generated datasets in various missing data scenarios, we show that the good old KNN algorithm is still competitive (nay better) than powerful deep-learning algorithms for tabular data imputation. This work advocates for an appropriate and reasonable use of machine learning.
INTRODUCTION
Missing values constitute serious issue in data science. Incomplete datasets result from uncol- lected, lost or voluntarily deleted observations. Most machine learning algorithms cannot handle datasets with missing values, and meticulous data preprocessing is therefore needed. The most com- mon preparatory processing consists of Listwise deletion, where only complete observations are kept for further data analysis. This solution, although straightforward to implement, has two main disadvantages: it significantly reduces the size of the dataset and induces a bias, which may lead to false positive research claims (Tall (2066).
Data imputation is an alternative preprocessing method which involves estimation and replacement of the missing values. allows to preserve the whole dataset for subsequent analysis but requires careful handling as it can also introduce bias in the imputed dataset (Gelman & Hill (006)). In particular, only "well-behaved" values can be imputed and missing values of outlier observations will get overlooked (Binder ([inder(1999)) (1996)).
Several data imputation algorithms have been proposed over the last 70 years: Hot-deck. Cold- deck, Mean-value substitution (Kalton & Kasprzyk (1866). C4.5 algorithm (Salzberg 1994)). )) CN2 induction algorithm (Clark & Niblett ( (1989)), KNN (originally developed in 1951 Fix & Hodges - and later expanded by Altman (1992). These traditional machine learning algorithms have been shown useful in the past. Notably, KNN has shown best imputation quality amongst these
various data imputation algorithms (Batista & Monard 2002). Bertsimas et al. 2018 Poulos &
Valle 201 8 Jadhav et al. (2019)).
Over the last decade, Artificial Neural Networks (ANN) have transformed the fields of statisties, computer science and data analysis. More precisely, Generative Adversarial Networks (GANs) have shown spectacular results at generating new observations from a given data distribution. As such, GAN models have been developed to tackle the problem of missing values in data science. GAIN (Yoon et al. (2018)) and MisGAN (Li et al.] (2019)) claim state-of-the-art data imputation results. While GAIN has been specifically developed for tabular data imputation, MisGAN is primarily designed to impute degraded images and can be adapted to work with tabular data.
This work intends compare data imputation performances of -fate-of-the-art GAN models (GAIN and MisGAN) with the standard already established K-nearest neighbours algorithm (KNN). We evaluate the imputation quality in MCAR, MAR and MNAR scenarios (see Section 2). The present study is restricted to tabular data, that is numerical data we can arrange into rows and columns in the form of table with cells.
Section 2 explains the various missing data scenarios and their properties. Section 3 presents the algorithms used in this work. Section 4 introduces the datasets. Section 5 details the method of this

---

Let denote the complete vector of given observation and m its missing value binary mask, so that the actual available data is the element-wise product I = 2 & m. Data imputation methods seek estimate the missing values of by taking advantage of patterns the dataset and using observed properties of T. The probability distribution of is referred to as the missing data mechanism. Under review as a conference paper at ICLR 2022
study. Section shows the data imputation results. Section 7 includes summary, a discussion and final recommendations.
MISSING DATA SETTINGS
Following the classification established by Little & Rubin (2014), missing data scenarios belong to one of the following three settings.
2.1 MISSING COMPLETELY AT RANDOM (MCAR)
In MCAR scenarios, the missing data mechanism is assumed to be independent of the intrinsic probability distribution x, and occurs completely at random, such that p(m[z) = p(m). We can generate MCAR missing values in complete dataset by hiding each cell with given probability, for example 10%. There exist statistical tests to assess whether the MCAR assumption holds or not (Little (1988)).
Apart from the obvious information loss, MCAR scenarios are convenient assumptions. Indeed, the subset of complete observations is free of bias, and may therefore safely proceed with further data analysis techniques. However, MCAR is often an unrealistically strong assumption in practice.
2.2 MISSING AT RANDOM (MAR)
MAR scenarios occur when the missing data mechanism can be fully explained by the observed data. We may write p(m[z) (z) = p(m [). For example, males are less likely to complete a depression survey, regardless of their depression level. As long as the gender of every participant is known, the dataset can be treated as MAR by assuming that missingness only depends on maleness.
MAR assumption is more general than MCAR, but cannot be statistically tested and one must rely on its fairness. Subsequent statistical analyses may include biases which can theoretically be corrected under the MAR assumption.
2.3 MISSING NOT AT RANDOM (MNAR)
Finally, MNAR scenarios encompass every other scenario: the reason why data is missing depends on unobserved variables. the previous example, this is the case respondents do not fill the depression survey because of their actual depression level, because of another unrecorded variable like their income for instance. As MCAR and MAR are convenient assumptions to facilitate the treatment of incomplete datasets, most missing data mechanisms are actually distributed in MNAR setting.
While being truthful, MNAR scenarios have no systematic ways handle missing values. One strategy is to accumulate more data in the hope of understanding the missing data mechanism.
DATA IMPUTATION ALGORITTMM
We select, analyse and compare the performances of the best algorithms to date. While some tabular data imputation methods learn probability distribution a complete dataset before performing data imputation, we focus on algorithms which use the entire dataset (including missing values) to perform in in-place imputations. On the one hand, GAIN (Yoon et al. (2018) and MisGAN (Li et al. (2019)) are new state-of-the-aas art ANN generative models tailored for data imputation. On the other hand, the KNN algorithm is non-parametric method which can be used for data imputation.
2

---

Under review as a conference paper at ICLR 2022
3.1 PRESENTATION OF GAIN
Generative Adversarial Imputation Nets (GAIN) have been proposed in 2018 as generative model specifically designed for tabular data imputation problems. GAIN generalises the well-established architecture of GAN models by looking at individual cells rather than complete rows. The authors claim state-of-the art imputation quality performances (Yoon et al. (2018)).
Given an incomplete observation, GAIN works as follows:
1. Cells with missing values filled by random noise while observed values kept un- changed. A binary mask telling whether each cell is genuine or comes from the random noise is concatenated the observation, therefore doubling its length.
2. This vector is provided the GAIN generator which outputs generated vector. The original data vector is imputed by conserving genuine cells and replacing missing cells by their corresponding values in the generated vector.
3. The imputed vector presented the GAIN discriminator which tells apart genuine values from imputed values. The GAIN discriminator works a cell-by-cell basis instead of considering the row as whole. Both GAIN generator's and discriminator's parameters are trained following the standard adversarial process used by GAN models.
GAIN's parameters updated minimise the binary cross-entropy loss function for the discrimi- nator given by
(n(1)= = Σ myy log(Uis) + (1 - m)g log(1 Vij i.by=0
where myj cell (i,j) is missing or otherwise, bij O indicates cells that have not been hinted the discriminator, and Vij is the discriminator prediction for cell (i, and combination of binary cross-entropy (for the generated cells) and RMSE for the discriminator given by
La(i) = Lcc + aLc,
with (1) (1- log(
and
LG (-) miy Eay --
xzT)
j.big
where Fiy is the generator output and Tay the actual observation for cell (i.j)-
GAIN has two hyperparameters to manually tune: the hint fraction h of true labels provide the discriminator to stabilise learning, and a the relative weight between cross-entropy and RMSE for the generator loss. After several trails, we decide et al. (0188). Besides, the most relevant hyperparameter for GAIN is the number of training epochs, which we discuss in Section 5.
the recommended default values of Yoon
3.2 PRESENTATION OF MISGAN
MisGAN has been introduced in 2019 another GAN model framework capable of handling com- plex datasets with missing values. Primarily developed for image completion, it can be adapted to handle tabular data. MisGAN claims state-of-the-an imputation quality for images (Li et al. (2019).
In its original design, MisGAN is composed of three pairs of generator and discriminator. A first pair (Ga, D.)) attempts to model the probability distribution of the data, while another pair (Gm, Dm) tries to model the missing data mechanism. Both generators G,, and Gm are used in parallel to pro- duce fake deteriorated observations. Finally, a third pair (G,, D.) is used to perform the imputation of missing values.
When adapted to tabular data, the missing data mechanism has a simple distribution and does not require complex neural networks be modelled. Instead, choose to draw missing masks directly from the observations the dataset.
Like for GAIN, the performances of MisGAN mostly depends the number of training epochs. Section 5 explains how we fix the number training epochs for MisGAN, The artificial neural network architecture of GAIN and MisGAN are both presented in Appendix A.
3

---

Under review as a conference paper at ICLR 2022
3.3 PRESENTATION OF KNN
The K-Nearest Neighbours algorithm is a non-parametric method, originally developed for classi- fication tasks in 1951 (Altman (1992)). For data imputation tasks, the KNN algorithm selects the K nearest neighbours of a given incomplete observation, and uses available data from the selected neighbours to estimate missing values. Despite its simplicity and its age, the KNN algorithm has been shown to outperform traditional data imputation algorithms (Batista & Monard )
KNN imputes missing values using weighted average of the selected neighbours. The most two common weighting systems are uniform and distance, respectively defined by X (where K is the number of neighbours) and 1_ (where dı is the distance between observations and We das respectively refer to these algorithms with the names ENN-uniform and KNN-distance.
DATASETS
Real-world and well-behaved simulated datasets are both used this comparative study. The real- world datasets are taken from the open access UCI Machine Learning Repository (Dua & Graff (211))) For the sake of reproducibility, we make use of the same datasets used by authors of GAIN Yoon al. (2018). And for the sake variety, wine and red wine datasets. Table[[ the size of the nine datasets.
additional datasets have been selected: white
We insist that comprehensive understanding of the data play is the most important preliminary step before any data analysis. Accordingly, Appendix B provides all essential details about the data used this work. worth noting that all following datasets are originally complete, i.e. they do not have missing values in the first place.
METHODS
This section describes the methodology of this work. We begin with finding the appropriate hy- perparameters of the algorithms used. Then, we conduct several experiments with varying missing rates, missing data scenarios and datasets. Throughout the following experiments, we first inject missing values and immediately scale the datasets in the interval [0, 1]. After imputation, the perfor- mances are computed using the RMSE between ground truth and imputed missing values directly on the scaled variables. Each experiment is repeated 20 times to allow for the computation of the mean and the standard deviation of the RMSE reported in each subsequent plot.
5.1 TUNING THE HYPERPARAMETRE
We use the generated mixture of three Gaussians dataset (c.f. mydata2 dataset, Appendix B.7) in MCAR scenario with 20% missing rate select appropriate hyperparameters for the various algorithms of this comparative study. For the following hyperparameter search, we first introduce missing values and then use the algorithms with their pre-selected hyperparameters values and repeat this experiemtn 20 times. We report the mean and standard deviation of the imputation RMSE in Appendix C to select the best hyperparameter values.
GAIN and MisGAN need the number of training epochs to be specified. We train from 1,000 to 20,000 epochs by steps of 1,000 epochs and choose the number of training epochs which minimises the imputation RMSE (see Appendix C): 20,000 epochs for GAIN and 5,000 epochs for MisGAN. MisGAN becomes unstable after 7,000 epochs. As GAIN performances keep decreasing, we train with more epochs (see Appendix C.2) and decide to stick with 20,000 epochs.
For KNN, the hyperparameter is the number of neighbours K. We use from 2 to 300 neighbours by steps of 2, and select K = 50 neighbours to minimise the imputation RMSE for both weight systems (see Appendix C.3).
Note that these hyperparameters may strongly depend on the dataset size. The dataset used for hyperparameters tuning has 1,000 rows. For following experiments, we compute a multiplicative factor to preserve similar proportions Tooo where is the number of rows in the current dataset.
The number of training epochs accordingly becomes 5000 7 for MisGAN and 20000 for GAIN, and _
4

---

Under review as a conference paper at ICLR 2022
the number of neighbours is 50f for both KNN algorithms. As example, dataset two times larger will use twice as many neighbours and half the number of training epochs. We experimented two datasets: wine white and breast. The former has about five times more
this approach entries than mydata2 dataset, while the later has about half. Appendix C.4 discuss the reliability of the hyperparameter selection tuning process.
5.2 EXPERIMENTS
The following experiments are conducted 20 times to compute the mean and standard deviation of the imputation RMSE. Results and interpretation are provided in the next section.
Varying the missing rate MCAR setting from 10% to 80% by steps of 10,, with the generated mixture of three Gaussians dataset.
2. Performances real-world datasets in MCAR setring with 20% missing rate.
3. Performances over all datasets in MAR setting, with overall 20% and 45% missing rates.
4. Performances over datasets in MNAR setting, with overall 20% and 45% missing rates.
To generate MAR missing values, we select column that we keep untouched throughout the full MAR experiment. We compute the quantiles of the selected column, scale the quantiles between and 2μ (where y is the overall missing rate) and interpret these values as common missing rates for all other columns. For example, in MAR setting with an overall 20% missing rate, the quantiles of the selected column are scaled in the range [0%, 40%]. The selected variable for each dataset along with its meaning are reported in Table
Table 1:
List of the selected variables for MAR settings. The datasets' sizes are also provided as well as the column number of the variable used to generate MAR missing data.
Dataset name
Size
MAR column
Meaning
breast
(569, 30)
0
mean cell radius
credit
(30000, 14)
customer's age
letter
(20000, 16)
total number of black pixels
news
(39644, 44)
0
number of words in the title
spam
(4601, 57)
54
average length of uppercase letters
wine_red
(1599, 12)
fixed acidity
wine white
(4898, 12)
0
fixed acidity
mydatal
(1000, 5)
0
No meaning
mydata2
(1000, 5)
No meaning
To generate MNAR missing values, we compute the quantiles for every individual column (in the same way MAR setting) which we scale in the interval [0, 24]. We interpret these values as the missing probability for every corresponding cell. MNAR settings, the higher the value of a cell relatively to its column, the more likely it will be missing. Looking at the first column of the mixture of three Gaussian dataset for example, data from the orange cluster is more likely to be missing than data from the blue cluster (see Fig. 5 Appendix B).
RESULTS AND INTERPRETATION
6.1 VARYING THE MISSING RATE IN MCAR SETTING
The first experiment consists in changing the missing rate from 10% to 80% (by steps of 10%) on the generated mixture of three Gaussians dataset in MCAR setting (mydata2). The imputation results are shown in Fig. T (a). As the RMSE for MisGAN is comparatively large, we only display Fig. I in Appendix D.I). We decide not to show the results of MisGAN from now on (they are provided in the supplementary materials, in Appendix D.1). For numerical values, Tables in Appendix G.
the results of GAIN and KNN for clarity (for complete results and discussion MisGAN,
5

---

Under review as a conference paper at ICLR 2022
Mixture of 3 gaussians dataset
[MCAR setting] mean missing_rad rate of 20%
0.250
0.14
T
GAIN
0.225
KNN-unifoom
0.200
0.12
KNN-distance
I
I
0.175
0.11
H
RMSE
RMSE
0.10
I
0.150
0.09
H
0.125
0.100
I
0.07
0.075
0.06
H
0.1
0,2
0.3
0.4
0.6
0.7
0.8
breast
credit
letter
spam
wine red
wine white
Missing rate
Figure 1: MCAR scenario experiments. (a) Left panel: imputation RMSE for the mixture of gaussians dataset (mydat a2) with varying missing rates. (b) Right panel: imputation RMSE for all real-world datasets (except the news dataset).
Not surprisingly, the performances of GAIN and KNN deteriorate with increasing missing rates. GAIN remains competitive with both KNN algorithms. is worth noting that KNN-uniform per- forms better low missing rates, while KNN-distance performs better high missing rates. Intu- itively, the weight system provides KNN-distance with better resilience to bad neighbours (which are more common with high missing rates).
6.2 REAL-WORLD DATASETS MCAR SETTING
The next experiment uses real-world datasets in MCAR setting with 20% missing rate. Fig. (b) shows the imputation quality results, where we can see that both KNN algorithms perform better than GAIN over all available datasets. From now, also decide not to report the RMSE for the news dataset since the performances are chaotic regardless of the imputation method (for explanation, discussion and the complete results, Fig. 12 in Appendix D.2).
We could not reproduce GAIN results. Intriguingly, the imputation of GAIN this work outper- forms the imputation from the original authors the credit dataset: RMSE of 0.063 here, and 0.186 in Yoon et al. (2018). For the other datasets, the performances of GAIN in this work are slightly lower than the original authors' reported RMSE. Exhaustive comparison is available in Ap- pendix E.
6.3 MAR EXPERIMENTS
This section refers to Fig. With a missing rate of 20% in MAR setting (left hand panel), both KNN algorithms perform slightly better than GAIN over all datasets. When increase the missing rate to 45% (right hand panel), we see that the imputation quality overall decays. The performances of GAIN, KNN-uniform and KNN-distance
comparable, with sometimes nificantly performing better. We notice that the trends and orders magnitude are being preserved across datasets when the missing rate increases. Look the breast dataset for instance: it has high RMSE with GAIN both with 20% or 45% missing rate data.
method sig-
6.4 MNAR EXPERIMENTS
This paragraph refers to Fig. MNAR setting with missing rate 20% (left hand panel), there is no clear best imputation method. All imputation RMSEs remain below 0.15 on average, and are very close to the imputation RMSES obtained in the MAR setting (cf. Fig. 2 (a)). With a missing rate of 45% (right hand panel), the RMSE becomes We can argue that MNAR missing values are harder to recover than MAR missing values, since the missing data mechanism hides itself in the process of generating missing values. GAIN performs
larger than with the MAR experiments.

---

Under review as a conference paper at ICLR 2022
[MAR setting] mean missing_rate rate of 20%
[MAR setting] mean missing rate of 45%
0.225
I
KNN-uniform KNN- distance
0.200
i
0.175
RMSE
0.150
[
H
I
0.125
0.100
H
0.075
I
0.050
breast
credit
letter
spar
wine, red
wine white
mydstaz
mydata2
breast
credit
letter
spar
wine red
wine white
mydstal
mydataz
Figure MAR scenario experiments for all datasets (except dataset). (a) Left panel: MAR setting with average missing rate of 20%. (b) Right panel: MAR setting with average missing rate 45%.
better imputations than KNN in MNAR setting with 45% missing rate. For the large variance in the spam dataset RMSE, see Appendix D.2.
worth noting that KNN algorithms perform poorly against GAIN with the generated Gaussians mixture dataset, on which we benchmarked the hyperparameters. The mixture of three Gaussians dataset indeed suffers a lot from the strong case of MNAR mechanism using quantiles, as clusters with large value variables will undergo higher missing rates than clusters high low value variables. Consequently, some clusters concentrate most of the missing data and become challenging to impute with KNN as they have few neighbours with complete observations.
[MNAR setting] mean missing_ratee of 20%
[MNAR setting) mean missing_rad rate 45%
I GAIN
0.35
I KNN-uniform
KNN-distance
0.30
0.25
RMSE
0.20
I
0.15
T
0.10
breast
cradit
letter
spam
wine red
wine white
mydatal
mydata2
breast
credit
letter
spam
wine red
wine white
mydatal
mydata2
Figure 3: MNAR scenario experiments for all datasets (except news dataset). (a) Left panel: MNAR setting with average missing rate of 20%. (b) Right panel: MNAR setting with average missing rate of 45%. For the high variability in the imputation RMSE of the spam dataset, Appendix D.2.
7 SUMMARY, DISCUSSION AND MESSAGE
7.1 SUMMING UP
MisGAN was initially designed to impute missing rectangular pixel blocs images. While Mis- GAN claims state-of-the-at results for image imputation, this framework does not perform well for tabular data imputation, with large confidence intervals in its predictions. That being said, MisGAN remain competitive well behaved datasets with low missing rates. GAIN has been developed

---

Under review as a conference paper at ICLR 2022
for tabular data imputation and shows good imputation quality, especially in MNAR settings with high missing rates. Like most ANN models, tuning the hyperparameters of MisGAN and GAIN is key to avoid over-training or under-training Adversarial Network architectures can learn complex relationships between variables. This feature explain the performances of GAIN in MAR and MNAR settings, where the network could find meaningful patterns in the missingness.
Unlike lazy learning algorithms like KNN, Generative
MCAR an assumption that can be tested (as opposed to MAR assumption). Little's Test of Missing Completely At Random @Litle (1888) can be used to assess whether the MCAR hypothesis holds. The MAR and MNAR missing data scenarios used in this study are strong cases of dependence between the data distribution and the missing data mechanism. Lighter dependencies exist (see supplementary materials for GAIN, Yoon al. (2018)). Choosing a missing data mechanism subjective (except for MCAR) and obviously affects the subsequent data imputation quality.
This study shows that the use of massive deep-learnigg algorithms are not justified for tabular data imputation, and reinforces previous results: [Bertsimas et al. (2018), Jadhav et al. (2019), PPoulos & Valle 20. Jäger et al. (2021). Tabular data imputation problems do not require complex machine learning al algorithms as it was already established by Batista & Monard (2002).
Lastly, it appears that the imputation quality of an algorithm rather depends on the dataset itself than on the missing data mechanism per better than GAIN, and mydata2 (mixture of three Gaussian dataset) where GAIN performs better than KNN most of the time. For this reason, recommend to start any data processing with thorough investigation of the dataset at play, like we show in Appendix B.
the breast dataset where KNN systematically performs
7.2 DISCUSSION ON MACHINE LEARNING
The development of ANN along with "Deep Learning" is unquestionably an important milestone in Data Science. For example, Convolutional Neural Networks (CNNs) have shown outstanding results in image processing where they can outperform most other traditional methods when properly used. But as for tabular data imputation, traditional and well- cell-established statistical methods are not yet obsolete. There is also inherent limitation in data imputation problems: are left with what the (observed) data is. Consequently, it is particularly challenging to recover missing outliers.
On one hand, KNN algorithms require distance matrix between each observation in the dataset to be computed. This step can take a couple of minutes if the dataset is large (see Appendix E). Once computed, we can instantaneously try any desired number of neighbours for the imputation, which make KNN easily to fine tune. Also, KNN is a deterministic algorithm that can be straight- forwardly reproduced and its results are easy interpret. On the other hand, takes several minutes to train GAIN or MisGAN (for 20,000 and 5,000 epochs respectively, on dataset with 1,000 ob- servations). Fitting massive ANN models can become time and resource consuming. In addition, the ANN "black-box"" architecture is often criticised. Stochastic imputations can be useful to derive probability distributions, but prevent from unambiguous replicability. Table [ gives a summary of this comparison. For more details on the computational training time, see Appendix E.
Table 2: Comparison and recapitulation
GAIN
MisGAN
KNN
Image imputation
No
Yes
No
Tabular data imputation
Yes
Inconsistent
Yes
Training time
(for 1000 observations)
190 sec
80 sec
< sec
Optimal hyperparameter search
Need to train again from scratch with new hyperparameters
Exhaustive and instantaneous
"Black-box" model
Easy to interpret
Results
Stochastic imputation
Deterministic imputation
Reproducibility
Algorithmically complex
Straightforward
8

---

Under review as a conference paper at ICLR 2022
Advances in Machine Learning tend showcase several srate-of-thh-ane algorithms, although only one may exist by definition. Exhaustive comparative studies allow to share common benchmarks and fairly assess the performances of each method.
Finally, Machine Learning comes alongside with the improvement computing power, smaller processors and digital era with huge amount of data. But this era is also associated with over- consumption, resources problems and alarming ecological consequences. Digital sobriety advocates for using the Internet and technologies in a mindful and responsible way. In light of it, we urge researchers in Machine Learning to develop and promote appropriate and reasonable tools, adapted the data and problem that we aim address.
7.3 TABULAR DATA ANALYSIS RECIPE
Acknowledging and handling missing values is of major importance in data analysis. Since most Data Science algorithms cannot handle missing values, a straightforward solution consists in delet- ing incomplete observations. We strongly discourage from doing so: the reason why data is missing may tell lot about an observation, and a missing value could even provide more information than an actual value, if utilised properly.
goes without saying that a great understanding of the dataset the most important factor for successful data analysis. We suggest to follow these steps before any tabular data analysis with missing values:
1. Do not discard incomplete observations.
2. If possible, handle individual missing values by hand.
3. Look for pattern in the missingness, and understand the missing data mechanism.
4. Test whether the MCAR assumption holds. If not, assess (empirically) the MAR assump- tion seems valid.
5. Perform the imputation with a suitable algorithm.
KNN imputation is almost instantaneous while deep-learning models take time and energy to be trained. For its simplicity, reproducibility and intuitive interpretation, we encourage to use KNN for data imputation when possible.
ETHICS STATEMENT
We assure that this review is the authors' own original work and reflect the authors' own research and analysis in a truthful and complete manner. The developers of the algorithms used in this study appropriately credited.
We also declare no conflict of interest or discrimination bias in this study. This work reflects fairness concerns and responsible research practices. We are grateful the developers of GAIN for their valuable input regarding the hyperparameters' tuning of their algorithm.
REPRODUCIBILITY STATEMENT
The code and the data used in this work are provided as supplementary materials in a zip folder. We have used pseudorandom number generator seed, that interested readers can faithfully reproduce all the plots of this work. We also welcome readers to change the seed to assess the statistical consistency of our results.
REFERENCES
N. S. Altman. An introduction kernel and nearest-neighbor nonparametrie regression. American Statistician, 46, 1992. ISSN 15372731. doi: 10. 10.108000003110...9922 10475778.
Gustavo E. A. A. Batista and Maria C. Monard. A study of k-nearest neighbour as an imputation method. Frontiers in Artificial Intelligence and Applications, 87, 2002.

---

Under review as a conference paper at ICLR 2022
Kristin P. Bennett and O. L. Mangasarian. Robust linear programming discrimination of two linearly inseparable sets. Optimization Methods and Software, 1, 1992. ISSN 10294937. doi: 10.1080/ 10556789008855004.
Dimitris Bertsimas, Colin Pawlowski, and Ying Daisy Zhuo From predictive methods to missing data imputation: An optimization approach. Journal of Machine Learning Research, 18, 2018. ISSN 15337928.
David A. Binder. On variance estimation with imputed survey data: Comment. Journal of the American Statistical Association, 91, 1996. ISSN 01621459, doi: 10.2307/2291639.
Peter Clark and Tim Niblett. The cn2 induction algorithm. Machine Learning, 3, 1989. ISSN 15730565. doi: 1000022/
Paulo Cortez, António Cerdeira, Fernando Almeida, Telmo Matos, and José Reis. Modeling wine preferences by data mining from physicochemical properties. Decision Support Systems, 47, 2009. ISSN 01679236. doi: 20000000
D. Dua and C Graff. Uci machine learning repository: Data sets. Irvine, CA: University of Cali- forna,, School of Information and Computer Science,, 2019. URL hattp://archiie ics.
uci edu/ml
Kelwin Fernandes, Pedro Vinagre, and Paulo Cortez. proactive intelligent decision support system for predicting the popularity of online news. Lecture Notes in Computer Science (including sub- series Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics), 9273, 2015.
ISSN 161 13349, doi:
Evelyn Fix and J. L. Hodges. Discriminatory analysis. nonparametric discrimination: Consistency properties. International Statistical Review / Revue Internationale de Statistique, 57, 1989. ISSN 03067734. doi: 10.2307/1403797
Peter W. Frey and David J. Slate. Letter recognition using holland-style adaptive classifiers. Machine Learning. 6, 1991. ISSN 0885-6125. doi: 10.1007/bf00114100
Andrew Gelman and Jennifer Hill. Data Analysis Using Regression and Multileve//Hieerarhhica Models. Cambridge University Press, 2006. doi: 10. 101 10000///b970001100000 1790942.
Anil Jadhav, Dhanya Pramod, and Krishnan Ramanathan. Comparison of performance of data im-
putation methods for numeric dataset. Applied Artificial Intelligence, 33, 2019. ISSN 10876545. doi: 10.1080008839514
Sebastian Jäger, Arndt Allhorn, and Felix Bießmann. A benchmark for data imputation methods. Frontiers in Big Data, 4, 2021. ISSN 2624909X. doi: 0..338/ftdaaa.2221.633677
Graham Kalton and Daniel Kasprzyk, The treatment of missing survey data. Survey methodology, 12, 1986.
Ranjit Lall. How multiple imputation makes difference. Political Analysis, 24, 2016. ISSN 14764989, doi: 10. 10.1093/pan/mpw020.
Steven Cheng Xian Li, Benjamin M. Marlin, and Bo Jiang. Misgan: with generative adversarial networks. 7th International Conference on Learning Representations, ICLR 2019, 2019.
Learning from incomplete data
Roderick J.A. Little. A test of missing completely at random for multivariate data with missing values. Journal of the American Statistical Association, 83, 1988. ISSN 1537274X. doi: 10. 1080/01621459.1988 10477872..
Roderick J.A. Little and Donald B. Rubin. Statistical analysis with missing data. Statistical Analysis with Missing Data, 2014. doi: 00.1002/97811100111
Jason Poulos and Rafael Valle. Missing data imputation for supervised learning. Applied Artificial Intelligence, 32, 2018. ISSN 10876545. doi: 10. (180000839514.2000 2018.14488 1448143.
10

---

Under review as a conference paper at ICLR 2022
Steven L. Salzberg. C4.5: Programs for machine learning by ross quinlan. morgan kaufmann publishers, inc., 1993. Machine Learning, 16, 1994. ISSN 0885-6125. doi: 10.1007//f00999309..
I. Cheng Yeh and Che hui Lien. The comparisons of data mining techniques for the predictive accuracy of probability of default of credit card clients. Expert Systems with Applications, 36, 2009. ISSN 09574174, doi: 10.
Jinsung Yoon, James Jordon, and Mihaela Van Der Schaar. Gain: Missing data imputation using generative adversarial nets. 35th International Conference on Machine Learning. ICML 2018, 13: 9042-9051, 2018.

---

Under review as a conference paper at ICLR 2022
Supplementary materials
A APPENDIX: GAIN AND MIISGAN ARCHITECTURES
Table and Table Table44pesse present the architecture of both the generator and the discriminator of GAIN and MisGAN models respectively. Following the architecture of GAIN (Yoon et al. (2018)), we adapted the input and output layers to the number с of columns in the dataset.
Table 3: Neural network architecture for GAIN
Layer
Input shape
Output shape
Nb. params
Fully connected
2c
64
128c + 64
Fully Connected
64
128
8320
Fully Connected
128
128
16512
Fully Connected
128
129c
Table Neural network architecture for MisGAN
Layer
Input shape
Output shape
Nb. params
Fully connected
128
128c + 128
Fully Connected
128
128
16512
Fully Connected
128
128
16512
Fully Connected
128
C
129c
B APPENDIX: DATASETS DESCRIPTION
Real-world and well-behaved simulated datasets both used in this comparative study. Seven real-world datasets are taken from the open access UCI Machine Learning Repository (Dua Graff) (2019)) Because an exhaustive comprehension of the data at play the most important preliminary step before any data analysis, this appendix provides essential details about the data used in this work. Note that all following datasets are complete: they do not show missing values in the first place. The first two columns of Table [ shows the dataset names and sizes.
B. BREAST CANCER WISCONSIN (DIAGNOSTIC) DATA SET ARRAST
Ten features are computed from a digitised image a fine-needle aspiration of breast mass. They describe characteristics of the cell nuclei shown in the image (Bennett & Mangasarian (1992)). Fine- needle aspiration diagnostic procedure which consists in sampling and examining cells under microscope (biopsy), in the search for lumps or tumours.
The dataset contains 596 observations and 32 columns. The first two columns are the ID number and the diagnosis of the cell (malignant benign). The last 30 columns comprised of the mean, the standard error and the mean of the largest three values of the following ten cell features: radius, texture, perimeter, area, smoothness, compactness, concavity, concave points, symmetry and fractal dimension. These last 30 columns are kept for data imputation comparison.
B.2 DEFAULT OF CREDIT CARD CLIENTS DATA SET CREDIT
Default payments of 30000 customers in Taiwan have been recorded over six months. The dataset comprises the following explanatory variables: total amount of given credit, gender, education, mar- ital status, age, past payment records, amount of bill statement and amount previous statements Yeh & hui Lien (2009))
12

---

Under review as a conference paper at ICLR 2022
Regarding data imputation purposes, the categorical socio-demographic variables (gender, education and marital status) and the past payment records are removed. In the end, the dataset includes 30000 rows and 14 columns.
B.3 LETTER RECOGNITION DATA SET LETTER
Black and white images of the 26 uppercase letters from the roman alphabet produced using 20 different font styles and random distortions. Sixteen features have heen computed on the original images such that the resulting dataset comprises 20000 rows and 17 columns, where the first column tells the corresponding letter and is therefore discarded for this study. The following 16 columns contain integer values ranging from 0 to 15, that we treat as continuous. These 16 columns include features like the height and length of the image in pixels, the number of black pixels in the image, the horizontal and vertical average of black pixels or higher statistical moments (Frey & Slate (991)).
B.4 ONLINE NEWS POPULARITY DATA SET NEWS
A large number of online news articles published by Mashable have been collected over the course of two years (Fernandes et al. (2015)). The actual articles can be seen been accessing their corre- sponding URL link provided in the dataset. A series of 58 features have been computed on the news articles. They include 44 continuous variables like number of words in the title and 14 binary variables indicating whether the article has been published a Monday, for example.
The dataset has 39644 rows and 61 columns. Three additional columns have been discarded for data imputation purposes. They include the URL link of the article and the time delay between the news article publication and the data acquisition (both of which are non-informative) as well as the number shares of the news article (which the original target variable).
B.5 SPAMBASE DATA SET SPAM
Features are extracted from both spam and non-spam e-mails. The dataset comprises 4601 obser- vations and 58 columns corresponding to the following attributes. The first 48 attributes the frequency (in percent of words in the e-mail) preselected words like "will", "you" or "free". The next six columns are the frequency (in percent) of the following preselected characters: ";' 'C. ',, "r'''"' 'S', "#' The next three columns show the average length uninterrupted sequence of uppercase letters, the length of the longest uninterrupted sequence of uppercase letters, and the total number of uppercase letters in the e-mail respectively. Finally, the last column indicates whether the corre- sponding e-mail was considered spam not and has been discarded for data imputation purposes.
B.6 WINE QUALITY DATA SET WINE_RED AND WINE.WHITE
Various variables have been collected for red and white variants of the Portuguese Vinho Verde wines (Cottez et al. (2009). The red wines dataset contains 1599 observations while the white wines one has 4989 observations. Both datasets have 12 columns corresponding to the following attributes: fixed acidity, volatile acidity, citric acid, residual sugar, chlorides, free sulfur dioxide, total sulfur dioxide, density, pH, sulphates, alcohol and a final score between 0 and 10 for the wine quality.
B.7 GAUSSIAN DATASETS MYDATA1 AND MYDATA2
Two Gaussian datasets have been generated for comparison. We referred them as "well behaved" datasets, in the sense that they do not show outliers as the other seven real-world datasets. Also, their intrinsic probability distributions are perfectly determined. Both Gaussian datasets have 1000 rows and 5 columns. Correlation matrices with high coefficients have been obtained by using random factors (see Appendix F).
The first Gaussian dataset comprises a single multivariate Gaussian distribution (see pairplot, Fig. 4). The second Gaussian dataset has been generated with a mixture of Gaussians from independent correlation matrices obtained with with random factors (see pairplot, Fig, (5). The first class (in blue) has 150 observations, the second class (in orange) has 300 observations and the third class (in green) has 550 observations.
13

---

Under review as a conference paper at ICLR 2022
Figure 4: Multivariate Gaussian dataset mydatal
ABMAA
Figure 5: Mixture of 3 Gaussians dataset mydata2
14

---

Under review as a conference paper at ICLR 2022
C
APPENDIX: HYPERPARAMETERS SELECTION
C.I
CHOOSING THE NUMBER OF EPOCHS FOR GAIN AND MISGAN
Mixture of three Gaussian dataset [1000x5]
0.24
[
0.22
0.20
H
0.18
RMSE
0.16
0.14
0.12
0.10
GAIN
I MisGAN
0.08
2500
5000
7500
10000
12500
15000
17500
20000
Nb epochs
Figure 6: Imputation RMSE for both GAN models (MisGAN and GAIN) on mydat a2 dataset in the MCAR scenario with 20% missing rate. We select the optimal number of epochs to be 5,000 for MisGAN and 20,000 for GAIN.
See Fig. While the imputation performances for GAIN consistently decreasing with low variance when we increase the number training epochs, MisGAN shows poor performances with high variability after 7,000 training epochs (notice the large standard deviations). Consequently, we respectively select 20,000 epochs and 5,000 epochs for GAIN and MisGAN hyperparameters. As the RMSE steadily decreases from 1,000 to 20,000 epochs in the case of GAIN, we decide to investigate more training epochs. See Appendix C.2 below.
C.2 ADDITIONAL NUMBER OF EPOCHS FOR GAIN
See Fig. If there is clear improvement from 10,000 to 20,000 epochs, there is significant change between 20,000 and 100,000 training epochs for GAIN imputation performances. We decide stick with 20,000 training epochs (per 1,000 observations) for GAIN that the computational cost does not become prohibitive.
C.3 CHOOSING THE NUMBER OF NEIGHBOURS FOR KNN
See Fig. Both KNN algorithms have better imputation quality when we increase the number of neighbours from about 50. KNN-uniform shows better imputation quality than KNN-distana for the generated mixture of Gaussians dataset with MCAR setting at 20% missing rate. KNN- uniform becomes worse than KNN-distance when the number of neighbours becomes large. Indeed, with the distance weight system, KNN-distance is less sensitive to newly included poor predictors when the number of neighbours becomes unnecessarily large.
Despite the global minimum being around 70 neighbours for KNN-uniform and 120 for KNN- distance, we decide to select K = 50 neighbours (per 1,000 observations) as the hyperparameter for both KNN algorithms. The reason behind this choice is that the RMSE plateaus large number of neighbours and its standard deviation is too large to justify meticulous fine-tuning on the number of neighbours.
15

---

Under review as a conference paper at ICLR 2022
Mixture of three Gaussian dataset [1000x5]
f
GAIN
0.12
0.11
RMSE
0.10
20000
40000
60000
80000
100000
Nb epochs
Figure 7: Imputation RMSE for additional training epochs with GAIN on mydata2 dataset in the MCAR scenario with 20% missing rate. There is no significant improvement after 20,000 epochs on the imputation RMSE in spite of a large computation cost increasc.
Mixture of three Gaussian dataset [1000x5]
KNN-uniform
KNN-distance
0.16
0.14
RMSE
0.12
0.10
0.08
50
100
200
250
300
Nb neighbaurs
Figure Imputation RMSE on mydata2 dataset MCAR scenario with missing rate with KNN models while varying the number of neighbours. The optimal hyperparameters seems to lie close to N=50 neighbours.
C.4 VERIFICATION OF THE HYPERPARAMETER SCALING PROCESS
We see in Fig. g and 10 the imputation RMSE for GAIN with respect to the training epochs on the left panel, and for both KNN algorithms with respect to the number of neighbours on the right panel. the wine-white tt dataset first, the scaling process introduced
Looking at Fig. 9 corresponding 5000 x 1000 s 1000 epochs, and 50 x TOOD
in Section 5.1 recommends to use 4898
4898
~ 250 neighbours.
We see that these approximations do not corresponds to the optimal hyperparameters. The optimal hyperparameters correspond to 5000 epochs for GAIN and approximately 25 neighbours for KNN. Now looking at Fig. [0] which uses the breast dataset. the recommend values obtained by the scaling process correspond to the optimal values for the algorithms hyperparameters. Indeed, for
16

---

Under review as a conference paper at ICLR 2022
White dataset Shape (4898, 12)
0.090
0.120
IM55
RM55
0.115
0.000
0.110
1000
2000
3000
4000
3030
50
100
150
200
250
300
No spoctss
No niihhtour
Figure 9: Imputation RMSE for the wine-white white dataset in MCAR scenario with 20% missing rate depending the hyperparameter selection. The scaling process gives approximately 4000 epochs for GAIN and 250 neighbours for the KNN algorithms.
Breast dataset
Shape (569, 30)
I
GAIN
Kin-unffom
0.33
KAn-cistance
0.085
0.11
0.090
ONSE
0.10
0.080
0.09
0.075
0.08
2300
5000
7500
10000
12500
15000
17500
20000
50
100
150
200
250
300
ND epochs
neighbours
Figure 10: Imputation RMSE for the breast dataset in MCAR scenario with 20% missing rate depending on the hyperparameter selection. The selected scaling process gives approximately 10,000 epuchs for GAIN and 25 neighbours for the KNN algorithms.
GAIN the number of epochs obtained with the scaling process is frac5000569 1000 9000 and for KNN the computed number of epochs is 50 number epochs for GAIN is 10,000 epochs, and the optimal number of neighbours for the KNN algorithms is 6 (which is lower than our estimate of 28).
369 1000 = 28. Looking at the RMSE, the optimal
Even if see that the scaling process does not always yield optimal results, we decide to perform the whole analysis of this work following this scaling method. The reason behind this choice is that an extensive hyperparameter search is not only extremely time and energy consuming, but also impedes unambiguous reproducibility.
D APPENDIX: SKIPPED OVER PERFORMANCES
D.1 POOR IMPUTATION QUALITY WITH MISGAN
Fig. shows the imputation quality the four algorithms presented in this study (including Mis- GAN), on the mixture of three Gaussians dataset with varying missing rates in MCAR setting. The imputation quality of MisGAN rapidly deteriorates when missing rate increases. Given the poor performances of MisGAN, we decide not to include any of its results in the study for clarity. This can be explained because MisGAN has been developed as a deee-learning architecture tailored for dete- riorated images restoration, while this work focuses tabular data. We refer the interested reader to Li et al. (2019) for further details on MisGAN, Imputations with MisGAN have been conducted with all experiments of this work (see Fig. 12 but the results plotted for clarity.
17

---

Under review as a conference paper at ICLR 2022
Mixture of three Gaussian dataset [1000x5)
H
GAIN
T
1
MisGAN
0.5
H
KNN-uniform
H
KNN-distance
0.4
RMSE
0.3
H
0.2
I
0.1
E
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
Missing rate
Figure 11: Imputation RMSE on mydata2 dataset in MCAR setting for all data imputation methods with varying missing rates. The performances of MisGAN are particularly poor in comparison to the other three methods.
D.2 CHAOTIC IMPUTATION PERFORMANCES FOR THE NEWS DATASET
Fig. 12 presents the imputation performances of the four algorithms in this study for the seven real-world datasets in MCAR setting with 20% missing rate. We see that MisGAN systematically performs worst, and we therefore decide not to include its results for other experiments.
Missing rate 20% (MCAR)
f GAIN
1.2
T MisCAN
10-
KNN-unniform
KNN distancce
0.8
RASE
0.6
0.4
0.2
breast
credit
letter
hews
sparn
wine red
Figure 12: Imputation RMSE for all real-world datasets (including the news dataset) in the MCAR scenario with 20% missing rate. The performances the four selected algorithms extremely variable on the news dataset.
We note the huge variability in the imputation performances for the four algorithms over the news dataset. The reason is that the news dataset has few extreme outliers, whose prediction will be particularly bad when randomly hidden before scaling the remaining data between [0, 1]. The same phenomenon happens on the spam dataset in the MNAR setting with an average missing rate of 45%: extreme outliers consequently more likely to become missing values in the hiding process (because of the chosen MNAR setting using quantiles).
We would like to mention that the news and spam datasets not really suited for data imputation problems. Using descriptive statistics (not presented in this work), we see that these two datasets are sparse with lot of zeroes, and have extreme outliers. This is not problem per se, but think that
18

---

Under review as a conference paper at ICLR 2022
manual treatment of missing values in such datasets is more appropriate. Unfortunately, they have been extensively used to benchmark Machine Learning algorithms. We decided to include them in this comparative study since they have been used by Yoon et al. (2018).
E APPENDIX: QUANTITATIVE ANALYSIS ON THE TRAINING TIME
Recall that the training strategy consists of using 20,000 and 5,000 training epochs per 1,000 ob- servations for GAIN and MisGAN respectively, and using 50 neighbours per 1,000 observations for both KNN algorithms. Table TableB]sshoe shows the average training time for the algorithms used in this study. KNN-uniform and KNN-distance having really similar training procedures, with grouped them into common column.
Table Average training time per dataset four algorithms in this study
Dataset name
Size
GAIN (in Sec.)
MisGAN (in Sec.)
KNN (in sec.)
breast
(569, 30)
221.9
88.6
0.0
credit
(30000, 14)
194.3
86.9
44.8
letter
(20000, 16)
196.9
85.2
24.6
news
(39644, 44)
213.3
129.4
210.7
spam
(4601,
220.5
89.0
3.3
wine.red
(1599, 12)
200.0
82.8
0.1
wine_white
(4898, 12)
197.1
82.8
1.1
mydatal
(1000,
190.5
80.4
0.2
mydata2
(1000, 5)
190.1
80.9
0.0
The training time for GAIN and MisGAN remains nearly constant across datasets. This is because of the multiplicative factor (introduced in Section 5.1) used to keep similar proportion when training the ANN models. To trian GAIN for example, which has 1,000 observations, but we use only 1,000 training for the letter dataset which shows 20,000 observations.
20,000 training epochs for the dataset mydata a
However, the training time for the KNN algorithms is strongly dependent on the dataset size. Indeed, computing the distance matrix between each pair of rows can become computationally expensive for large datasets. For the largest dataset, news, it takes about 3 minutes and 30 seconds to compute the distance matrix. That being said, there no need compute the distance matrix again when tuning the hyperparameter K of the number of neighbours used for imputation, while we need to retrain GAIN or MisGAN from scratch if we change the number of training epochs, their hyperparameter in this case.
F APPENDIX: RANDOM FACTOR METHOD FOR CORRELATION MATRICES
Generating random correlation matrices can be easily done by transforming a random square matrix into a symmetric positive definitive matrix. In our case, we draw the entries of from a standard Gaussian distribution, and we compute AT A symmetric positive definite matrix. However, this method yields matrices with non-diagonal elements close to zeros.
To obtain symmetric definite positive matrix of dimension n with higher correlation coefficients outside the diagonal, we start by choosing a number factors 0 < < n. We draw a rectangular matrix A of size (k, n. whose entries are independently sampled from standard Gaussian distribu- tion. We compute the matrix AT A which is symmetric of size (n, n), but not full rank. We draw diagonal matrix D whose entries have random positive noise, such that W A + D is invertible.
The code generate the Gaussian data used is this study is provided as supplementary material.
G APPENDIX: NUMERICAL IMPUTATION RMSE RESULTS
19

---

Under review as a conference paper at ICLR 2022
Table 6: Numerical results corresponding to Fig. RMSE are given percents (%)
(a). The mean and standard deviation of the imputation
Missing rate
GAIN
KNN-uniform
KNN-distance
MisGAN
10%
9.9 (= 0.59)
7.05 (# 0.33)
7.8 (= 0.61)
13.98 (# 2.81)
20%
9.82 91 0.69)
8.47 (# 0.42)
10.57 ( 0.71)
28.98 (#: 2.07)
30%
11.1T 0.63)
10.34 (= 0.51)
12.83 (#: 0.64)
39.09 (#: 1.83)
40%
12.33 (# 0.71)
(2
14.5 0.62)
46.07 (# 1.55)
50%
13.63 (# 0.48)
13.89 (= 0.49)
15.5 0.55)
49.62 (# 1.53)
60%
13.74 (4 0.57)
16.07 (= 0.45)
16.75 0.41)
51.57 (# 1.16)
70%
18.07 (# 0.64)
19.27 (= 0.76)
18.26 (# 0.56)
53.36 (# 1.4)
80%
21,47 (# 0.74)
23.49 = 0.66)
20.63 (# 0.56)
54.16 (# 1.75)
Table 7: Numerical results corresponding to Fig. RMSE are given percents (%)
(b). The mean and standard deviation of the imputation
Dataset name
GAIN
KNN-uniform
KNN-distance
MisGAN
breast
9.39 (+ 0.77)
8.46 (= 0.47)
8.35 (= 0.47)
13.01 (= 0.96)
credit
( 0.28)
6.1 0.2)
6.2 0.22)
8.33 (# 0.74)
letter
13.44 (3.4 0.21)
11.27 (# 0.05)
10.09 (1.09 0.07)
22.32 (# 1.26)
news
63.58 (+: 65.42)
60.83 (+ 67.0)
60.69 (+ 67.09)
66.56 (+: 63.94)
spam
7.38 (# 0.72)
6,05 (#E 0.31)
6,06 (# 0.29)
7.64 (# 0.66)
wine_red
12.07 (#: 0.43)
11.11 (# 0.4)
9.78 (# 0.4)
15.98 (E 0.89)
wine_white
10.44 (# 0.49)
9.92 (EE 0.46)
8.65 (#E 0.45)
15.2 (E 1.49)
Table Numerical results corresponding to Fig. RMSE are given in percents (%).
(a). The mean standard deviation the imputation
Dataset name
GAIN
KNN-uniform
KNN-distance
MisGAN
breast
14.35 (# 1.14)
9.63 (# 0.88)
9.49 (EE 0.87)
15.66 1.11)
credit
5.21 (E 0.35)
5.15 (1: 0.36)
5.4 0.4)
7.56 (Z 0.43)
letter
13.04 0.14)
11.26 0.07)
11.42 0.1)
27.34 1.18)
news
77.51 (#: 68.59)
75.47 (# 70.0)
75.34 (#: 70.09)
82.9 (#: 65.24)
spam
7.03 (#: 0.49)
6.32 0.44)
5.73 (#: 0.42)
7.62 0.52)
wine_red
12.16 0.48)
(+
9.71 (# 0.48)
18.83 (# 1.23)
wine-white
10.43 (# 0.41)
9.86 (#: 0.43)
8.89 (#: 0.43)
17.22 (+ 1.05)
mydatal
9.9 (# 0.5)
9.5 (EE 0.28)
10.03 (E 0.34)
21.76 (# 1.61)
mydata2
9.71 (+ 0.68)
10.05 (+ 0.47)
13.35 (+ 0.73)
25.18 (+ 1.86)
Table 9: Numerical results corresponding to Fig. RMSE are given percents (%).
2 (b). The mean and standard deviation of the imputation
Dataset name
GAIN
KNN-uniform
KNN-distance
MisGAN
breast
21.26 (+ 1.14)
14.43 (# 0.72)
14.07 (+ 0.7)
29.22 (+ 2.29)
credit
7.02 (+ 0.44)
7.15 (+ 0.36)
8.06 0.44)
11.48 (+: 0.91)
letter
14.45 (+ 0.33)
12.99 (+: 0.14)
13.04 (+ 0.14)
31.61 (+ 1.06)
news
56.19 (=: 36.05)
55.07 (#: 36.81)
55.55 36.48)
62.73 (#: 32.42)
spam
9,89 (=t 0.7)
7.9 0.55)
9.74 0.45)
8.69 (#t 0.67)
wine.red
14.83 (+ 0.94)
12.97 (# 0.49)
14.6 0.58)
26.91 (# 1.12)
wine_white
12.92 (# 0.54)
12.39 0.6)
13.65 (# 0.68)
22.94 (# 1.44)
mydatal
11.6 (# 0.47)
11.25 (+ 0.38)
11.78 (# 0.34)
34.02 (+ 1.82)
mydata2
14.8 (= 0.59)
17.33 (# 0.38)
19.96 (# 0.52)
37.31 (# 1.55)
20

---

Under review as a conference paper at ICLR 2022
Table 10: Numerical results corresponding to Fig. RMSE are given in percents (%)
(a). The mean and standard deviation of the imputation
Dataset name
GAIN
KNN-uniform
KNN-distance
MisGAN
breast
14.75 (# 0.98)
12.14 I.I)
11.98 ( 1.11)
20.97 (#: 1.69)
credit
7.85 (# 0.44)
8.1 (= 0.29)
8.17 (= 0.3)
11.2 (+ 0.73)
letter
13.29 (= 0.18)
12.83 (# 0.04)
11.91 (# 0.06)
29.74 (# 1.08)
news
130.86 (=: 60.0)
130.55 (+: 60.37)
130.5 (# 60.42)
134.06 (+: 57.62)
spam
9.15 (+ 0.32)
8.88 (+ 0.37)
8- 0.32)
9.98 (+ 0.42)
wine red
13.18 (# 0.83)
13.9 (# 0.82)
0.82)
24.33 (# 1.5)
wine_white
I1.59 (#= 0.37)
11.86 (# 0.48)
10.6 (# 0.45)
22.15 (#: 1.67)
mydatal
9.98 (#: 0.54)
10.5 (# 0.53)
11.38 (= 0.54)
27.86 (# 3.4)
mydata2
9.91 (# 0.56)
9.99 (# 0.63)
12.01 (# 0.65)
27.24 (# 3.06)
Table 11: Numerical results corresponding to Fig. RMSE given in percents (%)
(b). The mean and standard deviation of the imputation
Dataset name
GAIN
KNN-uniform
KNN-distance
MisGAN
breast
30.3 (# 1.52)
25.41 (# 1.23)
25.13 ( 1.24)
38.35 (+: 1.73)
credit
12.14 (# 0.55)
13.06 0.51)
13.09 (# 0.51)
15.99 (+ 0.84)
letter
17.36 (+ 0.59)
18.95 (# 0.2)
19.1 0.21)
32,23 ( 0.66)
news
158.89 7.9)
158.97 (+ 7.89)
159.0
161.34 (+ 7.86)
spam
23.95 (=: 12.63)
23.2 (+: 12.84)
23.11 12,85)
23.69 (+ 12.68)
wine-red
19,9 (#E 1.03)
23.6 (E 0.97)
25.02 (E 1,17)
31.94 (EE 1.44)
wine white
17.16 (+t 1.15)
20.24 1.12)
21.4 ( 1.18)
30.16 (#: 1.82)
mydatal
13.69 (# 0.95)
(1.8800.00 0.67)
17.33 (# 0.77)
35,03 ( 2.31)
mydata2
15.04 (# 0.8)
25.38 (+ 1.19)
24.84 ( 1.03)
38.8 (EE 2.58)
21


Reviews:
[{"review_id": "_e0lKm9pkSV", "reviewer": "Reviewer_m2kd", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "This paper considers the problem of data imputation, which is important in handling tabular data, and shows interesting results regarding imputation methods, with some potential practical implications regarding the efficient methods to use. However, there are some concerns regarding the presented results:\n- The authors state that they did not manage to reproduce the results of GAIN from the original paper, although they used the same datasets (in one case it was much better in their measurement, in the others worse). They also do not provide reference results for any of the other classical algorithms that appeared in the original paper (such as MICE and MissForest). It is therefore unclear whether their setting was correct and whether this comparison was valid. \n- Since datasets behave differently, it is unclear whether setting the hyperparameters based on one small dataset is the right approach. Note that GAIN is better than KNN on this dataset (my_data2), possibly due to some over-fitting, but not as good on the other datasets. It may be better to set the hyperparameters based on several datasets with different mechanisms of missing data. Maybe there could also be some fine-tuning on a subset of the relevant dataset (although it has missing data), rather than assuming a linear scaling based on the number of samples. Also, it should be noted that more hyperparameter options were checked for KNN than for GAIN.\n- The difference between the algorithms is shown in figures, but it is not quantified. The paper should state what is the average difference between GAIN and each of the KNN methods, per dataset, per scenario, and overall.\n- The paper uses a small number of datasets, and all the datasets used have a small number of features. KNN is known to suffer from the \"curse of dimensionality\". It would therefore be interesting to see if this result holds for a couple of datasets with hundreds of features or more. It would also be interesting to see the results for a dataset with a larger number of samples.\n\n", "summary_of_the_paper": "This paper considers the problem of tabular data imputation to handle missing numeric data. It compares the RMSE of recently proposed GAN-based methods (GAIN and mis-GAN) with the classical KNN method (either through uniform or distance-weighted imputation). It considers 9 datasets, two are simulated by a mixture of Gaussians and 7 are real datasets, 5 of them used in the GAIN paper. The datasets have 500-30000 samples and 5-57 features (most are below 20). It also considers three different mechanisms of missing data (MCAR - completely random, MAR - random based on the observed data, MNAR - based on unobserved data), each with one or two rates of missing data (more rates are tested for MCAR with the simulated data). The hyperparameters of all the models are tuned once on the simulated data (1000 samples), and scaled based on the number of samples for each dataset (scaling the number of neighbors for KNN, or epochs for the GANs). The models are applied to each imputation case 20 times, presenting the mean and standard deviation for each dataset in each scenario. The paper shows that mis-GAN, originally developed for filling missing image pixels, usually performs much worse than KNN. Furthermore, KNN generally performs roughly on-par compared to GAIN, depending on the dataset and exact scenario (in many cases KNN is better). Based on the presented figures, the RMSE differences are usually within 0.02, but may reach 0.05. The compute required by KNN is much lower, and the paper calls for using the cheaper option, avoiding waste of resources.", "summary_of_the_review": "The paper considers an interesting problem and presents interesting results that encourage trying KNN. However, due to the concerns mentioned above regarding the correctness and representativeness of the results, and since it does not have a significant technical contribution other than comparing existing methods, it is below the bar for ICLR.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "KbZpTvLur6t", "reviewer": "Reviewer_zJAq", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "**List strong and weak points of the paper. Be as comprehensive as possible.**\n\n**Pros:**\n\n* All missingness scenarios (MCAR, MAR, MNAR) are taken into account.\n* The (real-world) datasets' shapes range from 12 to 57 attributes and 569 to 39644 examples and, therefore, cover a wide range of scenarios.\n* All datasets are fully-observed and the missing values are artificially induced. This helps to eliminate interfering effects.\n* Structure of the paper is clear\n\n**Cons:**\n\n* The paper claims to be \"the first to compare state-of-the-art deep-learning models with the well-established KNN algorithm\", which seems like a bit of a bold statement given the number of publications on empirical work in that space in the past years:\n  * papers that present imputation methods sometimes use KNN as baseline (see: <https://arxiv.org/pdf/1808.01684.pdf>)\n  * there is a number of papers at a recent ICML workshop (see <https://artemiss-workshop.github.io/>), where several papers presented empirical work on KNN imputations as well as more recent generative deep learning approaches\n  * there there is recent benchmark that compares both: <https://www.frontiersin.org/articles/10.3389/fdata.2021.693674/full>\n  * there is a recent paper comparing deep learning approaches with AutoML imputations, and KNN is one of the models used in that AutoML solution <https://arxiv.org/abs/2106.11189> - interestingly that publication finds that if regularized properly, deep nets can actually outperform more classical discriminative methods (such as KNN)\n* The algorithms hyperparameters are optimized on a synthetic dataset and then “scaled” to the dataset’s size. Hyperparameters should always be tuned on the same that that the models are applied to. In the very setting investigated by the authors, it is fair to assume that KNN, with fewer hyperparameters, has an advantage due to that procedure\n* an important experimental setting is whether the imputation model was trained on fully observed data (a relevant but rather academic/lab-conditions setting) or on data with missing values (which is more relevant in real-world applications). \n* Minor:\n  * Section 7.2 abbreviation \"CNN\" should be introduced\n\n\n**Clearly state your recommendation (accept or reject) with one or two key reasons for this choice.**\n\nI vote for reject since there are already comparisons between KNN and state-of-the-art models. Further, the hyperparameter optimization can explain the better performance of KNN. \n\n**Ask questions you would like answered by the authors to help you clarify your understanding of the paper and provide the additional evidence you need to be confident in your assessment.**\n\n* Although, the method used to induce MAR missingness is clear to me, I do not understand the column “MAR column” in Table 1.\n* Is there a good reason why experiment 1+2 differ from the third and fourth?\n\n**Provide additional feedback with the aim to improve the paper. Make it clear that these points are here to help, and not necessarily part of your decision assessment.**\n\n* Given an overview of the datasets' attributes data types would help to get more insights. Perhaps you should consider to take them into account for the discussion.\n* Add information about the data split or cross-validation used to find hyperparameters would increase the credibility.\n* Figures 3+4 and 5+6 could be side-by-side with the same y-axis; in my opinion, figures 1+2 too to support readability.\n* I personally prefere extensive figure captions.\n* Lastly, adding more \"simple\" imputation methods, e.g. random forests, neural networks, would be interesting, since there are some empirical evaluations that come to different results\n* the authors are right in that there are few peer reviewed papers at the big ML conferences that directly compare latest generative deep learninig methods with classical methods, like kNN. This is however in part true because in my experience these papers do not get accepted. I have been reviewing imputation papers for the past couple years and I've seen several cases where an empirical evaluation of this kind, even though the experimental setting was absolutely flawless, did not make it. ", "summary_of_the_paper": "The paper compares two generative deep learning based data imputation methods (GAIN and MisGAIN) with KNN based imputation. The authors conducted experiments on seven real-world and two synthetic datasets in  all missingness scenarios (MCAR, MAR, MNAR) and multiple missingness rates.", "summary_of_the_review": "The experimental setting has a significant limitation, the hyper parameter optimization is not conducted on the right data set, which can explain the superior performance of KNN. The authors claim that they are the first to report these results, which is not exactly true. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "2yWX-1SrQj", "reviewer": "Reviewer_VqfN", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strength:\n- The text is clear; the experiments are robust; the messages are important.\n- This paper has also an important message for the new method developers within the community suggesting to compare first their new methods with simple standard baselines. The authors might want to emphasize more this point in their summary.\n- This study presents a nice guideline for imputing missing values in tabular data.\n\nWeaknesses:\n- The experimental design can be improved. Currently, the authors mainly focused on the effect of missing data mechanisms (MCAR, MAR, MNAR) on the performance of data imputation strategies. This is while there are also other important avenues to explore. For example, the authors could also evaluate the effect of different hyperparameter settings on the performance of GAIN and KNN. Currently, we have such analysis in Appendix C but only for simulated data and a fixed number of samples (1000). What about the effect of hyperparameters when imputing real data?  I suggest focussing on one real dataset as a case study and evaluating the effect of hyperparameters (K in KNN and epoch number in GAIN) on the performance of imputation. My hypothesis is that K in KNN is less sensitive to sample size compared to the number of epochs in GAIN. It is also interesting to see that how the optimal value for the hyperparameters changes from one dataset to another. Furthermore, I wonder why the authors decided to include the misGAN in the main text (section 3.2) while its results are below the bar and only presented in the appendix. I suggest saving some space for extra analysis by moving all information about misGAN to appendices. \n- The introduction consists few strong (of course valid) arguments and sentences without proper referencing. For example 1) \"This solution, although straightforward to implement, has two main disadvantages: it can significantly reduce the size of the dataset and induce a bias [Ref.?].\" 2) \"It allows to preserve the whole dataset for subsequent analysis but requires careful handling as it can also introduce a bias in the imputed dataset [Ref.?].\" 3) \"In particular, only ”well-behaved” values can be imputed and missing values of outlier observations will get overlooked [Ref.?].\"\n- While the argument is to compare the recently introduced complex GAN-based imputation with their simpler classic alternatives, I wonder why the authors did not include the MICE method in their analysis. MICE performs in some ways similar to GAIN and would be interesting to see a comparison. Of course, MICE does not apply to the MNAR scenario (due to its MAR assumption).\n\nMinor comments and questions:\n- The last paragraph on page 1 is not informative and can be removed in favor of more space for extra analysis.\n- \"We argue that all real-world datasets have missing values distributed in MNAR settings; MCAR and MAR are convenient assumptions to facilitate the treatment of incomplete datasets.\" This is an unsupported argument that should be supported either by some empirical evidence or by including some references.\n- Sec 5: the data is scaled to interval [0,1]: is it only for outputs or includes also inputs? Should be clarified in the text because this rescaling might not be the best choice for the input features. I assume this is done only for outputs to make the resulting RMSE comparable across datasets. In that case, again I would suggest skipping rescaling and using instead standardized MSE (SMSE). It is a very minor comment though.\n- It is not clear how the other GAIN hyperparameters ($h$ and $\\alpha$) are decided. The authors say after several trials they have decided to use the defaults in the original study. Would be nice to include those trials in the appendix.\n- Page 3: \"parameters are trained trained following...\" => \"parameters are trained following...\"\n- Page 3: \"After several trails, ...\"=> \"After several trials,...\"\n- Some aspects of this study overlap with recent work at https://doi.org/10.3389/fdata.2021.693674 . I wonder why the authors did not mention this study.\n \n", "summary_of_the_paper": "This manuscript presents a nice empirical comparison between the classic KNN imputation with two state-of-the-art GAN-based imputations for tabular data imputation. The experiments are performed on both simulated and publicly available real data. The results, overall, show that the KNN, despite its simplicity, provides very competitive results when compared to its more complex and computationally expensive alternatives. In summary, this paper has a few important messages for both method developers and reviewers in the missing data community: i) first try the simplest; ii) more complex does not mean better; iii) understand your data before data imputation;  ", "summary_of_the_review": "This is a nice study with important messages for the community. The experimental section can be enriched with a more in-depth evaluation of the effect of hyperparameters on the performance of data imputation approaches.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper presents a study of methods for tabular data imputation. In particular, the authors compare deep learning methods with k-NN based approaches. Experiment results demonstrate k-NN to be competitive with deep learning methods.

Reviewers have concerns about the characteristics of datasets used in the experiments and hyperparameters used for evaluation, which I agree with. They also think that the main contribution of comparing k-NN and deep learning methods is not strong enough for acceptance to ICLR. I recommend rejecting this paper.
Decision: Reject