Numerical Data Imputation for Multimodal Data Sets: A Probabilistic Nearest-Neighbor Kernel Density Approach
Authors: Florian Lalande
Kenji Doya
Venue: Accepted by TMLR ()

Abstract: Numerical data imputation algorithms replace missing values by estimates to leverage incomplete data sets. Current imputation methods seek to minimize the error between the unobserved ground truth and the imputed values. But this strategy can create artifacts leading to poor imputation in the presence of multimodal or complex distributions. To tackle this problem, we introduce the $k$NN$\times$KDE algorithm: a data imputation method combining nearest neighbor estimation ($k$NN) and density estimation with Gaussian kernels (KDE). 
We compare our method with previous data imputation methods using artificial and real-world data with different data missing scenarios and various data missing rates, and show that our method can cope with complex original data structure, yields lower data imputation errors, and provides probabilistic estimates with higher likelihood than current methods. We release the code in open-source for the community.

Published in Transactions on Machine Learning Research (06/2023)
Numerical Data Imputation for Multimodal Data Sets: A Probabilistic Nearest-Neighbor Kernel Density Approach
Lalande Florian
florian @oist.jp
Neural Computation Unit
Okinawa Institute of Science and Technology 1919-1 Tancha, Onna-son, Okinawa, JAPAN
Doya Kenji
doya Doist.p
Neural Computation Unit
Okinawa Institute of Science and Technology 1919-1 Tancha, Onna-son, Okinawa, JAPAN
Reviewed on OpenReview: https: // openreview. net/ formm id= KgR3rgooXt
Abstract
Numerical data imputation algorithms replace missing values by estimates to leverage in- complete data sets. Current imputation methods seek to minimize the error between the unobserved ground truth and the imputed values. But this strategy can create artifacts leading to poor imputation in the presence of multimodal tackle this problem, introduce the &NNxKDE algorithm: data. imputation method combining nearest neighbor estimation (kNN) and density estimation with Gaussian kernels (KDE). We compare our method with previous data imputation methods using artificial and real-world data with different data missing scenarios and various data missing rates, and show that our method can cope with complex original data structure, yields lower data imputation errors, and provides probabilistic estimates with higher likelihood than current. methods. We release the code in open-source for the community
complex distributions. To
1 Background and related work
As sensors now ubiquitous and the Internet of Things has become widespread and found numerous applications, Big Data is often referred to as the "Gold of the 21st Century". However, along with the proliferation of numerical databases, missing data has become a pervasive problem: they can introduce a bias, lead to wrong conclusions, or even prevent from using data analysis tools that require complete data sets.
To mitigate this issue, data imputation algorithms have been developed. From the straightforward mcan/mode imputation (Little & Rubin 2014] to recent generative adversarial networks (GAN) models (Yoon el al.] 2018), wide range of tools are available to impule incomplete data sets. As the variely and specificity of available data imputation algorithms can be overwhelming for practitioners, flexible pack- ages like DataWig allow optimal imputation results by sweeping through several methods and automatically perform
tuning (Bießmann et al. 2019).
Data imputation mast popular application consists of recovering missing parts of an image, also known as inpainting. Deep learning methods have shown promising results for image inpainting and are therefore the preferred solutions for image recovery (Xiang et al.) 2023). However, typical image features differ from tabular data. This study focuses on tabular numerical data sets, that is numerical real-valued data arranged in rows and columns in form of matrix. For numerical data sets, recent benchmarks argue that deep- learning imputation methods do not perform better than simple traditional algorithms (Bertsinas el. al.
https://github.co http:///gttuuub.cc///taaaaaoooot

---

Published in Transactions on Machine Learning Research (06/2023)
2018
Poulos & Valle 2018 Jachav et al. 2019 Woznica & Biecek 2020 Jäger et al., 2021 Lalande & Doya
2022
Grinszlajn et al. 2022 These studies show that the &NN-Imputer (Troyanskaya et al. 2001 | and
MissPorest (Stekhoven & Bühlmann 2012). in spite of being simple algorithmss, generally perform better over a large range of data scts in various missing data scenarios. In the presence of linear dependencies, Multiple
Imputation using Chained Equations (MICE) and its variants (van Buuren & Groothuis-
2011
Khan & Hoque 2020) can show good imputation performances.
We denote R² the complete ground truth for observation in dimension D 2 2, and m {0, 1). the missing mask. The observed data is presented as ã O m, where @ denotes the element wise product. Data may be missing because it was not recorded, the record has been lost, degraded, or data may alternatively be censored. The exercise now consists in retrieving x from ã, while allowing incomplete data for modeling, and not only complete data.
The probability distribution of the missing mask, p(m) is referred to as the missing data mechanism (or missingness mechanism), and depends on missing data scenarios. Following the usual classification of Little and Rubin, missing data scenarios are split into three types (Little & Rubin 2014 missing completely at random (MCAR), missing at random (MAR) and missing not at random (MNAR).
In MCAR the missing data mechanism is assumed to be independent of the data set and we can write p(m]ax== = p(m)- In MAR, the missing data mechanism is assumed to be fully explained by the observed variables, such that p[m[(x) = p(m] The MNAR scenario includes every other possible scenarios, where the reason why data is missing may depend on the missing values themselves.
Numerical data imputation methods are usually evaluated using the normalized RMSE (NRMSE) between the imputed value and the ground truth. The higher the average NRMSE, the poorer the imputation re- sults. This approach is intuitive, but is too restrictive for multimodal data sets: it assumes that there exists a unique answer for a given set of observed variables, which is not true for multimodal distributions. For multimodal data scts, density cstimation methods like the Kernel Density Estimation (KDE) (Rosenblatt,
1956 Parzen 1962) appear of interest for data imputation. But despite some allempls (Titterington & Mill, Leibrandt & Günnemann 2018), density estimation methods with missing values remain computa- tionally expensive and not suitable for practical imputation purposes, mostly because they do not generalize well to real-world data sets in spite of an interesting theoretical framework.
1983
Alternatively, other works have developed Gaussian mixture density estimates with Expectation- Maximization (EM) training (Delalleuu et al.] 2012; McCaww et al.) 2020) as well as Gaussian processes for Kernel Principal Component Analysis (KPCA) (Sanguinetti & Lawrence 2006). but these methods also do not generalize well do heterogeneous numerical data sets in practice. Also, if the mathematical frame- work of the Missingness Aware Gaussian Mixture Models (MGMM) of McCCa et al. (2020) is interesting, it requires to manually search for the optimal number of Gaussians in the mixture, and is primarily focused on classification tasks. More recently, variants of collaborative filtering algorithms for Matrix Completion prob- lems have been developed (Lee et al.) 2016 Li ct al. 2020) and can be used for numerical data imputation as well. However, these methods do not seem to perform better than the traditional SoftImpute algorithm (Hastie et al. 2015) for Matrix Completion.
This work focuses on concurrently learning from incomplete data to model and recover missing numerical values. We first look at three simple data sets to illustrate the shortcomings of current data imputation methods with multimodal distributions. We address these issues by introdueing local density estimator that is flexible to accommodate multimodal data structures. By leveraging the convenient properties of the ANNNImpute and the KDE framework, we develop the kNNxKDE: a simple yet efficient algorithm for density estimation and data imputation of missing values in numerical data sets.
Using heterogeneous real-world and simulated data sets, we show that our method performs equally or better than tatt--of-the-art numerical imputation methods, while providing better density estimates for missing values. The code and data used in this work are provided in open-access for the community.
2

---

Published in Transactions on Machine Learning Research (06/2023)
2 Problems of current imputation methods with multimodal data sets
In this section, we illustrate problems of current numerical data imputation methods with multimodal data sets. For this purpose, we generate three synthetic data sets two-dimensional space and qualitatively dis- cuss the imputation performances of four stale-of-the-art numerical imputation algorithms with two bench- mark methods (column mean and column median).
2.1 Three synthetic data sets
The first. data set, called 2d
2d_ linear, is noisy linear distribution. 31 sampled from a mollified uniform distribution on [0, 1] with standard deviation 0.05. Then Z2 I1 E; where ~ (0,00.1)
4TU, where u is drawn from a mollified distribution on [0, 1] with standard deviation a = 0.05. Then 22 = sin 22 + e, where & ~ (O,, 0.2). The noisy surjection allows to show that most imputation algorithms perform well in the unambiguous case (when 22 is missing), but not with multimodal distributions (when æj is missing).
The second data set, 2d ra_sine, is a sine wave with noise. We sample
Finally, 2d ring displays ring with noise, has been generated polar coordinates where 0 ~ (1 2T) and 1.0 + E, with N(O, 0.1). Euclidean coordinates are cos f and T2 sin
These three simple data sets have N 500 observations and are plotted in Figure 1 The code used for generation and the data sets themselves are available on the online repository. We have used a mollified uniform distribution for 21 in 2d _i_erre and 2d_sine to prevent from zero likelihood computation problems at the edges of the uniform distribution.
linear
2d__ins sine
2d so_ring
0.5
0.0
0.0
0.5
1.0
0
10
i
Figure Three basic synthctic data sets with N 500 obsorvations. 2d 1 linear a bijection, 2d_sing sine is surjection, and 2d_ring displays ring and therefore not function in the euclidean space.
2.2 Five state-of-the-art numerical data imputation methods
Here, we present four data imputation methods used in this work: the KNN-Imputer, Forsso,est, MICE and GAIN, This choice is course arbitrary, but illustrates well the current state of affairs regarding tabular
data imputation (Bertsimas al. 2018 Poulos Valle 2018: Yoon et al.) 2018 Jadhav al. 2019 Woznical
& Biecek 2020 Jäger et al. 2021 Lalande Doya 2022 Grinsztajn al. 2022)
The ANN-Imputer "Troyanskaya al. 2001 computes distances between pairs of observations using the
NaN-Euclidean distance, which can handle missing values. It imputes missing cells one column at time by
averaging over the k nearest neighbors that have an observed value for the given feature. Therefore, different neighbors can be used to impute various missing entries for the same observation. The hyperparameter k for the number neighbors is to be optimized.
FissForest (Stekhoven & Bühlmann 2012) is an iterative imputation algorithm. MissForest starts by filling all missing values with initial estimates (typically the column mean), and loops through all columns,
3

---

Published in Transactions on Machine Learning Research (06/2023)
one at time, performing regression of that specific column onto all other columns using Random Forests. It stops when the imputed data set is stable enough (following user-defined threshold) or when a fixed number of iterations has been performed. The number of trees used in the Random Forest algorithm is the hyperparameter to be tuned.
MICE stands for Multiple Imputation Chained Equations (van Buuren & Groothuis-Oudsshone 2011 Similar to MissForest, is an iterative imputation algorithm. MICE strictly refers to the algorithmic method which consists of filling missing values using iterative series of regression models one variable time. In this work, use the standard version of MICE that uses linear regressions predict cach column successively. This algorithm has no hyperparameter to optimize. MICE has shown good imputation results and is appreciated for its simplicity and absence of hyperparameter tuning, but it fails at capturing non-linear dependencies.
regressor to
Ioft a matrix complction algorithm (Hastic et al., 2015). It works by finding a low-rank ap- proximation of the matrix with missing values while promoting sparsily through a regularization term with coefficient A. The algorithm uses an iterative procedure to minimize the objective function. In each iteration, the observed entries of the matrix are used to estimate the missing entries. The estimated entries are then used to update the low-rank approximation of the matrix. This process is repeated until convergence.
Finally, GAIN is GAN artificial neural network tailored for tabular numerical data imputation which claims shaate--ff-hh-are numerical data imputation results (Yoon et al. 2018 GAIN smartly revisits the GAN architecture by working with individual cells rather than entire observations. It has recently benefited from a lot of attention for numerical data imputation. However, recent. benchmarks show that its performances are
mediocre in practice (Jäger et al. 2021 Lalande & Doya 2022. Grinsztajn et al. 2022). GAIN has several hyperparameters to tune: batch size, hint rate (amount of correct labels provided to the discriminator)) number of training iterations, and weight parameter used in the generator loss.
2.3 Imputation results
We introduce missing values in MCAR setting with 20% missing rate. an observation has both features removed, we repeat the process until at least, one feature is present. After missing values have been inserted, we normalize the data set the range [0, 1] using min/max normalization.
For each data imputation algorithm and for each data set represented matrix of size (N, grid search of the hyperparameter than best minimizes the NRMSE:
D), we perform
N
D
NRMSE
ΣΣ ==1==1
[zij - 21r)2 (1-11mmm)) miy)
(1)
Nunise
where myy = 1 if cell (i.j) is observed (mi, 0 if missing) and Nuise EN Ep= mmy) is the total number of missing entries in the data set. Imputation results provided by the best hyperparameters are plotted in Figure
Figure provides a concise insight into the current state of numerical data imputation. The scientific con- sensus is that the kNN-Imputer and MissForest overall provide the best numerical data imputation quality, which is somewhat recovered here. MICE uses linear regression between features and cannot capture non- linear dependencies. SoftImpute uses low-rank matrix completion, hence the straight lines as well. Despite its flexible architecture, GAIN performs poorly, even on 2d, linear. GAIN, like all generative adversarial networks, is difficult to optimize because of training instabilities, mode collapse problems, potential impos- sibility to converge, or not well defined loss function (Saxena & Cao 2020).
Both the ANN-Imputer and MissForest average over several predictions. This is why the imputation of Ti for the 2d_sine data set lies between the two sine waves, and imputed values for both æ; and 22 for the 2d_ bd_ring data are inside the ring. While averaging over several predictions often leads better estimates, this strategy deteriorates the imputation quality if the missing values distribution is not unimodal.
4

---

Published in Transactions on Machine Learning Research (06/2023)
KNN--mputer
MissForest
MICE
Softimpute
GAIN
1.0
10-
1.0
1.0
P__innar
0.5
0.5
1.0
0.0
0.5
1.5
0.0
0.5
1.5
0.0
0.5
1.0
0.0
0.
10
22 nina
Figure 2: Imputation results for the three synthetic data sets by the four selected imputation methods with optimized hyperparameters. Missing dala have been injected in MCAR scenario with 20% missing rate. Blue dots correspond to complete observations; orange dots have observed T2 and imputed 21 red dots have observed 21 and imputed Ø2- The &NN-Imputer, MissForest and MICE perform well on 2d_linear. For 2d_ _d_singe sine, the kNN-Imputer and MissForest can impute 22, but fail at recovering 21. No method can properly impute 2d 2d_ring.
MICE performs imputation by assuming linear relations between features of the data Tt is therefore surprise that MICE can very well impute data set 2d linear, but fails at imputing data sets 2d_ sine and 2d ring. Similarly, SoftImpute uses linear combinations of the observed values as matrix completion algorithm.
GAIN provides surprisingly disappointing imputation results. While deep-learning models are flexible meth- ods, the generator and the discriminator of GAIN fail capture the relationship between ø1 and 22 in all data sets. Yet innovative, the complex architecture of GATN (and GANs is general) is problematic to train. This leads to bad imputation results well as large variability between runs.
3 The kNN*KDE algorithm
To address the above-mentioned issues related to multimodal distributions, we propose a local stochastic imputation algorithm inspired by the kNN-Imputer and kernel density estimation. We adapt the KDE algorithm missing data settings such that the conditional density of missing features given observed features is estimated.
We use methodology analogous to the KNN-Imputer to look for neighbors, but we work with missing pallerns instead of working column by column. The reason of this choice is that working with one column at a time may lead to imputation artifacts as the selected neighbors for various imputed features can different. Therefore, imputed observations may be incompatible with the original data structure. On the contrary, we are guarantced to preserve the original data structure if we impute all missing features of an observation at once.
5

---

Published in Transactions on Machine Learning Research (06/2023)
For data set with D columns, we have up to 2 2 possible missing patterns. Indeed, each cell may either be missing not (hence 2 choices) but we do not account for complete cases (nothing to impute) and completely unobserved cases (without even an observed feature).
We first normalize each column of the data set to fit within the range of [0, 1]. We refer to this process as the min-max normalization. For imputation the data in row i, we compute the distance di with all other rows j, using the distance
dig
Σ (Its Izk)² + Σ
(2)
DeDaba
D
where Dobs = [k E [1, | mi mik = 1l the set of indices for commonly observed features observations and j. Dmiss = [k [1, magMik = 0} is the set of indices for features where at least one observation or is missing, and o is the standard deviation of feature k computed over all observed cells. We call this distance metric the MaN-std-Euclies
Distance, in contrast to the original NaN-Buclideds Distance used by the &NN-Impuler (Dionn 1979). See Appendix D for discussion on this metric properties.
The pairwise distances are then passed to softmax function to define probabilities:
Pij = Σ5--us/-
(3)
We use the "soft' version of the &NN algorithm, and introduce the temperature hyperparameter T which can interpreted as the effective neighborhood diameter. Instead selecting a fixed number of neighbors per observation, we consider all observations but give nearest neighbors stronger weight. In similar fashion as Frosst al. (2019). the notion of temperature controls the tightness of each observation's neighborhood. See Appendix A.1 for a discussion the temperature hyperparameter.
Given missing pattern, we first select all rows to impute and all the rows corresponding to potential donors. The data to impute is the subset of data which has the current missing pattern, and potential donors are the subset of data whore at least all columns in the current missing pattern are observed. For an incomplete observation in the subset of data to impute, Pt is the probability of choosing observation j from the subset of potential donors. We have Σ,Pr
Pig = Algorithm 1 shows the pseudo-code of the xNNxKDE.
Algorithm 1: Pseudo-code for the ANNxKDE Hyper-parameters: Softmax temperature h; Kernel bandwidth h; Nb draws Naraws
Data: Incomplete numerical data set X
min-max normalization in the interval [O, 1]; for each missing pattern do
Ximp +- linta_tt_ipput (X, missing pattern);
Xdon potential donors (X, missing pattern); _eaatial_ddoors_se
d; NaN _td__ttedd_ddaaa Distance (Ximps X
Pij + softmax (-d;/T); for each row in Ximp do
+ sample Ndraws rows from Xdon with probabilities Ply - sample noise Naraws times from e e W(0,)) with dimcnsion K;
imputation Satptaaiiiinssssseeeeaeeee +- Xdon[r]
end end
min-max denormalization;
Return: emputations_samples

---

Published in Transactions on Machine Learning Research (06/2023)
The &NNxKDE has three hyperparameters:: the temperature for the softmax probabilities, the (shared) standard deviation h of the Gaussian kernels, and Ndrows the number of imputed samples to draw for each missing cell. The effects of these three hyperparameters are discussed in Appendix A
For observation with a missing value in column k, the probability distribution of the missing cell rae is given by
p(.ak) Σ Piy Pi (Zul)z5s;h)
(4)
where Pij are the softmax probabilities defined in Equation with Piş = if observation j is not in the subset of potential donors for observation i, and N (-Ip;)) denotes the density function of a univariate Gaussian with mean y and standard deviation a:
N (0l1;0)= = V2T02
e +(
If observation has K missing values, in columns k1, f2.... kg: then the subset of potential donors will likely be smaller and the joined probability distribution for all missing values is given by
p(Ziki Tike .. Tikk) )
Σ
Pij II N (aaa, (z)
(6)
where the index k rums from 1 to K to denote the successive missing columns, and Pi; = 0 if observation j is not in the subset of potential donors for observation like above. As can been seen from Equation G the weights Pij are shared such that imputed cells for the same observation have a joined probability that reflects the structure of the original data set.
Note that the pseudo-code of the ANNxKDE presented in Algorithm D uses Ndraws samples for each missing cell. We could instead use the softmax probabilities Pij as weights for the mixture of Gaussians with all potential donors, which would ideally lead to direct probability distributions. We have tried this approach but found that this requires a much larger computational cost, and is only tractable in practice with small data sets. We therefore contimue to sample Ndraws times to show the returned probability distributions of the &NNxKDE.
4 Results on the synthetic toy data sets
We show that the pseudo-code the algorithm presented the proposed method provides imputation samples that preserve the structure of the original data sets. For now, missing data are inserted in MCAR scenario with 20% missing rate, and the hyperparameters of the kNNxKDE are fixed their default values: h 0.03, 1/7 50.0 and Naraws 10, 000.
The upper panels of Figure 3 show the imputation with sub-sampling size No. 10. The sub-sampling size is only used for plotting purposes. If x1 is missing, we sample N possible values given 22 (see the orange horizontal trails of dots), and if 22 is missing, we draw Nas possible values given o (see the red vertical trails dots).
Of course, it is worth mentioning that if we decide to average over the returned samples by the xNDx then similar artifacts as the ones presented in Figure 2 will arise again. For instance, single point estimates for the 2d 2d_rrii_ ring data set will fall inside the ring.
Another way to visualize the imputation distribution for each missing value is to look at the univariate density provided by the kNNxKDE algorithm. For each data set, we have selected two observations: one with missing I1 and one with missing Zg. The lower panels of VigureBsshor the univariate densities returned by the ANN XDD algorithm with default hyperparametors. The upper left corner of each panel shows the observed value and a thick dashed line indicates the (unknown) ground truth to be imputed. We see that the ground truth always falls in one of the modes of the estimated imputation density.
7

---

Published in Transactions on Machine Learning Research (06/2023)
2d linear
2d sine
2d ring
1.2
1.5
1.0
1.0
0.8
0.5
2 0.0
0.4
0.2
-0.5
0.0
-1.0
-1.5
0.0
0.2
0.4
0.6
0.8
1.0
0.0
2.5
5.0
7.5
10.0
12.5
-1.0
-0.5
0.0
0.5
1.0
Densiry Xs
X2 0.11
2.
X2 =.101
0.4
-1.0
-0.5
X1
X1 0.94
- 11.77
X1 -0.511
0.6
0.7
0.8
0.9
1.0
1.5 1.0
0.0
0.5
1.5
1.0
0.5
0.5
1.0
X2
Figure 3: Imputation results from the BNNNKDE algorithm on the three synthetic data sets. Missing data are inserted in MCAR setting with 20% missing rate, Each missing entry has been imputed by the kNNxKDE with default hyperparameters Nos = 10 times for plotting purposes. The imputed values follow the structure of the original data sets. The histograms in the lower panels have Ndraws = 10000 samples. Thick dashed lines correspond the (unobserved) ground truth and the observed value is shown in the the upper-left corner. The kNNxKDE returns probability distribution for each missing cell which captures the original data multi-modality structure.
For the 2d_sine data set, when I1 missing (central middle panel of Figure the ANN*KDE returns a multimodal distribution. Indeed, given the observed T2 = -0.88, three separate ranges of values could correspond to the missing 21. Similarly, the 2d_ farring data shows bimodal distributions both for ø1 or 22, corresponding to the two possible ranges of values allowed by the ring structure.
5 Performances on heterogeneous data sets
Now, we assess the practical performances of our method on larger data sets, using both synthetic and real-world data sets from UCI and other repositories. See Appendix for a comprehensive description of the data sets.
We present imputation results using metrics: Subsection 5.] presents the normalized root mean square errors (NRMSE) commonly used for comparing numerical data imputation methods; Subsection 5.2 shows the mean log-likelihood score of the (unknown) ground truth under each imputation model computed over the normalized data in the range [0, 1] for fair comparison. In both cases we test four missing data settings: 'Ful MCAR', 'MCAR', 'MAR', and 'MNAR', and six missing rales: 10%, 20%, 30%, 40%, 50%, and 60%. While Full MCAR includes missing data from multiple columns as defined in Section 1, 'MCAR" assumes only one column missing, as Jääer et al. (2021). See Appendix C for missing data sccnario details. For each data set, each missing data setting, and each missing rate, we repeat the imputation NB_ REPEAT-20 times to compute the mean and the standard deviation of the chosen metric.
8

---

Published in Transactions on Machine Learning Research (06/2023)
5.1 Imputation results with NRMSE
This subsection presents the imputation results evaluated by the NRMSE, defined in Equation 1 For the LNN-Imputer, FossForest, MICE, the Mean, and the Median imputation schemes, we use the implementation provided by the Python package sklearn" Pedregosa et al. (2011). ). For GAIN, we use the original GitHub repository of the authors of GAIN Yoon el al. (2018). As the original package for Sopul is in R, we use a more recent Pythor implementation provided by Muzellec et al. (2020).
When minimizing the NRMSE for a given data set, a given missing data scenario, and a given missing rate, we perform a hyperparameter search except for MICE, Mean, and Median imputation methods, which do not have hyperparameters. We consider the following lists for the other methods' hyperparameters:
For ANNxKDE the inverse lemperature 1/7 E [10, 20, 50, 100, 250, 500, 1000]
For the ANN-Imputer, the number of neighbors kt E [1, 10, 20, 20,50, 100]
For MissForest, the number of regression trees Nirees E 2,, 5, 10, 15, 20]
For SoftImpute, the regularization term À E [0.1, 0.2, 0.5, 1.0, 2.0,5.0, 10.0]
For GAIN, the number of training epochs Niter. [100, 200, 400, 700, 1000, 2000, 4000]
When computing the NRMSE for the NDE, we impute with the imputation sample mean.
Tables 2 and show the mean imputation NRMSE for each method and each data set with the missing rate 20%. For each data set, the top three methods that achieve lowest imputation NRMSE have been colored in green, ycllow, and orange. We provide the numerical results for the 20% missing rate case as this is often the default missing rate for tabular data imputation benchmarks. The results for all missing rates are available in the online repository for this project.
In order to provide more concise overview of the imputation NRMSE results, we rank the proposed methods from 1 (best.) to 8 (worst) for each data set. For example, looking at the 4th row of Table D we have for the geyser data set in Full MCAR setting with 20% missing rate: the kNN-Imputer (1), the
(2), MICE (3), MissForest (4), SoftImpute (5), GAIN (6), Mean (7) and Median (8). Now, for each missing data setting and each missing rate, we compute the mean and the standard deviation for each method ranks over the 15 data sets. Figure shows the average rank for each method.
These results reinforce the previous reports that Deep Learning methods do not perform better than tradi- tional methods on tabular numerical data sets (Bertsimas et al. 2018 Poulos & Valle, 2018 Jadhav et al. 2019 Woznica & Biecek 2020 Jäger et al. 2021 Lalande & Doya, 2022 Grinsztajn et al. 2022). The proposed kNNxKDE consistently achieves the best rank, tightly followed by MissForest. The KNN-Imputer and MICE come next. SoftImput,, GAIN, the column Mean, and the column Median are always in the group of the four last methods.
Rankings for MissForest show large error bars because this method is confident in the provided imputation. In other words, the performances of MissForest can vary lot depending on the nature of the data set (see the standard deviation in the reported NRMSE results in Tables I to 4 Alternatively, the ANNxKDE and the kNN-Imputer have lower rank error bars, indicating that these mothods are more consistent across data sets. This can been seen in the lower NRMSE standard deviation in Tables to 4
It is worth noting that the KNN* seems to suffer from the curse of dimensionalityy, especially in the 'Full MCAR' scenario high missing rate. Looking at Table [ the xNNxKDE has higher NRMSE compared to other methods for the breast and the sylvine data sets. Indeed, in Fuul MCAR' scenario at high missing rates for data sets in high dimension, the subset of potential donors for a specific missing pattern can be very low, or even empty therefore preventing from sampling.
While Figure 4 provides the overall ranks, note that the imputation NRMSE can vary greatly between two consecutive ranks. Using again the abalone data set NRMSE provided in the first row of Table 1 to
1 http::///ssskkt.eemr.rrr//l/llld
http:///gtthbbbcmm//rmoo222/1111 http:///ghhhaaccooooooiiiaaaaala
9

---

Published in Transactions on Machine Learning Research (06/2023)
NRMSE rankings
1
kNNxKDE
Full MCAR ranks
2
kNN-Imputer
MissForest
Softimpute
5
E GAIN
E MICE
6
Mean
Median
8
10%
20%
30%
40%
50%
60%
1
2
3
MCAR
8
10%
20%
30%
40%
50%
60%
1
2
MAR ranks
3
4
6
7
8
10%
20%
30%
40%
50%
60%
2
3
MNAR
5
6
10%
20%
30%
40%
50%
60%
Missing rate
Figure 4: Average NRMSE rank for each data imputation method in various missing data settings and missing rates. Our proposed method, the xNNxKDE, is consistently the best method, regardless of the missing rate of missing data setting. Second comes MissPorest. The kNN-Imputer and MICE come next. Besides the column mean median numerical imputation methods, GAIN and SoftImpute invariably under-perform every other methods.
exemplify, the NRMSE remains below 4.00 for the top four methods, then jumps to 5.29 for GAIN, and finally gets close 15.00 for the column Mean and Median imputation methods.
Finally, stress that though the KDE overall provides minimal NRMSE, the framework used here computes the distribution mean to return point estimate. Calculating point estimate brings back the original problem of choosing a single estimate lo impute missing data, depicted in Figure 2 For instance, looking at the 2d_ring data set Table [ we see that the KNNxKDE does not perform much better than the ANN-Imputer or MissForest, which are considered stato-of-tho-art numerical imputation methods. Therefore, we decide to also measure the performances of the imputation methods with the log- liglliill score.
10

---

Published in Transactions on Machine Learning Research (06/2023)
5.2 Performances by log- regliiklihood score
Next, we look at the log- loglliiel of missing values under the probabilistic model provided by each method. For the ANN x KDDE a probability distribution for each missing cell is obtained as described in Section 8 and illustrated in Section g For the ANN-Imputer, we compute the mean and the standard deviation of the k selected neighbors, and calculate the log-likelihood of the ground truth assuming a Gaussian distribution. Similarly for the Mean imputation method, we compule the column mean and standard deviation and assume a Gaussian distribution. For MICE and MissForest, the stochastic nature of these two Iterative Imputer methods allows us to repeat the imputation N = 5 times, compute the mcan and the standard deviation for each missing value, and calculate the log- ligllikeihoodd of the ground truth assuming Gaussian distribution.
Despite being a generative model, GAIN systematically returns a unique value once trained, such that the variability in GAIN's predictions cannot be taken into account. Therefore, we decided not to include GAIN for the likelihood comparative study. We also do not consider the column Median anymore, as we already use the Mean for likelihood computation. We finally discard Ioftum from this section as well because it showed mediocre performances on the NRMSE rankings and there is no straightforward way to implement probabilistic version of the SoftImpute algorithm.
When computing the log-likelihood, we do not perform hyperparameter tuning for MissForest, the ANNxKDE, and kNN-Imputer. Instead, we choose the hyperparameter that best minimized the imputation NRMSE in the previous subsection.
Following the same approach as Subsection 5.1 we present the average log-likclihodd scores (computed over all missing cells) for each data set and each method with 20% missing rates in Tables 6 and S The numerical results for other missing rates are available online.
In similar fashion as before, we compute the ranks of the proposed methods using the mean log-likelihood for each missing data scenario and missing rate. For example, looking at the abalone data set in Full MCAR mean log-likelihood provided in the first raw of Table 5 we have the following rankings: ANN-Imputer (1); kNNxKDE (2), MICE (3), Mean (4), and MissForest (5). We average the ranks over all 15 data sets, and present the aggregated results in Figure
The kNNxKDE provides the overall best mean log-likelihood score, and the ANN-Imputer comes next. Tn Full MCAR' missing data sctting high missing rates, the &NN-Imputer model returns a higher likelihood score Ihan the >NNx A tentative explanation is that high missing rates in 'Full MCAR' missing setting create sparse observations from which sampling with the softmax probabilities of the LNNxKDE can become challenging. In contrast, the ANN-Imputer uses independent Gaussian distributions for cach column, which may lead to better results when a lot of cells are missing. On a similar note, notice how the column Mean provides greater log-likelihood scores than the MICE algorithm at high missing rates in 'Full MCAR' scenario.
As before, the KNNxKDE algorithm can suffer from high missing rates in 'Full MCAR' scenario for high dimensional data sets (see Table 5 for instance) as the subset of potential donors can be small, or even empty. But contrary to the NRMSE case, the likeliol score is not as severely affected. Data sets that exhibit a multi-modality structure tend to have much better log- lielliieldood score results under the XNNxKDE probability distribution. These data sets can be identified by looking at the average Dip test p-value for the test of unimodality (sec Appendix B
Despite MissForest showing interesting results with the NRMSE as performance metric, it now always scores last when averaging over multiple data sets. This is because the estimates provided by MissForest have low variability over different runs. As a consequence, the standard deviation used for the Gaussian distribution to model the probability distribution for each missing cell is small, and the resulting shape of the probability distribution is therefore very narrow. In the rare cases where the ground truth falls within 1o or 2o of the mean provided by MissForest, the likelihood will be high; but in most cases, the ground truth more than 3 away from the MissForest mean, therefore leading to small likelihood of the (unknown) ground truth under the Fosssorest model.
11

---

Published in Transactions on Machine Learning Research (06/2023)
Log-likelihood rankings
KNNxKDE kNN-imputer MissForest
Full MCAR ranks
MICE
3
Mean
5
10%
20%
30%
40%
50%
60%
MCAR ranks
2
4
5
10%
20%
30%
40%
50%
60%
1
MAR ranks
2
3
5
10%
20%
30%
40%
50%
60%
1
MNAR ranks
2
5
10%
20%
30%
40%
50%
60%
Missing rate
Figure 5: Average log-likelihood scores rank for each data imputation method in various missing data settings and missing rates. The kNNxKDE ranks best in all cases, except for the Full MCAR scenario at high missing rates, where the kNN-Imputer is best. MissForest consistently returns the lowest log-likelihood score, because its predictions do not allow for much variability.
Looking at Tables 5 to we see that for 20% missing rate, the ANNxKDE provides the best log-likelin logelikelihood score, especially for data sets with smaller dimension. As mentioned carlier, both the &NN and the &NN-Imputer can suffer from the curse of dimensionality because of the computation of the Euclidean distance in large dimensional spaces.
6 Discussion
This work proposes the XDE, new approach using a soft version the ANN algorithms to derive weights for the Kernel Density Estimation method. The ANNxKDE has been developed for numerical data imputation, especially for low dimensional data sets in the presence of multimodality complex
12

---

Published in Transactions on Machine Learning Research (06/2023)
dependencies. Here, we discuss of the limits and strengths of the ANN*KDE, conclude work, and provide directions for subsequent works.
6.1 Limits & Strengths
A substantial drawback is that the kNNxKDE becomes computationally expensive in the presence with large data sets. However, it remains faster than MissForest in practice, since it works with missing patterns instead of looping through the data set column by column. This strategy enables to compute only necessary pairwise distances. See Appendix
L for quantitative results on computation time.
Another drawback of the KNN*KDE is that cannot impute certain data sets with too many features in FFull MCAR and when the missing rate is high. Indeed, in 'Full MCAR' scenario with 60% missing rate for instance, the subset of potential donors (sce Algorithm 1 may be empty. In such cases, working column-by-colunn basis, like the standard KNN-Impuler, may be interesting solution.
Now, the great advantage of the KNN is that it preserves the original data structure, which is of major importance when working with multimodal data sets. Our method returns an imputation sample that provides information about the missing data distribution, which is better than a point estimate. By working with missing patterns and imputing all missing features at the same time, the kNN XDDE provides a sample of entirely imputed observations that are consistent with the original data set, which is not the case with Iterative Imputation methods (like MissForest and MICE) the INN-tmp
Finally, even though our method consistently achieves the average best imputation NRMSE in all missing data scenarios and at all considered missing rates (see Figure 4. using the sample mean of the returned imputation samples brings the original problem with multimodal distributions back. Looking at the 2d_ faringg data set in Table g we see that the &NNxKDE, does not perform better than other methods because of the imputation sample mean. However, we see on Table 0 that the kNN*KDE is the only method capable of providing good density estimation (and therefore high log-likelihood score) for the 2d od_ring data set. interested
This problem essentially boils down to asking why imputation is needed in the first place: in subsoquent downstream regression or classification tasks; or are we solcly interested in estimating missing values? The common approach of first imputing and then performing downstream tasks may be sub-optimal depending on the choosing imputation strategy (Le Morvan et al. 2021). Instead, the conditional probability distributions returned by the &NNxKDE allow to postpone the decision of imputing or not to a later stage. Imputation can subsequently be performed freely: with the mean (to minimize the root mean square error), with the mode (to minimize the absolute mean error), by random sampling (which would prevent from artifacts in the presence of multimodal datasets), with any other relevant statistic.
6.2 Future work
We decided to derive kernel version the traditional kNN-Imputer, and developed the proposed &NNxKDF. Alternatively, it could be interesting to look into another kernel method (or at least any other way to perform density estimation) using Random Forests, since MissFoorest achieves good results even in its current form.
Another possible extension of this work would be to include an end-to-end treatment of categorical variables within the framework of the kNE. As this study makes use of numerical imputation methods that cannot handle categorical features (e.g. GAIN SoftImpute), we decided to exclude categorical variables from the scope of this paper. However, tabular data imputation can include numerical and categorical variables in practice and further work may be needed in this direction.
Finally, the Edn-Eucideann metric. A possible explanation is that this new metric penalizes sparse observations (with a large number of missing values) by using the feature standard deviation when the entry is missing, therefore preventing to use artificially close neighbours for imputation (see Appendix D). Further investigation of this metric, and experimental results with the standard kNN-Imputer may yield interesting insights.
metric appear to yield better results compared to the commonly used
13

---

Published in Transactions on Machine Learning Research (06/2023)
6.3 Conclusion
The motivation behind this work was to design algorithm capable of imputing numerical values in data sets with heterogeneous structures, In particular, multimodality makes imputation ambiguous, as distinct values may be valid imputations. Now, minimizing the imputation RMSE is an intuitive objective for numerical data imputation, it does not capture the complexity of multimodal data sets. Instead averaging over several possible imputed values like traditional methods, the kNNx offers look at the probability density of the missing values and choose how to perform the imputation: sampling, mean, median, etc.
Ultimately, this work advocates for qualitative approach of numerical data imputation, rather than the current quantitative one. The online repository for this work provides all algorithms, all data, and few Jupiter Notebooks to test the proposed method, and we recommend trying it for practical numerical data imputation in various domains.
Acknowledgments
We would like to thank the three anonymous referees for their time and helpful remarks during the review of our manuscript. In addition, we would like to express our gratitude to Alain Celisse and the good people at the SAMM (Statistique, Analyse et Modélisation Multidisciplinaire) Seminar of the University Paris 1 Panthéon-Sorbonne, for their insightful comments during the development of the ANNxKDE.
This rescarch was supported by internal funding from the Okinawa Institute of Science and Technology Graduate University to K. D.
14

---

Published in Transactions on Machine Learning Research (06/2023)
Table 1: Imputation NRMSE (in %) with 20% missing rate in Full MCAR scenario. MissForest and the ANNx XDDE overall perform best
Median
0.94
+ 1.11
0.87
31.34 1 1.92
23.28 1 1.01
0.21
17.29 0.91
14.98 + 0.35
0.19
23.40 0.21
14.22 1. 0.47
0.33
0.07
20.08 0.14
0.71
at minimizing the NRMSE, followed by MICE and the &NN- Imputer. GAIN does not compete against other numerical data imputation methods.
Median
1.57
= 1.13
+ 1.41
21
= 2.51
0.64
18 04 1.35
16.60 = 0.03
20.57 n.24
19.36 36 = 0.32
15.80 = 0.71
8. 8.38 0.88
16. 80 0..0
= 0.64
17.09 J. 1.44
24.49
26.36
29.74
14.28
18 18.90
11 11.28
16.21
21 66.66
26 09
29. 47
36. 18
23. 23.55 55.5
14. 8L.5
14. 1.93
15.
24.42 + 0.91
26.29 + 1.11
0.81
29.02 1 1.10
22.78 1 0.83
0.21
2
14.87 + 0.35
0.11
21.94 + 0.11
14.00 1 0.46
11.19 0.54
0.07
19.62 1 0.13
15.29 + 0.08
= 1.36
26.05 + 1.13
1.40
33 03. + 1.27
23.02 + 2.04
14.85 1. 0.64
17.29 29 + 0.95
16.37 + 0.48
20.52 + 0.24
18.98 + 0.29
15.45 + 0.70
8.37 + 0.89
16.7 + 0.30
14.62 0.57
16.60 1: 1.29
Mean
Mean
29 60
14.28
15.98
17.88
16.18
21 62.6222
29.4
Table 2: Imputation NRMSE (in %) with 20% missing rate in MCAR scenario.
7.63 0.34
25 22 + 0.82
29 + 0.84
12.74 + 7.
15.19 1 0.61
0.21
10.22 + 0.77
3,86 + 0.33
0.32
9.24 0.12
10.08 1 0.43
8.98 + 0.30
11.49 0.08
17.61 1 0.13
5. 5.59 0.32
0.46
24.64 + 1.12
1.40
14.76 1.18
11.01 1 1.27
4.10 4.10 0.15
23 0.37
+ 0.19
6.35 + 0.11
0.32
5.51 1.13
14.62 = 0.57
0.31 1 0.04
MICE
MICE
6.30 +
10. +
10.13
10.93
6.80
29.
2. 20
5.48
20.73 5.62
26.82 + 1.46
30.21 # 1.92
21.95 + 7-43
18.37 1 1.02
13.61 + 1.58
12.07 + 0.80
5.29 + 0.70
14.13 + 1.81
11.61 + 1.52
12.03 1 0.61
10.81 + 0.63
13.35 1 0.45
19.01 1 0.20
7.44 = 0.51
13.81 + 0.43
26.13 + 1.21
29.54 1.49
27.72 4 11.60
14.09 + 2.00
9.11 2.08
4.02 = 1.00
8.29 = 0.81
1.32
8.83 = 1.07
7.97 = 1.06
0.70
0.00
2.99 1 0.51
GAIN
GAIN
9. 9.69
9.05
15.43
14.
Soft Doftumeute
11.82 0.77
43.24 + 1.97
.7. # 1.39
17.81 + 0.07
66 1 1.54
17.54 + 0.28
1.17
6.34 + 0.19
14.04 + 0.14
0.19
12.49 1 0.43
U.37
14.32 1 0.07
18.84 1 0.09
5.79 0.31
Soft Coftlmmute
12. 25.45 1.39
42.85 = 2.77
43.50 + 1.10
= 1.31
15.04 0.42
0.83
4.12 + 0.13
8.82 +: 0.16
3.02 + 0.23
7.69 + 0.34
3.58 4: 1.46
0.31
0.40
0.75 0.14
14.21
15.38
11.51
21.05
13.09
10.10
13. 13.26
15.92
Miss Forest
9.79 0.58
25 29 1.57
60
12.98 + 1.14
14.47 1 0.78
10. 0.20
9.16 + 1.02
3.32 + 0.39
6.58 + 0:26
8.58 1 0.30
7.30 + 0.36
16.98 1 0.14
6.39 + 0.54
MissForest
8. 0.03
8.63 + 0.75
40.61 2.42
1.18
9.94 + 0.90
4.49 1. 0.19
6.97 + 0.37
2.54 = 0.16
1.82 +: 0.08
4.55 + 0.09
5.04 + 0.45
4.72 1. 0.58
15.21 = 0.53
0.87 1. 0.32
5.35 =
7.49 1
38, 33
12.54
6.96
ANN- Imputer
7.72 + 0.41
19.21 + 0.96
29. 77 + 0.84
10.77 0.90
13.61 1 0.75
11.82 + 0.19
11.19 + 0.77
3.73 + 0.36
10.01 + 0.17
10.50 1 0.44
3,75 1 0.41
18.20 1 0.14
8.28 + 0.60
ANN- Imputer
0.83 83 0.50
7.26 - 0.70
29. 29.64 + 1.39
23
9.20 = 0.08
4.92 1. 0.26
7.58 0.57
2.54 = 0.17
2.02 + 0.30
4.61 0.08
6.36 + 0.32
6.21 1. 0.67
6.02 0.21
14.70 + 0.57
1.37 1. 0.45
6.60 +
8.78 1
11.01
XKDE
7.63 + 0.30
18.85 + 0.94
29 67 + 0.53
10.78 + 0.93
12.99 1 0.65
10.41 + 0.21
9.77 + 0.07
3.44 + 0.30
5.85 0.36
5.47 = 0.00
9.78 1 0.47
8.64 0:42
7.97 + 0.00
18.62 1 0.13
9.17 = 0.59
ANN x
6. 80 = 0.49
6.94 + 0.57
+ 1.42
3.00 + 0.33
1.83 1. 0.24
7.60 7,60 + 0.32
2.51 + 0.17
1,92 # 0.09
4.6 6.69 + 0.08
5.43 + 0.35
5.43 4: 0.65
5.34 + 0.15
+ 0.38
0.49
KNN
29.53 .2.5
10.89
14.69
1.39
Dim.
2
2
2
4
6
7
11
11
12
20
30
Dim.
2
2
2
2
4
5
6
7
8
11
11
12
20
30
japanese vowels
japanese yowels
2d 1inear
24 sine
2d. ring
geyser
penguin
pollen
p planets
abalons one
sulfur
gauss lans
wine _red
wine white
sylvine
breast
2d linear
24 sine
2d. ring
geyser
penguin
pollen
planets
abal one
sulfur
gaussian ans
wine red
wine white
sylvine
breast
15

---

Published in Transactions on Machine Learning Research (06/2023)
22. 12.18 1.50
1.06
= 1.30
2:90
2.
14.61 =: 0.68
14. 4.97 0.95
15.35 = 0.42
21.16 = 0.24
17.60 = 0.31
15.15 0.92
8.64 1 1.11
16. 17 0.26
14. 84.7 0.38
18.36 = 1.40
22. 08 1.85
1: 1.18
= 1.13
2.75
26.33 + 1.95
15.00 = 0.67
15.37 1.07
15. 48 0.59
21.27 = 0:21
18. 0.38
18. 18.29 29 =: n.72
10.04 = 1.41
16. 16.92 0.34
85 0.34
19. = 1:33
Median
Median
25. 25.78 79
28 62
30 12.32-22
26. 79
26 26.96 66.96
29. 42
31 11.33
21.85 + 1.48
+ 1.05
= 1.37
31.43 # 1.58
24.82 + 2.05
14.63 + 0.68
15. 15.90 + 0.87
15.97 - 0.43
20.62 + 0.24
17.90 + 0.29
15.07 = 0.74
8.63 1 1.10
16, 16.43 +. 0:27
14.64 0.36
17.51 + 1.28
1.70
1: 1.04
29. 29.09 = 1.09
32.33 + 0.98
24.26 + 1.70
15.07 - 0.68
16.60 + 0,02
16.11 1. 0.62
20.62 # 0.18
18.90 + 0.40
16.84 + 0.73
9.65 = 1.36
16.99 + 0.35
0.26
18. 18.59 + 1.21
Mean
Mean
25. 25.80
28.56
21.70
26. 26.52
13.65
Table 3: Imputation NRMSE (in %) with 20% missing rate in MAR scenario.
Table 4: Imputation NRMSE (in %) with 20% missing rate in MNAR scenario.
6.82 0.37
1.12
28. = 1.37
13.91 # 1.02
11.30 + 0.94
4.07 1 0.17
7.91 0.7.3
5.79 0.122
6.43 4.122
0.31
6.07 T 1.58
10.10 1 0.20
0.36
0.31 1 0.04
6.67 0.38
24.67 1. 1.13
29. 09 = 1.09
14.06 + 1.19
11.64 + 1.08
4.11 0.18
0.86
1 0.20
+ 0.10
0.35
5.99 # 1.45
10.17 0.24
13.65 0.26
0.34 0.05
MICE
MICE
24.40
2.61
5 5.25
14.64
8.59
2 2.68
6.01
5.97
5.92
7.46
1.03
+ 1.64
82.82 4 13.20
3.73
8.94 = 2.11
9.50 = 1.32
8.74 = 1.53
8.61 = 0.92
8.57 = 1.59
8.00 1 1.09
14.46 1 1.44
0.72
2.92 = 0.23
14.92 7:50
26.43 1: 1. 1.20
29. 42 + 2.07
25. .2 + 12.47
2.61
9.38 = 2.76
9.75 = 0.73
0.35
8.28 = 1.30
8.53 = 1.31
9.32 0.94
9.47 = 1.37
14.73 0.77
2.92 = 0.32
GAIN
GAIN
00
26.09
28.94
15.03
4.00
15.50
14.03
3.43
14.84
SoftImpute
13.38 0.91
38.72 2.03
41.55 = 2.38
22.01 = 1.57
2.28
15.05 = 0.51
9.95 + 0.90
4.08 + 0.22
9.29 + 0.14
8.15 + 0.19
7.37 - 0.32
8.67 1. 1.49
12.87 = 0.28
0.29
0.81 = 0.22
SoftImpute
13.56 1.29
38 a: 2.19
50.03 = 1.07
22.58 = 1.57
16.26 + 1.74
17.06 = 0.73
11.18 0.87
4.08 1. 0.13
+ 0.18
8 3.67 0.21
9.19 +: 0.75
9.97 + 1.60
14.76 = 0.33
0.22
0.96 = 0.26
15.42
16.06
9. 0.66
16.82
MissForest
8 3.66 0.58
3.85 + 0.62
38 38.57 57 =: 03
12.38 = 1.35
4.41 +. 0.18
6.96 0.63
2. 2.60 + 0.12
1.78 + 0.07
4.44 0.30
4.74 + 0.48
4.74 1. 0.75
6.95 0.13
15.30 = 0.40
0.88 + 0.27
Misssorest
8.57 0.62
8.81 1. 0.64
39.34 = 40
11.97 1.18
10.44 0.99
4.47 + 0.20
7.57 0.37
2.67 1. 0.17
1.79 + 0.09
4.24 +
5.26 + 0.42
5.51 # 0.80
7.08 0.19 19
14.54 d: 28
1.17 0.44
10.18
ANN- Imputer
ANN- Imputer
6,95 0.66
0.54
28.81 + 1.45
1:11
0.75
1.78 1: 23
7.38 0.78
2.61 + 0.12
1.95 = 0.00
4.49 0.00
5.84 + 0.47
6.41 1. 0.84
0.30
0.37
4.62 + 0.46
6.74 0.70
0.67
+ 1.24
1.02
0.97
4 4.90 + 0.28
8.21 0.84
2.68 0.18
L.98 = 0.14
6.77 +: 0.42
7.21 + 1.11
13.77 + 0.27
4.75 4 0.50
4.34 =
6.26 =
7.55
11.18
9.39 3.39
6.08
14.71
7.28
29. 29.41
10.65
9.84
DE
6.86 + 0.64
7.10 + 0.52
28.63 # 1.40
11.02 4 1.10
9.24 = 0.80
4.70 d 0.25
7.40 + 0.71
1.87 + 0.08
4.56 # 0.08
5.02 = 0.43
5.58 1 0.79
5.38 + 0.16
+ 0.36
+ 0.40
DDE
6.72 + 0.03
7.01 +. 0,68
29 26 + 1.16
10.55 + 1.03
9.68 + 0.95
4.83 = 0.28
0. + 0.71
2.66 1. 0.17
1.91 # 0.14
4.41 0.11
5.67 + 0.47
6.34 # 1.01
5.59 + 0.21
13.75 de 0.25
0.47
2.59 =
ANN
ANN
14.70
4.57
4.61
Dim.
Dim.
2
2
2
2
4
5
6
7
7
11
11
12
20
30
2
2
2
2
4
5
6
7
7
8
11
11
20
30
japanese yowels
japanese vowels
2d, linear
2d sine
24 ring
geyser
penguin
pollen
planets
a ooee
sulfur
gaussians
wine reed
wine white
sylvine
breast
linear
2d sine
2d ring
geyser
penguin
pollen
planets
abalone
sulfur
gaussiand
wine xed
wine white
sylvine
breast
16

---

Published in Transactions on Machine Learning Research (06/2023)
Table 5: Mean log- likelihood scores with 20% missing rate in Full MCAR scenario. The average log- ligeiiild of the missing observations is higher under the &NN KDE density model. The &NN- &NN- Imputer comes next. MissForest, the KNN- Imputer, MICE, or the Mean imputation method are prone to generate artillcts in the imputed data sets, therefore leading to a lower log- likelhodd of the ground-truth under their density models.
0.02 1 0.030
-0 0.20 0.028
-0.18 + 0.030
0 + 0.048
0.53 + 0.019
0.55 + 0.045
0.028
0.58 + 0.012
0.14 + 0.010
0.66 + 0.023
0.91 + 0.049
0.41 + 0.005
0.32 +. 0.006
0.55 0.023
0.10 + 0.087
40.07 1. 0.046
-0.17 + 0.041
-0 0.30 0.050
0.07 1. 0.071
0.51 + 0.038
0.35 + 0.076
0.38 + 0.037
0.16 + 0.010
0.24 + 0.020
0.44 1. 0.037
1.06 +: 0.123
0 0.36 0.018
0.51 0.028
0.37 + 0.041
Mean
Mean
-0.09
Table 6: Mean log- lee-likclihodd scores with 20% missing rate in MCAR sconario.
0.61
0.65 1 0.182
-0.61 0.170
-0, 82 + 0.117
0.12 + 0.202
-0.03 0.106
0.62 0.028
0.77 + 0.067
0.040
0.69 + 0.019
0.53 + 0.018
0. 0.60 + 0,062
0 0.89 + 0.052
0.33 + 0.012
0.18 + 0,013
1.74 0.042
0.81 0.228
0.181
-0. 69 + 0.184
-0.11 + 0.288
0.31 0.286
1.28 + 0.083
0.101
0.086
0.89 + 0.058
0.050
0.97 0.100
1.18 + 0.122
0.37 0.032
0.03 + 0.068
3.37 0.134
MICE
MICE
1.83
-0 -0.2
0.67
1.84
0.59
MisssFrest
-5.48 1 0.380
-5.91 + 0.348
-10.67 + 0.432
-3.71 + 0.102
-0.71 + 0.291
-1.18 0.086
-0.67 + 0.058
-2.87 1 0.022
-2. -2.98 + 0.128
-2.77 + 0.081
-2.10 # 0.026
0.115
MissForest
-8.3 8.37 0.359
-8. 8. 0.454
8.85 +: 0.490
-1133 + 0.176
-1.93 0.572
-3.66 + 0.229
-3 3.46 0.572
- 2.28 0.198
0.47 + 0.129
2.97 0.122
-4.32 0.381
-4.57 +: 0.426
-2.04 2.04 0.103
-4.49 0.208
1.26 + 0.401
-4.66 #
-1.51
-3.07
-1.51
KNN-Imputer
ANN- Imputer
1.11 1 0.077
-0.09 + 0.081
0.81 + 0.060
0.77 + 0,020
1.98 1 0.226
2.22 1 0.020
1.24 + 0.017
1.37 1 0.011
1.05 + 0.027
122 + 0.050
1.09 + 0.007
0.008
1.17 + 0.034
1.18 0.113
1.13 0.113
-0.03 + 0.046
0.81 0.097
0.91 0.107
1.59 = 0.047
0.180
0.052
0.45 + 0.111
1.71 0.023
1.26 0.131
-2.37 + 0:218
0.100
0.027
1660 + 0.183
0.64 +
0.51
0.47
1.09
2.30
-0.14
0.50
ANN XKDE
1.13 = 0.008
0.077
0.28 = 0.027
0.81 = 0.063
0.67 = 0.096
0.94 = 0.025
1.33 = 0.090
2.02 0.019
2.04 = 0.016
1.50 = 0.013
1.20 0.022
1.34 = 0.048
1.10 = 0.014
0.50 0.009
1.01 = 0,048
ANN XKDE
1.19 0,102
0.103
0.32 = 0.052
0.102
1.53 = 0.044
1.04 0.143
0.025
2.36 = 0.013
0.019
0.075
1.77 = 0.094
1.70 = 0.027
0.026
1,56 = 0.056
0.82 =
0.89
1.11
0.85
2.10
1.61
1.65
0.60
Dim.
Dim.
2
2
2
2
4
5
6
7
7
8
11
11
12
20
30
2
2
5
7
11
11
12
20
30
Japanese vowels
japanese vowels
2d linear
2d sine
2d ring
geyser
penguin
pollen
planeta
abalone
sulfur
gauss ans
wine white
sylvine
breast
2d linear
2d_ sine
2d. ring
geyser
penguin
pollen
planets
abalone
sulfur
gausslans
wine red
wine whits
sylvine
breast
17

---

Published in Transactions on Machine Learning Research (06/2023)
0. 1: 0.058
-0. 0. 0.042
-0. 0.041
-0.26 + 0.031
-0.01 + 0.085
0.017
0.42 1 0.043
0.42 0.027
0.16 1 0.010
0.30 + 0.012
0.48 0.042
1.00 + 0.167
0.37 + 0.025
0.50 0.028
0.33 + 0.070
0,09 + 0.080
-0.09 = 0.066
-0.19 1: 0.048
-0. 2.28 0.040
+0.03 0.113
0.47 1. 0.039
0.37 + 0.061
0.42 + 0.034
0.16 + 0.007
0.24 + 0.017
0.37 1 0.050
0.94 0.176
0.35 0.017
0.56 0.019
0.25 + 0.133
Mean
Mean
Table 8: Mean log- wirllikeehood scores with 20% missing rate in MNAR scenario.
0. 48
Table 7: Mean log- log-iikelhhoo scores with 20% missing rate in MAR scenario.
0.60 1. 0.213
0.204
-0.73 + 0.107
0.196
1.28 0.050
0.63 1 0.106
1.05 1 0.062
0.82 + 0.058
0.097
1.13 + 0.186
0.36 + 0.062
0.03 + 0.056
3.42 1 0.112
0.72 0.273
-0.58 0.164
-0.82 1: 0.238
-0.07 0:240
0.26 0.271
1.29 1 0.088
0.63 0.164
0.088
0.98 + 0.053
0.94 0.010
0.96 0.108
1.07 1. 0.205
0.30 0.044
0.10 + 0.040
3.40 + 0.167
MICE
MICE
-0.65
0.04
0.21
1.79
1. 1.05
1.80
MissForest
88.7 1. 0.410
-8. 27 0.529
-8.64 + 0.487
-11.39 + 0.122
-3.48 + 0.257
33.99 1 0.316
0.32 + 0.138
-2.86 1 0.144
4.44 = 0.197
-.166 + 0.088
-446 0.135
1.40 + 0.433
MissForest
-8. 8.24 0.391
88 25 + 0.720
-8.54 1. 0.425
11.39 + 0:221
-4.43 1. 0.838
-3.51 1. 0.155
-3. 3.81 0.395
-2.6 0.234
0.39 + 0.125
2.78 0.157
4.09 0-405
-4.62 1. 0.378
-2. 2.20 0.151
-4.87 0.150
1.87 + 0.356
-5.02
-2. 2.40
4.32
INp-tep
1.19 1 0.114
-0.05 + 0.042
0.82 + 0.106
1.57 1 0.060
1.19 1 0.118
0.43 1 0.144
1.74 + 0.017
-2.03 353
1.04 + O. 160
-0.18 + 107
1.50 1 0.170
ANN- Imputer
1.18 0.118
1.09 0.185
-0.05 1 0.054
0. 0.093
0.113
1.58 1 0.061
1.04 0.38G
2. 2.24 0.049
0.48 + 0.136
0.024
0.009
-2.54 1. 0.264
0.071
0.021
1.43 4 0.194
1.11
0.92
224
0.50
0.90
1.77
1.19
-0.26
0.55
NNN XDD
0.068
0.068
0.31 = 0.043
0.83 0.098
0.81 = 0.125
1.52 = 0.036
1.13 1 0.128
2,08 + 0.024
2.35 = 0.012
1.63 = 0.012
1.6 1,69 0.047
1.70 = 0.147
1.67 = 0.032
0.024
1.53 = 0.063
ANN XKDE
1,20 0.087
0.126
0.31 = 0.056
0.82 = 0.070
0.84 0.101
1.52 4 0.048
1.05 0.111
2.09 0.021
2.35 = 0.012
0.019
0.030
0.139
1.66 = 0.031
0.70 0.020
1.47 + 0.075
1.12
0.60
1.06
1.65
1.61
1.64
Dim.
Dim.
2
2
2
2
4
5
6
7
7
11
11
12
20
30
2
2
5
11
11
20
30
japanese vowels
japanese vowels
2d linear
2d, sine
24 ring
geyser
penguin
pollen
planets
abalone
sulfur
gaussi ans
wine white
sylvine
breast
2d linear
2d sine
2d ring
geyser
penguin
pollen
planets
aba one
sulfur
gaussiant ans
wine #ine_red red
wine, white
sylvine
breast
18

---

Published in Transactions on Machine Learning Research (06/2023)
References
R. L. Akeson, X. Chen, D. Ciardi, M. Crane, J. Good, M. Harbut, E. Jackson, S. R. Kane, A. C. Laity, S. Leifer, M. Lynn, D. L. McElroy, M. Papin, P. Plavchan, S. V. Ramírez, R. Rey, K. von Braun, M. Wittman, M. Abajian, B. Ali, C. Beichman, A. Beekley, G. B. Berriman, S. Berukoff, G. Bryden, B. Chan, S. Groom, C. Lau, A. N. Payne, M. Regelson, M. Saucedo, M. Schmitz, J. Stauffer, P. Wyatt, and A. Zhang, The nasa exoplanet archive: Data and tools for exoplanet research. Publications of the Astronomical Society of the Pacific, 125, 2013. ISSN 00046280. doi: 10. 10.1086/672273.
A. Azzalini and A. W. Bowman. A look at some data on the old faithful geyser. Applied Statistics, 39, 1990. ISSN 00359254. doi: 10.2307/2347336
Kristin P. Bennett and O. L. Mangasarian. Robust linear programming discrimination of two linearly insepa- rable sets. Optimization Methods and Software, 1, 1992. ISSN 10294937. doi: 10.1080/105550 1000000115567792000555500
Dimitris Bertsimas, Colin Pawlowski, and Ying Daisy Zhuo. From predictive methods to missing data imputation: An optimization approach. Journal of Machine Learning Research, 18, 2018. ISSN 15337928.
Felix Bießmaan, Tammo Rukat, Phillipp Schmidt, Prathik Naidu, Sebastian Schelter, Andrey Taptunov, Dustin Lange, and David Salinas. Datawig: Missing value imputation for tables. Journal of Machine Learning Research, 20, 2019. ISSN 15337928.
Paulo Cortez, António Cerdeira, Fernando Almeida, Telmo Matos, and José Reis. Modeling wine preferences by data mining from physicochemical properties. Decision Support Systems, 47, 2009. ISSN 01679236. doi:
Olivier Delalleau, Aaron C. Courville, and Yoshua Bengio. Efficient EM training of gaussian mixtures with missing data. CoRR, abs/1209.0522 2012. URL http:///wwvv..r///a////0///0//
John K. Dixon. Pattern recognition with partly missing data. IEEE Transactions on Systems, Man and Cybernetics, 9, 1979. ISSN 21682909. doi: (0.1109/SSMC.199...00
D. Dua and C Graff. Uci machine learning repository: Data sets Ireine, CA: University of California, School of Information and Computer Science., 2019. URL http://archive. ics. ucc.edu/ml
Luigi Fortuna, Salvatore Graziani, Alessandro Rizzo, and Maria G. Xibilia. Soft Sensors for Moniloring and Control of Industrial Processes. Springer, 2007. doi: 11.1007/978-1-84660 848846888---99,
Nicholas Frosst, Nicolas Papernot, and Geoffrey Hinton. Analyzing and improving representations with the soft nearest neighbor loss. In ICML2019,, volume 2019-June, 2019.
Leo Grinsztajn, Edouard Oyallon, and Gael Varoquaux. Why do tree-based models still outperform deep learning on tabular data? OpenAccess, 2022. doi: 10.48550 wwt..iiv..2200....
F. Maxwell Harper and Joseph A. Konstan. The movielens datasets: History and context. ACM Transactions on Interactive Intelligent Systems, 5, 2015. ISSN 21606463. doi: 10.1145/2827872.
J. A. Hartigan and P. M. Hartigan. The Dip Test of Unimodality. The Annals of Statistics, 13(1):70 - 84, 1985. doi: 10.1214/aos/ /oss/111634657.7 URL https://doi. wttg//..111///s///11/r1 20/11/////////34657777
Trevor Hastie, Rahul Mazumder, Jason D. Lee, and Reza Zadeh. Matrix completion and low-rank svd via fast alternating least squares. Journal of Machine Learning Research, 16, 2015. ISSN 15337928.
Allison Marie Horst, Alison Presmanes Hill, and Kristen B Gorman. yobmerpenguins: Palmer Archipelago (Antarctica) penguin data, 2020. URL https://alllsohhrsss.gg htt.://occctttttttttceoooooooaaa R package version 0.1.0.
Anil Jadhav, Dhanya Pramod, and Krishnan Ramanathan. Comparison of performance of data imputation methods for numeric dataset. Applied Artificial Intelligence, 33, 2019. ISSN 10876545. doi: 10.1080/ 08839514..019..13311
19

---

Published in Transactions on Machine Learning Research (06/2023)
Sebastian Jäger, Arndt Allhorn, and Felix Bießmann. A benchmark for data imputation methods. Frontiers in Big Dala, 4, 2021. ISSN 2624909X. doi: (33389//data2221 0693677.7..
Shahidul Islam Khan and Abu Sayed Md Latiful Hoque Sice: an improved missing data imputation tech- nique. Journal of Big Data, 2020. ISSN 21961115.
doi: 10.1166//440377-22000033333
Mineichi Kudo, Jun Toyama, and Masaru Shimbo. Multidimensionall curve classification using passing- through regions. Pattern Recognition Letters, 20, 1999. ISSN 01678655. doi: 10.1016/S0167-8855/99 00077-X.
Florian Lalande and Kenji Doya. Tabular data imputation: Choose knn over deep-learning. SIGKDD, 2022. doi: 10.48550/ar Nw..ivv.07.022337.
Marine Le Morvan, Julie Josse, Erwan Scornet, and Gaël Varoquaux. What's a good imputation to predict with missing values? In Advances in Neural Information Processing Systems, volume 14, 2021.
Christina E. Lee, Yihua Li, Devavrat Shah, and Dogyoon Song. Blind regression: Nonparametric regres- sion for latent variable models via collaborative filtering. In Advances in Neural Information Processing Systems, 2016.
Richard Leibrandt and Stephan Günnemann. Making kernel density estimation robust towards missing values in highly incomplete multivariate data without imputation. In SIAM2018, 2018. doi: 10.1137/1.
9781611975321.84
Yihua Li, Devavrat Shah, Dogyoon Song, and Christina Lee Yu. Nearest neighbors for matrix estimation interpreted as blind regression for latent variable model. IEFE Transactions on Information Theory, 66, 2020. ISSN 15579654. doi: N0.1109/TIT.20 TTT.TTTT...11102000 29992995
Roderick J.A. Little and Donald B. Rubin. Statistical analysis with missing data. Wiley, 2014. doi: 10.1002/ 9781119013563.
Zachary Ryan McCaw, Hanna Julienne, and Hugues Aschard. Mgmm: An package for fitting gaussian mixture models incomplete data. bioRriv, 2020.
Boris Muzellec, Julie Josse, Claire Boyer, and Marco Cuturi. Missing data imputation using optimal trans- port. In International Conference on Machine Learning, pp. 7130-7 (1407 PMLR, 2020.
Warwick Nash, T.L. Sellers, S.R. Talbot, A.J. Cawthorn, and W.B. Ford. 7he population biology of abalone (haliotis species) in tasmania. i. blacklip abalone (h. rubra) from the north coast and islands of bass strait. Sea Fisheries Division, Techmical Report No, 48, 01 1994.
Emanuel Parzen. On estimation of probability density function and mode. The Annals of Mathematical Statistics, 33, 1962. ISSN 0003-4851. doi: 10.1214/aomm //4am//mm//// 1 1177704472.
F. Podrogosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel, P. Prettenhofer, R. Weiss, V. Dubourg, J. Vanderplas, A. Passos, D. Cournapeau, M. Brucher, M. Perrot, and E. Duchesnay. Scikit-learn: Machine learning in Python. Journal of Machine Learning Research, 12:2825 2830, 2011.
Jason Poulos and Rafael Valle. Missing data imputation for supervised learning. Applied Artificial Intelli- gence, 32, 2018. ISSN 10876545. doi: 10.1089/0N639554.2201. 1448148..4
Murray Rosenblatt. Remarks on some nonparametric estimates of a density function. The Annals of Math- ematical Statistics, 27, 1956. ISSN 0003-4851. doi: (0.1214/aoms/1177/25
Guido Sanguinetti and Neil D. Lawrence. Missing data in kernel pea. In Johannes Fürnkranz, Tobias Scheffer, and Myra Spiliopoulou (eds.), Machine Learning: ECML 2006, pp. 751 758, Berlin, Heidelberg, 2006. Springer Berlin Heidelberg. ISBN 978-3-540-46056-5.
Divya Saxena and Jiannong Cao. Generative adversarial networks (gans): Challenges, solutions, and future directions. OpenAccess, 2020. doi: 1..45550/ariv..005....
20

---

Published in Transactions on Machine Learning Research (06/2023)
Daniel J. Stekhoven and Peter Bühhmann. Missforet-noo--praaaeti missing value imputation for mixed- type data. Bioinformatics, 28, 2012. ISSN 13674803. doi: 10. (000///boifformatis///tbttee
Elizabeth Tasker, Matthieu Laneuville, and Nicholas Guttenberg. Estimating planetary mass with deep learning. The Astronomical Journal, 159:41, 2020. ISSN 1538-3881. doi: 10.3847/1538-3388//bbb3 /abbbb.
D. M. Titterington and G. M. Mill. Kernel-based density estimates from incomplete data. Journal of the Royal Statistical Society: Series B (Methodological), 45, 1983. doi: 10.111/12557-60011198..88
Olga Troyanskaya, Michael Cantor, Gavin Sherlock, Pat Brown, Trevor Hastie, Robert Tibshirani, David Botstein, and Russ B. Altman. Missing value estimation methods for dna microarrays. Bioinformatics, 17, 2001. ISSN 13674803. doi: htng://iioooooooaoss//.....
Stef van Buuren and Karin Groothhis-Ondsshon.. mice: Multivariate imputation by chained equations in r. Journal of Statisticol Software, 45, 2011. ISSN 15487660. doi: 10.1 12118667js..045....
Kalarzyna Woznica and Przemyslaw Biecek. Does imputation malter? benchmark for predictive models. Artemiss2020, ICML workshop, 2020. doi: 10.48550//rriv..20 200.0x00...000....7.7..
Hanyu Xiang, Qin Zou, Muhammad Ali Nawaz, Xianfeng Huang, Fan Zhang, and Hongkai Yu. Deep learning for image inpainting: A survey. Pattern Recognition, 134:109046, 2023. ISSN 0031-3203. doi: https:// doi.org/ 10. 11.11101116.. patco..22. 5003132032200526X
URL httpss//ww...scccccccteeeettsrsr
Jinsung Yoon, James Jordon, and Mihaela Van Der Schaar. Gain: Missing dala imputation using generalive adversarial nets. 35th International Conference on Machine Learning, ICML 2018, 13:9042-990 9051, 2018.
21

---

Published in Transactions on Machine Learning Research (06/2023)
A Discussion on the hyperparameters of the KNN X XDE
This appendix section discusses about the qualitative effects for the three hyperparameterss of the NNN XDDE. We only focus on the 2d_ sine data set for visualization purposes. Similar Figure in the main text, each figure subsample size aa 10 for plotting purposes, complete observations are in blue, observations where is missing are in orange, and observations where x2 is missing are in red. As before, the subsample size No 10 leads to red vertical or orange horizontal trails of points.
A.1 Softmax temperature
Figure shows the imputation quality as a function of the softmax temperature 7, where the Gaussian kernels bandwidth fixed to = 0.03 and the number of drawn samples Ndraws 10000. For interpretability reason, we report the inverse temperature 1/T.
1/t - 10.0
1/1 500.0
1/r - 25000.0
1.0
0.3
0.5
0.5
0.0
0.0
0.0
-0.5
-0.
-0.5
12
0.4
0.10
0.05
0.2
0.00
0.0
0.0
Figure Change in the imputation quality with varying softmax temperature т. When 1/T is too low (meaning the temperature is too high), the imputation distribution has higher variance, hence a large scatter. If 1/T is too high (meaning the temperature is too low), the imputation distribution will be biased towards the nearest neighbor. The dashed lines show the ground truth coordinates for randomly selected observation [2) = (7.50, 1.03)
On the top-left panel, we see that the neighborhood of each imputed cell is too broad with 1/r = 10.0. resulting in irrelevant neighbors being sampled for imputation and a large scatter. Conversely, the top-right panel shows the imputed samples with an inverse temperature of 1/r = 25000.0 which might be too much and leads an overfit of the observed data.
The three panels in the bottom focus on randomly selected observation with with observed a2 and missing (7.50, 1.03), and 21 is missing. The vertical dashed line in
21. In our case, the ground truth is (æ1, the upper panels shows the observed 22 1.03, and the horizontal dashed line in the lower panels shows the unknown ground truth æ1 = 7.50 to be estimated. The lower pancls show the histogram for the returned distribution by the ANN KDE for this specific cell. Qualitatively, we see that when the lemperature is too high, the imputation distribution is too broad and bridges appear between the two modes of the distribution. On the contrary, when the temperature 7 too low, the imputation distribution biased towards the nearest neighbor, resulting in a unimodal distribution. For 1// 500.0, the imputation distribution is bimodal which reflects the original data structure, and the ground truth correctly falls in of the modes.
22

---

Published in Transactions on Machine Learning Research (06/2023)
A.2 Gaussian kernels shared bandwidth h
Figure 7 shows the imputation quality function of the shared Gaussian kernel bandwidth h. Here, the
softmax inverse lemperature is fixed to 1/rr 300.0 and the number of drawn samples is Ndraws 10000.
h- 0.01
0.03
- 0.1
1.0
0.5
0.5
0.5-
0.0
0.0
0.0
-05
-0.5
-0.5
12
12
01
12
0.4-
0.10
0.1
0.05
0.0
0.0
0.00
Figure Change in the imputation quality with varying kernel bandwidth h. When too small, the imputation distribution is sharp. When too large, the imputation distribution blureed. The dashed lines show the ground truth coordinates for randomly selected observation (x1, (2.88, 0.34)
The loft two pancls show the imputation distribution with a kernel bandwidth h 0.01. narrow bandwidth results in a light to the observed data, therefore leading to a "spiky" imputation distribution on the bottom- left panel. In the limit where + 0.0, the returned distribution become multimodal with probabilities provided by the softmax function. On the other hand, the right two pancls show the imputation distribution with large bandwidth of h 0.1, leading to large scatter around the complete observations.
Like above, the bottom three panels work with a randomly selected observation where is observed and rı is missing. This time, the ground truth is x2) = (2.88, 0.34). With narrow bandwidth, we can clearly see the five possible modes corresponding to Ø2 = 0.34. As the bandwidth becomes larger, modes get closer and uvlly merge. The bottom-right panel has only three modes left. Tn any case, the (unobserved) ground truth Φ 2.88 always falls in a mode of the imputation distribution returned by the KNN*KDE.
Note that the Gaussian kernel bandwidth is shared throughout the algorithm and is therefore the same for all features. As the data set is originally min-max normalized in the [0, 1] interval, the bandwidth adapts to varying feature magnitudes. However, it does not adapt to features scatter, where some features can show a higher standard deviation than others.
Ultimately, we do not optimize the kernel bandwidth in this work. It has been fixed to default value 0.03 throughout all experiments in this paper, but could be fine-tuned obtain higher liceliood scores.
A.3 Number of imputation samples Naraws
The imputation distribution returned by the ANNxKDE as function of the number of samples Naraws is shown Figure Now, the softmax temperature is fixed to 1/r 300.0 and the Gaussian kernel bandwidth is fixed to h 0.03.
23

---

Published in Transactions on Machine Learning Research (06/2023)
1.5
1.5
1.5
1.0
-055
-0.5
-10-
-1.0
11.0
-1.5
-1.5
-1.5
12
12
Density
0.2
0.1
0.1
12
0 4
12
12
Figure 8: Returned distribution as a function of the number of imputation samples Ndraws- The bottom three imputation distributions have a higher resolution with larger Naraws The dashed lines show the ground truth coordinates for a randomly selected observation (I1, 22) = (9.15,
In that case, only the number of samples returned by the ANNxKDE changes. Therefore, there is no statistical difference between the top three panels besides the random subsamples size N.s = 10 used for plotting purposes.
Like with the hyperparameters 7 and h, an observation with observed zg and missing 21 has been randomly selected. On Figure D the ground truth is (21, (9.15, 0.07). The dashed lines the top panels show see that the
that there are five possible values for æ1 given that T2 0.07. On the bottom panels, effect of the number of impulation samples drawn by the ANN*KDE defines the resolution of the returned probability distribution. In all cases, the (unobserved) ground truth ø1 9.15 falls in one of the modes of the imputation distribution.
There no drawback at selting a large Ndrawn, besides the obvious computational cost. On the lower- left panel, we see that the value of the probability distribution density can be poorly approximated with Naraws 200. In this work, always compute the likelihood interval, and we use 120 evenly spaced bins from -0.1 to 1.1 to allow for outliers. For the imputation with the mean of XNN distributions, we used Naraws 1000 since high resolution of the imputation distribution is not necessary. However, we used Ndraws 10000 when computing the likelihood of the ground truth because a low resolution can lead to likelihood computation error in this case.
the normalized data sets in the [0, 1]
B Presentation of the data sets
Both real-world and simulated data sets are used in this work. Eleven real-world data sets have been downloaded, most of them from the open access UC Irvine Machine Learning Repository (Dua & Graff 2019 or the OpenML online repository. Four data sets have been simulated.
This appendix provides summary details about the data. Note that all data sets complete: they originally do not have missing values. Table g presents the data sets name, size, and source. Mcaningless rows (like patient ID or row ID) have been removed for data imputation task. Data sets and their description are provided in the online GitHub repository.
In addition, Table 10 provides the mean and standard deviation for the Pearson correlation coefficient, the Spearman correlation coefficient, and the Hartigan Dip Test of unimodality p-value (Hartigan & Hartigan 1985).
24

---

Published in Transactions on Machine Learning Research (06/2023)
Table 9: Name, sizes, and origin of the data sets. Sorted by increasing dimension D.
Data set name
Size
Source
2d_ linear
(500, 2)
Simulated
2d_ sine
(500, 2)
Simulated
2d_r ring
(500, 2)
Simulated
geyser
(272, 2)
Online AAzzalini & Bowman 1990)
penguin
(342, 4)
Online (Horst et al. 2020
pollen
(3848, 5)
OpenML
planets
(550, 6)
NASA Exoplanet Archive (Akeson et al. 2013
abalone
(4177, 7)
OpenML (Nash et al.) 1994
sulfur
(10081, 6)
OpenML Fooruuna et al. 2007)
gaussians
(10000, 8)
Generated with
random factors
wine_red
(1599, 11)
UCI ML (Cortez et al. 2009
wine wine_white white
(4898, 11)
UCI ML (Cortez et al. 2009
vowels
(9960, 12)
OpenML
(Kudo et al. 1999
sylvine
(5124, 20)
OpenML
breast
(569, 30)
UCI ML (Bennett & Mangasarian 1992)
Given data set of size (N, and two columns hy and bg in [1, DI, the Pearson correlation coefficient is defined as:
EM (Ta.k) Tb (Fink - Tkg)
rp (X Xta))
( TRi) Ei xk2)²
where Tg denotes the mean of column & E [1,
The Spearman correlation coefficient is obtained by computing the Pearson correlation coefficient over the rank variables. If we denote R(Xx) the rank variable for column [1, D], we can write
r (x Xx,) = (R(X,)) R(Xxs))
The Pearson correlation cocfficient measures the lincar correlation between two columns, while the Spearman correlation coefficient quantifies the monotonic relationship (whether linear or not) between two variables. For each data set, we computed both correlation coefficients between all columns, and took their absolute values to compute the mean and standard deviation which we report in Table I0
As for the Dip Test of unimodality, it tries to assess whether a (univariate) distribution is unimodal not. The Dip statistic corresponds to the maximum difference between the empirical cumulative distribution function of sample and the unimodal cunulative distribution function that minimizes that maximum difference. The test computes p-value, which is the probability of obtaining the observed statistic value under the nssumption that the distribution actually unimodal. Lower p-values indicate that the distribution of that fcature likely to be multimodal. The last column of Table I0 reports the mean and the standard deviation of the p-value computed over the D numerical features for each data set.
B.1 Simulated Two-Dimensional Data Sets 2d 2d_linear, 2d_sine, 2d 2d_ring
Three simple data sets in two-dimension are used in this work, named 2d, linear, 2d sine,, and 2d ring. See Section 2.1 for more details.
B.2 Abalone Data Set abalone
This data set is used to predict the age of abalones species of marine snails) from physical measurements: sex, shell length, shell diameter, shell height, whole weight, weight of meat, viscera weight, shell weight, and
25

---

Published in Transactions on Machine Learning Research (06/2023)
number of rings which translates to the abalone age Nash et al, 1994). There are 4,177 observations. The abalone age (target) and the abalone sex have been removed for this work, leading to 7 features.
B.3 Breast Cancer Wisconsin (Diagnostic) Data Set breast
Ten features are computed from a digitized image of a needle aspiration of a breast mass (Bennett & Mangasarian 1992). The data set contains 596 observations and 32 columns. The first two columns are removed for numerical data imputation purposes. The other 30 columns are comprised of the mean, the standard error and the mean of the largest three values of the following ten cell features: radius, texture, perimeter, area, smoothness, compactuess, concavity, concave points, symmetry and fractal dimension.
B.4 Simulated Mixture Gaussians Data Set - gaussians
We use a mixture three multivariate gaussians, generated with random factors method using & = 4 factors in dimension 8. The gaussians data set has 10,000 observations and 8 numerical columns.
B.5 Old Faithful Geyser Data Set geyser
Two features indicate the waiting time between cruptions and the duration of the eruption for the Old Faithful geyser in Yellowstone National Park, Wyoming, USA (Azaalini & Bowman 1990). This data set is commonly used in machine learning and can easily be found online. It has 272 observations and only 2 features.
B.6 Japanese Vowels Data Set japanese_
The data has been collected assess the performances of a multidimensionall time series classifier. Nine male speakers uttered two Japanese vowels /ae/ successively. For each utterance, 12-degree linear prediction coefficients (LPC) analysis is applied, leading to 12 numerical features. Each speaker has various time series for each of its LPC features, which amounts to 9,960 observations.
B.7 Palmer Archipelago Antartica Penguin Data Set penguin
A total of 342 penguins with features (beak length, beak depth, flipper length and body mass) are organized similar the famous iris data set.
in classes (Horst et al. 2020). This data
B.8 NASA Confirmed Exoplanets Archive planets
All confirmed exoplanets according to the NASA Exoplanet Archive of January 2020 have been downloaded AAkeson et al. 2013). This data set has been generated for a study on exoplanets with the intent to retrieve planetary masses Tasker et al. 2020 Six planet features have been selected: planet radius, planet mass, planet orbital period, planet equilibrium temperature, host star mass, and number of planets in the system. Only complete observations have been kept, resulting 550 rows and columns.
B.9 Pollen Data Set pollen
This is synthetic data set provided by RCA Laboratorics at Princcton, New Jerscy. This data set contains 5 geometric features from 3,848 generated pollen grains, namely the length along x-dimension, length along y-dimension, length along z-dimension, pollen grain weight, and pollen density. We could not identify clearly the origin of this data set.
B.10 Sulfur Recovery Unit Data Set sulfur
The Sulfur Recovery Unit (SRU) is used remove environmental pollutants from acid gas stream before they released into the atmosphere. The data set provides variables for 10,081 measures from industrial
26

---

Published in Transactions on Machine Learning Research (06/2023)
processes. These variables describe gas and air flows (ootunn et al. 2007 The target variables (amount of sulfur) have been removed for imputation purposes.
B.11 Sulfur Recovery Unit Data Set sylvine
This data set has been generated for a supervised learning data challenge in machine learning where the goal was to perform classification and regression tasks without human intervention. These data has been generated by computing 6 features over broad variety of other data sets from various domains, which amounts to 5,124 observations.
B.12 Wine Quality Data Set - wiinereed and wine_white
Typical features (e.g. fixed acidity, citric acid, chlorides, pH, alcohol, ...) have been computed for red and white variants of the Portuguese Vinho Verde wines (Cortez al.) 2009). The red wines data set contains 1,599 observations while the white wines has 4,989 observations. Both data sets have 11 numerical features, with additional column indicating the overall wine quality score. This last column has been removed for our imputation work.
Table 10: Mean and standard deviation of Pearson and Spearman correlation coefficients for all data sets.
Data set.
Pearson
Spearman correlation
Dip Test
Dim.
Name
correlation
p-value
_d_linear
2
0.95 = 0.000
0.952 0.000
0.605 + 0.268
2d_sine
0.323 = 0.000
0.325 0.000
0.437 + 0.436
2d_ring
2
0.0117 1 0.000
0.014
0.000
0.000 1 0.000
geyser
0.901 = 0.000
0.778 + 0.000
0.001 + 0.001
penguin
4
0.569 = 0.192
0.546
0.193
0.226 +: 0.259
pollen
0.297 1 0.240
0.287
0.236
0.953 1 0.066
planets
6
0.501 = 0.200
0.534
0.186
0.390 + 0.438
abalone
7
0.891 1 0.058
0.941
0.031
0.337 1: 0.398
sulfur
0.239 1 0.255
0.235
0.236
0.303 1: 0.435
gaussians 8
0.588 + 0.246
0.499
0.203
0.125 +: 0.331
wine_ree
11
0.2
0.187
0.214
0.193
0.022 1 0.047
wine white
0.178 + 0.188
0.194
0.203
0.010 + 0.024
_apanese_vowell vowels 12
0.226 1 0.152
0.228
0.152
0.995 1 0.003
sylvine 20
0.0512 1 0.124
0.048
0.121
0.245
1 0.382
breast
30
0.395 = 0.264
0.422
0.257
0.925
+ 0.093
C Missing data scenarios
We have noticed that the terminology 'Missing Completely At Random' (MCAR) is equivocal. In this work, we consider 4 types of missing data scenarios, namely 'Full MCAR', 'MCAR', 'MAR', and 'MNAR'
In 'Full MCAR', the missing data are inserted completely at random the entire data set and in all columns. This follows the definition of "MCAR" in the work presenting GAIN YYon et al. 2018 for instance.
For 'MCAR', 'MAR', and 'MNAR', we follow the methodology used by the mumerical data. imputation benchmark of Jäger et al. in which only single column is masked (Jäger et al. 2021). For this purpose, we select two columns for each data set: the column miss_col col will be imputed, and the column cond_ col will be used lo compute missing probabilities for the Missing At Random scenario. The selected columms can be seen in the online code. In 'MCAR', missing data are inserted completely at random in column miss miss_col. In 'MAR', we use the quantiles of column cond_co col to provide conditional probabilities for obscrvations miss_coo col to be missing. In 'MNAR', the quantiles of column miss_col col thenselves are used compute missing probabilities.
27

---

Published in Transactions on Machine Learning Research (06/2023)
D Study of the new metric
The ANN-Imputer makes use of the Buclidan Distance to look for neighbors in the presence of missing data. Given a data set represented as matrix of shape (N,)), the NaN-Euclidean Distance is defined as:
di - V |Dohs
D
Σ ReDo
(I xzk]²
where i,j [] are indices, Dots (k E [[1, DJ mik mik 1} is the set of column indices for commonly observed features in observations and j and (Dohsl denotes its cardinality (Dixon 1979
The NaN-Euclidean Distance is, in essence, scaled version of the traditional Fuclidean distance which compute pairwise distance only when possible. However, this metric can generate artificially small distances when the [Dabs| is low compared to D. For example, let us consider = partially observed rows in dimension 5.
--1
6
NaN
8
NaN
NaN
NaN
-2
NaN
Suppose interested in ostimating the missing value 23,5 NaN. With the NaN-Euclidean Distance, the distances are
d13 1/ (02 12 12) ~ 1.83
22,3 = V / (02) = 0.0
meaning that observations and 23 are at an artificially small distance of d2,3 = 0.0, and the missing value will most likely be imputed with 22,5 4 rather than with 21,5 = 8 even though commonly observed cells in rows In and 2y appear quitc similar. In short, the FaN-Euclidean Distance can lead to crroncously small dist listances between observations with few similar commonly observed features.
To address this problem, introduce the SaN-std-Eucliden Distance defined as:
di
Σ (atk T2 + kED
Σ
Diiis
where Dobs {k [], m. MyR 13 is as before, Diiss = {k E [1, mg.45s = 0} is the complement of Dobs, and dg. the standard deviation for column k.
Now suppose that 012,,44,55 1.5, the distances become
d1,3 /0= + 12 102+122
12 + o oz 2.55
22. - of o3 + + + of o = 3.0
such that row In now the closest neighbor of row æ3.
Using the column standard deviation when pairwise distances cannot be computed allows to penalize obser- vations with lot of missing values. The standard deviation may be different for each column, which means that missing data for features with a large variance are more greatly penalized that missing data for features with small variance.
While this section introduces the idea behind our new metric with a simple example, we have noticed that it greatly improves the performances of the RNNxKDE in practice. Before the change of metric, the NRMSE imputation results of the ANNxKDE were closer the ones from the ANN-Imputer, while they are now slightly better than FissForest.
28

---

Published in Transactions on Machine Learning Research (06/2023)
E
Experimental computation time
For the computational time evaluation, we only focus on the 'Full MCAR' scenario with 20% missing rate. For each data set, and for each numerical data imputation ion method, we repeat N = 3 times the main loop of our algorithm. Results are reported in Tables [I and [2] along with the data set sizes, These results have been obtained using a 2.3 GHz Dual-Core Intel Core i5 with 16 GB of RAM.
Table 11: Computational time during experiments, seconds.
Data set name
Size
ANNxKDY
ANN--mputer
MissForest
SoftImpute
2d_ linear
(500, 2)
0.281 + 0.015
0.026 + 0.001
0.334 + 0.026
0.744 + 0.007
2d_sine
(500, 2)
0.267 + 0.013
0.026 + 0.001
1.161 + 0.003
0.733 + 0.005
2d_ ed_ring
(500, 2)
0.327 1 0.043
0.027 1 0.000
1.172 1 0.034
0.765 1 0.015
geyser
(272, 2)
0.153 + 0.006
0.014 + 0.000
0.238 + 0.029
0.667 + 0.008
penguin
(342, 4)
0.417 T. 0.013
0.029 T 0.000
2.153 T. 0.245
0.924 T 0.036
pollen
(3848, 5)
7.843 + 0.115
3.056 +: 0.043
12.417 + 0.117
4.062 1 0.005
planets
(550, 6)
1.062 + 0.047
0.074 + 0.003
3.323 + 0.824
1.417 + 0.054
abalone
(4177, 7)
11.962 1 0.112
5.118 1. 0.109
10.471 1. 1.222
6.066 T 0.022
sulfur
(10081, 7)
38.834 +: 0.052
36.327 +: 0.579
54.081 + 1.069
13.477 + 0.096
gaussians
(10000, 8)
42.094 + 0.195
38.594 + 0.241
54.304 + 8.646
15.551 + 0.093
wine_reat
(1599, 11)
6.744 1: 0.146
0.872 1: 0.023
20.097 1 0.105
4.328 1: 0.063
thhe_
(4898, 11)
22.912 +: 0.051
10.047 +: 0.096
53.790 + 0.217
11.043 + 0.066
Sraanss_vowelle
(9960, 12)
60.341 + 0.092
51.194 +: 0.559
136.634 + 0.514
24.030 + 0.073
sylvine
(5124, 20)
58.984 + 0.217
16.962 +: 0.105
178.992 +: 0.157
23.380 + 0.156
breast
(569, 30)
6.101 + 0.064
0.286 + 0.004
48.564 + 0.968
5.955 + 0.170
Table 12: Computational time during experiments, in seconds. (continue)
Data set name
Size
GAIN
MICE
Mean
Median
_d_linear
(500, 2)
13.134 + 0.125
0,005 + 0.000
0.000 + 0.000
0.001 + 0.000
2d_sine
(500, 2)
12.885 1 0.072
0.003 1 0.000
0.001 0.000
0.001 1 0.000
2d_ring
(500, 2)
13.025 + 0.040
0.003 = 0.000
0.001 + 0.000
0.001 + 0.000
geyser
(272, 2)
12.876 1 0.123
0.005 1 0.000
0.000 1 0.000
0.001 1 0.000
penguin
(342, 4)
13.174 1 0.167
0.012 1 0.002
0.001 0.000
0.001 1: 0.000
pollen
(3848, 5)
13.793 + 0.116
0.023 + 0.000
0.001 +:: 0.000
0.002 + 0.000
planets
(550, 6)
13.349 1 0.107
0.020 1 0.000
0.001 0.000
0.001 1 0.000
abalone
(4177, 7)
15.041 + 0.058
0.041 1 0.008
0.001
0.000
0.003 + 0.001
sulfur
(10081, 7)
16.010 + 0.142
0.054 1:: 0.001
0.002 + 0.000
0.007 + 0.000
gaussians
(10000, 8)
15.443 1 0.273
0.087 1 0.021
0.003
0.000
0.008 1 0.001
wine wine_red
(1599, 11)
15.679 + 0.164
0.046 + 0.001
0.001
0.000
0.002 + 0.000
wine
wine_white white
(4898, 11)
16.008 1 0.121
0.095 1 0.021
0.002 1 0.000
0.004 1 0.000
japanese vowels
(9960, 12)
16.681 + 0.069
0.137 1 0.015
0.003
0.000
0.011 + 0.000
sylvine
(5124, 20)
17.412 + 0.124
0.275 = 0.094
0.002 + 0.000
0.008 = 0.000
breast
(569, 30)
18.793 1 0.068
0.208 1 0.008
0.001 0.000
0.002 1 0.000
For large data sets, the KDE, the ANN-Imputer, and MissForcst become time-consuming. MissForest becomes particularly slow when using more than 20 regression trees for data sets where number total cells (number of variables times number of observations) is large.
29

---

Published in Transactions on Machine Learning Research (06/2023)
F Benchmark results on the MovieLens dataset
We tested our method on the MovicLens datasct (Harper & Konstan, 2015 which is known for having inherent missing values with high missing rate. Note that the Matrix Completion problem associated to the MovieLens dataset includes approximately 6,000 users and 4,000 movies with categorical features, and the missing rate associated with this problem about 96%. The xNNxKI categorical features, but we could make it work by treating the gender and the age of the users as numerical features and looking for neighbours within users.
not designed to work with
We used the leave-one-out cross-validation scheme to assess the performances of the ANNxKDE the MovieLens dataset. We randomly hide one value (corresponding to one random rating for one random user) and use all other users having a rating observed for the randomly selected movie. This process is repeated = 100, 000 times to compute the average RMSE. The ANNx provides imputation with an averago RMSE 0.975. We refer the interested readers to the online repository for the script.
30


Reviews:
[{"review_id": "SxKx14soZZ", "reviewer": "Reviewer_PGgQ", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n- overall I found the paper easy to follow\n- the proposed method is simple and easy to understand\n\nWeaknesses:\n- no comparison to methods commonly used for matrix completion, which is also doing imputation such as SoftImpute (Hastie et al. 2015); there's also a somewhat recently developed nearest-neighbor-inspired algorithm that can be used for imputation by Lee et al 2016/Li et al 2019; I think commenting on how the proposed method kNN-KDE compares with these matrix completion/collaborative filtering approaches would be helpful\n- no experiments on real datasets with real missingness (for example, in collaborative filtering literature, it's common to use MovieLens/Netflix/Yahoo datasets, which essentially have MNAR missingness)\n- I think providing more theoretical analysis of the proposed method would be helpful to give a better sense of when and why we should expect it to work well\n- I think it's important to also consider what imputation is even being used for. Often it is used as just a preprocessing step in which the \"real\" problem that might be of interest is some downstream prediction task. In this case, there is already a fairly strong theoretical result on what a \"good\" imputation even is (Morvan et al 2021). I think providing commentary on this would also be helpful.\n\nReferences:\n- Trevor Hastie, Rahul Mazumder, Jason D Lee, Reza Zadeh. \"Matrix completion and low-rank SVD via fast alternating least squares\". JMLR 2015.\n- Christina E Lee, Yihua Li, Devavrat Shah, Dogyoon Song. \"Blind Regression: Nonparametric Regression for Latent Variable Models via Collaborative Filtering\". NeurIPS 2016.\n- Yihua Li, Devavrat Shah, Dogyoon Song, Christina Lee Yu. \"Nearest neighbors for matrix estimation interpreted as blind regression for latent variable model\". IEEE Transactions on Information Theory 2019.\n- Marine Le Morvan, Julie Josse, Erwan Scornet, Gael Varoquaux. \"What’s a good imputation to predict with missing values?\". NeurIPS 2021.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper proposes an imputation method called kNN-KDE and demonstrates its effectiveness in various simulated datasets and real datasets with simulated missingness.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Please address the weaknesses that I mentioned.", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "W7hi79A6JT", "reviewer": "Reviewer_RUkZ", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "# Strengths\n\n+ The idea to use density estimates on well performing existing imputation algorithms is attractive\n\n+ The approach proposed is appealing in its simplicity. In fact I believe it would be interesting to simplify it further and apply that principle also to other imputation methods, such as forests or other ensemble methods, replacing the inner sampling loop in algorithm 1. That way, one could also try to use the proposed method as a ‚base estimator‘ for MICE?\n\n+ The authors selected kNN as the base imputation algorithm in their proposed solution, which has been shown to work well for many tabular data sets\n\n+ The experimental evaluation on a variety of data sets is very well designed and implemented\n\n+ Probing the algorithm under various missingness conditions and missingness ratios helps to demonstrate the empirical performance comprehensively\n\n+ the idea of drawing bootstrap samples in the inner loop of the proposed algorithm is convincing - especially considering the scaling problems associated with KNN\n\n# Potential improvements\n\n- minor: the title contains multimodal and this often refers to combinations of different data sources, images, text, tables. Here it refers to multimodal distributions, right? \n\n- The empirical comparisons do not appear to strongly support the conclusion that the proposed method performs better than established imputation techniques such as MissForest or kNN. The results in table 1 through 8 indicate that the imputation performance is overall not substantially improved. \n\n- The synthetic data examples shown in fig. 2 could be explained a bit better; my intuition from very similar experiments is that the kNN and the random forest should be able to capture the conditional distributions of one column conditioned on the other column well. If the imputations in fig. 3 are computed with i.e. more training data, other missingness patterns or generally different experimental conditions as those in fig. 2, this could be highlighted. As motivating examples it would be helpful if fig. 2 demonstrated more clearly how exactly the proposed solution solves the problem that e.g. the kNN imputation has and why the gaussian density assumption would help on those orange dots, that are pretty far off from the ground truth.\n\n- The study considers only numerical data. Most of the relevant related literature on tabular data imputation considers at least also categorical data, some also ordinal data. It would strengthen the study to include at least some data sets with more heterogeneous data types. This complicates the analysis of the empirical results, but as the relevant figures are based on rank statistics, also imputation based on classification methods would be simple to include - and most methods compared with support that sort of data out of the box. \n\n- One reason why kNN imputations have not been adopted in practice as much as other solutions is their poor scalability to large data sets. I might have missed that, but if not the authors could mention that limitation. The bootstrap sampling in the inner loop of algorithm 1 could address this problem but it is not entirely clear how this approach would trade off imputation quality with N_{draws}. \n\n- the bootstrap sampling in algorithm 1., inner loop, could be / is part of other imputation implementations, e.g. forest based ones, too. It would be interesting to see how much of the improvements that were observed in some cases for the proposed method are due to this resampling. And whether existing other methods could improve with that, as well. I think that would strengthen the proposed method, if it’s applicable and tested for other ‚base‘ models than kNN, too. ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The authors present an imputation approach for numerical tables based on a combination of a kNN Models and a kernel density estimate of the to-be-imputed attribute, conditioned on the observed attributes. \n\nExtensive empirical comparisons with other established imputation methods investigate the performance of the proposed approach under a variety of missingness patterns and conditions. \n\nThe strongest parts of this work in my opinion are the extensive experiments under a variety of missingness conditions and solid comparisons with many other methods. The empirical evaluation is very convincing - it could be strengthened further though if the authors would have considered tables with categorical / ordinal attributes as well. \n\nAnother positive aspect is the simplicity of the proposed approach. \n\nThere were a couple aspects that could be improved, but the main point on that list is the empirical comparison. While the empirical evaluations were done very well, the results do not seem to be very convincing overall. And the results on the synthetic data could be explained a bit better - in my experience kNN and random forests should yield a better imputation result on those nonlinear tasks in fig. 2 and it is not clear how and why the proposed approach performs better in these two scenarios. \n", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "- include data sets with categorical attributes\n- Explain synthetic data results better\n- Better illustration of the empirical performance of the proposed approach (maybe scatterplots between proposed method and RF / kNN to better assess visually that the proposed approach really is consistently better?)", "broader_impact_concerns": "If imputation methods are easy to use, as this one here, other researchers will use it on any data which can lead to potentially wrong or biased imputations. If these imputations will be taken as real data in downstream analyses, this could lead astray conclusions of researchers who are not aware of the imputation operations performed. ", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "DJ7y2gEEy9", "reviewer": "Reviewer_VbMm", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\nThis paper proposes a new numerical imputation method from the probabilistic perspective, which can preserve the distribution characteristics of the data and handle different missing data patterns.\nThe authors have conducted extensive numerical experiments on synthetic toy data sets and heterogeneous data sets to verify the superiority of KNN$\\times$KDE over existing numerical imputation methods in imputing missing data with multi-modal distributions.\n\nWeaknesses:\n\n1. In Appendix A.3, the authors indicate that a larger imputation sample size has no disadvantages other than increased computational cost. However, increasing the imputation sample size not only increases computational cost but also leads to the overfitting of regression or classification algorithms since the imputed samples are concentrated around the missing data. The authors need to further discuss the selection of the imputation sample size and its impact on the performance of learning algorithms.\n2. In Algorithm 1, the KNN$\\times$KDE algorithm depends on many parameters, including the selection of $\\tau$, $h$, and $N_{draws}$. The authors minimize the NRMSE to select these parameters. However, NRMSE brings back the original problem of choosing a single estimate to impute missing data and can not reflect the multi-modal distribution of missing data. Therefore, my question is whether there are better rules to select these parameters, such as using the log-likelihood score mentioned in Section 5.2.\n3. Some presentation typos:\n(i) Algorithm 1 should add a line to represent the input parameters of the algorithm;\n(ii) On page 10, line 30, \"kNN$\\times$\" should be changed to \"kNN$\\times$KDE\".\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper proposes a numerical imputation method that combines KNN with KDE, which can effectively impute missing data with multi-modal distributions. For different missing data patterns, the proposed algorithm uses soft k nearest neighbors of potential donor data to construct a kernel density estimation of the missing data and then draws samples from the density estimation. Compared with existing numerical imputation methods that can only fit unimodal distributions, the proposed method can effectively impute data with complex dependencies and multi-modal distributions. The paper is well-written, logically structured, and easy to follow.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "I suggest that the author refer to my suggestions in weakness to improve the quality of the paper.", "broader_impact_concerns": "No broader impact concerns regarding the submission.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The method is supported by exhaustive experiments: "The authors have conducted extensive numerical experiments on synthetic toy data sets and heterogeneous data sets" [VbMm]; "Extensive empirical comparisons with other established imputation methods investigate the performance of the proposed approach under a variety of missingness patterns and conditions" [RUkZ].

Despite its simplicity, the performances of kNN*KDE are very good: "the superiority of KNNKDE over existing numerical imputation methods in imputing missing data with multi-modal distributions" [VbMm]; "The approach proposed is appealing in its simplicity", "The empirical evaluation is very convincing" [RUkZ]; "the proposed method is simple and easy to understand", "demonstrates its effectiveness in various simulated datasets and real datasets with simulated missingness" [PGgQ].

The reviewers praised the clarity of the paper: "The paper is well-written, logically structured, and easy to follow." [VbMm]; "I found the paper easy to follow" [PGgQ].

The following important points were raised by the Reviewers and led to discussion:
- no experiments with categorical variables [RUkZ]: the authors acknowledge that kNN*KDE cannot handle categorical variables yet, which is a limitation of the method.
- calibration of the hyperparameters [VbMm]: the authors clarified this point: "the authors make additional explanations for the mild weakness in the original version of the paper" [VbMm].
- lack to comparison with matrix completion method [PGgQ]: some experiments were included by the authors.

The code is available online and can easily be used by practitioners. Its performance is good. Thus, I believe the method is of certain practical interest, and I support the publication of the paper in TMLR. This opinion is supported by [VbMm] and [PGgQ]. However, [RUkZ] still believe that some problems should be fixed before publication. I will thus opt for "minor revision", so that the authors can address these problems.

I copy and paste the comments of [RUkZ] in his/her recommendation. Please address them as best as you can before sending the revised version.

>> The synthetic data experiments suggest that the baseline methods are generally not able to model the data sets with non-linear dependency structure, while the proposed method is (cf figures 2 and 3).

>> I think I understand the arguments of the authors but I still feel that the argument and illustration should be made such that readers do not get the impression that the nonlinear baselines (kNN, forests) cannot model these manifolds while the proposed method can. This is not true and presenting synthetic experiments in a way that suggest that was the reason for me to give the „leaning to reject“ vote.

>> On a different note, the authors emphasized in the discussions and the manuscript that their method is superior as it allows to model the posterior distribution of to-be-imputed values - this is also true for some other baselines that perform poorly in these experiments. Also here the reason for that worse performance could have been explained better. For instance the MICE implementation uses a linear model, which explains its poor performance.

>> So to summarize the comparisons with the baselines in real-data experiments suggest that the proposed method is not really much better than existing well established, simple and more scalable methods. And the synthetic experiments are at least a bit misleading: Some advantages of the proposed method are shared with some baseline methods (and do not help) and some differences in performance are suggesting that baseline methods are worse than they could be. The authors acknowledged these points but did not change the relevant figures 2 and 3.
Decision: Accept (poster)