Imputation for prediction: beware of diminishing returns.
Authors: Marine Le Morvan
Gael Varoquaux
Venue: ICLR 2025 Spotlight (2025)
Keywords: imputation
missing
Primary area: datasets and benchmarks
TL;DR: We show that improving imputation often offers limited benefits for predictive performances with missing values: its beneficial impact is reduced with expressive models, the use of missingness indicators, and real-world (non-linear) data.

Abstract: Missing values are prevalent across various fields, posing challenges for training and deploying predictive models. In this context, imputation is a common practice, driven by the hope that accurate imputations will enhance predictions. However, recent theoretical and empirical studies indicate that simple constant imputation can be consistent and competitive. This empirical study aims at clarifying 
*if* and *when* investing in advanced imputation methods yields significantly better predictions. Relating imputation and predictive accuracies across combinations of imputation and predictive models on 19 datasets, we show that imputation accuracy matters less i) when using expressive models, ii) when incorporating missingness indicators as complementary inputs, iii) matters much more for generated linear outcomes than for real-data outcomes. Interestingly, we also show that the use of the missingness indicator is beneficial to the prediction performance, even in MCAR scenarios. Overall, on real-data with powerful models, imputation quality has only a minor effect on prediction performance. Thus, investing in better imputations for improved predictions often offers limited benefits.

Published as a conference paper at ICLR 2025
IMPUTATION FOR PREDICTION: BEWARE OF DIMINISHING RETURNS.
Marine Le Morvan Soda, Inria Saclay marine.. 1e ree-orvanninniie fr
Gaël Varoquaux Soda, Inria Saclay gael. www.vvvvouuuuuiiiea fr
ABSTRACT
Missing values are prevalent across various fields, posing challenges for training and deploying predictive models. In this context, imputation a common practice, driven by the hope that accurate imputations will enhance predictions. However, recent theoretical and empirical studies indicate that simple constant imputation can be consistent and competitive. This empirical study aims at clarifying if and when investing in advanced imputation methods yields significantly better predictions. Relating imputation and predictive accuracies across combinations of imputation and predictive models on 19 datasets, we show that imputation accuracy matters less i) when using expressive models, ii) when incorporating missingness indicators as complementary inputs, iii) matters much more for generated linear outcomes than for real-data outcomes. Interestingly, the missingness indicator beneficial to the prediction performance, even in MCAR scenarios. Overall, real-data with powerful models, improving imputation only has a minor effect on prediction performance. Thus, investing in better imputations for improved predictions often offers limited benefits.
also show that the use of
INTRODUCTION
Databases are often riddled with missing values due to faulty measurements, unanswered question- naire items or unreported data. This typical of large health databases such as the UK Biobank (Sudlow et al., 2015), the National Health Interview Survey (Blewett et al., 2019) and others (Perez- Lebel et al., 2022). Statistical analysis with missing values has been widely studied, particularly to estimate parameters such as means and variances (Little & Rubin, 2019). However, how to best deal with missing values for prediction has been less studied, Since most machine learning models do not natively handle missing values, common practice is to impute missing values before training model on the completed data, often with the expectation that "good" imputation improves predictions. Considerable efforts have been dedicated to improving imputation techniques, utilizing Generative Adversarial Networks (Yoon et al., 2018), Variational AutoEncoders (Mattei & Frellsen, 2019), opti- mal transport (Muzellec al., 2020) or AutoMI--nhhane iterative conditional imputation (Jarrett et al., 2022) among others. Most of these studies concentrate on imputation accuracy without assess- ing performance on subsequent tasks. However, theoretical arguments suggest that good imputation is not needed for good prediction (Le Morvan et al., 2021; Josse et al., 2019). These arguments are asymptotic and whether they hold in typical cases is debatable. To address the discrepancy between this theory and the emphasis imputation efforts, there is critical need for empirical studies to determine whether better imputations actually lead to better predictions.
Theory does establish that in some scenarios, better imputations imply better predictions. For instance with linearly- generrtde outcome, the optimal prediction a linear model of the optimally-impute data (Le Morvan et al., 2021). Thus, when using a linear model for prediction, better imputations generally lead to better predictions. However, theoretical results suggest that in very high-dimensional settings (Ayme et al., 2023) or in small dimensions with uncorrelated features (Ayme al., 2024), simple constant imputations can be sufficient for linear models. Beyond linear models, empirical studies real data have shown the competitiveness of simple imputations -such the mean- (Paterakis et al., 2024; Perez-Lebel et al., 2022; Shadbahr et al., 2023; Luengo et al., 2012) aligning with theoretical arguments. However, their findings may be driven by "predictive" missingness

---

Published as a conference paper at ICLR 2025
(Missing Not At Random data Little & Rubin, 2019; Josse et al., 2019), for which most imputation methods are invalid.
Drawing robust and broadly applicable conclusions from existing empirical research is challenging, as several key experimental factors can influence conclusions. One critical factor is the missingness mechanism, which falls into three categories: MCAR (Missing Completely At Random), MAR (Missing At Random), and MNAR (Missing Not At Random). MCAR is the simplest case, where missing entries occur with a fixed probability, independent of observed or unobserved data. In MAR, missingness depends solely on observed variables, whereas in MNAR, it is related to the unobserved values themselves. making it informative. Naturally occurring missing values are generally assumed to be MNAR. However, most imputation algorithms are not valid in MNAR scenarios, questioning the utility of advanced imputation algorithms in such cases. MCAR allows for higher quality imputations but their benefit for downstream performance needs to be evaluated. Beyond the missingness mechanism, other influential factors include (i) the missing rate low proportions of missing values may have a limited impact predictive performance; (ii) the use of missingness indicators- concatenating a binary indicator for missing values to the original sample is common practice for prediction, as implemented in seikit-learn (Pedregosa al., 2011); 1); (iii) the choice of downstream model more flexible models may better compensate for poor imputation; and (iv) the proportion of categorical features- for categorical features, the most effective approach is often to treat missing values as a separate category, bypassing imputation altogether.
To draw actionable conclusions taking into account these potentially influential factors, we quantify the change in predictive performance resulting from a gain in imputation accuracy across controlled experimental settings. First, we focus on MCAR as a best-case scenario since it allows for high- quality imputations that are most likely to improve downstream performance. If only limited gains are observed under these ideal conditions. the benefits general scenarios are likely smaller. Thus we aim to establish an upper bound on the potential benefits of imputation. Results under MNAR conditions are nonetheless included (subsection 4.4 and Appendix L.) and confirm that imputation provides less benefits in such cases. Following a similar rationale, we focus on numerical features, where imputation quality is most likely to affect downstream performances, as missing categorical values are better handled separate category. As for the remaining factors, of downstream prediction model, the use of the missingness indicator, as well as the missing rate to condition our conclusions on the exprimental setting. Rather than identifying the "best" imputation and prediction pipeline, our goal is to rigorously assess the impact of imputation quality on predictive performance.
vary the choice
Section 2 introduces related work, covering both benchmarks for prediction with missing values and available theory. Section 3 details our experimental procedures, specifically the methods examined. Section 4 presents our findings, relating gains in imputation to prediction performance. Finally, section 5 summarizes the lessons learned.
2 RELATED WORK
Benchmarks. Several benchmark studies have investigated imputation in prediction context (Paterakis et al., 2024; Jäger et al., 2021; Ramosaj et al., 2022; Wožnica & Biecek, 2020; Perez-Lebel et al., 2022; Poulos & Valle, 2018; Shadbahr et al., 2023; Li et al., 2024; Luengo et al., 2012; Bertsimas et al., 2024). However, drawing definitive conclusions from most studies is challenging due to various limitations in scope and experimental choices. For example, Bertsimas et al. (2018);
Li et al. (2024) trained imputation methods using both the training and test sets, rather than applying the imputation learned on the training set to the test set, which is not possible with many imputation packages. This approach creates data leakage. Woénica & Biecek (2020) trained imputers separately on the train and test sets, which creates an "imputation shift" -a situation where the imputation patterns between the train and test sets differ, causing inconsistencies in the data used for model training versus model evaluation. Jäger et al. (2021) discards and imputes values in a single column of the test set, chosen at random and fixed throughout the experiments. Yet as they note, conclusions can change drastically depending on the importance of the to-be-t-mppeee be-imputed column for the prediction task or its correlation with other features. Some studies (Poulos & Valle, 2018; Ramosaj al., 2022) use a small number of datasets (resp. and 5 datasets from the UCI machine learning repository respectively), thus limiting the significance of their conclusions. Wožnica & Biecek (2020) do
2

---

Published as a conference paper at ICLR 2025
not perform hyperparameter tuning for the prediction models, while Ramosaj et al. (2022) tunes hyperparameters on the complete data, though it is unclear whether the best hyperparameters on complete data are also the best incomplete data. Furthermore, some benchmarks focus specific types of downstream prediction models, such as linear models (Jäger et al., 2021), AutoMI. models (Paterakis et al., 2024) Support Vector Machines (Li et al., 2024), meaning their conclusions should not be generalized to all types of downstream models. Finally, only Perez-Lebel et al. (2022) and Paterakis et al. (2024) evaluate the use of the missingness indicator as complementary input features.
Among the benchmarks with largest scope, Paterakis et al. (2024) recommend mean/mode imputation with the indicator as the default option in AutoML settings, both for native and simulated missingness. It is among the top- perfoffrfming approaches, never statistically significantly outperformed, and is also the most cost-effective. They also show that using the missingness indicator as input improves perfor- mances slightly but significantly for most imputation methods on naturally-occurring missingness. Perez-Lebel et al. (2022) focus on predictive modeling for large health databases, which contain many missing values. They compare various imputation strategies (mean, median, k-nearest neighbors imputation, MICE) combined with mudieen--bossted trees (GBTs) for prediction, as well as GBTs with native handling of missing values. Similarly to Paterakis et al. (2024), they find that appending the indicator to the imputed data significantly improves performances, which may reflect MÑAR data. While they recommend resorting to the native handling of missing values as is relatively cheap, their results further indicate that no method is significantly better than using the mean as imputation method together with the indicator. Shadbahr et al. (2023) also find that the best imputations do not necessarily result in the best downstream performances. Using an analysis of variance, they show that the choice of imputation method has a significant but small effect on the classification performance.
Whether better imputation leads to better prediction may vary depending on factors like the choice of downstream model, the missingness rate, or the characteristics of the datasets. Yet. many studies seek a definitive conclusion across diverse settings. Only Paterakis et al. (2024) conducted a meta-ama meta-analysis analysis, but it did not determine when more advanced imputation strategies are beneficial compared to mean or mode imputations. Identifying scenarios in which better imputations are more likely to improve predictions is however of strong practical interest.
Theoretical insights. Previous works have addressed this question from theoretical point of view. Le Morvan et al. (2021) showed that for all missingness mechanisms and almost all deterministic imputation functions, universally consistent algorithms trained imputed data asymptotically achieve optimal performances in prediction. This is in particular true for simple imputations such as the mean (Josse al., 2019), thereby providing rationale to favor simple imputations over more accurate ones. Essentially, optimal prediction models can be built on mean-imputed data by modeling the mean special value encoding for missingness. Ayme et al. (2023) also provide theoretical support for the use of simple imputations, as they advocate for the use of zero imputation in high- dimensional settings. They show that, for linear regression problem and MCAR missingness, learning on zero- irroiidd data instead of complete data incurs an imputation bias that goes to zero when the dimension increases. This holds given certain assumptions on the covariance matrix, which intuitively impose some redundancy among variables. Finally, Van Ness et al. (2023) prove that in MCAR settings, the best linear predictor assigns zero weights to the missingness indicator, whereas these weights are non-zer non-zero in MNAR settings. Their theoretical results imply that the missingness indicator neither degrades nor enhances performances asymptotically in MCAR.
EXPERIMENTAL SETUP.
Imputation methods. We chose four imputation models to cover wide range of imputation qualities, in order to facilitate the estimation of effects and correlations between imputation and prediction accuracies.
mean each missing value is imputed with the mean of the observed values in a given variable. provides useful baseline for assessing the effectiveness of advanced techniques.
iterativeBR. - each feature is imputed based on the other features in a round-robin fashion using a Bayesian ridge regressor. This method is related to mi ce (Van Buuren & Groothuis-Oudshoom.. 201 1) as it also relies on a fully conditional specification (Van Buuren, 2018). It is implemented in sei milkiitllle -larnn's Iterat Imputtr (Pedregosa et al., 2011).
3

---

Published as a conference paper at ICLR 2025
missforest (Stekhoven & Bühlmann, 2012)- operates in manner analogous to iterativeBR, wherein it imputes one feature using all others and iteratively enhances the imputation by se- quentially addressing each feature multiple times. The key distinction lies in its utilization of a random forest for imputation rather than a linear model. We used scikit-le ran's as estimators. Default parameters for Missforest were set to n_est imators- and max _eax__deeh11555551 for the random forests (the higher the better) to keep a reasonable computational budget. Note that in Impute (Jarrett et al., 2022), random forests are more limited: 10 trees and a maximum depth of 4. condexp- uses the conditional expectation formula of multivariate normal distribution to impute the missing entries given the observed ones. The mean and covariance matrix of the multivariate normal distribution are estimated with (pairwise) available-case estimates (Little & Rubin, 2019, section 3.4), i.e., the ji entry of the covariance matrix is estimated solely from samples where both variables and j are observed. This approach offers computational advantages over more iesource-intensive approaches such the Expectation- MataatiinnMaiiiiiaiionn (EM) algorithm. It is related to Buck's method (Buck, 1960; Little & Rubin, 2019, section 4.2).
Tterat ve Imputer with
Mean imputation can be expected to give the worst imputation, with other methods offering varying improvements. In particular, mi s forest often delivers top-tier performance on tabular data (Waljee et al., 2013; Jarrett al., 2022; Yoon al., 2018; Mateii & Frellsen, 2019; Jäger al., 2021).
Models: As the effect of imputation on prediction quality can be modulated by the predictive model used, we included three predictive models. We took care to include both a deep learning and tree-based representative, as the prediction functions produced by these models have different properties, for example regarding their smoothness. These representatives were chosen because they were identified as ofate-of-the of-thh--rttttttt aat in their category according to recent benchmarks (Borisov et al., 2022; Grinsztajn et al., 2022).
MLP: basic Multilayer Perceptron with ReLU activations, serve simple baseline. SAINT (Somepalli et al., 2021): Self-Attention and Intersample Attention Transformer (SAINT) is deep tabular model that performs both row and column attention. The numerical features are first embedded to a d-dimensional space before being fed to the transformer. We chose SAINT as it has been shown to be state-of-the-ar of-the-art among deep learning approaches for tabular data in several surveys (Borisov et al., 2022: Grinsztajn et al., 2022).
XGBoost (Chen & Guestrin, 2016): We chose XGBoost as it a popular ofate-of-the-ar boosting method, and it has been shown to be the best tree-based model on regression tasks with numerical features only in Grinsztajn et al. (2022).
For XGBoost and the MLP, hyperparameters were tuned using Optuna (Akiba et al., 2019) with 50 trials, i.e, Optuna draws 50 sets of hyperparameters, trains a model for each of these hyperparameter sets, and retains the best one according to the prediction performance on the validation set. For SAINT, we used the default hyperparameters provided by its authors (Somepalli et al., 2021) for computational reasons. Tables to 4 in the Appendix provide the hyperparameter spaces searched, default hyperparameters as well as optimization details.
Native handling of missing values: Both SAINT and XGBoost can directly be applied on incom- plete data, without prior imputation of the missing values, each with its own strategy. XGBoost uses the Missing Incorporated in Attribute (MIA) (Twala et al., 2008; Josse et al., 2019) approach. When splitting, samples with missing values in the split feature can go left, right, form their own Ieaf. MIA retains the option that minimizes the prediction error. SAINT, numerical features are embed- ded in d-dimensional space using simple MLPs. In case of missing value, learnable d-dimensional embedding is used to represent the NaN. Each feature has its own missingness embedding.
The datasets We use a benchmark created by Grinsztajn al. (2022) for tabular learning. comprises 19 datasets (listed in table 1), each corresponding to regression task with continuous and ordinal features. Missing data is generated according to a MCAR mechanism with either 20% or 50% missing rate. Specifically, each value has 20% or 50% chance of being missing. Continuous features are gaussianized using scikit-learn's 's Quant ientillerrrn aanns s f omer while ordinal features are standard scaled to have zero mean and unit variance. This is true for all imputation and model combination except for XGBoost with native handling of missing values, as it is not expected to benefit from normalization. The outputs y are also standard scaled. In all cases, the parameters of these data
4

---

Published as a conference paper at ICLR 2025
normalizations are learned on the train set with missing values. We also provide experiments semi-synthetic data where the response is simulated as a linear function the original data X. The coefficients of the linear function are all taken equal and scaled that the variance of 3 is equal to 1. Noise is added with a signal-to-noise ratio 10.
Evaluation strategy Each dataset is randomly split into 3 folds (train 80%, validation 10% and test - 10%), and each split is furthermore capped at 50,000 samples (table 1). Train, validation and test sets are imputed using the same imputation model trained on the train set. Prediction models are then trained on the imputed train set. When the indicator is used, it is appended as extra features the imputed data, it is not leveraged for the imputation stage. We run all combinations of the 3 prediction models with the 4 imputation techniques, with and without the indicator, resulting in x 3 x 24 models which we add XGBoost and SAINT with native handling of missing values. This results in a total of 26 models displayed in Figure 1. Finally, the whole process is repeated with 10 different train/validation/tes splits. For reprodueibility, the code is available at https: /ttps://giie thub. com/ ma r neLM/ Iwpptatiiiitoooooo for oaroooofr__errreaaaeeeeeeee ction cason__enchhaaa rk.
Computational resources. Properly benchmarking methodologies with missing values is particu- larly resource-intensive, as already emphasized previous works (Jäger al., 2021; Perez-Lebel et al., 2022). Computing costs are driven on the one hand by the need to run the imputation and prediction pipeline across multiple train-test splits (which is important to account for benchmark variance, Bouthillier et al., 2021). and on the other hand by the combinatorics of imputation and prediction models, hyperparameter optimization for both, inclusion and exclusion of the missingness indicator, and varying missing rates. Multiplying the number of models (26) with the number of datasets (19 19 linear versions), the hyperparameter tuning (50 trials), the number of repetitions of the experiments (10), and the 2 missing rates, get very large number of runs (around 1,000,000). As some methods are computationally expensive -such as missforest for imputing, as well as SAINT notably when the indicator is used-, these experiments required total of 325 CPU days for the MCAR experiments. fifth of this time was dedicated to imputation.
missing_
missing rete 50%
+.05
Relalive predictin R²
-06
MLP
SAINT
XGBoos!
MLP
SAINT
XGBoost
Impulation
Add missingness indicator
maan
BeradvvBR
condaxp
missforest
Nana
False
True
Figure I: Relative prediction performances across datasets for different imputations, predictors, and use of the missingness indicator. Each boxplot represents 200 points (20 datasets with 10 repetitions per dataset). The performances shown are R² scores on the test set relative the mean performance across all models for given dataset and repetition. A value of 0.01 indicates that a given method outperforms the average performance on a given dataset by 0.01 on the 2² score. Corresponding critical difference plots in figs. and 7.
5

---

Published as a conference paper at ICLR 2025
4 RESULTS: DETERMINANTS OF PREDICTIONS PERFORMANCE WITH MCAR MISSINGNESS
4.1 BENEFITS OF SOPHISTICATED IMPUTATION, THE INDICATOR, AND XGBOOST AMID HIGH VARIANCE.
Figure summarizes the relative performance of the various predictors combined with the different imputation schemes across the 19 datasets. Some trends emerge: more sophisticated imputers tend to improve prediction, with foisssorrett predictors often outperforming those using condexp or iterativeBR imputers, which in turn outperform predictors based on mean imputation. However, using the missingness indicator decreases this effect. Additionally, while less powerful models like MLPs show greater improvements with more advanced imputations, this effect is barely noticeable for the strongest predictor, XGBoost, which maintains its advantages on tabular data (as described in Grinsztajn et al., 2022) even in the presence of missing values.
That the best predictor barely benefits from fancy imputers brings us back to our original question: should efforts go into imputation? Drawing a conclusion from figure I would be premature: the variance across datasets is typically greater than the difference in performance between methods (critical difference diagram in figs. and 7). For example, missforest + XGBoost indicator outperforms all other methods in only 4 out of 19 datasets. Additionally, XGBoost indicator does not perform significantly better with missforest than with condexp at 50% missingness, while mean imputation does not always lead to the worst prediction. In what follows, we focus quantifying the effects of improved imputation accuracy on predictions different scenarios.
4.2 A DETOUR THROUGH IMPUTATION ACCURACIES: HOW DO IMPUTERS COMPARE?
Although comparing imputers is not our main objective, it is enlightening for our prediction purpose to characterize their relative performance range. Figure (left) gives imputation performances measured as the R² score between the imputed and ground truth values relative to the average across imputations, for each dataset. At 20% missing rate, missForest is the best imputer, followed by condexp and iterativeBR, which are nearly tied, while mean imputation performs significantly worse. At a 50% missing rate, the imputation accuracy of all but mean imputation drop, but condexp is much less affected. It is interesting that such a simple method performs best. It is notably two orders of magnitude faster than missforest (figure 2 right), which makes an imputation technique worth considering. is possible that the gaussianization of the features helped condexp, although feature-wise gaussianization does not produce jointly Gaussian dataset.
In order to highlight the link between imputation and prediction quality, it necessary achieve varying imputation qualities. Here the high range of imputation accuracy between the best and worst
missing
Dalasel
mosiall charges
ReraiiveBR
damardk
condexp
missforest
dolays_z-ron zirkon Jransport
houses
hosses
-0.4
-0.2
0.0
myorse green-de--2016
cuc_y
missing rate 50%
Manihouure aaa
soles
RerativeBR
opu,ast
condexp
Alwrrs
missforest
yprpp 4_1
superoontust
-04
-0.2
0.0
Relative impulation R² score
Imputation time
Figure 2: Left: Imputer performance for recovery. Performances given H² scores for each dataset relative to the mean performance across imputation techniques. A negative value indicates that method perform worse than the average of other methods. Right: Imputation time.

---

Published as a conference paper at ICLR 2025
Figure 3: Example fit of prediction performance as function of impu- tation accuracy, for the Bike_ Se___Sharinn___eeee dataset and missing rate of 50%: on the left using MLP predictor, and on the right an MLP with missingness indicator.
MLP
MLP + indicator
Relative prediction R²
correlation: 0.64
correlation: -0.13 slope: -0.01
slope: 0.23
Imputation iterativeBR condexp
missforest
-0.2
0.0
0.2
-0.2
0.0
0.2
Relative imputation recovery R²
methods (an average difference of 0.5 R² points at 20% and 0.3 R² points at 50%) allows capturing differences in prediction performance.
4.3 LINKING IMPUTATION ACCURACY AND PREDICTION PERFORMANCES.
Combining the four imputation techniques with 10 repetitions of each experiment yields 40 (im- putation Rº, prediction It²) pairs for each model and dataset. To quantify how improvements in imputation accuracy translate into downstream prediction performance, we fit a linear regression using these 40 points for each model and dataset'. Figure 3 gives two examples of such fit: on the
dataset, for missing rate of 50%, the prediction H² increases as function of the imputation A² the effect is greater for the MLP, for which the fit gives slope of 0.23, than for the MLP with indicator for which the slope is -0.01.
Figure summarizes the slopes estimated using the aforementioned methodology across all datasets, predictors with and without the indicator, and varying missing rates. Firstly, the fact that most slopes are positive indicates that better imputations correlate with better predictions, aligning with common beliefs. However, this observation should be interpreted with nuance in light of the effect sizes.
Gains in prediction R² are 10 or less of the gains in imputation R². Figure shows that the slopes are typically small, rarely exceeding 0.1. This implies that an improvement of 0.1 in imputation n² typically leads to an improvement in prediction It' that is 10 times smaller, i.e. a gain of 0.01 in prediction R², even less. For XGBoost, the average slope across datasets in rather close to 0.025 or less (even zero without the mask at 20% missing rate). Thus, an enhancement of 0.3 in imputation R², which represents the average difference between the best of the worst imputer in this scenario (mean vs condexp in fig. 2), implies gain in prediction H² of only 0.0075.
'The repetition identifier is also used as a covariate in the linear regression to account for the effects of the various train/test splits on prediction performance.
missing_res rate = 20%
missing rate = 50%
Regression coefliclett linking
prediction to imputalion R2
0.20
Miss maninsn indicator
Fiess
house
delays, zurich
0.15
Coof sign Ecently
quality
yea
Misnittuuinn
0.10
house salva
house
medical charpes
clevators
diamanda
spu act pol
0.00
sulfur
Bike. Sharing
Allerons
-0.08
abalone
yproo_4.
Br arz lan houses
suporconduct
M.P
SAINT
XGBooM
MLP
SAINT
XGBoost
Figure Effect of the imputation recovery on the prediction performance. We report the slope of the regression line where imputation quality is used to predict prediction performance. A coefficient is marked as significantly greater than zero (cirele) if the associated p- p-value (one-sided T-test) is below 0.05 after Bonferroni correction for multiple testing.
7

---

Published as a conference paper at ICLR 2025
missing_ rate 20%
missing_ree rate 50%
1.0
Outcome
ryo-laxi
Correaation between
prodiction and imputation
0.8
rewl
Manmiousinng
qusityy
Catasets
house salee
medicall chargax
house 10H
diamonds
eleaaooss
0.2
opu
Bike Sharing
pol
0.0
abalene
Allernna
aaallan
0.2
dedays
superoonduct
houses
MLP
SAINT
XGDoost
MLP
SAINT
xGB
Figure 5: Correlation between imputation quality and prediction performance. A correlation close to 1 indicates that the quality of imputations stronly associated the quality of predictions, while a correlation close to zero means that the quality of predictions is not linked to the quality of imputations. Each correlation is computed using 40 different imputation/perfformace pairs, made of 4 imputation methods (mean, iterativeBR, missforest, condexp) repeated 10 times.
Good imputations matter less for more expressive predictors. Comparison between models shows a decrease in slope from MLP to SAINT, to XGBooost. A one-sided Wilcoxon signed-rank test assessing whether the median of each boxplot in Figure 4 is significantly greater than 0 reveals that the positive effect of imputation on prediction is significant for the MLP, but not for SAINT or XGBoost (case without missingness indicator). These results illustrate the idea that a powerful model can compensate for the simplicity or inaccuracy of an imputation (in our case, the MLP can be considered the least expressive model, and XGBoost the most expressive). Le Morvan et al. (2021) gives formal proof in an extreme case: given enough samples, a sufficiently expressive model can always build a Bayes optimal predictor even on the simplest imputations (e.g. a constant).
Good imputations matter less when adding the indicator. Figure shows that adding the missingness indicator clearly decreases the effect size: imputing better has less impact on performance when the indicator is used (we discuss this effect further in section 4.5).
Good imputations matter less when the response is non-linear. When the response y is a linear function of the input X, the best predictor can be built using a linear model on the most accurate simple imputation. However, when the responses are non-linear, it may be difficult to learn the best possible predictor as becomes a discontinuous function even with the most accurate imputation (Le Morvan et al., 2021). There are thus reasons to believe that response non non-lineate -on-iinerr which is common real data, alters the relationship between imputation accuracy and prediction performance. To investigate this, we compare the real datasets with matching semi-simulated datasets where is simulated as linear function of the input X. We also measure correlation (fig. 5) in addition the slope, to quantify the reliability of the association: correlation captures not only the effect size (slope) but also the amount of noise (appendix C recalls this classic result) in the relationship. While the effects similar between real and linear outcomes (fig. 11 gives effects in the semi-simulated case), the correlation between imputation accuracy and prediction performance, averaged across all datasets, is systematically smaller for real outcomes than for linear ones (fig. 5). The average decrease in correlation lies between 0. and 0.3 across models. Moreover, the variance in correlations for real outcomes is much larger, with many datasets with near-zero correlation. This shows that the gains expected in prediction from better imputation are much more reliably achieved when the response is linear.
4.4 LOWER EFFECT OF IMPUTATION ACCURACY ON PREDICTION PERFORMANCE WITH MNAR MISSINGNESS.
We argue that MCAR provides best-case scenario study the potential benefit of imputation on prediction. It is indeed the easiest setting for imputation, as the missingness does not depend on the data. In addition, expecting general conclusions to apply universally across all MAR and MNAR
SSpeiffically we use the partial correlation, partialing out the effect of repeated train/test splits.
8

---

Published as a conference paper at ICLR 2025
mechanisms is unrealistic, these categories encompass families of missingness mechanisms with infinitely many variations (see, for example. Pereira et al. (2024)). Instead, the effects will depend on the specific MAR MNAR model used. For instance, in self-censoring (a MNAR mechanism), the probability of a feature being missing can follow a hard-thresholding function based on underlying values, making imputation very hard, or an almost flat function, which is much closer an easy MCAR scenario.
To illustrate the difference between MCAR and MNAR scenarios, we re-ran all experiments with self-censoring mechanism, where the probability of missingness increases smoothly from 0 to over the support of the data, according chosen to be neither too hard (e.g. hard thresholding), nor too easy (e.g almost MCAR). Figure 34 shows that the estimated effects are consistently lower than in the MCAR case. When the missingness indicator is not used, improving imputation quality even has, on average, negative effect on prediction accuracy of XGBoost and SAINT. This is mainly because mean imputation performs well compared to more advanced strategies, likely because the mean allows to retain the information that a value was imputed. Overall, these experiments show that MCAR is a best-case scenario and that imputation in other settings will bring less benefits.
probit function (details Appendix L.1). This mechanism was
4.5 WITY THE INDICATOR BENEFICIAL, EVEN WITH MCAR DATA?
In general, we find that adding the missingness indicator improves prediction. While it is expected that adding the indicator is beneficial in MÑAR scenarios, as the missingness is informative, it is less obvious in the MCAR settings studied here. Indeed, the indicator contains absolutely relevant information for predicting the outcome. To the best of our knowledge, the benefit of using an indicator in MCAR has not yet been established.
possible theoretical explanation for this finding lies in the challenge of learning optimal prediction functions on imputed data. The best possible predictor in the presence of missingness can always be expressed as the composition of an imputation and a prediction function (Le Morvan et al., 2021). But, in general, the best prediction function the imputed data can be challenging learn, even for perfect conditional imputation. In fact, it often displays discontinuities on imputed points. We hypothesize that adding the missingness indicator simplifies modeling functions that exhibit discontinuities at these points, as the indicator can act as switch to encode these discontinuities.
To assess the role of the information encoded in the missingness indicator, we repeated the experiments with shuffled missingness indicator, where the columns of the indicator were randomly shuffled for each sample. This preserves the total number of missing values per sample but removes information about which specific features are missing. Figure 15a demonstrates that using shuffled missingness indicator harms prediction performance, except for XGBoost for which performances are unchanged. In contrast, the true missingness indicator improves performances (fig. 1). Furthermore, the shuffled indicator does not affect the relationship between imputation accuracy and prediction accuracy (fig. 15b), whereas the true indicator reduces the effect size (fig. 4). These results confirm that the benefit of the missingness indicator is not due to a regularization or merely encoding the number of missing values. Experiments on feature importance further show that the importance of feature drops when imputed compared to when observed (Appendix H), suggesting that imputations do not contribute predictions as effectively as observed values.
The case of XGBoost in Figure illustrates the importance of keeping the missingness information encoded. For 50% missing rate, in the absence of an indicator, no imputation benefits prediction with XGBoost, and the best option is to use the native handling of missing values. This suggests that XGBoost benefits from knowing which values are missing. With advanced imputations. distinguishing between imputed and observed values becomes challenging. Appending the indicator to the imputed data reinstates the missingness information unambiguously. which enables XGBoost to benefit from more advanced imputations, in particular missforest.
CONCLUSION
Imputation matters for prediction, but only marginally. Prior theoretical work showed that in extreme cases (asymptotics), imputation does not matter for predicting with missing values. We quantified empirically the effect of imputation accuracy gains prediction performance across
9

---

Published as a conference paper at ICLR 2025
many datasets and scenarios. We show that in practice, imputation does play role. But various factors modulate the importance of better imputations for prediction: investing in better imputations will be less beneficial when flexible model is used, when missing-valus indicator is used, and if the response is non-linear. These results are actually in line with the theoretical results suggesting that imputation does not matter, these hold for very flexible models (ie universally consistent). A notable new insight is that adding a missing-value indicator as input is beneficial for prediction performances even for MCAR settings, where missingness is uninformative.
We show that large gains in imputation accuracy translate into small gains in prediction performance. These results were drawn from favorable MCAR setting, and is likely that with native missingness, often Missing Non At Random (MNAR), the performance gains are even smaller. As novel imputation methods usually provide small gains in imputation accuracy compared to the state-of-the art,, the corresponding gains in downstream prediction tasks are likely to be even smaller.
There are multiple potential reasons why imputation gains do not always correlate with performance gains. For instance, some features may be well recovered, but not useful in the prediction because they are not predictive. Or even with accurate imputations, may still be difficult to learn a predictor that performs well for all missing data patterns (Le Morvan et al., 2021). Finally, the imputation accuracy is also probably an imperfect measure of the potential gains in prediction: in our experiments on 50% missing rate, missforest and iterativeBR performs comparable on average yet missforest-based predictors tend to outperform those based on iterativeBR.
Limitations and future work. It would be valuable investigate whether imputations based on random draws outperform deterministic imputations for downstream prediction tasks, and the useful- ness of multiple imputations (Perez-Lebel et al., 2022). A related question is whether reconstructing well the data distribution is important for better predictions. Shadbahr et al. (2023) shows that it does not seem crucial for classification performances but may compromise more seriously model interpretabiliity Finally, appending the indicator to the input improves predictions, but doubles the feature count and may not be the most effective encoding for enhancing downstream performance. Alternative approaches, such missingness-awan missingness embeddings (Somepalli et al., 2021), or missingness- awarre layers (Le Morvan et al., 2020), have been proposed, but further investigation is needed in these directions.
feature encodings (Lenz et al., 2024), learned
Outlook Improving imputation is often a difficult way of improving prediction. On top of im- putation, future research inherently handle missing values and effectively incorporate missingness indicators to improve predictive performance.
could focus more on developing advanced modeling techniques that can
REFERENCES
Takuya Akiba, Shotaro Sano, Toshihiko Yanase, Takeru Ohta, and Masanori Koyama. Optuna: A next-generation hyperparameter optimization framework. In Proceedings of the 25th ACM SIGKDD international conference on knowledge discovery data mining, pp. 2623-2631, 2019.
Alexis Ayme, Claire Boyer, Aymeric Dieuleveut, and Erwan Scornet. Naive imputation implicitly regularizes dighedimmnsional linear models. In International Conference Machine Learning, pp. 1340. PMLR, 2023.
Alexis Ayme, Claire Boyer, Aymeric Dieuleveut, and Erwan Scornet. Random features models: a way to study the success of naive imputation. arXiv preprint arXiv:2402.03839, 2024.
Dimitris Bertsimas, Colin Pawlowski, and Ying Daisy Zhuo From predictive methods to missing data imputation: an optimization approach. Journal of Machine Learning Research, 18(196): 2018.
Dimitris Bertsimas, Arthur Delarue, and Jean Pauphilet. Simple imputation rules for prediction with missing data: Theoretical guarantees vs. empirical performance. Transactions Machine Learning Research, 2024.
Lynn A Blewett. Julia A Rivera Drew, Miriam L King, Kari CW Williams, Natalie Del Ponte, and Pat Convey. lpums health surveys: National health interview survey. Minneapolis, MN: IPUMS, 2019.
10

---

Published as a conference paper at ICLR 2025
Vadim Borisov, Tobias Leemann, Kathrin Seßler, Johannes Haug. Martin Pawelezyk, and Gjergji Kasneci. Deep neural networks and tabular data: A survey. IEEE Transactions on Neural Networks and Learning Systems, 2022.
Xavier Bouthillier, Pierre Delaunay, Mirko Bronzi, Assya Trofimov, Brennan Nichyporuk, Justin Szeto, Nazanin Mohammadi Sepahvand. Edward Raff. Kanika Madan. Vikram Voleti, al. Accounting for variance in machine learning benchmarks. Proceedings of Machine Learning and
Systems, 3:747-769, 2021.
Samuel F Buck. A method of estimation of missing values in multivariate data suitable for use with an electronic computer. Journal of the Royal Statistical Society: Series B (Methodological), 22(2): 302-306, 1960.
Tianqi Chen and Carlos Guestrin. XGBoost: A scalable tree boosting system. In Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, KDD 16, pp. 785-794, New York, NY, USA, 2016. ACM.
Léo Grinsztajn, Edouard Oyallon, and Gaäl Varoquaux. Why do tree-based models still outperform deep learning typical tabular data? Advances in Neural Information Processing Systems, 35: 507-520, 2022.
Sebastian Jäger, Arndt Allhorn, and Felix Bießmann. A benchmark for data imputation methods. Frontiers in big Data, 4:693674, 2021.
Daniel Jarrett, Bogdan C Cehere, Tennison Liu, Alicia Curth, and Mihaela van der Schaar. Hyperim- pute: Generalized iterative imputation with automatic model selection. In International Conference on Machine Learning, pp. 9916-9937. PMLR, 2022.
Julie Josse, Nicolas Prost, Erwan Scornet, and Gaël Varoquaux. On the consistency of supervised learning with missing values. arXiv preprint arXiv:1902.069331 2019.
Marine Le Morvan, Julie Josse, Thomas Moreau, Erwan Scornet, and Gaël Varoquaux, Neumiss networks: differentiable programming for supervised learning with missing values. Advances in Neural Information Processing Systems, 33:5980-5990, 2020.
Marine Le Morvan, Julie Josse, Erwan Scornet, and Gael Varoquaux. What's a good imputation to predict with missing values? In M. Ranzato, A. Beygelzimer, Y. Dauphin, P.S. Liang, and J. Wortman Vaughan (eds.), Advances in Neural Information Processing Systems, volume 34, pp. 11530- 11540. Curran Associates, Inc., 2021.
Oliver Urs Lenz, Daniel Peralta, and Chris Corelis. Polar encoding: A simple baseline approach for classification with missing values. IEEE Transactions Fuzzy Systems, 2024.
JiaHang Li, ShuXia Guo, RuLin Ma, Jia He, XiangHui Zhang, DongSheng Rui, YuSong Ding, Yu Li, La Jian, Jing Cheng, et al. Comparison of the effects of imputation methods for missing data in predictive modelling of cohort study datasets. BMC Medical Research Methodology, 2:11):41, 2024.
Roderick JA Little and Donald B Rubin. Statistical analysis with missing data, volume 793. John Wiley & Sons, 2019.
Julián Luengo, Salvador Gareía, and Francisco Herrera. On the choice of the best imputation methods for missing values considering three groups of classification methods. Knowledge and information systems, 32:77-108, 2012.
Pierre-Alexanee Alexandre Mattei and Jes Frellsen. Miwae: Deep generative modelling and imputation of incomplete data sets. International conference on machine learning, pp. 4413-4423. PMLR, 2019.
Boris Muzellee, Julie Josse, Claire Boyer, and Marco Cuturi. Missing data imputation using optimal transport. In International Conference Machine Learning, pp. 7130-7140. PMLR, 2020.
11

---

Published as a conference paper at ICLR 2025
George Paterakis, Stefanos Fafalios, Paulos Charonyktakis, Vassilis Christophides, and loannis Tsamardinos. Do we really need imputation in automl predictive modeling? ACM Transactions on Knowledge Discovery from Data, 2024.
Fabian Pedregosa, Gaël Varoquaux, Alexandre Gramfort, Vincent Michel, Bertrand Thirion, Olivier Grisel, Mathieu Blondel, Peter Prettenhofer, Ron Weiss, Vincent Dubourg, et al. Scikit-learn: Machine learning python. the Journal of machine Learning research, 12:2825-2830, 2011.
Ricardo Cardoso Pereira, Pedro Henriques Abreu, Pedro Pereira Rodrigues, and Mário AT Figueiredo. Imputation of data missing not at random: Artificial generation and benchmark analysis. Expert Systems with Applications, 249:1,3654, 2024.
Alexandre Perez-Lebel, Gaäl Varoquaux, Marine Le Morvan, Julie Josse, and Jean- Tran-Baptiste Poline. Benchmarking missing-values approaches for predictive models on health databases. GigaScience, 11:giac013, 2022.
Jason Poulos and Rafael Valle. Missing data imputation for supervised learning. Applied Artificial Intelligence, 32(2):186-19) 2018.
Burim Ramosaj, Justus Tulowietzki, and Markus Pauly. On the relation between prediction and imputation accuracy under missing covariates. Entropy, 24(3):386, 2022.
Tolou Shadbahr, Michael Roberts, Jan Stanczuk, Julian Gilbey, Philip Teare, Sören Dittmer, Matthew Thorpe, Ramon Viñas Torné, Evis Sala, Pietro Lió, et al. The impact of imputation quality on machine learning classifiers for datasets with missing values. Communications Medicine, 3(1):139, 2023.
Gowthami Somepalli. Micah Goldblum, Avi Schwarzschild, C Bayan Bruss, and Tom Goldstein. Saint: Improved neural networks for tabular data via row attention and contrastive pre-training. arXiv preprint arXiv:2106.013:: 2021.
Daniel Stekhoven and Peter Bühlmann. Missforest non--onn--ane- con-aeaaetrric missing value imputation for mixed-type data. Bioinformatics, 28(1):112-111 2012.
Cathie Sudlow, John Gallacher, Naomi Allen, Valerie Beral, Paul Burton, John Danesh, Paul Downey, Paul Elliot, Jane Green, Martin Landray, et al. Uk biobank: an open access resource for identifying the causes of wide range of complex diseases of middle and old age. PLoS medicine, 12(3): c1001779, 2015.
Bheki ETH Twala, MC Jones, and David J Hand. Good methods for coping with missing data in decision trees. Pattern Recognition Letters, 29(7):950-956, 2008.
Stef Van Buuren. Flexible imputation of missing data. CRC press, 2018.
Stef Van Buuren and Karin Proothhis-Oudshooee mice: Multivariate imputation by chained equations in r. Journal of statistical software, 45:1-67,200 2011.
Mike Van Ness, Tomas M Bosschieter, Roberto Halpin- Dreein--rreoroo,, and Madeleine Udell. The missing indicator method: From low to high dimensions. In Proceedings of the 29th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, pp. 5004-5015, 2023.
Akbar K Waljee, Ashin Mukherjee, Amit G Singal, Yiwei Zhang, Jeffrey Warren, Ulysses Balis, Jorge Marrero, Ji Zhu, and Peter DR Higgins. Comparison of imputation methods for missing laboratory data in medicine. BMJ open, 380)840 2013.
Katarzyna Wožnica and Przemyslaw Biecek. Does imputation matter? benchmark for predictive models. arXiv preprint arXiv:2007.02837, 2020.
Jinsung Yoon, James Jordon, and Mihaela Schaar. Gain: Missing data imputation using generative adversarial nets. In International conference on machine learning, pp. 5689-5698. PMLR, 2018
12

---

Published as a conference paper at ICLR 2025
Appendix
Table of Contents
A I List of datasets.
14
B Hyperparameter search spaces
14
C Link between correlation and effect size.
16
D Critical Difference diagrams.
16
E Prediction performances the semi-synthetic data.
F Regression slopes for the semi-synthetic data.
19
G Effect of the missing rate.
20
Feature importance depending on the missingness status.
21
Investigating the role the missingness indicator.
22
1.1 Prediction performance gains when using the missingness indicator versus imputa- tion accuracy
22
1.2 Shuffling the missingness indicator.
23
J Computation times per method.
24
K Scatterplots of prediction R² vs imputation R² for each model and dataset.
26
L MNAR scenario.
38
L.1 Description self-censoring mechanism.
L.2 Results under MNAR missingness.
39
13

---

Published as a conference paper at ICLR 2025
A LIST OF DATASETS.
This benchmark was created by Grinsztajn et al. (2022), and is available on OpenML. at https://www.opett tasks_ iaaks_iicluuedddd-333
Table I: Dataset dimensions.
dataset
n_ train
test
house_1de 16H
16
18185
2274
epu_act
6553
820
elevators
16
13279
1661
wine qualetyuaayy
11
5197
651
Brazilian houses
8553
1070
house_sales
15
17290
2162
sulfur
8064
1009
Ailerons
33
11000
1375
Bike Sharing Demand
13903
1739
diamonds
43152
5394
14450
houses
16512
2064
medical chargee__hhheeeeeee
50000
50000
MiamiHousing20066
13
11145
1394
myc-taxi-green--ee-2016
50000
50000
pol
12000
superconduct
79
17010
2127
yprop_4. __1
42
7108
889
abalone
3341
419
B HYPERPARAMETER SEARCH SPACES
Table 2: XGBoost hyperparameter space. We used the XGBRegressor from the xgboost Python library. The hyperparameters optimized are commonly accepted as the most important ones. The variation ranges are inspired by the ones used in Grinsztajn (2022), while the default hyperparameters are those of the xgboost library.
parameter
range
log scale
default
esstimatorss
[100, 2000
no
100
max _ex__dephh
[1.6 6
no
learning_rate
[10-5,0.1 0.7
yes
0.3
reg_alpha
10 8 10a
yes
10-8
reg_ leg_lamdda
[1.4]
yes
boilysstopinggrounde
20
14

---

Published as a conference paper at ICLR 2025
Table MLP hy pypraarameeerr space. We implemented the MLP in PyTorch. The parameter d for the width of the MLP represents the number of features. When d >> 1024, the width is taken equal to the number of features
parameter
range
default
MLP
depth
[0.
3
width
[d, min (10d, 1024)
3d
dropout rate
[0, 0.5
0.2
Optimizer
name
Adam W
weight decay
10
learning rate
10
Scheduler name
ReduceLROnPLateau
factor
0.2
patience
10
threshold
10
General
max nb. epochs
2000
early stopping
Yes
batch size
256
Table 4: SAINT default hyperparameters. We used the implementation provided by Somepalli et al. (2021). refers to the number of features of the dataset. We did not use scheduler with SAINT, We followed the default configuration provided by the paper introducing SAINT (Somepalli al., 2021)
when there is both intersample and feature attention (i.e. attention_type colrow') )
parameter
default
SAINT
dim depth heads attn aut__dropout ff ff_dropout attentiontype name weight decay learning rate max nb. epochs early stopping batch size
32 if d 70, if d E 200), 4 if d > 200
4
0.8
0.8
colrow
Optimizer
Adam W
1
10
General
100
yes
256
15

---

Published as a conference paper at ICLR 2025
C LINK BETWEEN CORRELATION AND EFFECT SIZE.
For completeness, we recall below the relationship between correlation and effect size.
Proposition C.1 (Link between correlation and effect size.). Let X1 IR be a random variable, and E R parameter. Furthermore, define:
X2 = 3X1 -E
where X[c[X1] = 0, var (e) = o².
Then:
1
cor X2)
Proof. Let's first derive the expression the variance of X2:
Var (X2) = (X2 - E[x
=E =E ( + [X[(1))²
= E. = (3(1 - E(x,1))² +2²
var + o²
follows that:
cor (X2, X1) B[(X, B(X2]) (X, - B[X]]]
/var var (x) var (X2)
E -X E 2
var (x) var (X2)
3 var (X1)
//va((X)) (X2)
var (X1)
var (X1) +
1
o²
In this work, look the effect of imputation accuracy (X1) on prediction performance (Xxg). Hence, in a case where the imputation accuracy X1 covers a wider range of values, i.c., var(X;) larger, but the effect I and the noise o² stay the same, then the correlation between imputation accuracy and prediction performance increases.
D CRITICAL DIFFERENCE DIAGRAMS.
Figures to 9 give the Critical Difference diagrams across all predictors and imputers of average score ranks for significance level of 0.05. The difference ranks for all methods covered by same black crossbar are not statistically significant according to Nemenyi test for multiple pairwise comparisons. The colors encode the imputation type, the markers identify the model, and the line types encode the presence absence of an indicator.
16

---

Published as a conference paper at ICLR 2025
Average score ranks ladder thighte is beller).
255
7.5
10.0
12.5
15.0
22,5
-XGBoost
MLP maek (6.5
20 missforest +XCBoost
mean SAINT (8.2)
19: iteraliveßR XGBoost mask
IeestheSBR MLP (3.
16: condexp XGBoost mask
condexp MLP (3.8)
misaforest: .SAINT
llerativeRR
None. SAINT
None XGBrost
condexp MLP mask (10
16) missforest MLP
mean SAINT mask
(10
15) condexp XG8ooet
BeraliveßR SAINT
15) mean XGBcost maak
missforess XGSBoost
mask
SAINT
llerliieBR SAINT (14
meen XGBoost
Figure 6: Critical Difference diagram 20% missingness rate.
Average ladder (higher better). 10.3
20.0
mean MLP (4.45
21) missforest XGBoost mask
IteretvviiR
condexp -CHHoost mask
SAINT
MLP mask
None XGBoost
mean SAIN
17) Aeralivabli + XGHHoost mask
IlerativRRR
p mack
condexg XCiRoost
ilerativBRR XGBoost
mean XGEoost mask
SAINT
condexp SAINT
condexp MLP
missforest + XGBoost
None SANNT
15) mean XGDoust
missforesi SAINT
missforest MLP mask
condaxp mask
133 nerativebR SAINT maak
Figure 7: Critical Difference diagram- 50% missingness rate.
Average score ranks lodder (higher is better).
15
mean + MLP
24) missforest MLP mask
mean SANNT (4.7
23 misoforre MLP
mean XGBooet 8.88
miesforest XGBooot mask
mean XGBoost mask
missforest -SINNT
None XGBooet
missforest XGBoost
mean SAINT mask
missforest mask
condkop
MLP
MLP
XCBoost
MLP
tterfiivBlt XGSocst (12)
ferativeßR MLP
condexp SAINT mask [13
TenaliveBR SAINT
sondaxp XGRosst mask (13)
faraliveRR + --ANNT mask
iterabivSik XGBoost mask (14)
14) condexg SAINT
Figure Critical Difference diagram 20% missingness rate, semi-synthetic data with linear outcomes.
17

---

Published as a conference paper at ICLR 2025
lacddrr (higher
20
mean MLP (4.3)
20) condexo MLP + mask
mean SAINT
coodeoc -GiBoost mask
RerativaBR SAINT
condexo MLP
mean SAIN | mask 110
18) condexa *SAINI
mean XGRonst
18) condies -GGRoost
iterativeRR XGBocst
missforest XGBoost mask
condexa SAINT
None SAIN
mean XGBoost mask 111
leraliveBR XGBoost mask
missfores SNNT 112
14 lenalive@R MLP L mask
missforest MLP (13)
feraliveBR SAINT mask
None XCBoost (13)
misstorest XGBoost
Figure 9: Critical Difference diagram 50% missingness rate, semi-synthetic data with linear outcomes.
18

---

Published as a conference paper at ICLR 2025
E PREDICTION PERFORMANCES FOR THE SEMI-SYNTHETIC DATA.
missing_nen rale 20%
missing rale = 50%
0.04
Relative prediction R
0.02
-0.02
-0.04
MLP
SAINT
XGBoast
M.P
SAINT
XGBoost
Imputation
Add missingnees indicator
HeraifveBR
False
Figure 10: Relative prediction performances for the semi-synthetic data with linear outcomes across datasets for different imputations, predictors, and use the missingness indicator. Each boxplot represents 200 points (20 datasets with 10 repetitions per dataset). The performances shown are It2 scores on the test set relative to the mean performance across all models for a given dataset and repetition. A value of 0.01 indicates that given method outperforms the average performance on a given dataset by 0.01 on the R score.
F REGRESSION SLOPES FOR THE SEMI-SYNTHETIC DATA.
missing_rast Baatnng__reea_ 20%
missing_rate 50%
0.25
Regression coefficlent linking
prediction R² to imputation R*
Missingness
delays_
0.20
False
houses
nyc-taxd
Coof significatlly
wine quality
yes
Housing
house_aae _aales
Dalasees
house_1e
0.05
medical charges
slevators
diamondd
cpu_acc
0.00
sulfur
Bik Shar iigingg
Allerons
0.05
abalone
4_1
Brazilian houses
suparsonduct
MLP
SAINT
XGBoost
MLP
SAINT
XGBoost
recovery the prediction performance for the semi- cemi-synthetic data with linear outcomes. We report the slope of the regression line where imputation quality used to predict prediction performance.
Figure 11: Effect of the imputation
19

---

Published as a conference paper at ICLR 2025
G
EFFECT OF THE MISSING RATE.
Real outcome
Linear outcome
Effect of imputation recovery on prediction accuracy
0.12
0.10
0.08
0.06
0.04
0.02
Model
MLP
0.00
SAINT
XGBoost
Correlation b/w imputation and prediction accuracy
0.8
Missingness ind.
0.7
False
True
0.6
0.4
0.3
0.2
20
50
70 20
50
70
missing rate
missing rate
Figure 12: Increasing missing rates lead to larger effects (slopes) but noisier associations (lower correlation). The values reported are median across datasets, for each model. Data MCAR.
Figure 12 shows that effects are larger for higher missing rates: this is particularly clear for linear outcomes, but less for real outcomes. This suggests that imputation matters more at higher missing rates, although for real outcomes and powerful models, these effects are still very small. By contrast, correlations decrease when the missing rate increases, i.e, the association is noisier (less likely to be significant).
20

---

Published as a conference paper at ICLR 2025
H
FEATURE IMPORTANCE DEPENDING ON THE MISSINGNESS STATUS.
1.2
Missingness indicator
False
True
festure more important when imputed
Dataset
1.0
Ailerons
feature important Imputed
Sharing Demand
Frazilian_houess
Miamitousiing2016
Importance when missing Imporrtencs observes
0.8
abalone
cpu_act act_act
00
delays_aurich transport
diamonds
hause 16H
house sales
0.4
houses
medical ccaaggeaareeee
nyc taxi green 2016
pel
0.2
sulfur
superconduct
wine _etee_ullttyyyy yprop 4 1
0.0
condexp
missforest
Imputation
Figure 13: Ratio of importances (when missing over when observed) for the two most important
features of each dataset. Importances are calculated with feature permutation. The 2 most important features per dataset are identified based on the whole test set. For each feature j. a permutation importance is then calculated based on the subset of test samples where feature ÿ is missing, and the subset where point refers to one feature, and its color identifies the dataset it belongs to. A ratio of 1 indicates that the feature is as important whether it imputed or observed (red line). A ratio of 0.1 means that it is 10 times less important when it is imputed compared to when it is observed.
observed. The ratio between these two values is then reported on the figure, where
This experiment was conducted using XGBoost with condexp or missforest imputation, both with and without the mask. Figure 13 indicates that average, feature is half as important when imputed compared to when observed, with considerable variability (i.e., many features are 10 times less important when imputed, and some features remain important when imputed). When mask is used, importances drop significantly more with missforest imputation compared to when no mask is used (Wilcoxon signed-rank test p-value 0.01). However, this effect significant with condexp imputation.
21

---

Published as a conference paper at ICLR 2025
I INVESTIGATING THE ROLE OF THE MISSINGNESS INDICATOR.
1.1 PREDICTION PERFORMANCE GAINS WHEN USING THE MISSINGNESS INDICATOR VERSUS IMPUTATION ACCURACY
missing_ rate 20%
missing_ rate 50%
Datasets
hause 16H
Regression coefficient linking imputation to the gain in prediction R due to the mask
apu act
elevators
wine quality
Brazilian houses
0.1
house_sales
sullur
Allerons
Bike Sharing
diamonds
houses
medical charges
-0.1
MiamiHousing
nyc-taxi
pol
superconduct
0.2
abalone delays, zurich
yprop_4
MLP SAINT XGBoost
MLP
SAINT XGBoost
Real outcome.
missing_ rraee 20%
missing_ rate 50%
Datasets
house_ 16H
Regression coefficient linking imputation to the gain in prediction R² due the mask
cpu_at act
0.000
elevators
wine_quality quality
wine
0.025
Brazilian houses
house,
-0.050
Allerons
Bike Sharing
diamonds
-0.075
houses
medical charges
MiamiHousing
-0.100
nyc-taxi
-0.125
superconduct
abalone delays zurich
-0.150
yprop 4
MLP
SAINT XGBoost
MLP
SAINT XGBoost
(b) Semi-synthetic linear outcome.
Figure 14: Efffec of imputation accuracy the improvement in prediction when using the mask, compared to not using it.
Most effects are negative, indicating that using the missingness indicator brings the largest boost in prediction performance when imputations have low accuracy. Moreover, effects are strongest for the MLP and smallest for XGBoost, meaning that with more powerful models, prediction boosts due to the missingness indicator are less pronounced.
22

---

Published as a conference paper at ICLR 2025
1.2
SHUFFLING THE MISSINGNESS INDICATOR.
Performances with shuffled missingness indicators.
+.05
Relative predictioon R
-05
MLP shuffled missingness indicator True
SAINT
XGBoost
Imputation
missforest
RoratveBR condexp
None
(a) Effect appending a shuffled mask on prediction performances. Real outcome, 50% MCAR missingness.
missing rate
50%
0.25
Regression coefficient linking
prediction R2 to imputation R²
Add shuffled missingness indicator
delays_ zurich
0.20
False
houses
True
nyc-taxi
0.15
Coef significantly >
wine Line_quallyy
MiamiHoousing
0.10
no
house_ssd sales
0.05
Datasets
house_16 16H
medical medical_charges
elevators
0.00
diamonds
opu_act
sulfur
pol
0.05
Bike Shaarinh
Allerons
abalone
yprop_4 4_1
-0.10
Brazilia_houses
superconduct
MLP
SAINT
XGBoost
(b) Comparing effect sizes of imputation accuracy on prediction accuracy, with a shuffled mask versus without mask. Real outcome, 50% MCAR missingness.
Figure 15: Effects of appending a shuffled mask
We repeat experiments with shuffled missingness indicator, where the columns of the indicator shuffled for each sample. This preserves the total number of missing values per sample but removes information about which specific features missing.
Figure 15a demonstrates that using a shuffled missingness indicator harms prediction performance, except for XGBoost for which performances unchanged. In contrast, the true missingness indicator improves performances (fig. 1). Furthermore, the shuffled indicator does not affect the relationship between imputation accuracy and prediction accuracy (fig. 15b), whereas the true indicator reduces the effect size (fig. 4).
These results confirm that the benefit of the missingness indicator due regularization or merely encoding the number of missing values. Prediction models effectively leverage information about which features are missing, even though under MCAR, this information is unrelated to the unobserved values.
23

---

Published as a conference paper at ICLR 2025
J COMPUTATION TIMES PER METHOD.
add mask = False | mmsssnng_rake rate 20%
add mask False missing. rate 50%
10"
lime (secondss
10
imputatin
mean
missing_rate
missing_rakt
iteraliveBR
condexp
missforest
None
time (secandss
10²
MLP
SAINT
XGBoost
MLP
SAINT
XGBoost
Figure 16: Running time for each model, including the 50 iterations of hyperparameter search for XGBoost and MLP.
24

---

Published as a conference paper at ICLR 2025
add_mask = False | mmsssingraate rate = 20%
add_ mask False missing_ rate 50%
time (secards)
10
10²
Imputation
mean
add mask Iissinn_rate 20%
True missing_ratt 50%
IterativaßR condexxp
None
lime (secoonds)
10
10²
MLP
SAINT
XGBoost
MLP
SAINT
XGBoost
model
model
Figure 17: Running time for each model for the semi-synthetic data with linear outcomes, includ- ing the 50 iterations of hyperparameter search for XGBoost and MLP.
25

---

Published as a conference paper at ICLR 2025
K SCATTERPLOTS OF PREDICTION R VS IMPUTATION R FOR EACH MODEL
AND DATASET.
MLP
house 16H
cpu_act
elevators
wine quality
Brazilian houses
3.25
0.05
3.000
0.0
0.00
-0.1
-0.02
-0
-0.25
0.00
-0.25
-0.2
house sales
sulfur
Allerons
Bike Sharing
diamonds
0,0
3.000
0.000
Regression R²
-0.02
-0.1
-3.025
-0.05
-0.005
Imputation
iterative
houses
medical medlical_charges
MiamiHousing
nyc-taxi
pol
condexp
misstorest
0.00
0.000
0.00
0.0
-0.025
-0.04
-3.25
-0.1
00
-0.1 0.0
superconduct
abalone
delays_zure zurrchhuuchhhhhoo
yprop_4. 4_1
0.06
3.002
0.00
0.00
3.000
0.00
-0.05
-0.05
-3.007
-0 10
-0.10
-0.5
0.0
-0.5
-0.25
0.00
-0.25 0.00
Imputation R
Figure 18: Prediction R² vs imputation R for MLP missing rate 50%. The R² scores are given relative to the mean R² score, with the effects of experiment repetitions climinated (i.c. the effect of the train/test splits on the performance)
26

---

Published as a conference paper at ICLR 2025
MLP+mask
house 16H
cpu_as act
elevators
wine Lune_qullity
Brazilian houses
0.02
0.35
0.005
8.00
0.0
0.000
-0.05
-0.005
-0.02
-0.1 0.0 a.1
-0.2
0.0
-0.25
0.00
-0.2
0.0
-0.1 0.0 0.1
house sa
sulfur
Allerons
Bikee Sharing
diamonds
0.05
0.002
0.0%
0.02
0.000
8000
0.00
+%
0.30
2.
Regression R²
-0.0%
Imputation
-0.1
0.0
-0.2
0.0
0.2
-0.25
0.00
-0.2
0.0
-0.5
0.0
mean
iterative
houses
medical charges
MiamiHousing
nyc-taxi
pol
condexp
0.001
0.00
0.1
missforest
omm
0.30
-0.01
-0.02
-0.002
2.2
superconduct
abal delays_zurics zurich
yprop_4 ___
0.025
0.002
0.00
0.000
0.30
-0.025
-0.02
-0.05
-0.050
Imputation R²
Prediction R² vs imputation Rº for a MLP indicator missing rate 50%. The R² scores are given relative the mean R² score, with the effects of experiment repetitions eliminated (i.e. the effect of the train/test splits on the performance)
Figure 19:
27

---

Published as a conference paper at ICLR 2025
SAINT
house 16H
cpu_act
elevators
wine quality
Brazilian houses
0.05
0BS
0.02
0.01
0.00
0.00
0.00
-0.05
-0.02
-0.02
-0.01
-0.10
-0.02
-0.1 0.0 0.1
-0.2
0.0
-0.25
0.00
-0.2
0.0
3.0
0.2
house sales
sulfur
Allerons
Bike Sharing
diamonds
0.05
0,002
ana
0.00
Regression R²
-0.002
-0.02
-0
-0.02
-0.05
Imputation
-9.1
0.0
-0.2
0.0
0.2
-0
25
0.00
0.23
-0.2
0.0
-0.5
0.0
mean
iterative
houses
medical__harges
MiamiHousing
nyc-taxi
pol
condexp
0.01
0.05
0.02
missforest
0.02
0.00
0.00
0.05
0.00
-0.01
-0.05
superconduct
abalone
delays_zurich
yprop_4 4__
0.05
0.05
0.00
-0.002
Imputation R²
Figure 20: Prediction R² vs imputation Ra for SAINT missing rate 50%. The R scores are given relative to the mean R² score, with the effects of experiment repetitions eliminated (i.e. the effect of the train/test splits on the performance)
28

---

Published as a conference paper at ICLR 2025
SAINT
house 16H
cpu_act
elevators
wine, quality
Brazilian
Brazilian_houses houses
0.05
0.05
0.02
0.005
0.00
0.00
0.000
-0.05
-0.005
-0.00
-0.05
-0.3
0.0 0.1
-0.25
0.00
25
0:00
-0.2
0.0
-0.1 n.0
0.1
house sales
sulfur
Allerons
Bike Sharing
diamonds
0.02
0.025
0.01
0.00
0.60
Regression R²
-0.025
Imputation
-0.01
-0.1
0.0
-0.23
0.00
-0.25 0.00
-0.25
0.00
-0.5
0.0
mean
iterative
houses
medical chaares
MiamiHousing
nyc-taxi
pol
condexp
0.02
0.001
0.050
0.0022
0.30
missforest
0.025
0.0000
0.00
-0.01
-0.001
-0.0025
-0.05
0.0
superconduct
abalone
delays_zuriin zurich
yprop_4_
0.005
0.001
0.000
0.00
0.800
0.80
-0.05
-0.001
-0.02
Imputation R²
Prediction R² vs imputation Rº for SAINT indicator missing rate 50%. The R² scores are given relative the mean R score, with the effects of experiment repetitions eliminated (i.e. the effect of the train/test splits on the performance)
Figure 21:
29

---

Published as a conference paper at ICLR 2025
XGBoost
house 16H
cpu act
elevators
wine wine_ qunaquality
Brazilian houses
0.02
3.5
0.05
0.01
2.00 0.00
0.00
-0.05
0.01
-0.1 0.0 a.1
-0.2
0.0
-0.25
0.00
-0.2
0.0
-0.1 0. 0.1
house sales
sulfur
Allerons
Bike Sharing
diamonds
0.02
0,002
0.00
3.00
0.00
0.00
Regression R²
-0.02
Imputation
-0.09
-0.002
-0.3
0.0
-0.2
0.0
0.2
-0
0.00
-0.2
3.0
-0.5
0.0
mean
iterative
houses
medical charges
MiamiHousing
nyc-taxi
pol
condexp
0.05
0.02
missforest
0.05
0.02
0.00
0.00
0.00
-0.02
-002
-0
0000000
0.05
3.2
superconduct
abalone
delays_zurich
yprop_4 4_
2.00
0.000
-0.025
-0.05
-0.002
Imputation R²
Figure 22: Prediction R² vs imputation R for XGBoost missing rate 50%. The R scores are given relative the mean R² score, with the effects of experiment repetitions eliminated (i.e. the effect of the train/test splits on the performance)
30

---

Published as a conference paper at ICLR 2025
XGBoost+mask
house 16H
cpu_act
elevators
wine quality
Brazilian houses
0.02
0.02
0.025
0.02
0.055
0.000
0.00
0.000
-0.02
0.025
-0.02
-0.035
-0.3 6.0 0.1
-0.25
0.00
-0.25
0.00
-0.2
0.0
-0.3 0.0
0.1
house sales
sulfur
Allerons
Bike Sharing
diamonds
0.04
0.01
0.02
0.001
0.00
0.00
0.90
0.000
Regression R²
-0.02
-0.02
-0.001
Imputation
-9.1
0.0
25
0.00
2G
0.00
-0.20
0.00
-0.5
0.0
mean
iterative
houses
medical chargges
MiamiHousing
nyc-taxi
pol
condexp
0.003
0.050
0.01
missforest
0.025
0.025
0.000
0.00
0.030
-0.01
-0.001
-0.025
-0.01
-0.025
-0.25
superconduct
abalone
delays_zurich
yprop_4_1
0.00
0.001
0.00
0.800
-0.05
-0.00
-0.001
-0.10
Imputation R²
Figure 23: Prediction R2 vs imputation R2 for XGBoost indicator missing rate 50%. The R² scores are given relative the mean R² score, with the effects of experiment repetitions eliminated (i.e. the effect of the train/test splits on the performance)
31

---

Published as a conference paper at ICLR 2025
MLP
house 16H
cpu_act act
elevators
wine quality
Brazilian houses
0,22
0.00
0.00
-0.02
-0.02
-0.2
-0.2
0.0
-0
0.00
-0.2
0.0
-0.3 0.0
house_
sulfur
Allerons
Bike Sharing
diamonds
0.00
0.00
0.00
0.00
Regression R²
-0.51
-0.01
Imputation
-0.02
-0.02
-0.1
-0.25
-0.5
mean
iterative
houses
medical chaaggee_easssee
MiamiHousing
nyc-taxi
pol
condexp
0.02
missforest
0.00
0.00
-0.02
-0.02
-0.04
superconduct
abalone
delays_
yprop 4
0.02
0.00
0.00
-0.025
-0.02
-0.02
-0.050
-0.5
0.0
-0.5
-0.2
-0.2
0.0
Imputation R²
Figure 24: Prediction R² vs imputation R for a MLP semi-synthetic data with linear outcomes, missing rate 50%. The n² scores are given relative to the mean R² score, with the effects of experiment repetitions eliminated (i.e. the effect of the train/test splits the performance)
32

---

Published as a conference paper at ICLR 2025
MLP+mask
house_ 16H
cpu_act _ct
elevators
wine quality
Brazilian houses
0.01
0.01
0.02
0.005
0.00
0,00
0.000
-0.01 -0.01
-0.02
-0.1 30
-0.25
0.00
+0.
1100
-0.2
0.0
11.0
house sales
sulfur
Ailerons
Bike Sharing
diamonds
0.01
0.010
0.0025
0.00
0.005
0.0030
0.00
Regression R²
-0.02
-0.01
-0.005
Imputation
-0.1
0.0
-0.25
0.00
-0.25 0.00
-0.25
3.00
-0.5
0.0
mean
houses
medical charges
MiamiHousing
nyc
pol
condexp
0.005
0.02
0.05
misstorest
0000
0,000
0.00
0m0
0.00
-0.001
-0.04
superconduct
abalone
delays_zurich
yprop_1_1
0.07
0.00
-0.01
-0.02
-0.02
Imputation R²
Prediction R imputation R² for MLP + indicator semi-synthetic data with linear outcomes, missing rate 50%. The R² scores are given relative to the mean H² score, with the effects of experiment repetitions climinated (i.c. the effect of the train/test splits on the performance)
Figure 25:
33

---

Published as a conference paper at ICLR 2025
SAINT
house 16H
cpu cpu_act act
elevators
wine quality
Brazilian houses
0.04
1.7
0.00
0.00
0.00
00.02
-0.01
-0.01
-0.92
0.01
-0.2
0.0
-0.25
0.00
20
0.00
-01
0.0
-0.10.0
house sales
sulfur
Allerons
Bike Sharing
diamonds
0.01
0.600
0.00
Regression R²
-0.005
Imputation
-0.02
mean
iterative
houses
medical medical_chargee _charges
MiamiHousing
nyc-taxi
pol
condexp
0.81
0.01
missforest
0.00
0.025
0.00
0.00
-0.01
0,000
-0.92
-0.025
-0.05
superconduct
abalone
delays zurich
yprop_4 4
0.01
0.01
0.09
0.00
0.00
0.00
0001
0.001
-0.92
-0.02
-0.02
-0.01
-0.5
-0.5
0.0
-0.25
0.00
-0.25 0.00
Imputation 8²
Figure 26: Prediction R² vs imputation RD for SAINT semi-synthetic data with linear outcomes, missing rate 50%. The n² scores are given relative to the mean R² score, with the effects of experiment repetitions eliminated (i.e. the effect of the train/test splits the performance)
34

---

Published as a conference paper at ICLR 2025
SAINT maskk
house_1611
cpu_act acc
elevators
wine quality
Brazilian houses
0.01
0.02
0.01
0.00
0.00
0.30
-0.01
-0.01
-0.01
-0.01
-0.2
0.0
-0.2
0.0
-D 25
0.00
-0.2
-0.1 u.a u.1
house sales
sulfur
Ailerons
Bike Sharing
diamonds
0.01
0.01
0.000
Regression R²
-0.01
Imputation
-0.1
0.0
-0.2
0.0
-0.25 0.00
-0.2
0.0
0.2
-0.5
0,0
mean
houses
medical charges
MiamiHousing
nyc
pol
condexp misstorest
0.050
0.03
0.025
0.000
0.00
0.000
-0.001
0.000
-0.002
-0.05
-0,002
-0.025
-0.2
superconduct
abalone
yprop_4,
0.02
0.01
0.00
0.00
0.00
-0.91
-0.0
-0.04
Imputation R²
Prediction R² imputation R² for SAINT indicator semi-synthetic data with linear outcomes, missing rate 50%. The R² scores are given relative to the mean H² score, with the effects of experiment repetitions climinated (i.c. the effect of the train/test splits on the performance)
Figure 27:
35

---

Published as a conference paper at ICLR 2025
XGBoost
house_ 15M
cpu acc
elevators
wine, qualityaltyyyy
Brazilian houses
0.01
0.01
0.02
0.00
1100
0.00
-0.01
-0.01
-0.02
-0.01
-0.2
0.0
-0.2
0.0
-0.25
0.00
-0.2
0.0
1000
022
house sales
sulfur
Ailerons
Bike Sharing
diamonds
0.01
0.02
0.003
0.01
0.000
Regression R²
-0.01
-0.01
0.01
-0.003
Imputation
-9.1
0.0
-0.2
0.0
-0.25 0.00
-0.2
0.0
-0.3
0.0
mean
houses
medical charges
MiamiHousing
nyc
condexp
D.s
0.02
misstorest
0.00
0.00
-0.01
-0.02
-0.00
-0.01
-0.04
superconduct
abalone
delays_zurice
yprop_4
0.01
0.01
0.02
0.00
-0.0
-0.02
Imputation R²
R² for XGBoost semi-synthetic data with linear outcomes, missing rate 50%. The R² scores are given relative to the mean H² score, with the effects of experiment repetitions climinated (i.c. the effect of the train/test splits on the performance)
Figure 28:
Prediction
R²
vs
imputation
36

---

Published as a conference paper at ICLR 2025
X
house 16H
cpu_act
cpu act
elevators
wine quality
Brazilian houses
0.00
0.00
0.30
0.000
-0.01
0.02
-1.2
6.0
-0.2
0.0
20.25
0.00
-0.2
-0.1 cuu o.1
house sales
sulfur
Ailerons
Bike Sharing
diamonds
6.01
0.0050
0.00
0.00
0.0000
Regression R²
Imputation
-0.01
-0.01
-0.0023
-0.1
0.0
-3.2
0.9
-3.25 0.00
-0.2
0.0
-0.5
0,0
mean
medical charges
MiamiHousing
nyC
pol
condexp misstorest
0.005
0.02
0.000
0.000
0.001
0.30
-0.005
-0.002
-0.02
-0.005
-0.02
superconduct
abalone
delays_zurice zurich
yprop_1.
0.21
0.02
0.00
0.00
0.30
-0.02
Imputation R²
Figure 29: Prediction R imputation R² for XGBoost + indicator semi-synthetic data with linear outcomes, missing rate 50%. The R² scores are given relative to the mean H² score, with the effects of experiment repetitions climinated (i.c. the effect of the train/test splits on the performance)
37

---

Published as a conference paper at ICLR 2025
L MNAR SCENARIO.
L.1 DESCRIPTION OF THE SELF-CENSORING MECHANISM.
The MNAR mechanism implemented probit self self-masking, defined for any feature j as: P(M, 1 ( cj)
where & denotes the probit function and X E R, Cj E TE. its slope and bias.
1. to have missingness Denoting by oj the standard deviation of feature j, we chose X3 = 2ng
probability that smoothly increases over the support of the data. The bias is then fixed to impose desired missing rate r based on proposition L...
Proposition L.1 (Achieving targeted missing rate with probit celf-censoring). Assume that the random variable X R follows a Gaussian distribution and is affected by probit self-masking mechanism, i.e,
X ~ ((X and P(M = 1|X) = 4XX cc),
where X E R and c E R control the slope and shift of the self-masking function. Given a fixed slope Ag a missing rate r is achieved by choosing:
CO do 11- (1)
Proof.
== P(M =1)
(1)
-
P(M = 1XX)X(X)XX
(2)
-
$(AX - -N
M-1
- r= $
Vx-2+0²
- c= (p - (r)( -2 +
(5)
where eq. (4) is obtained according to equation 4.152 in Bishop.
Experiments show that the target missingness probability is achieved even though the features are Gaussian.
38

---

Published as a conference paper at ICLR 2025
L.2
RESULTS UNDER MNAR MISSINGNESS.
missing rate- 20%
missing rate 50%
Relalive prediclion R²
MLP
SAINT
XGBoost
MLP
SAINT
XGBoost
Impulation
Add missingness indicalor Falsa
Figure 30: Relative prediction performances across datasets for different imputations, predictors, and use the missingness indicator under MNAR missingness. Each boxplot represents 200 points (20 datasets with 10 repetitions per dataset). The performances shown are Ht scores on the test set relative to the mean performance across all models for a given dataset and repetition. A value of 0.01 indicates that given method outperforms the average performance on the n² score. Corresponding critical difference plots in figs. 31 and 32.
given dataset 0.01
Awerage score (higher better).
2.5
7.5
10.0
12.5
15.0
17.5
20.0
22.5
meen MLP 13.4
22 missforest XGBoost mask
condexo
SaraliveHR XGHooost mask
XGRonst
terativelSR MLP
condexo XGiRoost
condexp SAINT (8.5
19 missforest + SANT mask
mean MLP mask
17) mean XGBoost mask
condexg XGBoost
MerralivBR -SINT mask
miseforest MLP
16) missforest MLP mask
condexp
MerativeBR
XGBoost
iterativei XGBonst
14) Nonc SAIN!
mean SAINT mask (13)
14) missforest SAINT
Figure 31: Critical Difference diagram 20% missingness under MNAR missingness.
Average score ranks ladder (higher better).
14
16
mean
missforest -GBBoost
IeeattivesR
XGibonst
IterativBR SAINT 17.4
20) condexx + XGBBoost mask
condesp MLP (8.3
19 missforest SANT mask
missforest
18 reraliveliR XGBoost mask
condexp SAINT
moan SAINT
mask
ilera ReralveRR XGDoosl
XGDoost
moan SAINT (10
161 None SAINT
condexp XGBoost 11
16) BerallveBR SAINT maok
mean MLP mask
missforest MLP mask
mean MLP
missoorest XGBooott
Figure 32: Critical Difference diagram 50% missingness rate under MNAR missingness.
39

---

Published as a conference paper at ICLR 2025
missing
Delasel
medkor charges
BeraliveBR
dimmadds
Shar_Shhrng_eeeo Demaard
condexp
missforest
dolays transport
houses
0.25
myp-lae grean-ees 4016
rn cuall
missing
M snoHooshgg0016
house, sales
house 161
lterativBRR
condexp
missforest
ypmp, 4 superooniudt
imputation R²
Imputation time
Figure 33: Left: Imputer performance for recovery under MNAR missingness. Performances are given as R² scores for each dataset relative to the mean performance across imputation techniques. A negative value indicates that method perform worse than the average of other methods. Right: Imputation time under MNAR missingness.
missinggnée
misknggratee
0.5
Missingraos indioetor
nyostai
tn coelloerd lnking
prodecton "5 mppbttinn
Falee
wne_qqaiity
MamiHouusing
house salus
medial charges
hause 16H
dismonds
slewsirns
opuaact
0.05
Bike, Sharing
pol
abralone
Nierons
0.00
Braziian
delays superconduct
-0.05
housea
MLP
SAINT
XGBoost
MLP
SAINT
XGBoost
Figure 34: Effect of the imputation recovery on the prediction performance under MNAR missingness. We report the slope of the regression line where imputation quality is used predict prediction performance.
40


Reviews:
[{"review_id": "uKPhvbD8ab", "reviewer": "Reviewer_ETS8", "summary": "This paper empirically studies the effects of various missing value imputation methods, in particular whether better imputation accuracy yields higher prediction performance. The authors conduct experiments on nineteen benchmark datasets using four imputation methods, three different models, as well as variations such as missingness indicators and semi-synthetic linear labels. They conclude that imputation accuracy does affect prediction performance, although the gains are quite small, and the effect is further reduced if the model is more expressive, missing value indicators are used, or the target variable has a non-linear relationship to the covariates.", "questions": "The observation about good imputations having less effect when using missingness indicators was very interesting. To test the intuition mentioned in Section 4.5, I think it would be interesting to use explainability techniques on these models trained with missingness indicators to see if the importance/weights of features drop when they are missing (i.e. inputs with the corresponding missingness indicator on).\n\nDoes the correlation of prediction performance and imputation accuracy (e.g, Figures 3 and 4) also show any difference by imputation method?\n\nWould larger missingness rates show more significant correlations?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "The paper presents an empirical study that can help us better understand the effect of various imputation techniques for different model/problem scenarios, which is an important question as missing values are ubiquitous and often handled by imputation. The paper specifically considers the link between imputation accuracy and prediction performance, which have been studied theoretically in some scenarios but not empirically for real-world problems.\n\nThe experimental setup is clearly described in detail for reproducibility, and the authors mention that code will also be available upon publication. The experiments are also thorough in terms of datasets, models, and imputation methods.\n\nRelated work discussion is clear, including both theoretical and empirical studies about missing value imputation methods.", "weaknesses": "Many claims/conclusions in the paper are based on small differences in average values with overlapping confidence intervals, and there are no statistical tests for significance. For example, the authors claim that the imputation techniques considered in the experiments show diverse imputation performance (Figure 2), but apart from mean imputation being worse than others, there doesn’t seem to be a definitive difference in quality by the other three imputers. \n\nIn addition, the claim about correlation of imputation accuracy and prediction performance is made based on the small but positive slope of the regression line (Figure 4). However, the slope seems very small in most cases to suggest this, especially in the 20% missing case, and the correlation may still be very weak (small correlation coefficient). On the other hand, the authors suggest that good imputations matter more in the linear response case (semi-simulated data), based on the slightly higher correlation (Fig 5), but the slope seems still very small (Fig 11).\n\nIn about a third of the benchmark datasets, the dimension is small such that 20% missing means just one missing feature. I would expect the difference between simple and more complex imputation techniques to be pretty small in such cases. \n\nThe experiment in MNAR scenario (Section 4.4) didn’t feel very connected to the rest of the paper. As the authors also noted, it is well known that most imputation methods (without considering causal structures) are not valid under MNAR.\n\n-------------\nPost-rebuttal: Thanks to the authors for the detailed response and revision. I think the additional experiments and statistical tests made the results of this paper much more convincing, and I am happy to raise my score to 6.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "CB8MiupMdC", "reviewer": "Reviewer_JGDK", "summary": "The authors use an empirical approach to investigate whether advanced imputation techniques significantly improve predictive accuracy in models with missing data. The study finds that while sophisticated imputation can enhance prediction in certain contexts, the benefits are modest, especially when more expressive models and/or missingness indicators are used. They conclude with the assertion that resources might be better allocated to models that inherently handle missing data rather than focusing extensively on imputation improvements.", "questions": "Please eliminate redundant sentences, remove unnecessary adverbs (\"really\", \"interestingly\", etc), remove unnecessary metadiscourse (e.g., let me tell you what I'm going to tell you- the end of section 1 ) and acronyms that were defined not on their first occurrence (or never at all). Missingness indicators, which are a prominent concept in the paper, should be defined in one sentence in the introduction. Also, the quality and clarity of the writing varies greatly from paragraph to paragraph - can you make it more consistent? \n\nLine 140: what do you mean by a \"universally consistent\" algorithm?\n\nWhy is the 'semi-synthetic' data (top of page 5) 'semi'? Can you elaborate on that?\n\nLine 341 uses the word \"nuanced\" in a strange way - so strange I couldn't figure out what the sentence in which it appears actually means.\n\nYou say 'comparing imputers is not our main objective' (twice), but it is an important set of results, right? Clearly, it is a salient result that a mean imputer does worse than other methods for both 20% and 50% missingness rates, and you could envision people citing this paper for that result. Perhaps these results should not be downplayed? \n\nPlease re-read and consider linearizing the order/structure of your sentences. For example, line 456 reads \"For prediction, imputation matters but marginally.\" You will minimize the amount of cognitive bandwidth / patience of your readers if you linearize the sentence structure, e.g.: 'Imputation matters marginally for prediction.\" If that is too much of a change of meaning, then consider 'Imputation matters for prediction, but only marginally.' These kinds of reverse-order sentences occur throughout the manuscript and, in aggregate, bog down the reader.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 1, "contribution": 3, "strengths": "The value of this paper arises from its significance, not its originality. What I mean by this is that the paper effectively reports a \"null\" result: something that *doesn't* work. This kind of work is valuable because it saves the community time by not re-inventing square wheels. We need to know what doesn't work. \n\nIf you count papers that provide evidence that a method doesn't work (I do) as original, then yes this paper is original. This definition of originality fits under the notion of originality as \"a new definition or problem formulation,\" because the authors are recommending that the data imputation research solves the wrong problem if the goal is improved prediction.\n\nThe quality of the work was adequate, they performed a comprehensive study using several different models and many different datasets to arrive at a convincing analysis of model performance.", "weaknesses": "I did not mention clarity in the Strengths section because the paper needs a lot of stylistic work. The authors would benefit from reading Strunk & White's \"Elements of Style\" as well as Steven Pinker's \"A Sense of Style.\" There were many redundant sentences, unnecessary adverbs (\"really\", \"interestingly\", etc), unnecessary metadiscourse (e.g., let me tell you what I'm going to tell you) and acronyms that were defined not on their first occurrence (or never at all). Missingness indicators, which are a prominent concept in the paper, should be defined in one sentence in the introduction. In addition, different paragraphs seemed to have different authors, where at least one author appears to not have an adequate grasp of English. I'm not judging that (I don't speak any other language, hats of to them if that is the case) but there are resources to help non-native English speakers (even ChatGPT now does a reasonably decent job: write a paragraph, input it along with \"improve:\" and then see how the output paragraph is written- I find this helpful for succinctness but it can also help with sentence structure/word choices). All of these weaknesses combined to make reading the paper feel arduous.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "WQl9kvClCD", "reviewer": "Reviewer_RyBn", "summary": "This paper explores the link between imputation quality and downstream performance. Through multiple experiments, the paper demonstrates that under MCAR patterns, the quality of reconstruction error is not always linked with performance, depending upon modelling strategies.", "questions": "None", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "The paper is well written and easy to read, presenting novel results even under the simple MCAR setting.", "weaknesses": "The critical insight that adding missing indicators can be beneficial, even under MCAR, needs more justification (with a potential theoretical justification). Intuitively, I suggest exploring the correlation between reconstruction error and gain from adding the indicator. I believe the correlation should be strong as the model is able to 'discard' badly imputed data. Additionally, an experiment with a random mask appended would demonstrate that this result is not just a product of a larger number of parameters in the model or some regularisation.\n\nOther works have explored this relation. I would recommend comparing the results with work such as Bertsimas 2024, where the authors conclude, \"While common sense suggests that a 'good' imputation method produces datasets that are plausible, we show, on the contrary, that, as far as prediction is concerned, crude can be good.\"\n\nBertsimas D, Delarue A, Pauphilet J. Simple Imputation Rules for Prediction with Missing Data: Theoretical Guarantees vs. Empirical Performance. Transactions on Machine Learning Research. 2024 Jun 5.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "C8HiIFFYAq", "reviewer": "Reviewer_9esu", "summary": "The paper investigates the empirical importance of accurate imputation in tabular, continuous data with MCAR missingness. It compares four common imputation techniques and three model architectures (2x NN, 1x tree-based) across 19 real-world, medium-sized (<50,000 samples), fully-observed datasets augmented with simulated missingness.\n\nThe authors conclude that good imputation yields marginal improvements in terms of prediction performance and that missingness indicators may be beneficial even in MCAR scenarios.", "questions": "Given that both key findings are consistent with established theory and have been described before empirically, what are the novel contributions that would justify acceptance of the paper?", "limitations": "", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "The study performs an extensive experiment across a variety of imputation methods, prediction models, and datasets.", "weaknesses": "The authors conclude that both better imputations and missingness indicators improve prediction performance. Neither of these findings is particularly surprising if we consider the optimal prediction with MCAR data: \n\n$\\mathbb{E}[Y \\mid X_o, M] = \\int_{X_m} f^\\star(X_m, X_o)p(X_m \\mid X_o)dX_m,$\n\nwhere, in line with Le Morvan et al. (2021), $Y$ is the outcome, $X_o$ and $X_m$ are the observed respectively missing covariates, and $f^\\star$ is the underlying full-data function. For the type of conditional mean imputation used in this work, we do not target $p(X_m \\mid X_o)$ but instead estimate $\\mathbb{E}[X_m \\mid X_o]$. In the extreme, we have perfect knowledge of this expectation through an oracle, and Le Morvan et al. (2021) in their Figure 4 already demonstrated that the performance of such an oracle > imputation via chained equations > mean impute. In the same figure, they also show that missing indicators are beneficial even in MCAR settings. The authors must be aware of this work, since they cite it as their main reference in their section \"4.5 WHY IS THE INDICATOR BENEFICIAL, EVEN WITH MCAR DATA?\", where they exclusively lean on Le Morvan et al. (2021) to explain why missingness indicators, even if they contain no information about the outcome, may still aid learning an often discontinuous optimal predictor.\n\nReferences: \nMarine Le Morvan, Julie Josse, Erwan Scornet, and Gael Varoquaux. What’s a good imputation to predict with missing values?  Advances in Neural Information Processing Systems, volume 34, pp. 11530–11540. 2021.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "G57EvGEU3R", "reviewer": "Reviewer_NNbq", "summary": "This study quantifies the relationship between imputation accuracy and prediction performance across various datasets, showing that improvements in prediction are typically only 10% or less of the gains in imputation \\(R^2\\). It finds that advanced imputation methods are less beneficial with more flexible models (e.g., XGBoost) and when using missingness indicators, which improve predictions even in MCAR scenarios. The authors highlight that in MNAR settings, the impact of better imputation is likely even smaller.", "questions": "- Pg. 2., line 70: The MNAR definition could be clarified by stating that missingness is related to the unobserved values themselves, making it informative. In this case, the probability of missing data is related to the actual values that are missing. Also, the MAR and MNAR abbreviations should be spelled out in the first instance. \n\n- Pg. 2, line 103: the sentence could be clarified, e.g., Woźnica & Biecek (2020) trained imputers separately for the training and test datasets, which led to an 'imputation shift'—a situation where the imputation patterns between the two datasets differ, causing inconsistencies in the data used for model training versus model evaluation.\n\n- Pg. 5, it may be informative to state in the “Computational resources” section or in the Appendix the computational hardware (e.g,. number of CPUs) used to conduct the experiments. \n\n**Typos/grammar**:\n- Pg. 1, line 52: missing parentheses\n- Pg. 2, line 85: “mportant”\n- Pg. 5, line 223: It’s been previously stated that for XGBoost and the MLP, hyperparameters are optimized using Optuna (Akiba et al., 2019)\n- Pg. 7, line 348: proper casing for figures (“fig. 2”)\n- Pg. 8, line 415: citation for Pereira 2024 not included in the reference list", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "The paper effectively bridges theoretical work on the minimal impact of imputation in asymptotic settings with empirical studies of the \"impute-then-predict\" pipeline in supervised learning, offering a rigorous evaluation of when imputation accuracy affects prediction. The authors position their work within the existing literature by addressing both empirical evaluations of imputation methods and theoretical frameworks. The evaluation spans 19 datasets, multiple imputation methods, and prediction models across MCAR and MNAR settings, with detailed analysis of relative prediction performance and imputation time. The use of critical difference plots establishes upper bounds on the benefits of imputation, particularly in best-case MCAR scenarios.", "weaknesses": "- When drawing conclusions from Figs. 1 and 2 (last paragraph of Sec. 4.1), the authors could address the conditions under which advanced imputers do provide a benefit. For example, while XGBoost might not benefit much from MissForest, other less flexible models like MLP show greater improvements when paired with more advanced imputations. \n\n    The authors might also explore non-linear or hierarchical relationships within the datasets where advanced imputations could outperform simpler methods more consistently. In cases where feature interactions are complex, MissForest could have greater effects.\n\n- Regarding “Good imputations matter less when the response is non-linear” (pgs. 7-8), the authors could strengthen the argument about non-linearities disrupting the relationship between imputation quality and prediction performance by incorporating insights from Le Morvan et al. (2021), which shows that *even with high-quality imputation*, non-linear functions can introduce discontinuities, making them harder to learn optimally. They could further illustrate how the *non-continuous nature of the regression function* after conditional imputation (as discussed in the paper) increases the complexity of the learning process in non-linear settings, which explains why non-linearities amplify the noise in the imputation-prediction relationship.\n\n- On the question of whether the missingness indicator is the optimal way to represent missingness (pg. 9),  the authors could discuss specific limitations of the indicator in more detail, such as potential over-reliance on the indicator or the risk of introducing additional noise into the model. The authors cite the polar encoding paper of Lenz et. al. 2024, but could  suggest additional methods to represent missingness more effectively, such as embedding missingness directly within the model architecture or using neural architectures like NeuMiss that incorporate missing patterns in a more structured way (Le Morvan et al., 2021).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Spotlight)