Imputation for prediction: beware of diminishing returns.

## Abstract

Abstract Missing values are prevalent across various fields, posing challenges for training and deploying predictive models. In this context, imputation is a common practice, driven by the hope that accurate imputations will enhance predictions. However, recent theoretical and empirical studies indicate that simple constant imputation can be consistent and competitive. This empirical study aims at clarifying if and when investing in advanced imputation methods yields significantly better predictions. Relating imputation and predictive accuracies across combinations of imputation and predictive models on 20 datasets, we show that imputation accuracy matters less i) when using expressive models, ii) when incorporating missingness indicators as complementary inputs, iii) matters much more for generated linear outcomes than for real-data outcomes. Interestingly, we also show that the use of the missingness indicator is beneficial to the prediction performance, even in MCAR scenarios . Overall, on real-data with powerful models, improving imputation only has a minor effect on prediction performance. Thus, investing in better imputations for improved predictions often offers limited benefits.

## 1 Introduction

Databases are often riddled with missing values due to faulty measurements, unanswered questionnaire items or unreported data. This is typical of large health databases such as the UK Biobank (Sudlow et al., ), the National Health Interview Survey (Blewett et al., ) and others (Perez-Lebel et al., ). Statistical analysis with missing values has been widely studied, particularly to estimate parameters such as means and variances (Little and Rubin, ). However, how to best deal with missing values for prediction has been less studied. Since most machine learning models do not natively handle missing values, common practice is to impute missing values before training a model on the completed data, often with the expectation that “good” imputation improves predictions. Considerable efforts have been dedicated to improving imputation techniques, utilizing Generative Adversarial Networks (Yoon et al., ), Variational AutoEncoders (Mattei and Frellsen, ), optimal transport (Muzellec et al., ) or AutoML-enhanced iterative conditional imputation (Jarrett et al., ) among others. Most of these studies concentrate on imputation accuracy without assessing performance on subsequent tasks. However, theoretical arguments suggest that good imputation is not needed for good prediction (Le Morvan et al., ; Josse et al., ). These arguments are asymptotic and whether they hold in typical cases is debatable. To address the discrepancy between this theory and the emphasis on imputation efforts, there is a critical need for empirical studies to determine whether better imputations actually lead to better predictions.

Theory does establish that in some scenarios, better imputations imply better predictions. For instance with a linearly-generated outcome, the optimal prediction writes as a linear model on the optimally-imputed data Le Morvan et al. (). Thus, using a linear model for prediction, better imputations typically yield better predictions.
Interestingly though, theoretical results indicate that for linear models, simple constant imputations suffice in very high dimensions (Ayme et al., ), or in small dimensions with uncorrelated features (Ayme et al., ). Beyond linear models, empirical studies on real data have shown the competitiveness of simple imputations –such as mean– (Paterakis et al., ; Perez-Lebel et al., ; Shadbahr et al., ) aligning with theoretical arguments. However, their findings may be driven by “predictive” missingness (Missing Not At Random data, for which most imputation methods are invalid Little and Rubin, ; Josse et al., ). In addition, these studies have not quantified the effect of imputation accuracy on prediction performance nor how these effects are modulated by the choice of downstream model, the use of a missingness indicator, the missingness mechanism or dataset-related features. Our work fills this gap and provides actionable conclusions.

We study settings where good imputations are most likely to benefit downstream predictive performance, in order to obtain an upper bound on the potential benefits of improved imputation in general cases. If only minor benefits are observed in this context, it would suggest even smaller benefits in general. Specifically, there are 3 types of missingness mechanisms (Rubin, ): (i) Missing Completely At Random (MCAR), where each entry is missing with a fixed probability; (ii) Missing At Random (MAR), where the probability of a value being missing depends only on observed variables, (iii) Missing Not At Random settings (MNAR), where the missingness is informative and encompasses all other scenarios. Most imputation algorithms are valid under MCAR but not under MNAR. Therefore, we focus on the MCAR settings where imputation algorithms are most likely to produce accurate imputations. Additionally, we focus on datasets with only numerical values. For categorical features, handling missing values by treating them as a new category is a popular and effective approach that does not involve a notion of imputation accuracy. In this setting, we characterize the relationship between imputation accuracy and predictive performance for several downstream prediction models, with and without the missingness indicator. Properly benchmarking methodologies with missing values is particularly resource-intensive, as already emphasized in previous works (Jäger et al., ; Perez-Lebel et al., ). Indeed, computing costs are driven by the combinatorics of imputation and prediction models, hyperparameter optimization for both, inclusion and exclusion of the indicator, and varying missing rates.

[Section 2](#S2) introduces related work, covering both benchmarks for prediction with missing values and available theory. [Section 3](#S3) details our experimental procedures, specifically the methods examined. [Section 4](#S4) presents our findings, relating gains in imputation to prediction performance. Finally, [section 5](#S5) summarizes the lessons learned.

## 2 Related work

#### Benchmarks.

Several benchmark studies have investigated imputation in a prediction context (Paterakis et al., ; Jäger et al., ; Ramosaj et al., ; Woźnica and Biecek, ; Perez-Lebel et al., ; Poulos and Valle, ; Shadbahr et al., ; Li et al., ). However, drawing definitive conclusions from most studies is challenging due to various limitations in scope and experimental choices. For example, Bertsimas et al. (); Li et al. () trained imputation methods using both the training and test sets, rather than applying the imputation learned on the training set to the test set, which is not possible with many imputation packages. This approach creates data leakage. Woźnica and Biecek () trained imputers separately on the train and test sets, which creates an “imputation shift” between the training and test data. Jäger et al. () discards and imputes values in a single column of the test set, chosen at random and fixed throughout the experiments. Yet as they note, conclusions can change drastically depending on the importance of the to-be-imputed column for the prediction task or its correlation with other features. Some studies (Poulos and Valle, ; Ramosaj et al., ) use a small number of datasets (resp. 2 and 5 datasets from the UCI machine learning repository respectively), thus limiting the significance of their conclusions. Woźnica and Biecek () do not perform hyperparameter tuning for the prediction models, while Ramosaj et al. () tunes hyperparameters on the complete data, though it is unclear whether the best hyperparameters on complete data are also the best on incomplete data. Furthermore, some benchmarks focus on specific types of downstream prediction models, such as linear models (Jäger et al., ), AutoML models (Paterakis et al., ) or Support Vector Machines (Li et al., ), meaning their conclusions should not be generalized to all types of downstream models. The data used in benchmarks also affects the scope of the conclusions. For instance, Paterakis et al. () considers 35 datasets, most with only a few hundred samples and a 50% train-test split, making their conclusions more relevant to small data scenarios. Finally, only Perez-Lebel et al. () and Paterakis et al. () evaluate the use of the missingness indicator as complementary input features.

Among the benchmarks with largest scope, Paterakis et al. () recommend mean/mode imputation with the indicator as the default option in AutoML settings, both for native and simulated missingness. It is among the top-performing approaches, never statistically significantly outperformed, and is also the most cost-effective. They also show that using the missingness indicator as input improves performances slightly but significantly for most imputation methods. Perez-Lebel et al. () focus on predictive modeling for large health databases, which contain many naturally occurring missing values. They compare various imputation strategies (mean, median, k-nearest neighbors imputation, MICE) combined with gradient-boosted trees (GBTs) for prediction, as well as GBTs with native handling of missing values. Similarly to Paterakis et al. (), they find that appending the indicator to the imputed data significantly improves performances, which may reflect MNAR data. While they recommend resorting to the native handling of missing values as it is relatively cheap, their results further indicate that no method is significantly better than using the mean as imputation method together with the indicator.
Shadbahr et al. () also find that the best imputations do not necessarily result in the best downstream performances. Using an analysis of variance, they show that the choice of imputation method has a significant but small effect on the classification performance.

Whether better imputation leads to better prediction may vary depending on factors like the choice of downstream model, the missingness rate, or the specific datasets. Yet, many studies seek a definitive conclusion across diverse settings. Only Paterakis et al. () conducted a meta-analysis, yet it did not determine when more advanced imputation strategies are beneficial compared to mean or mode imputations. Identifying scenarios in which better imputations are more likely to improve predictions is however of strong practical interest.

#### Theoretical insights.

Previous works have also addressed this question from a theoretical point of view. Le Morvan et al. () showed that for all missingness mechanisms and almost all deterministic imputation functions, universally consistent algorithms trained on imputed data asymptotically achieve optimal performances in prediction. This is in particular true for simple imputations such as the mean (Josse et al., ), thereby providing rationale to favor simple imputations over more accurate ones. Essentially, optimal prediction models can be built on mean-imputed data by modeling the mean as a special value encoding for missingness. Ayme et al. () also provide theoretical support for the use of simple imputations, as they advocate for the use of zero imputation in high-dimensional settings. They show that, for a linear regression problem and MCAR missingness, learning on zero-imputed data instead of complete data incurs an imputation bias that goes to zero when the dimension increases. This holds given certain assumptions on the covariance matrix, which intuitively impose some redundancy among variables. Finally, Van Ness et al. () prove that in MCAR settings, the best linear predictor assigns zero weights to the missingness indicator, whereas these weights are non-zero in MNAR settings. Their theoretical results imply that the missingness indicator neither degrades nor enhances performances asymptotically in MCAR.

## 3 Experimental setup.

#### Imputation methods.

We chose four imputation models to cover a wide range of imputation qualities. Importantly, our goal *is not* to extensively benchmark different imputation strategies but to understand the link between imputation quality and prediction performance.

- each missing value is imputed with the mean of the observed values in a given variable. It provides a useful baseline for assessing the effectiveness of advanced techniques.

- each feature is imputed based on the other features in a round-robin fashion using a Bayesian ridge regressor. This method is related to mice (Van Buuren and Groothuis-Oudshoorn, ) in that it is also based on a fully conditional specification (Van Buuren, ). It is implemented in scikit-learn’s IterativeImputer (Pedregosa et al., ).

(Stekhoven and Bühlmann, ) - operates in a manner analogous to iterativeBR, wherein it imputes one feature using all others and iteratively enhances the imputation by sequentially addressing each feature multiple times. The key distinction lies in its utilization of a random forest for imputation rather than a linear model. We used scikit-learn’s IterativeImputer with RandomForestRegressor as estimators. Default parameters for Missforest were set to n_estimators=30 and max_depth=15 for the random forests (the higher the better) to keep a reasonable computational budget. Note that in HyperImpute (Jarrett et al., ), random forests are more limited: 10 trees and a maximum depth of 4.

- uses the conditional expectation formula of a multivariate normal distribution to impute the missing entries given the observed ones. The mean and covariance matrix of the multivariate normal distribution are estimated with (pairwise) available-case estimates (Little and Rubin, , section 3.4), i.e., the $(i,j)^{th}$ entry of the covariance matrix is estimated solely from samples where both variables $i$ and $j$ are observed. This approach offers computational advantages over more resource-intensive approaches such as the Expectation-Maximization (EM) algorithm. It is related to Buck’s method (Buck, ; Little and Rubin, , section 4.2).

Mean imputation can be expected to give the worst imputation, with other methods offering varying improvements. In particular, missforest often delivers top-tier performance on tabular data.

#### Models:

As the effect of imputation on prediction quality can be modulated by the predictive model used, we included three predictive models. In particular, we took care to include both deep learning and tree-based representatives, as the prediction functions produced by these models have different properties, for example regarding their smoothness.

- •
MLP: a basic Multilayer Perceptron with ReLU activations, to serve as a simple baseline.
- •
SAINT (Somepalli et al., ): Self-Attention and Intersample Attention Transformer (SAINT) is a deep tabular model that performs both row and column attention. The numerical features are first embedded to a $d$-dimensional space before being fed to the transformer. We chose SAINT as it has been shown to be state-of-the-art among deep learning approaches for tabular data in several surveys (Borisov et al., ; Grinsztajn et al., ).
- •
XGBoost (Chen and Guestrin, ): We chose XGBoost as it is a popular state-of-the-art boosting method, and it has been shown to be the best tree-based model on regression tasks with numerical features only in Grinsztajn et al. ().

We tuned hyperparameters with Optuna (Akiba et al., ) for XGBoost and MLP. For SAINT, we used the default hyperparameters provided by its authors (Somepalli et al., ) for computational reasons. [Tables 3](#A2.T3), [4](#A2.T4) and [2](#A2.T2) give all details on hyperparameter tuning and default hyperparameters.

#### Native handling of missing values:

Both SAINT and XGBoost can directly be applied on incomplete data, without prior imputation of the missing values, each with its own strategy. XGBoost uses the Missing Incorporated in Attribute (MIA) (Twala et al., ; Josse et al., ) approach. When splitting, samples with missing values in the split feature can go left, right, or form their own leaf. MIA retains the option that minimizes the prediction error. In SAINT, numerical features are embedded in a $d$-dimensional space using simple MLPs. In case of missing value, a learnable $d$-dimensional embedding is used to represent the NaN. Each feature has its own missingness embedding.

#### The datasets

We use a benchmark created by Grinsztajn et al. () for tabular learning. It comprises 20 datasets (listed in [table 1](#A1.T1)), each corresponding to a regression task with continuous and ordinal features. Missing data is generated according to a MCAR mechanism with either 20% or 50% misssing rate. Continuous features are gaussianized using scikit-learn’s QuantileTransformer while ordinal features are standard scaled to have a zero mean and unit variance. This is true for all imputation and model combination except for XGBoost with native handling of missing values, as it is not expected to benefit from a normalization. The outputs $y$ are also standard scaled. In all cases, the parameters of these data normalizations are learned on the train set with missing values.
We also provide experiments on semi-synthetic data where the response $y$ is simulated as a linear function of the original data $X$. The coefficients $\beta$ of the linear function are all taken equal and scaled so that the variance of $\beta^{\top}X$ is equal to 1. Noise is added with a signal-to-noise ratio of 10.

#### Evaluation strategy

Each dataset is randomly split into 3 folds (train - 80%, validation - 10% and test - 10%), and each split is furthermore capped at 50,000 samples (table [1](#A1.T1)). Train, validation and test sets are imputed using the same imputation model trained on the train set. Prediction models are then trained on the imputed train set. For XGBoost and the MLP, hyperparameters are optimized using Optuna (Akiba et al., ) with 50 trials, i.e, Optuna draws 50 sets of hyperparameters, trains a model for each of these hyperparameter sets, and retains the best one according to the prediction performance on the validation set.
When the indicator is used, it is appended as extra features to the imputed data, it is not leveraged for the imputation stage. We run all combinations of the 3 prediction models with the 4 imputation techniques, with and without the indicator, resulting in $4\times 3\times 2=24$ models to which we add XGBoost and SAINT with native handling of missing values. This results in a total of 26 models displayed in Figure [1](#S4.F1). Finally, the whole process is repeated with 10 different train/validation/test splits. For reproducibility, the code will be available on GitHub upon publication of the preprint.

#### Computational resources.

Multiplying the number of models (26) with the number of datasets (20 + 20 linear versions), the hyperparameter tuning (50 trials), the number of repetitions of the experiments (10), and the 2 missing rates, we get a very large number of runs (around 700,000). As some methods are computationally expensive –such as missforest for imputing, as well as SAINT notably when the indicator is used–, these experiments required a total of 2.5 CPU years. A quarter of this time was dedicated to imputation.

## 4 Results: Determinants of predictions performance with MCAR missingness

Figure: Figure 1: Relative prediction performances across datasets for different imputations, predictors, and use of the missingness indicator. Each boxplot represents 200 points (20 datasets with 10 repetitions per dataset). The performances shown are $R^{2}\,$scores on the test set relative to the mean performance across all models for a given dataset and repetition. A value of 0.01 indicates that a given method outperforms the average performance on a given dataset by 0.01 on the $R^{2}\,$score. Corresponding critical difference plots in [figs. 6](#A4.F6) and [7](#A4.F7).
Refer to caption: /html/2407.19804/assets/x1.png

### 4.1 Benefits of sophisticated imputation, the indicator, and XGBoost amid high variance.

Figure [1](#S4.F1) summarizes the relative performance of the various predictors combined with the different imputation schemes across the 20 datasets. Some trends emerge: more sophisticated imputers tend to improve prediction, with missForest-based predictors often outperforming those using condexp or iterativeBR imputers, which in turn outperform predictors based on mean imputation. However, using the missingness indicator decreases this effect. Additionally, this effect is barely noticeable for the best predictor, XGBoost, which appears to maintain its advantages on tabular data (as described in Grinsztajn et al., ) even in the presence of missing values.

That the best predictor barely benefits from fancy imputers brings us back to our original question: should efforts go into imputation?
Drawing a conclusion from [fig. 1](#S4.F1) would be premature:
the variance across datasets is typically greater than the difference in performance between methods (critical difference diagram in [figs. 6](#A4.F6) and [7](#A4.F7)). For example, missforest + XGBoost + indicator outperforms all other methods on only a third of datasets. Additionally, XGBoost + indicator does not perform significantly better with missforest than with condexp, while mean imputation does not always lead to the worst prediction. In what follows, we focus on quantifying the effects of improved imputation accuracy on predictions in different scenarios.

Figure: Figure 2: Left: Imputer performance for recovery. Performances are given as $R^{2}\,$scores for each dataset relative to the mean performance across imputation techniques. A negative value indicates that a method perform worse than the average of other methods. Right: Imputation time.
Refer to caption: /html/2407.19804/assets/x2.png

### 4.2 A detour through imputation accuracies: how do imputers compare?

Although comparing imputers is not our main objective, it is enlightening for our prediction purpose to characterize their relative performance range.
Figure [2](#S4.F2) (left) gives imputation performances measured as the $R^{2}\,$score between the imputed and ground truth values, relative to the average across methods and missing rates for each dataset. At 20% missing rate, the best imputer is missforest, followed by condexp and iterativeBR, nearly tied, and far behind mean imputation. At 50% missing rate, the imputation accuracy of all but mean imputation drop, but interestingly condexp is much less affected. It is interesting that such a simple method performs best. It is notably two orders of magnitude faster than missforest (figure [2](#S4.F2) right), which makes it an imputation technique worth considering. It is possible that the gaussianization of the features helped condexp, although a feature-wise gaussianization does not produce a jointly Gaussian dataset.

This work does not aim to compare or identify the best imputers, but rather to achieve varying imputation qualities to highlight the link between imputation and prediction quality. In this regard, the high range of imputation accuracy between the best and worst methods (an average difference of 0.5 $R^{2}\,$points at 20% and 0.3 $R^{2}\,$points at 50%) allows capturing differences in prediction performance.

### 4.3 Linking imputation accuracy and prediction performances.

Figure: Figure 3: Example fit of prediction performance as a function of imputation accuracy, for the Bike_Sharing_Demand dataset and a missing rate of 50%: on the left using an MLP as predictor, and on the right an MLP with missingness indicator.
Refer to caption: /html/2407.19804/assets/x3.png

Figure: Figure 4: Effect of the imputation recovery on the prediction performance. We report the slope of the regression line where imputation quality is used to predict prediction performance.
Refer to caption: /html/2407.19804/assets/x4.png

Combining the four imputation techniques with 10 repetitions of each experiment yields 40 (imputation $R^{2}\,$, prediction $R^{2}\,$) pairs for each model and dataset. To quantify how improvements in imputation accuracy translate into downstream prediction performance, we fit a linear regression using these 40 points for each model and dataset(^1^11The repetition identifier is also used as a covariate in the linear regression to account for the effects of the various train/test splits on prediction performance.). [Figure 3](#S4.F3.fig1) gives two examples of such fit: on the Bike_Sharing_Demand dataset, for a missing rate of 50%, the prediction $R^{2}\,$increases as a function of the imputation $R^{2}\,$; the effect is greater for the MLP, for which the fit gives a slope of 0.24, than for the MLP with indicator for which the slope is only 0.03.
[Figure 4](#S4.F4) summarizes the slopes estimated using the aforementioned methodology across all datasets, predictors with and without the indicator, and varying missing rates. Firstly, the fact that most slopes are positive indicates that better imputations correlate with better predictions, aligning with common beliefs. However, this observation should be nuanced by the size of the effects.

#### Gains in prediction R 2 superscript 𝑅 2 R^{2}\, are 10% or less of the gains in imputation R 2 superscript 𝑅 2 R^{2}\, .

[Figure 4](#S4.F4) shows that the slopes are typically small, rarely exceeding $0.1$. This implies that an improvement of $0.1$ in imputation $R^{2}\,$typically leads to an improvement in prediction $R^{2}\,$that is 10 times smaller, i.e. a gain of $0.01$ in prediction $R^{2}\,$, or even less. For XGBoost, the average slope across datasets in rather close to 0.025 or less (even zero without the mask at 20% missing rate). Thus, an enhancement of 0.3 in imputation $R^{2}\,$, which represents the average difference between the best of the worst imputer in this scenario (mean vs condexp in [fig. 2](#S4.F2)), implies a gain in prediction $R^{2}\,$of only 0.0075.

#### Good imputations matter less for more expressive predictors.

Comparison between models shows a decrease in slope from MLP to SAINT, to XGBooost. These results illustrate the idea that a powerful model can compensate for the simplicity or inaccuracy of an imputation (in our case, the MLP can be considered the least expressive model, and XGBoost the most expressive). Le Morvan et al. () gives a formal proof in an extreme case: given enough samples, a sufficiently expressive model can always build a Bayes optimal predictor even on the simplest imputations (e.g. a constant).

#### Good imputations matter less when adding the indicator.

Figure [4](#S4.F4) shows that adding the missingness indicator clearly decreases the effect size: imputing better has less impact on performance when the indicator is used (we discuss this effect further in [section 4.4](#S4.SS4)).

Figure: Figure 5: Correlation between imputation quality and prediction performance. A correlation close to 1 indicates that the quality of imputations is stronly associated to the quality of predictions, while a correlation close to zero means that the quality of predictions is not linked to the quality of imputations. Each correlation is computed using 40 different imputation/performance pairs, made of 4 imputation methods (mean, iterativeBR, missforest, condexp) repeated 10 times.
Refer to caption: /html/2407.19804/assets/x5.png

#### Good imputations matter less when the response is non-linear.

When the response $y$ is a linear function of the input $X$, the best predictor can be built using a linear model on the most accurate simple imputation. However, when the responses are non-linear, it may be difficult to learn the best possible predictor even with the most accurate imputation (Le Morvan et al., ). There are thus reasons to believe that response non-linearity, which is common in real data, alters the relationship between imputation accuracy and prediction performance. To investigate this, we compare the real datasets with matching semi-simulated datasets where $y$ is simulated as a linear function of the input $X$. We also measure correlation(^2^22Specifically we use the partial correlation, partialing out the effect of repeated train/test splits.) ([fig. 5](#S4.F5)) in addition to the slope, to quantify the reliability of the association: correlation captures not only the effect size (slope) but also the amount of noise ([appendix C](#A3) recalls this classic result) in the relationship.
While the effects are similar between real and linear outcomes ([fig. 11](#A6.F11) gives effects in the semi-simulated case), the correlation between imputation accuracy and prediction performance, averaged across all datasets, is systematically smaller for real outcomes than for linear ones ([fig. 5](#S4.F5)). The average decrease in correlation lies between 0.1 and 0.3 across models. Moreover, the variance in correlations for real outcomes is much larger, with many datasets with a near-zero correlation. This shows that the gains expected in prediction from better imputation are much more reliably achieved when the response is linear.

### 4.4 Why is the indicator beneficial, even with MCAR data?

In general, we find that adding the missingness indicator really helps prediction. While it is expected that adding the indicator is beneficial in MNAR scenarios, as the missingness is informative, it is less obvious in the MCAR settings studied here. Indeed, the indicator contains absolutely no relevant information for predicting the outcome. To the best of our knowledge, the benefit of using an indicator in MCAR has not yet been established. Below, we propose a theoretical insight to explain this finding.

The best possible predictor in the presence of missingness can always be expressed as the composition of an imputation and a prediction function (Le Morvan et al., ). But, in general, the best prediction function on the imputed data can be challenging to learn, even for perfect conditional imputation. In fact, it often displays discontinuities on imputed points.
We hypothesize that adding the missingness indicator simplifies modeling functions that exhibit discontinuities at these points, as the indicator can act as a switch to encode these discontinuities.

The case of XGBoost in Figure [1](#S4.F1) illustrates the importance of keeping the missingness information encoded. For 50% missing rate, in the absence of an indicator, no imputation really benefits prediction with XGBoost, and the best option is to use the native handling of missing values. This suggests that XGBoost benefits from knowing which values are missing. With advanced imputations, distinguishing between imputed and observed values becomes challenging. Appending the indicator to the imputed data reinstates the missingness information unambiguously, which enables XGBoost to benefit from more advanced imputations, in particular missforest.

## 5 Conclusion

#### For prediction, imputation matters but marginally.

Prior theoretical work showed that in extreme cases (asymptotics), imputation does not matter for predicting with missing values.
We quantified empirically the effect of imputation accuracy gains on prediction performance across many datasets and scenarios. We show that in practice, imputation does play a role.
But various factors modulate the importance of better imputations for prediction: investing in better imputations will be less beneficial when a flexible model is used, when a missing-value indicator is used, and if the response is thought to be non-linear. These results are actually in line with the theoretical results suggesting that imputation does not matter, as these hold for very flexible models (ie universally consistent). A notable new insight is that adding a missing-value indicator as input is beneficial for prediction performances *even for MCAR settings*, where missingness is uninformative.

We show that large gains in imputation accuracy translate into small gains in prediction performance. These results were drawn from a favorable MCAR setting, and it is likely that with native missingness, often Missing Non At Random (MNAR), the performance gains are even smaller. As novel imputation methods usually provide small gains in imputation accuracy compared to the state-of-the-art, the corresponding gains in downstream prediction tasks are likely to be even smaller.

There are multiple potential reasons why imputation gains do not always correlate with performance gains. For instance, some features may be well recovered, but not useful in the prediction because they are not predictive. Or even with accurate imputations, it may still be difficult to learn a predictor that performs well for all missing data patterns. Finally, the imputation accuracy is also probably an imperfect measure of the potential gains in prediction: in our experiments on 50% missing rate, missforest and iterativeBR performs comparable on average yet missforest-based predictors tend to outperform those based on iterativeBR.

#### Limitations and future work.

A remaining question is whether better imputations matter more for higher-dimensional datasets. In our study, the 3 datasets with higher dimension (superconduct, isolet and year) show high correlations of imputation with prediction gains, and larger effects, notably for isolet. It would also be useful to investigate whether imputations based on random draws outperform deterministic imputations for downstream prediction tasks, and the usefulness of multiple imputations (Perez-Lebel et al., ). A related question is whether reconstructing well the data distribution is important for better predictions. Shadbahr et al. () shows that it does not seem crucial for classification performances but may compromise more seriously model interpretability. Finally, adding the indicator to the input enables better predictors to be learned, yet it may not be the best way to represent the missingness information, and further investigation is needed in this direction.

#### Outlook

We have seen that often, improving imputation is a difficult way of improving prediction. On top of imputation, future research could focus more on developing advanced modeling techniques that can inherently handle missing values and effectively incorporate missingness indicators to improve predictive performance.

## Appendix A List of datasets.

This benchmark was created by Grinsztajn et al. [], and is available on OpenML at [https://www.openml.org/search?type=benchmark&study_type=task&sort=tasks_included&id=297](https://www.openml.org/search?type=benchmark&study_type=task&sort=tasks_included&id=297).

**Table 1: Datasets dimensions.**
| dataset | d | n_train | n_test |
| --- | --- | --- | --- |
| house_16H | 16 | 18185 | 2274 |
| cpu_act | 21 | 6553 | 820 |
| elevators | 16 | 13279 | 1661 |
| wine_quality | 11 | 5197 | 651 |
| Brazilian_houses | 8 | 8553 | 1070 |
| house_sales | 15 | 17290 | 2162 |
| sulfur | 6 | 8064 | 1009 |
| Ailerons | 33 | 11000 | 1375 |
| Bike_Sharing_Demand | 6 | 13903 | 1739 |
| california | 8 | 16512 | 2064 |
| diamonds | 6 | 43152 | 5394 |
| fifa | 5 | 14450 | 1807 |
| houses | 8 | 16512 | 2064 |
| isolet | 613 | 6237 | 781 |
| medical_charges | 3 | 50000 | 50000 |
| MiamiHousing2016 | 13 | 11145 | 1394 |
| nyc-taxi-green-dec-2016 | 9 | 50000 | 50000 |
| pol | 26 | 12000 | 1500 |
| superconduct | 79 | 17010 | 2127 |
| year | 90 | 50000 | 50000 |

## Appendix B Hyperparameter search spaces

**Table 2: XGBoost hyperparameter space. We used the XGBRegressor from the xgboost Python library. The hyperparameters optimized are commonly accepted as the most important ones. The variation ranges are inspired by the ones used in Grinsztajn et al. [], while the default hyperparameters are those of the xgboost library.**
| parameter | range | log scale | default |
| --- | --- | --- | --- |
| n_estimators | $\left[100,2000\right]$ | no | 100 |
| max_depth | $\left[1,6\right]$ | no | 6 |
| learning_rate | $\left[10^{-5},0.7\right]$ | yes | 0.3 |
| reg_alpha | $\left[10^{-8},10^{2}\right]$ | yes | $10^{-8}$ |
| reg_lambda | $\left[1,4\right]$ | yes | 1 |
| early_stopping_rounds | - | - | 20 |

**Table 3: MLP hyperparameter space. We implemented the MLP in PyTorch. The parameter $d$ for the width of the MLP represents the number of features. When $d>1024$, the width is taken equal to the number of features $d$.**
|  | parameter | range | default |
| --- | --- | --- | --- |
| MLP | depth | $\left\llbracket 0,6\right\rrbracket$ | 3 |
|  | width | $\left[d,\min\left(10d,1024\right)\right]$ | 3d |
|  | dropout rate | $\left[0,0.5\right]$ | 0.2 |
| Optimizer | name | - | AdamW |
|  | weight decay | - | $10^{-6}$ |
|  | learning rate | - | $10^{-3}$ |
| Scheduler | name | - | ReduceLROnPlateau |
|  | factor | - | 0.2 |
|  | patience | - | 10 |
|  | threshold | - | $10^{-4}$ |
| General | max nb. epochs | - | 2000 |
|  | early stoppping | - | Yes |
|  | batch size | - | 256 |

**Table 4: SAINT default hyperparameters. We used the implementation provided by Somepalli et al. []. $d$ refers to the number of features of the dataset. We did not use a scheduler with SAINT. We followed the default configuration provided by the paper introducing SAINT [Somepalli et al., ] when there is both intersample and feature attention (i.e. attention_type = ’colrow’).**
|  | parameter | default |
| --- | --- | --- |
| SAINT | dim | 32 if $d<70$, 16 if $d\in\left[70,200\right]$, 4 if $d\geq 200$ |
|  | depth | 1 |
|  | heads | 4 |
|  | attn_dropout | 0.8 |
|  | ff_dropout | 0.8 |
|  | attentiontype | colrow |
| Optimizer | name | AdamW |
|  | weight decay | $10^{-2}$ |
|  | learning rate | $10^{-4}$ |
| General | max nb. epochs | 100 |
|  | early stopping | yes |
|  | batch size | 256 |

## Appendix C Link between correlation and effect size.

For completeness, we recall below the relationship between correlation and effect size.

###### Proposition C.1 (Link between correlation and effect size.) .

Let $X\in\mathbb{R}$ be a random variable, and $\beta\in\mathbb{R}$ a parameter. Furthermore, define:

$$ $Y=\beta X+\epsilon\quad\text{where}\quad\mathbb{E}\!\left[\epsilon|X\right]=0,\;\text{var}\left(\epsilon\right)=\sigma^{2}.$ $$

Then:

$$ $\text{cor}\left(X,Y\right)=\frac{1}{\sqrt{1+\frac{\sigma^{2}}{\beta^{2}\text{var}\left(X\right)}}}$ $$

###### Proof.

Let’s first derive the expression of the variance of $Y$:

$$ $\displaystyle\text{Var}\left(Y\right)$ $\displaystyle=\mathbb{E}\!\left[\left(Y-\mathbb{E}\!\left[Y\right]\right)^{2}\right]$ $\displaystyle=\mathbb{E}\!\left[\left(\beta X+\epsilon-\beta\mathbb{E}\!\left[X\right]\right)^{2}\right]$ $\displaystyle=\mathbb{E}\!\left[\left(\beta\left(X-\mathbb{E}\!\left[X\right]\right)\right)^{2}+\epsilon^{2}\right]$ $\displaystyle=\beta^{2}\text{var}\!\left(X\right)+\sigma^{2}$ $$

It follows that:

$$ $\displaystyle\text{cor}\left(Y,X\right)$ $\displaystyle=\frac{\mathbb{E}\!\left[\left(Y-\mathbb{E}\!\left[Y\right]\right)\left(X-\mathbb{E}\!\left[X\right]\right)\right]}{\sqrt{\text{var}\!\left(X\right)\text{var}\!\left(Y\right)}}$ $\displaystyle=\frac{\mathbb{E}\!\left[\beta\left(X-\mathbb{E}\!\left[X\right]\right)^{2}\right]}{\sqrt{\text{var}\!\left(X\right)\text{var}\!\left(Y\right)}}$ $\displaystyle=\beta\frac{\sqrt{\text{var}\!\left(X\right)}}{\sqrt{\text{var}\!\left(Y\right)}}$ $\displaystyle=\beta\frac{\sqrt{\text{var}\!\left(X\right)}}{\sqrt{\beta^{2}\text{var}\!\left(X\right)+\sigma^{2}}}$ $\displaystyle=\frac{1}{\sqrt{1+\frac{\sigma^{2}}{\beta^{2}\text{var}\!\left(X\right)}}}$ $$

∎

Hence, in a case where the imputation accuracy $X$ covers a wider range of values, i.e., $\text{var}\!\left(X\right)$ is larger, but the effect $\beta$ and the noise $\sigma^{2}$ stay the same, then the correlation increases.

###### Proposition C.1 (Link between correlation and effect size.) .

###### Proof.

## Appendix D Critical Difference diagrams.

[Figures 6](#A4.F6), [7](#A4.F7), [8](#A4.F8) and [9](#A4.F9) give the Critical Difference diagrams across all predictors and imputers of average score ranks for a significance level of 0.05. The difference in ranks for all methods covered by the same black crossbar are not statistically significant according to a Nemenyi test for multiple pairwise comparisons. The colors encode the imputation type, the markers identify the model, and the line types encode the presence or absence of an indicator.

Figure: Figure 6: Critical Difference diagram - 20% missingness rate.
Refer to caption: /html/2407.19804/assets/CD_plot_M_20.png

Figure: Figure 7: Critical Difference diagram - 50% missingness rate.
Refer to caption: /html/2407.19804/assets/CD_plot_M_50.png

Figure: Figure 8: Critical Difference diagram - 20% missingness rate, semi-synthetic data with linear outcomes.
Refer to caption: /html/2407.19804/assets/CD_plot_yM_20.png

Figure: Figure 9: Critical Difference diagram - 50% missingness rate, semi-synthetic data with linear outcomes.
Refer to caption: /html/2407.19804/assets/CD_plot_yM_50.png

## Appendix E Prediction performances for the semi-synthetic data.

Figure: Figure 10: Relative prediction performances for the semi-synthetic data with linear outcomes across datasets for different imputations, predictors, and use of the missingness indicator. Each boxplot represents 200 points (20 datasets with 10 repetitions per dataset). The performances shown are $R^{2}\,$scores on the test set relative to the mean performance across all models for a given dataset and repetition. A value of 0.01 indicates that a given method outperforms the average performance on a given dataset by 0.01 on the $R^{2}\,$score.
Refer to caption: /html/2407.19804/assets/x6.png

## Appendix F Regression slopes for the semi-synthetic data.

Figure: Figure 11: Effect of the imputation recovery on the prediction performance for the semi-synthetic data with linear outcomes. We report the slope of the regression line where imputation quality is used to predict prediction performance.
Refer to caption: /html/2407.19804/assets/x7.png

## Appendix G Computation times per method.

Figure: Figure 12: Running time for each model, including the 50 iterations of hyperparameter search for XGBoost and MLP.
Refer to caption: /html/2407.19804/assets/x8.png

Figure: Figure 13: Running time for each model for the semi-synthetic data with linear outcomes, including the 50 iterations of hyperparameter search for XGBoost and MLP.
Refer to caption: /html/2407.19804/assets/x9.png

## Appendix H Scatterplots of prediction R 2 superscript 𝑅 2 R^{2} vs imputation R 2 superscript 𝑅 2 R^{2} for each model and dataset.

Figure: Figure 14: Prediction $R^{2}$ vs imputation $R^{2}$ for a MLP - missing rate 50%. The $R^{2}\,$scores are given relative to the mean $R^{2}\,$score, with the effects of experiment repetitions eliminated (i.e. the effect of the train/test splits on the performance)
Refer to caption: /html/2407.19804/assets/partial_correlation_per_dataset_M_MLP_rate50.png

Figure: Figure 15: Prediction $R^{2}$ vs imputation $R^{2}$ for a MLP + indicator - missing rate 50%. The $R^{2}\,$scores are given relative to the mean $R^{2}\,$score, with the effects of experiment repetitions eliminated (i.e. the effect of the train/test splits on the performance)
Refer to caption: /html/2407.19804/assets/partial_correlation_per_dataset_M_MLP+mask_rate50.png

Figure: Figure 16: Prediction $R^{2}$ vs imputation $R^{2}$ for SAINT - missing rate 50%. The $R^{2}\,$scores are given relative to the mean $R^{2}\,$score, with the effects of experiment repetitions eliminated (i.e. the effect of the train/test splits on the performance)
Refer to caption: /html/2407.19804/assets/partial_correlation_per_dataset_M_SAINT_rate50.png

Figure: Figure 17: Prediction $R^{2}$ vs imputation $R^{2}$ for SAINT + indicator - missing rate 50%. The $R^{2}\,$scores are given relative to the mean $R^{2}\,$score, with the effects of experiment repetitions eliminated (i.e. the effect of the train/test splits on the performance)
Refer to caption: /html/2407.19804/assets/partial_correlation_per_dataset_M_SAINT+mask_rate50.png

Figure: Figure 18: Prediction $R^{2}$ vs imputation $R^{2}$ for XGBoost - missing rate 50%. The $R^{2}\,$scores are given relative to the mean $R^{2}\,$score, with the effects of experiment repetitions eliminated (i.e. the effect of the train/test splits on the performance)
Refer to caption: /html/2407.19804/assets/partial_correlation_per_dataset_M_XGBoost_rate50.png

Figure: Figure 19: Prediction $R^{2}$ vs imputation $R^{2}$ for XGBoost + indicator - missing rate 50%. The $R^{2}\,$scores are given relative to the mean $R^{2}\,$score, with the effects of experiment repetitions eliminated (i.e. the effect of the train/test splits on the performance)
Refer to caption: /html/2407.19804/assets/partial_correlation_per_dataset_M_XGBoost+mask_rate50.png

Figure: Figure 20: Prediction $R^{2}$ vs imputation $R^{2}$ for a MLP - semi-synthetic data with linear outcomes, missing rate 50%. The $R^{2}\,$scores are given relative to the mean $R^{2}\,$score, with the effects of experiment repetitions eliminated (i.e. the effect of the train/test splits on the performance)
Refer to caption: /html/2407.19804/assets/partial_correlation_per_dataset_yM_MLP_rate50.png

Figure: Figure 21: Prediction $R^{2}$ vs imputation $R^{2}$ for a MLP + indicator - semi-synthetic data with linear outcomes, missing rate 50%. The $R^{2}\,$scores are given relative to the mean $R^{2}\,$score, with the effects of experiment repetitions eliminated (i.e. the effect of the train/test splits on the performance)
Refer to caption: /html/2407.19804/assets/partial_correlation_per_dataset_yM_MLP+mask_rate50.png

Figure: Figure 22: Prediction $R^{2}$ vs imputation $R^{2}$ for SAINT - semi-synthetic data with linear outcomes, missing rate 50%. The $R^{2}\,$scores are given relative to the mean $R^{2}\,$score, with the effects of experiment repetitions eliminated (i.e. the effect of the train/test splits on the performance)
Refer to caption: /html/2407.19804/assets/partial_correlation_per_dataset_yM_SAINT_rate50.png

Figure: Figure 23: Prediction $R^{2}$ vs imputation $R^{2}$ for SAINT + indicator - semi-synthetic data with linear outcomes, missing rate 50%. The $R^{2}\,$scores are given relative to the mean $R^{2}\,$score, with the effects of experiment repetitions eliminated (i.e. the effect of the train/test splits on the performance)
Refer to caption: /html/2407.19804/assets/partial_correlation_per_dataset_yM_SAINT+mask_rate50.png

Figure: Figure 24: Prediction $R^{2}$ vs imputation $R^{2}$ for XGBoost - semi-synthetic data with linear outcomes, missing rate 50%. The $R^{2}\,$scores are given relative to the mean $R^{2}\,$score, with the effects of experiment repetitions eliminated (i.e. the effect of the train/test splits on the performance)
Refer to caption: /html/2407.19804/assets/partial_correlation_per_dataset_yM_XGBoost_rate50.png

Figure: Figure 25: Prediction $R^{2}$ vs imputation $R^{2}$ for XGBoost + indicator - semi-synthetic data with linear outcomes, missing rate 50%. The $R^{2}\,$scores are given relative to the mean $R^{2}\,$score, with the effects of experiment repetitions eliminated (i.e. the effect of the train/test splits on the performance)
Refer to caption: /html/2407.19804/assets/partial_correlation_per_dataset_yM_XGBoost+mask_rate50.png

## Peer Reviews

confidence: 3
contribution: 2
initial_rating: 3
presentation: 3
questions: The observation about good imputations having less effect when using missingness indicators was very interesting. To test the intuition mentioned in Section 4.5, I think it would be interesting to use explainability techniques on these models trained with missingness indicators to see if the importance/weights of features drop when they are missing (i.e. inputs with the corresponding missingness indicator on).

Does the correlation of prediction performance and imputation accuracy (e.g, Figures 3 and 4) also show any difference by imputation method?

Would larger missingness rates show more significant correlations?
soundness: 2
strengths: The paper presents an empirical study that can help us better understand the effect of various imputation techniques for different model/problem scenarios, which is an important question as missing values are ubiquitous and often handled by imputation. The paper specifically considers the link between imputation accuracy and prediction performance, which have been studied theoretically in some scenarios but not empirically for real-world problems.

The experimental setup is clearly described in detail for reproducibility, and the authors mention that code will also be available upon publication. The experiments are also thorough in terms of datasets, models, and imputation methods.

Related work discussion is clear, including both theoretical and empirical studies about missing value imputation methods.
summary: This paper empirically studies the effects of various missing value imputation methods, in particular whether better imputation accuracy yields higher prediction performance. The authors conduct experiments on nineteen benchmark datasets using four imputation methods, three different models, as well as variations such as missingness indicators and semi-synthetic linear labels. They conclude that imputation accuracy does affect prediction performance, although the gains are quite small, and the effect is further reduced if the model is more expressive, missing value indicators are used, or the target variable has a non-linear relationship to the covariates.
weaknesses: Many claims/conclusions in the paper are based on small differences in average values with overlapping confidence intervals, and there are no statistical tests for significance. For example, the authors claim that the imputation techniques considered in the experiments show diverse imputation performance (Figure 2), but apart from mean imputation being worse than others, there doesn’t seem to be a definitive difference in quality by the other three imputers. 

In addition, the claim about correlation of imputation accuracy and prediction performance is made based on the small but positive slope of the regression line (Figure 4). However, the slope seems very small in most cases to suggest this, especially in the 20% missing case, and the correlation may still be very weak (small correlation coefficient). On the other hand, the authors suggest that good imputations matter more in the linear response case (semi-simulated data), based on the slightly higher correlation (Fig 5), but the slope seems still very small (Fig 11).

In about a third of the benchmark datasets, the dimension is small such that 20% missing means just one missing feature. I would expect the difference between simple and more complex imputation techniques to be pretty small in such cases. 

The experiment in MNAR scenario (Section 4.4) didn’t feel very connected to the rest of the paper. As the authors also noted, it is well known that most imputation methods (without considering causal structures) are not valid under MNAR.

-------------
Post-rebuttal: Thanks to the authors for the detailed response and revision. I think the additional experiments and statistical tests made the results of this paper much more convincing, and I am happy to raise my score to 6.

---

confidence: 4
contribution: 3
initial_rating: 8
presentation: 1
questions: Please eliminate redundant sentences, remove unnecessary adverbs ("really", "interestingly", etc), remove unnecessary metadiscourse (e.g., let me tell you what I'm going to tell you- the end of section 1 ) and acronyms that were defined not on their first occurrence (or never at all). Missingness indicators, which are a prominent concept in the paper, should be defined in one sentence in the introduction. Also, the quality and clarity of the writing varies greatly from paragraph to paragraph - can you make it more consistent? 

Line 140: what do you mean by a "universally consistent" algorithm?

Why is the 'semi-synthetic' data (top of page 5) 'semi'? Can you elaborate on that?

Line 341 uses the word "nuanced" in a strange way - so strange I couldn't figure out what the sentence in which it appears actually means.

You say 'comparing imputers is not our main objective' (twice), but it is an important set of results, right? Clearly, it is a salient result that a mean imputer does worse than other methods for both 20% and 50% missingness rates, and you could envision people citing this paper for that result. Perhaps these results should not be downplayed? 

Please re-read and consider linearizing the order/structure of your sentences. For example, line 456 reads "For prediction, imputation matters but marginally." You will minimize the amount of cognitive bandwidth / patience of your readers if you linearize the sentence structure, e.g.: 'Imputation matters marginally for prediction." If that is too much of a change of meaning, then consider 'Imputation matters for prediction, but only marginally.' These kinds of reverse-order sentences occur throughout the manuscript and, in aggregate, bog down the reader.
soundness: 3
strengths: The value of this paper arises from its significance, not its originality. What I mean by this is that the paper effectively reports a "null" result: something that *doesn't* work. This kind of work is valuable because it saves the community time by not re-inventing square wheels. We need to know what doesn't work. 

If you count papers that provide evidence that a method doesn't work (I do) as original, then yes this paper is original. This definition of originality fits under the notion of originality as "a new definition or problem formulation," because the authors are recommending that the data imputation research solves the wrong problem if the goal is improved prediction.

The quality of the work was adequate, they performed a comprehensive study using several different models and many different datasets to arrive at a convincing analysis of model performance.
summary: The authors use an empirical approach to investigate whether advanced imputation techniques significantly improve predictive accuracy in models with missing data. The study finds that while sophisticated imputation can enhance prediction in certain contexts, the benefits are modest, especially when more expressive models and/or missingness indicators are used. They conclude with the assertion that resources might be better allocated to models that inherently handle missing data rather than focusing extensively on imputation improvements.
weaknesses: I did not mention clarity in the Strengths section because the paper needs a lot of stylistic work. The authors would benefit from reading Strunk & White's "Elements of Style" as well as Steven Pinker's "A Sense of Style." There were many redundant sentences, unnecessary adverbs ("really", "interestingly", etc), unnecessary metadiscourse (e.g., let me tell you what I'm going to tell you) and acronyms that were defined not on their first occurrence (or never at all). Missingness indicators, which are a prominent concept in the paper, should be defined in one sentence in the introduction. In addition, different paragraphs seemed to have different authors, where at least one author appears to not have an adequate grasp of English. I'm not judging that (I don't speak any other language, hats of to them if that is the case) but there are resources to help non-native English speakers (even ChatGPT now does a reasonably decent job: write a paragraph, input it along with "improve:" and then see how the output paragraph is written- I find this helpful for succinctness but it can also help with sentence structure/word choices). All of these weaknesses combined to make reading the paper feel arduous.

---

confidence: 4
contribution: 3
initial_rating: 8
presentation: 4
questions: None
soundness: 3
strengths: The paper is well written and easy to read, presenting novel results even under the simple MCAR setting.
summary: This paper explores the link between imputation quality and downstream performance. Through multiple experiments, the paper demonstrates that under MCAR patterns, the quality of reconstruction error is not always linked with performance, depending upon modelling strategies.
weaknesses: The critical insight that adding missing indicators can be beneficial, even under MCAR, needs more justification (with a potential theoretical justification). Intuitively, I suggest exploring the correlation between reconstruction error and gain from adding the indicator. I believe the correlation should be strong as the model is able to 'discard' badly imputed data. Additionally, an experiment with a random mask appended would demonstrate that this result is not just a product of a larger number of parameters in the model or some regularisation.

Other works have explored this relation. I would recommend comparing the results with work such as Bertsimas 2024, where the authors conclude, "While common sense suggests that a 'good' imputation method produces datasets that are plausible, we show, on the contrary, that, as far as prediction is concerned, crude can be good."

Bertsimas D, Delarue A, Pauphilet J. Simple Imputation Rules for Prediction with Missing Data: Theoretical Guarantees vs. Empirical Performance. Transactions on Machine Learning Research. 2024 Jun 5.

---

confidence: 5
contribution: 2
initial_rating: 3
presentation: 3
questions: Given that both key findings are consistent with established theory and have been described before empirically, what are the novel contributions that would justify acceptance of the paper?
soundness: 3
strengths: The study performs an extensive experiment across a variety of imputation methods, prediction models, and datasets.
summary: The paper investigates the empirical importance of accurate imputation in tabular, continuous data with MCAR missingness. It compares four common imputation techniques and three model architectures (2x NN, 1x tree-based) across 19 real-world, medium-sized (<50,000 samples), fully-observed datasets augmented with simulated missingness.

The authors conclude that good imputation yields marginal improvements in terms of prediction performance and that missingness indicators may be beneficial even in MCAR scenarios.
weaknesses: The authors conclude that both better imputations and missingness indicators improve prediction performance. Neither of these findings is particularly surprising if we consider the optimal prediction with MCAR data: 

$\mathbb{E}[Y \mid X_o, M] = \int_{X_m} f^\star(X_m, X_o)p(X_m \mid X_o)dX_m,$

where, in line with Le Morvan et al. (2021), $Y$ is the outcome, $X_o$ and $X_m$ are the observed respectively missing covariates, and $f^\star$ is the underlying full-data function. For the type of conditional mean imputation used in this work, we do not target $p(X_m \mid X_o)$ but instead estimate $\mathbb{E}[X_m \mid X_o]$. In the extreme, we have perfect knowledge of this expectation through an oracle, and Le Morvan et al. (2021) in their Figure 4 already demonstrated that the performance of such an oracle > imputation via chained equations > mean impute. In the same figure, they also show that missing indicators are beneficial even in MCAR settings. The authors must be aware of this work, since they cite it as their main reference in their section "4.5 WHY IS THE INDICATOR BENEFICIAL, EVEN WITH MCAR DATA?", where they exclusively lean on Le Morvan et al. (2021) to explain why missingness indicators, even if they contain no information about the outcome, may still aid learning an often discontinuous optimal predictor.

References: 
Marine Le Morvan, Julie Josse, Erwan Scornet, and Gael Varoquaux. What’s a good imputation to predict with missing values?  Advances in Neural Information Processing Systems, volume 34, pp. 11530–11540. 2021.

---

confidence: 4
contribution: 3
initial_rating: 8
presentation: 3
questions: - Pg. 2., line 70: The MNAR definition could be clarified by stating that missingness is related to the unobserved values themselves, making it informative. In this case, the probability of missing data is related to the actual values that are missing. Also, the MAR and MNAR abbreviations should be spelled out in the first instance. 

- Pg. 2, line 103: the sentence could be clarified, e.g., Woźnica & Biecek (2020) trained imputers separately for the training and test datasets, which led to an 'imputation shift'—a situation where the imputation patterns between the two datasets differ, causing inconsistencies in the data used for model training versus model evaluation.

- Pg. 5, it may be informative to state in the “Computational resources” section or in the Appendix the computational hardware (e.g,. number of CPUs) used to conduct the experiments. 

**Typos/grammar**:
- Pg. 1, line 52: missing parentheses
- Pg. 2, line 85: “mportant”
- Pg. 5, line 223: It’s been previously stated that for XGBoost and the MLP, hyperparameters are optimized using Optuna (Akiba et al., 2019)
- Pg. 7, line 348: proper casing for figures (“fig. 2”)
- Pg. 8, line 415: citation for Pereira 2024 not included in the reference list
soundness: 4
strengths: The paper effectively bridges theoretical work on the minimal impact of imputation in asymptotic settings with empirical studies of the "impute-then-predict" pipeline in supervised learning, offering a rigorous evaluation of when imputation accuracy affects prediction. The authors position their work within the existing literature by addressing both empirical evaluations of imputation methods and theoretical frameworks. The evaluation spans 19 datasets, multiple imputation methods, and prediction models across MCAR and MNAR settings, with detailed analysis of relative prediction performance and imputation time. The use of critical difference plots establishes upper bounds on the benefits of imputation, particularly in best-case MCAR scenarios.
summary: This study quantifies the relationship between imputation accuracy and prediction performance across various datasets, showing that improvements in prediction are typically only 10% or less of the gains in imputation \(R^2\). It finds that advanced imputation methods are less beneficial with more flexible models (e.g., XGBoost) and when using missingness indicators, which improve predictions even in MCAR scenarios. The authors highlight that in MNAR settings, the impact of better imputation is likely even smaller.
weaknesses: - When drawing conclusions from Figs. 1 and 2 (last paragraph of Sec. 4.1), the authors could address the conditions under which advanced imputers do provide a benefit. For example, while XGBoost might not benefit much from MissForest, other less flexible models like MLP show greater improvements when paired with more advanced imputations. 

    The authors might also explore non-linear or hierarchical relationships within the datasets where advanced imputations could outperform simpler methods more consistently. In cases where feature interactions are complex, MissForest could have greater effects.

- Regarding “Good imputations matter less when the response is non-linear” (pgs. 7-8), the authors could strengthen the argument about non-linearities disrupting the relationship between imputation quality and prediction performance by incorporating insights from Le Morvan et al. (2021), which shows that *even with high-quality imputation*, non-linear functions can introduce discontinuities, making them harder to learn optimally. They could further illustrate how the *non-continuous nature of the regression function* after conditional imputation (as discussed in the paper) increases the complexity of the learning process in non-linear settings, which explains why non-linearities amplify the noise in the imputation-prediction relationship.

- On the question of whether the missingness indicator is the optimal way to represent missingness (pg. 9),  the authors could discuss specific limitations of the indicator in more detail, such as potential over-reliance on the indicator or the risk of introducing additional noise into the model. The authors cite the polar encoding paper of Lenz et. al. 2024, but could  suggest additional methods to represent missingness more effectively, such as embedding missingness directly within the model architecture or using neural architectures like NeuMiss that incorporate missing patterns in a more structured way (Le Morvan et al., 2021).

Meta-review: The paper proposes an empirical study that analyses the effectiveness of advanced imputation methods for missing values.
All reviewers are happy about the findings of the paper and agree to accept the paper.

Decision: Accept (Spotlight)