Simple Imputation Rules for Prediction with Missing Data: Theoretical Guarantees vs. Empirical Performance
Authors: Dimitris Bertsimas
Arthur Delarue
Jean Pauphilet
Venue: Accepted by TMLR ()

Abstract: Missing data is a common issue in real-world datasets. This paper studies the performance of impute-then-regress pipelines by contrasting theoretical and empirical evidence. We establish the asymptotic consistency of such pipelines for a broad family of imputation methods. While common sense suggests that a 'good' imputation method produces datasets that are plausible, we show, on the contrary, that, as far as prediction is concerned, crude can be good. Among others, we find that mode-impute is asymptotically sub-optimal, while mean-impute is asymptotically optimal. We then exhaustively assess the validity of these theoretical conclusions on a large corpus of synthetic, semi-real, and real datasets. While the empirical evidence we collect mostly supports our theoretical findings, it also highlights gaps between theory and practice and opportunities for future research, regarding the relevance of the MAR assumption, the complex interdependency between the imputation and regression tasks, and the need for realistic synthetic data generation models.

Published in Transactions on Machine Learning Research (06/2024)
Simple Imputation Rules for Prediction with Missing Data: Theoretical Guarantees vs. Empirical Performance
Dimitris Bertsimas
dbertsim@mmit.co edu
Sloon School of Management
Massachusetls Institute of Technology, Cambridge (MA), USA
Arthur Delaruc
arthur seiturddelruee@isse..gee... edu
H. Millon Stewart School of Industrial and Systems Engineering Georgia Institute of Technology, Atlanta (GA), USA
Jean Pauphilet
jpouphitett@oonddnnccoo edu
London Business School, London, UK
Reviewed on OpenReview: https: // openreview. net/ forum? id= IKH5ziX9dk
Abstract
Missing data is a common issue in real-world datasets. This paper studies the perfor- mance of impute-then-regress pipelines by contrasting theoretical and empirical evidence. We establish the asymptotic consistency of such pipelines for broad family of imputation methods. While common sense suggests that a 'good' imputation method produces datasets that are plausible, we show, on the contrary, that, as far as prediction concerned, crude can be good. Among others, we find that mode-impute is asymptotically sub-optimal, while mcan-impute is asymptotically optimal. We then exhaustively assess the validity of these theoretical conclusions on large corpus of synthetic, semi-real, and real datasets. While the empirical evidence collect mostly supports theoretical findings, it also highlights gaps between theory and practice and opportunities for future research, regarding the relc- vance of the MAR assumption, the complex interdependency between the imputation and regression tasks, and the need for realistic synthetic data generation models.
1 Introduction
Real-world datasets are plagued with missing values. For inference purposes, the key assumption that data entries are missing at random (MAR) ie., the fact that feature is missing and its (unobserved) value are independent, conditional on the observed features (Rubin) 1976). Under the MAR assumption, valid inference can be performed by using tailored FM algorit that can handle missing values (e.g.. Dempster et al. 1977 Ibrahim et al. 2005 Jiang et al. 2020). Alternatively, one can impute the missing values, estimate the parameter of interest on the imputed dataset, and obtain valid confidence intervals by accounting for imputation errors (e.g., via multiple imputations as in Rubin 1987). A myriad of imputation techniques have been proposed based on mean and mode imputation (Little & Rubin 2019). k-nearest neighbors (Troyanskaya
al. 2001 Brás & Menezes 2007). least square regression (Bø et al. 2004 Kim
et al. 2005 Cai et al. 2006 Zhang et al. 2008), support vector maahinee/rgrreseee
(Wang et al. 2006
Bertsimas et al. 2018). decision trees (Bertsimas et al. 2018), neural networks (Yoon et al. 2018). factor
analysis and other dimension reduction techniques (Mohamed et al. 2009 Husson et al.] 2019). All these
methods assume that the MAR assumption holds, hence missing values can be guessed accurately from non- missing values. However, without any further assumptions about the data, the MAR assumption cannot tested nor refuted from the data Little 1988 Jacger 2006, for some additional assumptions amenable statistical testing). In addition, most inference guarantees become invalid as soon the data is not missing at random (NMAR).

---

Published in Transactions on Machine Learning Research (06/2024)
Prediction, however, is different task from inference (Shmueli 2010 that requires different treatment for missing data. Besides tree-based methods that can natively handle missing values (see (Josse et al. 2019, Section 6 and references therein), prediction with missing data is usually performed following a similar approach as the one for inference, namely an sepptt-then--egnee values, and then trains a model on the imputed dataset. Accordingly, the MAR assumption, which is required for the imputation step be unbiased, often perceived as a requirement for impute-then-rogross approaches.
approach in which one first imputes missing
Our paper contributes to the recent literature on the analysis of thenuteethhe--rgress pipelines and the rele- vance of the MAR assumption. Our analysis starts by theoretical guarantee on the asymptotic consistency of impute-then-regres pipelines for broad class of imputation methods, which generalizes (Josse et al. 2019) and complements (Le Morvan et al.) 2020b 2021 the existing literature. To the best of our knowl- odge, our result the first to cover the imputation of categorical features, and not only continuous ones. In particular, our result characterizes the (asymptotically) "optimal" imputation mothods as those that en- code for missingness as obviously as possible. For simple imputation rules, conclusions from the theory are thus clear: mode imputation (though often used in practice, see Jääer et al., 2021) is sub-optimal, while mcan-imputation is asymptotically optimal. We then exhaustively assoss the validity of these conclusions on synthetic, semi-real, and real datasets, with the double objective of supporting our theoretical findings with empirical evidence as well as eliciting gaps between theory and practice that could motivate future research in the area.
The rest of the paper is organized as follows:
We study the consistency of generic imputation rules in the infinite-data regime in Section 2 While inference requires the imputed values to be unbiased estimates of the missing entries (a property usually satisfied under MAR), we show that, for prediction, impute-then-regres works the imputed value codifies missingness, i.e., if the imputed values are conspicuous as possible. Our analysis applies to contimous well as categorical missing features and to a broad family of imputation rules, including mode and mean imputation.
For mode and mean imputation, we contrast these theoretical consistency results with their empirical performance on a large corpus of synthetic, semi-real (i.e., real-world design matrix and missingness patterns but synthetic signals), and real-world datasets.
For discrete variables (Section 3.2 our theoretical analysis suggests that mode imputation cannot lead consistent predictions and that encoding missingness its own category should be preferred. Empirical evidence on synthetic and semi-real datasets strongly supports the validity of these findings, although the evidence on real data is not conclusive.
For continuous variables (Section 3.3), rules as simple as mean imputation are tcelretically consistent. We compare the performance of mean imputation with non-linear iterative imputa- tion mothods (van Buuren & Sroothnis-Oudshoomm 2010).. As suggested by theory, we observe no clear downside from using mean imputation on average. However, we observe that the missingness mechanism, the fraction of missing entries, and the complexity of the downstream predictive model have significant impact on which imputation model performs best.
Finally, we discuss the limitations of our thcoretical and empirical findings in Section ! with the hope lo guide future research. In particular, the contrast between theory and practice highlights interesting directions related with the relevance of the MAR assumption in predictive setting; the complex interactions between the imputation and the predictive models and its impact on finite- sample performance; and the need for realistic generative models.
Notation. We denote scalars by lowercase characters (x) and random variables by uppercase characters (X). Boldfaced characters denote vectors (e.g., x is a vector and X is random vector). The symbol designates independent random variables. For any positive integer n, let [n] (I....,n).
2

---

Published in Transactions on Machine Learning Research (06/2024)
Code. All code used to implement and evaluate imputation strategies is available on Github
2 Theoretical Consistency with Generic Imputation Rules
2.1
Setting
Generalities and notation. We consider the task of predicting target or dependent variable, modeled as a random variable Y, from a set of features, modeled as random vector X. For each feature X,, the binary random variable M; indicates whether it is missing (1) or observed (0). Our predictive model is trained using data: n iid. samples (z); m, yr), E [nl], where 2; E R is the vector of covariates and m, E {0, 1] is a vector indicating the missing covariates maj 1 if Tij is missing, 0, otherwise- and Wi R is the output of interest. For every data point ||m=]lo :- E, mij covariates are missing, We refer to m, the missingness indicator or missingness pattern of sample i. We further denote o(z,, m,) the vector of observed covariates (Seaman et al. 2013).
(d -
For the task of predicting Y given covariates X, predictor fn trained dataset of n observations is called (aspmptotically) consistent if lim4-40 (x) E[Y|X æ/. Moreover, fn is universally consistent if the previous statement holds for any distribution of (X,Y).
Objective. Our goal in this section is to investigate the asymptotic consistency of impute-then-regress strategies for a broad family of imputation rules.
For ease of exposition, we consider a simplified setting in d dimensions where only the first covariate X1 is missing. The optimal (consistent) predictor for the target variable Y is
E = x, M1 = 0], E
if m1 0,
(1)
X2xx 2.d F2.d; M1 1],
if mm
However, like in Josse et al. (2019), our analysis could be extended to the case with more than one missing covariate. Here, we concisely denote 32.d the (d - 1)-dimensional vector (I2,...,,
We study the common practice of imputing a deterministic value for X₁ whenever it is missing. In particular, we allow for the imputed value be a deterministic function of the other covariates, denoted p(æ2a)) This model captures mean and mode imputation, well conditional mean and mode imputation. For clarity, will denote X the random variable obtained from M) after p-imputation, i.e., X Xi if Mi 0, X = μ( (2.a) if M; = 1, and X 2dd X2.d-
2.2
Asymptotic prediction function
We now introduce the main result of this section, which characterizes the asymptotic performance of impute- then-regress strategies based on deterministic imputation rules.
Theorem 2.1. Consider universally consistent learning algorithm when trained on any fully observed dataset. Imputing p(æza)) for X2 X2d D2dd when X1 is missing (M1 = 1) on the training set and training a predictor on the impuled datasel lends, in the limit with infinite data, to the following prediction rule, denoted fu-tmpuse(2x) and equal almost everywhere to
E = M1 =0],
if 21 # p(zza))
(p-impute(w) =
((xx)) E| 22dds Mi =1]
+ -1--ah ( X1 u((2dd) X2.d B2.di M1 0], otherwise.
where
7(x) PP = 2 F2.d) is the probability that X, is missing given X
http:://gsttubb.com/aaaat
3

---

Published in Transactions on Machine Learning Research (06/2024)
Pu(x) = P(X1 ju(zz.d), M1 X|:d D2.a) is the probability for the true X1 to take the imputed value (u(22:d) and not be missing, given the other covoriales,
n(x))
and a(x) n)(x) + Pp(a) is the posterior probability that X1 was missing before imputation, given that it lakes the value u((r2ad) after impulation, i.e., PPM = 1 = p( X2.1 = T22dd).
Theorem 2.1 characterizes the effect of an arbitrary deterministic imputation rule on the downstream predic- tion rule in the infinite-data setting. Consider applying fn-impute new observation æ. Either #1 * p(æzaa) 21 p(=22d). Recall that in the infinite-data regime, only the training data in the neighborhood of x affects the prediction. In the first case, æ is almost, surely surrounded by points with no missing entries, so Ju-impate predicts the conditional expectation E[Y|X= x, M1 = Oj. In the second case, however, the points in the neighborhood of æ come from a mixture of two distributions, depending on whether I pu(22a)) occurs artificially due to imputation, or naturally in the data: some points have a missing first feature im- puted by yu-), and therefore are sampled from (m(#2:d), M2 = 1; other points are sampled from the original distribution (X1, X2:a)| O. The predicted outcome is a weighted average of both conditional expectations, with (xx) being the proper weighting factor. We defer the detailed proof to Appendix A
Interestingly, we observe that a(x) # n)x=)= = P(M = 11 2:d 22.a) in general. Even when X1 and M1 are conditionally independent (i.e., MAR assumption), imputation induces correlation between X and M1.
We note that Theorem 2.1 generalizes Josse et al. (2019 Theorem 4), which only applies to constant im-
putation for continuous features and requires the MAR assumption. Theorem 2.1 (or its implications for asymptotic consistency in Corollary 2.2) can also be viewed as a weaker version of Le Morvan et al. (2021, Theorem 3.1) since it applies to more restrictive setting with only single missing covariate. However, Le Morvan et al. [2021 Theorem 3.1) applies to continuous features, while our result equally applies to discrete features. Theorem 2.1 elicits an explicit condition the imputation function (through the quantity a<x)) that drives asymptotic consistency, hence providing further intuition on what permits an impute-ther then- regress pipeline to learn the Bayes-optimal predictor.
2.3 Implications of Theorem 2.1
We now apply Theorem 2.1 to study the out-of-sample predictions from a learner trained on p-imputed data. For new observation (o(x, m1), m1), we apply p-imputation and then predict according to fu-impute(2") (x"). For the sake of the discussion, assume that [E = M2 = 1] + E u(x2.a), X2.d B2dd M1 = 0), otherwise the imputation is obviously harmless.
If, on one hand, æ is not originally missing (m² = 0), then x = 21 and the impute- thentettthe-prrdicct rule agrees with the Bayes-optimal predictor, [X = x, Mi = 0), almost everywhere, if either Iı * (( almost surely (px(x) + a(x) 1) or axx)
On the other hand, if m 1, then xt ((x2a)), so the impute-then-pree preedicct rule agrees with the Bayes-optimal estimator X2.d = 22.d; M1 1] if and only if a<x: 1.
Again, axx) corresponds to the posterior probability that X, was missing in the original observation given that X = u(æ2.a). In other words, Theorem 2.1 indicates that consistency is achieved as long as the predictor can almost surcly de-impute, that is properly guess after imputation whether X1 was originally missing or not. This discussion can be summarized by the following corollary:
Corollary 2.2. Under the assumptions and notations of Theorem 2.1 cimputaaion--ten-neee asymptot- ically (i.e., in the infimite-data regime) leads to Bayes-optimal estimates at x if and only if a(xx) = or | = 22.d, M1 = 1] =E|Y|X = ,4, X2.1 = x2.d; M1 = 0).
Despile the simplicity of the underlying inluition, Theorem 2.1 challenges common practice. Indeed, one could think that a 'good' imputation method should produce datasets that are plausible, i.e., where imputed and non-imputed observations are indistinguishable (a(x) = 0). On the contrary, as far as predictive power is concerned, Theorem 2.1 speaks in favor of imputation methods that can be almost surely de-imputed (a<()) 1); because they can be used as encoding for missingness. In short, as far prediction is
4

---

Published in Transactions on Machine Learning Research (06/2024)
concerned, crude is good. This conclusion also supports the common practice of adding as part of the predictive features, a simple but powerful idea which is gaining traction in the deep learning community (Van Ness et al. 2023 Van Ness & Udell 2023).
3 Empirical Validation
3.1
Setting
Much of the literature on missing data relies on synthetic data for validation, mostly because grants the experimenter full knowledge of the missing values themselves. However, as result, missing data patterns may not match those found in the real world. In this paper, try to bridge the gap between real and synthetic data by creating diverse corpus of synthetic, semi-real, and real datasets. We briefly summarize our mothodology here; more details can be found in Appendix C For the synthotic datasets, we generate the design matrix X with n observations (ranging from 40 to 1,000) and synthetic signals of the form Y == f() + where the function f is either linear or the output of a two-layer neural network (NN). We ampute (i.e., drop) data either completely at random (MCAR) or by censoring extreme values (which is a special case of NMAR), and consider 8 different proportions of missing entries. In total, generate 49 datasets for each of the 2 x 2 x 8 = 32 configurations. For the semi-real and real datasets, we assemble corpus of 63 publicly available datasets with missing data, from the UCI Machine Learning Repository and the RDatasets Repository. We consider the design matrix and the missingness patterns from the real datasets. For semi-real instances, we generate synthetic response variables Y (again, according to a linear or neural network modcl). We control how Y depends X and M, and consider relationships corresponding to missing at random (MAR), not missing at random (NMAR) or adversially missing (AM). Experiments each dataset are replicated 10 times, with different training/test splits.
3.2 Categorical variables: mode imputation is inconsistent
For discrete features, choosing μ as the mode of the distribution of X1|M, 0 (a.k.a. mode imputation) is one of the advised methods in practico. However, with this choice of μt, for any æ, we have (x) > 0 so axx) < 1 and mode impute-then-reggres cannot be asymptotically consistent by Corollary Conversely, choosing f outside of the original support of X1, iee, encoding missingness as a new category/valuu provides consistency.
Theory.
2.2
The practical implications Corollary 2.2 for categorical variables are thus clear: We should encode missing- ness as a new category instead of imputing the mode. Accordingly, we numerically compare the out-of-sample performance of these two approaches.
Experimental setting. We evaluate the performance of mode imputation on our synthetic, semi-real, and real datasets. For the semi-real and real instances, we consider the 41 datasets with at least one missing catogorical fcature described in Tables C.1 and C.3 Appendix C.2 Real signals Y are available for 34 out of the 41 datasets. For training predictive models, missing numerical features are mean-imputed and missing categorical are either encoded as their own category or mode-imputed. The downstream predictive model is chosen among regularized linear, a tree, a random forest, two-layer neural network, and an XGBoost model (using 5-fold cross- validatio on the training data).
Results. To quantitatively assess the effect of mode imputation, we compare the out-of-sample accuracy (measured in R² or AUC) with mode imputation with that obtained when encoding missingness as its own category, using paired t-test (difference in means) and paired Wilcoxon test (difference in pseudo- medians). Results are reported in Table We observe that mode imputation has significant and negative (detrimental) effect on predictive power on instances with synthetic signals (for both synthetic and real design matrix and across missingness mechanisms), hence corroborating the insights from Theorem 2.1 In terms of magnitude, the average reduction in R² RR/AUCC mostly occurs at the second decimal. Yet, we do not observe a significant effect when we predict the real signal Y, which suggests that other factors beyond missing data might impact the validity of Theorem 2.1 in practice (e.g., finite amount of samples, limited class of predictors).
5

---

Published in Transactions on Machine Learning Research (06/2024)
Table 1: Difference in means and in pseudo-median (with two-sided p-values) in out-of-sample accuracy from t and Wilcoxon test applied to assess the impact of mode imputation on downstream accuracy. A negative value means that mode imputation reduces accuracy.
Design Signal Y
Missingness
# comp.
A mean (p-value)
A pseudo-median (p-value)
matrix X
MCAR
3,920
-0.0758
(***)
-0.0762
(***)
Syn. Linear
Censoring
3,920
-0.1270
(***)
-0.1248
(***)
Syn.
MCAR
3,920
-0.0697 (***)
-0.0722 (+*4))
Syn. NN
Censoring
3,920
-0.1214 (***)
-0.1153
MAR
1,760
-0.0303 (**)
-0.0079
Syn. Linear
NMAR
1,760
-0.0277 (**)
-0.0083
AM
1,760
-0.0303 (**)
-0.0071
Real
MAR
1,760
-0.0320
(**)
-0.0057
Syn. NN
NMAR
1,760
-0.0357
(**)
-0.0070 (***)
AM
1,760
-0.0321
(***)
-0.0064 (***)
Roal
Real
Real
340
0.0073 (0.29)
-0.001 (0.19)
Note: p-values *** 10020) 20; ** 10 10; *x< 10=9;
3.3
Continuous variables: mean imputation is consistent
Theory. For continuous features, one of the simplest and most widely used imputation rule is mean imputation, namely using = | = 0]. Further assume that X2 = 22.a. M₁ 0 is continuous (i.e., if the 'observed' X1 is continuous, conditioned on the other covariates), then, conditioned on X2.d = 22dd and M1 = 0, the probability that X2 takes any specific value is 0 so Pp (z) = and a<x)) = 1. Consequently, asymptotically consistent, as already proved by Josse et al. (2019 Theorem 4). Intuitively, systematically imputing p for X1 creates a discontinuity in the distribution of X and the events (Xf μ4} and {M1 =} are equal almost surcly. A universally encoding for missingness. Observe that any constant imputation rule (e.g., 0-imputation out-of-range imputation) satisfies this property, because it creates mass in an otherwise-contimouss distribution.
Corollary 2.2 guarantees that mean imputatioon-then-rerree
consistent downstream predictive model is then able to learn this pattern and view X =
In practice, this result suggests that sophisticated imputation methods for continuous variables are not needed, and may indeed be counter-productive. We now empirically evaluate the validity of this finding.
Experimental setting. Our analysis comprises the same synthetic, semi-real, and real datasets. For the semi-real and real instances, consider the 52 datasets with at least one missing numerical feature described in Tables C.3 and C.2 in Appendix C.2 Real signals Y are available for 36 out of the 52 datasets. Missing categorical features are encoded as their own category and missing mumerical features are either mean-imputed or imputed using the complex iterative method mice (van Buuren & Groothnis-Oudshoomm) 2010). The downstream predictive model is chosen among a regularized linear, a tree, a random forest, a two-layer neural network, and an XGBoost model (using 5-fold cross-validation on the training data).
compute the empirical mean the training data and use it to impute the missing values on both the training and testing data, in order to have the same imputation rule for the training and test set. For mice, we first impule the training set alone, and then impute the test set with the impuled training data. We discuss alternative implementations in Appendix D
For mcan-imputation,
Results. Table 2 reports the output from paired and Wilcoxon tests to compare out-of-sample accuracy obtained when using mcan-impute vs. mice (a negative value indicates that mice performs worsc). We pri- marily comment on the differences in average accuracy (t-test), the difference in median accuracy supporting similar conclusions.
6

---

Published in Transactions on Machine Learning Research (06/2024)
Table 2: Difference in means and in pseudo-mediame (with two-sided p-values) in out-of-sample accuracy from t and Wilcoxon test applied to assess the impact of mice imputation on downstream accuracy. A negative value means that mice reduces accuracy compared with mean impute.
Design matrix X
Signal Y
Missingness
# comp.
A mean (p-value)
A pseudo-median (p-value)
MCAR
3,920
0.0043 (0.001)
0.0112
***)
Syn. Linear
Censoring
3,920
-0.1047 (***)
-0.0908
(***)
Syn.
MCAR
3,920
0.0318 (***)
0.0339
Syn. NN
Censoring
3,920
-0.1275 (***)
-0.1193
MAR
2,180
-0.0256 (*)
-0.0115
Syn. Linear
NMAR
2,390
-0.1890 (**)
-0.0106
AM
2,390
-0.0020 (0.31)
-0.0029
Real
MAR
2,390
-0.0168 (*)
-0.0088
Syn. NN
NMAR
2,390
-0.0196 (**)
-0.0134 (***+)
AM
2,390
-0.0110 (+)
-0.0069 (***)
Roal
Real
Real
360
-0.0043 (0.38)
-0.0028 (0.008)
Note: p-values ** 10 20; ** 10 10, *
10
10"9;
On the fully synthetic datasets, mice imputation leads more accurate predictions when the data is MCAR, but is less accurate than mean impute when the data is censored. Both observations are highly statistically significant. To elicit the mechanisms at play, we report in Figure | the average out-of-sample R² obtained by each method as a function of the fraction of missing entries in the data (which is one of the parameters we control), for the four design scttings. Across all scttings, mice imputation is preferable when the proportion of missing entries is low and seaa--impute--he--rgeeeee larger. The only difference between the MCAR and Censoring settings is the value of the break-even point that makes both approaches comparablc. This behavior can be explained by the fact that the proportion of missing entries is directly related to the nunber of observations available to calibrate the imputation model: As the fraction of missing entries increases, there are fewer observations available to learn how to impute so complex models like mice are likely to overfit and perform poorly. However, we do not observe such clear pattern for the impact of the sample size directly (Figure E.3
is stronger when the proportion of missing entries is
On the semi-real and real instances, we observe that the impact of mice-then-regress is negative and signifi- cant (p-value 10-5) in five cases out of 7, and not statistically significant in the remaining cases. These mixed results are consistent with our theoretical findings: According to Corollary 2.2 both mean impute and mice could be asymptotically consistent so there is reason to expect the other. Nonetheless, the strong performance of mcan-impute, despite its simplicity, is remarkable. Yet, we should note that the maguilude of the effect in average accuracy is smaller than for mode imputation in the previous section.
be systematically better than
4 Discussion
We now discuss the implications and limitations our findings. We first discuss how our analysis calls into question the relevance of the MAR assumption for prediction. We then highlight two limitations of our approach: the infinite-data assumption in Thcorem 2.1 and the discrepancy in numerical results between real and synthetic data. We propose simple ways to alleviate these limitations, and identify avenues for further research on these topics.
Relevance of the MAR assumption. A striking feature of the theoretical consistency guarantee for impute-then-regres pipelines (Thcorem 2.1p is the absence of the MAR assumption (the same observation holds for Theorem 3.1 in Le Morvan et al. (022). Empirically, however, we observe that the relative per- formance of mice mean imputation synthetic data depends strongly on the missingness mechanisms.
7

---

Published in Transactions on Machine Learning Research (06/2024)
Out-il-sampp R2
Dut of R2
0.5
Mean Ampute-then-Reeee Regress mice imgute-then-Regies
03
Mean Impute then- Regress mice Impute-then-Reeeee
0.0
0.2
0.4
0.6
0.8
0.0
0.2
0.4
0.6
0.8
Proportion missing values
Proportion of missing values
(a) Y linear MCAR
(b) Y linear Censoring
0.6
0.5
Oat-l--mmpee R2
I
Out sfample R2
0.4
0.4
0.3
0.
Mean then Regress mice impute-then-Rggress
Mean Impute then Regress mice Impute-then-Regress
0.0
0,2
0.4
0.
0.8
0.0
0.2
0.4
0.6
0.8
Proportion missing
Proportion missing values
(c) YNN- MCAR
Y NN Censoring
Figure 1: Average oft-lle R of then-eeens and seea--imputethhh--reeee on fully synthetic data, as the proportion of missing entries increases. Results are averaged over 50 different sample size and 10 training/tes splits.

---

Published in Transactions on Machine Learning Research (06/2024)
Thus, one might wonder what the true role of the missingness mechanism is. Given generative model for (X, Y); the missingness mechanisms impact the shape and regularity of the function [ M) hence its learnability. Since Theorem 2.1 considers a universally consistent predictive model, we can always guarantee consistency with infinite data. For a finite amount of data, however, the shape and regularity of the function and so, indirectly, the missingness mechanism will impact the performance of method. Based on this observation, one might wonder whether alternative assumptions would be more relevant to study and develop methods for prediction with missing data.
In particular, inftuenced by the insights from inference, practitioners tend to consider MAR as favorable situation compared with NMAR. Indeed, textbooks in the field, such as Hastic et al. (2001 Chapter 9.6) or Kuhn et al. (2013] Chapter 3.4), tend to present MAR favorably. However, one should emphasize that, as far as prediction is concerned, NMAR can be a blessing and not a curse, because missingness can sometimes be used as a powerful predictor of the outcome of interest (i.c., predictive missingness). We formalize this observation in the sctting of Theorem 2.1 (proof in Appendix B
Theorem 4.1. Consider two missingness mechanisms M1 and Mi leading to the same proportion of missing entries PM, 1) P(M = 1) = p. Further assume that, conditioned on X2.d Mi is independent of and X1 (MAR). Then the optimal prediction rule achieves a lower prediction error under M1 than under Mi if and only if the following condition holds:
(1 (1-1) (x, x) + p21// (X2rd: 22 p(1 ()) X) p(1 - - X2.a) 2 0,
where
1 B) E (E E M | =i] 2
We now provide simple examples of each situation.
Example 4.2 (Prediction benefits from NMAR). Let X1 be a Bernoulli random variable with parameter 1/2, and Y = X1. We compare two missingness patterns: M1 X1, and Mi 1 X1,, another Bernoulli random variable with parameter 1/2. Then, the optimal predictor under M1 is Y Xı when M1 = 0 and Y M1 = 1 when Mj = 1, with empirical risk R = 0. Meanwhile, the Bayes-optimal predictor under Mí is Y == X1 when M = 0 and Y 1/2 when Mí- = 1, with empirical risk R' = 1/8.
Example 4.3 (Prediction benefits from MAR). Let d = 2, and let X2, U, V be three independent Bernoulli random variables with parameter 1/2. Define X₁ XU = V = 1) and let Y X1. We compare two missingness patterns: M1 = U, and Mj an independent Bernoulli random variable with parameter 1/2. Then the Bayes-optimal learner under M, is X2 when Mj = 0 and 1/2 when M1 1, with empirical risk R = 1/8. In contrast, the Baycs-optimal learner under M1 is X1 when M 0, and X2/2 + 1/4 when Mi 1, 1, with empirical risk R' = 3/32.
Such result highlights the fact that the MAR/NMAR distinction does not correctly classify missing data into "goodd//"bd"" cases and that a better taxonomy is needed. This insight is valuable because missing data mechanisms are inherently unobservable and because the MAR assumption is obviously violated in many
industrial applications, e.g., in pricing and revenue management (Alles et al. 2000: Wang
2022
Pauphilet
2022
Fimite-sample and parametric models. One limitation of Theorem 2.1 is that it relics on two restrictive assumptions, namely that the number of observations grows to infinity and that the rewnstreem predictive model is universally consistent. Universally consistent models, however, require large datasets, while real- world data is often limited. Hence, in practice, one might favor parametric classes of predictors (instead of universally consistent learning algorithms), for which Theorem 2.1 does not apply. More work is needed to design tailored classes of parametric models for prediction with missing data (such as Le Morvan et al. 2020a) or to understand the finit-sample performance of impute-then-regre pipelines.
In the latter case, we believe that the interactions between the model complexity of the imputation rule and that of the downstream regression are not yet vell-understood and provide exciting grounds for future research. Figure 2 illustrates this phenomenon by representing the out-of-sample R² of four impute-then- regress pipelines (using mice/mean imputation and linear/randon forest regressor), as the number of
9

---

Published in Transactions on Machine Learning Research (06/2024)
Out-of-sample R2
0.3
Mean impute then Linear Regression Mean impute then Random Forest impute mice im impute then Random Forest
Regression
0.2
200
400
600
800
Number of samples
synthetic data with linear signal, NMAR data, and 40% of missing entries, the number of samples increases. We report the performance of two different downstream predictors: Linear (T.ASSO) regression and random forest. Results are averaged over 10 training/test test splits.
Figure 2: Out-of-sample R mi ci-rreeress and mann-imputte-then--eeesss
0.5
0.5
Out sample R2
Out-ofsample
0.4
0.3
0.2
0.2
Mean impute then XGBoost mice impute then XGBoost XGBoostt
Mean impute then XGBoost mice impute then XGBoost XGBoost
200
400
200
400
800
Number samples
Number of samples n
(a) Y non-linear MCAR
(b) Y non-linear Censoring
Figure Average oft-of-sample R² of XGBoost with mean impute, mice, or no imputation method, on synthetic data with non-linear signal, NMAR missing data, and 40% of missing entries, as the number of samples n increases. Results are averaged over 10 training/ter test splits.
observations increases, for our synthetically generated datasets, with censored data and non-linear synthetic signal Y. When a linear regression model is used, find that mice consistently outperforms mean impute (across all values of n)) and is strictly more accurate asymptotically. When using random forest regressor, however, we observe the exact opposite. These empirical observations complement some theoretical findings from Le Morvan et al. (Section (2021) and demonstrate the complexity of the inter-dependencies between the classes of models used for imputation and regression.
This observation holds also for models that can handle missing data natively such as tree-based methods. For XGBoost, for example, compare how the out-of-sample accuracy increases with the number n of training samples, for mean impute followed by XGBoost, mice impute followed by XGBoost, and XGBoost applied on missing data directly in Figure 3 We observe that the relative performance of the no-imputation strategy (especially compared with an imputation method like mice) varies drastically with the missingness pattern. We can make a similar observation for random forest (Figure E.4. These observations highlight that, although "hassleffree', methods which can handle missing data directly come at a cost in terms of predictive power.
10

---

Published in Transactions on Machine Learning Research (06/2024)
Realistic synthetic data. A second limitation is that, while our empirical experiments largely confirm the theoretical findings when synthetic data is used, the conclusions real data (and to a lesser extent on semi-real data) are noticeably less obvious. This observation raises the question of the validity of analysis conducted with synthetic data in the literature, especially for research related with missing data. Algorithms for missing data imputation are almost always validated on synthelically generated data or at. least synthel- ically amputed data, settings where the gains (both in imputation error and downstream accuracy) can be significant (e-g., Bertsimas al., 2018 2021). On the contrary, we observe on real-world design matrices and missingness patterns (both for synthetic and real signals Y) that imputation method sophisticated as mice leads, on average, to less accurate predictions than simple mean imputation. Synthetic data are useful to researchers because they often allow for more extensive numerical validation but this is only true if they accurately represent how data go missing in practice. Our findings thus highlight the need for more realistic and accessible models for missingness to allow researchers to benchmark and develop models that are better suited to real-world data with missing entrics.
References
Michael Alles, Amin Amershi, Srikant Datar, and Raina Sarkar. Information and incentive effects of inven- tory in JIT production. Management Science, 46(12):1528 1544, 2000.
Dimitris Bertsimas, Colin Pawlowski, and Ying Daisy Zhuo. From predictive methods to missing data imputation: An optimization approach. Journal of Machine Learning Research, 18(196):1 39, 2018,
Dimitris Bertsimas, Agni Orfanoudaki, and Colin Pawlowski. Imputation of clinical covariates in time scries. Machine Learning, 110:185 248, 2021.
Trond Hellem Bø, Bjarte Dysvik, and Inge Jonassen. Lsimpute: accurate estimation of missing values in microarray data with least squares methods. Nucleic Acids Research, 32(3):e34 e34, 2004.
Lígia P Brás and José C Menezes. Improving cluster-based missing value estimation of DNA microarray data. Biomolecular Engineering, 24(2):273 282, 2007.
Zhipeng Cai, Maysam Heydari, and Guohui Lin. Iterated local least squares microarray missing value
imputation. Journal of Bioinformatics and Computational Biology, 4(05):335-99) 957, 2006.
Arthur P Dempster, Nan M Laird, and Donald B Rubin. Maximum likelihood from incomplete data via the em algorithm. Journal of the Royal Statistical Sociely: Series B (Methadological), 39(1):1 22, 1977.
Trevor Hastie, Robert Tibshirani, and Jerome Friedman. The Elements of Statisticol Learning, volume 2. Springer series in statistics New York, 2001.
François Husson, Julie Josse, Balasubramanian Narasimhan, and Geneviève Robin. Imputation of mixed data with multilevel singular value decomposition. Journal of Computational and Grophical Statistics, 28 (3):552 566, 2019.
Joseph G Ibrahim, Ming-Hut Hui Chen, Stuart Lipsitz, and Amy H Herring. Missing- data methods for generalized linear models: A comparative review. Journal of the American Statisticol Association, 100 (469):332-34) 346, 2005.
Manfred Jaeger. On testing the missing at random assumption. In European Conference on Machine Learning, pp. 671-678. Springer, 2006.
Sebastian Jäger, Arndt Allhorn, and Felix Bießmann. A benchmark for data imputation methods. Frontiers in big Data, 4:693674, 2021.
Wei Jiang, Julie Josso, Mare Lavielle, and TraumaBase Group. Logistic regression with missing covari-
ates parameter estimation, model selection and prediction within a joint-modeling framework. Compu- tational Statistics & Data Analysis, 145:106907, 2020.
11

---

Published in Transactions on Machine Learning Research (06/2024)
Julie Josse, Nicolas Prost, Erwan Scornet, and Gaäl Varoquaux. On the consistency of supervised learning with missing values. arXie preprint
2019.
Hyunsoo Kim, Gene Golub, and Haesun Park. Missing value estimation for DNA mieroarray gene expres- sion data: local least squares imputation. Bioinformatics, 21(2):187 198, 2005.
Max Kuhn, Kjell Johnson, al. Applied Predictive Modeling, volume 26. Springer, 2013.
Marine Le Morvan, Julie Josse, Thomas Morcau, Erwan Scornet, and Gačl Varoquaux. Neumiss networks: dileerentiable programming for supervised learning with missing values. Advances in Neural Information Processing Systems, 33:5980-505 -9900, 2020a.
Marine Le Morvan, Nicolas Prost, Julie Josse, Erwan Scornet, and Gaël Varoquaux. Linear predictor on linearly-generated data with missing values: non consistency and solutions. In International Conference on Artificial Intelligence and Statistics, pp. 2165-3174) PMLR, 2020b.
Marine Le Morvan, Julie Josse, Erwan Scornet, and Gaël Varoquaux. What's good imputation to predict with missing values? Advonces in Neural Information Processing Systems, 34:11530-11555 2021.
Roderick JA Little. A test of missing completely at random for multivariate data with missing values. Journal of the Americon Statistical Association, 83(404):1198-122,, 1988.
Roderick JA Little and Donald Rubin. Statistical Analysis with Missing Data, volume 793. Wiley Sons, 2019.
Shakir Mohamed, Zoubin Ghahramani, and Katherine A Heller, Bayesian exponential family pca. In Advances in Newral Information Processing Systems, pp. 1089-1096, 2009.
Jean Pauphilet. Robust and heterogenous odds ratio: Estimating price sensitivity for unbought items. Manufacturing & Service Operations Management, 2022.
Donald B. Rubin. Inference and missing data. Biometrika, 63(3):581-592, 1976.
Donald B Rubin. Multiple Imputation for Nonresponse in Surveys, volume 81. Wiley & Sons, 1987.
Shaun Seaman, John Galati, Dan Jackson, and John Carlin. What is meant by "missing at random"? Statistical Science, pp. 257-268, 2013.
Galit Shmueli. To explain or to predict? Statistical Science, 25(3):289 310, 2010.
Daniel J Stekhoven and Peter Bühlmann. Mosnforett missing value imputation for mixed- type data. Bioinformatics, 28(1):112 118, 2012.
Olga Troyanskaya, Michael Cantor, Gavin Sherlock, Pat Brown, Trevor Hastie, Robert Tibshirani, David Botstein, and Russ B Altman. Missing value estimation methods for DNA microarrays. Bioinformatics, 17(6):520-53) 520,5525222220 2001.
Stef van Buuren and Karin Sardduus-Oudshooon.. mice: Multivariate imputation by chained equations in r. Journal of Statistical Software, pp. 1-68, 2010.
Mike Van Ness and Madeleine Udell. In defense of zero imputation for tabular deep learning. In NeurIPS 2023 Second Toble Representotion Learning Workshop, 2023.
Mike Van Ness, Tomas M Bosschieter, Roberto SeoppnnGreeoroo,, and Madeleine Udell. The missing indicator method: From low to high dimonsions. In Proceedings of the 29th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, pp. 5004 5015, 2023.
Ruxian Wang. What the impact of nonrandomness on random choice models? Manufacturing & Service Operations Management, 24(1)=485-58) 2022.
12

---

Published in Transactions on Machine Learning Research (06/2024)
Xian Wang, Ao Li, Zhaohui Jiang, and Huanqing Feng. Missing value estimation for DNA microarray gene expression data by support vector regression imputation and orthogonal coding scheme. BMC Bioinfor- matics, 7(1):22, 2006.
Jinsung Yoon, James Jordon, and Mihaela Van Der Schaar. Gain: Missing data imputation using generative adversarial nets. arXic preprint orXiv: 02920, 2018.
Xiaobai Zhang, Xiaofeng Song, Huinan Wang, and Huanping Zhang. Sequential local least squares imputa- tion estimating missing value of microarray data. Computers in Biology and Medicine, 38(10)1112-1120) 2008.
13

---

Published in Transactions on Machine Learning Research (06/2024)
A Proof of Theorem 2.1
Proof. Since the downstream predictive model is trained after p-imputation, the learner is trained not on
(o(X, Mi), Mi) but on its imputed version X*. Accordingly, asymptotically, Jysimpute (x) ) = X = x]
We now relate the conditional expectation of Y| = ax, X2.d = æ2.d with those of Y M1 = 22d = 22.d
and M| = 0, X1 = oी. X2.d = 22.d-
For random variables Z1, Zo, we denote oz, (2)) the distribution function of Zi and gz, (z Zg 22)d21 the distribution function of Z1| 7/2 22-
We condition on the event {X 2:d 22.1 for some 22,d such that Ix.. ( > 0. All our reasoning is conditioned on {X2.d 22:dh, e.g., all distributions/ cetpuutiios/pppbbbaa probabilities are conditional
but we omit the dependency on D2.d for simplicity. We also denote p=:= = Ju( (x2.a))
(Im + 1m =0 =0 of==)) ) 9(x, 9(x. Mi (xr, mi )dxf da dm, By inte-
The joint density for (X1, Mr, X grating over Mi), we obtain the density of X²::
h( = P(Mi 1)1 3(x.,M)) O)dx"
Note that
M,),,,,) 0)dx
if x + n, x = n
(af)da
PPM = 1)dxf + 90x.M) O)daf'
Furthermore, let us denote x(y, the joint density of (Y, ()) Since the joint density for X1, M1, X is (y = xI, Mi m1) (1m (1m 127= + 1m 1m1=0l2+= =0 (() m mdydd do ddmi, we have
h(y, (t,.4))de dydde = - = 1)P(MI = 1)dyda + gy (ulx = xf, M1 = ()) O)dydx (ddddddd Accordingly,
J ghly,,t)ds P(Mi = -1) Vg 1)dy + 9(x) / ygr ( = = Mi -)dy
= -1 1) = 1] (4 = 0, X1 = =
All together, distinguishing the case where (X1, M2) is contimuous//diccree and denoting
P(M 1)
a = P = P) M1 = 0)
we obtain
E| =f M
if if + p. if if = p.
X = =
= +1 o1 M, -]
B Proof of Theorem 4.1 and Examples
The quantities (A, B) in Theorem measure the distance between EE [Y]A] and E E Mj] conditional Mj i. In other words, these quantities compare the predictive power (on Y) of fcature set A and feature set (B, M. We note from the signs in the condition in Theorem 4.1 that sufficient condition for the NMAR
setting to yield higher predictive power is E (E [Y] - E | X M2)) | = 0, corresponding to the case where Mj is a perfect substitute for X,- This condition not necessary since the first three terms can be strictly positive: in this case, NMAR can lead to higher predictive power long as replacing the value X1 with the fact that it is missing preserves enough information about the value of Y.
14

---

Published in Transactions on Machine Learning Research (06/2024)
Proof. Proof of Theorem 4.1 Let R (R') designate the optimal empirical risk under missingness M1 (M))
= (1 (1- 1) [
| X, M. M1 = 0] + pl [[Y E [ X M = 1]
R' = (1 - P) [YY LE | X, Mi1 M = oj + + [(Y (Y X Mil) | = 1]
Because M/ is independent of Y given X2dd, we can write:
R' -1 (Y ( =0] + (Y EEY-EEYA 2 =
(1 (1- (Y [Y| pË [(Y E Y 2.d))%)
where the last equality follows from the MAR assumption. Then we can apply the tower rule, conditioning on Mf:
R' (1 p) (YY E X [M]] + E (Y E (
(1 D) [(Y [X ) M1 = +0000(1(1111) [(Y E ) 1]
+ p11 - p) [(Y [E X 2 [0] +p²B [( - E. Y 2d M| =
=: (1 p) + p(1 ( + C) +D²D.
Then, we can modify the terms A, B, C, and D as follows:
A = E [Y - E EY xI) M = 0]
E [(Y IE [Y] Mi] + EE [Y X Mi] E X)) = o]
EE [(Y IE [Y] M M = 0] + B ( Mj] - { 0]
2E [(Y E MI)
[Y] 0J
E [Y - IE [E MI) 0 (
since X Mi] - [ is constant when
conditioning on X and M1, and
E - Y x, M) M,,
.M] = 0. Similarly, we can show that
B E [(Y B[ X 2nd, | =1] 1/11 (X, X2.d),
C [(Y X 2 = 0] + +voo X2.d; X);
D = E [(Y - E Y X2:d, M | =1] + the 2:dd X2:d)-
Putting it all together yields:
R' = (1 - p) E X M)) = 0] + (Y Y X
, =1] X)
+ vv [X,X] X) + p² wh X 2:d. X2.a) + p(1 p)
p)
X 2:d=
p(1 p)411 (X, X2.a)-
Recognizing that the first two terms in the above expression are equal to R completes the proof.
C Description of the Data and Evaluation Methodology
In this section, we describe the datasets we used in our numerical experiments, as well as various implemen- fation details. In line with other works in the literature, we conduct some of our experiments on synthetic data, where have full control over the design matrix X, the missingness pattern M, and the signal Y. We also contrast the results obtained on these synthetic instances with real world instances from the UCT Machine Learning Repository and the RDatasets Repository
Note that all experiments were performed on a Intel Xeon E5 2690 v4 2.6GHz CPU core using 8 GB RAM.
htttps//arcchvee. ics. uci. and hww.://///gttta....aa/ccceeeeeee
15

---

Published in Transactions on Machine Learning Research (06/2024)
C.1 Synthetic data generation
As in Le Morvan et al. 22020b; Section 7), we generale a multivariate vector X from multivariale Gaussian with mean 0 and covariance matrix Σ == BB" ell where B E Rd with iid. standard Gaussian entries and < > 0 chosen small enough so that E > O. We fix d = 10 and r = in our experiments. For experiments requiring Saacrete/categorii sampling each entry according to Ber(logit (X,)))(X))))),) We generate n observations, the training data and 5, 000 observations for the test data.
features (Section 3.2), we convert the entries X into {0, 1] by {40, 60,..., 1000), for
We then generate signals Y f(X)+E, where f(-) is predefined function and is centered normal random noise whose variance is calibrated to achieve target signal-to-noise ratio SN R. We choose SNR = 2 in our experiments. We use functions f(-) of the following forms:
Linear model: f(x) + w where b ~ 10, and wy ~ ~(([-1,]])
Neural Network (NN) model: f(x) corresponds to the output function of a 2-layer neural network with 10 hidden nodes, ReLU activation functions, and random weights and intercept for each node.
We compute f(x) using a random subset of out of the d features only, with h 5.
Finally, for a given fraction of missing entries p, we generate missing entries according to mechanisms
Missing Completely At Random (MCAR): For each observation and for each feature j E (l....,d), d}, we sample Mj ~ Bern(p) independently (for each feature and each observation).
Not Missing At Random (NMAR) Censoring: We set Mj whenever the value of X, is above the (1 p)th percentile.
For the fraction of missing entries, we consider the different values p E (0.1,0......,...
With this methodology, generate total of 49 training sets, with 2 categories of signal Y, missingness mechanisms, and 8 proportion of missing entries, i.e., 1, 568 different instances.
We measure the predictive power of a method in terms of average oft-ol-sample R We use R², which is a scaled version of the mean squared error, to allow for a fair comparison and aggregation of the results across datasets and generative models.
C.2 Real-world design matrix
In addition to synthetic data, we also assemble a corpus of 63 publicly available datasets with missing data, from the UCI Machine Learning Repository and the RDatasets Repository. Tables C.I C.2 and C.3 present summary statistics for the datasets with only categorical features missing, only continuous features missing, and both categorical and continuous features missing respectively. We use the datasets presented in C.1 and in Section 3.2 to empirically validate that, for categorical features, missingness should be encoded as category instead of using mode imputation. Then, the datasets from Tables C.2 and in Section 3.3 to compare the performance of different impute-then-regree strategies and our adaptive regression models.
Tables
For these datasets, we consider two categories of signal Y, real-world and synthetic signals.
16

---

Published in Transactions on Machine Learning Research (06/2024)
Datasct
n
# features
#missiig cont.
#missing cat.
M
d
Eedat-Males
4360
38
mushroom
8124
117
bin.
post-operativv-patieet
90
24
bin.
breast-cancer
286
0
43
bin.
heart-dieeuse-cloolaadd
303
28
0
8
3
30
bin.
COUNT-loomis
384
0
g
4
12
cont.
Zelig-coalition2
314
24
0
14
2
25
NA
serell--landing-contte
15
16
0
16
6
21
bin.
pongressional-votinge records
435
32
32
60
48
bin-
lung-cancer soybean-large
32
157
33
3
159
bin.
307
98
0
98
132
bin.
Table C.1: Description of the datasels our library where the features affected by missingness are categorical features only, n denotes the number of observations. The columns '#features,, '#missing cont., and "#missing cat.' report the total number of features, the number of continuous features affected by missingness, and the number of categorical features affected by missingness, respectively. [M] correspond to the number of unique missingness patterns m E {0, 1) observed, where d is the total number of features after one-hot-encoding of the categorical features. The final column Y indicates whether the dependent variable is binary or continuous (if available).
Datasct
T
features
#missing cont.
#misssing cat.
[M
d
auto-mpg
398
13
cont.
breast- wweast-cancer-wife
699
9
0
bin.
creast-cancer-wiecossii pww.eccsiiiii--onnnttttii
198
32
0
2
32
bin.
dermatology
366
130
O
2
130
cont.
sgplot2-movies
58788
34
0
2
34
NA
indian-liver-patinn
583
11
0
2
11
bin.
sear-carrtest.frran
60
81
0
2
81
bin.
Ecdat-MCAS
180
13
0
3
13
cont.
MASS-Carsf13
93
64
3
64
cont.
car-Davis
200
6
NA
car-Freedman
110
4
car-Hartnagel
37
8
datasets-airquality
153
mee-Rev-eseem
1905
300
bin.
Bodatt-RetScooo
3078
37
cont.
arrhythmia
452
391
0
391
cont.
boot-neuro
469
6
0
reshape2-french frenchhrrccc..c fries
696
9
D
D
HA
survival-mgus
241
15
11
bin.
sem-'Tests
32
6
6
8
NA
robustasseammiinttOXcCh
366
13
13
45
13
NA
Table C.2: Description of the 22 datasets in our library where the features affected by missingness are numerical features only. n denotes the number of observations. The columns '#feature',, '#missing cont.' and *#iisiig cat.' report the total number of features, the number of continuous features affected by missingness, and the number of categorical features affected by missingness, respectively. | correspond to the number of unique missingness patterns m E {0, 1)d observed, where d is the total number of features after one-hot-encoding of the categorical features. The final column indicates whether the dependent variable is binary or continuous (if available).
17

---

Published in Transactions on Machine Learning Research (06/2024)
Dataset
#features
#missing cont.
#missing cat.
[M]
d
Y
pecl-ollticalllffomatiio
1800
1440
1
1431
1441
bin.
car-SLID)
7425
2
3
NA
rpart-stagec:
146
15
2
3
16
NA
EoclatSchoollig
3010
51
2
8
9
53
cont.
mammographic-mass
961
15
2
13
18
bin.
Muster-plantTTratts
136
68
2
37
16
85
NA
mimRev-star
24613
122
72
19
128
cont.
car-Chile
2532
3
bin.
17
bin.
beart-dieaase-siizzeeladdd
18
bin.
agplot2-msleep
29
survival-cancer
cont.
heart-discaso-va
bin.
MASS-survey
237
NA
hepatitis
155
5
20
42
hin.
automobile
205
2
7
70
cont.
ochocardiogram
132
2
13
bin.
Muyroid-discase-alll
2800
52
6
46
25
54
bin.
pyyrodddissasee aaleeesse-llllyeeee allhyper
2800
52
6
46
25
54
hin.
hhyoodddlsease-- plpppo
2800
52
6
46
25
54
hin.
pyyroddddisess--llll
2800
52
6
46
25
54
bin.
thyroid- -iieeeiissss--iiss
2800
62
6
46
25
54
bin.
-rridddiisess-sickk
2800
52
6
46
25
54
bin.
survival-pbe
418
27
7
17
32
bin.
3163
43
7
36
23
44
hin.
horse-colic
300
bin.
plyr-baseball
21699
14
297
communitie--nn-ccrine
1994
22
127
cont.
cont.
wikidhe
cont.
Table C.3: Description of the 30 datasets in our library where the features affected by missingness are numerical and categorical. n denotes the number of observations. The columns '#feature',, '#missing cont.', and #miising cat.' report the total number of features, the number of continuous features affected by missinguess, and the number categorical features affected by missingness, respectively. [M] correspond the number of unique missingness patterns m E {0, 1)d observed, where the total number of features after one-hot-encoding of the categorical features. The final column Y indicates whether the dependent variable is binary continuous (if available).
18

---

Published in Transactions on Machine Learning Research (06/2024)
C.2.1 Real signal Y
46 out of the 63 datasets had an identified target variable Y, which could be continuous or binary. If is categorical with more than category, we considered the binary one-vs-all classification task using the first (alphabetical order) category. For regression (resp. classification) tasks, we use the mean squared error (resp. logistic log-likelihood) as the training loss and measure predictive power in terms of R (resp. 2 x -UC Again, choose this measure over mean square error (resp. accuracy or AUC) because it is normalized between O and 1, and can be more safely compared and aggregated across datasets.
C.2.2 Synthetic signal Y
To generate synthetic signals Y, we use the same three generative models as with synthetic data in Section However, this requires knowledge of the fully observed input matrix, while we only have access to obser- vations with missing entries, (o(x1(), ml'), m()), ) i= I,....7t. n. Therefore, we first generate a fully observed version of the data by performing missing data imputation using the package missForest (Stekhoven &
Biihlmann 2012), obtaining a new dataset {(>/ull he(m) We use this dataset to generate synthetic signals using the three types of signals described in Section C.1 linear, tree, and neural network.
Regarding the relationship between the missingness pattern M and the signal Y, we consider three mecha- nisms:
In this setting, pass h: = min(10, d) coordinates of ærull as input to the function f()). Out of those k features, we explicitly control liisssin E (O....,k) k}, the number of features contributing to the signal that are affected by missingness. Hence, in this setting, the resulting response Y depends directly on the covariates X but not on the missingness pattern M. However, we do not control the correlation between X and M for two reasons: First, they both come from a real-world dataset which might not satisfy the MAR assumption. Second, previously observed (Section 2.2 imputation does induce some correlation between the imputed dataset f and M.
MAR:
NMAR: In the second setting, in addition to k 10 coordinates of Dfull, we also pass kmissing coordinates of m, so that Y is now function of both X and M.
Adversarially Missing (AM): The third setting generates Y in the same way as the MAR setting. After Y is generated, however, reallocate the missingness patterns across observations as ensure the data is NMAR. Formally, we consider the observations (o(*))) m (1)))) m( ((o))) i E [n], where o is the permutation maximizing the total sum of missing values Σn= 2 m m.))
For each weal-world dataset, this methodology generates up to 3 x 11 66 different instances.
All together, we obtain four experimental settings, with both synthetic and real signals Y. They differ in the relationships between the missingness pattern M, the design matrix X and the signal Y as sunmarized on Figure C.1
C.3 Evaluation pipeline
In our numerical experiments, we compare a series of impute-then--egress methods where the imputation step is performed either via mode/mm moodemmeanna imputation or using the chained equation method mice (van Buuren & Groothnis-Ouddhoomm Oudshoorn 2010). We implement these approaches with linear, tree, random forest model for the downstream predictive model. We treat the type of model as typer-parames We used the default parameter values for number of imputations and number of itcrations in mice.
For linear predictors, the hyper-parameters are the Lasso penalty À and the amount of ridge regularization a (ElasticNet).. For tree predictors, the exper-paramete is the maximum depth. For the random forest prodictors, the hyper-parametory are the maximum depth of cach tree and the number of troes the forest.
All hyper-parameters are cross- validated using 5-fold cross- validatio procedure the training set.
19

---

Published in Transactions on Machine Learning Research (06/2024)
X
X
Y
?
Y
Y
?
Y
M
M
M
M
(a) Syn-MAR
(b) Syn-NMAR
(c) Syn-AM
(d) Real
Figure C.I: Graphical representation of the 4 experimental designs implemented in our benchmark simula- tions with real-world design matrix X. Solid (resp. dashed) lines correspond to correlations explicitly (resp. not explicitly) controlled in our experiments.
We report out-ol--ample predictive power on the test set. For synthetic data, the test set consists of 5, 000 obsorvations. For real data, we hold out 30% of the observations as a test set.
All experiments replicated 10 times, with different (random) split into training/ Lest sets between repli- cations.
D Implementation of Impute-then-Regree Pipelines with MICE
To achieve the best achievable predictive power, Theorem 2.1 requires the imputation rule to be the the training set (on which the downstream predictive model is then trained) many imputation models (especially the best performing ones) do not impute missing values as a simple function of the observed features (p(#2;)) in the statement of Theorem 2.1 but rely on an iterative process where, at each iteration, current imputed values are used to train a new imputation model and then updated. Consequently, practice, we cannot guarantee that the exact same imputation rule used in training and testing.
the test set. However,
We consider three implementations of impute-then--egree
(V1) In the first variant, we simultaneously impule the Lrain and test set, before training the model.
(V2) Secondly, we impute the training set alone, and then impute the test set with the imputed training data.
(V3) As a third option, impute the training set alone and then the test set with the original training set.
Intuitively, (V1) should lead to the most consistent imputation across the training and test set but is not practical for predictive models that are meant be used in production. Indeed, the behavior of the model the test set should mimic its behavior should not be used) at any stage of the calibration process. (V2) is the variant compared mean-imputation against in Section 3.3 We intuit that (V3) will be less powerful than (V2) because the rules learned for imputing the test set might differ from the ones used for the training set.
future observations, which, by definition, are unavailable (hence
We conduct a regression analysis (Table D.1.. 1 to assess the relative benefit using (V1) and (V3) over (V2).
As expected, we observe that (V1) leads to higher accuracy than (V2) on the synthetic instances. On the semi-real and real data instances, however, observe no strong statistically significant difference wetw the two variants. Regarding (V3) vs. (V2), when statistically significant (7 out of 11 cases), find that the comparison is in favor of (V2). Henceforth, we recommend in practice to use (V2) since it mimics more closely the production pipeline and the theoretical requirements from Theorem 2.1
20

---

Published in Transactions on Machine Learning Research (06/2024)
Table D.1: Regression output for predicting the out-of-accuracy (R² or AUC) based on the implementation of the impute- then-regress method. We include dataset and kmisssing fixed effects. We report regression coefficient values (and clustered standard errors).
Design matrix
X
Signal Y
Missingness
VI V2 coeff.
V3 vs. V2 coeff.
Adjusted g
MCAR
0.0049 (0.0017)**
-0.0002 (0.0002)
0.3483
Syn. Lincar
Consoring
0.0047 (0.00144)***
-0.0013 (0. (0.0004)***
0.3484
Syn.
MCAR
0.0038 (0.0010)**
0.0 (0.0002)
0.4289
Syn. NN
Censoring
0.0052 (0.0023)
-0.0024 (0.0002)
0.4920
MAR
0.0076 (0.0066)
-0.0067 (0.0024) **
0,2325
Syn. Linear
NMAR
0.0020 (0.0030)
-0.0076 (0.0027) **
0.3982
AM
-0.0021 (0.0027)
-0.0017 (0.00255***
0.3468
Real
MAR
0.0067 (0.0060)
-0.0028 (0.0012)*
0,2805
Syn. NN
NMAR
0.0030 (0.0036)
-0.0037 (0.0017)*
0,3146
AM
0.0008 (0.0020)
-0.0005 (0.0015)
0.3362
Real
Real
Real
0.0104 (0.0070)
-0.0016 (0.0020)
0.8496
Controls:
Datasct, Kmiissng (if available); p-value: ":< 0.1, *":< 0.01, *** 0.001
E Additional Numerical Results
In this section, we report complementary numerical results to Sections 3
In Table we evaluate the benefit. of mode imputation (vs. encoding missingness as a new category) for missing categorical variables on a collection of synthetic, semi-real, and real datasets. For the 34 real datasets, we consider 10 different training/ test splits and compute the difference in accuracy between the two imputation strategies and conduct a paired and Wilcoxon test across all datasets to assess which method is more accurate (on average and more often). Aggregating differences in accuracy across all datasets might be fallacious because they cannot be considered as identically distributed. Actually, both tests were inconclusive on these datasets. Alternatively, we can do this analysis at a dataset level. For each dataset, we can compute the average difference in means across the 10 training/ /test splits and compute the associated t-statistics (or oquivalently the associated z-scorc). Figure E.1 reports the distribution of the difference in mcan accuracy (left panel) and of the z-scores across the datasets. First, we observe that despite our best effort to aggregate accuracy metrics that are on the same scale (2 AUC for classification and R² for regression tasks), the difference in means are more concentrated for regression problems than for classification ones. Second, these distributions confirm the absence of clear conclusion in these settings with almost many datasets where mode imputation is clearly beneficial (difference in means strongly positive, z-score close to 0) as those where it is detrimental (difference in means strongly negative, z-score close to 1). On this regard, we should note that the mode imputation seems more detrimental for regression tasks than classification ones, although the limited number of datasets involve prevents us from drawing a strong conclusion (see Table E.1
Table E.1: Difference in means and in pseudo-medians (with two-sided p-values) in out-of-sample accuracy from a t and Wilcoxon test applied to assess the impact of mode imputation on downstream accuracy, on real datasets only. A negative value means that mode impute reduces accuracy compared with encoding missingn a new category.
Task
# comp. mean (p-value) A pseudo-median (y-value)
Classification
250
0.0118 (0.210)
0.0000 (0.98)
Regression
90
-0.0049 (0.092)
-0.0002 (0.015)
Figure D2
reports the result of the same analysis for the comparison between mice and mcan impute.
21

---

Published in Transactions on Machine Learning Research (06/2024)
222
135
proh NJO -20009
242
125
-10
181
(a) Difference in means
(b) Z-scores
Figure E.1: Distribution of average difference in out-of-sample accuracy (left panel) and of the z-scores (right panel) across the 34 real datasets used to evaluate the effect of mode impute over encoding missingness as a new category. On the left panel, a negative value implies that mode imputation performs worse on average. On the right panel, a z-score close to (resp. 0) implies that mode imputation has detrimental (resp. beneficial) effect with strong confidence.
Prob: M0L1 1- -teett
Deta in reams
2.02
-28
Caeiitrats
Regieroon
Dascliation
Reginicss
(a) Difference in means
(b) Z-scores
Figure E.2: Distribution of average difference in out-of-sample accuracy (left panel) and of the z-scores (right panel) across the 36 real datasets used evaluate the effect of mice impute over mean impute. On the left panel, a negative value implies that mice performs worse on average. On the right panel, a z-score close (resp. 0) implics that mice has detrimental (resp. beneficial) effect with strong confidence.
complements Figure by displaying the difference in accuracy between mi mice-then-regres and as both the number of observations and the proportion of missing entries vary. Unlike the proportion of missing entries, we do not observe a clear pattern in how sample size affects the relative performance of each approach.
Figure F.3
22

---

Published in Transactions on Machine Learning Research (06/2024)
940
0.20
940
Number of observations
840
840
740
-0.10
740
640
0.05
540
540
440
-.05
440
Number
-0.1
240
240
0.2
0.15
140
140
-0.20
0.3
40
40
0.1
0.3
0.4
0.5
0.6
0.7
0.8
0.1
0.2
0.3
0.4
0.5
0.7
0.8
Proportion missing values
Proportion of missing values
(a) Y linear -CAA
(b) Y linear Censoring
940
940
Number of observations
840
-0.075
840
-0.550
640
-0.255
640
540
540
440
0.025
440
340
0.05C
340
-01
240
0.075
240
0.2
140
0.100
140
-03
0.4 Proportion of missing values
0.1
0.8
Proportion of missing values
(c) Y NN MCAR
(d) Y NN Censoring
Figure E.3: Difference in out-of-sample R² between mice- and mean-impptt--heen-reeee on fully synthetic data, as the proportion of missing entries and the sample size vary. green/ /positive value indicates that mean impute more accurate than mice.
Figure E.4 replicates Figure for random forest regressor instead. reports the out-of-sample accuracy achieved by random forest the number of observations increases, for three different treatment of missing data: no treatment, mean impute, and mice.
0.5
Out-ofsample R2
sor-ot-
0.4
0.4
0.3
0.3
impute Random mice impute then Forest Random Forest
mice impute Random Forest Forest
200
400
600
800
500
800
Number samples
Number samples n
(a) Y non-linear MCAR
(b) non-linear Censoring
Figure E.4: Average out-of-sample R² of XGBoost with mean impute, mice, or imputation method, on synthetic data with non-lineat signal, NMAR missing data, and 40% of missing entries, as the number of samples n increases. Results are averaged over 10 training/tess splits.
23


Reviews:
[{"review_id": "Asy7VYh2Ik", "reviewer": "Reviewer_o3iN", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths: The problem is important, the results seem interesting, and the experiments seem to confirm the suggested interpretation.\n\nWeaknesses: The paper is extremely confusing. The exact training method is ambiguous, and the explanations of the theoretical results are incomprehensible.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The authors study imputation of missing data. They first report a theoretical result which relates the outputs of an imputed-trained predictor to various expectations. An implication of this result is that the imputed-trained predictor will be consistent (expected output equal to the true expected value for a given input, IIUC) under certain conditions, which mostly relate to how well the predictor can detect missingness and imputation.. \n\nExperiments seem to confirm the implications of the theory. For discrete data, mode-imputation is shown to reduce accuracy in comparison to indicating missingness with a separate category. For continuous data, mean-inputation (which effectively indicates missingness under mild assumption) is competitive with a much more complex method, at least when much data is missing.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "- What, exactly is the proposed training method? In Theorem 2.1 we read \"systematically imputing\" x1. Does this apply to all x1, or only to the missing one? That is, do we replace the non-missing x1 with their imputed values in the training set or not?\n\n- Also in this theorem, the first equation seems to suggest that the method will produce the correct result when mu != x1, i.e. precisely when the imputation is incorrect?? Is that actually the case?\n\n- When the imputation is incorrect, the output will be a weighted mean of the value predicted while ignoring x1, and trhe value predicted while inputing x1? But isn't the latter exactly what we are doing? \n\n- Basically please just explain carefully what is going on there and what the terms actually mean.\n\n- In 3.3, to be clear, the reasoning would apply to any imputation method resulting in values outside the distribution of the true non-missing X1's? Even simply mu = +inf for any input would produce \"consistent\" results?", "broader_impact_concerns": "I do not see any broader impacts.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "jr9Pa0ojzO", "reviewer": "Reviewer_PEFs", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "## Strengths\n\n+ **Free from the MAR assumption.** The main technical contribution, Theorem 2.1, is shown without the MAR assumption, a common assumption in the field of missing data imputation yet extremely challenging to verify in practice. The optimal classifier is derived without this assumption. The proof is based on a simple calculus and is not hard to follow.\n+ *Large-scale experiments.\" The theoretical findings and discussions are tested with many real datasets. Though the performances are not as good as expected, the reason is carefully discussed.\n\n## Weaknesses\n\n+ **Unclear why $f\\_{\\\\mu-\\\\text{impute}}$ is needed.** In Eq. (1), it is shown that all we need is to regress the observed covariates $\\\\boldsymbol{x}\\_{2:d}$ to the outcome $y$ when $x\\_1$ is missing ($m\\_1=1$). This can be done without imputation by simply regressing $\\\\boldsymbol{x}\\_{2:d} \\\\mapsto y$, and hence, the necessity of the intermediate imputation is unclear if we are interested solely in the downstream regression task. Additionally, I'm not sure whether the Bayes predictor in Eq. (1) can be derived straightforwardly from the general one shown in Eq. (3) in Le Morvan et al. (2021).\n+ **Theory in Section 3.3.** The justification of the mean imputation hinges on the claim \"$X\\_1|\\\\boldsymbol{X}\\_{2:d} = \\\\boldsymbol{x}\\_{2:d}$, $M\\_1 = 0$ is continuous [...] then, conditioned on $\\\\boldsymbol{X}\\_{2:d} = \\\\boldsymbol{x}\\_{2:d}$ and $M\\_1 = 0$, the probability that $X\\_1$ takes any specific value is $0$ so $p\\_\\\\mu(\\\\boldsymbol{x}) = 0$ and $\\\\alpha(\\\\boldsymbol{x}) = 1$\" (in the first paragraph in Section 3.3). While this is correct in the measure-theoretic viewpoint, does it mean any imputation methods can attain $\\\\alpha(\\\\boldsymbol{x}) = 1$? It seems that any method can avoid this measure-zero event $X\\_1 = \\\\mu(\\\\boldsymbol{x}\\_{2:d})$. This point can be discussed slightly more.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper deals with the impute-then-regress framework, where missing data imputation is targeted to the downstream regression task. Unlike the pure missing data imputation task, the authors reveal that \"crude\" imputation works well in light of the downstream task. Specifically, Theorem 2.1 shows the form of the optimal predictor given by a universally consistent learning algorithm applied to imputed data, which is a convex combination of the imputed and non-imputed regressors. This optimal classifier becomes closer to the Bayes predictor in the form of Eq. (1) when it is easier to discern whether a given data is imputed. Eventually, the authors empirically tested this finding with the mode imputation for discrete data and mean imputation for continuous data, and the latter works well because the realization of imputed means would never precisely match the unobserved feature value (almost surely), i.e., detecting whether a given data is imputed or not is fairly easy.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "The above comments in Weaknesses are expected to be addressed.\n\nIn the experiments, the following points can be clarified/improved.\n\n+ In Table 1, it is not evident how 41 datasets used for \"Real\" are aggregated.\n+ In the experiments in Section 3.2, \"the out-of-sample accuracy [without] mode imputation\" should be clarified. Is it simply regression from $\\\\boldsymbol{x}\\_{2:d}$ to $y$?\n+ In the experiments in Table 2, why not compare the mean imputation with a non-imputed regressor instead of the baseline mice? This comparison looks more natural because we want to see whether the mean imputation improves the downstream regressor over the vanilla regressor.\n\nBelow are minor suggestions.\n\n+ In the third line on p.3, a space needs to be added at \"For ease of exposition,[ ]we consider a simplified setting ...\"\n+ In Theorem 2.1, the hyphen in $f\\_{\\\\mu-\\\\text{impute}}$ can be better typeset by \"f\\_{\\mu\\mathchar`-\\text{impute}}\".\n+ Theorem/section headers in citations should be capitalized. For example, \"Josse et al. (2019, [T]heorem 4)\" in the first line of the last paragraph on p.4.\n+ In 8th line in Section 3.1, \"ampute\" -> \"impute\"\n+ In 11th line in Section 3.1, \"UCI Machine Learning and RDatasets repositories\" -> \"UCI Machine Learning Repository\"\n+ In the 2nd line in Section 3.2, it should be better to have \"for any $\\\\boldsymbol{x}$\" for the statement \"$p\\_\\\\mu(\\\\boldsymbol{x}) > 0$ so $\\\\alpha(\\\\boldsymbol{x}) < 1$\".", "broader_impact_concerns": "The current work mainly focuses on the theory of missing data analysis, and no ethical implications would be applicable.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "CV8MFN1RCO", "reviewer": "Reviewer_Wsdy", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\n1. The motivation is clear.\n\n2. The paper is easy to follow. \n\n3. The insights are significant.\n\nWeaknesses:\n\n1. It seems like the authors missed the related works. I suggest the authors add this section to provide an overview of the development of this research topic (i.e., Imputation).\n\n2. For the experiment in Section 3.2, the authors designed the baselines (means and pseudo-medians). I am not very familiar with this domain. However, I suppose there are lots of established methods from existing papers on this topic. Why do the authors not compare with the existing baselines? \n\n3. The predictive models are traditional machine learning models. No neural networks (NNs) are considered. The authors only use NNs to generate synthetic data. I would like to know the performance of advanced predictive models such as NNs.\n\n4.  It is better to have a discussion about the limitations of this work.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper discussed that mode-impute is asymptotically sub-optimal, while mean-impute is asymptotically optimal. The authors use theoretical proofs to support this statement and empirical study to demonstrate its correctness.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Add Related Works. Add NNs as one type of predictive model. Add limitation discussion. Compare with the existing methods.", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper analyzed the impute-then-regress framework, where the missing data imputation is performed for the downstream regression task.  Validity of the theory is empirically assessed on artificial and real datasets.

The reviewers found significant contributions with criticisms on the clarity, which have been well addressed by the authors.
Decision: Accept (poster)