Feature Importance Metrics in the Presence of Missing Data
Authors: Henrik von Kleist
Joshua Wendland
Ilya Shpitser
Carsten Marr
Venue: ICML 2025 poster (2025)
Keywords: feature importance
missing data
leave-one-covariate-out
feature acquisition
full data
observed data
Primary area: general_machine_learning
TL;DR: Addressing feature importance in the presence of missing data by contrasting full and observed data perspectives and introducing a novel metric to guide feature acquisition decisions.

Abstract: Feature importance metrics are critical for interpreting machine learning models and understanding the relevance of individual features. However, real-world data often exhibit missingness, thereby complicating how feature importance should be evaluated.  We introduce the distinction between two evaluation frameworks under missing data: (1) feature importance under the full data, as if every feature had been fully measured, and (2) feature importance under the observed data, where missingness is governed by the current measurement policy. While the full data perspective offers insights into the data generating process, it often relies on unrealistic assumptions and cannot guide decisions when missingness persists at model deployment. Since neither framework directly informs improvements in data collection, we additionally introduce the feature measurement importance gradient (FMIG), a novel, model-agnostic metric that identifies features that should be measured more frequently to enhance predictive performance. Using synthetic data, we illustrate key differences between these metrics and the risks of conflating them.

Feature Importance Metrics in the Presence of Missing Data
Henrik von Kleist 123
Joshua Wendland 4
Ilya Shpitser Carsten Marr'
Abstract
feature in the prediction model, typically given a certain baseline of other features. These metrics help clarify the relevance of features in applications like medical decision- making, where they can provide insights into the diagnostic and prognostic value of medical tests (Yoo et al., 2025; Rai et al., 2024; Ma et al., 2023). In practice, however, particularly clinical contexts, features are often measured inconsistently. leading to missing values (Zamanian et al., 2024). This raises important questions about how define and interpret feature importance in such scenarios.
Feature importance metrics are critical for inter- preting machine learning models and understand- ing the relevance of individual features. However, real-world data often exhibit missingness, thereby complicating how feature importance should be evaluated. We introduce the distinction between two evaluation frameworks under missing data: (1) feature importance under the full data, as if every feature had been fully measured, and (2) feature importance under the observed data, where miss- ingness is governed by the current measurement policy. While the full data perspective offers in- sights into the data generating process, it often relies on unrealistic assumptions and cannot guide decisions when missingness persists at model de- ployment. Since neither framework directly in- forms improvements in data collection, we addi- tionally introduce the feature measurement impor- tance gradient (FMIG), a novel, model-agnostie metric that identifies features that should be mea- sured more frequently to enhance predictive per- formance. Using synthetic data, we illustrate key differences between these metrics and the risks of conflating them.
To ground our discussion, we present three simplified sce- narios in heart attack diagnosis, each illustrating a distinct perspective on how to reason about feature importance in the presence of missing data:
A biomedical researcher aims to under- stand the intrinsic relationship between troponin levels, a biomarker typically measured via blood test, and the oc- currence of heart attacks independent of hospital testing protocols. Using retrospective dataset with missing values, the researcher is interested in feature importance as if every feature had been fully measured.
Scenario 1:
Scenario 2:
A machine learning engineer develops a heart attack prediction model and evaluates the relevance of tro- ponin levels. Given a retrospective dataset with missingness, the model be trained for deployment in a setting where such missingness will persist due unchanged hospital testing protocols. The engineer is thus interested in fea- ture importance under the current missingness where features sometimes assume a special value "?" representing missingness.
1. Introduction
Feature importance metrics play a crucial role in supervised machine learning, offering a means to evaluate the impact of individual features on model's performance. A subset of these metries, known as global feature importance metrics (Ewald et al. 2024), determine the value of including
Scenario 3:
Since troponin tests are costly and time- consuming, but provide valuable diagnostic information, an analyst seeks to optimize their use after deploying the prediction model. The analyst wants to understand how prediction performance would change troponin was mea- sured more frequently. They are thus interested in feature importance under an increase in measurement probabil- ity.
'Institute of AI for Health, Helmholtz Munich German Re- scarch Center for Envirommental Health, Neuherberg, Germany 2TUM School Computation, Information and Technology, Tech- nical University of Munich, Garching, Germany Department of Computer Science, Johns Hopkins University, Baltimore, MD, USA Faculty of Computer Science, Ruhr University Bochum Bochum, Germany. Correspondence to: Henrik von Kleist <hen- dee.i.t@tum.e>.
In this work, we examine how feature importance metrics can be defined and estimated in such scenarios. To clar- ify the structure of the problem, we organize our analysis
Proceedings of the 12nd International Conference on Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

Feature Importance Metries in the Presence of Missing Data
around the classical three-step framework of statistical in- ference: (i) Estimand definition, which entails specifying the target parameter to be computed; (ii) Identification, which assesses whether the estimand can be expressed as a function of the observed data; and (iii) Estimation, which concerns how the estimand is calculated from the observed data.
ture contains no excessive information, leading to distorted feature importance estimates. Consequently, reported metrics may misrepresent the true underlying importance of features.
We find that current practices for reporting feature impor- tance metrics in scenarios with missing data lack rigor across all three steps:
To address the lack of estimand specification, we introduce the distinction between two concepts: (1) Full data feature importance metrics, which refer to the ground truth data had there been no missingness, and (2) Observed data fea- ture importance metries, which refer to the observed data and thus analyze feature importance in the context of the current measurement policy. We illustrate the implications of these two perspectives using a simple yet practical fea- ture importance metric: the leave-one-covariat-oou (LOCCO metric (Lei et al., 2018).
Estimands are not clearly specified: When transition- ing from fully observed dataset to one with missing data, the target parameter - i.e. the feature importance metric must be redefined to reflect the data structure. In current practice, analysts do not report based on which data the feature importance is calculated. For ex- ample, is the metric intended to reflect the importance of features in the observed data with missingness, or in the full data that would have been available had there been no missingness?
We then turn the problem of identification. While ob- served data feature importance metrics are directly identifi- able from the observed data distribution, the identification of full data feature importance metrics relies on strong and often unrealistic assumptions. In particular, we examine the role of the positivity assumption, which can critically limit the reliability of estimation in practice.
Identification assumptions are not stated, and identi- fication not performed: Analysts often fail to spec- ify the necessary assumptions required for identifica- tion. This includes independence assumptions about the missingness process, such as whether data are as- sumed to be Missing Completely at Random (MCAR), Missing at Random (MAR), or Missing Not at Random (MNAR) (Bhattacharya et al., 2020; Nabi et al., 2020; Mohan & Pearl, 2021). It also includes the positiy- ity assumption (Petersen et al., 2012), which ensures sufficient overlap in feature availability across subpop- ulations. As result, it remains unclear under which as- sumptions the reported feature importance metrics hold true. Examples of this issue can be found in Yoo et al. (2025); Rai et al. (2024); Ma et al. (2023); Beld et al. (2024); Peng et al. (2023); Guan et al. (2023); Zhou et al. (2023): Qi et al. (2022); Xie et al. (2023): Bao et al. (2023); Yang et al. (2022); Zhang et al. (2023).
Next, we address the problem of estimation by reviewing commonly used approaches for computing feature impor- tance metrics under missing data. We classify these methods according to the estimands they target and assess their abil- ity to produce unbiased results under various missingness mechanisms. We demonstrate that estimation results can differ drastically between full data and observed data fea- ture importance metries. For instance, consider a costly or invasive medical test that provides a highly informative feature: if performed for every patient, the feature would be an excellent predictor, yielding high full data feature importance. However, due to cost constraints, it is rarely conducted in practice - leading to a much lower observed data feature importance.
Estimation methods are often biased: A variety of methods are used in practice to handle missing data when computing feature importance metrics, includ- ing but not limited to mean/median/zeroo imputation (Zhuang et al., 2023; Lucini et al., 2023; Ishii et al., 2023; Huang et al., 2023; Danilatou et al., 2022). condi- tional mean imputation (Vo et al., 2024), complete case analysis (Zhang et al., 2022), and multiple imputation (Peng et al., 2023; Guan et al., 2023; Qi et al., 2022; Xie et al., 2023; Chen et al., 2023; Zeng et al., 2023). It is often unclear to which estimand these methods corre- spond to and whether they introduce bias. For instance, some methods implicitly assume that the missing fea-
While full data and observed data feature importance met- rics are appropriate for the questions posed in Scenarios I and 2, they fall short in addressing Scenario 3, where the goal is to assess a features importance under changes in the measurement process. To fill this gap, we introduce the feature measurement importance gradient (FMIG), a novel, model-agnostic, and computationally efficient esti- mand. FMIG quantifies the sensitivity of prediction perfor- mance to small increases in the measurement probability of given feature. Unlike full data feature importance metrics, FMIG does not require a positivity assumption, making it more robust and practically applicable in real-world settings with high rates of missingness.
2

---

Feature Importance Metries in the Presence of Missing Data
2. Related Methods
trinsically handle missing data (Vo al., 2024). Among imputation-based methods, conditional mean imputation and similar approaches like KNN-imputation (Smit et al., 2022) or SVD imputation are commonly used (Srinivasan et al., 2016; Hapfelmeier, 2012; Ergul Aydin & Kamisli Oz- turk, 2024; Smit et al., 2022), but introduce bias (Seijo- Pardo et al., 2018).
In Appendix A, we provide an overview of feature im- portance metrics. In the following, we discuss related ap- proaches for assessing feature importance and the closely related problem of feature selection in the presence of miss- ing data.
2.1. Estimand Specification
In contrast, multiple imputation (MI) is recognized as an unbiased method for estimating feature importance under missing data (Moons et al., 2006). However, prior work has emphasized that the label must be included in the imputa- tion model to ensure validity (Moons et al., 2006). Despite this recommendation, it is still common in practice to ex- clude the label (Srinivasan et al., 2016; Hapfelmeier, 2012; Ergul Aydin & Kamisli Ozturk, 2024; Zeng et al., 2023).
To the best of our knowledge, prior studies have explicitly differentiated between observed and full data feature impor- tance metries. However, some papers incidentally report both types of metrics because they use multiple methods- some of which align with observed data feature importance metries and others with full data feature importance with- out noticing the difference in the underlying estimands (Vo et al., 2024; Srinivasan et al., 2016; Rai et al., 2024). The same issue arises in feature selection (Fan et al., 2023; Mera- Gaona et al., 2021; Ergul Aydin & Kamisli Ozturk, 2024). Notably, only Hapfelmeier (2012) acknowledged that mul- tiple imputation addresses a different question than their proposed method, but the difference in estimands remained unexplored. In contrast, our work is the first to formally define these concepts and clarify their implications.
While these methods have been analyzed in various studies, no work has systematically categorized them based on the type of feature importance metric they compute- whether observed data or full data feature importance. In addition to this categorization. we identify which estimation meth- ods introduce bias, further clarifying their limitations and appropriate applications.
2.4. Action Importance Metries in Reinforcement Learning and Causal Inference
2.2. Identification
Challenges related estimating feature importance or per- forming feature selection under different missingness mech-
The feature measurement importance gradient (FMIG), in- troduced in this work, quantifies how predictive perfor- mance would change if the probability of measuring a given feature were slightly increased. This metric frames the mea- surement process as a decision-making problem, where the decision is whether to measure a feature in a specific context defined by the already-measured features. This perspective naturally aligns with concepts in reinforcement learning (RL.) and causal inference, which study the importance of actions and their effects.
anisms, such as MCAR, MAR, and MNAR, have been dis- cussed in some studies (Vo et al., 2024; Hapfelmeier, 2012;
Srinivasan et al., 2016; Fan et al., 2023; Beld et al., 2024; Zhao & Long, 2017), but MNAR remains particularly un- derexplored (Zhao & Long, 2017). Many papers fail to specify the missingness assumptions under which their re- ported metrics are valid (Yoo al., 2025; Rai et al., 2024; Ma et al., 2023; Beld et al., 2024; Peng et al., 2023; Guan et al., 2023; Zhou et al., 2023; Qi et al., 2022; Xie et al., 2023; Bao et al., 2023; Yang et al., 2022; Zhang et al., 2023). Moreover, no prior work has, to the best of our knowledge, addressed the role of the positivity assumption in the context of feature importance metrics under missing data.
Shapley values, which are widely used in feature impor- tance evaluation (Lundberg & Lee, 2017), have been also applied in RL to assess the importance of individual actions (Beechey et al., 2023). Concepts like the credit assignment problem in RL (Pignatelli et al., 2024), which focus on attributing rewards specific actions, highlight related chal- lenges in quantifying the importance of actions and suggest potential connections to feature measurement tasks.
2.3. Estimation
Various studies have investigated methods for estimating feature importance metries (Vo et al., 2024; Srinivasan et al., 2016) or conducting feature selection (Gunn et al., 2023; Seijo-Pardo et al., 2018; 2019; Mera-Gaona et al., 2021; Hu et al., 2021; Genossar et al., 2024; Ergul Aydin & Kamisli Ozturk, 2024; Doquire & Verleysen, 2012) in the presence of missing data. These approaches include imputation- based methods (Zhao& & Long, 2017; Vo et al., 2024; Seijo-Pardo et al., 2018; 2019) and methods that in-
FMIG also shares a conceptual link with incremental propen- sity score interventions introduced by Kennedy (2019), which provide an interpretable framework for analyzing the effects of small changes in action probabilities (propensity scores). We adapt this idea to the setting of feature mea- surement and, by focusing only on the gradient, simplify estimation while retaining an interpretable metric.
3

---

Feature Importance Metries in the Presence of Missing Data
3. Full Data and Observed Data Feature Importance Metrics
3.2. Estimand Definition: Full Data and Observed Data
LOCO Metries
In this section, we introduce notation, define the full data and observed data LOCO metries, and discuss identification and estimation. We consider a classification setting, but our results naturally extend to regression.
For full data, the LOCO metric for feature j evaluates as follows.
Full data LOCO:
LOCO =B
3.1. Notation
[D]]
We define the key variables and functions as follows:
Here, the full data LOCO metric measures the expected loss difference if feature is excluded compared to if is included, with all other features fully observed.
Observed features: X (RU U ("?")){, where X, rep- resents the value of the i-th feature. If X; = "?", the feature missing: otherwise, X; E R.
Missingness indicators: R E {0, 1j², where R, in- dicates that feature X; is observed, and R; 0 indicates it is missing.
In contrast, the observed data LOCO metric evaluates as follows.
Observed data LOCO:
Ground truth features: (1) E IRd representing the true values of all features, had they all been observed. Using potential outcome notation (Rubin, 2005), X(). denotes the value of feature X,, potentially contrary to fact, had R 1. We obtain:
It differs from LOCO in that replaces the ground truth features X()) with their observed proxies X and the missing- ness indicators R, which embeds the feature importance in the context of the measurement policy that was used in the acquisition of the dataset. The observed data LOCO metric LOCO thus quantifies the loss difference when feature is excluded compared to if it is included, but only when measured under the current measurement policy, with other features partially observed under their respective measure- ment policies.
((Da "?"
if 1, if R 0.
X;
Label: {0, K, 1}, the categorical outcome associated with the feature Xet where is the number of possible classes. We do not consider label missing- ness in this work.
Missingness process: :: Rd x
[0, 1]"", representing the mechanism that determines the missingness indicator R. In the most general case <(R[X(1>> Y) depends on both the ground truth features (()) and the label Y.
Remark 3.1. Note that the observed data LOCO metric, LOCO does not capture the contrast in loss that would occur if feature j were longer measured. We intro- duce alternative estimand, termed saav-oone-ovvaiate unmeasured (LOCU), which describes this scenario in detail in Appendix B. The LOCU metric coincides with OOCO only under certain restrictive independence assumptions about the missingness process.
Classifier: fa (IR U ["?"])" x [0, - [0, 1]", de- fines a mapping from the observed features and miss- ingness indicators to the label probabilities. While X already encodes missingness through the presence of "?" values, we explicitly include the missingness indica- tors R in the notation to emphasize the dependence the missingness pattern. Using missingness indicators in the classifier is common practice in many settings (Van Ness et al., 2023; Singh et al., 2021).
3.3. Identification
The observed data LOCO metric is already identified, mean- ing is expressed as a function of the observed data. This is not the case for the full data LOCO metric, which depends the unobserved ground truth features X(1)- To identify the full data LOCO metric, missing data identification methods can be applied. Generally, the following assumptions required:
Loss function: L: {0, 1,. ......,,- K- -
[0, IR de- fines a loss function, for example cross entropy, for the classifier fa based on the true labels and the predicted label probabilities.
We further let X-j. R-j. X())-j denote the reduced dimen- sion observed features, missingness indicators and ground truth features when feature is excluded. Similarly, we let fel.y: (R d" x {0, 1 + [0, 1] denote the classifier that excludes information from feature j.
Stable Unit Treatment Value Assumption (SUTVA): This assumption ensures that the counterfactual feature values are well-defined, as described by Eq. 1. Specifically, requires that the observed features are equal the counterfactual

---

Feature Importance Metries in the Presence of Missing Data
features when R 1. Additionally, it assumes that data points do not interfere with one another- specifically, that missingness in one instance is not influenced by the features or missingness indicators of other instances.
distribution is expressed as:
p(R - (1) Y) P(X TT 1
This formulation is valid if the propensity score Λ( i [()), Y) = p(R = I X()) Y) is identifiable, as p(R = ī, (1) Y) p(R = ī,x, Y) is already function of the observed data. When the missingness mechanism is MCAR, this estimator simplifies to complete case analysis.
Independence assumptions about the missingness process: These assumptions describe the relationship between the missingness process and the data:
Missing Completely at Random (MCAR): The miss- ingness process is independent of both observed and unobserved data, (()] Y) = π(R).
Multiple Imputation (MI): The MI estimator reconstructs ((X Y) by decomposing X(1) into its observed compo- nents X, and missing components X,. as follows:
Missing at Random (MAR): The missingness process depends only on the observed data.
Missing Not Random (MNAR): The missingness process may depend on unobserved data, such as the ground truth features (())-
p(X Y)= x)=Σ>)xxxx) E Xx,, ,,R))
Here, p(x o. Y,R) corresponds to the observed data, while p(x,, Xx,, Y,R) must be identified and estimated.
Positivity assumption: The positivity assumption requires that rt(R- i V) 0 for all ((), Y with (1) Y) > 0. In essence, it ensures that for every ground truth data point, there is positive probability that it is fully observed. The positivity assumption is particularly critical in many real-world settings, as it is often violated. Consider, for example, a time-series medical prediction setting. The as- sumption requires that for every patient, there is chance that all medical tests are being performed jointly and at every time point- scenario that is rarely, if ever, met in practice. As a result, full data feature importance metric estimates become highly unreliable in such contexts.
To ensure unbiased estimation, the imputation model must thus include the label Y an input. This was also verified in experiments (Moons et al., 2006). However, this introduces practical concern: if the imputation model is misspecified, the imputed values may contain excessive information about Y, potentially leading to inflated prediction performance estimates.
To mitigate this risk, practitioners often exclude Y from the imputation model (Srinivasan et al., 2016; Hapfelmeier, 2012; Ergul Aydin & Kamisli Ozturk, 2024). Imputing X, using p(Xm R) assumes X,, TL Y I,,,RR, implying that missing features contribute no additional information for predicting Y beyond X, and R. Such a practice effectively dismisses the potential importance of missing features be- fore the computation even begins, rendering the approach fundamentally flawed.
3.4. Estimation
In the following, we categorize various estimation methods for both full data and observed data feature importance metrics. Note that feature importance is defined relative to a given classifier fai and thus reflects the true importance of a feature only to the extent that the classifier accurately captures the relationship between X and Y.
3.4.2. Estimation of the Observed Data LOCO Metric
Estimating observed data feature importance metrics is more straightforward because the observed data LOCO metric is already identified. The primary requirement is ensuring that the classifier can appropriately handle missing values. Crucially, when assessing observed data feature importance metrics, additional information should be introduced for the missing feature. This can be achieved either by applying the classifier directly (if it supports "?" values) or by using simple imputation methods (c.g., mean or median imputa- tion) that do not inject additional predictive information. In the case of imputation, it is particularly important to include the unchanged missingness indicators in the classifier's input to preserve the informativeness of the missingness
3.4.1. Estimation of the Full Data LOCO Metric
Inverse probability weighting (Seaman& & White, 2013) and multiple imputation (Sterne et al., 2009) are widely used methods from the missing data literature for estimating full data target parameters. They can thus also be used to com- pute the full data LOCO metric. Both approaches recon- struct samples from the ground truth distribution p(X(,). Y), enabling the estimation of full data LOCO metrics.
Inverse-Probability Weighting (IPW): The IPW estimator reconstructs samples from the complete cases by applying appropriate weights. Using Bayes' rule, the ground-truth
X2 differs from because X contains placeholders (e.g., "?" for missing values) that encode information about
5

---

Feature Importance Metries in the Presence of Missing Data
pattern. While mean imputation is often considered biased approach (Mer--Gaon al., 2021), this is true only when full data feature importance method is of interest. For observed data feature importance, mean imputation remains unbiased. A more rigorous mathematical justification for this result is provided in Appendix C.
done correctly, LOCOED is evaluated on dataset that mir- rors the original dataset as if no missingness had occurred. In this case, the classifier can leverage the imputed values to potentially improve performance. For LOCOOD, evaluation occurs under conditions where missingness is present. Since the observed data already reflects the classifier's working conditions, no imputation is required. However, one may opt for impute-then-regres. classifiers, where imputation serves purely means to handle missing values elegantly. In such cases, the imputation step does not introduce ad- ditional information beyond what is already present in the observed input features.
Other popular approaches to estimate feature importance metrics can handle missing data intrinsically. These include for example tree-based methods such XGBoost (Vo et al., 2024; Chen & Guestrin, 2016) and are often based on principle known as missingness incorporated in attributes (MIA) (Twala et al., 2008). These methods merely handle the missing values within the classification and do not imag- ine counterfactual, "what everything had been observed", scenarios. They are therefore also included in the observed data feature importance category.
Remark 3.2. While the LOCO metrics in Eqs. 2 and 3 are valid for any classifier fai the choice of training method is essential for interpretability. Classifiers must be trained for the specific conditions under which they will be deployed. To evaluate observed data feature importance metries, the classifier must perform optimally in the presence of miss- ing data. In contrast, full data feature importance metries require a classifier trained for settings where all features are available. The implications of this distinction are discussed in Appendix D.
3.4.3. Biased Estimation Methods
A common but highly problematic approach to computing feature importance metrics is conditional mean imputation. This method replaces missing values with
X[X,, m Xx, Y,R],
or, more commonly (when Y is excluded), with
4. Feature Measurement Importance Gradient
E[[mm m X R].
The observed data and full data LOCO metrics assess the im- pact of completely omitting X, or X from the prediction model, providing an "all-or-nothing" perspective on feature importance. However, in practice, improving predictive per- formance often requires more -ne-grained considerations. Rather than asking whether a feature is important in abso- lute terms, a more actionable question is: which feature, if measured more frequently, would yield the greatest im- provement in prediction performance? For instance, if we could increase the measurement frequency of certain fea- tures, how should we prioritize them to maximize predictive gains? Answering this question provides practical guidance for optimizing data collection strategies.
While this simplification may appear reasonable, it intro- duces substantial bias when the goal is to estimate either full data or observed data feature importance metrics. We provide a detailed mathematical justification for a range of feature importance metrics in Appendix C and offer some intuition below.
Including Y may create deterministic relationship between y and the imputed values, potentially allowing a classifier to exploit this relationship and achieve artificially high pre- diction accuracy. Excluding Y, on the other hand, can still introduce a deterministic relationship between Xm and X,, thereby transferring predictive information from X, to the imputed values. This severely skews feature importance estimates. This method is thus neither suited to estimate full data, nor observed data feature importance metrics. Note that the same holds for all similar imputation methods that do not model the full density for the imputed features. This includes for example KNN-imputation (Zhang, 2012) which also frequently used to report feature importance under missing data (Zhou et al., 2023).
To address this, we introduce the feature measurement im- portance gradient (FMIG), novel metric that quantifies how prediction performance changes with small increases in the measurement probability of a given feature. We for- malize FMIG by defining perturbation interventions on the measurement policy and discuss its identification and esti- mation.
Specifically,, FMIG captures the sensitivity of predictive performance to marginal changes in measurement proba- bility. To this end, we introduce a perturbation ö, to the current measurement policy π,, increasing the probability of measuring feature j. We denote the perturbed policy as TT,
In conclusion, when using imputation for the estimation of "D and OD the missing values must be imputed in manner consistent with the classifiers respective "work- ing conditions." " For L the classifier operates under full data availability. Therefore, missing values must be im- puted using samples from the full data distribution. When
Given this perturbation, we can determine the gradient of the negative loss with respect to the perturbation to rank
6

---

Feature Importance Metries in the Presence of Missing Data
features in their importance:
4.3. Identification
Using the perturbation interventions, the feature measure- ment importance gradient Gj can be identified as follows: Lemma 4.1. (Identification of the Feature Measurement Importance Gradient (FMIG))
Feature Measurement Importance Gradient (FMIG)
Gj:
G-= Vo (=
(4)
where 87 is the value of 5, that represents no perturbation. Here, ((s,)) and ((x,,) denote the counterfactual observed features and missingness indicators, had the perturbed policy Te, been applied. We take the gradient with respect to the negative loss so that the FMIG quantifies the expected reduction in loss when the feature is measured slightly more frequently and thus leads to positive feature importances.
Under SUTVA and the assumption about the missingness process from Eq. 5, the feature measurement importance gradient (FMIG) for feature is identified as:
G = V 8, -L ( (Y.fa(X(ss)) R,(ss)))
-LCY,,_aax, -
= E
(1- (R
The proof is provided in Appendix E. It demonstrates why, unlike full data feature importance metrics, the FMIG does not require a positivity assumption due to the chosen per- turbation. This allows for far more reliable estimation in real-world settings with high fractions of missing values.
4.1. Measurement Process Assumptions
The definition of interventions on the measurement policy relies on the data's measurement process. Here, we consider special MAR process. Features are measured sequentially, where the measurement of feature depends on all prior missingness indicators and observed features.
4.4. Estimation
Under these assumptions, the measurement policy factorizes as:
The functional Gy. as identified in Lemma 4.1, is straightfor- ward to estimate. requires only the measurement policy of the feature of interest to be learned. An estimator is given by:
[() Y) (R RR 1
(5)
= -
where , = ,,... R , (X, 1x...... x and we
let RD= O1 and X, = 0
where f8, is the empirical mean over dataset of size n, and f, is model learned to approximate TT.. When using flexi- ble machine learning models for fty, these should be trained on a separate data split. To restore full data efficiency, a cross-fitting scheme can be employed (Chernozhukov et al., 2018).
4.2. Measurement Policy Interventions
To define perturbation interventions, leverage the factor- ization from Eq. 5:
TB. (R)X Y)= =
We extend the theory of the feature measurement importance gradient to time-series settings in Appendix F. In this setting, each feature is assumed to be measured at fixed time points, with predictions made for time-series label at each time point.
= 7,,6 (R (R
π [=11/]]
n.
Here, the intervention modifies the measurement policy only for feature j. We adapt the perturbation from incremental propensity score interventions (Kennedy, 2019), defined as:
5. Experiments
6, m
1, 6, n. +1 n
for 0 < <6,- 00,
Missing data estimation methods cannot be tested on real- world data with real missingness due to the unavailability of ground truth features X(1)- To address this limitation, we perform a series of synthetic experiments to illustrate the differences between feature importance metrics, the impact of positivity violations, and the significance of appropriate estimation methods.
where n, = v. (R, 11
The parameter 5, is the
odds ratio of the intervention:
6, = TT,,, (1-,,//1111,,))
odds, TT,/(1 TT,) odds,, (R = 1 1 1
odds R (R, 1|B, 1-1)
Values of 6, < 1 correspond to a decrease in the probability of measuring feature j. while 5, > 1 increases the probabil- ity. When 6, = 57 = 1, the original measurement policy is recovered: -T,8==1 = T...
5.1. Experiment Design
We consider a time-series prediction task where features x E (??")) (t E (1,2,3) are used to predict labels
7

---

Feature Importance Metries in the Presence of Missing Data
A)
Experiment 1: MCAR
B)
Experiment 2: MNAR
Feature Importance
0.5
LOCO LOCO
0.4
0.3
0.2
1
X2
X3
X1
X2
X3
Features
Features
Figure 1. Comparison of feature importance metrics (ground truth estimates). Experiment I: The full data LOCO metric increases from X, to x3, reflecting the features' growing predictive importance. However, the observed data LOCO remains flat because the increasing predictive value offset by higher missingness rates from X, to X3. B) Experiment Due MNAR missingness (X(1). R2), the observed data LOCO of X3 is further reduced because the missingness mechanism induces correlations between ((),a and X, X₂ via Ri R2. Confidence intervals were computed using the non-parametris bootstrap with samples.
yf at each time step. We use an "impute-then-regres, clas- sifier with zero imputation and a temporal convolutional network (TCN) (Bai al., 2018). Missingness mechanisms (MCAR, MNAR) are introduced at varying missingness rates across experiments. Full experimental details pro- vided in Appendix G. Additionally, Appendix H presents experiment highlighting the importance of training classi- fiers appropriately for the metric under evaluation.
decreases as its information is "leaked" into correlated vari- ables (R,, R2). The experiment highlights how informative missingness redistributes feature importance in observed data.
Experiment 3: Positivity violations introduce bias in estimation
Design:
Using the same dataset as Experiment 1, we sim-
ulate stronger MCAR missingness: p(R) = 1) = 0.4, p(R₁ = 1) 0.2, pRR = 1) 0.1. This results a near-complete absence of fully observed cases, effectively violating the positivity assumption.
5.2. Synthetic Experiments and Results
Experiment 1: Rarely measured variables have reduced observed data feature importance
Results: Figure 2 illustrates biases in estimating compo- nents of the full data LOCO metric. The complete case analysis, which generally unbiased under MCAR, pro- duces unbiased estimates in Figure 2A) for Experiment 1. However, in Experiment 3, where the positivity assump- tion is violated, the complete case estimator exhibits strong bias (Figure 2B). These results highlight the limitations of standard estimation methods for the full data LOCO metric when positivity is violated.
Design: We evaluate three variables with increasing predic- tive importance but decreasing measurement probabilities under an MCAR scenario: p(R 1) 0.75, pRR] = 0.5, p(R) = =] 0.3.
While the full data LOCO reflects the increas- ing predictive importance from X, to X3. the observed data LOCO remains flat (Figure 1A). This occurs because the gains in predictive performance from more informative fea- tures are counteracted by their higher missingness rates. The experiment illustrates how observed data feature importance metrics lead different conclusions than full data feature importance metrics.
Results:
Experiment 4: FMIG provides additional insights
Design: We evaluate three variables with equal full data feature importance, using an MCAR mechanism. We have p(R 1) = 1, P(R)
0.5 and constant feature X(1, with PR = 1)= = 1 and p(RC 1) 0.5 for 2 2.
Experiment 2: Informative missingness affects observed feature importance
Results: Figure 3 illustrates the FMIG metric is zero for X1 as is already fully observed. FMIG is positive for X2 which could, measured more often, improve predic- tion performance. Measuring the constant X multiple times provides no predictive improvement. Since the ad- ditional feature measurement is completely uninformative, the FMIG even becomes slightly negative. The insight that measuring X(1)3 more often will not improve prediction per-
Design:
Using the same dataset as Experiment 1, we sim- ulate MNAR missingness where X()x influences missing- of features at subsequent time points. Average miss- ingness probabilities remain equal to Experiment 1 with p(R = 1) & 0.75, p(R 1) & 0.5, PRR = 1) = 0.3.
Results: Figure IB) shows that the full data LOCO remains unchanged. However, the observed data LOCO of X(1),
8

---

Feature Importance Metries in the Presence of Missing Data
A)
Experiment Valid positivity assumption
B) Experiment Violated positivity assumption
20
Meen Imputation
Camplete Case Analysis
Ground Truth
Loss
1.2
0.0
All features
All features, except X₃
All festures, except X₂
All features, except X3
All features
All features, except X
All features, except X2
All features, except Xg
Components of LD
Components of LOCO
Figure 2. Effects of the positivity assumption violations the estimation of full data LOCO components. The plot shows estimates using (f'sff =1]] (denoted "All features") and B - -== (Y', ffa,,,(i,..-, R', 1))
different estimators for [ Σ E[-E: for each feature (denoted as "All features, except "All features, except X2" ", and "All features, except X3 ". A) Experiment 1: Under valid positivity assumption, the complete case estimator provides acceptable estimate of the ground truth. B) Experiment 3: Under violations of the positivity assumption, the complete cases analysis provides biased estimate of the ground truth. Note that the classificrs used evaluate feature importance metrics were trained the observed data from each experiment. estimates differ slightly, with lower loss in Experiment due to less missingness during classifier training. Confidence intervals were computed using the non- pa-aaaraettii bootstrap with 50 samples.
result, even the ground truth
Experiment 4: MCAR with constant X₃
their target estimands.
Feature Importance
Neither observed nor full data feature importance directly informs data collection. To address this, introduce the feature measurement importance gradient (FMIG), novel metric that identifies which features, if measured more fre- quently, would yield the greatest improvement in predictive performance. Synthetic experiments illustrate the distinct insights these metrics provide, emphasizing the need for tailored approaches in missing data settings. FMIG is thus not alternative to the observed data full data LOCO metrics, but additional tool to assess the importance of feature under change in measurement probability.
0.3
0.2
LD
LOCOfD
FMIG (-5.15)
X1
X2
X3
Features
Figure X(1, is constant and fully observed at 1, but less frequently at later times. The feature measurement importance gradient FMIG low observed data and full data feature importance.
While FMIG provides information on which features should measured more frequently. it does not preseribe an opti- mal measurement policy. Active feature acquisition (AFA) (von Kleist al., 2025; 2023), on the other hand, explic- itly balances measurement costs against predictive value derive optimal policies. FMIG thus complements AFA by identifying features worth prioritizing for optimization.
feature, even though it has high
formance is apparent from observed or full data LOCO metrics, but only from the FMIG.
6. Discussion & Conclusion
Our experimental evaluation is limited to synthetic data. Demonstrating the usefulness feature importance metrics in real-world settings particularly where prior knowledge or separately collected, fully observed datasets are available important direction for future work. Such demonstrations could highlight the practical relevance of these metrics and inform their application in domains with varying measure- ment constraints.
In this work, we examined feature importance in the pres- ence of missing data. Full data feature importance metrics, which assume no missingness, offer insights into the data- generating process but often difficult to estimate and impractical when missingness persists at deployment. To address this, we introduced observed data feature impor- tance metrics, which assess importance under a given mea- surement policy and highlighted key differences between these metries, categorizing estimation methods based on
9

---

Feature Importance Metries in the Presence of Missing Data
Acknowledgements
Frontiers in cardiovascular medicine, 10(101653388)
11 1119699, 2023.
Joshua Wendland was supported by the European Re- search Council (ERC) Starting Grant DEUCE, grant no. 101077178. Πya Shpitser acknowledges support from grants ONR N000142412701 and NSF CAREER 1942239. Carsten Marr acknowledges funding from the ERC under the European Unions Horizon 2020 research and innovation program (grant agreement no. 866411) and support from the Hightech Agenda Bayern.
Chernozhukov, V., Chetverikov, D., Demirer, M., Duflo, E., Hansen, C.,, Newey, W., and Robins, J. Double/debiased machine learning for treatment and structural parameters. The Econometrics Journal, 21(1):C1-C66 2018.
Danilatou, V., Nikolakakis, S., Antonakaki, D., Tzagkarakis, C., Mavroidis, D., Kostoulas, T., and loannidis, S. Out- come Prediction in Critically-III Patients with Venous Thromboembolism and/or Cancer Using Machine Learn- ing Algorithms: External Validation and Comparison with Scoring Systems. International journal of molecular sciences, 23(13), 2022.
Impact Statement
Our method enhances the explainability of AI models by providing a principled way to assess feature importance under missing data, thereby improving trust and reliabil- ityespecially in high-stakes applications such as medical decision- moeiii.n-making.. By clarifying biases in existing methods, our work contributes to more transparent, interpretable, and fair Al systems, ultimately fostering more informed and responsible deployment of machine learning models.
Doquire, G. and Verleysen, M. Feature selection with miss- ing data using mutual information estimators. Neurocom- puting, 90:3-11, 2012.
Ergul Aydin, Z. and Kamisli Ozturk, Z. Filter-based fea- ture selection methods in the presence of missing data for medical prediction models. Multimedia Tools and Applications, 83(8))24187-244 24216, 2024.
References
Ewald, K., Bothmann, L., Wright, M. N., Bischl, B., Casalicchio, G., and König, G. A Guide to Feature Importance Methods for Scientific Inference. arXiv:2404. 128622222, 2024.
Bai, S., Kolter, J. Z.,, and Koltun, V. An Empirical Evalua- tion of Generic Convolutional and Recurrent Networks for Sequence Modeling, arXiv:1803.012 [cs], 2018.
Bao, C.. Deng, F., and Zhao, S. Machine-learning mod- els for prediction of sepsis patients mortality. Medicina intensiva, 1556):3 2023.
Fan, M., Peng. X., Niu, X., Cui, T., and He, Q. Missing data imputation, prediction, and feature selection in diagnosis of vaginal prolapse. BMC Medical Research Methodol- ogy, 23(1):259, 2023.
Beechey, D., Smith, M. S., and imek, Explain- ing Reinforcement Learning with Shapley Values. arXiv:2306.055 055800 [cs]. 2023.
Genossar, B., On, T., Islam, M. M., Eliav, B., Roy, S. B., and Gal, A. MISFEAT: Feature Selection for Subgroups with Systematic Missing Data. arXiv:2412.06711 [cs], 2024.
Beld, J.-J. v. d., Pathak, S., Geerdink, J., Hegeman, H., and Seifert, C. Feature importance explain multimodal prediction models. A clinical use case. arXiv:2404. 18631 [cs]. 2024.
Guan, C., Ma. F,, Chang, S., and Zhang J. Interpretable machine learning models for predicting venous throm- boembolism in the intensive care unit: an analysis based on data from 207 centers. Critical care (London, Eng- land), 27(1):406, 2023.
Bhattacharya, R., Nabi, R., Shpitser, I., and Robins, J. M. Identification In Missing Data Models Represented By Directed Acyclic Graphs. In Proceedings of The 35th Uncertainty in Artificial Intelligence Conference, pp- 1149-1158. PMLR, 2020.
Gunn, H. J., Hayati Rezvan, P., Fernández, M. I., and Co- mulada, W. S. How to apply variable selection machine learning algorithms with multiply imputed data: A miss- ing discussion. Psychological Methods, 28(2):452 4711, 2023.
Chen, T. and Guestrin, C. XGBoost: A Scalable Tree Boost- ing System. In Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pp. 785-794, San Francisco California USA, 2016. ACM.
Hapfelmeier, A. Random Forest variable importance with missing data. 2012.
Hu, L., Joyce Lin, J.-Y., and Ji, J. Variable selection with missing data in both covariates and outcomes: Imputation and machine learning. Statistical Methods in Medical Research, 30(12)2661-2677, 2021.
Chen, Z., Li, T., Guo, S., Zeng, D., and Wang, K. Ma- chine learning-basee based io-hospital mortality risk prediction tool for intensive care unit patients with heart failure.
10

---

Feature Importance Metries in the Presence of Missing Data
Huang, C.-T., Wang, T.-J., Kuo, L.-K., Tsai, M.-J,, Cia, C.- T,, Chiang, D.-H., Chang, P.-.,, Chong, I.-W., Tsai, Y.-S., Chu, Y.-C., Liu, C.-J., Chen, C.-H., Pai, K.-C,, and Wu, C.-L. Federated machine learning for predicting acute kidney injury in critically ill patients: a multicenter study in Taiwan. Health information science and systems, 11 (1):48, 2023.
Moons, K. G. M., Donders, R. A. R. T., Stijnen, T., and Harrell, F. E. Using the outcome for imputation of miss- ing predictor values was preferred. Journal of Clinical Epidemiology. 19(10):1092-110: 2006.
Nabi, R., Bhattacharya, R., and Shpitser, I. Full law identifi- cation in graphical models of missing data: Completeness results. In International Conference on Machine Learn- ing, pp. 7153-7163. PMLR, 2020.
Ishii, B., Nawa, N., Hashimoto, S., Shigemitsu, H., and Fuji- wara, T. Development, validation, and feature extraction of a deep learning model predicting in-hospital mortality using Japan's 8 largest national ICU database: a validation framework for transparent clinical Artificial Intelligence (cAI) development. Anaesthesia, critical care & pain medicine, 42(2):101167, 2023.
Peng, C.,, Yang, F,, Li, I.., Peng. I.., Yu, J., Wang, P., and Jin, Z. A Machine Learning Approach for the Prediction of Severe Acute Kidney Injury Following Traumatic Brain Injury. Neurocritical care, 38(2):335-34 2023.
Petersen, M. L., Porter, K. E.,, Gruber, S., Wang. Y., and van der Laan, M. Diagnosing and responding viola- tions in the positivity assumption. Statistical methods in medical research, 21(1):31-54 2012.
Kennedy, E. H. Nonparametrie Causal Effects Based on Incremental Propensity Score Interventions. Journal of the American Statistical Association, 1 114(526)644--656, 2019.
Pignatelli, E.,, Ferret, J., Geist, M., Mesnard, T., Hasselt, H. v., Pietquin, O., and Toni, L. A Survey of Tempo- ral Credit Assignment in Deep Reinforcement Learning, 2024.
Le Morvan, M., Josse, J., Scornet, E., and Varoquaux, G. Whats a good imputation to predict with missing values? In Advances in Neural Information Processing Systems, volume 34, pp. 11530-11540, 2021.
Qi, J., Lei, J., Li, N., Huang, D., Liu, H., Zhou, K., Dai, Z., and Sun, C. Machine learning models to predict in- hospital mortality in septic patients with diabetes. Fron- tiers in endocrinology. 13(101555782) 1034251, 10322555,,,, 2022.
Lei, J., GSell, M., Rinaldo, A., Tibshirani, J. and Wasser- man, L. Distribution- Free Predictive Inference for Re- gression. Journal of the American Statistical Association, 113(523):1994-1111 2018
Rai, T., Shen, Y., He, J., Mahmud, M., Brown, D. J., Kaur, J., ODowd, E., Baldwin, D. R., and Hubbard, R. Under- standing Feature Importance of Prediction Models Based on Lung Cancer Primary Care Data. In 2024 Interna- tional Joint Conference on Neural Networks (LJCNN), pp. 1-8, 2024.
Lucini, F. R., Stelfox, H. T., and Lee, J. Deep Learning- Based Recurrent Delirium Prediction Critically III Pa- tients. Critical care medicine, 51(4):492-502, 2023.
Robins, J. A new approach to causal inference in mortality studies with a sustained exposure periodapplication to control of the healthy worker survivor effect. Mathemati- cal Modelling, (00)1393-1511 1986.
Lundberg, S. M. and Lee, S.-I. A Unified Approach to Interpreting Model Predictions. In Advances in Neural Information Processing Systems, volume 30, Curran As- sociates, Inc., 2017.
Rubin, D. B. Causal Inference Using Potential Outcomes:
Ma, I.,, Zhang, C., Gao, J., Jiao, X., Yu, Z., Ma, X., Wang, Y., Tang, W., Zhao, X., Ruan, W., and Wang, T. Mortality Prediction with Adaptive Feature Impor- tance Recalibration for Peritoneal Dialysis Patients: a deep-learning-bais based study on a real-world longitudinal follow-up dataset. arXiv:2301.077 007 [cs], 2023.
Design, Modeling, Decisions. Journal of the American Statistical Association, 100(469):322-331, 2005.
Seaman, S. R. and White, R. Review of inverse probabil- ity weighting for dealing with missing data. Statistical methods in medical research, 22(3):278 22577 2013.
Jeijo-Pardo, B., Alonso- Betanzos, A., Bennett, K., Bolon- Canedo, V., Guyon, I., Josse, J., and Saeed, M. Analysis of imputation bias for feature selection with missing data. Computational Intelligence, 2018
Mera-Gaona, M., Neumann, U., Vargas-Canas, R., and López, D. M. Evaluating the impact of multivariate im- putation by MICE in feature selection. PLOS ONE, 16 (7):00254720, 2021.
Seijo-Pardo, B., Allonso-Betanzal A., Bennett, K. P., Bolón- Canedo, V., Josse, J., Saeed, M., and Guyon, I. Biases in feature selection with missing data. Neurocomputing. 342:97-112, 2019.
Mohan, K. and Pearl, J. Graphical Models for Process- ing Missing Data. Journal of the American Statistical Association, 1 116(53)):1033-1033 2021.

---

Feature Importance Metries in the Presence of Missing Data
Singh, J., Sato, M., and Ohkuma, T. On missingness fea- tures in machine learning models for critical care: obser- vational study. JMIR Medical Informatics, 912):e2502, 2021.
Vo, T. L.., Nguyen, T., Hammer, H. L., Riegler, M. A., and Halvorsen, P. Explainability of Machine Learning Models under Missing Data. arXiv:2407.0000 240.:04400000 [cs], 2024.
Von Kleist, H., Zamanian, A., Shpitser, I., and Ahmidi, N. Evaluation of Active Feature Acquisition Methods for Static Feature Settings. arXiv:2312.03619 [cs, stat], 2023.
Smit, J. M., Krijthe, J. H., Endeman, H., Tintu, A. N., de Rijke, Y. B., Gommers, D. A. M. P.J., Cremer, O. L.,, Bosman, R. J., Rigter, S., Wils, E.-J., Frenzel, T., Don- gelmans, D. A., De Jong, R., Peters, M. A. A., Kamps, M. J. A., Ramnarain, D., Nowitzky, R., Nooteboom, F. G. C. A., De Ruijter, W. Urlings-Strop, I.. C., Smit, E. G. M., Mchagnoul-Schipper, D. Dormans, T., De Jager, C. P. C., Hendriks, S. H. A., Achterberg. S., Oostdijk, E., Reidinga, A. C., Festen-Spanjer, B., Brunnekreef, G. B., Cornet, A. D., Van den Tempel. W. Boelens, A. D., Koet- sier, P. Lens, J. A., Faber, H. J., Karakus, A., Entjes, R., De Jong, P. Rettig, T. C. D., Arbous, M. S., Lal- isang, R. C. A., Tonutti, M., De Bruin, D. P., Elbers, P. W. G., Van Bommel, J., and Reinders, M. J. T. Dy- namic prediction of mortality in COVID-19 patients in the intensive care unit: A retrospective multi-center co- hort study. Intelligence-bll based medicine, 6(101771737): 100071, 2022.
von Kleist, H., Zamanian, A., Shpitser, I., and Ahmidi, N. Evaluation of Active Feature Acquisition Methods for Time-varying Feature Settings. Journal of Machine Learning Research, 26(60):1-84, 2025.
Xie, W., Li, Y., Meng, X., and Zhao, M. Machine learning prediction models and nomogram to predict the risk of in-hospital death for severe DKA: A clinical study based MIMIC-IV, eICU databases, and college hospital ICU. International journal of medical informatics, 174 (ct4, 9711057): 2023.
Yang, B., Zhu, Y., Lu, X., and Shen, C. A Novel Composite Indicator of Predicting Mortality Risk for Heart Failure Patients With Diabetes Admiteed to Intensive Care Unit Based on Machine Learning. Frontiers in endocrinology, 13(101555722:9918838, 2022.
Srinivasan, K., Currim, F,, Ram, S., Lindberg, C., Sternberg, E., Skeath, P., Najafi, B., Razjouyan, J., Lee, H.-K., Foe- Parker, C., Goebel, N., Herzl, R., Mehl, M. R., Gilligan, B., Heerwagen, J., Kampschroer, K., and Canada, K. Fea- ture Importance and Predictive Modeling for Multi- Healthcare Data with Missing Values. In Proceedings of the 6th International Conference on Digital Health Conference, DH '16 pp. 47-54, New York, NY, USA, 2016. Association for Computing Machinery.
Yoo, S.-K., Fitzgerald, C. W. Cho, B. A., Fitzgerald, B. G., Han, C., Koh, E. S., Pandey, A., Sfreddo, H., Crowley, F., Korostin, M. R., Debnath, N., Leyfman, Y., Valero, C., Lee, M., Vos, J. L.,, Lee, A. S., Zhao K., Lam, S., Olumuyide, E., Kuo, F., Wilson, E. A., Hamon, P., Hen- nequin, C., Saffern, M., Vuong, L., Hakimi, A. A., Brown, B., Merad, M., Gnjatic, S., Bhardwaj, N., Galsky, M. D., Schadt, E. E., Samstein, R. M., Marron, T. U., Gönen, M., Morris, L. G. T., and Chowell, D. Prediction of check- point inhibitor immunotherapy efficacy for cancer using routine blood tests and clinical data. Nature Medicine, pp. 1-12, 2025.
Sterne, J. A. C., White, R., Carlin, J. B., Spratt, M.. Roys- ton, P., Kenward, M. G., Wood, A. M., and Carpenter, J. R. Multiple imputation for missing data in epidemio- logical and clinical research: potential and pitfalls. BMJ, 338:b2393, 2009,
Zamanian, A., von Kleist, H., Ciora, O.-A., Piperno, M., Lancho, G., and Ahmidi, N. Analysis of Missingness Scenarios for Observational Health Data. Journal of Personalized Medicine, 14(5):514, 2024.
Twala, B. E. T. H., Jones, M. C., and Hand, J. methods for coping with missing data in decision trees. Pattern Recognition Letters, (9(7):950-956 2008
J. Good
Van Ness, M., Bosschieter, T. M., Udell, M. The Missing Indicator Method: From Low to High Dimensions. In Proceedings of the 29th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, KDD '23, pp. 2023. Association for Computing Machinery.
R., and
Zeng, Z., Liu, Y., Yao, S., Liu, J., Xiao, B., Liu, C., and Gong, X. Neural networks based on attention architecture are robust to data missingness for early predicting hospital mortality intensive care unit patients. Digital health, 9 (101 (101690866)20552722111777777 177147772211181822 2023.
New York, NY, USA,
Zhang, L., Wang, Z., Zhou, Z., Li, S., Huang, T., Yin, H., and Lyu, J. Developing an ensemble machine learning model for early prediction of sepsis-associated acute kid- ney injury. iScience, 2509):104932, 2022.
Verdinelli, I. and Wasserman, L. Feature Impor- tance: A Closer Look at Shapley Values and LOCO. arXiv:2303. 05598/ [stat], 2023.
12

---

Feature Importance Metries in the Presence of Missing Data
Zhang, S. Nearest neighbor selection for iteratively ANN imputation. Journal of Systems and Software, 85(11): 2541-2552, 2012.
Zhang, Y., Hu, J., Hua, T., Zhang. J., Zhang, Z., and Yang, M. Development of machine learning- baase prediction model for sepsis-assoiaatd delirium in the intensive care unit. Scientific reports, 13(1):12697, 2023.
Thao, Y. and Long, Q. Variable selection in the presence of missing data: imputation- based methods, WIREs Compu- tational Statistics, ee 1402, 2017.
Zhou, H., Liu, L.., Zhao, Q., Jin, X., Peng. Z., Wang. W., Huang, L.., Xie, Y., Xu, H., Tao, L.., Xiao, X., Nie, W., Liu, Li, L., and Yuan, Q. Machine learning for the prediction of all-cause mortality in patients with sepsis- associated acute kidney injury during hospitalization. Frontiers in immunology, 14(101560660): 1021111440 140755, 2023.
Zhuang, J., Huang, H., Jiang. S., Liang, J., Liu, Y., and Yu, X. A generalizable and interpretable model for mortality risk stratification of sepsis patients in intensive unit. BMC medical informatics and decision making, 23(1):
185, 2023.
13

---

Feature Importance Metries in the Presence of Missing Data
A. Background on Feature Importance Metrics
Feature importance metrics can be categorized into local and global approaches (Ewald et al., 2024). Local metries focus on explaining individual predictions, while global metries provide insights into the aggregate importance of features in the prediction process. This work centers on global, model-agnostic methods that can be applied to any predictive model.
Feature importance metrics assign an importance value to each feature, reflecting its contribution to prediction performance. Beyond their role in explaining model behavior, feature importance metrics are often used to derive reliable, population- insights about the data-generating process. They are also closely linked to feature selection methods, as both seek to evaluate the relevance of features within a predictive model. In scenarios involving missing data, the challenges faced in assessing feature importance and selecting features often overlap, allowing similar methodologies to be applied across these tasks.
The field of feature importance metrics is dominated by three main approaches (Ewald al., 2024). Permutation feature importance evaluates the impact of randomly permuting a features values on model performance. It is model-agnostic and widely used for its simplicity but may underestimate feature importance when features are highly correlated. Marginalization based methods assess importance by marginalizing out the feature of interest, often requiring strong assumptions about the data-generating process to be valid. Finally, model refitting methods compare model performance before and after removing specific features. Techniques like LOCO (Lei et al., 2018) fall into this category, directly assessing the dependency of predictions on a given feature.
B. The Caaee-OOn--ovarrat---nneeee
(LOCU) Metric
In this appendix, we introduce alternative estimand to the observed data LOCO metric, LOCO Unlike LOCO which evaluates the impact of leaving one covariate out of the input to the classifier, this new metric considers the scenario where the measurement of a covariate is entirely omitted.
We denote this alternative metric as lenue-one-courriiee-mmmmsssse
(LOCU). (LCCU)
defined as follows:
LOCU:
LOCU=
where ((-j) denotes the counterfactual value would have taken R, = O. This represents a scenario where all features are measured according to the current measurement policy, except for feature j, which is no longer measured. LOCU thus quantifies the change in prediction loss if feature is omitted from measurement. This can provide valuable insights for data collection decisions. For instance, in medical settings, if particular test is costly, the metric can help assess the impact of discontinuing the test on prediction performance.
LOCU differs from the observed data LOCO metric 0D in cases where the measurement process of other features depends on the mcasurement of feature j (i.c., there are causal arrows from R2 to other missingness indicators). For example, suppose feature i is measured only if a test on feature j yields a positive result. In this situation, omitting the measurement of feature j would also prevent the measurement of feature potentially leading to a more substantial change in prediction performance.
While causal inference methods can be used to identify and estimate LOCU in such scenarios, this aspect is considered outside the scope of this work.
C. Proofs for Unbiased Estimation Results for the LOCO Metrics
In this Appendix, we add mathematical proofs that complement the discussion about estimation for LOCO and LOCOED We define 0 as general estimand and ê as its corresponding estimator. The bias of the estimator is given by:
Bias( = E[ - 0.
Our results extend to a broader class of full data feature importance metrics of the form:
F =
14

---

Feature Importance Metries in the Presence of Missing Data
where & can be any function and & represents the of all feature subsets. This definition encompasses wide range of metrics, including LOCO and Shapley values. Notably, Shapley values be expressed weighted average of LOCO values across submodels (Verdinelli & Wasserman, 2023). Additionally, we consider observed data feature importance metrics goD of the same form, with X()), and R₃ = replaced by X, and R..
C.1. Conditional Mean Imputation for Full Data Feature Importance Metries
Finstly, we demonstrate that conditional mean imputation results in biased estimation of full data feature importance metrics. Our analysis is based on the formulation used for the unbiased multiple imputation estimator:
0
Σ g =
-
Σ XR.
=)) (, Y,R),
where ,X,, Y, R,R) represents the imputation density. When the imputation model is learned, one can apply Monte Carlo integration obtain unbiased estimator f for @FD
Conditional mean imputation, however, simplifies the above expression to:
Σ
(E X = :::: (,, X
which assumes that the expectation operator can be pulled inside the functions g and fat This assumption is only valid if both functions are linear, which is generally not the case. Consequently, the use of conditional mean imputation introduces bias.
C.2. Mean and Conditional Mean Imputation for Observed Data Feature Importance Metrics
Next, we demonstrate that mean imputation (or a broader class of imputation methods) does not introduce bias for observed data feature importance metrics goD Since god is defined as a function of the classifier fat the reported feature importance metric reflects classífier-specific importance rather than a general global measure.
commonly used class of classifiers, referred to as impute-then- regress classifiers (Le Morvan et al., 2021), first impute the missing values and subsequently classify. If the classifier is sufficiently flexible and the dataset is large enough, the choice of imputation method becomes inconsequential, R, to Y can be used, including those employing mean imputation, without inducing bias.
information is introduced. Thus, any classifier that maps X, and
However, this conclusion holds only if imputation is performed within the classifier itself using only its input features. If conditional mean imputation is applied to the entire dataset before choosing the subset for the classifier, bias arises. Let the imputed features be:
X [XIXX R=
if = 1, if if 0.
X
In this case, the imputed variable X effectively becomes a function of other features, X fX,,, R= = Ī). The resulting bias the estimator given by:
Bias(@oD) = =
15

---

Feature Importance Metries in the Presence of Missing Data
This bias is generally nonzero because X depends all features rather than just X,. Consequently, applying conditional mean imputation before classification results in biased estimates.
D. Impact of Classifier Training on LOCO Metries
The LOCO metrics in Eqs. 2 and highlight the importance of aligning classifier training with the conditions of deployment. Specifically:
Full data feature importance metrics require classifier f designed to handle fully available features: fa (=) [=
Observed data feature importance metries require classifier fa capable of handling missing data:
fa (x)(X,)) ( = p(V RX,
Training these classifiers introduces unique challenges. For f.D. the absence of (()) in the training data creates missing data problem, complicating the training process. In contrast, training f involves dealing with missing values ("?") in X. While some models inherently handle missing data (Twala et al., 2008). common alternative is to impute missing values. However, this approach can introduce discontinuities, which hinder the learning process.
To address these challenges, thenrreethen--egress classifiers (Le Morvan et al., 2021) provide an effective solution. These classifiers leverage the following decomposition:
(x) = p(Y X = E[X, ,),
Here, missing values (Xm) are replaced with their conditional expectations based on observed values X, (and missingness indicators R). This imputation is intrinsic to the classifier and differs from post hoc imputation used during feature importance evaluation. Conditional mean imputation is often a practical and effective choice in this context.
The improper training of classifiers can introduce additional bias in LOCO metric estimations, particularly for full data feature importance metrics when fOD is used instead of f.D These biases are exacerbated missing data scenarios such as MAR and MNAR, compared to MCAR. Awareness of these limitations is crucial for ensuring robust and interpretable feature importance assessments.
E. Proof of Lemma 4.1
In this appendix, we prove Lemma 4.1 which states the form of the feature measurement importance gradient.
Proof. The gradient simplifies as follows:
G, = V E L Y, V. (Y,f((X Ren,
Σ
LYY,fa ná, (R)X ()p()
R R,y,%
6,=1
d R. (X)
Σ
[
1.,, (R RR
p(X
R.YX
i=1
6,=1
: Σ (X))
-, - (
R.Y.
1=1
E (1-
with the following explanations:
16

---

Feature Importance Metries in the Presence of Missing Data
In 1) we use the g-formula (Robins, 1986), which requires the SUTVA assumption, exchangeability (ensured by the MAR assumption) and in general positivity. The last assumption of positivity is, however, always guaranteed due the form of our intervention. Positivity in general necessitates that - with p(B = -- = > 0 and
J,55 (R, ryl مー x 0, that must also have (j rj r [ 1 x- 0. This
always holds, as setting with 7Tj 0 yield
5,.0 =0
1,6
6, 0
settings with T = 1 give
5,.
3,,,, 6, 1+110 1111 1.
and settings with 23 E (0, 1) imply
6, 12j
j,6, 6 T1 + Rj
>0 0
since 6,. n, > O, and ,, n7 1 73 holds.
In +1), we also sums for the integrals to keep notation simple, but for continuous X one can replace the sums with proper integrals.
In +2), we used the following simplifications of the gradient of the perturbed measurement policy. We first look at the case R = 1:
6, Jj
"2 Ií(6, F -
(6, m7)) (TT,)) (6, Tj + - x2)
V,, 1,, (R, = 1|R )-) 15,=1
Vo,
8, TT +1
7Tj 8
6
nT(1 (1,1)
= TT, - R,,)
(6;
6,=1
where #2.1) follows from the quotion rule of differentiation.
For R₁ = O, we find:
V 8, T1, (R, = 0|R
6,-1 = V v5 13-111111) TT,5, = 1 R 1- X x-])]]
6,-1 =- J,, = =1|R, x x 1,-1
nTy) = - (1- π (R, = 01R -( πI (R 01R = -- (1-
Bringing both together, we find the transformation in +2) and thus completes the proof.
F. Feature Measurement Importance Gradient for Time-series Settings
In this appendix, we extend the feature measurement importance gradient to time-series settings. For time-step t, we denote the observed features as X E (R u ("?")) and all features up to time t as X [X'',..,'''; The time-dependent label y E {0, 1}. classifier predicts y at each time step. This classifier is ff(X', (R), where R denotes the missingness indicators up to
We redefine the FMIG for the time-series settings as:
Feature Measurement Importance Gradient (FMIG) Gj:
V V E Σ (
fa(x (m)))
(6)
6,=8
where 83 is the value of 6, that represents no perturbation.
17

---

Feature Importance Metries in the Presence of Missing Data
F.1. Measurement Process Assumptions
We make the following assumptions about the factorization of the missingness process:
1 x (R' -x,")=))))
where we assume the multivariate missingness indicators further factorize into conditionally independent terms:
x:' (R'IR x-- -
(R )
(8)
i-1
We thus assume that at each time-point, the measurement of feature will depend on all observed features in the past.
F22 Measurement Policy Interventions
To redefine perturbation interventions for the time-series setting by now jointly intervening feature at all time-steps. We obtain:
T.. (RIX Y) y)= M
33,,, (RIIR"1 x TT) x(
(9)
t=1
where each individual intervention follows the same odds ratio parameterization in the main body.
F.3. Identification
Using the perturbation interventions, the feature measurement importance gradient Gj can identified for the time-series setting as the following corollary states:
Corollary F.1. (Identification of the Feature Measurement Importance Gradient (FMIG) for Time-series Settings) Under SUTVA and the assumptions about the missingness process from Eqs. 7 and 8, the feature measurement importance gradient (FMIG) for feature j is identified as:
- V Y
C
Birra)
1,,=1
Σ
(1- (1-
{ 2
E
(10)
18

---

Feature Importance Metries in the Presence of Missing Data
Proof. The gradient simplifies follows:
-,= 5
E
L
Y f( (
mi, Bime)) 15,=1
V,
Y f (m5)))
(X(() Bi
Bimm)) )) Fo, (R) ()((((I))))
R.F.()))
6y-1
E V [ N3 (R 1---
T . (1x
R,Y,
i=1
i J (R
Σ
-y
V 6, J (
1=1,1
x y)=
E
RY
R 1 (xx)))
Σ
y ff RR)) Σ (-1)) (1- x
R.Y.
(R RT x= 1 I
T. (R x x |p(X r)=
i-1
(1- (1- (1
= E
L MY
with the following explanations:
In +1) +1), we used the fact that only the part of π that models missingness for feature j depends on the gradient. We further use the fact that - (Y',ff(Xx.E) RR)) independent of any missingness indicators R with
In +2), we apply the product rule of differentiation.
In #3), we leverage the same simplification of V T,, (R 1xxx1x was used in Appendix E for the simpler static feature setting.
G. Experiment Details
In this appendix, we provide detailed description of the experimental setup, including the data generation process, missingness mechanisms, training configurations, and the methods used to estimate feature importance metrics. These details align with the experiments presented in Section 5 and Figures 2, 3 and 4.
G.I. Experiment Setup
The experiments are based on a synthetic time-series dataset with three features (d = 3) and three time-points (T = 3). For time-step t, we denote the observed features as X E (IR U ("?"))d and all features up to time t as X = (X ,... The iime-dependent label is y E (0, 13. A classifier predicts yt at each time step, and for the observed data, this classifier is FoR (x RR), where R' denotes the missingness indicators up to z.
19

---

Feature Importance Metries in the Presence of Missing Data
The data generation process is defined as follows. Features (1),i are independent across dimensions and generated recursively:
(xx (1) + +11 (De i 1, 1
X
where ej -/~ ~(0,0 1) and y; controls the temporal dependence for feature i. We let x() = Ö. Labels Y are binary and determined as:
1, if Σ WiX + +E W.X >0, p(r' = 1) 0.2, otherwise.
(1)
This label distribution simulates a scenario where not all data points are equally easy to classify. Parameters W represent feature importances, which vary across experiments. We generate 100,000 data points and split them into 30% for training the classifier, 30% for training the measurement policy, and 40% for testing.
G.2. Missingness Mechanisms
The missingness process in these experiments follows an MCAR or MNAR scenario. For the MNAR case, the missingness probabilities are modeled as:
T. (R 1xx = o(a + azx()))
where denotes the logistic function. In this example, X directly influences the missingness of RŞ, making the process MNAR. The exact missingness probabilities used in each experiment are summarized in Table 1.
G.3. Training and Evaluation
We used an "impute-then-regres" classifier (Le Morvan et al., 2021) with zero imputation and temporal convolutional network (TCN) (Bai et al., 2018) to classify labels y The classifier uses four layers, with 32 channels per layer, a batch size of 2,000, dropout rate of 0.2, and learning rate of 0.001.
Feature importance is evaluated using the observed and full data LOCO and the feature measurement importance gradient (FMIG). When we report the full data LOCO metric and do not mention the estimation method, refers to the ground truth feature importance, computed from the complete dataset without missingness. Binary cross-entropy used the loss function when calculating the LOCO metrics and the FMIG. We report confidence intervals using the nonparametric bootstrap with 50 samples. However, due to computational cost, we do not retrain the classifier for each bootstrap iteration. As result, the confidence intervals may be overly optimistic.
G.4. Experiment Configurations
The detailed configurations for each experiment, including the data-generating process parameters (W,y) and missingness mechanisms (T)), are provided in Table 1.
H. Experiment Regarding Classifier Training Procedures
To illustrate the impact of proper classifier training. we modified the design of Experiment by training the classifier on both the observed dataset (denoted as fd and the full dataset (denoted as (P)) This experiment, conducted under violation of the positivity assumption and a high percentage of missingness, demonstrates how improper classifier training affects feature importance metrics. The results, shown in Figure 4, reveal that classifiers trained on observed data yield lower full-data LOCO metrics compared to those trained on the full dataset. Importantly, both LOCO metrics were evaluated on the ground truth dataset without missingness, highlighting how improper training introduces additional bias on top of the bias already inherent in estimating full data LOCO metrics from observed data. The consequences are expected to be even stronger for MAR and MNAR missingness scenarios, compared to this MCAR experiment.
These findings underscore the critical importance of aligning classifier training with the target metric ensure accurate and meaningful feature importance evaluations. In practice, however, full alignment is often unfeasible due to the absence of ground truth data. Nonetheless, being aware of this limitation is crucial, as it allows for adjustments of training approaches to mitigate its consequences.
20

---

Feature Importance Metries in the Presence of Missing Data
Experiment
Data-Generating Process
Missingness Mechanisms
Exp
W 61 = 0.2
1.
<(R] = 1) = 0.75
22 8 Y2
<(R] 1) 0.5
W3 3 Ya 0.2
<<RR 1) 0.3
Exp 2
W1 8: Y1 0.2
T[R -
W2 8: Y2 0.2
<<R; -
W3 E: 0.2
TR = 1) = 0.3
Exp 3
W1- T 0.2
TRR = 1) = 0.4
W2 : Y/ = 0.2
<<Ri = 1) = 0.2
W3 2: Y33 0.2
T(R] 1) 0.1
Exp 4
W1 }: Y1
<<Ri 1) 1.0
W2 3: Y2 0.2
<<RRi 1) 0.5
W3- 3:13 Y33 1.0
TRRR) 1) 1.0; <(R) 1) = 0.5 (for > 1)
Table Experiment-spocifee details of the data-generating processes and missingness mechanisms.
Influence of the Classifier on Feature Importance
L for #P
LOCO for 150
Feature importance
3.
X
X Features
X,
Figure 4. Impact of classifier training feature importance: The full data LOCO metric derived from classifier trained on observed data underestimates feature importance compared to a classifier trained on ground ground-truth data without missingness. Both LOCO metrics estimated on the ground truth data.
21


Reviews:
[{"review_id": "AH2ST1nud4", "reviewer": "Reviewer_rtga", "summary": "This paper introduces a conceptual framework that distinguishes between feature importance methods under missing data: (1) full-data feature importance evaluates each feature's importance if all feature values were present; (2) observed-data feature importance evaluates each feature's importance based on the actual observed data, which include missing features; and (3) feature measurement importance evaluates each feature's importance based on model improvement when the feature is measured with a higher probability.\n\nEmpirically, this paper instantiates LOCO (leave-one-covariate-out) for full-data and observed-data feature importance, as well as introduces the FMIG (Feature Measurement Importance Gradient) for feature measurement importance. With synthetic datasets, the authors demonstrate that these three methods offer complementary insights. Finally, the authors demonstrate that violations of the positivity assumption can introduce bias for full-data feature importance.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate the reviewer's positive feedback and valuable suggestions for improvement. Please find our responses below.\n\n## Claims and Evidence\n\n### Definition of Bias\n\nWe define $\\theta$ as an estimand and $\\hat{\\theta}$ as its estimator. The bias of the estimator is given by:\n\\begin{equation}\n    \\text{Bias}(\\hat{\\theta}) = \\mathbb{E}[\\hat{\\theta}] - \\theta.\n\\end{equation}\n### Generalization of Metrics\n\nOur results extend to a broader class of full data feature importance metrics:\n\\begin{equation}\n    \\theta^{FD} = \\mathbb{E} \\left[ g( \\lbrace f_{cl,s}(X_{(1),s}, R_{s} = \\vec{1}) : s \\in \\mathcal{S} \\rbrace, Y ) \\right],\n\\end{equation}\nwhere $g$ can be any function and $\\mathcal{S}$ is the set of all feature subsets. This definition includes a wide range of metrics, including LOCO and Shapley values. Notably, Shapley values can be expressed as a weighted average of LOCO values across submodels (Verdinelli and Wasserman, 2023).\n\nAdditionally, we consider observed data feature importance metrics $\\theta^{OD}$ of the same form, with $X_{(1),s}$ and $R_{s} = \\vec{1}$ replaced by $X_s$ and $R_s$.\n\n### Full Data Feature Importance Metrics \n\nAs requested, we demonstrate that conditional mean imputation results in biased estimation of full data feature importance metrics. Our analysis is based on the formulation used for the unbiased MI estimator:\n\\begin{equation}\n    \\theta^{FD} = \\sum_{X_{(1)},Y} g( \\lbrace f_{cl,s}(X_{(1),s}, R_{s} = \\vec{1}) : s \\in \\mathcal{S} \\rbrace , Y ) p(X_{(1)},Y) \n    = \\sum_{X_m, X_o, Y, R} g( \\lbrace f_{cl,s}(X_{m \\cap s}, X_{o \\cap s}, R_{s} = \\vec{1}) : s \\in \\mathcal{S} \\rbrace, Y ) p(X_m|X_o,Y, R) p(X_o,Y, R),\n\\end{equation}\nwhere $p(X_m|X_o,Y, R)$ represents the imputation density. When the imputation model is learned, one can apply Monte Carlo integration to obtain the unbiased estimator.\n\nConditional mean imputation, however, simplifies the above expression to:\n\\begin{equation}\n    \\theta^{FD} \\approx \\sum_{ X_o, Y, R} g( \\lbrace f_{cl,s}( \\mathbb{E}[X_{m \\cap s}|X_o,R], X_{o \\cap s}, R_{s} = \\vec{1}) : s \\in \\mathcal{S}   \\rbrace, Y ) p(X_o,Y, R),\n\\end{equation}\nwhich assumes that the expectation operator can be pulled inside the functions $g$ and $f_{cl}$. This assumption is only valid if both functions are linear, which is generally not the case. Consequently, the use of conditional mean imputation introduces bias.\n\n### Observed Data Feature Importance Metrics\n\nNext, we demonstrate that mean imputation (or a broader class of imputation methods) does not introduce bias for observed data feature importance metrics $\\theta^{OD}$. Since $\\theta^{OD}$ is defined as a function of the classifier $f_{cl}$, the reported feature importance metric reflects classifier-specific importance rather than a general global measure.\n\nA commonly used class of classifiers, referred to as impute-then-regress classifiers, first impute the missing values and subsequently classify. If the classifier is sufficiently flexible and the dataset is large enough, the choice of imputation method becomes inconsequential, as no new information is introduced. Thus, any classifier that maps $X_s$ and $R_s$ to $Y$ can be used, including those employing mean imputation, without inducing bias.\n\nHowever, this conclusion holds only if imputation is performed within the classifier itself using only its input features. If conditional mean imputation is applied to the entire dataset before choosing the subset for the classifier, bias arises. Let the imputed features be:  $X'_i = \n        X_i$  if  $R_i = 1$ and $X'_i =  \\mathbb{E}[X_i| X_o, R = \\vec{1}]$ if $R_i = 0$.\n\nThe resulting estimator contains terms $f_{cl,s}(X^\\prime_s, R_s)$ and is thus no longer a function of $X_{s}$, and $R_{s}$, but of the whole $X_o$. Consequently, applying conditional mean imputation before classification results in biased estimates.\n\n## Theoretical Claims:\n\nWe added the following to complete the proof for the positivity assumption: \n\n\"In settings with $ \\pi_j \\in (0,1)$, we find \n$ \\pi_{j,\\delta_j} = \\frac{\\delta_j \\cdot \\pi_j}{\\delta_j \\cdot \\pi_j + 1 - \\pi_j} > 0\n$\nsince $\\delta_j \\cdot \\pi_j$ > 0.\"\n\n## Other Strengths and Weaknesses:\n\nThe reviewer asked for clarification on FMIG’s practical use and the value of ‘descriptive’ insights. We have updated the text to avoid the ambiguous term ‘descriptive’. FMIG will inform us about what features should be measured more frequently, but doesn’t suggest specific policy interventions.\n\n## Other Comments or Suggestions:\n\nWe've switched experiments 3 and 4 according to the reviewer's request. We also included $R$ in the full data classifier for clarity. We've also clarified that we mean  $R_{-0} \\equiv \\lbrace \\rbrace$.", "claims_and_evidence": "- In 3.4.2, it is claimed that mean imputation for observed-data feature importance is unbiased. It is unclear what bias means for observed-data feature importance. Hence, it is also unclear why predictive information can lead to bias.\n\n- In 3.4.3, it is strongly claimed that conditional mean imputation introduces bias for both full-data and observed-data feature importance. For observed-data feature importance, I have the same concern as in the first bullet point. For full-data feature importance, given how strong the claim is, although the intuitive explanation makes sense, more solid theoretical or empirical evidence should be presented to support the strong claim. \n\n- For 3.4.3, the authors should discuss whether these biases are also present for methods that consider feature interactions (e.g., the Shapley values). Otherwise, the authors should be clear that their exposition focuses on LOCO.\n\n- Overall, the claims about biases should be strengthened by clearly defining what biases mean for full-data feature importance, observed-data feature importance, and even feature measurement importance. Also, it should be made clear whether those claims are specific to LOCO or applicable to feature importance methods generally.", "methods_and_evaluation_criteria": "The problem focuses on demonstrating different qualitative insights gained from full-data LOCO, observed-data LOCO, and FMIG. Hence, the qualitative evaluation using synthetic datasets makes sense.", "theoretical_claims": "I checked all the proofs.\n\n- In lines 790-800 of the proof for Lemma 4.1, it's only shown that $\\pi_{j, \\delta_j} = 0$ implies $\\pi_{j} = 0$, and that $\\pi_{j, \\delta_j} = 1$ implies $\\pi_{j} = 1$. Does $\\pi_{j, \\delta_j} \\in (0, 1)$ implies $\\pi_j > 0$? This is also necessary to prove that positivity holds.\n\n- If the above issue is addressed, then the proof for Corollary E.1 seems correct to me.", "experimental_designs_or_analyses": "I checked all the experimental designs and analyses, and I found no particular issues.", "relation_to_broader_scientific_literature": "Although prior methods exist for computing feature importance scores under missing data, this paper provides a clear conceptual framework that distinguishes between the purposes of those methods (i.e, full-data feature importance, observed-data feature importance, vs. feature measurement importance). This conceptual framework can bring clarity when new methods are developed to address feature importance under missing data.", "essential_references_not_discussed": "N/A", "other_strengths_and_weaknesses": "**Strengths**\n\n- The formalism that distinguishes full-data vs. observed-data feature importance is clear and empirically useful (as demonstrated by the synthetic experiments).\n\n**Weaknesses**\n\n- The experiments are done for only one set of data-generating and missingness-generating processes. Other generation processes should also be included to make the empirical claims stronger.\n\n- The authors noted that FMIG provides descriptive insights for feature measurement importance. It is unclear how such descriptive insights are useful in practice. Isn't the main point of feature measurement importance to prescribe which features to measure more often?", "other_comments_or_suggestions": "- Experiment 4 should be presented before Experiment 3. This will complete the analyses for full-data and observed-data feature importance before introducing additional insights from FMIG.\n\n- The full-data classifier doesn't take the missingness indicators as inputs. This should be noted in the subsection labeled **Classifier** in 3.1 for clarity.\n\n- In Equation (5), it is unclear how $\\pi_1(R_1 | R_{-0}, X_{-0})$ is defined. I think a special case is needed when $i=1$.", "questions_for_authors": "N/A", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ffwpHo336W", "reviewer": "Reviewer_sya3", "summary": "## update after rebuttal: 2 --> 3\n\n\nWhen applying feature importance (FI) methods as explanation techniques for machine learning (ML) models, the presence of missing data is typically not considered. The authors highlight the issue that missing values can impact FI method results and demonstrate this using the leave-one-covariate-out (LOCO) approach. They adapt LOCO to account for missing values by incorporating the missingness process into the computation. Furthermore, the authors introduce the new \"feature measurement importance gradient\" (FMIG) as a novel metric to assess how the increased measurement or observation of a feature would influence predictive performance. They support their claims with several synthetic examples.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank the reviewer for their detailed feedback, which has helped us to improve our paper.\n## Methods and Evaluation Criteria:\nFor clarification on estimation, imputation, and the definition of bias, please refer to our response to reviewer rtga. To address the question about a distinction between the metrics, we have provided a simplified diagnostic scenario involving heart attacks and troponin levels in the introduction. Please find a slightly shorter version below: \n### Scenario 1: Biological Relationship Analysis (Full Data LOCO)\nA biomedical researcher aims to understand the intrinsic biological relationship between troponin levels, measured via a blood test, and heart attacks, independent of clinical testing protocols. The full data $LOCO$ metric is ideal to assess the explanatory power of troponin values, as it isolates the pure biological relationship, unaffected by real-world data missingness.\n\n### Scenario 2: Clinical Prediction Model Development (Observed Data LOCO)\n\nA machine learning engineer develops a heart attack prediction model and evaluates the relevance of troponin levels under current real-world missingness patterns. The observed data $LOCO$ metric is used to assess feature importance, ensuring the model aligns with the hospital’s existing testing protocols.\n\n### Scenario 3: Testing Protocol Optimization (FMIG and LOCU)\n\nAfter deploying the prediction model, an analyst seeks to optimize troponin testing by evaluating how different testing frequencies affect diagnostic performance. They use FMIG to quantify the benefits of increasing test frequency and LOCU to assess the impact of discontinuing troponin testing entirely. \n\n## Theoretical Claims:\nWe have integrated the relevant appendix sections (A.2) into the main body and expanded the derivations of the FMIG formulation.\n## Experimental Design and Analysis:\n### 1. Experiment Design:\nWe chose a time-series setting due to its prevalence in medical prediction tasks. Presenting it directly in the main paper would have been overly complex. \n### 2. Experiments 1 and 2:\nThe reviewer mentions that the experiment results confirm what is expected. We agree and want to emphasize that our goal is not to provide novel insights on missingness but to underscore the crucial difference between observed and full data feature importance metrics, challenging current practices that misapply estimators.\n### 3. Experiment 4:\nExperiment 4 mirrors Experiment 1 but introduces stronger missingness. We've clarified this in Table 4 of the appendix and revised the description in the main text. \n\nThe differences in ground truth between Figures 2A and 2B stem from variations in missingness patterns influencing classifier training. Our ground truth reflects feature importance for a classifier trained on the available data which differs between experiments. \n## Strengths and Weaknesses:\nWe thank the reviewer for pointing out errors in our notation. The following adjustments have been made:\n\nWe updated the classifier and loss function definitions. The classifier is now $ f_{cl} \\colon (\\mathbb{R} \\cup \\lbrace \"?\" \\rbrace )^d \\times \\lbrace 0,1 \\rbrace ^d \\to [0,1]^K $, and the loss function is $ L \\colon \\{0, 1, \\ldots, K-1\\} \\times [0,1]^K \\to \\mathbb{R} $.\n\nTo clarify when a feature is excluded, we define $ X_{-j}, R_{-j}, X_{(1),-j} $ as the reduced dimension observed features, missingness indicators, and ground truth features, and the classifier excluding feature $ j $ is $ f_{cl, -j} \\colon (\\mathbb{R} \\cup \\lbrace \"?\" \\rbrace )^{d-1} \\times \\lbrace 0,1 \\rbrace ^{d-1} \\to [0,1]^K $.\n\nWhile we acknowledge the redundancy of $ R $ in the classifier, we kept it to highlight the classifier's dependence on missingness. We now explicitly include $ R$ in all classifier definitions, including the full data case. \n\nRegarding $ X_{(0)} $, we added a reference for readers unfamiliar with potential outcomes: (Rubin, 2005).\n\nFinally, in response to the comment about $ \\pi $'s mapping, we corrected the definition of $ \\pi$ to represent probabilities. It now maps into $ [0,1]^{2^d}$, rather than $ \\lbrace 0,1 \\rbrace ^d$.\n## Questions for Authors:\n### 1. Distinction Between Metrics:\nIf no missingness exists, $LOCO^{OD} = LOCO^{FD}$ since $R = \\vec{1}$ and $X = X_{(1)}$. However, $LOCU$ remains generally unidentifiable without knowledge of how leaving feature $j$ unmeasured affects other measurements.\n### 2. FMIG when Features Are Rarely Observed:\nIf a feature is never observed, it is impossible to estimate its measurement impact, violating positivity assumptions. FMIG is based on an odds ratio, so if initial measurement probability is zero, the intervention effect is also zero. This does not mean increased measurement is unhelpful—just that our definition precludes estimation in such cases. When a feature is rarely observed, identification holds, but finite-sample issues can cause non-robust estimates (see Figure 2).", "claims_and_evidence": "While I agree that the issue the authors raise here is an interesting one, especially for applied data science work, the paper is often not well written (especially but not limited to formal notation, see further below) and the experimental results are not extremely insightful.", "methods_and_evaluation_criteria": "LOCO is examined from three different perspectives: (1) full data LOCO, (2) observed data LOCO, and (3) leave-one-covariate-unmeasured (LOCU), with a primary focus on the first two.\n\nThe distinction between these perspectives is not entirely clear, leaving some open questions. For instance, both LOCO^FD and LOCO^OD can be approximated by imputing missing values and differ only in the imputation method used. This raises the question of how to distinguish between simple and non-simple imputation methods.\n\nFurthermore, concrete application examples illustrating when each perspective is of particular interest would be valuable, especially in differentiating between LOCO^OD and LOCU.", "theoretical_claims": "I have reviewed the proof of Lemma 4.1 in the appendix. Unfortunately, I am only partially familiar with Robins' g-formula, which prevents me from fully understanding this (fundamental) step.\n\nThe definition of the FMIG is motivated by literature in the appendix (A.2). This should be part of the main text. A more detailed derivation of the formulation and construction of the newly introduced measure (FMIG) should be provided, at least in the appendix.", "experimental_designs_or_analyses": "General comment w.r.t. exp setup: I am not exactly sure why the authors mix in time-dependent components in their experiment without discussing them before in the paper (with only a short ref to the appendix where this is covered more). This is certainly not \"illegal\" but somewhat unmotivated and / or one wonders why the i.i.d. scenario is not explored at all. Also, the experiments are not very \"large\" (I do not mean data size here!), so not very many different scenarios are explored, so it is unclear how much the results generalize. \n\nExperiments 1 and 2: The experiment setup seems ok to me, but the experiments pretty much say what we knew already? Missingness reduces the predictive value of features. And \"informed missingness\", well…. \"informs\", so it influences our loss-based LOCO result.  \n\nExperiment 3: I would rather call this more or less a \"sanity check\" for the FMIG measure. Which is fine. That LOCO cannot provide the same information seems clear, as it was never constructed for this. LOCO assesses a feature's importance, FMIG reflects the importance or added value of increasing a feature’s measurement probability. \n\nExperiment 4: The presentation is somewhat confusing, as it initially follows the scenario from Experiment 1 but is later adapted in the results section. Additionally, the ground truth is unclear (as it differs between Figures 2A and 2B).\n\nDetailed information about the experimental setup is given in the appendix, but primarily for experiments 1+2; experiments 3+4 lack detailed information.", "relation_to_broader_scientific_literature": "The authors state that this is the first work to evaluate LOCO in the context of missing values. I am also not aware of any other research addressing this topic. The paper cites works from both the feature importance and missing values literature, evaluating the advantages and disadvantages of various approaches in the missing values research field.", "essential_references_not_discussed": "See above.", "other_strengths_and_weaknesses": "The paper has severe problems in notation and description. This makes it sometimes very hard to follow, and the burden on the reader too heavy. One should not have to \"guess\" things means. Some examples: \n\nCross-entropy is not defined on \"labels\" x \"labels\", but \"label\" x \"probs\". For the same reason, the classifier f_cl also cannot output labels but must output probs.\n\nIt is often confusing that the authors conflate the \"training procedure\" of the model with the \"predictor\" of the model. I know that it can be annoying to distinguish between the two in notation, but it would help.\nJust look at formula (2): The authors write f(X_{(1), -j}). First of all, by definition, f would always take R. Here, it does not. Then, via \"-j\" the input loses one dimension. The definition of f does not allow this. I can GUESS what is LIKELY meant here —> But I should not. And this makes other areas of the paper even harder to follow. \n\nWhy does the classifier also take R? Additionally, the features are defined as \"R union ?\".\nThis is redundant? Also, is training or prediction meant here...?\n\nX_(0) is never defined but is used later when we do X_(pi).\n\nThere is a difference in the missingness-process between the observed missingness label \\in {0,1} and its associated probability. I had the impression that the authors conflate the two.", "other_comments_or_suggestions": "Appendix F, line 938: \"eps_i\" is used twice here, in the initialization line (2nd) and the recursion (1st). I would GUESS these should be independent noise variables? So different? Defined like this, we convex-combine the same stuff and the recursion would never change the value of X^t ??  \n\nAccording to Table 1, Appendix F.4, the probability of observing X3 in experiment 1 is 0.25; in the experiment description in section 5.2 for experiment 1 it is said to be 0.3. Which number is correct?\n\nFurther suggestions for improvements:\nA. The role of imputation in missing data should be mentioned in the introduction.\nB. The approach is only formulated for classification. Why? I see no reason for that restriction and the authors do not even discuss it. \nC. I would highly appreciate a simple example clarifying FMIG.", "questions_for_authors": "1. Section 3.4.1 (Unbiased Estimation of the Full Data LOCO Metric) suggests that unbiased estimation is impossible. A conclusive statement reinforcing this would help. What does unbiased refer to?\n\n2. As written above, the distinction between Observed Data LOCO and LOCU is not entirely clear. Providing examples for each scenario would help clarify how the expectation values differ. How do the LOCO measures relate to each other if you have a dataset that contains no missing values? Is LOCO^{OD} = LOCU? \n\n3. If a feature has nearly never been observed, can FMIG still assign it a measurement importance? If yes, how? Is it assumed that the learner has encountered such data points during training? Can we infer a feature importance value from this, or would it simply be a measurement importance? I would expect the latter.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "UgB5cdm12f", "reviewer": "Reviewer_3Vm4", "summary": "This paper tackles the challenge of determining feature importance in realistic scenarios where data is missing. It is the first to explicitly formulate this problem and in doing so, introduces FMIG, a novel gradient-based metric that quantifies how small increases in the frequency of feature measurement can improve prediction performance. The approach is backed by theoretical derivations and validated through synthetic experiments under various missing data scenarios.\n\n## update after rebuttal\nAfter reading the other reviewers' comments and the authors' rebuttal, I have decided to keep my original score. I appreciate the authors’ thoughtful response to my review. While I still believe it may be possible to obtain or semi-generate a more realistic setting to evaluate the proposed method, I think the paper meets the current experimental standards for more theoretical works at this conference and should be accepted.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We greatly appreciate the reviewer's thoughtful review and positive feedback. We recognize the value of conducting additional experiments, particularly those utilizing datasets with established prior knowledge of expected effects, to further validate our results. Unfortunately, we do not have access to such datasets. As a result, we will explicitly address this limitation in the Conclusion section, highlighting the potential for future research to explore these opportunities.", "claims_and_evidence": "The paper clearly lays out its theoretical derivations and presents synthetic experiments that support its main claims. It also clearly defines the theoretical settings in which the method applies (MAR vs. NMAR). However, the claim that FMIG can effectively guide data acquisition in practice is less supported, since all experiments are synthetic. The assumption that the gradient approach can accuractly capture the effects on data in the real world need to be validated.", "methods_and_evaluation_criteria": "The methods and evaluation criteria are reasonable for an initial validation of the proposed approach. The paper’s synthetic experiments help illustrate how both the full data and observed data metrics, as well as the FMIG, behave under controlled conditions. However, further testing on benchmark or real-world datasets would be necessary to confirm the practical applicability of these methods.\n\nOne useful suggestion would be to induce synthetic missingness in real-world data by syntactically perturbing the missingness mechanism. Additionally, it would be beneficial to run an experiment with real dataset where features are missing and where there is prior knowledge of the expected effects. This would allow the authors to assess whether the insights provided by FMIG and the other metrics align with known expectations, even without direct access to the full data or comparison to the full data scenario.", "theoretical_claims": "I have briefly checked the main claims and they seem valid. In particular, the proofs related to the identification of FMIG and the derivation of the LOCO metrics appear solid. The theoretical ground is strong, and the settings and assumptions are clearly stated and highlighted as expected.", "experimental_designs_or_analyses": "As I mentioned in the Methods and Evaluation section, the synthetic experiments are well demonstrating the differences between full and observed data scenarios, as well as the insights that could be captured by FMIG. However, further testing on real-world data would help validate these findings in practice.", "relation_to_broader_scientific_literature": "In my view, this paper makes a very important contribution, as the entire field of feature importance has generally focused only on the case where all data is always available which is very often not true in practice. The examples provided, especially in medical contexts, are highly relevant and highlight the need for further research in this area. Moreover, the authors offer an initial solution backed by a strong theoretical foundation, which I believe could pave the way for both theoretical and empirical research with significant impact.", "essential_references_not_discussed": "I'm not familiar enough with the latest works in that field to provide a solid review or opinion on this part.", "other_strengths_and_weaknesses": "In addition to the strengths mentioned above, I would like to emphasize one more strength: the use of the MAR formulation is well justified  as it provides a robust model for real-world scenarios where sampling decisions are often based on the outcomes of other features. Ideally having a method for NMAR could be very useful, but as stated much harder to tackle.", "other_comments_or_suggestions": "-", "questions_for_authors": "-", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper studies the problem of feature importance when data is not fully observed. Feature importance is a central tool in explainable AI; however, the challenge of handling missing data in the computation of feature importance has been largely overlooked. Therefore, this paper makes a notable contribution by formulating the problem and proposing initial solutions. The reviewers provided suggestions for additional experiments and presentation enhancements, which the authors are encouraged to consider.
Decision: Accept (poster)