SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Ahmad Chamma
Inria, Universite Paris Saclay, CEA ahmad.chamma@inria.fr
Denis A. Engemann
Roche Pharma Research and Early Development, Neuroscience and Rare Diseases, Roche Innovation Center Basel, F. Hoffmann-La Roche Ltd., Basel, Switzerland denis.engemann@roche.com
Inria, Universite Paris Saclay, CEA bertrand.thirion@inria.fr
Variable importance assessment has become a crucial step in machine-learning applications when using complex learners, such as deep neural networks, on large-scale data. Removal-based importance assessment is currently the reference approach, particularly when statistical guarantees are sought to justify variable inclusion. It is often implemented with variable permutation schemes. On the flip side, these approaches risk misidentifying unimportant variables as important in the presence of correlations among covariates. Here we develop a systematic approach for studying Conditional Permutation Importance (CPI) that is model agnostic and computationally lean, as well as reusable benchmarks of state-of-the-art variable importance estimators. We show theoretically and empirically that CPI overcomes the limitations of standard permutation importance by providing accurate type-I error control. When used with a deep neural network, CPI consistently showed top accuracy across benchmarks. An experiment on real-world data analysis in a large-scale medical dataset showed that CPI provides a more parsimonious selection of statistically significant variables. Our results suggest that CPI can be readily used as drop-in replacement for permutation-based methods.
Machine learning is an area of growing interest for biomedical research [Iniesta et al., 2016, Taylor and Tibshirani, 2015, Malley et al., 2011] for predicting biomedical outcomes from heterogeneous inputs [Hung et al., 2020, Zheng and Agresti, 2000, Giorgio et al., 2022, Sechidis et al., 2021]. Biomarker development is increasingly focusing on multimodal data including brain images, genetics, biological specimens and behavioral data [Coravos et al., 2019, Siebert, 2011, Ye et al., 2008, Castillo-Barnes et al., 2018, Yang et al., 2022]. Such high-dimensional settings with correlated inputs put strong pressure on model identification. With complex, often nonlinear models, it becomes harder to assess the role of features in the prediction, aka variable importance [Casalicchio et al., 2019, Altmann et al., 2010]. In epidemiological and clinical studies, one is interested in population-level feature importance, as opposed to instance-level feature importance.
In that context, variable importance is understood as conditional importance, meaning that it measures the information carried by one variable on the outcome given the others, as opposed to the easily accessible marginal importance of the variables. Conditional importance is necessary e.g. to assess
whether a given measurement is worth acquiring, on top of others, for a diagnostic or prognostic task. As the identification of relevant variables is model-dependent and potentially unstable, point estimates of variable importance are misleading. One needs confidence intervals of importance estimates or statistical guarantees, such as type-I error control, i.e. the percentage of non-relevant variables detected as relevant (false positives). This control depends on the accuracy of the p-values on variable importance being non-zero [Cribbie, 2000].
Within the family of removal-based importance assessment methods [Covert et al., 2022], a popular model-agnostic approach is permutation variable importance, that measures the impact of shuffling a given variable on the prediction [Janitza et al., 2018]. By repeating the permutation importance analysis on permuted replicas of the variable of interest, importance values can be tested against the null hypothesis of being zero, yielding p-values that are valid under general distribution assumptions. Yet, statistical guarantees for permutation importance assessment do not hold in the presence of correlated variables, leading to selection of unimportant variables [Molnar et al., 2021, Hooker et al., 2021, Nicodemus et al., 2010, Stigler, 2005]. For instance, the method proposed in [Mi et al., 2021] is a powerful variable importance evaluation scheme, but it does not control the rate of type-I error.
In this work, we propose a general methodology for studying the properties of Conditional Permutation Importance in biomedical applications alongside tools for benchmarking variable importance estimators:
- Building on the previous literature on CPI, we develop theoretical results for the limitations regarding Permutation Importance (PI) and advantages of conditional Permutation Importance (CPI) given correlated inputs (section 3).
- We propose a novel implementation for CPI allowing us to combine the potential advantages of highly expressive base learners for prediction (a deep neural network) and a comparably lean Random Forest model as a conditional probability learner (section 4).
- We conduct extensive benchmarks on synthetic and heterogeneous multimodal real-world biomedical data tapping into different correlation levels and data-generating scenarios for both classification and regression (section 5).
- We propose a reusable library for simulation experiments and real-world applications of our method on a public GitHub repo https://github.com/achamma723/Variable_Importance.
A popular approach to interpret black-box predictive models is based on locally interpretable, i.e. instance-based, models. LIME [Ribeiro et al., 2016] provides local interpretable model-agnostic explanations by locally approximating a given complex model with a linear model around the instance of interest. SHAP [Burzykowski, 2020] is a popular package that measures local feature effects using the Shapley values from coalitional game theory.
However, global, i.e. population-level, explanations are better suited than instance-level explanations for epidemiological studies and scientific discovery in general. Many methods can be subsumed under the general category of removal-based approaches [Covert et al., 2022]. Permutation importance is defined as the decrease in a model score when the values of a single feature are randomly shuffled [Breiman, 2001]. This procedure breaks the relationship between the feature and the outcome, thus the drop in model performance expresses the relevance of the feature. Janitza et al. [2018] use an ensemble of Random Forests with the sample space equally partitioned. They approximate the null distribution based on the observed importance scores to provide p-values. Yet, this coarse estimate of the null distribution can give unstable results. Recently, a generic approach has been proposed in [Williamson et al., 2021] that measures the loss difference between models that include or exclude a given variable, also applied with LOCO (Leave One Covariate Out) in the work by Lei et al. [2018]. They show the asymptotic consistency of the model. However, their approach is intractable, given that it requires refitting the model for each variable. A simplified version has been proposed by Gao et al. [2022]. However, relying on linear approximations, some statistical guarantees from [Williamson et al., 2021] are potentially lost.
Another recent paper by Mi et al. [2021] has introduced model-agnostic explanation for black-box models based on the permutation approach. Permutation importance [Breiman, 2001] can work with
any learner. Moreover, it relies on a single model fit, hence it is an efficient procedure. Strobl et al. [2008] pointed out limitations with the permutation approach in the face of correlated variables. As an alternative, they propose a conditional permutation importance by shuffling the variable of interest conditionally on the other variables. However, the solution was specific to Random Forests, as it is based on bisecting the space with the cutpoints extracted during the building process of the forest.
With the Conditional Randomization Test proposed by Candes et al. [2017], the association between the outcome  $y$  and the variable of interest  $x^j$  conditioned on  $\mathbf{x}^{-\mathbf{j}}$  is estimated. The variable of interest is sampled conditionally on the other covariates multiple times to compute a test statistic and p-values. However, this solution is limited to generalized linear models and is computationally expensive. Finally, a recent paper by [Watson and Wright, 2021] showed the necessity of conditional schemes and introduced a knockoff sampling scheme, whereby the variable of interest is replaced by its knockoff to monitor any drop in performance of the leaner used without refitting. This method is computationally inexpensive, and enjoys statistical guarantees from [Lei et al., 2018]. However, it depends on the quality of the knockoff sampling where even a relatively small distribution shift in knockoff generation can lead to large errors at inference time.
Other work has presented comparisons of select models within distinct communities [Liu et al., 2021, Chipman et al., 2010, Janitza et al., 2018, Mi et al., 2021, Altenmüller et al., 2021], however, lacking conceptualization from a unified perspective. In summary, previous work has established potential advantages of conditional permutation schemes for inference of variable importance. Yet, the lack of computationally scalable approaches has hampered systematic investigations of different permutation schemes and their comparison with alternative techniques across a broader range of predictive modeling settings.
3 Permutation importance and its limitations
Notations We will use the following system of notations. We denote matrices, vectors, scalar variables and sets by bold uppercase letters, bold lowercase letters, script lowercase letters, and calligraphic letters, respectively (e.g.  $\mathbf{X}$ ,  $\mathbf{x}$ ,  $x$ ,  $\mathcal{X}$ ). We call  $\mu$  the function that maps the sample space  $\mathcal{X} \subset \mathbb{R}^p$  to the sample space  $\mathcal{V} \subset \mathbb{R}$  and  $\hat{\mu}$  is an estimate of  $\mu$ . Permutation procedures will be represented by (perm). We denote by  $\llbracket n\rrbracket$  the set  $\{1, \dots, n\}$ .
Let  $\mathbf{X} \in \mathbb{R}^{n \times p}$  be a design matrix where the  $i$ -th row and the  $j$ -th column are denoted  $\mathbf{x}_i$  and  $\mathbf{x}^j$  respectively. Let  $\mathbf{X}^{-j} = (\mathbf{x}^1, \ldots, \mathbf{x}^{j-1}, \mathbf{x}^{j+1}, \ldots, \mathbf{x}^p)$  be the design matrix, where the  $j^{th}$  column is removed, and  $\mathbf{X}^{(j)} = (\mathbf{x}^1, \ldots, \mathbf{x}^{j-1}, \{\mathbf{x}^j\}^{perm}, \mathbf{x}^{j+1}, \ldots, \mathbf{x}^p)$  the design matrix with the  $j^{th}$  column shuffled. The rows of  $\mathbf{X}^{-j}$  and  $\mathbf{X}^{(j)}$  are denoted  $\mathbf{x}_i^{-j}$  and  $\mathbf{x}_i^{(j)}$  respectively, for  $i \in [[n]]$ .
Problem setting Machine learning inputs are a design matrix  $\mathbf{X}$  and a target  $\mathbf{y} \in \mathbb{R}^n$  or  $\in \{0,1\}^n$  depending on whether it is a regression or a classification problem. Throughout the paper, we rely on an i.i.d. sampling train / test partition scheme where the  $n$  samples are divided into  $n_{train}$  training and  $n_{test}$  test samples and consider that  $\mathbf{X}$  and  $\mathbf{y}$  are restricted to the test samples - the training samples were used to obtain  $\hat{\mu}$ .
3.2 The permutation approach leads to false detections in the presence of correlations
A known problem with permutation variable importance is that if features are correlated, their importance is typically over-estimated [Strobl et al., 2008], leading to a loss of type-I error control. However, this loss has not been precisely characterized yet, which we will work through for the linear case. We use the setting of [Mi et al., 2021], where the estimator  $\hat{\mu}$ , computed with empirical risk minimization under the training set, is used to assess variable importance on a new set of data (test set). We consider a regression model with a least-square loss function for simplicity. The importance of variable  $\mathbf{x}^{\mathrm{j}}$  is computed as follows:
$$
\hat {m} ^ {j} = \frac {1}{n _ {\text {t e s t}}} \sum_ {i = 1} ^ {n _ {\text {t e s t}}} \left(\left(y _ {i} - \hat {\mu} \left(\mathbf {x} _ {\mathbf {i}} ^ {(\mathbf {j})}\right)\right) ^ {2} - \left(y _ {i} - \hat {\mu} (\mathbf {x} _ {\mathbf {i}})\right) ^ {2}\right). \tag {1}
$$
Let  $\varepsilon_{i} = y_{i} - \mu (\mathbf{x_{i}})$  for  $i\in [[n_{test}]]$  . Re-arranging terms yields
$$
\hat {m} ^ {j} = \frac {1}{n _ {\text {t e s t}}} \sum_ {i = 1} ^ {n _ {\text {t e s t}}} \left(\hat {\mu} \left(\mathbf {x} _ {\mathbf {i}}\right) - \hat {\mu} \left(\mathbf {x} _ {\mathbf {i}} ^ {(\mathbf {j})}\right)\right) \left(2 \mu \left(\mathbf {x} _ {\mathbf {i}}\right) - \hat {\mu} \left(\mathbf {x} _ {\mathbf {i}}\right) - \hat {\mu} \left(\mathbf {x} _ {\mathbf {i}} ^ {(\mathbf {j})}\right) + 2 \varepsilon_ {i}\right). \tag {2}
$$
Mi et al. [2021] argued that these terms vanish when  $n_{test} \to \infty$ . But it is not the case as long as the training set is fixed. In order to get tractable computation, we assume that  $\mu$  and  $\hat{\mu}$  are linear functions:  $\mu(\mathbf{x}) = \mathbf{x}\mathbf{w}$  and  $\hat{\mu}(\mathbf{x}) = \mathbf{x}\hat{\mathbf{w}}$ . Let us further consider that  $\mathbf{x}^{\mathrm{j}}$  is a null feature, i.e.  $w^{j} = 0$ . This yields  $\mathbf{x}\mathbf{w} = x^{j}w^{j} + \mathbf{x}^{-\mathrm{j}}\mathbf{w}^{-\mathrm{j}} = \mathbf{x}^{-\mathrm{j}}\mathbf{w}^{-\mathrm{j}}$ . Denoting the standard dot product by  $\langle ., \rangle$ , this leads to (Detailed proof of getting from Eq. 2 to Eq. 3 can be found in supplement section A)
$$
\hat {m} ^ {j} = \frac {2 \hat {w} ^ {j}}{n _ {t e s t}} \left\langle \mathbf {x} ^ {\mathbf {j}} - \{\mathbf {x} ^ {\mathbf {j}} \} ^ {p e r m}, \mathbf {X} ^ {- \mathbf {j}} \left(\mathbf {w} ^ {- \mathbf {j}} - \hat {\mathbf {w}} ^ {- \mathbf {j}}\right) + \varepsilon \right\rangle \tag {3}
$$
as  $(\| \mathbf{x}^{\mathbf{j}}\|^{2} - \| \{\mathbf{x}^{\mathbf{j}}\}^{perm}\|^{2}) = 0$ . Next,  $\frac{1}{n_{test}}\langle \{\mathbf{x}^{\mathbf{j}}\}^{perm},\mathbf{X}^{-\mathbf{j}}(\mathbf{w}^{-\mathbf{j}} - \hat{\mathbf{w}}^{-\mathbf{j}})\rangle \to 0$  and  $\frac{1}{n_{test}}\langle \mathbf{x}^{\mathbf{j}}-$ $\{\mathbf{x}^{\mathbf{j}}\}^{perm},\varepsilon \rangle \to 0$  when  $n_{test}\to \infty$  with speed  $\frac{1}{\sqrt{n_{test}}}$  from the Berry-Essen theorem, assuming that the first three moments of these quantities are bounded and that the test samples are i.i.d. Let us assume that the correlation within  $\mathbf{X}$  takes the following form:  $\mathbf{x}^{\mathrm{j}} = \mathbf{X}^{-\mathrm{j}}\mathbf{u} + \delta$ , where  $\mathbf{u}\in \mathbb{R}^{p - 1}$  and  $\delta$  is a random vector independent of  $\mathbf{X}^{-\mathbf{j}}$ . By contrast,  $\frac{2\hat{w}^j}{n_{test}}\langle \mathbf{x}^{\mathrm{j}},\mathbf{X}^{-\mathrm{j}}(\mathbf{w}^{-\mathrm{j}} - \hat{\mathbf{w}}^{-\mathrm{j}})\rangle$  has a non-zero limit  $2\hat{w}^j\mathbf{u}^T Cov(\mathbf{X}^{-\mathbf{j}})(\mathbf{w}^{-\mathbf{j}} - \hat{\mathbf{w}}^{-\mathbf{j}})$ , where  $Cov(\mathbf{X}^{-\mathbf{j}}) = \lim_{n_{test}\to \infty}\frac{\mathbf{X}^{-\mathbf{j}}\mathbf{T}\mathbf{X}^{-\mathbf{j}}}{n_{test}}$  (remember that both  $\mathbf{w}^{-\mathrm{j}}$  and  $\hat{\mathbf{w}}^{-\mathrm{j}}$  are fixed, because the training set is fixed). Thus, the permutation importance of a null but correlated variable does not vanish when  $n_{test}\to \infty$ , implying that this inference scheme will lead to false positives.
4 Conditional sampling-based feature importance
4.1 Main result
We define the permutation of variable  $x^j$  conditional to  $\mathbf{x}^{-\mathbf{j}}$ , as a variable  $\tilde{x}^j$  that retains the dependency of  $x^j$  with respect to the other variables in  $\mathbf{x}^{-\mathbf{j}}$ , but where the independent part is shuffled;  $\tilde{\mathbf{x}}^{(\mathbf{j})}$  is the vector  $\mathbf{x}$  where  $x^j$  is replaced by  $\tilde{x}^j$ . We propose two constructions below (see Fig. E1). In the case of regression, this leads to the following importance estimator:
$$
\hat {m} _ {C P I} ^ {j} = \frac {1}{n _ {t e s t}} \sum_ {i = 1} ^ {n _ {t e s t}} \left((y _ {i} - \hat {\mu} (\tilde {\mathbf {x}} _ {\mathbf {i}} ^ {(\mathbf {j})})) ^ {2} - (y _ {i} - \hat {\mu} (\mathbf {x} _ {\mathbf {i}})) ^ {2}\right). \tag {4}
$$
As noted by Watson and Wright [2021], this inference is correct, as in traditional permutation tests, as long as one wishes to perform inference conditional to  $\hat{\mu}$ . However, the following proposition states that this inference has much wider validity in the asymptotic regime.
Proposition. Assuming that the estimator  $\hat{\mu}$  is obtained from a class of functions  $\mathcal{F}$  with sufficient regularity, i.e. that it meets conditions (A1, A2, A3, A4, B1 and B2) defined in supplementary material, the importance score  $\hat{m}_{CPI}^j$  defined in (4) cancels when  $n_{train} \to \infty$  and  $n_{test} \to \infty$  under the null hypothesis, i.e. the  $j$ -th variable is not significant for the prediction. Moreover, the Wald statistic  $z^j = \frac{\text{mean}(\hat{m}_{CPI}^j)}{\text{std}(\hat{m}_{CPI}^j)}$  obtained by dividing the mean of the importance score by its standard deviation asymptotically follows a standard normal distribution.
This implies that in the large sample limit, the p-value associated with  $z^j$  controls the type-I error rate for all optimal estimators in  $\mathcal{F}$ .
The proof of the proposition is given in the supplement (section C). It consists in observing that the importance score defined in (4) is 0 for the class of learners discussed in [Williamson et al., 2021], namely those that meet a certain set of convergence guarantees and are invariant to arbitrary change of their  $j^{th}$  argument, conditional on the others. In the supplement, we also restate the precise technical conditions under which the importance score  $\hat{m}_{CPI}^j$  used is (asymptotically) valid, i.e. leads to a Wald-type statistic that behaves as a standard normal under the null hypothesis.
It is easy to see that for the setting in Sec. 3.2, all terms in Eq. 4 vanish with speed  $\frac{1}{\sqrt{n_{test}}}$ .
4.2 Practical estimation
Next, we present algorithms for computing conditional permutation importance. We propose two constructions for  $\tilde{x}^j$ , the conditionally permuted counterpart of  $x^{j}$ . The first one is additive: on test samples,  $x^{j}$  is divided into the predictable and random parts  $\tilde{x}^j = \mathbb{E}(x^j |\mathbf{x}^{-\mathbf{j}}) + (x^j -\mathbb{E}(x^j |\mathbf{x}^{-\mathbf{j}}))^{\textit{perm}}$  where the residuals of the regression of  $x^{j}$  on  $\mathbf{x}^{-\mathbf{j}}$  are shuffled to obtain  $\tilde{x}^j$ . In practice, the expectation is obtained by a universal but efficient estimator, such as a random forest trained on the test set.
The other possibility consists in using a random forest (RF) model to fit  $x^{j}$  from  $\mathbf{x}^{-\mathbf{j}}$  and then sample the prediction within leaves of the RF.
Random shuffling is applied B times. For instance, using the additive construction, a shuffling of the residuals  $\tilde{\epsilon}^{\mathbf{j},\mathbf{b}}$  for a given  $b\in [[B]]$  allows to reconstruct the variable of interest as the sum of the predicted version and the shuffled residuals, that is
$$
\tilde {\mathbf {x}} ^ {\mathbf {j}, \mathbf {b}} = \hat {\mathbf {x}} ^ {\mathbf {j}} + \tilde {\epsilon} ^ {\mathbf {j}, \mathbf {b}}. \tag {5}
$$
Let  $\tilde{\mathbf{X}}^{\mathbf{j},\mathbf{b}} = (\mathbf{x}^{\mathbf{1}},\dots ,\mathbf{x}^{\mathbf{j} - 1},\tilde{\mathbf{x}}^{\mathbf{j},\mathbf{b}},\mathbf{x}^{\mathbf{j} + 1},\dots ,\mathbf{x}^{\mathbf{p}})\in \mathbb{R}^{n_{test}\times p}$  be the new design matrix including the reconstructed version of the variable of interest  $\mathbf{x}^{\mathrm{j}}$  . Both  $\tilde{\mathbf{X}}^{\mathbf{j},\mathbf{b}}$  and the target vector  $\mathbf{y}$  are fed to the loss function in order to compute a loss score  $l_i^{j,b}\in \mathbb{R}$  defined by
$$
l _ {i} ^ {j, b} = \left\{ \begin{array}{l} y _ {i} \log \left(\frac {S (\hat {y} _ {i})}{S (\tilde {y} _ {i} ^ {b})}\right) + (1 - y _ {i}) \log \left(\frac {1 - S (\hat {y} _ {i})}{1 - S (\tilde {y} _ {i} ^ {b})}\right) \\ (y _ {i} - \tilde {y} _ {i} ^ {b}) ^ {2} - (y _ {i} - \hat {y} _ {i}) ^ {2} \end{array} \right. \tag {6}
$$
for binary and regression cases respectively where  $i \in [[n_{test}]], j \in [[p]], b \in [[B]]$ ,  $i$  indexes a test sample of the dataset,  $\hat{y}_i = \hat{\mu}(\mathbf{x_i})$  and  $\tilde{y}_i^b = \hat{\mu}(\tilde{\mathbf{x}}_i^{j,\mathbf{b}})$  is the new fitted value following the reconstruction of the variable of interest with the  $b^{th}$  residual shuffled and  $S(x) = \frac{1}{1 + e^{-x}}$ .
The variable importance scores are computed as the double average over the number of permutations  $B$  and the number of test samples  $n_{test}$  (line 15 of Alg. 1), while their standard deviations are computed as the square root of the average over the test samples of the quadratic deviation over the number of permutations (line 17). Note that, unlike Williamson et al. [2021], the variance estimator is non-vanishing, and thus can be used as a plugin. A  $z^{j}_{CPI}$  statistic is then computed by dividing the mean of the corresponding importance scores with the corresponding standard deviation (line 18). P-values are computed using the cumulative distribution function of the standard normal distribution (line 19). The conditional sampling and inference steps are summarized in Algorithm 1. This leads to the CPI-DNN method when  $\hat{\mu}$  is a deep neural network, or CPI-RF when  $\hat{\mu}$  is a random forest. Supplementary analysis reporting the computational advantage of CPI-DNN over a remove-and-relearn alternative a.k.a. LOCO-DNN, can be found in supplement (section D), which justifies its computational leanness.
In all experiments, we refer to the original implementation of the different methods in order to maintain a fair comparison. Regarding Permfit-DNN, CPI-DNN and CPI-RF models specifically, our implementation involves a 2-fold internal validation (the training set of further split to get validation set for hyperparameter tuning). The scores from different splits are thus concatenated to compute the final variable importance. We focus on the Permfit-DNN and CPI-DNN importance estimators that use a deep neural network as learner  $\hat{\mu}$ , using standard permutation and algorithm 1, respectively. All experiments are performed with 100 runs. The evaluation metrics are detailed in the supplement (section E).
5.1 Experiment 1: Type-I error control and accuracy when increasing variable correlation
We compare the performance of CPI-DNN with that of Perfit-DNN by applying both methods across different correlation scenarios. The data  $\{\mathbf{x_i}\}_{i = 1}^n$  follow a Gaussian distribution with a prescribed covariance structure  $\boldsymbol{\Sigma}$  i.e.  $\mathbf{x_i}\sim \mathcal{N}(0,\boldsymbol {\Sigma})\forall i\in [[n]]$  . We consider a block-designed covariance matrix  $\boldsymbol{\Sigma}$  of 10 blocks with an equal correlation coefficient  $\rho \in \{0,0.2,0.5,0.8\}$  among the variables of
Algorithm 1 Conditional sampling step: The algorithm implements the conditional sampling step in place of the permutation approach when computing the p-value of variable  $x^j$
Require:  $\mathbf{X} \in \mathbb{R}^{n_{test} \times p}$ ,  $\mathbf{y} \in \mathbb{R}^{n_{test}}$ ,  $\hat{\mu}$ : estimator,  $l$ : loss function,  $\mathrm{RF}_j$ : learner trained to predict  $x^j$  from  $\mathbf{x}^{-j}$