Statistically Valid Variable Importance Assessment through Conditional Permutations
Authors: Ahmad Chamma
Denis Engemann
Bertrand Thirion
Venue: NeurIPS 2023 poster (2023)
Keywords: Interpretability
Variable Importance
Machine Learning
Deep Learning
Statistical Inference

Abstract: Variable importance assessment has become a crucial step in machine-learning applications when using complex learners, such as deep neural networks, on large-scale data. Removal-based importance assessment is currently the reference approach, particularly when statistical guarantees are sought to justify variable inclusion. It is often implemented with variable permutation schemes. On the flip side, these approaches risk misidentifying unimportant variables as important in the presence of correlations among covariates. Here we develop a systematic approach for studying Conditional Permutation Importance (CPI) that is model agnostic and computationally lean,  as well as reusable benchmarks of state-of-the-art variable importance estimators. We show theoretically and empirically that \textit{CPI} overcomes the limitations of standard permutation importance by providing accurate type-I error control. When used with a deep neural network, \textit{CPI} consistently showed top accuracy across benchmarks. An experiment on real-world data analysis in a large-scale medical dataset showed that \textit{CPI} provides a more parsimonious selection of statistically significant variables. Our results suggest that \textit{CPI} can be readily used as drop-in replacement for permutation-based methods.

Statistically Valid Variable Importance Assessment through Conditional Permutations
Ahmad Chamma Inria, Universite Paris Saclay, CEA ahmad. chamma@inria.ft
Denis A. Engemann
Roche Pharma Research and Early Development,
Neuroscience and Rare Diseases,
Roche Innovation Center Basel,
F. Hoffmann-Laa Roche Ltd., Basel, Switzerland
denis engemann@roche.com
Bertrand Thirion
Inria, Universite Paris Saclay, CEA bertrand. wwwiiin@iiiiiii......mmmmmmmaaai
Abstract
Variable importance assessment has become crucial step in machine-learning applications when using complex learners, such as deep neural networks, on large-scale data. Removal-based importance assessment is currently the reference approach, particularly when statistical guarantees are sought to justify variable inclusion. It is often implemented with variable permutation schemes. On the flip side, these approaches risk misidentifying unimportant variables as important in the presence of correlations among covariates. Here we develop systematic approach for studying Conditional Permutation Importance (CPI) that is model agnostic and computationally lean, well reusable benchmarks of state-of-the of-thh--rtttttt variable importance estimators. We show thcoretically and empirically that CPI overcomes the limitations of standard permutation importance by providing accurate type-I error control. When used with a deep neural network, CPI consistently showed top accuracy across benchmarks. An experiment scale medical dataset showed that CP/ provides a more parsimonious selection statistically significant variables. Our results suggest that CPI can be readily used as drop-in replacement for permutation-based methods.
real-world data analysis in large-
Introduction
Machine learning is an area of growing interest for biomedical research [Iniesta et al. 2016 Taylor
and Tibshirani 2015 Malley et al. 2011 for predicting biomedical outcomes from heterogeneous
inputs [Hung et al. 2020 Zheng and Agresti 2000, Giorgio et al., 2022 Sechidis et al. 2021
Biomarker development mereasingly focusing multimodal data inc including brain images, genetics,
biological specimens and behavioral data [Coravos et al. 2019 Siebert 2011 Ye et al. 2008 Castillo Barnes al. 2018 Yang al. 2022). Such high-dimensional settings with correlated inputs put strong pressure on model identification. With complex, often nonlinear models, it becomes harder to assess the role of features in the prediction, aka variable importance [Casalicchio al. 2019 Altmann et al. 2010). In cpidemiological and clinical studies, one is interested in population-level Teature importance, opposed instance-level feature importance.
In that context, variable importance is understood as conditional importance, meaning that it measures the information carried by variable the outcome given the others, opposed to the easily accessible marginal importance of the variables. Conditional importance is necessary e.g. to assess
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

whether given measurement is worth acquiring, on top of others, for a diagnostic or prognostic task. As the identification of relevant variables is model-dependent and potentially unstable. point estimates of variable importance are misleading. One needs confidence intervals of importance estimates or statistical guarantees, such as type-I error control, i.e. the percentage of nen-velevant variables detected as relevant (false positives). This control depends on the accuracy of the p-values on variable importance being non-zero [Cribbic 2000].
2022 a popular Within the family of removal-based importance assessment methods [Covert et al. model-agnostic approach is permutation variable importance, that measures the impact of shuffling given variable on the prediction [Janitza et al. (2088). By repeating the permutation importance analysis on permuted replicas of the variable of interest, importance values can be tested against the null hypothesis of being zero, yielding p-values that are valid under general distribution assumptions. Yet, statistical guarantees for permutation importance assessment do not hold in the presence of correlated variables, leading to selection of unimportant variables [MoInar et al. [221] Hooker et al., 2021, Nicodemus et al., 2010, Stigler 2005]. For instance, the method proposed [Mi et al. 2021] a powerful variable importance evaluation scheme, but it does not control the rate of type-I error.
In this work, we propose general methodology for studying the properties of Conditional Permuta- tion Importance in biomedical applications alongside tools for benchmarking variable importance estimators:
Building on the previous literature on CPI, we develop theoretical results for the limita- tions regarding Permutation Importance (PI) and advantages of conditional Permutation Importance (CPI) given correlated inputs (section 3
We propose a novel implementation for CPI allowing us to combine the potential advantages of highly expressive base learners for prediction (a deep neural network) and a comparably lean Random Forest model as a conditional probability learner (section 4
We conduct extensive benchmarks on synthetic and heterogeneous multimodal real-world biomedical data tapping into different correlation levels and data-generating scenarios for both classification and regression (section 5)
We propose reusable library for simulation experiments and real-world applications of our method on a public GitHub repo https: /tt:://gitttt wow.accammm2233//////ca /2rr///aa///aaaeaeea
Importance
2 Related work
popular approach to interpret black-box predictive models is based on locally interpretable, i.e. instance-based, models. LIME (Ribeiro et al. 2016] provides local interpretable madel-agnostic explanations by locally approximating a given complex model with a linear model around the instance of interest. SHAP [Burzykowski 2020) is popular package that measures local feature effects using the Shapley values from coalitional game theory.
However, global, i.e. population-level, explanations are better suited than instance-level explanations for epidemiological studies and seientific discovery in general. Many methods can be subsumed under the general category of removal- based approaches [Covert et al. [2022]. Permutation importance is defined as the decrease model score when the values of a single feature are randomly shuffled [reiman| [001]. This procedure breaks the relationship between the feature and the outcome, thus the drop model performance expresses the relevance of the feature. Janitza et al. [2018) use an ensemble of Random Forests with the sample space equally partitioned. They approximate the null distribution based on the observed importance scores to provide p-values. Yet. this coarse estimate of the null distribution can give unstable results. Recently, generic approach has been proposed in [Williamson et al. 202T that measures the loss difference between models that include or exclude a given variable, also applied with LOCO (Leave One Covariate Out) in the work by Lei et al. 2018). They show the asymptotic consistency of the model. However, their approach is intractable, given that it requires refitting the model for each variable. A simplified version has been proposed by Gao et al.) [002). However, relying on linear approximations, some statistical guarantees from [Williamson
et al. 2 are potentially lost.
Another recent paper by Mi et al. [202T] has introduced model-agnostic explanation for black-box models based on the permutation approach. Permutation importance [Breiman 2001) can work with
2

---

any learner. Moreover, it relies on single model fit, hence it is an efficient procedure. Strobl et al. 2008) pointed out limitations with the permutation approach in the face of correlated variables. As an alternative, they propose conditional permutation importance by shuffling the variable of interest conditionally on the other variables. However, the solution was specific to Random Forests, as based on bisecting the space with the cutpoints extracted during the building process of the forest.
With the Conditional Randomization Test proposed by Candes et 2017]. the association between the outcome and the variable of interest x conditioned on x is estimated. The variable of interest is sampled conditionally on the other covariates multiple times to compute a test statistic and p-values. However, this solution is limited to generalized linear models and is computationally expensive. Finally, a recent paper by [Watson and Wright 2021] showed the necessity of conditional schemes and introduced knockoff sampling scheme, whereby the variable of interest is replaced by its knockoff to monitor any drop in performance of the leaner used without refitting. This method computationally inexpensive, and enjoys statistical guarantees from from [Lei et al. [2088). However, depends on the quality of the knockoff sampling where even a relatively small distribution shift in knockoff generation can lead to large errors inference time.
Other work has presented comparisons of select models within distinct communities Liu al. 2021 Chipman et al.) 2010 Janitza el al., 2018 Mi et al. 202T Altenmüller et al. [021], however, lacking conceptualization from a unified perspective. In summary, previous work has established potential advantages of conditional permutation schemes for inference of variable importance. Yet, the lack of computationally scalable approaches has hampered systematic investigations of different permutation schemes and their comparison with alternative techniques across broader range of predictive modeling settings.
Permutation importance and its limitations
3.1 Preliminaries
Notations We will use the following system of notations. We denote matrices, vectors, scalar variables and sets by bold uppercase letters, bold lowercase letters, script lowercase letters, and calligraphic letters, respectively (e.g. X, x, x, X). We call μu the function that maps the sample space X C R' the sample space C I and j an estimate of μ. Permutation procedures will be represented by (perm). We denote by [n|] the set n.).
Let X E RTxp be a design matrix where the i-th row and the j-th column are denoted xạ and x respectively. Let X = (x1 ...... x² x ..... xP) be the design matrix, where the jth column is removed, and ( = (xl xl {xd]nerm xi+1 xP) the design matrix with the jth column shuffled. The rows of X-j and x() are denoted x, and x(") respectively, for e [n]].
Problem setting Machine learning inputs are design matrix X and a target y E R or E f0, 1)" depending on whether it is regression or a classification problem. Throughout the paper, we rely on an i.i.d. sampling train / test partition scheme where the n samples are divided into Arrain training and Neast test samples and consider that X and y are restricted to the test samples the training samples were used to obtain jt.
3.2 The permutation approach leads to false detections in the presence of correlations
A known problem with permutation variable importance is that if features are correlated, their importance is typically over-estimated [Strobl et al., 2008). leading loss of type-I error control. However, this loss has not been precisely characterized yet, which we will work through for the linear case. We use the setting of [Mi et al., (02T], where the estimator ft, computed with empirical risk minimization under the training set, is used to assess variable importance on a new set of data (test set). We consider a regression model with a least-square loss function for simplicity. The importance of variable x is computed as follows:
Riesi
m
- ((x))((()))= (v: -((xi))2)
(I)
Wlast
3

---

Let E; - ((xx)) for i [[tees]. I Re- Re-arranging terms yields
m
- Mxxx - f(x1) - (xxx + 26,).
Deest
1=1
Mi et al. [022]] argued that these terms vanish when Tltest - co. But it is not the case as long as the training is fixed. In order to get tractable computation, we assume that μ and j are linear functions: xx) = xw and jx() = xw. Let us further consider that x is a null feature, i.e. w O. This yields xw = w'w' + JJ = x-Jw Denoting the standard dot product by (...), this
leads to (Detailed proof of getting from Eq. 2 to Eq. can be found in supplement section A
2 (xe)
i
{xx) perm x x--(w- w- + E)
Ntest
as (||=)|| 2 [ = O. Next, 1 (fxl)perm X J(w w أ)) - and {x) peerm, E) + 0 when Frrsst that the first three moments of these quantities are bounded and that the test samples are i.idd Let us assume that the correlation within X takes the following form: x = X u--to 8, where u E IR and
(x)
Aiseat
from the Berry-Essen theorem, assuming with speed Vilceal
5 is random vector independent of X By contrast,
21d"'
(x)) X
w)) has non-zero [
Preeve
limit 2nd u Tu Cooo(X (w w 3), where Coo(X
(remember that both w and w - are fixed, because the training set is fixed). Thus, the permutation importance of null but correlated variable does not vanish when Meeat + oo, implying that this inference scheme
lim
Pheast
will lead to false positives.
4
Conditional sampling-based feature importance
4.1 Main result
We define the permutation of variable Iº conditional to xJ_ as a variable j/ that retains the dependency of z with respect to the other variables in x but where the independent part is shuffled; x3) is the vector x where x² is replaced by ă², We propose two constructions below (see Fig. ET In the case of regression, this leads to the following importance estimator:
Deest
m
Σ (M
((x)))²)
Thast
As noted by Watson and Wright [202], this inference is correct, as in traditional permutation tests, as long as one wishes to perform inference conditional to ft. However, the following proposition states that this inference has much wider validity in the asymptotic regime.
Proposition. Assuming that the estimator fe is obtained from class of functions F with sufficient regularity, iee. that meets conditions (AI, A2, A3, A4, Bl and B2) defined in supplementary material, the importance score meri defined in (4) cancels when Revain - 30 and Deest - co under the null hypothesis, i.e. the j-th variable is not significant for the prediction. Moreover, the Wald statistic 2 mean((ieee)) obtained by dividing the mean of the importance score by its standard deviation std(merr))
urymptotically follows a standard normal distribution.
This implies that the large sample limit, the p-value associated with 2 controls the type-I error rate for all optimal estimators in F.
The proof of the proposition is given in the supplement (section C It consists in observing that the importance score defined in (4) is 0 for the class of learners discussed in [Williamson et al. 2021]. namely those that meet a certain of convergence guarantees and are invariant arbitrary change of their jth argument, conditional on the others. In the supplement, we also restate the precise technical conditions under which the importance score vigrr used is (asymptotically) valid, i.e. leads to Wald-type statistic that behaves as a standard normal under the null hypothesis.
It is easy to see that for the setting in Sec. 3.2 all terms in Eq. vanish with speed Viteat"

---

4.2 Practical estimation
Next, we present algorithms for computing conditional permutation importance. We propose two con- structions for 25, the conditionally permuted counterpart of x² The first one is additive: on test sam- ples, x is divided into the predictable and random parts .² = B(x x) + (ar E(r ()) ) where the residuals of the regression of x on x are shuffled to obtain ø In practice, the expec- tation is obtained by a universal but efficient estimator, such as a random forest trained on the test set.
The other possibility consists in using random forest (RF) model to xy from x and then sample the prediction within leaves of the RF
Random shuffling is applied B times. For instance, using the additive construction, shuffling of the residuals z for given E [B] allows to reconstruct the variable of interest as the sum of the predicted version and the shuffled residuals, that
xJ.
+ J..
(5)
xP) R x be the new design matrix including the reconstructed version of the variable of interest x!. Both X and the target vector y are fed to the loss function in order to compute loss score
Let Xi.b = (x' x xi+1
12.0 R defined by
SGO S(GP)
1-3(8)) + (1 - 1i) log (-7()))
log
(6)
(3/1
77 )²
33 01)²
for binary and regression cases respectively where i E [ E [p], b E [B]] i indexes test sample of the dataset, ú: ((x;)) and of p(x(1)) is the fitted value following the reconstruction of the variable of interest with the 5t residual shuffledd and S(r) T+e
The variable importance scores are computed as the double average over the number of permutations B and the number of test samples Nsaest (line 15 of Alg. D. while their standard deviations computed as the square root of the average over the test samples of the quadratic deviation over the number of permutations (line 17 Note that, unlike Williamson et al. (202T), the variance estimator non-vanishing, and thus can be used as a plugin. ECPI statistic is then computed by dividing the mean of the corresponding importance scores with the corresponding standard deviation (8.. P-values are computed using the cumulative distribution function of the standard normal distribution (line 19). The conditional sampling and inference steps are summarized in Algorithm [ This leads to the CPI-DNN method when jí is a deep neural network, or CPI-RF when ju is a random forest. Supplementary analysis reporting the computational advantage of CPI-DNN over alternative a.k.a. LOCO-DNN, can be found in supplement (section D) which justifies its computational leanness.
Experiments & Results
In all experiments, refer to the original implementation of the different methods in order to maintain a fair comparison. Regarding Permfit-DNV CPI-DNN and CPI-RF models specifically, our implementation involves 2-fold internal validation (the training set of further split to get validation set for hyperparameter tuning). The scores from different splits are thus concatenated to compute the final variable importance. We focus the Permfit- -DN and CPI-DNN importance estimators that use a deep neural network as learner f., using standard permutation and algorithm|] [ respectively. All experiments performed with 100 runs. The evaluation metrics are detailed in the supplement (section P.
5.1 Experiment 1: Type-I error control and accuracy when increasing variable correlation
We compare the performance of CPI-DNN with that of Permfi-DNN by applying both methods across different correlation scenarios. The data {x1] follow Gaussian distribution with a prescribed covariance structure Σ i.e. x; NO Σi [m]]. We consider a dlckkddsiggeed covariance matrix Σ of 10 blocks with equal correlation coefficient p {0, 0.2, 0.5, 0.8) among the variables of
5

---

Algorithm 1 Conditional sampling step: The algorithm implements the conditional sampling step place of the permutation approach when computing the p-value of variable nJ
Require: X E R xp, y E ER ju: estimator, 1: loss function, RFy: learner trained to predict x from x-
< number of permutations
13: end for
2: X t X with j-th column removed
14: end for
3: for fori= 1 to Reest do
15: mean(m
1/.6
4: :/ t- R-RFF R
Teeest
b=1
5: end for
6: Residuals eع t- xi x3
16: 7. =
1
Σ
p.b
mean((foor))
B
7: for b = to B do
8: Z..b +- Random Shuffling(e)
17: std(m))rr)
Piast
9: xl.b t x2 + 2
Deeust
10: for to nratt do
1=1
5 - ((x{1))
18: =<P1- = mmmam mpp
12:
compute 1000
19: p² + 1-1 cdf (c(P1)
A
CPI-DNN Permiit-DNN
о
coooooooo
Correlation strength
CPI-DNN
foooa..oo
Permfit-DNN
CPI-DNN
po-1
0:00:00
Permfit-DNN
00000.0
Method
CPI-DNN
CPI-DNN
1.00
Permfit-DNN
Permfit- DNN
0.1
0.2
0.3
0.4
0.50
0.60
0.70
0.80
0.90
1.00
Type-1 error
AUC
Figure 1: CPI-DNN Permfit-DNN: Performance detecting important variables simulated
data with n = 300 and p= 100. (A): The type-I error quantifies to which extent the rate of low p-values (p < 0.05) exceeds the nominal false positive rate. (B): The AUC score measures which extent variables are ranked consistently with the ground truth. Dashed line: targeted type-I error rate. Solid line: chance level.
each block. In this experiment, p = 100 and n = 300. The first variable of each of the first blocks is chosen to predict the target y with the following model, where (O, I):
y/ = - + log (1 + 2(x + (x?" + 1)2) + x x11 + Ess E [n]]
The AUC score and type-I error are presented in Fig. I Power and computation time are reported in the supplement Fig. I ISSI Based on the AUC scores, Permfit-DNN and CPI-DNN showed virtually identical performance. However, Permfit-DNN lost type-I error control when correlation in X is increased, while CPI-DNN always controlled the type-I error at the targeted rate.
5.2 Experiment 2: Performance across different settings
In the second setup, we check if CPI-DNN and Permfit-DDe DNN control the type-I error with an increasing total number of samples n. The data are generated as previously, with a correlation p = 0.8. We fix
the number of variables p to 50 while the number of samples n increases from 100 to 1000 with a step size of 100. We use 5 different models to generate the outcome y from X: classification, Plain linear, Regression with ReLu, Interactions only and Main effects with interactions. Further details regarding each data- generainn scenario can be found in supplement (section G)

---

Method
Scenario
CPI-DNN Permfie-NM
Classification
Plain linear
Regi regression with ReLu
Interactions only
Main effects and lateractions
Type- Temor
0.4
0.6
0.4
0.5
0.2
0.0
0.0
0.0
0.0
400
400
800
400
800
400
800
400
800
AUC B
1.0
1.0
1.0
1.0
1.0
0.8
0.8
0.9
0.8
0.8
0.6
0.6
0.6
0.6
0.4
0.4
0.4
0.4
0.4
400
800
400
800
400
800
400
800
400
800
Number samples
Figure 2: Model comparisons across data-generating scenarios: The (A) type-I eπror and (B) AUC scores of Permfit-DNN and CPI-DNN are plotted as function of sample size for five different settings. The number of samples increased from 100 to 1000 with step size of 100. The number of variables p was set to 50. Dashed line: targeted type-I error rate. Solid line: chance level.
The AUC score and type-I error of Permfi-DNN and CPI-DNN are shown function of sample size Fig. 2 The accuracy of the two methods was similar across data-generating scenarios, with slight reduction in the AUC scores of Permfit-DNN as compared to CPI-DNN. Only CPI-DNN controlled the rate of type-I error the different scenarios at the specified level of 0.05. Thus, CPI-DNN provided accurate ranking of the variables according their importance score while, at the same time, controlling for the type-I error in all scenarios.
5.3 Experiment 3: Performance benchmark across methods
In the third setup, we include Permfit-DNN -NN and CPI-DNN in a benchmark with other state-of-the-an methods for variable importance using the same setting as in Experiment 2, while fixing the total number of samples to 1000. We consider the following methods:
Marginal Effects: A univariate linear model is fit explain the response from each of the variables separately. The importance scores are then obtained from the ensuing p-values.
Conditional-RE [Stobb et al. 2008 A conditional variable importance approach based on Random Forest model. This method provides p-values.
CRCT [Liu et al. distillation, using sparse linear or logistic learner.
2021 Nguyen et al. 2022 The Conditional Randomization Test with
Lazy VI [Gao et al.
2022).
Permfit-DNN Mi et al. 2021
LOCO Lei et al.
2018]:
This method applies the remove-and-retrain approach.
cpi-knockoof [Watson and Wright,, 2021]: Similar to CPI-RE, but permutation steps are replaced by sampling step with knockoff sampler.
CPI-RF: This corresponds to the method in Alg. [ where ğ is Random Forest.
CPI-DNN: This corresponds to the method in Alg. where fi is DNN.
The extensive benchmarks on baselines and competing methods that provide p-values are presented in Fig. 3 For type-I error, do CRT, CPI-RF. CPI-DNN, LOCO and cpi-knockoff provided reliable control, whereas Marginal effects, Permfit-DNN Conditional-RR and Lazy VI showed less consistent results across scenarios. For AUC, we observed that marginal effects performed poorly, as they do not proper predictive model. LOCO and cpi-knockoff behave similarly. do CRT performed well when the data- tata-generating model was linear and did not include interaction effects, Conditional-F and CPI-RF showed reasonable performance across scenarios. Finally, Permfit- DNN and CPI-DNN outperformed all the other methods, closely followed by Lazy VI.
Additional benchmarks popular methods that do not provide p-values, e.g. BART Chipman et al. 2010 or local and instance-based methods such as Shapley values [Kumar al. 2020), are reported the supplement (section H.. The performance of these methods in terms of power and computation
7

---

Scenario
o Main effects and Interactions
Interactions only
Regression with ReLu
Plain linear
Classification
A
Method
CPI-DNN Permfit-DNN
CPI-RF cpi-knockoff
LOCO
Lazy VI Conditional-RF dOCRT
Marginal
1.0
Type-1 error
9
090
0.2
0.0
004R0
50000
Q.Q...
B
1.0
0.9
AUC
0.7
0.6
o 00980
P00
0.5
0.4
Figure 3: Extended model comparisons: CPI-DNN and Permfit-DNN were compared to baseline models (outer columns) and competing approaches across data-generating scenarios (inner columns). Prediction tasks were simulated with 1000 and p 50. (A): Type-I error. (B): AUC scores. Dashed line: targeted type-I error rate. Solid line: chance level.
time are reported the supplement Figs. 3 & 3 respectively. Additional inspection of power showed that across data generating scenarios, CPI-DNN, Permfit-DNN and conditional-Re showed strong results. Marginal and dOCRT performed only well scenarios without interaction effects. CPI-RF, cpi-knockoff. LOCO and Lazy VI performed poorly. Finally, to put estimated variable importance in perspective with model capacity, we benchmarked prediction performance of the underlying learning algorithms in the supplement Fig. S4
5.4 Experiment 4: Permfit-DNN CPI-DNN on Real Dataset UKBB
Large-scale simulations comparing the performance of CPI-DNN and Permfit-DNN are conducted in supplement (section D We conducted an empirical study of variable importance in a biomedical application using the non-conditional permutation approach Permfit-DNN (no statistical guarantees for correlated inputs) and the safer CPI-DNN approach. A recent real-world data analysis of the UK Biobank dataset reported successful machine learning analysis of individual characteristics. The UK Biobank project (UKBB) curates phenotypic and imaging data from prospective cohort of volunteers drawn from the general population of the UK [Constantinescu et al. [0222]. The data is provided by the UKBB operating within the terms of an Ethics and Governance Framework. The work focused age. cognitive function and mood from brain images and social variables and put the ensuing models in relation to individual life-style choices regarding sleep, exercise, alcohol and tobacco (Dadi al. 2021]
coarse analysis of variable importance was presented, which entire blocks of features were removed. It suggested that variables measuring brain structure or brain activity were less important for explaining the predictions of cognitive or mood outcomes than aocio-demographic characteristics. On the other hand, brain imaging phenotypes were highly predictive of the age of person, in line with the brain-age literature [Cole and Franke 2017]. In this benchmark, we explored variable-level importance rankings provided by the CPI-DNN and PermfitDNN methods.
The real-world empirical benchmarks on predicting personal characteristics and life-style are summa- rized in Fig. 4 Results in panel (A) suggest that highest agreement for rankings between CPI-DNN and Permfit-DNN was achieved for social variables (bottom left, orange squares). At the same time, CPI-DNN flagged more brain-related variables as relevant (bottom right, circles). We next computed counts and percentage and broke down results by variable domain (Fig. B). Naturally, the total relevance for brain versus social variables varied by outcome. However, seemed more selective as flagged fewer variables as important (blue) beyond those flagged as impor- tant by both methods (light bluc). This was pronounced for social variables where CPI-DNN
tendency, CPI-DNN

---

Brain
Social
domain
Age B
Relevance
Rank (CPI-DNN)
60
Social
Brain
Alsahol
Important
Cai
40
Relevance
Cpi
Outzome
Cognition
Unimpprtant
Permfit
-20
Important Permftt
Exercise
Mood
Unimportant
0
Smoke
20
0%
40% 80% 0%
40%
80%
Rank (PermfitDNN)
Percentage
Percentage
Figure 4: Real-world empirical benchmark: Prediction of personal characteristics (age, cognition, mood) and life-style habits (alcohol consumption, sleep, exercise & smoking) from various sociode- mographic and brain-imaging derived phenotypes in sample of n 8357 volunteers from the UK Biobank. (A) plots variable rankings for Cermfit-DNN (x axis) versus CPI-DNN (y axis) across all outcomes. Color: variable domain (brain versus social). Shape: variables classified by both methods as important (squares), unimportant (crosses) or by only one of the methods, i.e., CPI-DNN (circles) or Permfit-DNY (triangles). (B) presents a detailed breakdown of percentage and counts of variable classification split by variable domain.
sometimes added no further variables. As expected by the impact of aging on brain structure and function, brain data was most important for age- pre-pddiitionn compared to other outcomes. Interestingly, most disagreements between the methods occurred in this setting as CPI rejected 16 out of 66 brain inputs that were found as important by Permfit. This outlines the importance of correlations between brain variables, that lead spurious impostanc importance findings with Permfit. We further explored the utility of our approach for age-prediction from neuromagnetic recordings (Engemann et al.) [020)] and observed that CPI-DNN readily selected relevant frequency bands without fine-tuning the approach (section M in the supplement).
Discussion
In this work, we have developed framework for studying the behavior of marginal and conditional permutation methods and proposed the CPI-DNN method, that inspired by the limitations of the Permfit-DNN approach. Both methods build on top of an expressive DNN learner, and both methods turned superior competing methods detecting relevant variables, leading high AUC scores across various simulated scenarios. However, our theoretical results predicted that Permfi-DNN would not control type-I errоr with correlated data, which was precisely what our simulation-bart based analyzes confirmed for different data-generating scenarios (Fig. ! 2). Other popular methods (Fig. B) showed similar failures type-I error control across scenarios or only worked well in subset of tasks. Instead, CPI-DNN achieved control of type-I errors by upgrading the permutation to conditional permutation. The consequences were pronounced for correlated predictive features arising from generative models with product terms, which was visible even with small fraction of data points for model training. Among alternatives, the Lazy approach [Gao al. 2022] obtained an accuracy almost good as Permfit- DNN and CPI-DNN but with an unreliable type-I error control.
Taken together, our results suggest that CPI-DNN may be practical default choice for variable importance estimation in predictive modeling. A practical validation of the standard normal dis- tribution assumption for the non important variables can be found in supplement (section Np. The CPI approach is generic and can be implemented for any combination of learning algorithms as base learner or conditional means estimator. CPI-DNN has a linear and quadratic complexity in the number of samples and variables, respectively. This is of concern when modeling the conditional distribution of the variable of interest which lends itself high computational complexity. In our work, Random Forests proced to be useful default estimators as they are computationally lean and their model complexity, given reasonable default choices implemented standard software, can be well controlled by tuning the tree depth. In fact, our supplementary analyses (section O) suggest that proper hyperparameter tuning was sufficient obtain good calibration of p-values. As potential limitation, it is noteworthy the current configuration of our approach uses a deep neural network as

---

the base learner. Therefore, in general, more samples might be needed for good model performance, hence, improved model interpretation.
Our real-world data analysis demonstrated that CPI-DNN readily applicable, providing similar variable rankings Permfit-DNN. The differences observed hard to judge as the ground truth is not known in this setting. Moreover, accurate variable selection is important to obtain unbiased interpretations which are relevant for data-rich domains like econometries, epidemiology, medicine, genetics or neuroscience. In that context, it is interesting that recent work raised doubts about the signal complexity in the UK biobank dataset [Schulz al. 2020 which could mean that underlying predictive patterns are spread out over correlated variables. In the subset of the UK biobank that we analysed, most variables actually had low correlation values (Fig. [4), which would explain why CPI-DNN and Cermfit-DNN showed similar results. Nevertheless, our empirical results seem compatible with our theoretical results as CPI-DNN flagged fewer variables as important, pointing at stricter control of type-l errors, which is welcome property for biomarker discovery.
When considering two highly correlated variables X1 and X2, the corresponding conditional impor- lance of both variables is O. This problem is linked to the very definition of conditional importance, and to the CPI procedure itself. The only workaround is eliminate, prior to importance analysis, degenerate cases where conditional importance cannot be defined. Therefore, possible future directions include inference on groups of variables, e.g. gene pathways, brain regions, while preserving statistical control offered by CPI-DNN.
the KARAIB AI chair This work has been supported by Bertrand Thirion and is supported by and the H2020 Research Infrastructures Grant EBRAIN- 101058516. D.E. is full-time employee of F. Hoffmann-Laa Roche Ltd.
Acknowledgement
References
Marlene Sophie Altenmüller, Leonie Lucia Lange, and Mario Gollwitzer. When research is me-search: How researchers' motivation to pursue topic affects laypeople's 's trust in science. PLoS One, 16 (7):00253911 1, July 2021. ISSN 1932-6203. doi: 10. 1371
André Altmann, Laura Toloşi, Oliver Sander, and Thomas Lengauer. Permutation importance: A corrected feature importance measure. Bioinformatics, 96(10):1340-1347 May 2010. ISSN 1367-4803. doi: 10. 2..199//ioiiffomaaii//ttgg
Andrew P. Bradley. The use of the area under the ROC curve in the evaluation of machine learning algorithms. Pattern Recognition, 30(7): 1159, July 1997. ISSN 0031-3203. doi: 10.1016/ 50033--3003(9600112222
Leo Breiman. Random Forests. Machine Learning, 45(1):5-32, October 2001. ISSN 1573-0565. doi: 10.1023/4110109334444444
Przemyslaw Biecek and Tomasz Burzykowski. Explanatory Model Analysis. December 2020.
Emmanuel Candes, Yingying Fan, Lucas Janson, and Jinchi Lv. Panning for Gold: Model-X Knockoff's for High-dimensional Controlled Variable Selection. arXiv: 1rXiv:/610.023// [math, stat], December 2017.
Giuseppe Casalicchio, Christoph Molnar, and Bernd Bischl. Visualizing the Feature Importance for Black Box Models. In Michele Berlingerio, Francesco Bonchi, Thomas Gärtner, Neil Hurley, and Georgiana Ifrim, editors, Machine Learning and Knowledge Discovery Databases, Lecture Notes in Computer Science, pages 655-670, Cham, 2019. Springer International Publishing. ISBN 978-3-030--1022-77 doi: 10. 111007/978-3-0341 (0925-7_40)
Diego Castillo-Barnes, Javier Ramírez, Fermín Segovia, Francisco J. Martinez-Murcia, Diego Salas- Gonzalez, and Juan M. Górriz. Robust Ensemble Classification Methodology for 1123-lofee SPECT Images and Multiple Heterogeneous Biomarkers in the Diagnosis of Parkinson's 's Disease. Frontiers in Neuroinformatics, 12:53, 2018. ISSN 1662-5196. doi: 20.3388//min.22188.00 00053..
Hugh A. Chipman, Edward George, and Robert E. McCulloch. BART: Bayesian additive regression trees. Ann. Appl. Stat., 4(1), March 2010. ISSN 1932-6157. doi: 10.121409--AAA285.
10

---

James H. Cole and Katja Franke. Predicting Age Using Neuroimaging: Innovative Brain Ageing Biomarkers. Trends Neurosci, 40(12))681-69)) -600, December 2017. ISSN 1878-108X. doi: 10.1016/j. tins. (i..2007..10.00
Andrei-Enin Constantinescu, Ruth E. Mitchell, Jie Zheng, Caroline J. Bull, Nicholas J. Timpson, Borko Amulic, Emma E. Vincent, and David A. Hughes. A framework for research into continental ancestry groups of the UK Biobank. Human Genomics, 16(1):3, January 2022. ISSN 1479-7364.
doi: 10.1 16//400266--22-00338---
Andrea Coravos, Sean Khozin, and Kenneth D. Mandl. Developing and adopting safe and effective digital biomarkers to improve patient outcomes. npj Digit. Med., 2(1):1-5, March 2019. ISSN 2398-6352 doi: (0.10388641144-009-00099
Ian Covert, Scott Lundberg, and Su-In Lee. Understanding Global Feature Contributions With Additive Importance Measures, October 2020.
Ian Covert, Scott Lundberg, and Su-In Lee. Explaining by Removing: A Unified Framework for Model Explanation, May 2022.
Robert A. Cribbie. Evaluating the importance of individual parameters in structural equation modeling:
The need for type error control. Personality and Individual Differences, 29()):66-5777, September 2000. ISSN 0191-8869,
doi: 10. (1101///0111-8889999900000
Kamalaker Dadi, Gaël Varoquaux, Josselin Houenou, Danilo Bzdok, Bertrand Thirion, and Denis Engemann. Population modeling with machine learning can enhance measures of mental health. GigaScience, 10(10)giaa007,, October 2021. ISSN 2047-217X. doi: (0.100/gigaccinnee/iaa0771.
Denis A Engemann, Olch Kozynets, David Sabbagh, Guillaume Lemaître, Gael Varoquaux, Franziskus Liem, and Alexandre Gramfort. Combining magnetoeneephalography with mag- netic resonance imaging enhances learning of surrogate-biomarkess. eLife, 9ce54055, May 2020. ISSN 2050-084X. doi: 10.7554/eLife.54055.
Yue Gao, Abby Stevens, Rebecca Willet, and Garvesh Raskutti. Lazy Estimation of Variable Importance for Large Neural Networks, July 2022.
Joseph Giorgio, William J. Jagust, Suzanne Baker, Susan M. Landau, Peter Tino, Zoe Kourtzi, and Alzheimer's Disease Neuroimaging Initiative. A robust and interpretable machine learning approach using multimodal biological data to predict future pathological tau accumulation. Nar Commun, (3(1): 1887, April 2022. ISSN 2041-1723. doi: 10. 10.1038//4 1477-022-28795-77
Giles Hooker, Lucas Mentch, and Siyu Zhou. Unrestricted Permutation forces Extrapolation: Variable Importance Requires least One More Model, or There Is No Free Variable Importance, October 2021.
Jui-Long Hung, Kerry Rice, Jennifer Kepka, and Juan Yang. Improving predictive power through deep learning analysis of K-12 online student behaviors and discussion board content. In- formation Discovery and Delivery, 48(4):199-21 212, January 2020. ISSN 2398-6247. doi:
10.1 10.1108/IDD-22 2-W-DDDD---222200199999
R. Iniesta, D. Stahl, and P. McGuffin. Machine learning, statistical learning and the future of biological research in psychiatry. Psychol Med, 46(12):2455-244) September 2016. ISSN 1469-8978. doi:
10.1077/0003329116000033
Silke Janitza, Ender Celik, and Anne-Laure Boulesteix. A computationally fast variable importance test for random forests for high-dimensional data, Adv Data Anal Classif, 12(4):885-915, December 2018. ISSN 1862-5355. doi: 10. 10.10077/1116444-1--77777
I. Elizabeth Kumar, Suresh Venkatasulrramanian Carlos Scheidegger, and Sorelle Friedler. Problems with Shapley-value- hased explanations as feature importance measures, June 2020.
Jing Lei, Max G'Sell, Alessandro Rinaldo, Ryan J. Tibshirani, and Larry Wasserman. Distribution- Free Predictive Inference for Regression. Journal of the American Statistical Association, 113 (524):1094-1111 July 2018. ISSN 0162-1459. doi: 10. 10.1000011224459.0 4000

---

Molei Liu, Eugene Katsevich, Lucas Janson, and Aaditya Ramdas. Fast and Powerful Conditional Randomization Testing via Distillation. arXiv:2006. 03980 [stat], June 2021.
Gilles Louppe, Louis Wehenkel, Antonio Sutera, and Pierre Geurts. Understanding variable impor- tances in forests of randomized trees. In Advances in Neural Information Processing Systems, volume 26. Curran Associates, Inc., 2013.
James D. Malley, Karen G. Malley, and Sinisa Pajevic. Statistical Learning for Biomedical Data. Cambridge University Press, February 2011. ISBN 978-1-139-49885-8
Xinlei Mi, Baiming Zou, Fei Zou, and Jianhua Hu. Permutation-basel identification of important biomarkers for complex diseases via machine learning models. Nat Commun, 12(1):3008, May 2021. ISSN 2041-1723. doi: 10. 1038//41146777 467-0022226-
Christoph Molnar, Gunnar König. Julia Herbinger, Timo Freiesleben, Susanne Dandl, Christian A. Scholbeck, Giuseppe Casalicchio, Moritz Grosse- Wentrup, and Bernd Bischl. General Pitfalls of Model-Agnostic Interpretation Methods for Machine Learning Models, August 2021.
Binh T. Nguyen, Bertrand Thirion, and Sylvain Arlot. A Conditional Randomization Test for Sparse Logistic Regression in Nigh--Dmension May 2022.
Tuan-Binh Nguyen, Serômee-Aeeii Chevalier, Bertrand Thirion, and Sylvain Arlot. Aggregation of
Multiple Knockoffs. WrXiv:2002.09266 [math, stat], June 2020.
Kristin K. Nicodemus, James D. Malley, Carolin Strobl, and Andreas Ziegler. The behaviour of random forest permutation-based variable importance measures under predictor correlation. BMC Bioinformatics, 11(1):110, February 2010. ISSN 1471-2105. doi: 10.1 -
Marco Tulio Ribeiro, Sameer Singh, and Carlos Guestrin. "Why Should Trust You?": Explaining the Predictions of Any Classifier, August 2016.
Marc-Andre Schulz, B. T. Thomas Yeo, Joshua T. Vogelstein, Janaina Kather, Konrad Kording, Blake Richards, and Danilo Bzdok. Different scaling of linear models and deep learning in UKBiobank brain images versus machine-learning datasets. Nat Commun, 11 (1):4238, August 2020. ISSN 2041-1723. doi: 10.
Jakob N.
18037-z.
Konstantinos Sechidis, Matthias Kormaksson, and David Ohlssen. Using knockoffs for controlled predictive biomarker identification. Statistics in Medicine, 40(25))5553-5473 2021.
Janet Siebert. Integrated biomarker discovery: Combining heterogencous data. Bioanalysis, 3(21): 2369-2372, November 2011. ISSN 7757-6180. doi: 10.4155/bio.11.221
Stephen Stigler. Correlation and causation: A comment. Perspect Biol Med, 48(1 SuppI):S88-94 2005. ISSN 0031-5982.
Carolin Strobl, Anne-Laure Boulesteix, Thomas Kneib, Thomas Augustin, and Achim Zeileis. Conditional variable importance for random forests. BMC Bioinformatics, 9(1):307, July 2008. ISSN 1471-2105. doi: 10. 121116/14711-205---990
Jonathan Taylor and Robert J. Tibshirani. Statistical learning and selective inference. Proceedings of the National Academy of Sciences, 112(25):762-7766 June 2015. doi: htw.3//paa.11055811111
David S. Watson and Marvin N. Wright. Testing conditional independence in supervised learning algorithms. Mach Learn, 1100))2107-2129, August 2021. ISSN 1573-0565. doi: 10.1007/ s10994-021-0003006
Brian D. Williamson, Peter B. Gilbert, Noah R. Simon, and Marco Carone. A General Framework for Inference on Algorithm-Agnostic Variable Importance. Journal of the American Statistical Association, 0(0):1-14, November 2021. ISSN 0162-1450 1459. doi:
Yuzhe Yang. Yuan Yuan, Guo Zhang, Hao Wang. Ying-Cong Chen, Yingcheng Liu, Christopher G. Tarolli, Daniel Crepeau, Jan Bukartyk, Mithri R. Junna, Aleksandar Videnovic, Terry D. Ellis, Melissa C. Lipford, Ray Dorsey, and Dina Katabi. Artificial intelligence-enabled detection and assessment of Parkinson's 's disease using nocturnal breathing signals. Nat Med, 28110)2207--2215 October 2022. ISSN 1546-170X. doi: 00.1038/441591-02--09993
12

---

Jieping Ye, Kewei Chen, Teresa Wu, Jing Li, Zheng Zhao, Rinkal Patel, Min Bae, Ravi Janardan, Huan Liu, Gene Alexander, and Eric Reiman. Heterogeneous data fusion for alzheimer's disease study. In Proceedings of the 14th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, KDD '08, pages 1025-1033, New York, NY, USA, August 2008. Association for Computing Machinery. ISBN 978- 1-8---6600 1-0-1--605881995 193-4.
doi: 10.1 14..14449110...40000000
B. Zheng and A. Agresti. Summarizing the predictive power of a generalized linear model. Stat Med, doi: 00.102//007700555/20000111001111
19(13):1771-1781, July 2000. ISSN 0277-6715.
hid=sim488533..co2222
13

---

A Supplement proof getting from Eq. to to Eq. 3
me
(n(x)) - (xxx (xxx))) - ((xi) p(x; )= + 25:)
Theest
Risatt
(xp W x2 ui x {1;] perm aه (2x, -w 2x, -w (2) {x.] (هs) + 22:)
Destt
2
Rieest
(x) [xm])arnn [x] ( ( J w 1) + =;) の ((x (fx (
Tteatt
2
Ricat
Jorrm (xy (x- w- + +;)
Titest
B Diagram of CPI constructions
x
Shuftling
X
X
Shuffling
DNN
Mean
p.values
B
Leafs of
Std
Max voting
pla
Sampling
DNN
RF
Leafs xf
Mean voting
Figure E1: CPI-DNN's constructions: Constructing the variable of interest ž is done either (1) by the additive construction (top block) where shuffled version of the residuals is added to the predicted version using the remaining predictors with the mean of a random forest (RF) or (2) by the sampling construction (bottom block) using random forest (RF) model fit x from X and then sample the prediction within the leaves of the RF.
C Conditional Permutation Importance (CPI) Wald statistic asymptotically controls type-I errors: hypotheses, theorem and proof
Outline The proof relies on the observation that the importance score defined in (4) is 0 in the asymptotic regime, where the permutation procedure becomes sampling step, under the assumption that variable is not conditionally associated with y. Then all the proof focuses on the convergence of the finite-sample estimator the population one. To study this,
the framework developed in [Williamson et al. 202T]. Note that the major difference with respect to other contributions [Watson] and Wright 2021 is that the ensuing inference is no longer conditioned on the estimated learner fi. Next, we first restate the precise technical conditions under which the different importance scores considered are asymptotically valid, i.e. lead to a Wald-type statistic that behaves as standard normal under the null hypothesis.
Notations
Let F represent the class of functions from which learner μ x + y is sought.
Let Po be the data data-generating distribution and P.. is the empirical data distribution observed after drawing n samples (noted Atrriin in the main text; in this section, we denote notations). The separation between train and test samples is actually only relevant to alleviate
to simplify
14

---

some technical conditions on the class of learners used. .M is the general class of distributions from which Pi..... Pn: ......PP..
Po are drawn. R := {c(P) P2) c E [0, (0, space of finite signed measures generated by M. Let / be the loss function used to obtain μ.. Given f E F, I(ʃ; Po) = [((f(x),)) ( Pa(z)dz, where = (x, y). Let l40 denote population solution to the estimation problem Mo E argmin pegmingerllla [[; Po) and ii, finite sample estimate Pnn E argmin P.)
M) is the
((xx,))ep,,((())
Let us denote by i(ss, Po; h) the Gâteaux derivative of P + (1, P) at Po in the direction h E R. and define the random function In [ppns Po; dg Po) [m0, Po; 82 Po), where 82 is the degenerate distribution on = (x. y).
Hypotheses
(AI) (Optimality) there exists some constant C' > 0, such that for each sequence M41 M2, E F given that I/ - Poll + 0, [[(,,, Po) - 1( Po)l < Cl - poll3 for each n large enough.
(A2) (Differentiabilityy) there exists some constant K. > 0 such that for each sequence e1, -. IR and h1, /22 ... R satisfying En + and ||h, h60 II + 0, it holds that Mu, Po + enl)n) - (1, Po) i(u,, Po: km))
sup
pET: Ip- p
fn
(A3) (Continuity of optimization) ||AP moll= = O()) for each E R. (A4) (Continuity of derivative) μr + 1(14 Po; h) is continuous at An relative to ||-1|>- for each h E R.
(BI) (Minimum rate of convergence) [A Polly op(n
(B2) (Weak consistency) J In (z))2)P((() = (r()).
(B3) (Limited complexity) there exists some Po Po-Donsker class Go such that Polgn. E Go) 1.
2021 If the above conditions hold, I(ū., Pn) an asymptotically linear estimator of [(4o, Po) and I[in. P.) is non-parametric efficient.
Proposition (Theorem Williamson
Let Pi be the distribution obtained by sampling the j-th coordinate of from the conditional distribution of go (x xx >>)), obtained after marginalizing over y:
S Po(x, y)
qo(2"|x Po dydr'd
Po u) (0 1x (o y)dx' Similarly. let Pa denote its finite-sample counterpart. It turns out from the definition of m/ in Eq. that mipp = U(pn Ph) l(ūn; Pa). It is thus the fimal-sample estimator of the population quantity mèp1 UPo, - U(Ao, Po).
Given that mop: = 1(ūn P.)) - I(Po. PT) - (1(p., P..) - I(Au; Po)) - l(Po: Pt) - U(ao Po), the estimator mpr1 is asymptotically linear and non- paramaraetticc efficient.
The crucial observation is that under the j-null hypothesis. is independent of σd given x Indeed, in that case Po(x, y) 4o (x (o) Pe = Po. Henee, mean/variance of miryy distribution provide valid confidence intervals for m/ and mean(mn)r)) n- standard normal distribution, implying that the ensuing test valid.
and Po(='|x y) Po J)- so that
0. Thus, the Wald statistic & defined in section (4.2) converges to a
In practice, hypothesis (B3), which is likely violated, is avoided by the use of cross-fitting as discussed in [Williamson et al., 2021]: : stated in the main text, variable importance is evaluated set of samples not used for training. An interesting impact of the cross-fitting approach is that it reduces the hypotheses to (A1) and (A2), plus the following two:
(B') I) (Minimum rate of convergence) || - Holls = or(n-1/4 on each fold of the sample splitting scheme.
(B2') (Weak consistency) f g.
= op(1) each fold of the sample splitting scheme.
15

---

D
Computational scaling of CPI-DNN and leanness
Method CPI-DNN
LOCO-DNN
AUC score
Type-I error
Power
Time (seconds)
CPI-DNN
LOCO-DNN
0.4 0.6 0.8 1.0 0.0 0.4
0.8
0.0
0.4
0.8
1,000
100,000
Figure E2: CPI-DNN vs LOCO-DNN: Performance detecting important variables on simulated data with n 1000, 50 and p = 0.8 in terms of (AUC score), Type-I error, Power and Time. Dashed line: targeted type-I error rate. Solid line: chance level.
Computationally lean refers to two facts: (I) there is need to refit the costly MLP learner to predict unlike LOCO-DNN (A removal-basee method provided with our learner) as seen in Fig. Both CPI-DNN and LOCO-DMI achieved high AUC score and controlled the Type-I error highly correlated setting (f=0.8). However, terms of computation time, CPI-DNN is far ahead of LOCO-DNN, which validates
of the permutation scheme. (2) The conditional estimation step involved for the conditional permutation procedure done with an efficient RF estimator, leading to small time difference wrt Permfft-DNN; Overall obtain the accuracy of LOCO-type procedures for the cost of a basic permutation scheme.
E
Evaluation Metrics
AUC score [Bradley 1997]: The variables are ordered by increasing p-values, yielding family of splits into relevant and non-relevant at various thresholds. AUC score measures the consistency of this ranking with the ground truth (Psignats predictive features versus p Psiguats).
Type-I error Some methods output p-values for each of the variables, that measure the evidence against each variable being a null variable. This score checks whether the rate of p-values of null variables exceeds the nominal false positive rate (set to 0.05).
This score reports the average proportion of informative variables detected (when consid- ering variables with p-value 0.05).
Power
Computation time
The average computation time per core on 100 cores.
Prediction Scores As some methods share the same core to perform inference and with the data
divided into train/test scheme, evaluate the predictive power for the different cores on the test set.
16

---

Supplement Figure 1 Power & Computation time
Method
CPI-DNN
Permfit-DNN
A
B
CPI-DNN
0.3
Pemmfi-DNN
Correlation strength
CPI-DNN
CPI-DNN
5
Permfi-DNN
CPL-DNN
0.2
Permfit-DNN
PemafttDNN
CPI-DNN
Permfit-DNN
0.50
300
700
Power
Time (seconds)
Figure 1 S1: Permfit-DNN CPI-DNN: Performance detecting important variables simulated 100. (A): The power reports the average
data under the setting of experiment I, with n = 300 and proportion of informative variables detected (p-value 0.05). (B): The computation time in seconds with (log scale) per core on 100 cores.
Based Fig. ST. both methods Permfit-DNN and CPI-DNN have almost similar power. In high correlation regime, CerffitDDNY yields more detections, but does not control type-I errors (Fig. [ Regarding computation time, CPI-DNN is slightly more computationally expensive than Permftt-DNN.
G Supplement Experiment 5.2 Models
The signal X,3main turned to binomial variables using the probit function 4. gmain and 3ruad are the two vectors with different lengths of regression coefficients having only nsignal = 20 non-zero coefficients, the true model. gmain is used with the main effects while gruad is involved with the interaction effects. Following (Janitza et al. 2018), the B values fmmain gruad) are drawn i.i.d. from the B {#3, =2, +0.5).
Classification
Ys Binomial (((z, )), Vi [[]]]]
Plain linear model We rely on linear model, where gmain is drawn as previously and e is the Gaussian additive noise (O, I) with magnitude a ||xamain 22
Yi = x;gmain +oss, Vi
SNRVn
[n].
Regression with ReLu An extra ReLu function is applied to the output of the Plain linear model: Relulxx,maat + o6), Vi E [n].
Interactions only model We compute the product of each pair of variables. The corresponding values are used as inputs to linear model: y/ quad(x,, gruad) + ae, Vi E [n]]. where
Paianats
quad(æ;, gruad
Equad
The magnitude of the noise is set to l|quad( X, x.ggudd)
SN R,
The non-zero 3quad coefficients drawn uniformly from B.
17

---

Main effects with Interactions We combine both Main and Interaction effects. The magnitude a the noise is set
XX gggri +-qund ,gooad
2; 1/i
X Smaain + quad (quad) t Vi E [m]]
SNR/A
H
Supplement Figure 3 Extended model comparisons
We also benchmarked the following methods deprived of statistical guarantees:
Knockoff's [Candes et al., 2017 Nguyen al., 2020 The knockoff filter is a variable selection method for multivaritte models that controls the False Discovery Rate. The first step of this procedure involves sampling extra null variables that have correlation structure similar to that of the original variables. A statistic is then calculated to measure the strength of the original variables versus their knockoff counterpart. We call this the knockoff statistic {wi 1 that is the difference between the importance of given feature and the importance of its knockoff.
Approximate Shapley values (Burzykowski 2020) SHAP being instance method, relied on an aggregation (averaging) of the per-sample Shapley values.
Shapley Additive Global importancE (SAGE) [Covert et al. 2020]: Whereas SHAP focuses the local interpretation by aiming to explain model's ' individual predictions, SAGE is an extension to SHAP assessing the role of each feature a global interpretability manner. The SAGE values are derived by applying the Shapley value to function that represents the predictive power contained in subsets features.
Mean Decrease of Impurity [Louppe et al. 2013]: The importance scores are related to the impact that each feature has the impurity function in each of the nodes.
BART [Chipman et al. 20 BART is an ensemble of additive regression trees. The trees are built iteratively using back-fitting algorithm such as MCMC (Markov Chain Monte Carlo). By keeping track of covariate inclusion frequencies, BART can identify which components are more important for explaining
Based on AUC, we observe SHAP, SAGE and Mean Decrease of Impurity (MDI) perform poorly, These approaches are vulnerable to correlation. Next, mmockoff-Deep and Knockoff-Laa Lasso perform well when the model does not include interaction effects. BART and Knockoff Bart show fair performance overall.
Scenario
effects
only
Regression ReLu
Plain linear
Claesification
Method
Knockotf-Lasso
Knockoff-Barr
Knockolf- Deep
SHAP
SAGE
MDI
BART
1.0
AUC score
0.9
0.8
0.7
0.6
0.5
Figure S1: Extended model comparisons: State-of-the methods for variable importance providing statistical guarantees in terms of p-values are compared (outer columns) and to competing approaches across data-ged seaareeenntiinggg scenarios (inner columns) using the settings of experiments 2 and 3. Prediction tasks were simulated with n 1000 and p 50. Solid line: chance level.
18

---

I Supplement Figure Power
Scenario
Main eflects and Interactions
Interactions only
Regression with ReLu
Plain linear
Classification
Method
CPI-DNN
PermfilDNN
cpi-knockoff
Lazy Condilional-RF
Marginal
1.0
0.8
000
Power
0.6
0.2
0.0
Figure 3 S2: Extended model comparisons: CPI-DNN and Permfit-DNV compared to baseline models (outer columns) and to competing approaches across data-generating scenarios (inner columns). Convention about power as in Fig. SI Prediction tasks were simulated with 1000 and 50.
Based on the power computation, Permfit-DNN DNN and CPI-DNN outperform the alternative methods. Thus, the use of the right learner leads to better interpretations.
J Supplement Figure 3 Computation time
CPI-DNN
Permfit-DNN
CPI-RF
cpi-knockoof
LOCO
Lazy VI-
Methods
Conditional-RF
d0CRT
Marginal
Knockoff-Lasso
Knockoff-Bart
Knockoff-Deep
SHAP
SAGE
MDI
BART
1020
100
1,000
10,000
Time (seconds)
Figure 3 S3: Extended model comparisons: The computation times for the different methods (with and without statistical guarantees in terms of p-values) reported in seconds with (log 10 scale) per core on 100 cores. Prediction tasks were simulated with n = 1000 and p= 50.
The computation time of the different methods mentioned in this work (with and without statistical guarantees) is presented in Fig. B S3 in seconds with (log 10 scale). First, compare CPI-RF. cpi-knockoff and LOCO based on a Random Forest learner with p=50. We see that cni-knoc and LOCO are faster than CPI-DNN. A possible reason is that CPI-DNN uses inner 2-fold internal validation for hyperparameter tuning (learning rate, LI and L2 regularization) unlike the alternatives. Next, The DNN-based methods (CPI-DNN and Permfit-Ne that control type-I error (d,RRT, cpi-knockoff and 1OCO) despite the use of computationally lean leamners in the latter.
are competitive with the alternatives
19

---

K
Supplement Figure 3 Prediction scores on simulated data
Scenario
Classification
Plain linear
BART
BART
DNN
H/|
DNN
Lasso
Lasso
Marginal
H
Marginal
Random Forest
+|+
Random Forest
0.0
0.4
0.8
0.0
0.4
0.8
Regression with ReLu
Interactions only
BART
BART
DNN
H||
DNN
Lasso
Lasso
Marginal H
Marginal
Random Forest
Random Forest
0.0
0.4
0.8
0.0
0.4
0.8
Prediction score (R2, AUC)
Main effects and Interactions
Data Problem
BART
Classification
DNN
Interactions only
Lasso
Main effects and Interactions
Plain linear
Marginal
Regression with ReLu
Random Forest
0.0
0.4
0.8
Prediction score (R2, AUC)
Figure 3 $4: Evaluating predictive power: Performance different base learners used variable importance methods (Marginal (Marginal effects), Lasso (Knockoff-Lasso). Random Forest = IMDI, dOCRT, CPI-RF, Conditional- RF. cpi-knockoff, LOCO). BART = Knockoff BAAT., BART) and DNN 1000 and p 50 terms of ROC-AUC score for the classification and R2 score for the regression.
Seroccofff-Deepp Permfit- CPI-DNN, Lazy VIJ) simulated data with
The results for computing the prediction accuracy using the underlying learners of the different methods are reported in Fig. 3 S4 Marginal inference, performs poorly, as it is not predictive approach. Linear models based on Lasso show good performance in the no-interaction effect scenario. Non-linea models based on Random Forest and BART improve on the Nevertheless, they fail achieve good performance in scenarios with interaction effects. The models equipped with deep learner outperform the other methods.
models.
20

---

L Large scale simulations
Method
CPI-DNN
Permfit-DNN
AUC score
Type-I error
Power
Time (seconds)
CPI-DNN
Permfit-DNN
0.4
0.6
0.8
1.0
0.0
0.4
0.8
0.0
0.4
0.8
10,000
100,000
800
200
-0.2
0.0
0.2
04
0.6
1.0
Figure E3: Semi-simulation with UK Biobank: (Top panel) Performance of CPI-DNN and Permfit- DNN is compared in terms of AUC score, Type-I error, Power and Time on the data from UKBB with 8357 and p 671. (Bottom panel) Correlation strength among the variables in the UKBB dataset.
Method CPI-DNN
Permfit-DNN
AUC score
Type-I error
Power
Time (seconds)
CPI-DNN
Permfit-DNN
0.4 0.6
0.8
1.0
0.0
0.4
0.8
0.0
0.4
0.8
10,000
100,000
Figure E4: Large scale simulation: Performance of CPI-DNN and Permfit-DNN is compared in terms of AUC score, Type-I error, Power and Time on simulated data with 10000, 50 and p= 0.8.
we provide comparison of the performance of both Permfit-DNM and CPI-DNN on the semi-simulated data from UK Biobank, with the design matrix consisting of the variables UK BioBank and the outcome is generated following random selection of the true support, where n=8357 and p=671, and a large scale simulation with n=10000, p = 50 and Block-based correlation of coefficient p 0.8. For the UKBB-bases simulation, AUC score and Power. However, both methods control the type-I error at the targeted level. To better understand the reason, we plotted (Fig. [3 Bottom panel) the histogram of the correlation values within the UKBB data: in this case, we consider a low-correlation setting which explains the good -NN. control for Permfit- DNN. In the large scale simulation where the correlation coefficient is set to 0.8, the difference is clear and only CPI-DNN controls the type-I error.
In Figs. D3
and 34
that CPI-DNN achieves higher
21

---

M
Age prediction from brain activity (MEG) in Cam-CAN dataset
Age-prediction from MEG (MAE=7.89, R2=0.69)
low
2.68 %
% of significant brain regions
delta
1.12 %
theta
1.56 %
Frequency band
alpha
0.89
beta low
3.57
beta high
2.23
gamma mid
1.12 %
gamma high
significant
1.34 %
false true
gamma low
1.12 %
0.0
0.5
1.0
1.5
2.0
3.0
3.5
4.0
log 10p value)
Figure E5: Age prediction from brain activity: Predicting age from brain activity in different frequencies with 536 and p 4032.
Following the work of [Engemann et al. 2020 we have applied CPI-DNN to the problem of age prediction from brain activity different frequencies recorded with magnetoencephaalggaaly (MEG) in the Cam-CAN dataset. Without tweaking, the DNN learner reached a prediction performance on par with the published results as seen Fig. E5 The p-vulues formally confirm aspects of the exploratory analysis in the original publication (importance of beta band).
22

---

N
Practical validation of the normal distribution assumption
CPI-DNN
Permfit-DNN
1.0
0.8
Observed p-values
0.6
0.4
0.2
0.0
0.00
0.25
0.50
0.75
1.00
0.00
0.25
0.50
0.75
1.00
Expected p-values
Expected p-values
Figure E6: CPI-DNN Permfit-DNN p-values calibration: p-values vs the uniform distribution with 1000 and 50.
Q-Q plot for the distribution of the
CPI-DNN
Permfit-DNN
125
100
75
50
25
0
-0.2
0.0
0.2
-0.2
0.0
0.2
Figure E7: Normal distribution assumption: Histogram plots of the distribution of the importance scores of random picked non-significant variable with 1000 and p 50.
In Fig. E7 we compared the distribution of the importance scores of a random picked non-significant variable using CPI-DNN and Bermfit-DNV through histogram plots, and can emphasize that the normal distribution assumption holds in practice.
Also, in Fig. E6 we plot the distribution of the p-values provided by CPI-DNN and Permfit-DNN vs the uniform distribution through QQ-plot. We can see that the p-values for CPI-DNN are well calibrated and slightly deviated towards higher values. However, with PermfitDNN the p-values not calibrated.
23

---

0
Random Forest for modeling the conditional distribution and resulting calibration
CPI-DNN
Type-1 Error
1.0
Depth
0.4
5
0.8
Observed p-values
0.3
0.2
0.4
0.1
0.2
0.0
0.0
0.0
0.2
0.4
0.6
0.8
1.0
2
10
Expected p-values
Depth
Figure E8: Random forest calibration: Calibration of the p-values for CPI-DNN (left panel) and the control of type-I error (right panel) as function of the complexity of the Random Forest (the max depth of the trees). Dashed line: targeted type-I error rate. Solid line: uniform distribution.
The use the Random Forest model was to maintain a good non-linear model with time benefits for the prediction of the conditional distribution of the variable of interest. In Fig. E8 We reducing the depth to or 2, thus making the model overly simple, breaks the control of the type-I errors the targeted level. With larger depths, the model becomes more conservative. Therefore, the max depth of the Random Forest chosen based on the performance with 2-fold cross validation.
that
24


Reviews:
[{"review_id": "8mWBI6Qs7J", "reviewer": "Reviewer_2imb", "summary": "The paper proposes a statistical testing framework for more effectively determining variable importance in generic learning settings. Using a conditional permutation test, they show that correlated variables are better handled theoretically and empirically compared to existing unconditional setups. ", "questions": "How good is the normal distribution assumption in practice? It might be interesting or valuable to plot the estimated null distribution in a few different settings with no variable importance.\n\nI'm leaning towards acceptance for this work, if the above issues are well addressed. It is quite neat, and exposed this reviewer to a number of interesting ideas. ", "limitations": "It's not clear if some version of existing Shapley/LIME/etc estimators may achieve similar results.\n\nThe efficiency claim is not well evaluated and does not seem true.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The proposed test is asymptotically and practically efficient, and while some of the theory (and assumptions) may seem strong, the resulting algorithm and test procedure are extremely clean. They take advantage of an interesting existing result in [Williamson et al 2021] and show that it can be implemented quite directly in modern learning settings.\n\nThe summary of related work seems complete.\n\nThe experimental evaluation is strong in supporting the claim that conditional importance is relevant and taking it properly into account leads to performance gains with respect to identifying variable importance compared to unconditional approaches, as well as the suite of selected other methods in the space.", "weaknesses": "1. It's unclear to me how the performance of the method would stack up against aggregrated Shapley or LIME-like methods. As the authors say they are quite popular, and while \"local\" or \"instance-based\", it would be interesting to see if some naive (or sophisticated) application of those is interesting here or not.\n\n2. Figure 4 and the related discussion around L283 is somewhat difficult to parse. It's also not clear what is meant to be taken away from this; is one in better agreement with scientific consensus? What is the interpretation here?\n\n3. The discussion of computation time in Section 7 suggests that it is significant. The result in the supplement suggests it was an additional hour or so (~450-525 min). How does this scale with number of variables? Number of samples? The abstract claims \"computationally lean\", I think if the authors want to state this, support is necessary in the main paper.\n\nMinor:\n\n4. The organization of the experimental section makes it quite difficult to read. I would suggest merging Sections 5 and 6 so that a reader does not need to flip back and forth between experimental setup and result.\n\n5. Might be worth summarizing the power result from the Supplement in the main paper.\n\n6. A few of the references are missing identifying information such as arXiv or DOI links, or venue.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "W1: “​It's unclear to me how the performance of the method would stack up against aggregated Shapley or LIME-like methods. (...).”\n\nR_W1: In Results (section 6) L.278-280, we mention that “Additional benchmarks on popular methods that do not provide p-values, e.g. BART [Chipman et al., 2010] or Shapley values [Kumar et al., 2020], are reported in the supplement (section E)\". In this plot, we examined the AUC score for an aggregated version of SHAP (Shapley values) also for SAGE - an extension for SHAP within a global interpretability manner - and we’ve observed that both methods perform rather poorly across the data-generating scenarios. We now highlighted in the text that we also benchmarked local and instance-based methods.\n\nW2: “Figure 4 and the related discussion around L283 is somewhat difficult to parse. It's also not clear what is meant to be taken away from this; is one in better agreement with scientific consensus? What is the interpretation here?”\n\nR_W2: We thank the reviewer for pointing out the opportunity for clarification. The analysis is motivated by the idea that it can be worthwhile revisiting an exploratory analysis of variable importance with two methods based on the same estimator, one safer method with guarantees and a “naive” approach, which may help detect debatable cases of spurious importance.\n\nWe have modified the main text to explain the context better (first sentence under Experiment 4): “We conducted an empirical study of variable importance in a biomedical application using the non-conditional permutation approach Permfit-DNN (no statistical guarantees for correlated inputs) and the safer CPI-DNN approach. Differences between the methods might point at the confounding presence of correlations between inputs whereas consensus between the methods would point at safer conclusions of importance.”\n\nWe’ve extended the results section to better highlight interesting cases of divergences between the methods: “As expected by the impact of aging on brain structure and function, brain data was most important for age-prediction compared to other outcomes. Interestingly, most disagreements between the methods occurred in this setting as CPI rejected 16 out of 66 brain inputs that were found as important by Permfit. This may point at the importance of correlations between brain variables, that may provoke spurious importance findings with Permfit.'.\n\nW3: The discussion of computation time in Section 7 suggests that it is significant. The result in the supplement suggests it was an additional hour or so (~450-525 min). How does this scale with the number of variables? Number of samples? The abstract claims \"computationally lean\", I think if the authors want to state this, support is necessary in the main paper.\n\nR_W3: This was due to a bug with the plotting (cf. global response - G4). We’ve corrected the time unit in the supplement figures, it should be in seconds rather than minutes. Therefore, we are talking here about a difference of up to 1 min. Please refer to the global response for the response regarding the computation scaling of CPI-DNN and the calibration of the used Random Forest model for the prediction of the conditional distribution of the variable of interest (G1 and G2).\n\nW4: “The organization of the experimental section makes it quite difficult to read. I would suggest merging Sections 5 and 6 so that a reader does not need to flip back and forth between experimental setup and result.”\n\nR_W4: We merged both sections 5 and 6 in order to make it clearer for the readers.\n\nW5: “Might be worth summarizing the power result from the Supplement in the main paper.“\n\nR_W5: We’ve added in the main paper regarding the power result: “Additional inspection of power (supplement section F) showed that across data generating scenarios, CPI-DNN, Permfit-DNN and conditional-RF showed strong results. Marginal and d0CRT performed only well in scenarios without interaction effects. CPI-RF, cpi-knockoff, LOCO and Lazy VI performed poorly.\"\n\nW6: “A few of the references are missing identifying information such as arXiv or DOI links, or venue.”\n\nR_W6: We thank the reviewer for raising this point and have repaired the relevant bibliographic records.\n\nQ1: “How good is the normal distribution assumption in practice? It might be interesting or valuable to plot the estimated null distribution in a few different settings with no variable importance.”\n\nR_Q1: An excellent suggestion. In $\\textbf{PDF Fig. R2}$, we compared the distribution of the importance scores of a random picked non-significant variable using CPI-DNN and Permfit-DNN through histogram plots and we can emphasize that the normal distribution assumption holds in practice. Also, in $\\textbf{PDF Fig. R1}$, we plot the distribution of the p-values provided by CPI-DNN and Permfit-DNN vs the uniform distribution through QQ-plot. We can see that the p-values for CPI-DNN are well calibrated and slightly deviated towards higher values. However, with Permfit-DNN the p-values are not calibrated.\n"}, {"review_id": "pMVWc22HVt", "reviewer": "Reviewer_pFgF", "summary": "The authors present an approach that aims to tackle the problem with the default permutation importance technique of performing variable importance. The default setting is prone to detecting correlated variables even as important even when they are not (Type I error). The authors present Conditional permutation importance: an approach that fixes this problem by conditionally sampling the variables based on the correlations. The results indicate the method is robust to type I errors, model agnostic and computationally efficient. ", "questions": "- The figure text is a bit small and hard to read if not zoomed in on a PDF\n- The paragraph formatting is strange is some places. For examples compare the strange linebreak in lines 26-27 with the standard Neurips format seen between paragraphs 74-75.\n- Is permutation performance (general setting) prone to type II errors for certain models? One can imagine a hypothetical setting where a variable, age, is duplicated, i.e. full correlation between the two variables. A certain type of model can learn to predict solely based on “age copy”, since it includes all the required information and “age original” is redundant. “Age original” will therefore have 0 importance from a PI analysis. Are these not considered as the models that collapse onto one feature are rare? ", "limitations": "There is a notable lack of discussion of the limitations of this method. One of the few weaknesses. The authors have many discussion points to touch on: computational complexity, sensitivity to the data structure, missing data, types of correlation, model configuration, just to name a few.\n\nThe discussion section is fairly short, with a very shallow discussion on the limitations. With this part expanded and made more comprehensive, this paper could go from strong to very strong.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The paper has a number of strengths including a novel approach to solving the correlation issues with standard permutation importance methods, a rigorous analysis of the proposed method with clear results. The paper is very well written and formatted leading to a well oriented reading. The related works are discussed appropriately with similar methods compared as baselines. Its value to the field is clear and its applications broad. This is a strong paper.", "weaknesses": "- The clarity of the experiments can be improved. Firstly the selections of models, datasets and settings can be motivated better. Secondly, the first paragraph of section 5 mentiones that “default behavior” was implemented. It’s not very clear what this default behavior is.\n- The contributions mention a computationally lean method, is this analysis presented somewhere? I cannot see it in the paper.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "W1: “The clarity of the experiments can be improved. Firstly the selections of models, datasets and settings can be motivated better. Secondly, the first paragraph of section 5 mentioned that “default behavior” was implemented. It’s not very clear what this default behavior is.”\n\nR_W1: First of all, to improve clarity and readability, we have merged the sections 5-6 (Experiments and Results). In Related Work (section 2) L.90-92, we review the state of the art giving rise to the models selected for our work. As a result, we found strong interest in DNNs as function estimators in permutation algorithms. In Experiment 3 (section 5.3) L.222-223, we state that “we include Permfit-DNN and CPI-DNN in a benchmark with other state-of-the-art methods for variable importance.“ Our benchmarks include all major ideas presented in our review. We substantially extended the basic synthetic data setting from Mi et al. 2021 to include different degrees of correlation, non-linearities and prediction tasks. The UK Biobank example was chosen as a recent example (Dadi et al 2021) of predicting real-world outcomes from heterogenous, potentially correlated inputs. By “default behavior”, we wanted to highlight the fact that the different methods were used according to their original implementations without any modifications. As for Permfit-DNN, CPI-DNN and CPI-RF, the default implementation is based on a special internal 2 fold-cross validation for the significance study. As an improvement, we’ve changed “the default behavior of the different methods is implemented in order to maintain a fair comparison” to “we refer to the original implementation of the different methods in order to maintain a fair comparison”\n\nW2: “The contributions mention a computationally lean method, is this analysis presented somewhere? I cannot see it in the paper.”\n\nR_W2: Please refer to the global response (Regarding the computational scaling of CPI-DNN and leanness - G1) where we are introducing the new plots that explain the reason we used this term in our paper also (Random Forest for modeling the conditional distribution and resulting calibration - G2) for the calibration of the Random Forest model used to predict the conditional distribution of the variable of interest.\n\nQ1: “The figure text is a bit small and hard to read if not zoomed in on a PDF”\n\nR_Q1: We increased the font sizes in order to maintain a better experience for the reader.\n\nQ2: “The paragraph formatting is strange in some places. For example, compare the strange line break in lines 26-27 with the standard Neurips format seen between paragraphs 74-75.”\n\nR_Q2: The paragraph formatting is corrected following the NeurIPS standards.\n\nQ3: “Is permutation performance (general setting) prone to type II errors for certain models? One can imagine a hypothetical setting where a variable, age, is duplicated, i.e. full correlation between the two variables. A certain type of model can learn to predict solely based on “age copy”, since it includes all the required information and “age original” is redundant. “Age original” will therefore have 0 importance from a PI analysis. Are these not considered as the models that collapse onto one feature are rare?”\n\nR_Q3: Please refer to the global response (Groups needed for very highly correlated settings - G3) where we provide the discussion related to the need for groups in very highly correlated settings.\n\nL1: “There is a notable lack of discussion of the limitations of this method. One of the few weaknesses. The authors have many discussion points to touch on: computational complexity, sensitivity to the data structure, missing data, types of correlation, model configuration, just to name a few. The discussion section is fairly short, with a very shallow discussion on the limitations.”\n\nR_L1: We thank you for bringing this comment around the discussion part. We rephrase this section to include the following ideas: (1) In the global response, we conduct the response related to the computation scaling of CPI-DNN and why it is a “computationally lean” method - G1 therefore, we can add that “CPI-DNN has a linear and quadratic complexity as a function of the number of samples and variables respectively as seen in the Supplement”, while adding the $\\textbf{PDF Fig. R6}$ to the Supplement. (2) “The efficiency of the CPI method is affected by the increase in the correlation coefficient between the variable i.e. two very high correlated variables will have a conditional importance of 0. Therefore, as a future direction, we suggest the use of group-based feature importance where groups of highly correlated variables are pre-defined”. (3) “Modeling the conditional distribution of the variable of interest is of great impact in this work where there is a need for a good non-linear model with time benefits. Random Forest is a good example which, given reasonable default choices implemented in standard software, benefits largely from tuning the depth of its trees as a single hyperparameter” - we’ve also added the $\\textbf{PDF Fig.R3}$ to show the impact of the depth on the calibration of the results. (4) We do mention in the Discussion part (section 7) L.311 that the prediction could also “lead to inaccuracies and biased inference if the number of variables becomes too large”.\n"}, {"review_id": "kNJqRdmi9K", "reviewer": "Reviewer_XsJL", "summary": "The authors argue for the importance of using conditional sampling in permutation importance (CPI) methods. \nThey show that when features are correlated, using non-conditional sampling for permutation importance (on a finite training set) leads to a bias in the estimated feature importance. (Specifically, the bias is discussed in the case of null features having positive importance). \nAs their main result, they show that under regularity conditions (similar to those of Williamson 2021), the conditional permutation importance estimator is approximately normal and therefore the resulting p-values / intervals would be valid. \nThey produce an implementation of CPI, which they test against the permfit-DNN algorithm of based on Mi et al in a series of experiments. Here the main result is that indeed the non-conditional permutations introduce bias (when interpreted as p-values) and a little loss in \"sorting\" accuracy (that is in the relative rank of the null examples).", "questions": "See above.", "limitations": "I think one limitation not discussed is the difficulty in modeling the conditional distribution, and what happens if there are shortcomings in this model. \n\n(No societal impact problems :)", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "Strengths: \nClarity: The paper is in general well written and quite easy to follow. The arguments and proofs are surveyed well. The experiments and results are easy to follow. \n\nQuality: \nThe theoretical derivations are convincing and seem correct. The experimental results are well set to show the point that methods can break if they don’t account for the correlation of the variables. The comparison to other explainability methods is quite comprehensive in terms of the survey.  \n\n\nNovelty: \nThere are some novel aspects to this paper: first and foremost, the method implements a code base for producing conditional inference without assuming a specific type of correlation between the variables, and with the learning model implemented at the same time. This might make the method attractive to users. \n\nIn addition, we have two main theoretical results. The first is a detailed critical discussion of the assumptions of Mi et al, showing an unaccounted for bias term in their analysis. The analysis is nice, but I am not sure it holds the paper. The second is an extension of the Williamson result for condition permutation importance, showing that the resulting statistic is approximately normal under some conditions. IT is important for the estimator, but as far as I can tell, applies the proofs in the Williamson paper to our estimator. \n\nHowever, here it should be said that in my view the paper significantly overclaims, and does not describe well the current knowledge, See more in “weaknesses”. \n\n\n\nSignificance: The problem of identifying important features that contribute to an outcome of interest is fundamental \nto scientific research, in particular in the biosciences. Recent years have seen many methods that try to adapt\nthese ideas so they would work well for non-linear models. This paper proposes a new method (and software) \nfor computing conditional permutation importance, and provides some theoretical and empirical evidence for the superiority \nof the conditional sampling over the marginal sampling. Because conditional sampling is harder, having out-of the box\ncode for performing this analysis may be important.", "weaknesses": "Weakness:\n\nThe main weakness of this paper in my view is that it is not honest enough in representing its contribution to the literature. At a first read, and perhaps for researchers who are not experts in the field, it may sound as if the authors were the first to propose conditional permutation importance. (See the structure of the introduction - the idea of conditional permutation importance is not mentioned, except under their contribution). \n“””In this work, we introduce an improvement of permutation importance called conditional per46 mutation importance. This method uses a conditional sampling step to compute the conditional  importance, taking into account the correlation between the variables. In section 3 we analyze  the limitations of the permutation approach. In section 4 we describe a conditional sampling based variable importance framework and establish that it controls type-I errors. In section 5, we  systematically compare the most popular variable importance measures on various classification 51 and regression tasks, with a focus on type-I error control.”””\nWhen they acknowledge other CPIs, in passing in Section 2, they only mention in the context of Strobl et al. \n\nIn practice, CPIs have long been discussed in the general framework of feature importance. See for example Chapter 8.2 of Fisher et al in JMLR. More generally, the debate whether features removed (in a simple removal as in Covert 2022, or as in SHAP) should be imputed from the marginal distribution or from the conditional distribution has been ongoing. For example, the original definition of SHAP is designed to sample conditionally, but for computational reasons resorts to marginal sampling. In Tree-SHAP, the sampling is conditional. \n\nIn implying that this paper introduces conditional permutation inference, the authors are also doing a disservice to the readers in neglecting to more clearly stake their actual innovations, allowing the reader to evaluate them. For example, is the Mi result important so that showing its fault is important? Articulate this. \n(In my view, the most important part here is the software with out-of-box estimators for the conditional distribution.  But other arguments by the authors would make their case more clear, Currently, reading the abstract, I understand their main innovation to be introducing CPI, which obviously is not true.\n(Comment: My current very low grade is due to this weakness. A satisfactory rewrite of this would probably allow me to increase the score to the area of just below or above acceptance). \n\n\nOther than that, it seems to be a paper with rather modest innovation, outside of the software itself. Perhaps my view is dimmed by the overclaiming, and put in proper context I will be happy to re-evaluate. \n\n\n3.Because in my view the most important part is the software tool, I would have loved to see more about how it works compared to other methods. In particular, more sophisticated relations between variables; how sensitive it is to an increase in p compared to other methods; An interesting question that goes unanswered is how much tweeking goes into making the method work well for new data; perhaps an analysis of multiple datasets would help here, (though I understand the added costs in writing that paper).  \n\n\nOther comments: \n\nOut of the box methods for conditional inference could backfire if the relation between the variables is hard to model correctly, or if there are too many variables / not enough samples so that variables are almost completely determined by other variables. I would imagine that a conditional method would provide some quality control and safeguards regarding the accuracy of modeling one variable from the rest. \n\nBoth the type-1 error statistics and the AUC are defined in terms of telling appart important and unimportant variables. Is there any sense where we can understand the accuracy or variability in ranking for the non-null variables? Feature important values can usually be interpreted for non-null cases as well. \n\nSmaller comments:\n\nFigure 4: color scheme is not consistent between left and right figures (same colors, different meanings). On the right color mean intersection of model results. On the left, the same colors mean source of features, whereas the intersection of model results are marked by the symbols. Also, crosses and circles almost identical; top rank is 1 not p which should be mentioned; Right side is a bit hard to parse.\n\n#### Changes ####\n\nFollowing discussions, the authors suggested specific changes to the writing and to the citations that I believe better reflect their contribution to the field. I will therefore update the score to borderline reject.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "S1: “The first is a detailed critical discussion of the assumptions of Mi et al (...) ”\n\nR_S1: Mi et al. 2021 demonstrated the intuitive estimation of variable importance in the biomedical context using an effective brute-force approach. This can work well thanks to their high-capacity function approximator (DNN), yet, lacks statistical guarantees (under the standard permutation approach). Our work shows how to overcome the statistical failure modes of Mi et al. while preserving their effective DNN model.\n\nS2: “ The second is an extension of the Williamson result (...).”\n\nR_S2: Williamson et al. 2021 provides the general framework for variable importance based on the removal of the variable of interest with a theoretical proof. We used their regularity conditions mentioned in L.152 and Appendix A. We carefully translated the proof to our setting (large-scale biomarker discovery). Importantly, we found that, in our case  (L.178-179) “unlike Williamson (...), the variance estimator is non-vanishing, and thus can be used as a plugin”, which adds support to the validity of our usage of this result in our setting. \n\nW1: “The main weakness of this paper in my view is that it is not honest enough (...).“\n\nR_W1: The reviewer flags a potential misunderstanding. We have studied previous contributions and did not intend to claim inventorship for the CPI idea. In Related Work (section 2) L.77-89, we discuss Strobl et al. 2008 (exclusive for Random Forests), Candes et al. 2017 (the variable of interest is sampled conditionally on the other covariates multiple times to compute a test statistic and p-values) and Watson and Wright 2021 (the necessity of conditional schemes via knockoff sampling). \nTo address the concern, we carefully rephrased introduction, related work and method sections to clarify how we build on top of the CPI literature.\nIn fact, we have not found any deep discussion of the practical implications of using CPI methods, nor a proof of the validity that would hold unconditionally with respect to estimated learners $\\hat{\\mu}$. Neither have we found systematic reusable benchmarks of CPI approaches against alternatives. An important suggestion made in our work is that a relatively efficient conditional sampling schema can be used, rendering CPI more affordable.\n\nW2: “ In implying that this paper introduces conditional permutation inference, (...) is the Mi result important (...)? (...).“\n\nR_W2: Our work studies the properties of CPI techniques, leading to a more general algorithm based on a thorough analysis of the limitations of the standard permutation scheme. To extend CPI to support the family of DNN learners, i.e. to make the proof more general, we adapted Williamson’s proof to our work as shown in Appendix A. Our work provides systematic benchmarks for CPI methods and helps clarify their respective advantages and failure modes. Of note, the LOCO approach in our work implements the method of Williamson et al 2021 and performed worse than the proposed CPI method, cf. Fig 3 in main text, $\\textbf{PDF Figs R4-6}$ (rebuttal), which is non-trivial. \nWe will provide a usable general implementation supporting any scikit-learn compatible estimators as base learners and conditional probability learners. The work of Mi et al. was inspiring, as it provides a state-of-the-art estimator with strong AUC-scores (cf. R_S1 above).\n\nW3: “Because in my view the most important part is the software tool, (...) multiple datasets would help here”\n\nR_W3: We thank the reviewer for the enthusiasm. Our work provides experts in variable importance and practitioners a reusable toolkit for composing and testing novel combinations of permutation algorithms and conditional probability estimators across various data-generating scenarios.\nTo make our tools more usable, our GitHub repository will provide a high-level notebook upholding the usage of different methods, including CPI algorithms. Inspired by the reviewer, we have applied CPI-DNN to the problem of age prediction from brain activity in different frequencies recorded with magnetoencephalography (MEG) in the Cam-CAN dataset (cf. Engemann et al 2020, eLife). Without tweaking, the DNN learner reached a prediction performance on par with the published results ($\\textbf{PDF Fig. R5}$). The p-values formally confirm aspects of the exploratory analysis in the original publication (importance of beta band). We have included this result in the paper.\n\nW4: “Out of the box methods for conditional inference (...) determined by other variables.”\n\nR_W4: We mention among the limitations (section 7) L.311 that the prediction could also “lead to inaccuracies and biased inference if the number of variables becomes too large”. To explore a related point, we have studied the model calibration when systematically reducing the random forest depth ($\\textbf{PDF Fig. R3}$), which could provide a tool to study such issues in practice.\n\nW5: “Is there any sense where we can understand the accuracy or variability (...)”\n\nR_W5: The effect size corresponds to the effect on y that can be attributed to variable $x_{j}$, conditionally to the others. Since we have the variance, we can define a confidence interval associated with it. We have added a note to the main text to clarify this point.\n\nW6: “Figure 4: color scheme is not consistent between left and right figures (...)”\n\nR_W6: Indeed. To avoid confusion, we have now chosen two distinct color sets.\n\nL1: “I think one limitation not discussed is the difficulty in modeling the conditional distribution (...)”\n\nR_L1: We now mention this limitation in the main text. To explore it, we have studied model calibration when systematically reducing the random forest depth ($\\textbf{PDF Fig. R3}$)-Global response G2, showing how a shallow random forest model actually breaks the model calibration while higher capacity increases conservatism. In practice, the depth is under control of grid search. We have added the result to the paper (supplement).\n"}, {"review_id": "Vlr7798Hjt", "reviewer": "Reviewer_HX9z", "summary": "In this work, the authors proposed a novel conditional permutation importance approach (CPI) that can well estimate the variable importance with the p-value provided under the condition that correlation exists across features. The authors theoretically justify the inflation of the feature permutation-based approach and justify the calibration of CPI. Empirically, CPI yields calibrated results across different settings with good power, which outperforms existing methods in the related field. ", "questions": "1. It is surprising to see the significant difference in AUC between CPI and cpi-knockoff. Do the authors have insight into what makes the difference?\n\n2. Getting from equation (2) to equation (3) is not intuitive to me. It would be great if the authors could provide more detailed proof.\n\n3. How did the authors compute the actual feature ranking? Which scheme did they use (i.e. Did you take the integration of equation (4); use Shapley value; or rank by coefficients?)\n\n", "limitations": "The authors have addressed several limitations of this work:\n1. The limitation of using RF to approximate the conditional independence\n2. The limitation of sample size for the trainer\n\nAdditionally, the theoretical calibration of the CPI relies on several hypotheses (A1, A2, A3, A4, B1, and B2) defined in the supplementary. It would be beneficial to have further discussion on the extent to which each hypothesis can be met in reality. Furthermore, if any of the hypotheses cannot be satisfied, it would be helpful to know the degree to which the calibration would be affected.\n", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "This paper focuses on addressing two key issues in interpretable machine learning: statistical calibration and scalability. The proposed method, CPI, shows a significant improvement in statistical calibration compared to the naive feature permutation-based approach. The paper also provides theoretical justification for CPI's ability to effectively control type-I errors. In simulation datasets, CPI outperformed other methods that addressed feature dependencies in terms of the AUC that measures the consistency of feature ranking.", "weaknesses": "The existing evidence is insufficient to convince me that CPI is well-calibrated. Several items can improve the soundness of the paper:\n1. The authors should also provide the qqplot with a reasonable amount of tests (>=100) to demonstrate the calibration. \n2. The authors used RF to approximate the feature j's conditional distribution; more justification is needed for why this is a good choice. (i.e., Does a simpler model achieve the same FPR and AUC level?)\n3. The sample sizes in simulations are fairly small. It is unclear whether the well-controlled FPR was due to the calibration or the lack of power to detect inflation. A sample size of at least 10K is suggested w.r.t the given feature size: 50. If computation is a problem, I would suggest either precomputing the SVD of the covariance matrix and then transposing the data matrix to get $X$ with desired correlation. Or it would be even better if the authors could (also) perform the simulation on the UK Biobank genotype dataset (they have more realistic feature dependencies needed for this work).\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "W1: “The authors should also provide the qqplot with a reasonable amount of tests (>=100) to demonstrate the calibration.”\n\nR_W1: In $\\textbf{PDF Fig. R1}$, we provided the qq-plot for the distribution of the p-values provided by Permfit-DNN and CPI-DNN for one randomly picked non-significant variable (the calibrated p-values should theoretically follow the uniform distribution). We can see that the p-values obtained by CPI-DNN are following the uniform distribution with a slight deviation towards higher p-values (which what we aim for as this variable is non-significant). However, with Permfit-DNN the p-values are not calibrated.\n\nW2: “The authors used RF to approximate the feature j's conditional distribution; more justification is needed for why this is a good choice. (i.e., Does a simpler model achieve the same FPR and AUC level?)”\n\nR_W2: Please refer to the global response (Random Forest for modeling the conditional distribution and resulting calibration - G2) for the discussion around the use of the Random Forest and its calibration.\n\nW3: “The sample sizes in simulations are fairly small. It is unclear whether the well-controlled FPR was due to the calibration or the lack of power to detect inflation. A sample size of at least 10K is suggested w.r.t the given feature size: 50. If computation is a problem, I would suggest either precomputing the SVD of the covariance matrix and then transposing the data matrix to get X with desired correlation. Or it would be even better if the authors could (also) perform the simulation on the UK Biobank genotype dataset (they have more realistic feature dependencies needed for this work).“\n\nR_W3: In $\\textbf{PDF Fig. R4-a}$ and $\\textbf{PDF Fig. R4-b}$, we provide a comparison of the performance of both Permfit-DNN and CPI-DNN on the semi-simulated data from UKBB where $n$=$8357$ and $p$=$671$, and a large scale simulation with $n$=$10000$, $p$ = $50$ and block-based correlation of coefficient $\\rho$ = $0.8$. For the UKBB-based simulation, we see that CPI-DNN achieves a higher AUC score and Power. However, both methods control the type-I error at the targeted level. To better understand the reason, we plotted (just below) the histogram of the correlation values within the UKBB data and indeed we are faced with a low-level correlation setting which explains the good control for Permfit-DNN. In the large scale simulation where the correlation coefficient is set to 0.8, the difference is clear and only CPI-DNN controls the type-I error. \n\nQ1: “It is surprising to see the significant difference in AUC between CPI and cpi-knockoff. Do the authors have insight into what makes the difference?”\n\nR_Q1: We can refer to two facts: (1) cpi-knockff is based on the use of a random forest model with the ranger implementation while CPI-DNN is equipped with a DNN, which could explain its lack of power and (2) cpi-knockoff is based on the use of Knockoff sampling to replace the variable of interest. Thus, one possible reason for this difference is the quality of the knockoff sampling. As we know, people often generate Gaussian knockoffs but this could be inaccurate (as shown our process is more accurate). We have observed that even a relatively small distribution shift in knockoff generation can lead to large errors at inference time. \n\nQ2: “Getting from equation (2) to equation (3) is not intuitive to me. It would be great if the authors could provide more detailed proof.”\n\nR_Q2: We’ve added the following part in the supplement for further clarifications:\n$\\mu = \\mathbf{xw} = x^{j} w^{j} + \\mathbf{x^{-j} w^{-j}} = \\mathbf{x^{-j} w^{-j}}$ because $x^{j}$ is a null feature i.e. $w^{j} = 0$, and $\\hat{\\mu} = \\mathbf{x\\hat{w}}$ are linear functions therefore:\n\n$ \\hat{m}^j\n= \\frac{1}{n_{test}} \\sum_{i=1}^{n_{test}}  (\\hat{\\mu}(\\mathbf{x_i}) - \\hat{\\mu}(\\mathbf{x_i^{(j)}})) (2\\mu(\\mathbf{x_i}) - \\hat{\\mu}(\\mathbf{x_i}) - \\hat{\\mu}(\\mathbf{x_i^{(j)}}) + 2 \\varepsilon_i)$\n\n$= \\frac{1}{n_{test}} \\sum_{i=1}^{n_{test}}  (\\cancel{\\mathbf{x_{i}^{-j}} \\mathbf{\\hat{w}^{-j}}} + x_{i}^{j} \\hat{w}^{j} - \\cancel{\\mathbf{x_{i}^{-j}} \\mathbf{\\hat{w}^{-j}}} - \\\\{x_{i}^{j}\\\\}^{perm} \\hat{w}^{j}) (2 \\mathbf{x_{i}^{-j}} \\mathbf{w^{-j}} - 2 \\mathbf{x_{i}^{-j}} \\mathbf{\\hat{w}^{-j}} - (x_{i}^{j}\\hat{w}^{j} + \\\\{x_{i}^{j}\\\\}^{perm} \\hat{w}^{j}) + 2 \\varepsilon_i)$\n\n$= \\frac{2\\hat{w}^{j}}{n_{test}} \\sum_{i=1}^{n_{test}}  (x_{i}^{j} - \\\\{x_{i}^{j}\\\\}^{perm}) (\\mathbf{x_{i}^{-j}} (\\mathbf{w^{-j}} - \\mathbf{\\hat{w}^{-j}})  + \\varepsilon_i) - \\cancel{\\hat{w}^{j}((x_{i}^{j})^{2} - (\\\\{x_{i}^{j}\\\\}^{perm})^{2})}$\n\n$= \\frac{2\\hat{w}^{j}}{n_{test}} \\sum_{i=1}^{n_{test}}  (x_{i}^{j} - \\\\{x_{i}^{j}\\\\}^{perm}) (\\mathbf{x_{i}^{-j}} (\\mathbf{w^{-j}} - \\mathbf{\\hat{w}^{-j}})  + \\varepsilon_i)$\n\nQ3: “How did the authors compute the actual feature ranking? Which scheme did they use (i.e. Did you take the integration of equation (4); use Shapley value; or rank by coefficients?)”\n\nR_Q3: As described in Practical estimation (section 4.2), the equation 4 (in the regression case) is used through the loss score defined in equation 6 for B drawings. We thus compute a Wald-type statistic as stated L.179-181 by “dividing the mean of the importance scores by the corresponding standard deviation. P-values are computed using the cumulative distribution function of the standard normal distribution”. These p-values are used to perform the ranking which is equivalent to rank based on the Wald statistic or standardized importance scores.\n\nL1: “It would be beneficial to have further discussion on the extent to which each hypothesis can be met in reality. Furthermore, if any of the hypotheses cannot be satisfied, it would be helpful to know the degree to which the calibration would be affected.”\n\nR_L1: Please refer to the global response (Groups needed for very highly correlated settings - G3) where we provide the discussion related to the need for groups in very highly correlated settings.\n"}]

Author rebuttal:
We thank all four reviewers for the time they spent studying our proposed work and for their thoughtful comments. We would also like to thank the reviewers for meaningfully recognizing the strengths of our work while giving us an opportunity to improve it based on their constructive criticism informing about shortcomings and weaknesses. 

We have identified a few common points of concern raised by the four reviewers which we will address jointly in what follows. For later reference, our global responses are named G1-G4.

G1

$\textbf{Computational scaling of CPI-DNN and leanness}$:  Following Alg. 1 in the main text and the Random Forest complexity analysis, the complexity can be computed as: $O(n_{estimators} n p^{2} d + B n p)$ where $n_{estimators}$ the number of estimators for Random Forest model, $n$ the number of samples, $p$ the number of variables,$d$ the depth of the trees, $B$ the number of draws, and $mtry=p$ under worst case assumption. Thus, the complexity is linear and quadratic with the number of samples and variables respectively. We are providing a plot summarizing the computation scaling of CPI-DNN as a function of the number of samples and variables ($\textbf{PDF, Fig. R6}$). We can see empirically that CPI-DNN has a linear complexity with the number of samples and a quadratic complexity with the number of variables. As for “computationally lean”, it refers to two facts: (1) there is no need to refit the costly MLP learner to predict y unlike LOCO-DNN (A removal-based method provided with our learner). This can be seen in the $\textbf{PDF Fig. R5}$ where both CPI-DNN and LOCO-DNN achieved a high AUC score and controlled the Type-I error in a highly correlated setting ($n$=$1000$, $p$=50 and $\rho$=$0.8$). However, in terms of computation time, CPI-DNN is far ahead of LOCO-DNN which validates our use of the permutation scheme. (2) The conditional estimation step involved for the conditional permutation procedure is done with an efficient RF estimator, leading to small time difference wrt Permfit-DNN; Overall we obtain the accuracy of LOCO-type procedures for the cost of a basic permutation scheme. Of note, LOCO as used in the main paper, implements the original approach by Williamson et al. 2021. 

G2

$\textbf{Random Forest for modeling the conditional distribution and resulting calibration}$: The use of the Random Forest model was to maintain a good non-linear model with time benefits for the prediction of the conditional distribution of the variable of interest. To better frame this idea, in the $\textbf{PDF Fig. R3}$, we are plotting the calibration of the p-values for CPI-DNN (left panel) and the control of type-I error (right panel) as a function of the complexity of the Random Forest (the max depth of the trees). We can see that reducing the depth to 1 or 2, thus making the model overly simple, breaks the control of the type-I errors at the targeted level. With larger depths, the model becomes more conservative. Therefore, the max depth of the Random Forest is chosen based on the performance with 2-fold cross validation. Moreover, the prediction could also “lead to inaccuracies and biased inference if the number of variables becomes too large” as stated in Discussion (section 7) L.311.

G3

$\textbf{Groups needed for very highly correlated settings}$: If we have the case of two highly correlated variables X1 and X2, the corresponding conditional importance of both variables is 0. This problem is linked to the very definition of conditional importance, and not to the CPI procedure itself described in the paper. The only workaround is to eliminate, prior to importance analysis, degenerate cases where conditional importance cannot be defined. This is why, among the future directions, we suggested the use of group-based variable importance where the process is preceded by defining the groups of correlated variables.

G4

$\textbf{Fixes in the plots}$: We’ve fixed the time computation reported in the supplement figures which should be in seconds rather than minutes. Also, for the second row of Fig 3, the results were shifted between the methods. Therefore, we’ve corrected this error in $\textbf{PDF Fig. R7}$. This figure is showing the analysis reported in section 6 (Results). 

For convenience, have compiled and enumerated all new results in support of our rebuttal in the PDF attached below. We will refer to the new additional results according to the numbering in the PDF.
For the detailed reviewer-specific responses below, we enumerate to reviewer statements by section, i.e., S for strength, W for weakness, L for limitation and Q for question. Our respective responses are indicated by the prefix “R_”.

Meta-review: Except one reviewer, the scores for this paper are quite good overall. As highlighted by Reviewer XsJL, the discussion of the literature was incomplete and it is surprising that the authors have missed several easy to find papers on this topic. This reviewer seems however to be satisfied by the proposed changes. While the general idea of conditional permutations is not novel, the implementation as done in the paper is original and the experimental comparison is extensive. Despite limited novelty, given the positive scores of all other reviewers, I recommend acceptance. Please make sure however to implement all changes promised during the rebuttal, in particular concerning the discussion of related works.
Decision: Accept (poster)