SlowGuess/NeurIPS_2023_OCR huggingface dataset:
(A3) (Continuity of optimization)  $\| \mu_{P_0 + \epsilon h} - \mu_0\|_{\mathcal{F}} = O(\epsilon)$  for each  $h\in \mathcal{R}$
(A4) (Continuity of derivative)  $\mu \mapsto \dot{l} (\mu ,P_0;h)$  is continuous at  $\mu_0$  relative to  $\| .\|_{\mathcal{F}}$  for each  $h\in \mathcal{R}$ .
(B1) (Minimum rate of convergence)  $\| \hat{\mu}_n - \mu_0\|_{\mathcal{F}} = o_P(n^{-1 / 4})$
(B2) (Weak consistency)  $\int g_{n}(\mathbf{z})^{2}dP_{0}(\mathbf{z}) = o_{P}(1)$ .
(B3) (Limited complexity) there exists some  $P_0$ -Donsker class  $\mathcal{G}_0$  such that  $P_0(g_n \in \mathcal{G}_0) \to 1$ .
Proposition (Theorem 1 in [Williamson et al., 2021]) If the above conditions hold,  $l(\hat{\mu}_n, P_n)$  is an asymptotically linear estimator of  $l(\mu_0, P_0)$  and  $l(\hat{\mu}_n, P_n)$  is non-parametric efficient.
Let  $P_0^\star$  be the distribution obtained by sampling the j-th coordinate of  $\mathbf{x}$  from the conditional distribution of  $q_{0}(x^{j}|\mathbf{x}^{-j})$ , obtained after marginalizing over  $y$ :
$$
q _ {0} (x ^ {j} | \mathbf {x} ^ {- \mathbf {j}}) = \frac {\int P _ {0} (\mathbf {x} , y) d y}{\int P _ {0} (\mathbf {x} , y) d x ^ {j} d y}
$$
$P_0^\star (\mathbf{x},y) = q_0(x^j |\mathbf{x}^{-\mathbf{j}})\int P_0(\mathbf{x},y)dx^j.$  Similarly, let  $P_{n}^{\star}$  denote its finite-sample counterpart. It turns out from the definition of  $\hat{m}_{CPI}^{j}$  in Eq. 4 that  $\hat{m}_{CPI}^{j} = l(\hat{\mu}_{n},P_{n}^{\star}) - l(\hat{\mu}_{n},P_{n})$  . It is thus the final-sample estimator of the population quantity  $m_{CPI}^{j} = l(\hat{\mu}_{0},P_{0}^{\star}) - l(\hat{\mu}_{0},P_{0}).$
Given that  $\hat{m}_{CPI}^{j} = l(\hat{\mu}_{n},P_{n}^{\star}) - l(\hat{\mu}_{0},P_{0}^{\star}) - (l(\hat{\mu}_{n},P_{n}) - l(\hat{\mu}_{0},P_{0})) + l(\hat{\mu}_{0},P_{0}^{\star}) - l(\hat{\mu}_{0},P_{0})$ , the estimator  $\hat{m}_{CPI}^{j}$  is asymptotically linear and non-parametric efficient.
The crucial observation is that under the j-null hypothesis,  $y$  is independent of  $x^{j}$  given  $\mathbf{x}^{-\mathbf{j}}$ . Indeed, in that case  $P_{0}(\mathbf{x},y) = q_{0}(x^{j}|\mathbf{x}^{-\mathbf{j}})P_{0}(y|\mathbf{x}^{-\mathbf{j}})P_{0}(\mathbf{x}^{-\mathbf{j}})$  and  $P_{0}(x^{j}|\mathbf{x}^{-\mathbf{j}},y) = P_{0}(x^{j}|\mathbf{x}^{-\mathbf{j}})$ , so that  $P_0^{\star} = P_0$ . Hence, mean/variance of  $\hat{m}_{CPI}^{j}$ 's distribution provides valid confidence intervals for  $m_{CPI}^{j}$  and mean  $(\hat{m}_{CPI}^{j})\underset {n\to \infty}{\rightarrow}0$ . Thus, the Wald statistic  $\hat{z}_{CPJ}^{j}$  defined in section (4.2) converges to a standard normal distribution, implying that the ensuing test is valid.
In practice, hypothesis (B3), which is likely violated, is avoided by the use of cross-fitting as discussed in [Williamson et al., 2021]: as stated in the main text, variable importance is evaluated on a set of samples not used for training. An interesting impact of the cross-fitting approach is that it reduces the hypotheses to (A1) and (A2), plus the following two:
(B'1) (Minimum rate of convergence)  $\| \hat{\mu}_n - \mu_0\|_{\mathcal{F}} = o_P(n^{-1 / 4})$  on each fold of the sample splitting scheme.
(B2') (Weak consistency)  $\int g_{n}(\mathbf{z})^{2}dP_{0}(\mathbf{z}) = o_{P}(1)$  on each fold of the sample splitting scheme.
D Computational scaling of CPI-DNN and leanness
CPI-DNN LOCO-DNN
Figure E2: CPI-DNN vs LOCO-DNN: Performance at detecting important variables on simulated data with  $n = 1000$ ,  $p = 50$  and  $\rho = 0.8$  in terms of (AUC score), Type-I error, Power and Time. Dashed line: targeted type-I error rate. Solid line: chance level.
Computationally lean refers to two facts: (1) there is no need to refit the costly MLP learner to predict y unlike LOCO-DNN (A removal-based method provided with our learner) as seen in Fig. E2. Both CPI-DNN and LOCO-DNN achieved a high AUC score and controlled the Type-I error in a highly correlated setting  $(\rho = 0.8)$ . However, in terms of computation time, CPI-DNN is far ahead of LOCO-DNN, which validates our use of the permutation scheme. (2) The conditional estimation step involved for the conditional permutation procedure is done with an efficient RF estimator, leading to small time difference wrt Perfit-DNN; Overall we obtain the accuracy of LOCO-type procedures for the cost of a basic permutation scheme.
E Evaluation Metrics
AUC score [Bradley, 1997]: The variables are ordered by increasing p-values, yielding a family of  $p$  splits into relevant and non-relevant at various thresholds. AUC score measures the consistency of this ranking with the ground truth ( $p_{\text{signals}}$  predictive features versus  $p - p_{\text{signals}}$ ).
Type-I error : Some methods output p-values for each of the variables, that measure the evidence against each variable being a null variable. This score checks whether the rate of low p-values of null variables exceeds the nominal false positive rate (set to 0.05).
Power : This score reports the average proportion of informative variables detected (when considering variables with p-value  $< 0.05$ ).
Computation time : The average computation time per core on 100 cores.
Prediction Scores : As some methods share the same core to perform inference and with the data divided into a train/test scheme, we evaluate the predictive power for the different cores on the test set.
F Supplement Figure 1 - Power & Computation time
Method
CPI-DNN Perfit-DNN
A
B
Figure 1 - S1: Permfit-DNN vs CPI-DNN: Performance at detecting important variables on simulated data under the setting of experiment 1, with  $n = 300$  and  $p = 100$ . (A): The power reports the average proportion of informative variables detected (p-value  $< 0.05$ ). (B): The computation time is in seconds with (log10 scale) per core on 100 cores.
Based on Fig. 1 - S1, both methods Permfit-DNN and CPI-DNN have almost similar power. In high correlation regime, Permfit-DNN yields more detections, but it does not control type-I errors (Fig. 1). Regarding computation time, CPI-DNN is slightly more computationally expensive than Permfit-DNN.
G Supplement Experiment 5.2 - Models
Classification The signal  $\mathbf{X}\beta^{main}$  is turned to binomial variables using the probit function  $\Phi$ .  $\beta^{main}$  and  $\beta^{quad}$  are the two vectors with different lengths of regression coefficients having only  $n_{\mathrm{signal}} = 20$  non-zero coefficients, the true model.  $\beta^{main}$  is used with the main effects while  $\beta^{quad}$  is involved with the interaction effects. Following [Janitza et al., 2018], the  $\beta$  values  $\in \{\beta^{main},\beta^{quad}\}$  are drawn i.i.d. from the set  $\mathcal{B} = \{\pm 3,\pm 2,\pm 1,\pm 0.5\}$ .
$$
y _ {i} \sim \operatorname {B i n o m i a l} (\Phi (\boldsymbol {x} _ {i} \beta^ {\text {m a i n}})), \forall i \in [   [ n ]   ]
$$
Plain linear model We rely on a linear model, where  $\beta^{main}$  is drawn as previously and  $\epsilon$  is the Gaussian additive noise  $\sim \mathcal{N}(0,\mathbf{I})$  with magnitude  $\sigma = \frac{\|\mathbf{X}\boldsymbol{\beta}^{main}\|_2}{SNR\sqrt{n}}$ :  $y_{i} = \mathbf{x}_{i}\boldsymbol{\beta}^{main} + \sigma \epsilon_{i}, \forall i \in [n]$ .
Regression with ReLu An extra ReLu function is applied to the output of the Plain linear model:  $y_{i} = \text{Relu}(\mathbf{x}_{\mathrm{i}}\boldsymbol{\beta}^{\text{main}} + \sigma \epsilon_{i}), \forall i \in [[n]]$ .
Interactions only model We compute the product of each pair of variables. The corresponding values are used as inputs to a linear model:  $y_{i} = \mathrm{quad}(\mathbf{x}_{i},\boldsymbol{\beta}^{\mathrm{quad}}) + \sigma \epsilon_{i}, \forall i \in [[n]]$ , where  $\mathrm{quad}(\boldsymbol{x}_i,\boldsymbol{\beta}^{\mathrm{quad}}) = \sum_{\substack{k,j = 1\\ k < j}}^{p_{\mathrm{signals}}} \beta^{\mathrm{quad}}_{k,j}x_i^k x_i^j$ . The magnitude  $\sigma$  of the noise is set to  $\frac{||\mathrm{quad}(\boldsymbol{X},\boldsymbol{\beta}^{\mathrm{quad}})||_2}{SNR\sqrt{n}}$ .
The non-zero  $\beta^{quad}$  coefficients are drawn uniformly from  $\mathcal{B}$ .
Main effects with Interactions We combine both Main and Interaction effects. The magnitude  $\sigma$  of the noise is set to  $\frac{||\mathbf{X}\boldsymbol{\beta}^{\mathrm{main}} + \mathrm{quad}(\mathbf{X},\boldsymbol{\beta}^{\mathrm{quad}})||_2}{SNR\sqrt{n}}$ :  $y_{i} = \mathbf{x}_{i}\boldsymbol{\beta}^{main} + \mathrm{quad}(\mathbf{x}_{i},\boldsymbol{\beta}^{\mathrm{quad}}) + \sigma \epsilon_{i}$ ,  $\forall i\in [[n]]$ .
H Supplement Figure 3 - Extended model comparisons
We also benchmarked the following methods deprived of statistical guarantees:
- Knockoffs [Candes et al., 2017, Nguyen et al., 2020]: The knockoff filter is a variable selection method for multivariate models that controls the False Discovery Rate. The first step of this procedure involves sampling extra null variables that have a correlation structure similar to that of the original variables. A statistic is then calculated to measure the strength of the original variables versus their knockoff counterpart. We call this the knockoff statistic  $\mathbf{w} = \{w_{j}\}_{j=1}^{p}$  that is the difference between the importance of a given feature and the importance of its knockoff.
- Approximate Shapley values [Burzykowski, 2020]: SHAP being an instance method, we relied on an aggregation (averaging) of the per-sample Shapley values.
- Shapley Additive Global importance (SAGE) [Covert et al., 2020]: Whereas SHAP focuses on the local interpretation by aiming to explain a model's individual predictions, SAGE is an extension to SHAP assessing the role of each feature in a global interpretability manner. The SAGE values are derived by applying the Shapley value to a function that represents the predictive power contained in subsets of features.
- Mean Decrease of Impurity [Louppe et al., 2013]: The importance scores are related to the impact that each feature has on the impurity function in each of the nodes.
- BART [Chipman et al., 2010]: BART is an ensemble of additive regression trees. The trees are built iteratively using a back-fitting algorithm such as MCMC (Markov Chain Monte Carlo). By keeping track of covariate inclusion frequencies, BART can identify which components are more important for explaining y.
Based on AUC, we observe SHAP, SAGE and Mean Decrease of Impurity (MDI) perform poorly. These approaches are vulnerable to correlation. Next, Knockoff-Deep and Knockoff-Lasso perform well when the model does not include interaction effects. BART and Knockoff-Bart show fair performance overall.
Figure 3 - S1: Extended model comparisons: State-of-the-art methods for variable importance not providing statistical guarantees in terms of p-values are compared (outer columns) and to competing approaches across data-generating scenarios (inner columns) using the settings of experiments 2 and 3. Prediction tasks were simulated with  $n = 1000$  and  $p = 50$ . Solid line: chance level.
I Supplement Figure 3 - Power
Figure 3 - S2: Extended model comparisons: CPI-DNN and Perfit-DNN were compared to baseline models (outer columns) and to competing approaches across data-generating scenarios (inner columns). Convention about power as in Fig. 1 - S1. Prediction tasks were simulated with  $n = 1000$  and  $p = 50$ .
Based on the power computation, Pernfit-DNN and CPI-DNN outperform the alternative methods. Thus, the use of the right learner leads to better interpretations.
J Supplement Figure 3 - Computation time
Figure 3 - S3: Extended model comparisons: The computation times for the different methods (with and without statistical guarantees in terms of p-values) are reported in seconds with (log10 scale) per core on 100 cores. Prediction tasks were simulated with  $n = 1000$  and  $p = 50$ .
The computation time of the different methods mentioned in this work (with and without statistical guarantees) is presented in Fig. 3 - S3 in seconds with (log10 scale). First, we compare CPI-RF, cpi-knockoff and LOCO based on a Random Forest learner with  $p = 50$ . We see that cpi-knockoff and LOCO are faster than CPI-DNN. A possible reason is that CPI-DNN uses an inner 2-fold internal validation for hyperparameter tuning (learning rate, L1 and L2 regularization) unlike the alternatives. Next, The DNN-based methods (CPI-DNN and Perfit-DNN) are competitive with the alternatives that control type-I error ( $d_0CRT$ , cpi-knockoff and LOCO) despite the use of computationally lean learners in the latter.
K Supplement Figure 3 - Prediction scores on simulated data
Regression with ReLu
Prediction score (R2, AUC)
Main effects and Interactions
Prediction score (R2, AUC)
Figure 3 - S4: Evaluating predictive power: Performance of the different base learners used in the variable importance methods (Marginal = {Marginal effects}, Lasso = {Knockoff-Lasso}, Random Forest = {MDI, d0CRT, CPI-RF, Conditional-RF, cpi-knockoff, LOCO}, BART = {Knockoff-BART, BART} and DNN = {Knockoff-Deep, Perfit-DNN, CPI-DNN, Lazy VI}) on simulated data with  $n = 1000$  and  $p = 50$  in terms of ROC-AUC score for the classification and R2 score for the regression.
Data Problem
Classification
Interactions only
Main effects and Interactions
Plain linear
Regression with ReLu
The results for computing the prediction accuracy using the underlying learners of the different methods are reported in Fig. 3 - S4. Marginal inference, performs poorly, as it is not a predictive approach. Linear models based on Lasso show a good performance in the no-interaction effect scenario. Non-linear models based on Random Forest and BART improve on the lasso-based models. Nevertheless, they fail to achieve a good performance in scenarios with interaction effects. The models equipped with a deep learner outperform the other methods.
L Large scale simulations
Figure E3: Semi-simulation with UK Biobank: (Top panel) Performance of CPI-DNN and Perfit-DNN is compared in terms of AUC score, Type-I error, Power and Time on the data from UKBB with  $n = 8357$  and  $p = 671$ . (Bottom panel) Correlation strength among the variables in the UKBB dataset.
Figure E4: Large scale simulation: Performance of CPI-DNN and Perfit-DNN is compared in terms of AUC score, Type-I error, Power and Time on simulated data with  $n = 10000$ ,  $p = 50$  and  $\rho = 0.8$ .
In Figs. E3 and E4, we provide a comparison of the performance of both Permfit-DNN and CPI-DNN on the semi-simulated data from UK Biobank, with the design matrix consisting of the variables in the UK BioBank and the outcome is generated following a random selection of the true support, where  $n = 8357$  and  $p = 671$ , and a large scale simulation with  $n = 10000$ ,  $p = 50$  and block-based correlation of coefficient  $\rho = 0.8$ . For the UKBB-based simulation, we see that CPI-DNN achieves a higher AUC score and Power. However, both methods control the type-I error at the targeted level. To better understand the reason, we plotted (Fig. E3 Bottom panel) the histogram of the correlation values within the UKBB data: in this case, we consider a low-correlation setting which explains the good control for Permfit-DNN. In the large scale simulation where the correlation coefficient is set to 0.8, the difference is clear and only CPI-DNN controls the type-I error.
M Age prediction from brain activity (MEG) in Cam-CAN dataset
Age-prediction from MEG (MAE=7.89, R2=0.69)
Figure E5: Age prediction from brain activity: Predicting age from brain activity in different frequencies with  $n = 536$  and  $p = 4032$ .
Following the work of Engemann et al. [2020], we have applied CPI-DNN to the problem of age prediction from brain activity in different frequencies recorded with magnetoencephalography (MEG) in the Cam-CAN dataset. Without tweaking, the DNN learner reached a prediction performance on par with the published results as seen in Fig. E5. The p-values formally confirm aspects of the exploratory analysis in the original publication (importance of beta band).
N Practical validation of the normal distribution assumption
Figure E6: CPI-DNN vs Perfit-DNN p-values calibration: Q-Q plot for the distribution of the p-values vs the uniform distribution with  $n = 1000$  and  $p = 50$ .
Figure E7: Normal distribution assumption: Histogram plots of the distribution of the importance scores of a random picked non-significant variable with  $n = 1000$  and  $p = 50$ .
In Fig. E7, we compared the distribution of the importance scores of a random picked non-significant variable using CPI-DNN and Perfit-DNN through histogram plots, and we can emphasize that the normal distribution assumption holds in practice.
Also, in Fig. E6, we plot the distribution of the p-values provided by CPI-DNN and Permfit-DNN vs the uniform distribution through QQ-plot. We can see that the p-values for CPI-DNN are well calibrated and slightly deviated towards higher values. However, with Permfit-DNN the p-values are not calibrated.
O Random Forest for modeling the conditional distribution and resulting calibration
Figure E8: Random forest calibration: Calibration of the p-values for CPI-DNN (left panel) and the control of type-I error (right panel) as a function of the complexity of the Random Forest (the max depth of the trees). Dashed line: targeted type-I error rate. Solid line: uniform distribution.
The use of the Random Forest model was to maintain a good non-linear model with time benefits for the prediction of the conditional distribution of the variable of interest. In Fig. E8, We can see that reducing the depth to 1 or 2, thus making the model overly simple, breaks the control of the type-I errors at the targeted level. With larger depths, the model becomes more conservative. Therefore, the max depth of the Random Forest is chosen based on the performance with 2-fold cross validation.