SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Theorem 1. Let Assumption 1 hold for distributional distance  $\rho(A_1, A_2)$  between distributions  $A_1$  and  $A_2$ . For any  $t > 0$ ,  $j \in \{0, \dots, p\}$  as  $\rho(LD^*(\cdot; \ell, n, \lambda), RLD(\cdot; \varepsilon, \mathcal{F}, \ell, \mathcal{P}_n, \lambda)) \to 0$  and  $B \to \infty$ ,
$$
\mathbb {P} \left(\left| \widehat {R I D} _ {j} (k; \varepsilon , \mathcal {F}, \ell , \mathcal {P} _ {n}, \lambda) - R I D _ {j} (k; \varepsilon , \{g ^ {*} \}, \ell , \mathcal {P} _ {n}, \lambda) \right| \geq t\right)\rightarrow 0.
$$
For a set of models that performs sufficiently well in terms of loss,  $\widehat{RID}_j$  thus recovers the CDF of variable importance for the true model across all reasonable perturbations. Further, we can provide a finite sample bound for the estimation of a marginal distribution between  $\widehat{RID}_j$  and  $RID_j$  for the model class  $\mathcal{F}$ , as stated in Theorem 2. Note that this result does not require Assumption 1.
Theorem 2. Let  $t > 0$  and  $\delta \in (0,1)$  be some pre-specified values. Then, with probability at least  $1 - \delta$  with respect to bootstrap samples of size  $n$ ,
$$
\left| \widehat {R I D} _ {j} (k; \varepsilon , \mathcal {F}, \ell , \mathcal {P} _ {n}, \lambda) - R I D _ {j} (k; \varepsilon , \mathcal {F}, \ell , \mathcal {P} _ {n}, \lambda) \right| \leq t \tag {3}
$$
with number of bootstrap samples  $B \geq \frac{1}{2t^2} \ln \left(\frac{2}{\delta}\right)$  for any  $k \in [\phi_{\min}, \phi_{\max}]$ .
Because we use a bootstrap procedure, we can control the number of bootstrap iterations to ensure that the difference between  $RID_{j}$  and  $\widehat{RID}_{j}$  is within some pre-specified error. (As defined earlier,  $RID_{j}$  is the expectation over infinite bootstraps, whereas  $\widehat{RID}_{j}$  is the empirical average over  $B$  bootstraps.) For example, after 471 bootstrap iterations, we find that  $\widehat{RID}_{j}(k; \varepsilon, \mathcal{F}, \ell, \mathcal{P}_{n}, \lambda)$  is within 0.075 of  $RID_{j}(k; \varepsilon, \mathcal{F}, \ell, \mathcal{P}_{n}, \lambda)$  for any given  $k$  with  $90\%$  confidence. It also follows that as  $B$  tends to infinity, the estimated  $RIV_{j}$  will converge to the true value.
Since we stably estimate the entire distribution of variable importance values, we can create (1) stable point estimates of variable importance (e.g., expected variable importance) that account for the Rashomon effect, (2) interquantile ranges of variable importance, and (3) confidence regions that characterize uncertainty around a point estimate of variable importance. We prove exponential rates of convergence for these statistics estimated using our framework in Section B of the supplement.
Because our estimand and our estimation strategy (1) enable us to manage instability, (2) account for the Rashomon effect, and (3) are completely model-agnostic and flexibly work with most existing variable importance metrics,  $RID$  is a valuable quantification of variable importance.
4 Experiments With Known Data Generation Processes
Figure 3: (Top) The proportion of features ranked correctly by each method on each data set represented as a stacked barplot. The figures are ordered by method performance across the four simulation setups. (Bottom) The proportion of independent DGP  $\phi^{(sub)}$  calculations on 500 new datasets from the DGP that were contained within the box-and-whiskers range computed using a single training set (with bootstrapping in all methods except VIC) for each method and variable in each simulation. Underneath each method's label, the first row shows the percentage of times across all 500 independently generated datasets and variables that the DGP's variable importance was inside of that method's box-and-whiskers interval. The second row shows the percentage of pairwise rankings correct for each method (from the top plot). Higher is better.
4.1 RID Distinguishes Important Variables from Extraneous Variables
There is no generally accepted ground truth measure for variable importance, so we first evaluate whether a variety of variable importance methods can correctly distinguish between variables used to generate the outcome (in a known data generation process) versus those that are not. We consider the following four data generation processes (DGPs). Chen's DGP [9]:  $Y = \mathbb{1}[-2\sin(X_1) + \max(X_2, 0) + X_3 + \exp(-X_4) + \varepsilon \geq 2.048]$ , where  $X_1, \ldots, X_{10}, \varepsilon \sim \mathcal{N}(0, 1)$ . Here, only  $X_1, \ldots, X_4$  are relevant. Friedman's DGP [17]:  $Y = \mathbb{1}[10\sin(\pi X_1X_2) + 20(X_3 - 0.5)^2 + 10X_4 + 5X_5 + \varepsilon \geq 15]$ , where  $X_1, \ldots, X_6 \sim \mathcal{U}(0, 1)$ ,  $\varepsilon \sim \mathcal{N}(0, 1)$ . Here, only  $X_1, \ldots, X_5$  are relevant. The Monk 1 DGP [42]:  $Y = \max(\mathbb{1}[X_1 = X_2], \mathbb{1}[X_5 = 1])$ , where the variables  $X_1, \ldots, X_6$  have domains of 2, 3, or 4 unique integer values. Only  $X_1, X_2, X_5$  are important. The Monk 3 DGP [42]:  $Y = \max(\mathbb{1}[X_5 = 3 \text{ and } X_4 = 1], \mathbb{1}[X_5 \neq 4 \text{ and } X_2 \neq 3])$  for the same covariates in Monk 1. Also,  $5\%$  label noise is added. Here,  $X_2, X_4$ , and  $X_5$  are relevant.
We compare the ability of  $RID$  to identify extraneous variables with that of the following baseline methods, whose details are provided in Section D of the supplement: subtractive model reliance  $\phi^{\mathrm{sub}}$  of a random forest (RF) [6], LASSO [20], boosted decision trees [16], and generalized optimal sparse decision trees (GOSDT) [26]; conditional model reliance (CMR) [15]; the impurity based model reliance metric for RF from [7]; the LOCO algorithm reliance [25] for RF and Lasso; the Pearson and Spearman correlation between each feature and the outcome; the mean of the partial dependency plot (PDP) [19] for each feature; the SHAP value [28] for RF; and mean of variable importance clouds (VIC) [12] for the Rashomon set of GOSDTs [49]. If we do not account for instability and simply learn a model and calculate variable importance, baseline models generally perform poorly, as shown
in Section E of the supplement. Thus, we chose to account for instability in a way that benefits the baselines. We evaluate each baseline method for each variable across 500 bootstrap samples and compute the median VI across bootstraps, with the exception of VIC — for VIC, we take the median VI value across the Rashomon set for the original dataset, as VIC accounts for Rashomon uncertainty. Here, we aim to see whether we can identify extraneous (i.e., unimportant variables). For a DGP with  $C$  extraneous variables, we classify the  $C$  variables with the  $C$  smallest median variable importance values as extraneous. We repeat this experiment with different values for the Rashomon threshold  $\varepsilon$  in Section E of the supplement.
Figure 3 (top) reports the proportion of variables that are correctly classified for each simulation setup as a stacked barplot.  $RID$  identifies all important and unimportant variables for these complex simulations. Note that four other baseline methods – MR RF, RF Impurity, RF Permute, and VIC – also differentiated all important from unimportant variables. Motivated by this finding, we next explore how well methods recover the true value for subtractive model reliance on the DGP, allowing us to distinguish between the best performing methods on the classification task.
4.2 RID Captures Model Reliance for the True Data Generation Process
$RID$  allows us to quantify uncertainty in variable importance due to both the Rashomon effect and instability. We perform an ablation study investigating how accounting for both stability and the Rashomon effect compares to having one without the other. We evaluate what proportion of subtractive model reliances calculated for the DGP on 500 test sets are contained within uncertainty intervals generated using only one training dataset. This experiment tells us whether the intervals created on a single dataset will generalize.
To create the uncertainty interval on the training dataset and for each method, we first find the subtractive model reliance  $\phi^{(sub)}$  across 500 bootstrap iterations of a given dataset for the four algorithms shown in Figure 3 (bottom) (baseline results without bootstrapping are in Section E of the supplementary material). Additionally, we find the VIC for the Rashomon set of GOSDTs on the original dataset. We summarize these model reliances (500 bootstraps  $\times$  28 variables across datasets  $\times$  4 algorithms  $+8,247$  models in VIC's  $+10,840,535$  total models across Rsets  $\times$  28 variables from  $RID$ ) by computing their box-and-whisker ranges ( $1.5 \times$  Interquartile range [46]). To compare with "ground truth," we sample 500 test datasets from the DGP and calculate  $\phi^{(sub)}$  for the DGP for that dataset. For example, assume the DGP is  $Y = X^2 + \varepsilon$ . We would then use  $f(X) = X^2$  as our predictive model and evaluate  $\phi^{(sub)}(f, \mathcal{D}^{(n)})$  on  $f$  for each of the 500 test sets. We then check if the box-and-whisker range of each method's interval constructed on the training set contains the computed  $\phi^{(sub)}$  for the DGP for each test dataset. Doing this allows us to understand whether our interval contains the true  $\phi^{(sub)}$  for each test set.
Figure 3 (bottom) illustrates the proportion of times that the test variable importance values fell within the uncertainty intervals from training. These baselines fail to capture the test  $\phi^{(sub)}$  entirely for at least one variable ( $< 0.05\%$  recovery proportion). Only  $RID$  both recovers important/unimportant classifications perfectly and achieves a strong recovery proportion at  $95\%$ .
4.3  $RID$  is Stable
Our final experiment investigates the stability of VIC and MCR (which capture only Rashomon uncertainty but not stability) to  $RID$ , which naturally considers data perturbations. We generate 50 independent datasets from each DGP and compute the box-and-whisker ranges (BWR) of each uncertainty metric for each dataset; for every pair of BWRs for a given method, we then calculate the Jaccard similarity between BWR's. For each generated dataset, we then average the Jaccard similarity across variables. Figure 4 shows these intervals for each non-extraneous variable from Chen's DGP. Supplement E.4 presents a similar figure for each DGP, showing that only  $RID$ 's intervals overlap across all generations for all datasets.
Figure 5 displays the similarity scores between the box and whisker ranges of MCR, VIC, and  $RID$  across the 50 datasets for each DGP. Note that Monk 1 has no noise added, so instability should not be a concern for any method. For datasets including noise, MCR and VIC achieve median similarity below 0.55;  $RID$ 's median similarity is 0.69; it is much more stable.
Figure 4: We generate 50 independent datasets from Chen's DGP and calculate MCR, BWRs for VIC, and BWRs for RID. The above plot shows the interval for each dataset for each non-null variable in Chen's DGP. All red-colored intervals do not overlap with at least one of the remaining 49 intervals.
Figure 5: Median Jaccard similarity scores across 50 independently generated MCR, VIC, and RID box and whisker ranges for each DGP; 1 is perfect similarity. Error bars show  $95\%$  confidence interval around the median.
5 Case Study
Having validated  $RID$  on synthetic datasets, we demonstrate its utility in a real world application: studying which host cell transcripts and chromatin patterns are associated with high expression of Human Immunodeficiency Virus (HIV) RNA. We analyzed a dataset that combined single cell RNAseq/ATACseq profiles for 74,031 individual HIV infected cells from two different donors in the aims of finding new cellular cofactors for HIV expression that could be targeted to reactivate the latent HIV reservoir in people with HIV (PWH). A longer description of the data is in [29]. Finding results on this type of data allows us to create new hypotheses for which genes are important for HIV load prediction and might generalize to large portions of the population.
To identify which genes are stably importance across good models, we evaluated this dataset using  $RID$  over the model class of sparse decision trees using subtractive model reliance. We selected 14,614 samples (all 7,307 high HIV load samples and 7,307 random low HIV load samples) from the overall dataset in order to balance labels, and filtered the complete profiles down to the top 100 variables by individual AUC. We consider the binary classification problem of predicting high versus low HIV load. For full experimental details, see Section D of the supplement. Section E.5 of the supplement contains timing experiments for  $RID$  using this dataset.
Figure 6 illustrates the probability that  $RID$  is greater than 0 for the 10 highest probability variables (0 is when the variable is not important at all). We find that LINC00486 - a less explored gene - is the most important variable, with  $1 - RID_{LINC00486}(0) = 78.4\%$ . LINC00486 is a long non-coding RNA (i.e., it functions as an RNA molecule but does not encode a protein like most genes), and there is no literature on this gene and HIV, making this association a novel one. However, recent work [45] has shown that LINC00486 can enhance EBV (Epstein-Barr virus) infection by activating NF- $\kappa$ B.
Figure 6: Probability of each gene's model reliance being greater than 0 across Rashomon sets across bootstrapped datasets for the ten genes with the highest  $\mathbb{P}(RIV_j > 0)$ . We ran 738 bootstrap iterations to ensure that  $\mathbb{P}(\widehat{RIV}_j > 0)$  is within 0.05 of  $\mathbb{P}(RIV_j > 0)$  with  $95\%$  confidence (from Theorem 2).
It is well established that NF- $\kappa$ B can regulate HIV expression [32, 23, 4], suggesting a possible mechanism and supporting future study. Notably,  $RID$  also highlighted PDE4D, which interacts with the Tat protein and thereby HIV transcription [39]; HNRNPA2B1, which promotes HIV expression by altering the structure of the viral promoter [38]; and MALAT1, which has recently been shown to be an important regulator of HIV expression [35]. These three findings validate prior work and show that  $RID$  can uncover variables that are known to interact with HIV.
Note that previous methods – even those that account for the Rashomon effect – could not produce this result. MCR and VIC do not account for instability. For example, after computing MCR for 738 bootstrap iterations, we find that the MCR for the LINC00486 gene has overlap with 0 in  $96.2\%$  of bootstrapped datasets, meaning MCR would not allow us to distinguish whether LINC00486 is important or not  $96.2\%$  of the time. Without  $RID$ , we would not have strong evidence that LINC00486 is necessary for good models. By explicitly accounting for instability, we increase trust in our analyses.
Critically,  $RID$  also found very low importance for the majority of variables, allowing researchers to dramatically reduce the number of possible directions for future experiments designed to test a gene's functional role. Such experiments are time consuming and cost tens of thousands of dollars per donor, so narrowing possible future directions to a small set of genes is of the utmost importance. Our analysis provides a manageable set of clear directions for future work studying the functional roles of these genes in HIV.
We introduced  $RID$ , a method for recovering the importance of variables in a way that accounts for both instability and the Rashomon effect. We showed that  $RID$  distinguishes between important and extraneous variables, and that  $RID$  better captures the true variable importance for the DGP than prior methods. We showed through a case study in HIV load prediction that  $RID$  can provide insight into complicated real world problems. Our framework overcomes instability and the Rashomon effect, moving beyond variable importance for a single model and increasing reproducibility.
$RID$  can be directly computed for any model class for which the Rashomon set can be found – at the time of publishing, decision trees, linear models, and GLMs. A limitation is that currently, there are relatively few model classes for which the Rashomon set can be computed. Therefore, future work should aim to compute and store the Rashomon set of a wider variety of model classes. Future work may investigate incorporating Rashomon sets that may be well-approximated (e.g., GAMs, [10]), but not computed exactly, into the  $RID$  approach. Nonetheless, sparse trees are highly flexible, and using them with  $RID$  improves the trustworthiness and transparency of variable importance measures, enabling researchers to uncover important, reproducible relationships about complex processes without being misled by the Rashomon effect.
We gratefully acknowledge support from NIH/NIDA R01DA054994, NIH/NIAID R01AI143381, DOE DE-SC0023194, NSF IIS-2147061, and NSF IIS-2130250.
[45] Wen-Fang Wang, Hui-Juan Zhong, Shu Cheng, Di Fu, Yan Zhao, Hua-Man Cai, Jie Xiong, and Wei-Li Zhao. A nuclear NKRF interacting long noncoding RNA controls EBV eradication and suppresses tumor progression in natural killer/T-cell lymphoma. Biochimica et Biophysica Acta (BBA)-Molecular Basis of Disease, page 166722, 2023.
[46] Wikipedia contributors. Box plot, 2023. URL https://en.wikipedia.org/wiki/Box_plot. [Online; accessed 14-May-2023].
[47] Brian Williamson and Jean Feng. Efficient nonparametric statistical inference on population feature importance using shapley values. In International Conference on Machine Learning, pages 10282-10291. PMLR, 2020.
[48] Brian D Williamson, Peter B Gilbert, Noah R Simon, and Marco Carone. A general framework for inference on algorithm-agnostic variable importance. Journal of the American Statistical Association, pages 1–14, 2021.
[49] Rui Xin, Chudi Zhong, Zhi Chen, Takuya Takagi, Margo Seltzer, and Cynthia Rudin. Exploring the whole rashomon set of sparse decision trees. Advances in Neural Information Processing Systems, 35: 22305-22315, 2022.
[50] Bin Yu. Stability. Bernoulli, 19(4):1484 - 1500, 2013. doi: 10.3150/13-BEJSP14.
[51] Bin Yu and Karl Kumbier. Veridical data science. Proceedings of the National Academy of Sciences, 117 (8):3920-3929, 2020. doi: 10.1073/pnas.1901326117.
[52] Lu Zhang and Lucas Janson. Floodgate: inference for model-free variable importance. arXiv preprint arXiv:2007.01283, 2020.