SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Jon Donnelly*
Durham, NC 27708
jon.donnelly@duke.edu
Srikar Katta*
srikar.katta@duke.edu
Cynthia Rudin
cynthiaa.rudin@duke.edu
Edward P. Browne
Chapel Hill, NC 27599
epbrowne@email.unc.edu
Quantifying variable importance is essential for answering high-stakes questions in fields like genetics, public policy, and medicine. Current methods generally calculate variable importance for a given model trained on a given dataset. However, for a given dataset, there may be many models that explain the target outcome equally well; without accounting for all possible explanations, different researchers may arrive at many conflicting yet equally valid conclusions given the same data. Additionally, even when accounting for all possible explanations for a given dataset, these insights may not generalize because not all good explanations are stable across reasonable data perturbations. We propose a new variable importance framework that quantifies the importance of a variable across the set of all good models and is stable across the data distribution. Our framework is extremely flexible and can be integrated with most existing model classes and global variable importance metrics. We demonstrate through experiments that our framework recovers variable importance rankings for complex simulation setups where other methods fail. Further, we show that our framework accurately estimates the true importance of a variable for the underlying data distribution. We provide theoretical guarantees on the consistency and finite sample error rates for our estimator. Finally, we demonstrate its utility with a real-world case study exploring which genes are important for predicting HIV load in persons with HIV, highlighting an important gene that has not previously been studied in connection with HIV.
Variable importance analysis enables researchers to gain insight into a domain or a model. Scientists are often interested in understanding causal relationships between variables, but running randomized experiments is time-consuming and expensive. Given an observational dataset, we can use global variable importance measures to check if there is a predictive relationship between two variables. It is particularly important in high stakes real world domains such as genetics [44, 34], finance [37], and criminal justice [15, 24] where randomized controlled trials are impractical or unethical. Variable
*Jon Donnelly and Srikar Katta contributed equally to this work.
(a) The number of models in each Rashomon set
(b) Model reliance range across each Rashomon set
Figure 1: Statistics of Rashomon sets computed across 500 bootstrap replicates of a given dataset sampled from the Monk 3 data generation process [42]. The original dataset consisted of 124 observations, and the Rashomon set was calculated using its definition in Equation 1, with parameters specified in Section D of the supplement. The Rashomon set size is the number of models with loss below a threshold. Model reliance is a measure of variable importance for a single variable — in this case,  $X_{2}$  — and Model Class Reliance (MCR) is its range over the Rashomon set. Both the Rashomon set size and model class reliance are unstable across bootstrap iterations.
importance would ideally be measured as the importance of each variable to the data generating process. However, the data generating process is never known in practice, so prior work generally draws insight by analyzing variable importance for a surrogate model, treating that model and its variable importance as truth.
This approach can be misleading because there may be many good models for a given dataset – a phenomenon referred to as the Rashomon effect [7, 40] — and variables that are important for one good model on a given dataset are not necessarily important for others. As such, any insights drawn from a single model need not reflect the underlying data distribution or even the consensus among good models. Recently, researchers have sought to overcome the Rashomon effect by computing Rashomon sets, the set of all good (i.e., low loss) models for a given dataset [15, 12]. However, the set of all good models is not stable across reasonable perturbations (e.g., bootstrap or jackknife) of a single dataset, with stability defined as in [50]. This concept of stability is one of the three pillars of veridical data science [51, 13]. Note that there is wide agreement on the intuition behind stability, but not its quantification [22, 33]. As such, in line with other stability research, we do not subscribe to a formal definition and treat stability as a general notion [22, 33, 50, 51]. In order to ensure trustworthy analyses, variable importance measures must account for both the Rashomon effect and stability.
Figure 1 provides a demonstration of this problem: across 500 bootstrap replicates from the same data set, the Rashomon set varies wildly – ranging from ten models to over ten thousand – suggesting that we should account for its instability in any computed statistics. This instability is further highlighted when considering the Model Class Reliance (MCR) variable importance, which is the range of model reliance (i.e., variable importance) values across the Rashomon set for the given dataset [15] (we define MCR and the Rashomon set more rigorously in Sections 2 and 3 respectively). In particular, for variable  $X_{2}$ , one interval – ranging from -0.1 to 0.33 – suggests that there exist good models that do not depend on this variable at all (0 indicates the variable is not important); on the other hand, another MCR from a bootstrapped dataset ranges from 0.33 to 0.36, suggesting that this variable is essential to all good models. Because of this instability, different researchers may draw very different conclusions about the same data distribution even when using the same method.
In this work, we present a framework unifying concepts from classical nonparametric estimation with recent developments on Rashomon sets to overcome the limitations of traditional variable importance measurements. We propose a stable, model- and variable-importance-metric-agnostic estimand that quantifies variable importance across all good models for the empirical data distribution and a corresponding bootstrap-style estimation strategy. Our method creates a cumulative density function (CDF) for variable importance over all variables via the framework shown in Figure 2. Using the CDF, we can compute a variety of statistics (e.g., expected variable importance, interquartile range, and credible regions) that can summarize the variable importance distribution.
The rest of this work is structured as follows. After more formally introducing our variable importance framework, we theoretically guarantee the convergence of our estimation strategy and derive error bounds. We also demonstrate experimentally that our estimand captures the true variable importance for the data generating process more accurately than previous work. Additionally, we illustrate the
Step 1: Bootstra
Figure 2: An overview of our framework. Step 1: We bootstrap multiple datasets from the original. Step 2: We show the loss values over the model class for each bootstrapped dataset, differentiated by color. The dotted line marks the Rashomon threshold; all models whose loss is under the threshold are in the Rashomon set for that bootstrapped dataset. On top, we highlight the number of bootstrapped datasets for which the corresponding model is in the Rashomon set. Step 3: We then compute the distribution of model reliance (variable importance – VI) values for variable  $j$  across the Rashomon set for each bootstrapped dataset. Step 4: We then average the corresponding CDF across bootstrap replicates into a single CDF (in purple). Step 5: Using the CDF, we compute the marginal distribution (PDF) of variable importance for variable  $j$  across the Rashomon sets of bootstrapped datasets.
generalizability of our variable importance metric by analyzing the reproducibility of our results given new datasets from the same data generation process. Lastly, we use our method to analyze which transcripts and chromatin patterns in human T cells are associated with high expression of HIV RNA. Our results suggest an unexplored link between the LINC00486 gene and HIV load.
Code is available at https://github.com/jdonnelly36/Rashomon_Importance_Distribution.
The key difference between our work and most others is the way it incorporates model uncertainty, also called the Rashomon effect [7]. The Rashomon effect is the phenomenon in which many different models explain a dataset equally well. It has been documented in high stakes domains including healthcare, finance, and recidivism prediction [14, 30, 15]. The Rashomon effect has been leveraged to create uncertainty sets for robust optimization [43], to perform responsible statistical inference [11], and to gauge whether simple yet accurate models exist for a given dataset [40]. One barrier to studying the Rashomon effect is the fact that Rashomon sets are computationally hard to calculate for non-trivial model classes. Only within the past year has code been made available to solve for (and store) the full Rashomon set for any nonlinear function class – that of decision trees [49]. This work enables us to revisit the study of variable importance with a new lens.
A classical way to determine the importance of a variable is to leave it out and see if the loss changes. This is called algorithmic reliance [15] or leave one covariate out (LOCO) inference [25, 36]. The problem with these approaches is that the performance of the model produced by an algorithm will not change if there exist other variables correlated with the variable of interest.
Model reliance (MR) methods capture the global variable importance (VI) of a given feature for a specific model [15]. (Note that MR is limited to refer to permutation importance in [15], while we use the term MR to refer to any metric capturing global variable importance of a given feature and model. We use VI and MR interchangeably when the relevant model is clear from context.) Several methods for measuring the MR of a model from a specific model class exist, including the variable importance measure from random forest which uses out-of-bag samples [7] and Lasso regression coefficients [20]. Lundberg et al. [28] introduce a way of measuring MR in tree ensembles using SHAP [27]. Williamson et al. [48] develop MR based on the change in performance between the optimal model and the optimal model using a subset of features.
In addition to the metrics tied to a specific model class, many MR methods can be applied to models from any model class. Traditional correlation measures [20] can measure the linear relationship (Pearson correlation) or general monotonic relationship (Spearman correlation) between a feature and predicted outcomes for a model from any model class. Permutation model reliance, as discussed by [1, 15, 21], describes how much worse a model performs when the values of a given feature are permuted such that the feature becomes uninformative. Shapley-based measures of MR, such as those
of [47, 27], calculate the average marginal contribution of each feature to a model's predictions. A complete overview of the variable importance literature is beyond the scope of this work; for a more thorough review, see, for example, [2, 31]. Rather than calculating the importance of a variable for a single model, our framework finds the importance of a variable for all models within a Rashomon set, although our framework is applicable to all of these model reliance metrics.
In contrast, model class reliance (MCR) methods describe how much a class of models (e.g., decision trees) relies on a variable. Fisher et al. [15] uses the Rashomon set to provide bounds on the possible range of model reliance for good models of a given class. Smith et al. [41] analytically find the range of model reliance for the model class of random forests. Zhang and Janson [52] introduce a way to compute confidence bounds for a specific variable importance metric over arbitrary models, which Aufiero and Janson [3] extend so that it is applicable to a broad class of surrogate models in pursuit of computational efficiency. These methods report MCR as a range, which gives no estimate of variable importance – only a range of what values are possible. In contrast, Dong and Rudin [12] compute and visualize the variable importance for every member of a given Rashomon set in projected spaces, calculating a set of points; however, these methods have no guarantees of stability to reasonable data perturbations. In contrast, our framework overcomes these finite sample biases, supporting stronger conclusions about the underlying data distribution.
Related to our work from the stability perspective, Duncan et al. [13] developed a software package to evaluate the stability of permutation variable importance in random forest methods; we perform a similar exercise to demonstrate that current variable importance metrics computed for the Rashomon set are not stable. Additionally, Basu et al. [5] introduced iterative random forests by iteratively reweighting trees and bootstrapping to find stable higher-order interactions from random forests. Further, theoretical results have demonstrated that bootstrapping stabilizes many machine learning algorithms and reduces the variance of statistics [18, 8]. We also take advantage of bootstrapping's flexibility and properties to ensure stability for our variable importance.
3.1 Definitions and Estimands
Let  $\mathcal{D}^{(n)} = \{(X_i,Y_i)\}_{i=1}^n$  denote a dataset of  $n$  independent and identically distributed tuples, where  $Y_i \in \mathbb{R}$  denotes some outcome of interest and  $X_i \in \mathbb{R}^p$  denotes a vector of  $p$  covariates. Let  $g^*$  represent the data generating process (DGP) producing  $\mathcal{D}^{(n)}$ . Let  $f \in \mathcal{F}$  be a model in a model class (e.g., a tree in the set of all possible sparse decision trees), and let  $\phi_j(f,\mathcal{D}^{(n)})$  denote a function that measures the importance of variable  $j$  for a model  $f$  over a dataset  $\mathcal{D}^{(n)}$ . This can be any of the functions described earlier (e.g., permutation importance, SHAP). Our framework is flexible with respect to the user's choice of  $\phi_j$  and enables practitioners to use the variable importance metric best suited for their purpose; for instance, conditional model reliance [15] is best-suited to measure only the unique information carried by the variable (that cannot be constructed using other variables), whereas other metrics like subtractive model reliance consider the unconditional importance of the variable. Our framework is easily integrable with either of these. We only assume that the variable importance function  $\phi$  has a bounded range, which holds for a wide class of metrics like SHAP [27], permutation model reliance, and conditional model reliance. Finally, let  $\ell(f,\mathcal{D}^{(n)};\lambda)$  represent a loss function given  $f,\mathcal{D}^{(n)}$ , and loss hyperparameters  $\lambda$  (e.g., regularization). We assume that our loss function is bounded above and below, which is true for common loss functions like 0-1 classification loss, as well as for differentiable loss functions with covariates from a bounded domain.
In an ideal setting, we would measure variable importance using  $g^{*}$  and the whole population, but this is impossible because  $g^{*}$  is unknown and data is finite. In practice, scientists instead use the empirical loss minimizer for a specific dataset  $\hat{f}^{*} \in \arg \min_{f \in \mathcal{F}} \ell(f, \mathcal{D}^{(n)})$ ; however, several models could explain the same dataset equally well (i.e., the Rashomon effect). Rather than using a single model to compute variable importance, we propose using the entire Rashomon set. Given a single dataset  $\mathcal{D}^{(n)}$ , we define the Rashomon set for a model class  $\mathcal{F}$  and parameter  $\varepsilon$  as the set of all models in  $\mathcal{F}$  whose empirical losses are within some bound  $\varepsilon > 0$  of the empirical loss minimizer:
$$
\mathcal {R} (\epsilon , \mathcal {F}, \ell , \mathcal {D} ^ {(n)}, \lambda) = \left\{f \in \mathcal {F}: \ell \left(f, \mathcal {D} ^ {(n)}; \lambda\right) \leq \min  _ {f ^ {\prime} \in \mathcal {F}} \ell \left(f ^ {\prime}, \mathcal {D} ^ {(n)}; \lambda\right) + \varepsilon \right\}. \tag {1}
$$
We denote this Rashomon set by  $\mathcal{R}_{\mathcal{D}^{(n)}}^{\varepsilon}$  or "Rset" (this assumes a fixed  $\mathcal{F}$ ,  $\ell$  and  $\lambda$ ). As discussed earlier, the Rashomon set can be fully computed and stored for non-linear models (e.g., sparse decision trees [49]). For notational simplicity, we often omit  $\lambda$  from the loss function.
While the Rashomon set describes the set of good models for a single dataset, Rashomon sets vary across permutations (e.g., subsampling and resampling schemes) of the given data. We introduce a stable quantity for variable importance that accounts for all good models and all permutations from the data using the random variable  $RIV$ .  $RIV$  is defined by its cumulative distribution function (CDF), the Rashomon Importance Distribution (RID):
$$
\begin{array}{l} R I D _ {j} (k; \varepsilon , \mathcal {F}, \ell , \mathcal {P} _ {n}, \lambda) = \mathbb {P} _ {\mathcal {D} _ {b} ^ {(n)} \sim \mathcal {P} _ {n}} \left(R I V _ {j} (\varepsilon , \mathcal {F}, \ell , \mathcal {D} _ {b} ^ {(n)}; \lambda) \leq k\right) \tag {2} \\ := \mathbb {E} _ {\mathcal {D} _ {b} ^ {(n)} \sim \mathcal {P} _ {n}} \left[ \frac {| \{f \in \mathcal {R} _ {\mathcal {D} _ {b} ^ {(n)}} ^ {\varepsilon} : \phi_ {j} (f , \mathcal {D} _ {b} ^ {(n)}) \leq k \} |}{| \mathcal {R} _ {\mathcal {D} _ {b} ^ {(n)}} ^ {\varepsilon} |} \right] \\ = \mathbb {E} _ {\mathcal {D} _ {b} ^ {(n)} \sim \mathcal {P} _ {n}} \left[ \frac {\operatorname {v o l} \text {o f R s e t s . t . v a r i a b l e j ' s i m p o r t a n c e i s a t m o s t} k}{\operatorname {v o l} \text {o f R s e t}} \right], \\ \end{array}
$$
where  $\phi_{j}$  denotes the variable importance metric being computed on variable  $j$ ,  $k \in [\phi_{\min}, \phi_{\max}]$ . For a continuous model class (e.g., linear regression models), the cardinality in the above definition becomes the volume under a measure on the function class, usually  $\ell_2$  on parameter space. RID constructs the cumulative distribution function (CDF) for the distribution of variable importance across Rashomon sets; as  $k$  increases, the value of  $\mathbb{P}(RIV_j(\varepsilon, \mathcal{F}, \ell, \mathcal{D}_b^{(n)}; \lambda) \leq k)$  becomes closer to 1. The probability and expectation are taken with respect to datasets of size  $n$  sampled from the empirical distribution  $\mathcal{P}_n$ , which is the same as considering all possible resamples of size  $n$  from the originally observed dataset  $\mathcal{D}^{(n)}$ . Equation (2) weights the contribution of  $\phi_{j}(f, \mathcal{D}_{b}^{(n)})$  for each model  $f$  by the proportion of datasets for which this model is a good explanation (i.e., in the Rashomon set). Intuitively, this provides greater weight to the importance of variables for stable models.
We now define an analogous metric for the loss function  $\ell$ ; we define the Rashomon Loss Distribution (RLD) evaluated at  $k$  as the expected fraction of functions in the Rashomon set with loss below  $k$ . Here, RLV is a random variable following this CDF.
$$
\begin{array}{l} R L D (k; \varepsilon , \mathcal {F}, \ell , \mathcal {P} _ {n}, \lambda) = \mathbb {P} _ {\mathcal {D} _ {b} ^ {(n)} \sim \mathcal {P} _ {n}} \left(R L V (\varepsilon , \mathcal {F}, \ell , \mathcal {D} _ {b} ^ {(n)}; \lambda) \leq k\right) \\ := \mathbb {E} _ {\mathcal {D} _ {b} ^ {(n)} \sim \mathcal {P} _ {n}} \left[ \frac {| \{f \in \mathcal {R} _ {\mathcal {D} _ {b} ^ {(n)}} ^ {\varepsilon} : \ell (f , \mathcal {D} _ {b} ^ {(n)}) \leq k \} |}{| \mathcal {R} _ {\mathcal {D} _ {b} ^ {(n)}} ^ {\varepsilon} |} \right] \\ = \mathbb {E} _ {\mathcal {D} _ {b} ^ {(n)} \sim \mathcal {P} _ {n}} \left[ \frac {| \mathcal {R} (k - \min _ {f \in \mathcal {F}} \ell (f , \mathcal {D} _ {b} ^ {(n)} , \mathcal {F} , \ell ; \lambda) |}{| \mathcal {R} (\varepsilon , \mathcal {F} , \ell ; \lambda) |} \right]. \\ \end{array}
$$
This quantity shows how quickly the Rashomon set "fills up" on average as loss changes. If there are many near-optimal functions, this will grow quickly with  $k$ .
In order to connect  $RID$  for model class  $\mathcal{F}$  to the unknown DGP  $g^{*}$ , we make a Lipschitz-continuity-style assumption on the relationship between  $RLD$  and  $RID$  relative to a general model class  $\mathcal{F}$  and  $\{g^{*}\}$ . To draw this connection, we define the loss CDF for  $g^{*}$ , called  $LD^{*}$ , over datasets of size  $n$  as:
$$
L D ^ {*} (k; \ell , n, \mathcal {P} _ {n}, \lambda) := \mathbb {E} _ {\mathcal {D} _ {b} ^ {(n)} \sim \mathcal {P} _ {n}} \left[ \mathbb {1} [ \ell (g ^ {*}, \mathcal {D} _ {b} ^ {(n)}) \leq k ] \right].
$$
One could think of  $LD^{*}$  as measuring how quickly the DGP's Rashomon set fills up as loss changes. Here,  $LD^{*}$  is the analog of RLD for the data generation process.
Assumption 1. If
$$
\rho \left(R L D (\cdot ; \varepsilon , \mathcal {F}, \ell , \mathcal {P} _ {n}, \lambda), L D ^ {*} (\cdot ; \ell , n, \mathcal {P} _ {n}, \lambda)\right) \leq \gamma \text {t h e n}
$$
$$
\rho \left(R I D _ {j} (\cdot ; \varepsilon , \mathcal {F}, \ell , \mathcal {P} _ {n}, \lambda), R I D _ {j} (\cdot ; \varepsilon , \{g ^ {*} \}, \ell , \mathcal {P} _ {n}, \lambda)\right) \leq d (\gamma)
$$
for a function  $d:[0,\ell_{\max} - \ell_{\min}] \to [0,\phi_{\max} - \phi_{\min}]$  such that  $\lim_{\gamma \to 0}d(\gamma) = 0$ . Here,  $\rho$  represents any distributional distance metric (e.g., 1-Wasserstein).
Assumption 1 says that a Rashomon set consisting of good approximations for  $g^{*}$  in terms of loss will also consist of good approximations for  $g^{*}$  in terms of variable importance. More formally, from this assumption, we know that as  $\rho(LD^{*}, RLD) \to 0$ , the variable importance distributions will converge:  $\rho(RID_{j}(\cdot, \varepsilon, \mathcal{F}, \ell, \mathcal{P}_{n}, \lambda), RID_{j}(\cdot, \varepsilon, \{g^{*}\}, \ell, \mathcal{P}_{n}, \lambda)) \to 0$ . We demonstrate that this assumption is realistic for a variety of model classes like linear models and generalized additive models in Section C of the supplement.
3.2 Estimation
We estimate  $RID_{j}$  for each variable  $j$  by leveraging bootstrap sampling to draw new datasets from the empirical data distribution: we sample observations from an observed dataset, construct its Rashomon set, and compute the  $j$ -th variable's importance for each model in the Rashomon set. After repeating this process for  $B$  bootstrap iterations, we estimate  $RID$  by weighting each model  $f$ 's realized variable importance score (evaluated on each bootstrapped dataset) by the proportion of the bootstrapped datasets for which  $f$  is in the Rashomon set and the size of each Rashomon set in which  $f$  appears. Specifically, let  $\mathcal{D}_b^{(n)}$  represent the dataset sampled with replacement from  $\mathcal{D}^{(n)}$  in iteration  $b = 1,\ldots ,B$  of the bootstrap procedure. For each dataset  $\mathcal{D}_b^{(n)}$ , we find the Rashomon set  $\mathcal{R}_{\mathcal{D}_b^{(n)}}^{\varepsilon}$ . Finally, we compute an empirical estimate  $\widehat{RID_j}$  of  $RID_{j}$  by computing:
$$
\widehat {R I D} _ {j} (k; \varepsilon , \mathcal {F}, \ell , \mathcal {P} _ {n}, \lambda) = \frac {1}{B} \sum_ {b = 1} ^ {B} \left(\frac {\left| \left\{f \in \mathcal {R} _ {\mathcal {D} _ {b} ^ {(n)}} ^ {\varepsilon}: \phi_ {j} (f , \mathcal {D} _ {b} ^ {(n)}) \leq k \right\} \right|}{\left| \mathcal {R} _ {\mathcal {D} _ {b} ^ {(n)}} ^ {\varepsilon} \right|}\right).
$$
Under Assumption 1, we can directly connect our estimate  $\widehat{RID}(k; \varepsilon, \mathcal{F}, \ell, \mathcal{P}_n, \lambda)$  to the DGP's variable importance distribution  $RID(k; \ell_{\max}, \{g^*\}, \ell, \mathcal{P}_n, \lambda)$ , which Theorem 1 formalizes.