Title: The Rashomon Importance Distribution: Getting RID of Unstable, Single Model-based Variable Importance
Venue: NeurIPS 2023
Accepted: yes
Keywords: Rashomon Effect

Variable Importance

XAI

Stability

Interpretable Machine Learning
Abstract: Quantifying variable importance is essential for answering high-stakes questions in fields like genetics, public policy, and medicine. Current methods generally calculate variable importance for a given model trained on a given dataset. However, for a given dataset, there may be many models that explain the target outcome equally well; without accounting for all possible explanations, different researchers may arrive at many conflicting yet equally valid conclusions given the same data. Additionally, even when accounting for all possible explanations for a given dataset, these insights may not generalize because not all good explanations are stable across reasonable data perturbations. We propose a new variable importance framework that quantifies the importance of a variable across the set of all good models and is stable across the data distribution. Our framework is extremely flexible and can be integrated with most existing model classes and global variable importance metrics. We demonstrate through experiments that our framework recovers variable importance rankings for complex simulation setups where other methods fail. Further, we show that our framework accurately estimates the _true importance_ of a variable for the underlying data distribution. We provide theoretical guarantees on the consistency and finite sample error rates for our estimator. Finally, we demonstrate its utility with a real-world case study exploring which genes are important for predicting HIV load in persons with HIV, highlighting an important gene that has not previously been studied in connection with HIV.

The Rashomon Importance Distribution: Getting RID of Unstable, Single Model-based Variable Importance

Abstract

Quantifying variable importance is essential for answering high-stakes questions in fields like genetics, public policy, and medicine. Current methods generally calculate variable importance for a given model trained on a given dataset. However, for a given dataset, there may be many models that explain the target outcome equally well; without accounting for all possible explanations, different researchers may arrive at many conflicting yet equally valid conclusions given the same data. Additionally, even when accounting for all possible explanations for a given dataset, these insights may not generalize because not all good explanations are stable across reasonable data perturbations. We propose a new variable importance framework that quantifies the importance of a variable across the set of all good models and is stable across the data distribution. Our framework is extremely flexible and can be integrated with most existing model classes and global variable importance metrics. We demonstrate through experiments that our framework recovers variable importance rankings for complex simulation setups where other methods fail. Further, we show that our framework accurately estimates the true importance of a variable for the underlying data distribution. We provide theoretical guarantees on the consistency and finite sample error rates for our estimator. Finally, we demonstrate its utility with a real-world case study exploring which genes are important for predicting HIV load in persons with HIV, highlighting an important gene that has not previously been studied in connection with HIV.

1 Introduction

Variable importance analysis enables researchers to gain insight into a domain or a model. Scientists are often interested in understanding causal relationships between variables, but running randomized experiments is time-consuming and expensive. Given an observational dataset, we can use global variable importance measures to check if there is a predictive relationship between two variables. It is particularly important in high stakes real world domains such as genetics [44, 34], finance [37], and criminal justice [15, 24] where randomized controlled trials are impractical or unethical. Variable

ModelReliance

0.4

LargestMinModelReliance

一

一

0.2

SmallestMaxModelReliance

0.0

10

100

1000

10000

-0.2

BootstrapIteration

(a) The number of models in each Rashomon set

(b) Model reliance range across each Rashomon set

importance would ideally be measured as the importance of each variable to the data generating process. However, the data generating process is never known in practice, so prior work generally draws insight by analyzing variable importance for a surrogate model, treating that model and its variable importance as truth.

This approach can be misleading because there may be many good models for a given dataset - a phenomenon referred to as the Rashomon effect [7, 40] - and variables that are important for one good model on a given dataset are not necessarily important for others. As such, any insights drawn from a single model need not reflect the underlying data distribution or even the consensus among good models. Recently, researchers have sought to overcome the Rashomon effect by computing Rashomon sets , the set of all good (i.e., low loss) models for a given dataset [15, 12]. However, the set of all good models is not stable across reasonable perturbations (e.g., bootstrap or jackknife) of a single dataset , with stability defined as in [50]. This concept of stability is one of the three pillars of veridical data science [51, 13]. Note that there is wide agreement on the intuition behind stability, but not its quantification [22, 33]. As such, in line with other stability research, we do not subscribe to a formal definition and treat stability as a general notion [22, 33, 50, 51]. In order to ensure trustworthy analyses, variable importance measures must account for both the Rashomon effect and stability.

Figure 1 provides a demonstration of this problem: across 500 bootstrap replicates from the same data set, the Rashomon set varies wildly - ranging from ten models to over ten thousand -suggesting that we should account for its instability in any computed statistics. This instability is further highlighted when considering the Model Class Reliance (MCR) variable importance, which is the range of model reliance (i.e., variable importance) values across the Rashomon set for the given dataset [15] (we define MCR and the Rashomon set more rigorously in Sections 2 and 3 respectively). In particular, for variable X 2 , one interval - ranging from -0.1 to 0.33 - suggests that there exist good models that do not depend on this variable at all (0 indicates the variable is not important); on the other hand, another MCR from a bootstrapped dataset ranges from 0.33 to 0.36, suggesting that this variable is essential to all good models. Because of this instability, different researchers may draw very different conclusions about the same data distribution even when using the same method.

In this work, we present a framework unifying concepts from classical nonparametric estimation with recent developments on Rashomon sets to overcome the limitations of traditional variable importance measurements. We propose a stable, model- and variable-importance-metric-agnostic estimand that quantifies variable importance across all good models for the empirical data distribution and a corresponding bootstrap-style estimation strategy. Our method creates a cumulative density function (CDF) for variable importance over all variables via the framework shown in Figure 2. Using the CDF, we can compute a variety of statistics (e.g., expected variable importance, interquartile range, and credible regions) that can summarize the variable importance distribution.

The rest of this work is structured as follows. After more formally introducing our variable importance framework, we theoretically guarantee the convergence of our estimation strategy and derive error bounds. We also demonstrate experimentally that our estimand captures the true variable importance for the data generating process more accurately than previous work. Additionally, we illustrate the

Step 1:

Step 2:

Step 3:

Step 4:

Step 5:

Bootstrap

Find Importances

Find CDF

Find PDF

Density of RID;

(VI≤k)

RIDj

loss(

PDF ofRID

fEModelClass

fERashomon Set

k E [Φmin, Pmax]

k E [Φmin, Pmax]

generalizability of our variable importance metric by analyzing the reproducibility of our results given new datasets from the same data generation process. Lastly, we use our method to analyze which transcripts and chromatin patterns in human T cells are associated with high expression of HIV RNA. Our results suggest an unexplored link between the LINC00486 gene and HIV load.

Code is available at https://github.com/jdonnelly36/Rashomon_Importance_Distribution.

2 Related Work

The key difference between our work and most others is the way it incorporates model uncertainty, also called the Rashomon effect [7]. The Rashomon effect is the phenomenon in which many different models explain a dataset equally well. It has been documented in high stakes domains including healthcare, finance, and recidivism prediction [14, 30, 15]. The Rashomon effect has been leveraged to create uncertainty sets for robust optimization [43], to perform responsible statistical inference [11], and to gauge whether simple yet accurate models exist for a given dataset [40]. One barrier to studying the Rashomon effect is the fact that Rashomon sets are computationally hard to calculate for non-trivial model classes. Only within the past year has code been made available to solve for (and store) the full Rashomon set for any nonlinear function class - that of decision trees [49]. This work enables us to revisit the study of variable importance with a new lens.

A classical way to determine the importance of a variable is to leave it out and see if the loss changes. This is called algorithmic reliance [15] or leave one covariate out (LOCO) inference [25, 36]. The problem with these approaches is that the performance of the model produced by an algorithm will not change if there exist other variables correlated with the variable of interest.

Model reliance (MR) methods capture the global variable importance (VI) of a given feature for a specific model [15]. (Note that MR is limited to refer to permutation importance in [15], while we use the term MR to refer to any metric capturing global variable importance of a given feature and model. We use VI and MR interchangably when the relevant model is clear from context.) Several methods for measuring the MR of a model from a specific model class exist, including the variable importance measure from random forest which uses out-of-bag samples [7] and Lasso regression coefficients [20]. Lundberg et al. [28] introduce a way of measuring MR in tree ensembles using SHAP [27]. Williamson et al. [48] develop MR based on the change in performance between the optimal model and the optimal model using a subset of features.

In addition to the metrics tied to a specific model class, many MR methods can be applied to models from any model class . Traditional correlation measures [20] can measure the linear relationship (Pearson correlation) or general monotonic relationship (Spearman correlation) between a feature and predicted outcomes for a model from any model class. Permutation model reliance, as discussed by [1, 15, 21], describes how much worse a model performs when the values of a given feature are permuted such that the feature becomes uninformative. Shapley-based measures of MR, such as those

of [47, 27], calculate the average marginal contribution of each feature to a model's predictions. A complete overview of the variable importance literature is beyond the scope of this work; for a more thorough review, see, for example, [2, 31]. Rather than calculating the importance of a variable for a single model, our framework finds the importance of a variable for all models within a Rashomon set, although our framework is applicable to all of these model reliance metrics.

In contrast, model class reliance (MCR) methods describe how much a class of models (e.g., decision trees) relies on a variable. Fisher et al. [15] uses the Rashomon set to provide bounds on the possible range of model reliance for good models of a given class. Smith et al. [41] analytically find the range of model reliance for the model class of random forests. Zhang and Janson [52] introduce a way to compute confidence bounds for a specific variable importance metric over arbitrary models, which Aufiero and Janson [3] extend so that it is applicable to a broad class of surrogate models in pursuit of computational efficiency. These methods report MCR as a range, which gives no estimate of variable importance - only a range of what values are possible. In contrast, Dong and Rudin [12] compute and visualize the variable importance for every member of a given Rashomon set in projected spaces, calculating a set of points; however, these methods have no guarantees of stability to reasonable data perturbations. In contrast, our framework overcomes these finite sample biases, supporting stronger conclusions about the underlying data distribution.

Related to our work from the stability perspective, Duncan et al. [13] developed a software package to evaluate the stability of permutation variable importance in random forest methods; we perform a similar exercise to demonstrate that current variable importance metrics computed for the Rashomon set are not stable. Additionally, Basu et al. [5] introduced iterative random forests by iteratively reweighting trees and bootstrapping to find stable higher-order interactions from random forests. Further, theoretical results have demonstrated that bootstrapping stabilizes many machine learning algorithms and reduces the variance of statistics [18, 8]. We also take advantage of bootstrapping's flexibility and properties to ensure stability for our variable importance.

3 Methods

3.1 Definitions and Estimands

Let D ( n ) = { ( X i , Y i ) } n i =1 denote a dataset of n independent and identically distributed tuples, where Y i ∈ R denotes some outcome of interest and X i ∈ R p denotes a vector of p covariates. Let g ∗ represent the data generating process (DGP) producing D ( n ) . Let f ∈ F be a model in a model class (e.g., a tree in the set of all possible sparse decision trees), and let ϕ j ( f, D ( n ) ) denote a function that measures the importance of variable j for a model f over a dataset D ( n ) . This can be any of the functions described earlier (e.g., permutation importance, SHAP). Our framework is flexible with respect to the user's choice of ϕ j and enables practitioners to use the variable importance metric best suited for their purpose; for instance, conditional model reliance [15] is best-suited to measure only the unique information carried by the variable (that cannot be constructed using other variables), whereas other metrics like subtractive model reliance consider the unconditional importance of the variable. Our framework is easily integrable with either of these. We only assume that the variable importance function ϕ has a bounded range, which holds for a wide class of metrics like SHAP [27], permutation model reliance, and conditional model reliance. Finally, let ℓ ( f, D ( n ) ; λ ) represent a loss function given f, D ( n ) , and loss hyperparameters λ (e.g., regularization). We assume that our loss function is bounded above and below, which is true for common loss functions like 0-1 classification loss, as well as for differentiable loss functions with covariates from a bounded domain.

In an ideal setting, we would measure variable importance using g ∗ and the whole population, but this is impossible because g ∗ is unknown and data is finite. In practice, scientists instead use the empirical loss minimizer for a specific dataset ˆ f ∗ ∈ arg min f ∈F ℓ ( f, D ( n ) ) ; however, several models could explain the same dataset equally well (i.e., the Rashomon effect). Rather than using a single model to compute variable importance, we propose using the entire Rashomon set. Given a single dataset D ( n ) , we define the Rashomon set for a model class F and parameter ε as the set of all models in F whose empirical losses are within some bound ε > 0 of the empirical loss minimizer:

We denote this Rashomon set by R ε D ( n ) or 'Rset' (this assumes a fixed F , ℓ and λ ). As discussed earlier, the Rashomon set can be fully computed and stored for non-linear models (e.g., sparse decision trees [49]). For notational simplicity, we often omit λ from the loss function.

While the Rashomon set describes the set of good models for a single dataset, Rashomon sets vary across permutations (e.g., subsampling and resampling schemes) of the given data. We introduce a stable quantity for variable importance that accounts for all good models and all permutations from the data using the random variable RIV . RIV is defined by its cumulative distribution function (CDF), the Rashomon Importance Distribution ( RID ) :

where ϕ j denotes the variable importance metric being computed on variable j , k ∈ [ ϕ min , ϕ max ] . For a continuous model class (e.g., linear regression models), the cardinality in the above definition becomes the volume under a measure on the function class, usually ℓ 2 on parameter space. RID constructs the cumulative distribution function (CDF) for the distribution of variable importance across Rashomon sets; as k increases, the value of P ( RIV j ( ε, F , ℓ, D ( n ) b ; λ ) ≤ k ) becomes closer to 1. The probability and expectation are taken with respect to datasets of size n sampled from the empirical distribution P n , which is the same as considering all possible resamples of size n from the originally observed dataset D ( n ) . Equation (2) weights the contribution of ϕ j ( f, D ( n ) b ) for each model f by the proportion of datasets for which this model is a good explanation (i.e., in the Rashomon set). Intuitively, this provides greater weight to the importance of variables for stable models.

We now define an analogous metric for the loss function ℓ ; we define the Rashomon Loss Distribution ( RLD ) evaluated at k as the expected fraction of functions in the Rashomon set with loss below k . Here, RLV is a random variable following this CDF.

This quantity shows how quickly the Rashomon set 'fills up' on average as loss changes. If there are many near-optimal functions, this will grow quickly with k .

In order to connect RID for model class F to the unknown DGP g ∗ , we make a Lipschitz-continuitystyle assumption on the relationship between RLD and RID relative to a general model class F and { g ∗ } . To draw this connection, we define the loss CDF for g ∗ , called LD ∗ , over datasets of size n as:

One could think of LD ∗ as measuring how quickly the DGP's Rashomon set fills up as loss changes. Here, LD ∗ is the analog of RLD for the data generation process.

Assumption 1. If

for a function d : [0 , ℓ max -ℓ min ] → [0 , ϕ max -ϕ min ] such that lim γ → 0 d ( γ ) = 0 . Here, ρ represents any distributional distance metric (e.g., 1-Wasserstein).

Assumption 1 says that a Rashomon set consisting of good approximations for g ∗ in terms of loss will also consist of good approximations for g ∗ in terms of variable importance. More formally, from this assumption, we know that as ρ ( LD ∗ , RLD ) → 0 , the variable importance distributions will converge: ρ ( RID j ( · , ε, F , ℓ, P n , λ ) , RID j ( · , ε, { g ∗ } , ℓ, P n , λ )) → 0 . We demonstrate that this assumption is realistic for a variety of model classes like linear models and generalized additive models in Section C of the supplement.

3.2 Estimation

We estimate RID j for each variable j by leveraging bootstrap sampling to draw new datasets from the empirical data distribution: we sample observations from an observed dataset, construct its Rashomon set, and compute the j -th variable's importance for each model in the Rashomon set. After repeating this process for B bootstrap iterations, we estimate RID by weighting each model f 's realized variable importance score (evaluated on each bootstrapped dataset) by the proportion of the bootstrapped datasets for which f is in the Rashomon set and the size of each Rashomon set in which f appears. Specifically, let D ( n ) b represent the dataset sampled with replacement from D ( n ) in iteration b = 1 , . . . , B of the bootstrap procedure. For each dataset D ( n ) b , we find the Rashomon set R ε D ( n ) b . Finally, we compute an empirical estimate ̂ RID j of RID j by computing:

Under Assumption 1, we can directly connect our estimate ̂ RID ( k ; ε, F , ℓ, P n , λ ) to the DGP's variable importance distribution RID ( k ; ℓ max , { g ∗ } , ℓ, P n , λ ) , which Theorem 1 formalizes.

Theorem 1. Let Assumption 1 hold for distributional distance ρ ( A 1 , A 2 ) between distributions A 1 and A 2 . For any t > 0 , j ∈ { 0 , . . . , p } as ρ ( LD ∗ ( · ; ℓ, n, λ ) , RLD ( · ; ε, F , ℓ, P n , λ )) → 0 and B →∞ ,

For a set of models that performs sufficiently well in terms of loss, ̂ RID j thus recovers the CDF of variable importance for the true model across all reasonable perturbations. Further, we can provide a finite sample bound for the estimation of a marginal distribution between ̂ RID j and RID j for the model class F , as stated in Theorem 2. Note that this result does not require Assumption 1.

Theorem 2. Let t > 0 and δ ∈ (0 , 1) be some pre-specified values. Then, with probability at least 1 -δ with respect to bootstrap samples of size n ,

with number of bootstrap samples B ≥ 1 2 t 2 ln ( 2 δ ) for any k ∈ [ ϕ min , ϕ max ] .

Because we use a bootstrap procedure, we can control the number of bootstrap iterations to ensure that the difference between RID j and ̂ RID j is within some pre-specified error. (As defined earlier, RID j is the expectation over infinite bootstraps, whereas ̂ RID j is the empirical average over B bootstraps.) For example, after 471 bootstrap iterations, we find that ̂ RID j ( k ; ε, F , ℓ, P n , λ ) is within 0.075 of RID j ( k ; ε, F , ℓ, P n , λ ) for any given k with 90% confidence. It also follows that as B tends to infinity, the estimated RIV j will converge to the true value.

Since we stably estimate the entire distribution of variable importance values, we can create (1) stable point estimates of variable importance (e.g., expected variable importance) that account for the Rashomon effect, (2) interquantile ranges of variable importance, and (3) confidence regions that characterize uncertainty around a point estimate of variable importance. We prove exponential rates of convergence for these statistics estimated using our framework in Section B of the supplement.

Because our estimand and our estimation strategy (1) enable us to manage instability, (2) account for the Rashomon effect, and (3) are completely model-agnostic and flexibly work with most existing variable importance metrics, RID is a valuable quantification of variable importance.

4 Experiments With Known Data Generation Processes

1.00

Classification

Proportion

0.75-

0.50

0.25

0.00

GOSDT

MR

RF

RF

RF

GOSDT

MR

CMR Lasso

LOCO

LOCO

RF

Pearson

Spearman

Stability +

Rashomon

Stability only

Rashomon

only

(via bootstrapping)

1.00

Recovery Proportion

0.75

Monk 1

0.50

Monk 3

Friedman's

0.25

0.00

Method

RID + GOSDT

VIC+GOSDT

AdaBoost

GOSDT

LASSO

RF

Recovery Correct (%)

(94.64%)

(81.16%)

(90.62%)

(93.49%)

(80.92%)

(84.98%)

Classifications Correct (%)

(100%)

(100%)

(83.33%)

(83.33%)

(83.33%)

(100%)

4.1 RID Distinguishes Important Variables from Extraneous Variables

̸

There is no generally accepted ground truth measure for variable importance, so we first evaluate whether a variety of variable importance methods can correctly distinguish between variables used to generate the outcome (in a known data generation process) versus those that are not. We consider the following four data generation processes (DGPs). Chen's DGP [9]: Y = 1 [ -2 sin( X 1 )+max( X 2 , 0)+ X 3 +exp( -X 4 )+ ε ≥ 2 . 048] , where X 1 , . . . , X 10 , ε ∼ N (0 , 1) . Here, only X 1 , . . . , X 4 are relevant. Friedman's DGP [17]: Y = 1 [10 sin( πX 1 X 2 ) + 20( X 3 -0 . 5) 2 +10 X 4 +5 X 5 + ε ≥ 15] , where X 1 , . . . , X 6 ∼ U (0 , 1) , ε ∼ N (0 , 1) . Here, only X 1 , . . . , X 5 are relevant. The Monk 1 DGP [42]: Y = max( 1 [ X 1 = X 2 ] , 1 [ X 5 = 1]) , where the variables X 1 , . . . , X 6 have domains of 2, 3, or 4 unique integer values. Only X 1 , X 2 , X 5 are important. The Monk 3 DGP [42]: Y = max( 1 [ X 5 = 3 and X 4 = 1] , 1 [ X 5 = 4 and X 2 = 3]) for the same covariates in Monk 1. Also, 5% label noise is added. Here, X 2 , X 4 , and X 5 are relevant.

̸

We compare the ability of RID to identify extraneous variables with that of the following baseline methods, whose details are provided in Section D of the supplement: subtractive model reliance ϕ sub of a random forest (RF) [6], LASSO [20], boosted decision trees [16], and generalized optimal sparse decision trees (GOSDT) [26]; conditional model reliance (CMR) [15]; the impurity based model reliance metric for RF from [7]; the LOCO algorithm reliance [25] for RF and Lasso; the Pearson and Spearman correlation between each feature and the outcome; the mean of the partial dependency plot (PDP) [19] for each feature; the SHAP value [28] for RF; and mean of variable importance clouds (VIC) [12] for the Rashomon set of GOSDTs [49]. If we do not account for instability and simply learn a model and calculate variable importance, baseline models generally perform poorly, as shown

in Section E of the supplement. Thus, we chose to account for instability in a way that benefits the baselines. We evaluate each baseline method for each variable across 500 bootstrap samples and compute the median VI across bootstraps , with the exception of VIC - for VIC, we take the median VI value across the Rashomon set for the original dataset, as VIC accounts for Rashomon uncertainty. Here, we aim to see whether we can identify extraneous (i.e., unimportant variables). For a DGP with C extraneous variables, we classify the C variables with the C smallest median variable importance values as extraneous. We repeat this experiment with different values for the Rashomon threshold ε in Section E of the supplement.

Figure 3 (top) reports the proportion of variables that are correctly classified for each simulation setup as a stacked barplot. RID identifies all important and unimportant variables for these complex simulations. Note that four other baseline methods - MR RF, RF Impurity, RF Permute, and VIC - also differentiated all important from unimportant variables. Motivated by this finding, we next explore how well methods recover the true value for subtractive model reliance on the DGP, allowing us to distinguish between the best performing methods on the classification task.

4.2 RID Captures Model Reliance for the True Data Generation Process

RID allows us to quantify uncertainty in variable importance due to both the Rashomon effect and instability. We perform an ablation study investigating how accounting for both stability and the Rashomon effect compares to having one without the other. We evaluate what proportion of subtractive model reliances calculated for the DGP on 500 test sets are contained within uncertainty intervals generated using only one training dataset. This experiment tells us whether the intervals created on a single dataset will generalize.

To create the uncertainty interval on the training dataset and for each method, we first find the subtractive model reliance ϕ ( sub ) across 500 bootstrap iterations of a given dataset for the four algorithms shown in Figure 3 (bottom) (baseline results without bootstrapping are in Section E of the supplementary material). Additionally, we find the VIC for the Rashomon set of GOSDTs on the original dataset. We summarize these model reliances (500 bootstraps × 28 variables across datasets × 4 algorithms + 8,247 models in VIC's + 10,840,535 total models across Rsets × 28 variables from RID ) by computing their box-and-whisker ranges (1.5 × Interquartile range [46]). To compare with 'ground truth,' we sample 500 test datasets from the DGP and calculate ϕ ( sub ) for the DGP for that dataset. For example, assume the DGP is Y = X 2 + ε . We would then use f ( X ) = X 2 as our predictive model and evaluate ϕ ( sub ) ( f, D ( n ) ) on f for each of the 500 test sets. We then check if the box-and-whisker range of each method's interval constructed on the training set contains the computed ϕ ( sub ) for the DGP for each test dataset. Doing this allows us to understand whether our interval contains the true ϕ ( sub ) for each test set.

Figure 3 (bottom) illustrates the proportion of times that the test variable importance values fell within the uncertainty intervals from training. These baselines fail to capture the test ϕ ( sub ) entirely for at least one variable ( < 0 . 05% recovery proportion). Only RID both recovers important/unimportant classifications perfectly and achieves a strong recovery proportion at 95% .

4.3 RID is Stable

Our final experiment investigates the stability of VIC and MCR (which capture only Rashomon uncertainty but not stability) to RID , which naturally considers data perturbations. We generate 50 independent datasets from each DGP and compute the box-and-whisker ranges (BWR) of each uncertainty metric for each dataset; for every pair of BWRs for a given method, we then calculate the Jaccard similarity between BWR's. For each generated dataset, we then average the Jaccard similarity across variables. Figure 4 shows these intervals for each non-extraneous variable from Chen's DGP. Supplement E.4 presents a similar figure for each DGP, showing that only RID 's intervals overlap across all generations for all datasets.

Figure 5 displays the similarity scores between the box and whisker ranges of MCR, VIC, and RID across the 50 datasets for each DGP. Note that Monk 1 has no noise added, so instability should not be a concern for any method. For datasets including noise, MCR and VIC achieve median similarity below 0.55; RID 's median similarity is 0.69; it is much more stable .

Chen's

MCR

VIC

RID

0.2

p

0.1

0.0

0.2

l

0.1

4

0.0

0

Dataset

Similarity

0.75

Method

0.50

MCR

VIC

0.25

RID

0.00

Friedman's

Monk 1

Monk 3

5 Case Study

Having validated RID on synthetic datasets, we demonstrate its utility in a real world application: studying which host cell transcripts and chromatin patterns are associated with high expression of Human Immunodeficiency Virus (HIV) RNA. We analyzed a dataset that combined single cell RNAseq/ATACseq profiles for 74,031 individual HIV infected cells from two different donors in the aims of finding new cellular cofactors for HIV expression that could be targeted to reactivate the latent HIV reservoir in people with HIV (PWH). A longer description of the data is in [29]. Finding results on this type of data allows us to create new hypotheses for which genes are important for HIV load prediction and might generalize to large portions of the population.

To identify which genes are stably importance across good models, we evaluated this dataset using RID over the model class of sparse decision trees using subtractive model reliance. We selected 14,614 samples (all 7,307 high HIV load samples and 7,307 random low HIV load samples) from the overall dataset in order to balance labels, and filtered the complete profiles down to the top 100 variables by individual AUC. We consider the binary classification problem of predicting high versus low HIV load. For full experimental details, see Section D of the supplement. Section E.5 of the supplement contains timing experiments for RID using this dataset.

Figure 6 illustrates the probability that RID is greater than 0 for the 10 highest probability variables (0 is when the variable is not important at all). We find that LINC00486 - a less explored gene - is the most important variable , with 1 -RID LINC 00486 (0) = 78 . 4% . LINC00486 is a long non-coding RNA (i.e., it functions as an RNA molecule but does not encode a protein like most genes), and there is no literature on this gene and HIV , making this association a novel one. However, recent work [45] has shown that LINC00486 can enhance EBV (Epstein-Barr virus) infection by activating NFκ B.

0.8

10 Most Important Genes

0.6

LINC00486

PPP3CA

(RIV;

0.4

PDE4D

TANK

ACTB

PTPRC

HNRNPA2B1

RBPJ

P

0.2

RPS19

MALAT1

0.0

Gene

It is well established that NFκ B can regulate HIV expression [32, 23, 4], suggesting a possible mechanism and supporting future study. Notably, RID also highlighted PDE4D, which interacts with the Tat protein and thereby HIV transcription [39]; HNRNPA2B1, which promotes HIV expression by altering the structure of the viral promoter [38]; and MALAT1, which has recently been shown to be an important regulator of HIV expression [35]. These three findings validate prior work and show that RID can uncover variables that are known to interact with HIV.

Note that previous methods - even those that account for the Rashomon effect - could not produce this result . MCR and VIC do not account for instability. For example, after computing MCR for 738 bootstrap iterations, we find that the MCR for the LINC00486 gene has overlap with 0 in 96 . 2% of bootstrapped datasets, meaning MCR would not allow us to distinguish whether LINC00486 is important or not 96 . 2% of the time. Without RID , we would not have strong evidence that LINC00486 is necessary for good models. By explicitly accounting for instability, we increase trust in our analyses.

Critically, RID also found very low importance for the majority of variables, allowing researchers to dramatically reduce the number of possible directions for future experiments designed to test a gene's functional role. Such experiments are time consuming and cost tens of thousands of dollars per donor , so narrowing possible future directions to a small set of genes is of the utmost importance. Our analysis provides a manageable set of clear directions for future work studying the functional roles of these genes in HIV .

6 Conclusion and Limitations

We introduced RID , a method for recovering the importance of variables in a way that accounts for both instability and the Rashomon effect. We showed that RID distinguishes between important and extraneous variables, and that RID better captures the true variable importance for the DGP than prior methods. We showed through a case study in HIV load prediction that RID can provide insight into complicated real world problems. Our framework overcomes instability and the Rashomon effect, moving beyond variable importance for a single model and increasing reproducibility.

RID can be directly computed for any model class for which the Rashomon set can be found - at the time of publishing, decision trees, linear models, and GLMs. A limitation is that currently, there are relatively few model classes for which the Rashomon set can be computed. Therefore, future work should aim to compute and store the Rashomon set of a wider variety of model classes. Future work may investigate incorporating Rashomon sets that may be well-approximated (e.g., GAMs, [10]), but not computed exactly, into the RID approach. Nonetheless, sparse trees are highly flexible, and using them with RID improves the trustworthiness and transparency of variable importance measures, enabling researchers to uncover important, reproducible relationships about complex processes without being misled by the Rashomon effect.

References

André Altmann, Laura Tolo¸ si, Oliver Sander, and Thomas Lengauer. Permutation importance: a corrected feature importance measure. Bioinformatics , 26(10):1340-1347, 2010.

Alejandro Barredo Arrieta, Natalia Díaz-Rodríguez, Javier Del Ser, Adrien Bennetot, Siham Tabik, Alberto Barbado, Salvador García, Sergio Gil-López, Daniel Molina, Richard Benjamins, et al. Explainable artificial intelligence (XAI): Concepts, taxonomies, opportunities and challenges toward responsible AI. Information Fusion , 58:82-115, 2020.

Massimo Aufiero and Lucas Janson. Surrogate-based global sensitivity analysis with statistical guarantees via floodgate. arXiv preprint arXiv:2208.05885 , 2022.

F Bachelerie, J Alcami, F Arenzana-Seisdedos, and JL Virelizier. HIV enhancer activity perpetuated by NFκ B induction on infection of monocytes. Nature , 350(6320):709-712, 1991.

Sumanta Basu, Karl Kumbier, James B Brown, and Bin Yu. Iterative random forests to discover predictive and stable high-order interactions. Proceedings of the National Academy of Sciences , 115(8):1943-1948, 2018.

Leo Breiman. Random forests. Machine learning , 45:5-32, 2001.

Leo Breiman. Statistical modeling: The two cultures (with comments and a rejoinder by the author). Statistical Science , 16(3):199-231, 2001.

Peter Bühlmann and Bin Yu. Analyzing bagging. The Annals of Statistics , 30(4):927-961, 2002.

Jianbo Chen, Mitchell Stern, Martin J Wainwright, and Michael I Jordan. Kernel feature selection via conditional covariance minimization. Advances in Neural Information Processing Systems , 30, 2017.

Zhi Chen, Chudi Zhong, Margo Seltzer, and Cynthia Rudin. Understanding and exploring the whole set of good sparse generalized additive models. Advances in Neural Information Processing Systems , 36, 2023.

Beau Coker, Cynthia Rudin, and Gary King. A theory of statistical inference for ensuring the robustness of scientific results. Management Science , 67(10):6174-6197, 2021.

Jiayun Dong and Cynthia Rudin. Exploring the cloud of variable importance for the set of all good models. Nature Machine Intelligence , 2(12):810-824, 2020.

James Duncan, Rush Kapoor, Abhineet Agarwal, Chandan Singh, and Bin Yu. Veridicalflow: a python package for building trustworthy data science pipelines with PCS. Journal of Open Source Software , 7(69): 3895, 2022.

Alexander D'Amour, Katherine Heller, Dan Moldovan, Ben Adlam, Babak Alipanahi, Alex Beutel, Christina Chen, Jonathan Deaton, Jacob Eisenstein, Matthew D Hoffman, et al. Underspecification presents challenges for credibility in modern machine learning. Journal of Machine Learning Research , 2020.

Aaron Fisher, Cynthia Rudin, and Francesca Dominici. All models are wrong, but many are useful: Learning a variable's importance by studying an entire class of prediction models simultaneously. Journal of Machine Learning Research , 20(177):1-81, 2019.

Yoav Freund and Robert E Schapire. A decision-theoretic generalization of on-line learning and an application to boosting. Journal of Computer and System Sciences , 55(1):119-139, 1997.

Jerome H Friedman. Multivariate adaptive regression splines. The Annals of Statistics , 19(1):1-67, 1991.

Yves Grandvalet. Stability of bagged decision trees. In Proceedings of the XLIII Scientific Meeting of the Italian Statistical Society , pages 221-230. CLEUP, 2006.

Brandon M Greenwell, Bradley C Boehmke, and Andrew J McCarthy. A simple and effective model-based variable importance measure. arXiv preprint arXiv:1805.04755 , 2018.

Trevor Hastie, Robert Tibshirani, Jerome H Friedman, and Jerome H Friedman. The elements of statistical learning: data mining, inference, and prediction , volume 2. Springer, 2009.

Giles Hooker, Lucas Mentch, and Siyu Zhou. Unrestricted permutation forces extrapolation: variable importance requires at least one more model, or there is no free variable importance. Statistics and Computing , 31(6):1-16, 2021.

Alexandros Kalousis, Julien Prados, and Melanie Hilario. Stability of feature selection algorithms. In Fifth IEEE International Conference on Data Mining (ICDM'05) , pages 8-pp. IEEE, 2005.

Marcus Kretzschmar, Michael Meisterernst, Claus Scheidereit, Gen Li, and Robert G Roeder. Transcriptional regulation of the HIV-1 promoter by NFκ B in vitro. Genes & Development , 6(5):761-774, 1992.

Jeff Larson, Surya Mattu, Lauren Kirchner, and Julia Angwin. How we analyzed the compas recidivism algorithm. ProPublica , May 2016. URL https://www.propublica.org/article/ machine-bias-risk-assessments-in-criminal-sentencing .

Jing Lei, Max G'Sell, Alessandro Rinaldo, Ryan J Tibshirani, and Larry Wasserman. Distribution-free predictive inference for regression. Journal of the American Statistical Association , 113(523):1094-1111, 2018.

Jimmy Lin, Chudi Zhong, Diane Hu, Cynthia Rudin, and Margo Seltzer. Generalized and scalable optimal sparse decision trees. In International Conference on Machine Learning , pages 6150-6160. PMLR, 2020.

Scott M Lundberg and Su-In Lee. A unified approach to interpreting model predictions. In I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, editors, Advances in Neural Information Processing Systems 30 , pages 4765-4774. Curran Associates, Inc., 2017.

Scott M Lundberg, Gabriel G Erion, and Su-In Lee. Consistent individualized feature attribution for tree ensembles. arXiv preprint arXiv:1802.03888 , 2018.

Ashokkumar Manickam, Jackson J Peterson, Yuriko Harigaya, David M Murdoch, David M Margolis, Alex Oesterling, Zhicheng Guo, Cynthia D Rudin, Yuchao Jiang, and Edward P Browne. Integrated single-cell multiomic analysis of HIV latency reversal reveals novel regulators of viral reactivation. bioRxiv , pages 2022-07, 2022.

Charles Marx, Flavio Calmon, and Berk Ustun. Predictive multiplicity in classification. In International Conference on Machine Learning , pages 6765-6774. PMLR, 2020.

Dang Minh, H Xiang Wang, Y Fen Li, and Tan N Nguyen. Explainable artificial intelligence: a comprehensive review. Artificial Intelligence Review , pages 1-66, 2022.

Christopher C Nixon, Maud Mavigner, Gavin C Sampey, Alyssa D Brooks, Rae Ann Spagnuolo, David M Irlbeck, Cameron Mattingly, Phong T Ho, Nils Schoof, Corinne G Cammon, et al. Systemic HIV and SIV latency reversal via non-canonical NFκ B signalling in vivo. Nature , 578(7793):160-165, 2020.

Sarah Nogueira, Konstantinos Sechidis, and Gavin Brown. On the stability of feature selection algorithms. The Journal of Machine Learning Research , 18(1):6345-6398, 2017.

Gherman Novakovsky, Nick Dexter, Maxwell W Libbrecht, Wyeth W Wasserman, and Sara Mostafavi. Obtaining genetics insights from deep learning via explainable artificial intelligence. Nature Reviews Genetics , pages 1-13, 2022.

Di Qu, Wei-Wei Sun, Li Li, Li Ma, Li Sun, Xia Jin, Taisheng Li, Wei Hou, and Jian-Hua Wang. Long noncoding RNA MALAT1 releases epigenetic silencing of HIV-1 replication by displacing the polycomb repressive complex 2 from binding to the LTR promoter. Nucleic Acids Research , 47(6):3013-3027, 2019.

Alessandro Rinaldo, Larry Wasserman, and Max G'Sell. Bootstrapping and sample splitting for highdimensional, assumption-lean inference. The Annals of Statistics , 47(6):3438 - 3469, 2019. doi: 10.1214/ 18-AOS1784.

Cynthia Rudin and Joanna Radin. Why are we using black box models in AI when we don't need to? A lesson from an explainable AI competition. Harvard Data Science Review , 1(2):10-1162, 2019.

Matteo Scalabrin, Ilaria Frasson, Emanuela Ruggiero, Rosalba Perrone, Elena Tosoni, Sara Lago, Martina Tassinari, Giorgio Palù, and Sara N Richter. The cellular protein hnRNP A2/B1 enhances HIV-1 transcription by unfolding LTR promoter G-quadruplexes. Scientific Reports , 7(1):1-13, 2017.

Paola Secchiero, Davide Zella, Sabrina Curreli, Prisco Mirandola, Silvano Capitani, Robert C Gallo, and Giorgio Zauli. Pivotal role of cyclic nucleoside phosphodiesterase 4 in tat-mediated CD4+ T cell hyperactivation and HIV type 1 replication. Proceedings of the National Academy of Sciences , 97(26): 14620-14625, 2000.

Lesia Semenova, Cynthia Rudin, and Ronald Parr. On the existence of simpler machine learning models. In 2022 ACM Conference on Fairness, Accountability, and Transparency , pages 1827-1858, 2022.

Gavin Smith, Roberto Mansilla, and James Goulding. Model class reliance for random forests. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems , volume 33, pages 22305-22315. Curran Associates, Inc., 2020.

S. Thrun, J. Bala, E. Bloedorn, I. Bratko, B. Cestnik, J. Cheng, K. De Jong, S. Dzeroski, R. Hamann, K. Kaufman, S. Keller, I. Kononenko, J. Kreuziger, R.S. Michalski, T. Mitchell, P. Pachowicz, B. Roger, H. Vafaie, W. Van de Velde, W. Wenzel, J. Wnek, and J. Zhang. The MONK's problems: A performance comparison of different learning algorithms. Technical Report CMU-CS-91-197, Carnegie Mellon University, Computer Science Department, Pittsburgh, PA, 1991.

Theja Tulabandhula and Cynthia Rudin. Robust optimization using machine learning for uncertainty sets. ECML/PKDD 2014 , page 121, 2014.

Fang Wang, Shujia Huang, Rongsui Gao, Yuwen Zhou, Changxiang Lai, Zhichao Li, Wenjie Xian, Xiaobo Qian, Zhiyu Li, Yushan Huang, et al. Initial whole-genome sequencing and analysis of the host genetic contribution to COVID-19 severity and susceptibility. Cell Discovery , 6(1):83, 2020.

Wen-Fang Wang, Hui-Juan Zhong, Shu Cheng, Di Fu, Yan Zhao, Hua-Man Cai, Jie Xiong, and Wei-Li Zhao. A nuclear NKRF interacting long noncoding RNA controls EBV eradication and suppresses tumor progression in natural killer/T-cell lymphoma. Biochimica et Biophysica Acta (BBA)-Molecular Basis of Disease , page 166722, 2023.

Wikipedia contributors. Box plot, 2023. URL https://en.wikipedia.org/wiki/Box_plot . [Online; accessed 14-May-2023].

Brian Williamson and Jean Feng. Efficient nonparametric statistical inference on population feature importance using shapley values. In International Conference on Machine Learning , pages 10282-10291. PMLR, 2020.

Brian D Williamson, Peter B Gilbert, Noah R Simon, and Marco Carone. A general framework for inference on algorithm-agnostic variable importance. Journal of the American Statistical Association , pages 1-14, 2021.

Rui Xin, Chudi Zhong, Zhi Chen, Takuya Takagi, Margo Seltzer, and Cynthia Rudin. Exploring the whole rashomon set of sparse decision trees. Advances in Neural Information Processing Systems , 35: 22305-22315, 2022.

Bin Yu. Stability. Bernoulli , 19(4):1484 - 1500, 2013. doi: 10.3150/13-BEJSP14.

Bin Yu and Karl Kumbier. Veridical data science. Proceedings of the National Academy of Sciences , 117 (8):3920-3929, 2020. doi: 10.1073/pnas.1901326117.

Lu Zhang and Lucas Janson. Floodgate: inference for model-free variable importance. arXiv preprint arXiv:2007.01283 , 2020.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a variable importance framework, the Rashomon importance distribution (RID), that is robust to both the Rashomon effect and to dataset resampling. It motivates the need for such a framework, and surveys existing related methods. It then provides a detailed theoretical explanation of the proposed variable importance (a distribution) and a method for its estimation. The paper presents several experiments which demonstrate that RID can distinguish between important and unimportant/extraneous variables more reliably than previous methods. It then presents a case study in immunology (HIV research) wherein RID was able to identify a variable importance that was previously undetected, creating a lead for future investigation.\n\nSTRENGTHS:\nThis is a very strong paper overall. \nThe quality of the writing and overall presentation (figures, equations, etc.) is excellent, within the top 5% of papers. \nThe authors are very familiar with the relevant prior work in the area, which is clearly demonstrated in the Related Work section.\nThe technical contributions are important. They further strengthen the arguments for using simple, interpretable model classes (like sparse trees).\nThe inclusion of a case study where the method creates real value in an important research area demonstrates its practical value.\n\nWEAKNESSES:\nThe paper’s main weakness is its (current) lack of practical applicability to non-linear model classes outside of sparse trees. Because of this, the paper’s impact at a venue where most researchers (the vast majority perhaps) are working primarily with neural networks and transformers may be more limited. That said, I think this work along with the related work it highlights nonetheless merits visibility. Collectively, this vein of research creates a compelling argument for using inherently interpretable models, and has the potential to shift the default tools used by data science practitioners. \n\nSuggestions:\n\nThe paper could be strengthened by running some of the experiments in Section 4 on real datasets for which a subset of the variables are known to be important/unimportant, (extraneous ones may also be added).  \n\nSome of the plot colors could be improved for greater visual clarity / contrast. For instance the greens/blues in figures 2 & 3 bleed into each other, especially when printed. \n\nIt would be good to refer the reader to Appendix Section D.3 in Section 4.3 on the stability of RID, possibly putting the discussion on robustness to $\\varepsilon$ there. \n\nIn the main text, it would be good to note that a timing analysis was conducted, referring the reader to Section D.5.\n\nIt would have been interesting to investigate the stability of RID across different dataset resample methods (e.g. bootstrap v.s. subsampling).\n\nIt would be good to explicitly list some of the model classes for which the RID can currently be computed.\n\nNitpicking a little…\nLine 140: Unless I’m mistaken, g* tells us P(X,Y), while f^* often aims to model P(Y|X), and needs to be used alongside some approximation of P(X) to be a surrogate for g*.\nLine 149: “the Rashomon set describes the set of good explanations for a single dataset” -- this is only true if the models themselves are interpretable. \nLines 162 and 187: seem to describe a model’s variable importance score as though it is fixed (independent of the dataset draw), i.e. by describing it as being a quantity weighted by the number of datasets for which f is in the Rashomon set, rather than something that can vary with each dataset (and only included when f is in the Rashomon set). The mathematical notation clarifies this however.\nLine 286: “For real datasets” -- I think you mean the remaining synthetic datasets that include noise\n\nQUESTIONS:\nIn Equation 2. (and others in the form of a ratio of model set cardinalities), did you consider a heterogenous weighting of the set members (the models), for instance a weight which decreases based on their deviance from optimality?\n\nLIMITATIONS:\nThe limitations could be emphasized a little more. For instance the claim in the abstract that “[Our framework] can be integrated with most existing model classes” is really only true in a theoretical sense. There are many model classes for which even estimating the Rashomon set is computationally prohibitive, let alone do so for hundreds of bootstrap resampled datasets.", "TITLE": "Official Review", "RECOMMENDATION": 8, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-04"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper studies the problem of quantifying variable importance in a stable manner. The authors argue that multiple models may explain a target outcome equally well for a given dataset. However, current methods to quantify variable importance only account for one of these models; therefore, without accounting for different explanations, different researchers may arrive at conflicting (and valid) conclusions. To solve this problem, the authors propose a framework to quantify variable importance that accounts for the set of all good models. The authors provide empirical and theoretical results to support their framework.\n\nSTRENGTHS:\n* The paper discusses an important and interesting problem of assigning variable importance while considering the whole set of good models and ensuring stability. This problem can interest the community studying the impacts of the Rashomon effect and, more broadly, the interpretability community. \n\n* The authors provide compelling experimental results in synthetical data, indicating that the proposed method can capture variable importance in data generation. They also compare with other methods in the literature and show that RID performs better or equal to state-of-the-art methods (Figure 3 top).\n\n* Theoretical results in Theorems 1 and 2 ensure that estimating by bootstrapping converges to the value of interest. \n\n* The Case study in Section 5 is fascinating. Unlike others in the literature, the authors demonstrate that their method associates a specific gene with HIV – previously an unknown relationship.\n\nWEAKNESSES:\n* The Rashomon set is still unknown for most model classes, making the application of the method infeasible using the tools presented in the paper. \n\n* Assumption 1 seems reasonable for model classes such as linear models and GAMs. However, how it will behave in more complicated classes is still unknown.\n\nQUESTIONS:\n* Are there any losses in using an approximation for the Rashomon set instead of the “True” empirical Rashomon set? Intuitively, there seems to be a tradeoff between the approximation for the Rashomon set and the RID. It would be interesting to have a result highlighting it in the main paper. \n\n* Can authors include an experiment like the one in Figure 3 top but for more complex data generation processes? For example, consider the ground truth DGP to be a deep neural network and calculate variable importance using decision trees.\n\nLIMITATIONS:\nThe authors discuss the limitations of the proposed method.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-04"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces a method for assessing the variable importance in prediction, when the goal is to understand variable importance as defined with respect to the underlying data-generating process, as opposed to a specific model.  To that end, a method is proposed which incorporates the concept of Rashomon sets (models whose performance is approximately optimal) and stability (e.g., considering Rashomon sets over different bootstrapped replications of the data) to construct a distribution of variable importance measures for each variable.\n\nSTRENGTHS:\nOverall, I rather enjoyed this paper, modulo some reservations that I outline in the \"weaknesses\" section. The presentation of the claimed contributions is clear, the contextualization to related work seems thoughtful, and the experiments support the main argument.\n\nFirst of all, I found much of the presentation to be quite clear.  Figures 1 and 2, for instance, give a fairly clear summarization of the motivating problem and the corresponding method.\n\nSecond, I found the technical contribution to be fairly clear relative to related work. Here, the main contribution appears to be the incorporation of bootstrapping to incorporate finite-sample uncertainty and improve stability, relative to related work that considers Rashomon sets for variable importance (e.g., citation [16] in this work).\n\nThird, while the main contribution (focusing on \"stability\") was not quite as formalized as I might have liked, the experiments in Section 4 seem to provide compelling evidence that the proposed approach more captures variable importance more reliably than baseline methods, including a fairly long list of alternative approaches.\n\nWEAKNESSES:\nThere are a few points in this paper that I felt were somewhat unclear.  I look forward to discussion with the authors during the response period, and I am willing to change my score.  I focus here on motivation for the given approach, and justification for the claims of \"stability\", which I found lacking in places. Generally, I found the technical results to be fairly straightforward consequences of the assumptions.\n\n### (W1) Motivation for \"stable\" variable importance somewhat unclear\n\nThe main weakness of this paper, in my view, lies in the motivation. In several places, the importance (no pun intended) of finding the \"ground truth\" variable importance measures is stated as an obvious fact, without justification.  For instance, on line 24 it is claimed that \"Variable importance would ideally be measured as the importance of each variable to the data generating process\".\n\nIt is not clear to me why this claim should be obvious - rather, it seems reasonable that we might want to understand variable importance in the context of a specific model, trained on a specific dataset, to better understand what drives the predictions of that model from an explainability perspective.\n\nFor instance, Fisher et al. 2019, cited in this paper as [16], give a fairly nuanced view when introducing the idea of Model Class Reliance (from my skim).  As I understood that work, the motivation for establishing a range of possible values for variable importance (VI) stems not necessarily from a desire to understand \"the data generating process\", but from the desire to understand variable importance for a single model. The catch is that the *model of interest may be proprietary and not available to the user* (e.g., in recidivism prediction).  In that setting, having upper and lower bounds on VI allows us to draw some conclusions (e.g., if the lower-bound is particularly high, we can conclude under some assumptions that the proprietary model depends on this feature).\n\nI think one could make a similar argument here, e.g., if the exact underlying dataset is also not available to us, we might be concerned about the sensitivity of our VI analysis to small differences in the data-generating process.  I would appreciate some comment from the authors on this question of the general motivation for finding stable variable importance measures.\n\n### (W2) Why consider distributions vs bounds?\n\nI had some trouble understanding the motivation for the Rashomon Importance Distribution.\n\nThis distribution seems to be something like \"averaging\" over the bootstrap replicates. As I understood it, if a variable's importance is at most k in one bootstrap sample, for all models in the Rashomon set, but greater than k for all models in the Rashomon set in 99 other bootstrap samples, then we will average these to say that the CDF at k is 1/100.\n\nIt wasn't clear to me how this approach would fit with the goal of establishing upper/lower bounds for importance measures.  I suppose that one could do this using quantiles of the resulting distribution, but it's not entirely clear to me what that would be measuring.\n\nTL;DR: If I were interested in something like \"the lower bound of the variable importance over the Rashomon set is higher than L with high probability\", is that something that could be read out from the RID?  That would seem like a more natural characterization of \"stability\" of results.\n\nAs an aside, it appears that [16] (Fisher et al. 2019) gives finite-sample / high-probability bounds on the upper/lower bounds of variable importance over the Rashomon set.  How should I think about the difference between the \"stability\" goal of this work (which is fundamentally a finite-sample concern) compared to the goal of providing high-probability bounds in that work?\n\n### (W3) What is stability, exactly, and why should we expect this method to achieve it?\n\nGiven that one of the main distinctions to prior work appears to be the focus on \"stability\", I was hoping to see some formal definition for what that means, and why the bootstrapping approach in this paper should be expected to achieve it.\n\nPresumably achieving stability is not the same as accurately estimating RID, since RID is defined in such as way that it can be estimated, for any sample size, with arbitrary precision. This fact is due to the definition of RID as an expectation over a bootstrapping distribution, with respect to our given dataset of size $n$ (see lines 158-161), allowing one to simply take more bootstrap replicates to estimate it precisely (as noted on lines 203-205).  If stability is a problem caused by finite samples sizes, then presumably it cannot be solved by simply re-sampling a small dataset.\n\nThe fact that the given strategy achieves stability seems to be taken as a given, e.g., RID is introduced on lines 150-151 as \"we define a stable quantity for variable importance\", it is said that \"intuitively, this [Eq 2] provides greater weight to the importance of variables for stable models\" (line 163), then it is concluded that \"since we stably estimate the entire distribution of variable importance values, we can create stable point estimates\" (lines 209-210), etc.\n\nI would be curious to get the thoughts of the authors on what stability means in this context, and why we should expect the approach in question to achieve it - is this simply a question of intuition, to be backed up by experiments?  If so, please just clarify that upfront.\n\n## Other minor points\n\nIt would be nice to discuss Assumption 1 in a bit more detail, explaining why it is necessary, when we might expect it to hold, etc.\n\nQUESTIONS:\nHere I have collected relevant questions that appear in my review above.  As stated previously, I am willing to update my score given compelling answers to some of these questions.\n\n1. (W1) How would you explain to a skeptical audience why we should be primarily concerned with variable importance measures that are not linked to a particular model?\n2. (W2) If I were interested in making claims like \"the lower bound of the variable importance over the Rashomon set is higher than L with high probability\", is that something that could be read out from the RID?\n3. (W2) It appears that [16] (Fisher et al. 2019) gives finite-sample / high-probability bounds on the upper/lower bounds of variable importance over the Rashomon set.  How should I think about the difference between the \"stability\" goal of this work (which is fundamentally a finite-sample concern) compared to the goal of providing high-probability bounds in that work?\n4. (W3) How would you formally define \"stability\" in this context, and why we should expect the approach in question to achieve it?\n\nOf these questions, I am most interested in the answers to 2-4. I understand that 1 is a bit more subjective, and despite appearing first, it has the smallest impact on my score.\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-07"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper presents a new method to find important predictive variables for a set of good prediction models (Rashomon set). The Ranshomon set is often unstable when some perturbations are added to the dataset. The present solution is based on an importance metric called Rashomon Importance Distribution (RID). Bootstrap sampling is proposed to estimate the cdf of RID. The method assesses the importance of each variable using the CDF that takes into account the Rashomon sets from different bootstrapped samples. This reduces the instability in variable selection due to the instability of the underlying Rashomon set.\n\nSTRENGTHS:\n1. Originality: The instability problem of variable selection for a Rashomon set is underexplored in the literature. The instability issue is more challenging than finding important variables for one model on one particular dataset. \n2. Quality: The paper is relatively well written. The proposed method is intuitive. A set of simulations and real-data studies demonstrate the value of the proposed method in applications.\n3. Clarity: The assumption of the proposed method is clear. Some basic consistency analysis is provided for the bootstrapping procedure.\n4. Significance: The paper makes a good contribution to addressing the instability problem that deserves more attention from the field.\n\nWEAKNESSES:\n1. Notation. For example in equation (2), the probability in the LHS is not indexed by n but the expectation in the RHS is an empirical distribution on the n observations. This is very confusing for me. What exactly the bootstrapping procedure is estimating? Is the unknown expectation assuming n observations drawn from an unknown distribution, or something else?\n2. Assumption 1. The connection between Rashomon Importance Distribution for a particular feature j and Rashomon Loss Distribution (RLD) is still not very convincing to me. The author should provide more intuitive explanations in the main text (e.g. using a linear model as an example). The current form of the assumption doesn't seem to hold for many model classes. \n3. The method is demonstrated in terms of finding the important variables. But does the method increases the false discovery of many non-important variables? More clarification and experiments can be provided on this aspect.\n\nQUESTIONS:\nSee weaknesses.\n\nLIMITATIONS:\nN.A.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-22"}