Implications of Model Indeterminacy for Explanations of Automated Decisions
Authors: Marc-Etienne Brunet
Ashton Anderson
Richard Zemel
Venue: NeurIPS 2022 Accept (2022)
Keywords: underspecification
Rashomon effect
explainability
robustness
epistemic uncertainty

Abstract: There has been a significant research effort focused on explaining predictive models, for example through post-hoc explainability and recourse methods. Most of the proposed techniques operate upon a single, fixed, predictive model. However, it is well-known that given a dataset and a predictive task, there may be a multiplicity of models that solve the problem (nearly) equally well. In this work, we investigate the implications of this kind of model indeterminacy on the post-hoc explanations of predictive models. We show how it can lead to explanatory multiplicity, and we explore the underlying drivers. We show how predictive multiplicity, and the related concept of epistemic uncertainty, are not reliable indicators of explanatory multiplicity. We further illustrate how a set of models showing very similar aggregate performance on a test dataset may show large variations in their local explanations, i.e., for a specific input. We explore these effects for Shapley value based explanations on three risk assessment datasets. Our results indicate that model indeterminacy may have a substantial impact on explanations in practice, leading to inconsistent and even contradicting explanations.

Implications of Model Indeterminacy for Explanations of Automated Decisions
Marc-Etienne Brunet University of Toronto Vector Institute
Ashton Anderson University of Toronto Vector Institute
Richard Zemel University of Toronto Columbia University
Vector Institute zemelêcs toronto.
mebrunet@es. toronto. edu
ashton@cs. toronto.edu
Abstract
There has been significant research effort focused on explaining predictive models, for example through post-hoc explainability and recourse methods. Most the proposed techniques operate upon single, fixed, predictive model. However, it is well-know that given dataset and a predictive task, there may be a multiplicity of models that solve the problem (nearly) equally well. In this work, investigate the implications of this kind of model indeterminacy on the post-hoc explanations of predictive models. We show how it can lead explanatory multiplicity, and we explore the underlying drivers. We show how predictive multiplicity, and the related concept of epistemic uncertainty, are reliable indicators of explanatory multiplicity. We further illustrate how set of models showing very similar test dataset may show large variations in their local explanations, i.e., for a specific input. We explore these effects for Shapley value based explanations on three risk assessment datasets. Our results indicate that model indeterminacy may have a substantial impact on explanations in practice, leading inconsistent and even contradicting explanations.
aggregate performance
Introduction
Data-driven decision making has become preferred practice within organizations. When the most important outcomes are readily quantifiable, entire decision making processes may even be automated using predictive models fit historic data. With much be gained from accurate predictions, powerful, modern, "black-box" machine learning (MI.) models have obvious appeal. Clearly, however, is understood that deciding whether deny access products like insurance and credit should not be taken lightly. Governing regulation may require a justification to be provided in conjunction with decision, such one from of adverse action codes (34). Even where such regulation is not required, there is increasing pressure to make automated decisions more transparent [31]. As result, considerable interest has recently been devoted explaining ML model decisions. Academics have proposed numerous new techniques [2]. new businesses have been founded to address this need, and regulators have invested significant effort provide guidance
Thus far, the focus of most machine learning explanation and transparency research has been on a single learned predictive model. However, it is well-known that given dataset and a predictive task, there may he a multiplicity of models that solve the problem (nearly) equally well 5 30) What are the implications of this kind of model indeterminacy for the explanations of automated decisions? It has been demonstrated that set of models with very similar performance on a test set may still show significant predictive multiplicity (20). which is say, the models make conficcing predictions on individual data points. Quantifying this kind of model uncertainty in machine learning predictions is the subject of considerable research interest. For instance it central work in Bayesian modelling, and to discussions of epistemic uncertainty [16]. But to what extent are explanations affected?
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

(e.g. Shapley value of feature k)
Aggregate Performance
P&ty [x)
set of approximalely equivalent models
Local Explana splanation at x
predictive variance
AUROC)
Model Output
(epistemic uncertainty)
gradient variance (local feature importance)
(e.g.
decision boundary variance (recourse)
approximately equivalent
Hypothesis space {f}
Input
Figure 1: [Left] An aggregate performance metric like AUROC (black solid line) and a local explanation like a Shapley value (blue dot-dash line) depend on a model's predictive function, f()), in different ways. Even if we see low variance in the aggregate performance across set of models, we may see high variance in the individual local explanations. [Right] Visualizing how model indeterminacy may affect predictive multiplicity (epistemic uncertainty), local feature-based explanations, and recourse separately. Consider a set of approximately equivalent models, and point in the imput space, The differences from the differences we may see in their gradients (with respect to Φ). The location of the nearest decision boundary may also change independently.
may in the model output probability at z are distinet
In practical applications, where the stakes are high, might model indeterminacy jeopardize the trustworthiness of an automated decision system by undermining transparency efforts? We explore
these questions in the context of risk assessment, i.e., insurance and credit applications, focusing on well-established class of explanation techniques which use Shapley values 7 analysis applies to a broader set systems and methods.
19 22
but our
Contributions In this work, we introduce the notion of explanatory multiplicity, and study it both analytically and experimentally. We show how a of models with very similar test-set performance may have large variations in their local explanations (Figure leftt. We further show how predictive multiplicity, not required for-or or even indicative of explaanatt multiplicity (Figure[[right) This is because in general, explanation methods are affected not just by the value of the predictive function point, but by its shape. We connect this analysis to an existing measure of model uncertainty: epistemic uncertainty [16]. Through experimentation, we explore explanatory multiplicity three risk assessment datasets. We measure the changes in the decisions and corresponding explanations that users would receive if an automated decision system switched between approximately equivalent models. Our results indicate that model indeterminacy may have a substantial impact on explanations in practice, leading to inconsistent and even contradicting explanations.
2 Setup and related work
Problem setup We consider the situation whereby user (applicant) is processed by a (partially) automated decision-making system, which ultimately accepts rejects them. We assume the system built around machine learning model that has been trained on relevant historical data. The model makes prediction which is used in the decision to accept reject the applicant, possibly in conjunction with other eligibility criteria or human expertise. However, we are concerned with explaining the model prediction to the user. The explanation should be local, in the sense that it should be specific to the user's ' input data, rather than describing a global property of the model.
Assume we have dataset of user-data outcome pairs, D {x,, W with multi-dimensional ..., Iss) X, and label y E y. We denote the of features (dimensions) of X' as M = {1,2.... .... M). Let f(z) be a predictive function from set of candidate functions (hypothesis space), F, mapping the users' data, X, to an output prediction score E C R used in determining their acceptance. A typical scenario may be that )' {0. 1}, with FF modelling the probability p(y = 1|x;,, which case = 00, 1]. We suppose that the system produces local, vector-valued explanations, of the form & F): + R This may be Jeature-importane vector such as the gradient [3] V.J. or those generated by LIME (2 or SHAP [19]. Alternatively,
input, x (x), (2,,22, I2,
this may be action vector determined by an algorithmic recourse method
32 in which case

---

hypothesis space all possible model instances
Figure 2: Model indeterminacy. The model in- stance that ends up in production is often not fully determined by the data, modelling objec- tives, and constraints. There likely be set of different models showing nearly indistin- guishable validation performance; this is known as the Rashomon effect. Even if the model class, hyperraaaameeess and optimization method are all specified, randomness in training may lead to substantial functional differences in the final model instance; this is known as underspeci- fication. Despite this potential for variability, most existing explanation techniques operate a single model instance.
Rashomon effect set "best performing" model instances source of variation: model class, hyperparameters, random seed
Underspecification set trained from the "best performing model class and hyperparameters source variation: random seed
further require (b(æ;) /) + x)) E X. either case, we (x;; J) denote the component of the explanation relevant to imput feature k E .M. We assume there exists dataset dependent model selection criteria, SD : F + {0, 1}. which filters F down to a set of approximately equivalent models, R R= (f F ; (f(f) = 1}. For example, the selection criteria may consist of limiting R to only include models which score above a certain threshold on an aggregate performance metric.
We are interested in the set of possible explanations that could be given to user across all approxi- mately equivalent models, (Φ(z; f): R]. We are particularly concerned with situations whereby there exist f, E R such that (x< < 0 < Φu (æ:; f"), as this would constitute contradicting explanations. user processed by I would be explained that feature contributed positively to their assessment (or should be increased in the case of recourse), but had the same user been processed by f', they would have been explained that feature & contributed negatively to their assessment (or should be decreased).
2.1 Model indeterminacy: Underspecification and the Rashomon effect
Model indeterminacy arises when there are many approximately equivalent predictive functions within the hypothesis space. We say that two models are approximately equivalent if, given context, we cannot justify why we should use one instead of the other- both would satisfy our model selection criteria. We consider two types of model indeterminacy: underspecification
studied by D' Ammour et al. [6]. and the Rashomon effect. as originally described by Breiman [5]. They arise in two slightly different commonplace contexts, but each may lead to sets of approximately equivalent models. The relationship between them is depicted in Figure 2 Underspecification arises when the model specification and training set are not enough to fully determine the model parameters. The training set, model class, hyperparameters, and optimization method are fixed, but the intentional use of randomness during training can lead to sets of approximately equivalent model parameters that effectively differ only because of the choice of random seeds. When the Rashomon effect is observed during model development, many models (possibly of different classes) show approximately equal performance. Choosing which model among them to use in deployment becomes an arbitrary choice; thus this set of best models may be considered approximately equivalent. The Rashomon effect is more general than underspecification, it does not require the model specification to be held constant. These types of model indeterminacy are very common in practical settings 6 5).
2.2 Shapley value based explanations
We focus on the class of local, feature-based explanations that use Shapley values to attribute importance to the model's input features. It has been shown that explanations satisfy a set of desirable criteria if and only if they are computed using Shapley values [9). This has put them in the spotlight
for regulatory purposes [34, (1). The core idea of using Shapley values for explanations is to explore all the possible subsets of features that the model could use to make a prediction, S M, where M is the set of all input features, and to calculate the average marginal effect that including feature k into has on that prediction. There are several related Shapley- methods, which vary in how they include/cxclude features from prediction. We present unifying formulation introduced by
3

---

Merrick and Taly [22] Let u( S)) be composite input function, which takes the values of æ for all the features in S M, and otherwise takes the values r. some other reference input; recall [M] = M. The Shapley value for feature l given input can then be written as
Φx o:
1 M
Σ
(15)
M 1 ER-DAI [f (u (æ, R, RU (k))) {u (x, (1)
(MM)(M)
The sum is over all subsets of features (excluding feature k). Each term the marginal change in the prediction when including feature k, divided by the binomial coefficient M - choose |S]. The reference distribution, Diet, is chosen to zero out the contribution of the excluded features. Typically, some variant of the input distribution is used. By simply varying the choice of reference distribution, we can recover several different Shapley- based explanation methods [22], including those used in popular techniques like KemelSHAA [9 and Quantitative Input Influence (7].
2.3 Uncertainty quantification
Model indeterminacy can be examined through the lens of uncertainty quantification. A popular, Dayesian-inspired framework used in this pursuit subdivides uncertainty into two categories: aleatoric and epistemic []4). Epistemic uncertainty aims to quantify the uncertainty in the model of the data generating process, and thus can be reduced by gathering more data. Aleatoric uncertainty aims to quantify the inherent (irreducible) noise in the data generating process. Kirsch al. [16] propose that the uncertainty in the prediction at input a of Bayesian model (with parameters 0) can be decomposed and categorized as
I O = - H DJ
E)(orD) H 0]]
(2)
epistemic
predictive
aleatoric
-
HI
Σ f(x 0.) π
(f((( 0,)
(3)
where H [Y sx, D] is the predictive entropy given the training data D. The mutual information term,
I[Y: 0x, D). which corresponds to the epistemic uncertainty. can be obtained by subtracting the aleatoric uncertainty from the predictive entropy. When we have access samples from p(0 D).
{0,) and f models the probability p(Y = y/z)) we can estimate Equation 2 with Equation 3 24]
The epistemic uncertainty at æ is high when the individual model instances, (f( 0.) confi-
dently disagree at æ. We use epistemic uncertainty as a principled way of measuring the variation in the prediction probabilities in sets of approximately equivalent models.
2.4 Related work
Local explanation methods have already attracted some criticism with regards to their robustness. For example, Alvarez-Melis and Jaakkola [I] have noted a lack of robustness small perturbations of the input being explained, while Lakkaraju and Bastani [8] show how this can be exploited to generate manipulative explanations. The effect of model indeterminacy on explanations has been explored in a philosophical sense by Hancox-Li [ Rudin (28) has argued against relying on post-hoc explanations of black box models in high stakes preisionnmakingg altogether. There has also been research into how the Rashomon effect can impact model interpretability. Semenova et al. 30 quantify the Rashomon effect, and differentiate it from existing ideas in the study of model complexity. Fisher et al. 9 consider the Rashomon set of "all well-performing models" within a specified class, and explore the extent to these models are necessarily dependent on any particular feature. They consider a global measure of feature importance called model reliance (MR), examining its range over the Rashomon set for each of the individual input features. Dong and Rudin [8] build on this work by characterizing the MR relationship between different features within the Rashomon set, using what they call the variable importance cloud (VIC). This set of work is highly relevant to our treatment of explanatory multiplicity, but it does not directly address the popular local explanation methods which explain the prediction for a specific user, i.e., a specific æ. These are currently the main candidates for explanations in practice. In contrast, global measures of feature importance like MR are tied to how important a feature is for model's aggregate performance on a whole dataset. Recently, Black et al. 4 examine the inconsistencies in model predictions and simple gradient explanations that arise from underspecification and leave-one-out changes to the training data. They propose using selective ensembles to make both predictions and explanations more consistent.

---

3 Explanatory multiplicity
We now examine how model indeterminacy can bring about explanatory multiplicity. First consider the relative consistency of performance metric, e.g., accuracy or AUROC, versus the value of a local explanation across a set of upproximately equivalent models, {6(œ; f) f S}. Regardless of the exact methods employed, the local explanation Φ<x;) f) will almost certainly depend on f differently than the aggregate performance metric. Therefore, even if set of approximately equivalent models show very similar aggregate performance, we should not expect consistency in their individual local explanations. The difference in dependency on f might transform a tight distribution over the performance metric into a wide, possibly multi-modal, distribution over the Shapley value for a feature, (x; f). This is illustrated in Figure (left).
Now consider how the form of the predictive function may vary in the region around some input across the approximately equivalent models. This is illustrated in F (right). We focus on how three local characteristics of the predictive function may vary within the set of approximately equivalent models. First, there can be variance in the value of the predictive function f f(z). Therefore, we may see considerable disagreement at any individual point in the input space. Strong disagreement would drive high epistemic uncertainty, per Equation [ Second, there can be variance in the gradient, (x)()), (or the discrete equivalent to this quantity if f is not differentiable). The gradient is an indicator of local feature importance. It is used directly in simple explanation techniques B, and it is closely connected to several others [29] including Shapley-value based methods in linear models. Finally, there can be variance in the position of the nearest decision boundary. This has implications for counterfactual explanations [33 and algorithmic recourse R2] It is possible that the nearest decision boundary to a user shift significantly between approximately equivalent models. This could nullify the validity of the recourse provided. Importantly, these three characteristics may vary independently. Therefore, while epistemic uncertainty may be a useful construct for indicating when we should expect to see variance in predictions, it is not indicative of when explanations will vary. Our experimental results presented below support this.
Illustrative example To illustrate these effects, we train 200 neural networks (MLPs) on a synthetic dataset (æ E RR2, y E {0, 1)). These model instances differ only because of differences in the random seeds used in training. We limit our consideration to the 3% highest-performing model instances, selecting those ranging between 93.3% and 93.6% validation accuracy. This constitutes set of approximately equivalent models. Because the models differ only in the random seeds, underspecifiiation is the source of model indeterminacy here. We plot the models' output (softmax probability) over region of the input space in Figure 15 in Appendix B Their functional forms may change considerably from one model instance to the next, in some cases, causing the nearest counterfactual boundary to shift significantly. A single model instance along with the (iid) test set is shown in Figure (left). The predictive multiplicity at each point in that region of the input space is calculated using the epistemic uncertainty in the set of approximately equivalent models (Equation 2 explained below) and is shown in Figure 3 (center). Notice it increases as we move toward regions of the input space for which we have less data. Finally,
compute the mean pairwise cosine similarity of the gradients (with respect to a:) between models over the same region of the input space, shown in Figure 3 (right). We see that predictive multiplicity is poor indicator of the similarity of the gradients across approximately equivalent models. For instance, consider the point at the orange star. We have low epistemic uncertainty (corresponding inter-model agreement in predictions), yet the gradients across the models at this point are very dissimilar (orange arrows).
Analysis in linear setting In a linear model class, it is possible to determine the precise conditions under which model indeterminacy can lead to contradicting explanations. We present a brief summary of this analysis here, and refer the reader Appendix C for more details. Consider the case where x E M y E TR and F is the set of ridge regression models of the form f(x; 0) = 0 x. with square error loss and &a penalty a O. The Rashomon set of all 0 which parameterize models within E of the optimal loss, is an ellipsoidal region, +,, centered at the OL.S solution, 0" We are interested in cireumstances whereby the sign of Shapley value reverses within this Rashomon set, i..., 3 0, 0' E 22 s.f. Φu (x; 0) > 0 > (2; 0').
It can be shown that the Shapley value of feature k in this ridge regression setting is dh 0) BL.KK Since x is fixed for a user, Φh changes sign if and only if 6g does. In the simple case where the features of x are independent, the ellipsoidal region O₂ is axis aligeel We show that fg is bounded in an interval centered at 0z having width 2(2/ ( Σn x2 + a)) For fixed e relaxation from
5

---

y=1
3.90
y-0
0.80
0.54
0.64
-56
0.32
-64
0.36
72
-80
output probability ply=1|x)
log epistemic uncertainty
gradient cosine similarity
Figure 3: [Left]] Softmax probability of an MLP over region of the input space. The test set is depicted as purple circles and yellow Xs. [Center]: Epistemic uncertainty over the same region the input space is computed using set of approximately equivalent model instances arising from the same training pipeline. [Right]: Mean pairwise cosine similarity of the gradients (w.r.t. the input) between models over the same region of the input space. The orange arrows are the gradients for (a subset of) the individual models at the point indicated by the orange star (left and center).
optimality and regularization coefficient a, the width of this interval decreases with the variance of the feature k. So Φb is candidate for sign reversal if zg is low variance independent feature, and 0% near zero.
Consider now if features j and k are correlated, with Cov(a, vx) > O, but all other features are independent. The ellipsoidal region is aligned for all axes {j. kj. but in the j, k plane, the cross-sectional ellipse is rotated. In this case, we show that 0g is also bounded in an interval centered 8- If we E, a, and the variance of features and k, we show that the width of this interval increases with Cov(z xx). Intuitively, this tells us that problems for which the input features are strongly correlated are more prone contradicting explanatory multiplicity. However, in complex hypothesis spaces, it is considerably more difficult to analyse explanatory multiplicity theoretically. We instead explore through experimentation.
4 Experiments
To explore the extent to which model indeterminacy may impact the consistency explanations in a practical setting, we conduct series of experiments. We use three different (binary) risk assessment datasets (all available on Kaggle): UCI Credit Card [35], Give Me Some Credit, and Porto Seguro's Safe Driver Prediction. Their details can be found in Appendix B. In each case, let = indicate the "bad" outcome: missed payment, serious delinquency, and insurance claim filed, respectively. We fit predictive models to each of these datasets, and consider their use in an automated decision system that filters out applicants (users) who pose an unacceptably high risk of a bad outcome. We then measure how the predictions and corresponding explanations the Rashomon effect and underspecification Our experiments were conducted on GPU accelerated computing cluster. ML models were written in PyTorch [26). and the analysis used NumPy [12 and Matplotiib 13
affected by
Preprocessing and model selection We first split each dataset into a development and a holdout set (70 30), and apply one-hot encoding and standard scaling. We then run model selection process with three model classes: logistic regression (LR), multi-layer perceptron (MLP), and tabular ResNet (TRN) recently proposed by Gorishniy et al. [10]. We sweep through range of hyperparameter settings, trying total of 408 model-hyperparrmeter configurations per dataset. For each configuration, we pick a random seed and use to control a shuffled split of the development dataset into train and validation sets (70/ 30). This seed also controls the randomness used in training (optimization). We fit the models using Adam [5] with a patience- patience-based stopping criteria on the validation set. We also up-weight the rare class, creating balanced loss. We repeat this process with 3 random seeds per configuration, obtaining a total of 1224 model instances per dataset.
1 'These packages are publicly available under BSD-style licenses.
2 Experimental source code will be made available at http:////eeeeeeeee///aaaeeeeeeee

---

Forming sets of approximately equivalent models To explore the effect of underspecification, we identify the best model-hyperpaare model instances from it. We refer to these as underspecification sets, Specifically, we use the AUROC on the validation sets (averaged over 3 random seeds) to determined the best hyprrratppre configuration for each dataset. We then train 100 models instances using that configuration. In this final training, we fix the number of epochs, and use the full development dataset for each. Thus the only source of variation between model instances is the random seed controlling initialization and mini-batch order. To explore the impact of the Rashomon effect, we simply skim off the top 3% of model instances (36 of 1 1224) from the model selection sweep, ranked according to their AUROC performance on their validation sets. We refer to these as Rashomon effect sets as they stray slightly from the definition of a "Rashomon set" by Semenova et al. [30] These sets include model instances from different model--yyperprrmeeeer configurations, and different controlling random seeds. After the holdout data to ensure that none of the model instances show
configuration and generate many approximately equivalent
forming of either type, unusually low performance out-of-sample.
Computing explanations For each dataset, we consider subset of 1000 instances from the holdout set. We generate Shapley value explanations on these points for each of the approximately equivalent models. Notice that in Equation T there is an expectation over the reference distribution, R ~ D, as well sum must be approximated for inputs with even a modest number of features, and the expectation converted to an empirical mean. Fortunately, there has been considerable effort put into developing estimators for Shapley values. this work we use a simple antithetic sampling approach described in (23] Importantly when comparing Shapley- based explanations across models are careful to use precisely the same set of samples, thereby ensuring that if two models agreed pointwise everywhere in their domains, their explanations would be identical We consider two different reference distributions, D in Equation [ each corresponding to different type of Shapley explanation. The first is point mass at the mean input, E [x).. In the language of Merrick and Taly (22), this amounts to considering a "single-referrece game"; ; the explanations contrast the user with the "average applicant". We refer to this as D We also consider the input distribution; this amounts to explaining with KernelSHAP. As is typical, to limit compute time, we approximate the input distribution using a small subset of the training data. We randomly select 100 reference inputs, and we reuse them for each explanation to ensure this is not source of variation. We refer to this as Dinput.
sum over all subsets of features in M which grows as (22) In practice this
4.1 Quantifying Consistency in Explanations
To quantify the consistency of explanations we focus on the top-k (in magnitude) Shapley values explaining a user's predicted risk (of a bad outcome). These correspond to the features which are most significant to the prediction. It recommended limit feature-based explanations to a subset of the most important features (27). thus these are most likely what would be shown to the user. If they are positive-valued, they indicate the feature adds the applicant's predicted risk. If they are negative-valued, they indicate the feature reduces that risk. We consider three metrics to quantify the consistency of the explanations. Each of these metrics takes in a pair of explanations and outputs a numerical value. For each user, we average these metrics over the explanations from every pair of approximately equivalent models ( [R] choose 2 pairs).
Signed Set Disagreement (SSD) We first consider a binary metric which takes value 0 if the sets of top-k features agree between two explanations, and value of 1 if they do not, i.e., they disagree. For agreement within the sets of top-k, we require that the sign of the Shapley values be the same, but not their numerical values or relative order. For example, suppose explanation A has Shapley values {Ø1 : 0.4, Ø2 : 0.7, Ø3 : -0.9, 64 -0.1). J. but explanation B has {Φ1 : - -0.5, d2 0.8, da -0.7, o. 0.3). Their top-1 Shapley values disagree: { vs. {d2). their top-2 Shapley values agree: {Φz, } but their 3 Shapley values disagree because Φ1 does not have the same sign in both explanations. When averaged over pairs of approximately equivalent models, the SSD estimates the probability that a user would see disagreement in the set of (signed) top-k explanatory features if the predictive model were switched to another approximately equivalent model.
Contradicting Direction of Contribution (CDC) We are most concerned about contradicting explanations, as these can be confusing or even harmful for user. For this we consider another
P We elaborate on this difference Appendix B.3
Further information about this estimate is available in Appendix
7

---

binary metric which takes value 1 if at least one of the top-k Shapley values changes sign (direction of contribution) across two explanations, and takes value otherwise. Thus the metric takes value 1 if there is at least one top-k feature, say feature-j, which is adding to the user's predicted risk in explanation A, but reducing their risk in explanation B. (Or vice-versa, going from reducing to adding). We require feature-j to be in the top-k features of explanation A or B, not necessarily both. When averaged over pairs of approximately equivalent models, the CDC estimates the probability that a user would see top-k explanatory feature change direction of contribution if the predictive model were switched to another approximately equivalent model.
Sign Agreement (SA) We also make use of the Sign Agreement (SA) metric proposed by Krishna al. [17]. This metric computes the fraction of top-k features that are not only common between two explanations, but also share the same sign (direction of contribution) in both the explanations. It takes fractional values from {0, K N 1}. Note that SSD indicates if SA < 1.
5 Results
Here we present the results of our experiments. Note that our figures focus on Dimput, underspecifica- tion, and the UCI Credit Card dataset. The corresponding figures for D mmaan- the Rashomon effect, and other datasets are presented in Appendix (D] We tabulate the results of the hyperparameter sweep and set formation in Table 2 in Appendix B.2 The probability that two approximately equivalent models disagree on a user acceptance decision is function of the decision threshold (risk appetite). With decision thresholds set to 50% user acceptance rate, disagreement ranges between 4.35-12.7% across our experimental setups. We report more details on the consistency of decisions across a range of user acceptance rates in Appendix D.I
140
top
120
500
tep- 3
Number of users
Number of users
100
600-
80-
400-
40-
200
02 0.4
1.9 0.0
06 Probability sign-change top-k (CDC)
Probability disagreement top-k (SSD)
Figure 4: Consistency of explanations. We compare the top-k Shapley values across approximately equivalent models for 1000 users in the hold-out set. [Left] Histogram over the probability of inter-model explanation disagreement (SSD) for each user. [Right] Histogram over the probability of moter-model contradiction (CDC) in the top-k Shapley values for each user. [Shapley type: Dimputi dataset: UCI Credit Card; source of indeterminacy: underspecificatiio)
Consistency Metrics The signed set disagreement (SSD) and the contradicting direction of contribution (CDC) metries are plotted in a (per-user) histogram in Figure They are also averaged over 1000 users and reported in Table In general, we see considerable inconsistency. If a user were shown explanations of a prediction they received from two approximately equivalent models, the probability that the set of top-3 Shapley disagree is over 35% in all our experimental settings, and over 50% in two thirds of them. The odds of a contradiction vary considerably across the experimental settings. They are remarkably high in some of the Rashomon sets, but roughly an order of magnitude lower in the underspecification sets. Nonetheless, the odds a contradiction in the top-5 Shapley values is over 10% in three quarters of our experimental settings, and over 20% in half of them, potentially alarming levels in real-world setting.
Visualizing (in)consistency Explanations are often reported in a visual form. We explore the visual consistency of the explanations across approximately equivalent models. To do so, we sort users by the sign agreement (SA) in their top-5 Shapley values across pairs of approximately equivalent models. High SA means the model explanations tend to agree. We select user, then we plot the explanation for each model within an approximately equivalent set separate trace. We limit the
8

---

Table 1: Consistency of explanations
UCI Credit Card top-3
Give Me Credit top-3
P.S.'s Safe Driver top-3
Shapley type: Pmcan
top-5
top-5
top-5
Rashomon effeet
Prob. of disagreement (SSD) (%)
83.5
92.8
66.9
70.1
78.4
92.1
Prob. of contradiction (CDC) (%)
46.6
62.7
6.6
19.5
19.2
34.2
Underspecitication
Prob. of disagreement (SSD) (%)
72.1
88.2
54.1
63.3
48.1
67.7
Prob. of contradiction (CDC) (%)
7.9
20.1
3.4
13.5
0.8
2.5
Shapley type: D mmputt
Rashomon effect
Prob. of disagreement (SSD) (%)
79.4
90.3
59.9
69.8
76.6
89.2
Prob. of contradiction (CDC) (%)
36.5
54.9
11.1
29.0
13.0
23.6
Underspecification
Prob. of disagreement (SSD) (%)
51.1
67.2
35.6
47.1
43.4
60.9
Prob. of contradiction (CDC) (%)
2.3
9.3
2.4
10.4
0.2
0.6
plot the features that in the top-5 for least model. An example of such plot for the user in the 10th percentile SA can be seen in F (left). We see a contradiction in the explanations for feature "pay-2". For some models, would be explained to the user that this feature was among the top-5 reasons for a decision, but if an approximately equivalent model were used instead, it could be explained that the same feature having the opposite effect on their decision.
0.30
softmax outpput "bad")
2.8
Signed Agreement (SA) in top-5
1.-
spearman 0.0
0.05
Shapley Value
2.0
-0.05
6.10
2.4
-6.20
-6.5
Hitt bal
ply.
pay_?
pay.
pay.
pay.
bill ami2
pay amii
eddcattont
aduciation.
Epistemic uncertainty
Figure 5: [Left and Center] Visualizing inconsistencies in explanations. Each trace corresponds to one model's explanation: The Shapley values for a subset of the most important features. Notice that "pay_2" crosses the line. For some models it would be presented among the top-5 reasons that decision made, yet for other approximately equivalent models, it would be explained having the opposite effect on that decision. In the center, we show the models' softmax outputs. The models tend to agree in their predictions, but not in their explanations. [Right] Predictive vs. explanatory multiplicity. Here we illustrate the relationship between predictive multiplicity and explanatory multiplicity. For 1000 users in the holdout set, we plot the (between pairs of model instances) vs. the (log) epistemic uncertainty in the predictions. The lack of correlation illustrates how epistemic uncertainty is not predictive of consistency in explanation. [Shapley type: Dinput; Dataset: UCI Credit Card; Source of indeterminacy: underspecification)
intersection of top-5 Shapley values
Variation in predictions vs. variation in explanations Within our set of approximately equivalent models, we explore the relationship between the variation in the predictions for user and the variation their explanations. We do not observe consistent relationship across our experimental setups. There are users with low explanatory agreement and high predictive agreement, and vice versa. To illustrate this, we plot the sign agreement (SA) the top-5 Shapley values vs. the epistemic uncertainty in the associated predictions (computed via Equation 3) for 1000 users. See Figure

---

(right). We also compute the Spearman correlation coefficient (p). In the UCI Credit Card and Safe Driver datasets, there is effectively no correlation; p ranges from-00, -0.17 to 0.23. In the Give Me Some Credit dataset there is weak to moderate negative correlation; p ranges from -0.34 to -0.58. One may wish to characterize the uncertainty in an explanation stemming from model indeterminacy. However, our results suggest that the epistemic uncertainty in prediction is not reliable indicator of the uncertainty in an explanation.
Summary and Analysis Our results indicate that 1) model indeterminacy can lead to substantial inconsistencies the local explanations of approximately equivalent models, even when models only differ because of underspecification: 2) variation in an explanation for a user across approximately equivalent models can occur with or without variation the prediction, i.e., whether the epistemic uncertainty in that user's prediction is high or low.
On the one hand, explanatory multiplicity may be viewed consequence of high fidelity expla- nations, which ought to show a sensitivity to the fluctuations of the predictive functions. However, when we consider providing these explanations to a user, whether as a justification for decision, or as information they can act on, the implications are concerning. Without some kind of guarantee about how long particular model instance will remain in use, the explanations may be of very little value, or even be detrimental to the users. (Additional results in Appendix D
6 Discussion
There are notable societal implications to this work. Consider being denied a loan then explained that your number of existing lines of credit was very detrimental to your application, only to learn that-under trivially different model-your existing lines credit would have been explained as having helped your application. How would this affect your trust in the decision-making system? Explanation techniques that use Shapley values have been rigorously motivated, and may represent a substantial operational cost in practice, requiring orders of magnitude more compute than just making a prediction. It is sensible to want them to be robust. We should at least communicate the limits of their robustness to the user, they understand exactly what is being explained, i.e., we should be explicit about the scope of the explanation. We elaborate on this discussion in Appendix
There is much to explore in future work. Our theoretical analysis is currently limited to a simple linear class of models; our experiments are limited to just two forms of Shapley value explanations. It would be important to explore the empirical effects of model indeterminacy methods including recourse. Notably, there wide range in the magnitude of the explanatory multiplicity measured across current experimental settings. It would be valuable to understand the drivers this variation so it may be preempted in practice. It may also be helpful to put the effeets of model indeterminacy into perspective by comparing them to human meakiss Human decisions, and the explanations of those decisions, are also liable to fluctuate arbitrarily. However, results remind us that just because decision or explanation was produced by "data driven" model, does not mean that they are fully determined by the data. We hope our work inspires a focused research effort into developing explanation techniques that more robust to the trivial or arbitrary model choices that occur in practice.
other explanation
Acknowledgments and Disclosure of Funding
Resources used in preparing this research were provided, in part, by the Province of Ontario, the Government of Canada through NSERC and CIFAR, and companies sponsoring the Vector Institute
We would like thank the Vector Institute's computing and administrative staff for their support, the anonymous reviewers for their insightful feedback, and Elliot Creager and Isaac Waller for their helpful comments.
http:///ectorinsituee.ll
10

---

References
[1] D. Alvarez-Melis and T. Jaakkola. On the Robustness of Interpretability Methods. arXiv preprint arXiv: /806 1806.0804,, 6 2018. URL httphttsss//aaiii.ororre.t
[2] P. P. Angelov, E. A. Soares, R. Jiang, N. I. Arnold, and P. M. Atkinson. Explainable artificial intelligence: analytical review. Wiley Interdisciplinary Reviews: Data Mining and Knowledge Discovery, 1 11(5,, 2021. ISSN 19424795. doi: 10.1002/widm.1444 1424.
[3] D. Bachrens, T. Schrocter, S. Harmeling, M. Kawanabe, K. Hansen, and K. R. Müller. How to explain individual classification decisions. Journal of Machine Learning Research, 11, 2010.
ISSN 15324435.
[4] E. Black, K. Leino, and M. Fredrikson. Selective Ensembles for Consistent Predictions. In International Conference on Learning Representations, 2022. URL https://opearrrviiie ierforuumiddnffyyyCCC
[5] L. Breiman. Statistical modeling: The two cultures. Statistical Science, 16(3):199-215, 2001. ISSN 08834237. doi: 10. 1/12214///00002117222
[6] A. D' D'Amour, K. Heller, D. Moldovan, B. Adlam, B. Alipanahi, A. Beutel, C. Chen, J. Deaton, Eisenstein, M. D. Hoffman, Hormozdiari, N. Houlsby, S. Hou, G. Jerfel, A. Karthike- salingam, M. Lucic, Y. Ma, C. Mcl.ean, D. Mincu, A. Mitani, A. Montanari, Z. Nado, V. Natara- jan, C. Niclson, T. F. Osborne, R. Raman, K. Ramasamy, R. Sayres, Schrouff, M. Seneviratne, S. Sequeira, H. Suresh, V. Veitch, M. Vladymyrov, X. Wang. K. Webster, S. Yadlowsky, T. Yun, X. Zhai, and D. Sculley. Underspecificatioon Presents Challenges for Credibility in Modern Machine Learning. 2020. Uttptp////axxi org/ haaggg/bs/22000 2ttp.//211..33333
[7] A. Datta, S. Sen, and Y. Zick. Algorithmic Transparency via Quantitative Input Influence: Theory and Experiments with Learning Systems BT 2016 IEEE Symposium on Security and Privacy, SP 2016, May 23, 2016 May 25, 2016. Security and Privacy (SP), 2016 IEEE Sym- posium on, pages 598-617. 2016. URL https: //wtp andrew cmu edu/ Busuuuuuur///ee wewtuuuser/aauuppaa danupam/ datta-sen-zicae
and16 wwtp://tttpp//////opppp //dx www.do...oo.o...o 1P09/SP 2016. .42
[8] Dong and C. Rudin. Exploring the cloud of variable importance for the set of all good models. Nature Machine Intelligence, 2 10 824, 2020. ISSN 25225839. doi: 1038/ 112256-020-00264-0 URL /outt://das doi org/10. 1tt///10.118//44410 1038/0022226662 00005566-000000026000 00264-0
[9] A. Fisher, C. Rudin, and F. Dominici. All models are wrong, but many are useful: Leaninng variable's importance by studying an entire class of prediction models simultaneously. Journal of Machine Learning Research, 20: 10: 2019, ISSN 15337928.
[10] Y. Gorishniy, I. Rubachev, V. Khrulkov, and A. Babenko. Revisiting deep learning models for tabular data. Advances Neural Information Processing Systems, 34, 2021. URL httpst
//arxiv. org/
11959
[11] L. Hancox-Li. Robustness in machine learning explanations: Does it matter? In FAT 2020 Proceedings of the 2020 Conference on Fairness, Accountability and Transparency, pages 640-647. Association for Computing Machinery, Inc, 2020. ISBN 9781450369367. doi:
10.1145/3311095533755
[12] C. R. Harris, K. J. Millman, S. J. van der Walt, R. Gommers, P. Virtanen, D. Cournapeau, E. Wieser, J. Taylor, S. Berg, N. J. Smith, R. Kern, M. Picus, Hoyer, M. H. van Kerkwijk, M. Brett, A. Haldane, J. F. del Río, M. Wiebe, P. Peterson, P. Gérard-Marchant, K. Sheppard, Reddy, W. Weckesser, H. Abbasi, C. Gohlke, and T. E, Oliphant. Array programming with NumPy, 2020. ISSN 14764687.
[13] J. D. Hunter. Matplotlib: A 2D graphics environment. Computing in Science and Engineering, 9()), 2007. ISSN 15219615. doi: 10. 10.100//CEE.2000.5555
[14] A. Kendall and Y. Gal. What Uncertainties Do We Need in Bayesian Deep Learning for Com- puter Vision? In Guyon, U. V. Luxburg, Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, editors, Advances in Neurul Information Processing Systems, volume 30. Cur- ran Associates, Inc., 2017. URL https: http///pooeeng...cooo www.ggs..uuiis.... http:/g.e...re.peeeeeeeee//////e pctpcc/ap/ppe//////eeel 0550609994666006655888888662222 Paper, pdf

---

[15] D. P. Kingma and J. L. Ba. Adam: A method for stochastic optimization. In 3rd International Conference on Learning Representations, ICLR 2015 Conference Track Proceedings. 2015.
[16] A. Kirsch, J. van Amersfoort, and Y. Gal. BatchBALD: Efficient and Diverse Batch Acqui- sition for Deep Bayesian Active Learning. In H. Wallach, H. Larochelle, A. Beygelzimer, Alché- Buce E. Fox, and R. Garnett, editors, Advances Neural Information Processing Systems, volume 32. Curran Associates, Inc., 2019. URL Paper. pdf
neurips
cc/paaper/011//// 20w566000/0000a022666eee0ee
[17] S. Krishna, T. Han, A. Gu, J. Pombra, S. Jabbari, S. Wu, and H. Lakkaraju. The Disagreement Problem in Explainable Machine Learning: A Practitioner's Perspective, 2022. URL https
7/arxiv. www.vv.or////o.oo abs/ 2202. 01602
[18] H. Lakkaraju and O. Bastani. "how do i fool you?": Manipulating user trust via misleading black box explanations. In AIES 2020 Proceedings of the AAAT/ACM Conference on AI, Ethics, and Society, 2020. doi: 10. 1e5111/43/677.33377777 33375883.
[19] S. M. Lundberg and S. I. Lee. A unified approach interpreting model predictions. Advances in Neural Information Processing Systems, 2017-Decem(Section 2):4766-4775, 2017. ISSN 10495258.
[20] C. T. Marx, F. D. P. Calmon, and B. Ustun. Predictive multiplicity in classification. In 37th International Conference on Machine Learning, ICML 2020, volume PartF168 147-9, 2020.
URL hotttttts///rxii> org/ abs/ 1909. 006777
[21] MAS. Veritas Document 3C FEAT Transparency Principles Assessment Methodol- ogy. Technical report, Monetary Authority of Singapore (MAS), 2 2022. URL https
7 //www mas. www.:aaw..o...... www.ggg//mmee http://aawaaaaaaaaaaaaaaaaaeeaaa
Priitas-Dccumnte -epeeentt-CCC-- --- - FEAT Trw.aaaappaaaegrroahiieeeeeeeeee Meghooooogy
pdf
[22] L. Merrick and A. Taly. The Explanation Game: Explaining Machine Learning Models Using Shapley Values. In Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics), volume 12279 LNCS, pages 17-38,
2020. ISBN 9783030573201. doi: 10.
[23] R. Mitchell, J. Cooper, E. Frank, and G. Holmes. Sampling Permutations for Shapley Value Estimation. Journal of Machine Learning Research, 23(43):1 2022.
prg/papers/v23/211 /tl/////211112 210/3/21-044 0439. html
[24] J. Mukhoti, A. Kirsch, J. van Amersfoort, P. H. S. Torr, and Y. Gal. Deterministic Neural Networks with Inductive Biases Capture Epistemic and Aleatoric Uncertainty. (iD)), 2021. URL http://arxiv. org. hbtg//bb/22/ 2102. 11582
[25] K. P. Murphy. Machine learning: A Probabilistic Perspective. 2012. ISBN 978-0-262-01802-9
[26] A. Paszke, S. Gross, F. Massa, A. Lerer, J. Bradbury, G. Chanan, T. Killeen, Z. Lin, N. Gimelshein, L. Antiga, A. Desmaison, A. Köpf, E. Yang, Z. DeVito, M. Raison, A. Tejani, S. Chilamkurthy, B. Steiner, L. Fang, J. Bai, and Chintala. PyTorch: An imperative style, nigh-performance deep learning library. In Advances in Neural Information Processing Systems, volume 32, 2019.
[27] M. T. Ribeiro, S. Singh, and C. Guestrin, "Why should i trust you?" Explaining the predictions of any classifier. Proceedings of the ACM SIGKDD International Conference Knowledge Discovery and Data Mining, 13-17-Augu: 133--1144, 2016. doi: 10.11457939977..993777
[28] C. Rudin. Stop explaining black box machine learning models for high stakes decisions and use interpretable models instead. Nature Machine Intelligence, 1(5):206-215, 2019. ISSN 25225839. doi: 10.1038/42256001190044.... URL //tp:://xxxooo org/10. 1038 0192560019-0044
[29] S. Samuel, V. Kamakshi, N. Lodhi, and N. Krishnan. Evaluation of Saliency- based Explainability Method, 2 2021. URL. https://uti adsabs...dd.......///.aa...a.. haw.aard.araardd..oo edu/ abs/
2021arXiv2106127739 abstract
12

---

[30] L. Semenova, C. Rudin, and R. Parr. A study in Rashomon curves and volumes: A new perspective on generalization and model simplicity in machine learning, pages 1 1-64, 2019. htt.htttt///xii.. org/ wttpg/abs1190..11155 01755
[31] A. Smith. Using Artificial Intelligence and Algorithms, 4 2020. URL
http:::/wwwww.t..gg//oo. htt.://gooreesseeeeteeee/eeeeaee wwwete/ebogg//eeeeeeeeee www.bbllooo///ooeeooooooooooooot 2020/04/ using-
[32] B. Ustun, A. Spangher, and Y. Liu. Actionable recourse in linear classification. FAT 2019 Proceedings of the 2019 Conference on Fairness, Accountability, and Transparency, pages 10-19, 2019. doi: 10. 10.11455/2775500022855..
[33] S. Wachter, Mittelstadt, and Russell. Counterfactual Explanations Without Opening the Black Box: Automated Decisions and the GDPR. Harvard Journal of Law and Technology, 31 doi: (1139//ssn.30632553
(2):842-884 887, 2018. ISSN 1556-5068.
[34] W. Wang, C. Lesner, A. Ran, M. Rukonic, J. Xue, and E. Shiu. Using small business banking data for explainable credit risk scoring. AAAI 2020 34th AAAI Conference on Artificial Intelligence, pages 13396-134011 2020. ISSN 2159-5399. doi: 10.1609/aaai, v34i08.7055 7055.
[35] I. C. Yeh and C. h. Lien. The comparisons of data mining techniques for the predictive accuracy of probability of default of credit card clients. Expert Systems with Applications, 36(2 PART 1), 2009. ISSN 09574174. doi:
Checklist
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's
contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes] For example, we note the limitations of our theoretical work, some experimental compromises we need make to limit compute resources, as well as the fact that our experiments are limited to one form of Shapley value explanations.
(c) Did you discuss any potential negative societal impacts of your work? [Yes] Our work directly engages with the (possible) negative societal impacts that model indeterminacy can have on ML transparency.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [Yes] pendix C
See Ap-
(b) Did you include complete proofs of all theoretical results? [Yes] See Appendix
C
3. If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material or as URL)? [Yes] We link to public GitHub repository and will make the experimental source code available before the conference.
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes] This discussed in Section 4 as well as the supplemental. The full details will be visible in our experimental code.
(c) Did you report error bars (e.g., with respect to the random seed after running experi- ments multiple times)? [Yes] We did not report error bars per say, however the subject of the paper and experiments expressly deals with variation due to arbitrary choices such the random seed.
13

---

(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, cloud provider)? [Yes] The type of resources used is mentioned in Section and will be further addressed in the acknowledgements. We do not estimate the total amount of compute used. It was helpful but not strictly necessary to have access to GPU cluster.
4. If you are using existing assets (e.g., code, data, models) or curating/relcasing new assets...
(a) If your work uses existing assets, did you cite the creators? [Yes] See Section
(b) Did you mention the license of the assets? [Yes] See footnote in Section
(c) Did you include any new assets either in the supplemental material or as URL.? [No]
However, the experimental source code will be on GitHub before publication.
(d) Did you discuss whether and how consent was obtained from people whose data you're re using/curating? [No] But the datasets we used publicly available on Kaggle and allow for research usage.
(c) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [No]
5. If you used crowdsourcing conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if applicable? [N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation? [N/A]
14


Reviews:
[{"review_id": "oFEgmHYqpk", "reviewer": "Reviewer_RULt", "summary": "This paper studies how model indeterminacy impacts explanation multiplicity. Two aspects of model indeterminacy are considered:  predictive multiplicity (underspecification, rashomon sets), and epistemic uncertainty. As models change, explanations are bound to change as the authors show via experiments. However, authors also show that predictive multiplicity or epistemic uncertainty are not good indicators of the extent to which the explanations agree.\n\nThere has been prior work on disagreement between explanations by Lakkaraju et al. where different local explainability methods with respect to a single ML model. There is also follow up bayesian work on quantifying the uncertainty of local explanations by the same authors. Rashomon sets have been used by Rudin et al as the authors point. ", "questions": "a) Can you clarify the key results ? \n\nb) Can you clarify the 3 terms better: predictive multiplicity, model indeterminacy, and epistemic uncertainty ?\n\nc) How do Rashomon sets as computed by you differ from the ones defined in Rudin, et al? \n\nc) How is epistemic uncertainty computed here. Are Rashomon sets or underspecification sets used to compute it ? \n\nd) How would this work/results generalise to other explanation methods -- LIME, counterfactual techniques, etc. \n\ne) Are there any results for case when predictions of multiple almost equivalent models disagree for a user, how do the explanations disagree in this setting. \n", "limitations": "See above.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The problem studied by authors is very useful one, especially in practise. The experiments are quite thorough and good. However the results and their implications are perhaps less clear. \n\nWhen multiple almost equivalent models have the same prediction for a user, even then local explanations/gradients tend to be different. This is a useful insight. Authors hypothesise that this may be due to the fact that explanations depend on the shape of the predictive function, also seen visually for experiments on small problem sizes. The almost equivalent models are characterised using Rashomon sets and underspecification sets (randomness in training). The results are for one local explanation method -- shap explainer that approximates shapley values. \n\nThe related insight is a bit confusing: i.e. the extent to which the explanations disagree is not determined by the extent to which the predictions of multiple almost equivalent models differ. Is this correct ? Pls clarify\n\nThe other insight is: If we consider epistemic uncertainty, this might not be a good predictor of explanation multiplicity for that user across the almost equivalent models. Is this correct ? Pls clarify\n\nThe main issues with the paper are: \n(a) Lack of clarity on following 3 terms: predictive multiplicity, model indeterminacy, and epistemic uncertainty. \n(b) Lack of clarity on results.\n(c) Experimental results are based on only 1 local explanation method:  SHAP. \n(d) The analytical results are based on linear setting alone. ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ["I don’t know"]}, "extra_text": {}}, {"review_id": "aZC3rUZN2u", "reviewer": "Reviewer_UP6W", "summary": "The authors show how model multiplicity (i.e. different models which have similar test set performance) can result in explanation multiplicity.  Using Shapely values for feature importance, often models that are underspecified (i.e. have the same hyper parameters setup but different initializations) but have similar test performance can result in different feature importance rankings and even entirely contradictory ones. The authors compare Rashomon effect (vastly different hyper parameters, similar test performance) and underspecification (same hyper parameters, different inits, similar test performance). The Rashomon effect has a much greater impact on agreement and contradiction (as expected), but underspecification also results in fairly high levels of contractiction.", "questions": "The authors note \"The odds of a contradiction vary considerably across the experimental settings. They are remarkably high in some of the Rashomon sets, but roughly an order of magnitude lower in the underspeciﬁcation sets\", could the authors speculate as to why this is. Are the models in the Rashomon set drastically different? Also, why are the contradict percentages fro the UCI Credit Card dataset in the top-3 so much higher than the other datasets? Did the models for that dataset evaluate more poorly on the test datasets?", "limitations": "The authors show how the arbitrary selection between similarly performing models can results in drastically different explanations. They couch the work in datasets related credit and insurance, an area which providing good explanations would allow the individual to modify their behavior such that they can score higher. In their work they show how the initial explanation would not often hold-up if a different (but similarly performing) model had been selected.", "rating": 8, "confidence": 3, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Very strong experimental design tackling a very important issue. Results felt rushed, things are pushed to the appendix that should have been kept in the main paper. I found Figure 2 to be confusing, but the caption was very clear. Also, the right hand side of Figure 5. Overall, paper is very well written and the work is very significant.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "1_UsBobvSn3", "reviewer": "Reviewer_77bi", "summary": "The paper introduces a notion of explanation multiplicity. This is similar to previous works that look at the effects of model indeterminacy such as predictive multiplicity. The authors first show how explanations can vary for models with similar performance on toy examples and theoretically in the linear case. The authors demonstrate that model multiplicity can have noticeable effects on the explanations using shapley sampling explanations, and this can lead to the shapley explanations contradicting in practice for models IRL.", "questions": "* how many samples do you use to compute explanations? \n* Sampling-based approaches introduce instability themselves and do you consider whether the contradictions come from not sampling enough or explanation multiplicity?", "limitations": "Authors have done a sufficient job", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n- This paper is very well written and easy to follow. It was a pleasure to read, and I had no issues understanding the work.\n- The intuition the paper builds with toy examples before providing real experiments is nice. I enjoyed this aspect. \n- The idea is well-founded. Previous works on things like predictive multiplicity are quite concerning and this work fits into that literature nicely.\n- The fig. 1 is quite nice, very clear + descriptive!\n- This work also makes sense in the context of explanations, where aspects like stability have been studied extensively in the past (i.e., small perturbations leading to noticeable differences or simply rerunning explanations leading to large differences). So, this work is a nice extension in that regard to a related, but different notion about how explanations that seemingly should be the same are in fact different.\n- The experiments are clear \n\nWeaknesses:\n\nMy main critique of this work is that the effects of multiplicity, in this case contradicting explanations, is quite a practical concern, and while the authors have shown this can appear in some cases, the extent to which explanation multiplicity appears in practice is still somewhat left unanswered.  The authors empirically demonstrate their technique using a sampling shap-based approach with the mean as the baseline distribution. Choices around the baseline distribution vary pretty widely and often other variants of shap are used like treeshap + deepshap practically because they're more efficient. It's nice that this work provides evidence this can be a problem in the case presented, but it would be really compelling to evaluate the degree to which explanation multiplicity occurs in other variants of shap that are practically used and other choices of baselines because I suspect readers will be looking for their IRL use cases.\n\nThis paper is a weak accept for me because I think the work is quite solid, but extending the evaluation to additional explanation use cases would make the impact much higher.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept