Selective Ensembles for Consistent Predictions
Authors: Emily Black
Klas Leino
Matt Fredrikson
Venue: ICLR 2022 Poster (2022)
Keywords: consistency
prediction consistency
model duplicity
inconsistent predictions
deep models
deep networks
explanations
saliency maps
gradient-based explanations
fairness
interpretability

Abstract: Recent work has shown that models trained to the same objective, and which achieve similar measures of accuracy on consistent test data, may nonetheless behave very differently on individual predictions. This inconsistency is undesirable in high-stakes contexts, such as medical diagnosis and finance. We show that this duplicitous behavior extends beyond predictions to feature attributions, which may likewise have negative implications for the intelligibility of a model, and one's ability to find recourse for subjects. We then introduce selective ensembles to mitigate such inconsistencies by applying hypothesis testing to the predictions of a set of models trained using randomly-selected starting conditions; importantly, selective ensembles can abstain in cases where a consistent outcome cannot be achieved up to a specified confidence level. We prove that that prediction disagreement between selective ensembles is bounded, and empirically demonstrate that selective ensembles achieve consistent predictions and feature attributions while maintaining low abstention rates. On several benchmark datasets, selective ensembles reach zero inconsistently predicted points, with abstention rates as low as 1.5%.

Published as a conference paper at ICLR 2022
SELECTIVE ENSEMBLES FOR CONSISTENT PREDICTIONS
Emily Black, Klas Leino, Matt Fredrikson (emi lybla, kleino, mf mfrediik) @cs cmu edu Carnegie Mellon University
ABSTRACT
Recent work has shown that models trained to the same objective, and which achieve similar measures of accuracy on consistent test data, may nonetheless behave very differently on individual predictions. This inconsistency is undesirable in high-stakes contexts, such as medical diagnosis and finance. We show that this inconsistent behavior extends beyond predictions to feature attributions, which may likewise have negative im- plications for the intelligibility of a model, and one's ability to find recourse for subjects. We then introduce selective ensembles to mitigate such inconsistencies by applying hypothesis testing to the predictions of a set of models trained using randomly-selected starting conditions; importantly, selective ensembles can abstain in cases where sistent outcome cannot be achieved up to a specified confidence level. We prove that that prediction disagreement between selective ensembles is bounded, and empirically demonstrate that selective ensembles achieve consistent predictions and feature attribu- tions while maintaining low abstention rates. On several benchmark datasets, selective ensembles reach zero inconsistently predicted points, with abstention rates as low 1.5%.
INTRODUCTION
Recent work has drawn attention to the fact that models that appear similar from aggregate quality measures, such as accuracy, often have markedly different behavior at the level of individual predictions (Black and Fredrikson] 202T Marx al. 2019). Further, in deep models, this inconsistency can arise even between closely-related models, such as those arising from different initializations, or from leave-one-ous ont differences in the training data (Black and Fredrikson 2021 D' Amour et al.] (2020). This behavior is undesirable in many high-stakes contexts, such as medical applications and credit-approving scenarios, as it may cast. doubt on the justifiability of the model's outcome and pose difficulties for reproducibility and comparison.
We begin by demonstrating that not only are the predictions of related deep models often dissimilar, but their feature attributions (Simonyan et al. 2014; (Sndaarraaann et al.) [017 [eino et al.) 2018)) are as well (Sec- 3). In particular, we show that there is little connection between a model's gradients, which are the basis for many deep attribution methods, and the labels that it predicts-moded with identical predictions can have arbitrarily different gradients almost everywhere (Theorem In practice, we show that this result oc- curs often on common datasets across closely-related models, leading to significant variation in attributions. This may be undesirable, feature attributions commonly used to provide explanations (Simonyan et af. 2014] Sundararaaan et al. 2017 Leino et al. 2018), debug model behavior (Adebayo et al. (020). and diagnose problems related privacy and faimess (Leino and Fredrikson] 2020; Datta al. 2016
Beyond these pragmatic concerns, this suggests that the salient factors behind these models' predictions on many points may have little in common, even when models appear do comparably well on test data.
To address inconsistency in both prediction and attribution, we then turn to ensembling, a well-known approach for reducing predictive variance (Meir et al. [1955 Naftaly al. Lincoln and Skrzypek
Fumerrumera Fumera et al. 22055] Hansen and Salamon 1990, Krogh and Vedelsby 1995). We introduce selective ensembles, which leverage a recent result on multinomial rank verification (Hung et all 20199-whi has
also been used recently for making certifiabll-robuss predictions (Cohen et al. 2019) to efficiently mitigate the problem of inconsistency with a probabilistic guarantee. Given a point to classify, selective ensemble returns the mode of the class labels predicted on that point, where the mode is sampled over models that vary according to a specified source of randomness in the training process. Importantly, if the mode cannot be inferred with sufficient confidence, then the selective ensemble abstains from prediction. This allows us to bound the probability that these ensembles do not return the true mode prediction (Theorem[4.) [4.I. and by extension, the rate of disagreement between selective ensembles (Corollary 4.3). In addition, we show that

---

Published as a conference paper at ICLR 2022
this also bounds the variance component in the ensembles' bias-variance error decomposition (Domingos 2000) (Corollary 4.2). providing guidance on how to effectively use of them in practice.
Our experiments show that seven benchmark datasets, selective ensembles of just ten models either agree on the entire test data across random differences in how their constituent models are trained, or abstain reasonably low rates (1-5% in most cases; Section 5.1p. Additionally, we show that simple ensembling doubles the agreement of attributions on key metrics on average, and when the variance of the constituent models high that selective ensembling further enhances this effect (Section 5.2
In summary, our contributions are: (/) we show that beyond predictions, feature attributions are not consis- tent across seemingly inconsequential random choices during learning (Section 3) (2) we introduce selec- tive ensembling, learning method that guarantees bounded inconsistency in predictions, (Section 4 and (3) we demonstrate the effectiveness of this approach on seven datasets, showing that selective ensembles consistently predict all points across models trained with different random seeds or leave-one-out differences in their training data, while also achieving low abstention rates and higher feature attribution consistency.
2 NOTATION AND PRELIMINARIES
We assume a supervised classification setting, with data points (x.yy E X x Y, drawn from data distribution, D, where x represents vector of features and y response. In order to capture the effects of arbitrary random events on a learned model model-ranging from randomness during training to randomness in the data selection process-
generalize the standard concept of learning rule that of learning pipeline. Specifically, a learning pipeline, PP is a procedure that outputs a model, h: X - Y, taking as input a random state, S~S S, containing all the information necessary for P to produce a model (including the architecture, training set, random coin flips used by the learning rule, etc.). Intuitively, S represents a distribution over random events that might impact the learned model. For example, S might capture randomness in sampling of the training set, or nondeterminism in the optimization process, e.g., the initialization of parameters, the order in which batches are processed, or the effects of dropout.
In our experiments, we model S to capture two specific types of random choices, namely (1) the initial parameters of the model, and (2) leave-one-out changes to the training data. As the initial parameters of the model tend to be determined by random seed, we will interchangeably refer to this as the selection of random seed. More generally, both of these types of choices instantiate a broader class of choices that could be considered arbitrary, despite the fact that they may impact the predictions (Black and Fredrikson| 102T Marx et al. 2019 Mehrer et al. 2020) (Section 5.1) and explanations (Section|5.2 of the resulting model.
3 INSTABILITY OF FEATURE ATTRIBUTIONS IN DEEP MODELS
Before we consider mitigating predictive inconsistency with ensembling, we first demonstrate that models' inconsistency across random choices in training exhibited not only through its predictions, but through its feature attributions as well. Feature attributions refer to numeric scores generated for some set of a model's features most commonly the model's input features which which are meant to connote how important each feature is in generating the model's prediction. Feature attributions are commonly used as a tool for explaining model behavior (Simonyan et al.) 2014 [eino et al., [2018] (Sundararajan al.) |017]] AAdebayo al. 2020) localized to given set of inputs. Thus, inconsistent feature attributions between models suggest the models differ the process by which they arrive at their predictions, even if the predictions are the same.
In deep models, many of the most popular attribution methods are based on the model's gradients at or around a given point (Simonyan et al. 2014 Sundararajan et al. (017). Accordingly, we will focus on the stability of gradients, and show via analysis and experiment that they are not stable in conventional deep models. First, we motivate our results by showing that even two deep models that predict the same labels on all points may have arbitrarily different gradients almost everywhere. Later, in our empirical evaluation (Section 5.2).. we demonstrate the extent of the differences between Saliency Maps (Simonyan) al. 2014) (i.e., input gradients) of deep networks even when the randomness of the learning pipeline is controlled to allow only one-point differences in the training set or differences in the random seed.
Predictions with Arbitrary Gradients. We show that even deep models that predict the exact same labels on all points cannot necessarily be expected have the same, or even similar, gradients; in fact, given binary classification model h, we can construct model ls which predicts the same labels as h,
2

---

Published as a conference paper at ICLR 2022
Hisjo1 Mer
Brx)mete) - mupiol-
Vou
WMMA
(igtwas)) india
Hrajoo
Reglon
Bigton
Figure 1: Intuitive illustration of how two models which predict identical classification labels can have arbitrary gradients. To show this, given a binary classifier H and an arbitrary function g. we construct classifier H' that predicts the same labels H, yet has gradients equal to g almost everywhere. We formally state this result in Theorem 3.1
but has arbitrarily different gradients everywhere except an arbitrarily small region around the boundary h (Theorem 3.1.
Theorem 3.1. Let H X x (-1.)
g: be binary classifier and 9: BR" + R. be an unrelated
function that bounded from above and below, contimuous, and piecewise differentiable. Then there exists another binary classifier H sign((h) such that for any c>0,
VXEX.
1. (<)
inf e'H((''+HH())
[1le- >//2//22-- = Vh(x)=V9(4)
The proof of Theorem 3.1 is given in Appendix A.I in the supplementary material. The proof is by construction of h: a sketch giving the intuition behind the construction is provided in Figure | In short, we first partition the domain into contiguous regions that are given the same label by H. We then construct from g by adjusting to lie above or below the origin to match the prediction behavior region. As these transformations merely shift g by constant in each region, they do not change Vy except near decision boundaries, where it is necessary to move across the origin.
each
Observations. The intuition stemming from Theorem 3.1 is that model's gradients at each point largely disconnected from the labels it predicts on distribution. As models that make identical predictions likely have similar loss given dataset, this theorem points the possibility that models of similar objective quality may still have arbitrarily different gradients. In Section [.22] we demonstrate that this outcome
only possible, but that it occurs in real models-foo example, on the German Credit dataset predicting credit risk, on average, individual models with similar accuracy agree on less than two out of the five most important features influencing their decision.
SELECTIVE ENSEMBLING
The results of Section suggests that models that retrained and redeployed, may exhibit substantially different behavior from their previous iterations. We build on the approach of ensembling for variance reduction by showing how these differences in behavior can be bounded via selective ensembling. However, whereas prior work which finds that more diversity among the constituent networks is beneficial for reducing overall error (Krogh and Vedelshy 1995 Hansen and Salamon 1990 Maclin al. 1995] Opitz and Shavlik 1996), our goal is to minimize, or al least place strict bounds on, the variance component. We show that ideas from robust classification, and in particular randomized smoothing (Cohen et al. 2019). which stem from recent results on multinomial hypothesis testing (Hung et al. 2019). can be used to enforce such a bound.
Mode Predictor. We may view the image of the learning pipeline, P, as distribution over possible mod- els induced by applying P to the random state, S~SS. The mode prediction on an input x, with respect to S, the expected label that would be predicted on by models drawn from this distribution. More formally, we define the mode predictor, 9p.s for a pipeline, P, and random state distribution, S, as given by Equation]
gP gP,s(z) =argmax
E S
1|P(S:))
yEY

---

Published as a conference paper at ICLR 2022
Algorithm 1: Selective Ensemble Creation
A Algorithm 2: Selective Ensemble Prediction
def ensemble predict (g. (P.S), a, x)
def train_ensembler (P, S~S"; n): return (P(S.) for iE[r]]
Y < Eneg.. one hot (h(())
RA, fi < top 2oY_2(YY if wwwooooonvaalllll p_ value (NA, nA + RB, Sa then
def sample_ensentn (P, S, n):
return YY)
S + sample_iid(S")
return wwwaitrraeeeeeeeeeeeee ensemble(e n))
else
return ABSTAIN
Note that while 9p.s is deterministic, and is therefore not sensitive to a specific state drawn from S, it does not necessarily produce the ground truth label on all inputs some learning pipelines may converge to stable loss minimum that misclassifies certain points.
explicit representation of the true mode predictor is, of course, unattainable-the non-convex loss surface of deep models and the complex interactions between the learning pipeline and the distribution of random states makes the expectation in Equation infeasible innfeasbbl to compute analytically. However, we approximate gp,s s1))(x) ( by computing the empirical mode prediction on z over a random sample of models produced by i.i.d. draws from P(S). But although ensembles with sufficiently many constituent models will more reliably output the mode prediction, for any fixed-size ensemble there will remain points on which the margin of the plurality vote is small enough to "flip" to runner-up in some set of nearby ensembles that differ on subset of their constituent;; in other words, these ensembles will not predict the mode prediction.
Approximation via Ensembling.
To rigorously bound the rate at which the ensemble will differ from the mode prediction, we allow the ensemble to abstain on points where the constituent predictions indicate a statistical toss-up between the two most likely classes. We call ensembles that may abstain in this way selective ensembles, borrowing the terminology from selective classification (EI- Yaniv et al. (010). We can think of of abstention as means of flagging unstable points on which the selective ensemble cannot accurately determine the mode prediction; whether this should be interpreted as a failed attempt at classification is an application-specific consideration.
Selective ensembles of n models predict according to the following procedure. First, the predictions of each of the models in the ensemble are collected. The constiteett models are derived from i.i.d. samples of P(S)) from S, as described in Algorithm [ From these predictions, we perform a two-sided statistical test to determine if the mode prediction was selected by statistically significant majority of
the constituent models. If the statistical test succeeds, we return the empirical mode prediction; otherwise abstain from predicting. Pseudocode for this prediction procedure is given in Algorithm 2 We will denote by 9n (P,S) (for S~ the output of train_ eeseennseele e in Algorithm 1 and by 9n (P.S a.²)
prediction produced by ensemble limibllppre peeeeddedeooo et in Algorithm 2 on 9r. (P,S).
Because of their ability to abstain from prediction, we can prove that with probability at least o, a selective ensemble will either return the true mode prediction abstain, where is chosen threshold for the statistical test to prevent prediction in the case of a toss-up. In other words, on any point on which it does not abstain, a selective ensemble will disagree with the mode predictor, gp,s, with probability at most as stated formally in Theorem
ang The statement of Theorem 4.1 make use of the relation, 7, where y1 7 y2 if and only if yı * ABSTAIN and 3/2 & ABSTAIN and yı Fy2. That is, ing captures disagreement between non- non-rejected predictions.
Theorem 4.1. Let P be learning pipeline, and let S be distribution over random states. Further, let gp.s . be the mode predictor, let (P,(P -.r S~ be a selective ensemble, and let a 200. Then,
VrEX
Pr Smng
In (P,S axx) 7 gp s(x) Sax
The proof (Appendix A) relies result from Hung and Fithian (Hung et al. 2019) which bounds the probability that a set of votes does not return the true plurality outcome, and we apply it in a similar fashion to how is used for making robust predictions in Randomized Smoothing (Cohen al. (011)).
Theorem 4.1 states that the probability that a selective ensemble makes prediction that does not match the mode prediction is small. However, one possible means of ensuring this is by providing prediction
4

---

Published as conference paper at ICLR 2022
8= 0.005
0.05
a=0.005
oF 0.05
1.0
abvontton (0)
0.5
100
1000
0.0
0.6
0.5
1.0 0.6
1.0 0.6
0.81
1.0
0.6
0.5
1.0
probuliiity of consiinent modil agrecment (g)
Figure 2: The left two plots show abstention rates a function of the underlying probability of agreement among models over S, ie.. the probability that any given model will return the mode prediction, with plots denoting varying numbers of constituent models. The right two graphs demonstrate the relationship between consistency the ensemble models as given by Corollary 4.3
in the first place, i.., if the selective ensemble abstains. Thus, the abstention rate is necessary to quantify the fraction of points on which the mode prediction will actually be produced.
In the 0-1 loss Dias-variance decomposition of Domingos (2000), the variance component of a classifier's loss is defined as the expected loss relative to the mode prediction (in our case, taken over the randomness in S). Thus, Theorem 4.1 leads to a direct bound this component, assuming a bound, B3, on the abstention rate. This is formalized in Corollary 4.2)
Corollary 4.2 Let P a learning pipeline, and be a distribution over random states, Further, let JP.s be the mode predictor, let In (P,S) for S S" be selective ensemble. Finally, let or 2 O, and let 2 an upper bound on the expected abstention rate of 9n Bn(P,)) Then, the expected loss variance, (zz,, over inputs, is bounded by a That is,
E
V(z)
=
Pr S
x)
<a+3
also address the original problem raised: that deep models often disagree on their predictions due to arbitrary random events the training pipeline. We show that, given bound, B, the abstention rate, the probability that two selective ensembles disagree in their predictions is bounded by 3( (Corollary 4.3). Intuitively, this suggests that the predictions of selective ensembles are more stable over different instantiations of the random decisions captured by S compared to individual models.
Consistency of Selective Ensembles. Using the result from Theorem
Corollary 4.3, Let P be a learning pipeline, and let be a distribution over random states. Further, let (P.(P))) for S~ be selective ensemble. Finally, let a > O, and let 3 2 20 be upper bound the expected abstention rate of In (PP (S). Then,
E
( (PSS a St. Sm
Pr
<2(a+
a>D
Corollary 4.3 tells that the agreement between any two selective ensembles is at least 1 2((x (). For a fixed ni, decreasing a will lead to higher abstention rate. Thus in order for a and / to both be small, as would be necessary for a high fraction of cansistently-prediiee number of constituent models, n. Figure 2 illustrates the trade-off between a, 3, and n, depending on the base level of agreement of the constituent models. In Section 5 show empirically that even with small values of n, abstention rates of selective ensembles are reasonably low in practice.
points, may require large
In summary, selective ensembles accomplish three primary things: (1) they identify points which the mode prediction cannot be determined, (2) they bound the fraction of points that can be inconsistently predicted, and (3) they provide a means of reliably inferring the mode prediction when the abstention rate can be kept sufficiently low.
5 EVALUATION
In this section, we demonstrate empirically that selective ensembles reduce instability in deep model predictions far below their theoretical bounds zero inconsistent predictions in the test set over 276

---

Published as a conference paper at ICLR 2022
meam accuracy + standard deviation
Randomness
Ger. Credit
Adult
Scizure
Warfarin
Tai. Credit
FMNIST
Colon
RS
.300 020
14 3
27 3
6861 3
1e 3
916133 3
927 33
LOO
7291.001
84347e-4
296422-
28
120
917
32643c-3
Table 1: Mean accuracy over 500 models trained over changes random initialization and leave-one-out differences in training data. German Credit stands as an outlier due to its small sample size (D) -80)..
mean of portion of test data with Paup >
Randomness
73
Ger. Credit
Adult
Scizure
Tai. Credit
Warfarin
FMNIST
Colon
RS
.570
.087
.060
.822
.098
.061
.037
RS
(5, 10, 15, 20)
0.0
0.0
0.0
0.0
0.0
0.0
0.0
LOO
.262
.063
.031
.031
.033
.084
.042
LOO
(5, 10, 15, 20)
0.0
0.0
0.0
0.0
0.0
0.0
0.0
Table 2: Percentage of points with disagreement between at least one pair of models (Pmip trained with different random seeds (RS) or leave-one-out differences (LOO) in training data, for single models (n=1) 1) and selective ensembles (n.>> 1). Results averaged over 10 runs of creating 24 selective ensemble models, standard deviations are in Appendix Selective ensemble results are together, as there is no disagreement.
accuracy
(abstain as emor) / abstention rote
S
Ger. Credit
Adult
Seizure
Warafin
Tai. Credit
FMNIST
Colon
RS
0.0/1.0
0.0/1.0
0.0/1.0
0.0/1.0
0.0/1.0
0.0/1.0
0.0/1.0
RS
10
576//201
820/ 043
960/ 002
660 .55
800/ 033
888/ 00.9
914/.032
RS
15
6366 205
827 033
9655 .018
008/0037
S0T 807/1028
897/1042
019//023
RS
20
664//115
830/ .044
.67/
6670/031
810/ .023
902/1036
.21//019
LOO
0.0/1.0
0.0/1.0
0.0/1.0
0.0/1.0
0.0/1.0
0.0/1.0
0.0/1.0
LOO
10
653/1151
827/ 0032
962/ 022
677//018
$12/.017
909/102
0112/006
LOO
15
6678/105
832/
.68//000 019
.679/0013
814//013
910/.016
916//027
LOO
20
.689/007 .079
834/0018 018
.970/001 0015
.680/0011
815/ .110
.12//012
.19//022 /.02
Table 3: Accuracy and abstention rate selective ensembles, with n E (5,10,15,20 constituents. Results are averaged over 24 randomly selected models; standard deviations are given in Table 8 Appendix
Iadbidaa with P4.
German Credit
FMNIST
Taiwancsc
German Credit
FMNIST
Taiwanese
375
025
Figure 3: Figure a Percentage of test data with non-zero disagreement rate in normal (i.c., not selective) ensembles. Horizontal axis depicts ensemble size. Figure bb) Average Spearman's 's Ranking coefficient, (For FMNIST, SSIM) between feature attributions for saliency maps generated for each individual test point (y-axis) over number ensemble models (x-axis). The lines indicated with (Sel) in the legend are the same metrics for selective ensembles.
pairwise comparisons of model predictions for each of tabular datasets, and 40 for image datasets. Additionally, following Theorem 3.1] we show that feature attributions of individual deep models are frequently inconsistent, and that ensembling effectively mitigates this problem.
Setup. To evaluate selective ensembling, we focus on two sources of randomness in the learning rule: (1) random initialization, and (2) leave-one-out changes to the training set. Our experiments consider seven datasets: UCI German Credit, Adult, Taiwanese Credit Default, Seizure, all from Dua and Karra Taniskidou (0077); the IWPC Warfarin Dosing Recommendation (International Warfarin Pharmacogenetic Consortium
2009, Fashion MNIST (Xiao et al. 2017). and Colorectal Histology (Kather et al. (016a)) All of these
datasets are either related to finance, credit approval, or medical diagnosis, except for FMNIST, which include it is common benchmark for image classification. Further details are in Appendix B

---

Published as a conference paper at ICLR 2022
Figure 4: Inconsistency of attributions on the same point across an individual (left) and ensembled (right) model (n= 15). The height of each bar on the horizontal axis represents the attribution score of a distinct feature, and each color represents different model. Features are ordered according to the attribution scores of one randomly-selecten model.
All experiments are implemented in TensorFlow 2.3. For each tabular, we train 500 models from independent samples of the relevant source of randomness (c.g. leave-one-out data variations or random seeds), and for each image dataset, we train 200 models from independent samples of each source of randomness. Details about the model architecture and hyperparameters used are given in Appendix C Table reports the mean accuracy for each dataset, along with the standard deviation.
For each non-image dataset we generate 24 random ensembles of size n E (5,10,15,20) by selecting uniformly without replacement among the 500 pre-trained models, as well 24 "singleton" models drawn uniformly from the 500 to use as point of comparison when measuring the stability of each ensemble. For image datasets, we generate random ensembles each size among 200 pre-trained models. We report ensemble predictions in the main paper using 2=0.05.
5.1 SELECTIVE ENSEMBLES: PREDICTION STABILITY AND ACCURACY
To measure prediction instability over either selective ensembles or singleton models, we compare the predictions of each pair of models on each point in the test set, amounting to 276 comparisons for tabular datasets, and 40 comparisons for image datasets, in total for each point, and record the rate of disagreement, Pfip- across these comparisons. We report mean and variance of this disagreement over 10 random re-samplings of constituent models to create ensemble models.
The results in Table and Figure 3a show the percentage of points with disagreement rate greater than zero We see that for singleton models, as many as 57% test points have Ptip > 0, indicating that disagreement in prediction is in some cases the norm rather than the exception, although more commonly this occurs on 500% of the data. Notably, selective ensembles completely mitigate this effect: even when as few as models are included in the ensemble, points experienced Phip O. Combined with the fact that abstention rates remain low (1-5% in all cases except where Phip was originally very high (c.g.. German Credit), this shows that selective ensembling can be a practical method for mitigating prediction instability,
Table shows the accuracy of selective ensembles, with abstention counted towards error, as well as accuracy of non-selective ensembles for comparison. Notably, all six models, with the exception of German Credit, the abstention rate drops to below 4% with 20 models in the ensemble. Accordingly, the accuracy the selective ensembles in these cases is oomparable-typicallyy within a few points- to that of the traditional ensemble. However, with just five models in the ensemble, the abstention rate is 100%; to achieve reasonable predictions with very few models, the threshold a needs be increased accordingly. Disagreement of non-selective ensembles are pictured in Figure Bal (with exact numbers in Appendix F): while they do lower prediction inconsistency, they are unable to eliminate as selective ensembles do.
5.2 ATTRIBUTION STABILITY
we demonstrate that feature attributions, which are usually computed for deep models using gradients (Simonyan et al.. 2014 (Sndaarraaaan et al. 2017] Leino al. 2018), are often inconsistent between similar models. We then show that, just as ensembling increases prediction stability, it also mitigates gradient instability, leading to more consistent attributions across models. For these experiments, we computed attributions using saliency maps (Simonyan et al.) 2014), which are simply the gradient of the model's 's prediction with respect to its input, as a simple and widely-used representative of gradient-based attribution methods.
Following up the theoretical result given in Theorem 3.1

---

Published as a conference paper at ICLR 2022
Random Seed
Leave-one-out
Dataset
Top-5
p
r
SSIM
Top-5
SSIM
German Credit
0,20, .27
0.01, 0.25
0.02, 0.28
0.20, 0.49
0.01, 0.59
0.02, 0.60
Adult
0.46, 0.83
0.09, 0.83
0.07, 0.93
0.46, 0.89
0.15,0.91
0.14,0.95
Seizure
0.14, 0.12
0.29, 0.32
0.30, 0.33
0.09, 0.25
0.23, 0.37
0.24, 0.59
Warfarin
0.37, 0.67
0.15, 0.72
0.12, 0.73
0.36, 0.92
0.12, 0.96
0.11.
Taiwanese Credit
0.55, 0.76
0.35, 0.75
0.36,0.83
0.56,0.91
0.35,0.95
0.37,0.96
FMNIST
0.00, 0.26
0.61, 0.61
0.50, 0.25
0.00, 0.57
0.90, 0.89
0.78, 0.62
Colon
0.00, 0.63
0.00, 0.92
0.18, 0.82
0.00, 0.61
0.00, 0.91
0.18,0881
Table 4: Average top-5 intersection, Spearman's 's Rank Correlation Coefficient (p), and Pearson's s Correlation Coefficient (r) to demonstrate attribution inconsistency on the same test points across different models. As baseline, compare against differences observed on different points in the same model. The baseline numbers are presented as: similarity baseline, similarity across models. For image models, we also report the Structural Similarity Index (SSIM). Standard deviations are included in Appendix H.2
Metrics. Following previous work (Dombrowski et al. 2019 Ghorbani et al. 2019), we measure the similarity between attributions using Spearman's 's Ranking Correlation (p) and the top-k intersection, with 5. For image datasets, we also display the Structural similarity metric (SSIM), discussed further in Ap- pendix [..]] Spearman's 's p is natural choice of metric attributions induce order of importance among features. We note that the top-k intersection is especially interesting in tabular datasets, as often only the most important features are of explanatory interest. To stay consistent with prior work, we also include Pearson's Correlation Coefficient (r)) Note that r and vary from 1 to I, denoting negative, zero, and positive correla- tion. We compute these metries over 276 pairwise comparisons of attributions for each size of ensemble (1, 5, 10, 15, and 20) for tabular datasets, and 40 pairwise comparisons for image datasets. For the top-k metric. we report the mean size of the intersection between each pair of attributions. More details are in Appendix [
Baselines. To contextualize the difference of attributions across models trained from distinct randomness, we also include the attribution similarity between 24 randomly chosen points in the same model (Table 4 We also present visual comparison of model attributions, for which we simply plot the attribution for the predicted class for a given point from nine randomly selected models out of the 24, and present the feature attributions in order of their magnitude according another randomly selected model (Figure 4.)
Singleton Models The left image in Figure demonstrates the inconsistency of model attributions of individual German Credit models on a random point in the test set. Each bar on the x-axis represents the at- tributions for feature, and each different-colore, bar represents a different randomly selected model. Thus, the disagreement between the sizes of the bars of different colors shows the disagreement between models on which features should be deemed important. Notably, some of the bars on the graph depicting individual models even have different signs, which means that models disagree on whether that feature counts towards or against the same prediction. Similar graphs for all other datasets are included in Appendix H.T
We demonstrate this inconsistency further in Table We see that German Credit and Seizure models have particularly unstable attributions, the top-k (and to lesser extent, p and r) scores of attributions of varying points in both the same model, and varying models on the same point, are quite similar. Feature attributions of individual models are inconsistent even highly weighted features: e.g., German Credit dataset has a top-k intersection of just over one attribute on average gegrage-Sugggetnngg that attributions generated through saliency maps on these sets of models may vary substantially over benign retrainings. Even on models where the metrics are higher, e.g. Taiwanese Credit, the baseline similarity between attributions higher as well-thus. we see that attributions between models of the same point are usually only 2-3x more related than those of random points within the same model.
This instability suggests that salient variables used to inform predictions across models are sensitive to ran- dom choices made during training. As previous work has argued in similar contexts D'Amour et al. (020)) this may be a result of a deep model's 's mnder-constrained search space with many local optima equivalent with respect to loss, with several minima corresponding to distinct rationales for making predictions.
Ensemble Models. We demonstrate that the similarity between saliency maps of ensembled models is greater than that of individual models, and that this similarity increases linearly with the number of models in the ensemble. For these experiments, we average each model's attributions toward the majority predicted class of the ensemble. On the right side of Figure 4 we see the feature attributions of various
8

---

Published as a conference paper at ICLR 2022
ensemble models of size 15 over the German Credit dataset. Note how the attributions of ensemble models are much more consistent than on the individual model.
We show this phenomenon more broadly in Figure
3b) where we display graphs of average Spearman's 's Rank Coefficient (p) (y-axis) between saliency maps on point in the test set. We see p increase as we increase the number of models in the ensemble (x-axis), for models generated over different random initializations and one-point differences in the training set. Selective ensembles can further increase stability of explanations by abstaining from unstable points, and this has marked effect when the abstention rate is high (e.g. German Credit). Similar graphs for the rest of metrics calculated are presented in Appendix H
6 RELATED WORK
Prior work has shown that deep models are inconsistent in their predictions across arbitrary random changes in their training pipeline, such as initialization parameters and makeup of the training set (Black)
and Fredrikson 202T MMhrer et al. 2020 D D'mourr et al.) 2020, Kolen and Pollack| 1991| Feldman 2019).
The problem of model sensitivity, particularly to variability in the training set, can lead to an increase
generalization error (Elisseeff et al. 2003) as well as to leaking training set information (Dwork 2006
Yeom et al. 2018). Thus, stability-enhancci learning rules have received significant attention in order to bolster desirable properties, such as privacy (Liu et al. 2020 Paperno et al. 2018 Wang et al. 2016).
One such approach is model ensembling, which has been used as variance reduction method since the advent of statistical learning (Zhou et al. [2002] Valentini et al. [2044] Opitz and Maclin [999] Tumeer and
Ghosh 1996
Dvomik et al. 2019 Hasan al. 2020 Freund and Schapire| 1997| (Sagi and Rokach| 20188
Polikar 2012 Che et al. 20 Perrone and Cooper 1992] Hansen and Salamon 1990). However, to our
knowledge, there is little work on providing guarantees about model disagreement using ensemble models
that may abstain from prediction. We relate our approach to the classic bias-variance decomposition of error (Domingos 2000), showing that it certifiably bounds the variance component.
Selective ensembles can be seen as a way to flag points that prone to inconsistency. Under this view, calibration and uncertainty estimation of deep model predictions (Lakshminarayanan al., 2016 Ovadia et al. (219) is a related stream of work, and one could potentially use these techniques to determine when to abstain from prediction. However, preventing inconsistent predictions and abstaining from uncertain predictions are different goals: in our setting, the aim is to predict the mode across models drawn from a certain distribution, whereas calibration is measured against predicting the true label. Moreover, prior work has shown that confidence scores may not be correlated with prediction consistency across models with different random initializations (Black and Fredrikson 202T). Finally, while abstaining on points with low confidence scores may lead to greater consistency, it may not yield guarantee, which this work provides.
Conformal inference (Linusson et al. 2020; Gupta et al. 2019 Löfström et al.] 2013). which rigorously as- signs confidence to predictions in settings where the data may differ from traimin, similarly related in that such a measure could be useful in identifying inconsistently predicted points. However, in this work, we aim to achieve consistent predictions across known distribution of models, as prior work, well suggest, even points conforming to past observations may still be predicted differently by different models.
results,
In addition to inconsistent predictions, this work demonstrates how feature attributions can differ substantially between individual deep models with inconsequential differences. Prior works investigating instability of gradient-basee
explanation techniques focus on an adversarial context (Dombrowski et al. 2019 Ghorbani et al. 2019 [Hoo et al. [2019 Wang et al. (2020). For example, (Anders et al (2020) develop attacks to create similar models that have differing gradient-based explanations. Contrastingly, this work focuses on the instability of counterfactual explanations between similar models that may occur naturally. As we demonstrate in Section 5.2 model gradients can be quite dissimilar without any adversary.
CONCLUSION
We show that similar deep models can have not only inconsistent predictions, but substantially different gradients as well. We introduce selective ensembles to mitigate this problem by bounding a model's inconsistency over random choices made during training. Empirically, we show that selective ensembles predict all points consistently over all datasets we studied. Selective ensembling may present a more reliable way of using deep models in settings where high model complexity and stability are required.

---

Published as a conference paper at ICLR 2022
ACKNOWLEDGMENTS
This work was developed with the support NSF grant CNS-1704845, NSF CNS- 1943016,, as well as by DARPA and the Air Force Research Laboratory under agreement number FA8750-15-2-0277. The U.S. Government is authorized to reproduce and distribute reprints for Govermmental purposes not with- standing any copyright notation thereon. The views, opinions, and/or findings expressed are those of the author(s) and should not be interpreted as representing the official views or policies of DARPA, the Air Force Research Laboratory, the National Science Foundation, the U.S. Govemment.
ETHICS STATEMENT
This work is motivated by the problem of model inconsistency over time in deployment settings- particularly settings impacting individuals' lives, where inconsistency may lead to confusion or even harm to users. The aim of this paper is to prevent harm to those impacted my model decisions occurring from inconsistent model outcomes. We note that in some high-stakes contexts, is possible that not supplying any outcome (i.e. abstaining) may be worse for an individual than an inconsistent outcome, and so we expect that selective ensembles will be used with a numan-in-tte-loow or other decision makinn framework to adjudicate over abstained-upon points during deployment.
Additionally, recent work has suggested that selective classification can amplify performance disparities between demographic groups (Jones et al. 2020). We investigate the extent of this behavior in selective ensembles and found that by and large, using selective ensembles does not exacerbate accuracy disparity by very much (within 1% of the original disparity). although they did not ameliorate disparities in accuracy that already existed within the performance of the algorithm. The results of these experiments are in Appendix
We note that the promise of stability from this paper may encourage machine learning practitioners to over-use highly complex models where a simpler model may be a better choice due to, e.g., transparency requirements. However, we hope that the prospect of increased stability that this paper introduces reduces the harm that can come from machine learning deployment.
REFERENCES
Structural similarity index.
transform: http:ploot__ss_eee wws.sssmm.. 1::....h00 stt_sssimm.mll
Julius Adebayo, Michael Muelly, Ilaria Liccardi, and Been Kim. Debugging tests for model explanations. In H. Larochelle, M. Ranzato, R. Hadsell, M. F. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems, volume 33, pages 700-712, Curran Asso- ciates, Inc., 2020. URL https: https::/prroeeeiiggs www.neu..uc..... httppppp..c//appr222///L.. 2abb0llec3d22daccco333 (as57788daa6flfd Paper. pdf
Christopher Anders, Plamen Pasliev, Ann-Kathrin Dombrowski, Klaus-Robert Müller, and Pan Kessel. Fairwashing explanations with off-manifold detergent. In International Conference Machine Learning, pages 314-323, PMLR, 2020.
Emily Black and Matt Fredrikson. Leave-one-out unfaimess. In Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency, pages 285-295, 2021.
Dongsheng Che, Qi Liu, Khaled Rasheed, and Xiuping Tao. Decision tree and ensemble learning algorithms with their applications in bioinformatics. Software tools and algoritluns for biological systems, pages 191-199, 2011.
Jeremy Cohen, Elan Rosenfeld, and Zico Kolter. Certified adversarial robustness via randomized smoothing. In Proceedings of the 36th International Conference on Machine Learning (ICML), 2019.
Alexander D' Amouu, Katherine Heller, Dan Moldovan, Ben Adlam, Babak Alipanahi, Alex Beutel, Christina Chen, Jonathan Deaton, Jacob Eisenstein, Matthew D) Hoffman, et al. Underspecification presents challenges for credibility in modern machine learning. arXiv preprint arXiv:201 1,03399, 2020.
Anupam Datta, Shayak Sen, and Yair Zick. Algorithmic transparency via quantitative input influence: Theory and experiments with learning systems. In IEEE Symposium on Security and Privacy, pages 598-617, 2016.
10

---

Published as a conference paper at ICLR 2022
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Fi Imagenet: A large-scale hierarchical image database. 2009 IEEE conference on computer vision and pattern recognition, pages 248-255.
Icec, 2009.
Ann-Kathrin Dombrowski, Maximilian Alber, Christopher Anders, Marcel Ackermann, Klaus-Robert Müller, and Pan Kessel. Explanations can be manipulated and geometry is to blame. arXiv preprint urXii:/906.. 07993, 2019.
Pedro Domingos. unified bias-variance decomposition. In Proceedings of 17th Interattonal Conference on Machine Learning, pages 231-238, 2000.
Dheeru Dua and Efi Karra Taniskidou. UCI machine learning repository. https:/ive.ics.uci..dei 2017.
Nikita Dvorikk Cordelia Schmid, and Julien Mairal. Diversity with cooperation: Ensemble methods for few-shot classification. In Proceedings of the IEEE/CVF International Conference Computer Vision, pages 3723-3731, 2019.
Cynthia Dwork.
Differential privacy, 2006.
Ran El-Yaniv al. On the foundations of noise-free selective classification. Journal of Machine Learning Research, 11(5), 2010.
André Elisseeff, Massimiliano Pontil, et al. Leave-one-out error and stability of learning algorithms with applications. NATO science series sub series ii computer and systems sciences, 2003.
Vitaly Feldman. Does learning require memorization? A short tale about long tail. CoRR, 1b6/1906. 2019.
Yoav Freund and Robert E Schapire. A decision-theoretic generalization on-line learning and an application to boosting. Journal of computer and system sciences, 55(1):119-13) 1997.
Giorgio Fumera, Fabio Roli, and Alessandra Serrau. Dynamics of variance reduction in bagging and other techniques based on randomisation. In International Workshop on Multiple Classifier Systems, 2005.
Amirata Ghorbani, Abubakar Abid, and James Zou. Interpretation of neural networks is fragile. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pages 3681-3688, 2019.
Chirag Gupta, Arun K Kuchibhotla, and Aaditya K Ramdas. Nested conformal prediction and quantile out-of-bag ensemble methods. arXiv preprint arXiv:/910 10562, 2019.
Lars Kai Hansen and Peter Salamon. Neural network ensembles. IEEE Iransactions on pallem analysis and machine intelligence, 12(10)9933-1001, 1990.
Md Kamrul Hasan, Ashraful Alam, Dola Das, Eklas Hossain, and Mahmudul Hasan. Diabetes pre- diction using ensembling of different machine learning classifiers. IEEE Access, 8:76516-76531, 2020.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770-778, 2016.
Juyeon Hco, Sunghwan Joo, and Taesup Moon. Fooling neural network interpretations via adversarial model manipulation. Advances in Neural Information Processing Systems, 32:2925-2936. 2019.
Kenneth Hung, William Fithian, et al. Rank verification for exponential families. Annals of Statistics, 47(2):758-782, 2019.
International Warfarin Pharmacogenetic Consortium. Estimation of the warfarin dose with clinical and pharmacogenetic data. New England Journal of Medicine, 360(8)7553-76,, 2009.
Erik Jones, Shiori Sagawa, Pang Wei Koh, Ananya Kumar, and Percy Liang. Selective classification can magnify disparities across groups. arXiv preprint arXiv:2010.1 144444400..4144 2020.
Jakob Nikolas Kather, Cleo-Aron Weis. Francesco Bianconi, Susanne M Melchers, Lothar R Schad, Timo Gaiser, Alexander Marx, and Frank Gerrtt ololener. Multi-class texture analysis in colorectal cancer histology. Scientific reports, 6:27988, 2016a.

---

Published as a conference paper at ICLR 2022
Jakob Nikolas Kather, Frank Gerrit Zöllner, Francesco Bianconi, Susanne M Melchers, Lothar R Schad, Timo Gaiser, Alexander Marx, and Cleo-Aron Weis. Collection of textures in colorectal cancer histology, May 2016b. URL Ballhttp:: /ttpttp:://eedd...oooo zenodo. crg/ wttp://ort//o./.ee..1.
YLkg2waKiqD
John Kolen and Jordan Pollack. Back propagation is sensitive to initial conditions. In Neural Information Processing Systems (NeurPPS), 1991.
Anders Krogh and Jesper Vedelsby. Neural network ensembles, cross validation, and active learning. Neural Information Processing Systems (NeurIPS), 1995.
Balaji Lakshminarayanan, Alexander Pritzel, and Charles Blundell. Simple and scalable predictive uncertainty estimation using deep ensembles. arXiv preprint drXh::161.00147,,200 2016.
Yann LeCun, LD Jackel, Léon Bottou, Corinna Cortes, John S Denker, Harris Drucker, Isabelle Guyon, Urs Muller, Eduard Sackinger, Patrice Simard, et Learning algorithms for classification: A comparison on handwritten digit recognition. Neural networks: the statistical mechanics perspective, 261:276, 1995.
Klas Leino and Matt Fredrikson. Stolen memories: Leveraging model memorization for calibrated white-box membership inference. 2020.
Klas Leino, Shayak Sen, Anupam Datta, Matt Fredrikson, and Linyi Li. Influence-directed explanations for deep convolutional networks. In IEEE International Test Conference (ITC), 2018.
William Lincoln and Josef Skrzypek. Synergy of clustering multiple back propagation networks. In Neural Information Processing Systems (NewrIPS), 1990.
Henrik Linusson, Ulf Johansson, and Henrik Boström. Efficient conformal predictor ensembles. Neurocomputing, 397:266-278, 2020.
Hongbin Liu, Jinyuan Jia, and Neil Zhenqiang Gong. On the intrinsic differential privacy of bagging. arXiv preprint arXiv:2008.098.0 2020.
Tuve Löfström, Ulf Johansson, and Henrik Boström. Effective utilization of data in inductive conformal prediction using ensembles of neural networks. In The 2013 International Joint Conference Neural Networks (JCNN), pages 1-8. IEEE, 2013.
Richard Maclin, Jude W Shavlik, al. Combining the predictions of multiple classifiers: Using competitive learning to initialize neural networks. In International Joint Conference on Artificial Intelligence (ICCAI), 1995.
Charles T. Marx, Flávio du Pin Calmon, and Berk Ustun. Predictive multiplicity in classification. CoRR, abs/1909.066l 096777 2019. URL http:://arrii..or//abb//19.. 06677
Johannes Mehrer, Courtney J Spoerer, Nikolaus Kriegeskorte, and Tim C Kietzmann. Individual differences among deep neural network models. Nature communications, 11(1):1-12, 2020.
Ronny Meir, G Tesauro, DS Touretzky, and TK Leen. Bias, variance and the combination of least squares estimators. Advances in neural information processing systems, pages 295-302, 1995.
Malik Sajjad Ahmed Nadeem, Jean- Cean-Daniel Zucker, and Blaise Hanczar. Accuracy-rejection curves (ares) for comparing classification methods with a reject option. Machine Learning in Systems Biology, pages 65-81. PMLR, 2009.
Ury Naftaly, Nathan Intrator, and David Horn. Optimal ensemble averaging of neural networks. Network: Computation in Neural Systems, 8(3):283-296, 1997.
David Opitz and Richard Maclin. Popular ensemble methods: An empirical study. Journal of artificial intelligence research, 11:169-198, 1999.
David Opitz and Jude Shavlik. Generating accurate and diverse members of a netwol- ensemble. In Advances in Neural Information Processing Systems, 1996.
12

---

Published as a conference paper at ICLR 2022
Yaniv Ovadia, Emily Fertig, Jie Ren, Zachary Nado, David Sculley, Sebastian Nowozin, Joshua V Dillon, Balaji Lakshminarayanan, and Jasper Snock. Can you trust your model's uncertainty? evaluating predictive uncertainty under dataset shift. arXiv preprint arXiv:190.002530, 2019.
Nicolas Papernot, Shuang Song, Ilya Mironov, Ananth Raghunathan, Kunal Talwar, and Ulfar Erlingsson. Scalable private learning with pate. In International Conference on Learning Representations, 2018.
Michael P Perrone and Leon N Cooper. When networks disagree: Ensemble methods for hybrid neural networks. Technical report, BROWN UNIV PROVIDENCE RI INST FOR BRAIN AND NEURAL SYSTEMS, 1992.
Robi Polikar.
Ensemble leaming. In Ensemble machine learning, pages 1-34. Springer, 2012.
Omer Sagi and Lior Rokach. Ensemble learning: survey. Wiley Interdisciplinary Reviews: Data Mining and Knowledge Discovery, 8(4))e1249 2018.
Karen Simonyan, Andrea Vedaldi, and Andrew Zisserman. Deep inside convolutional networks:
Visualising image classification models and saliency maps. In International Conference on Leurning Representations (ICLR), 2014.
Mukund Sundararajan, Ankur Taly, and Qiqi Yan. Axiomatic attribution for deep networks. In International Conference on Machine Learning, pages 3319-3328. PMLR, 2017.
Kagan Tumer and Joydeep Ghosh. Error correlation and enror reduction in ensemble classifiers. Connection science, 8(3-4):385-404, 1996.
Giorgio Valentini, Marco Muselli, and Francesca Ruffino. Cancer recognition with bagged ensembles of support vector machines. Neurocomputing, 56:461-466, 2004.
Yu-Xi Xu-Xiang Wang, Jing Lei, and Stephen E. Fienberg. Learning with differential privacy: Stability, learnability and the sufficiency and necessity of emm principle. Journal of Machine Learning Research, 17(183):1-40, 2016.
ht.
Zhou Wang, Alan C Bovik, Hamid R Sheikh, and Eero Simoncelli. Image quality assessment: from error visibility to structural similarity. IEEE transactions on image processing. 13(4))600-6) 612, 2004.
Zifan Wang, Haofan Wang, Shakul Ramkumar, Piotr Mardziel, Matt Fredrikson, and Anupam Datta. Smoothed geometry for robust attribution. In Advances Neural Information Processing Systems, 2020.
Han Xiao, Kashif Rasul, and Roland Vollgraf. Fashion-mnist: a novel image dataset for benchmarking machine learning algorithms, 2017.
S. Yeom, I. Giacomelli, M. Fredrikson, and S. Jha. Privacy risk in machine learning: Analyzing the connection to overfitting. In 2018 IEEE 31st Computer Security Foundations Symposium (CSF), 2018.
Zhi-Hus Hua Zhou, Jianxin Wu, and Wei Tang, Ensembling neural networks: many could be better than all. Artificial intelligence, 137(112):239 263, 2002.
13

---

Published as a conference paper at ICLR 2022
Awmmn
Risjo1 Mer
Vw mn
WwM
indio
Hrajon
Bigton
Figure 5: Intuitive illustration of how two models which predict identical classification labels can have arbitrary gradients. To show this, given a binary classifier H and an arbitrary function g. we construct classifier H' that predicts the same labels H, yet has gradients equal to g almost everywhere. We formally state this result in Theorem 3.1
A PROOFS
A.1 PROOF OF THEOREM 3.1
-R Theorem 3.1. Let : : x sign(h) () be a binary classifier and g: X- R be an unrelated function that bounded from above and below, contimuous, and piecewise differentiable. Then there exists another binary classifier H sign(h) such that for any c>0,
VxEX. H()))((()))
inf (MM {1le {11e- >2/2 = Vh(x)=
Proof. We partition X into regions {4) determined by the decision boundaries of H. That is, each I; represents a maximal contiguous region for which each x E I, receives the same label from H.
Recall we given function g: X ++R which is bounded from above and below. We create set of functions gi. I R such that
9r, (x) inf if (TT)) (() H(L))
where is some small constant greater than zero. Additionally, let d(x) be the &a distance from x to the nearest decision boundary of ie.
{1I [|x-xa1]]
Then, we define h to be:
[1,(() (I)
for æ El,if(()>>
h(x)=
01, (I)) 2d(w)
for El, if
And, as described above, we define Å signgnh).. First, we show that ((x)= Hz) V.eX. Without loss of generality, consider some I; where (xxx=11 for any E I, We first consider the case where d(z)> z.
By construction, for a E I, (i)==i)n(((()))) sign (zg) (1th::))--aaapaoooo---aaaaaaaa-a By definition of the infimum, inf)))infggg))) 20, and thus sign(g(z) -iffg(=))iinfffffl g() so H(=)=1=H(x))
Note that in the case where d(z) < 5. we can follow the same argument multiplication by positive constant does not affect the sign. symmetric argument follows for the case where for x I, H(x) = 11;
thus. (r) (() E X.
Secondly, we show that Vh(x) Vg(a)) Vi where d(z) 5. Consider the case where H(x) 1. By construction, ((x))= (x) = g(x) (x) Note that this means the infimum and are constants, so their gradients are zero. Thus, Vh(x) = Vg(x). A symmetric argument holds for the case where H(x)=-1.
14

---

Published as a conference paper at ICLR 2022
It remains to prove that h is continuous and piecewise differentiable, in order be realizable as a ReL.Unnttwor.. By assumption, g is piecewise differentiable, which means that 9i are piecewise differentiable as well, is Í((z)- d(x) Thus, is beecewis-diiffreeniiabl To see that h is continuous, consider the case where d(x)= </2 for some æ. Then ((z)). d(2 = =x) (x)). Additionally, consider the case where d(x) = 0, ie. is on decision boundary of h(xx), between two regions 4,14. Then -(z)) d(x) = 9. (2) (z)) This shows that the piecewise components of h come to the same value at their intersection. Further,, each piecewise component of h is equal to continuous function, gzr) is continuous by assumption. Thus, h is continuous, and we conclude our proof.
We include visual intuition of the proof in Figure
A.2 PROOF OF THEOREM 4.1
Theorem 4.1. Let P be learning pipeline, and let S be distribution random states. Further, let gp,s be the mode predictor, let 9n (P,5) (5) for S be a selective ensemble, and let a 20.. Then,
V
Pr 9n (P,S axx) 99 Sa
Proof. In (P,S) is an ensemble of n models. By the definition of Algorithm 2 9n (P.S) gathers vector of class counts of the prediction for from each model the ensemble. Let the class with the highest count be Ca, with counts n. and the class with the second highest count be called cB. with counts In (P.5) runs a two-sided hypothesis test to ensure that Pr[nA ~ Binomial(n, + +n,,0.5) < a, i.e. that CA the true mode prediction over S. See that
Pr(sps(t)+CC) (P..S
-P( Pr (P.. (p,
Pr 9n # [p..((9)
(4)
$ Pr (PSS # ( =a
By Hung et al, 2019
(5)
Thus,
Pr(ge.(t)++e) ;, sa
A.3 PROOF OF COROLLARY 4.2
Corollary 4.2. Let P be learning pipeline, and let S be distribution over random states. Further, let gP.s be the mode predictor, let (P.)).)) for S St selective ensemble. Finally, let a 2 0, and let 020 be upper bound on the expected abstention rate of 9n. (P.)). Then, the expected loss variance, V (z), over inputs, 2, is bounded by a That is,
E D-D
V(x)
E IND
Pr Sm.S
In :t)
sa+B
Proof.
Since gp,s never abstains, have by the law of total probability that
Pr (P,SS a a.x)) (())
S-S" g (P.S ax)) 2h sp &n (P.S: a..):
ShiSn
gn (P.S (.)) (
Pr smgn
(n (P.S a.,)) ABSTAIN
8m5n
15

---

Published as a conference paper at ICLR 2022
By Theorem 4.1
have that Pre-es (in : a. op sx)) sa;
thus
Pr Sm
In
(P.S:
Sat anD
E
Pr Sm.S"
Finally, since is an upper bound on the expected abstention rate of In (P,S), we conclude that
E
(n,P,S
sa+B
anD
S-5m
A.4 PROOF OF COROLLARY 4.3
Corollary 4.3. Let P be learning pipeline, and let S be distribution random states. Further, let 9n (P,S) for S~S" be a selective ensemble. Finally, let a 20, and let 3 0 be an upper bound on the expected abstention rate 9n (P.S). Then,
E
Pr
( (P ;ax)) and]]
<2(a+
R>D
Proof. For For [1,2), let A' the event that Gn (P,S* a,,) = ABSTAAN, and N be the event that (P. a,x) 7 9p,s- In the worst case, A and A?, and N and N are disjoint, that is, e.g., if In (PSt abstains on then 9n (P,S) a,,)) f (2,52 a.r).. In other words, we have that
Pr 81 S1 Sm
(P.S ( a.x) <Pr A A Nº N2)
which, by union bound implies that
Pr 81 Sm
On (P.S)
a.x) <Pr +Pr[A <Pr())++rPP+ +Pr[N*]
By Theorem 4.1
Pr NN <a. Thus have
Pr St S0
Gn (P,S' ;a,z) 4 ;a,))
< 2 E nnD
Pr[A"]
E anD
Pr |r||A]]
Finally, since is an upper bound on the expected abstention rate of 3n (P.S), we conclude that
E
Pr Sm
(P,S') (. ;a,)) am)
((
D
St
B DATASETS
The German Credit and Taiwanese data consist of individuals financial data, with binary response indicating their creditworthiness. For the German Credit dataset, there are 1000 points, and 20 attributes. We one-hot encode the data to get 61 features, and standardize the data to zero mean and unit variance using SKLearn Standard scaler. We partitioned the data intro training set of 700 and test set of 200. The Taiwanese credit dataset has 30,000 instances with 24 attributes. We one-hot encode the data to get 32 features and normalize the data be between zero and one. We partitioned the data intro training set of 22500 and a test set of 7500.
The Adult dataset consists of a subset of publicly-availabl US Census data, binary response indicating annual income > 50k. There are attributes, which we one-hot encode to get 96 features. We normalize the numerical features have values between 0 and 1. After removing instances with missing
16

---

Published as a conference paper at ICLR 2022
values, there are 30,162 examples which we split into training set of 14891, leave one out set of 100, and a test set of 1501 examples.
The Seizure dataset comprises time-series EEG recordings for 500 individuals, with binary response indicating the occurrence of a seizure. This is represented as 11500 rows with 178 features each. We split this into 7,950 train points and 3,550 test points. We standardize the numeric features zero mean and unit variance.
The Warfain dataset is collected by the International Warfarin Pharmacogeneties Consortium (International Warfarin Pharmacogenetic Consortium 2009) about patients who were prescribed warfarin. We removed rows with missing values, 48T9 patients remained in the dataset. The inputs the model are demographic (age, height, weight, race), medical (use of amiodarone, use of enzyme inducer), and genetic (VKORCI, CYP2C9) attributes. Age, height, and weight are real-valued and were scaled zero mean and unit variance. The medical attributes take binary values, and the remaining attributes were one-hot encoded. The output the weekly dose of warfarin in milligrams, which we encode as "low", "medium", or "high", following the recommendations set by International Warfarin Pharmacogenetic Consortium (2009)
Fashion MNIST contains images of clothing items, with a multilabel response of 10 classes. There are 60000 training examples and 10000 test examples. We pre-process the data by normalizing the numerical values in the image array to be between 0 and 1.
The colorectal histology dataset contains images of human colorectal cancer, with multilabel response of 8 classes. There are 5,000 images, which we divide into a training set of 3750 and a validation set of 1250. We pre-process the data by normalizing the numerical values in the image array to be between 0 and 1.
The UCI datasets as well as FMNIST are under an MIT license, the colorectal histology and Warfarin datasets under Creative Commons License. (Dua and Karra Taniskidou 2017 Kather et al. 2016b International Warfarin Pharmacogenetiic Consortium 2009).
C MODEL ARCHITECTURE AND HYPER- PARAMETERS
The German Credit and Seizure models have three hidden layers, of size 128, 64, and 16. Models on the Adult dataset have one hidden layer of 200 neurons. Models on the Taiwanese dataset have two hidden layers of 32 and 16. The Warfarin models have one hidden layer of 100. The FMNIST model a modified LeNet architecture (LeCun et al. 1995). This model trained with dropout. The Colon models are trained with modified, ResNet50 (He et al. 2016), pre-trained on ImageNet (Deng et al. 2009), available from Keras, German Credit, Adult, Scizure, Taiwanese, and Warfarin models are trained for 100 epochs; FMNIST for 50, and Colon models are trained for 20 epochs. German Credit models are trained with a batch size of 32; FMNIST 64; Adult, Seizure, and Warfarin models with batch sizes of 128; and Colon and Taiwanese Credit models with batch sizes of 512. German Credit, Adult, Seizure, Taiwanese Credit, Warfarin, and Colon are trained with keras' Adam optimizer with the default parameters. FMNIST models are trained with keras' SGD optimizer with the default parameters.
Note that we discuss train-test splits and data preprocessing above in Section B We prepare different models for the same dataset using Tensorflow 2.3.0 and all computations are done using Titan RTX accelerator on a machine with 64 gigabytes of memory.
D METRICS
We report similarity between feature attributions with Spearman's Ranking Correlation (p), Pearson's 's Correlation Coefficient (r), top-k intersection, & distance, and SSIM for image datasets. We use standard implementations for Spearman's 's Ranking Correlation (p) and Pearson's Correlation Coefficient (r") from scipy, and implement & distance as well as the top-k using numpy functions.
Note that and P vary from -1 to 1, denoting negative, zero, and positive correlation. We display top-k for k=5, and compute this by taking the number of features in the intersection of the top 5 between two models, and then diving this by 5. Thus top-k is between 0 and 1, indicating low and high correlation respectively.
The lg distance has a minimum of 0, but is unbounded from above, and SSIM varies from -1 to 1, indicating no correlation to exact correlation respectively.
17

---

Published as a conference paper at ICLR 2022
meam + sid. dev of portion of test data with Pmp >0
Randormesss
R
Ger. Credit
Adult
Seizure
Tui. Credit
Warfurin
FMNIST
Colon
RS
.5704 020
087=
.000
0822 002
.088 .003
.61 008
.0772.00
(5, 10, 15, 20)
0.0:0.00
0.0:10.0
0.0:10.0
0.0::..0
0.0:1..0
0.0:10.0
0.0::0.0
LOO
.2222 014
063=.001
001::001
0311.001
0332 003
0344.004
0424.005
LOO
(5, 10. 15, 20)
0.0=0.0
0.040.0
0.0:0.0
0.0
0.0:0.00
0.0+0.00
0.0::.00
Table 5: The percentage of points with disagreement between at least one pair of models (Pmip > 0) trained with different random seeds (RS) or leave-one-out differences in training data, for singleton models (n [n=]] and selective ensembles (n > 1). Results for selective ensembles all selective ensembles are shown together, as they all have no disagreement. Note that these results are for 0.01. But this different a also leads to zero disagreement between predicted points.
Note that compute these metrics between two different models the same point, for every point in the test set, over 276 different pairs of models for tabular datasets and over 40 pairs of models for image datasets. We average this result over the points in the test set and over the comparisons to get the numbers displayed in the tables and graphs throughout the paper.
D.I SSIM
Explanations for image models can be interpreted image (as there is attribution for each pixel). and are often evaluated visually (Leino et al., 2018 Simonyan et al.) [2014] Sundararajan et al. 2017). However, pixel-wise indicators for similarity between images (such as top-k similarity between pixel values, Spearman's 's ranking coefficient, or mean squared error) often do not capture how similar images visually, in aggregate. In order to give an indication if the entire explanation for image model, i.e. the explanatory image produced, is similar, we use the structural similarity index (SSIM) (Wang et al. 2004). We use the implementation from scikit immae (str). SSIM varies from to 1, indicating no correlation to exact correlation respectively.
E EXPERIMENTAL RESULTS FOR a 0.01
We include results on the prediction of selective ensemble models for 0.01 as well. We include the percentage of points with disagreement between at least one pair of models (Phip > trained with different random seeds (RS) or leave-one-out differences in training data, for singleton models (n=1) and selective ensembles (n > 1) in Table [S] Notice the number of points with PRip 0 is again zero. We also include the mean and standard deviation of accuracy and abstention rate for a 0.01 in Table
F SELECTIVE ENSEMBLING FULL RESULTS
We include the full results from the evaluation section, including enror bars the disagreement, accuracy, abstention rates of selective ensembles, in Table and Table respectivell We also include the results for all datasets on the accuracy of non-selective ensembling and their ability mitigate disagreement, in Table 7 and Table 6 respectivel.
G SELECTIVE ENSEMBLES AND DISPARITY IN SELECTIVE PREDICTION
In light the fact that prior work has brought to light the possibility selective prediction exacerbating model accuracy disparity between demographic groups Jones et al. (2020), we present the selective ensemble accuracy and abstention rate group-by-group for several different demographic groups across four datasets: Adult, German Credit, Taiwanese Credit, and Warfarin Dosing, Results are in Table 9
H EXPLANATION CONSISTENCY FULL RESULTS
We give full results for selective and non-selective ensembling's mitigation inconsistency feature atrributions.
18

---

Published as a conference paper at ICLR 2022
mewn accuracy (abstain as emorr/ std. dev
g
n
Ger. Crodit
Adult
Scizure
Wafarin
Tai. Credit
FMNIST
Colon
RS
5
0.0.10.0
0.0.1...0
0.0:10.0
0.0
0.0:10.0
0.0.10.0
0.0.10.0
RS
10
4611.016
807 3
.4 3
3461
.788.122 3
870 3
902.22c
RS
.589 015
8224
961
361 3
802-100
200
915-1e--3
20
.513=.011
822#
.61
162=1e-3
.83
1e-
326=1e-3
LO0
0.0+0.00
0.0++0.0
0.0+0.00
0.0+0.00
0.0++0.0
0.0++0.0
0.0+0.0
LOO
10
618-t.017
318-1e-3
.47 947-144e 4
674 2
1e-3
60
20-1122-33
LOO
15
65611.017
12 3
963 3
678.1
812.19c 4
908.1cc 3
.912122
LOO
20
6611.018 018
72
16411e-3
7e
812488
90
912222
mean abstention rate std dev
S
Ger. Credit
Adult
Seizure
Warfarin
Tai. Credit
FMNIST
Colon
10400.
104000
1.0:0.0
10::0.
10400.
10400.
100:0.0
RS
10
.494.021
26842e-3
.45=2e-3
078 3
263+2e-3
087
050 3
RS
15
278+.017
11-
125+1e-3
30
1e-3
230
RS
20
2270+0015
140+1-63
-04+1e-3
2047+2--3
1e-++1e-3
0054+9--4
138+1e-3
LOO
5
1.010.0
10.00.00
1.0:0.00
1.0:1.00
1.0.0..0
1.0.1.00
1.0.0.00
LOO
2215+0030
349-22-3
305=5e-3
20
1e-+1e-3
109++e--
.044+2--3
LO0
.144+0.040
-00++2-3
-026+1e-
-017+2--3
-01+-e--
+
-
LOO
20
.133:1.040
129-1c-3
.025
.017:1c-3
.01:1:2-3
-19-11e-3
.035:136-3
Table 6: Accuracy (above) and abstention rate (helow) of selective ensembles with n (5, 10, 15,20) 20) constituents. Results are averaged over 24 models, standard deviation presented. Note that these results for 0.01.
meam +: sid. dev of portion of test data with Pmp >00
Randomness
n
Ger. Credit
Adult
Seizure
Tui. Credit
Warfurin
FMNIST
Colon
RS
I
.570 020
007=:001
.000+
.822 002
.098 0003
.61 008
.037+.003
(5, 10, 15, 20)
0.0:0.0
0.0::0.0
0.0:00.0
0.0:0..0
0.0:10.0
0.0:10.0
0.0:0.0
LOO
.2222 014
.033 001
.314 001
.311
0331 003
.344 004
.4221.05
LOO
(5, 10. 15, 20)
0.0=0.0
0.040.0
0.0+0.00
0.0
0.0:1..0
0.0+0.0
0.0:0.00
Table 7: Percentage of points with disagreement between at least one pair of models (Pri >> 0) trained with different random seeds (RS) or leave-one-out differences in training data, for singleton models (n= 1) and selective ensembles (n > 1). We present the mean and standard deviation of this percentage over 10 runs of re-sampling ensemble models. Note that these results are for a= 0.05 and a = 0.01, since both resulted in inconsistent prediction over predicted points.
meon accuracy (abstoin as emory / std. dev
S
n
Ger. Credit
Adult
Seizure
Warfarin
Tai. Credit
FMNIST
Colon
RS
0.0:10.0
0.0:1:.0
0.0:10.0
0.0:10.0
0.0::0.0
0.0:10.0
0.0:10.0
RS
.5762.013
820
160=1e-3 3
.60
100++1e-3
888 3
914+1e-3
63611.017
827116e
165
668
807 190
897
91911e
RS
20
.6441.014
830150
.67
.67013
.801188
90211e
.21
LOO
0.0:0..0
0.0:0.00
0.0:00.0
0.0=00.0
0.0::0.0
0.0::0.0
0.0:0000
LOO
10
.65++.017
827+
.62
1e7
812+1e-3
409+
1922+1e-3
LOO
15
+714
432+7e-4
968 9
.679+ 94
814+9e-4
-10+1e-3
-16++2e-3
LOO
20
.8941.014
83417e-4
190 3
.80 7
815.188-4
991114c-4
.988
abstention rute std dev
S
Ger. Credit
Adult
Scizure
Warfarin
Tai. Credit
FMNIST
Colon
RS
5
1.0+1:.0
1.0.1..00
1.0:10.0
1.0:1:.0
1.0:1..0
1.0:1..0
1.0:1:.0
RS
10
2011.014
04311c 3
.2111c 3
0501
.09-1.2e 3
059112c 3
.32-13c
RS
15
2034 020
102-1e-33
10--1e--3
3037
0284+le-2
042-22
023422 3
.1654.015
02447-e4
.014=7e-4
.031
.033
1e-
.019
LO0
1.0+0.00
1.0+0.00
1.0+0..0
1.0+0.0
1.0+0.00
1.0+0.0
1.0+0.0
LOO
.151+:041
-
2
.018
-
5e-
LOO
105110.04
002-11 3
019-1c
.1311
.13312c 3
016.12c
027
LOO
20
.079+.009
.0881e-3
.015+1e-3
-011+2e-3
.00+1e-3
.012+8e-4
-023+2e-3
Table 8: Accuracy (above) and abstention rate (below) of selective ensembles with E (5,10,15, 10, constituents. Results are averaged over 24 models, standard deviation is presented. Note that these results for a =0055, which presented in the main paper.
19

---

Published as a conference paper at ICLR 2022
disagreement of non-abstaining ensembles Warfarin
S
Ger. Credit
Adult
Seizure
Tai. Credit
FMNIST
Colon
RS
.570:.020 .087+.001
.060:.01
.082:0002
.0981.003
0.113:.005
.066:.002
RS
5
.3051.017 .045:.001
.028:1.01
.082:.002
.054:0003
.046:1002
.022:1.001
RS
10
.234..014 .031#.001
.019#.001
.041#.001
.040=.002
.032#.002
.014#.002
RS
15
.185:.012 .026=.001
.015#.001
.030=.000
.033+#002
.028+.002
.012#
RS
20
.155+.010 .022++001
.013+-001
.021
.030+=002
.026+0001
.010+.001
LOO
.262:.014 .063+.001
.031#.001
.031
.033#.003
.0562.004
.0681.003
LOO
5
.142+.037 .033=.001
.028:.001
.019:.001
.018#.001
.032:.002
.030=1003
LOO
10
.111#.020 .023+.001
.020+f.001
.014:.001
.016+.001
.034+0002
.016+.003
LOO
15
.074+.02 .019+.001
.017+.001
.011=.001
.012+.001
.029+0001
.014+.002
LOO
20
.0671.003
.016:.001
.015:1.011
.010+.000
,011
.027:.001
.010::001
Figure 6: Mean and standard deviation of the percentage of test data with non-zero disagreement over 24 normal (i.c., not selective) ensembles. The mean and standard deviation are taken over ten re-samplings of 24 ensembles. While ensembling alone mitigates much of the prediction instability, it is unable eliminate it as selective ensembles do.
accuracy of non-abstaining ensembles
S
Ger. Credit
Adult
Seizure
Warfarin
Tai. Credit
FMNIST
Colon
RS
0.745
0.842+00000
0.905+00001
0.688- 0.0
0.919+0.001 0.802++0.001
0.027+0.001
RS
0.747=0.014
0.843+0.000
0.005
0.822++0.001 0.688=0.0
0.92++0.001
0.028+00001
RS
0.75=0.01
0.842440.001
0.975:10.001
0.688=0.0
0.822=10.001
0.92:40.001
0.028+0.0001
RS
20
0.747=0.001 0.842:10.0
0.975=+0.001
0.688=0.0
0.822=:0.001
0.92++0.001
0.9
LOO
5
0.728=0.011 0.84410.00
0.979110.001
0.685=0.0002
0.001
0.91810.00
0.927 100002
1.00
10
0.728
0.844+00001
0.978+0.001
0.686+0.002
0.821+-0.00
0.918+00.0
0.927+0.U0
LO0
15
0.733-0.00
0.844+0.0
0.979++0.001
0.685++0000
0.821+00.0
0.917++0.0
0.927+00000
LOO
20
0.73+
0.84310.00
0.979110.011
0.085+0.002
0.821+10.0
0.918=0.001
0.927
Figure Accuracy of non-sclective (regular) ensembles with n (5,10,15,20) constituents. Results are averaged over 24 models, standard deviation is presented.
H.1 ATTRIBUTIONS
We pictorially show the inconsistency of individual model feature attributions versus the consistency of attributions ensembles of 15 for each tabular dataset Figure & and Figure 9 The former shows inconsistency over differences in random initialization, the latter shows inconsistency over one-point changes to the training set.
H.2 SIMILARITY METRICS OF ATTRIBUTIONS
We display how Spearman's 's ranking coefficient (p), Pearson's Correlation Coefficient (r), top-5 intersection and E2 distance between feature attributions over the same point become more and more similar with increasing numbers of ensemble models. While the comparisons to generate the similarity score between two models on the same point, the result is averaged over this comparison for the entire test set. We average this over 276 comparisons between different models. In cases were abstention high, indicating inconsistency the dataset for the training pipeline, selective ensembling can further improve stability of attributions by not considering unstable points (see e.g. German Credit). We present the expanded results from the main paper, for all datasets, all four metrics (as SSIM is only computed for image datasets, and p is not computed for image datasets). We display error bars indicating standard deviation over the 276 comparisons between two models for tabular datasets, and 40 comparisons for image datasets.
I INTUITION BEHIND ENSEMBLE GRADIENT CONSISTENCY
In section 4 demonstrated how prediction inconsistency can be provably bounded in selective ensembles. Section 5.2 showed that ensembling also improves the consistency of the gradients; we now provide some theoretical insight why this is the case.
20

---

Published as a conference paper at ICLR 2022
dccuracy fabriai enor)/ obieetion rate
Adult Male
Adult Fem
Ger. Crod. Young
Ger. Cred. Old
Tal. Cred. Male
Tai. Cred. Fem.
Warf. Black
Warl. White
Warf. Asian
Base
8888
923V
.077/
.7771-
51
825V
665/
.88/
68W
RS
0.0/1.0
0,0/1.0
0.0/1.0
0001100
0.0/1.0
0.0/1.0
0,011.0
00/1.3
0.0/1.0
RS
10
7777/063
9122,023
507//394
436//254
791/048
807/1055
650/000
881 7-002
683/ DU1
RS
15
T86/
9152-015
.500/248
206/1168
798/
812/1025
664/001
0002
10GG
RS
20
1000
9117-013
.886 2016
7333/130
802/ 022
814/1020
061/009
[633 1002
689) LOG
Base
800
9222-
Jt97]
3077
81V-
821
8637
8867
G8SV
LO0
0.0/1.0
0,0/1.0
0.0/1.0
0,/110
0.0/LD
0,0/10
0,0/1.0
0,0/1,
0,0/10
10
709/0088
9033/0188
6612/116
18911138
802/
817,001
690/0019
6800 001
15
783/
91622012
.666. 101
.00//107
806/
819/001
8558
682/202 2013
LO0
20
7766 0022
9177-010
.881/0021
2114/084
.888 0L1
.820/000
.653 011
(62) 0011
683/1011
Table 9: We present the selective ensemble accuracy and abstention rate group-by-group for several different demographic groups actoss four datasets: Adult, German Credit, Taiwanese Credit, and Warfarin Dosing, We note that by and large, using selective ensembles did not exacerbate accuracy disparity by very much (within 1% of the original disparity), although they did not ameliorate disparities in accuracy that already existed within the performance of the algorithm. The only exception to this was German Credit, where we note, as in the remainder of our results, that the entire dataset is only 1000 points, so results may be slightly different this regime. Overall, we note that subgroup abstention rates can vary by dataset, and so it should be studied whenever selective ensembles used in sensitive setting.
At high level, we argue that by taking the average gradient (of the mode prediction, w.r.t the input), we reduce the variance in the ensemble gradient, stabilizing it towards the expected gradient over the distribution S. While it is difficult exactly characterize the distribution over gradients, we provide formalized intuition by making some simplifying assumptions about this distribution.
We model the variations in gradients from model to model Gaussian (i.e., the gradient of each model deviates from the expected gradient by some Gaussian noise). Formally, let f PP()) be a model produced by the pipeline on some random state, S, and let f -Es-s V (1:) be the expected gradient over S. We will assume that fz), the gradient of a particular model produced by the pipeline, is given by V [zx)) +% where (0,a))
Under this assumption, the fact that the gradient is stabilized by ensembling follows from the fact that the variance of the sample mean is smaller than the population variance. Specifically, the variance of the sample mean of the gradient with n samples is o²/n, which tends towards 0 as n increases (Equation 6)
n V((x)
Var
(6)
The metrics we measure for gradient consistency are not based on variance of the gradient (which Equation shows is reduced by ensembling), but rather ranking of features according to their gradients. To relate this to variance we can argue the following: first, let us take, for example, the top-ranked feature (the feature with the largest positive gradient). Assuming the expected gradient is not degenerate (in our case this means there is a unique top-ranked feature), we can consider the gap between the average gradient of the top-ranked feature, and the second-ranked feature, j. When the variance in the ensemble gradient is reduced sufficiently relative to this gap, it will be unlikely that this order will be switched.
Specifically, let 8 -=) (2)) be the gap between the average gradient of the top-ranked feature be the gap between the gradient of the same features on an ensemble of n models produced by the pipeline. Note that the ordering between features i and j is preserved provided that d(n) > O. Thus, we can quantify the probability that this ordering preserved according to Equation]] 7
and the second- -ccked- feature, and let 6(2)) = + E vf v,rr)),,fffe fz(()
where 1)s ~ (...V((V())) (i((t), = and ny ~ (V.V(((V))) {,)(s),
0²
Pr(p(n)>00 0|- Primx > O]
= Pr 8 Tkj
where Mly ~ ~N(11
202
+ 8
n2
20a
The same argument holds for other pairs of features.
21

---

Published as a conference paper at ICLR 2022
(a)
(b)
(c)
(d)
(c)
Figure 8: Inconsistency of attributions on the same point across an individual (left) and ensembled (right) model (n= 15), for all datasets, over differences in random seed chosen for initialization parameters before training. The height of each bar the horizontal axis represents the attribution score of distinct fcature, and each color represents different model. Features are ordered according to the attribution scores of one randomly-selected model. Figure @ depicts the German Credit Dataset, Figure b deppics Adult, Figure Seizure, Figure Taiwanese, and Figure è Warfarin. We do not include feature attribution for image datasets as the individual pixels are less meaningful than the feature attributions in tabular dataset.
J ACCURACY REJECTION CURVES
Figure I1 shows plots accuracy-rejection curves Nadeem et al. (2009) for selective ensembles sizes 5. 10. 15, and 20. Rejection rates were controlled by varying from O. In accordance with the convention of Nadeem et al.] we count the accuracy as 1.0 when the model abstains on all points.
The plots indicate that accuracy remained relatively high even with low abstention rates and did not increase substantially when more points were rejected, signifying that there was not a strong trade-off between accuracy and rejection. This is desirable hecause clearly prefer to have low rejection rate; meanwhile,
22

---

Published as a conference paper at ICLR 2022
(a)
(h)
(d)
(c)
Figure 9: Inconsistency of attributions on the same point across an individual (left) and ensembled (right) model (n=12 15), for all datasets, over leave-one-out differences in the training set. The height of each bar on the horizontal axis represents the attribution score of distinct feature, and each color represents differrntt model. Features are ordered according to the attribution scores of one randomly-sclecter model. Figure deppict the German Credit Dataset, Figure b depicts Adult, Figure e Figure | Taiwanese, and Figure (e) Warfarin. We do not include feature attribution for image datasets as the individual pixels less meaningful than the feature attributions in tabular dataset.
the purpose of abstention is to guarantee consistency, so we do not expect abstention to have a strong effect on accuracy. Furthermore, low rejection rates correspond more consistent predictions. Finally, we note that rejection rates are kept low even for small values of a by increasing the size of the ensemble.
23

---

Published as a conference paper at ICLR 2022
top
1.0
1.0
10
15
10
0.4
04
0.2
0.2
0.5
1.0
0.81
015
0.5
0.20
0.15
0.2
0.3
0.3
0.05
0.5
0.5
0.6
0.5
1.5
002
0.2
0.0
20
0.0
16
000
30
0.5
10
20
0.3
0.0
0.6
0.5
0.2
0.4
0.0
20
0.0
16
20
0.0
30
0.0
10
20
1.0
1.0
1.0
0.5
0.4
04
0.2
1.0
0.5
0.3
15
0.4
0.2
0.2
0.2
0.5
0.55
0.5
0.5
0.50
8:40
0.2
1F
0.2
0.2
0.0
00
Figure 10: We plot the average similarity across feature attributions for an individual point, averaged over 276 comparisons of feature attributions from two different models. This is aggregated across the entire validation split. The error bars represent the standard deviation over the 276 comparisons between models. Each row of plots constitutes the plots for given dataset, noted on the far left, and each column of plots is for given metric, noted at the top. Note that for image datasets, (FMNIST and Colon), we plot SSIM instead Spearman's 's Ranking Coefficient (p). The x-axis the number of models in the ensemble, starting with one, and the y-axis indicates the value of the similarity metric averaged over all 276 comparisons of individual points' in the validation split's attributions. The red and orange lines depict regular ensembles, and the green and blue represent selective ensembles.
24

---

Published as a conference paper at ICLR 2022
09-
0.
09-
077
0.7
0.7
57
1.0
colon
0.0
0.3
6.0
0.2
0.4
06
0
01.4
5.0
0.8
1.0
0.2
0.2
1.0
Figure 11: Graphs of accuracy (y) versus abstention (x) of ensembles of different size, gathered by calculating accuracy and abstention for 12 different values of a. Note that the y-axis begins 0.6.
25


Reviews:
[{"review_id": "VBAgNwqwilE", "reviewer": "Reviewer_hSPx", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper motivates a very practical problem of model and feature attribution instability, and proposed a solution with a plethora of theoretical and empirical guarantees. The theoretical development is sound, and the empirical study section is comprehensive and convincing. The paper is also well written in a clear and concise manner.\n\nThe reviewer's major concerns here are based on the novelty and the feasibility of the proposed approach. It is worth pointing out that neither of these two points prevents the paper from being considered as a highly completed work, and the reviewer would gladly update the rating should the concerns be concisely addressed.\n\n1. Regarding the novelty, it is not very surprising to see that bagging / ensembling helps with the prediction stability for classifiers, especially when equipped with reject option so that the most ambiguous (hard) examples can somehow evade the strict metric computation. What seems more novel here is the improvement of gradient stability, which is still expected (since taking gradient is still a linear operation inside the expectation) but has involved less study. The empirical evidence shown in Figure 4 is strong, and it would be even better if there were corresponding theoretical discussions.\n2. The proposed method constructs multiple (20 at most as in the empirical study) models. Since the discussion seems to be heavily focused on deep neural nets, two things to catch here:\n- It is unclear whether the proposed method would feasibly apply to cases involving giant model structures and a large amount of output classes. (This comment is slightly nitpicking)\n- On one hand, neural net ensembles can be technically constructed as a single neural net with an aggregating final layer and multiplicative numbers of parameters. On the other hand, neural nets themselves are showing behaviors of ensembles, especially when trained with dropout or similar mechanism. The proposed method does not take advantages of these existing relationships between neural nets and neural net ensembles.\n\nThe reviewer also finds two minor theoretical issues that might deserve a bit more attention\n- The power of the top_2 test is highly influenced by the data generating distribution (e.g. a fair coin flip would also trigger the reject option, and maybe the German Credit dataset), therefore \\alpha and \\beta are probably not two independent parameters. \n- For reject (abstention) options there is usually a penalty for saying \"I'm not sure\". It would influence the rates by a fraction of \\beta which is safe to ignore. \n\nThe paper is nicely written. The only editorial comments:\n1. Theorem 3.1 is also referred as Theorem A.1 in multiple places.\n2. Table 3, LF -> LOO\n3. Section 3, line 5, \"Feature attributions are a commonly used as a tool...\", redundant \"a\". \n\n\n\n=====\nThe reviewer would like to upgrade the rating to an 8 after the author response. \n", "summary_of_the_paper": "In the paper the authors proposed a selective ensemble whose constituents are classifiers with reject (abstention) option. They theoretically exposed the problem that differentiable models yielding identical predictions might remotely share identical gradients, biasing the feature attribution based on (integrated) gradient approaches. They theoretically demonstrated that the selective ensemble can contribute to model stability, and empirically showed that both the model stability and feature attribution could be improved as a result of the proposed method.", "summary_of_the_review": "The paper presents a highly completed study on the improvements of model stability and feature attribution thanks to the contribution from selective ensembles with abstention option. The paper could use a bit more theoretical discussions regarding the inner workings of the proposed method as well as the relationship between the work and existing ensemble studies to highlight its novelty. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "OSs_ZggL5uR", "reviewer": "Reviewer_dvdh", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The topic is interesting and the algorithms nice and simple. However, the empirical section is quite difficult to read due to insufficient descriptive details in the captions, as well as out of order referencing in the main text. \n\nstrengths:\n\n- Theoretically sound idea which is also simple in practice.\n\nweaknesses:\n\n- The authors demonstrate that selective ensembles remove disagreements by contrasting with standard ensembles, but don't show accuracy comparisons against standard ensembles. Table 1 could do with comparitive numbers from a standard ensemble.\n- Captions in general don't explain the plots/tables sufficiently. Table 4 in particular is uninterpretable as it is unclear what the two numbers\nin the cells actuall are. This is a significant issue for interpreting the feature attribution results.\n- Resampling can also involve features not just samples, like the random forest (an ensemble method) has demonstrated. This is not explored in the paper.\n\nminor:\n\n- I'm used to captions above tables and below figure, so I kept reading the wrong thing with consecuitive tables\n- Table 3/fig 3 seem to be discussed and referenced before table 1/fig 1; maybe swap the order\n- figure 4 as a bar chart is visually misleading due to overplotting of many colours; why not just plot the points with some x displacement to prevent overlaps?", "summary_of_the_paper": "The paper considers the problem of dealing with inconsistencies when forming (binary) classifier ensembles. The authors propose a mechanism whereby an ensemble can abstain from making a prediction based on a statistical test on the predictions from its constituents.", "summary_of_the_review": "While an interesting method the lack of clarity in the empirical experiments reduces the impact. Furthermore, the method reduces disagreement by essentially avoiding the problem through abstention; it's not really demonstrated that this has a practical advantage other than through highlighting specific cases for manual intervention or inspection.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "MC6ELo3N-ES", "reviewer": "Reviewer_5ZWm", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The authors address the problem, where models trained for the same purpose achieve similar accuracy on consistent test data but behave very differently in individual predictions, is highly problematic and essential in domains such as medicine and finance where a high degree of reliability and stability is required.\nTo address this problem, the authors introduce a selective ensemble that mitigates such inconsistencies by applying hypothesis testing to the predictions of a set of models trained with randomly selected starting conditions.\nIn the selective ensemble, predictions are aborted when the constituent models disagree sufficiently. To demonstrate the effectiveness of the proposed selective ensemble, we first prove theoretically that the prediction discrepancies between the selective ensembles are bounded. Furthermore, using seven benchmark datasets, we show that the selective ensemble has zero points with inconsistent predictions and a low abstention rate of 1.5%.\nThe proposed idea is simple but appears to work well for the problem being tackled.", "summary_of_the_paper": "In this paper, the authors address the problem that models trained for the same purpose may achieve similar accuracy on consistent test data but differently on individual predictions. Such inconsistent behavior is highly problematic, especially in the medical and financial domains. To address this problem, the authors introduce a selective ensemble that mitigates such inconsistencies by applying hypothesis testing to the predictions of a set of models trained on randomly selected starting conditions. First, they proved that the inconsistency of predictions between the selective ensembles is bounded. Additionally,  they empirically showed that the selective ensembles achieve consistent predictions and feature attributions while maintaining a low abstention rate. On seven benchmark datasets, the selective ensembles have zero points with inconsistent predictions. And an abstention rate as low as 1.5%. Thus, the selective ensemble proposed by the authors may present a more reliable method for using deep models in environments where high model complexity and stability are required.", "summary_of_the_review": "This paper addresses an important problem in machine learning that is of great interest to many researchers and practitioners. The authors propose a method called selective ensembles and evaluate its effectiveness from both theoretical and empirical perspectives to solve this problem. Experiments on benchmark data show that the proposed method seems to work well for the problem. In conclusion, we believe that this paper is worthy of acceptance.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "JczJmMh_osN", "reviewer": "Reviewer_AzPU", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Although the idea is interesting and theoretically sound, it is unclear in which aspect the proposed method is superior to existing methods. \n- If it's classification accuracy, the method should be compared with such methods as hard-voting prediction, soft-voting prediction, and other basic ensemble methods that aggregate individual predictions, and also individual model's prediction and MC-dropout.\n- If it's uncertainty quantification performance (or selective classification performance - trade-off between classification accuracy and abstention rate), various uncertainty quantificaiton measures can be compared, such as entropy, BALD, .... Deep ensemble can also be compared.\n- Lakshminarayanan, B., Pritzel, A., & Blundell, C. (2016). Simple and scalable predictive uncertainty estimation using deep ensembles. arXiv preprint arXiv:1612.01474.\n\nAnother contribution the authors mentioned is that the proposed method achives consistent feature attribution. i'm wondering what's the practical meaning of it. how does it relate to the use of a model in classifying data?\n\nFor the proposed method, the absention rate may be adjustable by using different criteria for the statistical testing. It would be interesting if the proposed method is evaluated in terms of accuracy-rejection curve to see the trade-off between classificaiton accuracy and abstention rate)\n- Nadeem, M. S. A., Zucker, J. D., & Hanczar, B. (2009, March). Accuracy-rejection curves (ARCs) for comparing classification methods with a reject option. In Machine Learning in Systems Biology (pp. 65-81). PMLR.\n\nMinor comments\nplease elaborate on what \"binom_p_value\", \"top_2\" function, n_A, and n_B are in Algorithm 2 to help readers to comprehend.\n", "summary_of_the_paper": "This paper presents a selective ensemble method. Given a set of models, it performs a statistical test to determine if the mode prediction was selected by a statistically significant majority of the constituent models. If the test succeeds, it returns the mode prediction, Otherwise, it abstains from prediction. the authors demonstrated the effectiveness of their method on improving classification accuracy by abstaining from those that are difficult to classify.", "summary_of_the_review": "It's unclear how the proposed method is beneficial in classification problems. The authors should provide the practical meaning of their method and compare their method with some proper baselines if necessary.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This is a well-done job which combines a few ideas to reach means to identify problematic cases and indicate this when classifying. It has raised doubts about the applicability, though I can see that a abstention rule can have multiple uses. While the work seems to be well done, it has not largely excited the committee members. It initially missed to be placed well wrt existing work to highlight the novelty, and the demonstration that the approach can be generally useful is not complete. Dealing with abstention rules always brings another facet to classification and comparisons are not trivial in many situations. Not surprisingly, this has landed as a borderline case, which I place on the inside (as I like the topic and I think it is interesting work) but it could become an outsider depending on the overall view of the selected papers for the conference and other constraints.
Decision: Accept (Poster)