Selective Ensembles for Consistent Predictions

1 INTRODUCTION . Recent work has drawn attention to the fact that models that appear similar from aggregate quality measures , such as accuracy , often have markedly different behavior at the level of individual predictions ( Black and Fredrikson , 2021 ; Marx et al. , 2019 ) . Further , in deep models , this inconsistency can arise even between closely-related models , such as those arising from different initializations , or from leave-one-out differences in the training data ( Black and Fredrikson , 2021 ; D ’ Amour et al. , 2020 ; Mehrer et al. , 2020 ) . This behavior is undesirable in many high-stakes contexts , such as medical applications and credit-approving scenarios , as it may cast doubt on the justifiability of the model ’ s outcome and pose difficulties for reproducibility and comparison . We begin by demonstrating that not only are the predictions of related deep models often dissimilar , but their feature attributions ( Simonyan et al. , 2014 ; Sundararajan et al. , 2017 ; Leino et al. , 2018 ) are as well ( Section 3 ) . In particular , we show that there is little connection between a model ’ s gradients , which are the basis for many deep attribution methods , and the labels that it predicts—models with identical predictions can have arbitrarily different gradients almost everywhere ( Theorem 3.1 ) . In practice , we show that this result occurs often on common datasets across closely-related models , leading to significant variation in attributions . This may be undesirable , as feature attributions are commonly used to provide explanations ( Simonyan et al. , 2014 ; Sundararajan et al. , 2017 ; Leino et al. , 2018 ) , debug model behavior ( Adebayo et al. , 2020 ) , and diagnose problems related to privacy and fairness ( Leino and Fredrikson , 2020 ; Datta et al. , 2016 ) . Beyond these pragmatic concerns , this suggests that the salient factors behind these models ’ predictions on many points may have little in common , even when models appear to do comparably well on test data . To address inconsistency in both prediction and attribution , we then turn to ensembling , a well-known approach for reducing predictive variance ( Meir et al. , 1995 ; Naftaly et al. , 1997 ; Lincoln and Skrzypek , 1990 ; Fumera et al. , 2005 ; Hansen and Salamon , 1990 ; Krogh and Vedelsby , 1995 ) . We introduce selective ensembles , which leverage a recent result on multinomial rank verification ( Hung et al. , 2019 ) —which has also been used recently for making certifiably-robust predictions ( Cohen et al. , 2019 ) —to efficiently mitigate the problem of inconsistency with a probabilistic guarantee . Given a point to classify , a selective ensemble returns the mode of the class labels predicted on that point , where the mode is sampled over models that vary according to a specified source of randomness in the training process . Importantly , if the mode can not be inferred with sufficient confidence , then the selective ensemble abstains from prediction . This allows us to bound the probability that these ensembles do not return the true mode prediction ( Theorem 4.1 ) , and by extension , the rate of disagreement between selective ensembles ( Corollary A.4 ) . In addition , we show that this also bounds the variance component in the ensembles ’ bias-variance error decomposition ( Domingos , 2000 ) ( Corollary A.3 ) , providing guidance on how to effectively use of them in practice . Our experiments show that on seven benchmark datasets , selective ensembles of just ten models either agree on the entire test data across random differences in how their constituent models are trained , or abstain at reasonably low rates ( 1-5 % in most cases ; Section 5.1 ) . Additionally , we show that simple ensembling doubles the agreement of attributions on key metrics on average , and when the variance of the constituent models is high that selective ensembling further enhances this effect ( Section 5.2 ) . In summary , our contributions are : ( 1 ) we show that beyond predictions , feature attributions are not consistent across seemingly inconsequential random choices during learning ( Section 3 ) ; ( 2 ) we introduce selective ensembling , a learning method that guarantees bounded inconsistency in predictions , ( Section 4 ) ; and ( 3 ) we demonstrate the effectiveness of this approach on seven datasets , showing that selective ensembles consistently predict all points across models trained with different random seeds or leave-one-out differences in their training data , while also achieving low abstention rates and higher feature attribution consistency . 2 NOTATION AND PRELIMINARIES . We assume a supervised classification setting , with data points ( x , y ) ∈X×Y , drawn from data distribution , D , where x represents a vector of features and y a response . In order to capture the effects of arbitrary random events on a learned model—ranging from randomness during training to randomness in the data selection process—we generalize the standard concept of a learning rule to that of a learning pipeline . Specifically , a learning pipeline , P , is a procedure that outputs a model , h : X→Y , taking as input a random state , S∼S , containing all the information necessary for P to produce a model ( including the architecture , training set , random coin flips used by the learning rule , etc. ) . Intuitively , S represents a distribution over random events that might impact the learned model . For example , S might capture randomness in sampling of the training set , or nondeterminism in the optimization process , e.g. , the initialization of parameters , the order in which batches are processed , or the effects of dropout . In our experiments , we model S to capture two specific types of random choices , namely ( 1 ) the initial parameters of the model , and ( 2 ) leave-one-out changes to the training data . As the initial parameters of the model tend to be determined by a random seed , we will interchangeably refer to this as the selection of random seed . More generally , both of these types of choices instantiate a broader class of choices that could be considered arbitrary , despite the fact that they may impact the predictions ( Black and Fredrikson , 2021 ; Marx et al. , 2019 ; Mehrer et al. , 2020 ) ( Section 5.1 ) and explanations ( Section 5.2 ) of the resulting model . 3 INSTABILITY OF FEATURE ATTRIBUTIONS IN DEEP MODELS . Before we consider mitigating predictive inconsistency with ensembling , we first demonstrate that models ’ inconsistency across random choices in training is exhibited not only through its predictions , but through its feature attributions as well . Feature attributions refer to numeric scores generated for some set of a model ’ s features—most commonly the model ’ s input features—which are meant to connote how important each feature is in generating the model ’ s prediction . Feature attributions are commonly used as a tool for explaining model behavior ( Simonyan et al. , 2014 ; Leino et al. , 2018 ; Sundararajan et al. , 2017 ; Adebayo et al. , 2020 ) localized to given set of inputs . Thus , inconsistent feature attributions between models suggest the models differ in the process by which they arrive at their predictions , even if the predictions are the same . In deep models , many of the most popular attribution methods are based on the model ’ s gradients at or around a given point ( Simonyan et al. , 2014 ; Sundararajan et al. , 2017 ) . Accordingly , we will focus on the stability of gradients , and show via analysis and experiment that they are not stable in conventional deep models . First , we motivate our results by showing that even two deep models that predict the same labels on all points may have arbitrarily different gradients almost everywhere . Later , in our empirical evaluation ( Section 5.2 ) , we demonstrate the extent of the differences between Saliency Maps ( Simonyan et al. , 2014 ) ( i.e. , input gradients ) of deep networks even when the randomness of the learning pipeline is controlled to allow only one-point differences in the training set or differences in the random seed . Predictions with Arbitrary Gradients . We show that even deep models that predict the exact same labels on all points can not necessarily be expected to have the same , or even similar , gradients ; in fact , given a binary classification model h , we can construct a model ĥ which predicts the same labels as h , but has arbitrarily different gradients everywhere except an arbitrarily small region around the boundary of h ( Theorem 3.1 ) . Theorem 3.1 . Let H : X→ { −1,1 } = sign ( h ) be a binary classifier and g : Rn→R be an unrelated function that is bounded from above and below , continuous , and piecewise differentiable . Then there exists another binary classifier Ĥ=sign ( ĥ ) such that for any > 0 , ∀x∈X . 1 . Ĥ ( x ) =H ( x ) 2. inf x′ : H ( x′ ) 6=H ( x ) { ||x−x′|| } > /2 =⇒ ∇ĥ ( x ) =∇g ( x ) The proof of Theorem 3.1 is given in Appendix A.1 in the supplementary material . The proof is by construction of ĥ ; a sketch giving the intuition behind the construction is provided in Figure 1 . In short , we first partition the domain into contiguous regions that are given the same label byH . We then construct ĥ from g by adjusting g to lie above or below the origin to match the prediction behavior of h in each region . As these transformations merely shift g by a constant in each region , they do not change∇g except near decision boundaries , where it is necessary to move across the origin . Observations . The intuition stemming from Theorem 3.1 is that a model ’ s gradients at each point are largely disconnected from the labels it predicts on a distribution . As models that make identical predictions are likely to have similar loss on a given dataset , this theorem points to the possibility that models of similar objective quality may still have arbitrarily different gradients . In Section 5.2 , we demonstrate that this outcome is not only possible , but that it occurs in real models—for example , on the German Credit dataset predicting credit risk , on average , individual models with similar accuracy agree on less than two out of the five most important features influencing their decision . 4 SELECTIVE ENSEMBLING . The results of Section 3 suggests that models that are retrained and redeployed , may exhibit substantially different behavior from their previous iterations . We build on the approach of ensembling for variance reduction by showing how these differences in behavior can be bounded via selective ensembling . However , whereas prior work which finds that more diversity among the constituent networks is beneficial for reducing overall error ( Krogh and Vedelsby , 1995 ; Hansen and Salamon , 1990 ; Maclin et al. , 1995 ; Opitz and Shavlik , 1996 ) , our goal is to minimize , or at least place strict bounds on , the variance component . We show that ideas from robust classification , and in particular randomized smoothing ( Cohen et al. , 2019 ) , which stem from recent results on multinomial hypothesis testing ( Hung et al. , 2019 ) , can be used to enforce such a bound . Mode Predictor . We may view the image of the learning pipeline , P , as a distribution over possible models induced by applyingP to the random state , S∼S . The mode prediction on an input x , with respect to S , is the expected label that would be predicted onx by models drawn from this distribution . More formally , we define the mode predictor , gP , S for a pipeline , P , and random state distribution , S , as given by Equation 1. gP , S ( x ) =argmax y∈Y { E S∼S [ 1 [ P ( S ; x ) =y ] ] } ( 1 ) Algorithm 1 : Selective Ensemble Creation def train_ensemble ( P , S∼Sn , n ) : return { P ( Si ) for i∈ [ n ] } def sample_ensemble ( P , S , n ) : S ← sample_iid ( Sn ) return train_ensemble ( P , S , n ) Algorithm 2 : Selective Ensemble Prediction def ensemble_predict ( ĝn ( P , S ) , α , x ) : Y ← ∑ h∈ĝn ( P , S ) one_hot ( h ( x ) ) nA , nB ← top_2 ( Y ) if binom_p_value ( nA , nA+nB , 0.5 ) ≤α then return argmax ( Y ) else return ABSTAIN Note that while gP , S is deterministic , and is therefore not sensitive to a specific state drawn from S , it does not necessarily produce the ground truth label on all inputs—some learning pipelines may converge to a stable loss minimum that misclassifies certain points . Approximation via Ensembling . An explicit representation of the true mode predictor is , of course , unattainable—the non-convex loss surface of deep models and the complex interactions between the learning pipeline and the distribution of random states makes the expectation in Equation 1 infeasible to compute analytically . However , we can approximate gP , S ( x ) by computing the empirical mode prediction on x over a random sample of models produced by i.i.d . draws from P ( S ) . But although ensembles with sufficiently many constituent models will more reliably output the mode prediction , for any fixed-size ensemble there will remain points on which the margin of the plurality vote is small enough to “ flip ” to runner-up in some set of nearby ensembles that differ on a subset of their constituents ; in other words , these ensembles will not predict the mode prediction . To rigorously bound the rate at which the ensemble will differ from the mode prediction , we allow the ensemble to abstain on points where the constituent predictions indicate a statistical toss-up between the two most likely classes . We call ensembles that may abstain in this way selective ensembles , borrowing the terminology from selective classification ( El-Yaniv et al. , 2010 ) . We can think of of abstention as a means of flagging unstable points on which the selective ensemble can not accurately determine the mode prediction ; whether this should be interpreted as a failed attempt at classification is an application-specific consideration . Selective ensembles of n models predict according to the following procedure . First , the predictions of each of the n models in the ensemble are collected . The constituent models are derived from n i.i.d . samples of P ( S ) from S , as described in Algorithm 1 . From these predictions , we perform a two-sided statistical test to determine if the mode prediction was selected by a statistically significant majority of the constituent models . If the statistical test succeeds , we return the empirical mode prediction ; otherwise we abstain from predicting . Pseudocode for this prediction procedure is given in Algorithm 2 . We will denote by ĝn ( P , S ) ( for S∼Sn ) the output of train_ensemble in Algorithm 1 , and by ĝn ( P , S ; α , x ) prediction produced by ensemble_predict in Algorithm 2 on ĝn ( P , S ) . Because of their ability to abstain from prediction , we can prove that with probability at least 1−α , a selective ensemble will either return the true mode prediction or abstain , where α is a chosen threshold for the statistical test to prevent prediction in the case of a toss-up . In other words , on any point on which it does not abstain , a selective ensemble will disagree with the mode predictor , gP , S , with probability at most α , as stated formally in Theorem 4.1 . The statement of Theorem 4.1 make use of the relation , ABS6= , where y1 ABS6= y2 if and only if y1 6=ABSTAIN and y2 6=ABSTAIN and y1 6=y2 . That is , ABS6= captures disagreement between non-rejected predictions . Theorem 4.1 . Let P be a learning pipeline , and let S be a distribution over random states . Further , let gP , S be the mode predictor , let ĝn ( P , S ) for S∼Sn be a selective ensemble , and let α≥0 . Then , ∀x∈X . Pr S∼Sn [ ĝn ( P , S ; α , x ) ABS6=gP , S ( x ) ] ≤α The proof ( Appendix A ) relies on a result from Hung and Fithian ( Hung et al. , 2019 ) which bounds the probability that a set of votes does not return the true plurality outcome , and we apply it in a similar fashion to how it is used for making robust predictions in Randomized Smoothing ( Cohen et al. , 2019 ) . Theorem 4.1 states that the probability that a selective ensemble makes a prediction that does not match the mode prediction is small . However , one possible means of ensuring this is by not providing a prediction in the first place , i.e. , if the selective ensemble abstains . Thus , the abstention rate is necessary to quantify the fraction of points on which the mode prediction will actually be produced . In the 0-1 loss bias-variance decomposition of Domingos ( 2000 ) , the variance component of a classifier ’ s loss is defined as the expected loss relative to the mode prediction ( in our case , taken over the randomness in S ) . Thus , Theorem 4.1 leads to a direct bound on this component , assuming a bound , β , on the abstention rate . This is formalized in Corollary A.3 . Corollary 4.2 . Let P be a learning pipeline , and let S be a distribution over random states . Further , let gP , S be the mode predictor , let ĝn ( P , S ) for S∼Sn be a selective ensemble . Finally , let α≥0 , and let β≥0 be an upper bound on the expected abstention rate of ĝn ( P , S ) . Then , the expected loss variance , V ( x ) , over inputs , x , is bounded by α+β . That is , E x∼D [ V ( x ) ] = E x∼D [ Pr S∼Sn [ ĝn ( P , S ; x ) 6=gP , S ( x ) ] ] ≤α+β Consistency of Selective Ensembles . Using the result from Theorem 4.1 , we can also address the original problem raised : that deep models often disagree on their predictions due to arbitrary random events over the training pipeline . We show that , given a bound , β , on the abstention rate , the probability that two selective ensembles disagree in their predictions is bounded by 2 ( α+β ) ( Corollary A.4 ) . Intuitively , this suggests that the predictions of selective ensembles are more stable over different instantiations of the random decisions captured by S compared to individual models . Corollary 4.3 . Let P be a learning pipeline , and let S be a distribution over random states . Further , let ĝn ( P , S ) for S∼Sn be a selective ensemble . Finally , let α≥0 , and let β≥0 be an upper bound on the expected abstention rate of ĝn ( P , S ) . Then , E x∼D [ Pr S1 , S2∼Sn [ ĝn ( P , S1 ; α , x ) 6= ĝn ( P , S2 ; α , x ) ] ] ≤2 ( α+β ) Corollary A.4 tells us that the agreement between any two selective ensembles is at least 1−2 ( α+β ) . For a fixed n , decreasing α will lead to a higher abstention rate . Thus in order for α and β to both be small , as would be necessary for a high fraction of consistently-predicted points , we may require a large number of constituent models , n. Figure 2 illustrates the trade-off between α , β , and n , depending on the base level of agreement of the constituent models . In Section 5 , we show empirically that even with small values of n , abstention rates of selective ensembles are reasonably low in practice . In summary , selective ensembles accomplish three primary things : ( 1 ) they identify points on which the mode prediction can not be determined , ( 2 ) they bound the fraction of points that can be inconsistently predicted , and ( 3 ) they provide a means of reliably inferring the mode prediction when the abstention rate can be kept sufficiently low .

Summary: In the paper the authors proposed a selective ensemble whose constituents are classifiers with reject (abstention) option. They theoretically exposed the problem that differentiable models yielding identical predictions might remotely share identical gradients, biasing the feature attribution based on (integrated) gradient approaches. They theoretically demonstrated that the selective ensemble can contribute to model stability, and empirically showed that both the model stability and feature attribution could be improved as a result of the proposed method.