Title: Explanation Shift: How Did Distribution Shift Impact the Model?
Venue: NeurIPS 2023
Accepted: no
Keywords: Model Monitoring

Distribution Shift

Explainable AI
Abstract: The performance of machine learning models on new data is critical for their success in real-world applications. However, the model's performance may deteriorate if the new data is sampled from a different distribution than the training data. Current methods to detect shifts in the input or output data distributions have limitations in identifying model behavior changes. In this paper, we define \emph{explanation shift} as the statistical comparison between how predictions from training data are explained and how predictions on new data are explained. We propose explanation shift as a key indicator to investigate the interaction between distribution shifts and learned models.  We introduce an Explanation Shift Detector that operates on the explanation distributions, providing more sensitive and explainable changes in interactions between distribution shifts and learned models. We compare explanation shifts with other methods based on distribution shifts, showing that monitoring for explanation shifts results in more sensitive indicators for varying model behavior. We provide theoretical and experimental evidence and demonstrate the effectiveness of our approach on synthetic and real data. Additionally, we release an open-source Python package, skshift, which implements our method and provides usage tutorials for further reproducibility.

Abstract

The performance of machine learning models on new data is critical for their success in real-world applications. However, the model's performance may deteriorate if the new data is sampled from a different distribution than the training data. Current methods to detect shifts in the input or output data distributions have limitations in identifying model behavior changes. In this paper, we define explanation shift as the statistical comparison between how predictions from training data are explained and how predictions on new data are explained. We propose explanation shift as a key indicator to investigate the interaction between distribution shifts and learned models. We introduce an Explanation Shift Detector that operates on the explanation distributions, providing more sensitive and explainable changes in interactions between distribution shifts and learned models. We compare explanation shifts with other methods based on distribution shifts, showing that monitoring for explanation shifts results in more sensitive indicators for varying model behavior. We provide theoretical and experimental evidence and demonstrate the effectiveness of our approach on synthetic and real data. Additionally, we release an open-source Python package, skshift , which implements our method and provides usage tutorials for further reproducibility.

1 Introduction

ML theory provides means to forecast the quality of ML models on unseen data, provided that this data is sampled from the same distribution as the data used to train and evaluate the model. If unseen data is sampled from a different distribution than the training data, model quality may deteriorate, making monitoring how the model's behavior changes crucial.

Recent research has highlighted the impossibility of reliably estimating the performance of machine learning models on unseen data sampled from a different distribution in the absence of further assumptions about the nature of the shift [1, 2, 3]. State-of-the-art techniques attempt to model statistical distances between the distributions of the training and unseen data [4, 5] or the distributions of the model predictions [3, 6, 7]. However, these measures of distribution shifts only partially relate to changes of interaction between new data and trained models or they rely on the availability of a causal graph or types of shift assumptions, which limits their applicability. Thus, it is often necessary to go beyond detecting such changes and understand how the feature attribution changes [8, 9, 10, 4].

The field of explainable AI has emerged as a way to understand model decisions [11, 12] and 31 interpret the inner workings of ML models [13]. The core idea of this paper is to go beyond the 32 modeling of distribution shifts and monitor for explanation shifts to signal a change of interactions 33 between learned models and dataset features in tabular data. We newly define explanation shift as the 34 statistical comparison between how predictions from training data are explained and how predictions 35 on new data are explained. In summary, our contributions are: 36

37

38

39

40

41

42

43

44

45

46

47

48

49

50

51

52

53

54

55

56

57

58

59

60

61

62

63

64

65

66

67

68

69

70

71

72

73

74

75

76

77

78

79

80

81

We propose measures of explanation shifts as a key indicator for investigating the interaction between distribution shifts and learned models.

We define an Explanation Shift Detector that operates on the explanation distributions allowing for more sensitive and explainable changes of interactions between distribution shifts and learned models.

We compare our monitoring method that is based on explanation shifts with methods that are based on other kinds of distribution shifts. We find that monitoring for explanation shifts results in more sensitive indicators for varying model behavior.

We release an open-source Python package skshift , which implements our ' Explanation Shift Detector ', along usage tutorials for reproducibility.

2 Foundations and Related Work

2.1 Basic Notions

Supervised machine learning induces a function f θ : dom ( X ) → dom ( Y ) , from training data D tr = { ( x tr 0 , y tr 0 ) . . . , ( x tr n , y tr n ) } . Thereby, f θ is from a family of functions f θ ∈ F and D tr is sampled from the joint distribution P ( X,Y ) with predictor variables X and target variable Y . f θ is expected to generalize well on new, previously unseen data D new X = { x new 0 , . . . , x new k } ⊆ dom ( X ) . We write D tr X to refer to { x tr 0 , . . . , x tr n } and D tr Y to refer to D tr Y = { y tr 0 . . . , y tr n } . For the purpose of formalizations and to define evaluation metrics, it is often convenient to assume that an oracle provides values D new Y = { y new 0 , . . . , y new k } such that D new = { ( x new 0 , y new 0 ) , . . . , ( x new k , y new k ) } ⊆ dom ( X ) × dom ( Y ) .

The core machine learning assumption is that training data D tr and novel data D new are sampled from the same underlying distribution P ( X,Y ) . The twin problems of model monitoring and recognizing that new data is out-of-distribution can now be described as predicting an absolute or relative performance drop between perf ( D tr ) and perf ( D new ) , where perf ( D ) = ∑ ( x,y ) ∈D ℓ eval ( f θ ( x ) , y ) , ℓ eval is a metric like 0-1-loss (accuracy), but D new Y is unknown and cannot be used for such judgment.

Therefore related work analyses distribution shifts between training and newly occurring data. Let two datasets D , D ′ define two empirical distributions P ( D ) , P ( D ′ ) , then we write P ( D ) ̸∼ P ( D ′ ) to express that P ( D ) is sampled from a different underlying distribution than P ( D ′ ) with high probability p > 1 -ϵ allowing us to formalize various types of distribution shifts.

Definition 2.1 (Data Shift) . We say that data shift occurs from D tr to D new X , if P ( D tr X ) ̸∼ P ( D new X ) .

Specific kinds of data shift are:

Definition 2.2 (Univariate data shift) . There is a univariate data shift between P ( D tr X ) = P ( D tr X 1 , . . . , D tr X p ) and P ( D new X ) = P ( D new X 1 , . . . , D new X p ) , if ∃ i ∈ { 1 . . . p } : P ( D tr X i ) ̸∼ P ( D new X i ) .

Definition 2.3 (Covariate data shift) . There is a covariate data shift between P ( D tr X ) = P ( D tr X 1 , . . . , D tr X p ) and P ( D new X ) = P ( D new X 1 , . . . , D new X p ) if P ( D tr X ) ̸∼ P ( D new X ) , which cannot only be caused by univariate shift.

The next two types of shift involve the interaction of data with the model

P

P

(

(

D

D

tr tr

X

)

)

θ

, which approximates the

f

θ

to refer to the multiset

{

f

θ

Definition 2.4 (Predictions Shift) . There is a predictions shift between distributions P ( D tr X ) and P ( D new X ) related to model f θ if P ( f θ ( D tr X )) ̸∼ P ( f θ ( D new X )) .

Definition 2.5 (Concept Shift) . There is a concept shift between P ( D tr ) = P ( D tr X , D tr Y ) and P ( D new ) = P ( D new X , D new Y ) if conditional distributions change, i.e. P ( D tr ) P ( D tr X ) ̸∼ P ( D new ) P ( D new X ) .

In practice, multiple types of shifts co-occur together and their disentangling may constitute a significant challenge that we do not address here [14, 15].

2.2 Related Work on Tabular Data

We briefly review the related works below. See Appendix A for a more detailed related work. 82

(

x

)

|

x

∈ D}

.

conditional

. Abusing notation, we write

f

(

D

)

83

84

85

86

87

88

89

90

91

92

93

94

95

96

97

98

99

100

101

102

103

104

105

106

107

108

109

110

111

112

113

114

115

116

117

118

119

120

121

122

123

124

125

126

127

128

129

130

131

Classifier two-sample test: Evaluating how two distributions differ has been a widely studied topic in the statistics and statistical learning literature [16, 15, 17] and has advanced in recent years [18, 19, 20]. The use of supervised learning classifiers to measure statistical tests has been explored by Lopez-Paz et al. [21] proposing a classifier-based approach that returns test statistics to interpret differences between two distributions. We adopt their power test analysis and interpretability approach but apply it to the explanation distributions.

Detecting distribution shift and its impact on model behaviour: A lot of related work has aimed at detecting that data is from out-of-distribution. To this end, they have created several benchmarks that measure whether data comes from in-distribution or not [22, 23, 24, 25, 26]. In contrast, our main aim is to evaluate the impact of the distribution shift on the model.

A typical example is two-sample testing on the latent space such as described by Rabanser et al. [27]. However, many of the methods developed for detecting out-of-distribution data are specific to neural networks processing image and text data and can not be applied to traditional machine learning techniques. These methods often assume that the relationships between predictor and response variables remain unchanged, i.e., no concept shift occurs. Our work is applied to tabular data where techniques such as gradient boosting decision trees achieve state-of-the-art model performance [28, 29, 30].

Impossibility of model monitoring: Recent research findings have formalized the limitations of monitoring machine learning models in the absence of labelled data. Specifically [3, 31] prove the impossibility of predicting model degradation or detecting out-of-distribution data with certainty [32, 33, 34]. Although our approach does not overcome these limitations, it provides valuable insights for machine learning engineers to understand better changes in interactions resulting from shifting data distributions and learned models.

Model monitoring and distribution shift under specific assumptions: Under specific types of assumptions, model monitoring and distribution shift become feasible tasks. One type of assumption often found in the literature is to leverage causal knowledge to identify the drivers of distribution changes [35, 36, 37]. For example, Budhathoki et al. [35] use graphical causal models and feature attributions based on Shapley values to detect changes in the distribution. Similarly, other works aim to detect specific distribution shifts, such as covariate or concept shifts. Our approach does not rely on additional information, such as a causal graph, labelled test data, or specific types of distribution shift. Still, by the nature of pure concept shifts, the model behaviour remains unaffected and new data need to come with labelled responses to be detected.

Explainability and distribution shift: Lundberg et al. [38] applied Shapley values to identify possible bugs in the pipeline by visualizing univariate SHAP contributions. In our work we go beyond debugging and formalize the multivariate explanation distributions where we perform a two-sample classifier test to detect distribution shift impacts on the model. Furthermore, we provide a mathematical analysis of how the SHAP values contribute to detecting distribution shift.

2.3 Explainable AI: Local Feature Attributions

Attribution by Shapley values explains machine learning models by determining the relevance of features used by the model [38, 39]. The Shapley value is a concept from coalition game theory that aims to allocate the surplus generated by the grand coalition in a game to each of its players [40]. The Shapley value S j for the j 'th player is defined via a value function val : 2 N → R of players in T :

In machine learning, N = { 1 , . . . , p } is the set of features occurring in the training data. Given that x is the feature vector of the instance to be explained, and the term val f,x ( T ) represents the prediction for the feature values in T that are marginalized over features that are not included in T :

The Shapley value framework satisfies several theoretical properties [12, 40, 41, 42]. Our approach is based on the efficiency and uninformative properties:

132

133

134

135

136

137

138

139

140

141

142

143

144

145

146

147

148

149

150

151

152

153

154

155

156

157

158

159

160

161

162

163

164

165

Efficiency Property. Feature contributions add up to the difference of prediction from x ⋆ and the expected value:

Uninformativeness Property. A feature j that does not change the predicted value has a Shapley value of zero.

Our approach works with explanation techniques that fulfill efficiency and uninformative properties, and we use Shapley values as an example. It is essential to distinguish between the theoretical Shapley values and the different implementations that approximate them. We use TreeSHAP as an efficient implementation for tree-based models of Shapley values [38, 12, 43], mainly we use the observational (or path-dependent) estimation [44, 45, 46], and for linear models, we use the correlation dependent implementation that takes into account feature dependencies [47].

LIME is another explanation method candidate for out approach [48, 49]. LIME computes local feature attributions and also satisfies efficiency and uninformative properties, at least in theoretical aspects. However, the definition of neighborhoods in LIME and corresponding computational expenses impact its applicability. In Appendix F, we analyze LIME's relationship with Shapley values for the purpose of describing explanation shifts.

3 A Model for Explanation Shift Detection

Our model for explanation shift detection is sketched in Fig. 1. We define it step-by-step as follows:

Definition 3.1 (Explanation distribution) . An explanation function S : F × dom ( X ) → R p maps a model f θ and data x ∈ R p to a vector of attributions S ( f θ , x ) ∈ R p . We call S ( f θ , x ) an explanation. We write S ( f θ , D ) to refer to the empirical explanation distribution generated by {S ( f θ , x ) | x ∈ D} .

We use local feature attribution methods SHAP and LIME as explanation functions S .

Definition 3.2 (Explanation shift) . Given a model f θ learned from D tr , explanation shift with respect to the model f θ occurs if S ( f θ , D new X ) ̸∼S ( f θ , D tr X ) .

Definition 3.3 (Explanation shift metrics) . Given a measure of statistical distances d , explanation shift is measured as the distance between two explanations of the model f θ by d ( S ( f θ , D tr X ) , S ( f θ , D new X )) .

We follow Lopez et al. [21] to define an explanation shift metrics based on a two-sample test classifier. We proceed as depicted in Figure 1. To counter overfitting, given the model f θ trained on D tr , we compute explanations {S ( f θ , x ) | x ∈ D val X } on an in-distribution validation data set D val X . Given a dataset D new X , for which the status of in- or out-of-distribution is unknown, we compute its explanations {S ( f θ , x ) | x ∈ D new X } . Then, we construct a two-samples dataset E = { ( S ( f θ , x ) , a x ) | x ∈ D val X , a x = 0 } ∪ { ( S ( f θ , x ) , a x ) | x ∈ D new X , a x = 1 } and we train a discrimination model g ψ : R p →{ 0 , 1 } on E , to predict if an explanation should be classified as in-distribution (ID) or out-of-distribution (OOD):

where ℓ is a classification loss function (e.g. cross-entropy). g ψ is our two-sample test classifier, 166 based on which AUC yields a test statistic that measures the distance between the D tr X explanations 167 and the explanations of new data D new X . 168

Explanation shift detection allows us to detect that a novel dataset D new changes the model's behavior. 169 Beyond recognizing explanation shift, using feature attributions for the model g ψ , we can interpret 170 how the features of the novel dataset D new X interact differently with model f θ than the features of the 171 validation dataset D val X . These features are to be considered for model monitoring and for classifying 172 new data as out-of-distribution. 173

174

175

176

177

178

179

180

181

182

183

184

185

186

187

188

189

190

191

192

193

194

195

196

197

198

199

200

201

202

203

Explanations

Train Classifier for

Two-Sample Test

No Explanation Shift

Explanation Shift

Explain

Explanation Shift Detector

Not

Reject

Reject

Explanations

Train Classifier

4 Relationships between Common Distribution Shifts and Explanation Shifts

This section analyses and compares data shifts, prediction shifts, with explanation shifts. Appendix B extends this analysis, and Appendix C draws from these analyses to derive experiments with synthetic data.

4.1 Explanation Shift vs Data Shift

One type of distribution shift that is challenging to detect comprises cases where the univariate distributions for each feature j are equal between the source D tr X and the unseen dataset D new X , but where interdependencies among different features change. Multi-covariance statistical testing is a hard taks with high sensitivity that can lead to false positives. The following example demonstrates that Shapley values account for co-variate interaction changes while a univariate statistical test will provide false negatives.

Example 4.1. ( Covariate Shift ) Let D tr ∼ N ([ µ 1 µ 2 ] , [ σ 2 X 1 0 0 σ 2 X 2 ]) × Y . We fit a linear model f θ ( x 1 , x 2 ) = γ + a · x 1 + b · x 2 . If D new X ∼ N ([ µ 1 µ 2 ] , [ σ 2 X 1 ρσ X 1 σ X 2 ρσ X 1 σ X 2 σ 2 X 2 ]) , then P ( D tr X 1 ) and P ( D tr X 2 ) are identically distributed with P ( D new X 1 ) and P ( D new X 2 ) , respectively, while this does not hold for the corresponding S j ( f θ , D tr X ) and S j ( f θ , D new X ) .

The detailed analysis of example 4.1 is given in Appendix B.2.

False positives frequently occur in out-of-distribution data detection when a statistical test recognizes differences between a source distribution and a new distribution, thought the differences do not affect the model behavior [28, 14]. Shapley values satisfy the Uninformativeness property, where a feature j that does not change the predicted value has a Shapley value of 0 (equation 4).

Example 4.2. Shifts on Uninformative Features. Let the random variables X 1 , X 2 be normally distributed with N (0; 1) . Let dataset D tr ∼ X 1 × X 2 × Y tr , with Y tr = X 1 . Thus Y tr ⊥ X 2 . Let D new X ∼ X 1 × X new 2 and X new 2 be normally distributed with N ( µ ; σ 2 ) and µ, σ ∈ R . When f θ is trained optimally on D tr then f θ ( x ) = x 1 . P ( D X 2 ) can be different from P ( D new X 2 ) but S 2 ( f θ , D tr X ) = 0 = S 2 ( f θ , D new X ) .

4.2 Explanation Shift vs Prediction Shift

Analyses of the explanations detect distribution shifts that interact with the model. In particular, if a prediction shift occurs, the explanations produced are also shifted.

̸

̸

Proposition 1. Given a model f θ : D X →D Y . If f θ ( x ′ ) = f θ ( x ) , then S ( f θ , x ′ ) = S ( f θ , x ) .

204

205

206

207

208

209

210

211

212

213

214

215

216

217

218

219

220

221

222

223

224

225

226

227

228

229

230

231

232

233

234

235

236

237

238

239

240

241

242

243

244

245

246

247

248

249

By efficiency property of the Shapley values [47] (equation ((3))), if the prediction between two instances is different, then they differ in at least one component of their explanation vectors.

The opposite direction does not always hold:

Example 4.3. ( Explanation shift not affecting prediction distribution ) Given D tr is generated from ( X 1 × X 2 × Y ) , X 1 ∼ U (0 , 1) , X 2 ∼ U (1 , 2) , Y = X 1 + X 2 + ϵ and thus the optimal model is f ( x ) = x 1 + x 2 . If D new is generated from X new 1 ∼ U (1 , 2) , X new 2 ∼ U (0 , 1) , Y new = X new 1 + X new 2 + ϵ , the prediction distributions are identical f θ ( D tr X ) , f θ ( D new X ) ∼ U (1 , 3) , but explanation distributions are different S ( f θ , D tr X ) ̸∼ S ( f θ , D new X ) , because S i ( f θ , x ) = α i · x i .

Thus, an explanation shift does not always imply a prediction shift.

4.3 Explanation Shift vs Concept Shift

Concept shift comprises cases where the covariates retain a given distribution, but their relationship with the target variable changes (cf. Section 2.1). This example shows the negative result that concept shift cannot be indicated by the detection of explanation shift.

̸

Example 4.4. Concept Shift Let D tr ∼ X 1 × X 2 × Y , and create a synthetic target y tr i = a 0 + a 1 · x i, 1 + a 2 · x i, 2 + ϵ . As new data we have D new X ∼ X new 1 × X new 2 × Y , with y new i = b 0 + b 1 · x i, 1 + b 2 · x i, 2 + ϵ whose coefficients are unknown at prediction stage. With coefficients a 0 = b 0 , a 1 = b 1 , a 2 = b 2 . We train a linear regression f θ : D tr X →D tr Y . Then explanations have the same distribution, P ( S ( f θ , D tr X )) = P ( S ( f θ , D new X )) , input data distribution P ( D tr X ) = P ( D new X ) and predictions P ( f θ ( D tr X )) = P ( f θ ( D new X )) . But there is no guarantee on the performance of f θ on D new X [3]

̸

̸

In general, concept shift cannot be detected because D new Y is unknown [3]. Some research studies have made specific assumptions about the conditional P ( D new ) P ( D new X ) in order to monitor models and detect distribution shift [7, 50].

In Appendix B.2.2, we analyze a situation in which an oracle - hypothetically - provides D new Y .

5 Empirical Evaluation

We perform core evaluations of explanation shift detection methods by systematically varying models f , model parametrizations θ , and input data distributions D X . We complement core experiments described in this section by adding further experimental results in the appendix that (i) add details on experiments with synthetic data (Appendix C), (ii) add experiments on further natural datasets (Appendix D), (iii) exhibit a larger range of modeling choices (Appendix E), and (iv) include LIME as an explanation method (Appendix F). Core observations made in this section will only be confirmed and refined, but not countered in the appendix.

5.1 Baseline Methods and Datasets

Baseline Methods. We compare our method of explanation shift detection (Section 3) with several methods that aim to detect that input data is out-of-distribution: (i) statistical Kolmogorov Smirnov test on input data [27], (ii) classifier drift [51], (iii) prediction shift detection by Wasserstein distance [7], (iv) prediction shift detection by Kolmogorov-Smirnov test[4], and (v) model agnostic uncertainty estimation [10, 52]. Distribution Shift Metrics are scaled between 0 and 1. We also compare against Classifier Two-Sample Test [21] on different distributions as discussed in Section 4, viz. (vi) classifier two-sample test on input distributions ( g ϕ ) and (vii) classifier two-sample test on the predictions distributions ( g Υ ):

Datasets. In the main body of the paper we base our comparisons on the UCI Adult Income dataset [53] and on synthetic data. In the Appendix, we extend experiments to several other datasets, which confirm our findings: ACS Travel Time [54], ACS Employment [54], Stackoverflow dataset [55].

250

251

252

253

254

255

256

257

258

259

260

261

262

263

264

265

266

267

268

269

270

5.2 Experiments on Synthetic Data

Our first experiment on synthetic data showcases the two main contributions of our method: ( i ) being more sensitive than prediction shift and input shift to changes in the model and ( ii ) accounting for its drivers. We first generate a synthetic dataset with a shift similar to the multivariate shift one (cf. Section 4.2). However, we add an extra variable X 3 = N (0 , 1) and generate our target Y = X 1 · X 2 + X 3 , and parametrize the multivariate shift between ρ = r ( X 1 , X 2 ) . We train the f θ on D tr using a gradient boosting decision tree, while for g ψ : S ( f θ , D val X ) →{ 0 , 1 } , we use a logistic regression for both experiments. In Appendix E we benchmark other estimators and detectors.

0.0

0.2

0.4

0.6

0.8

1.0

Correlation coefficient

0.45

0.50

0.55

0.60

0.65

0.70

0.75

0.80

AUC

Sensitivity to Covariate Shift of Classifier Two Sample Test

Explanation Distribution

g

Input Distribution

g

Prediction Distribution

g

0.0

0.2

0.4

0.6

0.8

1.0

Correlation coefficient

0.0

0.2

0.4

0.6

0.8

1.0

Distribution Shift Metrics

Sensitivy to Covariate Shift for Distribution Shift Methods

Explanation Shift

Input KS

Classifier Drift

Pred. KS

Uncertainty by CP

Table 1 and Figure 2 show the results of our approach when learning on different distributions. In our sensitivity experiment, we observed that using the explanation shift led to higher sensitivity towards detecting distribution shift. This is due to the efficiency property of the Shapley values, which decompose f θ ( D X ) into S ( f θ , D X ) . Moreover, we can identify the features that are causing the drift by extracting the coefficients of g ψ , providing global and local explainability.

The right image in Figure 2 compares our approach against Classifier Two Sample Testing for detecting multi-covariate shifts on different distributions. We can see how the explanations distributions have more sensitivity to the others. On the left image, the same experiment against other out-ofdistribution detection methods such statistical differences on the input data (Input KS, Classifier Drift)[51, 4], which are model-independent; uncertainty estimation methods[52, 10, 56], whose effectiveness under specific types of shift is unclear; and statistical changes on the prediction distribution (K-S and Wasserstein Distance) [57, 58, 7], which can detect changes in model but lack sensitivity and accountability of the explanation shift. All metrics produce output scaled between 0 and 1.

5.3 Experiments on Natural Data: Inspecting Explanation Shifts 271

272

273

In the following experiments, we will provide use cases of our approach in two scenarios with natural data: ( i ) novel group distribution shift and ( ii ) geopolitical and temporal shift.

274

275

276

277

278

279

280

5.3.1 Novel Covariate Group

The distribution shift in this experimental set-up relies on the appearance of a new unseen group at the prediction stage (the group feature is not present in the covariates). We vary the ratio of presence of this unseen group in D new X data. As estimators, we use a gradient-boosting decision tree and a logistic regression(just when indicated); we use a logistic regression for the detector. We compare different estimators and detectors' performance in AppendixE.1 for a benchmark and Appendix E.2 for experiments varying hyperparameters.

0.2

0.4

0.6

0.8

1.0

Fraction of data from previously unseen group

0.50

0.55

0.60

0.65

0.70

0.75

0.80

AUC of Explanation Shift Detector

Black

Am-Indian

Asian

Other

Mixed

0.2

0.4

0.6

0.8

1.0

Fraction of data from previously unseen group

0.50

0.55

0.60

0.65

0.70

0.75

0.80

AUC

g

= Explanations,

f

= XGB

g

= Explanations,

f

= Log

g

= Predictions,

f

= XGB

g

= Predictions,

f

= Log

g

= Input,

f

= XGB

g

= Input,

f

= Log

5.3.2 Geopolitical and Temporal Shift 281

0.50

0.55

0.60

0.65

0.70

0.75

AUC

0

50

100

150

200

AUC Kernel Density Distribution

AUC performance of the Explanation Shift Detector

In-Distribution (CA14)

NY18

TX18

HI18

KS18

MN18

PR18

CA18

PR18

KS18

HI18

MN18

TX18

NY18

CA18

PlaceOfBirth

Race

Occupation

WKHP

Relationship

Age

Education

COW

Marital

Sex

0.81

1.1

0.72

0.68

0.11

0.08

0.014

0.072

0.26

0.29

0.33

0.18

0.16

0.015

0.31

0.15

0.27

0.11

0.026

0.092

0.015

0.18

0.2

0.15

0.12

0.19

0.017

0.058

0.11

0.16

0.28

0.14

0.091

0.016

0.099

0.3

0.16

0.19

0.028

0.11

0.057

0.01

0.24

0.15

0.085

0.038

0.087

0.17

0.051

0.096

0.056

0.22

0.068

0.018

0.051

0.018

0.18

0.049

0.019

0.045

0.022

0.13

0.056

0.086

0.03

0.0039

0.018

0.035

0.012

0.015

Feature importance of the Explanation Shift detector (Wasserstein)

10

2

10

1

10

0

In this section, we tackle a geopolitical and temporal distribution shift, for this, we train the model f θ 282 in California in 2014 and evaluate it in the rest of the states in 2018. The model g θ is trained each 283 time on each state using only the D new X in the absence of the label, and a 50/50 random train-test split 284 evaluates its performance. As models, we use a gradient boosting decision tree[59, 60] as estimator 285 f θ , and using logistic regression for the Explanation Shift Detector . 286

We hypothesize that the AUC of the 'Explanation Shift Detector' on new data will be distinct from 287 on ID data due to the OOD model explanations. Figure 4 illustrates the performance of our method 288 on different data distributions, where the baseline is a hold-out set of ID -CA 14 . The AUC for 289

290

291

292

293

294

295

296

297

298

299

300

301

302

303

304

305

306

307

308

309

310

311

312

313

314

315

316

317

318

319

320

321

322

323

324

325

326

327

328

329

330

331

332

333

334

335

CA 18 , where there is only a temporal shift, is the closest to the baseline, and the OOD detection performance is better in the rest of the states. The most disparate state is Puerto Rico (PR18).

Our next objective is to identify the features where the explanations differ between D tr X and D new X data. To achieve this, we compare the distribution of linear coefficients of the detector between ID and New data. We use the Wasserstein distance as a distance measure, where we generate 1000 in-distribution bootstraps using a 63 . 2% sampling fraction from California-14 and 1000 bootstraps from other states in 2018. In the right image of Figure 4, we observe that for PR18, the most crucial feature is the citizenship status 1 .

Furthermore, we conduct an across-task evaluation by comparing the performance of the 'Explanation Shift Detector' on another prediction task in the Appendix D. Although some features are present in both prediction tasks, the weights and importance order assigned by the "Explanation Shift Detector" differ. One of this method's advantages is that it identifies differences in distributions and how they relate to the model.

6 Discussion

In this study, we conducted a comprehensive evaluation of explanation shift by systematically varying models ( f ), model parametrizations ( θ ), feature attribution explanations ( S ), and input data distributions ( D X ). Our objective was to investigate the impact of distribution shift on the model by explanation shift and gain insights into its characteristics and implications.

Our approach cannot detect concept shifts, as concept shift requires understanding the interaction between prediction and response variables. By the nature of pure concept shifts, such changes do not affect the model. To be understood, new data need to come with labelled responses. We work under the assumption that such labels are not available for new data, nor do we make other assumptions; therefore, our method is not able to predict the degradation of prediction performance under distribution shifts. All papers such as [3, 10, 61, 31, 32, 62, 7] that address the monitoring of prediction performance have the same limitation. Only under specific assumptions, e.g., no occurrence of concept shift or causal graph availability, can performance degradation be predicted with reasonable reliability.

The potential utility of explanation shifts as distribution shift indicators that affect the model in computer vision or natural language processing tasks remains an open question. We have used Shapley values to derive indications of explanation shifts, but other AI explanation techniques may be applicable and come with their advantages.

7 Conclusions

Commonly, the problem of detecting the impact of the distribution shift on the model has relied on measurements for detecting shifts in the input or output data distributions or relied on assumptions either on the type of distribution shift or causal graphs availability. In this paper, we have provided evidence that explanation shifts can be a more suitable indicator for detecting and identifying distribution shifts' impact in machine learning models. We provide software, mathematical analysis examples, synthetic data, and real-data experimental evaluation. We found that measures of explanation shift can provide more insights than input distribution and prediction shift measures when monitoring machine learning models.

Reproducibility Statement

To ensure reproducibility, we make the data, code repositories, and experiments publicly available 2 . Also, an open-source Python package skshift 3 is attached with methods routines and tutorials. For our experiments, we used default scikit-learn parameters [63]. We describe the system requirements and software dependencies of our experiments. Experiments were run on a 4 vCPU server with 32 GB RAM.

336

337

338

339

340

341

342

343

344

345

346

347

348

349

350

351

352

353

354

355

356

357

358

359

360

361

362

363

364

365

366

367

368

369

370

371

372

373

374

375

376

377

378

379

380

381

382

383

References

Shai Ben-David, Tyler Lu, Teresa Luu, and Dávid Pál. Impossibility theorems for domain adaptation. In Yee Whye Teh and D. Mike Titterington, editors, Proceedings of the Thirteenth International Conference on Artificial Intelligence and Statistics, AISTATS 2010, Chia Laguna Resort, Sardinia, Italy, May 13-15, 2010 , volume 9 of JMLR Proceedings , pages 129-136. JMLR.org, 2010.

Zachary C. Lipton, Yu-Xiang Wang, and Alexander J. Smola. Detecting and correcting for label shift with black box predictors. In Jennifer G. Dy and Andreas Krause, editors, Proceedings of the 35th International Conference on Machine Learning, ICML 2018, Stockholmsmässan, Stockholm, Sweden, July 10-15, 2018 , volume 80 of Proceedings of Machine Learning Research , pages 3128-3136. PMLR, 2018.

Saurabh Garg, Sivaraman Balakrishnan, Zachary Chase Lipton, Behnam Neyshabur, and Hanie Sedghi. Leveraging unlabeled data to predict out-of-distribution performance. In NeurIPS 2021 Workshop on Distribution Shifts: Connecting Methods and Applications , 2021.

Tom Diethe, Tom Borchert, Eno Thereska, Borja Balle, and Neil Lawrence. Continual learning in practice. ArXiv preprint, https://arxiv.org/abs/1903.05202 , 2019.

Cloudera Fastforward Labs. Inferring concept drift without labeled data. https:// concept-drift.fastforwardlabs.com/ , 2021.

Saurabh Garg, Sivaraman Balakrishnan, Zico Kolter, and Zachary Lipton. Ratt: Leveraging unlabeled data to guarantee generalization. In International Conference on Machine Learning , pages 3598-3609. PMLR, 2021.

Yuzhe Lu, Zhenlin Wang, Runtian Zhai, Soheil Kolouri, Joseph Campbell, and Katia P. Sycara. Predicting out-of-distribution error with confidence optimal transport. In ICLR 2023 Workshop on Pitfalls of limited data and computation for Trustworthy ML , 2023.

Krishnaram Kenthapadi, Himabindu Lakkaraju, Pradeep Natarajan, and Mehrnoosh Sameki. Model monitoring in practice: Lessons learned and open challenges. In Proceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining , KDD '22, page 4800-4801, New York, NY, USA, 2022. Association for Computing Machinery.

Johannes Haug, Alexander Braun, Stefan Zürn, and Gjergji Kasneci. Change detection for local explainability in evolving data streams. In Proceedings of the 31st ACM International Conference on Information & Knowledge Management , pages 706-716, 2022.

Carlos Mougan and Dan Saattrup Nielsen. Monitoring model deterioration with explainable uncertainty estimation via non-parametric bootstrap. In AAAI Conference on Artificial Intelligence , 2023.

Alejandro Barredo Arrieta, Natalia Díaz-Rodríguez, Javier Del Ser, Adrien Bennetot, Siham Tabik, Alberto Barbado, Salvador Garcia, Sergio Gil-Lopez, Daniel Molina, Richard Benjamins, Raja Chatila, and Francisco Herrera. Explainable artificial intelligence (xai): Concepts, taxonomies, opportunities and challenges toward responsible ai. Information Fusion , 58:82-115, 2020.

Christoph Molnar. Interpretable Machine Learning . ., 2019. https://christophm.github. io/interpretable-ml-book/ .

Riccardo Guidotti, Anna Monreale, Salvatore Ruggieri, Franco Turini, Fosca Giannotti, and Dino Pedreschi. A survey of methods for explaining black box models. ACM Comput. Surv. , 51(5), August 2018.

Chip Huyen. Designing Machine Learning Systems: An Iterative Process for Production-Ready Applications . O'Reilly, 2022.

Joaquin Quiñonero-Candela, Masashi Sugiyama, Neil D Lawrence, and Anton Schwaighofer. Dataset shift in machine learning . Mit Press, 2009.

384

385

386

387

388

389

390

391

392

393

394

395

396

397

398

399

400

401

402

403

404

405

406

407

408

409

410

411

412

413

414

415

416

417

418

419

420

421

422

423

424

425

426

427

428

429

430

431

432

433

434

435

436

Trevor Hastie, Robert Tibshirani, and Jerome Friedman. The Elements of Statistical Learning . Springer Series in Statistics. Springer New York Inc., New York, NY, USA, 2001.

Feng Liu, Wenkai Xu, Jie Lu, Guangquan Zhang, Arthur Gretton, and Danica J. Sutherland. Learning deep kernels for non-parametric two-sample tests. In Proceedings of the 37th International Conference on Machine Learning, ICML 2020, 13-18 July 2020, Virtual Event , volume 119 of Proceedings of Machine Learning Research , pages 6316-6326. PMLR, 2020.

Chunjong Park, Anas Awadalla, Tadayoshi Kohno, and Shwetak N. Patel. Reliable and trustworthy machine learning for health using dataset shift detection. In Marc'Aurelio Ranzato, Alina Beygelzimer, Yann N. Dauphin, Percy Liang, and Jennifer Wortman Vaughan, editors, Advances in Neural Information Processing Systems 34: Annual Conference on Neural Information Processing Systems 2021, NeurIPS 2021, December 6-14, 2021, virtual , pages 3043-3056, 2021.

Kimin Lee, Kibok Lee, Honglak Lee, and Jinwoo Shin. A simple unified framework for detecting out-of-distribution samples and adversarial attacks. In Samy Bengio, Hanna M. Wallach, Hugo Larochelle, Kristen Grauman, Nicolò Cesa-Bianchi, and Roman Garnett, editors, Advances in Neural Information Processing Systems 31: Annual Conference on Neural Information Processing Systems 2018, NeurIPS 2018, December 3-8, 2018, Montréal, Canada , pages 7167-7177, 2018.

Kun Zhang, Bernhard Schölkopf, Krikamol Muandet, and Zhikun Wang. Domain adaptation under target and conditional shift. In Proceedings of the 30th International Conference on Machine Learning, ICML 2013, Atlanta, GA, USA, 16-21 June 2013 , volume 28 of JMLR Workshop and Conference Proceedings , pages 819-827. JMLR.org, 2013.

David Lopez-Paz and Maxime Oquab. Revisiting classifier two-sample tests. In 5th International Conference on Learning Representations, ICLR 2017, Toulon, France, April 24-26, 2017, Conference Track Proceedings . OpenReview.net, 2017.

Pang Wei Koh, Shiori Sagawa, Henrik Marklund, Sang Michael Xie, Marvin Zhang, Akshay Balsubramani, Weihua Hu, Michihiro Yasunaga, Richard Lanas Phillips, Irena Gao, Tony Lee, Etienne David, Ian Stavness, Wei Guo, Berton Earnshaw, Imran Haque, Sara M. Beery, Jure Leskovec, Anshul Kundaje, Emma Pierson, Sergey Levine, Chelsea Finn, and Percy Liang. WILDS: A benchmark of in-the-wild distribution shifts. In Marina Meila and Tong Zhang, editors, Proceedings of the 38th International Conference on Machine Learning, ICML 2021, 18-24 July 2021, Virtual Event , volume 139 of Proceedings of Machine Learning Research , pages 5637-5664. PMLR, 2021.

Shiori Sagawa, Pang Wei Koh, Tony Lee, Irena Gao, Sang Michael Xie, Kendrick Shen, Ananya Kumar, Weihua Hu, Michihiro Yasunaga, Henrik Marklund, Sara Beery, Etienne David, Ian Stavness, Wei Guo, Jure Leskovec, Kate Saenko, Tatsunori Hashimoto, Sergey Levine, Chelsea Finn, and Percy Liang. Extending the WILDS benchmark for unsupervised adaptation. CoRR , abs/2112.05090, 2021.

Andrey Malinin, Neil Band, German Chesnokov, Yarin Gal, Mark JF Gales, Alexey Noskov, Andrey Ploskonosov, Liudmila Prokhorenkova, Ivan Provilkov, Vatsal Raina, et al. Shifts: A dataset of real distributional shift across multiple large-scale tasks. arXiv preprint arXiv:2107.07455 , 2021.

Andrey Malinin, Andreas Athanasopoulos, Muhamed Barakovic, Meritxell Bach Cuadra, Mark JF Gales, Cristina Granziera, Mara Graziani, Nikolay Kartashev, Konstantinos Kyriakopoulos, Po-Jui Lu, et al. Shifts 2.0: Extending the dataset of real distributional shifts. arXiv preprint arXiv:2206.15407 , 2022.

Andrey Malinin, Neil Band, Yarin Gal, Mark J. F. Gales, Alexander Ganshin, German Chesnokov, Alexey Noskov, Andrey Ploskonosov, Liudmila Prokhorenkova, Ivan Provilkov, Vatsal Raina, Vyas Raina, Denis Roginskiy, Mariya Shmatova, Panagiotis Tigas, and Boris Yangel. Shifts: A dataset of real distributional shift across multiple large-scale tasks. In Joaquin Vanschoren and Sai-Kit Yeung, editors, Proceedings of the Neural Information Processing Systems Track on Datasets and Benchmarks 1, NeurIPS Datasets and Benchmarks 2021, December 2021, virtual , 2021.

437

438

439

440

441

442

443

444

445

446

447

448

449

450

451

452

453

454

455

456

457

458

459

460

461

462

463

464

465

466

467

468

469

470

471

472

473

474

475

476

477

478

479

480

481

482

483

484

485

486

Stephan Rabanser, Stephan Günnemann, and Zachary C. Lipton. Failing loudly: An empirical study of methods for detecting dataset shift. In Hanna M. Wallach, Hugo Larochelle, Alina Beygelzimer, Florence d'Alché-Buc, Emily B. Fox, and Roman Garnett, editors, Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NeurIPS 2019, December 8-14, 2019, Vancouver, BC, Canada , pages 1394-1406, 2019.

Leo Grinsztajn, Edouard Oyallon, and Gael Varoquaux. Why do tree-based models still outperform deep learning on typical tabular data? In Thirty-sixth Conference on Neural Information Processing Systems Datasets and Benchmarks Track , 2022.

Shereen Elsayed, Daniela Thyssens, Ahmed Rashed, Lars Schmidt-Thieme, and Hadi Samer Jomaa. Do we really need deep learning models for time series forecasting? CoRR , abs/2101.02118, 2021.

Vadim Borisov, Tobias Leemann, Kathrin Seßler, Johannes Haug, Martin Pawelczyk, and Gjergji Kasneci. Deep neural networks and tabular data: A survey, 2021.

Lingjiao Chen, Matei Zaharia, and James Y. Zou. Estimating and explaining model performance when both covariates and labels shift. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho, editors, Advances in Neural Information Processing Systems , 2022.

Zhen Fang, Yixuan Li, Jie Lu, Jiahua Dong, Bo Han, and Feng Liu. Is out-of-distribution detection learnable? In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho, editors, Advances in Neural Information Processing Systems , 2022.

Lily H. Zhang, Mark Goldstein, and Rajesh Ranganath. Understanding failures in out-ofdistribution detection with deep generative models. In Marina Meila and Tong Zhang, editors, Proceedings of the 38th International Conference on Machine Learning, ICML 2021, 18-24 July 2021, Virtual Event , volume 139 of Proceedings of Machine Learning Research , pages 12427-12436. PMLR, 2021.

Joris Guerin, Kevin Delmas, Raul Sena Ferreira, and Jérémie Guiochet. Out-of-distribution detection is not all you need. In NeurIPS ML Safety Workshop , 2022.

Kailash Budhathoki, Dominik Janzing, Patrick Blöbaum, and Hoiyi Ng. Why did the distribution change? In Arindam Banerjee and Kenji Fukumizu, editors, The 24th International Conference on Artificial Intelligence and Statistics, AISTATS 2021, April 13-15, 2021, Virtual Event , volume 130 of Proceedings of Machine Learning Research , pages 1666-1674. PMLR, 2021.

Haoran Zhang, Harvineet Singh, and Shalmali Joshi. 'why did the model fail?': Attributing model performance changes to distribution shifts. In ICML 2022: Workshop on Spurious Correlations, Invariance and Stability , 2022.

Jessica Schrouff, Natalie Harris, Oluwasanmi O Koyejo, Ibrahim Alabdulmohsin, Eva Schnider, Krista Opsahl-Ong, Alexander Brown, Subhrajit Roy, Diana Mincu, Chrsitina Chen, Awa Dieng, Yuan Liu, Vivek Natarajan, Alan Karthikesalingam, Katherine A Heller, Silvia Chiappa, and Alexander D'Amour. Diagnosing failures of fairness transfer across distribution shift in real-world medical settings. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho, editors, Advances in Neural Information Processing Systems , 2022.

Scott M. Lundberg, Gabriel Erion, Hugh Chen, Alex DeGrave, Jordan M. Prutkin, Bala Nair, Ronit Katz, Jonathan Himmelfarb, Nisha Bansal, and Su-In Lee. From local explanations to global understanding with explainable ai for trees. Nature Machine Intelligence , 2(1):25225839, 2020.

Scott M. Lundberg and Su-In Lee. A unified approach to interpreting model predictions. In Isabelle Guyon, Ulrike von Luxburg, Samy Bengio, Hanna M. Wallach, Rob Fergus, S. V. N. Vishwanathan, and Roman Garnett, editors, Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, December 4-9, 2017, Long Beach, CA, USA , pages 4765-4774, 2017.

L. S. Shapley. A Value for n-Person Games , pages 307-318. Princeton University Press, 1953.

Eyal Winter. Chapter 53 the shapley value. In . , volume 3 of Handbook of Game Theory with 487 Economic Applications , pages 2025-2054. Elsevier, 2002. 488

Robert J Aumann and Jacques H Dreze. Cooperative games with coalition structures. Interna489 tional Journal of game theory , 3(4):217-237, 1974. 490

491

492

493

Artjom Zern, Klaus Broelemann, and Gjergji Kasneci. Interventional shap values and interaction values for piecewise linear regression trees. In Proceedings of the AAAI Conference on Artificial Intelligence , 2023.

494

495

496

497

498

499

500

501

502

503

504

505

506

507

508

509

510

511

512

513

514

515

516

517

518

519

520

521

522

523

524

525

526

527

528

529

530

531

532

533

534

Hugh Chen, Ian C. Covert, Scott M. Lundberg, and Su-In Lee. Algorithms to estimate shapley value feature attributions. CoRR , abs/2207.07605, 2022.

Christopher Frye, Colin Rowat, and Ilya Feige. Asymmetric shapley values: incorporating causal knowledge into model-agnostic explainability. In Hugo Larochelle, Marc'Aurelio Ranzato, Raia Hadsell, Maria-Florina Balcan, and Hsuan-Tien Lin, editors, Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual , 2020.

Hugh Chen, Joseph D. Janizek, Scott M. Lundberg, and Su-In Lee. True to the model or true to the data? CoRR , abs/2006.16234, 2020.

Kjersti Aas, Martin Jullum, and Anders Løland. Explaining individual predictions when features are dependent: More accurate approximations to shapley values. Artif. Intell. , 298:103502, 2021.

Marco Túlio Ribeiro, Sameer Singh, and Carlos Guestrin. "why should I trust you?": Explaining the predictions of any classifier. In Balaji Krishnapuram, Mohak Shah, Alexander J. Smola, Charu C. Aggarwal, Dou Shen, and Rajeev Rastogi, editors, Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, San Francisco, CA, USA, August 13-17, 2016 , pages 1135-1144. ACM, 2016.

Marco Tulio Ribeiro, Sameer Singh, and Carlos Guestrin. Model-agnostic interpretability of machine learning, 2016.

Jose M. Alvarez, Kristen M. Scott, Salvatore Ruggieri, and Bettina Berendt. Domain adaptive decision trees: Implications for accuracy and fairness. In Proceedings of the 2023 ACM Conference on Fairness, Accountability, and Transparency . Association for Computing Machinery, 2023.

Arnaud Van Looveren, Janis Klaise, Giovanni Vacanti, Oliver Cobb, Ashley Scillitoe, and Robert Samoilescu. Alibi detect: Algorithms for outlier, adversarial and drift detection, 2019.

Byol Kim, Chen Xu, and Rina Foygel Barber. Predictive inference is free with the jackknife+after-bootstrap. In Hugo Larochelle, Marc'Aurelio Ranzato, Raia Hadsell, Maria-Florina Balcan, and Hsuan-Tien Lin, editors, Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual , 2020.

Dheeru Dua and Casey Graff. UCI machine learning repository, 2017.

Frances Ding, Moritz Hardt, John Miller, and Ludwig Schmidt. Retiring adult: New datasets for fair machine learning. In Marc'Aurelio Ranzato, Alina Beygelzimer, Yann N. Dauphin, Percy Liang, and Jennifer Wortman Vaughan, editors, Advances in Neural Information Processing Systems 34: Annual Conference on Neural Information Processing Systems 2021, NeurIPS 2021, December 6-14, 2021, virtual , pages 6478-6490, 2021.

Stackoverflow. Developer survey results 2019, 2019.

Joseph D Romano, Trang T Le, William La Cava, John T Gregg, Daniel J Goldberg, Praneel Chakraborty, Natasha L Ray, Daniel Himmelstein, Weixuan Fu, and Jason H Moore. Pmlb v1.0: an open source dataset collection for benchmarking machine learning methods. arXiv preprint arXiv:2012.00058v2 , 2021.

535

536

537

538

539

540

541

542

543

544

545

546

547

548

549

550

551

552

553

554

555

556

557

558

559

560

561

562

563

564

565

566

567

568

569

570

571

572

573

574

575

576

577

578

579

580

581

582

583

584

585

586

587

Stanislav Fort, Jie Ren, and Balaji Lakshminarayanan. Exploring the limits of out-of-distribution detection. Advances in Neural Information Processing Systems , 34, 2021.

Saurabh Garg, Yifan Wu, Sivaraman Balakrishnan, and Zachary Lipton. A unified view of label shift estimation. In H. Larochelle, M. Ranzato, R. Hadsell, M. F. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems , volume 33, pages 3290-3300. Curran Associates, Inc., 2020.

Tianqi Chen and Carlos Guestrin. XGBoost: A scalable tree boosting system. In Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining , KDD '16, pages 785-794, New York, NY, USA, 2016. ACM.

Liudmila Ostroumova Prokhorenkova, Gleb Gusev, Aleksandr Vorobev, Anna Veronika Dorogush, and Andrey Gulin. Catboost: unbiased boosting with categorical features. In Samy Bengio, Hanna M. Wallach, Hugo Larochelle, Kristen Grauman, Nicolò Cesa-Bianchi, and Roman Garnett, editors, Advances in Neural Information Processing Systems 31: Annual Conference on Neural Information Processing Systems 2018, NeurIPS 2018, December 3-8, 2018, Montréal, Canada , pages 6639-6649, 2018.

Christina Baek, Yiding Jiang, Aditi Raghunathan, and J Zico Kolter. Agreement-on-the-line: Predicting the performance of neural networks under distribution shift. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho, editors, Advances in Neural Information Processing Systems , 2022.

John Miller, Rohan Taori, Aditi Raghunathan, Shiori Sagawa, Pang Wei Koh, Vaishaal Shankar, Percy Liang, Yair Carmon, and Ludwig Schmidt. Accuracy on the line: on the strong correlation between out-of-distribution and in-distribution generalization. In Marina Meila and Tong Zhang, editors, Proceedings of the 38th International Conference on Machine Learning, ICML 2021, 18-24 July 2021, Virtual Event , volume 139 of Proceedings of Machine Learning Research , pages 7721-7735. PMLR, 2021.

Fabian Pedregosa, Gaël Varoquaux, Alexandre Gramfort, Vincent Michel, Bertrand Thirion, Olivier Grisel, Mathieu Blondel, Peter Prettenhofer, Ron Weiss, Vincent Dubourg, et al. Scikitlearn: Machine learning in python. the Journal of machine Learning research , 12:2825-2830, 2011.

Dan Hendrycks and Kevin Gimpel. A baseline for detecting misclassified and out-of-distribution examples in neural networks. In 5th International Conference on Learning Representations, ICLR 2017, Toulon, France, April 24-26, 2017, Conference Track Proceedings . OpenReview.net, 2017.

Jie Ren, Peter J. Liu, Emily Fertig, Jasper Snoek, Ryan Poplin, Mark A. DePristo, Joshua V. Dillon, and Balaji Lakshminarayanan. Likelihood ratios for out-of-distribution detection. In Hanna M. Wallach, Hugo Larochelle, Alina Beygelzimer, Florence d'Alché-Buc, Emily B. Fox, and Roman Garnett, editors, Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NeurIPS 2019, December 8-14, 2019, Vancouver, BC, Canada , pages 14680-14691, 2019.

Weitang Liu, Xiaoyun Wang, John D. Owens, and Yixuan Li. Energy-based out-of-distribution detection. In Hugo Larochelle, Marc'Aurelio Ranzato, Raia Hadsell, Maria-Florina Balcan, and Hsuan-Tien Lin, editors, Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual , 2020.

Haoran Wang, Weitang Liu, Alex Bocchieri, and Yixuan Li. Can multi-label classification networks know what they don't know? In Marc'Aurelio Ranzato, Alina Beygelzimer, Yann N. Dauphin, Percy Liang, and Jennifer Wortman Vaughan, editors, Advances in Neural Information Processing Systems 34: Annual Conference on Neural Information Processing Systems 2021, NeurIPS 2021, December 6-14, 2021, virtual , pages 29074-29087, 2021.

Rui Huang, Andrew Geng, and Yixuan Li. On the importance of gradients for detecting distributional shifts in the wild. Advances in Neural Information Processing Systems 34: Annual Conference on Neural Information Processing Systems 2021, NeurIPS 2021 , abs/2110.00218, 2021.

588

589

590

591

592

593

594

595

596

597

598

599

600

601

602

603

604

605

606

607

608

609

610

611

612

613

614

615

Chunjong Park, Anas Awadalla, Tadayoshi Kohno, and Shwetak N. Patel. Reliable and trustworthy machine learning for health using dataset shift detection. Advances in Neural Information Processing Systems 34: Annual Conference on Neural Information Processing Systems 2021, NeurIPS 2021 , abs/2110.14019, 2021.

Chiara Balestra, Bin Li, and Emmanuel Müller. Enabling the visualization of distributional shift using shapley values. In NeurIPS 2022 Workshop on Distribution Shifts: Connecting Methods and Applications , 2022.

Johannes Haug and Gjergji Kasneci. Learning parameter distributions to detect concept drift in data streams. In 2020 25th International Conference on Pattern Recognition (ICPR) , pages 9452-9459. IEEE, 2021.

Yongchan Kwon, Manuel A. Rivas, and James Zou. Efficient computation and analysis of distributional shapley values. In Arindam Banerjee and Kenji Fukumizu, editors, The 24th International Conference on Artificial Intelligence and Statistics, AISTATS 2021, April 13-15, 2021, Virtual Event , volume 130 of Proceedings of Machine Learning Research , pages 793-801. PMLR, 2021.

Amirata Ghorbani and James Y. Zou. Data shapley: Equitable valuation of data for machine learning. In Kamalika Chaudhuri and Ruslan Salakhutdinov, editors, Proceedings of the 36th International Conference on Machine Learning, ICML 2019, 9-15 June 2019, Long Beach, California, USA , volume 97 of Proceedings of Machine Learning Research , pages 2242-2251. PMLR, 2019.

Jianbo Chen, Le Song, Martin J. Wainwright, and Michael I. Jordan. L-shapley and c-shapley: Efficient model interpretation for structured data. In 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019 . OpenReview.net, 2019.

Dylan Slack, Sophie Hilgard, Emily Jia, Sameer Singh, and Himabindu Lakkaraju. Fooling LIME and SHAP: adversarial attacks on post hoc explanation methods. In Annette N. Markham, Julia Powles, Toby Walsh, and Anne L. Washington, editors, AIES '20: AAAI/ACM Conference on AI, Ethics, and Society, New York, NY, USA, February 7-8, 2020 , pages 180-186. ACM, 2020.

Contents 616

658

659

660

661

662

663

664

665

666

667

668

669

670

671

672

673

674

675

676

677

678

679

680

681

682

683

684

A Extended Related Work

This section provides an in-depth review of the related theoretical works that inform our research.

A.1 Out-Of-Distribution Detection

Evaluating how two distributions differ has been a widely studied topic in the statistics and statistical learning literature [16, 15, 17], that have advanced recently in last years [18, 19, 20]. [27] provides a comprehensive empirical investigation, examining how dimensionality reduction and two-sample testing might be combined to produce a practical pipeline for detecting distribution shifts in real-life machine learning systems. Other methods to detect if new data is OOD have relied on neural networks based on the prediction distributions [57, 58]. They use the maximum softmax probabilities/likelihood as a confidence score [64], temperature or energy-based scores [65, 66, 67], they extract information from the gradient space [68], they fit a Gaussian distribution to the embedding, or they use the Mahalanobis distance for out-of-distribution detection [19, 69].

Many of these methods are explicitly developed for neural networks that operate on image and text data, and often they can not be directly applied to traditional ML techniques. For image and text data, one may build on the assumption that the relationships between relevant predictor variables ( X ) and response variables ( Y ) remain unchanged, i.e., that no concept shift occurs. For instance, the essence of how a dog looks remains unchanged over different data sets, even if contexts may change. Thus, one can define invariances on the latent spaces of deep neural models, which are not applicable to tabular data in a likewise manner. For example, predicting buying behavior before, during, and after the COVID-19 pandemic constitutes a conceptual shift that is not amenable to such methods. We focus on such tabular data where techniques such as gradient boosting decision trees achieve state-of-the-art model performance [28, 29, 30].

A.2 Explainability and Distribution Shift

Another approach using Shapley values by Balestra et al. [70] allows for tracking distributional shifts and their impact among for categorical time series using slidSHAP, a novel method for unlabelled data streams. In our work, we define the explanation distributions and exploit its theoretical properties under distribution shift where we perform a two-sample classifier test to detect

Haut et al. [71] track changes in the distribution of model parameter values that are directly related 685 to the input features to identify concept drift early on in data streams. In a more recent paper,Haug 686 et al. [9] also exploits the idea that local changes to feature attributions and distribution shifts are 687 strongly intertwined and uses this idea to update the local feature attributions efficiently. Their work 688 focuses on model retraining and concept shift, in our work the original estimator f θ remains unaltered, 689 and since we are in an unsupervised monitoring scenario we can't detect concept shift see discussion 690 in Section 6 691

B Extended Analytical Examples 692

693

This appendix provides more details about the analytical examples presented in Section 4.1.

B.1 Explanation Shift vs Prediction Shift 694

Proposition 2. Given a model f θ : D X →D Y . If f θ ( x ′ ) = f θ ( x ) , then S ( f θ , x ′ ) = S ( f θ , x ) . 695

̸

̸

̸

̸

Example B.1. Explanation shift that does not affect the prediction distribution Given D tr is generated from ( X 1 , X 2 , Y ) , X 1 ∼ U (0 , 1) , X 2 ∼ U (1 , 2) , Y = X 1 + X 2 + ϵ and thus the model is f ( x ) = x 1 + x 2 . If D new is generated from X new 1 ∼ U (1 , 2) , X new 2 ∼ U (0 , 1) , the prediction distributions are identical f θ ( D tr X ) , f θ ( D new X ) , but explanation distributions are different S ( f θ , D tr X ) = S ( f θ , D new X )

̸

B.2 Explanation Shifts vs Input Data Distribution Shifts

B.2.1 Multivariate Shift

Example B.2. Multivariate Shift Let D tr X = ( D new X 1 , D new X 2 ) ∼ N ([ µ 1 µ 2 ] , [ σ 2 x 1 0 0 σ 2 x 2 ]) , D new X = ( D new X 1 , D new X 2 ) ∼ N ([ µ 1 µ 2 ] , [ σ 2 x 1 ρσ x 1 σ x 2 ρσ x 1 σ x 2 σ 2 x 2 ]) . We fit a linear model f θ ( X 1 , X 2 ) = γ + a · X 1 + b · X 2 . D X 1 and D X 2 are identically distributed with D new X 1 and D new X 2 , respectively, while this does not hold for the corresponding SHAP values S j ( f θ , D tr X ) and S j ( f θ , D val X ) .

̸

̸

696

697

698

699

700

701

702

703

704

705

706

707

708

709

710

711

712

̸

B.2.2 Concept Shift

One of the most challenging types of distribution shift to detect are cases where distributions are equal between source and unseen data-set P ( D tr X ) = P ( D new X ) and the target variable P ( D tr Y ) = P ( D new Y ) and what changes are the relationships that features have with the target P ( D tr Y |D tr X ) = P ( D new Y |D new X ) , this kind of distribution shift is also known as concept drift or posterior shift [14] and is especially difficult to notice, as it requires labeled data to detect. The following example

713

714

715

716

717

718

719

720

721

722

723

724

725

726

727

728

729

compares how the explanations change for two models fed with the same input data and different target relations.

̸

Example B.3. Concept shift Let D X = ( X 1 , X 2 ) ∼ N ( µ, I ) , and D new X = ( X new 1 , X new 2 ) ∼ N ( µ, I ) , where I is an identity matrix of order two and µ = ( µ 1 , µ 2 ) . We now create two synthetic targets Y = a + α · X 1 + β · X 2 + ϵ and Y new = a + β · X 1 + α · X 2 + ϵ . Let f θ be a linear regression model trained on f θ : D X →D Y ) and h ϕ another linear model trained on h ϕ : D new X →D new Y ) . Then P ( f θ ( X )) = P ( h ϕ ( X new )) , P ( X ) = P ( X new ) but S ( f θ , X ) = S ( h ϕ , X ) .

̸

̸

C Further Experiments on Synthetic Data

This experimental section explores the detection of distribution shift on the previous synthetic examples.

C.1 Detecting multivariate shift

Given two bivariate normal distributions D X = ( X 1 , X 2 ) ∼ N ( 0 , [ 1 0 0 1 ]) and D new X =

( X new 1 , X new 2 ) ∼ N ( 0 , [ 1 0 . 2 0 . 2 1 ]) , then, for each feature j the underlying distribution is equally distributed between D X and D new X , ∀ j ∈ { 1 , 2 } : P ( D X j ) = P ( D new X j ) , and what is different are the interaction terms between them. We now create a synthetic target Y = X 1 · X 2 + ϵ with ϵ ∼ N (0 , 0 . 1) and fit a gradient boosting decision tree f θ ( D X ) . Then we compute the SHAP explanation values for S ( f θ , D X ) and S ( f θ , D new X )

Having drawn 50 , 000 samples from both D X and D new X , in Table 2, we evaluate whether changes in 730 the input data distribution or on the explanations are able to detect changes in covariate distribution. 731 For this, we compare the one-tailed p-values of the Kolmogorov-Smirnov test between the input data 732 distribution and the explanations distribution. Explanation shift correctly detects the multivariate 733 distribution change that univariate statistical testing can not detect. 734

735

736

737

738

739

740

741

742

743

744

745

746

747

748

749

750

751

752

753

754

755

756

757

758

759

760

761

762

763

764

765

766

767

768

C.2 Detecting concept shift

As mentioned before, concept shift cannot be detected if new data comes without target labels. If new data is labelled, the explanation shift can still be a useful technique for detecting concept shifts.

Given a bivariate normal distribution D X = ( X 1 , X 2 ) ∼ N (1 , I ) where I is an identity matrix of order two. We now create two synthetic targets Y = X 2 1 · X 2 + ϵ and Y new = X 1 · X 2 2 + ϵ and fit two machine learning models f θ : D X →D Y ) and h Υ : D X →D new Y ) . Now we compute the SHAP values for S ( f θ , D X ) and S ( h Υ , D X )

In Table 3, we see how the distribution shifts are not able to capture the change in the model behavior while the SHAP values are different. The 'Distinct/Not distinct' conclusion is based on the one-tailed p-value of the Kolmogorov-Smirnov test with a 0 . 05 threshold drawn out of 50 , 000 samples for both distributions. As in the synthetic example, in table 3 SHAP values can detect a relational change between D X and D Y , even if both distributions remain equivalent.

C.3 Uninformative features on synthetic data

To have an applied use case of the synthetic example from the methodology section, we create a three-variate normal distribution D X = ( X 1 , X 2 , X 3 ) ∼ N (0 , I 3 ) , where I 3 is an identity matrix of order three. The target variable is generated Y = X 1 · X 2 + ϵ being independent of X 3 . For both, training and test data, 50 , 000 samples are drawn. Then out-of-distribution data is created by shifting X 3 , which is independent of the target, on test data D new X 3 = D te X 3 +1 .

In Table 4, we see how an unused feature has changed the input distribution, but the explanation distributions and performance evaluation metrics remain the same. The 'Distinct/Not Distinct' conclusion is based on the one-tailed p-value of the Kolmogorov-Smirnov test drawn out of 50 , 000 samples for both distributions.

C.4 Explanation shift that does not affect the prediction

In this case we provide a situation when we have changes in the input data distributions that affect the model explanations but do not affect the model predictions due to positive and negative associations between the model predictions and the distributions cancel out producing a vanishing correlation in the mixture of the distribution (Yule's effect 4.2).

We create a train and test data by drawing 50 , 000 samples from a bi-uniform distribution X 1 ∼ U (0 , 1) , X 2 ∼ U (1 , 2) the target variable is generated by Y = X 1 + X 2 where we train our model f θ . Then if out-of-distribution data is sampled from X new 1 ∼ U (1 , 2) , X new 2 ∼ U (0 , 1)

In Table 5, we see how an unused feature has changed the input distribution, but the explanation distributions and performance evaluation metrics remain the same. The 'Distinct/Not Distinct' conclusion is based on the one-tailed p-value of the Kolmogorov-Smirnov test drawn out of 50 , 000 samples for both distributions.

769

770

771

772

773

774

775

776

777

778

779

780

781

782

783

784

785

1

D Further Experiments on Real Data

In this section, we extend the prediction task of the main body of the paper. The methodology used follows the same structure. We start by creating a distribution shift by training the model f θ in California in 2014 and evaluating it in the rest of the states in 2018, creating a geopolitical and temporal shift. The model g θ is trained each time on each state using only the X New in the absence of the label, and its performance is evaluated by a 50/50 random train-test split. As models, we use a gradient boosting decision tree[59, 60] as estimator f θ , approximating the Shapley values by TreeExplainer [38], and using logistic regression for the Explanation Shift Detector .

D.1 ACS Employment

The objective of this task is to determine whether an individual aged between 16 and 90 years is employed or not. The model's performance was evaluated using the AUC metric in different states, except PR18, where the model showed an explanation shift. The explanation shift was observed to be influenced by features such as Citizenship and Military Service. The performance of the model was found to be consistent across most of the states, with an AUC below 0.60. The impact of features such as difficulties in hearing or seeing was negligible in the distribution shift impact on the model. The left figure in Figure 5 compares the performance of the Explanation Shift Detector in different states for the ACS Employment dataset.

0.50

0.55

0.60

0.65

0.70

0.75

0.80

0.85

0.90

AUC

0

50

100

150

200

250

AUC Kernel Density Distribution

AUC performance of the Explanation Shift Detector

In-Distribution (CA14)

NY18

TX18

HI18

KS18

MN18

PR18

CA18

PR18

NY18

HI18

MN18

CA18

KS18

TX18

Citizenship

Relationship

Military

Age

Race

MobilityStat

Education

Disability

Ancestry

Employment

Marital

EaringDiff

VisionDiff

DREM

Sex

NATIVITY

2

0.032

0.041

0.024

0.0093

0.013

0.041

0.23

0.14

0.065

0.043

0.45

0.011

0.28

0.42

0.17

0.11

0.21

0.068

0.11

0.014

0.053

0.19

0.0077

0.25

0.086

0.2

0.064

0.19

0.08

0.24

0.011

0.014

0.068

0.0047

0.42

0.041

0.023

0.058

0.0069

0.022

0.0077

0.067

0.1

0.13

0.024

0.064

0.023

0.018

0.2

0.02

0.012

0.037

0.02

0.06

0.027

0.14

0.023

0.064

0.087

0.0092

0.018

0.012

0.24

0.015

0.016

0.015

0.0098

0.012

0.03

0.21

0.032

0.037

0.0097

0.022

0.0078

0.013

0.054

0.038

0.025

0.03

0.026

0.042

0.02

0.1

0.015

0.022

0.013

0.012

0.007

0.0095

0.02

0.016

0.037

0.013

0.016

0.023

0.037

0.041

0.0094

0.029

0.019

0.012

0.0063

0.011

Feature importance of the Explanation Shift detector (Wasserstein)

10

2

10

1

10

0

Additionally, the feature importance analysis for the same dataset is presented in the right figure in 786 Figure 5. 787

788

D.2 ACS Travel Time

The goal of this task is to predict whether an individual has a commute to work that is longer than 789 +20 minutes. For this prediction task, the results are different from the previous two cases; the state 790 with the highest OOD score is KS 18 , with the 'Explanation Shift Detector' highlighting features as 791 Place of Birth, Race or Working Hours Per Week. The closest state to ID is CA18, where there is 792 only a temporal shift without any geospatial distribution shift. 793

1

0.50

0.55

0.60

0.65

0.70

0.75

0.80

AUC

0

25

50

75

100

125

150

175

200

AUC Kernel Density Distribution

AUC performance of the Explanation Shift Detector

In-Distribution (CA14)

NY18

TX18

HI18

KS18

MN18

PR18

CA18

PR18

HI18

MN18

KS18

NY18

TX18

CA18

Race

Citizenship

PovertyIncome

WorkTravel

Age

Occupation

Relationship

Employment

MobilityStat

Sex

Education

Disability

Marital

0.14

0.48

0.45

0.38

0.11

0.15

0.031

0.89

0.15

0.26

0.23

0.039

0.13

0.0086

0.42

0.13

0.069

0.023

0.074

0.027

0.04

0.11

0.011

0.039

0.016

0.28

0.038

0.034

0.098

0.05

0.027

0.073

0.035

0.033

0.045

0.067

0.11

0.0039

0.02

0.054

0.034

0.0097

0.067

0.084

0.052

0.025

0.023

0.012

0.011

0.031

0.016

0.094

0.044

0.031

0.037

0.015

0.072

0.024

0.031

0.012

0.021

0.0093

0.011

0.022

0.023

0.036

0.033

0.02

0.006

0.0061

0.033

0.056

0.017

0.0033

0.0089

0.0068

0.016

0.04

0.0092

0.015

0.032

0.018

0.014

0.0086

0.033

0.0041

0.026

0.0061

0.012

0.011

0.0098

Feature importance of the Explanation Shift detector (Wasserstein)

10

2

10

1

D.3 ACS Mobility 794

The objective of this task is to predict whether an individual between the ages of 18 and 35 had the 795 same residential address as a year ago. This filtering is intended to increase the difficulty of the 796 prediction task, as the base rate for staying at the same address is above 90% for the population [54]. 797

The experiment shows a similar pattern to the ACS Income prediction task (cf. Section 4), where the 798 inland US states have an AUC range of 0 . 55 -0 . 70 , while the state of PR18 achieves a higher AUC. 799 For PR18, the model has shifted due to features such as Citizenship, while for the other states, it is 800 Ancestry (Census record of your ancestors' lives with details like where they lived, who they lived 801 with, and what they did for a living) that drives the change in the model. 802

As depicted in Figure 7, all states, except for PR18, fall below an AUC of explanation shift detection 803 of 0 . 70 . Protected social attributes, such as Race or Marital status, play an essential role for these 804 states, whereas for PR18, Citizenship is a key feature driving the impact of distribution shift in model. 805

0.50

0.55

0.60

0.65

0.70

0.75

0.80

0.85

0.90

AUC

0

20

40

60

80

100

120

140

160

AUC Kernel Density Distribution

AUC performance of the Explanation Shift Detector

In-Distribution (CA14)

NY18

TX18

HI18

KS18

MN18

PR18

CA18

PR18

MN18

HI18

KS18

NY18

TX18

CA18

Citizenship

Ancestry

Race

Relationship

EmploymentStatus

Education

WKHP

JWMNP

COW

PINCP

Marital

DREM

GCL

VisionDiff

Disability

Age

Military

EaringDiff

Sex

NATIVITY

Employment

1.2

0.15

0.11

0.12

0.034

0.0072

0.019

0.5

0.35

0.21

0.2

0.17

0.033

0.069

0.18

0.24

0.39

0.24

0.17

0.19

0.011

0.32

0.28

0.083

0.31

0.046

0.11

0.02

0.067

0.26

0.079

0.13

0.059

0.046

0.062

0.16

0.029

0.083

0.048

0.17

0.035

0.099

0.36

0.053

0.02

0.037

0.031

0.023

0.018

0.14

0.054

0.062

0.039

0.11

0.02

0.031

0.12

0.048

0.03

0.079

0.039

0.044

0.015

0.008

0.044

0.17

0.0066

0.043

0.053

0.053

0.059

0.09

0.067

0.0072

0.035

0.021

0.029

0.082

0.044

0.024

0.044

0.036

0.039

0.031

0.038

0.011

0.1

0.054

0.01

0.044

0.013

0.083

0.016

0.05

0.047

0.011

0.041

0.0085

0.028

0.068

0.024

0.0071

0.057

0.043

0.025

0.098

0.0076

0.077

0.027

0.018

0.011

0.0055

0.0057

0.046

0.042

0.023

0.01

0.069

0.01

0.02

0.031

0.021

0.025

0.01

0.015

0.0082

0.0095

0.023

0.032

0.0045

0.036

0.011

0.0098

Feature importance of the Explanation Shift detector (Wasserstein)

10

2

10

1

10

0

D.4 StackOverflow Survey Data: Novel Covariate Group 806

This experimental section evaluates the proposed Explanation Shift Detector approach on real-world 807 data under novel group distribution shifts. In this scenario, a new unseen group appears at the 808 prediction stage, and the ratio of the presence of this unseen group in the new data is varied. The 809 estimator used is a gradient-boosting decision tree or logistic regression, and a logistic regression 810 is used for the detector. The results show that the AUC of the Explanation Shift Detector varies 811 depending on the quantification of OOD explanations, and it show more sensitivity w.r.t. to model 812 variations than other state-of-the-art techniques. 813

The dataset used is the StackOverflow annual developer survey has over 70,000 responses from over 814 180 countries examining aspects of the developer experience [55]. The data has high dimensionality, 815 leaving it with +100 features after data cleansing and feature engineering. The goal of this task is to 816 predict the total annual compensation. 817

2

4

6

8

10

0.475

0.500

0.525

0.550

0.575

0.600

0.625

Explanation Shift AUC

Log. Reg. as Explanation Shift Detector

XGB

Decision Tree

Random Forest

2

4

6

8

10

0.45

0.50

0.55

0.60

0.65

0.70

0.75

Explanation Shift AUC

XGB as Explanation Shift Detector

XGB

Decision Tree

Random Forest

E Experiments with Modeling Methods and Hyperparameters

In the next sections, we are going to show the sensitivity or our method to variations of the estimator f , the detector g , and the parameters of the estimator f θ .

As an experimental setup, In the main body of the paper, we have focused on the UCI Adult Income dataset. The experimental setup has been using Gradient Boosting Decision Tree as the original estimator f θ and then as 'Explanation Shift Detector' g ψ a logistic regression. In this section, we extend the experimental setup by providing experiments by varying the types of algorithms for a given experimental set-up: the UCI Adult Income dataset using the Novel Covariate Group Shift for the 'Asian' group with a fraction ratio of 0 . 5 (cf. Section 5).

E.1 Varying Estimator and Explanation Shift Detector

OOD data detection methods based on input data distributions only depend on the type of detector used, being independent of the estimator. OOD Explanation methods rely on both the model and the data. Using explanations shifts as indicators for measuring distribution shifts impact on the model enables us to account for the influencing factors of the explanation shift. Therefore, in this section, we compare the performance of different types of algorithms for explanation shift detection using the same experimental setup. The results of our experiments show that using Explanation Shift enables us to see differences in the choice of the original estimator f θ and the Explanation Shift Detector g ϕ

E.2 Hyperparameters Sensitivity Evaluation

818

819

820

821

822

823

824

825

826

827

828

829

830

831

832

833

834

835

This section presents an extension to our experimental setup where we vary the model complexity by 836 varying the model hyperparameters S ( f θ , X ) . Specifically, we use the UCI Adult Income dataset 837 with the Novel Covariate Group Shift for the 'Asian' group with a fraction ratio of 0 . 5 as described 838 in Section 5. 839

In this experiment, we changed the hyperparameters of the original model: for the decision tree, we 840 varied the depth of the tree, while for the gradient-boosting decision, we changed the number of 841 estimators, and for the random forest, both hyperparameters. We calculated the Shapley values using 842

TreeExplainer [38]. For the Detector choice of model, we compare Logistic Regression and XGBoost 843 models. 844

2

4

6

8

10

0.45

0.50

0.55

0.60

0.65

0.70

Explanation Shift AUC

Log. Reg. as Explanation Shift Detector

XGB

Decision Tree

Random Forest

2

4

6

8

10

0.46

0.48

0.50

0.52

0.54

0.56

0.58

0.60

Explanation Shift AUC

XGB as Explanation Shift Detector

XGB

Decision Tree

Random Forest

The results presented in Figure 9 show the AUC of the Explanation Shift Detector for the ACS Income 845 dataset under novel group shift. We observe that the distribution shift does not affect very simplistic 846 models, such as decision trees with depths 1 or 2. However, as we increase the model complexity, 847 the out-of-distribution data impact on the model becomes more pronounced. Furthermore, when we 848 compare the performance of the Explanation Shift Detector across different models, such as Logistic 849 Regression and Gradient Boosting Decision Tree, we observe distinct differences(note that the y-axis 850 takes different values). 851

852

853

854

In conclusion, the explanation distributions serve as a projection of the data and model sensitive to what the model has learned. The results demonstrate the importance of considering model complexity under distribution shifts.

855

F LIME as an Alternative Explanation Method

Another feature attribution technique that satisfies the aforementioned properties (efficiency and 856 uninformative features Section 2) and can be used to create the explanation distributions is LIME 857 (Local Interpretable Model-Agnostic Explanations). The intuition behind LIME is to create a local 858 interpretable model that approximates the behavior of the original model in a small neighbourhood of 859 the desired data to explain [48, 49] whose mathematical intuition is very similar to the Taylor series. 860

861

862

863

864

865

866

867

868

869

870

871

872

873

874

875

876

877

878

879

880

881

882

883

884

885

886

887

888

889

890

891

892

893

894

In this work, we have proposed explanation shifts as a key indicator for investigating the impact of distribution shifts on ML models. In this section, we compare the explanation distributions composed by SHAP and LIME methods. LIME can potentially suffers several drawbacks:

Computationally Expensive: Its currently implementation is more computationally expensive than current SHAP implementations such as TreeSHAP [38], Data SHAP [72, 73] or Local and Connected SHAP [74], the problem increases when we produce explanations of distributions. Even though implementations might be improved, LIME requires sampling data and fitting a linear model which is a computationally more expensive approach than the aforementioned model-specific approaches to SHAP.

Local Neighborhood: The definition of a local 'neighborhood', which can lead to instability of the explanations. Slight variations of this explanation hyperparameter lead to different local explanations. In [75] the authors showed that the explanations of two very close points can vary greatly.

Dimensionality: LIME requires as a hyperparameter the number of features to use for the local linear approximation. This creates a dimensionality problem as for our method to work, the explanation distributions have to be from the exact same dimensions as the input data. Reducing the number of features to be explained might improve the computational burden.

0.0

0.2

0.4

0.6

0.8

1.0

Correlation coefficient

0.5

0.6

0.7

0.8

AUC Explanation Shift Detector

Sensitivity to Multicovariate Shift

Explanation Shift - SHAP

Explanation Shift - Lime

0.2

0.4

0.6

0.8

1.0

Fraction of OOD data

0.475

0.500

0.525

0.550

0.575

0.600

0.625

0.650

AUC of Explanation Shift Detector

ShapAsian

LimeAsian

Figure 10 compares the explanation distributions generated by LIME and SHAP. The left plot shows the sensitivity of the predicted probabilities to multicovariate changes using the synthetic data experimental setup from Figure 2 in the main body of the paper. The right plot shows the distribution of explanation shifts for a New Covariate Category shift (Asian) in the ASC Income dataset. The performance of OOD explanations detection is similar between the two methods, but LIME suffers from two drawbacks: its theoretical properties rely on the definition of a local neighborhood, which can lead to unstable explanations (false positives or false negatives on explanation shift detection), and its computational runtime required is much higher than that of SHAP (see experiments below).

F.1 Runtime

We conducted an analysis of the runtimes of generating the explanation distributions using the two proposed methods. The experiments were run on a server with 4 vCPUs and 32 GB of RAM. We used shap version 0 . 41 . 0 and lime version 0 . 2 . 0 . 1 as software packages. In order to define the local neighborhood for both methods in this example we use all the data provided as background data. As an estimator, we use an xgboost and compare the results of TreeShap against LIME. When varying the number of samples we use 5 features and while varying the number of features we use 1000 samples.

Figure 11, shows the wall time required for generating explanation distributions using SHAP and 895 LIME with varying numbers of samples and columns. The runtime required of generating an 896 explanation distributions using LIME is much higher than using SHAP, especially when producing 897

100

200

500

1000

5000

10

1

10

0

10

1

Time (s)

method

Tree Shap

Lime

5

10

15

20

25

10

1

10

0

10

1

Time (s)

method

Tree Shap

Lime

Number of samples

Number of features

explanations for distributions. This is due to the fact that LIME requires training a local model for 898 each instance of the input data to be explained, which can be computationally expensive. In contrast, 899 SHAP relies on heuristic approximations to estimate the feature attribution with no need to train a 900 model for each instance. The results illustrate that this difference in computational runtime becomes 901 more pronounced as the number of samples and columns increases. 902

903

904

905

906

We note that the computational burden of generating the explanation distributions can be further reduced by limiting the number of features to be explained, as this reduces the dimensionality of the explanation distributions, but this will inhibit the quality of the explanation shift detection as it won't be able to detect changes on the distribution shift that impact model on those features.

Given the current state-of-the-art of software packages we have used SHAP values due to lower 907 runtime required and that theoretical guarantees hold with the implementations. In the experiments 908 performed in this paper, we are dealing with a medium-scaled dataset with around ∼ 1 , 000 , 000 909 samples and 20 -25 features. Further work can be envisioned on developing novel mathematical 910 analysis and software that study under which conditions which method is more suitable. 911

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces a new concept called \"explanation shift\" for detecting shifts in data distributions with the changes of the attribution distributions on machine learning models. The authors argue that current methods for detecting shifts have limitations in identifying changes in model behavior. Explanation shift provides more sensitive and explainable indicators for these changes. The paper also compares the proposed method against other methods for detecting distribution shifts in both synthetic and real datasets.\n\nSTRENGTHS:\n+ Leveraging the changes of explanations as a manner of detecting the distribution shift is a novel idea.\n+ The authors provide a compreshensive analysis to show the connections between explanation shift and various distribution shifts, which could be helpful for readers to understand how to use explanation shift to detect distribution shift\n\nWEAKNESSES:\n+ The overall presentation is not clear and many key terminologies and notations are not well explained or defined. For example, in Equation (3), what is $x^*$? What is the formal definition of $S(f_{\\theta},x)$? What are the definitions of \"sensitivity\" and \"accountability\" which are used as evaluation metrics in Experiments? The lack of clear presentations of these terms makes me extremely hard to understand the key information in this paper\n+ Although the authors proposed a new concept called \"explanation shift\", the technical contribution is still very limited. First of all, the method proposed for detecting the explanation shift (i.e., Section 3) is very simple. But the authors failed to justify why this is an effective method from the theoretical perspective by comparing it against other methods. \n+  Some key empirical studies are missing. In Section 5.3, the authors evaluate their methods on some real datasets to detect novel group distribution shift and geopolitical and temporal shift. However, the authors did not perform the same experiments by using baseline methods. Thus it is unclear whether those baseline methods can discover the same types of shifts or not. If yes, then what are the benefits of the proposed method? If not, why are those baseline methods unable to find out those shifts?\n\nQUESTIONS:\n+ Some notations and terminologies should be clearly defined. For example,  in Equation (3), what is $x^*$? What is the formal definition of $S(f_{\\theta},x)$? What are the definitions of \"sensitivity\" and \"accountability\" which are used as evaluation metrics in Experiments?\n+ In terms of the experiments on the real datasets, can the baseline methods discover the same types of shifts?\n\nLIMITATIONS:\nNot applicable.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-06-11"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper uses explanation shift as a way to detect different types of distribution shift between the training set and unseen (test) data sets. The method is based on measuring the changes between the explanation provided by an explanation approach such as Shapley values, for the two data sets for a trained model. As such, the two data sets could be statistically similar but appear different from the model’s perspective. Overall, the proposed approach is novel and interesting but the paper needs to be improved.\n\nSTRENGTHS:\nTo the best of my knowledge, this is a novel approach that uses explanation to detection distribution shift. The proposed method is clear and the method seems to be effective in practice.\n\nWEAKNESSES:\nSection 4.1 provides examples where the proposed method works but simple distribution shift evaluation fails. But this does not provide any guarantee whether in general the proposed model is better or not. The same is true for Section 4.3. Section 4.2. provides a disposition but as mentioned by the authors, the prediction shift implies explanation shift, but the opposite is not true. Thus, no conclusion can be ae when there is an explanation shift.  \n\nEven though the authors compared their proposed model with the baselines on the synthetic data set in Section 5.1, they have not done it using any real data sets. The real data set is mainly used to study the sensitivity of the model on the parameters. \n\nThere is lack of consistency in notation used in the paper that makes it more difficult to follow. Notation changes from one section to another, and in some extreme cases from one example to another. Here are some instances:\n1-\tVal function is defined differently in Equation 1 and 2. \n2-\tEquation 3 is not clear and not explained either. What is the expected value is defined on? If it is X, why the notation differs from Equation 2?\n3-\tThere is a sign used in Example 4.2 which is not defined. \n\n\nThe paper benefits from a round of proof-reading.\nLine 143: out approach –> our approach\nLine 182: a hard tasks --> a hard task (the sentence that includes this is also not clear and needs explanation)\nLine 279: AppendixE.1 --> Appendix E.1\n\nQUESTIONS:\nCheck the Weaknesses.\n\nLIMITATIONS:\nNo.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-06"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nDetecting shifts in data distribution between training and deployment is critical for ensuring models function as intended and operate in their domain of applicability. However, detecting such shifts is challenging. In this paper, the authors propose an approach based on techniques from the explainability literature. They define the concept of explanation shift and introduce an Explanation Shift Detector. They validate their approach on a synthetic data and 4 tabular datasets, demonstrating improved performance over a range of baselines.\n\nSTRENGTHS:\nThe paper is well written, the introduction well motivated, and the formalism both precise and easy for the reader to follow. I found the method interesting, and the analysis of explanation shift detailed and informative. I think this work is a meaningful contribution to the literature.\n\nWEAKNESSES:\nThe experiments were only conducted on several, relatively simple, tabular datasets. Demonstrating the method for another modality would strengthen the paper.\n\nPlease see Questions below.\n\nQUESTIONS:\nL115-119 - Related work in explainability. Lundberg et al. is not the only work relating explainability and distributional shift. For example, Crabbe et al (2020) use example-based explanations to detect out-of-distribution samples, while Hinder et al. (2022) use contrasting explanations to explain concept drift. \n\nCrabbé, Jonathan, et al. \"Explaining latent representations with a corpus of examples.\" Advances in Neural Information Processing Systems 34 (2021): 12154-12166.\nHinder, Fabian, et al. \"Contrasting explanation of concept drift.\" 30th European Symposium on Artificial Neural Networks, Computational Intelligence and Machine Learning, ESANN. 2022.\n\nFigure 1 – I think there is a typo in “Explain Explanation Shift Detector” (unclosed bracket).\n\nL238-241 – It would benefit the reader to provide some additional explanation of some of the baseline methods, particularly those which don’t seem to be discussed elsewhere.\n\nL258 – how does Table 1 “show the results of [your] approach”? More generally Table 1 seems like it would fit more naturally with the discussion of related work. \n\nL263,L265 – I think left and right in Figure 2 have been switched. \n\nFigure 2, right – It is stated in the caption that “good indicators should follow a progressive stead positive slope…”, however this is not discussed in the text. I think the discussion would benefit from this, since it might not be immediately obvious to the reader.\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-06"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe submission proposes an approach to improve model monitoring by rather evaluating changes in explanations instead of input features. The authors provide synthetic examples to justify their method and compare it empirically to existing strategies on tabular datasets.\n\nSTRENGTHS:\n- The paper addresses an important topic as effective model monitoring based on unlabeled data only is a relevant problem.\n - Although rather, simplistic the synthetic examples help to get a rough idea about the potential benefits of explanation monitoring.\n - The authors provide code as well as tutorials on how to apply their method to ensure reproducibility.\n\nWEAKNESSES:\n- The theoretical analysis is extremely limited such that the overall assumptions under which the proposed method can be expected to yield actual benefits are too vague. Also, the basic notations section seems a bit inflated. \n - I think the novelty is limited as well. Monitoring feature attributions instead of input data is not new and is already offered by popular ML service providers. See for instance here the functionality implemented by Google (https://cloud.google.com/vertex-ai/docs/model-monitoring/monitor-explainable-ai). I would have also liked to see such an alternative approach to use explanations for monitoring somewhere included in the experiments.\n - The conducted numerical experiments are not sufficient to demonstrate the benefits of the proposed approach. If only considering tabular data I think including more than 3 actual datasets and 4 prediction tasks is necessary to be convincing. This is especially true for methods where rigorous theoretical analysis is challenging. See also the question below for further suggestions. \n - The evaluation section is hard to follow, and lacks formulation of insights derived from the experimental results, e.g., it is unclear what benefits can be derived from the feature importance in Figure 4. Given the lack of baselines and justification for those explanations, it is also not clear if they represent useful insights into the effect of a distribution shift on the model’s behavior.  Also, Table 1 comes out of nowhere and is not described sufficiently.\n - Given the limited theoretical and empirical investigation the submission does in my opinion not make a significant contribution to the field.\n\nQUESTIONS:\nWhy are only observational Shapley values considered? Have there also been conducted experiments based on the interventional Shapley Values? The interventional approach is more applicable in general and has been proven to result in Shapley values that are closer to explaining the true model behavior [1,2]. I would imagine that this should also have an impact on how distribution changes are reflected in the explanations. \n\nHas it also been tried to apply explanations techniques like SHAP and LIME to the two-sample test classifier instead of the presented approach in Figure 4 (right)? This would be much more general. \n\nHave all of the mentioned baseline methods also been applied to the considered real datasets? This is needed for a comprehensive evaluation.\n\nWhy are the investigations only limited to tabular data? Explanations are extremely popular for vision models and even Shapley values can also be approximated quite efficiently when basic feed-forward neural networks are used.  Adebayo et al. [3] evaluate a variety of different explanation methods on vision models for in-domain and out-of-domain instances. Their setup is a little different but maybe the authors can get some inspiration regarding similar experiments for explanation shifts on image data.\n\nExplanations are unstable and might change significantly already for minor input perturbations [4] or other desirable sanity checks that are not satisfied [5]. I wonder whether these phenomena also impact the capabilities of explanation monitoring and would have liked to see a corresponding investigation. \n\n[1] Janzing, Dominik, Lenon Minorics, and Patrick Blöbaum. \"Feature relevance quantification in explainable AI: A causal problem.\" International Conference on artificial intelligence and statistics. PMLR, 2020.\n[2] Chen, Hugh, et al. \"True to the model or true to the data?.\" arXiv preprint arXiv:2006.16234 (2020).\n[3] Adebayo, Julius, et al. \"Debugging Tests for Model Explanations.\" Advances in Neural Information Processing Systems 33 (2020): 700-712.\n[4] Alvarez-Melis, David, and Tommi S. Jaakkola. \"On the robustness of interpretability methods.\" arXiv preprint arXiv:1806.08049 (2018).\n[5] Adebayo, Julius, et al. \"Sanity checks for saliency maps.\" Advances in neural information processing systems 31 (2018).\n\nLIMITATIONS:\nI appreciate the discussion at the end of the paper that hints at some relevant limitations.", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-06"}