Bringing Order to Chaos:
Probing the Disagreement Problem in XAI
Hima Lakkaraju

Motivation
Model understanding is absolutely critical in several domains -particularly those involving high stakes decisions!

2[ Larson et. al. 2016 ]Motivation: Why Model Understanding?
Appointment timeand day are being
used to make theprediction!! These
are irrelevant.

Patient Details

Model Understanding

Model understanding helps assess if and when to trust model
predictions
when making decisions.
Day
CoughTime

Predictive
ModelDoctor

Prediction = COVID risk3

Motivation: Why Model Understanding?
Model Understanding

Patient Data
25, Female, Cold
32, Male, No
31, Male, Cough.
.
.
.

If gender = female,
if ID_num > 200, then sickIf gender = male,
if cold = true and cough = true, then sickThis model is using
irrelevant features when
predicting on female
subpopulation. Thiscannot be approved!

Model understanding allows us to vet models to determine
if they are suitable for deployment in real world.

Predictive
Model

Predictions

Healthy
Sick
Sick.
.
Healthy
HealthySick4

[ Letham and Rudin 2015; Lakkaraju et. al. 2016 ]Achieving Model Understanding
Take 1: Build inherently interpretable predictive models

5[ Ribeiro et. al. 2016, 2018; Lakkaraju et. al. 2019]

Achieving Model Understanding
Take 2: Explain pre-built models in a post-hoc mannerExplainer

6

[ Cireşan et. al. 2012, Caruana et. al. 2006, Frosst et. al. 2017, Stewart 2020]

Inherently Interpretable Models vs.
Post hoc Explanations

In certain settings, accuracy-interpretability trade offs may exist.
In certain settings, you may just have access to a (proprietary) black box.
7

Feature Attribution Based Local Explanations•Local explanations•explain individual predictions of any classifier•

Output feature attributions for individual instances, which capture theeffect/contribution of each feature on the black box prediction•

Examples: LIME, SHAP, Gradient, Gradient times Input, SmoothGrad,
Integrated Gradients

8

Disagreement Problem in XAI: Overview•

Study to understand:•if and how often feature attribution based explanation methods disagreewith each other in practice•

What constitutes disagreement between these explanations, and how toformalize the notion of explanation disagreement based on practitionerinputs?

•How do practitioners resolve explanation disagreement?

Krishna and Han et. al., 2022

9

Practitioner Inputs on Explanation Disagreement•30 minute semi-structured interviews with 25 data scientists•

84% of participants said they often encountered disagreement
between explanation methods•

Characterizing disagreement:••••

Top features are different
Ordering among top features is different
Direction of top feature contributions is different
Relative ordering of features of interest is different
10Practitioner Inputs on Explanation Disagreement•Participants typically characterize explanation disagreement
based on factors such as:•••••

mismatch in top features,feature ordering, anddirections of feature contributions,But NOT on the feature importance values output by different
explanation methods

24 out of 25 participants (96%) in our study opine that featureimportance values output by different explanation methods are
not directly comparable
11

Practitioner Inputs on Explanation Disagreement•Quote: “The values generated by different explanation methods are clearlydifferent. So, I would not characterize disagreement based on that. But, I
would at least want the explanations they output to give me consistentinsights. The explanations should agree on what are the most importantfeatures, the ordering among them and so on for me to derive consistentinsights. But, they don’t!”

12Formalizing the Notion ofExplanation Disagreement (Top K)

13

Formalizing the Notion of Explanation
Disagreement (Features of Interest)
Spearman rank correlation coeﬃcient computed over features of interest

14

Empirical Analysis ofExplanation Disagreement

COMPAS data, NN•

COMPAS data, NNWe carried out empirical analysis with 6 post hoc explanation methods, 4 real world datasets (tabular,NLP, images), 8 model classes, and found several disagreements between explanation methods

15

How do Practitioners Resolve Disagreements?

16

How do Practitioners Resolve Disagreements?
•Online user study where 25 users were shown explanations thatdisagree and asked to make a choice, and explain why•

Practitioners are choosing methods due to:•••

Associated theory or publication time (33%)
Explanations matching human intuition better (32%)
Type of data (23%)•E.g., LIME or SHAP are better for tabular data

17

How do Practitioners Resolve Disagreements?

18

Insights and Moving Forward•

Feature attribution methods often disagree in practice w.r.t. basic
insights, and practitioners adopt ad hoc heuristics to resolve
those disagreements!

•

Why do feature attribution methods disagree?

•

Given that feature attribution methods disagree, which
explanation method should we choose for different kinds of data
and applications?
19Why do Feature Attribution MethodsDisagree?
•

Various feature attribution methods (e.g., LIME, C-LIME,KernelSHAP, Occlusion, Vanilla Gradients, Gradient times Input,
SmoothGrad, Integrated Gradients) are essentially local function
approximations.

•But…
Han et. al., NeurIPS 2022

20

Why do Feature Attribution Methods
Disagree?
•But, they adopt different loss functions, and local neighborhoods

Han et. al., 2022

21

Why Do Feature Attribution Methods
Disagree?
•No Free Lunch Theorem for Explanation Methods: No single
method can perform optimally across all neighborhoods

22Which Method Should We Choose?: Take 1•

A guiding principle based on function approximation: choose amethod which recovers the underlying model when the model isa member of the explanation function class•

For continuous data, use additive continuous noise methods (e.g.
SmoothGrad, Vanilla Gradients, C-LIME) or multiplicative
continuous noise methods (e.g. Integrated Gradients, Gradient xInput). For binary data, use binary noise methods (e.g. LIME,KernelSHAP, Occlusion).
23

Which Method Should We Choose?: Take 2•OpenXAI: open-source framework to readily evaluate and benchmark
post hoc explanation methods•

Systematic, efficient, and reproducible evaluations of post hoc
explanation methods on various datasets•

Assessing reliability of post hoc explanation methods from diverse
perspectives (e.g., faithfulness, stability, fairness)•(Customizable) dashboards to compare existing and new methods
across various datasets easily
Agarwal et. al., NeurIPS 2022; OpenXAI: Towards a Transparent Evaluation of Model Explanations; open-xai.github.io

24

Conclusions and Summary•

Several methods proposed to “explain” machine learning models
in prior research•

Important to characterize these methods, and understand which
methods can be useful under what circumstances•

Critical to bridge the gaps between researchers and practitioners

25

Thank You!
•Email: hlakkaraju@hbs.edu; hlakkaraju@seas.harvard.edu;•Webpage: https://himalakkaraju.github.io•

Course on interpretability and explainability: https://interpretable-ml-class.github.io/•

Multiple tutorials on explaining ML models (ranging from 1 hour to 3 hours): explainml-tutorial.github.io•

Trustworthy ML Initiative: https://www.trustworthyml.org/•Lots of resources and seminar series on topics related to explainability, fairness, adversarial robustness,differential privacy, causality etc.

26