Explainable AI in Industry
AAAI 2020 Tutorial
Freddy Lecue, Krishna Gade, Sahin Cem Geyik,

Krishnaram Kenthapadi, Varun Mithal, Ankur Taly,
Riccardo Guidotti, Pasquale Minervini

https://xaitutorial2020.github.io1

Outline2Agenda●●Part I: Introduction and Motivation○

Motivation, Definitions, Properties, Evaluation○

Challenges for Explainable AI @ Scale

Part II: Explanation in AI (not only Machine Learning!)○

From Machine Learning to Knowledge Representation and Reasoning and Beyond●

Part III: Explainable Machine Learning (from a Machine Learning Perspective)●

Part IV: Explainable Machine Learning (from a Knowledge Graph Perspective)●

Part V: Case Studies from Industry○

Applications, Lessons Learned, and Research Challenges
3

Scope4

AI Adoption: Requirements

Introduction and Motivation

6

Explanation - From a Business Perspective

7

Business to Customer AI

Critical Systems (1)Critical Systems (2)… but not only Critical Systems (1)COMPAS recidivism black bias

… but not only Critical Systems (2)▌… but not only Critical Systems (3)▌Black-box AI creates business risk for Industry

Black-box AI creates confusion and doubt
Can I trust our AI
decisions?
Business Owner

Why I am getting thisdecision?

How do I answer thiscustomer complaint?
Customer Support

Poor Decision

Black-box
AI

How do I monitor anddebug this model?
IT & Operations

How can I get abetter decision?

Is this the best modelthat can be built?
Data Scientists

Are these AI systemdecisions fair?
Internal Audit, Regulators

Explanation - From a Model Perspective

16

Why Explainability: Debug (Mis-)Predictions

Top label: “clog”

Why did the network label thisimage as “clog”?
17Why Explainability: Improve ML Model

18

Credit: Samek, Binder, Tutorial on Interpretable ML, MICCAI’18

Why Explainability: Verify the ML Model / System

19

Credit: Samek, Binder, Tutorial on Interpretable ML, MICCAI’18

Why Explainability: Learn New Insights

20

Credit: Samek, Binder, Tutorial on Interpretable ML, MICCAI’18

Why Explainability: Learn Insights in the Sciences

21

Credit: Samek, Binder, Tutorial on Interpretable ML, MICCAI’18

Explanation - From a Regulatory Perspective

22

Why Explainability: Laws against Discrimination
Citizenship

Sex

Race

Age

Disability status

And more...

23

24

GDPR Concerns Around Lack of Explainability in AI“
Companies should commit to ensuringsystems that could fall under GDPR, including
AI, will be compliant. The threat of sizeablefines of €20 million or 4% of global turnoverprovides a sharp incentive.
Article 22 of GDPR empowers individuals withthe right to demand an explanation of how
an AI system made a decision that affectsthem.

”- European Commision

VP, European Commision

26

27

Why Explainability: Growing Global AI Regulation●

GDPR: Article 22 empowers individuals with the right to demand an explanation of how an
automated system made a decision that affects them.

●

Algorithmic Accountability Act 2019: Requires companies to provide an assessment of the risks posed by
the automated decision system to the privacy or security and the risks that contribute to inaccurate, unfair,biased, or discriminatory decisions impacting consumers●

California Consumer Privacy Act: Requires companies to rethink their approach to capturing,storing, and sharing personal data to align with the new requirements by January 1, 2020.

●

Washington Bill 1655: Establishes guidelines for the use of automated decision systems to protect
consumers, improve transparency, and create more market predictability.

●

Massachusetts Bill H.2701: Establishes a commission on automated decision-making,transparency, fairness, and individual rights.

●

Illinois House Bill 3415: States predictive data analytics determining creditworthiness or hiring
decisions may not include information that correlates with the applicant race or zip code.

SR 11-7 and OCC regulations for Financial Institutions

29

“Explainability by Design” for AI products
Model Debugging
Model Visualization

Feedback Loop
Train
Model Diagnostics
Root Cause Analytics
Debug

Model Evaluation
Compliance Testing
QA
Performance monitoring
Fairness monitoring

Explainable
AI

Monitor

DeployA/B Test
Model Comparison
Cohort Analysis

Predict

Explainable Decisions
API Support

Model Launch Signoff
Model Release Mgmt

AI @ Scale - Challenges for Explainable AI

31

LinkedIn operates the largest professional
network on the Internet
645M+ members

30M+

35K+

companies are
represented
on LinkedIn

skills listed

280B

20M+
open jobs
on LinkedIn
Jobs

90K+
schools listed(high school &college)

Feed updates

The AWS ML Stack
Broadest and most complete set of Machine Learning capabilities
AI SERVICES
VISION

AmazonRekognition

SPEECH

AmazonPolly

TEXT

AmazonTranscribe

AmazonTextract

AmazonAmazonComprehend Translate

SEARCH NEWCHATBOTSPERSONALIZATIONFORECASTINGFRAUD NEWDEVELOPMENT NEW

AmazonKendraAmazonLex

AmazonPersonalize

AmazonForecast

AmazonFraud Detector

AmazonCodeGuruNEW+Medical+Medical

CONTACT CENTERS
NEW

Contact Lens
For Amazon
Connect

ML SERVICES
SageMaker Studio IDE NEW

Amazon SageMaker

GroundTruth

Augmented
AI

Built-inalgorithms

SageMaker
Notebooks NEW

SageMaker
Experiments NEW

ModeltuningSageMaker
Debugger NEW

SageMaker
Autopilot NEW

Modelhosting

SageMaker
Model Monitor NEW

SageMaker
Neo

ML FRAMEWORKS & INFRASTRUCTURE
NEW

NEW

NEW

Deep Learning
AMIs & Containers

GPUs &CPUs

Elastic
Inference

Inferentia© 2019 Amazon Web Services, Inc. or its affiliates. All rights reserved |

FPGA

33

Explanation - In a Nutshell

34

What is Explainable AI?
Black Box AI

DataBlack-Box
AI

Decision,
Recommendation

AI productConfusion with Today’s AI Black
Box
●
●
●
●

Why did you do that?
Why did you not do that?
When do you succeed or fail?
How do I correct an error?

Explainable AI
Feedback

Data

Explainable
AI

Explainable
AI Product

Clear & Transparent Predictions
Decision
Explanation

●
●
●
●

I understand why
I understand why not
I know why you succeed or failI understand, so I trust youExample of an End-to-End XAI System

Humans may have follow-up questions
Explanations cannot answer all users’ concerns

How to Explain? Accuracy vs. Explainability

Neural Net
GAN

CNN

•
•
•

RNN

Ensemble
MethodXGB
RandomForest•••Decision
Tree
Statistical
Model
Graphical Model

AOG

•

•
•

SVM

Bayesian
Belief Net
SLR
CRF

HBN
MLN

MarkovModel

LinearModel

XAI Definitions - Explanation vs. Interpretation
Oxford Dictionary ofEnglishOn the Role of Data in XAI

TextEvaluation (1) - Perturbation-based Approaches

Evaluation (2) - Human (Role)-based Evaluation is
Essential… but too often based on size!
Evaluation criteria for Explanations [Miller, 2017]○ Truth & probability○ Usefulness, relevance○ Coherence with prior belief○ Generalization
Cognitive chunks = basic explanation units○ Which basic units for explanations?
○ How many?
○ How to compose them?
○ Uncertainty & end users?

(for different explanation needs)

Evaluation (3) - XAI: One Objective, Many Metrics

Explanation in AI (not only Machine Learning!)

43

XAI: One Objective, Many ‘AI’s, Many Definitions, Many Approaches

XAI: One Objective, Many ‘AI’s, Many Definitions, Many Approaches

XAI: One Objective, Many ‘AI’s, Many Definitions, Many Approaches

XAI: One Objective, Many ‘AI’s, Many Definitions, Many Approaches

XAI: One Objective, Many ‘AI’s, Many Definitions, Many Approaches•••

XAI: One Objective, Many ‘AI’s, Many Definitions, Many Approaches•••

XAI: One Objective, Many ‘AI’s, Many Definitions, Many Approaches•••

XAI: One Objective, Many ‘AI’s, Many Definitions, Many Approaches•••

XAI: One Objective, Many ‘AI’s, Many Definitions, Many Approaches•••

XAI: One Objective, Many ‘AI’s, Many Definitions, Many Approaches•••

XAI: One Objective, Many ‘AI’s, Many Definitions, Many Approaches

•
•
•

•
•

XAI: One Objective, Many ‘AI’s, Many Definitions, Many Approaches

•
•
•

•
•

Overview of Explanation in Machine Learning (1)

•
•
•
•
•
•

Overview of Explanation in Machine Learning (2)● Artificial Neural Network

Overview of Explanation in Machine Learning (3)● Computer Vision

Overview of Explanation in Different AI Fields (1)● Game Theory

Overview of Explanation in Different AI Fields (1)● Game Theory

Overview of Explanation in Different AI Fields (1)● Game TheoryŠOverview of Explanation in Different AI Fields (2)•Overview of Explanation in Different AI Fields (2)•Overview of Explanation in Different AI Fields (3)•Overview of Explanation in Different AI Fields (3)•Overview of Explanation in Different AI Fields (3)•Overview of Explanation in Different AI Fields (4)•● Multi-agent Systems

Overview of Explanation in Different AI Fields (4)•● Multi-agent Systems

Overview of Explanation in Different AI Fields (4)•● Multi-agent Systems

Overview of Explanation in Different AI Fields (5)•••Overview of Explanation in Different AI Fields (5)•••Overview of Explanation in Different AI Fields (5)•••Overview of Explanation in Different AI Fields (6)•● Planning and SchedulingOverview of Explanation in Different AI Fields (6)•● Planning and SchedulingOverview of Explanation in Different AI Fields (7)•Overview of Explanation in Different AI Fields (7)•Overview of Explanation in Different AI Fields (8)•Explainable Machine Learning(from a Machine Learning Perspective)78

Achieving Explainable AI
Approach 1: Post-hoc explain a given AI model●

Individual prediction explanations in terms of input features, inﬂuential examples,
concepts, local decision rules●

Global prediction explanations in terms of entire model in terms of partial
dependence plots, global feature importance, global decision rules

Approach 2: Build an interpretable model●

Logistic regression, Decision trees, Decision lists and sets, Generalized Additive
Models (GAMs)

79

Integrated
Gradients

Slide credit: https://twitter.com/chandan_singh96/status/1138811752769101825

Achieving Explainable AI
Approach 1: Post-hoc explain a given AI model●

Individual prediction explanations in terms of input features, inﬂuential examples,
concepts, local decision rules●

Global prediction explanations in terms of entire model in terms of partial
dependence plots, global feature importance, global decision rules

Approach 2: Build an interpretable model●

Logistic regression, Decision trees, Decision lists and sets, Generalized Additive
Models (GAMs)

81

Top label: “clog”

Why did the network label thisimage as “clog”?

82Top label: “ﬁreboat”

Why did the network label thisimage as “ﬁreboat”?

83Credit Lending in a black-box ML world

Credit Line Increase?

Request Denied

Bank

Query AI System

Credit Lending Score = 0.3

Why? Why not?
How?

Fair lending laws [ECOA, FCRA] require credit decisions to be explainable

Credit Lending Model

The Attribution Problem
Attribute a model’s prediction on an input to features of the input
Examples:●Attribute an object recognition network’s prediction to its pixels●

Attribute a text sentiment network’s prediction to individual words●

Attribute a lending model’s prediction to its features

A reductive formulation of “why this prediction” but surprisingly usefulApplication of Attributions●

Debugging model predictions
E.g., Attribution an image misclassiﬁcation to the pixels responsible for it●

Generating an explanation for the end-user
E.g., Expose attributions for a lending prediction to the end-user●

Analyzing model robustness
E.g., Craft adversarial examples using weaknesses surfaced by attributions●

Extract rules from the model
E.g., Combine attribution to craft rules (pharmacophores) capturing prediction
logic of a drug screening network86

Next few slides
We will cover the following attribution methods**● Ablations● Gradient based methods (speciﬁc to differentiable models)● Score Backpropagation based methods (speciﬁc to NNs)

We will also discuss game theory (Shapley value) in attributions**Not a complete list!
See Ancona et al. [ICML 2019], Guidotti et al. [arxiv 2018] for a comprehensive survey

87Ablations
Drop each feature and attribute the change in prediction to that feature
Pros:●

Simple and intuitive to interpret

Cons:
●

Unrealistic inputs●

Improper accounting of interactive features●

Can be computationally expensive

88Feature*GradientAttribution to a feature is feature value times gradient, i.e., xi* 𝜕y/𝜕xi●

Gradient captures sensitivity of output w.r.t. feature●

Equivalent to Feature*Coeﬃcient for linear models○●

First-order Taylor approximation of non-linear models

Popularized by SaliencyMaps [NIPS 2013], Baehrens et al. [JMLR 2010]
Gradients in thevicinity of the input
seem like noise?

89Local linear approximations can be too local

score

Interesting gradients

uninteresting gradients(saturation)

1.0

0.0

“ﬁreboat-ness” of image
90

Score Back-Propagation based Methods
Re-distribute the prediction score through the neurons in the network●

LRP [JMLR 2017], DeepLift [ICML 2017], Guided BackProp [ICLR 2014]
Easy case: Output of a neuron is a linear functionof previous neurons (i.e., ni = ⅀ wij * nj)e.g., the logit neuron●

Image credit heatmapping.org

Re-distribute the contribution in proportion to
the coeﬃcients wij

91

Score Back-Propagation based Methods
Re-distribute the prediction score through the neurons in the network●

LRP [JMLR 2017], DeepLift [ICML 2017], Guided BackProp [ICLR 2014]
Tricky case: Output of a neuron is a non-linear
function, e.g., ReLU, Sigmoid, etc.

Image credit heatmapping.org●

Guided BackProp: Only consider ReLUs thatare on (linear regime), and which contribute
positively●

LRP: Use ﬁrst-order Taylor decomposition tolinearize activation function●

DeepLift: Distribute activation difference
relative a reference point in proportion toedge weights

92

Score Back-Propagation based Methods
Re-distribute the prediction score through the neurons in the network●

LRP [JMLR 2017], DeepLift [ICML 2017], Guided BackProp [ICLR 2014]

Pros:● Conceptually simple● Methods have been empirically validated toyield sensible result

Image credit heatmapping.org

Cons:● Hard to implement, requires instrumenting
the model● Often breaks implementation invariance
Think: F(x, y, z) = x * y *z andG(x, y, z) = x * (y * z)Baselines and additivity●

When we decompose the score via backpropagation, we imply a normative
alternative called a baseline○●●“Why Pr(ﬁreboat) = 0.91 [instead of 0.00]”

Common choice is an informationless input for the model○

E.g., Black image for image models○

E.g., Empty text or zero embedding vector for text models

Additive attributions explain F(input) - F(baseline) in terms of input features

Another approach: gradients at many points
score

Interesting gradients

uninteresting gradients(saturation)

1.0

intensity

0.0

Baseline… scaled inputs ...

… gradients of scaled inputs ….

Input

Integrated Gradients [ICML 2017]
Integrate the gradients along a straight-line path from baseline to input

IG(input, base) ::= (input - base) * ∫0 -1▽F(𝛂*input + (1-𝛂)*base) d𝛂
Original image

Integrated Gradients

Integrated Gradients in action97Why is this image labeled as “clog”?
Original image“Clog”

Why is this image labeled as “clog”?
Original image

Integrated Gradients(for label “clog”)“Clog”

Detecting an architecture bug●

Deep network [Kearns, 2016] predicts if a molecule binds to certain DNA site●

Finding: Some atoms had identical attributions despite different connectivity

Detecting an architecture bug●

Deep network [Kearns, 2016] predicts if a molecule binds to certain DNA site●

Finding: Some atoms had identical attributions despite different connectivity●

Bug: The architecture had a bug due to which the convolved bond features
did not affect the prediction!

Detecting a data issue●

Deep network predicts various diseases from chest x-rays

Original image

Integrated gradients(for top label)Detecting a data issue●

Deep network predicts various diseases from chest x-rays●

Finding: Attributions fell on radiologist’s markings (rather than the pathology)
Original image

Integrated gradients(for top label)Cooperative game theory in attributions

104

Shapley Value [Annals of Mathematical studies,1953]
Classic result in game theory on distributing gain in a coalition game●

Coalition Games○

Players collaborating to generate some gain (think: revenue)○

Set function v(S) determining the gain for any subset S of players

Shapley Value [Annals of Mathematical studies, 1953]
Classic result in game theory on distributing gain in a coalition game●●

Coalition Games○

Players collaborating to generate some gain (think: revenue)○

Set function v(S) determining the gain for any subset S of players

Shapley Values are a fair way to attribute the total gain to the players based on
their contributions○

Concept: Marginal contribution of a player to a subset of other players (v(S U {i}) - v(S))○

Shapley value for a player is a specific weighted aggregation of its marginal over all
possible subsets of other players

Shapley Value for player i = ⅀S⊆N w(S) * (v(S U {i}) - v(S))(where w(S) = N! / |S|! (N - |S| -1)!)

Shapley Value Justification
Shapley values are unique under four simple axioms●

Dummy: If a player never contributes to the game then it must receive zero attribution●

Efficiency: Attributions must add to the total gain●

Symmetry: Symmetric players must receive equal attribution●

Linearity: Attribution for the (weighted) sum of two games must be the same as the(weighted) sum of the attributions for each of the games

Shapley Values for Explaining ML models
SHAP [NeurIPS 2018], QII [S&P 2016], Strumbelj & Konenko [JMLR 2009]●●

Define a coalition game for each model input X○

Players are the features in the input○

Gain is the model prediction (output), i.e., gain = F(X)

Feature attributions are the Shapley values of this gameShapley Values for Explaining ML models
SHAP [NeurIPS 2018], QII [S&P 2016], Strumbelj & Konenko [JMLR 2009]●●

Define a coalition game for each model input X○

Players are the features in the input○

Gain is the model prediction (output), i.e., gain = F(X)

Feature attributions are the Shapley values of this game

Challenge: Shapley values require the gain to be defined for all subsets of players●

What is the prediction when some players (features) are absent?
i.e., what is F(x_1, <absent>, x_3, …, <absent>)?

Modeling Feature Absence
Key Idea: Take the expected prediction when the (absent) feature is sampled froma certain distribution.
Different approaches choose different distributions●[SHAP, NIPS 2018] Use conditional distribution w.r.t. the present features●[QII, S&P 2016] Use marginal distribution●[Strumbelj et al., JMLR 2009] Use uniform distribution

Preprint: The Explanation Game: Explaining Machine Learning Models with Cooperative Game
TheoryComputing Shapley Values
Exact Shapley value computation is exponential in the number of features●

Shapley values can be expressed as an expectation of marginals
𝜙(i) = ES ~ D [marginal(S, i)]●●

Sampling-based methods can be used to approximate the expectation●

See: “Computational Aspects of Cooperative Game Theory”, Chalkiadakis et al. 2011

The method is still computationally infeasible for models with hundreds offeatures, e.g., image models

Non-atomic Games: Aumann-Shapley Values and IG●

Values of Non-Atomic Games (1974): Aumann and Shapley extend their
method → players can contribute fractionally●

Aumann-Shapley values calculated by integrating along a straight-line path…
same as Integrated Gradients!

●

IG through a game theory lens: continuous game, feature absence is modeledby replacement with a baseline value●

Axiomatically justified as a result:○Integrated Gradients is the unique path-integral method satisfying: Sensitivity, Insensitivity,
Linearity preservation, Implementation invariance, Completeness, and Symmetry

Lessons learned: baselines are importantBaselines (or Norms) are essential to explanations [Kahneman-Miller 86]●

E.g., A man suffers from indigestion. Doctor blames it to a stomach ulcer. Wife blames
it on eating turnips. Both are correct relative to their baselines.

●

The baseline may also be an important analysis knob.

Attributions are contrastive, whether we think about it or not.

Some limitations and caveats for attributions

Attributions don’t explain everything
Some things that are missing:●

Feature interactions (ignored or averaged out)●

What training examples influenced the prediction (training agnostic)●

Global properties of the model (prediction-specific)

An instance where attributions are useless:●

A model that predicts TRUE when there are even number of black pixels and
FALSE otherwise

Attributions are for human consumption●

Humans interpret attributions and generate insights○●

Doctor maps attributions for x-rays to pathologies

Visualization matters as much as the attribution technique
Naive scaling of attributions
from 0 to 255

Attributions have a large
range and long tailacross pixels

After clipping attributionsat 99% to reduce range

Other individual prediction explanation methods

Local Interpretable Model-agnostic Explanations(Ribeiro et al. KDD 2016)

Figure credit: Ribeiro et al. KDD 2016
Figure credit: Anchors: High-Precision Model-Agnostic
Explanations. Ribeiro et al. AAAI 2018

118

Anchors

Figure credit: Anchors: High-Precision Model-Agnostic Explanations. Ribeiro et al. AAAI 2018119Influence functions●●

Trace a model’s prediction through the learning algorithm andback to its training data
Training points “responsible” for a given prediction

Figure credit: Understanding Black-box Predictions via Influence Functions. Koh and Liang. ICML 2017120

Example based Explanations

Learned prototypes and criticisms from Imagenet dataset (two types of dog breeds)●

Prototypes: Representative of all the training data.

●

Criticisms: Data instance that is not well represented by the set of prototypes.

Figure credit: Examples are not Enough, Learn to Criticize! Criticism for Interpretability. Kim, Khanna and Koyejo. NIPS 2016

121

Global Explanations

122

Global Explanations Methods●

Partial Dependence Plot: Shows
the marginal effect one or two features
have on the predicted outcome of amachine learning model

123

Global Explanations Methods●

Permutations: The importance of a feature is the increase in the prediction error of the model
after we permuted the feature’s values, which breaks the relationship between the feature and thetrue outcome.

124

Achieving Explainable AI
Approach 1: Post-hoc explain a given AI model●

Individual prediction explanations in terms of input features, inﬂuential examples,
concepts, local decision rules●

Global prediction explanations in terms of entire model in terms of partial
dependence plots, global feature importance, global decision rules

Approach 2: Build an interpretable model●

Logistic regression, Decision trees, Decision lists and sets, Generalized Additive
Models (GAMs)

125

Decision TreesIs the person fit?
Age < 30 ?
Yes
Eats a lot of pizzas?
Yes

No

Unfit

Fit

No

Exercises in the morning?
Yes

No

Fit

Unfit

126

Decision Set

Figure credit: Interpretable Decision Sets: A Joint Framework for Description and Prediction, Lakkaraju, Bach,Leskovec

127

Decision Set

128

Decision List

Figure credit: Interpretable Decision Sets: A Joint Framework for Description and Prediction, Lakkaraju, Bach,Leskovec

129

Falling Rule List
A falling rule list is an ordered list of if-then rules (falling rule lists are a type ofdecision list), such that the estimated probability of success decreases
monotonically down the list. Thus, a falling rule list directly contains thedecision-making process, whereby the most at-risk observations are classiﬁed
ﬁrst, then the second set, and so on.

130Box Drawings for Rare Classes

Figure credit: Box Drawings for Learning with Imbalanced. Data Siong Thye Goh and Cynthia Rudin

131

Supersparse Linear Integer Models for Optimized
Medical Scoring Systems

Figure credit: Supersparse Linear Integer Models for Optimized Medical Scoring Systems. Berk Ustun and Cynthia Rudin
132

K- Nearest Neighbors

Explanation in terms of nearest training
data points responsible for the decision133

GLMs and GAMsIntelligible Models for Classification and Regression. Lou, Caruana and Gehrke KDD 2012
Accurate Intelligible Models with Pairwise Interactions. Lou, Caruana, Gehrke and Hooker. KDD 2013134

Explainable Machine Learning(from a Knowledge Graph Perspective)

Freddy Lécué: On the role of knowledge graphsin135
explainable AI. Semantic Web 11(1): 41-51 (2020)

Knowledge Graph (1)August 28th, 2019

Tutorial on Explainable AI

136

Freddy Lécué: On the role of knowledge graphs in
explainable AI. Semantic Web 11(1): 41-51 (2020)

Knowledge Graph (2)August 28th, 2019

Tutorial on Explainable AI

137

Freddy Lécué: On the role of knowledge graphs in
explainable AI. Semantic Web 11(1): 41-51 (2020)

Knowledge Graph Construction

August 28th, 2019

Tutorial on Explainable AI

138

Freddy Lécué: On the role of knowledge graphs in
explainable AI. Semantic Web 11(1): 41-51 (2020)

Knowledge Graph in Machine Learning (1)Freddy Lécué: On the role of knowledge graphs in
explainable AI. Semantic Web 11(1): 41-51 (2020)

Knowledge Graph in Machine Learning (2)Freddy Lécué: On the role of knowledge graphs in
explainable AI. Semantic Web 11(1): 41-51 (2020)

Knowledge Graph in Machine Learning (3)Training
Data
Input(unlabeledimage)
Neurons respondto simple shapes
Neurons respond to
more complex
structures
Neurons respond tohighly complex,
abstract concepts

1st Layer

Low-levelfeatures tohigh-levelfeatures

2nd Layer

nth Layer

Freddy Lécué: On the role of knowledge graphs in
explainable AI. Semantic Web 11(1): 41-51 (2020)

Knowledge Graph in Machine Learning (4)Training
Data
Input(unlabeledimage)
Neurons respondto simple shapes
Neurons respond to
more complex
structures
Neurons respond tohighly complex,
abstract concepts

1st Layer

Low-levelfeatures tohigh-levelfeatures

2nd Layer

nth Layer

Freddy Lécué: On the role of knowledge graphs in
explainable AI. Semantic Web 11(1): 41-51 (2020)

Knowledge Graph in Machine Learning (5)Freddy Lécué: On the role of knowledge graphs in
explainable AI. Semantic Web 11(1): 41-51 (2020)

Knowledge Graph in Machine Learning (6)After Human Reasoning…

•

•

Knowledge Graph in Machine Learning - An Implementation

XAI Tools on Applications,Lessons Learnt
and Research Challenges

160

161

Explainable Boosted Object Detection – Industry Agnostic
Challenge: Object detection is usually performed from alarge portfolio of Artificial Neural Networks (ANNs)architectures trained on large amount of labelled data.
Explaining object detections is rather difficult due to thehigh complexity of the most accurate ANNs.
AI Technology: Integration of AI related technologies
i.e., Machine Learning (Deep Learning / CNNs), andknowledge graphs / linked open data.
XAI Technology: Knowledge graphs and Artificial Neural
Networks

Context● Explanation in Machine Learning systems has been identiﬁed to be
the one asset to have for large scale deployment of Artiﬁcial
Intelligence (AI) in critical systems● Explanations could be example-based (who is similar), features-based(what is driving decision), or even counterfactual (what-if scenario) topotentially action on an AI system; they could be represented in manydifferent ways e.g., textual, graphical, visual

Goal● All representations serve different means, purpose and operators. We
designed the ﬁrst-of-its-kind XAI platform for critical systems i.e., theThales Explainable AI Platform which aims at serving explanations
through various forms

Approach: Model-Agnostic● [AI:ML] Grad-Cam, Shapley, Counter-factual, Knowledge graphDebugging Artificial Neural Networks – Industry Agnostic
Challenge: Designing Artificial Neural Networkarchitectures requires lots of experimentation(i.e., training phases) and parameters tuning(optimization strategy, learning rate, number oflayers…) to reach optimal and robust machinelearning models.
AI Technology: Artificial Neural Network
XAI Technology: Artificial Neural Network, 3DModeling and Simulation Platform For AI

Obstacle Identification Certification (Trust) - Transportation

Challenge: Public transportation is getting more and more
self-driving vehicles. Even if trains are getting more and moreautonomous, the human stays in the loop for critical decision,for instance in case of obstacles. In case of obstacles trains
are required to provide recommendation of action i.e., go onor go back to station. In such a case the human is required tovalidate the recommendation through an explanation exposedby the train or machine.
AI Technology: Integration of AI related technologies i.e.,
Machine Learning (Deep Learning / CNNs), and semantic
segmentation.
XAI Technology: Deep learning and Epistemic uncertainty

Explaining Flight Performance- Transportation
Challenge: Predicting and explaining
aircraft engine performance
AI Technology: Artificial Neural Networks
XAI Technology: Shapely Values

Explainable On-Time Performance - Transportation
Challenge: Globally 323,454 flights are delayed every year.
Airline-caused delays totaled 20.2 million minutes last year,
generating huge cost for the company. Existing in-house
technique reaches 53% accuracy for predicting flight delay,
does not provide any time estimation (in minutes as opposed
to True/False) and is unable to capture the underlying
reasons (explanation).
AI Technology: Integration of AI related technologies i.e.,
Machine Learning (Deep Learning / Recurrent neural
Network),
Reasoning(throughsemantics-augmented
case-based reasoning) and Natural Language Processing forbuilding a robust model which can (1) predict flight delays inminutes, (2) explain delays by comparing with historicalcases.
XAI Technology: Knowledge graph embedded Sequence
Learning using LSTMs

Explainable Risk Management - Finance

Challenge: Accenture is managing every year more than80,000 opportunities and 35,000 contracts with an expected
revenue of $34.1 billion. Revenue expectation does notmeet estimation due to the complexity and risks of criticalcontracts. This is, in part, due to the (1) large volume of
projects to assess and control, and (2) the existing
non-systematic assessment process.
AI Technology: Integration of AI technologies i.e., Machine
Learning, Reasoning, Natural Language Processing forbuilding a robust model which can (1) predict revenue loss,(2) recommend corrective actions, and (3) explain why suchactions might have a positive impact.
XAI Technology: Knowledge graph embedded Random
ForrestCopyright © 2017 Accenture. All rights reserved.

Explainable Anomaly Detection – Finance (Compliance)

Challenge: Predicting and explaining abnormally employee expenses (as high accommodation price in 1000+ cities).
AI Technology: Various techniques have been matured over the last two decades to achieve excellent results. However most methods address the problem
from a statistic and pure data-centric angle, which in turn limit any interpretation. We elaborated a web application running live with real data from (i) travel andexpenses from Accenture, (ii) external data from third party such as Google Knowledge Graph, DBPedia (relational DataBase version of Wikipedia) and social
events from Eventful, for explaining abnormalities.
XAI Technology: Knowledge graph embedded Ensemble Learning

Counterfactual Explanations for Credit Decisions (3) - FinanceExplanation of Medical Condition Relapse – Health
Challenge: Explaining medical condition relapse in thecontext of oncology.
AI Technology: Relational learning
XAI Technology: Knowledge graphs and Artificial Neural
Networks

Case Study:
Talent Search
Varun Mithal, Girish Kathalagiri, Sahin Cem Geyik

174

LinkedIn Recruiter●

Recruiter Searches for Candidates○●

Standardized and free-text search criteria

Retrieval and Ranking○○

Filter candidates using the criteria
Rank candidates in multiple levels using ML
models175

Modeling Approaches●●●

Pairwise XGBoost
GLMix
DNNs via TensorFlow●

Optimization Criteria: inMail Accepts○

Positive: inMail sent by recruiter, and positively responded by candidate■ Mutual interest between the recruiter and the candidate

176

Feature Importance in XGBoost

177

How We Utilize Feature Importances for GBDT●

Understanding feature digressions○○●

Introducing new features in bulk and identifying effective ones○○●

An activity feature for last 3 hours, 6 hours, 12 hours, 24 hours introduced (costly to compute)
Should we keep all such features?

Separating the factors for that caused an improvement○○●

Which a feature that was impactful no longer is?
Should we debug feature generation?

Did an improvement come from a new feature, or a new labeling strategy, data source?
Did the ordering between features change?

Shortcoming: A global view, not case by case178

GLMix Models●

Generalized Linear Mixed Models
○
○
○

●

Lots of parameters overall○●

Global: Linear Model
Per-contract: Linear Model
Per-recruiter: Linear Model

For a specific recruiter or contract the weights can be summed up

Inherently explainable○○

Contribution of a feature is “weight x feature value”
Can be examined in a case-by-case manner as well179

TensorFlow Models in Recruiter and Explaining Them●

We utilize the Integrated Gradients [ICML 2017] method●

How do we determine the baseline example?
○
○
○
○
○
○

Every query creates its own feature values for the same candidate
Query match features, time-based features
Recruiter affinity, and candidate affinity features
A candidate would be scored differently by each query
Cannot recommend a “Software Engineer” to a search for a “Forensic Chemist”
There is no globally neutral example for comparison!

180

Query-Specific Baseline Selection●

For each query:
○
○
○
○

●

Score examples by the TF model
Rank examples
Choose one example as the baseline
Compare others to the baseline example

How to choose the baseline example○○○○

Last candidate
Kth percentile in ranking
A random candidate
Request by user (answering a question like: “Why was I presented candidate x above
candidate y?”)181

Example

182

Example - Detailed
FeatureDescription

Difference (1 vs 2)

Contribution

Feature……….

Description……….

-2.0476928-2.144455602

Feature……….

Description……….

-2.3223877

1.903594618

Feature……….

Description……….

0.11666667

0.2114946752

Feature……….

Description……….

-2.1442587

0.2060414469

Feature……….

Description……….

-14

0.1215354111

Feature……….

Description……….

1

0.1000282466

Feature……….

Description……….

-92-0.085286277

Feature……….

Description……….

0.9333333

0.0568533262

Feature……….

Description……….

-1-0.051796317

Feature……….

Description……….

-1-0.050895940
183

Pros & Cons●●

Explains potentially very complex models
Case-by-case analysis
○
○
○
○

●

Why do you think candidate x is a better match for my position?
Why do you think I am a better fit for this job?
Why am I being shown this ad?
Great for debugging real-time problems in production

Global view is missing○○

Aggregate Contributions can be computed
Could be costly to compute184Lessons Learned and Next Steps●

Global explanations vs. Case-by-case Explanations○○●

Integrated gradients worked well for us○○●

Global gives an overview, better for making modeling decisions
Case-by-case could be more useful for the non-technical user, better for debugging
Complex models make it harder for developers to map improvement to effort
Use-case gave intuitive results, on top of completely describing score differences

Next steps○

Global explanations for Deep Models

185

Case Study:
Model Interpretation for Predictive Models in B2B
Sales Predictions
Jilei Yang, Wei Di, Songtao Guo

186

Problem Setting●

Predictive models in B2B sales prediction○○●

E.g.: random forest, gradient boosting, deep neural network, …
High accuracy, low interpretability

Global feature importance → Individual feature reasoning

187

Example

188

Revisiting LIME●

Given a target sample 𝑥𝑘, approximate its prediction 𝑝𝑟𝑒𝑑(𝑥𝑘) by building asample-speciﬁc linear model:𝑝𝑟𝑒𝑑(𝑋) ≈ 𝛽𝑘1 𝑋1 + 𝛽𝑘2 𝑋2 + …, 𝑋 ∈ 𝑛𝑒𝑖𝑔ℎ𝑏𝑜𝑟(𝑥𝑘)●E.g., for company CompanyX:
0.76 ≈ 1.82 ∗ 0.17 + 1.61 ∗ 0.11+…

189

xLIME

Piecewise Linear
Regression

Localized Stratified
Sampling

190

Piecewise Linear Regression
Motivation: Separate top positive feature inﬂuencers and top negative feature inﬂuencers

191Impact of Piecewise Approach●●

Target sample 𝑥𝑘=(𝑥𝑘1, 𝑥𝑘2, ⋯)Top feature contributor○○●

Top positive feature inﬂuencer○○●

LIME: large magnitude of 𝛽𝑘𝑗 ⋅ 𝑥𝑘𝑗
xLIME: large magnitude of 𝛽𝑘𝑗− ⋅ 𝑥𝑘𝑗
LIME: large magnitude of 𝛽𝑘𝑗
xLIME: large magnitude of negative 𝛽𝑘𝑗− or positive 𝛽𝑘𝑗+

Top negative feature inﬂuencer○○

LIME: large magnitude of 𝛽𝑘𝑗
xLIME: large magnitude of positive 𝛽𝑘𝑗− or negative 𝛽𝑘𝑗+

192

Localized Stratified Sampling: Idea
Method: Sampling based on empirical distribution around target value at each feature level

193

Localized Stratified Sampling: Method●●

Sampling based on empirical distribution around target value for each feature
For target sample 𝑥𝑘 = (𝑥𝑘1 , 𝑥𝑘2 , ⋯), sampling values of feature 𝑗 according to
𝑝𝑗 (𝑋𝑗) ⋅ 𝑁(𝑥𝑘𝑗 , (𝛼 ⋅ 𝑠𝑗 )2)
○
○
○
○

●

𝑝𝑗 (𝑋𝑗) : empirical distribution.
𝑥𝑘𝑗 : feature value in target sample.
𝑠𝑗 : standard deviation.
𝛼 : Interpretable range: tradeoff between interpretable coverage and local accuracy.

In LIME, sampling according to 𝑁(𝑥_𝑗 , 𝑠𝑗2).

194Summary

195

LTS LCP (LinkedIn Career Page) Upsell●

A subset of churn data○○●

Total Companies: ~ 19K
Company features: 117

Problem: Estimate whether there will be upsell given a set of features about
the company’s utility from the product196

Top Feature Contributor

197

198

Top Feature Influencers

199

Key Takeaways●

Looking at the explanation as contributor vs. inﬂuencer features is useful○○●xLIME aims to improve on LIME via:○○●

Contributor: Which features end-up in the current outcome case-by-caseInfluencer: What needs to be done to improve likelihood, case-by-case

Piecewise linear regression: More accurately describes local point, helps with finding correct
influencers
Localized stratified sampling: More realistic set of local points

Better captures the important features200

Case Study:
Relevance Debugging and Explaining @Daniel Qiu, Yucheng Qian

201

Debugging Relevance Models

202Architecture203

What Could Go Wrong?

204Challenges

205Solution

206

Call Graph

207

Timing208

Features

209

Advanced Use Cases

210

Perturbation

211

Comparison

212

Holistic Comparison

213

Granular Comparison

214

Replay215

Teams

●
●
●
●
●
●

SearchFeed
Comments
People you may know
Jobs you may be interested in
Notiﬁcation

216

Case Study:
Building an Explainable AI Engine @Luke Merrick

217

Fiddler’s Explainable AI Engine
Mission: Unlock Trust, Visibility and Insights by making AI Explainable in every enterprise

All your data

Any data warehouse

Custom Models

Fiddler Modeling Layer

Explainable AI for everyone

APIs, Dashboards, Reports, Trusted Insights

Example: Credit Lending in a black-box ML world

Credit Line Increase?

Request Denied

Bank

Query AI System

Credit Lending Score = 0.3

Why? Why not? How?

Fair lending laws [ECOA, FCRA] require credit decisions to be explainable

Credit Lending ModelExplain individual predictions (using Shapley Values)
How Can This Help…
Customer Support
Why was a customer loan
rejected?

Bias & Fairness
How is my model doing
across demographics?

Lending LOB
What variables should theyvalidate with customers on“borderline” decisions?

Explain individual predictions (using Shapley Values)
How Can This Help…
Customer Support
Why was a customer loan
rejected?

Bias & Fairness
How is my model doing
across demographics?

Lending LOB
What variables should theyvalidate with customers on“borderline” decisions?

Explain individual predictions (using Shapley Values)
How Can This Help…
Customer Support
Why was a customer loan
rejected?

Probe themodel oncounterfactuals

Bias & Fairness
How is my model doing
across demographics?

Lending LOB
What variables should theyvalidate with customers on“borderline” decisions?

Integrating explanations
How Can This Help…
Customer Support
Why was a customer loan
rejected?
Why was the credit card limit
low?
Why was this transaction
marked as fraud?

Slice & Explain

How Can This Help…
Global Explanations
What are the primary featuredrivers of the dataset on mymodel?

Region Explanations
How does my model perform
on a certain slice? Where
does the model not performwell? Is my model uniformly
fair across slices?

Model Monitoring: Feature Drift

Time slice

Feature distribution fortime slice relative totraining distribution

Investigate Data Drift Impacting Model PerformanceModel Monitoring: Outliers with Explanations

How Can This Help…
Operations

Outlier

Individual
Explanations

Why are there outliers inmodel predictions? Whatcaused model performance
to go awry?

Data Science
How can I improve my MLmodel? Where does it not dowell?

Some lessons learned at Fiddler● Attributions are contrastive to their baselines● Explaining explanations is important (e.g. good UI)● In practice, we face engineering challenges as much as
theoretical challenges

227

Recap●●Part I: Introduction and Motivation○

Motivation, Definitions & Properties○

Evaluation Protocols & Metrics

Part II: Explanation in AI (not only Machine Learning!)○

From Machine Learning to Knowledge Representation and Reasoning and Beyond●

Part III: Explainable Machine Learning (from a Machine Learning Perspective)●

Part IV: Explainable Machine Learning (from a Knowledge Graph Perspective)●

Part V: XAI Tools on Applications, Lessons Learnt and Research Challenges

228

Challenges & Tradeoffs●

Lack of standard interface for ML models
makes pluggable explanations hard●

Explanation needs vary depending on the typeof the user who needs it and also the problem

Fairness

Performanceat hand.
●

The algorithm you employ for explanations
might depend on the use-case, model type,?

data format, etc.
●

There are trade-offs w.r.t. Explainability,

Transparency

User Privacy

Performance, Fairness, and Privacy.
229

Explainability in ML: Broad Challenges
Actionable explanations
Balance between explanations & model secrecy
Robustness of explanations to failure modes (Interaction between ML
components)
Application-specific challenges
Conversational AI systems: contextual explanations
Gradation of explanations

Tools for explanations across AI lifecycle
Pre & post-deployment for ML models
Model developer vs. End user focusedThanks! Questions?
●Feedback most welcome :-)○freddy.lecue@inria.fr, krishna@fiddler.ai, sgeyik@linkedin.com,
kenthk@amazon.com, vamithal@linkedin.com, ankur@fiddler.ai,
luke@fiddler.ai, p.minervini@ucl.ac.uk, riccardo.guidotti@unipi.it●

Tutorial website: https://xaitutorial2020.github.io●

To try Fiddler, please send an email to info@ﬁddler.ai●

To try Thales XAI Platform , please send an email to freddy.lecue@thalesgroup.com

https://xaitutorial2020.github.io 231

Appendix

232

Case Study:
Talent Platform“Diversity Insights and Fairness-Aware Ranking”
Sahin Cem Geyik, Krishnaram Kenthapadi

233

Guiding Principle:“Diversity by Design”

“Diversity by Design” in LinkedIn’s Talent SolutionsRepresentative Ranking for Talent Search
S. C. Geyik, S. Ambler,
K. Kenthapadi, Fairness-Aware
Ranking in Search &Recommendation Systems with
Application to LinkedIn Talent
Search, KDD’19.

[Microsoft’s AI/ML
conference(MLADS’18). DistinguishedContribution Award]
Building Representative
Talent Search at LinkedIn(LinkedIn engineering blog)

Intuition for Measuring and Achieving Representativeness
Ideal: Top ranked results should follow a desired distribution ongender/age/…
E.g., same distribution as the underlying talent pool

Inspired by “Equal Opportunity” definition [Hardt et al, NIPS’16]

Defined measures (skew, divergence) based on this intuition

Desired Proportions within the Attribute of Interest
Compute the proportions of the values of the attribute (e.g., gender,
gender-age combination) amongst the set of qualified candidates“Qualified candidates” = Set of candidates that match thesearch query criteria
Retrieved by LinkedIn’s Galene search engine●●

Desired proportions could also be obtained based on legal
mandate / voluntary commitment

Fairness-aware Reranking Algorithm (Simplified)Partition the set of potential candidates into different buckets foreach attribute value
Rank the candidates in each bucket according to the scoresassigned by the machine-learned model
Merge the ranked lists, balancing the representation requirements
and the selection of highest scored candidates
Representation requirement: Desired distribution on gender/age/…
Algorithmic variants based on how we achieve this balanceValidating Our Approach
Gender RepresentativenessOver 95% of all searches are representative compared to thequalified population of the search●Business Metrics
A/B test over LinkedIn Recruiter users for two weeks
No significant change in business metrics (e.g., # InMails sentor accepted)●●

Ramped to 100% of LinkedIn Recruiter users worldwide• Post-processing approach desirable• Model agnostic• Scalable across different model choices
for our application• Acts as a “fail-safe”• Robust to application-specific businesslogic• Easier to incorporate as part of existing
systems• Build a stand-alone service or component
for post-processing• No significant modifications to the existing
components• Complementary to efforts to reduce bias fromtraining data & during model training• Collaboration/consensus across key stakeholders

Engineering for Fairness in AI Lifecycle
Does the model encourage
feedback loops that can
produce increasingly unfair
outcomes?
Are we deploying our model
on a population that we did
not train/test on?

Feedback

Problem
Formation

Is an algorithm an
ethical solution to our
problem?
Does our data include enoughminority samples?

DatasetDeployment

Construction

Are there unequal effects
across users?

Have we evaluated the modelusing relevant fairness
metrics?

Are there missing/biased
features?
Do we need to apply debiasingalgorithms to preprocess ourdata?

Algorithm
Selection

Testing
Process

Do we need to include fairness
constraints in the function?

Training
Process
Credit: K. Browne & J. Draper

247

Algorithmic Bias▪

▪

Ethical challenges posed by AI
systems
Inherent biases present insociety– Reflected in training data–

AI/ML models prone toamplifying such biases▪

ACM FAT* conference /KDD’16 & NeurIPS’17
Tutorials

248

Example: Facebook adds Explainable AI to build TrustAxioms●

Insensitivity: A variable that has no effect on the output gets no attribution●

Sensitivity: If baseline and input differ in a single variable, and have different
outputs, then that variable should receive some attribution●

Linearity preservation: Attributions(ɑ*F1 + ß*F2) = ɑ*Attributions(F1) +ß*Attributions(F2)●Implementation invariance: Two networks that compute identical functions for
all inputs get identical attributions●

Completeness: Sum(attributions) = F(input) - F(baseline)●

Symmetry: Symmetric variables with identical values get equal attributions