Ankur Taly (Google Inc.) Joint work with Mukund Sundararajan and Qiqi Yan Some Deep Learning Successes Google weaves machine learning into new Google Photos features Facebook Uses Deep Learning To Make Faster Translations Google Al Matches Diabetic Retinopathy Screens Google’s AlphaGo Al beats the world’s best human Go player Posted May 23, 2017 by Darrell Etherington (@etherington) Sighthound . "optricity +erraloupe # pilot.ai CORE AI: VOICE INTERFACE Hobvoi
出口可可MaluubA ROBOTICS & AUTO N netradyne  AdasWorks drıve.aı   Rokid 7 comma.ai HEALTHCARE [ Π  L E Λ P M I N D   Arya 2 twentybn 😊 KOViD True AI
E-COMMERCE
:K) Reflektion V i S E N Z E CQUIRED Source: https://www.cbsinsights.com
Output (Image label, next word, next move, etc.) Input (Image, sentence, game position, etc.) Flexible model for learning arbitrary non-linear, non-convex functions Transform input through a network of neurons Each neuron applies a non-linear activation function (σ) to its inputs
Output (Image label, next word, next move, etc.) Input (Image, sentence, game position, etc.) Highly expressive model ● Many different architectures o Can encode linear models, decision trees, markov models, and their combinations Less feature engineering needed o Lower layers can automatically extract complex features from raw inputs and feed them above
Output (Image label, next word, next move, etc.) Input (Image, sentence, game position, etc.) Used to be notoriously hard to train Rapid advances over the last few years ● Stochastic gradient descent ● Lots of training data ● Several “training” tricks ● Better hardware and software We can now train networks with millions of parameters over billions of training examples!
We understand them enough to: Design architectures for complex learning tasks ● For both supervised, unsupervised training datasets o ● Train these architectures to favorable optima ● Help them generalize beyond training set (prevent overfitting) But, a trained network still remains a black box to humans
Understanding the input-output behavior of Deep Networks i.e., we ask why did it make this prediction on this input? Benefits: ● Debug and understand models ● Build trust in the model ● Surface an explanation to the end-user ● Intellectual curiosity Why did the network label this image as  “drilling platform”?
Retinal Fundus Image
Why does the network label this image with “mild” Diabetic Retinopathy?
Inception architecture: 1.6 million parameters Modern architectures are way too complex for analytical reasoning ● The meaning of individual neurons is not human-intelligible ● ● Faithfulness vs. Interpretability
● Explain the network’s behavior in terms of the input ● What “features” of the input were important for this prediction? (Getting the right problem statement is the hard part here)
Distribute the prediction score to each input feature in proportion to its contribution to the prediction with respect to a certain baseline input ● Input features could be pixels, words etc. ● Baseline input is one where the prediction is neutral, e.g., black image ● The amount assigned to each feature is its attribution Large attribution indicates feature importance
● Our attribution method: Integrated Gradients ● Applications of the method ● Justifying Integrated Gradients ● Discussion
Ablate each input feature and measure the change in prediction ● Costly, especially for dense models with (224*224*3) pixel features Over or under attribution of interactive features o E.g., Query="Facebook"  AND  Domain="facebook.com"   IMPLIES high click through rate o Unrealistic inputs ●
Attribute using gradient* of the output w.r.t each base input feature Attribution for feature x, is x,* ∂y/∂x, Standard approach for understanding linear models ● Same as feature weights o First-order approximation for non-linear models ●
Visualization: Use (normalized) attribution as mask/window over image
Prediction
Average pixel gradient (normalized)
Average pixel gradient (normalized)
● Compute gradient for images that range from black to actual image ● Use these gradients as attributions ● Gif the sequence of resulting visualizations ● Blue screen starts animation
● across images o Not just the two images we discussed ● across layers o Not just the output layer ● across networks o Not just Inception on ImageNet o Severity varies (see this paper for details) Integrated Gradients
Construct a sequence of images interpolating from a baseline (black) to the actual image Average the gradients across these images
Mathematically, IG;(image) =  image; * ∫ 0 -1 ∇ F;(α*image) dα where: o F is the prediction function for the label ● image, is the intensity of the ith pixel IG (image) is the integrated gradient w.r.t. the ● ith pixel, i.e., attribution for ith pixel
Original image (Drilling platform) Gradient at image Integrated gradient
Original image (Drilling platform) Gradient at image Integrated gradient Original image Original image Original image Top label: school bus Score: 0.997033 Integrated gradients Gradients at image Top label: stopmatch Score: 0.998507 Integrated gradients Gradients at image Integrated gradients Gradients at image
Human label:  accordion Network’s top label: toaster
Human label:  accordion Network’s top label: toaster Integrated gradient
Diabetes complication that causes damage to blood vessels in the eye due to excess blood sugar. An Inception-based network for predicting  diabetic retinopathy grade from retinal fundus images achieves 0.97 AUC [JAMA paper] On what basis, does the network predict the DR grade?
Predicted DR grade: Mild
● We have a data set of questions and answers o Answer types include numbers, strings, dates, and yes/no ● Can we predict the answer type from the question? o Answer: Yes using a simple feedforward network ● Can we tell which words were indicative of the answer type? Enter attributions o ● Key issue: What is the analog of the black image? o Answer: the zero embedding vector
how many townships have a population above 50 ? [prediction: NUMERIC] what is the difference in population between fora and masilo [prediction: NUMERIC] how many athletes are not ranked ? [prediction: NUMERIC] what is the total number of points scored ? [prediction: NUMERIC] which film was before the audacity of democracy ? [prediction: STRING] which year did she work on the most films ? [prediction: DATETIME] what year was the last school established ? [prediction: DATETIME] when did ed sheeran get his first number one of the year ? [prediction: DATETIME] did charles oakley play more minutes than robert parish ? [prediction: YESNO] Red is positive attribution Blue is negative attribution Shades interplolate
Several sensible results, can almost harvest these as grammar rules how many townshipアルバa population above 50 ? [prediction: NUMERIC] what is the difference in population between fora and masilo [prediction: NUMERIC] how many athletes are not ranked ? [prediction: NUMERIC] what is the total number of points scored ? [prediction: NUMERIC] which film was before the audacity of democracy ? [prediction: STRING] which year did she work on the most films ? [prediction: DATETIME] what year was the last school established ? [prediction: DATETIME] when did ed sheeran get his first number one of the year ? [prediction: DATETIME] did charles oakley play more minutes thas robert parish ? [prediction: YESNO] Overfitting? Negative signals too

●
Drug Discovery
o Why part of the molecule causes it to bind to this protein? ● Search Ranking o What makes one result rank higher than another?
●
Language translation o Which input word does this output word correspond to? Justifying Integrated Gradients How do you evaluate an attribution method?
Our approach: ●   Define a set of reasonable axioms for attribution methods ● Show that integrated gradients satisfies them
Sensitivity: If starting from baseline, varying a variable changes the output, then the variable should receive some attribution. Pure gradients do not satisfy this when predictions saturate. ● Insensitivity: A variable that has no effect on the output gets no attribution.
Implementation Invariance: Two functionally equivalent networks have identical attributions for all inputs and baseline Linearity: If the function F is a linear combination of two functions F 4 , F 2 then the attributions for F are a linear combination of the attributions for F 1 , F 2 Symmetry: If a function is symmetric across two input variables then the variables should receive identical attribution
Completeness: Sum(attributions) = F(input) - F(baseline) Break down the predicted click through rate (pCTR) of an ad like: 55% of pCTR is because it's at position 1 ● 25% is due to its domain (a popular one) ● … ●
Theorem: Integrated Gradients is the unique method satisfying: ● Sensitivity, Insensitivity ● Implementation Invariance, Linearity, Symmetry o Completeness up to the errors from approximating integration. Historical note: It’s essentially the Aumann-Shapley method in cost sharing, which has a similar ● characterization. (Friedman 2004)
● The need for a baseline is central to any explanation method 。 In a sense, it is the counterfactual for causal reasoning ● The network must have a truly neutral prediction at the baseline input
● Easy to implement Gradient calls on a bunch of scaled down inputs o No instrumentation of the network, no new training o ● Widely applicable ● Backed by an axiomatic guarantee