What Explainable AI fails to explain (and how we fix that) | by Alvin Wan | Towards Data Science

Don’t take it from me. Take it from IEEE Fellow Cuntai Guan, who recognizes “many machine decisions are still poorly understood”. Most papers even suggest a rigid dichotomy between accuracy and interpretability.
Explainable AI (XAI) attempts to bridge this divide, but as we explain below, XAI justifies decisions without interpreting the model directly. This means practitioners in applications such as finance and medicine are forced into a dilemma: pick an un-interpretable, accurate model or an inaccurate, interpretable model.
Defining explainability or interpretability for computer vision is challenging: What does it even mean to explain a classification for high-dimensional inputs like images? As we discuss below, two popular definitions involve saliency maps and decision trees, but both approaches have their weaknesses.
Many XAI methods produce heatmaps known as saliency maps, which highlight important input pixels that influence the prediction. However, saliency maps focus on the input and neglect to explain how the model makes decisions.
For more on saliency maps, see these saliency tutorials and Github repositories.
To illustrate why saliency maps do not fully explain how the model predicts, here is an example: Below, the saliency maps are identical, but the predictions differ. Why? Even though both saliency maps highlight the correct object, one prediction is incorrect. How? Answering this could help us improve the model, but as shown below, saliency maps fail to explain the model’s decision process.
Another approach is to replace neural networks with interpretable models. Before deep learning, decision trees were the gold standard for accuracy and interpretability. Below, we illustrate the interpretability of decision trees, which works by breaking up each prediction into a sequence of decisions.
For accuracy, however, decision trees lag behind neural networks by up to 40% accuracy on image classification datasets2. Neural-network-and-decision-tree hybrids also underperform, failing to match neural networks on even the dataset CIFAR10, which features tiny 32x32 images like the one below.
As we show in our paper (Sec 5.2), this accuracy gap damages interpretability: high-accuracy, interpretable models are needed to explain high-accuracy neural networks.
We challenge this false dichotomy by building models that are both interpretable and accurate. Our key insight is to combine neural networks with decision trees, preserving high-level interpretability while using neural networks for low-level decisions, as shown below. We call these models Neural-Backed Decision Trees (NBDTs) and show they can match neural network accuracy while preserving the interpretability of a decision tree.
NBDTs are as interpretable as decision trees. Unlike neural networks today, NBDTs can output intermediate decisions for a prediction. For example, given an image, a neural network may output Dog. However, an NBDT can output both Dog and Animal, Chordate, Carnivore (below).
NBDTs achieve neural network accuracy. Unlike any other decision-tree-based method, NBDTs match neural network accuracy (< 1% difference) on 3 image classification datasets3. NBDTs also achieve accuracy within 2% of neural networks on ImageNet, one of the largest image classification datasets with 1.2 million 224x224 images.
Furthermore, NBDTs set new state-of-the-art accuracies for interpretable models. The NBDT’s ImageNet accuracy of 75.30% outperforms the best competing decision-tree-based method by a whole ~14%. To contextualize this accuracy gain: A similar gain of ~14% for non-interpretable neural networks took 3 years of research4.
The most insightful justifications are for objects the model has never seen before. For example, consider an NBDT (below), and run inference on a Zebra. Although this model has never seen Zebra, the intermediate decisions shown below are correct — Zebras are both Animals and Ungulates (hoofed animal). The ability to see justification for individual predictions is quintessential for unseen objects.
Furthermore, we find that with NBDTs, interpretability improves with accuracy. This is contrary to the dichotomy in the introduction: NBDTs not only have both accuracy and interpretability; they also make both accuracy and interpretability the same objective.
For example, the lower-accuracy ResNet6 hierarchy (left) makes less sense, grouping Frog, Cat, and Airplane together. This is “less sensible,” as it is difficult to find an obvious visual feature shared by all three classes. By contrast, the higher-accuracy WideResNet hierarchy (right) makes more sense, cleanly separating Animal from Vehicle — thus, the higher accuracy, the more interpretable the NBDT.
With low-dimensional tabular data, decision rules in a decision tree are simple to interpret e.g., if the dish contains a bun, then pick the right child, as shown below. However, decision rules are not as straightforward for inputs like high-dimensional images.
As we qualitatively find in the paper (Sec 5.3), the model’s decision rules are based not only on object type but also on context, shape, and color.
To interpret decision rules quantitatively, we leverage an existing hierarchy of nouns called WordNet7; with this hierarchy, we can find the most specific shared meaning between classes. For example, given the classes Cat and Dog, WordNet would provide Mammal. In our paper (Sec 5.2) and pictured below, we quantitatively verify these WordNet hypotheses.
Note that in small datasets with 10 classes i.e., CIFAR10, we can find WordNet hypotheses for all nodes. However, in large datasets with 1000 classes i.e., ImageNet, we can only find WordNet hypotheses for a subset of nodes.
Interested in trying out an NBDT, now? Without installing anything, you can view more example outputs online and even try out our web demo. Alternatively, use our command-line utility to run inference (Install with pip install nbdt). Below, we run inference on a picture of a cat.
nbdt https://images.pexels.com/photos/126407/pexels-photo-126407.jpeg?auto=compress&cs=tinysrgb&dpr=2&w=32  # this can also be a path to local image
This outputs both the class prediction and all the intermediate decisions.
Prediction: cat // Decisions: animal (99.47%), chordate (99.20%), carnivore (99.42%), cat (99.86%)
You can load a pretrained NBDT in just a few lines of Python as well. Use the following to get started. We support several neural networks and datasets.
from nbdt.model import HardNBDTfrom nbdt.models import wrn28_10_cifar10model = wrn28_10_cifar10()model = HardNBDT(    pretrained=True,    dataset='CIFAR10',    arch='wrn28_10_cifar10',    model=model)
For reference, see the script for the command-line tool we ran above; only ~20 lines are directly involved in transforming the input and running inference. For more instructions on getting started and examples, see our Github repository.
The training and inference process for a Neural-Backed Decision Tree can be broken down into four steps.
Construct a hierarchy for the decision tree. This hierarchy determines which sets of classes the NBDT must decide between. We refer to this hierarchy as an Induced Hierarchy.This hierarchy yields a particular loss function, that we call the Tree Supervision Loss5. Train the original neural network, without any modifications, using this new loss.Start inference by passing the sample through the neural network backbone. The backbone is all neural network layers before the final fully-connected layer.Finish inference by running the final fully-connected layer as a sequence of decision rules, which we call Embedded Decision Rules. These decisions culminate in the final prediction.
Construct a hierarchy for the decision tree. This hierarchy determines which sets of classes the NBDT must decide between. We refer to this hierarchy as an Induced Hierarchy.
This hierarchy yields a particular loss function, that we call the Tree Supervision Loss5. Train the original neural network, without any modifications, using this new loss.
Start inference by passing the sample through the neural network backbone. The backbone is all neural network layers before the final fully-connected layer.
Finish inference by running the final fully-connected layer as a sequence of decision rules, which we call Embedded Decision Rules. These decisions culminate in the final prediction.
For more detail, see our paper (Sec 3).
Explainable AI does not fully explain how the neural network reaches a prediction: Existing methods explain the image’s impact on model predictions but do not explain the decision process. Decision trees address this, but unfortunately, images7 are kryptonite for decision tree accuracy.
We thus combine neural networks and decision trees. Unlike predecessors that arrived at the same hybrid design, our neural-backed decision trees (NBDTs) simultaneously address the failures (1) of neural networks to provide justification and (2) of decision trees to attain high accuracy. This primes a new category of accurate, interpretable NBDTs for applications like medicine and finance. To get started, see the project page.
By Alvin Wan, *Lisa Dunlap, *Daniel Ho, Jihan Yin, Scott Lee, Henry Jin, Suzanne Petryk, Sarah Adel Bargal, Joseph E. Gonzalez
where * denotes equal contribution
[0] Designed by author Alvin Wan. Footnote exists to clarify we have rights to use this graphic.
[1] There are two types of saliency maps: one is white-box, where the method has access to the model and its parameters. One popular white-box method is Grad-CAM, which uses both gradients and class activation maps to visualize attention. You can learn more from the paper, “Grad-CAM: Visual Explanations from Deep Networks via Gradient-based Localization” http://openaccess.thecvf.com/content_ICCV_2017/papers/Selvaraju_Grad-CAM_Visual_Explanations_ICCV_2017_paper.pdf. The other type of saliency map is black-box, where the model does not have access to the model parameters. RISE is one such saliency method. RISE masks random portions of the input image and passes this image through the model — the mask that damages accuracy the most is the most “important” portion. You can learn more from the paper “RISE: Randomized Input Sampling for Explanation of Black-box Models”, http://bmvc2018.org/contents/papers/1064.pdf.
[2] This 40% gap between decision tree and neural network accuracy shows up on TinyImageNet200.
[3] The three datasets in particular are CIFAR10, CIFAR100, TinyImageNet200.
[4] This ImageNet accuracy gain is significant: for non-interpretable neural networks, a similar 14% gain on ImageNet took 3 years of research. To make this comparison, we examine a similar accuracy gain which took 3 years, from AlexNet in 2013 (63.3%) to Inception V3 (78.8%). The NBDT improves on previously state-of-the-art results by ~14% at around the same range, from NofE (61.29%) to our NBDTs (75.30%). There are other factors at play, however: One obvious one is that compute and deep learning libraries were not as readily available in 2013. A fairer comparison may to be use the latest the latest 14%-gain on ImageNet. The latest 14% gain took 5 years, starting from VGG-19 in 2015 (74.5%) and leading up to FixEfficientNet-L2 in 2020 (88.5%). However, this technically isn’t comparable either since large gains are harder at higher accuracies. Despite this lack of perfectly comparable benchmark progress, we just took the minimum of the two ranges in time, to try and illustrate how large of a gap 14% is.
[6] ResNet10 achieves 4% lower accuracy than WideResNet28x10 on CIFAR10.
[7] WordNet is a lexical hierarchy of various words. A large majority of words are nouns, but other parts of speech are included as well. For more information, see the official website.
[8] To understand the basic idea for a Tree Supervision Loss: Horse is just one class. However, it is also an Ungulate and an Animal. (See the figure in “Justifications for Individual Predictions”.) At the root node, the Horse sample thus needs to be passed to the child node Animal. Furthermore, the node Animal needs to pass the sample to Ungulate. Finally, the node Ungulate must pass the sample to Horse. Train each node to predict the correct child node. We call the loss that enforces this the Tree Supervision Loss.
[9] In general, decision trees perform best with low-dimensional data. Images are the antithesis of this best-case scenario, being extremely high-dimensional.