Visualizing AI. Deconstructing and Optimizing the SHAP... | by Wai On | Towards Data Science

Explainability in AI is currently a hot topic. As AI systems play an increasing role in decision making, there is a general agreement that these systems need to provide information underlying their decisions or advice.
There is now a burgeoning industry and a vast amount of research interest into providing Explainable AI (XAI) tools and services that aim to help AI systems explain their decisions and actions to users. At the same time, governments are beginning to mandate the provision of explanations for individuals who may be affected by automated decisions. Indeed, Europe’s General Data Protection Regulation states that where an individual is subject to “a decision based solely on automated processing” that produces “legal effects” or result of similar significance, the individual is entitled to “meaningful information about the logic involved”.
It’s important to note that this last quote opens up the idea that not only do these systems need to provide explanations but that individuals should be able to come away with a clear understanding of the information provided. No doubt, this part of the regulation will be debated fiercely in time to come. However, from an immediate practical point of view, this (and other legislations in the works) should be seen as a call not only for these systems to provide information on how the results are derived but also for us to pay close attention to how they are presented so that individuals affected can understand them effectively.
In the past, results of XAI tools were typically presented in rudimentary forms. For example, with a table of numbers and/or a bar chart of some kind.
The results were easy to interpret but they were limited in their explanation power. This partly explains why these tools are often categorized as “Interpretable” AI rather than “Explainable” AI since they reveal what the underlying algorithm is doing but stop short at communicating what is going on in a comprehensive way so that users can come away with a “meaningful” understanding of the explanation.
Recently, some XAI tools have started to address this issue. For example, by providing rich visualizations as part of a more comprehensive explanation of the results1. These visualizations are powerful in that they go beyond the traditional summary statistics that come with basic graph types. However, for the uninitiated they are harder to interpret and to understand.
In part 1 of this article, I will describe how SHAP, a popular XAI tool, presents its results. Specifically, I will focus on describing one of the rich visualizations that comes with it: the Summary Plot. In part 2, I will give an example of its use in explaining a black box Machine Learning model before offering some ideas for improving the visualization. The goal is to help the reader interpret the visualization, optimize it, and to arrive at a deeper understanding of the results.
This article is intended for anyone who has a basic understanding of Machine Learning concepts and is interested in understanding the results of XAI tools more thoroughly. In terms of user types, this article would be most useful for a Data Scientist responsible for delivering reports to end users, or an end user decision maker or end user consumer2 who is going to receive the results of XAI tools.
SHAP stands for SHapley Additive exPlanations3. It belongs to a class of XAI tools known as feature attribution or feature importance. These tools show the workings of Machine Learning models by uncovering how strongly different input features affect the output. The approach that SHAP uses is a solution in cooperative game theory called Shapley values (named after Lloyd Shapley, an American mathematician and Nobel laureate, who came up with the idea).
There are lots of great articles about what SHAP is, what it does, how to use it, as well as the mathematical underpinnings behind Shapley values, so I will not attempt to rehash them in detail here.
SHAP has been implemented on a number of platforms, but the general process for using SHAP is as follow:
Model your data using a Machine Learning model (e.g. Random Forest, Neural Network).Feed the resulting model and your data into SHAP.SHAP calculates the influences of the model (using Shapley values) and output a set of SHAP values.You can then display the output using visualizations provided by SHAP.
Model your data using a Machine Learning model (e.g. Random Forest, Neural Network).
Feed the resulting model and your data into SHAP.
SHAP calculates the influences of the model (using Shapley values) and output a set of SHAP values.
You can then display the output using visualizations provided by SHAP.
SHAP is not the only tool set available that implements Shapley values (e.g. IML). However, one of things that sets it apart in my view is the interesting array of visualizations that it provides. Unfortunately, most of the current articles that talk about SHAP say very little about them and how they can help users to understand the result.
Rather than exhaustively go through each of the SHAP visualizations, I will focus on one called the Summary Plot. The reason for choosing this is because I feel this is one of the most interesting visualizations and because it has come under some criticisms for being difficult to understand. I will first describe the SHAP Summary Plot thoroughly step-by-step before using it in an example for explaining a black box Machine Learning model in part 2.
There are currently four types of Summary Plots: dot, bar, violin, and compact dot. In this article, I will focus on the “dot” type, which is the default Summary Plot for a single output model.
The SHAP Summary Plot provides a high-level composite view that shows the importance of features and how their SHAP values are spread across the data. The Summary Plot is a cross between a Swamp Plot and a Violin Plot in that all the instances are displayed and the resulting shapes show the frequencies and distributions of the data.
To show a Summary Plot, you simply invoke the Summary Plot function with the data to be explained and its corresponding SHAP values:
shap.summary_plot(shap_values, X)
Here, I am using the Python version of the SHAP package. The above plot is produced using a 100 by 5 matrix of random numbers:
shap.summary_plot(np.random.randn(100, 5), np.random.randn(100, 5))
So what you are seeing is a model with 100 instances, each with 5 features. The model has a single output which is a real number, as you would expect from a regression model.
How do we begin to understand this visualization? What is it telling us about the model? Let’s start with the axes. On the y-axis, features are listed in descending order of importance. In other words, the features that are most influential are listed at the top and least influential at the bottom.
On the x-axis, there is a scale for SHAP values with a vertical line at zero. Positive values are to the right of the line and negative values are to the left of the line. A point’s position in the plot represents a SHAP value for a particular feature of an instance. Values that are pulling the model’s prediction lower are to the left and values that are pushing the prediction higher are to the right.
The plot above shows one data instance with 5 features. It shows that Feature 2 is the most influential feature. The next most influential is Feature 4, and so on. Feature 2 has a SHAP value that is positive, around 2.1. Feature 4, despite it being the second most influential, is a negative number, around -1.5. This is because feature importance is about the absolute influence of the features rather than whether they are influencing the model’s output one way or another. So in this example, these two features are pulling the model’s output in opposite directions.
If another instance has the same SHAP values, then it will be stacked vertically. However, the vertical stacking only goes a certain distance within the boundary of the row. After which, it starts to overlap. In this way, the Summary plot can efficiently scale to hundreds of thousands of data points in a small space.
Let’s now turn to colors and what they mean. Colors differentiate the relative size of the feature values between instances. High values (relative to other instances) are colored red; low values are colored blue; and in-between values are colored shades of purple according to the color map on the right of the plot.
As more data points are added, their positions show the range and distribution of SHAP values for each feature. For example, does it cluster around the middle like a normal curve (but also mirrored), or does it skew towards one end?
The clustering of features with similar values creates “patches of colors” in the plot. The position of a patch gives the user a sense of how the magnitude of the values are impacting the model’s output. For example, if a red patch (high values) is to the left of zero then it means that high feature values are collectively pulling the model’s output towards a lower number. If they are to the right then it means they are pushing the model’s output towards a higher number. Similarly, depending on its position, a blue patch (low values) may be influencing the model’s output to be lower or higher.
Here’s an example that illustrates the color patches.
If we look at the first feature, the plot shows that low values of Hormonal Contraceptives Years (blue patch) are associated with lower predicted cancer risk, high values (red points) are associated with higher risk.
Like the Seaborn Swarm Plot and the Violin Plot, the SHAP Summary Plot can be thought of as a plot of plots. We can examine the data as a whole or look at each category (in this case, features) and examine the details within. For example, we can see in the plot above that the distribution of the values for the first feature, Hormonal Contraceptives Years, is skewed to the left, suggesting that on the whole the feature is influencing the model’s output to a lower number.
Features whose SHAP values cluster around zero typically means that most of the values do not have much of an impact on the model’s output (ignoring interactions between features at this point). The lower half of the features in the plot above is like that, which is why they are not as influential as the features in the top half of the plot.
To summarize, the Summary Plot answers the following questions:
Which features are most influential in the model’s output?
Features are listed in descending order of importance on the left hand side of the plot.
Does a feature influence the output to be a lower or higher number?
If the cluster of points is on the right of the zero line, the feature influences the model’s output to be a higher number. If a cluster of points is on the left, the feature influences the model’s output to be a lower number.
How strong are the influences?
The further the cluster of points is on the right, the stronger the influence on the output to be a higher number. The further the cluster of points is on the left, the stronger the influence on the output to be a lower number.
What are the distributions for SHAP values?
The shape of the points indicates the distribution of the SHAP values for each feature. That is, whether the values follow a normal distribution, skewed, or are scattered.
How are the relative sizes of feature values influencing the model?
The positions of blue color patches (small feature values) or red color patches (large feature values) show the different sizes of feature values and whether they are influencing the output of the model to be a higher number or a lower number.
The Summary Plot is an effective visualization for quickly answering these questions in a compact space.
I hope this article has provided a helpful discussion on the various components that make up the SHAP Summary Plot and how they work together. As a result, you are able to understand quickly the influences behind a Machine Learning model’s output from this visualization.
I find the Summary Plot to be a tremendously useful addition to other visualizations when understanding how different input features affect the output of a model. However, the design of the Summary Plot is not without challenges. In part 2 of this article, I will go through an example of its use in explaining a black box Machine Learning model before offering some ideas for tackling some of the challenges.
[1] S.M. Lundberg, G. G. Erion and S. Lee, Consistent Individualized Feature Attribution for Tree Ensembles (2019), University of Washington.
[2] H. Michael, Explaining explainable AI (2019), XRDS: Crossroads, April 2019.
[3] S.M. Lundberg, S. Lee, A Unified Approach to Interpreting Model Predictions (2017).