Introducing SHAP Decision Plots. Visualize the inner workings of machine... | by Floid Gilbert | Towards Data Science

The SHAP (SHapley Additive exPlanations) framework has proved to be an important advancement in the field of machine learning model interpretation. Developed by Scott Lundberg and Su-In Lee, SHAP combines several existing methods to create an intuitive, theoretically-sound approach to explain predictions for any model.
SHAP builds model explanations by asking the same question for every prediction and feature: “How does prediction i change when feature j is removed from the model?” So-called SHAP values are the answers. They quantify the magnitude and direction (positive or negative) of a feature’s effect on a prediction. As this article will show, SHAP values can produce model explanations with the clarity of a linear model.
The Python software package shap, developed by Scott Lundberg et al., provides utilities to calculate and plot SHAP values. The project’s main page demonstrates typical SHAP plots and provides links to example notebooks.
Recently, a new class of plots known as decision plots have been added to the shap package. Decision plots offer a detailed view of a model’s inner workings; that is, they show how models make decisions. This article briefly presents several use cases for decision plots:
Show a large number of feature effects clearly.Visualize multioutput predictions.Display the cumulative effect of interactions.Explore feature effects for a range of feature values.Compare and contrast predictions for several models.Identify typical prediction paths.
Show a large number of feature effects clearly.
Visualize multioutput predictions.
Display the cumulative effect of interactions.
Explore feature effects for a range of feature values.
Compare and contrast predictions for several models.
Identify typical prediction paths.
For additional details and source code, see the decision plot documentation/vignette or download the notebook.
Update: In addition to learning SHAP, I recommend exploring Explainable Boosting Machines (EBMs) in the interpretML package (official docs, video). EBMs are inherently explainable and the associated package includes interactive plots. interpretML also integrates SHAP, among other methods/models, facilitating side-by-side comparisons.
To date, force plots (introduced in Nature BME) have been the default method for visualizing individual model predictions via the shap package. Consider the following plot. It shows the prediction for a LightGBM model trained on the UCI Adult Income data set. The prediction is the probability that the individual makes over $50K a year. Red arrows represent feature effects (SHAP values) that drive the prediction value higher while blue arrows are those effects that drive the prediction value lower. Each arrow’s size represents the magnitude of the corresponding feature’s effect. The “base value” (see the grey print towards the upper-left of the image) marks the model’s average prediction over the training set. The “output value” is the model’s prediction: probability 0.64. The feature values for the largest effects are printed at the bottom of the plot. Overall, the force plot provides an effective summary for this prediction.
Compare the force plot above to the decision plot below. The decision plot’s straight vertical line marks the model’s base value. The colored line is the prediction. Feature values are printed next to the prediction line for reference. Starting at the bottom of the plot, the prediction line shows how the SHAP values (i.e., the feature effects) accumulate from the base value to arrive at the model’s final score at the top of the plot. (Roughly speaking, this is similar to a statistical linear model where the sum of effects, plus an intercept, equals the prediction.) Decision plots are literal representations of SHAP values, making them easy to interpret.
The force plot and the decision plot are both effective at explaining the foregoing model’s prediction. The magnitude and direction of the major effects are easy to identify. However, as we will show next, decision plots are more effective than force plots when many significant features are involved.
The next example uses a LightGBM model trained on the UCI Communities and Crime data set to predict the number of non-violent crimes per 100,000 people in the population. The model uses 101 features. The same prediction is explained using a force plot and a decision plot. Notice that the force plot’s horizontal format prevents it from showing all the significant features clearly.
In contrast, the decision plot’s vertical format can display the effects of any number of features clearly. This plot was configured to show the 30 most important features out of the total 101 features.
The foregoing examples show that decision plots are similar to force plots in their explanation, but the linear, vertical format of the decision plot allows for more effects to be described.
Decision plots can effectively explain multioutput models. This example explains a Catboost classification model trained on the UCI Heart Disease data set. There are five classes that indicate the extent of the disease for each observation: Class 1 indicates no disease; Class 5 indicates advanced disease.
The decision plot below shows the model’s multiple outputs for a single observation. In this case, the x-axis represents the raw score, not a probability. The scores are included in the plot’s legend to facilitate identification. The model predicts class 4 (the dotted line). The plot shows that the model confidently predicts that disease is present (Class 1 has a very low score), but the model cannot easily differentiate between classes 3, 4, and 5.
Those familiar with the shap package know that it produces a separate base value for each model output. The decision plot displays the average of the model’s base values and shifts the SHAP values accordingly to accurately reproduce the model’s scores. This approach simplifies the visualization and allows any number of outputs to be plotted and compared on the same scale.
Decision plots support SHAP interaction values: the first-order interactions estimated from tree-based models. In contrast to a dependence plot that shows a single interaction for many predictions, a decision plot displays all main effects and interactions together.
This decision plot explains a single prediction from the UCI Adult Income data set using both main effects and interactions. It is configured to show the 20 most important effects. The plot shows plainly that several interactions drive this prediction’s score higher.
Decision plots can expose a model’s behaviors in detail. In this example, we explore how the model’s predictions change as feature values change given a specific scenario. We start with the following record in the UCI Adult Income data set.
Age                                56Workclass                   Local-govEducation-Num                      13Marital Status     Married-civ-spouseOccupation               Tech-supportRelationship                  HusbandRace                            WhiteSex                              MaleCapital Gain                        0Capital Loss                        0Hours per week                     40Country                 United-States
We create a synthetic data set by duplicating the reference record several times, but varying the capital gain values from $0 to $10,000 by $100 increments. This approach allows us to evaluate and debug the model. Analysts may also find this method useful for presenting hypothetical scenarios. Keep in mind that the effects for capital gains shown in this example are specific to the reference record, and therefore cannot be generalized.
A dependence plot can show the change in SHAP values across a feature’s value range. The SHAP values for this model represent a change in log odds. This plot shows that there is a sharp shift in SHAP values around $5,000. It also shows some significant outliers at $0 and approximately $3,000.
Though the dependence plot is helpful, it is difficult to discern the practical effects of the SHAP values in context. For that purpose, we can plot the synthetic data set with a decision plot on the probability scale. First, we plot the reference record to establish context. The features are ordered manually to match the plot of the synthetic data that follows. The prediction is probability 0.76. Capital gain is zero, for which the model assigns a small, negative effect.
Now, we plot the synthetic data. The reference record is highlighted using a dashed line. The features are ordered via hierarchical clustering to group similar prediction paths. We see that, in practical terms, the effect of capital gain is largely polarized; only a handful of predictions lie between 0.2 and 0.8.
Upon further inspection of the predictions, we found a threshold around $4,300, but there are anomalies. Capital gains of $0, $3,000, and $3,100 contribute to unexpectedly high predictions; capital gains of $5,000 contribute to unexpectedly low predictions. These anomalies are plotted here with a legend to help identify each prediction. The predictions paths for $3,000 and $3,100 are identical.
Overall, model behaviors are easy to explain using decision plots simply because several predictions can be plotted side-by-side in full detail. The remaining examples in this article take advantage of this same feature.
Decision plots are useful for comparing predictions from different models or for interpreting the predictions from ensembles of models.
The predictions from an ensemble of five LightGBM models trained on the UCI Adult Income data set are plotted below. If probability 0.5 is the threshold for this binary classification task, we see that this observation is difficult to classify. However, Model 2 is confident that the individual makes less than $50K per year. If this is a typical observation, it is worth examining why this model is different.
A decision plot can expose a model’s typical prediction paths. Here, we plot all of the predictions from the UCI Adult Income data set in the probability interval [0.98, 1.0] to see what high-scoring predictions have in common. The features are ordered using hierarchical clustering to group similar prediction paths. The plot shows two distinct paths: one is dominated by capital gain while the other is dominated by capital loss. The effects related to relationship, age, and education are also notable. Alpha blending (partial transparency) is used to indicate areas of high and low density.
For comparison, a multi-prediction force plot is shown here. It is a combination of many individual force plots that are rotated 90 degrees and stacked horizontally. There are several differences between the plots; we will point out a few. The force plot shows that approximately three-quarters of the predictions follow the prediction path dominated by capital gain. The decision plot cannot convey this sense of proportion. In the force plot, it is not evident that the predictions on the left are dominated by capital loss unless it is viewed in interactive mode where feature labels appear on mouse-over. Furthermore, the general patterns for age and relationship do not stand out. The decision plot reveals these aspects more effectively.
Decision plots are a natural addition to the shap package. They effectively communicate model decisions with the simplicity and clarity afforded by the SHAP framework. They are also flexible, allowing for a variety of use cases ranging from single-prediction explanations to model diagnostics. Please comment if you discover a new way to use decision plots.