Introducing SHAP Decision Plots
By: Floid Gilbert
Tags: Machine Learning, Data Science, Towards Data Science, Data Visualisation, Model Interpretability

In contrast, the decision plot’s vertical format can display the effects of any number of features clearly. This plot was configured to show the 30 most important features out of the total 101 features.

The foregoing examples show that decision plots are similar to force plots in their explanation, but the linear, vertical format of the decision plot allows for more effects to be described.

Visualize multioutput predictions

Decision plots can effectively explain multioutput models. This example explains a Catboost classification model trained on the UCI Heart Disease data set. There are five classes that indicate the extent of the disease for each observation: Class 1 indicates no disease; Class 5 indicates advanced disease.

The decision plot below shows the model’s multiple outputs for a single observation. In this case, the x-axis represents the raw score, not a probability. The scores are included in the plot’s legend to facilitate identification. The model predicts class 4 (the dotted line). The plot shows that the model confidently predicts that disease is present (Class 1 has a very low score), but the model cannot easily differentiate between classes 3, 4, and 5.

Those familiar with the shap package know that it produces a separate base value for each output’s SHAP values. The decision plot displays the average of the model’s base values and shifts the SHAP values accordingly to accurately reproduce the model’s scores. This approach simplifies the visualization and allows any number of outputs to be plotted and compared on the same scale.

Display the cumulative effect of interactions

Decision plots support SHAP interaction values: the first-order interactions estimated from tree-based models. In contrast to a dependence plot that shows a single interaction for many predictions, a decision plot displays all main effects and interactions together for one or more predictions.

This decision plot explains a single prediction from the UCI Adult Income data set using both main effects and interactions. It is configured to show the 20 most important effects. The plot shows plainly that several interactions drive this prediction’s score higher.

Explore feature effects for a range of feature values

Decision plots can expose a model’s behaviors in detail. In this example, we explore how the model’s predictions change as feature values change given a specific scenario. We start with the following record in the UCI Adult Income data set.

Age 56

Workclass Local-gov

Education-Num 13

Marital Status Married-civ-spouse

Occupation Tech-support

Relationship Husband

Race White

Sex Male

Capital Gain 0

Capital Loss 0

Hours per week 40

Country United-States

We create a synthetic data set by duplicating the reference record several times, but varying the capital gain values from $0 to $10,000 by $100 increments. This approach allows us to evaluate and debug the model. Analysts may also find this method useful for presenting hypothetical scenarios. Keep in mind that the effects for capital gains shown in this example are specific to the reference record, and therefore cannot be generalized.

A dependence plot can show the change in SHAP values across a feature’s value range. The SHAP values for this model represent a change in log odds. This plot shows that there is a sharp shift in SHAP values around $5,000. It also shows some significant outliers at $0 and approximately $3,000.

Though the dependence plot is helpful, it is difficult to discern the practical effects of the SHAP values in context. For that purpose, we can plot the synthetic data set with a decision plot on the probability scale. First, we plot the reference record to establish context. The features are ordered manually to match the plot of the synthetic data that follows. The prediction is probability 0.76. Capital gain is zero, for which the model assigns a small, negative effect.

Now, we plot the synthetic data. The reference record is highlighted using a dashed line. The features are ordered via hierarchical clustering to group similar prediction paths. We see that, in practical terms, the effect of capital gain is largely polarized; only a handful of predictions lie between 0.2 and 0.8.

Upon further inspection of the predictions, we found a threshold around $4,300, but there are anomalies. Capital gains of $0, $3,000, and $3,100 contribute to unexpectedly high predictions; capital gains of $5,000 contribute to unexpectedly low predictions. These anomalies are plotted here with a legend to help identify each prediction. The predictions paths for $3,000 and $3,100 are identical.

Overall, model behaviors are easy to explain using decision plots simply because several predictions can be plotted side-by-side in full detail. The remaining examples in this article take advantage of this same feature.

Compare and contrast predictions for several models

Decision plots are useful for comparing predictions from different models or for interpreting the predictions for an ensembles of models.

The predictions from an ensemble of five LightGBM models trained on the UCI Adult Income data set are plotted below. If probability 0.5 is the threshold for this binary classification task, we see that this observation is difficult to classify. However, Model 2 is confident that the individual makes less than $50K per year. If this is a typical observation, it is worth examining why this model is different.

Identify typical prediction paths

A decision plot can expose a model’s typical prediction paths. Here, we plot all of the predictions from the UCI Adult Income data set in the probability interval [0.98, 1.0] to see what high-scoring predictions have in common. The features are ordered using hierarchical clustering to group similar prediction paths. The plot shows two distinct paths: one is dominated by capital gain while the other is dominated by capital loss. The effects related to relationship, age, and education are also notable. Alpha blending (partial transparency) is used to indicate areas of high and low density.

For comparison, a multi-prediction force plot is shown here. It is a combination of many individual force plots that are rotated 90 degrees and stacked horizontally. There are several differences between the plots; we will point out a few. The force plot shows that approximately three-quarters of the predictions follow the prediction path dominated by capital gain. The decision plot cannot convey this sense of proportion. In the force plot, it is not evident that the predictions on the left are dominated by capital loss unless it is viewed in interactive mode where feature labels appear on mouse-over. Furthermore, the general patterns for age and relationship do not stand out. The decision plot reveals these aspects more effectively.