Explainable AI (XAI) with SHAP - regression problem | by Idit Cohen | Towards Data Science

Model explainability becomes a basic part of the machine learning pipeline. Keeping a machine learning model as a “black box” is not an option anymore. Luckily there are tools that are evolving rapidly and becoming more popular. This guide is a practical guide for XAI analysis of SHAP open-source Python package for a regression problem.
SHAP (Shapley Additive Explanations) by Lundberg and Lee (2016) is a method to explain individual predictions, based on the game theoretically optimal Shapley values. Shapley values are a widely used approach from cooperative game theory that come with desirable properties. The feature values of a data instance act as players in a coalition. The Shapley value is the average marginal contribution of a feature value across all possible coalitions[1].
In this guide, we will use the Boston house prices dataset example from sklearn datasets. It is a simple regression problem[2].
boston = datasets.load_boston()X_train, X_test, y_train, y_test = model_selection.train_test_split(boston.data, boston.target, random_state=0)
After splitting the dataset to train and test, create the model and fit.
regressor = ensemble.RandomForestRegressor()regressor.fit(X_train, y_train);
With SHAP package the calculation is quite simple and straightforward. We only need the model (regressor) and the dataset (X_train).
# Create object that can calculate shap valuesexplainer = shap.TreeExplainer(regressor)# Calculate Shap valuesshap_values = explainer.shap_values(X_train)
After calculating the SHAP values we can plot several analyses that will help us understand the model.
As a first step, I recommend viewing the feature importance. It is a basic insight into the model. In the following figure, you can see a comparison between feature importance calculated by SHAP values (features with large absolute Shapley values are important) and feature importance computed as the mean and standard deviation of accumulation of the impurity decrease within each tree (using scikit-learn [3]). As you can see they look quite similar but they are not the same.
shap.summary_plot(shap_values, X_train, feature_names=features, plot_type="bar")
The summary plot combines feature importance with feature effects. Each point on the summary plot is a Shapley value of an instance per feature. The position on the y-axis is determined by the feature and on the x-axis by the Shapley value of each instance. You can see that the feature LSTAT is the most important feature, has a high Shapley value range. The color represents the value of the feature from low to high. Overlapping points are jittered in the y-axis direction, so we get a sense of the distribution of the Shapley values per feature. The features are ordered according to their importance.
shap.summary_plot(shap_values, X_train, feature_names=features)
In the summary plot, we see first indications of the relationship between the value of a feature and the impact on the prediction. But to see the exact form of the relationship, we have to look at SHAP dependence plots.
The partial dependence plot (PDP or PD plot) shows the marginal effect that one or two features have on the predicted outcome of a machine learning model (J. H. Friedman 2001 [4]). A partial dependence plot can show whether the relationship between the target and a feature is linear, monotonic, or more complex. The partial dependence plot is a global method: The method considers all instances and gives a statement about the global relationship of a feature with the predicted outcome. The PDP assumes that the first feature is not correlated with the second feature. If this assumption is violated, the averages calculated for the partial dependence plot will include data points that are very unlikely or even impossible.
A dependence plot is a scatter plot that shows the effect of a single feature on the predictions made by the model. In this example, the property value increases significantly when the average number of rooms per dwelling is higher than 6.
Each dot is a single prediction (row) from the dataset.
The x-axis is the actual value from the dataset.
The y-axis is the SHAP value for that feature, which represents how much knowing that feature value changes the output of the model for that prediction.
The color corresponds to a second feature that may have an interaction effect with the feature we are plotting (by default this second feature is chosen automatically). If an interaction effect is present between this other feature and the feature we are plotting it will show up as a distinct vertical pattern of coloring.
shap.dependence_plot(5, shap_values, X_train, feature_names=features)
In the above example, we can see that the average number of rooms per dwelling is above 7.5 the CRIM is always low. The Shapley values of these cases are high, meninge the results greatly. This insight is a function of dependence interaction between RM, CRIM features.
This plot gives us the explainability of a single model prediction. Force plot can be used for error analysis, finding the explanation to specific instance prediction.
i = 18shap.force_plot(explainer.expected_value, shap_values[i], X_test[i], feature_names = features)
From the plot we can see:
The model output value: 16.83The base value: this is the value that would be predicted if we didn’t know any features for the current instance. The base value is the average of the model output over the training dataset (explainer.expected_value in the code).The numbers on the plot arrows are the value of the feature for this instance. CRIM, per capita crime rate by town = 0.06664 and RM, the average number of rooms = 6.546Red represents features that pushed the model score higher, and blue representing features that pushed the score lower.The bigger the arrow, the bigger the impact of the feature on the output. The amount of decrease or increase in the impact can be seen on the x-axis.CRIM of 0.066 increases the property value, RM of 6.546 reduces the property value.
The model output value: 16.83
The base value: this is the value that would be predicted if we didn’t know any features for the current instance. The base value is the average of the model output over the training dataset (explainer.expected_value in the code).
The numbers on the plot arrows are the value of the feature for this instance. CRIM, per capita crime rate by town = 0.06664 and RM, the average number of rooms = 6.546
Red represents features that pushed the model score higher, and blue representing features that pushed the score lower.
The bigger the arrow, the bigger the impact of the feature on the output. The amount of decrease or increase in the impact can be seen on the x-axis.
CRIM of 0.066 increases the property value, RM of 6.546 reduces the property value.
If we want more global representation of the previous predictions, we can use a variant of the force plot. Here we can see the predictions for a group of rows placed vertically (rotated of 90°) and side by side. In the below figure we can see the first 5 rows in the dataset.
# visualize the first 5 predictions explanations with a dark red dark blue color map.shap.force_plot(explainer.expected_value, shap_values[0:5,:], X_test[0:5,:], plot_cmap="DrDb", feature_names=features)
The Decision Plot shows essentially the same information as the Force Plot. The grey vertical line is the base value and the red line indicates if each feature moved the output value to a higher or lower value than the average prediction.
This plot can be a little bit more clear and intuitive than the previous one, especially when there are many features to analyze. In the Force Plot, the information may look very condensed when the number of predictors is high.
shap.decision_plot(explainer.expected_value[0], shap_values[0], feature_names = list(features))
The decision plot’s straight vertical line marks the model’s base value. The colored line is the prediction. Feature values are printed next to the prediction line for reference. Starting at the bottom of the plot, the prediction line shows how the SHAP values accumulate from the base value to arrive at the model’s final score at the top of the plot. Decision plots are a literal representation of SHAP values, making them easy to interpret.
The force plot and the decision plot are both effective in explaining the foregoing model’s prediction. The magnitude and direction of the major effects are easy to identify.
Stacking decision plots together can help locate the outliers based on their SHAP values. In the figure above you can see an example of a different dataset, for outliers detection with SHAP decision plots.
The SHAP framework has proved to be an important advancement in the field of machine learning model interpretation. SHAP combines several existing methods to create an intuitive, theoretically sound approach to explain predictions for any model. SHAP values quantify the magnitude and direction (positive or negative) of a feature’s effect on a prediction[6]. I believe XAI analysis with SHAP and other tools should be an integral part of the machine learning pipeline. For more about XAI for multiclass classification problems with SHAP see the link. The code in this post can be found here.