Visualizing AI. Deconstructing and Optimizing the SHAP... | by Wai On | Towards Data Science

Now that you understand how the various components of the SHAP Summary Plot work together (part 1), I will provide an example of its use in explaining a black box Machine Learning model. In addition, I will discuss some of the problems with the visualization in the example before offering some ideas for improving it. The goal of these articles is to help the reader interpret the visualization, optimize it, and to arrive at a deeper understanding of the results.
The code for this example can be found here on GitHub.
The data I’m using is the HR data from IBM that is available at Kaggle. The original purpose of the data was to understand attrition, but here I’m using it to understand performance rating in the workplace.
To reduce the amount of data-prep necessary, I’ve simplified the data by pairing down the number of features. In addition, an examination of the data shows that the performance ratings are restricted to either a 3 or a 4 (out of a scale of 1–4). So in order to use the data to illustrate the capabilities of the Summary Plot for regression data, I’ve re-scaled the rating to a real number between 0–9 by applying a set of transformations (see data file on GitHub).
In the discussion of the results below, I will analyze this supposedly fictional data and hypothesize a possible story behind the data and the resulting model.
For this exercise, I used the Random Forest algorithm from scikit-learn and used the SHAP Tree Explainer for explanation.
model = RandomForestRegressor(max_depth=4, random_state=1,               n_estimators=10)model.fit(X_train, y_train)shap_values = shap.TreeExplainer(model).shap_values(X_train)
The following diagram shows the process of using SHAP in our example.
The bar graph below shows the feature names and their corresponding mean SHAP value.
A bar graph of this type is the typical presentation of results for feature attribution algorithms. Here, it lists the features in descending order of importance, according to the mean SHAP value.
As shown above, the feature with the highest mean SHAP value is “PercentSalaryHike”. This value is so dominant that it makes it difficult to tell the relative difference of other top influencers. In fact, “PercentSalaryHike” is so dominant that it suggests that something might be problematic with the data or our interpretation of it.
Let’s investigate further. The Summary Plot looks like this:
The Summary Plot above shows that high values (red points) of “PercentSalaryHike” are associated with positive SHAP values and low values (blue points) of “PercentSalaryHike” are, in general, associated with negative SHAP values.
Since the original data does not specify the timing of Salary Hike relative to Performance Rating, there are two interpretations of the result:
High performance rating is the result of high salary hike from the previous salary review.High salary hike is the result of high performance rating. In other words, employees are rewarded for good work performance with a good salary hike.
High performance rating is the result of high salary hike from the previous salary review.
High salary hike is the result of high performance rating. In other words, employees are rewarded for good work performance with a good salary hike.
In my view, the logical interpretation is probably #2. That is, employees received a high percentage increase in salary because they performed well. Of course, this is conjecture on my part but it serves as a good reminder that XAI tools reflect the nature of the underlying Machine Learning algorithms in general. Specifically, they operate in the realm of correlation and their results are ambiguous in terms of causal directions. The result, therefore, does not tell us that “PercentSalaryHike” is a cause of good work performance, it only tells us it is highly correlated. In this case, significantly more so than other features in the model.
Of the top five features, “Age” also seems to follow a pattern where high values are associated with high SHAP values and low values are associated with low SHAP values. This pattern also seems to be applied to “StockOptionLevel” in the Summary Plot, albeit with less contrast.
The SHAP values for the remaining features seem to cluster around zero but it’s hard to see the details because of scaling needed in the plot. That is, the visualization has to be compressed in order to accommodate the large values from the first feature.
Clearly, although the Summary Plot is useful as it is, there are a number of problems that are preventing us from understanding the result more easily. In this section, I will discuss some of these and to offer suggestions for tackling them in SHAP.
First and foremost is the use of the color red to indicate high values. Red is an intensely emotional color. It is typically used as a warning of danger and generally has a negative connotation in western cultures. The use of red in and off itself is not the problem. However, in our example, it is particularly problematic since we are dealing with monetary values, experience, and age, where a high value may not be a negative thing. Making sense of a plot in this type of situation is particularly difficult because of the cognitive interference due to interactions between color and feature value.
The second issue is that the transition from blue to red may cause differentiation issues for some users due to color insensitivity. In particular, the use of blue and purple next to one another being a combination to avoid.
The mistake that the Summary Plot designers made is similar to those that use the “rainbow color” scheme for data visualization (e.g. in “heat maps”). That is, the rainbow color palette looks attractive but is perceptually and semantically problematic.
There have been discussions on providing custom color palettes for the ‘dot’ type (primarily as a way to deal with business requirements) but so far it’s still forthcoming. To address the issues discussed, I tweaked the source code and removed the use of red. I also chose a single color but varied the luminosity to make it easier to see the color difference.
The new Summary Plot looks like this:
I think the resulting plot is easier to parse at a glance since there are fewer colors to decipher. In addition, switching to a more luminous blue as an indication of high numbers make the higher values standout. However, this may be a problem for users who are accustomed to seeing blue as the low number. If this is the case, I would recommend flipping the two end colors.
Following a well-known design strategy, I eliminated some of the information that was not useful. Since most of the values for the features are so close to each other and since most the SHAP values are close to zero, it makes sense for us to focus on the top set of features by setting the max_display parameter:
shap.summary_plot(shap_values, X_train, max_display=5)
As mentioned above, it’s unclear if “PercentSalaryHike” was a prior measure or a post measure of performance rating. If the employee received a high salary hike as a result of a good performance rating then this is less of an interest since we are interested in factors that contribute to good performance. Although it may also be interesting to see if a previous salary hike has any effect on performance rating, given it is unclear what the timing of salary increase is, it’s prudent to eliminate this altogether.
The above comments on “PercentSalaryHike” ambiguity can also be applied to other features associated with monetary values. For the top 5, this includes “StockOptionLevel”. For our analysis, we can assume that others (e.g. distanceFromWork, Age) stay relatively similar whether they are post or pre-measure of work performance rating.
Since “PercentSalaryHike” is so dominant, the scale for the x-axis has to extend to accommodate for the magnitude of the values. As a result, it is difficult to see how the SHAP values are distributed.
Luckily, removing the features “PercentSalaryHike” and “StockOptionLevel” from the top five features automatically address this issue. The new summary plot for our model is as shown below with a better suited magnification.
The resulting plot is simpler and easier to understand. The plot shows that higher values of total working years and age correlate with higher SHAP values (which in turn means higher performance ratings). In addition, low values also correlates with low SHAP values. This supports the thinking that maturity and work experience contribute to good work performance.
The third feature is more difficult to interpret. Higher values of “DistanceFromHome” (that is, living further from a place of work) seem to correlate with high SHAP values. Without understanding work from home policies and how employees feel about working from home, it’s hard to interpret the result. In addition, the contrast between the color patches isn’t particularly clear since the plot shows some high values also negatively impact performance rating (light blue dots to the left). This result shows this is an area worth looking into and suggests further research from HR in the areas of commuting and the effects of work from home policies in general.
Of course, this is fictitious data so any interpretation here is purely an exercise in demonstrating the potential analysis that could be done. In addition to the small size of the SHAP values and the small difference between them, the fact that the Summary Plot shows that most of the values tend to cluster around zero also reminds us that it’s best to interpret the results cautiously as well as the need to gather additional corroborative evidence.
Although designing better XAI systems, in general, is beyond the scope of this article, I want to briefly discuss some tentative ideas for improvements that current tools can make immediately; particularly in the areas of User Experience and Data Visualizations design.
Providing better contrast and color choice. Not only avoiding colors that might be confusing (e.g. using red to indicate a good outcome) but also providing choice in color palettes that are easily discernible and universal to all users.
Be able to interactively remove specific features from visualizations. Research from Social Science suggests that “explanations are selected” in the sense that we pick one or two causes from a set of possible causes as being the explanation. For XAI tools to be used by end users, visualizations need to become more interactive. They need to allow users to easily select a set of features and focus on them without the need to explicitly do so programmatically.
Support iterative exploration. A recent study has shown that much of the process of finding an explanation is iterative. Indeed, the process described above in localizing a handful of influential features follows the pattern of iterative analysis in finding an explanation. The ability to see results, explore different possibilities, contrast different features, examine their interaction, etc. and do this iteratively is immensely valuable for users at the receiving end of explanations.
Support transition to and from a high level view to an individual view at the feature level. Following up on the earlier point, the ability to zoom in/out, transition from focusing on one feature or contrast a group of features would allow users to follow their investigation as well as deal with scaling issues that we encountered in our example.
Without waiting for major research breakthroughs in XAI, I think there are some concrete improvements that can be made immediately so that they can be used to better explain the results of Machine Learning models. I hope to write more about this topic in the future.
In this article, I used the SHAP Summary Plot to explain the influences behind a black box model’s output. Using the visualization, I hypothesized a possible story behind the data and the resulting model. As part of the process of telling a hypothetical story, I identified a number of ambiguities in the data as well as problems with the design of the SHAP Summary Plot. I then offered some ideas for improving the visualization as well as identifying further work needed to clarify the data. Finally, I provided some tentative ideas for XAI systems so that they can be tailored to the way users behave when seeking explanations.
SHAP comes with a number of rich visualizations that deserve more attention than they are currently receiving. If you are a Data Scientist responsible for delivering systems or reports to end users, then hopefully this article has helped you to make the necessary adjustments to use the visualization more effectively. If you are an end user decision maker or end user consumer, you should now have a better understanding of the SHAP Summary Plot. This, in turn, will hopefully empower you to demand results to be presented in a way that is easier to understand.