What is Explainable AI (XAI)?. An introduction to building trust in... | by Nicklas Ankarstad | Towards Data Science

There is a lot of buzz around AI these days. Almost every company either has plans to incorporate AI, is actively using it, or is rebranding their old rule-based engines as AI-enabled technologies. As more and more companies embed AI and advanced analytics within a business process and automate decisions, there needs to have transparency into how these models make decisions grows larger and larger.
How do we achieve this transparency while harnessing the efficiencies AI brings. This is where the field of Explainable AI (XAI) can help.
Explainable AI refers to methods and techniques in the application of artificial intelligence technology (AI) such that the results of the solution can be understood by human experts.
It contrasts with the concept of the “black box” in machine learning where even their designers cannot explain why the AI arrived at a specific decision. XAI is an implementation of the social right to explanation [1]
In it’s simplest form, AI takes some inputs to produce an output. When we talk about Explainable AI, we are really talking about the input variables impact on the output.
AI has the power to automate decisions and those decisions have business impacts, both positive and negative. Much like hiring decision-makers in the organization, it's important to understand how AI makes decisions. A lot of organizations want to leverage AI but are not comfortable letting the model or AI make more impactful decisions because they do not yet trust the model. Explainability helps with this as it provides insights into how models make decisions.
One of the biggest challenges I’ve found in my experience isn’t building the right model, it’s getting stakeholder buy-in that the model makes a better decision than a human. A better decision is not the same as higher accuracy scores or lower RMSE. It’s using the right inputs to derive a good answer. Often times the decision-maker needs to understand the decision. They need see how the model reasoned to be comfortable handing it off to the model to make the decision.
We build trust in AI and models the same way we do as humans. We show our work. Let’s dive into a practical example of explainability.
What if we were to try and predict the mpg on the number of cylinders a car has using a linear regression model.
This can be easily done in any programming tool or even excel these days. We find the coefficient and the intercept and plot it. The nice thing here is that we can simply see how the overall model makes decisions (global level). At the same time, the individual predictions are consistent with the global level predictions.
import requestsimport pandas as pdfrom io import StringIOimport matplotlib.pyplot as pltimport numpy as npfrom sklearn import linear_model## The URL for the dataseturl = 'https://archive.ics.uci.edu/ml/machine-learning-databases/auto-mpg/auto-mpg.data' r = requests.get(url)## Read the file in and store as dffile = r.text.replace("\t"," ")  # list_labels written manually: list_labels = ['mpg', 'cylinders', 'displacement', 'horsepower', 'weight', 'acceleration', 'model year', 'origin','car name']df = pd.read_csv(StringIO(file),sep="\s+",header = None,names=list_labels)## Modelingregr = linear_model.LinearRegression()train_x = np.asanyarray(df[['cylinders']])train_y = np.asanyarray(df[['mpg']])regr.fit (train_x, train_y)## Plotting the modelplt.scatter(df.cylinders, df.mpg,  color='blue')plt.plot(train_x, regr.coef_[0][0]*train_x + regr.intercept_[0], '-r')plt.xlabel("Cylinders")plt.ylabel("MPG")
There is a negative linear relationship, but as the plot shows our predictions are way off. For example, in the 4 cylinder category, our estimate is about 28–29ish, but the values seem to range from 45+ to 19.
We may need to add more features or try a more complex model to achieve the desired model performance.
Increasing Performance through Complexity
We may want to try a more complex model such as Gradient Boosted Decision Trees. GBMs such as XGBoost, H20, LightGBM, AdaBoost, and Catboost consistently win data science competitions for tabular data.
GBM’s converts weak learners (decision trees) into strong learners. They have lower bias and lower variance by using a sampling technique called boosting.
On the spectrum of complexity, they are not as black-box as neural networks nor as transparent as logistic regression.
When we talk about impact, we differentiate between global and local importance:
Global importance can be thought of as overall how is the model making decisions.
Local importance is how is the model making decisions for this one person.
These distinctions may seem small, but they have a significant impact as the right to explanation legislation becomes more prevalent.
Increased Complexity Example
Let’s load up a new data set. In this case, we will use the credit card default prediction dataset from UCI. The dataset is from a Taiwanese bank and includes information
The columns include information on each borrower:
Demographics
Historical repayment history
Historical bill amount
Historical previous payment amount
We can use this dataset to predict payment defaults using XGBoost. Once we have built the model, we need to evaluate its performance and since this is credit decision, we are interested in how it makes decisions.
import requestsimport pandas as pdfrom io import StringIOimport matplotlib.pyplot as pltimport numpy as npfrom sklearn import linear_modelimport shapimport xgboost as xgbfrom xgboost import XGBClassifierfrom sklearn.model_selection import train_test_splitshap.initjs()import limeimport lime.lime_tabularimport matplotlib.pylab as plfrom sklearn.metrics import confusion_matrixfrom sklearn.metrics import roc_auc_scorefrom sklearn.metrics import accuracy_score## Read in the filedf = pd.read_excel('https://archive.ics.uci.edu/ml/machine-learning-databases/00350/default%20of%20credit%20card%20clients.xls',header = 1)## Store target in y variable and split to train testy = df['default payment next month'] df = df.drop(['default payment next month','ID'], axis =1) X_train, X_test, y_train, y_test = train_test_split(df, y, test_size=0.20, random_state=42)## Train XGBoost Modelxgb_model = XGBClassifier(learning_rate = 0.05, verbosity = 1, n_estimators = 200 ,scale_pos_weight =4, random_state = 2456 )xgb_model.fit(X_train, y_train)## Predicty_pred = xgb_model.predict(X_test) roc_auc_score(y_test,y_pred)
Feature importance plots are what many data scientist use to evaluate the impact the input variables have on the output variable. Notice that depending on which metric (cover or gain), different feature come to the top. How do you explain to a customer that you declined their application due to their marital status? Is that really how the model made its decision?
## Plot importance (cover)xgb.plot_importance(xgb_model, importance_type="cover")pl.title('xgboost.plot_importance(model, importance_type="cover")')pl.show()## Plot importance (gain)xgb.plot_importance(xgb_model, importance_type="gain")pl.title('xgboost.plot_importance(model, importance_type="gain")')pl.show()
The truth is that while these feature importance plots show the overall impact of the features, they are not able to explain how the model makes decisions for individual people.
Global interpretability of models entails seeking to understand the overall structure of the model. What if I want to understand how a model made a prediction for a specific person?
Local interpretability of models consists of providing detailed explanations for why an individual prediction was made.
Let’s talk about the first local interpretability framework, often referred to as LIME.
LIME randomly samples from the prediction and uses simpler models to explain the prediction. Since it is not tied to a specific model, it is considered model agnostic.
xgb_model.fit(X_train.as_matrix(), y_train)predict_fn_xgb = lambda x: xgb_model.predict_proba(x).astype(float)explainer = lime.lime_tabular.LimeTabularExplainer(X_train.to_numpy(), feature_names = X_train.columns)exp = explainer.explain_instance(X_test.to_numpy()[observation_1], predict_fn_xgb, num_features = 9)exp.show_in_notebook()
If you look closely you can see that the top feature is Pay > 0 meaning that the person is behind on their payments.
LIME uses a decision tree to explain the more complex models. Because LIME estimates the prediction using local approximations it can sometimes be risky to use as a ground truth explanation.
SHAP is another novel approach to explainability developed by Scott Lundberg at Microsoft and eventually opened sourced.
SHAP has a strong mathematical foundation based on Shapley values in game theory where each player in the cooperation is rewarded based on how important they are to cooperation.
Here we see the same most likely defaulting borrower and the impact each variable has on the output. These local explanations are accurate and can be used to explain how XGBoost makes its decision.
## Calculate SHAP Valuesexplainer = shap.TreeExplainer(xgb_model)expected_value = explainer.expected_value## Generate forceplotshap.force_plot(expected_value,shap_values[observation_1],features_display.iloc[observation_1] ,X_test.columns,link='logit', matplotlib=True)
SHAP can illustrate local importance and the really nice thing about SHAP is that it is globally consistent. Several model frameworks can spit out SHAP values (XGBoost for example) directly from the code.
shap.summary_plot(shap_values, X_test)
The last framework I’ll mention t is Explainable Boosting Machines.
Explainable Boosting Machines or EBMs is another Microsoft open source library. Instead of being a post-hoc explanation framework, they are considered more white-box than XGBoost, while reaching similar performance.
EBM uses a framework called GA2M, which is a generalized additive model with an interaction function added to it.
The models have been proven to be as accurate as XGBoost and a GA2M won a data science contest hosted by FICO a year or so ago.
from interpret.glassbox import ExplainableBoostingClassifierfrom interpret import show## Train Modelebm = ExplainableBoostingClassifier(scoring = 'auc')ebm.fit(X_train, y_train)## Calculate Global Explainationebm_global = ebm.explain_global()## Plot Global Variable Importanceshow(ebm_global)
InterpretML is still in alpha release right now, but for tabular data GA2M models they are starting to become the de-facto standard for white-box models.
There is a very nice user interface with a dashboard that explains the predictions from the global level down to the local level, much like SHAP.
## Local explainationsebm_local = ebm.explain_local(X_test, y_test)show(ebm_local)
You should use explainability techniques such as LIME and SHAP when you need to build trust with your stakeholders by being transparent in modeling. If you are a decision-maker, always ask your data scientist or vendor for explanations of how the model makes decisions. As with almost everything in life, the best model and explanation option usually depends on the situation.
[1] Explainable Artificial Intelligence, Wikipedia
[2] Lundberg, Scott M and Lee, Su-In, A Unified Approach to Interpreting Model Predictions (2017), Advances in Neural Information Processing Systems 30
[3] H. Nori, S. Jenkins, P. Koch, and R. Caruana, InterpretML: A Unified Framework for Machine Learning Interpretability (2019)