Interpretable vs Explainable Machine Learning
By: Conor O'Sullivan
Tags: Artificial Intelligence, Machine Learning, Data Science, Towards Data Science, Statistics

Following on from this example, suppose we want a model that predicts the maximum loan size given to a person (Y) using a person’s age and income. We use linear regression and come up with the following equation:

Y = 100*age + 10*income + 200

We see that for every year a person ages their predicted maximum loan increases by $100. Similarly, the loan size increases as income increases. For the person (26 years old, $3000 income ) the maximum loan size is predicted to be $32800.

So like the decision tree, we can look at this model’s parameters and understand exactly why it makes the predictions that it does. This is because these models are fairly simple. The decision tree only has a few nodes and the linear regression model only has 3 parameters. As models become more complicated we can no longer understand them in this way.

Explainable Machine Learning

You can think of an ML model as a function. The model features are the input and the predictions are the output. An explainable model is a function that is too complicated for a human to understand. Another name for this is a black-box model. We need an additional method/technique to be able to peer into the black-box and understand how the model works.

An example of such a model would be a Random Forest. Put simply, a Random Forest is made up of many Decision Trees where the predictions of all the individual trees are taken into account when making the final prediction. To understand how a Random Forest works we would have to simultaneously understand how all of the individual trees work. Even with a small number of trees, this would not be possible.

The Architecture of AlexNet (Source: Satya Mallick & Sunita Nayak)

Things get even more complicated when we start to look at algorithms like Neural Networks. To put it into perspective, AlexNet, a Convolutional Neural Network used for image recognition, has 62,378,344 parameters. In comparison, our simple regression model above has only 3 parameters. It is simply not possible for a human to comprehend how a model, like AlexNet, works by just looking at the parameter weights.

Techniques Used to Understand Explainable Models

We need some additional techniques to understand how algorithms like the ones discussed above make predictions. These could include techniques such as a feature importance score. These techniques try to determine how well each of the individual features can predict the target variable. The higher the score the more important the feature is to the model or in other words the more the feature influences the model’s prediction.

There are also a whole host of techniques that try to decompose individual predictions. So, unlike feature importance which looks at the predictions overall, these techniques would explain how each feature has contributed to one specific prediction. This includes methods, such as LIME and SHAP, which are model agnostic. Meaning they can be used to explain the predictions of any ML model. They also include methods created for specific models, such as DeepLIFT which is used for explaining the predictions of Neural Networks.

A level of caution should be taken when making any conclusions using these techniques. This is because these techniques can only provide approximations of how the model actually makes predictions. To validate any conclusions, multiple techniques can be used in combination or they could be validated using data visualisations. Domain knowledge can also be an important tool. Any results to the contrary of previous experience/knowledge should be analysed in more detail.

Interpretability

Up to this point, we have discussed models as either being interpretable or explainable but it may not make sense to apply this binary flag. This is because the interpretability of a model is on a spectrum where some models are more interpretable than others. In other words, interpretability is the degree to which a model can be understood in human terms. One model is more interpretable than another if it is easier for a human to understand how it makes predictions than the other model.

In Figure 2, we can see what is meant by an interpretability spectrum. A Convolutional Neural Network is less interpretable than a Random Forest which is less interpretable than a Decision Tree. Most models could generally be classified as interpretable or explainable where explainable models would fall to the right and interpretable models to the left of the spectrum. However, there is a grey area where you would find that people would disagree on the classification.