Explain your machine learning with feature importance | by Naren Santhanam | Towards Data Science

Let’s imagine that you’ve landed a consulting gig with a bank who have asked you to identify those who have a high likelihood of default on the next month’s bill. Armed with the machine learning techniques that you’ve learnt and practiced, let’s say you proceed to analyze the data set given by your client and have used a random forest algorithm that achieves a reasonably high accuracy. Your next task is to present to the business stakeholders from the client’s team how you achieved these results. What would you say to them? Will they be able to understand all the hyperparameters of the algorithm that you tweaked in order to land on your final model? How will they react when you start talking about the number of estimators and Gini criterion of the random forest?
Although it is important to be proficient in understanding the inner workings of the algorithm, it is far more essential to be able to communicate the findings to an audience who may not have any theoretical / practical knowledge of machine learning. Just showing that the algorithm predicts well is not enough. You have to attribute the predictions to the elements of the input data that contribute to your accuracy. Thankfully, the random forest implementation of sklearn does give an output called “feature importances” which helps us explain the predictive power of the features in the dataset. But, there are certain drawbacks to this method that we will explore in this post, and an alternative technique to assess the feature importances that overcomes these drawbacks.
As we saw in the introduction, we will be using the credit card default dataset from UCI Machine Learning repository, hosted on Kaggle. The dataset contains information on default payments, demographic factors, credit data, history of payment, and bill statements of credit card clients in Taiwan from April 2005 to September 2005, and comprises a good mix of continuous and categorical variables, as shown below:
The default.payment.next.month is our target variable. A value of 1 indicates that the customer will default the payment in the next month. Let us look at the distribution of this variable:
credit.groupby([‘default.payment.next.month’]).size()>default.payment.next.month>0    23364 >1     6636 >dtype: int64
This is an imbalanced dataset with ~22% of the records with default value of 1. Since our focus is on evaluating the feature importances, I will not go into examining the dataset and its features in this post. Rather, I will quickly set up a model and start looking at the scores. Let us first one-hot encode the categorical variables before setting up a model.
Let us see how a plain random forest model performs on this dataset:
Clearly we have an algorithm that is overfitting. The accuracy on the test set is lower than the training set, but not too bad. Since we are dealing with an imbalanced dataset though, the precision and recall numbers are more important. And there is a huge difference in the precision and recall between the training and test sets.
Let us examine the top feature importances behind this model:
It is surprising to note that age is the top feature contributing to the predictions of the model. We should also be mindful of an important aspect of the feature importances from the random forest model:
The feature importances from random forest are calculated based on the training data given to the model, not on predictions on a test dataset.
Thus, these numbers do not indicate the true predictive power of the model, especially one that overfits! Moreover, we saw that the performance of the model on the training data set and the test data set was vastly different. So, we need an alternative technique that will be able to calculate the feature importances on a test dataset, and also calculate it based on different metrics such as accuracy, precision or recall.
The permutation feature importance technique overcomes the drawbacks that we saw above, with a very simple concept: how does the model perform if the only the values of a feature are randomly shuffled? In other words, if we make a feature unusable for the model, what is the impact on the performance?
A few thoughts about this technique:
If we want to assess the performance of a model, why shuffle a feature in the dataset rather than removing it altogether? Removing the feature will mean that the training will need to be done all over again. We want to train the model with the feature, but test the model without it.
What if we make the feature all zeros? In some scenarios, zeros do have some value to the model. Moreover, shuffling the values around within a feature makes sure that the values stay within the range of the data.
For our scenario, let us define the permutation importance as the difference in the baseline metric caused by the permutation of a feature. The function below calculates the permutation importances when given a classifier that has already been trained:
Let us now plot the percentage change in the recall score, on the test dataset by permuting all the features:
Surprisingly, age does not feature in the top 25 features when we plot the permutation importances. This is because the feature importance method of random forest favors features that have high cardinality. In our dataset, age had 55 unique values, and this caused the algorithm to think that it was the most important feature.
Feature importances form a critical part of machine learning interpretation and explainability. Hopefully this article helped you learn more about it.
The code for this post is available on Github.