Stop Permuting Features. Permutation importance may give you... | by Denis Vorotyntsev | Towards Data Science

Data scientists need features importances calculations for a variety of tasks. Importances could help us to understand if we have biases in our data or bugs in models. Also, importance is frequently using for understanding the underlying process and making business decisions. Indeed, the model’s top important features may give us inspiration for further feature engineering and provide insights on what is going on.
There are a lot of ways how we could calculate feature importance nowadays. Some of them are based on the model’s type, e.g., coefficients of linear regression, gain importance in tree-based models, or batch norm parameters in neural nets (BN params are often used for NN pruning, i.e., neural network compression; for example, this paper addresses CNN nets, but the same logic could be applicable to fully-connected nets). While others are “universal”, they could be applied to almost any model: methods such as SHAP values, permutation importances, drop-and-relearn approach, and many others.
Although the model’s black box unboxing is an integral part of the model development pipeline, a study conducted by Harmanpreet et al. (Interpreting Interpretability: Understanding Data Scientists’ Use of Interpretability Tools for Machine Learning) showed that not all data scientists know how to do it correctly. The availability and simplicity of the methods are making them “golden hammer”. Indeed, if one could run “pip install lib, lib.explain(model)”, why bother on the theory behind?
In this post, I’d like to address a bias of over-using permutation importance for finding the influencing features. I will show that in some cases, permutation importance gives wrong, misleading results.
Permutation importance is a frequently used type of feature importance. It shows the drop in the score if the feature would be replaced with randomly permuted values. It is calculated with several straightforward steps.
Train model with training data X_train, y_train;Make predictions for a training dataset X_train — y_hat and calculate the score — score (higher score = better);To calculate permutation importance for each feature feature_i, do the following: (1) permute feature_i values in the training dataset while keeping all other features “as is” — X_train_permuted; (2) make predictions using X_train_permuted and previously trained model — y_hat_permuted; (3) calculate the score on the permuted dataset — score_permuted; (4) The importance of the feature is equal to score_permuted — score. Lower the delta — the more important the feature.The process is repeated several times to reduce the influence of random permutations and scores or ranks are averaged across runs.
Train model with training data X_train, y_train;
Make predictions for a training dataset X_train — y_hat and calculate the score — score (higher score = better);
To calculate permutation importance for each feature feature_i, do the following: (1) permute feature_i values in the training dataset while keeping all other features “as is” — X_train_permuted; (2) make predictions using X_train_permuted and previously trained model — y_hat_permuted; (3) calculate the score on the permuted dataset — score_permuted; (4) The importance of the feature is equal to score_permuted — score. Lower the delta — the more important the feature.
The process is repeated several times to reduce the influence of random permutations and scores or ranks are averaged across runs.
Code snippet to illustrate the calculations:
Permutation importance is easy to explain, implement, and use. Although calculation requires to make predictions on training data n_featurs times, it’s not a substantial operation, compared to model retraining or precise SHAP values calculation. Also, permutation importance allows you to select features: if the score on the permuted dataset is higher then on normal — it’s a clear sign to remove the feature and retrain a model. For those reasons, permutation importance is wildly applied in many machine learning pipelines.
While being a very attractive choice for model interpretation, permutation importance has several problems, especially when working with correlated features. Giles Hooker and Lucas Mentch combined them in their paper “Please Stop Permuting Features An Explanation and Alternatives”:
Strobl et al. (2007) investigate classification and note that the importance measures based on permuted out-of-bag (OOB) error in CART built trees are biased toward features that are correlated with other features and/or have many categories and further suggest that bootstrapping exaggerates these effects;Archer and Kimes (2008) explore a similar set-up and also note improved performance when true features — those actually related to the response — are uncorrelated with noise features;Nicodemus et al. (2010) investigates these claims of feature preference in a large scale simulation study and again find that the OOB measures overestimate the importance of correlated predictors.
Strobl et al. (2007) investigate classification and note that the importance measures based on permuted out-of-bag (OOB) error in CART built trees are biased toward features that are correlated with other features and/or have many categories and further suggest that bootstrapping exaggerates these effects;
Archer and Kimes (2008) explore a similar set-up and also note improved performance when true features — those actually related to the response — are uncorrelated with noise features;
Nicodemus et al. (2010) investigates these claims of feature preference in a large scale simulation study and again find that the OOB measures overestimate the importance of correlated predictors.
The possible explanation for this is the model’s extrapolation. Suppose, the model was trained using two highly positively-correlated features x1 and x2 (left plot on the illustration below). To calculate the importance of feature x1, we shuffle the feature and make predictions for a “shuffled” points (red points on the center plot). But the model hasn’t seen any training examples of x1 in the left upper corner and right bottom corner. Thus, to make predictions, it must extrapolate to previously unseen regions (right plot). All models extrapolate badly, thus making unexpected predictions. These “points from new regions” strongly affect the final score and hence, permutation importance.
Giles Hooker and Lucas Mentch proposed several alternatives to use instead of permutation importance:
Conditional Variable Importance — permute features conditional, based on the values of remaining features to avoid “unseen regions”;Dropped Variable Importance — equivalent to the leave-one-covariate-out methods explored in Lei et al: drop feature, retrain model, compare scores.Permute-and-Relearn Importance — the approach is taken in Mentch and Hooker: permute feature, retrain model, compare scores.
Conditional Variable Importance — permute features conditional, based on the values of remaining features to avoid “unseen regions”;
Dropped Variable Importance — equivalent to the leave-one-covariate-out methods explored in Lei et al: drop feature, retrain model, compare scores.
Permute-and-Relearn Importance — the approach is taken in Mentch and Hooker: permute feature, retrain model, compare scores.
To understand, how heavily features correlation influences permutation importance and other features importances methods, I conducted the following experiment.
At first, I generated a normally-distributed dataset with a specified number of features and samples (n_features=50, n_samples=10,000). The mean of all features was equal to 0, the standard deviation was equal to 1. All dataset features correlated one with each other with a max_correlation correlation.
After a dataset is generated, I added a uniformly-distributed noise to each feature. The noise magnitude for each feature was selected randomly from an uniform distribution between [-0.5*noise_magnitude_max, 0.5*noise_magnitude_max], noise_magnitude_max = var. This was done to decrease features correlation. Features for the task are ready!
Now we need to create a target. For each feature, I generated a weight, which was sampled from a gamma distribution with specified gamma and scale parameters (gamma=1, scale=1). Gamma distribution was selected because it looks very similar to a typical feature importance distribution. Each feature weight was then divided by the sum of weights, making the sum of weights equal to one. That was done to reduce the influence of random weights generation on the final results.
Then the logit of a target was calculated as a linear combination of feature and corresponding feature weight (a sign of feature weight was selected at random). A sigmoid function was applied to a standard-scaled logit of a target. To get the label, I rounded the result. The target is ready!
For a generated dataset and target, I trained a single LightGBM model with the following parameters:
learning_rate: 0.01n_estimators: 100random_state: 42
All other parameters were default. I calculate the importance rank of each feature using built-in gain importance, SHAP importance, and permutation importance (permutation is repeated five times, deltas scores are averaged).
Then I calculate the Spearman rank correlation between calculated importance and actual importances of features. Actual importances are equal to rank(-weights). The best possible correlation is 1.0, i.e. features importances are in the same order as actual importances (weights of features).
Data of each experiment (dataset correlation statistics, Spearman rank correlation between the model’s importance and actual importance of features for built-in gain importance, SHAP importance, and permutation importance) was saved for further analysis. The experiment is run fifty times with different seeds and with varying combinations of max_correlation and noise_magnitude_max. I also run the same experiment with “drop and relearn” and “permute and relearn” approaches but only five times due to required heavy computations.
The code and analysis of the experiment could be found in the repository of the project:
github.com
To make you familiar with what is going on, I’ll illustrate a single experiment. The experiment illustration notebook could be found here: experiment illustration. Parameters of the experiment are:
TASK = "classification"OBJECTIVE = "binary"METRIC = roc_auc_scoreMU = 0VAR = 1N_FEATUES = 50N_SAMPLES = 10_000NOISE_MAGNITUDE_MAX = 1SEED = 42
Part of the correlation matrix of the generated dataset:
We may see that the features are highly correlated with one with others (mean absolute correlation is about 0.96). Statistics of correlation:
Distribution of generated features weights:
Calculated Spearman rank correlation between calculated importance and actual importances of features:
Model's score [train data]: 0.9998Permutation spearman corr: 0.5724SHAP spearman corr: 0.4721LGB gain spearman corr: 0.4567
And the illustration of expected and calculated features importances ranks:
We may see several problems here (marked with green circles):
The most important and second most important features ranks are mismatched3rd most important feature according to permutation importance should be 9th;Actual 8th important features dropped to 39th position if we trust permutation importance.
The most important and second most important features ranks are mismatched
3rd most important feature according to permutation importance should be 9th;
Actual 8th important features dropped to 39th position if we trust permutation importance.
Here’s an illustration of expected and calculated features importances ranks for the same experiment parameters, except NOISE_MAGNITUDE_MAX, which is now equal to 10 (abs_correlation_mean dropped from 0.96 to 0.36):
Model's score [train data]: 0.9663Permutation spearman corr: 0.6430SHAP spearman corr: 0.7139LGB gain spearman corr: 0.6978
Still not perfect, but even visually much better, if we are talking about the top ten most important features.
I conducted the described experiments and plot results, which are presented in this section. A total of 1200 runs was made for “Permutations vs SHAP vs Gain” and 120 runs for “Permutations vs Relearning” experiments.
In this subsection, I compare feature importances ranks calculated using permutation importances, SHAP values, and built-in gain.
From the following plots, we may see that correlation between actual features importances and calculated using permutations, SHAP values, and gain is, as expected, negatively correlated with the mean and max of features correlation. Permutation importance suffers the most from highly correlated features. There is no difference between importance calculated using SHAP of built-in gain.
Also, we may see that that correlation between actual features importances and calculated depends on the model’s score: higher the score — lower the correlation (Figure 10 — Spearman features rank correlation = f(model’s score)). It’s not clear, why that happened, but I may hypothesis, that more correlated features lead to more accurate models (which could be seen from Figure 11— Model’s score= f(mean of feature correlations)), because of denser features spaces and fewer “unknown” regions.
In this subsection, I compare permutation importances with relearning approaches.
Surprisingly, relearning approaches performed significantly worse than permutation across all correlations, which could be seen from plots below. Also, relearning approaches took approximately n_features times more time to run.
Don’t use permutation importance for tree-based models interpretation (or any model which interpolates in unseen regions badly).Use SHAP values or built-in gain importance instead.Don’t use “permute-and-relearn” or “drop-and-relearn” approaches for finding important features.
Don’t use permutation importance for tree-based models interpretation (or any model which interpolates in unseen regions badly).
Use SHAP values or built-in gain importance instead.
Don’t use “permute-and-relearn” or “drop-and-relearn” approaches for finding important features.
In this post I described the permutation importance approach and problems associated with it. I showed how and why highly correlated features might affect permutation importance, which will give misleading results. I conducted an experiment, which showed that permutation importance suffers the most from highly correlated features (among importances calculated using SHAP values and gain). I also showed that, despite relearning approaches expected to be promising, they perform worse then permutation importances, and require much more time to run.
I hope this post will help data scientists to interpret their models correctly.
The topic of the post and conducted experiment were inspired by “Please Stop Permuting Features An Explanation and Alternatives”, work done by Giles Hooker and Lucas Mentch.
If you liked this, you might be interested in reading my other post on problems with LIME importance: