Which Of Your Features Are Overfitting? | by Samuele Mazzanti | Towards Data Science

What matters most in machine learning is making good predictions on new data.
When predictions are good on training data but bad on test data, it is said that the model is “overfitting”. It means that the model has learnt too many noisy patterns from training data, and so it’s unable to generalize well to data it has not seen before.
Who’s to blame for overfitting? In other words,
Which are the features (dataset’s columns) that prevent the model from generalizing well on new data?
In this article — with the aid of a real-world dataset — we will see an advanced method to answer this question.
If your answer to the question above was “I would look at feature importance”, try harder.
Feature importance says nothing about how the features will perform on new data.
In fact, it is just a proxy of what the model has learnt during the training phase. If the model has learnt many patterns about feature ‘Age’, then this feature will rank high in feature importance. This doesn’t say anything about whether those patterns are correct or not (by “correct”, I mean a pattern that is general enough to hold true also on new data).
So, we will need a different approach to solve the problem.
In order to explain the approach, I will use a dataset containing records from the German health registry from 1984 to 1988 (the dataset is accessible from the library Pydataset, under the MIT license).
Downloading the data is as simple as:
import pydatasetX = pydataset.data('rwm5yr')y = (X['hospvis'] > 1).astype(int)X = X.drop('hospvis', axis = 1)
This dataset consists of 19,609 rows, where each row contains some information about a patient for a given year. Note that patients are observed across different years, so the same patient may be found in different rows of the dataframe.
The target variable is:
`hospvis`: whether the patient has been more than 1 day in hospital during the corresponding year.
We dispose of 16 columns:
`id`: patient ID (1-7028);`docvis`: number of visits to doctor during year (0-121);`year`: year (1984-1988);`edlevel`: educational level (1-4);`age`: age (25-64);`outwork`: 1 if out of work, 0 else;`female`: 1 if female, 0 else;`married`: 1 if married, 0 else;`kids`: 1 if has children, 0 else;`hhninc`: household yearly income in marks (in Marks);`educ`: years of formal education (7-18);`self`: 1 if self-employed, 0 else;`edlevel1`: 1 if less than high school graduate, 0 else;`edlevel2`: 1 if high school graduate, 0 else;`edlevel3`: 1 if university/college, 0 else;`edlevel4`: 1 if graduate school, 0 else.
`id`: patient ID (1-7028);
`docvis`: number of visits to doctor during year (0-121);
`year`: year (1984-1988);
`edlevel`: educational level (1-4);
`age`: age (25-64);
`outwork`: 1 if out of work, 0 else;
`female`: 1 if female, 0 else;
`married`: 1 if married, 0 else;
`kids`: 1 if has children, 0 else;
`hhninc`: household yearly income in marks (in Marks);
`educ`: years of formal education (7-18);
`self`: 1 if self-employed, 0 else;
`edlevel1`: 1 if less than high school graduate, 0 else;
`edlevel2`: 1 if high school graduate, 0 else;
`edlevel3`: 1 if university/college, 0 else;
`edlevel4`: 1 if graduate school, 0 else.
Let’s split the data into a training and a test set. There are more sophisticated ways to do that, such as cross-validation, but let’s keep things simple. Since this is an experiment, we will (naively) treat all of the columns as numeric features.
from sklearn.model_selection import train_test_splitfrom catboost import CatBoostClassifierX_train, X_test, y_train, y_test = train_test_split(  X, y, test_size = .2, stratify = y)cat = CatBoostClassifier(silent = True).fit(X_train, y_train)
Once the model is trained, let’s give a look at feature importance:
import pandas as pdfimpo = pd.Series(cat.feature_importances_, index = X_train.columns)
Not surprisingly, `docvis` — the number of visits to a doctor — is important in predicting whether the patient has spent more than 1 day in hospital. Also `age` and `hhninc` (income) are somewhat obvious. But the fact that the patient’s id ranks 2nd in importance should make us suspicious, especially because we have treated it as a numeric feature.
Now, let’s calculate the model performance (area under ROC) both on the training set and on the test set.
from sklearn.metrics import roc_auc_scoreroc_train = roc_auc_score(y_train, cat.predict_proba(X_train)[:, 1])roc_test = roc_auc_score(y_test, cat.predict_proba(X_test)[:, 1])
It’s a huge difference! This is an indication of strong overfitting. But which features are “responsible” for that?
We have many metrics to measure how a model performs on some data. But how do we measure how a feature performs on some data?
The most powerful tool to do that is called “SHAP values”.
In general, to efficiently compute SHAP values for any predictive model, you can use the dedicated Python library. In this example, however, we will take advantage of Catboost’s native method:
from catboost import Poolshap_train = pd.DataFrame(  data = cat.get_feature_importance(    data = Pool(X_train),     type = 'ShapValues')[:, :-1],   index = X_train.index,   columns = X_train.columns)shap_test = pd.DataFrame(  data = cat.get_feature_importance(    data = Pool(X_test),     type = 'ShapValues')[:, :-1],   index = X_test.index,   columns = X_test.columns)
If you take a look at shap_train and shap_test, you will notice that they are the same shape of the respective datasets.
If you want to know more about SHAP values and how they work, you can start from here and here. But — for our purpose — all you need to know is that SHAP values give you an idea of the impact of each single feature on the final prediction made by a model, on one or more observations.
Let’s see a couple of examples:
Patient at row 12071 has had 0 visits. The corresponding SHAP value (-0.753) tells us that this piece of information lowers by -0.753 the probability (actually the log-odds) that he has been more than 1 day in hospital. On the contrary, patient at row 18650 has been 4 times to a doctor, this raises by 0.918 the log-odds that she has been more than 1 day in hospital.
Intuitively, a good proxy of the performance of a feature on a dataset is the correlation between the SHAP values of the feature and the target variable. In fact, if the model has found good patterns on a feature, the SHAP values of that feature must be highly positively correlated with the target variable.
For example, if we wish to calculate the correlation between the feature `docvis` and the target variable on the observations contained in the test set:
np.corrcoef(shap_test['docvis'], y_test)
However, SHAP values are additive, meaning that the final prediction is the sum of the SHAPs of all the features. So it makes more sense if we remove the effect of other features before calculating the correlation. This is exactly the definition of “partial correlation”. A convenient implementation of partial correlation is contained in the Python library Pingouin:
import pingouinpingouin.partial_corr(  data = pd.concat([shap_test, y_test], axis = 1).astype(float),   x = 'docvis',   y = y_test.name,  x_covar = [feature for feature in shap_test.columns if feature != 'docvis'] )
This snippet of code means “calculate the correlation between SHAP values of feature `docvis` and the target variable on the observations of the test set, after removing the effect of all the other features”.
Since everybody needs a name, I will call this formula “ParShap” (from “Partial correlation of Shap values”).
We can repeat this procedure for each feature, both on train set and test set:
parshap_train = partial_correlation(shap_train, y_train)parshap_test = partial_correlation(shap_test, y_test)
Note: you can find the definition of the function partial_correlation at the end of this article.
Now, let’s plot parshap_train on the x-axis and parshap_test on the y-axis.
plt.scatter(parshap_train, parshap_test)
If a feature lies on the bisector, it means it performs exactly the same on train set and on test set. It’s the ideal situation, when there is neither overfitting nor underfitting. On the contrary, if a feature lies below the bisector, it means it has performed worse on test set than it did on training set. This is the overfitting area.
Visually, it’s immediately apparent which features are underperforming: I have circled them in blue.
Therefore, the arithmetic difference between parshap_test and parshap_train (which equals the vertical distance between each feature and the bisector) gives us a measure of how much the feature is overfitting, for our model.
parshap_diff = parshap_test — parshap_train
How should we interpret this output? Based on what we have said above, the more negative this score, the more overfitting is brought by the feature.
It’s fine: you don’t have to! Can you figure out a way to check whether the intuition behind this article is correct?
In logic, if we remove the “overfitting features” from the dataset, we should be able to reduce the overfitting (namely, the distance between roc_train and roc_test).
So let’s try to drop one feature at a time and see how the area under ROC changes.
On the left, we are removing one feature at a time, sorted by feature importance. So, first the least important (`edlevel4`) is removed, then the two least important (`edlevel4` and `edlevel1`) are removed, and so on.
On the right, we are following the same process, but the order of removal is given by ParShap. So, first the most negative ParShap (`id`) is removed, then the two most negative ParShap (`id` and `year`) are removed, and so on.
As we had hoped, dropping features with most negative ParShap has led to a strong decrease in overfitting. In fact, roc_train is getting closer and closer to roc_test.
Note that this was only a test to check the correctness of our line of reasoning. In general, ParShap should not be used as a method for feature selection. Indeed, the fact that some features are prone to overfitting does not imply that those features don’t carry useful information at all! (For instance, income and age, in this example).
However, ParShap proves extremely helpful in giving us hints on how to debug our model. In fact, it allows us to focus the attention on those features that require some more feature engineering or regularization.
Full code used for this article (you may obtain slightly different results due to random seed):
# Import librariesimport pandas as pdimport pydatasetfrom sklearn.model_selection import train_test_splitfrom catboost import CatBoostClassifier, Poolfrom sklearn.metrics import roc_auc_scorefrom pingouin import partial_corrimport matplotlib.pyplot as plt# Print documentation and read dataprint('################# Print docs')pydataset.data('rwm5yr', show_doc = True)X = pydataset.data('rwm5yr')y = (X['hospvis'] > 1).astype(int)X = X.drop('hospvis', axis = 1)# Split data in train + testX_train, X_test, y_train, y_test = train_test_split(X, y, test_size = .2, stratify = y)# Fit modelcat = CatBoostClassifier(silent = True).fit(X_train, y_train)# Show feature importancefimpo = pd.Series(cat.feature_importances_, index = X_train.columns)fig, ax = plt.subplots()fimpo.sort_values().plot.barh(ax = ax)fig.savefig('fimpo.png', dpi = 200, bbox_inches="tight")fig.show()# Compute metricsroc_train = roc_auc_score(y_train, cat.predict_proba(X_train)[:, 1])roc_test = roc_auc_score(y_test, cat.predict_proba(X_test)[:, 1])print('\n################# Print roc')print('roc_auc train: {:.2f}'.format(roc_train))print('roc_auc  test: {:.2f}'.format(roc_test))# Compute SHAP values  shap_train = pd.DataFrame(  data = cat.get_feature_importance(data = Pool(X_train), type = 'ShapValues')[:, :-1],  index = X_train.index,   columns = X_train.columns)shap_test = pd.DataFrame(  data = cat.get_feature_importance(data = Pool(X_test), type = 'ShapValues')[:, :-1],  index = X_test.index,   columns = X_test.columns)print('\n################# Print df shapes')print(f'X_train.shape:    {X_train.shape}')print(f'X_test.shape:     {X_test.shape}\n')print(f'shap_train.shape: {shap_train.shape}')print(f'shap_test.shape:  {shap_test.shape}')print('\n################# Print data and SHAP')print('Original data:')display(X_test.head(3))print('\nCorresponding SHAP values:')display(shap_test.head(3).round(3))# Define function for partial correlationdef partial_correlation(X, y):  out = pd.Series(index = X.columns, dtype = float)  for feature_name in X.columns:    out[feature_name] = partial_corr(      data = pd.concat([X, y], axis = 1).astype(float),       x = feature_name,       y = y.name,      x_covar = [f for f in X.columns if f != feature_name]     ).loc['pearson', 'r']  return out# Compute ParShapparshap_train = partial_correlation(shap_train, y_train)parshap_test = partial_correlation(shap_test, y_test)parshap_diff = pd.Series(parshap_test - parshap_train, name = 'parshap_diff')print('\n################# Print parshap_diff')print(parshap_diff.sort_values())                         # Plot parshapplotmin, plotmax = min(parshap_train.min(), parshap_test.min()), max(parshap_train.max(), parshap_test.max())plotbuffer = .05 * (plotmax - plotmin)fig, ax = plt.subplots()if plotmin < 0:    ax.vlines(0, plotmin - plotbuffer, plotmax + plotbuffer, color = 'darkgrey', zorder = 0)    ax.hlines(0, plotmin - plotbuffer, plotmax + plotbuffer, color = 'darkgrey', zorder = 0)ax.plot(    [plotmin - plotbuffer, plotmax + plotbuffer], [plotmin - plotbuffer, plotmax + plotbuffer],     color = 'darkgrey', zorder = 0)sc = ax.scatter(    parshap_train, parshap_test,     edgecolor = 'grey', c = fimpo, s = 50, cmap = plt.cm.get_cmap('Reds'), vmin = 0, vmax = fimpo.max())ax.set(title = 'Partial correlation bw SHAP and target...', xlabel = '... on Train data', ylabel = '... on Test data')cbar = fig.colorbar(sc)cbar.set_ticks([])for txt in parshap_train.index:    ax.annotate(txt, (parshap_train[txt], parshap_test[txt] + plotbuffer / 2), ha = 'center', va = 'bottom')fig.savefig('parshap.png', dpi = 300, bbox_inches="tight")fig.show()# Feature selectionn_drop_max = 5iterations = 4features = {'parshap': parshap_diff, 'fimpo': fimpo}features_dropped = {}roc_auc_scores = {  'fimpo': {'train': pd.DataFrame(), 'test': pd.DataFrame()},  'parshap': {'train': pd.DataFrame(), 'test': pd.DataFrame()}}for type_ in ['parshap', 'fimpo']:  for n_drop in range(n_drop_max + 1):    features_drop = features[type_].sort_values().head(n_drop).index.to_list()    features_dropped[type_] = features_drop    X_drop = X.drop(features_drop, axis = 1)    for i in range(iterations):      X_train, X_test, y_train, y_test = train_test_split(X_drop, y, test_size = .2, stratify = y)      cat = CatBoostClassifier(silent = True).fit(X_train, y_train)      roc_auc_scores[type_]['train'].loc[n_drop, i] = roc_auc_score(y_train, cat.predict_proba(X_train)[:, 1])      roc_auc_scores[type_]['test'].loc[n_drop, i] = roc_auc_score(y_test, cat.predict_proba(X_test)[:, 1])        # Plot feature selectionfig, axs = plt.subplots(1, 2, sharey = True, figsize = (8, 3))plt.subplots_adjust(wspace = .1)axs[0].plot(roc_auc_scores['fimpo']['train'].index, roc_auc_scores['fimpo']['train'].mean(axis = 1), lw = 3, label = 'Train')axs[0].plot(roc_auc_scores['fimpo']['test'].index, roc_auc_scores['fimpo']['test'].mean(axis = 1), lw = 3, label = 'Test')axs[0].set_xticks(roc_auc_scores['fimpo']['train'].index)axs[0].set_xticklabels([''] + features_dropped['fimpo'], rotation = 90)axs[0].set_title('Feature Importance')axs[0].set_xlabel('Feature dropped')axs[0].grid()axs[0].legend(loc = 'center left')axs[0].set(ylabel = 'ROC-AUC score')axs[1].plot(roc_auc_scores['parshap']['train'].index, roc_auc_scores['parshap']['train'].mean(axis = 1), lw = 3, label = 'Train')axs[1].plot(roc_auc_scores['parshap']['test'].index, roc_auc_scores['parshap']['test'].mean(axis = 1), lw = 3, label = 'Test')axs[1].set_xticks(roc_auc_scores['parshap']['train'].index)axs[1].set_xticklabels([''] + features_dropped['parshap'], rotation = 90)axs[1].set_title('ParShap')axs[1].set_xlabel('Feature dropped')axs[1].grid()axs[1].legend(loc = 'center left')fig.savefig('feature_selection.png', dpi = 300, bbox_inches="tight")fig.show()
Thank you for reading! I hope you enjoyed this article. If you’d like, add me on Linkedin!