Extracting & Plotting Feature Names & Importance from Scikit-Learn Pipelines | by Kyle Gilde | Towards Data Science

If you have ever been tasked with productionalizing a machine learning model, you probably know that Scikit-Learn library offers one of the best ways — if not the best way — of creating production-quality machine learning workflows. The ecosystem’s Pipeline, ColumnTransformer, preprocessors, imputers & feature selection classes are powerful tools that transform raw data into model-ready features.
However, before anyone is going to let you deploy to production, you are going to want to have some rudimentary understanding of how the new model works. The most common way to explain how a black-box model works is by plotting feature names and importance values. If you have ever tried to extract the feature names from a heterogeneous dataset processed by ColumnTransformer, you know that this is no easy task. Exhaustive Internet searches have only brought to my attention where others have asked the same question or offered a partial answer, instead of yielding a comprehensive and satisfying solution.
To remedy this situation, I have developed a class called FeatureImportance that will extract feature names and importance values from a Pipeline instance. It then uses the Plotly library to plot the feature importance using only a few lines of code. In this post, I will load a fitted Pipeline, demonstrate how to use my class and then give an overview of how it works. The complete code can be found here or at the end of this blog post.
Note: My class may not work starting with the changes in scikit-learn 1.0.
There are two things I should note before continuing:
I credit Joey Gao’s code on this thread with showing the way to tackle this problem.My post assumes that you have worked with Scikit-Learn and Pandas before and are familiar with how ColumnTransformer, Pipeline & preprocessing classes facilitate reproducible feature engineering processes. If you need a refresher, check out this Scikit-Learn example.
I credit Joey Gao’s code on this thread with showing the way to tackle this problem.
My post assumes that you have worked with Scikit-Learn and Pandas before and are familiar with how ColumnTransformer, Pipeline & preprocessing classes facilitate reproducible feature engineering processes. If you need a refresher, check out this Scikit-Learn example.
For the purposes of demonstration, I’ve written a script called fit_pipeline_ames.py. It loads the Ames housing training data from Kaggle and fits a moderately complex Pipeline. I’ve plotted its visual representation below.
from sklearn import set_config from sklearn.utils import estimator_html_repr from IPython.core.display import display, HTML from fit_pipeline_ames import pipe   # create & fit pipelineset_config(display='diagram')display(HTML(estimator_html_repr(pipe)))
This pipe instance contains the following 4 steps:
The ColumnTransformer instance is composed of 3 Pipelines, containing a total of 4 transformer instances, including SimpleImputer, OneHotEncoder & GLMMEncoder from the category_encoders package. See my previous blog post for a full explanation of how I dynamically constructed this particular ColumnTransformer.The VarianceThreshold uses the default threshold of 0, which removes any features that contain only a single value. Some models will fail if a feature has no variance.The SelectPercentile uses the f_regression scoring function with a percentile threshold of 90. These settings retain the top 90% of features and discard the bottom 10%.The CatBoostRegressor model is fit to the SalesPrice dependent variable using the features created and selected in the preceding steps.
The ColumnTransformer instance is composed of 3 Pipelines, containing a total of 4 transformer instances, including SimpleImputer, OneHotEncoder & GLMMEncoder from the category_encoders package. See my previous blog post for a full explanation of how I dynamically constructed this particular ColumnTransformer.
The VarianceThreshold uses the default threshold of 0, which removes any features that contain only a single value. Some models will fail if a feature has no variance.
The SelectPercentile uses the f_regression scoring function with a percentile threshold of 90. These settings retain the top 90% of features and discard the bottom 10%.
The CatBoostRegressor model is fit to the SalesPrice dependent variable using the features created and selected in the preceding steps.
With the help of FeatureImportance, we can extract the feature names and importance values and plot them with 3 lines of code.
from feature_importance import FeatureImportancefeature_importance = FeatureImportance(pipe)feature_importance.plot(top_n_features=25)
The plot method takes a number of arguments that control the plot's display. The most important ones are the following:
top_n_features: This controls how many features will be plotted. The default value is 100. The plot's title will indicate this value as well as how many features there are in total. To plot all features, just set top_n_features to a number larger than the total features.
rank_features: This argument controls whether the integer ranks are displayed in front of the feature names. The default is True. I find that this aids with interpretation, especially when comparing the feature importance from multiple models.
max_scale: This determines whether the importance values are scaled by the maximum value & multiplied by 100. The default is True. I find that this enables an intuitive way to compare how important other features are vis-a-viz the most important one. For instance, in the plot of above, we can say that GrLivArea is about 81% as important to the model as the top feature, OverallQty.
The FeatureImportance class should be instantiated using a fitted Pipeline instance. (You can also change the verbose argument to True if you want to have all of the diagnostics printed to your console.) My class validates that this Pipeline starts with a ColumnTransformer instance and ends with a regression or classification model that has the feature_importance_ attribute. The Pipeline can have any number of or no instances of classes from sklearn.feature_selection as intermediate steps.
The FeatureImportance class is composed of 4 methods.
get_feature_names was the hardest method to devise. It iterates through the ColumnTransformer transformers, uses the hasattr function to discern what type of class we are dealing with and pulls the feature names accordingly. (Special Note: If the ColumnTransformer contains Pipelines and if one of the transformers in the Pipeline is adding completely new columns, it must come last in the pipeline. For example, OneHotEncoder, MissingIndicator & SimpleImputer(add_indicator=True) add columns to the dataset that didn't exist before, so they should come last in the Pipeline.)get_selected_features calls get_feature_names. Then it tests for whether the main Pipeline contains any classes from sklearn.feature_selection based upon the existence of the get_support method. If it does, this method returns only the features names that were retained by the selector class or classes. It stores the features that were not selected in the discarded_featuresattribute. Here are the 24 features that were removed by the selectors in my pipeline:
get_feature_names was the hardest method to devise. It iterates through the ColumnTransformer transformers, uses the hasattr function to discern what type of class we are dealing with and pulls the feature names accordingly. (Special Note: If the ColumnTransformer contains Pipelines and if one of the transformers in the Pipeline is adding completely new columns, it must come last in the pipeline. For example, OneHotEncoder, MissingIndicator & SimpleImputer(add_indicator=True) add columns to the dataset that didn't exist before, so they should come last in the Pipeline.)
get_selected_features calls get_feature_names. Then it tests for whether the main Pipeline contains any classes from sklearn.feature_selection based upon the existence of the get_support method. If it does, this method returns only the features names that were retained by the selector class or classes. It stores the features that were not selected in the discarded_featuresattribute. Here are the 24 features that were removed by the selectors in my pipeline:
feature_importance.discarded_features['BsmtFinSF2','BsmtHalfBath','Alley_Pave','LandContour_Lvl','Utilities_AllPub','Utilities_NoSeWa','LotConfig_Corner','LotConfig_FR2','Condition1_RRNe','Condition2_RRAe','Condition2_RRAn','HouseStyle_SLvl','RoofStyle_Mansard','RoofMatl_ClyTile','RoofMatl_Metal','RoofMatl_Roll','RoofMatl_Tar&Grv','ExterCond_Ex','Foundation_Stone','Foundation_Wood','BsmtFinType2_GLQ','Electrical_missing_value','FireplaceQu_Fa','GarageCond_Gd','MiscFeature_Gar2','SaleType_ConLI']
3. get_feature_importance calls get_selected_features and then creates a Pandas Series where values are the feature importance values from the model and its index is the feature names created by the first 2 methods. This Series is then stored in the feature_importance attribute.
4. plot calls get_feature_importance and plots the output based upon the specifications.
The original notebook for this blog post can be found here. The complete code for FeatureImportance is shown below and can be found here.
If you create a Pipeline that you believe should be supported FeatureImportanceby but is not, please provide a reproducible example, and I will consider making the necessary changes.
Stay tuned for further posts on training & regularizing models with Scikit-Learn ColumnTransformers and Pipelines. Let me know if you found this post helpful or have any ideas for improvement. Thanks!