Custom Implementation of Feature Importance for your Voting Classifier Model | by Satyam Kumar | Towards Data Science

Machine learning models are becoming increasingly employed in complex high settings such as financial technology, medical science, etc. Despite the increase in utilization, there’s a lack of techniques to explain the model. The higher the interpretability of the model, the easier it becomes for someone to comprehend the results. There are various advanced techniques and algorithms to interpret the models including LIME, SHAP, etc.
Feature Importance is the simplest and most efficient technique to interpret the importance of the feature for the estimator. Feature Importance can help to get a better interpretation of the estimator and lead to model improvements by employing feature selection. There are various techniques to compute the feature importance score of the estimator:
Scikit-Learn built-in implementation of Feature Importance
Feature Importance computed with the Permutation method
Feature Importance computed with the SHAP value
In this article, we will further discuss the first feature importance strategy. Scikit-Learn package comes up with a function model.feature_importances_ to compute the feature importance for most of the estimators except the Voting Classifier estimator. Scikit-learn employs different algorithms to compute the feature importance score. For Random Forest or Decision Tree models, it computes the importance score using Gini Importance, for Logistic Regression it uses the vector weight.
In this article, we will discuss the implementation to compute the Importance score for a Voting Classifier model.
Voting Classifier is an ensemble technique that combines the predictions of various models together predicts an output (class) based on their highest probability. The voting classifier algorithm simply aggregates the findings of each classifier passed into the model and predicts the output class based on the highest majority of voting.
Voting Classifier supports two types of voting techniques:
Hard Voting: The predicted output of the voting classifier estimator will be calculated based on the highest majority of votes i.e the class which had the highest probability of being predicted by each of the classifiers.
Soft Voting: The output class is the prediction based on the average probability given to that class.
To get a better understanding of the Voting Classifier, read an article by Mubarak Ganiyu:
towardsdatascience.com
Unlike other model estimators, Scikit-learn does not come up with the feature importance implementation of Voting Classifier. But each of the base estimators used for the Voting Classifier has its implementation. The idea is to combine the importance score of each of the estimators based on the weights.
The steps of the algorithm to compute the feature importance of the Voting Classifier is:
Compute Feature Importance score of each of the base estimatorsMultiply the weights of the base estimator to the importance score of each of the features.Average out the features importance score (from step 2) for each feature.
Compute Feature Importance score of each of the base estimators
Multiply the weights of the base estimator to the importance score of each of the features.
Average out the features importance score (from step 2) for each feature.
import pandas as pd
from sklearn.tree import DecisionTreeClassifier
from sklearn.ensemble import RandomForestClassifier
from sklearn.ensemble import VotingClassifier
from sklearn.model_selection import train_test_split
from sklearn.metrics import *
import numpy as np

df = pd.read_csv("DATA/creditcard.csv")
df.shape

(284807, 31)
y = df['Class']
X = df.drop(['Class'], axis=1)
X_train, X_test, Y_train, Y_test = train_test_split(X, y)

print(X_train.shape)
print(X_test.shape)

(213605, 30)
(71202, 30)

clf1 = DecisionTreeClassifier(random_state=42)
clf2 = DecisionTreeClassifier(random_state=42, max_depth=20)
clf3 = RandomForestClassifier(random_state=42, n_jobs=-1)

%%time

estimators = [('DT1', clf1), ('DT2', clf2), ('RF', clf3)]
voting_clf = VotingClassifier(estimators=estimators, voting='soft', weights=[1,1,2])
voting_clf.fit(X_train, Y_train)

Wall time: 1min 22s

VotingClassifier(estimators=[('DT1', DecisionTreeClassifier(random_state=42)),
                             ('DT2',
                              DecisionTreeClassifier(max_depth=20,
                                                     random_state=42)),
                             ('RF',
                              RandomForestClassifier(n_jobs=-1,
                                                     random_state=42))],
                 voting='soft', weights=[1, 1, 2])
def compute_feature_importance(voting_clf, weights):
    """ Function to compute feature importance of Voting Classifier """
    
    feature_importance = dict()
    for est in voting_clf.estimators_:
        feature_importance[str(est)] = est.feature_importances_
    
    fe_scores = [0]*len(list(feature_importance.values())[0])
    for idx, imp_score in enumerate(feature_importance.values()):
        imp_score_with_weight = imp_score*weights[idx]
        fe_scores = list(np.add(fe_scores, list(imp_score_with_weight)))
    return fe_scores

df = pd.DataFrame()
df['Feature'] = X_train.columns
df['Feature Importance'] = compute_feature_importance(voting_clf, [1, 1, 2])
df.sort_values('Feature Importance', ascending=False)

 

In this article, we have discussed a technique or hack to compute the feature importance score for the Voting Classifier ensemble model. Feature importance score can be helpful to understand the top important features for the model and may lead to model improvements by employing feature selection.
[1] Voting Classifier Documentation from Scikit-Learn: https://scikit-learn.org/stable/modules/generated/sklearn.ensemble.VotingClassifier.html
Thank You for Reading