Greater Than Our Parts
By: S. T. Lanier
Tags: Ensemble Method, Classification, Sklearn, Data Science, Vote

Photo by Element5 Digital on Unsplash

Greater Than Our Parts

I promise, this photo is the only (very on-the-nose) American politics clickbait I’ll use in this article. What I really want to talk about is how to setup a voting ensemble classifier from scikit-learn for the sake of eking out some additional performance from your already trained classifiers. Since you’re here, why don’t you pull up a seat, stay awhile? I won’t ask who you voted for.

On this glorious December day in the early twenty-first century, there is a smorgasbord of models available to the technician intent on classifying his data: support vector machines (SVMs), decision trees, random forests, k-nearest neighbors and logistic regression classifiers, just to name a few. Each of them brings its own strengths to the table, and often, the best you can do for yourself is just try a few out to see which is best handling the data. So you train a few models, find two or three or five that do the best job of classifying your data, and then what? Try to pick one favorite from amongst your many bright children? Mummy loves you all.

Something I wish I’d known earlier in my study of classification models is how easy it is to combine multiple of them into one streamlined machine, the voting ensemble classifier. If you’ve already dug into random forests and gradient/adaptive boosting (AdaBoost), this isn’t a novel idea: a random forest is so many decision trees trained and given votes, and with AdaBoost, for example, you can even change the classifiers populating your “forest”––they don’t have to be decision trees. By changing the base_estimator parameter, you could just as easily have an AdaBoost classifier made up of SVMs.

Screenshot of scikit-learn’s AdaBoost documentation.

A voting ensemble classifier allows you to do the same thing, but with even more control of the types of estimators in your ensemble, allowing you to mix-and-match. (Also, unlike with AdaBoost, you’ll probably be employing strong learners in your ensemble voter. Just for the record.) Let’s see some code. Say, for example, you’ve been trying out a few different classifiers and you’ve found a few that seem to be working well on the dataset. Here’s the accuracy on test data for each model, all of which were fit with GridSearchCV to find the best parameters for each.

Sigmoid SVC 0.880000

AdaBoost 0.866667

Decision Tree 0.866667

lr 0.853333

Random Forest 0.853333

RBF SVC 0.853333

Linear SVC 0.853333

XGBoost 0.840000

KNN 0.826667

Polynomial SVC 0.813333

dtype: float64

Seems like, on their own, the SVC with sigmoid kernel, AdaBoost, and the lone decision tree are performing best. Let’s try combining those three models into a voting classifier, using the best estimator from each respective GridSearch.

from sklearn.ensemble import VotingClassifier clf_votehard = VotingClassifier(

estimators=[('svcs', gs_svcs.best_estimator_),

('dt', gs_dt.best_estimator_),

('ab', gs_ab.best_estimator_)],

voting='hard') clf_votehard.fit(X_train, y_train) accuracy_score(y_test, clf_votehard.predict(X_test)) >> 0.866667

In this first example, voting has been set to 'hard' , meaning that each classifier gets a single vote — the SVM predicts a class, the decision tree predicts a class, and the AdaBoost classifier predicts a class, and those three ballots are counted to see what the winning class is. Weights can also be assigned to the classifiers, potentially giving some classifiers more votes than others.

The alternative would be to set voting to 'soft' , where the predicted probabilities from each classifier are added up , and then the most probable class is picked from there. For example, in a binary situation, maybe the SVM outputs 0.3 and 0.7 predicted probabilities for classes 0 and 1 respectively; 0.4 and 0.6 for the decision tree; and 0.9 and 0.1 for the AdaBoost classifier. The voting classifier would add up the probabilities for each class — 0.3+0.4+0.9=1.6 for class 0, 0.7+0.6+0.1=1.4 for class 1 — and choose the class with the highest sum of probabilities: class 0. This is the method the scikit-learn documentation recommends for well-calibrated classifiers such as these. You can add weights to the classifiers on top of that.

clf_votesoft = VotingClassifier(

estimators=[('svcs', gs_svcs.best_estimator_),

('dt', gs_dt.best_estimator_),

('ab', gs_ab.best_estimator_)],

voting='soft') clf_votesoft.fit(X_train, y_train) accuracy_score(y_test, clf_votesoft.predict(X_test)) >> 0.866667

For this particular dataset, for which you can find the code here, I got the best results by including two more classifiers (k-nearest neighbors and logistic regression), using soft voting, and some classifier weights.

clf_votesoft_final = VotingClassifier(

estimators=[('lr', baseline_classifiers['LogReg']),

('knn', gs_knn.best_estimator_),

('svcs', gs_svcs.best_estimator_),

('dt', gs_dt.best_estimator_),

('ab', gs_ab.best_estimator_)],

voting='soft',

weights=[1, 1, 3, 2, 2]) clf_votesoft_final.fit(X_train, y_train) accuracy_score(y_test, clf_votesoft_final.predict(X_test)) >> 0.8933333

Conclusion

After having found some classifiers that work well for the data and finding the best hyperparameters for each using something like GridSearchCV, you can combine their strengths into a voting ensemble classifier, giving each classifier some share of votes for each prediction the ensemble makes. With diverse enough classifiers curated, the strengths of one can help fill out the weaknesses of another, ultimately resulting in better performance from the ensemble than any single constituent classifier could achieve: in the example above, the highest accuracy any single classifier achieved was 0.880, but by combining five classifiers in a soft-vote, weighted voting ensemble classifier, accuracy was increased to 0.893.