Advanced Ensemble Learning Techniques | by Charu Makhijani | Towards Data Science

In my previous post about ensemble learning, I have explained what is ensemble learning, how it relates with Bias and Variance in machine learning and what are the simple techniques of ensemble learning. If you haven’t read the post, please refer here.
In this post I will cover ensemble learning types, advanced ensemble learning methods — Bagging, Boosting, Stacking and Blending with code samples. At the end I will explain some pros and cons of using ensemble learning.
Ensemble learning methods can be categorized into two groups:
1. Sequential Ensemble Methods
In this method base learners are dependent on the results from previous base learners. Every subsequent base model corrects the prediction made by its predecessor fixing the errors in it. Hence the overall performance can be increased via improving the weight of previous labels.
2. Parallel Ensemble Methods
In this method there is no dependency between the base learners and all base learners execute in parallel and the results of all base models are combined in the end (using averaging for regression and voting for classification problems).
Parallel Ensemble methods are divided in two categories-
1. Homogeneous Parallel Ensemble Methods- In this method a single machine learning algorithm is used as a base learner.
2. Heterogeneous Parallel Ensemble Methods- In this method multiple machine learning algorithms are used as base learners.
Bagging or Bootstrap Aggregation is a parallel ensemble learning technique to reduce the variance in the final prediction.
The Bagging process is very similar to averaging, only difference is that bagging uses random sub-samples of the original dataset to train same/multiple models and then combines the prediction, whereas in averaging the same dataset is used to train models. Hence the technique is called Bootstrap Aggregation as it combines both Bootstrapping (or Sampling of data) and Aggregation to form an ensemble model.
Bagging is a 3 step process-
2. Model is built (either a classifier or a decision tree) with every sample.
3. Predictions of all the base models are combined (using averaging or weighted averaging for regression problems or majority voting for classification problems) to get the final result.
All three steps can be parallelized across different sub-samples, hence the training can be done faster when working on larger datasets.
In bagging, every base model is trained on a different subset of data and all the results are combined, so the final model is less overfitted and variance is reduced.
Bagging is more useful when the model is unstable, with stable models bagging is not useful in improving the performance. Model is called stable when it’s less sensitive for small fluctuations in the training data.
Some examples of Bagging are — Random Forest, Bagged Decision Trees, Extra Trees. sklearn library also provides BaggingClassifier and BaggingRegressor classes to create your own bagging algorithms.
Let's see this in the example below-
LogisticRegression :::: Mean: 0.7995780505454071 , Std Dev: 0.006888373667690784Bagging LogisticRegression :::: Mean: 0.8023420359806932 Std Dev: 0.00669463780099821 DecisionTreeClassifier :::: Mean: 0.8119073077604059 , Std Dev: 0.005729415273647502Bagging DecisionTreeClassifier :::: Mean: 0.849639923635328 Std Dev: 0.0046034229502244905 RandomForestClassifier :::: Mean: 0.8489381115139759 , Std Dev: 0.005116577814042257Bagging RandomForestClassifier :::: Mean: 0.8567037712754901 Std Dev: 0.004468761007278419 ExtraTreesClassifier :::: Mean: 0.8414792383547726 , Std Dev: 0.0064275238258043816Bagging ExtraTreesClassifier :::: Mean: 0.8511317483045042 Std Dev: 0.004708539080690846 KNeighborsClassifier :::: Mean: 0.8238853221249702 , Std Dev: 0.006423083088668752Bagging KNeighborsClassifier :::: Mean: 0.8396364017767104 Std Dev: 0.00599320955270458 VotingClassifier :::: Mean: 0.8462174468641986 Std Dev: 0.006423083088668752
As we see in the example, Bagging Classifiers are improving the variance of ML models and reducing the deviation. Same is the case when using VotingClassifier which improves the average variance of ML models.
Boosting is a sequential ensemble learning technique to convert weak base learners to strong learner that performs better and is less bias. The intuition here is that individual models may not perform very well on the entire dataset, but they work well on some part of the entire dataset. Hence each model in the ensemble actually boosts the overall performance.
Boosting is an iterative method that adjusts the weight of an observation based on the previous classification. If an observation was classified incorrectly, then the weight of that observation is increased in the next iteration. In the same way, if an observation was classified correctly then the weight of that observation is reduced in the next iteration.
Boosting is used to decrease the bias error, but it can also overfit the training data. That is why the parameter tuning is an important part of boosting algorithms to make them avoid overfitting the data.
Boosting was originally designed for classification problems but extended for regression problems as well.
Some examples of Boosting algorithms are — AdaBoost, Gradient Boosting Machine (GBM), XGBoost, LightGBM, CatBoost.
Lets see Boosting with an example-
AdaBoostClassifier :::: Mean: 0.8604337082284473 Std Dev: 0.0032409094349287403GradientBoostingClassifier :::: Mean: 0.8644262257222698 Std Dev: 0.0032315430892614675XGBClassifier :::: Mean: 0.8641189579917322 Std Dev: 0.004561102596800773VotingClassifier :::: Mean: 0.864645581703271 Std Dev: 0.0032985215353102735
Stacking, also known as stacked generalization, is an ensemble learning technique that combines multiple machine learning algorithms via meta learning (either a meta-classifier or a meta-regressor).
The base level algorithms are trained on entire training dataset, and then the meta model is trained on the predictions from all the base level models as features. The base models are called level-0 models, and meta model which combines the base models predictions is called level-1 model.
The level-1 model training data is prepared via k-fold cross-validation of the base models, and out-of-fold predictions (real numbers for regression and class labels for classification) are used as the training dataset.
The level-0 models can be either diverse range of algorithms or same algorithm (most often they are diverse). The level-1 meta model is most often a simple model, like Linear Regression for regression problems and Logistic Regression for Classification problems.
Stacking method can reduce the bias or variance based on the algorithms used in level-0.
There are many libraries for Stacking, like — StackingClassifier, StackingRegressor, make_classification, make_regression, ML Ensemble, H20.
Lets see Stacking with an example using StackingClassifier-
LogisticRegression :::: Mean: 0.799198344149096 Std Dev: 0.004958323931953346DecisionTreeClassifier :::: Mean: 0.8130779055654346 Std Dev: 0.008467878845801694KNeighborsClassifier :::: Mean: 0.8251287819886122 Std Dev: 0.00634438876282278SVC :::: Mean: 0.8004562250294449 Std Dev: 0.005221775246052317GaussianNB :::: Mean: 0.7964780515718138 Std Dev: 0.004996489474526567StackingClassifier :::: Mean: 0.8376917712960184 Std Dev: 0.005593816155570199
As we see in this example, we have used different ML models in the level0 and stacked them with StackingClassifier using LogisticRegression in level1, and it has improved the variance.
Multi-levels Stacking is an extension of stacking where stacking is applied on multiple layers.
For example in a 3-level stacking, Level-0 is same where diverse range of base learners are trained using k-fold cross validation. In level-1, instead of single meta model, N such meta models are used. In level-2, the final meta model is used that takes the predictions from N meta models of level-1 .
Adding multiple levels is both data expensive (as lots of data needed to be trained) and time expensive (as each layer adds multiple models).
Blending is most often used interchangeable with Stacking. And it is almost similar to Stacking with only one difference, Stacking uses out-of-fold predictions for the training set whereas Blending uses a hold-out (validation) set (10–20% of the training set) to train the next layer.
Although Blending is simpler than stacking and uses less data, the final model may overfit on the holdout set.
1. Ensemble methods have higher predictive accuracy, compared to the individual models.
2. Ensemble methods are very useful when there is both linear and non-linear type of data in the dataset; different models can be combined to handle this type of data.
3. With ensemble methods bias/variance can be reduced and most of the times, model is not underfitted/overfitted.
4. Ensemble of models is always less noisy and is more stable.
1. Ensembling is less interpretable, the output of the ensembled model is hard to predict and explain. Hence the idea with ensemble is hard to sell and get useful business insights.
2. The art of ensembling is hard to learn and any wrong selection can lead to lower predictive accuracy than an individual model.
3. Ensembling is expensive in terms of both time and space. Hence ROI can increase with ensembling.
After looking at the fundamentals of ensemble learning techniques above and the pros/cons of ensemble learning, it won’t be wrong to say that if used correctly ensemble methods are great for improving the overall performance of ML models. When its hard to rely on one model, ensemble makes the life easier and that’s the reason why in most of the ML competitions ensemble methods are the choices for winners.
Although selecting the right ensemble methods and using them is not an easy job but this art can be learned with experience. The techniques described in this post are most often reliable source for ensembling, but other variants are also possible based on specific problems/needs.
To access the complete code for the Advanced Ensemble Techniques, please check this Github link.
Thanks for the read. If you like the story please like, share and follow for more such content. As always, please reach out for any questions / comments / feedback.
Github: https://github.com/charumakhijaniLinkedIn: https://www.linkedin.com/in/charu-makhijani-23b18318/