Ensemble Learning: Stacking, Blending & Voting | by Fernando López | Towards Data Science

We have heard the phrase “unity is strength”, whose meaning can be transferred to different areas of life. Sometimes correct answers to a specific problem are supported by several sources and not just one. This is what Ensemble Learning tries to do, that is, to put together a group of ML models to improve solutions to specific problems.
Throughout this blog, we will learrn what Ensemble Learning is, what are the types of Ensembles that exist and we will specifically address Voting and Stacking Ensembles. Therefore, this blog will be divided into the following sections:
What is Ensemble Learning?
Stacking
Blending
Voting
I invite you to download the code and try it for yourself, it will be worth: https://github.com/FernandoLpz/Stacking-Blending-Voting-Ensembles
Ensemble Learning refers to the use of ML algorithms jointly to solve classification and/or regression problems mainly. These algorithms can be the same type (homogeneous Ensemble Learning) or different types (heterogeneous Ensemble Learning). Ensemble Learning performs a strategic combination of various experts or ML models in order to improve the effectiveness obtained using a single weak model [1, 2]. Figure 1 provides a visual overview regarding the comparison of a model that does not implement Ensemble Learning and a model that does implement Ensemble Learning.
There are different types of Ensemble Learning techniques which differ mainly by the type of models used (homogeneous or heterogeneous models), the data sampling (with or without replacement, k-fold, etc.) and the decision function (voting, average, meta model, etc). Therefore, Ensemble Learning techniques can be classified as:
Bagging
Boosting
Stacking
In addition to these three main categories, two important variations emerge: Voting (which is a complement of Bagging) and Blending (a subtype of Stacking). Although Voting and Blending are a complement and a subtype of Bagging and Stacking respectively, these techniques are often found as direct types of Ensemble Learning.
In this blog we will specifically address the Stacking, Blending and Voting techniques, let’s go for it!
Better known as Stacking Generalization, it is a method introduced by David H. Wolpert in 1992 [3] where the key is to reduce the generalization error of different generalizers (i.e. ML models). The general idea of the Stacking Generalization method is the generation of a Meta-Model. Such a Meta-Model is made up of the predictions of a set of ML base models (i.e. weak learners) through the k-fold cross validation technique. Finally, the Meta-Model is trained with an additional ML model (which is commonly known as the “final estimator” or “final learner”).
The Stacking Generalization method is commonly composed of 2 training stages, better known as “level 0” and “level 1”. It is important to mention that it can be added as many levels as necessary. However, in practice it is common to use only 2 levels. The aim of the first stage (level 0) is to generate the training data for the meta-model, this is carried out by implementing k-fold cross validation for each “weak learner” defined in the first stage. The predictions of each one of these“weak learners” are “stacked” in order to build such such “new training set” (the meta-model). The aim of the second stage (level 1) is to train the meta-model, such training is carried out through an already determined “final learner”.
In figure 2 we see a graphical description of an architecture of a Stacking Generalization Classifier that is composed of 3 base models (weak learners) and a final estimator.
Perfect, so far we already know how the Stacking Generalization technique works. Now let’s see a small example of how we would do this in code (it is important to mention that this technique can be implemented directly from scikit-learn, however, in order to make the explanation more demonstrative, let’s see how we do it from scratch).
Let’s analyze the key parts, in line 4 we are defining 5 classifiers (weak learners) that will be the base models of our stack (which are trained at level 0). In line 11 we define the final classifier (which is the meta-model classifier). Now, level 0 training begins with the for loop defined in line 17. As we can see, in line 19 we are receiving the predictions of k-fold cross validation and in line 26 we are “stacking” these predictions (the which are forming the training data of the meta-model). On line 22 we are receiving the predictions from the test set which are “stacked” on line 33 to form the meta-model test data. Finally, in line 45 we carry out the level 1 training, that is, the meta-model training.
I invite you to download the code and try it for yourself, it will be worth: https://github.com/FernandoLpz/Stacking-Blending-Voting-Ensembles
Well, so far we already know how the Stacking Generalization technique works. As we mentioned, one of the key parts of this method is the use of the k-fold cross validation for the generation of the meta-model training data. However, there is a variation, we can omit k-fold cross validation and only use “one-holdout set”, this small but significant variation is called “Blending”.
Let’s see how it works!
Blending is a technique derived from Stacking Generalization. The only difference is that in Blending, the k-fold cross validation technique is not used to generate the training data of the meta-model. Blending implements “one-holdout set”, that is, a small portion of the training data (validation) to make predictions which will be “stacked” to form the training data of the meta-model. Also, predictions are made from the test data to form the meta-model test data.
In figure 3 we can see a Blending architecture using 3 base models (weak learners) and a final classifier. The blue boxes represent that portion of the training data that is used to generate predictions (yellow boxes) to form the meta-model. The green boxes represent the test data which is used to generate predictions to form the meta-model test data (purple boxes).
Great, now that you’re familiar with the Blending architecture, let’s see how we do this in code:
Let’s analyze the key parts of this model. In line 4 we are defining the 5 base classifiers that we will use (weak learners), in line 11 we define the final classifier, as in the previous example, we will use Logistic Regression. Level 0 training begins on line 17. As we can see, on line 20 we are receiving the predictions of the validation set (which will form the training data of the meta-model) and the predictions of the test data (the which will form the meta-model test data). Also, on lines 24 and 31 we are “stacking” the predictions of each base classifier. Finally, on line 43 we are moving to level 1 training, and that is it!
I invite you to download the code and try it for yourself, it will be worth: https://github.com/FernandoLpz/Stacking-Blending-Voting-Ensembles
As we can see, the Blending architecture is slightly simpler and more compact than Stack Generalization. Omitting k-fold cross validation can make us optimize the processing time.
Great, by now you already know the Stacked Generalization architecture and how it works as well as the variation that arises from it (Blending). The million dollar question remains: which technique is better? When should I apply Stacking or Blending? Well, that will depend 100% on the task you are trying to solve, the amount of data you have as well as the computing power and memory available.
Finally, let’s talk about an Ensemble Learning technique that is simple, intuitive and that can sometimes be a good option, let’s talk about Voting!
This type of ensemble is one of the most intuitive and easy to understand. The Voting Classifier is a homogeneous and heterogeneous type of Ensemble Learning, that is, the base classifiers can be of the same or different type. As mentioned earlier, this type of ensemble also works as an extension of bagging (e.g. Random Forest).
The architecture of a Voting Classifier is made up of a number “n” of ML models, whose predictions are valued in two different ways: hard and soft. In hard mode, the winning prediction is the one with “the most votes”. In Figure 2 we see an example of how the Voting Classifier works in hard mode.
On the other hand, the Voting Classifier in soft mode considers the probabilities thrown by each ML model, these probabilities will be weighted and averaged, consequently the winning class will be the one with the highest weighted and averaged probability. In Figure 3 we see an example of how the Voting Classifier works in the soft mode.
Ok, now that we know how the Voiting Classifier works, let’s see how to do this in code. On this occasion, since it is a simple and intuitive ensemble technique (compared to Stacking or Blending), let’s make use of the function provided by scikit-learn for the implementation of Voting, let’s do it!
In the code above we are creating a class which will contain different classifiers which are: Decision Tree, K-Nearest Neighbors, Logistic Regression and Voting Classifier (lines 24, 32, 40 and 48 respectively). To compare the effectiveness between “weak classifiers” and the Ensemble, we will make use of the “breast_cancer” toy dataset (line 8). We will use each classifier with its default values.
Let’s go directly to what interests us, Voting Classifier. As we can see, in lines from 51 to 53 we are defining the weak classifiers that we are going to use and, in a simple way, we are going to pass these classifiers as parameters to the Voting Classifier (line 56), in this example, we are using the “hard” mode. Finally, on line 59 we train the Ensemble, this is what we get:
Decision Tree, Train accuracy: 1Decision Tree, Test accuracy: 0.958K-Nearest Neighbors, Train accuracy: 0.930K-Nearest Neighbors, Test accuracy: 0.946Logistic Regression, Train accuracy: 0.953Logistic Regression, Test accuracy: 0.951Voting Classifier, Train accuracy: 0.981Voting Classifier, Test accuracy: 0.965
As we can see, the Test accuracy of the Voting Classifier is slightly better than that of the weak classifiers. It is very important to mention that, although Voting Classifier is a great alternative to improve the accuracy of your models, it may not always be the best option due to various factors, including processing time.
I invite you to download the code and try it for yourself, it will be worth: https://github.com/FernandoLpz/Stacking-Blending-Voting-Ensembles
Congratulations, we have reached the end of this great journey to learn a little more about Ensemble Learning.
In this blog we have seen what Ensemble Learning is and its most common techniques. On the other hand, we have delved a little into Stacking, Blending and Voting techniques. Blending being a technique derived from Stacking.
In the end, there is a great diversity of techniques to reduce the error of our ML models as well as to increase their effectiveness, it will be up to each developer to choose the one that best suits their problem taking into account several factors such as the amount of data available, the type of learning to be carried out, the amount of computational processing available, etc.