Ensemble Learning — Bagging & Random Forest (Part 2)
By: Irene Pham
Tags: Python, Machine Learning, Artificial Intelligence, Scikit Learn, Algorithms

By Romain Baise

This is a series of posts that explain Ensemble method in an easy-to-follow manner. In this post, we will discuss about Homogenous Ensemble- Bagging.

1/Heterogenous Ensemble

2/ Homogenous Ensemble — Bagging

3/ Homogenous Ensemble — Boosting

Homogeneous Ensemble

While Heterogenous Ensemble makes use of the brilliance of diverse algorithms, Homogenous Ensemble uses only one type of algorithm. The two main techniques for this method are Bagging and Boosting.

Bagging (Boostrap Aggregating)

Bagging, for bootstrap aggregation, is a machine learning method that makes prediction by combining several classifiers trained on bootstrapped subsets of your dataset.

Bootstrap

Firstly, we use bootstrap sampling to create a number of data subsets from our training data. The emphasis here is that those subsets of data are products of random sampling with replacement. Why does it matter with our model? The answer is that it ensures the effect of randomness and independence contributed to our model. Since randomness allows algorithms to attempt to avoid overfitting and generalise better on new observations, bootstrap effectively helps to fight against high variance which is a drawback for algorithms such as Decision Tree. An example of algorithm using Bagging technique is the well-known Random Forest that uses Decision Tree as the base classifier.

Aggregation

After creating a number of classifiers trained on bootstrapped data subsets, the output is then aggregated using voting or averaging method to achieve final prediction. We can also say Bagging is a sister of Voting ensemble method due to this characteristics while the powerful bootstrap technique of Bagging substitutes the strength of fine-tuned classifiers in Voting.

Classifiers’ characteristics

Another characteristics of Bagging worth to mention is that the number of learners should be large, usually between 100 to 500 classifiers, at the same time, each of them needs to achieve accuracy more than of 50%. Bagging employs the Condorcet’s Jury Theorem on the Application of Analysis to the Probability of Majority Decisions which states that:

“If p is greater than ½ (each voter is more likely to vote correctly), then adding more voters increases the probability that the majority decision is correct. In the limit, the probability that the major votes correctly approaches 1 as the number of voters increases” (Wikipedia, 2020).

Based on this theory, as long as our Bagging model meets those conditions, the final prediction accuracy may be as high as 100%.

Bagging is certainly a star classifier when we need to fight against variance, create a more stable and robust model that can be run parallel. The only limitation is that it is computationally expensive since it requires a high number of estimators. For that reason, we need to bear in mind that our collection of estimators should be light, fast, and can provide predictions just better than random (more than 50%).

Random Forest

Let’s see how Random Forest adapts Bagging technique to upgrade its tree-based model.

Step 1: Train a number of trees on different bootstrapped subsets of your dataset.

Step 2: Let those learners make predictions.

Step 3: Use Mode (for Classification) or Mean (for Regression) to aggregate predictions and return the final one.

Although Decision Tree is quite intuitive, easy to use and to interpret the result, one of its biggest limitation is instability. If we achieve a high accuracy for our model, there is no guarantee that we can have such metric in deployment. It is probably the universal problem for any other modelling technique but it is more true for Decision Tree. Since Decision Tree choose its split point based on our particular dataset (a subset of the real-world data population), there is no certainty that split point will work on the real-world data.

Random Forest solves this problem by having numerous trees run on different random samples of the training data. Using bootstrap sampling is the way Random Forest says: let’s simulate a bigger data subset based on our dataset which will be more representative of real-world data. By training many trees on a larger data subset, we can have more different split points, therefore, more perspectives to decide on and come up with a model that have a better generalising power.

Apart from that, Random Forest also searches for the best features among random subsets of features sampled at each nodes. This interesting property can be employed in Feature Selection for high-dimensional datasets to quickly come up with features more important for our model.

The powerful technique can be called easily with Scikit-learn, see the below code for your preference.

Overall, Bagging Ensemble is a powerful machine learning modelling technique. By implementing bootstrap, variance is reduced while bias level does not necessarily increase due to effect of aggregating the based models. Apart from that, our model also has higher stability and robustness which are indispensable characteristics of successful model in deployment. Even though Bagging can be computationally expensive if the number of estimators is exceptionally high, it seems to be a fair sacrifice for the sake of creating a better machine learning model.

Thanks for reading!

Don’t be shy, let’s connect if you are interested in more of my posts:

Medium: https://medium.com/@irenepham_45233

Instagram: https://www.instagram.com/funwithmachinelearning/