Types of Ensemble methods in Machine learning.
By: Anju Rajbangshi
Tags: Machine Learning, Data Science, Ensemble Learning, Predictive Analytics, Data Modeling

In my previous article, we discussed about the different types of regression in machine learning. So, today I am going to focus on the various ensemble methods in data science.

One thing I have experienced in the course of my career is that while learning any skill , things become easier when we try to relate concepts to events we face in our everyday lives. It’s effortless remembering things when we have a real life analogy. Hence, keeping that in mind, we will look at the various ensemble techniques by comparing it to everyday scenarios.

Table of contents

1. Introduction

2. Types of ensemble methods

3. Similarities between bagging and boosting methods

4. Differences between both the methods

5. Types of voting

6. Detailed understanding of the concepts

Bagging

Boosting

7. Conclusion

1. Introduction

What is an ensemble?

In simple English, ensemble refers to a group of items. For e.g: a group of ministers, a group of dancers etc.

What is ensemble method in machine learning?

An ensemble method is a technique which uses multiple independent similar or different models/weak learners to derive an output or make some predictions.

For e.g.

A random forest is an ensemble of multiple decision trees.

An ensemble can also be built with a combination of different models like random forest, SVM, Logistic regression etc.

So now, let’s compare it with a real life example.

Suppose a bill is passed in the parliament and a meeting is held for the same with a group of ministers.

Instead of the President or the prime minister taking the decision alone by themselves might not be a very good idea as this accounts for dictatorship and the results might not be so much in favor of the public. So, this analogy can be referred to a single model like Decision Tree, Logistic Regression etc in machine learning and they will always slightly perform less better than the ensemble methods.

However, if a meeting is held between different ministers , let’s say four of them, each of them will provide an opinion with pros and cons. And finally, the best opinion will be picked based on majority voting. This is exactly what ensemble method is.

Minister 1: based on his pros and cons, says the bill should be passed.

Minister 2: based on his pros and cons, says the bill is not useful and might trigger a lot of challenges. Hence should not be passed.

Minister 3: Says bill can be passed.

Minister 4: Also says bill can be passed.

Majority says ‘pass the bill’. Hence, the bill got passed in the parliament.

Similarly, we can see below that different models are used and based on the output of each, the final decision will be made on the majority voting.

Source: Google

Why do we need ensemble techniques?

As we all know, the errors and predictions in any machine learning models are adversely influenced by bias, variance and noise. In order to combat these disadvantages, ensemble methods are used.

2. Types of ensemble methods

There are many ensemble techniques available but we will discuss about the below two most widely used methods:

1. Bagging

2. Boosting

Let us first understand the similarities and differences between both of the terminologies.

3. Similarities between bagging and boosting methods

· Both the methods can be used for classification (discrete output) and regression (continuous output) problems.

· ­Both methods are better in performance compared to single models as they aggregate the output of all the weak learners via voting thereby leading to more accuracy in the predictions.

4. Differences between the two methods