Types of Ensemble methods in Machine learning | by Anju Rajbangshi | Towards Data Science

In my previous article, we discussed about the different types of regression in machine learning. So, today I am going to focus on the various ensemble methods in data science.
One thing I have experienced in the course of my career is that while learning any skill , things become easier when we try to relate concepts to events we face in our everyday lives. It’s effortless remembering things when we have a real life analogy. Hence, keeping that in mind, we will look at the various ensemble techniques by comparing it to everyday scenarios.
1. Introduction
2. Types of ensemble methods
3. Similarities between bagging and boosting methods
4. Differences between both the methods
5. Types of voting
6. Detailed understanding of the concepts
Bagging
Boosting
7. Conclusion
What is an ensemble?
In simple English, ensemble refers to a group of items. For e.g: a group of ministers, a group of dancers etc.
What is ensemble method in machine learning?
An ensemble method is a technique which uses multiple independent similar or different models/weak learners to derive an output or make some predictions.
For e.g.
A random forest is an ensemble of multiple decision trees.
An ensemble can also be built with a combination of different models like random forest, SVM, Logistic regression etc.
So now, let’s compare it with a real life example.
Suppose a bill is passed in the parliament and a meeting is held for the same with a group of ministers.
Instead of the President or the prime minister taking the decision alone by themselves might not be a very good idea as this accounts for dictatorship and the results might not be so much in favor of the public. So, this analogy can be referred to a single model like Decision Tree, Logistic Regression etc in machine learning and they will always slightly perform less better than the ensemble methods.
However, if a meeting is held between different ministers , let’s say four of them, each of them will provide an opinion with pros and cons. And finally, the best opinion will be picked based on majority voting. This is exactly what ensemble method is.
Minister 1: based on his pros and cons, says the bill should be passed.
Minister 2: based on his pros and cons, says the bill is not useful and might trigger a lot of challenges. Hence should not be passed.
Minister 3: Says bill can be passed.
Minister 4: Also says bill can be passed.
Similarly, we can see below that different models are used and based on the output of each, the final decision will be made on the majority voting.
Why do we need ensemble techniques?
As we all know, the errors and predictions in any machine learning models are adversely influenced by bias, variance and noise. In order to combat these disadvantages, ensemble methods are used.
There are many ensemble techniques available but we will discuss about the below two most widely used methods:
1. Bagging
2. Boosting
Let us first understand the similarities and differences between both of the terminologies.
· Both the methods can be used for classification (discrete output) and regression (continuous output) problems.
· ­Both methods are better in performance compared to single models as they aggregate the output of all the weak learners via voting thereby leading to more accuracy in the predictions.
There are two types of voting used to aggregate the output of all the weak learners.
Hard voting — If we receive the output of each weak learner in classes (for e.g: 0/1), and finally we select the output class which was returned in majority, then this method is called hard voting.
For instance: In our example of the group of ministers, since 3 ministers were in favor of the bill, the final decision was made to pass the bill. This is hard voting. Majority wins.
Soft voting — There are many algorithms which also provide us with prediction probabilities and if we receive the output in probabilities, we take the average of the probabilities of each of the classes and whichever class is having the highest average will be the final prediction. This way of prediction is called Soft voting and is better than hard voting as we receive more information in terms of probability.
Another important thing to remember is, this method of voting is used as a default technique for ensembles like random forest.
Let’s understand these two concepts using below diagrams where 1 is class 1 and 0 is class 0.
Also known as Bootstrap aggregating because the training set is divided into multiple sub samples and fed to each weak learner(for e.g.: each decision tree). And each sub sample contains a few set of features along with a few observations selected randomly.
Since the main idea of this article is to understand the concepts in Layman terms, let us take an easy example to comprehend bagging.
We all attend interviews right and the interview process is the most simple scenario I could come across which is analogous to Bagging.
Suppose you went to attend an interview and as soon as you enter the room, you see five stern looking people sitting there to lash out all their questions at you . What do you do? Feel nervous and storm out of the room !! No right? You went there because you need the job and you know you are bold enough , intelligent enough to face whatever comes your way and you greet them anyway and sit down in front of them. So, let’s face this.
Now consider each of the five interviewers are going to interview the candidate on five different aspects say aptitude , technical -basic ,technical- advanced, communication and management skills .
Also, the company might already have prepared a set of questions based on the different aspects to test the candidate’s skills. And let’s name this as ‘question set’.
So, the interview starts now. And our target variable here would be if the candidate gets selected or not.
Interviewer 1 — The interviewer 1 starts testing your aptitude skills taking a sample of aptitude based questions from the ‘question set’ and sets some probabilities whether you should be selected or not.
Interviewer 2 — The interviewer 2 also starts testing your basic technical skills, again taking a sample of preliminary technical questions from the’ question set’ and sets some probabilities whether you should be selected or not.
Interviewer 3 — Similarly, the interviewer 3 starts testing your advanced technical skills taking another sample of slightly difficult technical questions from the’ question set’ and again sets some probabilities whether you should be selected or not.
Now, It can also be possible that he asks you some questions picked randomly from the ‘question set’ but is similar and was already asked in basic technical test or any of the previous tests . Such a scenario is called bootstrap in bagging method as the same sample may be picked up again due to the use of random sampling with replacement.
Interviewer 4 & Interviewer 5: Interviewer 4 and interviewer 5 repeats the same process as above, testing the candidate’s communication and management skills taking a sample of related questions from the ‘question set’ and accordingly sets some probabilities whether to select or not.
Result: Finally the candidate is done with the interview and he comes out of the room leaving the interviewers to analyse their respective scores about his skills based on the sub samples of questions from the ‘question set’ that he was scored upon . Ultimately they aggregate all of their individual opinions and come out with the final decision whether to select the candidate or not.
This is exactly similar to ensemble methods.
Let’s try out a few ensemble methods using python and compare their accuracy corresponding to a few single models.
Note: The jupyter notebook with complete code is available in github.
I am using Advertisement Success dataset from Kaggle for this purpose and want to predict the success of an Ad.
a. Decision tree
# Decision tree classifier with grid seacrh CV and model evaluation using accuracy score, precision score and AUC/ROC curve.from sklearn.tree import DecisionTreeClassifierfrom sklearn.model_selection import GridSearchCVparameters = {'max_features': [0.5,0.6,0.7,0.8,0.9,1.0], 'max_depth': [2,3,4,5,6,7],'min_samples_leaf':[1,10,100],'random_state':[14]} clf = GridSearchCV(DecisionTreeClassifier(), parameters, cv=5, scoring='roc_auc')clf.fit(X_train, y_train)
Evaluation metrics of Decision Tree applied on example data :
AUC score is 85.84%.
Decision trees are always prone to overfitting if we don’t choose the right parameters like minimum sample of leaf, minimum sample of nodes, maximum depth of the tree as higher the depth , more minutely will the model capture the data points of the training set leading to excellent predictions in the training dataset itself but will fail on new data. Hence, ensemble methods are one of the techniques which helps to reduce the variance thereby solving overfitting encountered by a single decision tree.
b. Single Logistic Regression classifier
#Single Logistic Regressionfrom sklearn.linear_model import LogisticRegressionlog = LogisticRegression(random_state=0, solver='lbfgs') log.fit(X_train, y_train)y_pred = log.predict(X_test)
Evaluation Metric of Single Logistic regression classifier applied on example data:
AUC score is 83.84 %.
Few examples are Random Forest , Extra tree classifiers/regressors, ensemble of linear regressors, ensemble of logistic regression classifiers, ensemble of SVMs etc.
a. Random forest — an ensemble of multiple decision trees
from sklearn.ensemble import RandomForestClassifierparameters = {'n_estimators':[700],'n_jobs':[-1], 'max_features': [0.5,0.7,0.9], 'max_depth': [3,5,7],'min_samples_leaf':[1,10],'random_state':[14]} clf1 = GridSearchCV(RandomForestClassifier(), parameters, cv=5, scoring='roc_auc')clf1.fit(X_train, y_train)
Evaluation metrics of random forest applied on example data:
AUC score is 86.53 %.
b. Ensemble of logistic regression classifiers.
When we use an ensemble of same classifiers, Sklearn’s BaggingClassifier is used to get the OOB (Out of bag) score which is a very important score in bagging to examine the performance of the classifiers as it gives us an idea of the accuracy to be achieved on the test set. As we all know, during training, sub samples are fed to each independent classifier but in bagging technique, it can also happen that the same sample is passed two to three times to any of the other individual classifiers. So , in that scenario the samples which are new to a particular classifier i.e. the samples which are not already seen are called out of bag samples. It serves the same purpose of a test set as OOB instances are something not seen earlier. We can get the OOB score using ‘classifier.oob_score_’ function after building the ensemble using Bagging Classifier.
# Multiple logistic regression classifiers using bagging Classifier.# Number of logistic regression classifiers we are using here are 400.logbagClf = BaggingClassifier(LogisticRegression(random_state=0, solver='lbfgs'), n_estimators = 400, oob_score = True, random_state = 90)logbagClf.fit(X_train, y_train)
Evaluation metric of the bagging classifier(multiple Logistic Regression Classifiers):
AUC score is 84.44 %.
For this technique, we can use Sklearn’s VotingClassifier.
For my testing purpose, I am using an ensemble of RandomForestClassifier, Support vector machine and Logistic regression.
from sklearn.ensemble import RandomForestClassifier, VotingClassifierfrom sklearn.svm import SVCfrom sklearn.linear_model import LogisticRegressionfrom sklearn.naive_bayes import GaussianNBrfClf = RandomForestClassifier(n_estimators=500, random_state=0) # 500 trees. svmClf = SVC(probability=True, random_state=0) # probability calculationlogClf = LogisticRegression(random_state=0)#nbclf = GaussianNB(random_state=0)# constructing the ensemble classifier by mentioning the individual classifiers.clf2 = VotingClassifier(estimators = [('rf',rfClf), ('svm',svmClf), ('log', logClf)], voting='soft') # train the ensemble classifierclf2.fit(X_train, y_train)
Evaluation metrics of the Voting Classifier(Ensemble of multiple models):
AUC score is 84.92 %
The main idea behind boosting is to convert weak learners to strong learners in sequential iterations.
Let us again take the example of interviews to understand the concept boosting.
Suppose a candidate has attended five interviews in five different companies. And based on his performance in these five interviews, he wants to rate himself on the skill.
Interview 1 — company 1 — He could answer only a few of the questions but he got an idea of the type of questions he could be asked again. He comes back home and studies. His learning rate increased.
Interview 2 — company 2 — Here, he did better than the first company as he learns all the concepts he missed on the first interview but still not selected. He comes back home and studies more thereby correcting all the mistakes he made in the previous two interviews.
Interview 3 and interview 4 also, he faced the same issue and by the time, he attends the 5th interview, he became an expert and was almost able to answer all the questions.
Ultimately all we see here is the candidate being exposed to different sample of questions and him learning and improving himself gradually till he becomes an expert. The outcome is he could rate himself the best after learning from his mistakes and correcting himself. There is a learning rate involved in each step. Same is the concept of boosting.
I will implement boosting algorithm XGBoost on my dataset.
import xgboost as xgbxgb_clf = xgb.XGBClassifier(max_depth=3,n_estimators=300,learning_rate=0.05)    xgb_clf.fit(X_train,y_train)
Evaluation metrics of XGBoost applied on example data:
AUC score is 86.46 %.
Let’s compare the accuracies , precision and AUC score of the single and the ensemble models.
Accuracy Score: Boosting algorithm ‘XGBoost’ is leading on the accuracy score.
Accuracy Score: Boosting algorithm ‘XGBoost’ is leading on the accuracy score.
We can also notice that ensemble classifiers ‘random forest’ and ‘multiple logistic regression classifiers’ have better accuracies compared to single classifiers ‘Decision Tree’ and ‘Logistic Regression’ respectively.
2. Precision Score: Boosting algorithm ‘XGboost’ is leading on the precision score.
We can notice that ensemble classifiers ‘random forest’ and ‘multiple logistic regression classifiers’ have better precision compared to single classifiers ‘Decision Tree’ and ‘Logistic Regression’ respectively.
3. AUC Score: Bagging algorithm ‘Random forest’ is leading on the AUC score even though there is very minute difference with the Auc score of XGboost.
We can notice that ensemble classifiers ‘random forest’ and ‘multiple logistic regression classifiers’ have better AUC scores compared to single classifiers ‘Decision Tree’ and ‘Logistic Regression’ respectively.
Hence, we can conclude that boosting method performed best on our dataset and we can finally deploy XGBoost model as it will predict most precisely the success of an ad campaign which is the main agenda of our problem statement.
However, It depends on the type of data we are dealing with.
Also, we can infer that bagging and boosting methods always perform better with great accuracies as compared to the single models.
Kindly add any additional thoughts, comments as it will help me understand things better.
Thanks for reading !!