Improving the Performance of Machine Learning Model using Bagging | by Satyam Kumar | Towards Data Science

The performance of a machine learning model tells us how the model performs for unseen data-points. There are various strategies and hacks to improve the performance of an ML model, some of them are:
Fine tuning hyperparameter of the ML model
Using ensemble learning.
Ensemble Learning is a technique to combine multiple ML models to form a single model. The multiple ML models also referred to as base models or weak learners can be of different algorithms or same algorithms with a change in hyperparameters.
Like for classification tasks, multiple ML models can be Logistic Regression, Naive Bayes, Decision Tree, SVM, etc. For regression tasks, multiple ML models can be Linear Regression, Lasso Regression, Decision Tree Regression, etc.
Ensemble Learning combines the advantages of base models to form a single robust model with improved performance. Various types of ensemble learning techniques are:
Bagging (Bootstrap Aggregation)BoostingVotingCascadingStacking
Bagging (Bootstrap Aggregation)
Boosting
Voting
Cascading
Stacking
And many more. This article will cover the working and implementation of the Bagging Ensemble technique.
Bagging ensemble technique also known as Bootstrap Aggregation uses randomization to improve performance. In bagging, we use base models that are trained on part of the dataset. In bagging, we use weak learners (or base models) models as building blocks for designing complex models by combining several of them.
Most of the time, these base models do not perform well because they either overfit or underfit. Overfitting or underfitting of a model is decided by bias-variance tradeoff.
What is Bias-Variance Tradeoff ? [1]
The overall error of a model is dependent on the bias and variance of the model following the below equation:
For a good robust model, the error of the model is as minimum as possible. To minimize the error the bias and variance need to be minimum and irreducible error remains constant. The below plot of error vs model flexibility (degree of freedom) describes the change of bias and variance along with test and training error:
Analysis from the above image plot:
When the model is in the initial phase of training, training, and test error are very high.
When the model is enough trained, the training error is very low and the test error is high.
The phase where training and test error is high is Underfitting.
The phase where training error is low and the test error is high is Overfitting.
The phase where there is a balance between training and testing error is Best fitting.
An Underfit model has low variance and high bias.
An Overfit model has high variance and low bias.
Bagging Ensemble technique can be used for base models that have low bias and high variance. Bagging ensemble uses randomization of the dataset (will be discussed later in this article) to reduce the variance of base models keeping the bias low.
It is now clear that bagging reduces the variance of base models keeping the bias low. The variance of base models is reduced by combining strategies of bootstrap sampling and aggregation. The entire working of Bagging is divided into 3 phases:
Bootstrap SamplingBase ModelingAggregation
Bootstrap Sampling
Base Modeling
Aggregation
Below diagram describes all three steps for a sample dataset D having n rows:
A bootstrap sample is a smaller sample that is a subset of the initial dataset. The bootstrap sample is created from the initial dataset by sampling with replacement.
Suppose for a dataset having n rows and f features, we do a bootstrap sampling that refers to sampling with replacement into k different smaller datasets each of size m with the same f features. Each smaller dataset D_i formed sees a subset of the dataset. In the figure below initial dataset D of shape (n, f) is sampled to k dataset each of shape (m, f), where m<n.
The below image describes how to bootstrap the sample. The dataset D having 10 rows is sampled with replacement into k smaller dataset each having 5 rows. Here n=10 and m=5 from the above diagram.
It is observed that each of the datasets formed by bootstrapping sees only a part of the original dataset and all the datasets are independent of each other.
This is the 1st step of the bagging ensemble technique in which k smaller datasets are created by bootstrapping independent of each other.
Modeling is the 2nd step of bagging. After k smaller datasets are created by bootstrapping each of the k datasets is trained using ML algorithms. The algorithms used for training k dataset can be the same with or without a change in hyperparameters or different algorithms can be used.
For example,
Decision Tree algorithms can be used as base models with a change in hyperparameters such as ‘depth’.
A combination of different algorithms such as SVM, Naive Bayes, Logistic Regression can be used.
The models which are trained on each bootstrap dataset are called base models or weak learners. The below diagram describes the training of each dataset of separate models:
A final powerful robust model is creating by combining the k different base models. Since the base models are trained on a bootstrap sample so each model may have different predictions. The aggregation technique is different depending on the problem statement.
For a regression problem: The aggregation can be taking the mean of prediction of each base model.
Notation,prediction: Final Output of bagging ensemblek: number of base modelspred_i: prediction of ith base model
For a classification problem: The aggregation can be using majority voting, the class having the maximum vote can be declared as the final prediction.
Notation,prediction: Final Output of bagging ensemblepred_i: prediction target class of ith base model1,2,3...,c: c different target classC: Target Class having maximum vote
Random Forest is an example of bagging ensemble learning. In the Random Forest algorithm, the base learners are only Decision Trees. Random Forest uses bagging along with column sampling to form a robust model.
In other terms, RF uses an implementation of bagging ensemble learning, in addition to that, it does column sampling with replacement during bootstrapping step. The variation in the first step of bootstrapping is shown below:
The shape of the initial dataset D is (n rows, f features), in case of bootstrap sampling + column sampling, the k base dataset formed are both row and column sampled. The shape of the base dataset D_i is (m rows, d features) where n>m and f>d.
Implementation of Decision Tree and Random Forest classifier for Pima Indians Diabetes Dataset from Kaggle.
Observation of Decision Tree Classifier:
The decision tree classifier is trained for X_train (lines 16–19), and the performance of the DT classifier model (lines 21–23) is tested on X_test data. The observation of the performance of the DT model is:
Observation of Random Forest Classifier:
The Random Forest classifier for k=50 is trained for X_train (lines 25–28), and the performance of the RF classifier model (lines 30–32) is tested on X_test data. The observation of the performance of the RF model is:
Observation of Performance Improvement:
The accuracy of the model increases from 71% (for DT classifier) to 75% (RF Classifier).
The change in a decrease in FN and FP values and an increase in TP and FN values can be observed from the two confusion matrix.
References:
[1] (April 23, 2019), Ensemble methods: bagging, boosting and stacking: https://towardsdatascience.com/ensemble-methods-bagging-boosting-and-stacking-c9214a10a205
Thank You for Reading!