Bagging, Boosting, and Gradient Boosting | by Chirag Chadha | Towards Data Science

Bagging is the aggregation of machine learning models trained on bootstrap samples (Bootstrap AGGregatING). What are bootstrap samples? These are almost independent and identically distributed (iid) samples – in that there is low correlation between the samples and they are drawn from the same distribution. The process of bootstrapping is used to create these samples. This involves drawing samples with replacement from a dataset keeping in mind that if these samples are large enough, they will be representative of the dataset they are drawn from, under the assumption that the dataset they are being drawn from is also large enough to be representative of the population. Having a high number of observations in our dataset also means that the bootstrap samples are more likely to have low correlation between samples.
Why don’t we simply draw more samples from the population instead of taking samples from our dataset? We may not have the resources or capability to acquire further samples from the population – oftentimes, in research and in industry, we must solely rely on the data that we have been provided or have previously collected. In this scenario, bootstrapping has proved itself to be an accurate and computationally efficient method of estimating the distribution of the population.
For example, if we have data on whether an individual defaulted on their loan with Bernoulli distributed response variable (0 or 1 for whether the person defaulted or not), we can calculate the mean of the response variable to get a quick estimate of how well represented the classes are in our dataset. In order to then find the distribution of the mean of the response variable (to get an idea of its variability and how representative our dataset is of the population), we would have to resample a few times from the population and calculate the mean for each of those samples in order to obtain a good understanding of the true distribution of the response variable. However, this is not always possible and we are stuck solely with the one sample (our dataset). It has been shown empirically that it is possible to obtain a pretty accurate estimate of the true distribution of our sampling statistic (mean of response variable) through bootstrapping using only the data that we have available by drawing samples with replacement from the dataset itself.
The idea of bagging follows naturally from bootstrapping as independent, homogeneous, machine learning models are trained on the almost iid bootstrap samples and their results are aggregated through a weighted mean. The reasoning behind training on these bootstrap samples is that the samples have low correlation and thus we get a better representation of the population. The popular bagging algorithm, random forest, also sub-samples a fraction of the features when fitting a decision tree to each bootstrap sample, thus further reducing correlation between samples. Bagging provides a good representation of the true population and so is most often used with models that have high variance (such as tree based models).
On a high level, all boosting algorithms work in a similar fashion:
All observations in the dataset are initially given equal weights.A machine learning model is trained on this dataset.The model from step 2 is added to the ensemble with an error associated with it that is the sum of the weights for the misclassifed observations.Misclassified observations are given greater weights.Steps 2–4 are repeated for a number of epochs.
All observations in the dataset are initially given equal weights.
A machine learning model is trained on this dataset.
The model from step 2 is added to the ensemble with an error associated with it that is the sum of the weights for the misclassifed observations.
Misclassified observations are given greater weights.
Steps 2–4 are repeated for a number of epochs.
The idea here is to make misclassified observation stand out as being more important in subsequent learning iterations. It should be noted at this point that boosting is an immensely powerful ensembling technique but, due to its nature of focusing on difficult observations, it is prone to over-fitting. As such, it is often used with base models that are relatively biased, e.g. shallow trees.
The easiest boosting algorithm to understand is AdaBoost, which works as follows:
Initialise variables.
M = Number of models to fitN = Number of observations in datasetm = 1 # Iteration counter, m <= M.w_i = 1/N # For i = 1, 2, 3, ..., N.F_0 = 0 # Ensemble model.
Train a model f_m using the observation weights w_i that minimizes the weighted error e_m = sum(w_misclassified) .
Latest ensemble model F_m = F_m_minus_1 + (alpha_m * f_m) where alpha_m = log(1 — e_m) / e_m .
Updates weights of misclassified observations with an increase proportional to alpha_m .
Increment iteration counter.
m += 1if m <= M:    go to step 2
Final model is the sum of predictions from the individual models times their respective alpha coefficients.
F = (alpha_1 * f_1) + (alpha_2 * f_2) + ... + (alpha_m * f_m)
This method is named gradient boosting as it uses a gradient descent algorithm to minimise loss when adding models to the ensemble. Subsequent models are fit to pseudo-residuals instead of adjusting weights. Stochastic gradient descent (as used by XGBoost) adds randomness by sampling observations and features in each stage, which is similar to the random forest algorithm except the sampling is done without replacement.
The gradient boosting algorithm improves on each iteration by constructing a new model that adds an estimator h to provide a better model. The perfect h would then imply:
Where y is the target. Or simply:
This is a residual between the target and the predictions from the previous iteration’s model. In order to approximate this residual, we use the aforementioned pseudo-residual, which is explained in the following steps:
Initially, train on dataset as normal and obtain predictions for each observation.Compute pseudo-residuals as the negative of the partial derivative of the loss function with respect to the predictions from step 1.Use pseudo-residuals as new target for next model and obtain predictions for each observation as before.Repeat steps 2–3 for M iterations.
Initially, train on dataset as normal and obtain predictions for each observation.
Compute pseudo-residuals as the negative of the partial derivative of the loss function with respect to the predictions from step 1.
Use pseudo-residuals as new target for next model and obtain predictions for each observation as before.
Repeat steps 2–3 for M iterations.
In doing the above, we’re effectively predicting an approximation of the residuals in each iteration and as such, we get the following ensemble model:
with h as described in:
replaced by our pseudo-residual value.
James, G., Witten, D., Hastie, T. and Tibshirani, R. (n.d.). An introduction to statistical learning.