Boosting Algorithms Explained
By: Zixuan Zhang
Tags: Machine Learning, Data Science, Visualization, Mathematics, Data

Boosting Algorithms Explained

Theory, Implementation, and Visualization

Unlike many ML models which focus on high quality prediction done by a single model, boosting algorithms seek to improve the prediction power by training a sequence of weak models, each compensating the weaknesses of its predecessors.

One is weak, together is strong, learning from past is the best

To understand Boosting, it is crucial to recognize that boosting is a generic algorithm rather than a specific model. Boosting needs you to specify a weak model (e.g. regression, shallow decision trees, etc) and then improves it.

With that sorted out, it is time to explore different definitions of weakness and their corresponding algorithms. I’ll introduce two major algorithms: Adaptive Boosting (AdaBoost) and Gradient Boosting.

1.AdaBoost

1.1 Definition of Weakness

AdaBoost is a specific Boosting algorithm developed for classification problems (also called discrete AdaBoost). The weakness is identified by the weak estimator’s error rate:

In each iteration, AdaBoost identifies miss-classified data points, increasing their weights (and decrease the weights of correct points, in a sense) so that the next classifier will pay extra attention to get them right.The following figure illustrates how weights impact the performance of a simple decision stump(tree with depth 1)

How sample weights affect the decision boundary

Now with weakness defined, the next step is to figure out how to combine the sequence of models to make the ensemble stronger overtime.

1.2 Pseudocode

There are several different algorithms proposed by researchers. Here I’ll introduce the most popular method called SAMME, a specific method that deals with multi-classification problems. (Zhu, H. Zou, S. Rosset, T. Hastie, “Multi-class AdaBoost”, 2009).

AdaBoost trains a sequence of models with augmented sample weights, generating ‘confidence’ coefficients Alpha for individual classifiers based on errors. Low errors leads to large Alpha, which means higher importance in the voting.

the size of dots indicates their weights

1.3 Implementation in Python

Scikit-Learn offers a nice implementation of AdaBoost with SAMME (a specific algorithm for Multi classification).

Parameters:base_estimator : object, optional (default=None) The base estimator from which the boosted ensemble is built. If None , then the base estimator is DecisionTreeClassifier(max_depth=1) n_estimators : integer, optional (default=50) The maximum number of estimators at which boosting is terminated. In case of perfect fit, the learning procedure is stopped early. learning_rate : float, optional (default=1.) Learning rate shrinks the contribution of each classifier by learning_rate . algorithm : {‘SAMME’, ‘SAMME.R’}, optional (default=’SAMME.R’) If ‘SAMME.R’ then use the SAMME.R real boosting algorithm. base_estimator must support calculation of class probabilities. If ‘SAMME’ then use the SAMME discrete boosting algorithm. random_state : int, RandomState instance or None, optional (default=None)

2. Gradient Boosting

2.1 Definition of Weakness

Gradient boosting approaches the problem a bit differently. Instead of adjusting weights of data points, Gradient boosting focuses on the difference between the prediction and the ground truth.

weakness is defined by gradients

2.2 Pseudocode

Gradient boosting requires a differential loss function and works for both regression and classifications. I’ll use a simple Least Square as the loss function (for regression). The algorithm for classifications shares the same idea, but the math is slightly more complicated.(J. Friedman, Greedy Function Approximation: A Gradient Boosting Machine)

Gradient Boosting with Least Square

Following is a visualization of how weak estimators H are built over time. Each time we fit a new estimator (regression tree with max_depth =3 in this case) to the gradient of loss(LS in this case).

gradient is scaled down for visualization purpose

2.3 Implementation in Python

Again, you can find Gradient Boosting function in Scikit-Learn’s library.

Regression: loss : {‘ls’, ‘lad’, ‘huber’, ‘quantile’}, optional (default=’ls’) Classification: loss : {‘deviance’, ‘exponential’}, optional (default=’deviance’) The rest are the same learning_rate : float, optional (default=0.1) n_estimators : int (default=100) Gradient boosting is fairly robust to over-fitting so a large number usually results in better performance. subsample : float, optional (default=1.0) The fraction of samples to be used for fitting the individual base learners. If smaller than 1.0 this results in Stochastic Gradient Boosting. subsample interacts with the parameter n_estimators . Choosing subsample < 1.0 leads to a reduction of variance and an increase in bias. criterion : string, optional (default=”friedman_mse”) The function to measure the quality of a split.

Strength and Weakness

Easy to interpret: boosting is essentially an ensemble model, hence it is easy to interpret it’s prediction strong prediction power: usually boosting > bagging (random forrest) > decision tree resilient to overfitting: See this article sensitive to outliers: since each weak classifier is dedicated to fix its predecessors’ shortcomings, the model may pay too much attention to outliers hard to scale up: since each estimator is built on its predecessors, the process is hard to parallelize.

Summary

Boosting algorithms represent a different machine learning perspective: turning a weak model to a stronger one to fix its weaknesses. Now you understand how boosting works, it is time to try it in real projects!