XGBoost: A Deep Dive into Boosting
By: Rohan Harode
Tags: Xgboost, Data Science, Machine Learning, Gradient Boosting, Blog Post

Too complex to comprehend? We have got your back! Let’s first understand the rudiments of machine learning methods which leads us to XGBoost and the vitality of its performance.

A quick flashback to Boosting:

Boosting generally means increasing performance. In ML, Boosting is a sequential ensemble learning technique (another terminology alert, fret not! we will explain this as well) to convert a weak hypothesis or weak learners into strong learners to increase the accuracy of the model.

We can understand the need for boosting through a simple classification example: To classify a Twitter account as Bot or Human with the help of underlying rules (limited in extent):

● No account information and profile photo → Bot

● Username is gibberish → Bot

● Tweeting in more than one language → Bot

● It has adequate amount of activity and a picture→ Human

● An enormous amount of tweets per day → Bot

● Other social media accounts linked→ Human

As now we are aware of the rules, let’s apply these on the below example:

Scenario — An account posting tweets in English and French.

The 3rd rule will classify it as a bot, but this will be a false prediction as a person can know and tweet in multiple languages. Consequently, based on a single rule, our prediction can be flawed. Since these individual rules are not strong enough to make an accurate prediction, they are called weak learners. Technically, a weak learner is a classifier that has a weak correlation with the actual value. Therefore, to make our predictions more accurate, we devise a model that combines weak learners’ predictions to make a strong learner, and this is done using the technique of boosting.

Weak rules are generated at each iteration by base learning algorithms which in our case can be of two types:

● Tree as base learner

● Linear base learner

Generally, decision trees are default base learners for boosting.

Interested to know more about learners? Be assured, we will demonstrate how to use both the base learners to train XGBoost models in a later section.

First, let’s get to know about the ensemble learning technique mentioned above.

Ensemble Learning:

Ensemble learning is a process in which decisions from multiple machine learning models are combined to reduce errors and improve prediction when compared to a Single ML model. Then the maximum voting technique is used on aggregated decisions (or predictions in machine learning jargon) to deduce the final prediction. Puzzled?

Think of it as organizing efficient routes to your work/college/or grocery stores. As you can use multiple routes to reach your destination, you tend to learn the traffic and the delay it might cause to you at different times of the day, allowing you to devise a perfect route, Ensemble learning is alike!

This image shows a clear distinction of a single ML model with respect to Ensemble Learner:

Single Model Prediction vs Ensemble Learner

Types of Ensemble learning:

Ensemble learning methods can be performed in two ways:

● Bagging (parallel ensemble)

● Boosting (sequential ensemble)

Though both have some fascinating math under the cover, we do not need to know it to be able to pick it up as a tool. Our focus will be more on the understanding of Boosting due to its relevance to XGBoost.

Working of boosting algorithm:

The boosting algorithm creates new weak learners (models) and sequentially combines their predictions to improve the overall performance of the model. For any incorrect prediction, larger weights are assigned to misclassified samples and lower ones to samples that are correctly classified. Weak learner models that perform better have higher weights in the final ensemble model. Boosting never changes the previous predictor and only corrects the next predictor by learning from mistakes. Since Boosting is greedy, it is recommended to set a stopping criterion such as model performance (early stopping) or several stages (e.g. depth of tree in tree-based learners) to prevent overfitting of training data. The first implementation of boosting was named AdaBoost (Adaptive Boosting).

Capital F(i) is current model, F(i-1) is previous model and small f(i) represents a weak model

Internal working of boosting algorithm

Feeling confident? Take a look at two more algorithms (CART and Gradient Boosting) to understand the mechanics of XGBoost before we delve deeper into the topic.

Classification And Regression Trees (CART):

A decision tree is a supervised machine learning algorithm used for predictive modeling of a dependent variable (target) based on the input of several independent variables. It has a tree-like structure with the root at the top. CART which stands for Classification And Regression Trees is used as an umbrella term to refer to the following types of decision trees:

Classification Trees: where the target variable is fixed or categorical, this algorithm is used to identify the class/category within which the target would most likely fall.

Regression Trees: where the target variable is continuous and the tree/algorithm is used to predict its value, e.g. predicting the weather.

Gradient Boosting:

Gradient boosting is a special case of boosting algorithm where errors are minimized by a gradient descent algorithm and produce a model in the form of weak prediction models e.g. decision trees.

The major difference between boosting and gradient boosting is how both the algorithms update model (weak learners) from wrong predictions. Gradient boosting adjusts weights by the use of gradient (a direction in the loss function) using an algorithm called Gradient Descent, which iteratively optimizes the loss of the model by updating weights. Loss normally means the difference between the predicted value and actual value. For regression algorithms, we use MSE (Mean Squared Error) loss as an evaluation metric while for classification problems, we use logarithmic loss.

w represents the weight vector, η is the learning rate

Gradient Boosting Process:

Process flow of Gradient Boosting

Gradient boosting uses Additive Modeling in which a new decision tree is added one at a time to a model that minimizes the loss using gradient descent. Existing trees in the model remain untouched and thus slow down the rate of overfitting. The output of the new tree is combined with the output of existing trees until the loss is minimized below a threshold or specified limit of trees is reached.

Additive Modeling in mathematics is a breakdown of a function into the addition of N subfunctions. In statistical terms, it can be thought of as a regression model in which response y is the arithmetic sum of individual effects of predictor variables x.