Gradient Boosting Technique
By: Charanraj Shetty
Tags: Supervised Learning, Machine Learning, Gradient Boosting, Classification, Boosting

Gradient Boosting Technique

Introduction

Gradient Boosting Technique is a supervised Machine Learning Algorithm that belongs to the Ensemble Boosting Technique family. It is generally applied for both Regression and Classification problems to generate a model that evaluates the output features based on the combination output result of weak prediction learner models. It applies the concept of minimizing a loss function to optimize the model. The Sequential models are constructed using the decision trees until maximum accuracy is achieved.

When Gradient Boosting is used to predict a continuous output value for a given dataset then we are performing Gradient Boosting for Regression. Likewise, if we are predicting discrete-valued output then we are performing Gradient boosting for Classification.

Here in this article, we will focus primarily on Gradient Boosting using Regression.

Steps in Gradient Boosting

We are using the below dataset as an example where we are predicting the Salary of candidates based on factors such as age, degree, and years of exp.

Image by Author

Step1: Taking the average values of Output Feature

Here we calculate the average value of salary being offered to candidates and assume it to be our predicted output value. So we add a new column next to Salary(Actual output value) column.

=Sum of all the salary

=60+50+65 = 58

Image by Author

Step 2:Calculate the residual value

Next is the calculation of the difference between predicted and actual values. The difference is known as the residual value.

Residual = Actual value — Predicted value

Image by Author

Although it's visible from the above table that adding a Residual to Predicted Salary gives us the Expected Actual Salary but do you think it's feasible? Well, this might help us to save ourselves during the testing of the model but we might not be able to get a lot of variation during the testing of the model. So basically we might come across the issue of low Bias and High Variance while applying the model to unknown datasets. The solution to this is explained in the latter part of the article.

Step 3: Generating the Base Learner decision Tree

We then construct the Decision tree based on the residuals as output values and the given independent features as input parameters. To avoid the issue of High Variance we scale the residual output values using a learning parameter and add it to the Average Predicted Salary.

Image by Author

Step 4: Generating Predicted output for Decision Tree1

Image by Author

The R1 Residual values calculated by us in the above image show the actual values of the Decision Tree model. We can obtain the predicted values of the Decision tree outputs using the formula of the loss function. As we intend to minimize the loss between actual and predicted output, the Residual value R2 which represents the predicted output for the Decision Tree will always have a value less than its output.

Step 5: Predicting new o/p value for each sample

So the new Predicted value after applying the Decision Tree is given as follows:

Previous Predicted value + Learning rate * Current Predicted value

α = Learning rate = 0.1

Here the learning rate is introduced to resolve the High Variance problem introduced above in step2.

Now we consider record-1 in the above image to understand things better.

Here,

The actual output is initially given by Salary = 60k

The Predicted Salary is given by = 58k

The Predicted Salary of Tree 1 = 8k

So the new predicted value is given by

58+0.1*8 = 58.8

So by applying one decision tree the predicted value came closer to the actual value 60 from 58 to 58.8.

So the conclusion is that we need to keep on repeating the process of creating Decision trees until we reach a stage where the Predicted value is equal to the actual value.

We need to keep on performing the Iteration to achieve maximum accuracy.

Mathematical Intuition

To understand the above process we move forward to understand the Algorithm designed for the same. We will do step by step comparison for the algorithm with the numerical to grasp things better.

Image by Author

Step1:

It is the calculation of the average value which is defined as a constant value. We calculate the constant value or the first predicted Salary value by applying the concept of minimizing the difference between actual and predicted results i.e. Loss function. Mathematically minimizing anything means differentiation at that point is zero. Hence in the below image we

Image by Author

Now putting back the values of y for each record we can calculate the predicted value or average value for the entire dataset.

Image by Author

Step 2(a):

For Ῡ = F₀( x ) = predicted value we need to now calculate the residual or difference of predicted value w.r.t y.

To calculate residual we use the Decision Tree which is represented by m.

rᵢₘ = residual of record I for decision tree m.

Image by Author

Step 2(b):

We now construct decision tree 1 using the residual as outputs and the features x as inputs.

Step 2(c):

For each actual residual output which is obtained in the above step, we will now calculate the predicted output Ῡ.

We already have the predicted value of the model1 which is defined as the average value in step-1.

To obtain the residual predicted output in the Decision Tree we are using the concept of loss function of step-1.

j = leaf node in decision tree

yᵢ = actual output

Ῡ = predicted output

Fₘ-₁( x ) = previous model output

Image by Author

Step-2(d)

After obtaining the predicted value of the Decision tree we combine it with the previous model value. The loop will keep on repeating itself until the predicted value is equal to the actual value.

Image by Author

Thank you for reading out the article!!

You can reach me on LinkedIn