Title: Scaling Kaggle Competitions Using XGBoost: Part 3 - PyImageSearch

URL Source: https://pyimagesearch.com/2023/01/16/scaling-kaggle-competitions-using-xgboost-part-3/

Published Time: 2023-01-16T14:00:00+00:00

Markdown Content:
* * *

**Table of Contents**
---------------------

*   [Scaling Kaggle Competitions Using XGBoost: Part 3](#h2BPTitle)

*   [Gradient Boost at a Glance](#h3Glance)
*   [AdaBoost vs. Gradient Boosting](#h3AdaBoost)
*   [Gradient Boosting Dissected](#h3Dissected)
*   [Configuring Your Development Environment](#h3Configuring)
*   [Having Problems Configuring Your Development Environment?](#h3Problems)
*   [Setting Up Our Project](#h3Setting)
*   [Comparing XGboost and Gradient Boost Results](#h3Comparing)

*   [Summary](#h2Summary)

*   [Citation Information](#h3Citation)

* * *

We continue our journey into understanding XGBoost, but there is one penultimate stop we need to make before deep diving into the nitty gritty of Extreme Gradient Boosting. We are not quite at the stage of embracing the “Extreme” nature of Gradient Boosting yet, as we are yet to demystify Gradient Boosting itself.

[![Image 1](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%20700%20393'%3E%3C/svg%3E)](https://pyimagesearch.com/wp-content/uploads/2023/01/Scaling-Kaggle-Competitions-Using-XGBoost-Part-3_featured.png)

In this blog post, we will thoroughly understand what Gradient Boosting is and understand the math behind this beautiful concept.

In this tutorial, you will learn about Gradient Boosting, the final precursor to XGBoost.

This lesson is the 3rd of a 4-part series on **Deep Learning 108**:

1.  [_Scaling Kaggle Competitions Using XGBoost: Part 1_](https://pyimg.co/0c9pb)
2.  [_Scaling Kaggle Competitions Using XGBoost: Part 2_](https://pyimg.co/2wiy7)
3.  [**_Scaling Kaggle Competitions Using XGBoost: Part 3_**](https://pyimg.co/8he0y) **(this tutorial)**
4.  _Scaling Kaggle Competitions Using XGBoost: Part 4_

**To learn how to utilize Gradient Boosting,** **_just keep reading._**

![Image 2](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%20128%20128'%3E%3C/svg%3E)

#### Looking for the source code to this post?

[Jump Right To The Downloads Section](#download-the-code)

* * *

[**Scaling Kaggle Competitions Using XGBoost: Part 3**](#TOC)
-------------------------------------------------------------

* * *

### [**Gradient Boost at a Glance**](#TOC)

In the [first blog post](https://pyimg.co/0c9pb) of this series, we went through basic concepts like ensemble learning and decision trees. To refresh your memory, we recommend going through the [first blog post](https://pyimg.co/0c9pb) of this series once again.

To recap: ensemble learners are normally a group of weak algorithms working together to produce quality output. Throughout this series, we have investigated algorithms by applying them to decision trees. So, it is important that you have a clear idea of how decision trees work.

The concept of Gradient Boosting works very similarly to how AdaBoost operates (see [previous blog post](https://pyimg.co/2wiy7)). The errors generated by the past predictor tree are used as information by the next predictor tree to improve its result. However, the two algorithms are very different.

* * *

### [**AdaBoost vs. Gradient Boosting**](#TOC)

For a quick reminder on how AdaBoost works, take a look at **Table 1**.

[![Image 3](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%20686%20500'%3E%3C/svg%3E)](https://lh5.googleusercontent.com/Buo-vW0GA7mflGQ_w0aj_MINhloorJbo0NLTppNeJCFQ_qEnK9NCtcJfFDJh-zDsL52uLw_ugnxobeuTvpJQgfP7n7QvM7ZPF4tCB-WpDFn9-LuB6zXJ8nHTceCZ3ZV_QqHwNNJIJzKWby8pNEhxjZNoxf0Jd0D9t8LPu4nRRc74tRgH_bV4VpMsiHEJ1A)

**Table 1:** AdaBoost Table.

Each sample is given a weight. Once the initial decision stump is created based on the features, we have certain samples which will get wrongly classified in those particular stumps. The next step is to focus on correcting those wrongly classified samples and change the weights based on the samples that need more focus than others (**Table 2**).

[![Image 4](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%20688%20500'%3E%3C/svg%3E)](https://lh4.googleusercontent.com/MMvkyqojw_cVwYddkLFukQUpItnpa36vSgHjlSRydoNemdIo5DMze-4JXAaddyQH2SGI8yw6MSXFCKERFE_7_7PhHWYbQV6QCHGy_dIkUeL_yHMNC7f3WGtRg1oNMEmSJ7E8M_JEq2DSl4-vU1peNU6KIuxiHgZeavH5NEkCb1VO0HrO4EFstV_iNbSxVQ)

**Table 2:** New Weights for the Samples.

Instead of working with sample weights, Gradient Boosting works with “residuals.” In the case of regression, residuals are nothing but the difference between the predicted values and the actual labels. These residuals tell us how far off our predictions are from the actual labels and become the labels themselves for subsequent tree creations (we will work these out in detail in the next section).

In short, AdaBoost works with sample weights focusing on wrongly classified samples to make sure they are corrected in subsequent stumps. But the Gradient Boosting algorithm is created so that samples get individual focus without additional weights to affect preferences. Also, the significance of stumps in the final say of the output is a key point in AdaBoost, which is not present in Gradient Boosting.

If the terms thrown around for Gradient Boosting in this section has you “stumped,” don’t worry; a detailed analysis is in order.

* * *

### [**Gradient Boosting Dissected**](#TOC)

Consistent with our previous posts, we will consider a dummy dataset to better understand the concepts (**Table 3**).

[![Image 5](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%200%200'%3E%3C/svg%3E)](https://lh6.googleusercontent.com/1GoD9XbCma40QlwTR4F9gCsv6KNjng-gUP32ZIOrFl3FWMbgYS_wYJvbFNKr2wTs7uzAFY3xdkILOY8pbOkyT3sWNakYgf3L23Lvf7LowlJhiSqYaUVehVCfQQCRpH7bpfoegykGRWOkJ-ITspgEI0RZxw099hv5edgP_wS43o1oU1M4_H1d9fv06qMlgA)

**Table 3:** Dummy Dataset for Gradient Boosting.

Note that this first example will focus on Gradient Boosting for regression. Gradient Boosting for classification is slightly different but is based on the same foundational principles you’ll learn for the regression task.

So we start with a dummy dataset with 4 samples. The `Courses` and `Credit` columns are the features, while the `Score` column is the label. The mathematical way of expressing this is ![Image 6: {(x_i, y_i)}^n_{i=1}](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%2063%2020'%3E%3C/svg%3E), where ![Image 7: x_i](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%2013%2010'%3E%3C/svg%3E) and ![Image 8: y_i](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%2012%2010'%3E%3C/svg%3E) are nothing but features and labels of the ![Image 9: i](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%205%2012'%3E%3C/svg%3E)th sample, respectively.

If you check the Wikipedia page for [Gradient Boosting](https://en.wikipedia.org/wiki/Gradient_boosting), there’s a section dedicated to the math equations used in Gradient Boosting (**Figure 1**).

[![Image 10](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%20700%20317'%3E%3C/svg%3E)](https://lh4.googleusercontent.com/sVw2KL90d33NoaLcRAeIzlHuc1sIppi_dxX9GKS6-KE3DwChgrRYVQItXWiRjmHEXveU71BjnIfl4fGLOt0Nl8qMbRGkGXSZkQyayZr3Gk7OSM51GZ-jh7AzW1O_DuetdPtUHn1RFW_SnfjlPHXs-fC_rM2yhtCgVf2FYDJbvI0rNcT-H0NASpGRRJ_K9A)

**Figure 1:** The Working of Gradient Boosting ([source](https://en.wikipedia.org/wiki/Gradient_boosting)).

Now at a glance, this might seem super complicated but don’t worry, stick with us, and we will systematically break this down for you.

Since we already have spoken about the training set, the next thing to note here is the differentiable loss function ![Image 11: L(y, F(x))](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%2072%2018'%3E%3C/svg%3E). This is where ![Image 12: F(x)](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%2033%2018'%3E%3C/svg%3E) is simply the predictions, and ![Image 13: y](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%208%2010'%3E%3C/svg%3E) is the corresponding labels, with the algorithm being run for ![Image 14: M](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%2017%2011'%3E%3C/svg%3E) (a choice dependent on the user) iterations.

**Step 1** is to initialize a model (in our case, a tree with a singular node). Now the equation might feel confusing to you at first glance, but it really isn’t. ![Image 15: \text{arg min}_\gamma](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%2062%2018'%3E%3C/svg%3E) simply means that we need to solve for a prediction ![Image 16: \gamma](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%209%2010'%3E%3C/svg%3E) (The prediction) such that it minimizes the equation.

Before we jump into that, let’s talk about the loss function. The standard go-to here is the squared loss with a constant of ![Image 17: 1/2](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%2023%2016'%3E%3C/svg%3E) before it. So it comes to ![Image 18: (1/2)(\text{Observed} - \text{Predicted})^2](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%20209%2018'%3E%3C/svg%3E). Note that these functions have constants like this since the loss functions require to be differentiated for backpropagation, and the differentiated loss function will look like: ![Image 19: (2/2) (\text{Observed} - \text{Predicted})](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%20202%2018'%3E%3C/svg%3E).

To find the ideal value for ![Image 20: \gamma](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%209%2010'%3E%3C/svg%3E) for our dummy dataset, we will:

*   ![Image 21: -(y_1 - \gamma) - (y_2 - \gamma) - (y_3 - \gamma) - (y_4 - \gamma) \ = \ 0](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%20341%2018'%3E%3C/svg%3E)
*   ![Image 22: -(81 - \gamma) - (96 - \gamma) - (95 - \gamma) - (51 - \gamma) \ = \ 0](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%20348%2018'%3E%3C/svg%3E)
*   ![Image 23: 4 \times \gamma \ = \ 323](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%2094%2015'%3E%3C/svg%3E)

which gives us the value for ![Image 24: \gamma = 80.75](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%2068%2016'%3E%3C/svg%3E).

This becomes our first predictor, the average of all the labels.

**Step 2** runs for the number of iterations you set it to but don’t get scared at how the equation looks. Under **Step 2**, we first calculate the “pseudo-residuals.” Now, as we have established that the loss function differentiated simply gives us ![Image 25: (\text{Observed} - \text{Predicted})](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%20165%2018'%3E%3C/svg%3E) (**Table 4**).

[![Image 26](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%200%200'%3E%3C/svg%3E)](https://lh3.googleusercontent.com/CV0DHGpk4KC__TXkM0bmFFrV0rXWYdkWyNapyDT-ImmD8-1JUuKT9FctQf3EtCGVODaQc2ZxGyhxXUO3PyHciWbAC3FKsSBEr0K5Srn0NKEPiyYassNSoAnUFztpQm6urhn6soSa0ta3YJE3AVuxIfOAJMGvYMFC3HPP42CTfohkpfqAivpNour_gxNo9w)

**Table 4:** First Residuals.

This gives us our first tables and residuals with which to work. To make things easier, you can consider this averaged-out node as your baseline regressor. The second part of the step involves us creating another decision tree, but this time it would be to predict the residuals. Let’s see what it turns out to be (**Figure 2**).

[![Image 27](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%200%200'%3E%3C/svg%3E)](https://lh4.googleusercontent.com/sihoKNYSNLBGHQo6UDdyKTXZVhe6tSrtzCybcl07wi4DF8z2OmQBuRUidvfi2ledS585bykfYi3OC-n99sa7igCC40_L0TPAa-wHmofnhl9qgGeXodXL8xogciWnhVOQqC0PjmlsjDkvaPYE9_Zyjlsoaw4kJvfOxGXscTl_H7-0sF_rGjuYsCCillkadQ)

**Figure 2:** The First Residual Tree.

For ease of explanation, we will be using this simple tree. In practical cases, you will have much better trees. The problem with this tree is that one of the nodes has multiple labels, which can’t be a possibility.

Hence, we need to figure out a single output that represents the node. We can use the equation we have used in **Step 1** just for the green node and solve for ![Image 28: \gamma](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%209%2010'%3E%3C/svg%3E), which would give us the representative value as ![Image 29: (0.25+15.25+14.25)/3 = 9.91](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%20221%2018'%3E%3C/svg%3E).

Now, for any new sample with courses greater than ![Image 30: 4](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%208%2013'%3E%3C/svg%3E), the green node will be represented by a single value, ![Image 31: 9.91](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%2028%2013'%3E%3C/svg%3E).

But now the question remains: how will we make use of this? For this, we move to the final part of **Step 2**.

*   Till now, we have our baseline predictor (the average of labels).
*   Since the average will not predict correctly for most samples, we calculate the residuals. Note that if you add the residuals to our initial prediction, we get the perfect labels for each training sample. Unfortunately, that means we are severely overfitting on the training data. For this reason, we compute a “learning rate” for each subsequent tree to take small steps toward our eventual result.
*   We create a decision tree to predict the residuals and ensure each leaf node has a single output representation.
*   Now, our updated model becomes ![Image 32: F_{(m-1)}(x) + (\text{learning}\_\text{rate}) \times F_m(x)](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%20253%2020'%3E%3C/svg%3E).

Let’s see what happens in the case of our dummy data (**Table 5**):

[![Image 33](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%200%200'%3E%3C/svg%3E)](https://lh6.googleusercontent.com/zQFNBpyRT_b1N3MRf4SCv6a2dCPQRiGFnprSLJyuTYXiS3CnYxOc3RyOiGmxDvQkk1YkfoOaTEx8T8xlNuy784rHOS-RitjlrpTvfPRQ7R5FWOUb2-Cbc2ZwBYoYlFYPIglB9JFHe26s2BnBfM1IeKmJMJiUSzJ3i80e9EQGKSCq74Z1x8cA3FcNRCznRQ)

**Table 5:** Final Calculations.

So for the case of the data entry “Tom,” if we try to predict using our created trees, we get:

*   ![Image 34: 80.75](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%2037%2013'%3E%3C/svg%3E) from our baseline classifier
*   ![Image 35: 0.01 \times 9.91 =  0.0991](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%20144%2013'%3E%3C/svg%3E) from the second tree

Adding these two values, we get ![Image 36: 80.84](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%2037%2013'%3E%3C/svg%3E) as our prediction. Notice that it is still less than the actual label ![Image 37: 81](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%2015%2013'%3E%3C/svg%3E), but better than our initial baseline prediction. Hence, with our first tree, we have indeed made progress toward the actual labels. We keep repeating this process and calculating residuals to slowly inch closer to the actual labels.

This concludes the working of gradient boosting using regression. Next, we will use Gradient Boosting on an actual dataset and analyze the results.

* * *

### [**Configuring Your Development Environment**](#TOC)

To follow this guide, you need to have the OpenCV library installed on your system.

Luckily, OpenCV is pip-installable:

$ pip install opencv-contrib-python

**If you need help configuring your development environment for OpenCV, we _highly recommend_ that you read our** [**_pip install OpenCV_ guide**](https://pyimagesearch.com/2018/09/19/pip-install-opencv/) — it will have you up and running in a matter of minutes.

* * *

### [**Having Problems Configuring Your Development Environment?**](#TOC)

[![Image 38](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%20500%20334'%3E%3C/svg%3E)](https://pyimagesearch.com/pyimagesearch-university/)

**Figure 3:** Having trouble configuring your dev environment? Want access to pre-configured Jupyter Notebooks running on Google Colab? Be sure to join [PyImageSearch University](https://pyimagesearch.com/pyimagesearch-university/) — you’ll be up and running with this tutorial in a matter of minutes.

All that said, are you:

*   Short on time?
*   Learning on your employer’s administratively locked system?
*   Wanting to skip the hassle of fighting with the command line, package managers, and virtual environments?
*   **Ready to run the code** **_right now_** **on your Windows, macOS, or Linux system?**

Then join [PyImageSearch University](https://pyimagesearch.com/pyimagesearch-university/) today!

**Gain access to Jupyter Notebooks for this tutorial and other PyImageSearch guides that are** **_pre-configured_** **to run on Google Colab’s ecosystem right in your web browser!** No installation required.

And best of all, these Jupyter Notebooks will run on Windows, macOS, and Linux!

* * *

### [**Setting Up Our Project**](#TOC)

Like XGBoost, it is easy to plug in the Gradient Boost algorithm using the sci-kit library.

\# import necessary packages
import pandas as pd
import xgboost as xgb
from sklearn.datasets import load\_iris
from sklearn.model\_selection import train\_test\_split
from sklearn.metrics import mean\_squared\_error 
from sklearn import ensemble

On **Lines 2-7**, we import the necessary packages, including the Iris dataset, XGBoost, and sci-kit ensemble package.

\# load the iris dataset and create the dataframe
iris = load\_iris()
X = pd.DataFrame(iris.data, columns=iris.feature\_names)
y = pd.Series(iris.target) 
 
# split the dataset 
xTrain, xTest, yTrain, yTest = train\_test\_split(X, y)

Next, we do the preliminary data setup. On **Line 10**, we loaded the `iris` dataset and stored it inside a dataframe (**Lines 10 and 11**).

We split the dataset into training and test splits (**Line 15**).

Next, we will compare the scores of XGBoost and Gradient Boosting.

* * *

### [**Comparing XGboost and Gradient Boost Results**](#TOC)

We will initialize the two algorithms with similar parameters to make the testing conditions as equal as possible.

\# initialize the gradient boosting algorithm
gradientBooster = ensemble.GradientBoostingRegressor(n\_estimators = 50,
    max\_depth =  8,
    learning\_rate = 1,
    criterion = 'squared\_error'
)

On **Lines 18-22**, we have initialized the Gradient Boost algorithm with `50` trees, a maximum tree depth of `8`, a learning rate of `1`, and a simple squared error as our loss function.

\# fit the training data 
gradientBooster.fit(xTrain, yTrain)
 
# check the score of the algorithm on test data
gradientBooster.score(xTest, yTest)

Next, we simply fit the training data and score the trained algorithm on the test set (**Lines 25-28**). We shall analyze the scores later.

\# define the XGBoost regressor according to your specifications
xgbModel = xgb.XGBRegressor(
    n\_estimators=50,
    reg\_lambda=2,
    gamma=0,
    max\_depth=8
)

The XGBoost regressor object is initialized on **Lines 31-36**, with `50` trees, a regularization value of `2`, and a max tree depth of `8`. There are several other parameters we can tweak, but for now, we are choosing to keep it simple.

\# fit the training data in the model
xgbModel.fit(xTrain, yTrain)
 
# check the score of the algorithm on test data
xgbModel.score(xTest, yTest)

As we had done for the Gradient Boost object, we fit the training data on the algorithm and scored it on the test set (**Lines 39-42**).

The Gradient Boost algorithm obtained a testing accuracy of `96.4`, while the XGBoost algorithm obtained a `97.09`.

* * *

### What's next? We recommend [PyImageSearch University](https://pyimagesearch.com/pyimagesearch-university/?utm_source=blogPost&utm_medium=bottomBanner&utm_campaign=What%27s%20next%3F%20I%20recommend).

**Course information:**  
84 total classes • 114+ hours of on-demand code walkthrough videos • Last updated: February 2024  
★★★★★ 4.84 (128 Ratings) • 16,000+ Students Enrolled

**I strongly believe that if you had the right teacher you could _master_ computer vision and deep learning.**

Do you think learning computer vision and deep learning has to be time-consuming, overwhelming, and complicated? Or has to involve complex mathematics and equations? Or requires a degree in computer science?

That’s _not_ the case.

All you need to master computer vision and deep learning is for someone to explain things to you in _simple, intuitive_ terms. _And that’s exactly what I do_. My mission is to change education and how complex Artificial Intelligence topics are taught.

If you're serious about learning computer vision, your next stop should be PyImageSearch University, the most comprehensive computer vision, deep learning, and OpenCV course online today. Here you’ll learn how to _successfully_ and _confidently_ apply computer vision to your work, research, and projects. Join me in computer vision mastery.

**Inside PyImageSearch University you'll find:**

*   ✓ **84 courses** on essential computer vision, deep learning, and OpenCV topics
*   ✓ **84 Certificates** of Completion
*   ✓ **114+ hours** of on-demand video
*   ✓ **Brand new courses released _regularly_**, ensuring you can keep up with state-of-the-art techniques
*   ✓ **Pre-configured Jupyter Notebooks in Google Colab**
*   ✓ Run all code examples in your web browser — works on Windows, macOS, and Linux (no dev environment configuration required!)
*   ✓ Access to **centralized code repos for _all_ 536+ tutorials** on PyImageSearch
*   ✓ **Easy one-click downloads** for code, datasets, pre-trained models, etc.
*   ✓ **Access** on mobile, laptop, desktop, etc.

[Click here to join PyImageSearch University](https://pyimagesearch.com/pyimagesearch-university/?utm_source=blogPost&utm_medium=bottomBanner&utm_campaign=What%27s%20next%3F%20I%20recommend)

* * *

[**Summary**](#TOC)
-------------------

Although XGBoost had better accuracy, Gradient Boost came really close to beating it. As XGBoost is built on Gradient Boost principles, it is important to understand how it works. The simple idea is to utilize ensemble learnings and pseudo-linear residuals to improve the prediction each time.

The nature of this algorithm is such that each tree will, in most cases, improve upon the previous result, be it by a tiny step. With the concept of Gradient Boost under our grasp, we are finished with the precursors for XGBoost, which will be our final destination in the next part of this series.

* * *

### [**Citation Information**](#TOC)

**Martinez, H.** “Scaling Kaggle Competitions Using XGBoost: Part 3,**”** _PyImageSearch_, P. Chugh, A. R. Gosthipaty, S. Huot, K. Kidriavsteva, R. Raha, and A. Thanki, eds., 2023, [https://pyimg.co/8he0y](https://pyimg.co/8he0y)

@incollection{Martinez\_2023\_XGBoost3,
  author = {Hector Martinez},
  title = {Scaling {Kaggle} Competitions Using {XGBoost}: Part 3},
  booktitle = {PyImageSearch},
  editor = {Puneet Chugh and Aritra Roy Gosthipaty and Susan Huot and Kseniia Kidriavsteva and Ritwik Raha and Abhishek Thanki},
  year = {2023},
  note = {https://pyimg.co/8he0y},
}

* * *

![Image 39: Featured Image](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%20500%20276'%3E%3C/svg%3E)

### Unleash the potential of computer vision with Roboflow - Free!

*   Step into the realm of the future by [signing up or logging into your Roboflow account](https://roboflow.com/?ref=pyimagesearch). Unlock a wealth of innovative dataset libraries and revolutionize your computer vision operations.
*   Jumpstart your journey by choosing from our broad array of datasets, or benefit from PyimageSearch’s comprehensive library, crafted to cater to a wide range of requirements.
*   Transfer your data to Roboflow in any of the 40+ compatible formats. Leverage cutting-edge model architectures for training, and deploy seamlessly across diverse platforms, including API, NVIDIA, browser, iOS, and beyond. Integrate our platform effortlessly with your applications or your favorite third-party tools.
*   Equip yourself with the ability to train a potent computer vision model in a mere afternoon. With a few images, you can import data from any source via API, annotate images using our superior cloud-hosted tool, kickstart model training with a single click, and deploy the model via a hosted API endpoint. Tailor your process by opting for a code-centric approach, leveraging our intuitive, cloud-based UI, or combining both to fit your unique needs.
*   Embark on your journey today with absolutely no credit card required. Step into the future with Roboflow.

[Join Roboflow Now](https://roboflow.com/?ref=pyimagesearch)

* * *

**To download the source code to this post (and be notified when future tutorials are published here on PyImageSearch), _simply enter your email address in the form below!_**

![Image 40](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%20410%20567'%3E%3C/svg%3E)

#### Download the Source Code and FREE 17-page Resource Guide

Enter your email address below to get a .zip of the code and a **FREE 17-page Resource Guide on Computer Vision, OpenCV, and Deep Learning.** Inside you'll find my hand-picked tutorials, books, courses, and libraries to help you master CV and DL!