Title: Scaling Kaggle Competitions Using XGBoost: Part 4 - PyImageSearch

URL Source: https://pyimagesearch.com/2023/01/23/scaling-kaggle-competitions-using-xgboost-part-4/

Published Time: 2023-01-23T14:00:00+00:00

Markdown Content:
* * *

**Table of Contents**
---------------------

*   [Scaling Kaggle Competitions Using XGBoost: Part 4](#h2BPTitle)

*   [What Is XGBoost?](#h3What)
*   [Our Dummy Dataset](#h3Dummy)
*   [Breaking Down the Math](#h3Breaking)
*   [Configuring Your Development Environment](#h3Configuring)
*   [Having Problems Configuring Your Development Environment?](#h3Problems)
*   [Applying XGBoost on a Problem Statement](#h3Statement)
*   [Applying XGBoost to Our Dataset](#h3Dataset)

*   [Summary](#h2Summary)

*   [Citation Information](#h3Citation)

* * *

Over the last few blog posts of this series, we have been steadily building up toward our grand finish: deciphering the mystery behind eXtreme Gradient Boosting (XGBoost) itself. If you have stuck with us through the entire journey till now (Great work!), you already know that our approach in this series is math-heavy instead of code-heavy.

[![Image 1](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%20700%20393'%3E%3C/svg%3E)](https://pyimagesearch.com/wp-content/uploads/2023/01/xgboost-part4-feature.png)

The reasoning behind that is simple; whatever we have learned till now, be it adaptive boosting, decision trees, or gradient boosting, have very distinct statistical foundations which require you to get your hands dirty with the math behind them. The goal is to nullify the abstraction created by packages as much as possible.

Our final stop of this series will have a similar outlook.

In this tutorial, you will learn the magic behind the critically acclaimed algorithm: XGBoost.

This lesson is the last of a 4-part series on **Deep Learning 108**:

1.  [_Scaling Kaggle Competitions Using XGBoost: Part 1_](https://pyimg.co/0c9pb)
2.  [_Scaling Kaggle Competitions Using XGBoost: Part 2_](https://pyimg.co/2wiy7)
3.  [_Scaling Kaggle Competitions Using XGBoost: Part 3_](https://pyimg.co/8he0y)
4.  [**_Scaling Kaggle Competitions Using XGBoost: Part 4_**](https://pyimg.co/onx24) **(this tutorial)**

**To learn how to conquer the ingenious concept of XGBoost,** **_just keep reading._**

![Image 2](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%20128%20128'%3E%3C/svg%3E)

#### Looking for the source code to this post?

[Jump Right To The Downloads Section](#download-the-code)

* * *

[**Scaling Kaggle Competitions Using XGBoost: Part 4**](#TOC)
-------------------------------------------------------------

If you went through our previous blog post on Gradient Boosting, it should be fairly easy for you to grasp XGBoost, as XGBoost is heavily based on the original Gradient Boosting algorithm. We strongly recommend having a strong grip on Parts [1](https://pyimg.co/0c9pb) and [3](https://pyimg.co/8he0y) of the series and an overall gist of [Part 2](https://pyimg.co/2wiy7).

* * *

### [**What Is XGBoost?**](#TOC)

Although we have already addressed this in the [first blog post](https://pyimg.co/0c9pb) of this series, let’s go over it again.

XGBoost stands for eXtreme Gradient Boosting, an optimized solution for training in gradient boosting. As argued by many, one of the most powerful classical machine learning algorithms out there today incorporates several features, some of which are:

*   Has steps to prevent overfitting (regularization)
*   Can work in parallel processing
*   Has in-built cross-validation and early stopping
*   Results in highly optimized results (deeper trees)

Till now, we have broken down ensemble learning, adaptive boosting, and gradient boosting. But all of these algorithms, despite having a strong mathematical foundation, have some flaws or the other. For example, in the case of adaptive boosting, we see that in most cases, the trees are simple stumps (a single root node with its two children) or the fact that its idea is based on focusing on samples that are wrongly classified.

In the case of gradient boosting, we do see an approach better than the ones we have considered till now, with a solid ensemble progression as well as methods to prevent overfitting. But how to make it even better? That’s where the “Extremeness” of XGBoost comes in. Without further ado, let’s start with our journey.

* * *

### [**Our Dummy Dataset**](#TOC)

In practical cases, XGBoost is normally applied to huge datasets. For simplicity, we will use a small dummy dataset to understand how regression works in XGBoost (**Figure 1**).

[![Image 3](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%200%200'%3E%3C/svg%3E)](https://lh4.googleusercontent.com/9qF_93usXiJzpEuT_VwybUycBY2XkEZmjLEnsoLfVa2kJru_rhFJMroP1ZoJdWvu2f7RnJwoMh8NoSNzYf6gU8TCC9AJ-i5VtZ4oYT2DVhBx5E-HFoGZ5jiwQUHtYNCGhSkDUVkHWShSG6kVqGFxKiOX3Il2X7juGCchHWEe8VyakCYC6oFBBRNgXJIm1g)

**Figure 1:** Our Dummy Dataset.

In this dataset, we have the feature as `Years in the industry` of individuals, with the label being their income. The general notion we will have about this dataset is that more years in the industry will grant you more income. Which means the purple point is our outlier.

* * *

### [**Breaking Down the Math**](#TOC)

As we had done for Gradient Boosting, let us glance over the math that is behind XGBoost (**Figure 2**)

[![Image 4](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%20700%20451'%3E%3C/svg%3E)](https://lh4.googleusercontent.com/2qhLdHi4QAUFflvpycmzBBRVk-SFlqRGRdH8lD4Guzq0mUXPb84T0xEmKIJ0AHMkASi_RAv85rsd8lrz10RO7C8CZSmPxdDnU52cZLVwZobFq3wr2a9sUBM3-Qw9gEI_KZ4XscQ62LgrRfCqY8r3v7AP6Gi5JOcnzKmyIXgX9s_uMHAza5V-m0mlbM9Nfw)

**Figure 2:** General Equations for XGBoost ([source](https://en.wikipedia.org/wiki/XGBoost)).

Again, don’t be scared of these scary-looking equations, as they are as simple as they can be. Before we begin, just a few points. If you have never come across the term ![Image 5: h_m](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%2020%2014'%3E%3C/svg%3E) in **Step 2**, those are nothing but Hessians, which we get by taking the derivative of the gradients.

Firstly, we have the definition of the training set, which is ![Image 6: (x_i, y_i)_{i=1}^N](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%2063%2019'%3E%3C/svg%3E) refers to the training sample ![Image 7: i](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%205%2012'%3E%3C/svg%3E), which has ![Image 8: x](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%209%207'%3E%3C/svg%3E) features and ![Image 9: y](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%208%2010'%3E%3C/svg%3E) labels.

The loss function again is defined as ![Image 10: L(y, F(x))](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%2072%2018'%3E%3C/svg%3E), which is nothing but the loss between labels and prediction ![Image 11: f(x)](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%2030%2018'%3E%3C/svg%3E).

Our first step is to create the base model, which we have assumed to be of the value `25`. By default, in practice, it is always set to `0.5`. However, to better understand the process, we have chosen a value of our own (**Figure 3**).

[![Image 12](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%200%200'%3E%3C/svg%3E)](https://lh3.googleusercontent.com/aGH2x6MJM46alqgdIes0xZwkZ8JRJOyTpuUVAw3fx2028qHXFGGpImJNjJ6Sr8YiMIJKhih48CLFUYXYku6dcBrumOgE-BiqtaYflxcJl0nnezboBGxhjW5Zxq-0Bh9jv2S4_iaqkVW0b4WzclUXWFgIBQw9nB_cOS91ZJaz8pzA6BP2TpH1On5UT-5UsQ)

**Figure 3:** The Baseline Split.

The next step is calculating the residuals `(-15, 10, -10, 15)`. Up until now, you will find a lot of similarities with Gradient Boosting.

The difference starts when the first unique XGBoost tree is built. Assume our first node splits the residuals based on the feature being greater than/not greater than `1.5` “Years in the industry” (**Figure 4**).

[![Image 13](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%20700%20269'%3E%3C/svg%3E)](https://lh6.googleusercontent.com/n7Nc70KSlFrnRePtMlf1DTcucHV3OQA2Cff5sTDt9XTZvZxktqJCk7xxLkFpqJ2mAyqtgvCt_IcLU4PCeUZKWYtAl44RcItM98osaHomT_Kc472ZvIoO1qbVH7Wobgs4uL4Vbb8gdgX5Jo57yoUqNvBN-k_QBky9vFpJGxOouxI76GybZEZoUV3ls0YKYg)

**Figure 4:** The First Tree.

Note that trees are normally deeper and more complex, but we have taken this example for easier understanding. Before we progress further, let’s calculate something called the similarity score: ![Image 14: (\text{Sum of residuals})^2/(\text{number of residuals} + \lambda)](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%20323%2018'%3E%3C/svg%3E). Here, ![Image 15: \lambda](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%209%2011'%3E%3C/svg%3E) is the regularizer.

But before that, let us understand how we got there. In **Step 2** of **Figure 2**, we see a calculation of gradients and Hessians. As mentioned in the earlier blog, the loss function normally used here is ![Image 16: (1/2) (y_i - p_i)^2](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%20100%2018'%3E%3C/svg%3E). Taking the first-order derivative (gradient) gives us nothing but ![Image 17: -(y_i - p_i)](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%2069%2018'%3E%3C/svg%3E), which is the residual.

The Hessian is the second-order derivative, giving us the value `1` here. That makes the Output of a node which is represented by ![Image 18: -(\text{sum of gradients})/(\text{sum of Hessians} + \lambda)](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%20304%2018'%3E%3C/svg%3E) as ![Image 19: \text{Residuals}/(\text{number of residuals} + \lambda)](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%20255%2018'%3E%3C/svg%3E) (such insane simplification, right?).

In the second part of **Step 2**, a few equations need to be followed for the tree to be created. Here, ![Image 20: -\text{gradients}/\text{hessians}](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%20141%2017'%3E%3C/svg%3E) value is the Output value without the ![Image 21: \lambda](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%209%2011'%3E%3C/svg%3E). Solving for the optimization problem gives us nothing but ![Image 22: (\text{Sum of residuals})^2/(\text{number of residuals} + \lambda)](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%20323%2018'%3E%3C/svg%3E).

Let us assume that ![Image 23: \lambda](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%209%2011'%3E%3C/svg%3E) is `0`, so the score for the root node (purple) becomes `0`.

The score for the red node becomes `225`.

The score for the green node becomes `75`.

Next, we calculate something known as the ![Image 24: \text{Gain}](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%2035%2013'%3E%3C/svg%3E) value for this tree, which is ![Image 25: \text{Gain} = \text{left}_\text{similarity} + \text{right}_\text{similarity} - \text{root}_\text{similarity}](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%20329%2019'%3E%3C/svg%3E), which for this tree becomes `300`.

Let’s try another tree now, where the root node has a different split condition (**Figure 5**)

[![Image 26](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%20700%20284'%3E%3C/svg%3E)](https://lh4.googleusercontent.com/trqz6qlXED_bvlV9B7NF_2AdacVXzVbZ6B-JbgHiz08oZc1Y4dc_QbxhQzConDtW5AFvJ5saRLAgGY5z29sn-GftQK_jvizX0xErg4aHNKvJn3WSKj6Ua_Xjp6Zd8qfo43PJkG-d1axWkSZtHk6H9TffzFhM2kaHi2SRBW8bXQkAyZV4Ubt4la_RgdScwQ)

**Figure 5:** A New Tree.

Now, the score for the red node becomes `12.5`.

The score for the green node becomes `12.5` again. This brings the ![Image 27: \text{Gain}](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%2035%2013'%3E%3C/svg%3E) for the tree as `24`.

This way, we can actually estimate which tree has a better gain and hence will function better, and this way, we can estimate which tree branch to consider for our final structure. But the green branch node in **Figure 4** can be split further. Let’s repeat the process for it! (**Figure 6**).

[![Image 28](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%20700%20338'%3E%3C/svg%3E)](https://lh5.googleusercontent.com/hU7F_INtuTZT447_vJ-2C2H0ro613n9s2U4FL5cMLbmZhe3d9BAPk6b5psIhbDNIOt5cz7G9M6eO6G_-B9cPp_mjfuoYGdFn4zbhvk3to6_y_g9i0GqrOxr_4HnInI670KWjhTHJ4u2nqAkgrfq8J8lTLXh4RKgF1wGrHTOcq7z0YIGgFOmSa3zcC2XAbw)

**Figure 6:** Going Further.

For this particular split condition, the ![Image 29: \text{Gain}](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%2035%2013'%3E%3C/svg%3E) of the right subtree becomes ![Image 30: 100 + 12.5 - 75](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%20109%2015'%3E%3C/svg%3E), which comes to `37.5`.

Let’s try another split condition (**Figure 7**).

[![Image 31](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%20700%20330'%3E%3C/svg%3E)](https://lh5.googleusercontent.com/M5YcPBOmbM96NesoStUnO96a2V22WcSbyTgJwOzh_jqUSsYZO0STTdaGKdg2zughAY6HfdxSKUZP9tmCd5YuRjPAS5qYrfDqyz-ZtAd526Doz4JNLY1l5cPK20slZwSYrIq4tEf9ZR1GW_P7g2JniVz9dI9MSiwmwU6gRgo4N8FADsOP9VyqF3_dBbhcng)

**Figure 7:** New Feature Split.

For this particular split condition, the ![Image 32: \text{Gain}](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%2035%2013'%3E%3C/svg%3E) of the right subtree becomes ![Image 33: 225-75](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%2061%2013'%3E%3C/svg%3E), which equals `150`. This better ![Image 34: \text{Gain}](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%2035%2013'%3E%3C/svg%3E) enables us to go for this split condition instead.

But we are getting quite ahead of ourselves. We have the first tree, we have mapped the labels according to the split conditions, and in some nodes, we have reached singular values!

Unfortunately, we are missing a key concept. If you have followed the previous tutorial, recap that we didn’t want a single tree to overfit on our data points. Here, to calculate the output, we simply use the formula ![Image 35: \sum \text{residuals}/(\text{number of residuals} + \lambda)](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%20270%2019'%3E%3C/svg%3E), which we have spoken about earlier.

For our dummy scenario, the ![Image 36: \lambda](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%209%2011'%3E%3C/svg%3E) value is set to `0`. So for “Years in the industry,” equals `1` would give us the output value of ![Image 37: -15/1](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%2043%2017'%3E%3C/svg%3E), equaling ![Image 38: -15](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%2028%2014'%3E%3C/svg%3E). But this just means that the tree is severely overfitting on single data points.

This is where ![Image 39: \lambda](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%209%2011'%3E%3C/svg%3E) comes in. If we set any value for ![Image 40: \lambda](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%209%2011'%3E%3C/svg%3E), the Similarity scores also fall (due to ![Image 41: \lambda](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%209%2011'%3E%3C/svg%3E)) being in the denominator, and the output value is also less. For the example above, the output value becomes ![Image 42: -7.5](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%2033%2013'%3E%3C/svg%3E).

Couple this with a suitable value for the `learning_rate` (`0.3`), and we get, for “Years in the industry” equals to `1`, ![Image 43: 25 + 0.3 \times (-7.5)](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%20122%2018'%3E%3C/svg%3E), which is `22.75`. This is an improvement over the baseline prediction and a bit closer to the actual label of `10`! This way, the process is repeated, and we get closer and closer to our desired area of the label.

There’s another thing left to check out, which is called pruning. Pruning is taken in its literal sense of removing branches based on our requirements. So let’s jump back to when we had the initial ![Image 44: \text{Gain}](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%2035%2013'%3E%3C/svg%3E) values calculated for our tree.

The right subtree had a ![Image 45: \text{Gain}](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%2035%2013'%3E%3C/svg%3E) value of `150` (**Figure 7**). We compare it to a value ![Image 46: \gamma](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%209%2010'%3E%3C/svg%3E), and if the result is negative, we remove this subtree and move on to the root. If the root gain comparison also is negative, we remove the tree altogether.

The output calculation is similar to Gradient Boosting. We use our base predictor (`25` in our case) and then add the outputs we receive from each tree (coupled with the learning rate) to get our final result.

That concludes the math behind an XGBoost Regressor. There are many similarities with Gradient Boosting, but the way the trees are created is very different.

* * *

### [**Configuring Your Development Environment**](#TOC)

To follow this guide, you need to have the OpenCV library installed on your system.

Luckily, OpenCV is pip-installable:

$ pip install opencv-contrib-python

**If you need help configuring your development environment for OpenCV, we _highly recommend_ that you read our** [**_pip install OpenCV_ guide**](https://pyimagesearch.com/2018/09/19/pip-install-opencv/) — it will have you up and running in a matter of minutes.

* * *

### [**Having Problems Configuring Your Development Environment?**](#TOC)

[![Image 47](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%20500%20334'%3E%3C/svg%3E)](https://pyimagesearch.com/pyimagesearch-university/)

**Figure 8:** Having trouble configuring your dev environment? Want access to pre-configured Jupyter Notebooks running on Google Colab? Be sure to join [PyImageSearch University](https://pyimagesearch.com/pyimagesearch-university/) — you’ll be up and running with this tutorial in a matter of minutes.

All that said, are you:

*   Short on time?
*   Learning on your employer’s administratively locked system?
*   Wanting to skip the hassle of fighting with the command line, package managers, and virtual environments?
*   **Ready to run the code** **_right now_** **on your Windows, macOS, or Linux system?**

Then join [PyImageSearch University](https://pyimagesearch.com/pyimagesearch-university/) today!

**Gain access to Jupyter Notebooks for this tutorial and other PyImageSearch guides that are** **_pre-configured_** **to run on Google Colab’s ecosystem right in your web browser!** No installation required.

And best of all, these Jupyter Notebooks will run on Windows, macOS, and Linux!

* * *

### [**Applying XGBoost on a Problem Statement**](#TOC)

Now that we have finished deciphering XGBoost regression, let’s try it out on a problem statement! We will be using the [Wine Quality](https://www.kaggle.com/code/sergey18650/wine-quality-randomforest-acc-82/data) dataset from Kaggle and seeing how XGBoost performs on a dataset filled with many features from which to choose!

First, let us download the dataset from Kaggle into our local Colab session.

\# initialize colab file operations
from google.colab import files
files.upload() 
 
# make sure kaggle.json file is present
!ls -lha kaggle.json
 
# install the kaggle package
!pip install -q kaggle
!mkdir -p /.kaggle
!cp kaggle.json ~/.kaggle/
 
# set permissions for the kaggle.json
!chmod 600 ~/.kaggle/kaggle.json
 
# download the required dataset from kaggle
!kaggle datasets download -d yasserh/wine-quality-dataset
!unzip -qq wine-quality-dataset.zip

On **Line 2**, we will utilize the colab file operation and upload our own unique `kaggle.json` file in the session (**Line 3**). On **Line 6**, we have just kept an additional sanity check for the file.

Next, we install the `kaggle` package and set permissions for the session to use the file (**Lines 9-14**). Our final step is downloading the exact dataset we require (**Line 17 and 18**).

\# import necessary packages
import numpy as np 
import pandas as pd 
import seaborn as sns
import xgboost as xgb
from sklearn import ensemble
import matplotlib.pyplot as plt
from imblearn.over\_sampling import SMOTE
from sklearn.metrics import mean\_squared\_error
from sklearn.model\_selection import train\_test\_split
%matplotlib inline

Next, we import the necessary packages according to our needs. We have used packages like XGBoost, pandas, numpy, matplotlib, and a few packages from `scikit-learn`. For better plots, we have used the matplotlib magic function `inline` (**Line 30**).

* * *

### [**Applying XGBoost to Our Dataset**](#TOC)

Next, we will do some exploratory data analysis and prepare the data for feeding the model.

\# read the csv and check for unique values in the labels
wineDf = pd.read\_csv('/content/wine-quality-dataset/WineQT.csv')
wineDf\["quality"\].unique()
 
# check the label distribution 
lblDist = sns.countplot(x='quality', data=wineDf)

On **Lines 33 and 34**, we read the `csv` file and then display the unique labels we are dealing with. For good measure, let’s check the distribution of labels in this particular dataset (**Figure 9**).

[![Image 48](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%200%200'%3E%3C/svg%3E)](https://lh4.googleusercontent.com/k-rgrTbkl0JhW0Ye9aRae_KJJgy4clyDPaoXnYlWYYlAKa0xFRVdCFRnVJ8JgITTLbM2HencsxGDVCda1pQHMTJUS3vGkqXdKcmibNgHTHU6X4PRRGTmwwvD7cLhVThgz6A8US0qkIp4GlL6F0XrLLl3I8mQ4P4bVEhaIxyNKY1ZJNnKZGLjGXOGVmeWZQ)

**Figure 9:** The Distribution of Labels.

**Figure 9** shows that there are very few samples with labels `3`, `4`, and `8`. Will this be a problem for our XGBoost regressor? Let’s find out.

\# prepare data for model feeding
X, y = wineDf.iloc\[:, :11\].values, wineDf.iloc\[:, 11\]
xTrain, xTest, yTrain, yTest = train\_test\_split(X, y, test\_size=0.20)
 
# define the XGBoost regressor according to your specifications
xgbModel = xgb.XGBRegressor(
    n\_estimators=500,
    gamma=0,
    max\_depth=5
)

Next, we split the features and the labels on **Line 40** and used `scikit-learn` to split the dataset into training and testing sets (**Line 41**).

The XGBoost regressor is defined on **Lines 44-48**. We use `500` trees, with a ![Image 49: \gamma](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%209%2010'%3E%3C/svg%3E) value of `0` and a maximum depth of each tree of `5`. You are free to use different values and check how it affects the results.

\# fit the training data in the model
xgbModel.fit(xTrain, yTrain)
 
# check the score of the algorithm on test data
xgbModel.score(xTest, yTest)

On **Lines 51-54**, we fit the training data into our regressor and then score it on the test data. Unfortunately, the test data score has come only to 41% accuracy.

This goes against all the claims and praises we have heard about XGBoost, but don’t worry. Our regressor has no fault for this poor score. The distribution of data is such that XGBoost experiences extremely low samples of certain classes, due to which it fails to learn about them properly. The low number can also sometimes make it such that the regressor will face only 2-4 samples of a particular class in the training set.

To fix this, we will introduce intentional oversampling of our data to make the labels have a similar number of samples and not pose any problems.

\# prepare resampled data for model feeding
X, y = SMOTE().fit\_resample(X, y)
xTrain, xTest, yTrain, yTest = train\_test\_split(X, y, test\_size=0.20)
 
# fit the training data in the model
xgbModel.fit(xTrain, yTrain)
 
# check the score of the algorithm on test data
xgbModel.score(xTest, yTest)

To introduce oversampling, we will use the `imblearn` package. On **Line 57**, we prepare our dataset again for model feeding, but this time without the problem of poor distribution.

The train-test split is created again, and the training data is fed to the model (**Lines 58-61**). Upon scoring the predictions on the test data (**Line 64**), we see that we have now achieved 92% accuracy! Our task is now complete.

* * *

### What's next? We recommend [PyImageSearch University](https://pyimagesearch.com/pyimagesearch-university/?utm_source=blogPost&utm_medium=bottomBanner&utm_campaign=What%27s%20next%3F%20I%20recommend).

**Course information:**  
84 total classes • 114+ hours of on-demand code walkthrough videos • Last updated: February 2024  
★★★★★ 4.84 (128 Ratings) • 16,000+ Students Enrolled

**I strongly believe that if you had the right teacher you could _master_ computer vision and deep learning.**

Do you think learning computer vision and deep learning has to be time-consuming, overwhelming, and complicated? Or has to involve complex mathematics and equations? Or requires a degree in computer science?

That’s _not_ the case.

All you need to master computer vision and deep learning is for someone to explain things to you in _simple, intuitive_ terms. _And that’s exactly what I do_. My mission is to change education and how complex Artificial Intelligence topics are taught.

If you're serious about learning computer vision, your next stop should be PyImageSearch University, the most comprehensive computer vision, deep learning, and OpenCV course online today. Here you’ll learn how to _successfully_ and _confidently_ apply computer vision to your work, research, and projects. Join me in computer vision mastery.

**Inside PyImageSearch University you'll find:**

*   ✓ **84 courses** on essential computer vision, deep learning, and OpenCV topics
*   ✓ **84 Certificates** of Completion
*   ✓ **114+ hours** of on-demand video
*   ✓ **Brand new courses released _regularly_**, ensuring you can keep up with state-of-the-art techniques
*   ✓ **Pre-configured Jupyter Notebooks in Google Colab**
*   ✓ Run all code examples in your web browser — works on Windows, macOS, and Linux (no dev environment configuration required!)
*   ✓ Access to **centralized code repos for _all_ 536+ tutorials** on PyImageSearch
*   ✓ **Easy one-click downloads** for code, datasets, pre-trained models, etc.
*   ✓ **Access** on mobile, laptop, desktop, etc.

[Click here to join PyImageSearch University](https://pyimagesearch.com/pyimagesearch-university/?utm_source=blogPost&utm_medium=bottomBanner&utm_campaign=What%27s%20next%3F%20I%20recommend)

* * *

[**Summary**](#TOC)
-------------------

Our journey of uncovering XGBoost and its prerequisites finally ends here. We took on the task of deciphering why it worked so well for such huge competitions and tried to uncover the simple bits of mathematics it has. Starting with defining simple decision trees, we moved on to uncover why adaptive boosting was an important milestone in our journey. Then we touched upon Gradient Boosting Regression trees and built our foundational knowledge about their mathematics. We utilized those concepts and turned them to the Extreme level, where we finally addressed eXtreme Gradient Boosting.

The math behind these concepts was devastatingly simple, hiding behind a canopy of difficult-looking mathematical algorithms. Once we addressed them one by one, all of their facades fell like dominos. While they are extremely interesting to decipher, each concept has its own uniqueness that makes them stand apart.

We hope you have enjoyed this journey, but above all, be sure to try it out for yourself!

* * *

### [**Citation Information**](#TOC)

**Martinez, H.** “Scaling Kaggle Competitions Using XGBoost: Part 4,” _PyImageSearch_, P. Chugh, A. R. Gosthipaty, S. Huot, K. Kidriavsteva, R. Raha, and A. Thanki, eds., 2023, [https://pyimg.co/onx24](https://pyimg.co/onx24)

@incollection{Martinez\_2023\_XGBoost4,
  author = {Hector Martinez},
  title = {Scaling {Kaggle} Competitions Using {XGBoost}: Part 4},
  booktitle = {PyImageSearch},
  editor = {Puneet Chugh and Aritra Roy Gosthipaty and Susan Huot and Kseniia Kidriavsteva and Ritwik Raha and Abhishek Thanki},
  year = {2023},
  url = {https://pyimg.co/onx24},
}

* * *

![Image 50: Featured Image](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%20500%20276'%3E%3C/svg%3E)

### Unleash the potential of computer vision with Roboflow - Free!

*   Step into the realm of the future by [signing up or logging into your Roboflow account](https://roboflow.com/?ref=pyimagesearch). Unlock a wealth of innovative dataset libraries and revolutionize your computer vision operations.
*   Jumpstart your journey by choosing from our broad array of datasets, or benefit from PyimageSearch’s comprehensive library, crafted to cater to a wide range of requirements.
*   Transfer your data to Roboflow in any of the 40+ compatible formats. Leverage cutting-edge model architectures for training, and deploy seamlessly across diverse platforms, including API, NVIDIA, browser, iOS, and beyond. Integrate our platform effortlessly with your applications or your favorite third-party tools.
*   Equip yourself with the ability to train a potent computer vision model in a mere afternoon. With a few images, you can import data from any source via API, annotate images using our superior cloud-hosted tool, kickstart model training with a single click, and deploy the model via a hosted API endpoint. Tailor your process by opting for a code-centric approach, leveraging our intuitive, cloud-based UI, or combining both to fit your unique needs.
*   Embark on your journey today with absolutely no credit card required. Step into the future with Roboflow.

[Join Roboflow Now](https://roboflow.com/?ref=pyimagesearch)

* * *

**To download the source code to this post (and be notified when future tutorials are published here on PyImageSearch), _simply enter your email address in the form below!_**

![Image 51](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%20410%20567'%3E%3C/svg%3E)

#### Download the Source Code and FREE 17-page Resource Guide

Enter your email address below to get a .zip of the code and a **FREE 17-page Resource Guide on Computer Vision, OpenCV, and Deep Learning.** Inside you'll find my hand-picked tutorials, books, courses, and libraries to help you master CV and DL!