Title: Scaling Kaggle Competitions Using XGBoost: Part 1 - PyImageSearch

URL Source: https://pyimagesearch.com/2022/11/21/scaling-kaggle-competitions-using-xgboost-part-1/

Published Time: 2022-11-21T14:00:00+00:00

Markdown Content:
* * *

**Table of Contents**
---------------------

*   [Scaling Kaggle Competitions Using XGBoost: Part 1](#h2BPTitle)

*   [Preface](#h3Preface)
*   [XGBoost](#h3XGBoost)
*   [Configuring Your Development Environment](#h3Configuring)
*   [Having Problems Configuring Your Development Environment?](#h3Problems)
*   [Configuring the Prerequisites](#h3Prerequisites)
*   [Plugging in XGBoost](#h3Plugging)

*   [Summary](#h2Summary)

*   [Citation Information](#h3Citation)

* * *

Tackling deep learning topics in depth can be tough sometimes. You may try again and again, but much like Writer’s block, researcher’s block is very real too!

[![Image 1](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%20700%20393'%3E%3C/svg%3E)](https://pyimagesearch.com/wp-content/uploads/2022/11/XGBoost-1-featured.png)

In my initial days of traversing deep learning, I used to hit roadblocks on a semi-daily basis. Of course, we all say that unwavering dedication is the key to a breakthrough, but sometimes it’s good to take a different approach.

For me, a breath of fresh air was competing (casually) in Kaggle competitions. Since deep learning research and practical deep learning require vastly different thinking approaches, it was a good change for me as I gained a different perspective on problem-solving, as well as a look into how data scientists of the highest caliber attack real-life problems with deep learning.

Competitive “kaggling” is actually a very addictive habit where deep learners can easily fall. However, competitions that provide growth and several prizes are enough to entice people to dedicate themselves to being the best “kaggler” out there.

However, this only constitutes 50% of our blog post title. So what about the other half?

Kagglers are always looking for ways to make their competition submissions better than others. The caliber of today’s kagglers is such that almost everyone reaches 99% accuracy, but only the one with 99.1% accuracy reigns supreme.

XGBoost is a scalable gradient-boosting library that came to light when several winning Kaggle competition teams used it in their submissions. If you are unfamiliar with the term “gradient boosting”; don’t worry, we have you covered.

Today, we will dive into a preamble of approaching Kaggle competitions and see how XGBoost pushes your gradients to their maximum potential.

In this tutorial, you will learn **How to Scale Kaggle Competitions with XGBoost**.

This lesson is the 1st of a 4-part series on **Deep Learning 108**:

1.  [**_Scaling Kaggle Competitions U_**](https://pyimg.co/0c9pb)**_[s](https://pyimg.co/0c9pb)_**[**_ing XGBoost: Part 1_**](https://pyimg.co/0c9pb) **(this tutorial)**
2.  _Scaling Kaggle Competitions Using XGBoost: Part 2_
3.  _Scaling Kaggle Competitions Using XGBoost: Part 3_
4.  _Scaling Kaggle Competitions Using XGBoost: Part 4_

**To learn how to **scale Kaggle competitions using XGBoost**, _just keep reading._**

* * *

[**Scaling Kaggle Competitions Using XGBoost: Part 1**](#TOC)
-------------------------------------------------------------

* * *

### [**Preface**](#TOC)

To understand XGBoost, we first need to understand a few prerequisites. The first concept you need to be familiar with is a decision tree.

By this point, we are all familiar with linear regression. If not, just recount the equation of a line in your head, ![Image 2: y=mx+c](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%2081%2014'%3E%3C/svg%3E). Take a look at **Figure 1**.

[![Image 3](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%200%200'%3E%3C/svg%3E)](https://lh3.googleusercontent.com/JVd0hQMDKw2IOsdXVv14SneBQ3JVCKQvgMzYTctsE60ld-utIpAg5MvjjafrHNMDRhJveAkEDK6modT4cpthtX7TNWeR0XgNUKn_sL30fwetUv62h8OzCL4ERxrRDvT0NmmnXCoLamvWZId1c_FLbrRBbsnoMrDgmyQmYKJASGza8XVI0PlemBXCX8EaCg)

**Figure 1:** A Linear Setting.

Here, we are trying to model `Y` using `X`. We have a few data points which we have plotted. Since it’s an ideal setting, we have found a linear function that cuts through all the points.

Now, if we make things more complicated by moving into a proper classification task, we have to deal with several features corresponding to labels. Ideally, let’s say we have features `X1` and `X2` for a given dataset, which, when plotted, perfectly split the dataset into its corresponding labels (**Figure 2**).

[![Image 4](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%200%200'%3E%3C/svg%3E)](https://lh6.googleusercontent.com/yCaDzXyA2GNTFXDSoYoxhC0Y7PLrP5OntG2DKRx64iAcIJJUsjEs-OfWrbfX-3CdvGG2BvafNHq_VLTEKeTh-cHuMOzCY9lgfx7dgow8NrIMvi1uRtik2oWmSmPwZ4V6eQIGNojAxaaCp2p8-frp-ry4m_FxwzyJlBTFKnHyxFZgUFeSQpt-hYLawnIhBA)

**Figure 2:** `X1` and `X2` do split the data.

That would be every data scientist’s dream, as the results would come flowing in a jiffy. However, most real-life datasets will have extremely nonlinear data, where using linear functions to classify the dataset would be difficult (**Figure 3**).

[![Image 5](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%200%200'%3E%3C/svg%3E)](https://lh6.googleusercontent.com/ILLNWgZ8exsabK-yvOr7JMdcmMNKC3wIYHxSBOXD1idsXb4vsz4uzfCzGPV1gwkBLS6mwj8gNly1lQDV8l_DGttOHeNs9lqZw2GY7cwAIR2UqI4W-muCh2h0LAVJpCezjydJs1eR4578tSKPU-S0MmrAYiQ7IeWww1siGsVzfzaeWnDBcewQ69c9bZXoow)

**Figure 3:** A tough data split.

Thankfully, introducing nonlinearity in our regression functions for datasets like these is where we have become adept. As a result, decision trees are an extremely adept algorithm for tackling tricky datasets like these.

A decision tree can be considered a binary tree that recursively splits the dataset so that at least a leaf node contains a singular class. On a personal note, decision trees were the coolest and most deceptive algorithms I came across in classical machine learning.

My coolest, I mean its design, is such that even the most diverse datasets can be easily tamed using decision trees. But on the other hand, the practical usage of decision trees has led to my belief that it’s very easy to overfit on the training data when using decision trees, especially for complex datasets.

Let’s look at **Figure 4**, which displays a decision tree that splits the popular iris dataset into its corresponding classes.

[![Image 6](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%20535%20500'%3E%3C/svg%3E)](https://lh5.googleusercontent.com/ZqhLQZQzFglelIqIbpT3BrzXLS3Yw3k1lcZiQw7QLLZ8N0fimnyLZbd-JHakDPoM2cAGH_enL7-taT29MslbX5fKU99ax4cTshHTfYWwnbJDuUyjJPGqME818ROYejcEv2s1ZOUy3XOqOuGnPfrPUkR2rGmPxit60JfDPzEmI7qsxXgWL-CEfUjuDOYPhA)

**Figure 4:** A Decision Tree ([source](https://newbedev.com/is-it-possible-to-print-the-decision-tree-in-scikit-learn)).

Now in all likelihood, this picture might look super complex to you. So, let’s decipher what it means; Look at the root node. We don’t have enough information in the root node to separate each class. Hence we see a distribution.

The children of the root node are split based on the petal length condition. The data points satisfying less-than condition yield a pure node (i.e., a node referencing a singular class). In the right child, the data points satisfying the greater than or equal to condition fall, but it is still a distribution since there are two classes left.

So in the subsequent nodes, the distribution will further be split based on conditions created by the model, and as we move further down the tree, we see that leaf nodes have individual classes.

Let’s try to visualize a decision tree for our example dataset of **Figure 3**.

In **Figure 5**, we have tried creating the initial tree structure for a possible decision tree that would be formed for the given dataset.

[![Image 7](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%20700%20433'%3E%3C/svg%3E)](https://lh4.googleusercontent.com/qGFeiUrHArkEBMuQirx7JUwwUAftrTHx5hcwtiRqc45OIloX1s5CuJPQK86_2neHM7nX739hTMCLoEImYRWVaDXJeZ5nz1vsINKT7BYsjBwJOp56sAjeGfwYD2pPNyA9-tp1pjM7muAeyevG1zWcw0KL4ddsnW4OtP51itXJ65silMBGUyGoROeKxTv_tg)

**Figure 5:** An example of a decision tree.

Now, if you look closely, we have several conditions that are systematically splitting the data points. Although the general idea of decision trees can be cleared using these examples, intuitively, it should strike us that many conditions can drastically increase the chance of overfitting.

Let’s move on to the next important prerequisite that we require: Ensemble learning.

So decision trees provided an approach consisting of several decisions in which we could split the dataset into its corresponding classes. However, there were several cons to this approach, most notably that of overfitting.

But what if instead of using a single tree, we use multiple decision trees, which help each other, eliminating weak links and ensuring a robust cluster of trees that can also be used to prevent overfitting?

The answer lies in using multiple machine learning models to combine the insights obtained from their outputs for far more accurate and improved decisions. Let’s understand it better with the example of multiple decision trees.

Let us assume you have a dataset containing an `X` number of features. Let’s start by taking a `K` number of features from these `X` features and building a decision tree. If we repeat this process, we will have a random number of features for our tree each time, and the resultant tree will be different.

Now, all that is left is to bootstrap the results together, and our barebones ensemble model is ready. If we also introduce a random sampling of the training data for this cluster of decision trees, this becomes the very famous and still in use decision tree algorithm known as Random Forest (**Figure 6**).

[![Image 8](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%20700%20366'%3E%3C/svg%3E)](https://lh5.googleusercontent.com/e4azyAvumbMt1iWtu4u7zMKB9CSI-3lviBHcWhr6hL-Cz0xB8ZF4e8yVMAoyS7RcBPSA3hLLw9UpG4evuVydlb269vY1L8s4rHl-oKabwqikyWpztbrRiS5B_EjsoABm0CHLyUxif_BWV_pt-Z5F4bWn6hF3eOMydoH5P9RqjlAUU28bBLREH9jrJYwRrQ)

**Figure 6:** Random Forest ([source](https://www.researchgate.net/figure/The-simplified-architecture-of-our-random-forest-classifier_fig5_337433422)).

The third piece of this puzzle is understanding what gradient boosting is. While we will discuss the mathematical side of this concept in the later blogs of this series, let’s try to grasp the idea behind it.

Assume we have some basic dataset where a label is given based on 3 features. First, we will start by creating our base tree. If our task is a regression task, this tree can be a single node outputting the average of the labels. This isn’t much help, but this is our starting point.

The loss generated from our starting tree will tell us how far off we are from getting close to the labels. Now, we create a second tree which will be a proper decision tree. As we have seen, decision trees can end up overfitting, causing very low consideration of variance.

To mitigate this problem, we will determine how much a single tree will have in our overall result. The result will be the summation of the output of our initial root tree (the average of labels) and the output of the second tree, modified with a learning rate.

If we are going in the right direction, our loss generated from this summation will be lower than the initial loss. We repeat this process until the number of trees specified is fulfilled. If you want to understand this in simpler terms, a set of weak learners is combined into a singular strong learner.

* * *

### [**XGBoost**](#TOC)

With a basic understanding of all the prerequisites, we can move on to the highlight of this blog; XGBoost.

XGBoost stands for Extreme Gradient Boosting, an optimized solution for training in gradient boosting. Arguably the most powerful classical machine learning algorithm out there today, it has several features which combine concepts we have learned so far:

*   Has steps to prevent overfitting (regularization)
*   Can work in Parallel processing
*   Has in-built cross-validation and early stopping
*   Results in highly optimized results (deeper trees)

In the subsequent parts of this series, we will learn more about the math behind some of the concepts we have mentioned, but for now, let us use XGBoost in a practical scenario and marvel at how easy it is!

* * *

### [**Configuring Your Development Environment**](#TOC)

To follow this guide, you need the OpenCV library installed on your system.

Luckily, OpenCV is pip-installable:

$ pip install opencv-contrib-python

**If you need help configuring your development environment for OpenCV, we _highly recommend_ that you read our** [**_pip install OpenCV_ guide**](https://pyimagesearch.com/2018/09/19/pip-install-opencv/) — it will have you up and running in a matter of minutes.

* * *

### [**Having Problems Configuring Your Development Environment?**](#TOC)

[![Image 9](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%20500%20334'%3E%3C/svg%3E)](https://pyimagesearch.com/pyimagesearch-university/)

**Figure 7:** Having trouble configuring your dev environment? Want access to pre-configured Jupyter Notebooks running on Google Colab? Be sure to join [PyImageSearch University](https://pyimagesearch.com/pyimagesearch-university/) — you’ll be up and running with this tutorial in a matter of minutes.

All that said, are you:

*   Short on time?
*   Learning on your employer’s administratively locked system?
*   Wanting to skip the hassle of fighting with the command line, package managers, and virtual environments?
*   **Ready to run the code** **_right now_** **on your Windows, macOS, or Linux system?**

Then join [PyImageSearch University](https://pyimagesearch.com/pyimagesearch-university/) today!

**Gain access to Jupyter Notebooks for this tutorial and other PyImageSearch guides that are** **_pre-configured_** **to run on Google Colab’s ecosystem right in your web browser!** No installation required.

And best of all, these Jupyter Notebooks will run on Windows, macOS, and Linux!

* * *

### [**Configuring the Prerequisites**](#TOC)

Since we are working with Kaggle, we will use notebooks and Kaggle notebooks for our code. For our dataset today, we have directly used the `iris` dataset from `sklearn.datasets`. Our objective is to see how easy it is to use XGBoost in our code.

\# import necessary packages
import pandas as pd
import xgboost as xgb
from sklearn.datasets import load\_iris
from sklearn.model\_selection import train\_test\_split
from sklearn.metrics import mean\_squared\_error

In the imports, we have `pandas`, `xgboost`, and several helper packages from `sklearn`, which include the iris dataset and `train_test_split` (**Lines 2-6**).

* * *

### [**Plugging in XGBoost**](#TOC)

Now we will directly use XGBoost in our code.

\# load the iris dataset and create the dataframe
iris = load\_iris()
X = pd.DataFrame(iris.data, columns=iris.feature\_names)
y = pd.Series(iris.target)

On **Line 9**, we used the `load_iris` functionality to load the required dataset directly and created a dataframe consisting of our data (**Lines 10 and 11**).

\# split the dataset 
xTrain, xTest, yTrain, yTest = train\_test\_split(X, y)
 
# define the XGBoost regressor according to your specifications
xgbModel = xgb.XGBRegressor(
    n\_estimators=100,
    reg\_lambda=1,
    gamma=0,
    max\_depth=4
)

On **Line 14**, we use another `sklearn` function to create our training and test split. This is followed by defining the XGBoost regressor, which takes in the following parameters (**Lines 17-22**):

*   `n_estimators`: Number of trees
*   `reg_lambda`: Lambda value for L2 regularization
*   `gamma`: Minimum reduction of loss for a branch split
*   `max_depth`: Max depth of a tree

\# fit the training data in the model
xgbModel.fit(xTrain, yTrain)
 
# Store the importance of features in a separate dataframe
impFeat = pd.DataFrame(xgbModel.feature\_importances\_.reshape(1, -1), columns=iris.feature\_names)

On **Line 25**, we fit the data in our training into our regression model and let it train. XGBoost will automatically let us separate out the importance of features, which we have done on **Line 28**.

\# get predictions on test data
yPred = xgbModel.predict(xTest)
 
# store the msq error from the predictions
msqErr = mean\_squared\_error(yPred, yTest)

Finally, we can use the trained model to predict on the test data (**Line 31**) and manually calculate the mean-squared error from the predictions (**Line 34**).

* * *

### What's next? We recommend [PyImageSearch University](https://pyimagesearch.com/pyimagesearch-university/?utm_source=blogPost&utm_medium=bottomBanner&utm_campaign=What%27s%20next%3F%20I%20recommend).

**Course information:**  
84 total classes • 114+ hours of on-demand code walkthrough videos • Last updated: February 2024  
★★★★★ 4.84 (128 Ratings) • 16,000+ Students Enrolled

**I strongly believe that if you had the right teacher you could _master_ computer vision and deep learning.**

Do you think learning computer vision and deep learning has to be time-consuming, overwhelming, and complicated? Or has to involve complex mathematics and equations? Or requires a degree in computer science?

That’s _not_ the case.

All you need to master computer vision and deep learning is for someone to explain things to you in _simple, intuitive_ terms. _And that’s exactly what I do_. My mission is to change education and how complex Artificial Intelligence topics are taught.

If you're serious about learning computer vision, your next stop should be PyImageSearch University, the most comprehensive computer vision, deep learning, and OpenCV course online today. Here you’ll learn how to _successfully_ and _confidently_ apply computer vision to your work, research, and projects. Join me in computer vision mastery.

**Inside PyImageSearch University you'll find:**

*   ✓ **84 courses** on essential computer vision, deep learning, and OpenCV topics
*   ✓ **84 Certificates** of Completion
*   ✓ **114+ hours** of on-demand video
*   ✓ **Brand new courses released _regularly_**, ensuring you can keep up with state-of-the-art techniques
*   ✓ **Pre-configured Jupyter Notebooks in Google Colab**
*   ✓ Run all code examples in your web browser — works on Windows, macOS, and Linux (no dev environment configuration required!)
*   ✓ Access to **centralized code repos for _all_ 536+ tutorials** on PyImageSearch
*   ✓ **Easy one-click downloads** for code, datasets, pre-trained models, etc.
*   ✓ **Access** on mobile, laptop, desktop, etc.

[Click here to join PyImageSearch University](https://pyimagesearch.com/pyimagesearch-university/?utm_source=blogPost&utm_medium=bottomBanner&utm_campaign=What%27s%20next%3F%20I%20recommend)

* * *

[**Summary**](#TOC)
-------------------

In this introductory post, we went over some key concepts required to understand XGBoost. We learned the intuition behind decision trees, ensemble models, and gradient boosting and used XGBoost for a basic problem of solving the iris dataset.

The ease of use for an algorithm as powerful as XGBoost was noted as well as its ability to learn over any given dataset.

* * *

### [**Citation Information**](#TOC)

**Martinez, H.** “Scaling Kaggle Competitions Using XGBoost: Part 1,” _PyImageSearch_, P. Chugh, A. R. Gosthipaty, S. Huot, K. Kidriavsteva, R. Raha, and A. Thanki, eds., 2022, [https:](https://pyimg.co/0c9pb)[//pyimg.co/0c9pb](https://pyimg.co/0c9pb)

@incollection{Martinez\_2022\_XGBoost1,
  author = {Hector Martinez},
  title = {Scaling {Kaggle} Competitions Using {XGBoost}: Part 1},
  booktitle = {PyImageSearch},
  editor = {Puneet Chugh and Aritra Roy Gosthipaty and Susan Huot and Kseniia Kidriavsteva and Ritwik Raha and Abhishek Thanki},
  year = {2022},
  note = {https://pyimg.co/0c9pb},
}

* * *

![Image 10: Featured Image](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%20500%20276'%3E%3C/svg%3E)

### Unleash the potential of computer vision with Roboflow - Free!

*   Step into the realm of the future by [signing up or logging into your Roboflow account](https://roboflow.com/?ref=pyimagesearch). Unlock a wealth of innovative dataset libraries and revolutionize your computer vision operations.
*   Jumpstart your journey by choosing from our broad array of datasets, or benefit from PyimageSearch’s comprehensive library, crafted to cater to a wide range of requirements.
*   Transfer your data to Roboflow in any of the 40+ compatible formats. Leverage cutting-edge model architectures for training, and deploy seamlessly across diverse platforms, including API, NVIDIA, browser, iOS, and beyond. Integrate our platform effortlessly with your applications or your favorite third-party tools.
*   Equip yourself with the ability to train a potent computer vision model in a mere afternoon. With a few images, you can import data from any source via API, annotate images using our superior cloud-hosted tool, kickstart model training with a single click, and deploy the model via a hosted API endpoint. Tailor your process by opting for a code-centric approach, leveraging our intuitive, cloud-based UI, or combining both to fit your unique needs.
*   Embark on your journey today with absolutely no credit card required. Step into the future with Roboflow.

[Join Roboflow Now](https://roboflow.com/?ref=pyimagesearch)

* * *

**To download the source code to this post (and be notified when future tutorials are published here on PyImageSearch), _simply enter your email address in the form below!_**

![Image 11](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%20410%20567'%3E%3C/svg%3E)

#### Download the Source Code and FREE 17-page Resource Guide

Enter your email address below to get a .zip of the code and a **FREE 17-page Resource Guide on Computer Vision, OpenCV, and Deep Learning.** Inside you'll find my hand-picked tutorials, books, courses, and libraries to help you master CV and DL!