How to Make XGBoost 200x Faster | Towards Data Science

As Data Scientists, we like to run many time-intensive experiments. Reducing the training speed of our models means that we can conduct more experiments in the same amount of time. Moreover, we can also leverage this speed by creating bigger model ensembles, ultimately resulting in higher accuracy.
Chen and Guestrin (from the University of Washington) released XGBoost dates in 2016. They achieved significant speedups and increased predictive power compared to regular gradient boosting (see my book for a comparison, see scikit-learn for regular gradient boosting). This new model soon became data scientists' favorite on Kaggle.
Let’s run XGBoost ‘vanilla’ version on a dummy dataset with 10M rows and 30 columns.
times_cpu = []for trial in range(trials): start = time.time() XGB = XGBRegressor() XGB = XGB.fit(X,Y) print(time.time() — start) times_cpu.append(time.time() — start)
The running time (on my intel i7–7700K CPU @ 4.20GHz computer) is around 2000 seconds. I only tried this setup once since it takes about half an hour of running time.
LGBM was released in 2017 by a Microsoft team. Their groundbreaking idea to make the training faster was to bucket continuous features (read training data) in discrete bins (histograms).
Histogram-based boosting is now considered the best practice for boosting trees. It is now implemented by XGBoost, CatBoost, as well as scikit-learn.
You can use this new histogram-based method by setting tree_method = ‘hist’ when training XGBoost.
times_cpu_hist = []for trial in range(trials): start = time.time() XGB = XGBRegressor(tree_method = “hist”) XGB = XGB.fit(X,Y) print(time.time() — start) times_cpu.append(time.time() — start)
Running time is now 57.7 seconds on average. We just achieved a 35x speed-up!
XGBoost allows you to use your GPU to train models. This usually runs faster than regular CPUs and can be activated easily.
To use your GPU, change tree_method to ‘gpu_hist’
times_gpu = []for trial in range(trials): start = time.time() XGB = XGBRegressor(tree_method = “gpu_hist”) XGB = XGB.fit(X,Y) print(time.time() — start) times_gpu.append(time.time() — start)
Running time is now around 13.1 seconds (using an Nvidia GeForce GTX 1080). That’s 4.4 times faster than the CPU.
Here is how you can use your GPU to run XGBoost on your windows machine.
You can skip this part if you can already run XGBoost on your GPU.
If you installed XGBoost via conda/anaconda, you won’t be able to use your GPU. (If you are unsure how you got XGBoost on your machine, it is 95% likely you got it with anaconda/conda).
Instead, we will install it using pip install.
Open a console and type the two following prompts
First, uninstall XGBoost.
pip uninstall xgboost
Then, reinstall it.
pip install xgboost
Cuda is required to use an Nvidia GPU for machine learning and deep learning.
Download it here: https://developer.nvidia.com/cuda-downloads.
You need to install it using the standard procedure (using the express settings)—nothing specific to do here.
That’s it!
You need to set ‘single_precision_histogram’ to True.
times_gpu_single = []for trial in range(trials): start = time.time() XGB = XGBRegressor(tree_method = “gpu_hist”, single_precision_histogram=True) XGB = XGB.fit(X,Y) print(time.time() — start) times_gpu.append(time.time() — start)
Running time is now 8.7 seconds. That’s a reduction of around 32% compared to the standard GPU method.
⚠️ Note that the single-precision mode won’t always work. If your data exceed the limitations of single-precision floats, you will run into problems. When tracking the squared error of forecasting models, this happened to me: the squared error exceeded the maximum limit that single-precision floating numbers could handle.
Using our GPU (instead of a CPU), histogram-boosting, and single-precision floating numbers, we achieved a 228x training time speed up for XGBoost.
CPU: 2000 seconds
CPU & Histogram: 57.7 seconds
GPU & Histogram: 13.1 seconds
GPU, Histogram & single-precision: 8.7 seconds
🚀 You could speed up the experimentation process further by running experiments in parallel on your GPU and on your CPUs.
⚠️🔬 Don’t forget that this article is using a dummy dataset. The results on your datasets and machine might be different. You should experiment with GPU, histograms, and single-precision before choosing your favorite method.
You might also be interested in this article:
towardsdatascience.com
Nicolas Vandeput is a supply chain data scientist specialized in demand forecasting and inventory optimization. He founded his consultancy company SupChains in 2016 and co-founded SKU Science — a fast, simple, and affordable demand forecasting platform — in 2018. Passionate about education, Nicolas is both an avid learner and enjoys teaching at universities: he has taught forecasting and inventory optimization to master students since 2014 in Brussels, Belgium. Since 2020 he is also teaching both subjects at CentraleSupelec, Paris, France. He published Data Science for Supply Chain Forecasting in 2018 (2nd edition in 2021) and Inventory Optimization: Models and Simulations in 2020.