Comprehensive LightGBM Tutorial (2021) | Towards Data Science

Learn how to crush XGBoost in this comprehensive LightGBM tutorial.
I am confused.
So many people are drawn to XGBoost like a moth to a flame. Yes, it has seen some glorious days in prestigious competitions, and it’s still the most widely-used ML library.
But, it has been 4 years since XGBoost lost its top spot in terms of performance. In 2017, Microsoft open-sourced LightGBM (Light Gradient Boosting Machine) that gives equally high accuracy with 2–10 times less training speed.
This is a game-changing advantage considering the ubiquity of massive, million-row datasets. There are other distinctions that tip the scales towards LightGBM and give it an edge over XGBoost.
By the end of this post, you will learn about these advantages, including:
how to develop LightGBM models for classification and regression tasks
structural differences between XGBoost and LGBM
how to use early stopping and evaluation sets
enabling powerful categorical feature support for up 8x times speed increase
implementing successful cross-validation with LGBM
hyperparameter tuning with Optuna (Part II)
When LGBM got released, it came with ground-breaking changes to the way it grows decision trees.
Both XGBoost and LightGBM are ensebmle algorithms. They use a special type of decision trees, also called weak learners, to capture complex, non-linear patterns.
In XGBoost (and many other libraries), decision trees were built one level at a time:
This type of structure tends to result in unnecessary nodes and leaves because the trees continued to build until the max_depth reached. This led to higher model complexity and training cost runtime.
In contrast, LightGBM takes a leaf-wise approach:
The structure continues to grow with the most promising branches and leaves (nodes with the most delta loss), holding the number of the decision leaves constant. (If this doesn’t make sense to you, don’t sweat. This won’t prevent you from effectively using LGBM).
This is one of the main reasons LGBM crushed XGBoost in terms of speed when it first came out.
Above is a benchmark comparison of XGBoost with traditional decision trees and LGBM with leaf-wise structure (first and last columns) on datasets with ~500k-13M samples. It shows that LGBM is orders of magnitude faster than XGB.
LGBM also uses histogram binning of continuous features, which provides even more speed-up than traditional gradient boosting. Binning numeric values significantly decrease the number of split points to consider in decision trees, and they remove the need to use sorting algorithms, which are always computation-heavy.
Inspired by LGBM, XGBoost also introduced histogram-binning, which gave massive speed-up but still not enough to match LGBM’s:
We will continue exploring the differences in the coming sections.
Like XGBoost, LGBM has two APIs — core learning API and Sklearn-compatible one. You know I am a big fan of Sklearn, so this tutorial will focus on that version.
Sklearn-compatible API of XGBoost and LGBM allows you to integrate their models in the Sklearn ecosystem so that you can use them inside pipelines in combination with other transformers.
Sklearn API exposes LGBMRegressor and LGBMClassifier, with the familiar fit/predict/predict_proba pattern:
objective specifies the type of learning task. Besides the common ones like binary, multiclass and regression tasks, there are others like poisson, tweedie regressions. See this section of the documentation for the full list of objectives.
>>> reg.fit(X, y)LGBMRegressor()
The number of decision trees inside the ensemble significantly affects the results. You can control it using the n_estimators parameter in both the classifier and regressor. Below, we will fit an LGBM binary classifier on the Kaggle TPS March dataset with 1000 decision trees:
Adding more trees leads to more accuracy but increases the risk of overfitting. To combat this, you can create many trees (+2000) and choose a smaller learning_rate (more on this later).
Like in XGBoost, fitting a single decision tree to the data is called a boosting round.
Each tree in the ensemble builds on the predictions of the last tree — i.e., each boosting round is an improvement of the last.
If the predictions don’t improve after a sequence of rounds, it is sensible to stop the training of the ensemble even if we are not at a hard stop for n_estimators.
To achieve this, LGBM provides early_stopping_rounds parameter inside the fit function. For example, setting it to 100 means we stop the training if the predictions have not improved for the last 100 rounds.
Before looking at a code example, we should learn a couple of concepts connected to early stopping.
Early stopping is only enabled when you pass a set of evaluation sets to eval_set parameter of the fit method. These evaluation sets are used to keep track of the quality of the predictions from one boosting round to the next:
In each round of n_estimators, a single decision tree is fit to (X_train, y_train) and predictions are made on the passed evaluation set (X_eval, y_eval). The quality of predictions is measured with a passed metric in eval_metric.
The training stops at the 738th iteration because the validation score has not improved since the 638th one — early stopping of 100 rounds. Now, we have the luxury of creating as many trees as we want and early_stopping_rounds can discard the unnecessary ones.
Let’s establish a baseline score with what we know so far. We will do the same for XGBoost so that we can compare the results:
LGBM achieved a smaller loss in ~4 times less runtime. Let’s see a final LGBM trick before we move on to cross-validation.
Histogram binning in LGBM comes with built-in support for handling missing values and categorical features. TPS March dataset contains 19 categoricals, and we have been using one-hot encoding up to this point.
This time, we will let LGBM deal with categoricals and compare the results with XGBoost once again:
To specify the categorical features, pass a list of their indices to categorical_feature parameter in the fit method:
You can achieve up to 8x speed up if you use pandas.Categorical data type when using LGBM.
The table shows the final scores and runtimes of both models. As you can see, the version with default categorical handling beat others both in accuracy and speed. Cheers!
The most common way of doing CV with LGBM is to use Sklearn CV splitters.
I am not talking about utility functions like cross_validate or cross_val_score but splitters like KFold or StratifiedKFold with their split method. Doing CV in this way gives you more control over the whole process.
I have talked many times about the importance of cross-validation. You can read this post for more details.
Also, it enables you to use early stopping during cross-validation in a hassle-free manner. Here is what this looks like for the TPS March data:
First, create a CV splitter — we are choosing StratifiedKFold because it is a classification problem. Then, loop through each train/test sets using split. In each fold, initialize and train a new LGBM model and optionally report the score and runtime. That's it! That's how most people do CV, including on Kaggle.
In this post, we learned pure modeling techniques with LightGBM. Next up, we will explore how to squeeze every bit of performance out of LGBM models using Optuna.
Specifically, Part II of this article will include a detailed overview of the most important LGBM hyperparameters and introduce a well-tested hyperparameter tuning workflow. It is already out — read it here.