XGBoost, Light GBM and CatBoost
By: Octave - John Keells Group
Tags: Decision Tree, Lightgbm, Xgboost, Data Science, Catboost

XGBoost, Light GBM and CatBoost

Photo by Kirill Nechmonya on Unsplash

When developing a Machine Learning Model, Data Scientists and Data Engineers often apply the algorithm they believe to be the most efficient, hoping for fast performance and accurate predictions, without much thought on how well it caters to the problem at hand. However, it is important to choose the correct algorithm considering the nature of the problem, variability, complexity of the features, and processing limitations as this will lead to better options and improved results. This article discusses popular boosting algorithms, their construction, and their functionality when applied to a large data set.

What are Boosting Algorithms?

In simple terms, ‘Boosting’ is an approach that combines multiple base models into a standalone composite model. When more simple base models (which are called weak models or weak learners) are constructed, the composite model becomes a stronger predictor or a stronger learner. Boosting trains weak learners sequentially, each improving its predecessor. The strong learner is called an ‘ensembled model’ in Boosting Algorithms.

When combining weak learners, the average/weighted average or the error function of the previous weak learner is used to correct the next iteration of learning. Boosting focuses on the misclassified rules or high error resulting rules by proceeding weak rules. When more weights are provided, the data points to incorrect classifications in previous weak models. In the test phase, each model is evaluated based on the test error of each weak model and the prediction is weighted for voting. Boosting methods helps to decrease the bias of this prediction.

Figure: The process of Boosting

In the Machine Learning world, gradient boosting is used for regression, classification, and ranking problems. There are two main elements in this process:

I. Creation of the base learner (weak learner)

Decision trees are used as the weak learner in gradient boosting algorithms. The creation of weak learners is constrained within specific parameters such as maximum number of layers, splits, or leaf nodes, etc.

II. Calculating the loss function

In each iteration, the new model gradually tries to minimize the loss function while maximizing the accuracy estimation of the response variable. The loss function can be customized based on the requirement, but it needs to be a ‘differentiable’ to be additive.

When outliers present in the data sets are not pre-treated, those data points will carry larger residuals than non-outliers, so gradient boosting will help minimize this deviation.

Comparison of Boosting Algorithms; XGBoost, Light GBM and CatBoost

XGBoost

The most sought-after algorithm at Machine Learning competitions, XGBoost was released in 2014. It is a highly scalable, portable, and distributed Gradient Boosting library that provides parallel tree boosting to solve many Data Science problems fast and accurately. The few unique features of XGBoost algorithm are:

· Sparsity aware split finding mechanism

Potential causes for sparsity in a data set include missing values, frequent zero or null entries (common in most surveys), and methods used in feature engineering such as label encoding and one hot encoding. It is important for the algorithm to be aware of these sparsity patterns. In each tree node, XGBoost adds a default direction. That is, when a value is missing in the sparce matrix, this instance is classified to the default split node direction. Note: once the pattern is learnt, the model handles all sparce data instances in a unique way.

· Weighted quantile sketch

Here, weighted data are considered to be ‘weighted quantiles’. Especially in a large data set with many data points, a high processing power is required to traverse all the data points and learn their patterns. This obstacle is typically overcome through quantile distribution sketches, which are based on representative sample of original data. The quantile distribution sketches will provide the same statistical characteristics for each sampled quantile sketch relative to the original quantiles. These quantiles can be of equal weights or unequal weights.

These quantile sketches are the crucial components that aid the algorithm to find split points. Most existing tree-based algorithms can find split points when the data points are of equal weights using a quantile sketch algorithm. However, they are ill-equipped to handle weighted data, unlike XGBoost which has a distributed weighted quantile sketch algorithm to effectively handle it.

· Regularization

This helps define the minimum gain so when a split occurs, and is vital in preventing overfitting. A minimum loss reduction is required to make further partitions on a leaf node of the tree. This concept leads the model to avoid building complex trees which causes the model to memorize patterns instead of learning.

· Using Memory blocks for Parallel Learning

Tree algorithms spend the bulk of the training time sorting the data points in order, which might be time consuming when the data frames are of massive sizes. The special design of memory blocks helps to overcome this problem as it facilitates their easy storage in distributed machines. This enables reuse of the block data layout in subsequent iterations instead of computing it again. This feature is useful for split finding as well.

It must be noted that XGBoost does not have an inbuilt method of handling categorical variables. Therefore, the user needs to perform categorical feature encoding for the model by way of label encoding, one hot encoding, etc.

Light GBM

LightGBM (Light Gradient Boosting Machine) is a free and open source distributed gradient boosting framework for machine learning developed by Microsoft in 2016 with the following advantages:

i. Faster training speed and higher efficiency.

ii. Lower memory usage.

iii. Better accuracy.

iv. Support of parallel, distributed, and GPU learning.

v. Capable of handling large-scale data.

Conventional GBDT (Gradient Boosting Decision Trees) have major limitations in increasing computational complexity proportional to the number of features and data points, as the design of the algorithm forces all the data instances to be scanned for every feature to estimate the information gain of all possible split points. To tackle this challenge, two novel techniques have been implemented in LightGBM:

· Gradient-Based One-Side Sampling (GOSS) — Focus on number of data points

Different gradients play different roles when computing the information gained in a decision tree. Data instances with larger gradients contribute more to the information gained and vice versa. Therefore, the best approach to train a model is to keep instances with large gradients and cut out the small gradients when sampling the data set. This will achieve higher accuracy in the information gained than down-sampling a data set using a purely uniform random sampling technique. This treatment is useful if the data set has a large range of information gained across data points which is generally the case with large data sets.

· Exclusive Feature Bundling (EFB) — Focus on number of features

Bundling many features together depends on the sparsity of model features. Although there are many features, they might have been generated through a process of one-hot encoding or a similar transformation which makes them exclusive. In other words, features rarely take nonzero values simultaneously. Such exclusive features can be bundled to reduce the feature set through a lossless approach.

Catboost

This is based on gradient boosting and developed by Yandex researchers in 2017. This algorithm focuses on categorical features in a data set, which gave it the name ‘CatBoost’. As categorical features have a discrete set of categories, a popular technique to handle them in boosted trees is one hot encoding, where a new binary feature is added to the existing feature in each category. However, with high cardinality features, this technique may lead to an infeasibly large number of features.

· Ordered boosting and ordered TS (Target Statistics)

The Catboost Algorithm has an ordering principal that stops target leakage and outperforms other gradient boosting techniques. In this algorithm, ordered target statistics encoding is developed from target mean encoding. Ordered boosting removes the prediction shift problem which is resulted from a difference in the distribution of train and test data sets.

Features with fewer categories use one-hot encoding. The maximum number of categories for one-hot encoding can be controlled by the one_hot_max_size parameter. For the remaining categorical columns, CatBoost uses an efficient method of encoding, similar to mean encoding but with an additional mechanism aimed at reducing overfitting.

If the data set is significantly biased with the number of categorical features, the ‘explainability’ of the model should also be biased with tunable categorical features relative to the numerical features, which leads to the algorithm to provide promising results.

Comparison

Empirical Evidence of a Regression Problem: A Case Study

A beverage company with a footprint throughout its operating region is interested in understanding how sales at each outlet can be influenced via the promotions they offer, and how these promotions impact their sales uplift. When solving this business problem, an issue arises in comparing sales if the company doesn’t run promotions and sales if there are promotions. If this is possible, the incremental sales can be derived from the findings which lead the company to evaluate the ROI (Return on Investment) of their promotion schemes to better align them in the future.

But comparing the pre and post promotion periods is not easy as it seems. It is possible if the company had pre promotion periods for the same target customer sets, but realistically this is not possible with long-standing companies as it has run similar promotions targeting the same customer base for years to keep them satisfied.

Although most large companies release promotions on a regular basis, there can be a segment of customers that are rarely touched by them due to low potential sales, being new customers, or the irregular nature of running the promotions. Sometimes the customers who receive promotions will not follow through on all but only for a few selected items. The final goal is to predict the baseline sales (no promotion sales) for all different types of beverages on offer.

Data set

Assume that the data set available in this case study has a large volume, leaving room to evaluate the finer points of the three algorithms discussed above. The total data points are approximately 1.5 million records spanning several months and 30 engineered features (25 numeric attributes and 5 categorical attributes). These final features are the best features obtained through a PCA (Principal Component Analysis) exercise which can best explain the final sales of the problem. These features comprised of sales features, promotion features, beverage product attributes, and location features.

Since the problem demands predicting a continuous variable, its required to use Regressors such as XGBoost, CatBoost and LightGBM. To generate comparable results, similar hyper parameter values have been applied for these algorithms. The model evaluation metric used is MAPE (Mean Absolute Percentage Error). The experimental environment is Azure Databricks with a runtime version of 7.3 LTS ML (includes Apache Spark 3.0.1, Scala 2.12). Further this was configured with one driver node of 32GB memory/4 cores and ten worker nodes of 14GB memory/4 cores.

Results

In terms of accuracy, both XGBoost and CatBoost exhibit similar performances and can provide an average accuracy closer to 80%. The MAPE of LightGBM is noticeably lower than the rest. But accuracy might not be the single criteria that should be considered when selecting an algorithm.

The speed of performance is determined by the complexity of the data set and the model parameters that the model will be based on. The hyper parameters configured in this experiment and used to train the model are 6000 trees, maximum depth of tress being 16 and with a learning rate of 0.25.

The fastest algorithm is LightGBM and the XGBoost is far behind the average training speed and is 2X times slower in this scenario.

Summary

Written by Nadeesha Ekanayake, Senior Data Scientist.