LightGBM, XGBoost and CatBoost — Kaggle — Santander Challenge
By: Rohan Paul
Tags: Python, Kaggle, Data Science, Machine Learning

LightGBM Model Training

Fundamentals of LightGBM Model

It is a gradient boosting model that makes use of tree based learning algorithms. It is considered to be a fast processing algorithm.

While other algorithms trees grow horizontally, LightGBM algorithm grows vertically, meaning it grows leaf-wise and other algorithms grow level-wise. LightGBM chooses the leaf with large loss to grow. It can lower down more loss than a level wise algorithm when growing the same leaf.

Source of Image

Light GBM is prefixed as Light because of its high speed. Light GBM can handle the large size of data and takes lower memory to run.

Another reason why Light GBM is so popular is because it focuses on accuracy of results. LGBM also supports GPU learning and thus data scientists are widely using LGBM for data science application development.

Leaf growth technique in LightGBM

LightGBM uses leaf-wise (best-first) tree growth. It chooses to grow the leaf that minimizes the loss, allowing a growth of an imbalanced tree. Because it doesn’t grow level-wise, but leaf-wise, over-fitting can happen when data is small. In these cases, it is important to control the tree depth.

When to use LightGBM ?

LightGBM is not preferred for a small volume of datasets as it can easily overfit small data due to its sensitivity. Hence, it generally advised for data having more than 10,000+ rows, though there is no fixed threshold that helps in deciding the usage of LightGBM.

What are LightGBM Parameters?

While, LightGBM has more than 100 parameters that are given in the documentation of LightGBM, let’s checkout the most important ones.

Control Parameters

Max depth: It gives the depth of the tree and also controls the overfitting of the model. If you feel your model is getting overfitted lower down the max depth.

Min_data_in_leaf: Leaf minimum number of records also used for controlling overfitting of the model.

Feature_fraction: It decides the randomly chosen parameter in every iteration for building trees. If it is 0.7 then it means 70% of the parameter would be used.

Bagging_fraction: It checks for the data fraction that will be used in every iteration. Often, used to increase the training speed and avoid overfitting.

Early_stopping_round: If the metric of the validation data does show any improvement in last early_stopping_round rounds. It will lower the imprudent iterations.

Lambda: It states regularization. Its values range from 0 to 1.

Min_gain_to_split: Used to control the number of splits in the tree.

Core Parameters

Task: It tells about the task that is to be performed on the data. It can either train on the data or prediction on the data.

Application: This parameter specifies whether to do regression or classification. LightGBM default parameter for application is regression.

Binary: It is used for binary classification.

Multiclass: It is used for multiclass classification problems.

Regression: It is used for doing regression.

Boosting: It specifies the algorithm type.

rf : Used for Random Forest.

Goss: Gradient-based One Side Sampling.

Num_boost_round: It tells about the boosting iterations.

Learning_rate: The role of learning rate is to power the magnitude of the changes in the approximate that gets updated from each tree’s output. It determines the contribution of each tree on the final outcome and controls how quickly the algorithm proceeds down the gradient descent (learns); Typical values between 0.001–0.3. Smaller values make the model robust to the specific characteristics of each individual tree, thus allowing it to generalize well. Smaller values also make it easier to stop prior to overfitting; however, they increase the risk of not reaching the optimum with a fixed number of trees and are more computationally demanding. This hyperparameter is also called shrinkage. Generally, the smaller this value, the more accurate the model can be but also will require more trees in the sequence.

Num_leaves: It gives the total number of leaves that would be present in a full tree, default value: 31

Metric Parameter

It takes care of the loss while building the model. Some of them are stated below for classification as well as regression.

Mae: Mean absolute error.

Mse: Mean squared error.

Binary_logloss: Binary Classification loss.

Multi_logloss: Multi Classification loss.

def light_gbm_model_run(train_x, train_y, validation_x, validation_y, test_x):

params = {

"objective" : "regression",

"metric" : "rmse",

"num_leaves" : 100,

"learning_rate" : 0.001,

"bagging_fraction" : 0.6,

"feature_fraction" : 0.6,

"bagging_frequency" : 6,

"bagging_seed" : 42,

"verbosity" : -1,

"seed": 42

}

# Given its a regression case, I am using the RMSE as the metric.



lg_train = lgb.Dataset(train_x, label=train_y)

lg_validation = lgb.Dataset(validation_x, label=validation_y)

evals_result_lgbm = {}



model_light_gbm = lgb.train(params, lg_train, 5000,

valid_sets=[lg_train, lg_validation],

early_stopping_rounds=100,

verbose_eval=150,

evals_result=evals_result_lgbm )



pred_test_light_gbm = np.expm1(model_light_gbm.predict(test_x, num_iteration=model_light_gbm.best_iteration ))



return pred_test_light_gbm, model_light_gbm, evals_result_lgbm # Training and output of LightGBM Model

predictions_test_y_light_gbm, model_lgbm, evals_result = light_gbm_model_run(X_train_split, y_train_split, X_validation, y_validation, X_test_original)

print('Output of LightGBM Model training..') Training until validation scores don't improve for 100 rounds

[150] training's rmse: 1.66447 valid_1's rmse: 1.63996

[300] training's rmse: 1.5765 valid_1's rmse: 1.5927

[450] training's rmse: 1.49849 valid_1's rmse: 1.55466

[600] training's rmse: 1.42919 valid_1's rmse: 1.52339

[750] training's rmse: 1.36631 valid_1's rmse: 1.49837

[900] training's rmse: 1.30931 valid_1's rmse: 1.47791

[1050] training's rmse: 1.25734 valid_1's rmse: 1.46143

[1200] training's rmse: 1.20984 valid_1's rmse: 1.44818

[1350] training's rmse: 1.16678 valid_1's rmse: 1.43796

[1500] training's rmse: 1.12698 valid_1's rmse: 1.42969

[1650] training's rmse: 1.09049 valid_1's rmse: 1.42292

[1800] training's rmse: 1.05661 valid_1's rmse: 1.41849

[1950] training's rmse: 1.02528 valid_1's rmse: 1.41488

[2100] training's rmse: 0.995869 valid_1's rmse: 1.41222

[2250] training's rmse: 0.968211 valid_1's rmse: 1.40996

[2400] training's rmse: 0.941985 valid_1's rmse: 1.40807

[2550] training's rmse: 0.917269 valid_1's rmse: 1.40669

[2700] training's rmse: 0.893978 valid_1's rmse: 1.40569

[2850] training's rmse: 0.871822 valid_1's rmse: 1.40492

[3000] training's rmse: 0.850995 valid_1's rmse: 1.40427

[3150] training's rmse: 0.831253 valid_1's rmse: 1.40393

[3300] training's rmse: 0.812591 valid_1's rmse: 1.40376

Early stopping, best iteration is:

[3289] training's rmse: 0.813895 valid_1's rmse: 1.40374

Output of LightGBM Model training..

Hyper-Parameter Tuning in LightGBM

Parameter Tuning is an important part that is usually done by data scientists to achieve a good accuracy, fast result and to deal with overfitting. Let us see quickly some of the parameter tuning you can do for better results.

num_leaves: This parameter is responsible for the complexity of the model. I normally start by trying values in the range [10,100]. But if you have a solid heuristic to choose tree depth you can always use it and set num_leaves to 2^tree_depth — 1

LightGBM Documentation says in respect — This is the main parameter to control the complexity of the tree model. Theoretically, we can set num_leaves = 2^(max_depth) to obtain the same number of leaves as depth-wise tree. However, this simple conversion is not good in practice. The reason is that a leaf-wise tree is typically much deeper than a depth-wise tree for a fixed number of leaves. Unconstrained depth can induce over-fitting. Thus, when trying to tune the num_leaves, we should let it be smaller than 2^(max_depth). For example, when the max_depth=7 the depth-wise tree can get good accuracy, but setting num_leaves to 127 may cause over-fitting, and setting it to 70 or 80 may get better accuracy than depth-wise.

Min_data_in_leaf: Assigning bigger value to this parameter can result in underfitting of the model. Giving it a value of 100 or 1000 is sufficient for a large dataset.

Max_depth: Controls the depth of the individual trees. Typical values range from a depth of 3–8 but it is not uncommon to see a tree depth of 1. Smaller depth trees are computationally efficient (but require more trees); however, higher depth trees allow the algorithm to capture unique interactions but also increase the risk of over-fitting. Larger training data sets are more tolerable to deeper trees.

num_iterations: Num_iterations specifies the number of boosting iterations (trees to build). The more trees you build the more accurate your model can be at the cost of: — Longer training time — Higher chance of over-fitting So typically start with a lower number of trees to build a baseline and increase it later when you want to squeeze the last % out of your model.

It is recommended to use smaller learning_rate with larger num_iterations . Also, we should use early_stopping_rounds if we go for higher num_iterations to stop your training when it is not learning anything useful.

early_stopping_rounds — “early stopping” refers to stopping the training process if the model’s performance on a given validation set does not improve for several consecutive iterations. This parameter will stop training if the validation metric is not improving after the last early stopping round. It should be defined in pair with a number of iterations. If we set it too large we increase the chance of over-fitting. The rule of thumb is to have it at 10% of your num_iterations .

So for my above implementation of LightGBM, initially for two of the LightGBM parameters as below got me a score of 1.47953 (in Kaggle Public Board)

"num_leaves" : 40,

"learning_rate" : 0.004,

And now if I only tune these parameters as below

"num_leaves" : 100,

"learning_rate" : 0.001,

I got my score very very slightly updated to 1.4714 (in Kaggle Public Board)

I also tried the below one (keeping ‘num_leaves’ at 70 to avoid over-fitting)

"num_leaves" : 70,

"learning_rate" : 0.001,

With this — I got a score of 1.47234 (in Kaggle Public Board)

Features Importance in LightGBM