LightGBM Hyper Parameters Tuning in Spark | by Cao YI | Towards Data Science

LightGBM is very popular among data scientists in all industries. The lightgbm package is well developed in Python and R. When the data is growing bigger and bigger, people want to run the model on clusters with distributed data frames.
I was developing a recommendation system on Azure Databricks recently. LightGBM model was used in the project. The Hyper Parameter tuning part is not as smooth as it was in Python. In this blog, I will share 3 approaches I have tried when doing the tuning. The parameters being tuned are:
numLeaves
maxDepth
baggingFraction
featureFraction
minSumHessianInLeaf
lambdaL1
lambdaL2
The LightGBM package used here is mmlspark, Microsoft Machine Learning for Apache Spark.
Grid search is a brute force method. If you have unlimited computation powerful, this method can guarantee you the best hyperparameter setting. The following code shows how to do grid search for a LightGBM regressor:
We should know the grid search has the curse of dimension. As the number of parameters increases, the grid grows exponentially. In my practice, the grid setting above will never finish on my exploring cluster with the below setting:
The truth is, even dramatically shrink the grid, the computation is very likely to fail(due to memory issue).
To reduce the computation pressure, the essence is to reduce the dimension. So the idea of doing the tuning sequentially occurs. After doing some research, I found this Chinese blog, written by Byron NG, very helpful. The concept is to do the tuning step by step:
step 1: set a relatively high learning rate, and lower your number of iteration.
This allows you to do the tuning faster in the following steps. And when finishing the tuning, you can increase your iteration number and lower your learning rate to have a decent performance.
step 2: tune numLeaves & maxDepth
These two parameters control the complexity of the tree model. The higher the value, the more complex the model. theoretically, numLeaves ≤ 2^maxDepth. You can check if your tuned value meets this condition. Your model will probably have high variance and low bias. So in the next few steps, I will try to reduce the overfitting by tuning other variables
By the way, the CrossValidatorModel API is really unfriendly. You can output the bestModel, But you can’t easily check your final Hyperparameters.I wrote a function to easily extract the hyperparameters information to check.
step 3: tune minSumHessianInLeaf
This parameter has a lot of aliases, like min_sum_hessian_per_leaf, min_sum_hessian, min_hessian, min_child_weight. In short, it tells you “stop trying to split once you reach a certain degree of purity in a node and your model can fit it”. The right value for minSumHessianInLeaf can reduce overfitting.
step 4: tune baggingFraction & featureFraction
These two parameters must be tuned at the same time. baggingFraction controls the instances subsampling and featureFraction controls the features subsampling. They all serve the purpose of reducing overfitting. (smaller fractions also allow faster computing)
# similar code here
step 5: tune lambdaL1 & lambdaL2
L1 & L2 regulation parameters also help reduce overfitting
# similar code here
Step 6: adjust the learning rate & number of iteration to fit the model
Through this sequential procedure, though tedious, you can eventually get a reasonable tuning result. The main problem for this approach is that you need to initialize a reasonable size grid to start with. If the optimal is located outside of your grid, you need to reassign a proper range. A few attempts are inevitable. And you better be an experienced data scientist to do the tuning. Because each attempt cost a lot of computation resources.
“Hyperopt is a Python library for serial and parallel optimization over awkward search spaces, which may include real-valued, discrete, and conditional dimensions.”
“Currently three algorithms are implemented in hyperopt:
Random Search
Tree of Parzen Estimators (TPE)
Adaptive TPE”
Hyperopt has been designed to accommodate Bayesian optimization algorithms based on Gaussian processes and regression trees, but these are not currently implemented.”
Hyperopt is not widely used so far, I found some posts give instructive implementation in Python: 1. hyperparameter tuning part0, 2. hyperparameter tuning part2. However, though Azure Databricks has hyperopt ready for use, no spark implementation can be found on any website.
Fortunately, the implementation of hyperopt is quite straightforward. After a few tests, I was able to run the tuning on spark. And it’s really fast compared to all the methods I tried before. The tuning results are also very satisfying. Here is how I implemented in PySpark:
To sum up, to best of my knowledge, Hyperopt might be the best option right now to tune the Hyperparameters for LightGBM on a spark data frame. It’s much faster than the brute force grid search and more reliable than the sequential search. But it’s truly not well documented yet. Without the post I found for hyperopt in Python, I may not be able to implement it on spark.
I am keeping learning machine learning on spark. However, the material is limited online so far; most of the model documents are not adequate an outsider use. few sample codes for spark ml implementation are shared. Hope more and more data scientists can share their work and stories and help each other.
If you like a discussion, find me on LinkedIn.