BetaBoosting. XGBoost with a Funky Learning Rate | by Tyler Blume | Towards Data Science

TLDR: Traditionally, Gradient Boosting implementations such as XGBoost have used a static learning rate. We propose using a function whose parameters can be passed to a standard hyperparameter tuning process to learn the ‘optimal’ learning rate shape.
All code lives here: BetaBoost Github
At this point, we all know of XGBoost due to the massive success it has had in numerous Data Science competitions held on platforms like Kaggle. Along with its success, we have seen several variations such as CatBoost and LightGBM. All of these implementations are based on the Gradient Boosting algorithm developed by Friedman1, which involves iteratively building an ensemble of weak learners (usually decision trees) where each subsequent learner is trained on the previous learner’s errors. Let’s take a look at some general pseudo-code for the algorithm from Elements of Statistical Learning2:
However, this is not complete! A core mechanism which allows boosting to work is a shrinkage parameter that penalizes each learner at each boosting round that is commonly called the ‘learning rate’. It’s main function is to prevent overfitting and it is a nice callback to the Gradient Descent algorithm which inspired many aspects of Gradient Boosting. To utilize a learning rate, Friedman altered 2.d to:
Now we are multiplying each new round’s learner by a constant. But what constant should we use? This is a common question that is typically handled with hyperparameter tuning and usually results in a constant such as .1 or .01. Some number well below 1 or, in other words, a number that heavily penalizes each round. With that small number we will not be utilizing each learner to it’s fullest extent.
When looking specifically at the learning rate parameter, a natural next question (although one that is typically inconsequential) is why use a constant? Gradient Descent has momentum strategies that allow the learning rate to change. Why can’t Gradient Boosting use a similar idea?
Well, we can...
...it just doesn’t work.
A quick Google search reveals that there has been some work done utilizing a decaying learning rate, one that starts out large and shrinks at each round. But we typically see no accuracy gains in Cross Validation and when looking at test error graphs there are minimal differences between its performance and using a regular old constant. The reasoning behind why a method like this doesn’t work appears to be a mystery. This leads us to a core theme that inspired BetaBoost:
More research should be done in this area.
In keeping with our theme, research was done and a function that appears to be promising has been found: the beta function (with some guardrails of course!). Or, more specifically, the beta probability distribution function. This function was initially found via brute force methods and once it provided encouraging results some semi-coherent rationale was dredged.
So, what is the beta PDF? As the name suggests it is a probability distribution described primarily by 2 parameters: α and β. In addition to these parameters, several more were added along with some further ‘guardrails’ which seem to help. For our purposes, we largely don’t care about any of the properties of the function as a probability density function, only it’s results with boosting.
A clear advantage that the Beta PDF has over many other functions is the sheer amount of shapes you can achieve with it3:
If we wanted to do something similar to exponential decay, or create a large spike in the middle, it is only one parameter change away. Which means if we hand the whole process over to a hyperparameter tuning package, such as Hyperopt, we can discover the learning rate shape which ‘best’ fits (or overfits!) our data. Although, we predominantly see one shape coming out on top.
Ultimately, I have no idea why any of this would provide any benefits, so:
More research should be done in this area.
This all brings us to BetaBoosting. This is not a new method of building a tree or proposing splits. It simply is assigning a different learning rate at each boosting round using callbacks in XGBoost’s Learning API. Our specific implementation assigns the learning rate based on the Beta PDf — thus we get the name ‘BetaBoosting’.
The code is pip installable for ease of use and requires xgboost==1.5:
pip install BetaBoost==0.0.5
As mentioned previously, a specific shape seemed to do better than others. A shape that had a large spike early on and, due to our guardrails, quickly went to a ‘floor’ parameter. Let’s take a look at BetaBoost using the class default parameters. This function passes the default parameters to the scipy Beta PDF function, along with some additional guardrails. Here is a simple standalone function along with the actual BetaBoost implementation:
def beta_pdf(scalar=1.5,             a=26,             b=1,             scale=80,             loc=-68,             floor=0.01,             n_boosting_rounds=100):    """    Get the learning rate from the beta PDF    Returns    -------    lrs : list        the resulting learning rates to use.    """    lrs = [scalar*beta.pdf(i,                           a=a,                            b=b,                            scale=scale,                            loc=loc)            + floor for i in range(n_boosting_rounds)]    return lrs
For ease of use here is the same learning rates using the BetaBoost class:
from BetaBoost import BetaBoost as bbimport matplotlib.pyplot as pltbooster = bb.BetaBoost(n_boosting_rounds=100)plt.plot(booster.beta_kernel())plt.show()
Given that this shape does well, this suggests that taking the larger steps, not in the beginning or in the end, but in the ‘teenage years’ of the boosted tree’s development provides the most bang for our buck.
To further investigate, we can run this with simulated data and compare against other learning rates.
(The following example comes from a post years ago about using a decaying learning rate that I can not find, if anyone finds it let me know so I can cite the article!)
First we will do the imports and simulate some data:
import numpy as npimport xgboost as xgbimport matplotlib.pyplot as pltdef generate_data():    y = np.random.gamma(2, 4, OBS)    X = np.random.normal(5, 2, [OBS, FEATURES])    return X, ymax_iter = 300eta_base = 0.2eta_min = 0.1eta_decay = np.linspace(eta_base, eta_min, max_iter).tolist()OBS = 10 ** 4FEATURES = 20PARAMS = {    'eta': eta_base,    "booster": "gbtree",}X_train, y_train = generate_data()X_test, y_test = generate_data()dtrain = xgb.DMatrix(X_train, label=y_train)dtest = xgb.DMatrix(X_test, label=y_test)evals_result = {'train': dtrain}
Next, let’s use the decaying learning rate and save our results in a dictionary which we will eventually graph:
progress1 = dict()model1 = xgb.train(    maximize=True,    params=PARAMS,    dtrain=dtrain,    num_boost_round=max_iter,    early_stopping_rounds=max_iter,    evals=[(dtrain, 'train'),(dtest, 'test')],    evals_result=progress1,    verbose_eval=False,    callbacks=[xgb.callback.LearningRateScheduler(eta_decay)])
Now, a standard constant of 0.01:
progress2 = dict()model2 = xgb.train(    maximize=True,    params=PARAMS,    dtrain=dtrain,    num_boost_round=max_iter,    early_stopping_rounds=max_iter,    evals=[(dtrain, 'train'),(dtest, 'test')],    evals_result=progress2,    verbose_eval=False,    callbacks=[xgb.callback.LearningRateScheduler(list(np.ones(max_iter)*0.01))])
A constant of 0.1
progress3 = dict()model3 = xgb.train(    maximize=True,    params=PARAMS,    dtrain=dtrain,    num_boost_round=max_iter,    early_stopping_rounds=max_iter,    evals=[(dtrain, 'train'),(dtest, 'test')],    evals_result=progress3,    verbose_eval=False,    callbacks=[xgb.callback.LearningRateScheduler(list(np.ones(max_iter)*0.1))])
And finally, our BetaBoost where the fit method simply returns the same output as the train method in the XGBoost Learning API
#Here we call the BetaBoost, the wrapper parameters are passed in the class initbb_evals = dict()from BetaBoost import BetaBoost as bbbetabooster = bb.BetaBoost(n_boosting_rounds=max_iter)betabooster.fit(dtrain=dtrain,                maximize=True,                params=PARAMS,                early_stopping_rounds=max_iter,                evals=[(dtrain, 'train'),(dtest, 'test')],                evals_result=bb_evals,                verbose_eval=False)
Let’s review the results on each round’s test accuracy:
plt.plot(progress1['test']['rmse'], linestyle = 'dashed', color = 'b', label = 'eta test decay')plt.plot(progress2['test']['rmse'], linestyle = 'dashed', color = 'r', label = '0.01 test')plt.plot(progress3['test']['rmse'], linestyle = 'dashed', color = 'black', label = '0.1 test')plt.plot(bb_evals['test']['rmse'], linestyle = 'dashed', color = 'y', label = 'bb test')plt.legend()plt.show()
Looking at the minimal test-set error, we see that the decaying learning rates actually hit the minimum the fastest, but it also is an unstable result as it quickly begins overfitting the test set. Next to hit its ‘kink’ is the betaboosted test set. Unlike the decay test set however, we can see that it continues to decrease in error slowly as it converges. Eventually around iteration 300 it meets up with the error rate corresponding to the constant 0.01 learning rate. So it appears that we are getting the best of both worlds here: we converge to a near-optimal test accuracy quite quickly and then we are resistant to overfitting. But, does it actually perform the best?
No.
The static 0.01 outperforms it slightly over the next 100 iterations. However, this isn’t always the case and we will see some 5 fold CV results where the optimized BetaBooster actually outperforms optimized vanilla XGBoost on real world data in the next article!
Gradient Boosting using a Beta density function as it’s learning rate appears to give us a quicker convergence and a robustness to overfitting. The cost of this is a few more parameters to tune. Additionally, the idea of tuning some function as the learning rate can be added to major boosting packages such as XGBoost and LightGBM quite quickly using their callback functionality. This seems like a low-hanging fruit in a field where we are currently squeezing water from rock.
Ultimately, BetaBoosting begs the question:
Is there a better function out there?
Our answer is...
More research should be done in this area.
References
Friedman, J. H. “Stochastic Gradient Boosting.” (March 1999)Hastie, T., Tibshirani, R., & Friedman, J. H. (2009). The elements of statistical learning: data mining, inference, and prediction. 2nd ed. New York: Springer.https://en.wikipedia.org/wiki/Beta_distribution
Friedman, J. H. “Stochastic Gradient Boosting.” (March 1999)
Hastie, T., Tibshirani, R., & Friedman, J. H. (2009). The elements of statistical learning: data mining, inference, and prediction. 2nd ed. New York: Springer.