XGBoost: A Complete Guide to Fine-Tune and Optimize your Model | by David Martins | Towards Data Science

Initially started as a research project in 2014, XGBoost has quickly become one of the most popular Machine Learning algorithms of the past few years.
Many consider it as one of the best algorithms and, due to its great performance for regression and classification problems, would recommend it as a first choice in many situations. XGBoost has become famous for winning tons of Kaggle competitions, is now used in many industry-application, and is even implemented within machine-learning platforms, such as BigQuery ML.
If you’re reading this article on XGBoost hyperparameters optimization, you’re probably familiar with the algorithm. But to better understand what we want to tune, let's have a recap!
XGBoost (eXtreme Gradient Boosting) is not only an algorithm. It’s an entire open-source library, designed as an optimized implementation of the Gradient Boosting framework. It focuses on speed, flexibility, and model performances. Its strength doesn’t only come from the algorithm, but also from all the underlying system optimization (parallelization, caching, hardware optimization, etc...).
In most cases, data scientist uses XGBoost with a“Tree Base learner”, which means that your XGBoost model is based on Decision Trees. But even though they are way less popular, you can also use XGboost with other base learners, such as linear model or Dart. As this is by far the most common situation, we’ll focus on Trees for the rest of this article.
At that point, you probably have even more questions. What is a Decision Tree? What is Boosting? What the difference with Gradient Boosting?Don’t worry, we’ll recap it all!
Decision tree is one of the simplest ML algorithms.
It is a way to implement an algorithm that only contains conditional statements.
XGBoost uses a type of decision tree called CART: Classification and Decision Tree.
Classification Trees: the target variable is categorical and the tree is used to identify the “class” within which a target variable would likely fall.
Regression Trees: the target variable is continuous and the tree is used to predict its value.
CART leaves don’t simply contain final decision values, but also real-valued scores for each leaf, no matter if they are used for classification or regression.
Boosting is just a method that uses the principle of ensemble learning, but in sequential order.
If you’re not familiar with ensemble learning, it’s a process that combines decisions from multiple underlying models, and uses a voting technique to determine the final prediction.
Random forests and Bagging are two famous ensemble learning methods.
Boosting is a type of ensemble learning that uses the previous model's result as an input to the next one. Instead of training models separately, boosting trains models sequentially, each new model being trained to correct the errors of the previous ones. At each iteration (round), the outcomes predicted correctly are given a lower weight, and the ones wrongly predicted a higher weight. It then uses a weighted average to produce a final outcome.
Finally, Gradient Boosting is a boosting method where errors are minimized using a gradient descent algorithm. Simply put, Gradient descent is an iterative optimization algorithm used to minimize a loss function.
The loss function quantifies how far off our prediction is from the actual result for a given data point. The better the predictions, the lower will be the output of your loss function.
When we construct our model, the goal is to minimize the loss function across all of the data points. For example, Mean squared error (MSE) is the most commonly used loss function for regression.
Contrary to classic Boosting, Gradient boosting not only weight higher wrongly predicted outcomes, but also adjust those weights based on a gradient — given by the direction in the loss function where the loss “decreases the fastest”. If you want to learn more about Gradient Boosting, you can check out this video.
And as we said in the intro, XGBoost is an optimized implementation of this Gradient Boosting method!
There are 2 common ways of using XGBoost:
Learning API: It is the basic, low-level way of using XGBoost. Simple and powerful, it includes a built-in cross-validation method.
import xgboost as xgb X, y = #Import your datadmatrix = xgb.DMatrix(data=x, label=y) #Learning API uses a dmatrixparams = {'objective':'reg:squarederror'}cv_results = xgb.cv(dtrain=dmatrix,                     params=params,                     nfold=10,                     metrics={'rmse'})print('RMSE: %.2f' % cv_results['test-rmse-mean'].min())
Scikit-Learn API: It is a Scikit-Learn wrapper interface for XGBoost. It allows using XGBoost in a scikit-learn compatible way, the same way you would use any native scikit-learn model.
import xgboost as xgbX, y = # Import your dataxtrain, xtest, ytrain, ytest = train_test_split(X, y, test_size=0.2)xgbr = xgb.XGBRegressor(objective='reg:squarederror')xgbr.fit(xtrain, ytrain) ypred = xgbr.predict(xtest)mse = mean_squared_error(ytest, ypred)print("RMSE: %.2f" % (mse**(1/2.0)))
Note that when using the Learning API you can input and access an evaluation metric, whereas when using the Scikit-learn API you have to calculate it.
XGBoost is a great choice in multiple situations, including regression and classification problems. Based on the problem and how you want your model to learn, you’ll choose a different objective function.
The most commonly used are:
reg:squarederror: for linear regression
reg:logistic: for logistic regression
binary:logistic: for logistic regression — with output of the probabilities
How would an untuned model perform compared to a tuned model? Is it worth the effort? Before going deeper into XGBoost model tuning, let’s highlight the reasons why you have to tune your model.
As a demo, we will use the well-known Boston house prices dataset from sklearn, and try to predict the prices of houses.
Here how would perform our model without hyperparameter tuning:
import xgboost as xgbfrom sklearn.datasets import load_bostonfrom sklearn.model_selection import train_test_splitfrom sklearn.metrics import mean_squared_error boston = load_boston()X, y = boston.data, boston.targetdmatrix = xgb.DMatrix(data=x, label=y)params={'objective':'reg:squarederror'}cv_results = xgb.cv(dtrain=dmatrix, params=params, nfold=10, metrics={'rmse'}, as_pandas=True, seed=20)print('RMSE: %.2f' % cv_results['test-rmse-mean'].min())## Result : RMSE: 3.38
Without any tuning, we’ve got a RMSE of 3.38. Which isn’t bad, but let’s see how it would perform with just a few tuned hyperparameters:
import xgboost as xgbfrom sklearn.datasets import load_bostonfrom sklearn.model_selection import train_test_splitfrom sklearn.metrics import mean_squared_error boston = load_boston()X, y = boston.data, boston.targetdmatrix = xgb.DMatrix(data=x, label=y)params={ 'objective':'reg:squarederror',         'max_depth': 6,          'colsample_bylevel':0.5,         'learning_rate':0.01,         'random_state':20}cv_results = xgb.cv(dtrain=dmatrix, params=params, nfold=10, metrics={'rmse'}, as_pandas=True, seed=20, num_boost_round=1000)print('RMSE: %.2f' % cv_results['test-rmse-mean'].min())## Result : RMSE: 2.69
With just a little bit of tuning, we’ve now got a RMSE of 2.69. It’s a 20% improvement! And we could probably improve even more. Let’s see how!
A hyperparameter is a type of parameter, external to the model, set before the learning process begins. It’s tunable and can directly affect how well a model performs.
To find out the best hyperparameters for your model, you may use rules of thumb, or specific methods that we’ll review in this article.
Before that, note that there are several parameters you can tune when working with XGBoost. You can find the complete list here, or the aliases used in the Scikit-Learn API.
For Tree base learners, the most common parameters are:
max_depth: The maximum depth per tree. A deeper tree might increase the performance, but also the complexity and chances to overfit.The value must be an integer greater than 0. Default is 6.
learning_rate: The learning rate determines the step size at each iteration while your model optimizes toward its objective. A low learning rate makes computation slower, and requires more rounds to achieve the same reduction in residual error as a model with a high learning rate. But it optimizes the chances to reach the best optimum.The value must be between 0 and 1. Default is 0.3.
n_estimators: The number of trees in our ensemble. Equivalent to the number of boosting rounds.The value must be an integer greater than 0. Default is 100.NB: In the standard library, this is referred as num_boost_round.
colsample_bytree: Represents the fraction of columns to be randomly sampled for each tree. It might improve overfitting.The value must be between 0 and 1. Default is 1.
subsample: Represents the fraction of observations to be sampled for each tree. A lower values prevent overfitting but might lead to under-fitting.The value must be between 0 and 1. Default is 1.
Regularization parameters:
alpha (reg_alpha): L1 regularization on the weights (Lasso Regression). When working with a large number of features, it might improve speed performances. It can be any integer. Default is 0.
lambda (reg_lambda): L2 regularization on the weights (Ridge Regression). It might help to reduce overfitting. It can be any integer. Default is 1.
gamma: Gamma is a pseudo-regularisation parameter (Lagrangian multiplier), and depends on the other parameters. The higher Gamma is, the higher the regularization. It can be any integer. Default is 0.
A first approach would be to start with reasonable parameters and to play along. If you understood the meanings of each hyperparameter above, you should be able to intuitively set some values.
Let’s start with reasonable values. It would usually be:
max_depth: 3–10n_estimators: 100 (lots of observations) to 1000 (few observations)learning_rate: 0.01–0.3colsample_bytree: 0.5–1subsample: 0.6–1
Then, you can focus on optimizing max_depth and n_estimators.
You can then play along with the learning_rate, and increase it to speed up the model without decreasing the performances. If it becomes faster without losing in performances, you can increase the number of estimators to try to increase the performances.
Finally, you can work with your regularization parameters, usually starting with alpha and lambda. For gamma, 0 would mean no regularization, 1–5 are commonly used values, whereas 10+ would be considered as very high.
A second approach to find the best hyperparameters is through Optimization Algorithm. Since XGBoost is available in a Scikit-learn compatible way, you can work with Scikit-learn’s hyperparameter optimizer functions!
The two most common are Grid Search and Random Search.
A Grid Search is an exhaustive search over every combination of specified parameter values. If you specify 2 possible values for max_depth and 3 for n_estimators, Grid Search will iterate over 6 possible combinations:
max_depth: [3,6],n_estimators:[100, 200, 300]
Would result in the following possibilities:max_depth: 3, n_estimators: 100max_depth: 3, n_estimators: 200max_depth: 3, n_estimators: 300max_depth: 6, n_estimators: 100max_depth: 6, n_estimators: 200max_depth: 6, n_estimators: 300
Let’s use GridSearchCV() from Scikit-learn to tune our XGBoost model!
In the following examples, we’ll use a processed version of the Life Expectancy dataset available on Kaggle.
import pandas as pdimport xgboost as xgbfrom sklearn.model_selection import GridSearchCVdata = pd.read_csv("life_expectancy_clean.csv")X, y = data[data.columns.tolist()[:-1]],       data[data.columns.tolist()[-1]]params = { 'max_depth': [3,6,10],           'learning_rate': [0.01, 0.05, 0.1],           'n_estimators': [100, 500, 1000],           'colsample_bytree': [0.3, 0.7]}xgbr = xgb.XGBRegressor(seed = 20)clf = GridSearchCV(estimator=xgbr,                    param_grid=params,                   scoring='neg_mean_squared_error',                    verbose=1)clf.fit(X, y)print("Best parameters:", clf.best_params_)print("Lowest RMSE: ", (-clf.best_score_)**(1/2.0))
estimator: GridSearchCV is part of sklearn.model_selection, and works with any scikit-learn compatible estimator. We use xgb.XGBRegressor(), from XGBoost’s Scikit-learn API.
param_grid: GridSearchCV takes a list of parameters to test in input. As we said, a Grid Search will test out every combination.
scoring: It’s the metric(s) that will be used to evaluate the performance of the cross-validated model. In this case, neg_mean_squared_error is used in replacement for mean_squared_error. GridSearchCV is simply using a negative version of MSE for technical reasons — so it makes the function generalizable to other metrics where we aim for the higher score instead of the lower.
verbose: Controls the verbosity. The higher, the more messages.
More parameters as available, as you can find out in the documentation.
Finally, the lowest RMSE based on the negative value of clf.best_score_And the best parameters with clf.best_params_
Best parameters: {'colsample_bytree': 0.7, 'learning_rate': 0.05, 'max_depth': 6, 'n_estimators': 500}
A Random Search uses a large (possibly infinite) range of hyperparameters values, and randomly iterates a specified number of times over combinations of those values. Contrary to a Grid Search which iterates over every possible combination, with a Random Search you specify the number of iterations.
If you input 10 possible values for max_depth, 200 possible values for n_estimators, and choose to do 10 iterations:
max_depth: np.arrange(1,10,1),n_estimators: np.arrange(100,400,2)
Example of random possibilities with 10 iterations:1: max_depth: 1, n_estimators: 1102: max_depth: 3, n_estimators: 2223: max_depth: 3, n_estimators: 3064: max_depth: 4, n_estimators: 1025: max_depth: 1, n_estimators: 3986: max_depth: 6, n_estimators: 2907: max_depth: 9, n_estimators: 1028: max_depth: 6, n_estimators: 3109: max_depth: 3, n_estimators: 34410: max_depth: 6, n_estimators: 202
Now, let’s use RandomSearchCV() from Scikit-learn to tune our model!
import pandas as pdimport numpy as npimport xgboost as xgbfrom sklearn.model_selection import RandomizedSearchCVdata = pd.read_csv("life_expectancy_clean.csv")X, y = data[data.columns.tolist()[:-1]],       data[data.columns.tolist()[-1]]params = { 'max_depth': [3, 5, 6, 10, 15, 20],           'learning_rate': [0.01, 0.1, 0.2, 0.3],           'subsample': np.arange(0.5, 1.0, 0.1),           'colsample_bytree': np.arange(0.4, 1.0, 0.1),           'colsample_bylevel': np.arange(0.4, 1.0, 0.1),           'n_estimators': [100, 500, 1000]}xgbr = xgb.XGBRegressor(seed = 20)clf = RandomizedSearchCV(estimator=xgbr,                         param_distributions=params,                         scoring='neg_mean_squared_error',                         n_iter=25,                         verbose=1)clf.fit(X, y)print("Best parameters:", clf.best_params_)print("Lowest RMSE: ", (-clf.best_score_)**(1/2.0))
Most RandomizedSearchCV’s parameters are similar to GridSearchCV’s.
n_iter: It’s the number of parameter combinations that are sampled.The higher, the more combinations you’ll be testing. It trades off runtime and quality of the solution.
As for GridSearchCV, we print the best parameters with clf.best_params_And the lowest RMSE based on the negative value of clf.best_score_
In this article, we explained how XGBoost operates to better understand how to tune its hyperparameters. As we’ve seen, tuning usually results in a big improvement in model performances.
Using our intuition to tune our model might sometimes be enough. It is also worth trying Optimization Algorithms like GridSearch and RandomSearch.But most of the time, you’ll get an even better result with a mix of Algorithms and adjustments through testing and intuition!