A guide to XGBoost hyperparameters | by Mahbubul Alam | Towards Data Science

What is the one machine learning algorithm — if you ask — that consistently gives superior performance in regression and classification?
XGBoost it is. It is arguably the most powerful algorithm and is increasingly being used in all industries and in all problem domains —from customer analytics and sales prediction to fraud detection and credit approval and more.
It is also a winning algorithm in many machine learning competitions. In fact, XGBoost was used in 17 out of 29 data science competitions on the Kaggle platform.
Not just in businesses and competitions, XGBoost has been used in scientific experiments such as the Large Hadron Collider (the Higgs Boson machine learning challenge).
A key to its performance is its hyperparameters. While XGBoost is extremely easy to implement, the hard part is tuning the hyperparameters. In this article, I will talk about some of the key hyperparameters, their role and how to choose their values.
But before I go there, let’s talk about how XGBoost works under the hood.
XGBoost (or eXtreme Gradient Boost) is not a standalone algorithm in the conventional sense. It is rather an open-source library that “boosts” the performance of other algorithms. It optimizes the performance of algorithms, primarily decision trees, in a gradient boosting framework while minimizing overfitting/bias through regularization.
The key strengths of XGBoost are:
Flexibility: It can perform machine learning tasks such as regression, classification, ranking and other user-defined objectives.
Portability: It runs on Windows, Linux and OS X as well as on cloud platforms.
Languages support: It supports multiple languages including C++, Python, R, Java, Scala, Julia.
Distributed training on cloud systems: XGBoost supports distributed training on multiple machines, including AWS, GCE, Azure, and Yarn clusters.
Other important features of XGBoost include:
parallel processing capabilities for large dataset
can handle missing values
allows for regularization to prevent overfitting
has built-in cross-validation
Below I’ll first walk through a simple 5-step implementation of XGBoost and then we can talk about the hyperparameters and how to use them to optimize performance.
For this demo we do not need much. From sklearn library we can import modules for splitting training and testing data and the accuracy metrics. Note that, first you need to install (pip install) the XGBoost library before you can import it.
# loading datafrom sklearn.datasets import load_iris# to split data into training and testing setfrom sklearn.model_selection import train_test_split# XGBoost libraryimport xgboost as xgb# evaluation metricsfrom sklearn.metrics import accuracy_score
We are using the built-in iris dataset from Scikit Learn library, so we do not need to do data pre-processing. After loading data we just need to store input features and the target variable separately.
df = load_iris()X = df.datay = df.target
The dataset has 4 features (predictors):
df.feature_names>> ['sepal length (cm)', 'sepal width (cm)', 'petal length (cm)', 'petal width (cm)']
The target has 3 classes.
df.target_names>> array(['setosa', 'versicolor', 'virginica'], dtype='<U10')
First, we are going to split data into training (70%) and testing (30%) sets. As an extra step, we need to store data into a compatible DMatrix object for XGBoost compatibility.
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.3, random_state = 1)train = xgb.DMatrix(X_train, label = y_train)test = xgb.DMatrix(X_test, label = y_test)
Since this is a simple demonstration, let’s choose just 3 hyperparameters.
# specify hyperparametersparams = {    'max_depth': 4,    'eta': 0.3,    'num_class': 3}epochs = 10
5. Modeling
Now that you have specified the hyperparameters, rudding the model and making a prediction takes just a couple more lines.
# train modelmodel = xgb.train(params, train, epochs)# predictiony_pred = model.predict(test)
So even with this simple implementation, the model was able to gain 98% accuracy.
Now let’s get to the core of this article — the hyperparameters.
There are many different parameters in XGBoost and they are broadly classified into 3 types:
General parameters
Booster parameters
Task parameters
There are more or less 35 different hyperparameters listed in the XGBoost documentation. However, not all of them are equally important, some are more impactful than others.
Below are some parameters that are frequently tuned in a grid search to find an optimal balance.
Frequently tuned hyperparameters
n_estimators: specifies the number of decision trees to be boosted. If n_estimator = 1, it means only 1 tree is generated, thus no boosting is at work. The default value is 100, but you can play with this number for optimal performance.
subsample: it represents the subsample ratio of the training sample. A subsample = 0.5 means that 50% of training data is used prior to growing a tree. The value can be any fraction but the default value is 1.
max_depth: it limits how deep each tree can grow. The default value is 6 but you can try other values if overfitting is an issue in your model.
learning_rate (alias: eta): it is a regularization parameter that shrinks feature weights in each boosting step. The default value is 0.3 but people generally tune with values such as 0.01, 0.1, 0.2 etc.
gamma (alias: min_split_loss): it’s another regularization parameter for tree pruning. It specifies the minimum loss reduction required to grow a tree. The default value is set at 0.
reg_alpha (alias: alpha): it is the L1 regularization parameter, increasing its value makes the model more conservative. Default is 0.
reg_lambda (alias: lambda): L2 regularization parameter, increasing its value also makes the model conservative. Default is 1.
Special use hyperparameters
The above set of parameters are “general purpose” parameters that you can always tune to optimize model performance. There are some special purpose hyperparameters that are used in particular situations:
scale_pos_weight: This parameter is useful in case you have an imbalanced dataset, particularly in classification problems, where the proportion of one class is a small fraction of total observations (e.g. credit card fraud). The default value is 1, but you can use the following ratio: total negative instance (e.g. no-fraud)/ total positive instance (e.g. fraud).
monotone_constraints: You can activate this parameter if you want to increase the constraint on the predictors, for example, a non-linear, increasing likelihood of credit-loan approval with a higher credit score.
booster: you can choose what kind of booster method to use. You have three options: ‘dart’, ‘gbtree ’ (tree-based) and ‘gblinear ’ (Ridge regression).
missing: it’s not missing value treatment exactly, it’s rather used to specify under what circumstances the algorithm should treat a value as missing (e.g. a negative value of the age of a customer certainly is impossible, thus the algorithm treats it as a missing value).
eval_metric: it specifies what loss function to use, e.g MAE, MSE, RMSE for regression and log loss for classification.
Just 5 years ago not many people knew about XGBoost (it was initially released in 2014) and now it is revolutionizing the field of data science and machine learning and applied in all kinds of industries. It is super simple to train XGBoost but the hardest part is parameter tuning. In this article I’ve shown a simple demo implementation along with a discussion on the most frequently used hyperparameters.
Thanks for reading. Feel free to subscribe to get notified of my medium articles or connect with me via Twitter, or LinkedIn.