XGBoost: theory and practice. Understand how one of the most popular... | by Arthur Mello | Towards Data Science

XGBoost stands for eXtreme Gradient Boosting and it’s an open-source implementation of the gradient boosted trees algorithm. It has been one of the most popular machine learning techniques in Kaggle competitions, due to its prediction power and ease of use. It is a supervised learning algorithm that can be used for regression or classification tasks.
Regardless of its futuristic name, it’s actually not that hard to understand, as long as we first go through a few concepts: decision trees and gradient boosting. If you are already familiar with those, feel free to skip to “How XGBoost works”.
Decision trees are arguably the most easily interpretable ML algorithms you can find and, if used in combination with the right techniques, can be quite powerful.
A decision tree has this name because of its visual shape, which looks like a tree, with a root and many nodes and leaves. Imagine you take a list of the Titanic’s survivors, with some information such as their age and gender, and a binary variable telling who survived the disaster and who didn’t. You now want to create a classification model, to predict who will survive, based on this data. A very simple one would look like this:
As you can see, decision trees are just a sequence of simple decision rules that, combined, produce a prediction of the desired variable.
Boosting is an ensemble method, meaning it’s a way of combining predictions from several models into one. It does that by taking each predictor sequentially and modelling it based on its predecessor’s error (giving more weight to predictors that perform better):
Fit a first model using the original dataFit a second model using the residuals of the first modelCreate a third model using the sum of models 1 and 2
Fit a first model using the original data
Fit a second model using the residuals of the first model
Create a third model using the sum of models 1 and 2
Gradient boosting is a specific type of boosting, called like that because it minimises the loss function using a gradient descent algorithm.
Now that you understand decision trees and gradient boosting, understanding XGBoost becomes easy: it is a gradient boosting algorithm that uses decision trees as its “weak” predictors. Beyond that, its implementation was specifically engineered for optimal performance and speed.
Historically, XGBoost has performed quite well for structured, tabular data. If you are dealing with non-structured data such as images, neural networks are usually a better option.
What are the most important hyperparamaters to choose when implementing XGBoost, and how to tune them?
booster is the boosting algorithm, for which you have 3 options: gbtree, gblinear or dart . The default option is gbtree , which is the version I explained in this article. dart is a similar version that uses dropout techniques to avoid overfitting, and gblinear uses generalized linear regression instead of decision trees.
reg_alpha and reg_lambda are L1 and L2 regularisation terms, respectively. The greater these numbers, the more conservative (less prone to overfitting but might miss relevant information) the model becomes. Recommended values lie between 0–1000 for both.
max_depth sets the maximum depth of the decision trees. The greater this number, the less conservative the model becomes. If set to 0, then there is no limit for trees’ depth.
subsample is the size of the sample ratio to be used when training the predictors. Default is 1, meaning there is no sampling and we use the whole data. If set to 0.7, for instance, then 70% of the observations would be randomly sampled to be used in each boosting iteration (a new sample is taken for each iteration). It can help to prevent overfitting.
num_estimators sets the number of boosting rounds, which equals setting the number of boosted trees to use. The greater this number, the greater the risk of overfitting (but low numbers can also lead to low performance).
To show how XGBoost works in practice, let’s do a simple exercise by trying to actually predict Titanic survivors on a Kaggle competition, using Python.
After downloading our data from Kaggle, we’ll import all the necessary libraries and our training data:
IN:import pandas as pdfrom xgboost import XGBClassifierfrom sklearn.model_selection import train_test_splitfrom sklearn.metrics import accuracy_scoredf = pd.read_csv("C:/Users/p005520/Downloads/titanic/train.csv")
Please note from xgboost import XGBClassifier . That only works because we have previously installed xgboost on our computer by runningpip install xgboost from our terminal. XGBClassifier is used here because it’s a classification problem. For regression problems, use XGBRegressor instead. The other libraries used for treating our data and calculating performance metrics.
IN:dummies = pd.get_dummies(df['Sex'])df = pd.concat([df, dummies], axis=1)X = df[[‘Age’,’female’,’male’]]y = df[‘Survived’]seed = 42test_size = 0.3X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=test_size, random_state=seed)
We have many variables in our data but, this time, we’ll stick to the same two from our previous example: ‘Sex’ and ‘Age’.
The first two lines of this block create dummy variables from ‘Sex’. This is needed to turn ‘Sex’ from string to integers, becoming two different variables: ‘male’ and ‘female’, which are equal to 1 or 0 depending on the passenger’s sex.
The next 2 lines define our target variable (‘Survived’) and the variables we’ll use for predicting it. The last 4 lines are for splitting our train and test sets: our train set will be used to create our XGBoost model, while the test set will be used to measure its performance.
IN:model = XGBClassifier(subsample = 0.7, max_depth = 4)model.fit(X_train, y_train)print(model)y_pred = model.predict(X_test)accuracy = accuracy_score(y_test, y_pred)print(“Accuracy: %.2f%%” % (accuracy * 100.0))OUT:XGBClassifier(base_score=0.5, booster='gbtree', colsample_bylevel=1,              colsample_bynode=1, colsample_bytree=1, gamma=0,              learning_rate=0.1, max_delta_step=0, max_depth=4,              min_child_weight=1, missing=None, n_estimators=100, n_jobs=1, nthread=None, objective='binary:logistic', random_state=0,              reg_alpha=0, reg_lambda=1, scale_pos_weight=1, seed=None,silent=None, subsample=1, verbosity=1)Accuracy: 80.60%
Here’s where we actually train our XGBoost, in the first 2 lines. Note subsample = 0.7 and max_depth = 4 where I’m manually defining some of my hyperparameters. If you want to set new hyperparameters differently from the default option, just add them to that list. Our model yields a 80.6% accuracy in the test set, now let’s see how to apply it to new data:
submission_input = pd.read_csv("C:/Users/p005520/Downloads/titanic/test.csv")dummies = pd.get_dummies(submission_input[‘Sex’])submission_input = pd.concat([submission_input, dummies], axis=1)submission_X = submission_input[[‘Age’,’female’,’male’]]submission = model.predict(submission_X)submission_df = pd.DataFrame({‘PassengerId’:submission_input[‘PassengerId’], ‘Survived’:submission})submission_df.to_csv(‘submission.csv’, index = False)
In this block we have imported Kaggle’s submission file, done the same dummy treatment as in our training set, applied our trained predictor with model.predict(submission_X) and then saved it to a csv. How does it score on Kaggle’s leaderboard?
As you can see, we are far from the top of the leaderboard but we have still managed to predict survival correctly for 76% of people, using only two features, no feature engineering and no hyperparameter tuning, which would be the logical next steps for our work.
I hope you now understand how XGBoost works and how to apply it to real data. Despite XGBoost’s inherent performance, hyperparameter tuning and feature engineering can make a huge difference in your results.
If you want to learn more about feature engineering to improve your predictions, you should read this article, which outlines the main techniques you can use to transform your variables and create new ones:
towardsdatascience.com
Feel free to reach out to me on LinkedIn if you would like to discuss further, it would be a pleasure (honestly).