Ensemble Learning case study: Running XGBoost on Google Colab free GPU | by Gabriel Signoretti | Towards Data Science

This article is the second part of a case study where we are exploring the 1994 census income dataset. In the first part, we took a deeper look at the dataset, compared the performance of some ensemble methods and then explored some tools to help with the model interpretability.
In this second part, we will explore a technique called Gradient Boosting and the Google Colaboratory, which is a free Jupyter notebook environment that requires no setup and runs entirely in the cloud.
Gradient Boosting is a type of ensemble method, much like Bagging and Pasting that we discussed last time. However, Boosting differs from the previously mentioned methods in relation to how it does such a combination of models. It uses techniques to combine several weak learners into one strong learner.
While the other ensemble methods rely on the simultaneous construction of several isolated base learners that are then used to make the prediction, Boosting algorithms rely on a sequential process, where each model attempts to correct the mistakes of previous models. This leads to a major efficiency drawback, as the models are created one at a time and depend on the results obtained by previous iterations
Gradient Boosting employs the gradient descent algorithm to minimize errors in sequential models. So, in essence, it is an optimization problem where the objective is to minimize the error (loss) function.
XGBoost (eXtreme Gradient Boosting), in its turn, is an optimized implementation of the Gradient Boosting algorithm. It features (among other enhancements): parallelized tree building, cache-aware access, sparsity awareness, regularization (both L1 and L2), weighted quantile sketch. The library has become one of the go-to solutions for winning many data science competitions in recent years.
XGBoost hyperparameters were divided into 3 categories by the original authors:
General Parameters: hyperparameters that control the overall functioning of the algorithm;
Booster Parameters: hyperparameters that control the individual boosters (tree or regression) at each step of the algorithm;
Learning Task Parameters: hyperparameters that configure the optimization to be performed;
One of XGBoost greatest advantages is the amount of customization available. To check the full list of available hyperparameters (it is quite extensive), I highly recommend you to check out the project documentation page.
XGBoost provides a scikit-learn compatible API and some parameters have slightly different names, but they work the same as in the regular library API.
We will briefly go over some of the hyperparameters that we will tune in our case study to get a better understanding of them:
booster: allows us to choose each booster to use: gbtree, for tree-based models, or gblinear, for linear models.
objective: is part of the Learning Task hyperparameters, and it specifies the learning task (regression, classification, ranking, etc) and function to be used in the optimization process.
tree_method: we will use the option “gpu_exact” to run on the GPU
eval_metric: the metric used to evaluate performance on the training data. We can pass multiple metrics in the format of a python list and, thus, we will use “error” (the binary classification error) and “auc” in the case study.
learning_rate (eta): it scales newly added weights after each step of tree boosting. This technique is called “shrinkage” and is responsible for reducing the influence of each individual tree, leaving space for future trees to improve the model. Generally, learning_rates lower than 0.1 are found to yield better generalization errors.
gamma: a tree node is split when the resulting split generates a positive reduction of the loss function. The gamma parameter specifies the minimum reduction required to perform such splits. Its value depends on the loss function being used.
max_depth: limits the maximum depth that the trees are allowed to grow. Larger trees are prone to overfitting.
colsample_bytree: sets the fraction of features (dataset columns) to be randomly used when fitting each individual tree.
subsample: sets the fraction of observations (dataset rows) to be randomly used when fitting each individual tree.
reg_alpha: controls the L1 regularization. Since L1 regularization tends to pull the weights to zero, it is more recommended when using linear boosters or data with high dimensionality.
reg_lambda: controls L2 regularization. Since L2 regularization encourages lower weights (but not necessarily 0), it can be used for tree boosters.
n_estimators: the number of trees to fit
As you can see, there are multiple hyperparameters to tune, and the ones presented above are not even close to the whole of it.
First, we are going to fit an out-of-the-box XGBoost classifier to get a basic understanding of the model’s performance, meaning that we are not going to tune most of the model’s parameters.
The XGBoost library implements two main APIs for model training: the default Learning API, which gives more fine control over the model; and the Scikit-Learn API, a scikit-learn wrapper that enables us to use the XGBoost model in conjunction with scikit-learn objects such as Pipelines and RandomizedSearchCV.
We are going to be primarily focusing on the Scikit-Learn API for now. This API provides a method to assess the model’s performance variation with the incremental addition of new trees. To do so we must provide the following additional arguments to the fit method: eval_set, the evaluation sets to be used (usually the train and test sets), and eval_metric, the metrics to be used when performing the evaluation. If these two arguments are provided, the evaluation results will be available by the eval_results attribute of the fitted model.
But enough talk, let’s finally show some code! As this article is the part 2 of the case study, we are not going to be discussing the development of the preprocessing pipeline steps, as that was already covered in the part 1 of the series.
To begin, we load the dataset, transform it into the X feature matrix and the y target vector, perform the preprocessing steps and, finally, split the data into training and test sets. This process is highlighted on the code below:
# load the datasetincome = pd.read_csv("income.csv")# Create the X feature matrix and the y target vectorX = income.drop(labels=["high_income", 'fnlwgt'], axis=1)y = income["high_income"]# the only step necessary to be done outside of pipeline# convert the target column to categoricalcol = pd.Categorical(y)y = pd.Series(col.codes)# validate the preprocessing pipeline by passing data through itclean_X = preprocessing_pipeline.fit_transform(X)clean_X_df = pd.DataFrame(clean_X, columns=X.columns)# split the clean_X into train and test setsX_train, X_test, y_train, y_test = train_test_split(clean_X, y, test_size=0.20, random_state=seed, shuffle=True, stratify=y)
With this, we are finally ready to train our first XGBoost classifier! To get a better sense of the model evolution, we will set the n_estimators hyperparameter to 500. The model training is highlighted on the code snippet below:
import timefrom xgboost import XGBClassifier# create a default XGBoost classifiermodel = XGBClassifier(n_estimators=500, random_state=seed)# define the eval set and metriceval_set = [(X_train, y_train), (X_test, y_test)]eval_metric = ["auc","error"]# fit the model%time model.fit(X_train, y_train, eval_metric=eval_metric, eval_set=eval_set, verbose=False)
The %time jupyter magic command returns the time execution of a Python statement or expression. In this case, the training took 11.2 seconds to be completed, not bad for a start.
Then we can measure the model performance by making some predictions:
# final model assessmentpred_test = model.predict(X_test)pred_train = model.predict(X_train)print('Train Accuracy: ', accuracy_score(y_train, pred_train))print('Test Accuraccy: ', accuracy_score(y_test, pred_test))print('Classification Report:')print(classification_report(y_test,pred_test))
The default model already gave us a better test accuracy than the tunned random forest from the last article! Let’s see how the model behaved with the incremental addition of new estimators:
# retrieve performance metricsresults = model.evals_result()epochs = len(results['validation_0']['error'])x_axis = range(0, epochs)fig, ax = plt.subplots(1, 2, figsize=(15,5))# plot aucax[0].plot(x_axis, results['validation_0']['auc'], label='Train')ax[0].plot(x_axis, results['validation_1']['auc'], label='Test')ax[0].legend()ax[0].set_title('XGBoost AUC-ROC')ax[0].set_ylabel('AUC-ROC')ax[0].set_xlabel('N estimators')# plot classification errorax[1].plot(x_axis, results['validation_0']['error'], label='Train')ax[1].plot(x_axis, results['validation_1']['error'], label='Test')ax[1].legend()ax[1].set_title('XGBoost Classification Error')ax[1].set_ylabel('Classification Error')ax[1].set_xlabel('N estimators')plt.show()plt.tight_layout()
Well, it looks like our model is performing fine until about 300 estimators, then the error on the train set continues to drop while the error on the test set more or less remains stable. That might be a sign of overfitting! One way to avoid this is by using the early_stopping_rounds parameter to stop execution once no improvement is observed after a specified number of rounds.
So, it looks like our default model performed quite well with its out-of-the-box configuration. However, we can still squeeze more out of it by tunning some hyperparameters, but how should we do that?
Imagine if we wanted to tune 7 hyperparameters and test 3 different values for each of them. Tunning that by hand is just out of the question due to the massive number of different possible combinations. If we then decide to use a naive grid search to do the job, we would end um with 3ˆ7=2187 different models to fit! In that case, if we consider the time it took to fit our default model (11,2 seconds) as a general rule of thumb, it would take around 408 minutes or 6,8 hours to train!
Therefore, as we did last time, we will be using a randomized search to tune the parameters, it might not find the best solution but it will find a “good enough” one in a reasonable amount of time.
Using scikit-learn’s RandomizedSeaarchCV we configured the search space to include the learning_rate, colsample_bytree, subsample, max_depth, n_estimators, reg_lambda, and gamma hyperparameters. We also limited the number of iterations to be 50 so that the experiment would not take too long to run. The process of configuring and fitting this model is shown below:
# create a default XGBoost classifiermodel = XGBClassifier(    random_state=seed,     eval_metric=["error", "auc"])# Create the grid search parameter grid and scoring funcitonsparam_grid = {    "learning_rate": [0.1, 0.01],    "colsample_bytree": [0.6, 0.8, 1.0],    "subsample": [0.6, 0.8, 1.0],    "max_depth": [2, 3, 4],    "n_estimators": [100, 200, 300, 400],    "reg_lambda": [1, 1.5, 2],    "gamma": [0, 0.1, 0.3],}scoring = {    'AUC': 'roc_auc',     'Accuracy': make_scorer(accuracy_score)}# create the Kfold objectnum_folds = 10kfold = StratifiedKFold(n_splits=num_folds, random_state=seed)# create the grid search objectn_iter=50grid = RandomizedSearchCV(    estimator=model,     param_distributions=param_grid,    cv=kfold,    scoring=scoring,    n_jobs=-1,    n_iter=n_iter,    refit="AUC",)# fit grid search%time best_model = grid.fit(X_train,y_train)
Again we use the %time command to measure the execution speed of the training process. This time it took approximately 20 minutes and 19 seconds to return the best model after the execution of the 50 rounds. You can now have an idea of how this process can become really time-consuming quite fast if we used a larger search space and a bigger number of iterations.
We can then check the best model AUC score and hyperparameters:
print(f'Best score: {best_model.best_score_}')print(f'Best model: {best_model.best_params_}')
And finally, we can make some predictions to assess the model overall performance:
pred_test = best_model.predict(X_test)pred_train = best_model.predict(X_train)print('Train Accuracy: ', accuracy_score(y_train, pred_train))print('Test Accuraccy: ', accuracy_score(y_test, pred_test))print('\nConfusion Matrix:')print(confusion_matrix(y_test,pred_test))print('\nClassification Report:')print(classification_report(y_test,pred_test))
It looks like we got a little improvement on both train and test accuracies, but not that much to be honest.
We finally arrived at the section you were probably waiting for! We will use the Google Colaboratory free GPU access to hopefully speed up the process of fitting the XGBoost model.
But how do you configure your workspace? It’s quite a simple two-step process: first, you need to access edit > notebook preferences, then, on the hardware accelerator dropdown menu select GPU and press Save. That’s it! your runtime will be restarted and then will be ready to go.
On the XGBoost side, the only thing that changes is the inclusion of the tree_method hyperparameter while creating the model, everything else for the randomized search remains the same! The change on the model creatin can be seen on the code snippet below:
model = XGBClassifier(    tree_method = "gpu_hist",     random_state=seed,     eval_metric=["error", "auc"])
After retraining the model with this configuration, we verified that it took around 18 minutes to complete the whole process. That might not seem like a great improvement, but it was around 10% faster than the previous run! The gains of using the GPU can be more easily seen on larger datasets, where the parallelization overhead would be more compensated by the parallelization speed gains.
After that, we repeated the process to check the best model parameters and performance. The results are shown in the figure below:
We can see another slight increase in training and test accuracies, that’s always great, isn’t it!?
So, to help make more sense of the XGBoost model predictions, we can use any of the techniques presented in the last part of this series: inspect and plot the feature_importances_ attribute of the fitted model; use the ELI5 feature weights table and prediction explanations; and, finally, use SHAP plots.
However, the XGBoost library has another trick up its sleeve and provides a built-in plotting API for generating plots of feature importances and representations of individual trees used in the boosting!
To plot the global feature importance we can use the plot_importances method. It is possible to specify one of three metrics to calculate the individual feature scores:
Feature Weights: the score is calculated based on the number of times the feature appears in the trees
Gain: the score is calculated based on the average gain of splits which use the feature
Coverage: the score is calculated based on the average coverage (number of samples affected by the split) of splits which use the feature
The three examples are shown in the code below:
# store the winning model in a new variablexgc = best_model_gpu.best_estimator_# saving the feature names to the modelxgc.get_booster().feature_names = X.columns.to_list()# Create the feature importances plotfig, ax = plt.subplots(1, 3, figsize=(15,5))# plot importances with feature weightxgb.plot_importance(    booster=xgc,     importance_type='weight',    title='Feature Weight',    show_values=False,    height=0.5,    ax=ax[0],)# plot importances with split mean gainxgb.plot_importance(    booster=xgc,    importance_type='gain',    title='Split Mean Gain',    show_values=False,    height=0.5,    ax=ax[1])# plot importances with sample coveragexgb.plot_importance(    xgc,    importance_type='cover',    title='Sample Coverage',    show_values=False,    height=0.5,    ax=ax[2])plt.tight_layout()plt.show()
And, finally, to plot a specific tree we can use the plot_tree method as shown below:
# Create the feature importances plotfig, ax = plt.subplots(figsize=(20,20))# plot a decision tree from the boosterxgb.plot_tree(booster=xgc, num_trees=0, ax=ax, rankdir='LR')plt.tight_layout()plt.show()
With this, we make it to the end of this two-part series on ensemble learning, I hope it was enjoyable and maybe a little bit useful!
Thanks for reading! leave a comment with your thoughts below.