Implementing an XGBoost Model in R | by Michael Grogan | Towards Data Science

In this example, an XGBoost model is built in R to predict incidences of customers cancelling their hotel booking. The analysis is based on data from Antonio, Almeida and Nunes (2019): Hotel booking demand datasets.
The H1 dataset is used for training and validation, while H2 is used for testing purposes.
In order to predict customers that will cancel their booking (where variable IsCanceled = 1 means a cancellation, and IsCanceled = 0 means the customer follows through with the booking), an XGBoost model is built in R with the following features:
Lead time
Country of origin
Market segment
Deposit type
Customer type
Required car parking spaces
Week of arrival
In order to make the data suitable for analysis with the XGBoost model in R — some data manipulation procedures are required.
Firstly, the xgboost and Matrix libraries are loaded:
require(xgboost)library(Matrix)
A data frame of features is formed through defining the variables as.numeric, and also defining in factor format where appropriate. The data frame is then converted into Matrix format.
leadtime<-as.numeric(H1$LeadTime)country<-as.numeric(factor(H1$Country))marketsegment<-as.numeric(factor(H1$MarketSegment))deposittype<-as.numeric(factor(H1$DepositType))customertype<-as.numeric(factor(H1$CustomerType))rcps<-as.numeric(H1$RequiredCarParkingSpaces)week<-as.numeric(H1$ArrivalDateWeekNumber)df<-data.frame(leadtime,country,marketsegment,deposittype,customertype,rcps,week)attach(df)df<-as.matrix(df)
The labels (IsCanceled) is defined as a factor.
IsCanceled<-as.numeric(factor(H1$IsCanceled))
However, R labels the 0 values as 1 with the 1 values labelled as 2.
In order for the data to work with XGBoost, 0 and 1 labels must be defined.
Therefore, the label values are replaced accordingly as follows:
IsCanceled[IsCanceled == "1"] <- "0"IsCanceled[IsCanceled == "2"] <- "1"
The training and validation sets are split, and each is defined as a dgCMatrix — which is a special type of matrix that works with the XGBoost model in R.
train <- df[1:32000,]val <- df[32001:40060,]train=as(train, "dgCMatrix")trainval=as(val, "dgCMatrix")valIsCanceled_train=IsCanceled[1:32000]IsCanceled_val=IsCanceled[32001:40060]
The boosting model is defined and predictions are generated:
bst <- xgboost(data = train, label = IsCanceled_train, max.depth = 2, eta = 1, nthread = 2, nrounds = 2, objective = "binary:logistic")pred <- predict(bst, val)
Here are the training and validation errors:
> bst <- xgboost(data = train, label = IsCanceled_train, max.depth = 2, eta = 1, nthread = 2, nrounds = 2, objective = "binary:logistic")[1] train-error:0.252000 [2] train-error:0.222188> pred <- predict(bst, val)> print(length(pred))[1] 8060> print(head(pred))[1] 0.31375486 0.31375486 0.31375486 0.31375486 0.01498148[6] 0.01498148> prediction <- as.numeric(pred > 0.5)> print(head(prediction))[1] 0 0 0 0 0 0> err <- mean(as.numeric(pred > 0.5) != IsCanceled_val)> print(paste("val-error=", err))[1] "val-error= 0.1287841191067"
From the above output, we can see that the validation error of 0.12 is significantly lower than the two reported training errors of 0.25 and 0.22.
An importance matrix is also generated to identify the most important influencing factors on hotel cancellations:
# Importance Matriximportance_matrix <- xgb.importance(model = bst)print(importance_matrix)xgb.plot.importance(importance_matrix = importance_matrix)
Here is a more detailed look at the importance matrix:
> print(importance_matrix)         Feature      Gain     Cover Frequency1:       country 0.3589183 0.2266091 0.33333332:      leadtime 0.2767732 0.2807021 0.16666673:          rcps 0.2198606 0.2967366 0.33333334: marketsegment 0.1444479 0.1959522 0.1666667
As elaborated in the following StackExchange post — gain, cover and frequency illustrate the following:
Gain: Illustrates the contribution of a feature for each tree in the model, with a higher value illustrating greater importance for predicting the outcome variable.
Cover: Number of relative observations related to the feature in question.
Frequency: Number of times a feature occurs across the trees in the model.
The gain is the most important feature in assessing the relative contribution of a feature to the model. With a gain of 0.35 for the country variable, this is indicated as the most important feature in the model, i.e. a customer’s country of origin will have a significant impact in determining whether or not they ultimately cancel their hotel booking.
From looking at the training and validation errors, it looks as though the model may have performed quite well in predicting cancellations for this particular validation set, but there is no guarantee that it will do so on other datasets.
For this reason, a separate test set (H2) is used to verify if the XGBoost model will still perform well in predicting cancellation incidences.
The variables across the H2 set are defined once again:
leadtime<-as.numeric(H2$LeadTime)country<-as.numeric(factor(H2$Country))marketsegment<-as.numeric(factor(H2$MarketSegment))deposittype<-as.numeric(factor(H2$DepositType))customertype<-as.numeric(factor(H2$CustomerType))rcps<-as.numeric(H2$RequiredCarParkingSpaces)week<-as.numeric(H2$ArrivalDateWeekNumber)test<-data.frame(leadtime,country,marketsegment,deposittype,customertype,rcps,week)attach(test)test<-as.matrix(test)test=as(test, "dgCMatrix")testIsCanceled_H2<-as.numeric(factor(H2$IsCanceled))IsCanceled_H2[IsCanceled_H2 == "1"] <- "0"IsCanceled_H2[IsCanceled_H2 == "2"] <- "1"IsCanceled_H2<-as.numeric(IsCanceled_H2)
The XGBoost model built using the H1 dataset is now used to incorporate the feature data across H2:
pred <- predict(bst, test)
Here is the yielded test error:
> print(length(pred))[1] 79330> print(head(pred))[1] 0.3137549 0.7707853 0.7707853 0.7707853 0.7707853 0.7707853> > prediction <- as.numeric(pred > 0.5)> print(head(prediction))[1] 0 1 1 1 1 1> > err <- mean(as.numeric(pred > 0.5) != IsCanceled_H2)> print(paste("test-error=", err))[1] "test-error= 0.274335056094794"
The test error of 0.27 is higher than the validation error of 0.12, which we would reasonably expect when predicting new data.
That said, the original training errors of 0.25 and 0.22 are just slightly lower than the test error — indicating that the model does not show evidence of overfitting. Should we encounter a situation where the training error is much lower than the test error — then this is clearly a problem as it indicates that the model performs well in predicting the data used to build the model but not new data.
In this example, you have seen how to run an XGBoost model with R.
Specifically, you have seen:
How to transform data into dgCMatrix format in order to work with R’s XGBoost model
How to visualise feature importance through an Importance Matrix
Run an XGBoost model on test data to verify model accuracy
Many thanks for reading, and any questions or feedback are greatly appreciated. You can find the relevant repository for this example with the R code and dataset, as well as other useful references below.
Antonio, Almeida and Nunes (2019): Hotel booking demand datasets
Data Science StackExchange: How to interpret the output of XGBoost importance?
GitHub: MGCodesandStats/hotel-modelling
XGBoost Documentation: XGBoost R Tutorial