Towards Data Science
By: Rukshan Pramoditha
Tags: Regression, Xgboost, Data Science, Supervised Learning, Machine Learning

Welcome to the 3rd article of “A Journey through XGBoost” series. Up to now, we have completed 2 milestones. What we have done so far is that, basically, we discussed how to set up the system to run XGBoost on our own computers, and also we did a classification task with XGBoost and created a small (but useful) web app to communicate our results with end-users who don’t have much technical knowledge. If you come here for the very first time, I recommend you to read the first two articles of “A Journey through XGBoost” series. Here are the links.

More specially, toady, we will do a regression task on the “Boston house-prices” dataset with XGBoost. Here are the topics we discuss today.

Topics we discuss

Form a regression problem

Build an XGBoost regression model (Scikit-learn compatible API)

Describe ‘RMSE’ and ‘R-squared’ metrics

and metrics Create prediction error and residuals plots

Explain XGBoost Regressor hyperparameters

hyperparameters XGBoost’s objective function

Apply L2 regularization to our XGBoost model

The Boston house-prices dataset

The “Boston house-prices” dataset is a built-in dataset in Scikit-learn. To access the data, all you need to do is calling the load_boston() function and assign it to a variable called data which is a Python object. Then we call various properties of that object to get X (feature matrix), y (target vector) and column names. When we write the code, you will see how to do that. For now, just look at the structure and variable information of the “Boston house-prices” dataset.

The first 5 rows of the “Boston house-prices” dataset (Image by author)

The following image contains the variable information of the dataset returned by the Pandas DataFrame info() method.

Variable information on the “Boston house-prices” dataset (Image by author)

The dataset has no missing values. All the values are numerical. So, no preprocessing step requires and the dataset is ready to use. This dataset has 506 observations and 13 features (not including the “target” column). The target column (which can be accessed using the target attribute of data) contains the median house prices. To find more information about this dataset, please read its documentation.

Define the problem

Based on CRIM, ZN,…, LSTAT, we want to predict the median house prices of given houses (new instances). This is a regression task because its model predicts a continuous-valued output (house prices are continuous values).

The algorithm that we use to solve this regression problem is XGBoost. The XGBoost model for regression is called XGBRegressor. So, we will build an XGBoost model for this regression problem and evaluate its performance on test data (unseen data/new instances) using the Root Mean Squared Error (RMSE) and the R-squared (R²-coefficient of determination). We will also use various graphical techniques such as prediction error plot, residual plot and distribution of residuals to evaluate the regression model and verify its assumptions.

Let’s get hands-on experience by writing the Python code to build an XGboost regression model on the “Boston house-prices” dataset.

Building the XGboost regression model

Here, we use the XGBoost Scikit-learn compatible API. “Scikit-learn compatible” means that you can use the Scikit-learn .fit() / .predict() paradigm and almost all other Scikit-learn classes with XGBoost. Here is the code.

Wait till loading the Python code! (Code Snippet-1)

The output of the above code segment is:

The output of the Code Snippet-1

Note that how I have used several print() functions to print multiple values and graphs in the same output. To create the prediction error and residuals plots, I have used only two lines of code. Here, I have taken the advantage of the Yellowbrick Python library. Yellowbrick is an excellent machine learning visualization library. I recommend you to use it because you can make fancy visualizations with just one or two lines of code and it is fully compatible with Scikit-learn. If you’re interested, read its documentation.

Our model returns an RMSE value of 2.86 (in y-units) for the test data. Is that a good value? To find out, let’s look at some statistical measures of the target column (house prices).

Statistical measures of the target column

With the standard deviation of 9.197 and the mean of 22.53, the RMSE value (2.86) that we got is very good. The smaller the RMSE value, the better the model. On average, the price predictions of our model are 2.86 units away from the actual values.

R² value for our model is 0.89. It means that 89% of the variability observed in house prices is captured by our model and the other 11% is due to some other factors and, of course, randomness!

As graphical methods, we can use prediction error plot, residuals plot and distribution of residuals to evaluate our regression model and verify its assumptions.

Prediction error plot: We can see that most of the points are on a straight line. We can compare this plot against the 45-degree line, where the prediction exactly matches the model. In general, the predictions follow the actual house prices.

We can see that most of the points are on a straight line. We can compare this plot against the 45-degree line, where the prediction exactly matches the model. In general, the predictions follow the actual house prices. Residuals plot: We cannot see any pattern between predictions and residuals. So, we can verify that the residuals are uncorrelated or independent. This is a good sign for our model.

We cannot see any pattern between predictions and residuals. So, we can verify that the residuals are uncorrelated or independent. This is a good sign for our model. Distribution of residuals plot: By looking at this plot, we can verify that the residuals (actual values-predicted values) are approximately normally distributed.

The model we created follows the standard regression assumptions.

Now, it’s time to describe the XGBRegressor() hyperparameters. We have specified 7 hyperparameters.

max_depth=3: Here, the XGBoost uses decision trees as base learners. By setting max_depth=3 , each tree will make 3 times of splits and stop there.

Here, the XGBoost uses decision trees as base learners. By setting , each tree will make 3 times of splits and stop there. n_estimators=100: There are 100 trees (individual models) in the ensemble.

There are 100 trees (individual models) in the ensemble. objective=’reg:squarederror’: A name for the loss function used in our model. reg:squarederror is the standard option for regression in XGBoost. You may also use reg:linear which gives the same result. But reg:linear is now deprecated and will be removed in a future version.

A name for the loss function used in our model. is the standard option for regression in XGBoost. You may also use which gives the same result. But is now deprecated and will be removed in a future version. booster=’gbtree’: This is the type of base learner that the ML model uses every round of boosting. ‘gbtree’ is the XGBoost default base learner. With booster=‘gbtree ’, the XGBoost model uses decision trees, which is the best option for non-linear data.

This is the type of base learner that the ML model uses every round of boosting. is the XGBoost default base learner. With ’, the XGBoost model uses decision trees, which is the best option for non-linear data. n_jobs=2: Use 2 cores of the processor for doing parallel computations to run XGBoost.

Use 2 cores of the processor for doing parallel computations to run XGBoost. random_state=1: Controls the randomness involved in creating tress. You may use any integer. By specifying a value for random_state , you will get the same result at different executions of your code.

Controls the randomness involved in creating tress. You may use any integer. By specifying a value for , you will get the same result at different executions of your code. learning_rate=0.05: Shrinks the weights of trees for each round of boosting. Decreasing learning_rate prevents overfitting.

XGBoost’s objective function

The objective function of XGBoost determines how far a prediction is from the actual value. Our goal is to find a model that gives the minimum value for the objective function. XGBoost’s objective function consists of two parts:

The loss function

The regularization term

Mathematically, this can be represented as:

XGBoost’s objective function (Image by author)

The loss function

For a regression problem, the loss function is the Mean Squared Error (MSE). For a classification problem, the loss function is the Log Loss. The hyperparameter values for the common loss functions in XGBoost are:

reg:squarederror or reg:linear — for regression

or — for regression binary:logistic — for binary classification

— for binary classification multi:softprob — for multi-class classification

The regularization term

The regularization can be defined as the “control on model complexity”. We need to create a model considering both accuracy and simplicity. The regularization term is a penalty term to prevent overfitting the model. The main difference between XGBoost and other tree-based models is that XGBoost’s objective function includes a regularization term. The regularization parameters in XGBoost are:

gamma: The default is 0. Values of less than 10 are standard. Increasing the value prevents overfitting.

The default is 0. Values of less than 10 are standard. Increasing the value prevents overfitting. reg_alpha: L1 regularization on leaf weights. Larger values mean more regularization and prevent overfitting. The default is 0.

L1 regularization on leaf weights. Larger values mean more regularization and prevent overfitting. The default is 0. reg_lambda: L2 regularization on leaf weights. Increasing the value prevents overfitting. The default is 1.

Apply L2 regularization to our XGBoost model

Now, we apply L2 regularization to the XGBoost model we created above. We will try 4 different values for the reg_lambda hyperparameter. We can do it with a simple for loop.

Wait till loading the Python code! (Code Snippet-2)

The output is:

L2 regularization effect on our XGBoost model

Here, we can notice that as the value of ‘lambda’ increases, the RMSE increases and the R-squared value decreases.

Summary

So far, We have completed 3 milestones of the XGBoost series. Today, we performed a regression task with XGBoost’s Scikit-learn compatible API. As we did in the classification problem, we can also perform regression with XGBoost’s non-Scikit-learn compatible API.

In the next article, I will discuss how to perform cross-validation with XGBoost. Stay tuned for the updates!

Thanks for reading!

This tutorial was designed and created by Rukshan Pramoditha, the Author of Data Science 365 Blog.

Read my other articles at https://rukshanpramoditha.medium.com

2021–03–11