How I used a pipelined Gradient Boosting Regression to solve Ames House Price Competition Question
Tags: Python, Gradient Boosting, Kaggle, Machine Learning, Pipeline

How I used a pipelined Gradient Boosting Regression to solve Ames House Price Competition Question Tracyrenee Follow Nov 5, 2020 · 4 min read

The Kaggle Ames House Price Prediction competition is a very good competition to experiment on because it contains relatively small train and test files that can easily be predicted upon. Because the files are small, there is little chance of crashing the system when attempting to run a program, which has happened to me on numerous occasions.

Because the Ames House Price train and test datasets do not put undo burden on a computer system, they are an excellent choice when conducting experiments to attempt to find the best algorithm to use to attain an optimum accuracy.

In this blog post I would like to illustrate how I used sklearn’s GradientBoostinRegressor() to achieve a very good score in the competition. Gradient boosting is a machine learning technique for regression and classification problems, which produces a prediction model in the form of an ensemble of weak prediction models, typically decision trees. It builds the model in a stage-wise fashion like other boosting methods do, and it generalizes them by allowing optimization of an arbitrary differentiable loss function.

I know of two additional boosting models, being XGBoost and CatBoost, but in this post I will be using sklearn’s version of the model. Sklearn is a very good library to use because it’s user documentation is superior, giving numerous examples of how to use its functions.

The problem files for the Kaggle Ames House Price competition can be found in my github account:- https://github.com/TracyRenee61/House-Prices

Excerpts from the problem statement are:-

Ask a home buyer to describe their dream house, and they probably won’t begin with the height of the basement ceiling or the proximity to an east-west railroad. But this playground competition’s dataset proves that much more influences price negotiations than the number of bedrooms or a white-picket fence.

With 79 explanatory variables describing (almost) every aspect of residential homes in Ames, Iowa, this competition challenges you to predict the final price of each home.

The Ames Housing dataset was compiled by Dean De Cock for use in data science education. It’s an incredible alternative for data scientists looking for a modernized and expanded version of the often cited Boston Housing dataset.

The first thing I did was to import the standard libraries into the Kaggle’s Jupyter notebook, which is a notebook that can be freely used to write programs:-

I then imported the operating system and read the files that were used to complete the competition question:-

I then checked for any null values in both the train and test datasets. Because there are 79 columns of independent variables, there were quite a few null values:-

I replaced the missing values with the text ‘Not Listed’ in the object columns. In the numerical columns I replaced the null value with the median.

In the past I had replaced the null values with mode() but felt it ultimately detracted from the accuracy of the model:-

I used OrdinalEncoder() to encode the object columns. Sklearn documentation states that LabelEncoder() should only be used to encode the target variable, which is why I used OrdinalEncoder():-

I defined X, y and X_test variables using feature selection. I dropped some of the columns that detracted from the accuracy of the model:-

I split the train set up for training and validation, stipulating the validation set is 10% of the original train set:-

I used the make_pipeline() command, which is an abbreviated version of Pipeline(). I put the StandardScaler() and GradientBoostingRegressor() in the pipeline and when I fit it I achieved an accuracy of 99.96%:-

When I predicted on the validation set I achieved an accuracy of 95.21% and a r2_score of 99.99%:-

I then predicted upon the test dataset and prepared it for prediction by creating a dataframe and converting it into a csv file:-

When I submitted the submission.csv file to Kaggle, I attained a score of 15.314%, which was not a winning cross but high nevertheless:-

The code of this post can be found in its entirety in the Kaggle website:- https://www.kaggle.com/tracyporter/ames-house-prices-gradientboostingregressor?scriptVersionId=46204265