Foray into Feature Selection: How Accuracy Improved by Selecting Features in Kaggle’s Ames House Price Competition Question
Tags: Python, Data Science, Kaggle, Machine Learning, Artificial Intelligence

Foray into Feature Selection: How Accuracy Improved by Selecting Features in Kaggle’s Ames House Price Competition Question Tracyrenee Follow Oct 3, 2020 · 4 min read

In my last post I discussed how I was able to obtain an accuracy of 94.22% on the accuracy of Kaggle’s Ames House Price competition question by using the label encoding function in sklearn’s preprocessing library.

Whilst demonstrating the results I received by label encoding the object columns in the datasets and using an appropriate model to train the datasets, fit them, and make predictions, I pondered whether I could increase the accuracy even further by selecting the features that will give optimum performance to the model.

I therefore, through trial and error, dropped several columns from training and testing functions of the model and was able to achieve an accuracy of 95.95% on the validation set.

When I predicted on the test set and submitted those predictions to Kaggle, my scores were slightly lower, which is unfortunately a consequence of training and testing on the training dataset instead of the testing dataset.

Below is a description of my methodology in attaining the accuracy that I did:-

The problem statement for this competition question is stated as:-

Ask a home buyer to describe their dream house, and they probably won’t begin with the height of the basement ceiling or the proximity to an east-west railroad. But this playground competition’s dataset proves that much more influences price negotiations than the number of bedrooms or a white-picket fence.

With 79 explanatory variables describing (almost) every aspect of residential homes in Ames, Iowa, this competition challenges you to predict the final price of each home.

The first step was to load the libraries into the program:-

The next step was to import the datasets into Jupyter notebooks:-

I then loaded and read the training and testing datasets:-

I checked for null values in both the training and testing datasets:-

I then imputed the missing values by replacing them with the mode:-

I further filled missing values in the training and test sets with ‘None’ or zero:-

I checked to see which columns in the training and test sets are objects because these were label encoded:-

All of the columns in the training and testing datasets were then put into a label encoder so they would be converted to numerical data, which is that the CPU wants to see:-

I then defined X, y and X_test. I dropped the columns that were having a detrimental effect on the accuracy of the validation set. I determined which columns by dropping them and then running the program. If accuracy improved the column was dropped from the features to be trained, fit, and predicted upon:-

I split the training dataset into four sections so it could be trained, fit and tested on. I then checked the shape of the four datasets created:-

I then selected XGB Regressor as the model because it gave me a high accuracy score, being 99.97% of the training set:-

I predicted on the validation set and obtained 95.59% accuracy!

I then used the trained model to predict on the test dataset:-

I put the predictions on a dataframe and submitted it to Kaggle:-

Unfortunately, the score was not as high as it had been before I had employed feature selection, but the reason for this is because I trained the data on the training dataset. Kaggle only allows me to make ten submissions a day and it would have been totally impossible for me to feature select 79 columns of data and test them all in one day because I would not be allowed to make 79 submissions in one day.

The Jupyter notebook for this post has been made public and you can view it at:-

https://www.kaggle.com/tracyporter/ames-house-prices-xgb-regressor?scriptVersionId=43959901