Kaggle Basics: Walkthrough of My First Data Science Challenge
By: Shikhar Gupta
Tags: Machine Learning, Regression, Data Visualization, Data Science, Kaggle

Kaggle Basics: Walkthrough of My First Data Science Challenge

Kickstarter pack for beginners!

Photo by Ronaldo de Oliveira on Unsplash

The outbreak of COVID-19 pandemic has forced the whole world to bring major changes to their lifestyle by being indoors all the time. With all the extra time in hand, saved from commute and outings, I decided to pursue things I never could otherwise. One of them was Kaggle.

In all of my previous projects, I had worked on visual datasets so wanted to try my hand on something different. I wanted to develop an understanding of the complete pipeline, starting from data cleaning, to various transformations, to feature selection and finally Machine Learning modelling. To do so, I picked up a beginner level challenge with an extensive dataset: House Prices: Advanced Regression Techniques. I set a target of getting under 1k (~top 20%).

As expected, 90% of my time went into data cleaning and feature engineering and the rest 10% on applying Machine Learning models onto them. I understood how important it is to ‘know your data’. Blindly feeding it to the SOTA model without any data engineering will only take you so far. I had to read a lot of blogs and Kaggle notebooks but the one I religiously followed was this. In this one, I’ll give an outline of the process I followed in very simple words and with examples so that beginners like me could get a sense of where to start.

For reference, here are my notebooks with the complete code so that you could follow along with the explanations —

Data cleaning and feature engineering: https://github.com/Shikhargupta/kaggle_house_pricing/blob/master/Feat_Engg.ipynb

Feature importance and modelling: https://github.com/Shikhargupta/kaggle_house_pricing/blob/master/Ensemble.ipynb

The Ames Housing Dataset

Photo by Breno Assis on Unsplash

Before getting into the process, let’s get an overview of the dataset in question. It has 80 columns (features) each describing a particular aspect of the house (e.g. neighbourhood, utilities, roof type etc.) which may or may not play a role in deciding its price (target variable). It consists of a total of 2920 observations, 1460 of which will be used for fitting (training data) and the rest for determining how good is your model performing (test data). Our goal is to predict the SalePrice for a house as closely as possible by taking the features (might not be all of them) in consideration. I would suggest you go through the data description to get accustomed to what each field means. The criteria for evaluation is Root-Mean-Squared-Error (RMSE) between the logarithm of the predicted value and the logarithm of the observed sales price. Lower the score, better is your model at prediction!

Data Cleaning (Outliers and NAN’s)

Photo by The Creative Exchange on Unsplash

First and foremost, we’ll clean our dataset to avoid any errors or discrepancies when we feed it to our model.

Always keep in mind that the end goal is to train our machine learning model and the data should be prepared in the same way. Be aware of how any action we perform might at the end improve or degrade the efficiency of the regressor/classifier.

For example, the presence of a NAN field might send the training routine for a toss and throw an error. Another killer of our prediction could be an outlier. The housing data we have was collected and stored manually and is prone to human error. An obvious large (20 bedrooms) or small (10 sq. ft. of LotArea) observation could sway our model in the wrong direction and affect our score. We either need to get rid of these data points or impute the values appropriately.

As far as Ames Housing dataset is concerned, we are good on the outliers front. The author has already pointed out the suspected outliers in the paper which I have visualised here in this plot

Data points with GrLivArea > 4000 are outliers due to abnormally high SalePrices and LivArea and shall be removed. Although there is nothing else mentioned, I found one discrepancy in the feature ‘GarageYr’ which depicts the year in which the garage was built in the house. The value was 2027 which is way into the future. Instead of getting rid of the observation, I imputed it with the mode. Before doing that I made sure that the house does have a garage by checking the ‘GarageType’ attribute. It is important to ‘know your data’ before making any changes so be aware of these small but significant aspects.

For NAN’s I plotted a graph for all the features having at least one of those. Then I analysed them one by one and decided how to handle the missing value. Most of them were replaced by the Mode of that feature but there were some special cases like this -

In the absence of a basement, we can impute the missing value in basement square feet as 0

Feature Engineering

Photo by Isaac Smith on Unsplash

This is undoubtedly the most important and time-consuming part of the process. My approach involved taking them up one at a time (80 of them!) and apply the transformations. This might sound cumbersome but believe you me, it will go a long way. My methodology was to visualize them using different graphs (seaborn gives you pretty graphs!) and base my decision (to either drop or keep them with any changes) on it. This will be a good time to use your analytical powers. There is no rocket science here and the dataset is very intuitive (more the number of bedrooms, higher the price) and logical. I’ll explain what I did with some examples in broad categories.

Polynomials

If you are planning to regress on a huge dataset, linear shouldn’t be the only way to do it. Some features might have a relationship with the target variable which cannot be expressed by a singular degree. Therefore, I handpicked the 10 most correlated variables (from the correlation matrix) and created their square, cube and square root versions and added them to the dataset.

Continuous Features

These are the ones which have continuous values like LotArea, GarageArea, 1stFlrSF etc. My go-to tactic for this kind of features was to plot their distribution graphs and their scatter-plot with the SalePrice (our target variable). Here is a snippet of the ‘LotFrontage’

Left: Distribution graph; Right: Scatter plot with SalePrice

The scatter plot shows that the effect of LotFrontage might not be very significant on the SalePrice but it is something and we’ll keep it. The left graph shows how positively skewed this feature distribution is. Always keep in mind, more the distribution is closer to the normal, better your prediction model will perform. So, always try to transform your data to make it more normal-y. Techniques usually used to do so are — log transformation and binning. I’ll take a log here.

Some of the features may seem continuous but will better suit when converted to categorical. For example, features related to year (year when the house was built, year when it was remodelled etc.). Also, other integral features like — number of bedrooms, number of pools, cars that could fit in the garage. They all will make more sense if converted to categorical and I did the same.

Categorical Features

All non-continuous variables are categorical. The practice I follow here is to visualise the feature using strip-plot and the average SalePrice for each category. Let’s take HeatingQC (heating quality) as an example:

Left: Stripplot; Right: Average

The first one lets me visualise the number of houses in each category and points out any discrepancies (we’ll see later how). The second one helps me decide whether to treat it as an ordered feature (replace each category with an integer) or create dummy variables. This example is clearly ordinal (Ex being the highest rating and having the highest average price and so on) so we’ll replace them with ordered integers. Boxplots could also be an efficient way to visualize the data.

Drop (useless) Features

We want to keep our feature space as dense as possible to get the maximum efficiency from our model. So we try to get rid of the features we think might not contribute significantly enough to the target variable. I came across a few and here are some examples.

Street: Type of road access to the property

The strip plot reveals that most of the houses have ‘Pave’ as the street type and the count for ‘Grvl’ is negligible. This feature has almost constant behaviour for all house prices and hence do not contribute much to the target. We can drop this feature.

PoolArea: Pool area in square feet

Similarly for PoolArea, the value is 0 for most of the observations (almost all the houses do not have a pool) and hence wouldn’t serve us much in the prediction. We can drop it.

Feature Importance

Even after an extensive analysis of all the features, some redundant features might escape the naked eye. We will use XGBoost’s inbuilt feature importance function to further identify the insignificant columns and get rid of them before we fit our model. Let’s first get the index for all the 324 features we have after transformations.

Importance of features with respect to the SalePrice. Quite a lot of them are zero.

We can see that numerous of our features do not contribute towards determining the SalePrice and hence we’ll get rid of them. After dropping the features with score 0, our dataset got reduced from 324 columns to 218!

Machine Learning Modelling

Photo by Omar Flores on Unsplash

Finally, the part for which we started all this. Frankly, 90% of our job is already done. If you have a well structured and dense dataset, modelling will be a cakewalk. I prepared a list of the most famous regression models that I wanted to fit my data on (listed later in this section). Then I split my test data into test and validation sets. I trained the model on the test set and evaluated on validation. This helped me visualize which model is performing better. Here is an example of the XGBoost regressor.

If you notice, I have taken a log transform of the SalePrice as it gives a better distribution. While getting the results I have converted the prices back to the original scale by taking an exponential of the value.

Parameter Tuning

Each regression model has its own set of parameters which are highly sensitive to the dataset upon which it is trained. If you want highly optimized results (which you do!), then it is imperative to have the best performing set of parameters. The conventional way is to run a grid search using GridSearchCV but it requires a considerable amount of computation power. If you don’t have access to a GPU, better avoid it (my MacBookPro took more than 7 hours for XGBoost and still could complete the search). The alternate way is to do a ‘parameter sweep’ in which we evaluate our training by varying one parameter and keeping the rest as same. This requires one to have a fair idea of the range of the values that might be suitable for the parameter we are sweeping. Here is an example of finding alpha for RidgeRegressor.

alpha~75 gives the lowest RMSE and is the best possible value for this dataset

Ensemble

So till now, we have scores from the individual regression models. But wouldn’t it be better if we could weigh in the predictions of all the models and get a combined result? Every model has its own speciality. Some are better at picking up certain info from the data and some are more susceptible to others. It would be better if we ‘consult’ various models and then reach to a final prediction. This is called ensembling. In this, we train all our models over the training data, and while predicting we combine the results. This combination could be anything — an average of all the values, weighted mean etc. I used the weighted mean option and decided the weights according to how individual models performed over the validation set. Here is the code.

Conclusion

Although experimenting with different models on the surface might give you good results, but it is important to understand the math behind it so that in future you could choose a regressor by reasoning. Also, my way is surely not the best to approach this house pricing problem and you can devise your own, so, keep trying y’all!