Tricks I used to succeed on a famous Kaggle Competition
By: Kamal Chouhbi
Tags: Machine Learning, Predictive Analytics, Kaggle, Competition, Data Science

Tricks I used to succeed on a famous Kaggle Competition

Photo by Tom Rumble on Unsplash

As a fresh Data Scientist, and during my Master degree in Data Analytics, I was motivated to participate in many Kaggle competitions in order to put into work all what I’ve learned so far during my studies.

I will talk about my first competition, House Prices: Advanced Regression Techniques. This is a perfect competition for data science beginners or students who passed a course in machine learning and are looking to expand their skill set. The goal of this competition is to predict the sales price for a sample of houses. All the submissions are evaluated on Root Mean Squared Error (RMSE) between the logarithm of the predicted value and the logarithm of the observed sales price.

There are several factors that impact the overall price of the house, some of those factors are more tangible as the quality of the house or the overall size (area) of the house and other factors are more intrinsic such as the performance of the economy. Coming with an accurate model that predicts the actual value is an arduous task since there are both internal and external factors that will affect the price of each single house.

Here I will describe for you how I got from around 2800th place to top 150. You can find a detailed notebook on my GitHub profile : https://github.com/chouhbik/Kaggle-House-Prices

At the beginning, I started with building different models and make them train on the raw data that I have downloaded from Kaggle. Random Forest performed well with a score of 0.13825 enough to put me in the bottom of the leaderboard.

After that, I realized that the preliminary steps of EDA (Exploratory Data Analysis) and Feature Engineering are so important. Next I will show you how I did improve that …

Step 1: Log-transform of the dependent variable

There are two reasons to predict log of sales, not sales: first, the evaluation metric used in this competition depends on log of sales and, second, many monetary entities have distribution close to log-normal, i.e. heavy-tailed. Usually it is better to predict something which is not heavy tailed and most of machine learning models perform well on the data that is normally distributed.

(np.log(df_train["SalePrice"])).hist(bins = 40)

Histogram : Log of sales

Step 2: Re-coding predictors / missings / outliers

Some of the predictors which are ordinal (i.e. there is a natural order in their values) initially are stored as factors. I re-coded them in a sensible way (examples of such predictors are ExterQual and BsmtCond). Obviously I had to fill in missing values:

Missings in numeric predictors: I filled in with zeros when it made sense (e.g. if there are no baths in the basement, BsmtFullBath should be equal to zero),

Missings in categorical predictors: I filled in with medians or other typical values (mean or mode). Last,

Also it was worth removing some observations with outliers in predictors, like OverallQual and GrLivArea .

Step 3: Dealing with Non-linearities

As I planned to use some linear methods (Lasso, Ridge, SVM), I replaced all heavy-tailed predictors with their logs and for some of the predictors added their squares (i.e. we have predictor X and we add predictor X²). Replacing heavy-tailed predictors with their logs is motivated by the fact that:

Linear methods might fit such predictors with very small weights and most of the information contained in the values might be lost

Predictions when such predictors take very high values might be also very high or misleading. Adding squares is motivated by non-linearities in scatterplots “predictor vs. log of sales”

Step 4: Adding new predictors

I decided to add some indicator variables which are equal to something if respective predictors take some certain values.

For example:

# feture engineering a new feature "TotalSF"

all_data['TotalSF'] = all_data['TotalBsmtSF'] + all_data['1stFlrSF'] + all_data['2ndFlrSF'] all_data['YrBltAndRemod']=all_data['YearBuilt']+all_data['YearRemodAdd']



all_data['Total_sqr_footage'] = (all_data['BsmtFinSF1'] + all_data['BsmtFinSF2'] +all_data['1stFlrSF'] + all_data['2ndFlrSF'])



all_data['Total_Bathrooms'] = (all_data['FullBath'] + (0.5 * all_data['HalfBath']) + all_data['BsmtFullBath'] + (0.5 * all_data['BsmtHalfBath']))



all_data['Total_porch_sf'] = (all_data['OpenPorchSF'] + all_data['3SsnPorch'] + all_data['EnclosedPorch'] + all_data['ScreenPorch'] +all_data['WoodDeckSF']) all_data['haspool'] = all_data['PoolArea'].apply(lambda x: 1 if x > 0 else 0)

all_data['has2ndfloor'] = all_data['2ndFlrSF'].apply(lambda x: 1 if x > 0 else 0)

all_data['hasgarage'] = all_data['GarageArea'].apply(lambda x: 1 if x > 0 else 0)

all_data['hasbsmt'] = all_data['TotalBsmtSF'].apply(lambda x: 1 if x > 0 else 0)

all_data['hasfireplace'] = all_data['Fireplaces'].apply(lambda x: 1 if x > 0 else 0)

Step 5: Stacking models

As a next step I used 10-fold cross-validation and stacking: for each “run” of cross-validation, I tried to fit 5 models on 9 out of 10 folds (Lasso, Ridge, Elastic-net, GBM and XGBoost), make predictions on the left-out fold and use these five sets of predictions as an input into another Lasso model to forecast log of sales on that left-out fold (such Lasso model is called meta-model). In total we have 6*10=60 models (10 sets of 6 models). All these models I used to make final predictions: I took the test dataset, made predictions using 5 sub-models and then used outputs of these models as an input into respective meta-model to get set of predictions for given set of models. I repeated that process 10 times to get 10 sets of predictions and then averaged them using arithmetic mean to get data used for submission.

Step 6: Tuning parameters

For stacking I used 6 different models and each needed tuning (for each “run” of cross-validation I fit 6 models with always the same parameters. I spent a lot of time and submissions to fine-tune parameters (best improvement was by tuning min_samples_leaf and min_samples_split for GradientBoostingRegressor). Eventually I got smt around 0.1167 and was confident I won’t be able to improve it by tuning.

Step 7: More with missing values

Next I tried different strategies of filling in missing values (modes/means/medians/etc.). Best thing which worked was to some extent unexpected: it was based on mice package in R and is described here. I used Mice package to fill the NA for Numeric Variable using Random Forest.

Mice package is very handy in filling NAs. The result is not the best but it is better than using median of the each variable to fill the gap. What’s more, it is nice and easy to use.

Using R/Mice/SVM and python/stacking and taking geometric mean of these predictions gave me 0.1131, which was already great. It was unexpected that such simple and not-business-driven approach to fill in outliers worked. Note, that Mice might fill in missings in numeric predictors with something not equal to zero when they should be zero. I suppose that such approach helped me as it was quite different from what I got in python.

Step 8: Brutal force

I was already happy with 0.1131 but there was last thing I wanted to try. Regression often works bad for edge cases , for small or big values of predictors. I took train, ran R/mice/svm, python/stacking, averaged results using geometric mean, got final predictions for log of sales and plotted them against logs of real values:

It was obvious from that picture that for small final predictions we are overestimating log of sales, for big values of predictions we are underestimating.

Finally, I tried something which is quite brutal: I take predicted sales (not log-, but sales as they are), take 3 percentiles and manually increase/decrease forecasts.

I repeated this 8 times with different values of percentiles until it eventually gave me 0.10985. It is important to note that such brute force approach improves forecasts for small number of observations, but it seems that is is enough to improve score on the Leaderboard.

q1 = final_sub['SalePrice'].quantile(0.0025)

q2 = final_sub['SalePrice'].quantile(0.0045)

q3 = final_sub['SalePrice'].quantile(0.99)



final_sub['SalePrice'] = final_sub['SalePrice'].apply(lambda x: x if x > q1 else x*0.79)

final_sub['SalePrice'] = final_sub['SalePrice'].apply(lambda x: x if x > q2 else x*0.89)

final_sub['SalePrice'] = final_sub['SalePrice'].apply(lambda x: x if x < q3 else x*1.0)

Summary:

To conclude, I tried also some other things, but didn’t help:

PCA

Adding more and more predictors

Adding other models to stacking

Using keras on tensorflow and scaling, with dense neural nets.

I would say that productionized version would have evaluation metric of around 0.12, as it is possible to build simple model which will deliver the same results. Interesting things which helped: adding squares of some of the predictors, using Mice & SVM in R on top of the raw data and working with edge-predictions (very low of very high ones).

Hope you enjoyed this analysis of this project and the predictive models used to come with close to accurate predictions.