Feature engineering and ensembled models for the top 10 in Kaggle “Housing Prices Competition”
By: Gabriel Naya
Tags: Feature Engineering, Data Science, Housing Prices, Kaggle Competition, Machine Learning

Ensemble models

The regression algorithms have had no problem supporting the columns after categorization, so except in strictly inconvenient cases, it does not seem necessary or even a good idea to remove them. However, this is only one approach; at the end of this article, we will see other approaches to the problem very different, and as much or even more useful.

In this case, then, it is not necessary to reduce the dimensionality of the model since training time with different algorithms is not a problem, and brute force does not seem a bad strategy.

What does “ensemble models” mean?

“Ensemble models give us excellent performance and work in a wide variety of problems. They’re easier to train than other types of techniques, requiring less data with better results. In machine learning, ensemble models are the norm. Even if you aren’t using them, your competitors are … Ensemble models are comprised of several weak models that aren’t so great themselves, known as weak learners. When you combine them, they fill in each other’s weaknesses and render better performance than they would when deployed alone.”¹¹

Model metrics

The competition metric is the Mean Absolute Error (MAE).

For the initial measurement, RMSE and CV_RMSE used.

Because the CV_RMSE is negative, to apply the square root, it is necessary to flip it.¹²

def rmsle(y, y_pred):

return np.sqrt(mean_squared_error(y, y_pred)) def cv_rmse(model):

rmse = np.sqrt(-cross_val_score(model, X, y, scoring="neg_mean_squared_error", cv=kfolds))

return rmse

The models

It is an example of one of the original models used by Sandeep Kumar, where we can see lots of hyperparameters uses:

gbr = GradientBoostingRegressor(n_estimators=3000, learning_rate=0.05,max_depth=4, max_features='sqrt', min_samples_leaf=15, min_samples_split=10, loss='huber', random_state=42)

Making the ensemble models

In the same operation that we carried out for the GBR model, we can carry it out with other different regression models. Then in a function, we assign them an incidence coefficient on the total value, in such a way that the ratios add up to 1.

Sandeep Kumar blend models:

def blend_models_predict(X):

return (

(0.1 * elastic_model_full_data.predict(X)) + \

(0.1 * lasso_model_full_data.predict(X)) + \

(0.1 * ridge_model_full_data.predict(X)) + \

(0.1 * svr_model_full_data.predict(X)) + \

(0.1 * gbr_model_full_data.predict(X)) + \

(0.15 * xgb_model_full_data.predict(X)) + \

(0.1 * lgb_model_full_data.predict(X)) + \

(0.25 * stack_gen_model.predict(np.array(X))))

Mi blend model:

def blend_models_predict(X=X):

return (

(0.10 * elastic_model_full_data.predict(X)) +

(0.05 * lasso_model_full_data.predict(X)) +

(0.05 * ridge_model_full_data.predict(X)) +

(0.05 * svr_model_full_data.predict(X)) +

(0.20 * gbr_model_full_data.predict(X)) +

(0.20 * xgb_model_full_data.predict(X)) +

(0.10 * lgb_model_full_data.predict(X)) +

(0.25 * stack_gen_model.predict(np.array(X))))

We were able to use other models, such as linear regressor, tree regressors, etc. The importance assigned to each model depends on the confidence we have in the algorithm on the test dataset, sometimes algorithms such as GBR, XGB, or LGB work very well in the fit but fall by overfitting to use them in the test set. On the other hand, models like SVR or RIDGE tend to have less precision but are more stable.

Putting it all together and many hours of effort

Other approaches

When we managed to place our model in the top 10 of the world competition and to know that our knowledge was very scarce, with the desire to go for more but feeling on a plateau, I contacted Adam Suwik who was in the #3 at the time, and still remains there in the top positions:

Very kindly Adam agreed to share the approach of his model in a very illustrative post.

We transcribe here some of the most significant passages for this article:

“… This is no secret knowledge, only simple logical thinking (although specialist knowledge in the subject would be very helpful too). Adjusting the data to the model — most likely Linear Regression will be the # 1 choice. Unfortunately, most of the data does not make much sense and may adversely affect the result. For example, quality — it is not the case that excellent quality of a small house and a large villa has the same value — it should rather be replaced by “size in square feet in exellent quality”. And it seems to me that most of the data should be prepared in such a way — combined with the size of a plot, house area, basement area, pool size, etc. Combining data / simplifying the model — that’s what I use a lot. The world is not that complicated. In this case, it seems to me that 3 features would be enough — how big the house is?, how nice it is? how comfortable is it?, maybe a few additional ones like a swimming pool, an additional garage space, etc. … Dropping data — I use it a lot too. If I do not see value or think that the information is contained elsewhere, or if I have somehow engineered data, I drop whatever I can. Some expertise would be pretty useful here, but using just common sense, much of the data that seems to have an impact is really secondary. Nobody builds a luxury home on cheap foundations and viceversa … Clusters — not always the same rules apply to entire populations, if I am able to isolate a large group of similar cases, I will build a separate model for them. From life experience — if you look at the price per square foot you will notice that for smaller apartments it is definitely bigger. And I probably could describe a few other things, but it all comes down to one thing — analyze all the data carefully and think about how to engineer them to fit the model you use. I believe in one principle: “garbage in, garbage out”. What quality of data you provide, the quality of result you will get… NaN’s. Unfortunately, the answer is just as above — there are no secrets — do whatever seems to make sense. I like “0”, “none”, I really like my own predictions, I hate the average, it’s like admitting to the lack of any idea and just minimizing cost of assumed mistakes…”

It is another approach, and there are many more approaches to the web. To be in 1% of any competition is in itself a good enough model and a sign that we are on the right way.

Summary

We have seen how important it is to start a job supported in the preliminary work that others have shared, in this case, the transfer learning of the Sandeep Kumar model. Thanks him!

This previous transfer learning gave us a floor and essential knowledge from which to start, much more elaborated and refined than the experience that a rookie takes from online courses.

Then, after many hours of effort and systematic tests, we saw the impact of each technique on the different variables to make decisions about the engineering features and the ensemble of models.

One day I got that idea that makes the difference, being distracted the head still thinking, the idea was to calculate the value of square feet based on a given concept, and the satisfaction of having reached that top ten was an incredible moment.

The objective of this Kaggle competition is to learn

This post aimed to transmit the techniques that you can use if you are stuck until the mud in the game as I was a few months ago hopefully, help you get on an exact path towards improving your model, or maybe help you light that little lamp that makes the difference.

I will be grateful for comments and reports if something could be doing better.

References

[1] Kaggle is an online community of data scientists and machine learners, owned by Google LLC . Kaggle allows users to find and publish data sets, explore and build models in a web-based data-science environment, work with other data scientists and machine learning engineers, and enter competitions to solve data science challenges. Kaggle got its start by offering machine learning competitions and now also offers a public data platform, a cloud-based workbench for data science, and short form AI education. On 8 March 2017, Google announced that they were acquiring Kaggle.

[2] https://www.google.com/url?sa=t&source=web&rct=j&url=https://personal.us.es/vararey/adatos2/correlacion.pdf&ved=2ahUKEwiaytLbhqnlAhUGLLkGHeL2B24QFjAMegQIBxAB&usg=AOvVaw1mijaxh5F0qXIXMpeadD5j

[3] Incidence of correlation and time features in a regression model

[4] Statistical data preparation: management of missing values and outliers

“… There are basically three methods for treating outliers in a data set. One method is to remove outliers as a means of trimming the data set. Another method involves replacing the values of outliers or reducing the influence of outliers through outlier weight adjustments. The third method is used to estimate the values of outliers using robust techniques.

Trimming: Under this approach, a data set that excludes outliers is analyzed. The trimmed estimators such as mean decrease the variance in the data and cause a bias based on under- or overestimation. Given that the outliers are also observed values, excluding them from the analysis makes this approach inadequate for the treatment of outliers.

Winsorization: This approach involves modifying the weights of outliers or replacing the values being tested for outliers with expected values. The weight modification method allows weight modification without discarding or replacing the values of outliers, limiting the influence of the outliers. The value modification method allows the replacement of the values of outliers with the largest or second smallest value in observations excluding outliers.

Robust estimation method: When the nature of the population distributions is known, this approach is considered appropriate because it produces estimators robust to outliers, and estimators are consistent. In the recent years, many studies have proposed a variety of statistical models for robust estimation; however, their applications are sluggish due to complicated methodological aspects. “

[5] https://docs.scipy.org/doc/scipy/reference/generated/scipy.stats.skew.html

[6] https://docs.scipy.org/doc/scipy/reference/generated/scipy.special.boxcox1p.html

[7]https://docs.scipy.org/doc/scipy/reference/generated/scipy.stats.boxcox_normmax.html?highlight=normmax#scipy.stats.boxcox_normmax

[8] https://www.kaggle.com/rtatman/data-cleaning-challenge-scale-and-normalize-data

“One thing you can do is add a fixed value that’s just bigger than your smallest negative to make all your value normal. Another normalization method is the Johnson Transformation, which can handle negative values.

[9] https://towardsdatascience.com/choosing-the-right-encoding-method-label-vs-onehot-encoder-a4434493149b

[10] https://towardsdatascience.com/one-hot-encoding-multicollinearity-and-the-dummy-variable-trap-b5840be3c41a

[11] https://medium.com/@ODSC/ensemble-models-demystified-c871d5ee7793

[12] https://github.com/scikit-learn/scikit-learn/issues/2439

The Mean Square Error returned by sklearn.cross_validation.cross_val_score is always a negative. While being a designed decision so that the output of this function can be used for maximization given some hyperparameters, it’s extremely confusing when using cross_val_score directly. At least I asked myself how a the mean of a square can possibly be negative and thought that cross_val_score was not working correctly or did not use the supplied metric. Only after digging in the sklearn source code I realized that the sign was flipped.

This behavior is mentioned in make_scorer in scorer.py, however it’s not mentioned in cross_val_score and I think it should be, because otherwise it makes people think that cross_val_score is not working correctly