Feature engineering and ensembled models for the top 10 in Kaggle “Housing Prices Competition” | by Gabriel Naya | Towards Data Science

Kaggle1 is one of the most significant online communities of data scientists and machine learners. Within Kaggle, it is common to find competitions, where different users generate prediction models and submit their results to be position according to the score, in the leaderboard.
The competition that supports the online courses that every beginner performs is “Housing Prices Competition for Kaggle Learn Users” .
A global and learning-oriented competence, therefore, the material circulating on the web is profuse, even for those looking well can be found directly with complete codes of excellent models that come very close to 1% on the scoreboard.
This article is base on the code exposed by Sandeep Kumar in Kaggle. This code was our most important starting point in the competition. While this code alone is not enough to reach the top 10, it uses many exciting techniques. To the methods used there, we have added others that have made the difference to position our model in the top 10.
Because every detail counts when you start trying to improve within 1%, you need to be very methodical and thorough in handling data engineering as well as model parameters.
The dataset has 1460 rows of houses with their corresponding 79 features and the target variable: the price of each house. There is also a test set with 1459 other houses.
In this article, we are going to work them unified for the different techniques that we are going to apply and separating them before realizing the final prediction to raise the results.
The workflow with the dataset will then be
a) Import the train_features and test_features data.
b) Save and remove the index (Id) and prepare train_features without the objective variable
c) Create a data frame “features” that joint them
d) Perform data engineering
e) After finished in data engineering work, split again in X_train and X_test the datasets.
f) Make the models, fit, and prediction
g) If we apply any transformation to the target variable (Y), we must invert this transformation over the prediction.
h) Submit results
We are facing a regression exercise, and we must calculate the selling price of each house, of each row of the dataset.
Our objective then is the column SalePrice, first, let’s see its general behavior:
train['SalePrice'].describe()
Let’s now see its practice in any model using it directly, or applying the logarithm to it.
So we see the improvement is significant going down from 12934 to 8458 if we change the model training, taking as a result of the logarithm of the target variable instead of ‘SalePrice’ directly.
“The purpose of the correlation is to examine the direction and strength of the association between two quantitative variables. Thus we will know the intensity of the relationship between them and whether, as the value of one variable increases, the value of the other variable increases or decreases.”2
If you want to know more about the incidence of the correlation of variables and the target variable, you can review this previous issue3 for a detailed analysis.
We are going to create a data frame with the numeric columns and apply the Pandas “corr()” method to them and evaluate the SalePrice target:
data_corr = train.select_dtypes(include=[np.number])data_corr.head()
corr = data_corr.corr()corr.head(20)corr.sort_values(['SalePrice'], ascending=False, inplace=True)corr['SalePrice']
Here we have an ordered list of variables related to the target variable, the increase in model precision may be thought to decrease as we use variables less and less correlated to the target variable.
“... Outliers significantly affect the process of estimating statistics (e.g., the average and standard deviation of a sample), resulting in overestimated or underestimated values. Therefore, the results of data analysis are considerably dependent on the ways in which the missing values and outliers are processed...”4
In the article from which this quotation is token, excellent concepts about the treatment of outliers’ values can be found in-depth.
Let’s take as an example one of the variables that have the highest incidence in its correlation with a price: GrLivArea.
To visualize outliers, we usually use boxplots.
Here an excellent article about the use of boxplot and outliers detection5 , and his main features:
Box plots have box from LQ to UQ, with median marked
They portray a five-number graphical summary of the data Minimum, LQ, Median, UQ, Maximum
Helps us to get an idea on the data distribution
Helps us to identify the outliers easily
25% of the population is below first quartile,
75% of the population is below third quartile
If the box is pushed to one side and some values are far away from the box then it’s a clear indication of outliers
We perform a check on any model, using the entire training set, cutting the Outliers > 4500 and cutting the Outliers > 4000, obtaining the following values
The difference in performance between cutting at 4000 or 4500 does not seem significant; however, the improvement in accuracy when slice GrLivArea registers greater than 4500 is considerable.
In principle and on the training data, it would seem a suitable strategy. Still, it is necessary to look for a balance between what is slice in the entry of the model since it can generate overfitting or take away possibilities of generalization to our model.
If we deploy null values in the data frame before and after these operations with this code, we are going to obtain graphs as in the following image.
import matplotlib.pyplot as pltimport seaborn as snsplt.figure(figsize=(12, 6))sns.heatmap(features.isnull())plt.show()
Remember that all null values must be clear before entering a model.
Many of these data we can already know that we are not going to precise them or that the attempt to fill the null values is not viable for them; we apply the drop of the column directly.
features.drop(['Street'], axis=1, inplace=True)features.drop(['Fence'], axis=1, inplace=True)
Everyday use for filling in missing data is to fill it in with “None” or “zero,” depending on the type of data.
Let’s see here two examples to assign the most frequent value (Pandas mode function) or how to apply the amount of the average of the whole column or a cluster of it.
These examples are only for illustrative purposes; you can apply any Pandas function and make the grouping by any attribute that you want to make the cluster. It can also be used to create new features of the “cluster” type.
Example a: Application of the most frequent MSZoning value, grouped for each MSSubClass
features['MSZoning'] = features.groupby('MSSubClass')['MSZoning'].transform(lambda x: x.fillna(x.mode()[0]))
Example b: Application of the mean value of LotFrontage, grouped for each Neighborhood
features['LotFrontage'] = features.groupby('Neighborhood')['LotFrontage'].transform(lambda x: x.fillna(x.median()))
Example c: Apply the most frequent amount to complete your features
features['Exterior1st'] = features['Exterior1st'].fillna(features['Exterior1st'].mode()[0])
Finally, we must make sure that there is no value left with null so we can apply a fill to None or zero depending on the type of column:
objects = [col for col in features.columns if features[col].dtype == "object"]features.update(features[objects].fillna('None'))numerics = [col for col in features.columns if features[col].dtype in np.number]features.update(features[numerics].fillna(0))
You can create new variables to add specific information that helps the model define the data sets, or to simplify the model by creating averages, sums, or Boolean variables.
Each of these variables must be carefully selected, and the impact on the model must be measured to see whether or not it adds quality to the result, and in the case of being used to “simplify” if the loss of eliminated columns does not affect the accuracy of the model.
features['CondTot'] = (features['Condition1'] + features['Condition2']) / 2features['OverTot'] = features['OverallQual'] + features['OverallCond']features['TotalSF'] = features['TotalBsmtSF'] + features['1stFlrSF'] + features['2ndFlrSF']features['haspool'] = features['PoolArea'].apply(lambda x: 1 if x > 0 else 0)
For normally distributed data, the skewness should be about 0. For unimodal continuous distributions, a skewness value > 0 means that there is more weight in the right tail of the distribution. The function skewtest can be used to determine if the skewness value is close enough to 0, statistically speaking.5
We use the Scipy.boxcoxp1 6 that compute the Box-Cox transformation of 1 + x. The Box-Cox transformation computed by boxcox1p is:
y = ((1+x)**lmbda - 1) / lmbda  if lmbda != 0log(1+x)                        if lmbda == 0Returns nan if x < -1. Returns -inf if x == -1 and lmbda < 0.
For getting the lambda value to input in boxcoxp1, we use boxcox_normmax(features[ind)) from Scipy, that Compute optimal Box-Cox transform parameter for input data7
This transformation can be done only on features that have all their positive values; if you need to do a conversion on elements with negative values, see the reference note8 .
from scipy.stats import skewfrom scipy.special import boxcox1pfrom scipy.stats import boxcox_normmaxnumerics = [col for col in features.columns if features[col].dtype in np.number]skew_f = features[numerics].apply(lambda x: skew(x)).sort_values(ascending=False)highest_skew =  skew_f[skew_f > 0.5]highest_skew
skew_idx = highest_skew.indexfor ind in skew_idx:    features[ind] = boxcox1p(features[ind], boxcox_normmax(features[ind))features[highest_skew.index].head()
When creating variables related to the target variable, it is necessary to do them only on the train data set and then try to take them to the test dataset, since there we do not know the value of the target variable “Y”.
This type of variables with which we have added a significant improvement in the leaderboard of the competition
For example, we can calculate the price per square meter, linked to each neighborhood or class of housing, then we first figure it in the train set:
train['Sqr'] = train['SalePrice'] / train['LotArea']train['Sqr'] = train.groupby('Neighborhood')['Sqr'].transform(lambda x: x.median())
Then we make a dictionary with the prices for each neighborhood:
d = {}for indice_fila, x_train in train.iterrows():    d.update({x_train['Neighborhood']:x_train['Sqr']})
And finally, we created the feature in the test dataset
test['Sqr'] = 0.00for indice, x_test in test.iterrows():    test.loc[test.index == indice ,'Sqr'] = d[x_test['Neighborhood']]
“What one hot encoding does is, it takes a column which has categorical data, which has been label encoded and then splits the column into multiple columns. The numbers are replaced by “1” and “0”, depending on which column has what value... ” 9
Categorizing object features can be very convenient, the discussion of which columns to use to apply OHE to them is broad and their limits diffuse, since each different value within the column will be transformed into a new dimension of the matrix input to the model. Sometimes it may be convenient to create a new feature that groups different values to reduce dimensionality, for example, a set of neighborhoods that have similar characteristics and prices.
Some numeric columns, we may want to categorize them, for which it is convenient to change the type to string since finally, we will apply get_dummies to all the columns of type object.
features['MSSubClass'] = features['MSSubClass'].apply(str)features['MoSold'] = features['MoSold'].astype(str) final_features = pd.get_dummies(features).reset_index(drop=True)
“The dummy variable trap manifests itself directly from one-hot-encoding applied on categorical variables ... size of one-hot vectors is equal to the number of unique values that a categorical column takes up and each such vector contains exactly one ‘1’ in it. This ingests multicollinearity into our dataset ... “10
In other words, when performing the transformation, the set of features of the added OHE is aligned to the target variable, and therefore any column is usually removed. In our model, we have not applied this technique.
Sandeep Kumar’s original work includes at the end of feature engineering and applying get_dummies, a code that looks for those columns that have a lot of zero values (above 99.94%), which are drop.
This technique indirectly mitigates in categorized variables the effect of the dummy variable trap and also avoids including in the model columns that by the impact of the content of the categorization are irrelevant for a model that seeks to extract the main characteristics and generalize them.
overfit = []for i in X.columns:    counts = X[i].value_counts()    zeros = counts.iloc[0]    if zeros / len(X) * 100 > 99.94:       overfit.append(i)overfit = list(overfit)overfit.append('MSZoning_C (all)')overfit
X = X.drop(overfit, axis=1).copy()
The regression algorithms have had no problem supporting the columns after categorization, so except in strictly inconvenient cases, it does not seem necessary or even a good idea to remove them. However, this is only one approach; at the end of this article, we will see other approaches to the problem very different, and as much or even more useful.
In this case, then, it is not necessary to reduce the dimensionality of the model since training time with different algorithms is not a problem, and brute force does not seem a bad strategy.
“Ensemble models give us excellent performance and work in a wide variety of problems. They’re easier to train than other types of techniques, requiring less data with better results. In machine learning, ensemble models are the norm. Even if you aren’t using them, your competitors are ... Ensemble models are comprised of several weak models that aren’t so great themselves, known as weak learners. When you combine them, they fill in each other’s weaknesses and render better performance than they would when deployed alone.”11
The competition metric is the Mean Absolute Error (MAE).
For the initial measurement, RMSE and CV_RMSE used.
Because the CV_RMSE is negative, to apply the square root, it is necessary to flip it.12
def rmsle(y, y_pred):    return np.sqrt(mean_squared_error(y, y_pred))def cv_rmse(model):    rmse = np.sqrt(-cross_val_score(model, X, y, scoring="neg_mean_squared_error", cv=kfolds))    return rmse
It is an example of one of the original models used by Sandeep Kumar, where we can see lots of hyperparameters uses:
gbr = GradientBoostingRegressor(n_estimators=3000, learning_rate=0.05,max_depth=4, max_features='sqrt', min_samples_leaf=15, min_samples_split=10, loss='huber', random_state=42)
In the same operation that we carried out for the GBR model, we can carry it out with other different regression models. Then in a function, we assign them an incidence coefficient on the total value, in such a way that the ratios add up to 1.
Sandeep Kumar blend models:
def blend_models_predict(X):   return ((0.1 * elastic_model_full_data.predict(X)) + \(0.1 * lasso_model_full_data.predict(X)) + \(0.1 * ridge_model_full_data.predict(X)) + \(0.1 * svr_model_full_data.predict(X)) + \(0.1 * gbr_model_full_data.predict(X)) + \(0.15 * xgb_model_full_data.predict(X)) + \(0.1 * lgb_model_full_data.predict(X)) + \(0.25 * stack_gen_model.predict(np.array(X))))
Mi blend model:
def blend_models_predict(X=X):    return ((0.10 * elastic_model_full_data.predict(X)) +(0.05 * lasso_model_full_data.predict(X)) +(0.05 * ridge_model_full_data.predict(X)) +(0.05 * svr_model_full_data.predict(X)) +(0.20 * gbr_model_full_data.predict(X)) +(0.20 * xgb_model_full_data.predict(X)) +(0.10 * lgb_model_full_data.predict(X)) +(0.25 * stack_gen_model.predict(np.array(X))))
We were able to use other models, such as linear regressor, tree regressors, etc. The importance assigned to each model depends on the confidence we have in the algorithm on the test dataset, sometimes algorithms such as GBR, XGB, or LGB work very well in the fit but fall by overfitting to use them in the test set. On the other hand, models like SVR or RIDGE tend to have less precision but are more stable.
When we managed to place our model in the top 10 of the world competition and to know that our knowledge was very scarce, with the desire to go for more but feeling on a plateau, I contacted Adam Suwik who was in the #3 at the time, and still remains there in the top positions:
Very kindly Adam agreed to share the approach of his model in a very illustrative post.
We transcribe here some of the most significant passages for this article:
“... This is no secret knowledge, only simple logical thinking (although specialist knowledge in the subject would be very helpful too).
Adjusting the data to the model — most likely Linear Regression will be the # 1 choice. Unfortunately, most of the data does not make much sense and may adversely affect the result. For example, quality — it is not the case that excellent quality of a small house and a large villa has the same value — it should rather be replaced by “size in square feet in exellent quality”. And it seems to me that most of the data should be prepared in such a way — combined with the size of a plot, house area, basement area, pool size, etc.
Combining data / simplifying the model — that’s what I use a lot. The world is not that complicated. In this case, it seems to me that 3 features would be enough — how big the house is?, how nice it is? how comfortable is it?, maybe a few additional ones like a swimming pool, an additional garage space, etc. ...
Dropping data — I use it a lot too. If I do not see value or think that the information is contained elsewhere, or if I have somehow engineered data, I drop whatever I can. Some expertise would be pretty useful here, but using just common sense, much of the data that seems to have an impact is really secondary. Nobody builds a luxury home on cheap foundations and viceversa ...
Clusters — not always the same rules apply to entire populations, if I am able to isolate a large group of similar cases, I will build a separate model for them. From life experience — if you look at the price per square foot you will notice that for smaller apartments it is definitely bigger.
And I probably could describe a few other things, but it all comes down to one thing — analyze all the data carefully and think about how to engineer them to fit the model you use. I believe in one principle: “garbage in, garbage out”. What quality of data you provide, the quality of result you will get...
NaN’s. Unfortunately, the answer is just as above — there are no secrets — do whatever seems to make sense. I like “0”, “none”, I really like my own predictions, I hate the average, it’s like admitting to the lack of any idea and just minimizing cost of assumed mistakes...”
It is another approach, and there are many more approaches to the web. To be in 1% of any competition is in itself a good enough model and a sign that we are on the right way.
We have seen how important it is to start a job supported in the preliminary work that others have shared, in this case, the transfer learning of the Sandeep Kumar model. Thanks him!
This previous transfer learning gave us a floor and essential knowledge from which to start, much more elaborated and refined than the experience that a rookie takes from online courses.
Then, after many hours of effort and systematic tests, we saw the impact of each technique on the different variables to make decisions about the engineering features and the ensemble of models.
One day I got that idea that makes the difference, being distracted the head still thinking, the idea was to calculate the value of square feet based on a given concept, and the satisfaction of having reached that top ten was an incredible moment.
The objective of this Kaggle competition is to learn
This post aimed to transmit the techniques that you can use if you are stuck until the mud in the game as I was a few months ago hopefully, help you get on an exact path towards improving your model, or maybe help you light that little lamp that makes the difference.
I will be grateful for comments and reports if something could be doing better.
[1] Kaggle is an online community of data scientists and machine learners, owned by Google LLC . Kaggle allows users to find and publish data sets, explore and build models in a web-based data-science environment, work with other data scientists and machine learning engineers, and enter competitions to solve data science challenges. Kaggle got its start by offering machine learning competitions and now also offers a public data platform, a cloud-based workbench for data science, and short form AI education. On 8 March 2017, Google announced that they were acquiring Kaggle.
[2] https://www.google.com/url?sa=t&source=web&rct=j&url=https://personal.us.es/vararey/adatos2/correlacion.pdf&ved=2ahUKEwiaytLbhqnlAhUGLLkGHeL2B24QFjAMegQIBxAB&usg=AOvVaw1mijaxh5F0qXIXMpeadD5j
[3] Incidence of correlation and time features in a regression model
[4] Statistical data preparation: management of missing values and outliers
“... There are basically three methods for treating outliers in a data set. One method is to remove outliers as a means of trimming the data set. Another method involves replacing the values of outliers or reducing the influence of outliers through outlier weight adjustments. The third method is used to estimate the values of outliers using robust techniques.
Trimming: Under this approach, a data set that excludes outliers is analyzed. The trimmed estimators such as mean decrease the variance in the data and cause a bias based on under- or overestimation. Given that the outliers are also observed values, excluding them from the analysis makes this approach inadequate for the treatment of outliers.
Winsorization: This approach involves modifying the weights of outliers or replacing the values being tested for outliers with expected values. The weight modification method allows weight modification without discarding or replacing the values of outliers, limiting the influence of the outliers. The value modification method allows the replacement of the values of outliers with the largest or second smallest value in observations excluding outliers.
Robust estimation method: When the nature of the population distributions is known, this approach is considered appropriate because it produces estimators robust to outliers, and estimators are consistent. In the recent years, many studies have proposed a variety of statistical models for robust estimation; however, their applications are sluggish due to complicated methodological aspects. “
[5] https://docs.scipy.org/doc/scipy/reference/generated/scipy.stats.skew.html
[6] https://docs.scipy.org/doc/scipy/reference/generated/scipy.special.boxcox1p.html
[7]https://docs.scipy.org/doc/scipy/reference/generated/scipy.stats.boxcox_normmax.html?highlight=normmax#scipy.stats.boxcox_normmax
[8] https://www.kaggle.com/rtatman/data-cleaning-challenge-scale-and-normalize-data
“One thing you can do is add a fixed value that’s just bigger than your smallest negative to make all your value normal. Another normalization method is the Johnson Transformation, which can handle negative values.
[9] https://towardsdatascience.com/choosing-the-right-encoding-method-label-vs-onehot-encoder-a4434493149b
[10] https://towardsdatascience.com/one-hot-encoding-multicollinearity-and-the-dummy-variable-trap-b5840be3c41a
[11] https://medium.com/@ODSC/ensemble-models-demystified-c871d5ee7793
[12] https://github.com/scikit-learn/scikit-learn/issues/2439
The Mean Square Error returned by sklearn.cross_validation.cross_val_score is always a negative. While being a designed decision so that the output of this function can be used for maximization given some hyperparameters, it’s extremely confusing when using cross_val_score directly. At least I asked myself how a the mean of a square can possibly be negative and thought that cross_val_score was not working correctly or did not use the supplied metric. Only after digging in the sklearn source code I realized that the sign was flipped.
This behavior is mentioned in make_scorer in scorer.py, however it’s not mentioned in cross_val_score and I think it should be, because otherwise it makes people think that cross_val_score is not working correctly