Predicting Housing Prices using a Scikit-Learn’s Random Forest Model | Towards Data Science

Having a housing price prediction model can be a very important tool for both the seller and the buyer as it can aid them in making well informed decision. For sellers, it may help them to determine the average price at which they should put their house for sale while for buyers, it may help them find out the right average price to purchase the house.
To build a random forest regression model, which is able to predict the median value of houses. We will also briefly walk through some Exploratory Data Analysis, Feature Engineering and Hyperparameter tuning to improve the performance of our Random Forest model.
Our Machine Learning Pipeline can be broadly summarized into the following task:
Data AcquisitionData Pre-Processing and Exploratory Data AnalysisCreating a Base ModelFeature EngineeringHyperparameter TuningFinal Model Training and Evaluation
Data Acquisition
Data Pre-Processing and Exploratory Data Analysis
Creating a Base Model
Feature Engineering
Hyperparameter Tuning
Final Model Training and Evaluation
We will be using the Boston Housing dataset: https://www.kaggle.com/c/boston-housing/data.
#Importing the necessary libraries we will be using%load_ext autoreload%autoreload 2%matplotlib inlinefrom fastai.imports import *from fastai.structured import *from pandas_summary import DataFrameSummaryfrom sklearn.ensemble import RandomForestRegressorfrom IPython.display import displayfrom sklearn import metricsfrom sklearn.model_selection import RandomizedSearchCV#Loading the DatasetPATH = 'data/Boston Housing Dataset/'df_raw_train = pd.read_csv(f'{PATH}train.csv',low_memory = False)df_raw_test = pd.read_csv(f'{PATH}test.csv',low_memory = False)
2.1 Checking and handling if there are any missing data and outliers.
df_raw_train.info
Understanding the raw data:
From the raw training dataset above:
(a) There are 14 variables (13 independent variables — Features and 1 dependent variable — Target Variable). (b) The data types are either integers or floats.(c) No categorical data is present.(d) There are no missing values in our dataset.
2.2 As part of EDA, we will first try to determine the distribution of the dependent variable (MDEV).
#Plot the distribution of MEDVplt.figure(figsize=(10, 6))sns.distplot(df_raw_train['MEDV'],bins=30)
The values of MEDV follows a normal distribution with a mean of around 22. There are some outliers to the right.
2.3 Next, try to determine if there are any correlations between:  (i) the independent variables themselves (ii) the independent variables and dependent variable
To do this, let’s do a correlation heatmap.
# Plot the correlation heatmapplt.figure(figsize=(14, 8))corr_matrix = df_raw_train.corr().round(2)sns.heatmap(data=corr_matrix,cmap='coolwarm',annot=True)
(i) Correlation between independent variables:We would need to look out for features of multi-collinearity (i.e. features that are correlated with each other)as this will affect our relationship with the independent variable.
Observe that RAD and TAX are highly correlated with each other (Correlation score: 0.92) while there are a couple of features which are somewhat correlated with one another with a correlation score of around 0.70 (INDUS and TAX, NOX and INDUS, AGE and DIS, AGE and INDUS).
(ii) Correlation between independent variable and dependent variable:In order for our regression model to perform well, we ideally need to select those features that are highly correlated with our dependent variable (MEDV).
We observe that both RM and LSTAT are correlated with MEDV with a correlation score of 0.66 and 0.74 respective. This can also be illustrated via the scatter plot .
#Scatter plot to observe the correlations between the features that are highly correlated with MEDVtarget_var = df_raw_train['MEDV']plot1 = plt.figure(1)plt.scatter(df_raw_train['RM'],target_var)plt.xlabel('RM')plt.ylabel('MEDV')plot2 = plt.figure(2)plt.scatter(df_raw_train['LSTAT'],target_var)plt.xlabel('LSTAT')plt.ylabel('MEDV')
From the scatter plot above:i) MEDV increases linearly with the RM. This makes sense as we would expect the median price of a house would be more generally more expensive as the number of rooms increases.ii) MEDV decreases linearly with LSTAT. This also makes sense since we can expect the median price of a house would generally be less expensive at places of lower status.
Hold up! Before we create our base Random Forest Model, it very important to pick a suitable evaluation metric.
3.1 Picking the right evaluation metricPicking the right evaluation metric will help us to evaluate whether our model’s performance is good. For regression problems, the go-to evaluation metric is either Root Mean Square Error(RMSE) or Root Mean Square Logistic Error (RMLSE).
RMSE: It is a measure of the squared difference between the prediction from our model and the actual value.
where y’: predicted value, y: actual value
RMSLE: It is a measure of the squared difference between the log of the prediction from our model and the log of the actual value.
where y’: predicted value, y: actual value
The RMSLE might be a better evaluation metric as (1) it is robust enough to deal with outliers, which we saw is present in our dataset (2) RMSLE incurs a larger penalty for underestimation of the actual value. If we put ourselves in the sellers perspective, we do not want to underestimate the price as it would result in losses. However, for this project we shall not take anyone’s side and shall choose the RMSE as the evaluation metric as we would be using the Random Forest model which is immune to outliers.
3.2 Creating our base Random Forest Model
The next step would be to create a base model first without any feature engineering and hyperparameter tuning. We would use this model’s performance as a benchmark for comparison later on after we have done feature engineering and hyperparameter tuning later.
The machine learning model we have chosen is the Random Forest Regression Model and here is why:
Random Forests models require minimal data preparation. It is able to easily hand categorical, numerical and binary features without scaling or normalization required.Random Forests models can help us in performing implicit feature selections as they provide good indicators of the important features.Random Forest models are immune to outliers, which is present in our data, and they completely ignore statistical issues because unlike other machine learning models which perform much better after being normalized.
Random Forests models require minimal data preparation. It is able to easily hand categorical, numerical and binary features without scaling or normalization required.
Random Forests models can help us in performing implicit feature selections as they provide good indicators of the important features.
Random Forest models are immune to outliers, which is present in our data, and they completely ignore statistical issues because unlike other machine learning models which perform much better after being normalized.
We would like to create some helpful functions :
# A function to split our training data into a training set to train our  model and a validations set, which will be used to validate our model.def split_vals(a,n):    return a[:n],a[n:]# Functions that will help us calculate the RMSE and print the score.def rmse(x,y):    return math.sqrt(((x-y)**2).mean())def print_score(m):    res =[rmse(m.predict(X_train),y_train),rmse(m.predict(X_valid),y_valid),m.score(X_train,y_train),m.score(X_valid,y_valid)]    if hasattr(m,'oob_score_'):res.append(m.oob_score_)    print(res)
Splitting our training data further — Training data to train our Random Forest Model on and Validation data to validate the performance of our Model.
n_valid = 100n_train = len(df_raw_train)-n_validX_train,X_valid = split_vals(df_raw_train.drop('MEDV',axis=1),n_train)y_train,y_valid = split_vals(df_raw_train['MEDV'],n_train)X_test = df_raw_test
Creating and Fitting our Random Forest Model without feature selection and hyperparameter tuning.
From our base Random Forest model, we already get a very decent result with the training RMSE to be 1.394 while the validation RMSE is 3.021. However, notice that our model seems to be ovefitting since the validation RMSE is around 3 times higher than the training RMSE.
Hence, the baseline score to beat is validation RMSE 3.021!
Feature Engineering and Hyperparameter tuning are essential steps within a Machine Learning Pipeline.
4.1 Determining the important featuresThe features in our data are directly influencing our Random Forest model and the result it achieves (i.e. the better the features we prepare and choose, the better final result we will achieve!). Hence we are going to explore and fine tune our Random Forest model by determining which of the features it had deemed to be important in our base model earlier.
def feat_importance(m,df_train):    importance = m.feature_importances_    importance = pd.DataFrame(importance,index=df_train.columns,columns=["Importance"])    return importance.sort_values(by=['Importance'],ascending=False)importance = feat_importance(m,X_train)importance[:]
importance.plot(kind='barh')
We observe that the most importance features in predicting the MDEV are LSTAT and RM. Recall that these two parameters were previously found to be highly correlated with the MDEV.
4.2 Discarding non-important features The next step will be to try and discard features with importance coefficient less than 0.01 and use it to model our Random Forest again to see if there is an improvement in our prediction results.
#Discarding features with feature coefficients less than 0.01to_keep = importance[importance['Importance'] > 0.01].indexdf_raw_train_keep = df_raw_train[to_keep].copy()df_raw_test_keep = df_raw_test[to_keep].copy()#Splitting data into training and validation setX_train,X_valid = split_vals(df_raw_train_keep,n_train)# Fitting our Random Forest Model after discarding the less important features.
Our Random Forest Model performs slightly better after removing some of the redundant features (i.e. 6 features!). We have obtain the top 9 most important features in determining the dependent variable MEDV (LSTAT, RM, DIS,CRIM, TAX, NOX, PTRATIO,NOX and AGE).
Next, let’s see if there have been any changes to the rank of the features which are found to be important to our Random Forest Model.
def feat_importance(m,df_raw_train_keep):    importance = m.feature_importances_    importance =pd.DataFrame(importance,index=df_train.columns,columns=["Importance"])    return importance.sort_values(by=['Importance'],ascending=False)importance
We observe that the most important features after removing the redundant features previously are still LSTAT and RM. We would like to explore how dropping each of the remaining features one by one would affect our overall score.
Notice removing RM, LSTAT, DIS and CRIM results in a worse validation RMSE while removing AGE and PTRATIO gives us a slightly better score. Hence, we will further remove AGE and PTRATIO from the dataset before doing running the final model.
4.3 Removing Multi collinearity between independent features
In 2.3(i), there are some features which are correlated to each other. To improve the model’s performance, we would ideally like to remove any multi collinearity between the features.
To see how the features are correlated, we can plot a dendrogram diagram.
#Dendogram plotfrom scipy.cluster import hierarchy as hccorr = np.round(scipy.stats.spearmanr(df_raw_train_keep).correlation,4)corr_condensed = hc.distance.squareform(1-corr)z = hc.linkage(corr_condensed,method='average')fig = plt.figure(figsize=(16,10))dendogram = hc.dendrogram(z,labels=df_raw_train_keep.columns,orientation='left',leaf_font_size=16)plt.show()
From the dendrogram plot, in terms of ranking we can see that none of the features are measuring the same thing. The closest one we have in terms of ranking is NOX and CRIM.
If there were features which were very close to each other in terms of ranking, the next step would have been to remove these features, one at a time, and see if our model could be simplified further without impacting our validation RMSE score. However for this case, we don’t need to.
We have come to the final step (hooray!) before we can build our final Random Forest Model.
Hyperparameter tuning is an iterative process whereby we select the best configurations of the hyperparameters that give us the best model performance output. For Random Forest model, we will focus on tuning 3 hyperparameters:
(1) n_estimators : Number of trees to create and generalize over our Random Forest. Likely, the more trees we create, the better as it would enable us to sample more of the dataset and help us to generalize better. However, there would come a point whereby increasing the number of trees will only result in very small changes to our validation RMSE at the cost of computational power.
(2) min_samples_leaf : This indicates the number of samples that will be in our leaf node. Each time we double the min_sample_leaf, we are removing one layer from the tree and halving the number of leaf nodes. Hence, the result of increasing min_samples_leaf is that each of our leaf nodes will have more than one sample inside them so when we calculate the average on that leaf node, it would be more stable although we will get a little less depth and have smaller number of nodes.Hence, though each tree will not be less predictive and less correlated, our model should be able to generalize better and prevent overfitting.
(3) max_features : This indicates how many of the features to consider at each split.
To optimize and search for the best hyperparameters, we will be using the Randomized Grid Search method!
n_estimators = [int(x) for x in np.arange(start = 10, stop = 2000, step = 10)]max_features = [0.5,'auto', 'sqrt','log2']min_samples_leaf = [1, 2, 4]bootstrap = [True, False]random_grid = {'n_estimators': n_estimators,               'max_features': max_features,               'min_samples_leaf': min_samples_leaf,               'bootstrap': bootstrap}# First create the base model to tunem = RandomForestRegressor()# Fit the random search modelm_random = RandomizedSearchCV(estimator = m, param_distributions = random_grid, n_iter = 100, cv = 3, verbose=2, random_state=42, n_jobs = -1)m_random.fit(X_train, y_train)m_random.best_params_
From our Randomized Grid Search, we found that the most optimized hyperparameters for our Random Forest Model to be those above.
The last step — building our final mode. To do so, we are going to drop the AGE and PTRATIO feature as discussed earlier.
We have obtained a validation RMSE score of 2.847, which is better than our original base model validation RMSE score of 3.021!
In addition, the final validation R2 score is 0.87 and this is better than the validation R2 score of the base model of 0.85. The R2 score tells us how much our model is able to explain the variation in the dataset. Having a score of 0.87 indicates that our model can explain 87% of the variation in the dataset.
Hence, we see the simple final model even with less input features performs much better!
While we have created a much better model already, let’s take a step back and explore our results further.
Previously, we observed there is a linear relationship between the median price of houses increases (MDEV) and number of rooms (RM).
Let’s try to do ggplot.
from pdpbox import pdpfrom plotnine import *ggplot(df_raw_train, aes('RM', 'MEDV'))+stat_smooth(se=True, method='loess')
Based on the ggplot above, we observe that the relationship between the RM and MEDV is not what we expected. For instance, one would generally expect the price of the house to increase with the number of rooms. However, we see in a drop in price between 4 rooms and 5 rooms, likewise a price drop between 8 rooms to 9 rooms.
The issue here is we are looking at univariate relationships and there are a lot of interactions between the features that are being lost in univariate plots. For instance, why did the price drop for houses that are 5 rooms as compared to 4 rooms and the price of a 6 room house is almost similar to a 4 room house?
Hence, to find the true relationship between RM and MEDV, we would need to do a partial independence plot (i.e. assuming that all other features being equal, how does the price of the house vary with the number of rooms) to see the true relationship.
def plot_pdp(feat,clusters=None,feat_name=None):    feat_name = feat_name or feat    p = pdp.pdp_isolate(m,X_train,feat)    return pdp.pdp_plot(p,feat_name,plot_lines=True,cluster=clusters is not None,n_cluster_centers = clusters)plot_pdp('RM')
Notice from the Partial Dependence Plot (PDP) above, after removing all other externalities we observe that the relationship between the RM and MEDV is almost a straight line (i.e. roughly linear), which is what we would expect. The yellow line represents the average MEDV of all the transactions.
Now let’s explore how LSTAT and RM influences the median housing prices.
feats = ['LSTAT','RM']p = pdp.pdp_interact(m,X_train,feats)pdp.pdp_interact_plot(p,feats)
Hence, looking at the PDP plots, we can safely conclude that the number of rooms (RM) influences the median housing prices (MEDV) linearly. The status of the population (LSTAT) influences MEDV inversely.
We have built a Random Forest model which has validation RMSE score of 3.021 and validation R2 score of 0.87.
We have also determined from our Random Forest model the key features that affects the median housing prices (MEDV) in Boston are (1) LSAT : Percentage of the lower population status (2) RM: The average number of rooms per dwelling (3) NOX: Concentration of Nitrogen Oxide (4) CRIM: The crime rate per capita by town.
The final model we have built is far from perfect. For instance, observe that the training R2 score in our final model is much higher 0.98 than our validation R2 score of 0.87. This indicates the final model is able to explain 98% of the variations in the training data while only explain 87% of the variation in the validation data (i.e. the final model is still overfitting the training data and does not generalize as well to the validation data).
One simple way to address overfitting could be to try to increase the size of our dataset to train our model on. Currently, we only have 406 entries in our dataset which is simply insufficient.
Congratulations and thank you for making it to the end. I sincerely hope you have learnt something new! Happy learning!😬