Kaggle House Prices Prediction with Linear Regression and Gradient Boosting
By: Rohan Paul
Tags: Python, Kaggle, Machine Learning, Datascience, Kaggle Competition

My Kaggle Notebook Link is here

As I intended this Notebook to be published as a blog on Linear Regression, Gradient Descent function and some EDA, so in the first 50% to 60% of this notebook I have mainly discussed the theory around those topics from mainly a beginner perspective.

import numpy as np

import pandas as pd

import matplotlib.pyplot as plt

import seaborn as sns

from sklearn.pipeline import Pipeline

from sklearn.preprocessing import LabelEncoder, OneHotEncoder

from sklearn.compose import ColumnTransformer

from sklearn.linear_model import LinearRegression

from sklearn.linear_model import Ridge

from sklearn.linear_model import Lasso

from sklearn.ensemble import RandomForestRegressor

from sklearn.impute import SimpleImputer

from sklearn.preprocessing import StandardScaler

from sklearn.model_selection import train_test_split

from sklearn import metrics

from sklearn.metrics import mean_squared_log_error

from sklearn.metrics import mean_squared_error

from xgboost import XGBRegressor

import math

The evaluation criteria for this Kaggle Competition is RMSLE — “Submissions are evaluated on Root-Mean-Squared-Error (RMSE) between the logarithm of the predicted value and the logarithm of the observed sales price. (Taking logs means that errors in predicting expensive houses and cheap houses will affect the result equally.)”

The Root Mean Squared Log Error (RMSLE) can be defined using a slight modification on sklearn’s mean_squared_log_error function, which itself a modification on the familiar Mean Squared Error (MSE) metric.

def root_mean_squared_log_error(y_validations, y_predicted):

if len(y_predicted) != len(y_validations):

return 'error: mismatch in number of data points between x and y'

y_predict_modified = [math.log(i) for i in y_predicted]

y_validations_modified = [math.log(i) for i in y_validations]



return mean_squared_error(y_validations_modified, y_predict_modified, squared=False) df_train = pd.read_csv('../input/house-prices-advanced-regression-techniques/train.csv')

df_test = pd.read_csv('../input/house-prices-advanced-regression-techniques/test.csv') df_train.head()

df_test.head()

First, I am getting a basic description of the data, to look quickly at the overall data to get some simple, easy-to-understand information. This is just to get a basic feel for the data. Using describe() function to get various summary statistics that exclude NaN values.

df_train.describe()

Now that I have got a general idea about your data set, it’s also a good idea to take a closer look at the data itself. With the help of the head() and tail() functions to check out the first and last lines of my DataFrame, respectively.

df_train.tail()

# Get the overall concise summary of the DataFrame

df_train.info() <class 'pandas.core.frame.DataFrame'>

RangeIndex: 1460 entries, 0 to 1459

Data columns (total 81 columns):

# Column Non-Null Count Dtype

--- ------ -------------- -----

0 Id 1460 non-null int64

1 MSSubClass 1460 non-null int64

2 MSZoning 1460 non-null object

3 LotFrontage 1201 non-null float64

4 LotArea 1460 non-null int64

5 Street 1460 non-null object

6 Alley 91 non-null object

7 LotShape 1460 non-null object

8 LandContour 1460 non-null object

9 Utilities 1460 non-null object

10 LotConfig 1460 non-null object

11 LandSlope 1460 non-null object

12 Neighborhood 1460 non-null object

13 Condition1 1460 non-null object

14 Condition2 1460 non-null object

15 BldgType 1460 non-null object

16 HouseStyle 1460 non-null object

17 OverallQual 1460 non-null int64

18 OverallCond 1460 non-null int64

19 YearBuilt 1460 non-null int64

20 YearRemodAdd 1460 non-null int64

21 RoofStyle 1460 non-null object

22 RoofMatl 1460 non-null object

23 Exterior1st 1460 non-null object

24 Exterior2nd 1460 non-null object

25 MasVnrType 1452 non-null object

26 MasVnrArea 1452 non-null float64

27 ExterQual 1460 non-null object

28 ExterCond 1460 non-null object

29 Foundation 1460 non-null object

30 BsmtQual 1423 non-null object

31 BsmtCond 1423 non-null object

32 BsmtExposure 1422 non-null object

33 BsmtFinType1 1423 non-null object

34 BsmtFinSF1 1460 non-null int64

35 BsmtFinType2 1422 non-null object

36 BsmtFinSF2 1460 non-null int64

37 BsmtUnfSF 1460 non-null int64

38 TotalBsmtSF 1460 non-null int64

39 Heating 1460 non-null object

40 HeatingQC 1460 non-null object

41 CentralAir 1460 non-null object

42 Electrical 1459 non-null object

43 1stFlrSF 1460 non-null int64

44 2ndFlrSF 1460 non-null int64

45 LowQualFinSF 1460 non-null int64

46 GrLivArea 1460 non-null int64

47 BsmtFullBath 1460 non-null int64

48 BsmtHalfBath 1460 non-null int64

49 FullBath 1460 non-null int64

50 HalfBath 1460 non-null int64

51 BedroomAbvGr 1460 non-null int64

52 KitchenAbvGr 1460 non-null int64

53 KitchenQual 1460 non-null object

54 TotRmsAbvGrd 1460 non-null int64

55 Functional 1460 non-null object

56 Fireplaces 1460 non-null int64

57 FireplaceQu 770 non-null object

58 GarageType 1379 non-null object

59 GarageYrBlt 1379 non-null float64

60 GarageFinish 1379 non-null object

61 GarageCars 1460 non-null int64

62 GarageArea 1460 non-null int64

63 GarageQual 1379 non-null object

64 GarageCond 1379 non-null object

65 PavedDrive 1460 non-null object

66 WoodDeckSF 1460 non-null int64

67 OpenPorchSF 1460 non-null int64

68 EnclosedPorch 1460 non-null int64

69 3SsnPorch 1460 non-null int64

70 ScreenPorch 1460 non-null int64

71 PoolArea 1460 non-null int64

72 PoolQC 7 non-null object

73 Fence 281 non-null object

74 MiscFeature 54 non-null object

75 MiscVal 1460 non-null int64

76 MoSold 1460 non-null int64

77 YrSold 1460 non-null int64

78 SaleType 1460 non-null object

79 SaleCondition 1460 non-null object

80 SalePrice 1460 non-null int64

dtypes: float64(3), int64(35), object(43)

memory usage: 924.0+ KB

Lets check what are the numerical features in the df_train dataset

numeric_features = df_train.select_dtypes(include=[np.number])

numeric_features.columns Index(['Id', 'MSSubClass', 'LotFrontage', 'LotArea', 'OverallQual',

'OverallCond', 'YearBuilt', 'YearRemodAdd', 'MasVnrArea', 'BsmtFinSF1',

'BsmtFinSF2', 'BsmtUnfSF', 'TotalBsmtSF', '1stFlrSF', '2ndFlrSF',

'LowQualFinSF', 'GrLivArea', 'BsmtFullBath', 'BsmtHalfBath', 'FullBath',

'HalfBath', 'BedroomAbvGr', 'KitchenAbvGr', 'TotRmsAbvGrd',

'Fireplaces', 'GarageYrBlt', 'GarageCars', 'GarageArea', 'WoodDeckSF',

'OpenPorchSF', 'EnclosedPorch', '3SsnPorch', 'ScreenPorch', 'PoolArea',

'MiscVal', 'MoSold', 'YrSold', 'SalePrice'],

dtype='object')

numeric features head

numeric_features.head()

Now a general description of SalePrice

# Now, as we will be predicting 'SalePrice' lets see description of that column

df_train['SalePrice'].describe() count 1460.000000

mean 180921.195890

std 79442.502883

min 34900.000000

25% 129975.000000

50% 163000.000000

75% 214000.000000

max 755000.000000

Name: SalePrice, dtype: float64

For a regression problem, the most important thing to do is to understand the distribution of your target. If a target ranges between 1 and 10, and after training our model we get a mean absolute error of 5, we can tell that the error is large in this context. However, the same error for a target that ranges between 500,000 and 1,000,000 is negligible.

If you have numeric type dataset and want to visualize in histogram

then the seaborn histogram will help you. For this seaborn distplot function responsible to plot it.

On y-axis give the numeric dataset

On x-axis gives bins. It means distribute given dataset in a particular range and show in bars

e.g. bins= 18 and bins = 40

sns.distplot(df_train['SalePrice']) <matplotlib.axes._subplots.AxesSubplot at 0x7f832c513b50>

From the above distplot, the ‘SalePrice’ is dense between 100k and 250k, but there are larger outliers on the right side, i.e the expensive price range.

Now, let’s have a look at the greater living area (square feet) against the sale price:

# first wo quickly view all the column names in the data

# print(df_train.columns)

# for above I could also use - data.columns

# Checking - 'GrLivArea'

data = pd.concat([df_train['SalePrice'], df_train['GrLivArea']], axis=1)

data.plot.scatter(x='GrLivArea', y='SalePrice', ylim=(0, 800000), s=32) 'c' argument looks like a single numeric RGB or RGBA sequence, which should be avoided as value-mapping will have precedence in case its length matches with 'x' & 'y'. Please use a 2-D array with a single row if you really want to specify the same RGB or RGBA value for all points.











<matplotlib.axes._subplots.AxesSubplot at 0x7f8316e54bd0>

Its generally expected that larger living area comes with higher price. This chart shows thats generally true. But there are some 2–3 “cheap” houses with large living area Now lets explore the “TotalBsmtSF” column — Total square feet of the basement area.

# Checking - 'TotalBsmtSF'

data = pd.concat([df_train['SalePrice'], df_train['TotalBsmtSF']], axis=1)

data.plot.scatter(x='TotalBsmtSF', y='SalePrice', ylim=(0, 800000)) 'c' argument looks like a single numeric RGB or RGBA sequence, which should be avoided as value-mapping will have precedence in case its length matches with 'x' & 'y'. Please use a 2-D array with a single row if you really want to specify the same RGB or RGBA value for all points.











<matplotlib.axes._subplots.AxesSubplot at 0x7f83170ed210>

We can see that basement area indeed may have good predictive power for the model.

Now lets check “OverallQual” — overall material and finish quality. This one may be more subjective feature.

Now list of variables that contain year information, because I will compare the difference between All years feature with SalePrice

year_feature = [feature for feature in numeric_features if 'Yr' in feature or 'Year' in feature]

year_feature ['YearBuilt', 'YearRemodAdd', 'GarageYrBlt', 'YrSold'] # Checking - 'YearBuilt'

data = pd.concat([df_train['SalePrice'], df_train['YearBuilt']], axis=1)

data.plot.scatter(x = 'YearBuilt', y = 'SalePrice', ylim=(0, 800000)) 'c' argument looks like a single numeric RGB or RGBA sequence, which should be avoided as value-mapping will have precedence in case its length matches with 'x' & 'y'. Please use a 2-D array with a single row if you really want to specify the same RGB or RGBA value for all points.











<matplotlib.axes._subplots.AxesSubplot at 0x7f8317109410>

# Checking - 'YearBuilt'

data = pd.concat([df_train['SalePrice'], df_train['GarageYrBlt']], axis=1)

data.plot.scatter(x = 'GarageYrBlt', y = 'SalePrice', ylim=(0, 800000)) 'c' argument looks like a single numeric RGB or RGBA sequence, which should be avoided as value-mapping will have precedence in case its length matches with 'x' & 'y'. Please use a 2-D array with a single row if you really want to specify the same RGB or RGBA value for all points.











<matplotlib.axes._subplots.AxesSubplot at 0x7f83172890d0>

# Checking - 'YearBuilt'

data = pd.concat([df_train['SalePrice'], df_train['YearRemodAdd']], axis=1)

data.plot.scatter(x = 'YearRemodAdd', y = 'SalePrice', ylim=(0, 800000)) 'c' argument looks like a single numeric RGB or RGBA sequence, which should be avoided as value-mapping will have precedence in case its length matches with 'x' & 'y'. Please use a 2-D array with a single row if you really want to specify the same RGB or RGBA value for all points.











<matplotlib.axes._subplots.AxesSubplot at 0x7f832c65d090>

# Checking - 'YearBuilt'

data = pd.concat([df_train['SalePrice'], df_train['YrSold']], axis=1)

data.plot.scatter(x = 'YrSold', y = 'SalePrice', ylim=(0, 800000)) 'c' argument looks like a single numeric RGB or RGBA sequence, which should be avoided as value-mapping will have precedence in case its length matches with 'x' & 'y'. Please use a 2-D array with a single row if you really want to specify the same RGB or RGBA value for all points.











<matplotlib.axes._subplots.AxesSubplot at 0x7f832d333e90>

# Checking - TotRmsAbvGrd' i.e. total rooms above grade

data = pd.concat([df_train['SalePrice'], df_train['TotRmsAbvGrd']], axis=1)

data.plot.scatter(x = 'TotRmsAbvGrd', y = 'SalePrice', ylim=(0, 800000)) 'c' argument looks like a single numeric RGB or RGBA sequence, which should be avoided as value-mapping will have precedence in case its length matches with 'x' & 'y'. Please use a 2-D array with a single row if you really want to specify the same RGB or RGBA value for all points.











<matplotlib.axes._subplots.AxesSubplot at 0x7f830b806050>

data = pd.concat([df_train['SalePrice'], df_train['OverallQual']], axis=1)

f, ax = plt.subplots(figsize=(14, 8))

fig = sns.boxplot(x='OverallQual', y='SalePrice', data=data)

fig.axis(ymin= 0, ymax=800000) (-0.5, 9.5, 0, 800000)

# Now lets view the top 8 correlated features with the sale price:

corr_mat = df_train.corr()

f, ax = plt.subplots(figsize=(12, 9))

sns.heatmap(corr_mat, vmax=0.8, square=True) <matplotlib.axes._subplots.AxesSubplot at 0x7f830bbf9850>

Now a simple Feature Engineering — of finding out the top-most correlated columns with SalesPrice. Later while in our Model will use only these top-most important features to predict SalePrices.

Feature Engineering — Find out the most correlated columns with SalesPrice and use only those columns in modelNow the big beautiful scatter plot with a hell of lot of visual data in a single plot

top_correlations = df_train.corr()

top_feature_columns = top_correlations['SalePrice'][top_correlations['SalePrice'].values > 0.2].index.values

top_feature_columns array(['LotFrontage', 'LotArea', 'OverallQual', 'YearBuilt',

'YearRemodAdd', 'MasVnrArea', 'BsmtFinSF1', 'BsmtUnfSF',

'TotalBsmtSF', '1stFlrSF', '2ndFlrSF', 'GrLivArea', 'BsmtFullBath',

'FullBath', 'HalfBath', 'TotRmsAbvGrd', 'Fireplaces',

'GarageYrBlt', 'GarageCars', 'GarageArea', 'WoodDeckSF',

'OpenPorchSF', 'SalePrice'], dtype=object)

And now another heatmap with ONLY the top 12 correlated features And within the top 12 features I can see ‘GarageYrBlt’ included but, we know that this column has many empty fields. Hence I would like to fill them with zeros.

# Handling Missing Values for 19 features which have missing values mentioned above

df_train['GarageYrBlt'] = df_train['GarageYrBlt'].fillna(0)

# filling in missing GarageYrBuilt values with zeros.

# But this may not be the most logical approach - refer to this discussion below for mor perspective

# https://www.kaggle.com/c/house-prices-advanced-regression-techniques/discussion/60143



# similary fillingup na valuse for couple of other features

df_train['LotFrontage'] = df_train.groupby('Neighborhood')['LotFrontage'].transform(lambda x: x.fillna(x.median()))

df_train['MasVnrArea'] = df_train['MasVnrArea'].fillna(0)



heat_map_with_top_correlated_features = np.append(top_feature_columns[-12:], np.array(['SalePrice']))

pearson_correlation_coefficients = np.corrcoef(df_train[heat_map_with_top_correlated_features[::-1]].T)

plt.figure(figsize=(16,16))

sns.set(font_scale=1)

with sns.axes_style('white'):

sns.heatmap(pearson_correlation_coefficients, yticklabels=heat_map_with_top_correlated_features[::-1], xticklabels=heat_map_with_top_correlated_features[::-1], fmt='.2f', annot_kws={'size': 10}, annot=True, square=True, cmap=None)

Now lets glance over the above heatmap to understand some points just visualy looking at the correlation number

‘OverallQual’, ‘GrLivArea’ and ‘TotalBsmtSF’ have strong correlation with ‘SalePrice’.

While ‘GarageCars’ and ‘GarageArea’ also have strong correlation, but they are mostly mutually dependent i.e. they are NOT linearly independent of each other, i.e. there is a high-multicollinearity (0.88 as we can see in the figure) between them. Because, the number of cars that fit into the garage is dependent of the garage area. Hence, we just need one of these variables in our analysis (we can decide to keep ‘GarageCars’ as its correlation with ‘SalePrice’ is higher).

Same mutual dependence applies to the two features ‘TotalBsmtSF’ and ‘1stFloor’ . We will take only ‘TotalBsmtSF’ in our feature-engineering.

AND ALSO ‘TotRmsAbvGrd’ and ‘GrLivArea’, and we will only take ‘GrLivArea’

So from the below array of top correlated features lets make our selection

sns.set()

cols = ['SalePrice', 'OverallQual', 'GrLivArea', 'GarageCars', 'TotalBsmtSF', 'YearBuilt', 'FullBath', 'OpenPorchSF',

'WoodDeckSF', '2ndFlrSF', 'YearRemodAdd', 'MasVnrArea', 'LotFrontage', 'LotArea']

sns.pairplot(df_train[cols], height=4)

plt.show()

Checking out missing data