Exploratory Data Analysis!
By: Upasana Mahanta
Tags: Linear Regression, Predictive Analytics, Data Science, Data Analysis, Data Visualization

Ames Housing Project

By Unsplash

Overview

Predicting Ames House price by conducting some exploratory data analysis followed by processing the data based on the findings. This was an individual set up project to participate in Kaggle competition to build the best regression model to predict housing prices based on the test CSV. file provided to us.

For the data processing to start I was supplied two data sets one is training data sets (csv. file) containing all the information of Ames House Project, that data used to make a model which aimed at predicting the housing prices of the other csv. file labeled test.csv. The test.csv file has all the columns of the train.csv except it is missing the prices of each houses. Data sets contain total 2051 observations with 82 variables.

I imported all the required libraries panda, seaborn, matplotlib, pyplot, sklearn. I cleaned the datasets for further analysis. Next I split the train datasets into two separate datasets so I could build a model on one part of the data and test its performance on the other sets. This will be allowed me to predict how my model would perform on unseen data. My model was scored against the true sale price using Root Mean Squared Error and placed Twenty Fifth out of all General assembly fellows nationwide.

Reading data

I read the train data set, first converted this into a Data Frame to analyze.

Exploratory Data Analysis

· First started to do analysis on all features by correlation matrix to see the correlation amongst the features as well as the target price and then I created a heatmap to visualize it.

· Created distribution graph to see the distribution of all the features.

· Then started looking for features which could be used in my predictive model. Features that are strongly (positively and negatively) correlated with price will probably be good indicators/predictors of price. Features that have strong correlations amongst themselves have collinearity, from these features I will want to pick the strongest feature and exclude the others since a principle assumption of linear regression models is the independence of features.

You can see my code below:

def get_feature_groups():

""" Returns a list of numerical and categorical features,

excluding SalePrice and Id. """



num_features = df_train.select_dtypes(include=['int64','float64']).columns

num_features = num_features.drop(['Id','SalePrice']) # drop ID and SalePrice cat_features = df_train.select_dtypes(include=['object']).columns

return list(num_features), list(cat_features) num_features, cat_features = get_feature_groups() corr = df_train[['SalePrice'] + num_features].corr()

fig = plt.figure(figsize=(16,15))

ax = fig.add_subplot(111)

cmap = sns.diverging_palette(220, 10, as_cmap=True)

sns.heatmap(corr,

xticklabels=corr.columns.values,

yticklabels=corr.index.values,

cmap=cmap)

ax.xaxis.tick_top()

plt.setp(ax.get_xticklabels(), rotation=90)

plt.show()

Correlation between all features

corr = corr.sort_values('SalePrice', ascending=False)

plt.figure(figsize=(8,10))

sns.barplot( corr.SalePrice[1:], corr.index[1:], orient='h')

plt.show()

Correlation view of features by barplot

Cleaning

Null values are values which are missing. After checking the amount of null values which are in the features, I decided to drop these.

Feature Engineering

I tried some feature engineering on some interesting features and tried to find out there any impact on sale price. I found on neighborhood, year sold, Ground living area, year sold of the houses shows some interesting linearity on sale price. I created some plots on these.

You can find my all codes below

df_train.groupby('Neighborhood').Id.count().\

sort_values().\

plot(kind='barh', figsize=(10,10))

plt.title('What neighborhoods are houses in?')

plt.show()

From the plots it shows that It would also be interesting to augment the dataset with additional information as it relates to neighborhoods.

data = pd.concat(

[

df_train.groupby('Neighborhood').mean()['SalePrice'],

df_train.groupby('Neighborhood').count()['Id']

],



axis=1)

f, ax = plt.subplots()

sns.stripplot(data.sort_values(by='SalePrice').SalePrice, data.sort_values(by='SalePrice').index, orient='h', color='red');

(For an example)

#Neighborhood 1: MeadowV:BrDale

#Neighborhood 2: BrkSide:Mitchel

#Neighborhood 3: SawyerW:Veenker

#Neighborhood 4: Timber:NoRidge

Interestingly, I see four clusters of neighborhoods above. we can categorize them and can approach which neighborhoods showing more expensive. It Looks like a good chunk of houses are in North Ames, Collect Creek, and Old Town, with few houses in Bluestem, Northpark Villa and Veenker.

Based on the below (histogram) plot, I would identify the following construction periods.(Codes are below)

plt.hist(df_train['Year Built'])

Oldest house built in 1872. Newest house built in 2010. Not much action in the 80s apparently. Looks like majority of houses were built in the 50s and after, 1985–2010: Extended growth leading up to the Great Recession.

Also found some interesting area covers means how big houses are made in a lot area, but there are also found some outlier .(Codes below)

sqft_to_acres = 43560.

print('The average lot is {:,.2f} acres, the median {:,.2f} acres'.format(

df_train['Lot Area'].mean()/sqft_to_acres, df_train['Lot Area'].median()/sqft_to_acres))

print('The biggest lot is {:,.2f} acres, the smallest {:,.2f} acres'.format(

df_train['Lot Area'].max()/sqft_to_acres, df_train['Lot Area'].min()/sqft_to_acres))

(df_train['Lot Area']/sqft_to_acres).hist(bins=50, rwidth=.7, figsize=(8,4))

plt.title('How big are lots? (in acres)')

plt.show()

Below plot shows that Ames seems to have been immune at the Recession time.(Codes are below)

sns.boxplot(x=df_train['Yr Sold'], y=df_train['SalePrice'])

Then I found some interesting on month and year sold which shows the seasonality pattern reflection.

df_train.groupby(['Yr Sold','Mo Sold']).Id.count().plot(kind='bar', figsize=(14,4))

plt.title('When where houses sold?')

plt.show()

Below can find how houses are big with Square Footage wise.(Codes are below)

print('The average house has {:,.0f} sq ft of space, the median {:,.0f} sq ft'.format(

df_train['Gr Liv Area'].mean(), df_train['Gr Liv Area'].median()))

print('The biggest house has {:,.0f} sq ft of space, the smallest {:,.0f} sq ft'.format(

df_train['Gr Liv Area'].max(), df_train['Gr Liv Area'].min()))

df_train['Gr Liv Area'].hist(bins=21, rwidth=.8, figsize=(8,4))

plt.title('How big are houses? (in sq feet)')

plt.show()

Square Footage of Ground Living Area

After doing all of this, I built a linear regression model on my selected features and then used this model on the test split of the data and got an RMSE score, as well as R-squared score for the test data.

Results

My linear regression model did very well. The RMSE was 35792.99 which stands for the calculated difference between each predicted price and the actual price of all the homes.

It had a 0.8586 R-Squared score on the test and a R-Squared score of 0.7974 for the training data. This shows the model is doing a great job of predicting both the training set and the testing set since there is a small drop in the R-Squared score on the testing set. Model is little overfit. Overall, the model is accounting for 85.86% of the variance in the price.

The feature with the highest correlation to sale price was my engineered feature for total square footage. Other important features were the overall quality of the home, Year Re Modification or adding (another feature I engineered), Basement uniform Square feet and lot area.

This is a scatter plot of linearity between actual price against predicted price.(Codes are below)

plt.scatter(df_train['Total_Sqft'], df_train['SalePrice']);

Conclusion

At the end, my initial strategy for feature engineering and selection was good since it performed well. With a RMSE of 35792.99 I am confident that my model can perform adequately when predicting house prices in Ames, Iowa provided enough information about the house features.

Recommendation

By regularizing the model by Lasso or ridge ,model may get more accurate prediction.I would have also possibly looked into creating some interactions. By applying PCA(Principal Component Analysis) method on features may also can give the more accurate prediction .Another feature I could have created was an Age feature which stated the age in years of each home.

By applying my own intuition to my feature selection,I was able to make a model which predicted the prices of the competition data with an RMSE of 35792.99.

This submission placed 25th out of all General Assembly fellows nationwide and placed 2nd position out of all General Assembly fellows local wide.

For details Please visit my GitHub profile below:

https://github.com/upad0412/Kaggle-challenge_Ames-Housing-Project