House Prices in Ames, Iowa — Working with Kaggle
This summer kicked off my interest with Data Science, starting of a journey of self-learning and exploration.
Category: Data Science
Author: Raghav Ravisankar

House Prices in Ames, Iowa — Working with Kaggle
This summer kicked off my interest with Data Science, starting of a journey of self-learning and exploration.
House prices are an interesting dataset to look at because they usually have a lot of features in them and not all of the columns and values have a direct correlation to the price of the house. House prices depend on a lot of factors, mostly because of the general variation of preferences across the population, one thing however that is trivial for us to understand that bigger means more expensive. That is why house prices datasets are interesting to analyze and visualize, even more so, building a predictor to the prices of the house from the features of the house. This makes the dataset good to work with while learning the art of data science.
You can find my Jupyter notebook for the data analysis here and the dataset here.
“A large suburban house with a garden on a sunny day” by Jesse Roberts on Unsplash
Here I am using a very structured way to do this data analysis, and here are the steps I am taking. Each step will have a heading in this article.
Question or problem definition.
Acquire training and testing data.
Wrangle, prepare, cleanse the data.
Analyze, identify patterns, and explore the data.
Model, predict and solve the problem.
Visualize, report, and present the problem solving steps and final solution.
Supply or submit the results.
Question or problem definition
So of to Kaggle, where there is this amazing dataset is located. In this dataset, we have a list of house prices and information about the houses themselves from Ames, Iowa. Since it is a very descriptive dataset, there are a lot of things we can do with this dataset. However, to keep it short and sweet, let us define the scope of this analysis to be to complete a comprehensive exploratory data analysis (EDA) of the house prices and to build a regressor model of the prices of the houses based on the columns in the training set to predict house prices in the test set/validation set.
Before you get started with this notebook, it is highly recommended to go through this paper from the author of the dataset to get an overview of the dataset and how the usage is intended. This notebook uses techniques used by Alexandru Papiu in his Jupyter notebook. I figured learning how to approach a dataset with some help from the community is a good start.
Acquire training and testing data
Now let us get right down to it. Let us import all the required packages and training and test set first, and display the structure of the dataset using the shape and head functions
This is the how the dataset looks like returned after the calling df_train.head(5)
Here are the columns in the dataset, that we find out using this command.
These are the columns that we get from the command df_train.columns
Now that we are aware of the structure of the dataset, let us take a closer look at the dataset to analyze the features. This brings us to the next step, wrangling, prepare and cleanse the data.
Wrangle, prepare, cleanse the data
In this section we are going to go through multiple techniques used to wrangle, prepare, cleanse the data.
Here is a breakdown of the techniques we are going to use:
Finding out if there are any duplicate entries in the dataset
Finding out about the type of columns: numeric and categorical
Finding out the missing data in each column
Filling in missing values with relevant data
Finding and removing outliers from the data
Lets us get started right away!
Finding out if there are any duplicate entries in the dataset
Duplicate entries affect the accuracy of the predictor we intend to build, as the features of a specific house will affect the predictions of the classifier if there are duplicate entries of it. Here is the code to check if the data has duplicate columns.
Finding out about the type of columns: numeric and categorical
Now we have to see which columns have numerical columns and which columns have categorical columns in order to build features from each column. This would be used later in order to do normalization and fixing the numerical columns in the dataset.
These are the numerical and categorical columns in the dataset
Finding out the missing data in each column
Missing columns in the data can either signify lack of data or a specific negative value within a categorical column of the data.
Now we identify the columns in the dataset that are missing, and to find out to find out the missing data for each column, we would be using this piece of code to find out the number of missing rows in each column.
The Y-axis here is the number of missing entries for each column in the x-axis.
We can clearly see the columns PoolQC, MiscFeature, Alley, Fence have a lot of values that are missing. But taking a closer look at the data, we understand the some part of each of these columns have information regarding the Alley or PoolQC and Fence. We will be filling in the data with the mean value of each of the columns later during the analysis.
Filling in missing values with relevant data
We will be doing this later when preparing the data for the model. Now let us move on to removing the outliers in the data
Finding and removing outliers from the data
Here we are trying to identify the outliers in the data which we cannot use in the model because it skews the analysis towards values that are unlikely. Here we intend to visualize the data to find out the spread of the data so we can remove the outliers in the data.
This is the entire dataset with all the outliers in the data.
Here we are removing houses from the dataset that are larger than 4000 sqft and more than 700,000 in value because they look like outliers in the dataset.
Here is the data without the outliers
Analyze, identify patterns, and explore the data.
Finding out the distribution of data within the dataset
Now to find out how each of the variables are distributed amongst the dataset, we can pivot the table into its constituent features and find out how each variable is distributed amongst the dataset using pd.melt http://pandas.pydata.org/pandas-docs/stable/generated/pandas.melt.html
For this we can only use numeric features and not any categorical features because of the nature of sns.FacetGrid module we are going to use to find correlations between the dataset.
Distribution of various columns in the dataset and if we can model the data. Cleaner curves with rectangles within them are more accurately modeled.
House prices over the years
Now let us analyze the house prices over the course of the years and visualize this on a histogram. In this graph, each x column would represent the year and y-axis would represent the house prices for the year.
Prices on the y-axis and year of sale on the x-axis.
Overall condition of the house
Now let us plot the correlation between three columns: Year the house was sold, Overall condition of house and Living Area in the house. I wanted to visualize these leads in a seaborn pairplot so I used the following code to implement this.
A nice complex graph to find out the relation between these three columns
Correlation matrix
Now let us plot the correlation matrix for each of the columns in the dataset. This is done to have a clear picture on the correlation between columns in the dataset.
Correlation matrix for the dataset, the darker the color, the more the correlation between the two columns.
Normalization
To model and predict the problem, we first need to understand that our data by itself has to be represented in a form that is easy for the predictor to learn from, hence we have to prepare the data; normalize the numerical columns in the data and encode the categorical columns in the dataset, in order to build an effective model for the problem. This warrants a detailed explanation, and is based on the ability of the algorithm to work better given the need of the algorithm minimize the distance of the data point from the classifier’s prediction. This is also based on the ability of the computer to be able to deal with computation of smaller numbers more easily because of their representation.
One example of normalization is using a log function: Taking logs means that errors in predicting expensive houses and cheap houses will affect the result equally. Notice the x-axis in the following graphs, they indicate the range of values taken up by the column. This following code show us how to get the graph.
After and before of taking the log of the prices column
We take the log of the SalePrice in order to build a normalized model of the data, the reason why we do this is because predictors work better when the data is not distributed sparsely.
In this we are going to use the skew function to find out the columns in which the numerical data is not too skewed here.
I learned how to do this from https://www.kaggle.com/apapiu/regularized-linear-models where the skew function essentially returns the skewedness of the distribution of values in a column. We do not want columns with too much of skewedness because that affects the value returned by the log function.
In the following steps we are going to compute the skewedness and normalize the numerical columns in the data
Now let us encode the categorical columns in the data with the following code. This is done in order to convert the categorical columns into a feature based on the values in the dataset pd.get_dummies() takes a data of the form [a, b, b, c, null] and converts it into [0, 1, 1 , 2, 3]. This function has both the property of categorizing each value in the dataset and filling in the null values.
Now let us make the test and training set from the normalized data that we will use for fitting the model and predictions.
Model, predict and solve the problem
In this analysis we are going to try implement regression models from sklearn to try to fit this data. We can find more about these basic regression models in the link which has a detailed explanation for each model or look at sklearn documentation
In this analysis we would using the RMSE value to find out the error of the model for a given set of the data. The root mean square error (RMSE) of the dataset is the square root of sum of the squared distance between the predictions of the classifier and the actual value in the dataset. In simple terms, Root mean square error for a dataset is a way to measure accuracy of the classifier with respect to the dataset. Closer to 1, worse the model. Closer to 0 better the model.
Here we are going to define the RMSE on the training set so we can train our classifiers on the training set.
Ridge Regression
Now let us train our models on a Ridge regressor from sklearn.linear_model package. Ridge regularization has a parameter alpha which is used to tune the data to fit the model to the training or validation data. In this code below we are going to train the Ridge regressor and store the RMSE value in cv_ridge_train
Now let us plot the values of cv_ridge_train in order to find out the alpha for which we will have the lowest RMSE
Finding out the best regularization parameter, here its probably 5.
Now let us find out the lowest RMSE for the Ridge regressor.
So the RMSE is about 0.110, which is pretty good.
Lasso Regression
Implementing a lasso regressor is easy, this regressor works by training on columns that are relevant to the dataset and does not include data from columns that are not needed.
The RMSE is 0.108 which is lesser than the Ridge regressor model. The following code allows to me to find out the number of features that the Lasso model used for training and the columns its discarded.
Lasso picked 105 variables and eliminated the other 182 variables. Nice!
XGBoost model
Here we are going to try out the XGBoost model which is created as a package which implements the Gradient Boosting Regression. Here we use the XGBoost as a blackbox with some parameters. Here you can find a detailed explanation on how this model works. Here is the documentation for this package.
The following code is going to give us the RMSE for the training and test set for XGBoost model and plot the data.
Blue is the RMSE mean for the test set, Orange is the RMSE mean for the training set. Y-axis is the value of RMSE and x-axis signifies the number of epochs in training.
The following code below is going to fit the XGBoost model on the training data
The following code is going to predict values from the Lasso model and the XGBoost model and convert them back to normal after normalization.
The following code shows the correlation between the predictions of the two regressors, showing that the predictions from the two regressors have a pattern.
Correlation between predictions on the XGB model and predictions on the Lasso model.
Now using the predictions from two different predictors, we can combine the models using a simple linear function
Supply or submit the results.
Now we are going to export the data into a csv file to supply and submit the predictions
The end!
Thanks for reading my article, the code for this analysis is available here. You can follow me on Medium at Raghav Ravisankar. Please let me know your feedback through the comments!