King County House Sales Price Inference Model
By: Jan Abdullah
Tags: Python, Inference, Regression, Statistics, Data Science

Breakdown of each phase

Pre-Processing

In this stage we load the dataset for the first time and take some quick glances at the dataframe’s contents. The initial formatting consists of converting all null values to 0, and converting any date columns to Python’s date format. Then, we categorize the variables to their respective classifications, being either dependent, continuous, or categorical.

The next step is cleaning the data. The cleaning.py module contains useful functionality for displaying diagnostics reports on each variable, and calling methods to clean the variables as needed. Here’s an example of a report on the ‘price’ variable:

(The formatting for this will look a little bit different in the notebook in the GitHub repo, but the info should all still be easy to read.)

The report displays a frequency distribution histogram plot with kernel density estimation (KDE) overlay, a summary report containing summary statistics, a “Top 10” and “Bottom 10” value counts list, and a “Zeros” report. With this info, we want to, if necessary, specify a range of values of the variable to keep, so as to exclude outliers. Firstly, we would take a look at the distribution and note the shape, and deduce where the outliers may lie. Looking at the ‘price’ distribution, we note strong positive skewness and a highly leptokurtic distribution, signifying that the distribution has a lot of outliers on the right tail. In layman’s terms, this means that there are a number of very higher-than-average prices in the dataset. Another thing to note is the magnitude of the difference between the mean and median, and the direction of the deviation of the mean away from the median. Looking at the distribution, we can presume that the outliers on the right tail will push the mean to the right of the median. We see exactly this in the summary report, where median is ~450k yet mean is ~540k. The next place to look at is the value counts, which would flag any singular outrageously high outlier, and we see just that, there is one value in the variable with a value of 7.7 mio, this is also the maximum value. Then, we look at this maximum and compare it against the upper quantile (97.5% and 99.5%) figures, and we see that these are ~1.49 mio and ~2.45 mio respectively, further proving that this 7.7 mio is a heavy outlier. So we definitely want to get rid of that value. An additional step to keep this safe, but should be done in moderation, is to set the maximum in a way where more outliers can be caught and removed. After testing multiple ranges to find a balance between improving the distribution and not losing too much data, we settle on a range of 130k — 2 mio.

With that, we run the report again after removing the out-of-range values.

We can note that while the new cleaned distribution does not resemble a perfect standard normal distribution, it is still a great improvement from the original distribution. Remember, we do not want to lose too much data in hopes of achieving a standard normal distribution for variables, as it may make the model less reliable. In the case of housing prices, positive skewness and high kurtosis may just be an inherent nature of the subject, as housing prices can only start from 0 and increase continuously. We know in the real world that there are a lot of averagely valued houses, a good number of below-average valued houses, and some highly-above-average valued houses that may not be much in numbers, but when just focusing on the variable of ‘price’, can really skew the distribution.

Also note the ‘Zeros’ report — this piece of information just tells us how many 0s and the percentage of 0s are in the variable. If the percentage is too high, we may consider dropping the variable completely. Luckily, there are no 0s in the ‘price’ variable.

This cleaning methodology was used for all the variables in the dataset, and an in-depth explanation justifying the cleaning configuration for each variable can be found in the markdown of the preprocessing.ipynb notebook.

(Note: the output after cleaning is saved down to .csv, for EDA to be done on it in the EDA.ipynb notebook.)

Next, we’ll take a look at the correlation matrix of all the variables via a heatmap. We want to avoid multi-collinearity in our model so we’ll need to take a look at the highly correlated independent variable pairs and remove some variables accordingly.

Take a look at the heatmap below.

Let’s focus on the variable pairs with correlation coefficients greater than 0.75, below:

In the notebook, you’ll see that of all these variables, we only keep the ‘sqft_living’ and ‘sqft_lot’ variables. The other variables are a bit redundant. The ‘sqft_living’ variable represents the total house square footage and ‘sqft_lot’ variable represents the total lot square footage. The other variables represent info such as the total house square footage except for basement, or the square footages of the 15 nearest neighbors. While sizes of neighbors’ houses can vary from the size of the house in question, we deem it generally not enough to provide any insight we couldn’t otherwise receive from the house in question itself.

The final step in preprocessing is to transform variables to make it more interpretable by the model. This requires log transforming and standardizing continuous variables, and one-hot encoding categorical variables. Also, within categorical variables, we can also apply binning techniques to reduce the number of dummy variables created. In this project, we apply binning to the ‘zipcodes’ variable, and the binning configuration can be found in the JSON configs folder.

Let’s take a look at the distribution of continuous variables after transformation:

While not all of these resemble standard normal distributions, the standardization helps us nonetheless by creating a normalized scale. Also, some variables don’t have enough variety in the values themselves to resemble the distribution of an actual continuous variable. These variables are our “discrete” variables, there are a limited range of values these variables can naturally be, but we treat them as continuous variables for simplicity.

This completes the preprocessing phase.

EDA

In this stage, we take a look at the cleaned dataframe (variables are still in their original form, i.e. not log transformed or one-hot encoded), and formulate some questions and answer them. We expect to see the answers to these questions reflected in our final model, amongst other things.

For our EDA, we take a zipcode-centric approach and formulate questions around that. First, zipcode values were converted to their respective bin value per the binning configuration for ease of interpretation.

Questions:

1: What are the Month-on-Month(MoM) average sales prices by zipcode group? What are the trends?

The datasets’ date range spans May 2014 — May 2015, we calculate the mean sales price per zipcode bin, per month and graph them below:

We can see that trends for all zipcode bins are flat overall, and that zipcode C had the highest average sales prices for every month in this time period. The neighborhoods in this bin may be more socio-economically well-off relative to neighborhoods in other bins, thus these neighborhoods may overall be nicer, and house values hgiher. So in our model, we expect to see that the dummy variable for category ‘zipcode C’ will have a positive coefficient.

2: What does the distribution of total unit size (living size + lot size) look like by zipcode group?

Looking at the boxplot for each bin, we can see that each bin has a lot of outliers in the right tail, which at this point we agree is inherent nature of the dataset.

The bar graphs show the aggregate sum and averages respectively of all the houses in each zipcode bin. One interesting thing to note, is that zipcode A has the largest total house size, but the smallest average size. We can deduce that zipcode bin A may have a relatively higher concentration of houses that are modestly sized. Zipcode B came in 2nd for highest aggregate sum and 1st for highest average sum. Zipcode bin C, our bin of interest in question #1, came in 3rd for highest aggregate sum, and a close 4th (but also not too far off from 2nd) in terms of highest average house sizes. This further paints the picture that zipcode C house prices are high, and house sizes are large as well.

Question 3: Are waterfront properties generally higher priced? What are the concentrations of waterfront properties by zipcode group?

It seems on average, waterfront properties are higher priced, and that zipcode bin ‘E’ has the highest concentration of waterfront properties. Thus, in our model, we should expect to see a positive coefficient for the dummy variable ‘waterfront_1’ (i.e. waterfront == True).

Processing / Inspecting the Base Model

In this step, we take our pre-processed dataset, and we run and and examine single variable OLS regressions for each variable, as well as a multivariate OLS regression for all the variables combined (the “Base Model”). We also take a look at the sum of squared errors for each variable in the single variable OLS regression runs to rank the reliability of predictors. After examining, we take note of if there are any features we should drop manually (features will be systematically selected via stepwise selection in our final model), and we run our base model again to obtain our baseline R-squared.

After examination, we decide to drop the “lat”, “long”, and “date” variables. In summary, the coordinate variables were dropped because location is already somewhat explained by the zipcode variable, and also these variables were not log-transformed or standardized since it does not make sense to. These variables are more for description rather than holding an actual value, so keeping these variables may make the model harder to interpret. “Date” was dropped because it had the highest SSE when modeled against ‘price’, making it not a great predictor.

After running OLS regression on the re-based inputs, we see that our baseline R-squared is 69.1%. This R-squared will be improved in the next phase where we use a systematic feature selection approach and add interaction terms, achieving better model fit.

See below OLS regression summary of our base model.

Modeling

In this phase, we use stepwise feature selection to select features with p-values less than 0.05, and add in interaction terms for better model fit, and then we perform stepwise feature selection again to drop any interaction terms that were added that have a p-value greater than 0.05. We also run some tests to validate the fitted model.

We generate models using iterations of number of interaction terms to add. The iterations are 5, 10, 20 and 50. We see that as we increase the number of interaction terms, the adjusted R-squared increases, which is expected. However, it also makes the model more complex, so we decide to use +10 interaction terms for our final model (minus interaction terms that were added but have p-values greater than 0.5). The final model summary is below:

Let’s also take a look at the predicted vs actual Y scatter plot, as well as the distribution of model residuals.

Visually, we see more or less what we saw in the base model, but with improved skew, kurtosis and, JB value, meaning that the new distribution of model residuals is closer to standard normal than our base model, and our new model overall has higher accuracy.

Note that in our EDA, we concluded that the dummy variables ‘”zipcode C” and “waterfront” would have positive coefficients based on our preliminary analysis. The final model is indeed in agreement with that.

Now, we can put the model to use to derive sales prices given some inputs.

Pricing

Let’s generate some arbitrary inputs and see what the derived model price is, and plot the house on a map.

Example 1:

Example 2:

Example 3: