The importance of meaningful questioning during EDA
By: Fernando Aguilar
Tags: King County, Data Science, Flatiron School, Eda

Introduction

At the start of any project, it is useful to have a clear framework to not deviate from the main objective, find the most efficient solution to a given problem. As more projects are developed, it is clearer to see what data science process fits the most with our own personal preferences and way of work. So far, I have been very comfortable following the OSEMin model which stands for: Obtain, Scrub, Explore, Model and iNterpret.

The first part of every project is to define its goal, understand what is the purpose of taking on the project. Once this is clear, you start by gathering all data related to the project, it could be organically obtained or sourced from external databases. As you start gathering this data, is common that it will not be perfect and some kind of cleaning or preprocessing must be performed.

I want to focus in what happens as we go through exploring our data. Assuming we have obtained relevant data, and have started the cleaning and preprocessing stage, is time to explore it. At this point is very to start asking relevant questions that have real-life applications to uncover hidden value in the data.

Finding hidden value in the data

Let’s take for example the Kings County data set, (https://www.kaggle.com/harlfoxem/housesalesprediction), what questions can be answered with the variables given in this data set? This dataset contains house sale prices for King County, between May 2014 and May 2015. The objective of our project is to create a house price predictive model, but that does not mean we cannot try to solve other relevant questions since we already have access to the data.

Some relevant questions I thought might be useful to the stakeholders of the project are:

* Is it worth renovating your home?

* What zip codes have seen the most price increase over time?

* Does having a higher grade translate to a higher price?

I derived to this questions, since the project is targeted towards homeowners, developers, and real estate agents who want to know the relevant features that determine the price of a house.

Home renovations

I will only take the first question as an example, Is it worth renovating your home? The variable ‘id’ is per home and each record is a transaction. Hence, an id can be repeated if a home has had more than one transaction, in other words, sold more than once.

I found out there were 353 duplicated id’s, however not all of this homes had information regarding their year of renovation. Actually only 4 homes had information about the year they were renovated and sold more than once. Also, no home had a transaction before and after the renovation, hence a direct comparison between before and after renovation prices was not possible.

This situation did not prevent me for carrying out further analysis trying to arrive to a decent answer. Therefore, realizing that I did not have enough data to derive a significant analysis, I went ahead and compared overall prices of houses without renovations and houses with renovations.

From the box plots above it is noticeable that there is a clear difference between median prices of homes with and without renovations. Houses with renovations sell for a median price 35% higher than without renovations. This is very important information for a home seller. Apart from that, looking at the very low values of home with renovations, made me delve deeper into the variable plotting its histogram.

I found out that most homes do not have renovation information. From the variable name and the initial data scrubbing stages, it was clear that it was a discrete variable. However, I might have just created dummy variables for each of the values and input them into the model. By looking deeper, a better solution was to transform it into a binary categorical value. Has the home been renovated or not? This way, I will at least have more data in the dummy variable rather than a lot of dummy variables filled with mostly 0 values.

Conclusion

By posing meaningful questions while exploring the data, hidden value can be uncovered by providing relevant answers to project stockholders apart from the final model. This, also, helps to understand the nature of the data, as well as bring up vices contained in it and finding the appropriate measures to fix it.