First Project-Flatiron’s Data Science Program
By: David Burton
Tags: Housing Prices, Data Science, Flatiron School, Bootcamp

What were we were taught prior to project 1

Every Monday morning we are given 2 Sections of work, each section contained about 10 lessons on average. Each lesson took an average of 25 minutes to complete. Every 2 sections contained multiple labs and a project to demonstrate understanding. Throughout the week there are online study groups for the material being covered hosted by a knowledgable flatiron instructor. I’ve included a breakdown of material covered so far quoting the summary given at the introduction of each section.

SECTION 1 WEEK 1- SETTING THE STAGE

Summary Remember, it’s OK to feel a little uncomfortable. For some students, section 01 will be the most difficult as it introduces so many new concepts at once, but you’ll continue to practice these day after day, until they become second nature. For those of you with a strong background in computing, please bear with us as we help everyone else to catch up. We’ve tried to introduce additional challenges for students who are finding the labs easy, so look out for the “Level Up (optional)” section in some of the labs to get some deeper practice with Python and visualization.

SECTION 2 WEEK 2-CODING IN PYTHON, BASIC STATISTICS, CALCULATOR PROJECT

Summary This is another section where students will have very different experiences. If you’re already a professional software developer with a degree in a quantative subject, it’s not going to be the hardest section for you to complete. But please take the time to practice your skills and feel free to dig deeper with the optional “Level Up” content if you’re done more quickly than your peers. If you’re completely new to coding and/or stats, you will probably find this a fairly challenging section. We’re continuing to introduce core programming concepts at quite a pace. Don’t worry if you still feel overwhelmed as you work through this section or work on the projects. You’re going to get to practice most of these skills on a daily or weekly basis for the rest of the course, so just keep on keeping on — you can do this!

SECTION 3 WEEK 2 — WORKING WITH PANDAS, SUMMARY STATS, CUSTOMIZING VISUALIZATIONS, WELCOME TO KAGGLE

Summary In this section, we’re leveraging the programming and statistical concepts from earlier sections to provide you with additional practice working with various data sets.

SECTION 4 WEEK 2-WORKING WITH PANDAS, LAMBDA FUNCTIONS, COMBINING DATAFRAMES, PIVOT TABLES, DEALING WITH MISSING DATA, PROJECT-DATA CLEANING

Summary In this section, we’re digging deeper into the tools required to import, combine and clean up data.

SECTION 5 WEEK 3-ALL ABOUT SQL

Summary SQL is an incredible powerful language for working with highly structured data and is likely to be one of the key skills you’ll use on a reegular basis as a practicing data scientist.

SECTION 6 WEEK 3-OBJECT ORIENTED PROGRAMMING

Summary In this section, we’ll start to provide you with a solid introduction to Object Oriented Programming which you will find useful whenever you’re writing code.

SECTION 7 WEEK 4 — OOP CONTINUED

Summary In this section, we’re introducing some additional concepts that you’ll need to become comfortable with Object Oriented Programming, and we’re bringing them together in a project that shows you one way to use OOP to solve a data science problem.

SECTION 8 WEEK 4 — NUMPY AND FOUNDATIONAL MATHEMATICS

Summary NumPy, Probability and Combinatorics fit well together. None of them are as attention grabbing as Convolutional Nerual Networks (don’t worry — we’ll get to those later in the course), but each one of them helps to provide the foundation on which most Python machine learning algorithms are built!

SECTION 9 WEEK 5— STATISTICAL DISTRIBUTIONS

Summary In this section, we’re going to take a deeper dive into a range of foundational statistical concepts that we’ll need as we start to dig into machine learning later in the course.

SECTION 10 WEEK 5 — INTRO TO LINEAR EQUATIONS

Summary Congratulations! You’ve made it through much of the introductory data and we’ve finally got enough context to take our first look at our first machine learning model, while broading our experience of both coding and math so we’ll be able to introduce more sophisticated machine learning models as the course progresses.

SECTION 11 WEEK 6 — MULTIPLE REGRESSION

Summary In this section, we continue to add rigor and richness to your understanding of linear regression so that you can solve a wider range of problems more accurately. Many of the principles covered in this section will also apply in later modules as we move onto working with other machine learning models.

SECTION 12 WEEK 6 — COMPLETE DATA SCIENCE PROJECT

Summary In this section, we’re going to give you an end-to-end review of the data science process so that you’re ready for the “end of module 1” project that you’re going to work on after this section.

My Project

The complete github repo can be found at https://github.com/Burton-David/Final_project .

Load libraries I will use throughout the project.

Future projects will only use a few libraries. However, I am new to this and wanted to experiment and try as many different things as I could in order to grow.

# Load the libraries.

import pandas as pd

import numpy as np

import matplotlib.pyplot as plt

import statsmodels.formula.api as smf

import matplotlib.style as style

from matplotlib.lines import Line2D

%matplotlib inline import random from IPython.display import Image from matplotlib.pyplot import show,figure,subplot

from matplotlib import dates as dates from scipy import stats

from scipy.stats import kurtosis, skew from sklearn import metrics

from sklearn.feature_selection import RFE

from sklearn.metrics import mean_squared_error

from sklearn.linear_model import LinearRegression

linreg = LinearRegression()

from sklearn.model_selection import train_test_split #other datavisualization libraries

import seaborn as sns

from seaborn import lmplot #stop the deprecation warning from popping up

import warnings

warnings.filterwarnings("ignore", category=DeprecationWarning)

import the king's county housing data set

# import Kings County Housing Data as df

df = pd.read_csv(‘kc_house_data.csv’, ‘r’, delimiter=’,’) # double checking that everything imported correctly

df.head()

glance around the data set

I am looking for placeholders, as well as getting a general sense of what information is recorded and how. I particularly like using df.sample() since I figured the instructors were likely to add anomalies that were not within the first or last 10 rows. Running df.sample(20) a few times allowed me to randomly glance around the data as if I were skimming a book.

#displays first 5 rows

df.head() #displays the last 5 rows

df.tail() #displays random group of 20 rows within dataset

df.sample(20) #displays data types and non-null values of each column

df.info() # displays overview statistics of each column

np.round(df.describe()) #displays table dimensions as (rows, columns)

df.shape #(21597,21)

From here I took note of which columns contained null values, placeholders, too many zeros, incorrect data types.

Before actually cleaning the data I needed more information about each column. The project itself contained an additional file within the github repository, I just needed to load it. And Kings County Washington had a useful website https://info.kingcounty.gov/assessor/esales/Glossary.aspx?type=r#g I referenced often while looking at each feature.

Dealing with Null Values

# For each column, are there any NaN values?

df.isnull().any()

Features waterfront, view, and yr-renovated all have null-values. But, how many?

# for each column, how many rows are NaN?

df.isnull().sum()

waterfront = 2376

view = 63

yr_renovated = 3,842

I haven’t actually dealt with these yet, but there is something else that is bothering me. I noticed that some of the id’s were appearing more than once. Maybe I can find some of the missing values within the duplicates.

# save duplicated rows into separate dataframe for further analysis

# get rid of duplicated rows

duplicated_id = pd.concat(g for _, g in df.groupby("id") if len(g) > 1) # The last listed price for each ID is the most recently used information

# therefore the most accurate representation of housing market

df['id'].drop_duplicates(keep='last',inplace=True) # The ID has served its purpose and will deprecate regression models so drop this column first

df = df.drop(["id"], axis=1)

The duplicates did not actually contain any missing information that I needed. However, I pulled them into a separate dataframe to study later. These houses look as though they were bought twice within a 1 year period!

replace view’s NaN values with 0

# view is the correct datatype already

# However, there are missing values to be dealt with df['view'].isnull().sum()

#63 df['view'].value_counts()

#0.0 19422

#2.0 957

#3.0 508

#1.0 330

#4.0 317

#Name: view, dtype: int64 #replace NaN with 0

df['view'].fillna(0,inplace=True)

yr_renovated so many values of 0 on top of NaN that I am changing this feature into renovated and making it boolean.

# number of null values

df['yr_renovated'].isnull().sum()

#3842 #how many different values does it have? df['yr_renovated'].value_counts().head()

#0.0 17011

#2014.0 73

#2003.0 31

#2013.0 31

#2007.0 30 #replace NaN values with 0

df['yr_renovated'].fillna(0,inplace=True) #double check that the NaN values are replaced

df['yr_renovated'].isnull().sum() #change the data type to boolean

df['yr_renovated'] = df['yr_renovated'].astype(bool)

#change the column name to more accurately represent data

df['renovated'] = df['yr_renovated'] # waterfront also needs to be turned into a boolean value 1 has waterview, # 0 does not

# change datatype to Boolean

df['waterfront'] = df['waterfront'].astype('bool')

The Date Feature should be a datetime object

#Date converted into datetime

df['date'] = pd.to_datetime(df['date']) #double check everything looks right

df['date'].head() # query the data type for date column

type(df['date'][0]) #pandas._libs.tslibs.timestamps.Timestamp

One requirement of the project is that I use data visualization to answer at least 3 questions. I was curious which day of the week most sales took place. I assume that few or none take place during the weekend when banks are closed.

#create a column that uses the timestamp to determine day of the week

df['weekday'] = df['date'].dt.dayofweek #Monday=0, Sunday=6

#graph the amount of sales that took place in each day of the week df.weekday.hist()

Sales per Day of Week

At this point I pulled up a calendar to double check the days matched the dates. I could have written a forloop to change the int values into strings “Sunday”, “Monday”…

# rewrite the column using strings I can easily read

dayOfWeek={0:'Monday', 1:'Tuesday', 2:'Wednesday', 3:'Thursday', 4:'Friday', 5:'Saturday', 6:'Sunday'}

df['day'] = df['date'].dt.dayofweek.map(dayOfWeek)

Graph the information using a seaborn barplot.

day_count = df['day'].value_counts() plt.figure(figsize=(15,8))

sns.barplot(day_count.index, day_count.values, alpha=0.8)

plt.title('When Are Most Homes Bought?')

plt.ylabel('Number of Sales', fontsize=12)

plt.xlabel('Day of Week', fontsize=12)

plt.show()

Question 1: Which day of the week do most sales take place?

Answer: Tuesday

Earlier I saw that sqft_basement contained ‘?’ as a placeholder in addition to many values of zero. I used the difference of sqft_above and sqft_living to determine the size of some of those missing basement values.

df['sqft_basement'].value_counts().head();

#0.0 12826

#? 454

#600.0 217

#500.0 209

#700.0 208

#Name: sqft_basement, dtype: int64 #locate rows containing ?

df.loc[df['sqft_basement'] == '?',['sqft_living','sqft_above']]; #locate rows with a different value for sqft_above and sqft_living

df.loc[(df["sqft_basement"] == '?') & (df["sqft_above"]!=df['sqft_living'])]; #Replace ? with difference between sqft_above and sqft_living

df['sqft_basement'].replace(to_replace='?', value = abs(df['sqft_above']-df['sqft_living']), inplace=True) #double check that everything was replaced correctly

df['sqft_basement'].describe();

Since there are 12826 houses without a basement the sqft_basement category will work better as a boolean. Either a home has a basement or it does not. Running a regression model on this feature as is would not work well. Then drop obsolete columns. sqft_living should not include basement squared footage but does. Therefore the data is corrupt and should be purged.

#convert the column into a numeric in preperation for becoming boolean

df['sqft_basement'] = pd.to_numeric(df['sqft_basement']) #convert column into boolean

df['sqft_basement'] = df['sqft_basement'].astype(bool) #create column name to more accurately represent data

df['basement'] = df['sqft_basement']



#drop obsolete sqft_basement, sqft_living, and yr_renovated

df = df.drop(["sqft_basement", "sqft_living","yr_renovated"], axis=1)

Question 2: How are the houses spread out geographically in regards to ordinal variables such as grade, condition, number of floors, and view? I wanted to use the latitude and longitude data for something, so at this point I had to learn how to graph it. It may be worth noting here that on top of the Flatiron curriculum I set aside a bit of time every day to read articles on data visualization, and try to go through at least 1 datacamp tutorial a day.

#group all of the ordinal features together

ord_features = ["view", "condition",

"grade","bedrooms",

"bathrooms", "floors"] #run the features through lmplot as a forloop to build graph

for x in ord_features: lmplot(data=df, x="long", y="lat", fit_reg=False, hue=x, height=10)

plt.show()

I’ve just included view to save space in this post although 6 graphs were created. By graphing view I was able to work out that view represents the unobstructed line of sight from the house. View’s of 2 or lower have a highly obstructed view. This led to another visualization worth exploring further.

Question 3: Where are the waterfront properties located geographically? Is this data reliable and accurate?

Answer: No

Many of these properties are located far inland. This data is bogus and therefore I removed the feature.

# drop waterfront feature from dataframe

df = df.drop(["waterfront"], axis=1)

Normalize and scale continuous features

# Create a pandas Dataframe to store normalized values

data_log= pd.DataFrame([]) #log transform each of the values.

data_log["sqft_lot"] = np.log(df["sqft_lot"])

data_log["sqft_above"] = np.log(df["sqft_above"])

data_log["sqft_living15"] = np.log(df["sqft_living15"])

data_log["sqft_lot15"] = np.log(df["sqft_lot15"])

data_log["lat"] = np.log(df["lat"]) # log transform the absolute value since longitudes are negative

data_log["long"] = np.log(abs(df["long"])) #graph the results to verify normalization

data_log.hist(figsize = [6, 6]);

#store newly normalized values into variables for easier access

logabove = data_log["sqft_above"]

loglot = data_log["sqft_lot"]

loglot15 = data_log["sqft_lot15"]

loglive15 = data_log["sqft_living15"]

loglat = data_log["lat"]

loglong = data_log["long"] #use min/max scaling to standardize values between 0 and 1

df["sqft_above"] = (logabove-min(logabove))/(max(logabove)-min(logabove))

df["sqft_living15"] = (loglive15-min(loglive15))/(max(loglive15)-min(loglive15))

df["sqft_lot"] = (loglot-min(loglot))/(max(loglot)-min(loglot))

df["sqft_lot15"] = (loglot15-min(loglot15))/(max(loglot15)-min(loglot15))

df["long"] = (loglong-min(loglong))/(max(loglong)-min(loglong))

df["lat"] = (loglat-min(loglat))/(max(loglat)-min(loglat))

Eliminate outliers in ordinal data

I chose to only eliminate these outliers first hoping that they would subsequently eliminate the outliers in continuous features. I did it this way to minimize the restrictions of my model.

# df will only include 2-5 bedrooms houses

df= df[(df['bedrooms'] <= 5) & (df['bathrooms'] >=2)] #df will only include 2-3.5 bathroom houses

df= df[(df['bathrooms'] <= 3.5) & (df['bathrooms'] >=2)]

Bin categorical data- yr_built and zipcode

I binned the years built by eras of construction. This took a bit of research. However, this is the thought process behind what I did.

Sturge’s Rule: k = 1 + 3.322(log10 n),

k is the number of classes,

n is the size of the data.

k = 1 + 3.322(log10 12409) = 14.5993935983

Sturge’s Rule recommends 14 to 15 bins

However, I want my bins to contain closer to equal value_counts not equally spaced bins

So I am doing research on home construction in USA by date

Homes built between 1890 and 1940 — before World War II — were dubbed pre-war. At the time, houses started changing, shifting from functional farmhouses and Victorian mansions, to a happy medium of two-storied houses and a basement.

Housing from the Post-War Boom

America grew rapidly from the 1940s through the 1970s, starting with the end of the Great Depression until the recession in the early 1980s. (the 1980’s recession began July 1981 ending November 1982)

Most houses listed are after 1982. So, it makes sense to break these into decades.

# first, create bins for based on the values observed. 5 values will result in 4 bins

bins = [1800, 1940, 1982 , 1990, 2000, 2010, 2020] # use pd.cut to create variable to store bins

bins_year = pd.cut(df['yr_built'], bins) # using pd.cut returns unordered categories. Transform this to ordered categories.

bins_year = bins_year.cat.as_unordered()

bins_year.head() # inspect the result

bins_year.value_counts().plot(kind='bar'); # replace the existing "yr_built" column

df["yr_built"]=bins_year

View has far too many 0’s compared to all other values. Even creating a boolean will leave the feature heavily lopsided so I dropped it.

#checking to see if setting view as boolean would help with the heavy skew towards 0

df.view.astype(bool).value_counts().plot(kind='bar'); # drop view column

df = df.drop(["view"], axis=1)

Bin zipcode

#find the range of zipcodes

98199-98001 # I'll put 40 zipcodes in each bin and see what happens with 5 bins 198/10 # first, create bins for based on the values observed.

zipbins = [98001, 98021, 98031, 98041,

98061 ,98101, 98121,

98141, 98200]

# use pd.cut

bins_zip = pd.cut(df['zipcode'], zipbins) # using pd.cut returns unordered categories. Transform this to ordered categories.

bins_zip = bins_zip.cat.as_unordered()

bins_zip.head(2) # inspect the result

bins_zip.value_counts().plot(kind='bar'); zipbins = [98000, 98011, 98028, 98035, 98045,

98057 ,98073, 98103,

98117, 98143, 98200]

# use pd.cut

bins_zip = pd.cut(df['zipcode'], zipbins)

bins_zip = bins_zip.cat.as_unordered() # replace the existing "zipcode" column

df["zipcode"]=bins_zip #change the datatype of categorical features

df["yr_built"] = df["yr_built"].cat.codes

df["zipcode"] = df["zipcode"].cat.codes #create dummy columns to store cat.codes

zipcode_dummy = pd.get_dummies(bins_zip, prefix="ZIP")

year_dummy = pd.get_dummies(bins_year, prefix="YR") #drop now obsolete columns

df = df.drop(["yr_built","zipcode"], axis=1)df = pd.concat([df, zipcode_dummy, year_dummy], axis=1) #double check that everything is as it should be

df.head()

renovated is too lopsided as well so I need to drop that

#making the renovated column obsolete

df = df.drop(["renovated"], axis=1)

Scale all remaining features

#assign each column to variable for easier access

bed = df["bedrooms"]

bath = df["bathrooms"]

floors = df["floors"]

condition = df["condition"]

grade = df["grade"]

weekday= df['weekday'] # minmax scaling

df["bedrooms"] = (bed-min(bed))/(max(bed)-min(bed))

df["bathrooms"] = (bath-min(bath))/(max(bath)-min(bath))

df["floors"] = (floors-min(floors))/(max(floors)-min(floors))

df["condition"] = (condition-min(condition))/(max(condition)-min(condition))

df["grade"] = (grade-min(grade))/(max(grade)-min(grade))

df["weekday"] = (weekday-min(weekday))/(max(weekday)-min(weekday))

Check for multicollinearity

drop features that correlate greater than 0.70 one at a time, starting from the greatest. Check multicollinearity again, then drop the next highest until none are greater than 0.70 to one another.

correlation = df.corr()

plt.figure(figsize=(14, 12))

heatmap = sns.heatmap(correlation, annot=True,vmin=-1, linewidths=0, cmap="RdBu_r")

I dropped sqft_lot15 then sqft_living15. Then double checked my results using.

#correlations greater than .70 will return True

abs(df.corr())>0.70

Model fitting



from sklearn.feature_selection import RFE

from sklearn.linear_model import LinearRegression linreg = LinearRegression()

selector = RFE(linreg, n_features_to_select = 5)

selector = selector.fit(X, y.values.ravel())

selector.support_

selected_columns = X.columns[selector.support_ ]

linreg.fit(X[selected_columns],y)

LinearRegression(copy_X=True, fit_intercept=True, n_jobs=1, normalize=False)

yhat = linreg.predict(X[selected_columns])

SS_Residual = np.sum((y-yhat)**2)

SS_Total = np.sum((y-np.mean(y))**2)

r_squared = 1 - (float(SS_Residual))/SS_Total

adjusted_r_squared = 1 - (1-r_squared)*(len(y)-1)/(len(y)-X[selected_columns].shape[1]-1)

r1 = r_squared

a1 = adjusted_r_squared

print(r_squared)

print(adjusted_r_squared)

4 features showed an ideal r-squared value.

Choosing the 4 features to use

adjusted_r_squaredselector = RFE(linreg, n_features_to_select = 4)

selector = selector.fit(X, y.values.ravel())

selector.support_

selected_columns = X.columns[selector.support_ ]

selected_columns

Grade, Pre-War era, sqft_above, and 1990–2000 were the top 4 features.

Perform train-test split

train_mse = mean_squared_error(y_train, y_hat_train)

test_mse = mean_squared_error(y_test, y_hat_test)

print('Train Mean Squarred Error:', np.sqrt(train_mse))

print('Test Mean Squarred Error:', np.sqrt(test_mse))

print('difference: ' + str(np.round(abs(np.sqrt(train_mse)-np.sqrt(test_mse)),2)) )

So, my model should be able to accurately predict the price of a home built between 1990–2000, or 1800–1940(pre-war era) with 2–5 bedrooms and 2–3.5 baths within $1053.17

Interpert the coefficients

I did the work, what does it mean?

Remember our feature ranking?

Features sorted by their rank:

Grade

YR_(1800, 1940]

sqft_above

YR_(1990, 2000]

grade = 14146.42

For every 1percent grade increase, you can expect the price of a house to go up 14,146.42 dollars compared to an otherwise identical house. But grades do not go up by percents. They actually go up in incremental steps from 1–13. Each grade is actually a 100/13 percent change. Meaning that a single grade increase will raise the price of the house by $108,818.61!

YR_(1800, 1940] = 2685.93

Well, grade makes sense but you can’t apply the same formula to a qualitative variable. These either are or aren’t. If the house is a pre-war house you can expect to pay $ 2,685 more for it.

sqft_above = 3365.90

This is the most straight forward. A house that is 1percent larger will cost $ 3,365.90 more.

YR_(1990, 2000] = -750.05

Houses built between 1990 and 2000. When construction was terrible and houses were being thrown up and given to anybody who could put $ 0 down. Causing the bubble to burst and the world's economy to suffer for years and years. Those houses are 750.05 dollars less than an otherwise identical house.

In conclusion

There is a much easier way to do this project. However, I was able to work my way through in about 2 weeks and come up with a pretty accurate model. Considering that I did not know how to code a single line 2 months ago this project shows what can be accomplished in a relatively short amount of time. I am looking forward to where I will be at in 8 months once the bootcamp is finished and I begin applying for jobs.