EDA: Feature Engineering and Encoding Categorical Data | by Wesley Neill | Towards Data Science

Last week, we began our series on regression techniques with some exploration of the Ames, Iowa real-estate dataset, using scatterplots and bar plots. These techniques are meant to build basic intuition about patterns in our data. Plotting our data also allows us to sift through and get to know the variables we are working with. In this dataset, there are 79 different variables used to predict home sale prices. That’s a lot!
Then, we wrapped up with some ways to evaluate and fill in missing data. If you’d like to read the full article, you can find it here.
Now, load up your Jupyter Notebook from last week, and let’s get started! You can find my full notebook here, but like last time, I encourage you to develop your own as we move through this tutorial.
Before we get started, make sure you’ve completed our first exercise:
****************************************************************Exercise 1: If you haven’t already, fill in any missing data in the dataset using the techniques outlined last week. Then you’ll be ready to dive into feature engineering. *****************************************************************
The idea behind feature engineering is to first discover and then bring to the surface patterns in the data that may drive our predictor variable. This two-part process begins with an exploration of our intuition.
What gives a home value?
Here’s my shortlist:
Size. I’m talking total usable interior space.How big is the property the home sits on? I like to garden!Does it have an external structure such as a shed?How about a pool?
Size. I’m talking total usable interior space.
How big is the property the home sits on? I like to garden!
Does it have an external structure such as a shed?
How about a pool?
Notice that some of these questions have “Yes” or “No” answers, and the simple act of answering with “Yes” drives sale price up in your mind. It’s not necessarily any different when it comes to machine learning.
Now imagine a machine learning algorithm that can only answer questions based on the data it has. If we look at our data dictionary, we see only two variables that are associated with a home’s pool:
PoolArea — Area in square feet.PoolQC — Pool quality.
PoolArea — Area in square feet.
PoolQC — Pool quality.
A human can make the jump from a given home having an “Excellent” pool quality to “This home must have a pool”. There is no guarantee that your algorithm or model can, however. Therefore, we must bring this pattern to the surface through feature engineering.
Time for some code! I’m assuming you have your notebook ready to go, and the dataset loaded.
df['HasPool'] = df.PoolArea.apply(lambda x: 1 if x > 0 else 0)
This single line of code adds the feature HasPool to your Pandas data frame.
What if I want to know if a home has a big shed or not? What if I want to know if my potential new house has a tennis court? Okay, I’d never want to know that, but someone would. Let’s again look at our best friend, the data dictionary. I see a variable called MiscFeature which has the following possible values:
       Elev	Elevator       Gar2	2nd Garage (if not described in garage section)       Othr	Other       Shed	Shed (over 100 SF)       TenC	Tennis Court       NA	None
Well, that was easy. We can now extract several features that may or may not affect the value of a home:
df['HasShed'] =     f.MiscFeature.apply(lambda x: 1 if x == 'Shed' else 0)df['HasTennis'] =     df.MiscFeature.apply(lambda x: 1 if x == 'TenC' else 0)df['HasGar2'] =     df.MiscFeature.apply(lambda x: 1 if x == 'Gar2' else 0)
Keep in mind, that any variables you engineer in the training set also need to be created in the test set.
****************************************************************Exercise 2: Answer the question “How much above ground living area does the house have?” Is there a variable that describes this quantity? If not, figure out how to engineer a variable that does, using the data you do have and the techniques above.*****************************************************************
Another way of tackling feature engineering is to “Try to say the same thing differently”. You can take existing information and break it into smaller parts. Imagine explaining a concept with more detail to someone who doesn’t understand. Alternatively, you can combine existing variables, which would be analogous to explaining a concept in broader strokes.
Before we move away from feature engineering, let me just say this: Don’t trust your intuition. Intuition is typically built by experience and experimentation. When intuition fails due to a lack of experience, research is a great substitute. For example, I didn’t know that a finished basement adds less value per square foot than an above-ground room!
Before we can garner any statistical insights into our data, we need it all to be encoded in numerical format. This will allow us to perform calculations on our data. To get an intuition about how to do this, we need to consider that there are two types of categorical variables: Ordinal and Nominal.
Ordinal: The possible values for an ordinal variable are non-continuous but do have an underlying rank structure. An example of an ordinal variable would be one that classifies college students into one of four categories: Freshman, Sophomore, Junior, or Senior. These discrete categories have an underlying order.
Nominal: A nominal variable is one that has no obvious underlying rank or order. Color is a good example. ‘Red’, ‘Blue’, ‘Green’ might as well be listed ‘Blue’, ‘Green’, ‘Red’, as they have no intrinsic properties that give them rank.
Defining these two categories is important when we want to convert categorical data to numbers. If our categories are ordinal, we must provide a method that preserves the underlying rank structure.
***************************************************************Exercise 3: Sort your categorical data into 2 lists, like so:
ordinal = ['ExterQual', 'BsmtQual', 'GarageQual',            'KitchenQual', 'FireplaceQu', 'PoolQC',             'OverallQual', 'ExterCond', ... ]nominal = ['Neighborhood', 'MSZoning', 'Condition1',            'Condition2', 'RoofStyle', ... ]
***************************************************************
Once we have our ordinal/ranked data separated, we must figure out exactly how it is ranked. Again, we must refer to our data dictionary to do this. If you haven’t noticed, the data dictionary is an essential tool that you should probably have open at all times. For example, let’s take the variable BsmtExposure:
BsmtExposure: Refers to walkout or garden level walls (split levels or foyers typically score average or above)       Gd	Good Exposure       Av	Average Exposure 	       Mn	Mimimum Exposure       No	No Exposure       NA	No Basement
This variable describes how well exposed to the outside a basement is. Many split level homes have a furnished basement with external access and patio or porch. This is considered “good exposure”, and we must account for it when we encode our variables.
There are likely many ways to encode ordinal data to numbers, but I like to use a dictionary or map.
BsmtExposure_map = {    'Gd':4,    'Av':3,    'Mn':2,    'No':1,    'NA':0}df.BsmtExposure = df.BsmtExposure.map(BsmtExposure_map)
Note the use of Pandas.Series.map to transform our variable with rank intact.
Likewise, we can do the same thing with the variable Functional, which describes any major damage or other issues with the house. Values range from Sal to Typ, where Sal means the house has experienced catastrophic damage and is uninhabitable. Typ signifies normal wear and tear.
Functional_map = {    'Sal': 0,    'Sev': 1,    'Maj2': 3,    'Maj1': 4,    'Mod': 5,    'Min2': 6,    'Min1': 7,    'Typ': 8}df.Functional = df.Functional.map(Functional_map)
**************************************************************Exercise 3: Convert all of your ordinal variables in the same manner as the two examples above. Keep in mind, this needs to be done for both your training and test sets. ***************************************************************
Okay, everybody likes heat maps of correlation matrices. I’m as guilty as the next guy or gal, and I am guilty of spending way too much time on choosing the perfect color map to match the rest of my notebook. So here’s a sample of what we are going to do in the next article (after we talk about why we can’t trust a pretty face... or heatmap):
Which will bring us to statistical testing... If the first date is a heatmap, then performing statistical testing is digging deep to find out if your new crush is as pretty on the inside as she is on the outside.
Until then, feel free to leave a comment with any questions, comments, or corrections!