EDA: Variable Relationships and Filling Missing Values | by Wesley Neill | Towards Data Science

As part of my journey as a data scientist, end-to-end data exploration and model development has been a key to my learning success. This series will provide an end to end look at predicting home sale prices in Ames Iowa. The series will be broken down into four parts for readability:
EDA Part 1: Plotting Warmup and Filling in Missing Values
EDA Part 2: Feature Engineering and Variable Encoding
EDA Part 3: Statistical Tests
Predictive Modeling: Training and Tuning XGBoost
Stacking Models: Combining XGBoost with Enet, Ridge, and Lasso
Open up a Jupyter Notebook, download the data, and follow along by implementing the following strategies yourself. I’ve included a few ‘exercises’ to get you going. I’m hiding the full code to force you to focus on the thought process. Additionally, by typing the code out yourself, you will gain experience that will eventually lead to fluency. However, if you need help along the way, you can find my full notebook here.
If you already read my article “A Beginner’s Look at Kaggle”, you know what’s coming: Let’s get our hands dirty with EDA. It’s going to be the sole focus of this part of the series.
This first section will focus primarily on basic plotting and missing value imputation. My goal is to outline a process for building an intuition as to what variables are at play, and which have the greatest impact on sale price. I also want to evaluate relationships between variables because it will give insight on how we might intelligently fill missing data.
The focus of our data exploration throughout the 4 part series is as follows:
Visually Representing Data — Graphing and plotting our data can provide human insight into the real-world implications of our data. We have 79 variables at play, so it’s a great way to get a feel for what’s going on. All visualizations should be taken with a grain of salt, however. They don’t tell the whole story.
Finding and Dealing with Missing Data — In this phase, we will analyze what variables have missing values, why they are missing, and how we might deal with the missing data.
Feature Engineering — Can we find and define underlying patterns in the data that help us better describe and solve our problem?
Strength of association. Provided that a variable is related to Sale Price at all, how strong is that relationship?
ANOVA F-Test of categorical variables against sale price. Unlike continuous variables, we can not necessarily determine the traditional correlation metric between categories and continuous data. Through analysis of variance or ANOVA, we can gain some insight into whether there is a significant impact of a categorical variable such as neighborhood on the continuous variable sale price.
*****************************************************************Exercise 1: Sign up for a Kaggle account and download the data we will be using! I keep my data in a folder called data located in the same directory as my notebook.*****************************************************************
import pandas as pdtrain_df = pd.read_csv('data/train.csv', index_col='Id')test_df = pd.read_csv('data/test.csv', index_col='Id')train_df.describe().T.head(10)
To begin getting a feel for our data, we will plot our target variable distribution along with a couple of different transformations. We’ll do this for a couple of reasons:
We want to understand our data, and looking at it visually helps us build intuition of the underlying factors that affect an outcome. It gets the wheels turning in our mind, and hopefully spurs questions that might lead to valuable insights.Transforming the distributions of our variables is common industry practice. Transformation is not always necessary or helpful. The mathematical theory behind the effectiveness of data transformation goes deep and is often debated (especially in the realm of deep learning). It suffices to say “it depends” and then experiment!
We want to understand our data, and looking at it visually helps us build intuition of the underlying factors that affect an outcome. It gets the wheels turning in our mind, and hopefully spurs questions that might lead to valuable insights.
Transforming the distributions of our variables is common industry practice. Transformation is not always necessary or helpful. The mathematical theory behind the effectiveness of data transformation goes deep and is often debated (especially in the realm of deep learning). It suffices to say “it depends” and then experiment!
Before transformation, our sale price data is right-skewed, meaning that the mean is biased towards a higher price than the median. Applying a Yeo Johnson transformation via sklearn.preprocessing.PowerTransformer() we get a pretty good approximation of the standard normal distribution:
import sklearn.preprocessing as preimport seaborn as snsimport matplotlib.pyplot as pltpt = pre.PowerTransformer()ptd = pt.fit_transform(train_df.SalePrice.to_numpy().reshape(-1,1))plt.title(f'Yeo Johnson, Skew: {stats.skew(ptd.squeeze()):.3f}')sns.distplot(ptd, kde=True)
I also checked out using a log transformation, but we don’t get quite as close to a normal distribution:
These transformations alter the meaning of the data (notice the lack of the ‘SalePrice’ label on the horizontal axis). The main purpose of transforming the target variable is in hopes of assisting the training of a machine learning algorithm. Again, this requires experimentation and won’t be covered until Part 3 of this series.
As our second warm-up exercise, let’s check our intuition that it’s not just quantity but the quality that counts toward the final sale price:
fig, axes = plt.subplots(2,2, figsize=(12,12))sns.scatterplot(    x='GrLivArea',     y='SalePrice',     hue='OverallQual',     data=train_df, ax=axes[0,0])...plt.tight_layout()# Try reproducing the plot below using the above code as a hint.
In the first column, we compare the sale price to the above-ground living (aka GrLivArea) area and the overall quality of home build materials. What we see is that the sale price is graduated well by both living-area and by quality. This is evident in the strong linear trend in the scatterplot, the even graduation of hue, as well as the boxplot of overall quality.
Somewhat surprisingly, we see that the variable describing the overall condition, OverallCond, seems to have little effect on sale price. The hue of the data points in the second scatterplot is random, and the boxplot is unordered. This is counter-intuitive, and we can dig into whether or not what we are seeing makes sense a little later.
Also, notice the two outliers in the data. There are two homes seen in the scatterplots that have quite large living areas, yet somehow sold for very low prices relative to others of the same size.
*****************************************************************Exercise 2: Investigate your intuitions about other variables and their relation to the sale price and one another by plotting them. Check out Seaborn’s great utility for plotting relationships.*****************************************************************
Our next round of analysis will focus on our missing data. We want to know why the data is missing, and how we might be able to fill in the blanks. Lost data is lost information that might otherwise be useful to us. It is often worth attempting to fill in those blanks for better predictions.
First, we will address the gaps in our missing training data. I wrote a little function to visualize missing data in a data set. Look in my full notebook for a method called viz_missing(df):
The barplot shows a lot of missing data. This data shouldn’t be ignored if we wish to get really good results. This part of the process is tedious but necessary. I encourage you to focus on the following thought process:
Choose a variable with missing data you wish to fill.See if there is another variable, related to the one you are investigating.Determine if the missing data can be accounted for by the related variable.If you cannot determine the missing information by using steps 2 & 3, consider using mean, median, or mode. Be aware that this can be dangerous for small datasets.
Choose a variable with missing data you wish to fill.
See if there is another variable, related to the one you are investigating.
Determine if the missing data can be accounted for by the related variable.
If you cannot determine the missing information by using steps 2 & 3, consider using mean, median, or mode. Be aware that this can be dangerous for small datasets.
The variable with the greatest amount of missing data in the chart above is PoolQC which stands for ‘Pool Quality and Condition’.
After looking at our data dictionary, we should notice that there is only one other variable that describes a home’s pool. That variable is PoolArea. Take note that there is no missing data associated with that variable.
My intuition is that all of the homes that have PoolArea = 0 may be associated with homes that are missing PoolQC information. In other words, those homes don’t have pools, and that is why they are missing the pool quality variable. We can check that intuition with just a couple lines of code:
# Get number of homes with missing PoolQC that also have 0 PoolArea.np = train_df[train_df.PoolQc.isna() & (train_df.PoolArea == 0)]count = np.shape[0]# Get total number of missing PoolQC entriesmissing = train_df.PoolQC.isna().sum()count, missing>>> (1452, 1452)
So we see that for all 1452 homes missing PoolQC entries, the PoolArea of that home is 0. This proves our intuition correct. We can fill in the blanks with one more line of code:
train_df.PoolQC.fillna("NA", inplace=True)
LotFrontage, according to the data dictionary, is the measure of the number of feet of road that is connected to the property. This time we don’t have any additional variables that will tell us why LotFrontage is missing. This means we will have to approach filling the blanks differently.
We may reasonably assume that almost every piece of property touches at least a small part of some road or alley. This may not be universally true, but is probably the rule rather than the exception. We will use this intuition to fill any missing LotFrontage with the mean or median of all homes for which we do have this data:
train_df.LotFrontage.fillna(train_df.LotFrontage.median(), inplace=True)
I chose to fill the missing entries with the median, as it will be more robust against outliers.
*****************************************************************Exercise 3: Perform this same investigation on all the missing data in both the test and training sets. It should take you a while. A lot is missing!*****************************************************************
I’m going to leave off here until part 2 of the series. Completing exercises 1–3 should take you quite some time and are encouraged to code up these exercises before moving on.
If you have any questions, comments, or critiques, please feel to reach out to me in the comment section!