Steps Before Modeling: Cleaning to EDA | by lucas winter | Towards Data Science

Before being able to utilize the machine learning algorithms we all know and love, it is important to understand how to initially get the data in proper order and how to effectively analyze the data in order to produce the best model. This post will provide a basic roadmap that a data scientist can follow to perform exploratory data analysis (EDA) when working with a new dataset in Python.
For this post, we will examine SAT and ACT participation rates by state for 2017 and 2018. The datasets include statewide average scores for each exam section, composite/total scores for both exams and participation rates.
#Imports:import num               as npimport pandas            as pdimport matplotlib.pyplot as pltimport seaborn           as snsfrom scipy               import stats
After loading the file into a Pandas dataframe, it is useful to look at the first few rows and the size of the dataset to get a feel for what you are working with. This is best accomplished by using the .head() and .shape methods.
The .describe() method provides a quick statistical overview of the quantitative features in the dataset from the standard deviation of each feature to the minimum and maximum values. This can also be used to spot potential outliers. In the image below, we can quickly see that the minimum score for Science is 2.3 which does not look right when compared to the other values. This will need to be examined further.
act_2017.sort_values('Science').head()
Utilizing the sort_values() method, we can see that the average science score for Maryland is listed as 2.3. This is most likely a typo and will need to be corrected.
After describing the data, it is important to make sure there are no missing values and that all the data types are appropriate for each feature. This .isnull() and .info() methods are best used in this case.
act_2018.isnull().sum()State            2Participation    2Composite        2
From above, we can see that the 2018 ACT data contains 2 missing values in each column. The participation column below is listed as an object when it should be a float since the values are percents.
These Pandas methods provide us with an initial look at the data and what might need to be cleaned or corrected before we can begin modeling.
After our initial review, it is important to fix the errors we spotted. First, we will overwrite the Science score for Maryland to 23.2 by using .loc to isolate the specific location in the dataframe.
act_2017.loc[act_2017['State'] == "Maryland", 'Science'] = 23.2
Below, we can see the 2 rows which contained null values.
act_2018.drop(act_2018.index[[52,53]], inplace = True)
The above code can be used to drop the two rows of missing values. The .dropna() method is another useful tool when there are missing values that need to be removed.
Lastly, before we begin visualizing the data we need to correct the data type of the participation column. Using the .astype() method, we can update the values in the column to be floats and not objects.
sat_2018['Participation'] = sat_2018['Participation'].astype(float)
Creating visualization can help us understand the data in greater detail and allow us to spot underlying relationships that we would not be able to see by just describing the data. Visualizations can also reveal outliers that could potentially skew our results. We will be using the seaborn and matplotlib libraries to create our visualizations.
A correlation heatmap is a useful image to see the linear relationships between features. When deciding which features to use in a model, it is important to avoid multi-collinearity. In the image below, the lighter values represent features that are highly positively correlated and the darker images are highly negatively correlated. Scores and participation rates tend to be correlated for the two exams year over year.
sns.heatmap(final.corr(),mask=mask, vmax=.3, square=True, annot=True)
Scatter plots are another method to see if there is a linear relationship between two features.
# Establish the size of the figure.plt.figure(figsize=(16, 9))# Create scatter plot of SAT vs. ACT math scores for 2017.sns.regplot(stats.zscore(final['total_sat_2017']),            stats.zscore(final['composite_act_2017']),             ci = None,             line_kws = {'color': 'orange'})plt.title("SAT vs. ACT 2017 Overall Scores", fontsize = 25)plt.ylabel("ACT 2017 Composite ZScores", fontsize = 20)plt.xlabel("SAT 2017 Composite ZScores", fontsize = 20)plt.xticks(size = 12)plt.yticks(size = 12);
From the scatter plot, we can see that the ACT and SAT composite scores are negatively correlated. Therefore, doing well on one exam does not necessarily mean that you will perform well on the other.
Histograms are useful when looking at distributions of features. The code below creates two histograms on the same plot so we can analyze the features side by side.
plt.figure(figsize=(16, 8))plt.subplot(1, 2, 1)final['total_sat_2017'].hist()plt.title('2017 SAT Total Scores', size = 25)plt.xlabel('Scores', size = 20)plt.ylabel('Frequency', size = 20)plt.vlines(x = np.mean(final['total_sat_2017']), ymin = 0, ymax = 10, color = 'r')plt.xticks(size = 12)plt.yticks(size = 12)plt.subplot(1, 2, 2)final['total_sat_2018'].hist()plt.title('2018 SAT Total Scores', size = 25)plt.xlabel('Scores', size = 20)plt.ylabel('Frequency', size = 20)plt.vlines(x = np.mean(final['total_sat_2018']), ymin = 0, ymax = 10, color = 'r')plt.xticks(size = 12)plt.yticks(size = 12)
We can see from the histograms above that SAT total scores in 2017 and 2018 are not normally distributed are are skewed to the right. In both years, the frequency of scores drop between 1100 and 1200. After looking at this image, we may decide to do further research on why there is this gap.
Lastly, boxplots are also useful when looking at distributions of features, but they are also helpful to spot outliers.
#2017 ACT scores by sectionplt.figure(figsize = (16, 9))plt.title("ACT Subsection Scores 2017", size = 25)sns.boxplot(data = final[["math_act_2017", "science_act_2017", "english_act_2017", "reading_act_2017"]])plt.ylabel("Average Score", size = 20)plt.xlabel('Section', size = 20)plt.xticks(size = 16)plt.yticks(size = 16)
In the image above, we can see that there are no outliers remaining in our dataset for ACT subsection scores in 2017. The outlier in the Science section was previously removed while performing data cleaning.
Now that our data is clean and we have analyzed our features visually, we are ready to begin preprocessing and modeling.
Properly cleaning a dataset and performing EDA are critical steps in a data scientists workflow. Every dataset is different, but hopefully you learned some useful methods to follow the next time you are faced with a problem that requires analyzing a dataset.
Code for this post can be found on my Github.
You can also find me on LinkedIn.