Easy, Breezy, EDA — An Update to My Latest Function | by Katie Sylvia | Towards Data Science

In my latest blog post, I had written about a function I created while working on my latest project to simplify code and save time during the EDA process. For those not familiar with the post, this was the function I had created:
def initial_eda(df):    # List of categorical columns    cat_cols = df.select_dtypes('object').columns        for col in cat_cols:        # Formatting        column_name = col.title().replace('_', ' ')        title= 'Distribution of ' + column_name                # Unique values <= 12 to avoid overcrowding        if len(df[col].value_counts())<=12:             plt.figure(figsize = (8, 6))                    sns.countplot(x=df[col],                           data=df,                           palette="Paired",                          order = df[col].value_counts().index)            plt.title(title, fontsize = 18, pad = 12)            plt.xlabel(column_name, fontsize = 15)            plt.xticks(rotation=20)            plt.ylabel("Frequency",fontsize = 15)            plt.show();        else:            print(f'{column_name} has {len(df[col].value_counts())} unique values. Alternative EDA should be considered.')    returnSome examples of the visuals created from this code:
After running the Austin Animal Center dataset through this function, here were some of the visuals that were produced:
While this worked well for initial EDA, like I mentioned in my last post, there was plenty of room for improvement. For example, the function above only provides visuals for columns with twelve or fewer unique values. Those with more unique values provided these print statements instead:
While this does provide some information about each of these columns, there was a missed opportunity to extract insights about the data that could be valuable. Here is a new, updated version of this function:
def initial_eda(df):    # List of categorical columns    cat_cols = df.select_dtypes(['object','category']).columns        for col in cat_cols:        # Formatting        column_name = col.title().replace('_', ' ')        title = 'Distribution of ' + column_name        unique_values = len(df[col].value_counts())                # unique_values <=12 to avoid overcrowding        if unique_values<=12:             plt.figure(figsize = (10, 6))            sns.countplot(x=df[col],                           data=df,                           palette="Paired",                          order = df[col].value_counts().index)            plt.title(title, fontsize = 18, pad = 12)            plt.xlabel(column_name, fontsize = 15)            plt.xticks(rotation=20)            plt.ylabel("Frequency",fontsize = 15)            plt.show();        else:            print(f'{column_name} has {len(df[col].value_counts())} unique values. Here are the top 10:')            print()            col_count  = df[col].value_counts()            col_count = col_count[:10,]            plt.figure(figsize = (12, 6))            sns.barplot(x = col_count.index,                         y = col_count.values,                         palette="Paired")            plt.title(f'Top 10 {column_name}s', fontsize = 18, pad = 12)            plt.ylabel('Frequency', fontsize=15)            plt.xticks(rotation=20)            plt.xlabel(column_name, fontsize=15)            plt.show()                return
Rather than returning a print statement, the updated version of this function will now create a bar plot of the ten most common unique values if the column has more than twelve unique values. Some examples:
This addition to the function provides us with even more information and raises questions we may choose to pursue further. For example, from these visuals we can see that the animal with ID A721033 has entered the shelter more than thirty times. Why is that? What animal type is it? Was this animal a stray? Has this animal been adopted before? These additional visuals can provide valuable insights and pilot the direction of the project prior to the modeling process.
One final addition is made to the function to ensure the produced visuals are aesthetically pleasing. While the function as is provides us with meaningful information, a ‘one-figsize-fits-all’ approach can create bar plots that may be considered visually unappealing. For example:
Maybe this is me being picky, but I look at these graphs and know they can be improved. In my opinion, columns with two unique values should not have the same figure size as a column with twelve unique values. Here is the final, updated function that configures the figure size based off the number of unique values:
def initial_eda(df):    # List of categorical columns    cat_cols = df.select_dtypes(['object','category']).columns        for col in cat_cols:        # Formatting        column_name = col.title().replace('_', ' ')        title = 'Distribution of ' + column_name        unique_values = len(df[col].value_counts())                # If statements for figsize        if 2<=unique_values<3:             plt.figure(figsize = (4, 6))            sns.countplot(x=df[col],                           data=df,                           palette="Paired",                          order = df[col].value_counts().index)            plt.title(title, fontsize = 18, pad = 12)            plt.xlabel(column_name, fontsize = 15)            plt.xticks(rotation=20)            plt.ylabel("Frequency",fontsize = 15)            plt.show();        elif 3<=unique_values<8:            plt.figure(figsize = (8, 6))            sns.countplot(x=df[col],                           data=df,                           palette="Paired",                          order = df[col].value_counts().index)            plt.title(title, fontsize = 18, pad = 12)            plt.xlabel(column_name, fontsize = 15)            plt.xticks(rotation=20)            plt.ylabel("Frequency",fontsize = 15)            plt.show();        elif 8<=unique_values<=12:            plt.figure(figsize = (10, 6))            sns.countplot(x=df[col],                           data=df,                           palette="Paired",                          order = df[col].value_counts().index)            plt.title(title, fontsize = 18, pad = 12)            plt.xlabel(column_name, fontsize = 15)            plt.xticks(rotation=20)            plt.ylabel("Frequency",fontsize = 15)            plt.show();        else:            print(f'{column_name} has {len(df[col].value_counts())} unique values. Here are the top 10:')            print()            col_count  = df[col].value_counts()            col_count = col_count[:10,]            plt.figure(figsize = (12, 6))            sns.barplot(x = col_count.index,                         y = col_count.values,                         palette="Paired")            plt.title(f'Top 10 {column_name}s', fontsize = 18, pad = 12)            plt.ylabel('Frequency', fontsize=15)            plt.xticks(rotation=20)            plt.xlabel(column_name, fontsize=15)            plt.show()                return
Now, this function will present these visuals instead:
These are slightly more appealing to look at, in my opinion.
Exploratory Data Analysis (EDA) can shape the direction of a project. It is one of my favorite phases in the data science process as it is where one’s curiosity can be fully explored. While the “magic” of data science comes to life during the modeling phase, determining and optimizing a model to its fullest potential starts with proper data exploration.
My hope is that myself and others can utilize this function not just to save time, but more importantly to use the time saved to expand our curiosities.
The important thing is not to stop questioning. Curiosity has its own reason for existing. One cannot help but be in awe when he contemplates the mysteries of eternity, of life, of the marvelous structure of reality. It is enough if one tries merely to comprehend a little of this mystery every day. Never lose a holy curiosity. — Albert Einstein, LIFE Magazine (2 May 1955)
To read more about the Austin Animal Outcomes project, you can view it on my personal GitHub here.