A Beginner’s Dilemma in Exploratory Data Analysis
By: Ksv Muralidhar
Tags: Data Science, Tableau, Beginner, Data Analysis, Exploratory Data Analysis

A Beginner’s Dilemma in Exploratory Data Analysis

Exploratory data analysis (EDA) is an important step in a data science project where you get a feel for your data. KSV Muralidhar Jul 24·4 min read

For beginners, EDA might pose a few challenges. This article discusses one of such challenges that every beginner in exploratory data analysis may face at some point in time. This article assumes the reader has a basic knowledge of EDA.

Photo by Firmbee.com on Unsplash

Recently, I was playing with a toy dataset (Loan Data) acquired from Kaggle. I used this dataset to create a dashboard in Tableau and publish it to Tableau Public (you may find the published dashboard here). In the early days of my data science journey, I had a dilemma of choosing between the questions to get the right answer from the data. I felt it would be worth sharing this with the beginners in this field.

Let’s understand the dilemma with an example. In the loan dataset, assume we need to find out “which type of loan (‘purpose’ feature in the dataset) is most likely to see a default?”. I can think of two methods to find an answer.

Method 1

In this method, I first select only the borrowers who have defaulted (1 in the ‘not.fully.paid’ feature) and then plot a bar chart of loan type for these defaulted borrowers. Then the plot would look like the one below.

Loans taken by defaulters (Image by author)

The plot above shows that, most of the defaulters took ‘debt_consolidation’ loans. Hence, we may say ‘debt_consolidation’ loans have the highest number of defaults.

Method 2

In this method, we’ll plot a stacked bar chart to see the proportion of the defaults by loan type.

Proportion of default by loan type (Image by author)

The above plot says a different story. Here, ‘small_business’ loans have the highest percentage of defaults.

Which method is right?

Method 2 is the right way to answer the question “Which type of loan is most likely to see a default?”. Then, what’s wrong in the method 1?

Method 1 answers the question “Which type of loan was most popular among defaulters?”. In the method 1, we had ‘debt _consolidation’ loans at the top because they were the most popular type of loan offered by the lender (to both defaulters and non-defaulters). Let's see it in the plot below.

Frequency of loans (Image by author)

As we can see ‘debt_consolidation’ was the most popular loan offered by the lender. More than 50% of the loans taken were for debt consolidation. Since, debt consolidation loans are offered to many borrowers, the defaults are also more. It doesn’t mean that people who took debt consolidation loans have relatively more defaults.

How to overcome this dilemma?

1. Frame the question correctly

Framing the question correctly solves 90% of your dilemma. In the above example, we wanted to know “which loan type is most likely to see a default”, NOT “which loan was most popular among defaulters”.

2. Believe in proportions, not absolute numbers

Let’s try the method 2 with absolute numbers instead of proportion. We’ll see that we end up getting the same results as in method 1.

Frequency of defaults by loan type (Image by author)

In the above plot, we again see debt consolidation loans have the highest number of defaults. But, that’s incorrect. Since debt consolidation loans are offered by the lender in huge numbers, the defaults would also be high. It doesn’t mean that debt consolidation loans have the highest default rate.

It’s the same as saying Americans are more likely to lose their jobs at a company X in the U.S. compared to the Indians. Assume, there are 100 Americans and 10 Indians employed by X. X laid off 10 Americans and 5 Indians. It doesn’t mean that Americans would more likely lose their job. If you see in absolute terms, more Americans are being laid off, but also remember that they form the majority of the employee composition. But, if you see in percentage terms only 10% of the Americans were laid off, while 50% of the Indians were laid off. Hence, absolute numbers may be misleading, so believe in proportions.