My biggest learning while solving Classification problems in Machine Learning
I started my Data Science journey a few months back. I had been doing my share of learning before that. I thought I was ready as I had been…
Category: Data Science
Author: tanvi purohit


My biggest learning while solving Classification problems in Machine Learning
I started my Data Science journey a few months back. I had been doing my share of learning before that. I thought I was ready as I had been participating in hackathons and competitions before. But I must admit — working on real life data science problems is a different beast altogether.
So, what was my biggest learning in my new data science role?
It was that “Defining the problem statement for Classification Problems” can be very challenging. Now, I know — it might sound simple to you. It did to me when I first started, until I realized how difficult it could be.
Let me explain this in more detail
I got a chance to explore Indian Fintech and Insurance industry. And it was very critical to assess the risk well. Or in other words — “How to define good vs bad?”

This is never an issue when we participate in Data Hackathons, where you get Ready to use data. As you get a labeled data set which makes it easy to differentiate between Good and Bad.
For example: In the image below (Image 1.1), it is easy to see that loan_status 0 refers to Good customers while loan_status 1 refers to Bad customers.
Image 1.1- Data for Loan Prediction Hack
However, in real life problems, designing a definition that will efficiently classify the data into multiple or binary classes can end up being a real challenge.
Incorrect choice will put your further analysis and model for a toss. That means designing the criteria that defines 1 and 0 is one of the critical task on any given data problem.

Example — Credit Risk Modeling
Let us start with understanding, what is Credit Risk?
Credit risk refers to the probability of loss due to a borrower’s failure to make payments on any type of debt.
Credit Risk Management, meanwhile, is the practice of mitigating losses incurred due to credit defaults by understanding the adequacy of both a bank’s capital and loan loss reserves at any given time.
Why is credit risk important for a financial institution?
Without a doubt, Credit Risk Management is a process that has been a challenge for financial institutions, however a better developed process enables a financial institute to absorb the higher capital costs for credit risk. Better credit risk management also presents an opportunity to greatly improve overall performance and secure a competitive advantage.
Data available with financial institutes
Typically, data available for a given credit risk model will comprise of data for loans given by the financial institute to the customers in past. This will include:
Demographics
Income Details
Bureau details
Customer repayment transactions.
The idea is to use this data to understand which customers are defaulting on their payments, going in partial or complete delinquencies (not paying the due payments on time), their loss bookings, debt to income ratio and much more.
Image 1.2: Glimpse of how real life data looks like
So now, to assess the risk better — we need to define who are good customers are who are bad and this is where our challenge starts.
A naive approach
A simple way might be to say Good Customers as customers who have never defaulted on any payment and Bad Customers as At-least defaulted once. (Default here refers to No of days past due the EMI/Premium payment date).
Sound good — This is setback moment.
Not every time a definition as simple as this would work. One would have to go through multiple iterations with the business to crack this definition.
A set of bad definitions can be chosen in this situation could be:
Customers Defaulted ever and also led to loss bookings
Customers Defaulted ever and had high debt to income ratio
Customer Defaulted in first 3 months of their tenure at-least twice.
Customers Defaulted in first 6 months of their tenure at-least thrice
Customers Defaulted in first 12 months of their tenure consecutively thrice.
Customers Defaulted in first 18 months of their tenure consecutively thrice.
This is not an exhaustive list. Putting all options could be taxing.
Thus, to finalize on a “Definition for Good and Bad”, generally few prescribed definitions are tested over sample of data. Typically, the real challenge is to arrive at a definition that captures “MOST- BAD” customers in shortest span, and exactly here is where Roll Rate Analysis helps.
What is Roll Rate analysis?
In this exercise, we choose a period(Months/Years) for analysis as well a static pool of customers on whom analysis is to be done. (Customers who took loan at around same time refers to static pool. Eg: All the customers who took loan in March 2018 is defined as static pool for March)
One must choose the period for Roll Rate analysis very tactfully and must note that no grave external factors like Demonetization are affecting or biasing the results. Once this period is decided we create a matrix that will analyze this static pool of customers for different default rates.
Image 1.3: Matrix highlighting the percentages of defaults in last 18 months window
The above image(Image 1.3) tells us the percentages of customers in various default buckets at the end of 18 months performance.
The first line in matrix explains us that there were 87% customers in no due bucket of which, 5% went in 30 days due and so on..
Similarly the second row in the matrix says that at the start of analysis there were 10% customers in 30 days due and in the period if 18 months of them, 77% of them jumped below 30 days due while 13% jumped ahead of 30 days due and so on…
Building such matrix one can determine the best balance between Roll back and Roll forward and bench mark as the segmenting point for good vs bad. Thus above matrix states using 120 days past due is the best number.
However the complexity does not end here. Once we decide on appropriate days past due number we must check the optimal time for which we must check 120 days past due. ie, 120 days past due in first 4 months, 120 days past due in first 6 months , 120 days past due in first 9 months, 120 days past due in first 12 months etc. The below curve shall solve the issue(Image 1.2).
Image 1.3: Counts of customers in days past due for different static pools.
Here we see that for different vintages of customers, a flattening phenomenon is observed post 9 months of loan tenure. Thus this will further strengthen our definition making it more accurate as 120 Days past Due in first 9 months.
So, to summarize, I would like to emphasize choosing the correct Bad definition is indeed a critical task and definitely requires some iterations to zero in on. The Credit Risk problem discussed in this article is also applicable for transaction centric scenarios to understand the behavior of the customers in greater depth. Such an in depth approach shall help us define “MOST-BAD” customers “a Best Criterion for a Classification problem” in shortest span.
Have you encountered a situation like this where you were caught in defining the problem in the right manner? If so, let me know, what was your experience. I would love to hear and learn from your experience.