Insightful Loan Default Analysis. Exploring and crunching out the driving…
By: Sparsh Gupta
Tags: Towards Data Science, Data Science, Exploratory Data Analysis, Credit Risk, Data Visualization

Metrics Derivation

Issue date is not in the standard format also we can split the date into two columns with month and the year which will make it easy for analysis

Year in the datetime requires year between 00 to 99 and in some cases year is single digit number i.e. 9 writing a function which will convert such dates to avoid exception in date conversion.

def standerdisedate(date):

year = date.split("-")[0]

if(len(year) == 1):

date = "0"+date

return date from datetime import datetime

loan['issue_d'] = loan['issue_d'].apply(lambda x:standerdisedate(x))

loan['issue_d'] = loan['issue_d'].apply(lambda x: datetime.strptime(x, '%b-%y')) # extracting month and year from issue_date

loan['month'] = loan['issue_d'].apply(lambda x: x.month)

loan['year'] = loan['issue_d'].apply(lambda x: x.year) # get year from issue_d and replace the same

loan["earliest_cr_line"] = pd.to_numeric(loan["earliest_cr_line"].apply(lambda x:x.split('-')[1]))

Binning Continuous features:

# create bins for loan_amnt range

bins = [0, 5000, 10000, 15000, 20000, 25000, 36000]

bucket_l = ['0-5000', '5000-10000', '10000-15000', '15000-20000', '20000-25000','25000+']

loan['loan_amnt_range'] = pd.cut(loan['loan_amnt'], bins, labels=bucket_l) # create bins for int_rate range

bins = [0, 7.5, 10, 12.5, 15, 100]

bucket_l = ['0-7.5', '7.5-10', '10-12.5', '12.5-15', '15+']

loan['int_rate_range'] = pd.cut(loan['int_rate'], bins, labels=bucket_l) # create bins for annual_inc range

bins = [0, 25000, 50000, 75000, 100000, 1000000]

bucket_l = ['0-25000', '25000-50000', '50000-75000', '75000-100000', '100000+']

loan['annual_inc_range'] = pd.cut(loan['annual_inc'], bins, labels=bucket_l) # create bins for installment range

def installment(n):

if n <= 200:

return 'low'

elif n > 200 and n <=500:

return 'medium'

elif n > 500 and n <=800:

return 'high'

else:

return 'very high'



loan['installment'] = loan['installment'].apply(lambda x: installment(x)) # create bins for dti range

bins = [-1, 5.00, 10.00, 15.00, 20.00, 25.00, 50.00]

bucket_l = ['0-5%', '5-10%', '10-15%', '15-20%', '20-25%', '25%+']

loan['dti_range'] = pd.cut(loan['dti'], bins, labels=bucket_l)

The following bins are created:

(Image by Author)

Visualising Data Insights

# check for amount of defaults in the data using countplot

plt.figure(figsize=(14,5))

sns.countplot(y="loan_status", data=loan)

plt.show()

(Image by Author)

From above plot we can see that around 16% i.e. 5062 people are defaulters in total 35152 records.

Univariate Analysis

# function for plotting the count plot features wrt default ratio

def plotUnivariateRatioBar(feature, data=loan, figsize=(10,5), rsorted=True):

plt.figure(figsize=figsize)

if rsorted:

feature_dimension = sorted(data[feature].unique())

else:

feature_dimension = data[feature].unique()

feature_values = []

for fd in feature_dimension:

feature_filter = data[data[feature]==fd]

feature_count = len(feature_filter[feature_filter["loan_status"]==1])

feature_values.append(feature_count*100/feature_filter["loan_status"].count())

plt.bar(feature_dimension, feature_values, color='orange', edgecolor='white')

plt.title("Loan Defaults wrt "+str(feature)+" feature - countplot")

plt.xlabel(feature, fontsize=16)

plt.ylabel("defaulter %", fontsize=16)

plt.show() # function to plot univariate with default status scale 0 - 1

def plotUnivariateBar(x, figsize=(10,5)):

plt.figure(figsize=figsize)

sns.barplot(x=x, y='loan_status', data=loan)

plt.title("Loan Defaults wrt "+str(x)+" feature - countplot")

plt.xlabel(x, fontsize=16)

plt.ylabel("defaulter ratio", fontsize=16)

plt.show()

a. Categorical Features

# check for defaulters wrt term in the data using countplot

plotUnivariateBar("term", figsize=(8,5))

(Image by Author)

From above plot for ‘term’ we can infer that the defaulters rate is increasing wrt term, hence the chances of loan getting deaulted is less for 36m than 60m. is term benificial -> Yes

# check for defaulters wrt grade in the data using countplot

plotUnivariateRatioBar("grade")

(Image by Author)

From above plot for ‘grade’ we can infer that the defaulters rate is increasing wrt grade, hence the chances of loan getting deaulted increases with the grade from A moving towards G. is grade benificial -> Yes

# check for defaulters wrt sub_grade in the data using countplot

plotUnivariateBar("sub_grade", figsize=(16,5))

(Image by Author)

From above plot for ‘sub_grade’ we can infer that the defaulters rate is increasing wrt sub_grade, hence the chances of loan getting deaulted increases with the sub_grade from A1 moving towards G5. is sub_grade benificial -> Yes

# check for defaulters wrt home_ownership in the data

plotUnivariateRatioBar("home_ownership")

(Image by Author)

From above plot for ‘home_ownership’ we can infer that the defaulters rate is constant here (it is quite more for OTHERS but we dont know what is in there, so we’ll not consider it for analysis), hence defaulter does not depends on home_ownership is home_ownership benificial -> No

# check for defaulters wrt verification_status in the data

plotUnivariateRatioBar("verification_status")

(Image by Author)

From above plot for ‘verification_status’ we can infer that the defaulters rate is increasing and is less for Not Verified users than Verified ones, but not useful for analysis. is verification_status benificial -> No

# check for defaulters wrt purpose in the data using countplot

plotUnivariateBar("purpose", figsize=(16,6))

(Image by Author)

From above plot for ‘purpose’ we can infer that the defaulters rate is nearly constant for all purpose type except ‘small business’, hence rate will depend on purpose of the loan is purpose benificial -> Yes

# check for defaulters wrt open_acc in the data using countplot

plotUnivariateRatioBar("open_acc", figsize=(16,6))

(Image by Author)

From above plot for ‘open_acc’ we can infer that the defaulters rate is nearly constant for feature open_acc, hence rate will not depend on open_acc feature is open_acc benificial -> No

# check for defaulters wrt pub_rec in the data using countplot

plotUnivariateRatioBar("pub_rec")

(Image by Author)

From above plot for ‘pub_rec’ we can infer that the defaulters rate is nearly increasing as it is less for 0 and more for pub_rec with value 1, but as other values are very less as compared to 0 we’ll not consider this is pub_rec benificial -> No

b. Continuous Features

# check for defaulters wrt emp_length in the data using countplot

plotUnivariateBar("emp_length", figsize=(14,6))

(Image by Author)

From above plot for ‘emp_length’ we can infer that the defaulters rate is constant here, hence defaulter does not depends on emp_length is emp_length benificial -> No

# check for defaulters wrt month in the data using countplot

plotUnivariateBar("month", figsize=(14,6))

(Image by Author)

From above plot for ‘month’ we can infer that the defaulters rate is nearly constant here, not useful is month benificial -> No

# check for defaulters wrt year in the data using countplot

plotUnivariateBar("year")

(Image by Author)

From above plot for ‘year’ we can infer that the defaulters rate is nearly constant here, not useful is year benificial -> No

# check for defaulters wrt earliest_cr_line in the data

plotUnivariateBar("earliest_cr_line", figsize=(16,10))

(Image by Author)

From above plot for ‘earliest_cr_line’ we can infer that the defaulters rate is nearly constant for all purpose type except year around 65, hence rate does not depends on earliest_cr_line of the person is earliest_cr_line benificial -> No

# check for defaulters wrt inq_last_6mths in the data

plotUnivariateBar("inq_last_6mths")

(Image by Author)

From above plot for ‘inq_last_6mths’ we can infer that the defaulters rate is not consistently increasing with inq_last_6mths type, hence not useful is inq_last_6mths benificial -> No

# check for defaulters wrt revol_util in the data using countplot

plotUnivariateRatioBar("revol_util", figsize=(16,6))

(Image by Author)

From above plot for ‘revol_util’ we can infer that the defaulters rate is fluctuating where some have complete 100% ratio for defaulter and is increasing as the magnitude increases, hence rate will depend on revol_util feature is revol_util benificial -> Yes

# check for defaulters wrt total_acc in the data using countplot

plotUnivariateRatioBar("total_acc", figsize=(14,6))

(Image by Author)

From above plot for ‘total_acc’ we can infer that the defaulters rate is nearly constant for all total_acc values, hence rate will not depend on total_acc feature is total_acc benificial -> No

# check for defaulters wrt loan_amnt_range in the data using countplot

plotUnivariateBar("loan_amnt_range")

(Image by Author)

From above plot for ‘loan_amnt_range’ we can infer that the defaulters rate is increasing loan_amnt_range values, hence rate will depend on loan_amnt_range feature is loan_amnt_range benificial -> Yes

# check for defaulters wrt int_rate_range in the data

plotUnivariateBar("int_rate_range")

(Image by Author)

From above plot for ‘int_rate_range’ we can infer that the defaulters rate is decreasing with int_rate_range values, hence rate will depend on int_rate_range feature is int_rate_range benificial -> Yes

# check for defaulters wrt annual_inc_range in the data

plotUnivariateBar("annual_inc_range")

(Image by Author)

From above plot for ‘annual_inc_range’ we can infer that the defaulters rate is decreasing as with annual_inc_range values, hence rate will depend on annual_inc_range feature is annual_inc_range benificial -> Yes

# check for defaulters wrt dti_range in the data using countplot

plotUnivariateBar("dti_range", figsize=(16,5))

(Image by Author)

From above plot for ‘dti_range’ we can infer that the defaulters rate is increasing as with dti_range values, hence rate will depend on dti_range feature is dti_range benificial -> Yes

# check for defaulters wrt installment range in the data

plotUnivariateBar("installment", figsize=(8,5))

(Image by Author)

From above plot for ‘installment’ we can infer that the defaulters rate is increasing as with installment values, hence rate will depend on dti_range feature is installment benificial -> Yes

Therefore, following are the important feature we deduced from above Univariate analysis:

term, grade, purpose, pub_rec, revol_util, funded_amnt_inv, int_rate, annual_inc, dti, installment

Bivariate Analysis

# function to plot scatter plot for two features

def plotScatter(x, y):

plt.figure(figsize=(16,6))

sns.scatterplot(x=x, y=y, hue="loan_status", data=loan)

plt.title("Scatter plot between "+x+" and "+y)

plt.xlabel(x, fontsize=16)

plt.ylabel(y, fontsize=16)

plt.show()



def plotBivariateBar(x, hue, figsize=(16,6)):

plt.figure(figsize=figsize)

sns.barplot(x=x, y='loan_status', hue=hue, data=loan)

plt.title("Loan Default ratio wrt "+x+" feature for hue "+hue+" in the data using countplot")

plt.xlabel(x, fontsize=16)

plt.ylabel("defaulter ratio", fontsize=16)

plt.show()

Plotting for two different features with respect to loan default ratio on y-axis with Bar Plots and Scatter Plots.

# check for defaulters wrt annual_inc and purpose in the data using countplot

plotBivariateBar("annual_inc_range", "purpose")

(Image by Author)

From above plot, we can infer it doesn’t shows any correlation related - N

# check for defaulters wrt term and purpose in the data

plotBivariateBar("term", "purpose")

(Image by Author)

As we can see straight lines on the plot, default ratio increases for every purpose wrt term related - Y

# check for defaulters wrt grade and purpose in the data

plotBivariateBar("grade", "purpose")

(Image by Author)

As we can see straight lines on the plot, default ratio increases for every purpose wrt grade related - Y

# check for defaulters wrt loan_amnt_range and purpose in the data

plotBivariateBar("loan_amnt_range", "purpose")

(Image by Author)

As we can see straight lines on the plot, default ratio increases for every purpose wrt loan_amnt_range related - Y

# check for defaulters wrt loan_amnt_range and term in the data

plotBivariateBar("loan_amnt_range", "term")

(Image by Author)

As we can see straight lines on the plot, default ratio increases for every term wrt loan_amnt_range related - Y

# check for defaulters wrt annual_inc_range and purpose in the data

plotBivariateBar("annual_inc_range", "purpose")

(Image by Author)

As we can see straight lines on the plot, default ratio increases for every purpose wrt annual_inc_range related - Y

# check for defaulters wrt annual_inc_range and purpose in the data

plotBivariateBar("installment", "purpose")

(Image by Author)

As we can see straight lines on the plot, default ratio increases for every purpose wrt installment except for small_business related - Y

# check for defaulters wrt loan_amnt_range in the data

plotScatter("int_rate", "annual_inc")

(Image by Author)

As we can see straight lines on the plot, there is no relation between above mentioned features related - N

# plot scatter for funded_amnt_inv with dti

plotScatter("funded_amnt_inv", "dti")

(Image by Author)

As we can see straight lines on the plot, there is no relation between above mentioned features related - N

# plot scatter for funded_amnt_inv with annual_inc

plotScatter("annual_inc", "funded_amnt_inv")

(Image by Author)

As we can see slope pattern on the plot, there is positive relation between above mentioned features related - Y

# plot scatter for loan_amnt with int_rate

plotScatter("loan_amnt", "int_rate")

(Image by Author)

As we can see straight line patterns on the plot, there is no relation between above mentioned features related - N

# plot scatter for int_rate with annual_inc

plotScatter("int_rate", "annual_inc")

(Image by Author)

As we can see negative correlation pattern with reduced density on the plot, there is some relation between above mentioned features related - Y

# plot scatter for earliest_cr_line with int_rate

plotScatter("earliest_cr_line", "int_rate")

(Image by Author)

As we can see positive correlation pattern with increasing density on the plot, there is co-relation between above mentioned features related - Y

# plot scatter for annual_inc with emp_length

plotScatter("annual_inc", "emp_length")

(Image by Author)

As we can see straight line patterns on the plot, there is no relation between above mentioned features related - N

# plot scatter for earliest_cr_line with dti

plotScatter("earliest_cr_line", "dti")

(Image by Author)

Plotting for two different features with respect to loan default ratio on y-axis with Box Plots and Violin Plots.

# function to plot boxplot for comparing two features

def plotBox(x, y, hue="loan_status"):

plt.figure(figsize=(16,6))

sns.boxplot(x=x, y=y, data=loan, hue=hue, order=sorted(loan[x].unique()))

plt.title("Box plot between "+x+" and "+y+" for each "+hue)

plt.xlabel(x, fontsize=16)

plt.ylabel(y, fontsize=16)

plt.show()

plt.figure(figsize=(16,8))

sns.violinplot(x=x, y=y, data=loan, hue=hue, order=sorted(loan[x].unique()))

plt.title("Violin plot between "+x+" and "+y+" for each "+hue)

plt.xlabel(x, fontsize=16)

plt.ylabel(y, fontsize=16)

plt.show() # plot box for term vs int_rate for each loan_status

plotBox("term", "int_rate")

(Image by Author)

(Image by Author)

int_rate increases with term on loan and the chances of default also increases

# plot box for loan_status vs int_rate for each purpose

plotBox("loan_status", "int_rate", hue="purpose")

(Image by Author)

(Image by Author)

int_rate is quite high where the loan is defaulted for every purpose value

# plot box for purpose vs revo_util for each status

plotBox("purpose", "revol_util")

(Image by Author)

(Image by Author)

revol_util is more for every purpose value where the loan is defaulted and quite high for credit_card

# plot box for grade vs int_rate for each loan_status

plotBox("grade", "int_rate", "loan_status")

(Image by Author)

(Image by Author)

int_rate is increasing with every grade and also the defaulters for every grade are having their median near the non-defaulter 75% quantile of int_rate

# plot box for issue_d vs int_rate for each loan_status

plotBox("month", "int_rate", "loan_status")

(Image by Author)

(Image by Author)

int_rate for defaulter is increasing with every month where the defaulters for every month are having their median near the non-defaulter’s 75% quantile of int_rate, but is almost constant for each month, not useful

Therefore, following are the important feature we deduced from above Bivariate analysis:

term, grade, purpose, pub_rec, revol_util, funded_amnt_inv, int_rate, annual_inc, installment

Multivariate Analysis (Correlation)

# plot heat map to see correlation between features

continuous_f = ["funded_amnt_inv", "annual_inc", "term", "int_rate", "loan_status", "revol_util", "pub_rec", "earliest_cr_line"]

loan_corr = loan[continuous_f].corr()

sns.heatmap(loan_corr,vmin=-1.0,vmax=1.0,annot=True, cmap="YlGnBu")

plt.title("Correlation Heatmap")

plt.show()

(Image by Author)

Hence, important related feature from above Multivariate analysis are:

term, grade, purpose, revol_util, int_rate, installment, annual_inc, funded_amnt_inv

Final Findings

After analysing all the related features available in the dataset, we have come to an end, deducing the main driving features for the Lending Club Loan Default analysis:

The best driving features for the Loan default analysis are: term, grade, purpose, revol_util, int_rate, installment, annual_inc, funded_amnt_inv