Home Credit Default Risk
By: Ankit Yadav
Tags: Xgboost, Lightgbm, Classification, Time Series Analysis, Auc

TABLE OF CONTENTS:

BUSINESS PROBLEM BUSINESS CONSTRAINTS USE OF MACHINE LEARNING SOURCE OF DATA DATA DESCRIPTION EXPLORATORY DATA ANALYSIS (EDA)

6.1 NAME_CONTRACT_TYPE

6.2 CODE_GENDER

6.3 FLAG_OWN_CAR

6.4 FLAG_OWN_REALTY

6.5 CNT_CHILDREN

6.6 NAME_TYPE_SUITE

6.7 NAME_INCOME_TYPE

6.8 NAME_EDUCATION_TYPE

6.9 AMT_GOODS_PRICE

6.10 AMT_ANNUITY

6.11 AMT_CREDIT

6.12 DAYS_REGISTRATION

6.13 OWN_CAR_AGE

6.14 FLAG_MOBIL

6.15 ORGANIZATION_TYPE

6.16 EXT_SOURCE_1 , EXT_SOURCE_2 , EXT_SOURCE_3

6.17 DAYS_BIRTH

6.18 CREDIT_DAY_OVERDUE

6.19 CREDIT_TYPE

6.20 CREDIT_ACTIVE

6.21 NAME_CLIENT_TYPE

6.22 CODE_REJECT_REASON

6.23 CNT_INSTALMENT_FUTURE

6.24 NAME_CONTRACT_STATUS

6.25 CNT_INSTALMENT_MATURE_CUM

7. OUTLIERS

7.1 DAYS_EMPLOYED

7.2 CODE_GENDER

7.3 NAME_INCOME_TYPE

7.4 FLAG DOCUMENTS

7.5 DAYS_TERMINATION

7.6 DAYS_FIRST_DRAWING

7.7 DAYS_FIRST_DUE

8. FEATURE ENGINEERING

9. SPLITTING DATA

10. MODELLING

10.1 LOGISTIC REGRESSION

10.2 XGBOOST

10.3 RANDOM FOREST

10.4 LightGBM

11 COMPARING THE MODELS

12. KAGGLE SUBMISSION

13. FUTURE WORK

14. REFERENCES

15. GITHUB REPOSITORY

16. LINKEDIN PROFILE

1. BUSINESS PROBLEM

Home credit announce this kaggle competition with an aim to build a model on a data-set given by them which will predict whether a loan applicant is capable of repaying a loan.

2. BUSINESS CONSTRAINTS

No strict latency constraints

Given a loan application we can take a few minutes to predict the output rather then predicting it in second.

2. Interpretability is important

Since our model has predicted the output it is not critical but important to give feature importance in banking domain so as to why any particular applicant is capable for repaying a loan.

3. Miss-Classification cost is very high

If our model has predicted that an applicant is capable of repaying a loan but in real the applicant is not able to repay it and also the loan amount is big then it will be a huge loss for the bank.

3. USE OF MACHINE LEARNING

It is a binary class classification problem where 0 means person is capable of repaying a loan and 1 means person is not capable of repaying a loan.

Performance metric

AUC- It is good for binary class classification and imbalanced data set.

4. SOURCE OF DATA

The data set can be obtain from the below link.

5. DATA DESCRIPTION

application_{train|test}.csv - This is the main table, broken into two files for Train (with TARGET) and Test (without TARGET). Static data for all applications. One row represents one loan in our data sample.

This is the main table, broken into two files for Train (with TARGET) and Test (without TARGET). Static data for all applications. One row represents one loan in our data sample. bureau.csv - All client’s previous credits provided by other financial institutions that were reported to Credit Bureau (for clients who have a loan in our sample).For every loan in our sample, there are as many rows as number of credits the client had in Credit Bureau before the application date.

All client’s previous credits provided by other financial institutions that were reported to Credit Bureau (for clients who have a loan in our sample).For every loan in our sample, there are as many rows as number of credits the client had in Credit Bureau before the application date. bureau_balance.csv — Monthly balances of previous credits in Credit Bureau.This table has one row for each month of history of every previous credit reported to Credit Bureau — i.e the table has (#loans in sample * # of relative previous credits * # of months where we have some history observable for the previous credits) rows.

Monthly balances of previous credits in Credit Bureau.This table has one row for each month of history of every previous credit reported to Credit Bureau — i.e the table has (#loans in sample * # of relative previous credits * # of months where we have some history observable for the previous credits) rows. POS_CASH_balance.csv — Monthly balance snapshots of previous POS (point of sales) and cash loans that the applicant had with Home Credit. This table has one row for each month of history of every previous credit in Home Credit (consumer credit and cash loans) related to loans in our sample — i.e. the table has (#loans in sample * # of relative previous credits * # of months in which we have some history observable for the previous credits) rows.

Monthly balance snapshots of previous POS (point of sales) and cash loans that the applicant had with Home Credit. This table has one row for each month of history of every previous credit in Home Credit (consumer credit and cash loans) related to loans in our sample — i.e. the table has (#loans in sample * # of relative previous credits * # of months in which we have some history observable for the previous credits) rows. credit_card_balance.csv - Monthly balance snapshots of previous credit cards that the applicant has with Home Credit. This table has one row for each month of history of every previous credit in Home Credit (consumer credit and cash loans) related to loans in our sample — i.e. the table has (#loans in sample * # of relative previous credit cards * # of months where we have some history observable for the previous credit card) rows.

Monthly balance snapshots of previous credit cards that the applicant has with Home Credit. This table has one row for each month of history of every previous credit in Home Credit (consumer credit and cash loans) related to loans in our sample — i.e. the table has (#loans in sample * # of relative previous credit cards * # of months where we have some history observable for the previous credit card) rows. previous_application.csv - All previous applications for Home Credit loans of clients who have loans in our sample.There is one row for each previous application related to loans in our data sample.

All previous applications for Home Credit loans of clients who have loans in our sample.There is one row for each previous application related to loans in our data sample. installments_payments.csv - Repayment history for the previously disbursed credits in Home Credit related to the loans in our sample.There is a) one row for every payment that was made plus b) one row each for missed payment. One row is equivalent to one payment of one installment OR one installment corresponding to one payment of one previous Home Credit credit related to loans in our sample.

Repayment history for the previously disbursed credits in Home Credit related to the loans in our sample.There is a) one row for every payment that was made plus b) one row each for missed payment. One row is equivalent to one payment of one installment OR one installment corresponding to one payment of one previous Home Credit credit related to loans in our sample. HomeCredit_columns_description.csv- This file contains descriptions for the columns in the various data files.

Relation between table

6. EXPLORATORY DATA ANALYSIS (EDA)

Function to print bar plot

def stack_plot(data, xtick, col2, col3='total'):

ind = np.arange(data.shape[0])

%matplotlib inline

plt.figure(figsize=(10,3))

p1 = plt.bar(ind, data[col3].values)

p2 = plt.bar(ind, data[col2].values)

plt.ylabel('No of Loans')

plt.title('Number of peoples capable vs not capable of giving

loans')

plt.xticks(ind, list(data[xtick].values),rotation=-90)

plt.legend((p1[0], p2[0]), ('capable', 'not capable'))

plt.show()

#Count number of zeros in dataframe python:

https://stackoverflow.com/a/51540521/4084039

temp = pd.DataFrame(Df_application_train.groupby(col1)

col2].agg(lambda x: x.eq(1).sum())).reset_index()

#Pandas dataframe grouby count:

temp['total'] = pd.DataFrame(Df_application_train.groupby(col1)

[col2].agg({'total':'count'})).reset_index()

['total'] def univariate_barplots(data, col1, col2, top=False):#Count number of zeros in dataframe python:temp = pd.DataFrame(Df_application_train.groupby(col1)col2].agg(lambda x: x.eq(1).sum())).reset_index()#Pandas dataframe grouby count: https://stackoverflow.com

/a/19385591/4084039 temp['total'] = pd.DataFrame(Df_application_train.groupby(col1)[col2].agg({'total':'count'})).reset_index()['total'] temp['Avg'] = pd.DataFrame(Df_application_train.groupby(col1)

[col2].agg({'Avg':'mean'})).reset_index()['Avg']

temp.sort_values(by=['total'],inplace=True, ascending=False) if top:

temp = temp[0:top]

stack_plot(temp, xtick=col1, col2=col2, col3='total')

Function to plot pie chart

def labelupdate(autotexts,labels):

l=[]

for i in range(len(labels)):

l.append(str(autotexts[i]._text) + ' ' + str(labels[i]))

return(l) def plotpiechart(s):

%matplotlib inline

import random

labels=list(Df_application_train[s].unique())

labels=[x for x in labels if (str(x)!='nan')]

fracs1=[]

fracs2=[]

fracs3=[]

for x in labels:

count1 = sum(Df_application_train[s]==x)

fracs1.append(count1)

count2 =

sum(Df_application_train[Df_application_train['TARGET']==0]

[s]==x)

fracs2.append(count2)

count3 =

sum(Df_application_train[Df_application_train['TARGET']==1]

[s]==x)

fracs3.append(count3)

colors=["#9c9ede", "#7375b5", "#4a5584", "#cedb9c", "#b5cf6b",

"#8ca252", "#637939","#e7cb94", "#e7ba52", "#bd9e39",

"#8c6d31", "#e7969c", "#d6616b", "#ad494a","#843c39",

"#de9ed6", "#ce6dbd", "#a55194", "#7b4173","#1f77b4",

"#aec7e8","#ff7f0e", "#ffbb78", "#2ca02c", "#98df8a",

"#d62728", "#ff9896", "#9467bd","#c5b0d5", "#8c564b",

"#c49c94", "#e377c2", "#f7b6d2", "#7f7f7f",

"#c7c7c7","#bcbd22", "#dbdb8d", "#17becf",

"#9edae5","#1f77b4", "#ff7f0e", "#2ca02c","#d62728",

"#9467bd", "#8c564b", "#e377c2", "#7f7f7f", "#bcbd22",

"#17becf"]

colors=random.sample(colors, len(labels))

plt.title("Total distribution ")

patches,texts,autotexts=plt.pie(fracs1, labels=labels,

colors=colors, autopct='%1.1f%%',

shadow=True,radius=1,textprops={'size': 'smaller'})

l=labelupdate(autotexts,labels)

plt.legend(patches,l, bbox_to_anchor=(1,0.5), loc="best",

fontsize=10,bbox_transform=plt.gcf().transFigure)

plt.show()

plt.title("Loan payable ")

patches,texts,autotexts=plt.pie(fracs2, labels=labels,

colors=colors, autopct='%1.1f%%',

shadow=True,radius=1,textprops={'size': 'smaller'})

l=labelupdate(autotexts,labels)

plt.legend(patches,l, bbox_to_anchor=(1,0.5), loc="best",

fontsize=10,bbox_transform=plt.gcf().transFigure)

plt.show()

plt.title("Loan not payable ")

patches,texts,autotexts=plt.pie(fracs3, labels=labels,

colors=colors, autopct='%1.1f%%', shadow=True,textprops=

{'size': 'smaller'},radius=1)

l=labelupdate(autotexts,labels)

plt.legend(patches,l, bbox_to_anchor=(1,0.5), loc="best",

fontsize=10,bbox_transform=plt.gcf().transFigure)

plt.show()

6.1 NAME_CONTRACT_TYPE - Identification if loan is cash or revolving

univariate_barplots(Df_application_train, ‘NAME_CONTRACT_TYPE’, ‘TARGET’ , top=False)

observation:Many people took cash loans.

6.2 CODE_GENDER - Gender of the client

observation : Female took more no of loans then males.

6.3 FLAG_OWN_CAR- Flag if the client owns a car

observation : People who own a car took less no of loan.

6.4 FLAG_OWN_REALTY- Flag if client owns a house or flat

observation : People who owns a house or flat takes more no of loans.

6.5 CNT_CHILDREN- Number of children the client has

observation : Clients having no child take more no of loans and are also more defaulters.

6.6 NAME_TYPE_SUITE- Who was accompanying client when he was applying for the loan

observation : Client who came alone took more loan but also were more defaulters.

6.7 NAME_INCOME_TYPE- Clients income type (businessman, working, maternity leave)

observation : clients with income type working take more no of loans.

6.8 NAME_EDUCATION_TYPE- Level of highest education the client achieved

observation : people with highest degree as secondary and secondary special are more no of defaulters compare to people with other degree.

6.9 AMT_GOODS_PRICE- For consumer loans it is the price of the goods for which the loan is given

from math import log

count1 = Df_application_train[Df_application_train['TARGET']==0]['AMT_GOODS_PRICE']

count1 = count1.values

count1=[x for x in count1 if (str(x)!='nan')]

count2 = Df_application_train[Df_application_train['TARGET']==1]['AMT_GOODS_PRICE']

count2 = count2.values

count2=[x for x in count2 if (str(x)!='nan')]

plt.figure(figsize=(10,3))

sns.distplot(count1, hist=False, label="Capable ")

sns.distplot(count2, hist=False, label="Not Capable")

plt.title('Goods price')

plt.xlabel('price')

plt.ylabel('Density')

plt.legend()

plt.show()

Observation: Most no of loans are given for goods price below 10 lakhs

6.10 AMT_ANNUITY- Loan annuity

observation : Most people pay annuity below 50000 for the credit loan

6.11 AMT_CREDIT- Credit amount of the loan

observation : Credit amount of the loan is mostly less then 10 lakhs

6.12 DAYS_REGISTRATION- How many days before the application did client change his registration

Observation: Most of the applicant changed their registration 15000 days before the application.

6.13 OWN_CAR_AGE- Age of client’s car

observation : Most of the cars are in age group between 0–20 and less between 40–60.

6.14 FLAG_MOBIL- Did client provide mobile phone (1=YES, 0=NO)

There is only 1 client who did not provide the mobile information.

6.15 ORGANIZATION_TYPE- Type of organization where client works

observation : clients working in business and self employed are more no of defaulters but they also take more no of loan.

6.16 EXT_SOURCE_1 , EXT_SOURCE_2 , EXT_SOURCE_3- Normalized score from external data source

observation : In all the above 3 plots data is well seperated so these will be important features.

6.17 DAYS_BIRTH- Client’s age in days at the time of applicationClient’s age in days at the time of application

Observation : People of age group between 35–40 years takes maximum no of loans.

6.18 CREDIT_DAY_OVERDUE- Number of days past due on CB credit at the time of application for related loan in our sample

Observation: Most of the credits have close to 0 days overdue

6.19 CREDIT_TYPE- Type of Credit Bureau credit (Car, cash,…)

Observation: Consumer credit and Credit card are mostly registered credit in credit bureau.

6.20 CREDIT_ACTIVE- Status of the Credit Bureau (CB) reported credits

Observation: Most of the credit registered are in status closed. Sold and bad debt are very few.

6.21 NAME_CLIENT_TYPE- Was the client old or new client when applying for the previous application

Observation: There are more no of repeaters when applying for the previous application.

6.22 CODE_REJECT_REASON- Why was the previous application rejected

Observation: Most of the times bank has given not applicable as the reason for rejecting the loan.

6.23 CNT_INSTALMENT_FUTURE- Installments left to pay on the previous credit

Observation: installments left to pay mostly falls in the bracket of 0–20.

6.24 NAME_CONTRACT_STATUS- Contract status during the month

Observation: Most of the time name contract status is Active

6.25 CNT_INSTALMENT_MATURE_CUM- Number of paid installments on the previous credit

Observation: 75% of the paid installments on the previous credit are below 40.

7. OUTLIERS

There are some outliers in the data set we need to remove them so that we get a better auc for the model

7.1 DAYS_EMPLOYED

365243 is max value for DAYS_EMPLOYED and after dividing it by 365 it gives 1000.66 which is an outlier because how can a employ works for 1000 years.

7.2 CODE_GENDER

XNA is not a gender so it should be removed.

7.3 NAME_INCOME_TYPE

Income type can never be Maternity leave so this should be removed.

7.4 FLAG DOCUMENTS

All documents contains very less no of 1’s except FLAG_DOCUMENT_3 so we can remove them as they cannot give much information.

7.5 DAYS_TERMINATION

The min value is -2874 and max value is 365243. 365243 is equal to 1000 years hence it is an outlier. Nearly 225913 values are outliers here in column DAYS_TERMINATION

7.6 DAYS_FIRST_DRAWING

All the values above 0 are 365243 and they all are outliers. Nearly 934444 values in DAYS_FIRST_DRAWING column are outliers except 62705 values which are less than 0.the min value here is -2922

7.7 DAYS_FIRST_DUE

Observation: 365243 is outlier here.

8. FEATURE ENGINEERING

Domain related features

I have used some domain related features like for example DAYS_BIRTH/DAYS_EMPLOYED if DAYS_BIRTH=10 and DAYS_EMPLOYED=1 then ratio is 10 this says that if an employee age is more and if he worked for less no of days then then ratio willl be more and vice versa is less so this way it helps to classify the labels.

Aggregated features

I just found out the max, mean, sum of columns and apply aggregation on them and add as a new column.

Time related features

In this feature engineering technique i have taken out data from certain time let ‘s say last 1 year and then i apply all the aggregation again on this data set and merge it with the main dataframe using left outer join. For last month data i can simply give pr.DAYS_DECISION ≥-31 in above image.I will do this for all data set except application.csv because it does not contain any time related feature.

9. SPLITTING DATA

I have splitted the data with cv size=27650, test size=30723, train size=248850 and also apply SelectKBest to select top 300 features.I could have used all the features but the task was computation intensive so i took only 300 features.

10. MODELLING

10.1 Logistic regression

Logistic regression is a linear algorithm whose aim is to find a plane which can classify the points into different class.First i have to hyperparameter tune the value of ‘alpha’ and then i can train the model on best ‘alpha’ valueand get the auc score.

code for hyper parameter tuning

plot between auc and parameters

We can see that our best value of alpha is 10**-3 now we can train our model on this alpha value and get the auc score.

We can that our auc score on test data is 0.748 and there is very less overfitting.Let’s try out some other models.

10.2 XGBOOST

XGBoost is an ensemble method which uses boosting intuition to create a strong classifier with low variance and low bias by additive combining the weak classifier having low var and high bias. After every iteration it adds one weak classifier which aims to correct the error made by previous model.

After hyperparameter tuning with RandomizedSearchCV we got best value for hyperparameters ,here n-estimators is the no of weak classifiers and learning_rate is shrinkage value and max_depth is tree depth ,now we use these values to train the model.

We have got an auc of 0.779 with xgboost classifier.

10.3 RANDOM FOREST

Random forest is an ensemble method which uses bagging intuition to create a strong classifier having low variance and low bias by parallelly training many weak classifier having high variance and low bias on the bootstrapped samples and finally take majority vote to predict the output.

I have got the best parameter on which i train my model and find the auc score also here base learners are decision tree.

I have got an AUC of 0.741 which is less then the xgboost.

10.4 LightGBM (Light Gradient Boosting Machine)

LightGBM is similar to xgboost but has some advantage over it

Fast training efficiency because of histogram binning. Better accuracy because it uses leaf wise growth so it can form complex trees whereas XGBoost uses level wise growth so it cannot form complex trees. Low memory usage because it replaces continuous values to discrete bins which result in lower memory usage.

Now as we know that lightgbm use less memory and is fast so we can take all 2337 features to train the model and we will use Bayesian optimization to predict optimal parameters for the model.

Bayesian optimization

Bayesian optimization methods is used where we need to optimize the functions which can take too much time to execute. It just build a probability model(surrogate model) of the objective function to propose smarter choices for the next set of hyperparameters to evaluate.It gives different set of parameters to surrogate model and obtain the value like P(y/x) here x is set of parameters given and y is the predicted value. Whichever set gets good y value that set will be given to objective function and the value is predicted in this way we are only giving best set of parameters to the objective function hence saving the time.

The Bayesian Optimization package can be installed with the following command !pip install bayesian-optimization

code for predicting parameter using bayesian optimization

Best parameters

we have got the best parameters now i can train my lightGBM model with these parameters. I am using 10 Kfold cross validation so that it uses all data for training as well as for cross validation.

Training and saving the model

We have got an AUC score of 79.47 which is better from all the model

plot for feature importance

In EDA we noticed that EXT_SOURCE features,domain related features and aggregated features are important , now here is the proof of that as we got more features importance for these features. The most important feature is AMT_CREDIT/AMT_ANNUITY it tells me the length of annuity.

11. COMPARING THE MODELS

Among all the models we have trained LightGBM performs best with AUC of 79.48 so i can use this model on our test data.

12. KAGGLE SUBMISSION

I have uploaded my test file on kaggle and i got a leaderboard score of 0.79790 which brings me in top 2% of rank

13. FUTURE WORK

We can try more domain related features.

We can use stacking.

We can try deep learning models

14. REFERENCES

Bayesian optimization

https://www.kaggle.com/returnofsputnik/bayes-opt-again/code

https://www.dlology.com/blog/how-to-do-hyperparameter-search-with-baysian-optimization-for-keras-model/

https://www.kaggle.com/returnofsputnik/bayes-opt-again/code https://www.dlology.com/blog/how-to-do-hyperparameter-search-with-baysian-optimization-for-keras-model/ Time related features

https://www.kaggle.com/c/home-credit-default-risk/discussion/64593

https://www.kaggle.com/c/home-credit-default-risk/discussion/64593 Domain specific features

https://www.kaggle.com/aantonova/797-lgbm-and-bayesian-optimization

https://www.kaggle.com/aantonova/797-lgbm-and-bayesian-optimization Aggregation featurization

https://www.kaggle.com/jsaguiar/lightgbm-7th-place-solution

https://www.linkedin.com/pulse/winning-9th-place-kaggles-biggest-competition-yet-home-levinson/

https://www.kaggle.com/jsaguiar/lightgbm-7th-place-solution https://www.linkedin.com/pulse/winning-9th-place-kaggles-biggest-competition-yet-home-levinson/ https://www.appliedaicourse.com/

15. GITHUB REPOSITORY

https://github.com/ankuyadav17/Home-credit-default-risk

16. LINKEDIN PROFILE

https://www.linkedin.com/in/ankit-yadav-809773100/