Machine Learning: Predicting Bank Loan Defaults | by Gonçalo Guimarães Gomes | Towards Data Science

The dataset contains information about credit applicants. Banks, globally, use this kind of dataset and type of informative data to create models to help in deciding on who to accept/refuse for a loan.
After all the exploratory data analysis, cleansing and dealing with all the anomalies we might (will) find along the way, the patterns of a good/bad applicant will be exposed to be learned by machine learning models.
We’re dealing with a supervised binary classification problem. The goal is to train the best machine learning model to maximize the predictive capability of deeply understanding the past customer’s profile minimizing the risk of future loan defaults.
The metric used for the models’ evaluation is the ROC AUC given that we’re dealing with a highly unbalanced data.
The project divides into three categories:
EDA: Exploratory Data AnalysisData Wrangling: Cleansing and Feature SelectionMachine Learning: Predictive Modelling
EDA: Exploratory Data Analysis
Data Wrangling: Cleansing and Feature Selection
Machine Learning: Predictive Modelling
You can download the data set here.
id: Unique ID of the loan application.
grade: LC assigned loan grade.
annual_inc: The self-reported annual income provided by the borrower during registration.
short_emp: 1 when employed for 1 year or less.
emp_length_num: Employment length in years. Possible values are between 0 and 10 where 0 means less than one year and 10 means ten or more years.
home_ownership: Type of home ownership.
dti (Debt-To-Income Ratio): A ratio calculated using the borrower’s total monthly debt payments on the total debt obligations, excluding mortgage and the requested LC loan, divided by the borrower’s self-reported monthly income.
purpose: A category provided by the borrower for the loan request.
term: The number of payments on the loan. Values are in months and can be either 36 or 60.
last_delinq_none: 1 when the borrower had at least one event of delinquency.
last_major_derog_none: 1 borrower had at least 90 days of a bad rating.
revol_util: Revolving line utilization rate, or the amount of credit the borrower is using relative to all available revolving credit.
total_rec_late_fee: Late fees received to date.
od_ratio: Overdraft ratio.
bad_loan: 1 when a loan was not paid.
Importing the libraries and dependencies required:
import pandas as pdimport numpy as npimport seaborn as snsimport pingouin as pgimport scipyfrom scipy.stats import chi2from scipy.stats import chi2_contingencyfrom scipy.stats import pearsonr, spearmanrfrom sklearn.preprocessing import StandardScalerfrom sklearn.preprocessing import MinMaxScalerfrom sklearn.model_selection import train_test_splitfrom sklearn.model_selection import GridSearchCVfrom sklearn.ensemble import RandomForestClassifierfrom sklearn.linear_model import LogisticRegressionfrom sklearn.neighbors import KNeighborsClassifierfrom sklearn.svm import SVCfrom sklearn import treefrom sklearn.tree import export_graphvizfrom sklearn.externals.six import StringIOfrom sklearn.linear_model import Perceptronfrom sklearn.neural_network import MLPClassifierfrom sklearn.metrics import  precision_recall_curve, roc_auc_score, confusion_matrix, accuracy_score, recall_score, precision_score, f1_score,auc, roc_curve, plot_confusion_matrix​from category_encoders import BinaryEncoderfrom IPython.display import Imageimport pydotplusimport matplotlib.pyplot as plt%matplotlib inlinecolor = sns.color_palette()seed = 42
Loading and displaying the dataset:>> data = pd.read_csv('lending_club_loan_dataset.csv', low_memory=False)>> data.head()
Main stats of numeric attributes:>> data.describe().round(3)
The dataset has 2000 observations and 15 variables including the target, divided into 11 numeric and 4 categoric features.
There are variables with missing values: ‘home_ownership’ with 7.46%, ‘dti’ with 0.77%, and ‘last_major_derog_none’ with 97.13%.
From the difference between the mean and the median, and also the distance of maximum values of the variables ‘annual_inc’, ‘revol_util’ and ‘total_rec_late_fee’, it seems there are some outliers.
Main stats of categoric attributes:>> data.describe(include=[np.object])
# Checking data balance/proportionloan = data.bad_loan.value_counts().to_frame().rename(columns={"bad_loan":"absolute"})loan["percent"] = (loan.apply(lambda x: x/x.sum()*100).round(2))display(loan)---​# pie chartdata.bad_loan.value_counts().plot(kind='pie', subplots=True, autopct='%1.2f%%', explode= (0.05, 0.05), startangle=80, legend=True, fontsize=12, figsize=(14,6), textprops={'color':"black"})plt.legend(["0: paid loan","1: not paid loan"]);
Unbalanced data: target has 80% of default results (value 1) against 20% of loans that ended up by been paid/ non-default (value 0).
Type of variables:>> data.dtypes.sort_values(ascending=True)
id                         int64short_emp                  int64emp_length_num             int64last_delinq_none           int64bad_loan                   int64annual_inc               float64dti                      float64last_major_derog_none    float64revol_util               float64total_rec_late_fee       float64od_ratio                 float64grade                     objecthome_ownership            objectpurpose                   objectterm                      objectdtype: object
Couting variables by type:>> data.dtypes.value_counts()
float64    6int64      5object     4dtype: int64
Checking for missing values:
nulval = data.isnull().sum().to_frame().rename(columns={0:"absolute"})nulval["percent"] = (nulval.apply(lambda x: x/x.sum())*100).round(2)nulval
Describing all the features in the dataset using and abusing graphics. Start by defining a few functions for every chart: boxplot, histograms, bar and pie charts, scatterplots, pivot charts, as well as a statistic descriptions.
# General statisticsdef stats(x):    print(f"Variable: {x}")    print(f"Type of variable: {data[x].dtype}")    print(f"Total observations: {data[x].shape[0]}")    detect_null_val = data[x].isnull().values.any()    if detect_null_val:        print(f"Missing values: {data[x].isnull().sum()} ({(data[x].isnull().sum() / data[x].isnull().shape[0] *100).round(2)}%)")    else:        print(f"Missing values? {data[x].isnull().values.any()}")    print(f"Unique values: {data[x].nunique()}")    if data[x].dtype != "O":        print(f"Min: {int(data[x].min())}")        print(f"25%: {int(data[x].quantile(q=[.25]).iloc[-1])}")        print(f"Median: {int(data[x].median())}")        print(f"75%: {int(data[x].quantile(q=[.75]).iloc[-1])}")        print(f"Max: {int(data[x].max())}")        print(f"Mean: {data[x].mean()}")        print(f"Std dev: {data[x].std()}")        print(f"Variance: {data[x].var()}")        print(f"Skewness: {scipy.stats.skew(data[x])}")        print(f"Kurtosis: {scipy.stats.kurtosis(data[x])}")        print("")                # Percentiles 1%, 5%, 95% and 99%print("Percentiles 1%, 5%, 95%, 99%")        display(data[x].quantile(q=[.01, .05, .95, .99]))        print("")    else:        print(f"List of unique values: {data[x].unique()}")---# Variable vs. target chartdef target(x):    short_0 = data[data.bad_loan == 0].loc[:,x]    short_1 = data[data.bad_loan == 1].loc[:,x]        a = np.array(short_0)    b = np.array(short_1)        np.warnings.filterwarnings('ignore')        plt.hist(a, bins=40, density=True, color="g", alpha = 0.6, label='Not-default', align="left")    plt.hist(b, bins=40, density=True, color="r", alpha = 0.6, label='Default', align="right")plt.legend(loc='upper right')    plt.title(x, fontsize=10, loc="right")    plt.xlabel('Relative frequency')    plt.ylabel('Absolute frequency')    plt.show()---​# Boxplot + Hist chartdef boxhist(x):    variable = data[x]    np.array(variable).mean()    np.median(variable)f, (ax_box, ax_hist) = plt.subplots(2, sharex=True, gridspec_kw= {"height_ratios": (0.5, 2)})    mean=np.array(variable).mean()    median=np.median(variable)sns.boxplot(variable, ax=ax_box)    ax_box.axvline(mean, color='r', linestyle='--')    ax_box.axvline(median, color='g', linestyle='-')sns.distplot(variable, ax=ax_hist)    ax_hist.axvline(mean, color='r', linestyle='--')    ax_hist.axvline(median, color='g', linestyle='-')plt.title(x, fontsize=10, loc="right")    plt.legend({'Mean':mean,'Median':median})    ax_box.set(xlabel='')    plt.show()---# Histogramdef hist(x):    plt.hist(data[x], bins=25)    plt.title(x, fontsize=10, loc="right")    plt.xlabel('Relative frequency')    plt.ylabel('Absolute frequency')    plt.show()---# Pie chartdef pie(x):    data[x].value_counts(dropna=False).plot(kind='pie', figsize=(6,5), fontsize=10, autopct='%1.1f%%', startangle=0, legend=True, textprops={'color':"white", 'weight':'bold'});# Number of observations by classobs = data[x].value_counts(dropna=False)o = pd.DataFrame(obs)o.rename(columns={x:"Freq abs"}, inplace=True)o_pc = (data[x].value_counts(normalize=True) * 100).round(2)obs_pc = pd.DataFrame(o_pc)obs_pc.rename(columns={x:"percent %"}, inplace=True)obs = pd.concat([o,obs_pc], axis=1)display(obs)---​# Variable vs. target chartdef target(x):    short_0 = data[data.bad_loan == 0].loc[:,x]    short_1 = data[data.bad_loan == 1].loc[:,x]        a = np.array(short_0)    b = np.array(short_1)        np.warnings.filterwarnings('ignore')        plt.hist(a, bins=40, density=True, color="g", alpha = 0.6, label='Not-default', align="left")    plt.hist(b, bins=40, density=True, color="r", alpha = 0.6, label='Default', align="right")plt.legend(loc='upper right')    plt.title(x, fontsize=10, loc="right")    plt.xlabel('Relative frequency')    plt.ylabel('Absolute frequency')    plt.show()---​# Boxplot + Hist chartdef boxhist(x):    variable = data[x]    np.array(variable).mean()    np.median(variable)f, (ax_box, ax_hist) = plt.subplots(2, sharex=True, gridspec_kw= {"height_ratios": (0.5, 2)})    mean=np.array(variable).mean()    median=np.median(variable)sns.boxplot(variable, ax=ax_box)    ax_box.axvline(mean, color='r', linestyle='--')    ax_box.axvline(median, color='g', linestyle='-')sns.distplot(variable, ax=ax_hist)    ax_hist.axvline(mean, color='r', linestyle='--')    ax_hist.axvline(median, color='g', linestyle='-')plt.title(x, fontsize=10, loc="right")    plt.legend({'Mean':mean,'Median':median})    ax_box.set(xlabel='')    plt.show()----# Bar chartdef bar(x):    ax = data[x].value_counts().plot(kind="bar", figsize=(6,5), fontsize=10, color=sns.color_palette("rocket"), table=False)    for p in ax.patches:        ax.annotate("%.2f" % p.get_height(), (p.get_x() + p.get_width() / 2., p.get_height()), ha='center', va='center', xytext=(0, 5), textcoords='offset points')plt.xlabel(x, fontsize=10)    plt.xticks(rotation=0, horizontalalignment="center")    plt.ylabel("Absolute values", fontsize=10)    plt.title(x, fontsize=10, loc="right")    ---    # Barh chartdef barh(x):    data[x].value_counts().plot(kind="barh", figsize=(6,5), fontsize=10, color=sns.color_palette("rocket"), table=False)    plt.xlabel("Absolute values", fontsize=10)    plt.xticks(rotation=0, horizontalalignment="center")    plt.ylabel(x, fontsize=10)    plt.title(x, fontsize=10, loc="right")---    # Pivot_table_meandef pivot_mean(a, b, c):    type_pivot_mean = data.pivot_table(        columns=a,        index=b,        values=c, aggfunc=np.mean)    display(type_pivot_mean)# Display pivot_table    type_pivot_mean.sort_values(by=[b], ascending=True).plot(kind="bar", title=(b), figsize=(6,4),fontsize = 12);# Pivot_table_sumdef pivot_sum(a, b, c):    type_pivot_sum = data.pivot_table(        columns=a,        index=b,        values=c, aggfunc=np.sum)    display(type_pivot_sum)# Display pivot_table    type_pivot_sum.sort_values(by=[b], ascending=True).plot(kind="bar", title=(b), figsize=(6,4),fontsize = 12);---    # Scatter plotdef scatter(x, y):    targets = data["bad_loan"].unique()for target in targets:        a = data[data["bad_loan"] == target][x]        b = data[data["bad_loan"] == target][y]plt.scatter(a, b, label=f"bad loan: {target}", marker="*")        plt.xlabel(x, fontsize=10)    plt.ylabel(y, fontsize=10)    plt.title("abc", fontsize=10, loc="right")    plt.legend()    plt.show()
Visualization of the numeric distribution:
data.hist(figsize=(10,9), bins=12, ec="b", xlabelsize=8, ylabelsize=8, alpha=0.9, grid=False)plt.tight_layout()plt.show()
Visualization of the categoric distribution:
for col in data.select_dtypes(include=["object"]).columns:    data[col].value_counts().plot(kind="bar", color=sns.color_palette("rocket"))        plt.xlabel("Class", fontsize=10)    plt.xticks(rotation=90, horizontalalignment="center")    plt.ylabel("Count", fontsize=10)    plt.title(col, fontsize=10, loc="right")    plt.show()
It seems there is a typo in the ’36 Months’ class on the variable ‘term’. Let’s fix it by lowering the capitalized characters.
>> data.term = data.term.str.lower()>> data.term.value_counts()36 months    15001 60 months     4999Name: term, dtype: int64
LC assigned loan grade.
stats("grade")Variable: gradeType of variable: objectTotal observations: 20000Missing values? FalseUnique values: 7List of unique values: ['A' 'D' 'E' 'B' 'G' 'C' 'F']---bar("grade")
pivot_sum("home_ownership","grade","id")
When the grade classes decrease, the type of homeownership tends to shift from mortgage to rent. It is on grades B, C, and D that we see the type of own propriety as the highest class.
target("grade")
It is between the upper-grade classes that the highest not-default loans happen.
The self-reported annual income provided by the borrower during registration.
boxhist("annual_inc")
stats("annual_inc")Variable: annual_incType of variable: float64Total observations: 20000Missing values? FalseUnique values: 2566Min: 841225%: 47000Median: 6500075%: 88000Max: 1000000Mean: 73349.57835Std dev: 45198.56725472537Variance: 2042910481.8799326Skewness: 5.275648123592321Kurtosis: 66.72665803201564Percentiles 1%, 5%, 95%, 99%0.01     20519.50.05     30000.00.95    145000.00.99    225000.0Name: annual_inc, dtype: float64---target("annual_inc")
The histogram tells us that the higher the income, the higher is the trend of default.
scatter("annual_inc","dti")
The scatterplot shows a weak and negative association between ‘annual income’ and ‘debt to income ratio’.
data.annual_inc.corr(dti)>> -0.22853314935876534
The correlation value is of -0.23 meaning as the annual_inc decreases, the loans at instance 1 (default/ not paid) increases.
1 when employed for 1 year or less.
hist("short_emp")
The clients that have been employed for one or less years (instance 1) represent 11.25% whereas 88.75% of the clients were employed for more than 1 year.
stats("short_emp")Variable: short_empType of variable: int64Total observations: 20000Missing values? FalseUnique values: 2Min: 025%: 0Median: 075%: 0Max: 1Mean: 0.1125Std dev: 0.3159885163057429Variance: 0.09984874243710473Skewness: 2.4526820936006293Kurtosis: 4.015649452269171Percentiles 1%, 5%, 95%, 99%0.01    0.00.05    0.00.95    1.00.99    1.0Name: short_emp, dtype: float64---target("short_emp")
The segment employed for less than 1 year had loans on default more frequently than the other segment.
Employment length in years. Possible values are between 0 and 10 where 0 means less than one year and 10 means ten or more years.
boxhist("emp_length_num")
stats("emp_length_num")Variable: emp_length_numType of variable: int64Total observations: 20000Missing values? FalseUnique values: 12Min: 025%: 3Median: 775%: 11Max: 11Mean: 6.8214Std dev: 3.7742302898357223Variance: 14.24481428071344Skewness: -0.27964924120655704Kurtosis: -1.3664296257576731Percentiles 1%, 5%, 95%, 99%0.01     0.00.05     1.00.95    11.00.99    11.0Name: emp_length_num, dtype: float64---target("emp_length_num")
With a few exceptions, it’s amongst nonstop employed clients for more than 10 years that the not-default loans occur.
pivot_mean("bad_loan", "purpose", "emp_length_num")
Wedding and vacation are the two purposes in which, on average, the majority of loans ended up not been paid.
Type of home ownership.
stats("home_ownership")Variable: home_ownershipType of variable: objectTotal observations: 20000Missing values: 1491 (7.46%)Unique values: 3List of unique values: ['RENT' 'OWN' 'MORTGAGE' nan]---bar("home_ownership")
pie("home_ownership")
pivot_sum("bad_loan", "home_ownership", "id")
Proportionally, there is no major difference between the type of homeownership and the default loans.
A ratio calculated using the borrower’s total monthly debt payments on the total debt obligations, excluding mortgage and the requested LC loan, divided by the borrower’s self-reported monthly income.
boxhist("dti")
stats("dti")Variable: dtiType of variable: float64Total observations: 20000Missing values: 154Unique values: 3295Min: 025%: 10Median: 1675%: 22Max: 34Mean: 16.58784137861536Std dev: 7.585811951545168Variance: 57.544542964205505Skewness: nanKurtosis: nanPercentiles 1%, 5%, 95%, 99%0.01     1.78000.05     4.65000.95    29.69000.99    33.4355Name: dti, dtype: float64---target("dti")
The distribution of bad loans (default) has, on average, higher ‘dti’ values (debt to income ratio) in comparison with the good loans. The trend is: the higher the effort rate, the more frequently are the loans on default.
pivot_sum("home_ownership", "purpose", "dti")
It is notorious that the main purpose for the requested loans is ‘debt consolidation’ followed by ‘credit card’, for ‘mortage’ and ‘own’ as types of home ownership by more than 6000 and 4000 people respectivelly.
On the other hand, ‘moving’ and ‘wedding’ and the purpose less declared by 19 and 47 in both same segments of type of ownership. For those who live in a rented place, the number of people who requests loans is substancially inferior, proporcionally.
pivot_sum("bad_loan", "grade", "dti")
The trend is when the grade classes decrease, the probability of a default loan increases.
A category provided by the borrower for the loan request.
stats("purpose")Variable: purposeType of variable: objectTotal observations: 20000Missing values? FalseUnique values: 12List of unique values: ['credit_card' 'debt_consolidation' 'medical' 'other' 'home_improvement' 'small_business' 'major_purchase' 'vacation' 'car' 'house' 'moving' 'wedding']---barh("purpose")
pivot_sum("bad_loan", "purpose", "id")
The number of payments on the loan. Values are in months and can be either 36 or 60.
pie("term")
target("term")
Default loans occur inversely and more frequently over a 60 months term period.
pivot_mean("term", "grade", "annual_inc")
On average, the 36 months term is the most common amongst clients with the highest debt-to-income that belong to the lowest grade class.
1 when the borrower had at least one event of delinquency.
target("last_delinq_none")
pie("last_delinq_none")
stats("last_delinq_none")Variable: last_delinq_noneType of variable: int64Total observations: 20000Missing values? FalseUnique values: 2Min: 025%: 0Median: 175%: 1Max: 1Mean: 0.5466Std dev: 0.49783614979391533Variance: 0.24784083204162968Skewness: -0.18721487004502552Kurtosis: -1.9649505924340243Percentiles 1%, 5%, 95%, 99%0.01    0.00.05    0.00.95    1.00.99    1.0Name: last_delinq_none, dtype: float64---pivot_mean("bad_loan","purpose","last_delinq_none")
Loans on default are, on average, the more frequent between loans’ purposes such as vacation and house.
1 borrower had at least 90 days of a bad rating.
bar("last_major_derog_none")
stats("last_major_derog_none")Variable: last_major_derog_noneType of variable: float64Total observations: 20000Missing values: 19426 (97.13%)Unique values: 2Min: 025%: 1Median: 175%: 1Max: 1Mean: 0.759581881533101Std dev: 0.42771012441406686Variance: 0.18293595052629658Skewness: nanKurtosis: nanPercentiles 1%, 5%, 95%, 99%0.01    0.00.05    0.00.95    1.00.99    1.0Name: last_major_derog_none, dtype: float64---target("last_major_derog_none")
Revolving line utilization rate, or the amount of credit the borrower is using relative to all available revolving credit.
stats("revol_util")Variable: revol_utilType of variable: float64Total observations: 20000Missing values? FalseUnique values: 1030Min: 025%: 38Median: 5775%: 73Max: 5010Mean: 55.95814805499972Std dev: 42.117455872216155Variance: 1773.880089148075Skewness: 81.32716395041949Kurtosis: 9569.242123791564Percentiles 1%, 5%, 95%, 99%0.01     2.6990.05    14.5000.95    91.8000.99    97.300Name: revol_util, dtype: float64---scatter("annual_inc", "revol_util")
The lowest the clients’ annual income, the highest the amount of credit the borrower is using relative to all available revolving credit.
boxhist("revol_util")
Late fees received to date.
stats("total_rec_late_fee")Variable: total_rec_late_feeType of variable: float64Total observations: 20000Missing values? FalseUnique values: 166Min: 025%: 0Median: 075%: 0Max: 96Mean: 0.29062163999999996Std dev: 3.1086544166442467Variance: 9.663732282121781Skewness: 14.299156408331024Kurtosis: 262.30322072057277Percentiles 1%, 5%, 95%, 99%0.01     0.00.05     0.00.95     0.00.99    15.0Name: total_rec_late_fee, dtype: float64---target("total_rec_late_fee")
scatter("annual_inc", "total_rec_late_fee")
data.total_rec_late_fee.corr(annual_inc)>> -0.00975830114406848
The customers with the lowest annual income are the ones that have more late fees, especially the highest and heavy ones.
pivot_mean("bad_loan", "purpose", "total_rec_late_fee")
The late fees occur in a higher frequency amongst loan purposes such as a house, small business, or vacation. On the other hand, wedding and car are the credit purposes with the lowest late fees execution.
Overdraft ratio.
boxhist("od_ratio")
stats("od_ratio")Variable: od_ratioType of variable: float64Total observations: 20000Missing values? FalseUnique values: 20000Min: 025%: 0Median: 075%: 0Max: 0Mean: 0.5044303048872487Std dev: 0.2877201586666063Variance: 0.0827828897031371Skewness: -0.02052095981509419Kurtosis: -1.1914529752985776Percentiles 1%, 5%, 95%, 99%0.01    0.0098870.05    0.0514950.95    0.9516160.99    0.990142Name: od_ratio, dtype: float64---scatter("annual_inc", "od_ratio")
There is a higher overdraft rate between clients with the lowest annual income, the same applicants involved with the most frequent defaulted loans.
pivot_sum("bad_loan", "term", "od_ratio")
Proportionally, overdraft ratios are higher on a 60-month term amongst defaulted loans.
1 when a loan was not paid.
stats("bad_loan")Variable: bad_loanType of variable: int64Total observations: 20000Missing values? FalseUnique values: 2Min: 025%: 0Median: 075%: 0Max: 1Mean: 0.2Std dev: 0.40001000037498174Variance: 0.16000800039999288Skewness: 1.4999999999999996Kurtosis: 0.24999999999999956Percentiles 1%, 5%, 95%, 99%0.01    0.00.05    0.00.95    1.00.99    1.0Name: bad_loan, dtype: float64---bar("bad_loan")
mask = np.triu(data.corr(), 1)plt.figure(figsize=(19, 9))sns.heatmap(data.corr(), annot=True, vmax=1, vmin=-1, square=True, cmap='BrBG', mask=mask);
The heatmap shows there are some positive and negative correlations amongst variables.
Let’s now find out which numerical features are the most correlated with the target.
bad_loan_c = pg.pairwise_corr(data, columns=['bad_loan'], method='pearson').loc[:,['X','Y','r']]bad_loan_c.sort_values(by=['r'], ascending=False)
The variable that is most correlated with the target is ‘dti’ with a weak and positive correlation of 0.141884.
data_spear = data.copy()data_spear.drop(["bad_loan"], axis=1, inplace=True)---spearman_rank = pg.pairwise_corr(data_spear, method='spearman').loc[:,['X','Y','r']]pos = spearman_rank.sort_values(kind="quicksort", by=['r'], ascending=False).iloc[:5,:]neg = spearman_rank.sort_values(kind="quicksort", by=['r'], ascending=False).iloc[-5:,:]con = pd.concat([pos,neg], axis=0)display(con.reset_index(drop=True))
mask = np.triu(data_spear.corr(method='spearman'), 1)plt.figure(figsize=(19, 9))sns.heatmap(data_spear.corr(method='spearman'), annot=True, vmax=1, vmin=-1, square=True, cmap='BrBG', mask=mask);
By plotting a heatmap — spearman method — it’s easy to understand that the variables ‘last_major_derog_none’ and ‘last_delinq_none’ are the two most correlated features with the highest monotonic relationship (60%). Nevertheless, ‘last_major_derog_none’ has 19426 missing values (97%), which is too much.
In this scenario, the column ‘last_major_derog_none’ is to be dropped as well as all the nan values from the dataset. Thus, the next most correlated variables would be ‘emp_length_num’ and ‘short_emp’, with a Spearman value of -55%.
Let’s examine the data and check for any outliers. Starting by selecting and filtering numeric and categoric data.
data_ca = data.select_dtypes(exclude=["int64","float64"]).copy()data_nu = data.select_dtypes(exclude=["object","category"]).copy()
# Boxplot: Visualizing the numeric data dispersion
fig, axs = plt.subplots(ncols=3, nrows=4, figsize=(16, 8))index = 0axs = axs.flatten()for k,v in data_nu.items():    sns.boxplot(y=k, data=data_nu, ax=axs[index], orient="h")    index += 1    plt.tight_layout(pad=0.4, w_pad=0.5, h_pad=5.0)
display(data.describe().loc[["mean","50%","std"]].loc[:,["annual_inc","revol_util","total_rec_late_fee"]])
Definitely, there are some outliers in the variables ‘annual_inc’, ‘revol_util’ and ‘total_rec_late_fee’. Let’s detect and solve them.
print(data.annual_inc.describe())count      20000.000000mean       73349.578350std        45198.567255min         8412.00000025%        47000.00000050%        65000.00000075%        88000.000000max      1000000.000000Name: annual_inc, dtype: float64---boxhist("annual_inc")
The graph and the boxplot suggests this variable has too many data points outside the upper quartile. Outliers are plotted as those individual points beyond the boxplot whiskers. The method used here is the IQR score.
# Dealing with the outliers through IQR score methodQ1 = data['annual_inc'].quantile(0.25)Q3 = data['annual_inc'].quantile(0.75)IQR = Q3 - Q1data['annual_inc'] = data.annual_inc[~((data.annual_inc < (Q1 - 1.5 * IQR)) |(data.annual_inc > (Q3 + 1.5 * IQR)))]---print(data.annual_inc.describe())count     19074.000000mean      66792.117857std       27241.646991min        8412.00000025%       46000.00000050%       62000.00000075%       84000.000000max      149000.000000Name: annual_inc, dtype: float64---boxhist("annual_inc")
print(int(data_nu.annual_inc.describe()[0]) - int(data.annual_inc.describe()[0]),"outliers were removed with this operation.")>> 926 outliers were removed with this operation.
print(data.revol_util.describe())count    20000.000000mean        55.958148std         42.117456min          0.00000025%         38.80000050%         57.10000075%         73.900000max       5010.000000Name: revol_util, dtype: float64---boxhist("revol_util")
The graph and the boxplot analisys suggests that this variable has one data point far away from the upper quartile.
Outliers are plotted as those individual points beyond the boxplot whiskers but that doesn’t mean that every single data point outside the lower and upper quartiles are indeed outliers.
Better check it out and possibly remove that single outlier.
# Dealing with the 5010.0 outliervalue = data.revol_util.quantile([.99999])p = value.iloc[0]data = data[data["revol_util"] < p]---print(data['revol_util'].describe())​count    19999.000000mean        55.710434std         23.380722min          0.00000025%         38.80000050%         57.10000075%         73.900000max        128.100000Name: revol_util, dtype: float64---boxhist("revol_util")
print(int(data_nu.revol_util.describe()[0]) - int(data.revol_util.describe()[0]),"outlier was removed with this operation.")>> 1 outlier was removed with this operation.
Visualizing the data dispersion:
sns.boxplot(x=data['total_rec_late_fee'],data=data)plt.xlabel('total_rec_late_fee', fontsize=10)plt.show()
Removing outlier:
value = data.total_rec_late_fee.quantile([.989])p = value.iloc[0]data = data[data["total_rec_late_fee"] < p]
Checking results:
sns.boxplot(x=data['total_rec_late_fee'],data=data)plt.xlabel('total_rec_late_fee', fontsize=10)plt.show()
Although there is a significant number of data point distant from the upper quartile, I believe this is the case that those are not outliers, their values are significant for the target classification. Not removing any others.
for col in data[["annual_inc", "total_rec_late_fee", "revol_util"]].columns:    sns.boxplot(data[col])    plt.show()
Time to detect and eliminate them.
for column in data.columns:    if data[column].isna().sum() != 0:        missing = data[column].isna().sum()        portion = (missing / data.shape[0]) * 100        print(f"'{column}': number of missing values '{missing}' ---> '{portion:.3f}%'")>> 'annual_inc': number of missing values '915' ---> '4.626%'>> 'home_ownership': number of missing values '1476' ---> '7.462%'>> 'dti': number of missing values '152' ---> '0.768%'>> 'last_major_derog_none': number of missing values '19208' ---> '97.113%'
data.annual_inc.value_counts(dropna=False)NaN         915 <---60000.0     77150000.0     72965000.0     60770000.0     599           ... 109097.0      188621.0       150455.0       118300.0       196241.0       1Name: annual_inc, Length: 2349, dtype: int64---boxhist("annual_inc")
Strategy: Replacing missing values with the mean (average).
data["annual_inc"] = data.annual_inc.fillna(data.annual_inc.mean())print(f"Fillna done. Anomalies detected: {data.annual_inc.isnull().values.any()}")>> Fillna done. Anomalies detected: False
data.home_ownership.value_counts(dropna=False)MORTGAGE    9744RENT        6959OWN         1600NaN         1476 <---Name: home_ownership, dtype: int64---bar("home_ownership")
Strategy: Mode imputation (replacing NaN by most frequent value: Mortage).
data["home_ownership"] = data.home_ownership.fillna(data.home_ownership.value_counts().index[0])print(f"Imputation done. Missing values: {data.home_ownership.isnull().sum()}")>> Imputation done. Missing values: 0
data.dti.value_counts(dropna=False)NaN      152 <---19.64     2015.87     209.60      1913.49     19        ... 1.61       129.00      129.19      131.78      13.26       1Name: dti, Length: 3286, dtype: int64---boxhist("dti")
Strategy: Replacing missing values with the mean (average).
data["dti"] = data.dti.fillna(data.dti.mean())print(f"Fillna done. Missing values: {data.dti.isnull().values.any()}")>> Fillna done. Missing values: False
abs_mv = data.last_major_derog_none.value_counts(dropna=False)pc_mv = data.last_major_derog_none.value_counts(dropna=False, normalize=True) * 100pc_mv_df = pd.DataFrame(pc_mv)pc_mv_df.rename(columns={"last_major_derog_none":"Percent %"}, inplace=True)abs_pc = pd.concat([abs_mv,pc_mv_df], axis=1)abs_pc
Strategy: Drop ‘last_major_derog_none’ numerical variable (too many anomalies).
data.drop("last_major_derog_none", axis=1, inplace=True)print(f"All missing values are solved in the entire dataset: {data.notnull().values.any()}")>> All missing values are solved in the entire dataset: True
>> data.info()
Strategy: Drop ‘id’ numerical variable (irrelevant feature).
data.drop("id", axis=1, inplace=True)---data.shape>> (19779, 13)
Selecting numeric variables only:>> data_nu = data.select_dtypes(exclude=["object","category"]).copy()
Creating subsets:>> Xnum = data_nu.drop(["bad_loan"], axis= "columns") >> ynum = data_nu.bad_loan
# Identifying the predictive features using the Pearson Correlation p-valuepd.DataFrame(    [scipy.stats.pearsonr(Xnum[col],     ynum) for col in Xnum.columns],     columns=["Pearson Corr.", "p-value"],     index=Xnum.columns,).round(4)
Strategy: Drop ‘od_ratio’ (p-value > 0.05) → low information to the target, and keep all the others.
Selecting categoric variables only:
Xcat = data.select_dtypes(exclude=['int64','float64']).copy()
Creating subsets:
Xcat['target'] = data.bad_loanXcat.dropna(how="any", inplace=True)ycat = Xcat.targetXcat.drop("target", axis=1, inplace=True)
Chi-square test for independence:
for col in Xcat.columns:    table = pd.crosstab(Xcat[col], ycat)    print()    display(table)    _, pval, _, expected_table = scipy.stats.chi2_contingency(table)    print(f"p-value: {pval:.25f}")
Strategy: Keep all features (p-value < 0.05). The categorical variables have predictive power.
Let’s continue by encoding and transforming the categorical variables into numeric ones.
The feature ‘grade’ is a scale which means i’ll be mapping it to numbers. On the other hand, concerning the variables ‘term’, ‘home_ownership’ and ‘purpose’, we need to inspect and decide which procedure (OHE or Binary Encoding) is the best option.
>> data["grade"] = data.grade.map({"A":7, "B":6, "C":5, "D":4, "E":3, "F":2, "G":1})
One Hot Encoding and Binary Encoding will be both displayed so we can chose the best to apply.
df_term = data.termdf_home = data.home_ownershipdf_purp = data.purpose#termt_ohe = pd.get_dummies(df_term)bin_enc_term = BinaryEncoder()t_bin = bin_enc_term.fit_transform(df_term)#home_ownsershiph_ohe = pd.get_dummies(df_home)bin_enc_home = BinaryEncoder()h_bin = bin_enc_home.fit_transform(df_home)#purposep_ohe = pd.get_dummies(df_purp)bin_enc_purp = BinaryEncoder()p_bin = bin_enc_purp.fit_transform(df_purp)>> The results are:** COLUMNS OHE **term: 2 <--- besthome: 3 <--- bestpurp: 12** COLUMNS BINARY **term: 2home: 3purp: 5 <--- best
>> data = pd.get_dummies(data, columns=["term","home_ownership"])
>> bin_enc_purp = BinaryEncoder() >> data_bin = bin_enc_purp.fit_transform(data.purpose)
# Concatenating both datasetsdf = pd.concat([data,data_bin],axis=1)# Dropping 'purpose'df.drop(["purpose"], axis=1, inplace=True)# Lowering upper charactersdf.columns = [x.lower() for x in df.columns]# printing 5 first rowsdf.head()
At this point, we are ready to test and train some models!
We’re dealing with a supervised binary problem using classification techniques.
Given that we have unbalanced data, we’ll use AUC ROC as the best metric to evaluate the performance of the following models.
Let’s define a function of the AUC to plot and display the threshold between the true positive rate (TPR) and false positive rate (FPR).
# ROC Curve: Area Under the Curvedef auc_roc_plot(y_test, y_preds):    fpr, tpr, thresholds = roc_curve(y_test,y_preds)    roc_auc = auc(fpr, tpr)    print(roc_auc)    plt.title('Receiver Operating Characteristic')    plt.plot(fpr, tpr, 'b', label = 'AUC = %0.2f' % roc_auc)    plt.legend(loc = 'lower right')    plt.plot([0, 1], [0, 1],'r--'    plt.xlim([0, 1])    plt.ylim([0, 1])    plt.ylabel('True Positive Rate'    plt.xlabel('False Positive Rate')    plt.show()
# Making a copy of the datasetdf_lr = df.copy()---# Dividing the dataset in train (80%) and test (20%)train_set_lr, test_set_lr = train_test_split(df_lr, test_size = 0.2, random_state = seed)X_train_lr = train_set_lr.drop(['bad_loan'], axis = 1)y_train_lr = train_set_lr['bad_loan']X_test_lr = test_set_lr.drop(['bad_loan'], axis = 1)y_test_lr = test_set_lr['bad_loan']---# Normalizing the train and test datascaler_lr = MinMaxScaler()features_names = X_train_lr.columnsX_train_lr = scaler_lr.fit_transform(X_train_lr)X_train_lr = pd.DataFrame(X_train_lr, columns = features_names)X_test_lr = scaler_lr.transform(X_test_lr)X_test_lr = pd.DataFrame(X_test_lr, columns = features_names)---%%timelr = LogisticRegression(max_iter = 1000, solver = 'lbfgs', random_state = seed, class_weight = 'balanced' )parameters = {'C':[0.001, 0.01, 0.1, 1, 10, 100]}clf_lr = GridSearchCV(lr, parameters, cv = 5).fit(X_train_lr, y_train_lr)>>> CPU times: user 10.3 s, sys: 449 ms, total: 10.8 sWall time: 3.21 s---clf_lr>>> GridSearchCV(cv=5, error_score=nan, estimator=LogisticRegression(C=1.0, class_weight='balanced', dual=False, fit_intercept=True, intercept_scaling=1, l1_ratio=None, max_iter=1000, multi_class='auto', n_jobs=None, penalty='l2', random_state=42, solver='lbfgs', tol=0.0001, verbose=0, warm_start=False), iid='deprecated', n_jobs=None, param_grid={'C': [0.001, 0.01, 0.1, 1, 10, 100]}, pre_dispatch='2*n_jobs', refit=True, return_train_score=False, scoring=None, verbose=0)clf_lr.best_estimator_>>> LogisticRegression(C=0.1, class_weight='balanced', dual=False, fit_intercept=True, intercept_scaling=1, l1_ratio=None, max_iter=1000, multi_class='auto', n_jobs=None, penalty='l2', random_state=42, solver='lbfgs', tol=0.0001, verbose=0, warm_start=False)---y_preds_lr = clf_lr.predict_proba(X_test_lr)[:,1]---auc_roc_plot(y_test_lr, y_preds_lr)>> 0.7074872114784778
# Confusion Matrix displayplot_confusion_matrix(clf_lr, X_test_lr, y_test_lr, values_format=".4g", cmap="Blues");---# Creating assignments for Final Resultstn, fp, fn, tp = confusion_matrix(y_test_lr == 1, y_preds_lr > 0.5).ravel()tn_lr = tnfp_lr = fpfn_lr = fntp_lr = tp
# Making a copy of the datasetdf_knn = df.copy()---# Dividing the dataset in train (80%) and test (20%)train_set_knn, test_set_knn = train_test_split(df_knn, test_size = 0.2, random_state = seed)​X_train_knn = train_set_knn.drop(['bad_loan'], axis = 1)y_train_knn = train_set_knn['bad_loan']X_test_knn = test_set_knn.drop(['bad_loan'], axis = 1)y_test_knn = test_set_knn['bad_loan']---# Normalizing train and test datascaler_knn = MinMaxScaler()features_names = X_train_knn.columnsX_train_knn = scaler_knn.fit_transform(X_train_knn)X_train_knn = pd.DataFrame(X_train_knn, columns = features_names)X_test_knn = scaler_knn.transform(X_test_knn)X_test_knn = pd.DataFrame(X_test_knn, columns = features_names)---%%timefor k in range(1, 200, 5):    k = k + 1    knn = KNeighborsClassifier(n_neighbors = k).fit(X_train_knn, y_train_knn)    acc = knn.score(X_test_knn, y_test_knn)    print('Accuracy for k =', k, ' is:', acc)Accuracy for k = 2  is: 0.7965116279069767Accuracy for k = 7  is: 0.7944893832153691Accuracy for k = 12  is: 0.8066228513650152Accuracy for k = 17  is: 0.8066228513650152Accuracy for k = 22  is: 0.8088978766430738Accuracy for k = 27  is: 0.8081395348837209Accuracy for k = 32  is: 0.8106673407482305Accuracy for k = 37  is: 0.8094034378159757Accuracy for k = 42  is: 0.8116784630940344Accuracy for k = 47  is: 0.8119312436804853Accuracy for k = 52  is: 0.8109201213346815Accuracy for k = 57  is: 0.8104145601617796Accuracy for k = 62  is: 0.8096562184024267Accuracy for k = 67  is: 0.8101617795753286Accuracy for k = 72  is: 0.8101617795753286Accuracy for k = 77  is: 0.8104145601617796Accuracy for k = 82  is: 0.8109201213346815Accuracy for k = 87  is: 0.8106673407482305Accuracy for k = 92  is: 0.8104145601617796Accuracy for k = 97  is: 0.8106673407482305Accuracy for k = 102  is: 0.8104145601617796Accuracy for k = 107  is: 0.8104145601617796Accuracy for k = 112  is: 0.8104145601617796Accuracy for k = 117  is: 0.8104145601617796Accuracy for k = 122  is: 0.8104145601617796Accuracy for k = 127  is: 0.8106673407482305Accuracy for k = 132  is: 0.8104145601617796Accuracy for k = 137  is: 0.8104145601617796Accuracy for k = 142  is: 0.8101617795753286Accuracy for k = 147  is: 0.8101617795753286Accuracy for k = 152  is: 0.8101617795753286Accuracy for k = 157  is: 0.8104145601617796Accuracy for k = 162  is: 0.8104145601617796Accuracy for k = 167  is: 0.8104145601617796Accuracy for k = 172  is: 0.8104145601617796Accuracy for k = 177  is: 0.8104145601617796Accuracy for k = 182  is: 0.8104145601617796Accuracy for k = 187  is: 0.8104145601617796Accuracy for k = 192  is: 0.8104145601617796Accuracy for k = 197  is: 0.8104145601617796>> CPU times: user 1min 8s, sys: 883 ms, total: 1min 9sWall time: 1min 10s---%%timeknn = KNeighborsClassifier(n_neighbors = 47, weights='uniform').fit(X_train_knn, y_train_knn)y_preds_knn = knn.predict(X_test_knn)>> CPU times: user 1.29 s, sys: 16 ms, total: 1.31 sWall time: 1.31 s---auc_roc_plot(y_test_knn, y_preds_knn)>> 0.6670792264504056
# Confusion Matrix displayplot_confusion_matrix(knn, X_test_knn, y_test_knn, values_format=".4g", cmap="Blues");---​# Creating assignments for Final Resultstn, fp, fn, tp = confusion_matrix(y_test_knn == 1, y_preds_knn > 0.5).ravel()tn_knn = tnfp_knn = fpfn_knn = fntp_knn = tp
# Making a copy of the datasetdf_svm = df.copy()---# Dividing the dataset in train (80%) and test (20%)train_set_svc, test_set_svc = train_test_split(df_svm, test_size = 0.2, random_state = seed)X_train_svc = train_set_svc.drop(['bad_loan'], axis = 1)y_train_svc = train_set_svc['bad_loan']X_test_svc = test_set_svc.drop(['bad_loan'], axis = 1)y_test_svc = test_set_svc['bad_loan']---# Standardization of train and test datazscore_svc = StandardScaler()features_names = X_train_svc.columnsX_train_svc = zscore_svc.fit_transform(X_train_svc)X_train_svc = pd.DataFrame(X_train_svc, columns = features_names)X_test_svc = zscore_svc.transform(X_test_svc)X_test_svc = pd.DataFrame(X_test_svc, columns = features_names)---%%timesvc = SVC(random_state=seed, class_weight='balanced',probability=True, verbose=True)parameters = {'C':[0.1, 1, 10]}clf_svc = GridSearchCV(svc, parameters, cv = 5).fit(X_train_svc, y_train_svc)>> [LibSVM][LibSVM][LibSVM][LibSVM][LibSVM][LibSVM][LibSVM][LibSVM][LibSVM][LibSVM][LibSVM][LibSVM][LibSVM][LibSVM][LibSVM][LibSVM]CPU times: user 14min 34s, sys: 22.7 s, total: 14min 57sWall time: 15min 2s---%%timey_preds_svc = clf_svc.predict_proba(X_test_svc)[:,1]>> CPU times: user 2.95 s, sys: 17.9 ms, total: 2.97 sWall time: 3 s---auc_roc_plot(y_test_svc, y_preds_svc)>> 0.6754917862341443
# Confusion Matrix displayplot_confusion_matrix(clf_svc, X_test_svc, y_test_svc, values_format=".4g", cmap="Blues");---​# Creating assignments for Final Resultstn, fp, fn, tp = confusion_matrix(y_test_svc == 1, y_preds_svc > 0.5).ravel()tn_svc = tnfp_svc = fpfn_svc = fntp_svc = tp
# Making a copy of the datasetdf_trees = df.copy()---# Dividing the dataset in train (80%) and test (20%)train_set_dt, test_set_dt = train_test_split(df_trees, test_size = 0.2, random_state = seed)X_train_dt = train_set_dt.drop(['bad_loan'], axis = 1)y_train_dt = train_set_dt['bad_loan']X_test_dt = test_set_dt.drop(['bad_loan'], axis = 1)y_test_dt = test_set_dt['bad_loan']---%%timeclf_tree = tree.DecisionTreeClassifier(random_state = seed, max_depth = 10).fit(X_train_dt, y_train_dt)>> CPU times: user 203 ms, sys: 65.8 ms, total: 268 msWall time: 388 ms---clf_tree.score(X_test_dt, y_test_dt)>> 0.788675429726997---# Visualizing variables by importanceimportant_features = pd.DataFrame(data = clf_tree.feature_importances_, index = X_train_dt.columns, columns = ["value"])important_features.sort_values(by = "value", ascending = False)
y_preds_dt = clf_tree.predict_proba(X_test_dt)[:,1]---auc_roc_plot(y_test_dt, y_preds_dt)>> 0.6295855687253067
# Confusion Matrix displayplot_confusion_matrix(clf_tree, X_test_dt, y_test_dt, values_format=".4g", cmap="Blues");---# Creating assignments Final Resultstn, fp, fn, tp = confusion_matrix(y_test_dt == 1, y_preds_dt > 0.5).ravel()tn_dt = tnfp_dt = fpfn_dt = fntp_dt = tp
# Making a copy of the datasetdf_rf = df.copy()---# Dividing the dataset in train (80%) and test (20%)train_set_rf, test_set_rf = train_test_split(df_rf, test_size = 0.2, random_state = seed)X_train_rf = train_set_rf.drop(['bad_loan'], axis = 1)y_train_rf = train_set_rf['bad_loan']X_test_rf = test_set_rf.drop(['bad_loan'], axis = 1)y_test_rf = test_set_rf['bad_loan']---%%timerf = RandomForestClassifier(random_state = seed, class_weight = None).fit(X_train_rf, y_train_rf)parameters = {'n_estimators':[10, 100, 300, 1000]}clf_rf = GridSearchCV(rf, parameters, cv = 5).fit(X_train_rf, y_train_rf)>> CPU times: user 2min 11s, sys: 3.33 s, total: 2min 14sWall time: 2min 15s---y_preds_rf = clf_rf.predict_proba(X_test_rf)[:,1]---auc_roc_plot(y_test_rf, y_preds_rf)>> 0.6735905593678521
# Confusion Matrxi displayplot_confusion_matrix(clf_rf, X_test_rf, y_test_rf, values_format=".4g", cmap="Blues");---​# Creating assignments for Final Resultstn, fp, fn, tp = confusion_matrix(y_test_rf == 1, y_preds_rf > 0.5).ravel()tn_rf = tnfp_rf = fpfn_rf = fntp_rf = tp
# Making a copy of the datasetdf_nn = df.copy()---# Dividing the dataset in train (80%) and test (20%)train_set_nn, test_set_nn = train_test_split(df_nn, test_size = 0.2, random_state = seed)X_train_nn = train_set_nn.drop(['bad_loan'], axis = 1)y_train_nn = train_set_nn['bad_loan']X_test_nn = test_set_nn.drop(['bad_loan'], axis = 1)y_test_nn = test_set_nn['bad_loan']---# Normalization of the train and test datascaler_nn = MinMaxScaler()features_names = X_train_nn.columnsX_train_nn = scaler_nn.fit_transform(X_train_nn)X_train_nn = pd.DataFrame(X_train_nn, columns = features_names)X_test_nn = scaler_nn.transform(X_test_nn)X_test_nn = pd.DataFrame(X_test_nn, columns = features_names)---%%timemlp_nn = MLPClassifier(solver = 'adam', random_state = seed, max_iter = 1000 )parameters = {'hidden_layer_sizes': [(20,), (20,10), (20, 10, 2)], 'learning_rate_init':[0.0001, 0.001, 0.01, 0.1]}clf_nn = GridSearchCV(mlp_nn, parameters, cv = 5).fit(X_train_nn, y_train_nn)>> CPU times: user 25min 41s, sys: 41.4 s, total: 26min 22sWall time: 6min 53s---y_preds_nn = clf_nn.predict_proba(X_test_nn)[:,1]---auc_roc_plot(y_test_nn, y_preds_nn)>> 0.7081023081721772
# Confusion Matrix displayplot_confusion_matrix(clf_nn, X_test_nn, y_test_nn, values_format=".4g", cmap="Blues");​---# Creating assignments for Final Resultstn, fp, fn, tp = confusion_matrix(y_test_nn == 1, y_preds_nn > 0.5).ravel()tn_nn = tnfp_nn = fpfn_nn = fntp_nn = tp
# Creating performance tableresults_1 = {'Classifier': ['AUC ROC (%)','TN (%)','FP (%)','FN (%)','TP (%)'],'Logistic Regression (LR)': [aucroclr, (tn_lr/3956*100).round(2), (fp_lr/3956*100).round(2), (fn_lr/3956*100).round(2), (tp_lr/3956*100).round(2)],'K Nearest Neighbour (KNN)': [aucrocknn, (tn_knn/3956*100).round(2),(fp_knn/3956*100).round(2), (fn_knn/3956*100).round(2),(tp_nn/3956*100).round(2)],'Support Vector Machine (SVC)': [aucrocsvc, (tn_svc/3956*100).round(2),(fp_svc/3956*100).round(2), (fn_svc/3956*100).round(2),(tp_svc/3956*100).round(2)],'Decision Trees (DT)': [aucrocdt, (tn_dt/3956*100).round(2), (fp_dt/3956*100).round(2), (fn_dt/3956*100).round(2),(tp_dt/3956*100).round(2)],'Random Forest (RF)': [aucrocrf, (tn_rf/3956*100).round(2), (fp_rf/3956*100).round(2), (fn_rf/3956*100).round(2),(tp_rf/3956*100).round(2)],'Neural Networks (NN)': [aucrocnn, (tn_nn/3956*100).round(2), (fp_nn/3956*100).round(2),(fn_nn/3956*100).round(2),(tp_nn/3956*100).round(2)]}df1 = pd.DataFrame(results_1, columns = ['Classifier', 'Logistic Regression (LR)', 'K Nearest Neighbour (KNN)', 'Support Vector Machine (SVC)', 'Decision Trees (DT)', 'Random Forest (RF)', 'Neural Networks (NN)'])df1.set_index("Classifier", inplace=True)results = df1.Tresults
​# Creating table for graphic visualizationresults_2 = {'Classifier': ['ROC AUC'], 'Logistic Regression (LR)': [aucroclr], 'K Nearest Neighbour (KNN)': [aucrocknn], 'Support Vector Machine (SVC)': [aucrocsvc], 'Decision Trees (DT)': [aucrocdt], 'Random Forest (RF)': [aucrocrf], 'Neural Networks (NN)': [aucrocnn]}df2 = pd.DataFrame(results_2, columns = ['Classifier', 'Logistic Regression (LR)', 'K Nearest Neighbour (KNN)', 'Support Vector Machine (SVC)', 'Decision Trees (DT)', 'Random Forest (RF)', 'Neural Networks (NN)'])df2.set_index("Classifier", inplace=True)results_2 = df2---# Display tHe graphax = results_2.plot(kind="bar", title=("Evaluating models' performance"), figsize=(12,8) ,fontsize=10, grid=True)for p in ax.patches:    ax.annotate("%.2f" % p.get_height(), (p.get_x() + p.get_width() / 2., p.get_height()), ha='center', va='center', xytext=(0, 5), textcoords='offset points')plt.legend(fontsize=8.5, loc="upper right")plt.xlabel('')plt.xticks(rotation='horizontal')plt.ylabel('Relative frequency (%)')plt.show()
Best model: Support Vector Machine - Classifier (SVC): 75.21%.
The rule of thumb is very straightforward: the higher the value of the ROC AUC metric, the better. If a random model would show 0.5, a perfect model would achieve 1.0.
The academic scoring system stands as follows:
.9 -  1 = excellent  (A).8 - .9 = good       (B).7 - .8 = reasonable (C).6 - .7 = weak       (D).5 - .6 = terrible   (F)
The ratio between TPR and FPR determined by a threshold over which results in a positive instance puts the chosen model (SVC) at a reasonable level ( C ) with a ROC AUC score of 75.21%.
You can download the data set here.
Some other articles:
goncaloggomes.medium.com
towardsdatascience.com
goncaloggomes.medium.com
How to get in contact with me:
Linkedin
Twitter
Comment below
👏🏻 if you find worth of it.
✅ Thanks for reading!