Loan Approval Using Machine Learning Algorithm
By: Sarit Maitra
Tags: Bias Variance Tradeoff, Classification Algorithms, Machine Learning Ai

Machine learning & classification algorithms

Loan Approval Using Machine Learning Algorithm

Classification algorithms to determine application outcome

Image by author

Loans in terms of financial pay outs is an important aspect of banking business system. Several loan applications are scanned based on certain inputs to validate the eligibility for loan. Here our use-case is that, we want to automate the loan eligibility process (real time) based on customer detail obtained during loan application. This will lead to improved service and customer satisfaction.

Let us load the available data to check the information it contain.

Loading training data:

Here that the dependent / target variable is the Loan_Status and we need to develop a model using the rest of the features to predict the target variable.

Data Pre-processing:

Bar plot to visualize application outcome:

ax = (df['Loan_Status'].value_counts()*100.0 /len(df)).plot(kind='bar', stacked = True, rot = 0)

ax.yaxis.set_major_formatter(mtick.PercentFormatter())

ax.set_ylabel('% Loan application')

ax.set_xlabel('Status')

ax.set_ylabel('% Customers')

ax.set_title('Application Distribution')

totals = [] # finding the values and append to list

for i in ax.patches:

totals.append(i.get_width())

total = sum(totals)

for i in ax.patches:

ax.text(i.get_x()+.15, i.get_height()-3.5, \ str(round((i.get_height()/total), 1))+'%', color='white', weight = 'bold')

Missing value and outlier treatment:

numerical variables: imputation using mean or median

categorical variables: imputation using mode

There are very less missing values in Gender, Married, Dependents,

Credit_History and Self_Employed features so we fill them using the mode of the features.

If an independent variable in our dataset has huge amount of missing data e.g. 80% missing values in it, then we would drop the variable from the dataset.

# replace missing values with the mode

df['Gender'].fillna(df['Gender'].mode()[0], inplace=True)

df['Married'].fillna(df['Married'].mode()[0], inplace=True)

df['Dependents'].fillna(df['Dependents'].mode()[0], inplace=True)

df['Self_Employed'].fillna(df['Self_Employed'].mode()[0], inplace=True)

df['Credit_History'].fillna(df['Credit_History'].mode()[0], inplace=True)

df['Loan_Amount_Term'].fillna(df['Loan_Amount_Term'].mode()[0], inplace=True) # replace missing values with the median value due to outliers

df['LoanAmount'].fillna(df['LoanAmount'].median(), inplace=True) # replacing Y and N in Loan_Status variable with 1 and 0

df['Loan_Status'].replace('N', 0, inplace=True)

df['Loan_Status'].replace('Y', 1, inplace=True)

Loading test data:

Test data processing:

# replacing 3+ in Dependents variable with 3

df1['Dependents'].replace('3+', 3, inplace=True) # replace missing values in Test set with mode/median from Training set

df1['Gender'].fillna(df['Gender'].mode()[0], inplace=True)

df1['Dependents'].fillna(df['Dependents'].mode()[0], inplace=True)

df1['Self_Employed'].fillna(df['Self_Employed'].mode()[0], inplace=True)

df1['Credit_History'].fillna(df['Credit_History'].mode()[0], inplace=True)

df1['Loan_Amount_Term'].fillna(df['Loan_Amount_Term'].mode()[0], inplace=True)

df1['LoanAmount'].fillna(df['LoanAmount'].median(), inplace=True)

Model Development and Evaluation:

# drop Loan_ID

train = df.drop('Loan_ID', axis=1) # train

test = df1.drop('Loan_ID', axis=1) # test # drop "Loan_Status" and assign it to target variable

X = train.drop('Loan_Status', 1)

y = train.Loan_Status # adding dummies to the dataset

X = pd.get_dummies(X)

train = pd.get_dummies(train)

test = pd.get_dummies(test) print(X.shape, train.shape, test.shape)

Overfitting:

Less than half of applications classified as not accepted, therefore perform stratified cv would be a better approach to avoid overfitting. With this approach, we will have not one estimate but 5 estimates for the generalization error.

We shall try several classification algorithms and select the best one based on performance score.

kfold = StratifiedKFold(n_splits=5, random_state=42) # Logistic Regression

log_reg = LogisticRegression(solver='lbfgs', max_iter=5000)

log_scores = cross_val_score(log_reg, x_train, y_train, cv=kfold)

log_reg_mean = log_scores.mean() # SVC

svc_clf = SVC(gamma='auto')

svc_scores = cross_val_score(svc_clf, x_train, y_train, cv=kfold)

svc_mean = svc_scores.mean() # KNearestNeighbors

knn_clf = KNeighborsClassifier()

knn_scores = cross_val_score(knn_clf, x_train, y_train, cv=kfold)

knn_mean = knn_scores.mean() # Decision Tree

tree_clf = tree.DecisionTreeClassifier()

tree_scores = cross_val_score(tree_clf, x_train, y_train, cv=kfold)

tree_mean = tree_scores.mean() # Gradient Boosting Classifier

grad_clf = GradientBoostingClassifier()

grad_scores = cross_val_score(grad_clf, x_train, y_train, cv=kfold)

grad_mean = grad_scores.mean() # Random Forest Classifier

rand_clf = RandomForestClassifier(n_estimators=100)

rand_scores = cross_val_score(rand_clf, x_train, y_train, cv=kfold)

rand_mean = rand_scores.mean() # NeuralNet Classifier

neural_clf = MLPClassifier(alpha=1)

neural_scores = cross_val_score(neural_clf, x_train, y_train, cv=kfold)

neural_mean = neural_scores.mean() # Naives Bayes

nav_clf = GaussianNB()

nav_scores = cross_val_score(nav_clf, x_train, y_train, cv=kfold)

nav_mean = neural_scores.mean() # Create a Dataframe with the results.

d = {'Classifiers': ['Logistic Reg.', 'SVC', 'KNN', 'Dec Tree', 'Grad B CLF', 'Rand FC', 'Neural Classifier', 'Naives Bayes'],

'Crossval Mean Scores': [log_reg_mean, svc_mean, knn_mean, tree_mean, grad_mean, rand_mean, neural_mean, nav_mean]} result_df = pd.DataFrame(data=d) # Log Reg performed best

result_df = result_df.sort_values(by=['Crossval Mean Scores'], ascending=False)

print(result_df)

Feature importance:

# Random Forest Classifier

rand_clf = RandomForestClassifier(n_estimators=100).fit(x_train, y_train) # get importance

plt.barh(X.columns, rand_clf.feature_importances_)

plt.show()

We can see here that credit history, loan amount, co-applicant’s income and applicant’s income plays significant roles in determining the approval. This kind of fits our mental model too.

Confusion Matrix:

The main purpose of a confusion matrix is to see how our model is performing when it comes to classifying potential clients that are likely to subscribe to a term deposit. We will see in the confusion matrix four terms the True Positives, False Positives, True Negatives and False Negatives.

cm = confusion_matrix(y_test, y_test_pred)

print('Confusion Matrix :')

print(cm) print ('Report : ')

print (classification_report(y_test, y_test_pred)) names = ['TN','FP','FN','TP']

counts = ['{0:0.0f}'.format(value) for value in cm.flatten()]

percentages = ['{0:.2%}'.format(value) for value in cm.flatten()/np.sum(cm)]

labels = [f'{v1}

{v2}

{v3}' for v1, v2, v3 in zip(names, counts, percentages)]

labels = np.asarray(labels).reshape(2,2)

sns.heatmap(cm/np.sum(cm), annot=labels, fmt='')

plt.show()

Recall Precision Tradeoff:

As the precision gets higher the recall gets lower and vice versa. For instance, if we increase the precision from 30% to 60% the model is picking the predictions that the model believes is 60% sure.

If there is an instance where the model believes that is 58% likely to be a loan approval then the model will classify it as a “No.”

However, that instance was actually a “Yes” (potential client eligible for loan)

That is why the higher the precision the more likely the model is to miss instances that are actually a “Yes”.

ROC Curve (Receiver Operating Characteristic):

The ROC curve tells us how well our classifier is classifying between yes and no.

The X-axis is represented by False positive rates (Specificity) and the Y-axis is represented by the True Positive Rate (Sensitivity.)

As the line moves the threshold of the classification changes giving us different values.

The closer is the line to our top left corner the better is our model separating both classes.

# predict probabilities

yhat = log_reg.predict_proba(x_test)

skplt.metrics.plot_roc_curve(y_test, yhat)

plt.annotate('Minimum ROC Score of 50%

(This is the minimum score to get)', xy=(0.5, 0.5), xytext=(0.6, 0.3), arrowprops=dict(shrink=0.05))

plt.show()

Threshold for test from ROC-curve:

# retrieve just the probabilities for the positive class

probs = yhat[:, 1]

fpr, tpr, thresholds = metrics.roc_curve(y_test, probs) plt.subplots(figsize=(10, 6))

plt.plot(fpr, tpr, 'o-', label="ROC curve")

plt.plot(np.linspace(0,1,10), np.linspace(0,1,10), label="diagonal")

for x, y, txt in zip(fpr[::5], tpr[::5], thresholds[::5]):

plt.annotate(np.round(txt,2), (x, y-0.04)) rnd_idx = 27

plt.annotate('this point refers to the tpr and the

fpr at a probability threshold of {}'.format(np.round(thresholds[rnd_idx], 2)), xy=(fpr[rnd_idx], tpr[rnd_idx]), xytext=(fpr[rnd_idx]+0.2, tpr[rnd_idx]-0.25), arrowprops=dict(color='black', lw=2, arrowstyle='->'))

plt.legend(loc="upper left")

plt.xlabel("FPR"); plt.ylabel("TPR"); plt.show();

gmeans = sqrt(tpr * (1-fpr))# geometric means=sqrt(sensitivity * specificity)= sqrt(tpr * (1-fpr))

index = argmax(gmeans) # index of the largest g-mean

print('Best Threshold=%f, G-Mean=%.3f' % (thresholds[ix], gmeans[ix])) # plot the roc curve for the model

plt.plot([0,1], [0,1], linestyle='--', label='No Skill')

plt.plot(fpr, tpr, marker='.', label='Logistic')

plt.scatter(fpr[index], tpr[index], marker='o', color='red', label='Best') plt.xlabel('False Positive Rate'); plt.ylabel('True Positive Rate'); plt.legend(); plt.show();

Learning curve (Bias-Variance):

It is in-fact impossible to avoid the relationship between bias and variance for the sheer fact that-

Increasing the bias will decrease the variance.

Increasing the variance will decrease the bias.

So, we see there is a trade-off and to strike a balance we need to carefully select the algorithms and configure to work on reducible errors. Our target is to achieve low variance and low bias. Well, the detection is not that difficult, however, real work is to reduce the error to minimum. Several measures can be taken to reduce this error e.g.-

increase the complexity of the model

increase input features

decrease regularization term etc.

Learning curve plotted the prediction accuracy/error vs. the training set size to reflect how better does the model get at predicting the target as training instances are increased. Here both the training and test/validation performance are plotted together so we can diagnose the bias-variance tradeoff. This is contrast to ROC curve which does not show learning. Here y-axis is ‘score’, and higher the score, the better the performance of the model. Training score (red line) and Cross-validation score (green line) almost meeting at some point indicating model a good fit for the given data set.

Report generation

report = pd.read_csv("report_loan_approval.csv")

# fill the Loan_ID and Loan_Status

report['Loan_Status'] = y_test_pred

report['Loan_ID'] = df1['Loan_ID'] # replace with "N" and "Y"

report['Loan_Status'].replace(0, 'N', inplace=True)

report['Loan_Status'].replace(1, 'Y', inplace=True) from IPython.display import HTML

def hover(hover_color="#ffff99"):

return dict(selector="tr:hover", props=[("background-color", "%s" % hover_color)])

styles = [hover(), dict(selector="th", props=[("font-size", "150%"), ("text-align", "center")]), dict(selector="caption", props=[("caption-side", "bottom")])]

html = (report.style.set_table_styles(styles).set_caption("Hover to highlight."))

html

Conclusion

Here, several machine learning algorithms were tried e.g. Logistic Regression(LR), Decision Tree (DT) and Random Forest (RF) etc. are applied to predict the loan approval of customers. The experimental results conclude that the accuracy of Logistic Regression algorithm is better as compared others. However, great care is required when selecting the appropriate cross-validation technique.

Connect me here.

Note: The programs described here are experimental and should be used with caution for any commercial purpose. All such use at your own risk.