Bank Institution Term Deposit Predictive Model | by Glory Odeyemi | Towards Data Science

Courtesy of the 10 Academy training program, I’ve been introduced to many data science concepts by working on different projects, each of them challenging in their own way.
Bank Institution Term Deposit Predictive Model is a project I found interesting. Its main objective is to build a model that predicts the customers that would or would not subscribe to bank term deposits, and this article aims at sharing my step by step approach of building the model.
The Data
Exploratory Data Analysis
Data Preprocessing
Machine Learning Model
Comparing Results
Prediction
Conclusion
Further Study
The dataset (Bank-additional-full.csv) used in this project contains bank customers’ data. The dataset, together with its information, can be gotten here. The first step to take when performing data analysis is to import the necessary libraries and the dataset to get you going.
# importing the necessary librariesimport pandas as pdimport numpy as npimport matplotlib.pyplot as pltimport matplotlib as mplimport seaborn as snsimport warningswarnings.filterwarnings('ignore')#importing the datasetdataset = pd.read_csv('bank-additional-full.csv', sep=';')dataset.name = 'dataset'dataset.head()
EDA is an essential part of machine learning model development because it helps us in understanding our data and extract useful insights that will help in feature engineering. Some of the EDA performed in this project includes but not limited to the following;
Shape and size of dataset
# function to check the shape of a datasetdef data_shape(data):    print(data.name,'shape:',data.shape)# function to check the size of a datasetdef data_size(data):    print(data.name,'size:',data.size)# Getting the shape of the datasetdata_shape(dataset)# Getting the size of the datasetdata_size(dataset)
dataset shape: (41188, 21)dataset size: 864948
.shape returns the number of rows and columns of the dataset.
.size returns the number of elements in the data i.e the number of rows times number of columns.
Information and Statistical summary
# function to ckeck the information of a datasetdef data_info(data):    print(data.name,'information:')    print('---------------------------------------------')    print(data.info())    print('---------------------------------------------')# Getting the information of the datasetdata_info(dataset)
.info() is used to get concise summary of the dataset.
# Getting the statistical summarydataset.describe().T
.describe() is used to view some basic statistical details like percentile, mean, std etc. of numerical columns in the dataset.
Unique and missing values
# function to get all unique values in the categorical variablesdef unique_val(data):    cols = data.columns    for i in cols:        if data[i].dtype == 'O':            print('Unique values in',i,'are',data[i].unique())            print('----------------------------------------------')# Getting the unique values in the categorical columnsunique_val(dataset)
.unique() returns the unique values in a categorical column of the dataset.
# function to check for missing valuesdef missing_val(data):    print('Sum of missing values in', data.name)    print('------------------------------')    print(data.isnull().sum())    print('------------------------------')# Getting the missing values in the datasetmissing_val(dataset)
.isnull().sum() returns the sum of missing values in each column of the dataset. Luckily for us, our dataset does not have missing values.
Categorical and numerical variables
# Categorical variablescat_data = dataset.select_dtypes(exclude='number')cat_data.head()# Numerical variablesnum_data = dataset.select_dtypes(include='number')num_data.head()
.select_dtypes(exclude=’number) returns all the columns that does not have a numerical data type.
.select_dtypes(exclude=’number) returns all the columns that has a numerical data type.
Univariate and Bivariate Analysis
I made use of tableau (a data visualization tool) for the univariate and bivariate analysis and the tableau story can be found here.
Correlation
# using heatmap to visualize correlation between the columnsfig_size(20,10)ax = sns.heatmap(dataset.corr(), annot=True, fmt='.1g',                  vmin=-1, vmax=1, center= 0)# setting the parametersfig_att(ax, "Heatmap correlation between Data Features",         "Features", "Features", 35, 25, "bold")plt.show()
Correlation shows the relationship between variables in the dataset.
Outliers
Seaborn boxplot is one of the ways of checking a dataset for outliers.
# Using boxplot to identify outliersfor col in num_data:    ax = sns.boxplot(num_data[col])    save(f"{col}")    plt.show()
The code above visualizes the numerical columns in the dataset and outliers detected were treated using the Interquartile Range (IQR) method. The code can be found in this GitHub repository.
In the course of the EDA, I found out that our target variable ‘y’ — has the client subscribed to a term deposit? (binary: ‘yes’,’no’), is highly imbalanced and that can affect our prediction model. This will be taken care of shortly and this article gives justice to some techniques of dealing with class imbalance.
When building a machine learning model, it is important to preprocess the data to have an efficient model.
# create list containing categorical columnscat_cols = ['job', 'marital', 'education', 'default', 'housing',            'loan', 'contact', 'month', 'day_of_week', 'poutcome']# create list containing numerical columnsnum_cols = ['duration', 'campaign', 'emp.var.rate',"pdays","age",       'cons.price.idx', 'cons.conf.idx', 'euribor3m', 'nr.employed', 'previous']
The following preprocessing was done in this stage:
Encoding Categorical columns
Machine learning algorithms only read numerical values, which is why we need to change our categorical values to numerical values. I made use of pandas get_dummies method and type-casting to one-hot encode the columns.
# function to encode categorical columnsdef encode(data):    cat_var_enc = pd.get_dummies(data[cat_cols], drop_first=False)    return cat_var_enc# defining output variable for classificationdataset_new['subscribed'] = (dataset_new.y == 'yes').astype('int')
Rescaling Numerical columns
Another data preprocessing method is to rescale our numerical columns; this helps to normalize our data within a particular range. Sklearn preprocessing StandardScaler() was used here.
# import library for rescalingfrom sklearn.preprocessing import StandardScaler# function to rescale numerical columnsdef rescale(data):    # creating an instance of the scaler object    scaler = StandardScaler()    data[num_cols] = scaler.fit_transform(data[num_cols])    return data
Specifying Dependent and Independent Variables
To proceed in building our prediction model, we have to specify our dependent and independent variables.
Independent variables — are the input for a process that is being analyzed.
Dependent variable — Dependent variable is the output of the process.
X = data.drop(columns=[ "subscribed", 'duration'])y = data["subscribed"]
The column ‘duration’ was dropped because it highly affects the output target (e.g., if duration=0 then y=’no’).
Splitting the Dataset
It is reasonable to always split the dataset into train and test set when building a machine learning model because it helps us to evaluate the performance of the model.
# import library for splitting datasetfrom sklearn.model_selection import train_test_split# split the dataX_train,X_test,y_train,y_test = train_test_split(X,y,test_size = 0.1,random_state=1)
Dimensionality Reduction
In a case whereby we have a large number of variables, it is advisable to consider reducing these variables by keeping the most important ones, and there are various techniques for doing this, such as; PCA, TSNE, autoencoders, etc. For this project, we will be considering PCA.
# import PCAfrom sklearn.decomposition import PCA# create an instance of pcapca = PCA(n_components=20)      # fit pca to our datapca.fit(X_train)pca_train = pca.transform(X_train)X_train_reduced = pd.DataFrame(pca_train)
Class Imbalance
As earlier stated, we have a highly imbalanced class, and this can affect our prediction if not treated.
In this project, I made use of SMOTE (Synthetic Minority Oversampling Technique) for dealing with class imbalance.
# importing the necessary function from imblearn.over_sampling import SMOTE# creating an instancesm = SMOTE(random_state=27)# applying it to the training setX_train_smote, y_train_smote = sm.fit_sample(X_train_reduced, y_train)
Note: It is advisable to use SMOTE on the training data.
Whew!, we finally made it to building the model; data preprocessing can be such a handful when trying to build a machine learning model. Let’s not waste any time and dive right in.
The machine learning algorithm that was considered in this project includes;
Logistic Regression
XGBoost
Multi Layer Perceptron
and the cross validation (this is essential especially in our case where we have an imbalanced class) method used includes;
K-Fold: K-Fold splits a given data set into a K number of sections/folds where each fold is used as a testing set at some point.
Stratified K-Fold: This is a variation of K-Fold that returns stratified folds. The folds are made by preserving the percentage of samples for each class.
# import machine learning model librariesfrom sklearn.linear_model import LogisticRegressionfrom xgboost import XGBClassifierfrom sklearn.neural_network import MLPClassifier# import libraries for cross validationfrom sklearn.model_selection import KFoldfrom sklearn.model_selection import StratifiedKFoldfrom sklearn.model_selection import cross_validatemetrics = ['accuracy', 'roc_auc', f1', 'precision', 'recall']# function to build machine learning modelsdef model(model, cv_method, metrics, X_train, X_test, y_train):    if (model == 'LR'):        # creating an instance of the regression        model_inst = LogisticRegression()        print('Logistic Regression\n----------------------')    elif (model == 'XGB'):        # creating an instance of the classifier        model_inst = XGBClassifier()        print('XGBoost\n----------------------')    elif (model == 'MLP'):        # creating an instance of the classifier        model_inst = MLPClassifier()        print('Multi Layer Perceptron\n----------------------')        # cross validation    if (cv_method == 'KFold'):        print('Cross validation: KFold\n--------------------------')        cv = KFold(n_splits=10, random_state=100)    elif (cv_method == 'StratifiedKFold'):        print('Cross validation: StratifiedKFold\n-----------------')        cv = StratifiedKFold(n_splits=10, random_state=100)    else:        print('Cross validation method not found!')    try:        cv_scores = cross_validate(model_inst, X_train, y_train,                                    cv=cv, scoring=metrics)           # displaying evaluation metric scores        cv_metric = cv_scores.keys()        for metric in cv_metric:            mean_score = cv_scores[metric].mean()*100            print(metric+':', '%.2f%%' % mean_score)            print('')                except:        metrics = ['accuracy', 'f1', 'precision', 'recall']        cv_scores = cross_validate(model_inst, X_train, y_train,                                    cv=cv, scoring=metrics)        # displaying evaluation metric scores        cv_metric = cv_scores.keys()        for metric in cv_metric:            mean_score = cv_scores[metric].mean()*100            print(metric+':', '%.2f%%' % mean_score)            print('')    return model_inst
Evaluation Metrics
Accuracy: The number of correctly predicted data points. This can be a misleading metric for an imbalanced dataset. Therefore, it is advisable to consider other evaluation metrics.
AUC (Area under the ROC Curve): It provides an aggregate measure of performance across all possible classification thresholds.
Precision: It is calculated as the ratio of correctly predicted positive examples divided by the total number of positive examples that were predicted.
Recall: It refers to the percentage of total relevant results correctly classified by your algorithm.
F1 score: This is the weighted average of Precision and Recall.
K-Fold vs Stratified K-Fold
As can be seen from the table above, Stratified K-Fold presented a much better result compared to the K-Fold cross validation. The K-Fold cross validation failed to provide the AUC score for the Logistic Regression and XGBoost model. Therefore, for further comparison, Stratified K-Fold results would be used.
Machine Learning Models
From the result gotten, XGBoost proves to be a better prediction model than Logistic Regression and MLP because it has the highest percentage values in 4/5 of the evaluation metrics.
XGboost, being the best performing model, is used for prediction.
# fitting the model to the train datamodel_xgb = xgb.fit(X_train_smote, y_train_smote)# make predictionsy_pred = xgb.predict(X_test_pca)
The main objective of this project is to build a model that predicts customers that would subscribe to a bank term deposit, and we were able to achieve that by considering three different models and using the best one for the prediction. We also went through rigorous steps of preparing our data for the model and choosing various evaluation metrics to measure the performance of our models.
In the result gotten, we observe that XGBoost was the best model with high percentage values in 4/5 of the evaluation metrics.
In this project, I used only three machine learning algorithms. However, algorithms such as; SVM, Random Forest, Decision Trees, etc. can be explored.
A detailed code for this project can be found in this GitHub repository.
I know this was a very long ride, but thank you for sticking with me to the end. I also appreciate 10 Academy once again, and my fellow learners for the wonderful opportunity to partake in this project.
Grammar checker
Class Imbalance
Dataset
Feature Engineering
Principal Component Analysis (PCA)
Tableau
Machine Learning Algorithms