Bank Institution Term Deposit Predictive Model | by Lotome John | Towards Data Science

The investment and portfolio department of the Bank of Portugal would want to be able to identify their customers who potentially would subscribe to their term deposits. As there has been heightened interest of marketing managers to carefully tune their directed campaigns to the rigorous selection of contacts, the task at hand is to find a model that can predict which future clients who would subscribe to the Bank’s term deposit.In this post, am going to talk about data exploration, data cleaning, feature extraction, and dealing with class imbalance and developing a robust machine learning algorithm for predicting which potential customers would subscribe to the term deposit using different Machine learning library. The Github repository containing all the code is available here.
PACKAGES
The following are the different machine learning and python packages used.
Exploratory data analysis is usually the initial process of every data science project. It is usually critical to perform an initial investigation on data to identify patterns, outliers, imbalances and to test a prior hypothesis. Summary statistics and visualization are mainly used to perform exploratory data analysis.
“There are no routine statistical questions, only questionable statistical routines.” Sir David Cox
From the above output, some patterns of the data can be extracted. Grouping the data based on if the potential client subscribed to the term deposit or not indicates that the majority of clients did not subscribe to the term deposit, 88.7% decline while 11.3% accepted the subscription offer. This significant difference in the proportion of the class that we are trying to predict leads to imbalance data.
The above output shows clusters of clients who subscribe to the term deposit and those who decline based on marital status.
An outlier is a data point that differs significantly from the other observations which may be due to variability in the measurement or it may indicate experimental errors. The temptation of training a machine learning model without checking and dealing with outliers is usually a recipe for failure. Outliers in data can significantly impact the accuracy of the model, but simply removing outliers from your data without considering the impact on the results is a bad practice since outliers aren’t always a bad thing.
“Outliers are not necessarily a bad thing. These are just observations that are not following the same pattern as the other ones. But it can be the case that an outlier is very interesting. For example, if in a biological experiment, a rat is not dead whereas all others are, then it would be very interesting to understand why. This could lead to new scientific discoveries. So, it is important to detect outliers.” Pierre Lafaye de Micheaux, Author, and Statistician.
A common practice of detecting outliers is using box plots, histograms, and scatter plots. This methodology may not be efficient in large datasets. PyOD, a python library has different algorithms that can be used to detect and clean outliers. For this case, we use k-Nearest Neighbors Detectors.
Datasets normally contain numerical and categorical variables but a machine learning algorithm only understands numbers and not text that is why we need to convert or transform the categorical variables, a process is known as categorical encoding. There two types of categorical encoding:
Label Encoding
One-Hot Encoding
Label Encoding is useful where the categories in the categorical columns have data with some kind of order or hierarchy. One-Hot Encoding works by splitting the column to be encoded into multiple columns depending on the number of categories in that column. The categories are then replaced by 1s and 0s, depending on which column has what value.We also use StandardScaler to rescale all numerical columns. This is useful since some variables are often measured at different scales and would not contribute equally to model fitting which may lead the trained model to create some bias.
Training a machine learning model on an imbalanced dataset may lead to inaccurate predictions. In this case, we use a random sampling technique to deal with the imbalanced data. Random oversampling works by randomly sampling data from the minority class and duplicates it to create more samples, this process is repeated until when the distribution is equal. One of the cons of the Random Sampling technique is that it can potentially lead to overfitting since it duplicates the existing minority data set.
Dimensionality reduction is a technique that reduces the number of input variables or features in a dataset. More input features often make a predictive modeling task more challenging to a model, referred to as the curse of dimensionality. There are different dimensionality reduction techniques like AutoEncoder and t-SNE which is a probabilistic technique. For our case, we use Principal Component Analysis (PCA) for dimension reduction. PCA performs a linear mapping of the data to a lower-dimensional space in such a way that the variance of the data in the low dimensional representation is maximized.We only performed PCA on the features alone, without including the labels.
The code snippet below shows a method that I used in comparing the different modelling algorithms. Four parameters are required when initializing the parameters
Model type. For this case, I used XGBoost Classifier, Multi-layer Perceptron classifier, Logistic Regression and Support Vector Machine.Cross-Validation technique, that is either k-fold or Stratified k-foldX which is the dataset featuresy which is the label
Model type. For this case, I used XGBoost Classifier, Multi-layer Perceptron classifier, Logistic Regression and Support Vector Machine.
Cross-Validation technique, that is either k-fold or Stratified k-fold
X which is the dataset features
y which is the label
Initialization of the different models
In all the models selected even with hyperparameter tuning, XGBoost Classifier had higher accuracy scores. The figure below shows the different metric scores of the XGBoost model used. There was a slightly significant difference in the scores while using Stratified k-fold and k-fold cross-validation. Stratified k-fold had higher scores compared to k-fold.
Accuracy scores of the model: 0.88 Classification report of the model--------------------------------------              precision    recall  f1-score   support           0       0.89      0.87      0.88      3442           1       0.88      0.90      0.89      3537    accuracy                           0.88      6979   macro avg       0.89      0.88      0.88      6979weighted avg       0.89      0.88      0.88      6979 Confusion Matrix of the model--------------------------------------[[2993  449] [ 354 3183]]
Our model had an accuracy of 0.88 with precision and recall metrics above 0.87.
The ROC plot when using k-fold cross validation indicates a score of 0.88.
while using stratified k-fold, there was an increase in score by 0.01
Since the XGBoost had a higher precision, recall, and f1-score for the minority class which is a more important classification of whether a potential client was going to subscribe to the term deposit, the bank should use XGBoost model. With different hyperparameter tuning techniques, the model can lead to higher scores.
Respect and gratitude to all my fellow learners and staff at 10Academy for their support in putting this together.