Machine Learning Case Study: A data-driven approach to predict the success of bank telemarketing | by Sukanta Roy | Towards Data Science

In today’s world, data is the king. Use it in an effective way and it can create a huge impact on your business, don’t leverage it and you will be left behind in this fast paced world in no time. And one of the ways that a organisation can improve it’s performance in the market is to capture and analyse customer data in an efficient way to improve the customer experience.
One of the industries that is being transformed the most by the recent Machine learning advances is the finance industry. Be it predicting stock prices, or in our case predicting if a customer will subscribe to a term deposit Machine learning can be an incredibly useful tool for providing better profitability. With this, let’s jump right into our Case study.
We are given the data of direct marketing campaigns (phone calls) of a Portuguese banking institution. The classification goal is to predict if the client will subscribe a term deposit (target variable y). This case study is inspired by this research paper where the researchers have used a very similar dataset as the one we will be using throughout this case study for determining the success of Bank Telemarketing. The researchers in their paper have mentioned that the best result they have got was a AUC score of 0.8 and a ALIFT of 0.7. So as a goal we will try to produce a similar result in our case study.
As mentioned above, the dataset consists of direct marketing campaigns data of a banking institution. The dataset was picked from UCI Machine Learning Repository which is an amazing source for publicly available datasets. There were four variants of the datasets out of which we chose “ bank-additional-full.csv” which consists of 41188 data points with 20 independent variables out of which 10 are numeric features and 10 are categorical features. The list of features available to us are given below:
bank client data:
age (numeric)job : type of job (categorical: ‘admin.’,’blue-collar’,’entrepreneur’,’housemaid’,’management’,’retired’,’self-employed’,’services’,’student’,’technician’,’unemployed’,’unknown’)marital : marital status (categorical: ‘divorced’,’married’,’single’,’unknown’; note: ‘divorced’ means divorced or widowed)education (categorical: ‘basic.4y’,’basic.6y’,’basic.9y’,’high.school’,’illiterate’,’professional.course’,’university.degree’,’unknown’)default: has credit in default? (categorical: ‘no’,’yes’,’unknown’)housing: has housing loan? (categorical: ‘no’,’yes’,’unknown’)loan: has personal loan? (categorical: ‘no’,’yes’,’unknown’)
age (numeric)
job : type of job (categorical: ‘admin.’,’blue-collar’,’entrepreneur’,’housemaid’,’management’,’retired’,’self-employed’,’services’,’student’,’technician’,’unemployed’,’unknown’)
marital : marital status (categorical: ‘divorced’,’married’,’single’,’unknown’; note: ‘divorced’ means divorced or widowed)
education (categorical: ‘basic.4y’,’basic.6y’,’basic.9y’,’high.school’,’illiterate’,’professional.course’,’university.degree’,’unknown’)
default: has credit in default? (categorical: ‘no’,’yes’,’unknown’)
housing: has housing loan? (categorical: ‘no’,’yes’,’unknown’)
loan: has personal loan? (categorical: ‘no’,’yes’,’unknown’)
Related with the last contact of the current campaign:8. contact: contact communication type (categorical: ‘cellular’,’telephone’)
9. month: last contact month of year (categorical: ‘jan’, ‘feb’, ‘mar’, ..., ‘nov’, ‘dec’)
10. day_of_week: last contact day of the week (categorical: ‘mon’,’tue’,’wed’,’thu’,’fri’)
11. duration: last contact duration, in seconds (numeric). Important note: this attribute highly affects the output target (e.g., if duration=0 then y=’no’). Yet, the duration is not known before a call is performed. Also, after the end of the call y is obviously known. Thus, this input should only be included for benchmark purposes and should be discarded if the intention is to have a realistic predictive model.
other attributes:12. campaign: number of contacts performed during this campaign and for this client (numeric, includes last contact)
13. pdays: number of days that passed by after the client was last contacted from a previous campaign (numeric; 999 means client was not previously contacted)
14. previous: number of contacts performed before this campaign and for this client (numeric)
15. poutcome: outcome of the previous marketing campaign (categorical: ‘failure’,’nonexistent’,’success’)
social and economic context attributes16. emp.var.rate: employment variation rate — quarterly indicator (numeric)
17. cons.price.idx: consumer price index — monthly indicator (numeric)
18. cons.conf.idx: consumer confidence index — monthly indicator (numeric)
19. euribor3m: euribor 3 month rate — daily indicator (numeric)
20. nr.employed: number of employees — quarterly indicator (numeric)
As you would have probably guess it by now, this is a binary classification problem. Our two classes are “yes” denoting that the customer subscribed to a term deposit, and “no” denoting that the customer did not subscribe.
The performance metric used for this case study is AUC ROC score also known as AUROC (Area Under the Receiver Operating Characteristics).
Why use AUC?The reason we are choosing AUC over accuracy is because,as we will see in Exploratory data analysis, the dataset we are working with is an imbalanced dataset with the class “no” being the majority class. If we use accuracy as our metric, any random model can give us a very good accuracy. But at the end, it will be a random model. AUC gets over this problem by looking into both the True positive rate (TPR) and False positive rate (FPR). Only if both the TPR and FPR are well above the random line in the ROC curve, we will get a good AUC. Accuracy does not guarantee that.
Jumping into the modelling part without knowing anything about the problem is not a good idea. So as a start, we will do some EDA to find out more about what kind of data are we dealing with, if there is any pattern to the data. We will do some univariate analysis to find out which features can help us in our classification task and which variables are not that important. Now as it will be a very lengthy blog if we go on an do univariate analysis of all the 20 features one by one here, we will just look into some of the key aspects in this blog. If you want to go through my whole case study you can find it in this repository.
Distribution of class variable: First we will look into the class distribution to see if it is a balanced data or not.
We can see from the above plot that the dataset in imbalanced, where the number of negative class is close to 8 times the number of positive class.
Now we will do some Univariate analysis which is taking one feature at a time and checking how well it can distinguish between the two classes. For example, one of the features in our dataset is “Job” of the customer. We will see the distribution to find out if there is any specific category for this feature which subscribes to a term deposit more than other categories.
We will look into the Count Plots of each feature to find out the class distribution for each feature. First let’s define a function to plot the count plots which will help us as we go forward with the case study.
Job: The type of job the customers have. Let’s call the count plot function defined earlier to plot the count plot of the job feature.
From the above plot, we can see that the customers who have a job of admin have the highest rate of subscribing a term deposit, but they are also the highest when it comes to not subscribing. This is simply because we have more customers working as admin than any other profession.
Let’s look into some more features.
Marital: This simply denotes the customer’s marital status.
Majority of the customers are married. Followed by Single, divorced and unknown.
Default: Denotes if the customer has credit in default or not. The categories are yes, no and unknown.
housing: Denotes if the customer has a housing loan. Three categories are ‘no’, ’yes’, ’unknown’.
As we can see from the above plot, majority of the customers have a housing loan.
poutcome: This feature denotes the outcome of the previous marketing campaign.
For most of the customers, the previous marketing campaign outcome does not exists. It means that most of the customers are new customers who have not been contacted earlier. Also one thing to note here that, for the customers who had a successful outcome from the previous campaign, majority of those customers did subscribe for a term deposit. As it has the class distribution of 2.2% for positive class, and 1.2% for negative class. From this, we can make an assumption, that this feature may hold some value in predicting the target variable. specially poutcome_success category.
Let’s see some of the features that did not seem very helpful in the analysis.
day_of_week: This feature denotes the last contact day of the week (categorical: ‘mon’,’tue’,’wed’,’thu’,’fri’)
As we can see clearly from the above plot that, all the days have the similar distribution for both the classes. 17–18% of the customers on any given day refuse to subscribe to a term deposit, and 2.1–2.5% of customers agrees to subscribe. With this we can deduce that this feature will not be very helpful in predicting the target variable.
With this let’s look into some of the numerical features.
We will look into the box plot and the distribution for all the numerical features.
Age:
From the above boxplot we know that for both the customers that subscibed or didn’t subscribe a term deposit, has a median age of around 38–40. And the boxplot for both the classes overlap quite a lot, which means that age isn’t necessarily a good indicator for which customer will subscribe and which customer will not.
Duration: Now this feature is very interesting in our case study. It denotes the duration of the last contact, in seconds. It is mentioned in the source of the dataset:
Important note: This attribute highly affects the output target (e.g., if duration=0 then y=’no’). Yet, the duration is not known before a call is performed. Also, after the end of the call, the target variable y is obviously known. Thus, this input should only be included for benchmark purposes and should be discarded if the intention is to have a realistic predictive model.
So with this we know that this feature will be very interesting xD. And we also know that we will not be able to include this feature in our final models, as obviously we want to create a realistic predictive model that can be used by the business. But, we will surely implement a basic model with the duration feature just to see how much of an impact this feature makes. So with that, let’s look into the box plot and pdf of this feature.
From the above plot it is clear that, the duration (last contact duration) of a customer can be useful for predicting the target variable. It is expected because it is already mentioned in the data overview that this field highely affects the target variable and should only be used for benchmark purposes.
emp.var.rate: Employment variation rate, quarterly indicator. Even though this and some other features like “previous” etc are numerical features, they have a limited number of unique values which occur repeatedly. Here is a value count of the emp.var.rate feature.
euribor 3 month rate, daily indicator. The euribor denotes the basic rate of interest used in lending between banks on the European Union interbank market and also used as a reference for setting the interest rate on other loans.
From the above plot, we can clearly see the difference in median for both the classes. This indicates that the feature can be very useful for our case study. But we can validate the assumption only by applying models and extracting feature importance.
The idea of correlation matrix came from this blog. Correlation is a statistical measure that indicates the extent to which two or more variables fluctuate together. A positive correlation indicates the extent to which those variables increase or decrease in parallel; a negative correlation indicates the extent to which one variable increases as the other decreases. This can be a very useful tool to quickly check which features are more correlated and which pair of features are not.
The emp.var.rate, cons.price.idx, euribor3m and nr.employed features have very high correlation. With euribor3m and nr.employed having the highest correlation of 0.95!
Now with that basic EDA done, let’s move forward to the next step which is Data Preprocessing. If you want to go through the EDA of all the features you can go check out this repository.
As our dataset has 10 categorical features we will need to encode these features into a numerical representation to apply the machine learning models. For this case study we will look into two encoding schemes and compare the results of both the encoding schemes at the end of the case study. The two encoding schemes are:1. One hot Encoding: This is one of the most popular encoding schemes to deal with the categorical features. For each unique category it creates a separate column with binary values (0 or 1). In each of these new columns there is a value of 1 if the corresponding category is present in this row and 0 otherwise.2. Response Coding: As part of this technique, we represent the probability of the data point belonging to a particular class given a category. So for a K-class classification problem, we get K new features which embed the probability of the datapoint belonging to each class based on the value of categorical data. Mathematically speaking, we calculate —
P(class=X | category=A) = P(category=A ∩ class=X) / P(category=A)
But before encoding the categorical features let’s first make some basic checks to see if there are any missing values or duplicate rows etc, as they can impact the final model performance.
Thankfully in our dataset there are no missing values. If there were any missing values we would have either removed the rows (if the number of missing values were less compared to the amount of data we have) or used imputation to handle the missing values.
As we can see there are some duplicate rows in our datasets, we will remove the duplicates before moving forward.
After dropping duplicate rows, there are in total 41176 rows.
Now it is very important to split your dataset into train, test and CV datasets, before applying any encoding schemes. Otherwise we will have a data leakage problem. So let’s first split the data before encoding our categorical variables. We will apply 64%, 16% and 20% for train, cv and test datasets respectively.
To encode the categorical variables, we use CountVectorizer which is normally used for encoding text features into Bag Of Words (BoW) representation. We use countVectorizer instead of any other functions for One hot Encoding because of the problem of missing dimensions in test dataset. This is a problem you will face if you are say using pandas get_dummies function. The problem arises because when you are splitting the data into train, test there can be some categories of some features which will be in train and not be in the test dataset. If you use pd.get_dummies you will end up with different dimensions in train and test. To encounter this problem we use CountVectorizer and fit_transform on the train dataset and transform on the cv and test dataset as you can see in the above embedded code.
As we can already encoded our dataset, it will be very unfair to not apply t-sne on it to visualise the high dimensional data in 2D isn’t it. So let’s do just that.
As we can see, even though there seems to be a lot of overlap in the data, there seems to be some distinction between the two classes.
Now let’s use response coding on the categorical features. After we do the response coding, we will go to the modelling part, where we will compare the results of the models using both the encoding schemes to see if the encoding scheme can make a difference or not. Here is a pseudo code to encode categorical features using response coding:
# alpha is for laplace smoothing 1. Consider all unique values and the number of occurrences of given feature in train data frame2. Build a vector (1*2) [As this is a binary classification problem].3. The first element = (number of times it occured in class1 + 10*alpha / number of time it occurred in total data+20*alpha)4. Create a dict (say feat_dict) that will store every categorical data as key and store a (1*2) representation of response coded vector as the value5. Create a list (say res_fea) that will store the final response coded features5. for a value of feature in df:       if it is in train data:          we add the vector that was stored in 'feat_dict' look up          table to 'res_fea'              if it is not there in train data:           we add [1/2, 1/2] to 'res_fea' to indicate equal probability of both classes.return 'res_fea'
Now comes the part we all have been waiting for, the part where we apply the data to the machine learning models to predict the target variable. So let’s start with that.
Now as I promised, before jumping onto creating a realistic model that can be used, we will first look how the duration feature effects the output. For that we will just apply as simple logistic regression model without any hyper parameter tuning with and without the duration column and compare the results. After that we will remove the feature from our dataset once and for all, and apply various models.
Now we will apply Logistic regression on both the One hot encoded dataset and the Response coded dataset and compare the results.
On one hot encoded data:
As we can see from the above gist that the best AUC that we got by applying Logistic Regression on the dataset with one hot encoding scheme is 0.798. It is anyway very close to what the researchers have got which was 0.8. Let’s see how Logistic regression works on response coded dataset.
On Response coded data:
With response coding, Logistic regression did not work as well as One hot encoding and gave us the AUC of 0.77.
Similarly we will apply some other models and see which algorithm works the best and can get us the AUC of or above 0.8.
On one hot encoded data:
On Response coded data:
On one hot encoded data:
On Response coded data:
In case of Random Forest, we can see that even after hyper parameter tuning, the models are overfitting a lot with a train AUC of 0.99 and a test AUC of 0.77. It is not that strange as Random Forest does have a tendency to overfit in some cases.
Now let’s try one of my favourite algorithms, which is XGBoost to see if it can outperform all the other models to give us a test AUC score of over 0.8!
On one hot encoded data:
On Response coded data:
As we can see from the above two gists, that XGBoost gave a test AUC score of 0.803 on response coded data. We can also see at the end of the gists the feature importance of each feature in both the encoding schemes. Turns out that the most important features were the numerical features :P More on that in the final conclusion.
So, after applying all the models (I have also applied KNN and RBF kernal SVM which I did not include here) this is the final table of all the train and test AUC scores.
The top 3 key features or attributes that helped in the prediction of the class variable were:1. nr.employed2. euribor3m3. emp.var.rate
Other than these 3 cons.price.idx and the categorical feature month and poutcome were also held a lot of importance in our case study
So let’s quickly go through what we did in this case study to achieve the desired performance we were looking for.
The objective of this case study was to predict whether a customer will subscribe to a term deposit or not given the data of the customer.There were a lot of categorical variables and some numerical variables which capture various information about the customer and the bank-customer relationship.First we did EDA and figured out that there is no null values for the data, and the data is imbalanced, where “no” is the majority class.After doing univariate analysis of we figured that day_of_week does not help very much when it comes to predicting the target variable. But on the other hand, some numerical features tend to predict the target variable much better.After doing data visualization with T-SNE, it was clear that the data does not have a very well separation between the classes, as the two classes were highly overlapping even after trying different parameters.After basic data preprocessing, I encoded the categorical data into both One hot encoding method and also the response coding method.After implementing all the models, we saw that the model that gave the best performance was XGBoost with RandomizedSearchCV and the test AUC was 0.803 which was similar to the results the researchers got in the relevent research paper.
The objective of this case study was to predict whether a customer will subscribe to a term deposit or not given the data of the customer.
There were a lot of categorical variables and some numerical variables which capture various information about the customer and the bank-customer relationship.
First we did EDA and figured out that there is no null values for the data, and the data is imbalanced, where “no” is the majority class.
After doing univariate analysis of we figured that day_of_week does not help very much when it comes to predicting the target variable. But on the other hand, some numerical features tend to predict the target variable much better.
After doing data visualization with T-SNE, it was clear that the data does not have a very well separation between the classes, as the two classes were highly overlapping even after trying different parameters.
After basic data preprocessing, I encoded the categorical data into both One hot encoding method and also the response coding method.
After implementing all the models, we saw that the model that gave the best performance was XGBoost with RandomizedSearchCV and the test AUC was 0.803 which was similar to the results the researchers got in the relevent research paper.
If you want to go through the complete case study, you can check out this repository.
Hi my name is Sukanta Roy, I am a software developer and an aspiring Machine Learning Engineer, Former Google Summer of Code 2018 student and a huge psychology buff. If any of these things interest you, you can connect with me on medium and LinkedIn.