Loan Default Prediction with Berka Dataset | by Zhou (Joe) Xu | Towards Data Science

· Introduction· About the Dataset· Import Dataset into the Database· Connect Python to MySQL Database· Feature Extraction· Feature Transformation· Modeling· Conclusion and Future Directions· About Me
Note: If you are interested in the details beyond this post, the Berka Dataset, all the code, and notebooks can be found on my GitHub Page.
This post is just a hands-on practice building a loan default prediction model. If you are interested in this topic and want to see some more in-depth work that I accomplished for a client, using optimization to turn their loss into profit using such loan default prediction models, please see my other article here: Loan Default Prediction for Profit Maximization.
For banks, it is always an interesting and challenging problem to predict how likely a client is going to default the loan when they only have a handful of information. In the modern era, the data science teams in the banks build predictive models using machine learning. The datasets used by them are most likely to be proprietary and are usually collected internally through their daily businesses. In other words, there are not many real-world datasets that we can use if we want to work on such financial projects. Fortunately, there is an exception: the Berka Dataset.
The Berka Dataset, or the PKDD’99 Financial Dataset, is a collection of real anonymized financial information from a Czech bank, used for PKDD’99 Discovery Challenge. The dataset can be accessed from my GitHub page.
In the dataset, 8 raw files include 8 tables:
account (4500 objects in the file ACCOUNT.ASC) — each record describes static characteristics of an account.
client (5369 objects in the file CLIENT.ASC) — each record describes characteristics of a client.
disposition (5369 objects in the file DISP.ASC) — each record relates together a client with an account i.e. this relation describes the rights of clients to operate accounts.
permanent order (6471 objects in the file ORDER.ASC) — each record describes characteristics of a payment order.
transaction (1056320 objects in the file TRANS.ASC) — each record describes one transaction on an account.
loan (682 objects in the file LOAN.ASC) — each record describes a loan granted for a given account.
credit card (892 objects in the file CARD.ASC) — each record describes a credit card issued to an account.
demographic data (77 objects in the file DISTRICT.ASC) — each record describes demographic characteristics of a district.
Each account has both static characteristics (e.g. date of creation, address of the branch) given in relation “account” and dynamic characteristics (e.g. payments debited or credited, balances) given in the relations “permanent order” and “transaction”.
Relation “client” describes the characteristics of persons who can manipulate the accounts.
One client can have more accounts, more clients can manipulate with a single account; clients and accounts are related together in relation “disposition”.
Relations “loan” and “credit card” describe some services which the bank offers to its clients.
More than one credit card can be issued to an account.
At most one loan can be granted for an account.
Relation “demographic data” gives some publicly available information about the districts (e.g. the unemployment rate); additional information about the clients can be deduced from this.
This is an optional step since the raw files contain only delimiter-separated values, so it can be directly imported into data frames using pandas.
Here I wrote SQL queries to import the raw data files into MySQL database for simple and fast data manipulations (eg. select, join and aggregation functions) on the data.
Above is a code snippet showing how to create the bank database and import the Account table. It includes three steps:
Create and use database
Create a table
Load data into the table
There should not be any troubles in the first two steps if you are familiar with MySQL and the database systems. For the “Load data” step, you need to make sure that you have enabled the LOCAL_INFILE in MySQL. Detailed instruction can be found from this thread.
By repeating step 2 and step 3 on each table, all the data can be imported into the database.
Again, if you choose to import the data directly into Python using Pandas, this step is optional. But if you have created the database and become familiar with the dataset through some SQL data manipulations, the next step is to transfer the prepared tables into Python and perform data analysis there. One way is to use the MySQL Connector for Python to execute SQL queries in Python and make Pandas DataFrames using the results. Here is my approach:
After modifying the database info such as host, database, user, password, we can initiate a connection instance, execute the query and convert it into Pandas DataFrame:
Even though this is an optional step, it is advantageous in terms of speed, convenience, and good for experimentation purposes compared to directly import the files into Pandas DataFrames. Unlike other ML projects where we are only given with acsv file (1 table), this dataset is quite complicated and there is a lot of useful information hidden between the connections of tables, so this is another reason why I want to introduce the way of loading data into the database first.
Now the data is in MySQL server and we have connected it Python so that we can smoothly access the data in data frames. The next steps are to extract features from the table, transform the variables, load them into one array, and train a machine learning model.
Since predicting the loan default is a binary classification problem, we first need to know how many instances in each class. By looking at the status variable in the Loan table, there are 4 distinct values: A, B, C, and D.
A: Contract finished, no problems.
B: Contract finished, loan not paid.
C: Running contract, okay so far.
D: Running contract, client in debt.
According to the definitions from the dataset description, we can make them into binary classes: good (A or C) and bad (B or D). There are 606 loans that fall into the “good” class and 76 of them are in the “bad” class.
With the two distinct classes defined, we can look into the variables and plot the histograms to see if they correspond to different distributions.
The loan amount shown below is a good example to see the difference between the two classes. Even though both are right-skewed, it still shows an interesting pattern that loans with a higher amount tend to default.
When extracting features, they don’t have to be the existing variables provided in the tables. Instead, we can always be creative and come up with some out-of-the-box solutions on creating our own features. For example, when joining the Loan table and the Account table, we can get both the date of loan issuance and the date of account creation. We may wonder if the time gap between creating the account and applying for the loan plays a role, so a simple subtraction would give us a new variable consists of days between the two such activities on the same account. The histograms are shown below, where a clear trend can be seen that people who apply for the loan right after creating the bank account tend to default.
By repeating the process of experimenting with existing features and created features, I finally prepared a table that consists of 18 feature columns and 1 label column. The selected features are:
amount: Loan amount
duration: Loan duration
payments: Loan payments
days_between: Days between account creation and loan issuance
frequency: Frequency of issuance of statements
average_order_amount: Average amount of the permanent orders made by the account
average_trans_amount: Average amount of the transactions made by the account
average_trans_balance: Average balance amount after transactions made by the account
n_trans: Transaction number of account
card_type: Type of credit card associated with the account
n_inhabitants: Number of inhabitants in the district of account
average_salary: Average salary in the district of account
average_unemployment: Average unemployment rate in the district of account
entrepreneur_rate: Number of entrepreneurs per 1000 inhabitants in the district of account
average_crime_rate: Average crime rate in the district of account
owner_gender: Account owner’s gender
owner_age: Account owner’s age
same_district: A boolean that represents if the owner has the same district information as the account
After the features are extracted and put into a big table, it is necessary to transform the data so that they can be fed into the machine learning model in an organic way. In our case, we have two types of features. One is numerical, such as amount, duration, and n_trans. The other one is categorical, such as card_type and owners_gender.
Our dataset is pretty clean and there is any missing value, so we can skip the imputation and directly jumpy into scaling for the numerical values. The are several options of scalers from scikit-learn , such as StandardScaler , MinMaxScaler and RobustScaler . Here, I used MinMaxScaler to rescale the numerical values between 0 and 1. On the other hand, the typical strategy of dealing with categorical variables is to use OneHotEncoder to transform the features into binary 0 and 1 values.
The code below is a representation of the feature transformation steps:
The first thing in training a machine learning model is to split the train and test sets. It is tricky in our dataset because it is not balanced: there are almost 10 times more good loans than bad loans. A stratified split is a good option here because it preserves the ratio between classes in both train and test sets.
There are many good machine learning models for binary classification tasks. Here, the Random Forest model is used in this project for its decent performance and quick-prototyping capability. An initial RandomForrestClassifier model is fit and three distinct measures are used to represent the model performance: Accuracy, F1 Score, and ROC AUC.
It is noticeable that Accuracy is not sufficient for this unbalanced dataset. If we finetune the model purely by accuracy, then it would favor toward predicting the loan as “good loan”. F1 score is the harmonic mean between precision and recall, and ROC AUC is the area under the ROC curve. These two are better metrics for evaluating the model performance for unbalanced data.
The code below shows how to apply 5-fold stratified cross-validation on the training set, and calculate the average of each score:
Acc: 0.8973F1: 0.1620ROC AUC: 0.7253
It is clearly seen that the accuracy is high, almost 0.9, but the F1 score is very low because of low recall. There is room for the model to be finetuned and strive for better performance, and one of the methods is Grid Search. By assigning different values to the hyperparameters of theRandomForestClassifier such as n_estimators max_depth min_samples_split and min_samples_leaf , it will iterate through the combinations of hyperparameters and output the one with the best performance on the score that we are interested in. A code snippet is shown below:
Refitting the model with the best parameters, we can take a look at the model performance one the whole train set and the test set:
Performance on Train Set:Acc: 0.9706F1: 0.8478ROC AUC: 0.9952Performance on Test Set:Acc: 0.8927F1: 0.2667ROC AUC: 0.6957
The performance on the train set is great: more than 2/3 of the bad loans and all of the good loans are correctly classified, and all of the three performance measures are above 0.84. On the other hand, when the model is used on the test set, the result is not quite satisfying: most of the bad loans are labeled as “good” and the F1 score is only 0.267. There is evidence that overfitting is involved, so more effort should be put into such iterative processes in order to get better model performance.
With the model built, we can now rank the features based on their importance. The top 5 features that have the most prediction powers are:
Average Transaction Balance
Average Transaction Amount
Loan Amount
Average Salary
Days between account creation and loan application
There is not to much surprise here, since for many of these, we have already seen the unusual behaviors that could be related to the loan default, such as the loan amount and days between account creation and loan application.
In this post, I introduced the whole pipeline of an end-to-end machine learning model in a banking application, loan default prediction, with real-world banking dataset Berka. I described the Berka dataset and the relationships between each table. Steps and codes were demonstrated on how to import the dataset into MySQL database and then connect to Python and convert processed records into Pandas DataFrame. Features were extracted and transformed into an array, ready for feeding into machine learning models. As the last step, I fit a Random Forest model using the data, evaluated the model performance, and generated the list of top 5 features that play roles in predicting loan default.
This machine learning pipeline is just a gentle touch of the one application that could be used with the Berka dataset. It could go deeper since there is more useful information hidden in the intricate relationship among tables; it could also go wider since it can be extended to other applications such as credit card and client’s transaction behaviors. But if just focusing on this loan default prediction, there could be three directions to dive further in the future:
Extract more features: Due to the time limit, it is not possible to conduct a thorough study and have a deep understanding of the dataset. There are still many features in the dataset that are unused and a lot of the information has not been fully digested with knowledge in the banking industry.Try other models: Only the Random Forest model is used, but there are many good ones out there, such as Logistic Regression, XGBoost, SVM, or even neural networks. The models can also be improved further by finer tunings on hyperparameters or using ensemble methods such as bagging, boosting, and stacking.Deal with the unbalanced data: It is important to notice this fact that the default loans are only about 10% of the total loans, thus during the training process, the model will favor predicting more negatives than positive results. We have already used the F1 score and ROC AUC instead of just accuracy. However, the performance is still not as good as it could be. In order to solve this problem, other methods such as collecting or resampling more data can be used in the future.
Extract more features: Due to the time limit, it is not possible to conduct a thorough study and have a deep understanding of the dataset. There are still many features in the dataset that are unused and a lot of the information has not been fully digested with knowledge in the banking industry.
Try other models: Only the Random Forest model is used, but there are many good ones out there, such as Logistic Regression, XGBoost, SVM, or even neural networks. The models can also be improved further by finer tunings on hyperparameters or using ensemble methods such as bagging, boosting, and stacking.
Deal with the unbalanced data: It is important to notice this fact that the default loans are only about 10% of the total loans, thus during the training process, the model will favor predicting more negatives than positive results. We have already used the F1 score and ROC AUC instead of just accuracy. However, the performance is still not as good as it could be. In order to solve this problem, other methods such as collecting or resampling more data can be used in the future.
Again, This post is just a hands-on practice building a loan default prediction model from scratch. If you are interested in this topic and want to see some more in-depth work that I accomplished for a client, using optimization to turn their loss into profit using such loan default prediction models, please see my other article here: Loan Default Prediction for Profit Maximization.
Thank you for reading! If you like this article, please follow my channel (really appreciate it 🙏). I will keep writing to share my ideas and projects about data science. Feel free to contact me if you have any questions.
I am a data scientist at Sanofi. I embrace technology and learn new skills every day. You are welcome to reach me from Medium Blog, LinkedIn, or GitHub. My opinions are my own and not the views of my employer.
Please see my other articles:
Loan Default Prediction for Profit Maximization
Time Series Pattern Recognition with Air Quality Sensor Data
Build REST API for Machine Learning Models using Python and Flask-RESTful
Understanding Sigmoid, Logistic, Softmax Functions, and Cross-Entropy Loss (Log Loss) in Classification Problems