Natural Language Processing for Loan Risk
By: Ty Mick
Tags: NLP, Word Vectors, Neural Networks, Keras, Spacy

The story so far

A few months ago, I built a neural network regression model to predict loan risk, training it with a public dataset from LendingClub. Then I built a public API with Flask to serve the model’s predictions.

Then last month, I decided to put my model to the test and found out that my model can pick grade A loans better than LendingClub!

But I’m not done. Now that I’ve learned the fundamentals of natural language processing (I highly recommend Kaggle’s course on the subject), I’m going to see if I can eke out a bit more predictive power using a couple of freeform text fields in the dataset: title and desc (description).

This dataset includes 1,110,171 loans.

5 rows × 69 columns

This post, like its predecessors, was adapted from a Jupyter Notebook, so feel free to fork my notebook on Kaggle or GitHub if you’d like to follow along.

Exploratory data analysis

There isn’t too much exploratory data analysis left to do after how thoroughly I cleaned the data in my first post, but I do have a few quick questions about the title and desc fields I’d like to answer before I move on.

How many loans use each field?

Have these fields always been included in the loan application?

What is the typical length of each field (in number of words)?

If the most frequent desc value is empty (or maybe just whitespace), perhaps I need to convert all empty or whitespace-only values to NaN before continuing.

Thankfully that didn’t remove too many values, but this “Borrower added on [date]” deal worries me now. I’ll deal with that a little later.

`title` is used in 98% of loans in the dataset.

`desc` is used in 6% of loans in the dataset.

The title "Debt consolidation" is used in 51% of loans.

These fields may not be as useful as I had previously thought. Even though there are 35,860 unique titles used across the dataset, 51% of them just use “Debt consolidation”. Maybe the titles are more descriptive in the other 49%?

And the desc field is only used with 6% of loans.

Now to check and see when these fields were introduced.

Total date range:

min 2012-08-01

max 2018-12-01

Name: issue_d, dtype: datetime64[ns] `title` date range:

min 2012-08-01

max 2018-12-01

Name: issue_d, dtype: datetime64[ns] `desc` date range:

min 2012-08-01

max 2016-07-01

Name: issue_d, dtype: datetime64[ns]

Neither of these fields were introduced late, but they may have stopped using the desc field for the last two years of the database.

Now I’ll take a closer look at values in these fields.

Debt consolidation 573992

Credit card refinancing 214423

Home improvement 64028

Other 56166

Major purchase 20734

Medical expenses 11454

Debt Consolidation 10638

Business 10142

Car financing 9660

Moving and relocation 6806

Vacation 6707

Home buying 5097

Consolidation 4069

debt consolidation 3310

Credit Card Consolidation 1607

consolidation 1538

Debt Consolidation Loan 1265

Consolidation Loan 1260

Personal Loan 1040

Credit Card Refinance 1020

Home Improvement 1016

Credit Card Payoff 991

Consolidate 947

Green loan 626

Loan 621

...

House Buying Consolidation 1

Credit Card Deby 1

Crdit cards 1

"CCC" 1

Loan to Moving & Relocation Expense 1

BILL PAYMENT 1

creit card pay off 1

Auto Repair & Debt Consolidation 1

BMW 2004 1

Moving Expenses - STL to PHX 1

Pay off Bills 1

Room addition 1

Optimistic 1

Consolid_loan2 1

ASSISTANCE NEEDED 1

My bail out 1

myfirstloan 1

second home 1

Just consolidating credit cards 1

Financially Sound Loan 1

refinance loans and home improvements 1

credit cart refincition 1

Managable Repayment Plan 1

ccdebit 1

Project Pay Off Debt 1

Name: title, Length: 35863, dtype: int64

Interesting. It seems like there’s plenty of variety in loan titles in the other 49%. A lot of them seem to directly correspond to the purpose categorical field, but not so many as to make this field useless, I think.

Side note: I discovered at one point when perusing this column that someone entered the Konami Code as the title of their loan application, and their inclusion in this dataset means that the code apparently worked for them — they got the loan.

Borrower added on 03/17/14 > Debt consolidation<br> 9

Borrower added on 01/15/14 > Debt consolidation<br> 7

Borrower added on 02/19/14 > Debt consolidation<br> 7

Borrower added on 03/10/14 > Debt consolidation<br> 7

Borrower added on 01/29/14 > Debt consolidation<br> 7

..

Borrower added on 01/14/13 > Credit Card consolidation<br> 1

Borrower added on 03/14/14 > Debts consolidation and cash for minor improvements on condominium<br> 1

Borrower added on 03/02/14 > I lost a house and need to pay taxes nd have credit card debt thatI already pay $350 a month on and it goes nowhere.<br> 1

Borrower added on 04/09/13 > I want to put in a conscious effort in eliminating my debt by converting high interest cards to a fixed payment that can be effectively managed by me.<br> 1

Borrower added on 09/18/12 > Want to become debt free, because of several circumstances and going back to school I got into debt. I want to pay for what I have purchased without it having an effect on my credit. That is why I want to consolidate my debt and become debt free!<br> 1

Name: desc, Length: 70925, dtype: int64

Do all these descriptions start with “Borrower added on [date]”?

71,858 loan descriptions begin with that pattern. (85 do not.)

Well, now I need to check those other 85.

Debt Consolidation 2

I would like to pay off 3 different credit cards, at 12%, 17% and 22% (after initial 0% period is up). It would be great to have everything under one loan, making it easier to pay off. Also, once I've paid off or down the loan, I can start looking into buying a house. 1

loan will be used for paying off credit card. 1

This loan will be used to consolidate high interest credit card debt. Over the course of this past year my wife and I had our first child, purchase a home and received a large bonus from work. With the new home and the child on the way I chose to spread my tax withholdings on the bonus to all checks received in 2008 this caused my monthly income to fall by $1500. This in combination with an unexpected additional down payment for our home of $17,000 with only a weeks notice we were force to dip into our Credit Cards for the past several months. Starting January 1, 2009 I will be able to readjust my tax withholding and start to pay off the Credit Card debt we have racked up. This loan will help lower the interest rate during the repayment period and give one central place for payment. My wife and I have not missed a payment or been late for the past 5 years. My fico score is 670 mainly due to several low limit credit cards near their max. I manage the international devision of a software company and my wife is a kindergarten teacher, combined we make 140K a year. Thank you for your consideration and I look forward to working with you. 1

to pay off different credit cards to consolidate my debt, so I can have just one monthly payment. 1

..

Hello, I would like to consolidate my debt into a lower more convenient payment. I have a very stable career of more than 20 years with the same company. My community is in a part of the country that made it through the last few years basically unscathed and has a very promising future.<br>Thank You<br> 1

consolidate my debt 1

I am looking to pay off my credit card debts. 1

This loan is to help me payoff my credit card debt. I've done what I can to negotiate lower rates, but the interest is killing me and my monthly payments are basically just taking care of interest. Paying them off will give me the fresh start I need on my way to financial independence. Thank you. 1

I have been in business for a year and want to eliminate some personal debt and use the remainder of the loan to take care of business expenses. Also lessening the number of trade lines I have open puts me in a better position to pursue business loans since it will be based on my personal credit. A detailed report can be created to show where exactly the funds will go and this can be provided at any time during the course of the loan. 1

Name: desc, Length: 84, dtype: int64

It looks like the borrower may be able to add information to the description at different points in time. I should check and see if any of those dates come after the actual issue date of the loan.

2014-01-01 00:00:00 – Borrower added on 01/08/14 > I am tired of making monthly payments and getting nowhere. With your help, except for my mortgage, I intend to be completely debt free by 12/31/2016.<br>

2014-01-01 00:00:00 – Borrower added on 01/08/14 > We have been engaged for 2 1/2yrs and wanted to bring out blended family together as one. We are set to get married on 03/22/14 and we are paying for it on our own. We saved the majority of the budget unfortunately there were a few unexpected cost that we still need help with.<br>

2014-01-01 00:00:00 – Borrower added on 01/06/14 > I am getting married 04/05/2014 and I want to have a cushion for expenses just in case.<br>

2014-01-01 00:00:00 – BR called in to push payment date to 09/19/14 because of not having the exact amount of funds in their bank account. Payment was processing. Was able to cancel. It is within grace period.

2014-01-01 00:00:00 – Borrower added on 01/01/14 > This loan is to consolidate my credit cards debt. I made one year this past 11/28/2013 at my current job. I considered to have job security because I'm a good employee. I make all may credit cards payments on time.<br>

2013-05-01 00:00:00 – Borrower added on 04/27/13 > My father passed away 05/12/2012 and I had to pay for the funeral. My mother could not afford it. He was not ill so I could not have planned it. I paid with what I had in my savings and the rest I had to pay with my credit cards. I would like to pay off the CC & pay one monthly payment.<br><br> Borrower added on 04/27/13 > My paerents own the house so I do not pay rent. The utilities, insurance and taxes, etc my mother pays. She can afford that. I help when needed.<br>

2013-02-01 00:00:00 – Borrower added on 02/10/13 > I am getting married in a week (02/17/2013) and have made some large purchases across my credit cards. I would like to consolidate all of my debt with this low rate loan.<br><br> Borrower added on 02/10/13 > I will be getting married in a week (02/17/13) and have had to make some large purchases on my CC. I am financially sound otherwise with low debt obligations.<br>

2012-12-01 00:00:00 – Borrower added on 12/10/12 > Approximately 1 year ago I had a highefficency furnace /AC installed. The installing Co. used GECRB to get me a loan. If I payoff the loan within one year, I pay no interest. The interest rate if not payed by 12/23/2012 is 26.99%. A 6.62% rate sounds a lot better.<br>

2012-11-01 00:00:00 – Borrower added on 11/19/12 > Looking to finish off consolidating the rest of my bills and lower my payments on my exsisting loan. Thanks!!!<br><br> Borrower added on 11/20/12 > Thanks again for everyone who has invested thus far. With this loan it will give me the ability to have only one payment monthly besides utilities and I will be almost debt free by my wedding date of 12/13/14!! Thanks again everyone!<br>

2012-10-01 00:00:00 – Borrower added on 10/22/12 > Need money by 10/26/2012 to purchase property on discounted APR.<br>

Good, all of the dates that come after the month the loan is issued only come up because the borrower is talking about a future event.

Now to clean these desc values up a bit I’m going to remove the Borrower added on [date] > s and the <br> s, since those don’t add value to the description content.

Imputing missing values

Since only 2% of loans in this set are missing a title, and since most titles simply copy the loan’s purpose , I’m going to impute missing titles with their loan’s purpose.

Since only 6% of loans use a description, I’ll just impute missing descriptions with an empty string. I’m going to wait and include that as a pipeline step a little later, though.

Optimizing data types

I’d really love to get right to the fun part, converting these text fields into document vectors, but I ran into a problem the first several times I tried doing so. Manually adding two sets of 300-dimensional vectors to this 1,110,171-row DataFrame caused its size in memory to skyrocket, exhausting the 16GB Kaggle gives me.

My first attempt to fix this was optimizing my data types, which still didn’t solve the problem on its own, but it’s a worthwhile step to take anyway.

After removing the issue_d column, which is no longer needed, the dataset contains five types of data: float, integer, ordinal, (unordered) categorical, and text.

Reduced DataFrame size in memory by 69%.

Creating document vectors

Now the fun part. Wrapping my spaCy document vector function in a scikit-learn FunctionTransformer turned out to be the secret that kept this process within memory limits. Scikit-learn must just be way better optimized than whatever manual process I was using (go figure).

Building the pipeline

First, the transformer. I’ll use scikit-learn’s ColumnTransformer to apply different transformations to different kinds of data.

This model itself will be identical to my previous model, but I’ll use Keras callbacks and a tqdm progress bar to make the training logs much more concise.

Evaluating the model

/opt/conda/lib/python3.7/site-packages/pandas/core/strings.py:2001: UserWarning: This pattern has match groups. To actually get the groups, use str.extract.

return func(self, *args, **kwargs)

[Pipeline] ....... (step 1 of 3) Processing nlp_imputer, total= 0.4s

[Pipeline] .... (step 2 of 3) Processing nlp_vectorizer, total= 1.2min

[Pipeline] ........ (step 3 of 3) Processing nlp_scaler, total= 8.6s

[ColumnTransformer] ...... (1 of 7) Processing nlp_cols, total= 1.3min

[ColumnTransformer] .... (2 of 7) Processing emp_length, total= 0.2s

[ColumnTransformer] .......... (3 of 7) Processing term, total= 0.3s

[ColumnTransformer] (4 of 7) Processing home_ownership, total= 0.3s

[ColumnTransformer] ....... (5 of 7) Processing purpose, total= 0.3s

[ColumnTransformer] (6 of 7) Processing application_type, total= 0.3s

[ColumnTransformer] ..... (7 of 7) Processing remainder, total= 1.3s

Training model: 100%|██████████| 100/100 [23:41<00:00, 14.22s/epoch]

Final metrics: loss: 0.02365 - val_loss: 0.02360

Well, it didn’t overfit, but this model performed a bit worse than my original, which had settled around a loss of 0.0231. I bet the desc feature is getting in the way—zeroes spanning 300 columns of the input data on 94% of the rows is probably quite confusing to the model. I’ll see what happens if I repeat the process while leaving desc out (making the title vectors the only new feature of this model compared to my original).

/opt/conda/lib/python3.7/site-packages/pandas/core/strings.py:2001: UserWarning: This pattern has match groups. To actually get the groups, use str.extract.

return func(self, *args, **kwargs)

[Pipeline] ....... (step 1 of 3) Processing nlp_imputer, total= 0.1s

[Pipeline] .... (step 2 of 3) Processing nlp_vectorizer, total= 41.3s

[Pipeline] ........ (step 3 of 3) Processing nlp_scaler, total= 4.6s

[ColumnTransformer] ...... (1 of 7) Processing nlp_cols, total= 45.9s

[ColumnTransformer] .... (2 of 7) Processing emp_length, total= 0.2s

[ColumnTransformer] .......... (3 of 7) Processing term, total= 0.3s

[ColumnTransformer] (4 of 7) Processing home_ownership, total= 0.3s

[ColumnTransformer] ....... (5 of 7) Processing purpose, total= 0.3s

[ColumnTransformer] (6 of 7) Processing application_type, total= 0.3s

[ColumnTransformer] ..... (7 of 7) Processing remainder, total= 1.1s

Training model: 100%|██████████| 100/100 [22:26<00:00, 13.46s/epoch]

Final metrics: loss: 0.02396 - val_loss: 0.02451

Wow, still not good enough to beat my original model. Just for kicks, I also tried additional runs where I trained for 1,000 epochs, and others where I increased the numbers of nodes in the first two dense layers to 128 and 64. And I tried decreasing the batch size to 64. But still, none of these beat my original model. I suppose these text features have no predictive quality to them in terms of loan outcomes. Interesting.

Next steps

If adding these two features decreased predictive capability, then perhaps some of the other variables I was already using are doing the same thing. I should try using some of scikit-learn’s feature selection methods to reduce the dimensionality of the input data.

A more efficient method of hyperparameter optimization would be pretty useful as well. I should give AutoKeras a shot.