Creating a Flask App to Classify Consumer Complaints using Natural Language Processing (Multi-class Classification with TFIDF/Word2Vec) | by Xian Jin Seow | Towards Data Science

In part 1 of this series, I will cover the workflow in obtaining the working model that churns out these classifications. This post will be continued in part 2, which involves building the Flask App of this model. So if you’re solely interested in the Flask portion, click here (still WIP) to head on over to that post!
Without further ado, let’s jump into part 1 of this series.
I am dissatisfied with the current outcome of a dispute that was initiated with Discover Card regarding a single transaction that occurred on XXXX/XXXX/2015 in the amount of {$280.00}. I have corresponded with Discover Card at least four times since XXXX/XXXX/2015 ( which I have enclosed as an attachment to this complaint ). I believe that the credit card issuer has violated consumer protection laws by failing to implement the Special Rule for Credit Card Purchase protection despite overwhelming paperwork evidence submitted by me that shows the merchant has conducted business in bad faith less favorable to the consumer. I have sustained a monetary loss as a result of merchants bad faith and intent. I have patiently utilized the internal Discover Card dispute process over the past three months with the credit card issuer always favoring the merchant ; I have repeatedly submitted irrefutable paperwork evidence that has shown that the merchant has conducted business in bad faith. I have tried in good faith to address my complaint with the merchant and Discover Card but believe that I will not receive a favorable outcome.I suffered a work-related injury in XXXX and am now permanently XXXX. My income has dropped substantially in the past 5 years and I am now on a low fixed income. On XXXX separate occasions from XXXX to XXXX I had zero income and had to use my Sams Club card to purchase food, thus running up debt out of necessity. I am currently in the process of attempting to pay down my Sams Club ( Synchrony Bank ) card, and stopped using the card some time ago. I have always made at least minimum payments and have never missed a payment. Despite this, my interest rate has been unilaterally raised three times in the past two years, and is now at 23.15 %. I called a Sams Club account rep today to file a complaint over the phone, because I am never going to be able to pay down this card when I am paying almost {$50.00} a month in interest and can only afford to pay the minimum payment + {$4.00} or {$5.00} dollars. They would not work with me, which I expected. In my opinion, Synchrony Bank is taking unfair advantage of recent interest rate hikes to gouge customers, especially those who are financially unable to make substantial monthly payments on their accounts. Therefore I am contacting the CFPB to file a complaint through which I might receive some relief.
Did you read read the wall of text above? Chances are you didn’t (because I wouldn’t either). Who has the time to read a whole chunk of text with no headers, no summary and terrible formatting (bold, spacing, etc.) that potentially holds no relevance to you?
If you actually read the texts above, you’d realize they’re actually customer complaints. And as Mr Gates once mentioned:
There is so much we can learn from our customers’ complaints. The best brands always connect with their consumers and make decisions revolving around them (think Paramount, who actually pushed the release of the Sonic movie to a later date and changed the design of our favorite blue hedgehog because of the terrible reviews and the countless memes that ensued).
Understanding your customers is important and all, but it doesn’t change the fact that a wall of text from an angry customer is not only strenuous to read through, but also demotivating. And often times it’s not going to be just one complaint.
Typically, in banks (or any customer service department of large companies), they see thousands of complaints like these on a daily basis and they’re not exactly going to be as pleasantly phrased and formatted as the ones above.
Furthermore, the complaints can be filed into the wrong sections by the consumers themselves, and it is going to be a huge pain to go through every single complaint and direct them to the relevant departments to be dealt with. And needless to say, these slow processing times also lead to slow response times to the customers, leading to your already-angry customers becoming angrier.
Man, if only there were some way to quickly categorize each complaint in order to speed up your reading process.
Well — turns out there is, which is the very purpose of this project. In my project, I developed a model that can correctly classify a customer complaint according to it’s product type, with an accuracy of 83%.
The app utilizes a machine learning algorithm in conjunction with Natural Language Processing (NLP) methodologies to process text to make a prediction on the type of product the complaint is referring to.
This is essentially a supervised (labelled) text multi-class classification problem, of which our goal is to make a prediction (assign to correct category) with a new input (complaint). In the steps below, I will take you through the following:
Data Exploration (Understanding the dataset)
Data (text) Pre-processing and the general steps one might take in an NLP project
Model selection and why Micro-F1 is used as an evaluation metric
Selected model final test and results
In Part 2 of the Series...
Flask App demonstration & how to build one
All the code to this project can be found here at my github. Feel free to drop me a message should you need any clarification.
The data used to train this model is taken from here. And an explanation of what each column means can be found here. It is essentially a labelled dataset provided by the Consumer Financial Protection Bureau (CFPB) about complaints received about financial products and services.
We observe that the dataset contains 1,437,716 rows of data and many, many rows containing null values.
<class 'pandas.core.frame.DataFrame'>RangeIndex: 1437716 entries, 0 to 1437715Data columns (total 18 columns):Date received                   1437716 non-null objectProduct                         1437716 non-null objectSub-product                     1202551 non-null objectIssue                           1437716 non-null objectSub-issue                       887045 non-null objectConsumer complaint narrative    463991 non-null objectCompany public response         530104 non-null objectCompany                         1437716 non-null objectState                           1414039 non-null objectZIP code                        1304830 non-null objectTags                            196144 non-null objectConsumer consent provided?      831624 non-null objectSubmitted via                   1437716 non-null objectDate sent to company            1437716 non-null objectCompany response to consumer    1437715 non-null objectTimely response?                1437716 non-null objectConsumer disputed?              768482 non-null objectComplaint ID                    1437716 non-null int64dtypes: int64(1), object(17)memory usage: 197.4+ MB
However, since we are solely analyzing the complaint itself, we need only the Product and Consumer Complaint Narrative columns.
After only keeping the columns we need, removing the null values and renaming them into something nicer to work with, we are left with just 463,991 rows of data:
<class 'pandas.core.frame.DataFrame'>Int64Index: 463991 entries, 4 to 1437711Data columns (total 2 columns):PRODUCT               463991 non-null objectCONSUMER_COMPLAINT    463991 non-null objectdtypes: object(2)memory usage: 10.6+ MB
Doing a little bit of EDA (exploratory data analysis) on the PRODUCT column, we observe 18 classes with the following distribution:
Some of the classes (products), however, overlap. For instance, Credit card or prepaid card clash with Credit card and Prepaid card. Additionally, there is also a problem with having 463,991 rows to perform analysis on (I doubt your laptop will be able to handle the processing and model training at the later part of this project. Mine couldn’t).
After removing a few product classes, combining some classes together and reducing the number of observations by 90% (the specifics of which can be found in my jupyter notebook on my github), we are left with around 30,000 rows of complaints (of which is manageable for a laptop with 8GB ram).
<class 'pandas.core.frame.DataFrame'>Int64Index: 31942 entries, 72 to 1437623Data columns (total 2 columns):PRODUCT               31942 non-null objectCONSUMER_COMPLAINT    31942 non-null objectdtypes: object(2)memory usage: 748.6+ KB
With the following distribution:
Not too shabby. Now we are ready to do some text pre-processing on our complaint data!
Now that we have our data, we need to go through the cleaning process. For text data sets in general, this often includes:
Removal of punctuation
Removal of stop words (things like “the”, “this”, “what”)
Lowercase
Tokenization
Stemming/lemmatization (reducing words down to their base form by removing suffixes like “-ed”, “-ing”)
If you are dealing with online text and comments, you will require further removal of items like hyperlinks, usernames, and automated messages.
What pre-processing you do varies from project to project, and for this I achieved the best accuracy score doing all of the above except stemming and lemmatization. I considered using bi-grams, but that caused me to consistently run into memory issues, and I had to therefore move forward with only considering 1-gram (standard single-word tokenization).
Below is a sample illustration of punctuation removal, stop-word removal, lower-casing and thereafter tokenization.
To turn words into something a machine learning algorithm can understand and process, we need to do something called vectorization. Put simply, this is the process of turning words into multi-dimensional vectors in such a way that the meaning or context of the word is correlated with where that vector points. In a sense, vectorization allows computers to quantify the meaning of words by mapping similar word meanings to similar vector spaces.
For this application, we want context-based vectorization. This is where Term frequency inverse document frequency vectorization comes in (TF-IDF). This vectorization method looks at the number of times a word appears in a comment relative to the number of times it appears in other comments. Two things result in a higher TF-IDF score:
Higher frequency of the word within the specific complaint being scored.Lower frequency of the word across all other complaints.
Higher frequency of the word within the specific complaint being scored.
Lower frequency of the word across all other complaints.
In a multi-class scenario, it isn’t as simple as Precision, Recall and F1 anymore as per a binary-classification (as with my previous project on Classifying Defaulters). It’s on a whole new level. I recommend referring to Boaz Shmueli’s explanation on evaluation metrics in multi-class classifiers if you need a better understanding on the evaluation metrics used for models in a multi-classification scenario. He does a really good job explaining regarding precision, recall and F1 scoring on both binary and multi-class classification metrics.
As with this mutli-classifcation case, I have decided to use Micro-F1 as an evaluation metric between models for it measures the model’s overall accuracy.
I have decided to use the following models for the multi-class text classification:
Multinomial Naive BayesGaussian Naive BayesLogistic RegressionRandom ForestLinearSVC
Multinomial Naive Bayes
Gaussian Naive Bayes
Logistic Regression
Random Forest
LinearSVC
Each model is cross-validated on a stratified 5-fold split train-validation (of 80% of the whole dataset). The remaining 20% data was held out and untouched to simulate how our final chosen model would perform in real-life scenarios where it would have to deal with inputs it had never seen before.
As always, the code behind how this was done can be found on my jupyter notebook (Section 3 onwards in the notebook) on github.
From our preliminary training results, we see that:
Logistic Regression gives us the highest accuracy of 83%.
83% is a pretty good accuracy. But can we do better? We can’t be sure if it WILL be better, but we can always try and strive for improvement.
Instead of utilizing Vectorization (Tf-idfVectorizer that we used above), we can utilize WordEmbedding on the complaints before feeding them into our classifiers. To understand more about WordEmbeddings, you can find it here.
Some methods such as Word2Vec and spaCy involve pre-trained, multi-gigabyte models that are trained on hundreds of thousands, if not millions of documents and then reduce the meanings of words to a set of a couple hundred numbers. These models are excellent at preserving context and meaning of words in general, but they are slow and large.
I have chosen to use Standford’s GloVe (in increasing complexity) and Google’s Word2Vec for this project.
Compiling all results from the tests, we find that the vanilla TFidfVectorizer does way better than the WordEmbeddings, with the highest accuracy of 83%.
Thus, TfidfVectorizer using Logistic Regression is selected as the text pre-processing and model combination to use for this project.
Remember the 20% data we heldout earlier that we didn’t use in the cross-validation? Here comes the part where we use it as an estimate as to how well our model will perform on unseen data (i.e. how it would perform in real-life scenarios).
The model returns us a whooping 83% accuracy on unseen data! Signifying that it can correctly classify a complaint 83% of the time! Now that we have our model, we can finally see it in action!
To be able to utilize our model in a user-friendly way, I created a Flask app that allows a complaint to be typed in. Thereafter, the app will predict which product category the complaint belongs to. Refer to this post (WIP) to find out how the Flask app is created!
I am dissatisfied with the current outcome of a dispute that was initiated with Discover Card regarding a single transaction that occurred on XXXX/XXXX/2015 in the amount of {$280.00}. I have corresponded with Discover Card at least four times since XXXX/XXXX/2015 ( which I have enclosed as an attachment to this complaint ). I believe that the credit card issuer has violated consumer protection laws by failing to implement the Special Rule for Credit Card Purchase protection despite overwhelming paperwork evidence submitted by me that shows the merchant has conducted business in bad faith less favorable to the consumer. I have sustained a monetary loss as a result of merchants bad faith and intent. I have patiently utilized the internal Discover Card dispute process over the past three months with the credit card issuer always favoring the merchant ; I have repeatedly submitted irrefutable paperwork evidence that has shown that the merchant has conducted business in bad faith. I have tried in good faith to address my complaint with the merchant and Discover Card but believe that I will not receive a favorable outcome.
Remember this horrible wall of text we saw at the beginning of the post? Let’s put our model to the test. Take your time to read it and guess which of these categories it falls under.
Now let’s put our app to the test.
Did you come to the same conclusion as the app did? Now let’s try this with a more difficult complaint, full of typos, caps and millennial lingo.
Omg where is my money?????? AOSINIONSAD WHY DID YOU GUYS TAKE MY MONEY AWAY. SAOIDNSIOADNOIAODNNIOASDNSADNOSDNOASDNI TRANSFERRED IT LAST NIGHT AND I WOKE UP TO NOTHING IN!!!!!! MY BANK ACCOUNT. HELP PLEASE!!!!!!!!!!!!!!!!!!!!!!!!!!!!!I NEED THE MONEY OR ELSE I WILL BE HUNTED BY LOAN SHARKS!!!!!!!!YOU *(#&$) PEOPLE HAVE NO RIGHT TO DO THIS TO ME!!!! I NEED MY MONEY!!!!
We see that the app can handle messy complaints with no errors and correctly classify what product category they fall into as well.
The utilization of this app can bring about a plethora of benefits, from freeing up manpower to increasing efficiency of sifting through customer complaints.
As an augmentation to this app to improve it’s usefulness, we can perhaps look into the following:
Topic SummarizationCurrently, the app only classifies the complaint into the correct product category. While it does file the complaint into it’s correct category, the customer complaint officer would still have to read through the complaint itself. It would be nice for the app to be able to also provide a short abstract of what the complaint is about. This would be an interesting topic to look into and devise how to create this functionality.
Complaint PrioritizationWe can look into also implementing Sentiment Analysis to provide some sort of complaint prioritization. Angrier-sounding complaints (i.e., angry customers) can be given a higher priority, since angry customers tend to provide the most problems. Our app would therefore have an automated sorting for the customer complaint officer as to which complaints need to be urgently attended to.
ask if anyone would like to work on it together, can drop me a text
This has been by no means a short post, and hopefully an informative one in your journey in Data Science. Thank you for taking the time to read this far.
If you learned anything, find this helpful or think it might help someone in this field, do share this post with others!
As always, feel free to hit me up over on my Linkedin or my gmail at xianjinseow92@gmail.com!