An Overview of Building a Merchant Name Cleaning Engine with SequenceMatcher and spaCy | by Cheng | Towards Data Science

Merchant names cleaning can be a quite challenging problem. As different bank provides different quality of transaction data, there is not a very mature way to clean the data. Commonly, merchant names cleaning can be classified as a Named Entity Recognition (NER) task and be solved in a similar way as an entity extraction problem.
For a FinTech company, a merchant names cleaning step is important because developers need to utilize these cleaned merchant names out of originally messy transaction data to generate proper transaction categorization to deliver a better customer experience in terms of managing personal finance.
I found this topic very interesting and I have been searching resources for weeks to write up this overview with my own basic solution to this. Therefore, I hope some of my thoughts on this topic can be helpful for the readers to better solve this merchant names cleaning problem.
If you are searching for more readings on this topic, you’re welcome to check the Reference List at the end of this article.
For a basic merchant names cleaning engine, I plan to divide it into three layers:
1st Layer: Remove special characters & numbers and convert cases.
2nd Layer: Return name matches based on similarity scores.
3rd Layer: Train a spaCy model to detect patterns and clean the inputs.
Toolkit for this project includes regular expression operations in python, FuzzyWuzzy/SequenceMatcher (library) and some knowledge in the spaCy model algorithm.
As the reading progresses, I will also share some related readings I found helpful.
Removing all special characters & numbers will be the first step for the project. It’s useful because special characters and numbers often adds more complexity when we try to find merchant name matches and compute similarity scores. Completely removing all special characters and numbers might be a little aggressive. However, considering a dataset of thousands of raw merchant names, you may find that the majority of special characters and numbers can be removed without affecting any keyword in a merchant name.
Special characters and numbers removal can be efficiently done with the help of Python Re library. Some pre-requisite knowledge is regular expression in Python.
medium.com
medium.com
By successfully completing the above step, you now have a dataset of merchant names containing only the alphabet letters. However, you might still find that some merchant names are in different cases such as “AMAZON”, “amazon” or “Amazon”. To convert the cases, some useful string functions can be found in the following article:
towardsdatascience.com
For this layer, our main goal is to calculate a similarity score table and return the matched names with the top 3 largest similarity scores. This is a useful method to clean merchant names assuming that you already have a catalogue of names to match and also the raw inputs are non-messy.
FuzzyWuzzy is a Python library that uses Levenshtein Distance to calculate the differences between sequences in a simple-to-use package.
github.com
Some practical examples in use of FuzzyWuzzy are as followings:
Simple Ratio
>>> fuzz.ratio("this is a test", "this is a test!")    97
Partial Ratio
>>> fuzz.partial_ratio("this is a test", "this is a test!")    100
Token Sort Ratio
>>> fuzz.ratio("fuzzy wuzzy was a bear", "wuzzy fuzzy was a bear")    91>>> fuzz.token_sort_ratio("fuzzy wuzzy was a bear", "wuzzy fuzzy was a bear")    100
Token Set Ratio
>>> fuzz.token_sort_ratio("fuzzy was a bear", "fuzzy fuzzy was a bear")    84>>> fuzz.token_set_ratio("fuzzy was a bear", "fuzzy fuzzy was a bear")    100
towardsdatascience.com
towardsdatascience.com
Alternatively, SequenceMatcher is also a great tool often used to compute the similarity between inputs.
The basic idea is to find the longest contiguous matching subsequence that contains no “junk” elements. The same idea is then applied recursively to the pieces of the sequences to the left and to the right of the matching subsequence. This does not yield minimal edit sequences, but does tend to yield matches that “look right” to people.
>>> s = SequenceMatcher(lambda x: x == " ", "private Thread currentThread;", "private volatile Thread currentThread;")    >>> .ratio() returns a float in [0, 1], measuring the "similarity" of the sequences.  As a rule of thumb, a .ratio() value over 0.6 means the sequences are close matches>>> print(round(s.ratio(), 3))    0.866
towardsdatascience.com
For me, I choose the SequenceMatcher as the metrics of evaluating similarity. The process will be similar if you choose the FuzzyWuzzy library.
# define a function to calculate similarity between input sequencesdef similarity_map(word1, word2):        seq = difflib.SequenceMatcher(None,word1,word2)    d = seq.ratio()        return d
The above customized function takes two sequences as input and returns a ratio value of similarity score.
To further compute a table of similarity scores, I made a dataset that has raw merchant names as row indexes and a catalogue of cleaned merchant names as column names. By running the code cell below, it will generate a table of similarity scores for each pair of row index and column name.
# prepare a sample datasetdf = pd.DataFrame(data, index =['amazon co com', 'www netflix com', 'paypal payment', 'apple com bill', 'google play', 'facebook ads'],columns = ['amazon', 'netflix', 'paypal', 'apple', 'google', 'facebook'])   # print the data dffrom tqdm import tqdmfor i in tqdm(range(6)):    for j in range(6):        df.loc[df.index[i], df.columns[j]] = similarity_map(str(df.index[i]), str(df.columns[j]))        df.head()
Once you completed running the above cell, you should have a table of similarity scores looks like the following:
Based on the above table, we can further analyze insights by return merchant names that have top 3 highest similarity scores for each row.
Write a function top that takes above dataset as input and returns a dataset of top 3 names along with their similarity scores.
similarity = df.reset_index()similarity.head()def top(x):    x.set_index('index', inplace=True)    df = pd.DataFrame({'Max1Name':[],'Max2Name':[],'Max3Name':[],'Max1Value':[],'Max2Value':[],'Max3Value':[]})    df.index.name='index'    df.loc[x.index.values[0],['Max1Name', 'Max2Name', 'Max3Name']] = x.sum().nlargest(3).index.tolist()    df.loc[x.index.values[0],['Max1Value', 'Max2Value', 'Max3Value']] = x.sum().nlargest(3).values    return dftest = similarity.groupby('index').apply(top).reset_index(level=1, drop=True).reset_index()test.head()
stackoverflow.com
stackoverflow.com
By successfully implementing the above code cell, you should have a return dataset like the following:
Although this is just a test on a sample dataset, we may still find this method useful if we have non-messy inputs along with a catalogue of cleaned merchant names.
However, this method may not perform well for more complex merchant inputs. For instance, a merchant name paypal * card payment on booking.com toronto will likely return a low similarity score (less than 0.5) with respect to either PayPal or Booking.
In this case, it requires a more advanced method to detect the location of the “real” merchant name we want.
By completing first two layers, we are able to solve some of the merchant names cleaning problems such as names of mis-spelling, different cases, missing characters/spaces and even some of the non-messy merchant inputs by simply returning a similarity score table.
However, we are actually still in the phase of working with a rule-based cleaning engine, which means so far we still haven’t learned from the data. Furthermore, even by use of a typical machine learning model, the training phase still requires a large amount of time to perform feature engineering in creating more informative features.
... potentially informative transaction-level features such as dollar amount and category, while also generating word-level natural language features such as word position within the label (e.g., 1st, 2nd), word length, proportion of vowels, consonants, and alphanumeric characters, among others.
CleanMachine: Financial transaction label translation for wallet.AI
Therefore, I researched on how to use a deep learning model to create the cleaning engine. The advantage of using a deep learning model is that we are able to “skip” the feature engineering step and let the model itself detect any insightful patterns from the inputs.
A free short course on spaCy can be found as following:
course.spacy.io
According to the spaCy Guide:
spaCy is a library for advanced Natural Language Processing in Python and Cython. It’s built on the very latest research, and was designed from day one to be used in real products. spaCy comes with pre-trained statistical models and word vectors, and currently supports tokenization for 60+ languages.
It features state-of-the-art speed, convolutional neural network models for tagging, parsing and named entity recognition and easy deep learning integration. It’s commercial open-source software, released under the MIT license.
github.com
Since the merchant names cleaning problem can be classified under the topic of named entity recognition(NER), I feel confident that a spaCy model will have a good performance by feeding in a set of representative input data.
To train a spaCy model, we don’t just want it to memorize our examples — we want it to come up with a theory that can be generalized across other examples.
Therefore, the training data should always be representative of the data we want to process. For our project, we may want to select training data from different types of merchant names. Eventually, our training data will be in form of an entity list like the following:
TRAIN_DATA = [('Amazon co ca', {'entities': [(0, 6, 'BRD')]}),('AMZNMKTPLACE AMAZON CO', {'entities': [(13, 19, 'BRD')]}),('APPLE COM BILL', {'entities': [(0, 5, 'BRD')]}),('BOOKING COM New York City', {'entities': [(0, 7, 'BRD')]}),('STARBUCKS Vancouver', {'entities': [(0, 9, 'BRD')]}),('Uber BV', {'entities': [(0, 4, 'BRD')]}),('Hotel on Booking com Toronto', {'entities': [(9, 16, 'BRD')]}),('UBER com', {'entities': [(0, 4, 'BRD')]}),('Netflix com', {'entities': [(0, 7, 'BRD')]})]]
The training data I choose is just a sample. The model can take more complex inputs. However, it can be a little boring to annotate a long list of merchant names. I would like to recommend another data labeling tool UBIAI to complete this task, as it supports output in a spaCy format or even in an Amazon Comprehend format.
ubiai.tools
medium.com
medium.com
It may require some experience on how to select a representative data input. As you practice more and observe the way how a spaCy model learns, it will become clearer that “representative” probably means “different locations”. It’s the reason why we need to provide an entity start & end index in the input data, because it can help the model to learn the patterns from different contexts.
If a model is often trained with the location of the first word being a merchant name (Amazon ca), then it tends to believe that a merchant name only locates at the beginning of an input. This will likely cause a bias and result in a wrong prediction for input such as “Music Spotify” because “Spotify” happens to be the second word.
However, it’s also important to include various merchant names in the input. Just be aware that we don’t want our model to merely memorize them.
Once you have finalized tuning your training data, the remaining process will almost be automated.
import spacyimport randomdef train_spacy(data,iterations):    TRAIN_DATA = data    nlp = spacy.blank('en')  # create blank Language class    # create the built-in pipeline components and add them to the pipeline    # nlp.create_pipe works for built-ins that are registered with spaCy    if 'ner' not in nlp.pipe_names:        ner = nlp.create_pipe('ner')        nlp.add_pipe(ner, last=True)# add labels    for _, annotations in TRAIN_DATA:         for ent in annotations.get('entities'):            ner.add_label(ent[2])# get names of other pipes to disable them during training    other_pipes = [pipe for pipe in nlp.pipe_names if pipe != 'ner']    with nlp.disable_pipes(*other_pipes):  # only train NER        optimizer = nlp.begin_training()        for itn in range(iterations):            print("Statring iteration " + str(itn))            random.shuffle(TRAIN_DATA)            losses = {}            for text, annotations in TRAIN_DATA:                nlp.update(                    [text],  # batch of texts                    [annotations],  # batch of annotations                    drop=0.2,  # dropout - make it harder to memorise data                    sgd=optimizer,  # callable to update weights                    losses=losses)            print(losses)    return nlpprdnlp = train_spacy(TRAIN_DATA, 20)# Save our trained Modelmodelfile = input("Enter your Model Name: ")prdnlp.to_disk(modelfile)#Test your texttest_text = input("Enter your testing text: ")doc = prdnlp(test_text)for ent in doc.ents:    print(ent.text, ent.start_char, ent.end_char, ent.label_)
Above code is from the following Medium article, as I found it very helpful and it inspired me to test spaCy on a merchant names cleaning problem.
manivannan-ai.medium.com
www.machinelearningplus.com
By successfully completing the training step, we can monitor the model progress by checking its loss values.
Statring iteration 0{'ner': 18.696674078702927}Statring iteration 1{'ner': 10.93641816265881}Statring iteration 2{'ner': 7.63046314753592}Statring iteration 3{'ner': 1.8599222962139454}Statring iteration 4{'ner': 0.29048295595632395}Statring iteration 5{'ner': 0.0009769084971516626}
The model is then shown the unlabelled text and will make a prediction. Because we know the correct answer, we can give the model feedback on its prediction in the form of an error gradient of the loss function that calculates the difference between the training example and the expected output. The greater the difference, the more significant the gradient and the updates to our model.
To test your model, you can run the code cell below:
#Test your texttest_text = input("Enter your testing text: ")doc = prdnlp(test_text)for ent in doc.ents:    print(ent.text, ent.start_char, ent.end_char, ent.label_)
For instance, we can use “paypal payment” as input and test the model if it can detect “paypal” as the correct brand name.
The model did a great job considering PayPal did not appear in the training input.
This also concludes my project of building a merchant names cleaning engine with a spaCy model.
First of all, thank you for your time reading this long article and I sincerely hope you found it helpful~
I started from an introduction on why merchant names cleaning is important.
Then, I divided the cleaning engine into three layers.
Finally, for each layer, I explained the inputs & outputs as well as why each layer is necessary to have.
As an overview on this challenging problem, I didn’t fully expect to give a perfect solution to this. But I believe that it can be helpful to share my thoughts and any useful readings I found during my research.
Hence, I hope you enjoyed reading this article. Meanwhile, I added a Reference List section at the end of this article in case you are interested in learning more about this topic.
Thank you~
A little update on the spaCy model training results with around 1000 lines of input data. (1000 annotations)
I have set the model to 50 iterations each round of training and trained it for 10 rounds to see how the training loss varies. And it seems if we preprocess the data in a correct way, we should be able to reach a low & consistent training loss every time through 50 iterations, plotted as the following:
🤔 However, is there still any way to make our spaCy model more interactive? Would it be more user-friendly if we can integrate spaCy model with Streamlit into a web interface?
I was inspired by the spaCy documentation like the following:
It seems that spaCy supports many fancy tools including Streamlit. Therefore, I decided to try to integrate spaCy and Streamlit together for a web app.
If you would like more knowledge on Streamlit, the following article is a good start.
towardsdatascience.com
Since previously we have defined our spaCy training function as train_spacy, the remaining work will be within 10 lines of code. It’s also the reason why I think I should give an update under the same article 🤗
Assuming we have also prepared our input annotation list in the format of previous TRAIN_DATA.
A web app interface’s code in Streamlit will look like the following:
import pandas as pdimport numpy as np import randomimport spacyimport reimport warningsimport streamlit as st warnings.filterwarnings('ignore') # ignore warnings nlp = train_spacy(TRAIN_DATA, 50) # number of iterations set as 50# Save our trained Model # Once you obtained a trained model, you can switch to load a model for merchant name cleaningmodelfile = input("Enter your Model Name: ")nlp.to_disk(modelfile)# Load our saved Model # Load your model to clean a user input instead of training a new model once again when there is a new input# nlp = spacy.load(modelfile/) # path to the saved file foldertext = 'Amazon/ca' # default text input on web interfacest.title('Merchant Names Cleaning App') # web app title nameuser_input = st.text_input("Text", text) # input text placedoc = nlp(user_input) for ent in doc.ents:        st.write('Text:', ent.text) # display model output     st.write('Label:', ent.label_) # display model output
A successful web interface will look like the following:
Hope you enjoyed the reading!
[1] Cleaning up business names from Reddit
[2] Improving the classification of your transaction data with Machine Learning
[3] Making sense of messy bank data
[4] Standardizing +113 million Merchant Names in Financial Services with Greenplum Hadoop
[5] String Matching With FuzzyWuzzy
[6] Natural Language Processing for Fuzzy String Matching with Python
[7] An Ensemble Approach to Large-Scale Fuzzy Name Matching
[8] Hybrid Fuzzy Name Matching
[9] CleanMachine: Financial transaction label translation for wallet.AI
[10] Deep Learning Magic: Small Business Type
[11] Merchant Name Detection using Pytorch
[12] Categorize Banking Transaction Data
[13] Training spaCy’s Statistical Models
[14] How to Train spaCy to Autodetect New Entities (NER) [Complete Guide]
[15] Build a custom entity recognizer using Amazon Comprehend
[16] Developing NER models with Amazon SageMaker Ground Truth and Amazon Comprehend
[17] UBIAI Documentation
[18] How to Automate Job Searches Using Named Entity Recognition — Part 1
[19] Building a Job Entity Recognizer Using Amazon Comprehend
[20] Streamlit and spaCy: Create an App to Predict Sentiment and Word Similarities with Minimal Domain Knowledge