FuzzyWuzzy: Fuzzy String Matching in Python | Towards Data Science

If you have dealt with text data before, you know that its issues are the hardest to deal with. There is just no one-size-fits-all solution to text problems and for each dataset, you have to come up with new ways to clean your data. In one of my previous articles, I talked about the worst-case scenario of such problems:
For example, consider this worst-case scenario: you are working on a survey data conducted across the USA and there is a state column for the state of each observation in the dataset. There are 50 states in the USA and imagine all the damn variations of state names people can come up with. You are in even bigger problem if data collectors decide to use abbreviations: CA, ca, Ca, Caliphornia, Californa, Calfornia, calipornia, CAL, CALI, ... Such columns will always be filled with typos, errors, inconsistencies.
The problems related to text often arise because of free-text during data collection. They will be full of typos, inconsistencies, whatever you can name. Of course, the most basic problems can be solved using simple regular expressions or built-in Python functions but for cases like above, which occur very often, you have to arm yourself with more complex tools.
Today’s special is fuzzywuzzy, a package with a very simple API which helps us to calculate string similarity.
Get the notebook and data used in this article from this Kaggle notebook or from this GitHub repo.
To understand string matching, let’s get you up to speed with Minimum Edit Distance. As humans, we have no trouble at all if two or more strings are similar or not. To create this ability in computers, many algorithms were created and almost all of them depend on Minimum Edit Distance.
Minimum Edit Distance (MED) is the least possible amount of steps needed to transition from one string to another. MED is calculated using only 4 operations:
Insertion
Deletion
Substitution
Replacing consecutive characters
Consider these two words: Program and Sonogram:
To get from Program to Sonogram, we need 3 steps:
Add the letter ‘S’ to the beginning of ‘Program’.Substitute ‘P’ with ‘O’.Substitute ‘R’ with ‘N’.
Add the letter ‘S’ to the beginning of ‘Program’.
Substitute ‘P’ with ‘O’.
Substitute ‘R’ with ‘N’.
As I said, there are many algorithms to calculate MED:
Damerau-Levenshtein
Levenshtein
Hamming
Jaro Distance
Also, there are packages that use these algorithms: nltk, fuzzywuzzy, textdistance, difflib, ...
In this article, we will only cover fuzzywuzzy.
Even though the basic installation can be done easily with pip, there are some other options or caveats to fuzzwuzzy's installation:
Using PIP via PyPI (standard):
pip install fuzzywuzzy
The above method installs the default up-to-date version of the package. At first, I installed it using this method. But whenever I imported it, it started giving a warning saying that the package itself is very slow and I should install python-Levenshtein package for more speed. If you hate warnings in your Jupyter Notebook like me, here is how you can install extra dependencies:
Directly install python-Levenshtein:
pip install python-Levenshtein
or
pip install fuzzywuzzy[speedup]
Warning for Windows users: if you don’t have Microsoft Visual Studio build tools installed, installing python-Levenshtein fails. You can download MVS Build Tools from here.
To get started with fuzzywuzzy, we first import fuzz sub-module:
from fuzzywuzzy import fuzz
In this sub-module, there are 5 functions for different methods of comparison between 2 strings. The most flexible and best one for everyday use is WRatio (Weighted Ratio) function:
Here, we are comparing ‘Python’ to ‘Cython’. The output returns a percentage between 0 and 100, 0 being not similar at all and 100 being identical:
All the functions of fuzzywuzzy are case-insensitive:
WRatio is also very good for partial strings with different orderings:
Apart from WRatio, there are 4 other functions to compute string similarity:
fuzz.ratio
fuzz.partial_ratio
fuzz.token_sort_ratio
fuzz.token_set_ratio
fuzz.ratio is perfect for strings with similar lengths and order:
For strings with differing lengths, it is better to use `fuzz.patial_ratio’:
If the strings have the same meaning but their order is different, use fuzz.token_sort_ratio:
For more edge cases, there is fuzz.token_set_ratio:
As you see, these 5 functions are full of caveats. Their comparison is a whole another topic so I am leaving you a link to the article written by the package creators which explains their difference beautifully.
I think you already saw that WRatio function gives the middle ground for all the functions of fuzzywuzzy. For many edge cases and different issues, it is best to use WRatio for best results.
Now we have some understanding fuzzywuzzy's different functions, we can move on to more complex problems. With real-life data, most of the time you have to find the most similar value to your string from a list of options. Consider this example:
We have to find the best matches to Mercedez-Benz to replace them with the correct spelling of the cars. We can loop over each value but such a process could take too long if there are millions of options to choose from. Since this operation is so commonly used, fuzzywuzzy provides us with a helpful sub-module:
from fuzzywuzzy import process
With this sub-module, you can extract the best matches to your string from a sequence of strings. Let’s solve our initial problem:
The parameters of interest in process.extract are query, choices and limit. This function computes the similarity of strings given in query from a sequence of options given in choices and returns a list of tuples. limit controls the number of tuples to return. Each of these tuples contains two elements, the first one is the matching string and the second one is the similarity score.
Under the hood, process.extract uses the default WRatio function. However, depending on your case and knowing the differences between the 5 functions you can change the scoring function with scorer:
If you have many options, it is best to stick with WRatio because it is the most flexible.
In the process module, there are other functions that perform a similar operation. process.extractOne returns only one output which contains the string with the highest matching score:
Now we are ready to tackle a real-world problem. I will load the raw data to practice:
cars.shape(8504, 4)
I used this dataset in one of my personal projects and the task was to correct the spelling of each vehicle make and model according to the correct values given in another file:
After loading the pickle file, make_model is now a dictionary containing the correct spelling of each car make as keys and the correct spelling of models under each key.
For example, let’s see the spellings of makes and models of Toyota cars:
Now, let’s subset the raw data for Toyota cars:
>>> cars[cars['vehicle_make'] == 'TOYOTA']
The dataset contains up to a hundred unique car makes like Audi, Bentley, BMW and each one contains several models that are full of edge cases. We cannot just convert each one to title case or lower case. We also don’t know if these contain any spelling errors or inconsistencies and visual search is not an option for such big datasets. There are also some cases where make labels with more than one word divide the name with a space while others with a dash. If you have this many inconsistencies and there is not a clear pattern, use string matching.
Let’s start by cleaning up make labels. For comparison, here are the make labels in both datasets:
I think the differences are obvious. We will use process.extract to match each makes with the correct spelling:
As you see, the make labels which exist in the make_model got converted into their correct spelling. Now, it is time for model labels:
The last two code snippets were a little hairy. To fully understand how they are working, you should get some practice on process.extract.
There you go! If you did not know string matching, the task would have been impossible and even Regular Expressions would not have been able to help you.
Read more articles related to the topic: