How fuzzy matching improve your NLP model | by Edward Ma | Towards Data Science

One of the challenge when dealing with NLP tasks is text fuzzy matching alignment. You can still build your NLP model when skipping this text process text but the trade-off is you may not achieve good result. Someone may argue that there is not necessary to have preprocessing when using deep learning. From my experience, having proper text preprocessing can improve your model at least 3% ~ 5%.
So what kind of scenario we need to deal with text fuzzy match and what can we do? If input source (i.e. text) cannot be guaranteed that it is 100% correct or there are alias name such as “New York” and “New York, US”.
For case 1, input can be come from user and the typical examples is searching. Imagining you are building Google Search engine, you have to validate the input rather using input blindly.
Another possible scenario is the input is the output of other models. For example, input comes is the output of Optical Character Engine (OCR) and you better reduce the error from other models.
For case 2 which is alias name. Although someone argue that we can build a amazing word embeddings such that the vector of “New York” and “New York, US” are close enough for models. However, why do not use a simple way to tackle it?
FuzzyWuzzy is a simple but useful library to find the similar text with score. This libraries is provided by SeatGeek. It is very simple to use this library. You only need to pass target token and dictionary. For the following example, I try to search country and the list of countries can be downloaded from here.
from fuzzywuzzy import processprocess.extract('Hong Kong', countries, limit=3)
First parameter is your target word while the second parameter is dictionary.
[('Hong Kong SAR China', 90), ('Kongo', 72), ('Tonga', 54)]
After the calculation, fuzzywuzzy suggested that “Hong Kong SAR China” has the highest score with “Hong Kong”.
Fuzzywuzzy provides the following different algorithms for us to score strings. For detail implementation, you can visit source code.
ratio: A wrapper of SequenceMatcher. Calculating the score based on number of matched character blocks between 0 and 100.
Formula: 2*(Matched Characters) / (len(String A) + len(String B))
Taking “Edward” as target word, while “Edwards”, “Edwards2” and “drawdE” as dictionary. Number of characters are 7, 8 and 6 respectively.
process.extract('Edward', ['Edwards', 'Edwards2', 'drawdE'], scorer=fuzz.ratio)
The result is
[('Edwards', 92), ('Edwards2', 86), ('drawdE', 50)]
The result of “drawdE” is 50 and you can see that it is not simply calculate the number of matched characters but character blocks.
WRatio: Default scorer. Using different algorithms to score strings between 0 and 100.
To see whether we can query back expected location name, let see whether Japan (inputted “jepen”) and United States (inputted “United tates”) can be returned.
# Default scorer is Weighed Ratiofor location in ['Hong Kong', 'jepen', 'United tates']:    result = process.extract(location, countries, limit=2)    print(result)
The result is
[('Hong Kong SAR China', 90), ('Congo - Kinshasa', 57)][('Japan', 60), ('Yemen', 60)][('United States', 96), ('United Arab Emirates', 86)]
QRatio: A quick ratio comparison for strings.
# Partial Ratioprocess.extract('Hong Kong', countries, scorer=fuzz.QRatio, limit=3)
The result is
[('Hong Kong SAR China', 64), ('Kongo', 57), ('Togo', 46)]
There is UWRatio (Same as WRatio) and UQRatio (Same as QRatio) in case you have to deal with unicode.
To access project template, you can visit this github repo.
I am Data Scientist in Bay Area. Focusing on state-of-the-art in Data Science, Artificial Intelligence , especially in NLP and platform related. You can reach me from Medium Blog, LinkedIn or Github.
Fuzzywuzzy in Python (Original)
Fuzzywuzzy in Java
Country dataset
Word Embeddings Story