FuzzyWuzzy: Find Similar Strings within one column in Python | Towards Data Science

There are different ways to make data dirty, and inconsistent data entry is one of them. Inconsistent values are even worse than duplicates, and sometimes difficult to detect. This article presents how I apply FuzzyWuzzy package to find similar ramen brand names in a ramen review dataset (full Jupyter Notebook can be found on my GitHub). You may also find functions to shorten your codes here.
import pandas as pdimport numpy as npfrom fuzzywuzzy import process, fuzzramen = pd.read_excel('The-Ramen-Rater-The-Big-List-1-3400-Current-                    As-Of-Jan-25-2020.xlsx')ramen.head()
ramen.info()<class 'pandas.core.frame.DataFrame'>RangeIndex: 3400 entries, 0 to 3399Data columns (total 6 columns):Review #    3400 non-null int64Brand       3400 non-null objectVariety     3400 non-null objectStyle       3400 non-null objectCountry     3400 non-null objectStars       3400 non-null objectdtypes: int64(1), object(5)memory usage: 159.5+ KB
This dataset is very simple and easy to understand. Before moving to FuzzyWuzzy, I would like to check some more information. First of all, I remove leading and trailing spaces (if available) in every column, and then print out their number of unique values.
for col in ramen[['Brand','Variety','Style','Country']]:    ramen[col] = ramen[col].str.strip()    print('Number of unique values in ' + str(col) +': ' + str(ramen[col].nunique()))Number of unique values in Brand: 499Number of unique values in Variety: 3170Number of unique values in Style: 8Number of unique values in Country: 48
By sorting unique brand names, we can see if there are similar ones. The result only shows the first 20 brands.
unique_brand = ramen['Brand'].unique().tolist()sorted(unique_brand)[:20]['1 To 3 Noodles', '7 Select', '7 Select/Nissin', '7-Eleven / Nissin', 'A-One', 'A-Sha', 'A-Sha Dry Noodle', 'A1', 'ABC', 'Acecook', 'Adabi', 'Ah Lai', 'Aji-no-men', 'Ajinatori', 'Ajinomoto', 'Alhami', 'Amianda', 'Amino', "Annie Chun's", 'Aroi']
We can see some suspicious names right at the beginning. Next, let’s have some tests with FuzzyWuzzy.
FuzzyWuzzy has four scorer options to find the Levenshtein distance between two strings. In this example, I would check on the token sort ratio and the token set ratio since I believe they are more suitable for this dataset which might have mixed words order and duplicated words.
I pick four brand names and find their similar names in the Brand column. Since we’re matching the Brand column with itself, the result will always include the selected name with a score of 100.
The token sort ratio scorer tokenizes the strings and cleans them by returning these strings to lower cases, removing punctuations, and then sorting them alphabetically. After that, it finds the Levenshtein distance and returns the similarity percentage.
process.extract('7 Select', unique_brand, scorer=fuzz.token_sort_ratio)[('7 Select', 100), ('7 Select/Nissin', 70), ('Jinbo Selection', 61), ('Seven & I', 53), ('Lele', 50)]
This result means that 7 Select/Nissin has 70% similarity when referring to 7 Select. Not bad if I set the threshold at 70% to get the pair of 7 Select — 7 Select/Nissin.
process.extract('A-Sha', unique_brand, scorer=fuzz.token_sort_ratio)[('A-Sha', 100), ('Shan', 67), ('Nasoya', 55), ('Alhami', 55), ('Ah Lai', 55)]
Below 70%, there’s no match for A-sha.
process.extract('Acecook', unique_brand, scorer=fuzz.token_sort_ratio)[('Acecook', 100), ('Vina Acecook', 74), ('Yatekomo', 53), ('Sahmyook', 53), ('Panco', 50)]
Still good at 70% threshold.
process.extract("Chef Nic's Noodles", unique_brand, scorer=fuzz.token_sort_ratio)[("Chef Nic's Noodles", 100), ("Mr. Lee's Noodles", 71), ('Fantastic Noodles', 69), ('1 To 3 Noodles', 62), ("Mom's Dry Noodle", 59)]
Now, we have a problem here. Token sort ratio scorer will get the wrong pair of Chef Nic’s Noodles — Mr. Lee’s Noodles if I set 70% threshold.
process.extract('Chorip Dong', unique_brand, scorer=fuzz.token_sort_ratio)[('Chorip Dong', 100), ('ChoripDong', 95), ('Hi-Myon', 56), ('Mr. Udon', 56), ('Maison de Coree', 54)]
This one looks good enough.
The token set ratio scorer also tokenizes the strings and follows processing steps just like the token sort ratio. Then it collects common tokens between two strings and performs pairwise comparisons to find the similarity percentage.
process.extract('7 Select', unique_brand, scorer=fuzz.token_set_ratio)[('7 Select', 100), ('7 Select/Nissin', 100), ('The Ramen Rater Select', 86), ('Jinbo Selection', 61), ('Seven & I', 53)]
Since the token set ratio is more flexible, the score has increased from 70% to 100% for 7 Select — 7 Select/Nissin.
process.extract('A-Sha', unique_brand, scorer=fuzz.token_set_ratio)[('A-Sha', 100), ('A-Sha Dry Noodle', 100), ('Shan', 67), ('Nasoya', 55), ('Alhami', 55)]
Now we see A-Sha has another name as A-Sha Dry Noodle. And we can see this only by using token set ratio.
process.extract('Acecook', unique_brand, scorer=fuzz.token_set_ratio)[('Acecook', 100), ('Vina Acecook', 100), ('Yatekomo', 53), ('Sahmyook', 53), ('Panco', 50)]
This one got 100% just like the 7 Select case.
process.extract("Chef Nic's Noodles", unique_brand, scorer=fuzz.token_set_ratio)[("Chef Nic's Noodles", 100), ('S&S', 100), ('Mr. Noodles', 82), ("Mr. Lee's Noodles", 72), ('Tseng Noodles', 70)]
This one gets much worse when token set ratio returns 100% for the pair of Chef Nic’s Noodles — S&S.
process.extract('Chorip Dong', unique_brand, scorer=fuzz.token_set_ratio)[('Chorip Dong', 100), ('ChoripDong', 95), ('Hi-Myon', 56), ('Mr. Udon', 56), ('Maison de Coree', 54)]
We have the same result for this one.
Although the token set ratio is more flexible and can detect more similar strings than the token sort ratio, it might also bring in more wrong matches.
In this part, I employ token sort ratio first and create a table showing brand names, their similarities, and their scores.
#Create tuples of brand names, matched brand names, and the scorescore_sort = [(x,) + i             for x in unique_brand              for i in process.extract(x, unique_brand,     scorer=fuzz.token_sort_ratio)]#Create a dataframe from the tuplessimilarity_sort = pd.DataFrame(score_sort, columns=['brand_sort','match_sort','score_sort'])similarity_sort.head()
Since we’re looking for matched values from the same column, one value pair would have another same pair in a reversed order. For example, we will find one pair of EDO Pack — Gau Do, and another pair of Gau Do — EDO Pack. To eliminate one of them later, we need to find “representative” values for the same pairs.
similarity_sort['sorted_brand_sort'] = np.minimum(similarity_sort['brand_sort'], similarity_sort['match_sort'])similarity_sort.head()
Based on the tests above, I only care of those pairs which have at least 80% similarity. I also exclude those which match to themselves (brand value and match value are exactly the same) and those which are duplicated pairs.
high_score_sort = similarity_sort[(similarity_sort['score_sort'] >= 80) &                (similarity_sort['brand_sort'] !=  similarity_sort['match_sort']) &                (similarity_sort['sorted_brand_sort'] != similarity_sort['match_sort'])]high_score_sort = high_score_sort.drop('sorted_brand_sort',axis=1).copy()
Now, let’s see the result.
high_score_sort.groupby(['brand_sort','score_sort']).agg(                        {'match_sort': ', '.join}).sort_values(                        ['score_sort'], ascending=False)
From the score of 95 and above, everything looks good. In each pair, the two values might have typos, one missing character, or inconsistent format, but overall they obviously refer to each other. Below 95, it would be harder to tell. We can look at some examples by listing out data from each pair.
#Souper - Super - 91%ramen[(ramen['Brand'] == 'Souper') | (ramen['Brand'] == 'Super')].sort_values(['Brand'])
For this pair, we see that these two brands come from different manufacturers/countries, and there’s also no similarity in their ramen types or styles. I would say that these brands are not the same.
#Ped Chef - Red Chef - 88%ramen[(ramen['Brand'] == 'Ped Chef') | (ramen['Brand'] == 'Red Chef')].sort_values(['Brand'])
Here, we only have one record of the Ped Chef brand, and we also see the same pattern in its variety name in comparison with the Red Chef brand. I’m pretty sure these two brands are the same.
For a small dataset like this one, we can continue to check other pairs by the same method. From the threshold of 84% and below, we can ignore some pairs which are obviously different or we can make a quick check as above.
To apply the token set ratio, I can just go over the same steps; this part can be found on my Github.
The comparison result include names, their matches using token sort ratio and token set ratio, and the scores respectively.
Now we can see how different it is between two scorers. As expected, the token set ratio matches wrong names with high scores (e.g. S&S, Mr.Noodles). However, it does bring in more matches compared to the token sort ratio (e.g. 7 Select/Nissin, Sugakiya Foods, Vina Acecook). This means to get the most matches, one should use both of the scorers.
With your ramen knowledge and FuzzyWuzzy, can you pick out any correct match?