Fuzzywuzzy: Merging Datasets on Diversely Transcribed Names | by Megan Sorel | Towards Data Science

I recently did a project that involved merging datasets on names which had slightly different transcriptions for the same person. Some names were spelled wrong, included a nickname, middle name or suffix which didn’t match across datasets. In this case, I used the a string matching package, Fuzzywuzzy, to merge on the correct names.
In order to save you a lot of the extra work I did before merging, I want to explain the modules, methods and uses of Fuzzywuzzy that you can use.
Fuzzywuzzy calculates distance is using Levenshtein distance which, in short form, determines how many changes you would need to make to each character in a string to get to the other word.
For example, ‘dog’ and ‘dogs’ would require 1 change: to add ‘s’. While ‘dog’ and ‘dig’ would also require 1 change: the substitution of 1 letter. Whereas ‘start’ changed to ‘tarp’ would be 2 changes: delete ‘s’ and substitute ‘t’ for ‘p’.
One thing to note when implementing this calculation one names is that it struggles when comparing names to nicknames where the root of the word changed. For example `Bobby` and `Robert` require a lot of changes and may cause other, unrelated names to appear as better matches.
Fuzzywuzzy has two modules: process and fuzz .
Returns the similarity ratio of strings, between 0–100, the basic, most exact method is fuzz.ratio() .
fuzz.partial_ratio() compares a string to equal-length substrings within the longer string. For example: fuzz.partial_ratio(‘Dogs’, ‘Dog’) will have a 100 partial_ratio but would only have a 86 with fuzz.ratio().
fuzz.partial_ratio() compares a string to equal-length substrings within the longer string. For example: fuzz.partial_ratio(‘Dogs’, ‘Dog’) will have a 100 partial_ratio but would only have a 86 with fuzz.ratio().
2. fuzz.token_sort_ratio() measures strings regardless of the order, as you may guess from the inclusion of the word sort. It is useful when comparing names of different order, for example, comparing LastName, FirstName compared to FirstName LastName.
3. token_set_ratio() , as you may guess from .token_sort_ratio , makes a set of each token to compare. Repeated words will not affect the ratio. If we were looking for the song in a list of song names, and one song was called Who Let the Dogs Out and the song match we wanted was actually called Who Let the Dogs Out Who, Who, Who, Who we would get a 100 ratio score.
Grabbing the most similar words based on ratios.
process.extract() and process.extractOne() will take the most similar (extractOne) or n most similar (extract(limit = n)) words from a list of strings compared to the string you give it.
process.extract() and process.extractOne() will take the most similar (extractOne) or n most similar (extract(limit = n)) words from a list of strings compared to the string you give it.
At this point in my merging process, I had reordered the names in one of my datasets to be the same order as my second dataset, so I didn’t use fuzz in my process. Now we know we don’t have to do that for future attempts.
First, install the package and module you are using.
# pip install fuzzywuzzyfrom fuzzywuzzy import process
To merge on the names, I created a set of the names in the dataframe I was merging on, called names and a set of names I wanted to compare to in the second dataframe: member_names. I then made a dictionary of the 2 closest matches to names so I could manually check the scores and if the name matches were correct.
# make a dictionary of closest matches to names keys = {}for name in names: #names in smaller dataset to compare and match#get closest match of `name` compared to larger data `member_names`    keys[name] = ((process.extract(name, member_names, limit=2)))#you can limit to 1 with extractOne to take less time but I wanted to check if names were returning as correct.
Below is the first 5 rows of what keys returns.
Robert J. Wittman [('J Cox', 86), ('Rob Wittman', 86)]James R. Langevin [('James Risch', 86), ('James Comer', 86)]Gerald E. Connolly [('Gerry Connolly', 75), ('J Cox', 54)]Kathy Manning [('Kay Granger', 58), ('Kathy Castor', 56)]Michael K. Simpson [('Mike Simpson', 86), ('Michael Waltz', 66)]
As you can see, the top matches aren’t perfect, so from this point, I filtered and checked for names which were not matches. I checked names with ratios below or at 86%.
#checking names that might not match correctlyfor name in keys:    if keys[name][0][1] <= 86:  #this gives the second value in the first tuple, which is the score of the first name            print(name, keys[name])
At this point you can expand your limit to more than 2, as some matches for me were found at a lower ratio. Once you find the right parameters, you can make a list for names without a match, with a match at a different index, and iterate over the dictionary values to create a merge key filtering for names in the list if need be:
no_match = ['Michael Enzi', 'Kathy Manning', 'John Hickenlooper', 'William Hagerty', 'cott Franklin', 'Victoria Spartz',           'Marjorie Taylor Greene', 'Marie Newman', 'eter Meijer', 'David Cheston Rouzer', 'Robert "Bobby" Scott']second_match = ['Mitchell Mcconnell, Jr.', 'Harold Dallas Rogers', 'Neal Patrick Dunn MD, FACS']
Below, I provided the simplified version of matching code to make the merge_keys:
# getting list of the matches from the bigger dataset to the smaller one as keys to merge onmerge_key = [keys[name][1][0] if                      name in second_match \                      else None if name in no_match else                      keys[name][0][0] for name in \                      df['name']]df[‘merge_key’] = merge_keydf.merge(second_df, how = ‘left’, left_on = ‘merge_key’, right_on=’member_names’)
And there you have it! You can overwrite the original column in either dataset if you want, but I wanted to keep the original names in the original dataset as well as the names I wanted to use to merge on the second dataset.
Thanks for reading and I hope it was helpful!
You can read more about fuzzywuzzy or download it here.