Fuzzy matching people names. I’ve recently had to solve an... | by Vadim Markovtsev | Towards Data Science

I’ve recently had to solve an interesting problem: given two unordered lists with real people names, match identities in between.
Looks easy, right? Sort both lists, and you are done. Alas, my problem is about two independent lists of people names taken from separate and independent information sources — GitHub and JIRA users belonging to the same organization, to be precise. I have to deal with the following complications:
No perfect match exists. There are names in the first list belonging to people that are not present in the second list, and vice versa.
The lists have different lengths.
The names are arbitrarily formatted. For example, “V. MARKOVTSEV” or “Vadim” instead of “Vadim Markovtsev” (Vadim is my first name).
Although the names are written in Latin letters, there are all possible Unicode quirks with different normalizations, diacritical marks, and so on waiting for me.
List items can contain more than one name. Those are distinct signatures that the corresponding person used on the information resource. For example, a GitHub user may author commits differently on work and home computers.
Having spent some time on experiments, I came up with a “good enough” solver in Python that I published on GitHub: names-matcher. The rest of the article is about the implemented fuzzy matching algorithm.
Having a good dataset for evaluating ideas is an intrinsic ingredient of all good solutions. I’ve collected two: a private and a public one. The private dataset carries personal information of ~10 Athenian’s client organizations: their full names on GitHub and JIRA (Athenian is the company I am working for). Therefore I cannot disclose the data. The public dataset is a semi-artificial dataset that everybody can reproduce following my instructions or direct download the archive.
The idea of how to build the public dataset roots in my previous experience at source{d}: to squeeze maximum juice from the publicly available metadata of GitHub repositories. I leverage two sources: a 2016’s GitHub snapshot on Google BigQuery and GHTorrent. The former supplies real people’s names in Git commit signatures, while the latter offers the ground truth identities of those commits.
The following BigQuery SQL extracts GitHub accounts of relevant size:
SELECT SUBSTR(repo_name, 0, STRPOS(repo_name, '/') - 1), COUNT(*)FROM `bigquery-public-data.github_repos.languages`GROUP BY SUBSTR(repo_name, 0, STRPOS(repo_name, '/') - 1)HAVING COUNT(*) > 50;
Of course, many more accounts with more than 50 repositories exist five years later; however, my goal is not to produce an exhaustive list. The query yields 500-something account names belonging to both organizations and individual users. I filter about 20 entries at the top to exclude too variadic and too mono-authored ones like google, JuliaPackageMirrors, or sindresorhus. Then I launch a heavier query to fetch names in Git signatures of the commits in the remaining repositories:
SELECT DISTINCT SUBSTR(repo, 0, STRPOS(repo, '/') - 1),                commit,                author.name,                committer.nameFROM `bigquery-public-data.github_repos.commits`,     UNNEST(repo_name) AS repoWHERE SUBSTR(repo, 0, STRPOS(repo, '/') - 1) IN ('openstack', 'octoblu', ...);
Now I have a commit hash, the corresponding author and committer names, and the GitHub account name. 1.37 billion of them.
Unfortunately, BigQuery does not tell us which users are associated with each record. We have to mine the second data source, MySQL dump of GHTorrent.
GHTorrent is expired, too — the updates stopped in 2019, but that’s fine because what we really need is a dump after the publication of the BigQuery dataset. The reader may wonder why we need BigQuery. The answer is because the MySQL schema does not contain people’s names. They do exist in MongoDB dumps, but those are incremental and harder to work with in my experience.
This is how to download the dump and extract the file with commit hashes and the corresponding user IDs without storing the rest:
wget -O - http://ghtorrent-downloads.ewi.tudelft.nl/mysql/mysql-2016-09-05.tar.gz | tar -xzvO mysql-2016-09-05/commits.csv | gzip > commits.csv.gz
The next step is to associate each Git commit signature from BigQuery with a user ID from GHTorrent provided the commit hashes are the same. Time to let Python shine.
This long script produces two dictionaries: people with the mapping from user IDs to the set of their names, and acc_membership with the mapping from account ID to the mentioned user IDs. That is not what we want yet: how do we transition to our two lists? By random splitting. We randomly split each person's names into two evenly sized groups — when it is possible.
We end up with 338 pairs of lists with names. The overall number of pairs is 20882. There is a certain degree of bias toward the GitHub logins because many input them as their full name in Git. Besides, the lists in each pair are of equal length. Nevertheless, the dataset proves enough for testing different fuzzy name matching approaches. Examples (I replaced the real names preserving the distinctive properties):
["Dynatrace-VadimMarkovtsev"] - ["Vadim Markovtsev"]["Vadim Markovtsev"] - ["DYNATRACE\\vadim.markovtsev", "VadimM"]["Vadim M."] - ["vmarkovtsev-dt"]["Vadim Markovtsev"] - ["Dynatrace-Vadim-Markovtsev"]["vadimmarkovtsev"] - ["Vadim Markovtsev"]["Vadim Markovtsev", "Headless", "Vadim"] - ["mrkv-headless", "Markovtsev", "mrkv"]["Vałim Mórkovtsev"] - ["Valim Morkovtsev"]
I was initially inspired by these two blog posts: Python Tutorial: Fuzzy Name Matching Algorithms and Python Tutorial: A Name Lookup Table for Fuzzy Name Data Sets by Felix Kuestahler. They are a great introduction to the topic and a solid example of data-driven algorithm development. To summarize his posts, Felix suggested the following plan for fuzzy name matching:
Normalize the name: deal with Unicode quirks, convert to lower case, remove irrelevant symbols, collapse whitespaces. Split the resulting string by whitespace, and convert each to a double metaphone.Define the name’s fingerprint as a set of all possible combinations of those metaphones. Set the distance between two names to zero if their fingerprint sets have an intersection and 1 otherwise.Greedily match the pairs: we ignore possible multiple matches.
Normalize the name: deal with Unicode quirks, convert to lower case, remove irrelevant symbols, collapse whitespaces. Split the resulting string by whitespace, and convert each to a double metaphone.
Define the name’s fingerprint as a set of all possible combinations of those metaphones. Set the distance between two names to zero if their fingerprint sets have an intersection and 1 otherwise.
Greedily match the pairs: we ignore possible multiple matches.
I generalize these steps as:
Name normalization.Distance measure aggregation.Matching over the distance matrix.
Name normalization.
Distance measure aggregation.
Matching over the distance matrix.
I suggest various approaches and refinements in each step and evaluate my public dataset's final accuracy. The more powerful each next step is, the more it masks each previous one's shortcomings, so we should benchmark them independently.
There is a great Python library to normalize Unicode strings called Unidecode. I naturally choose it to remove accents, umlauts, rare letters, and non-typical Unicode normalizations. We evaluate three options coded below.
The first function does a minimal job: removes any character which is not a letter, a number, or a space, and collapses several spaces to one. The second additionally applies unidecode. The third function splits the result of the second by whitespace, converts each part to a metaphone, and concatenates them back.
What shall we do with several names per person? I remove duplicates, sort and concatenate the rest together.
Likewise, there is another great Python library to calculate the similarity between two strings: FuzzyWuzzy. It provides several algorithms that differ by how aggressive they are, i.e., their precision vs. recall trade-off values.
ratio is the traditional Levenshtein distance.
token_sort_ratio splits by whitespace, sorts the parts, concatenates them back, and then calculates ratio.
token_set_ratio splits by whitespace, builds two sets of parts, finds the intersection and the symmetric differences, concatenates the sorted elements of each of the three derivative sets, appends difference strings to the intersection string, and chooses the maximum pairwiseratio (code).
We evaluate five distance calculation methods. The following functions suppose that s1and s2 originate from the normalization step.
FuzzyWuzzy returns the similarity measure from 0 to 100, so I have to subtract it from 100 to pretend that we calculate a distance measure. I “pretend” because there are certain well-defined criteria of a distance measure, and the triangle inequality is not always guaranteed to hold. Luckily, it is not important for our problem at all.
distance_join_ratio nukes all the whitespace characters before calculating ratio. As I show later, that matters.
distance_max_ratio elaborates on the idea of token_set_ratio: I take a minimum of two distances: the token set ratio and distance_ratio_join.
Having compared each name in the first list to each name in the second, we are left with a distance matrix. We have to solve a classic linear assignment problem (LAP):
The problem instance has a number of agents and a number of tasks. Any agent can be assigned to perform any task, incurring some cost that may vary depending on the agent-task assignment. It is required to perform as many tasks as possible by assigning at most one agent to each task and at most one task to each agent, in such a way that the total cost of the assignment is minimized.
I wrote a blog post about LAP a few years ago and even published an open-source LAP solver on GitHub (the core algorithm is courtesy of Jonker and Volgenant). So yes, there is another great Python library to solve the linear assignment problem.
We evaluate two distance matrix matching solutions, the greedy choice and the LAP. The greedy solver does not need an explicit distance matrix.
names1 and names2 are the outputs of the normalization step: the normalized names in the two lists. compare is one of the functions from the distance measure aggregation step.
Important note: solving an exact LAP has cubic complexity over the matrix size. The elapsed time becomes impractical after ~10,000.
Let’s slip everything together. I propose 3 distinct normalization functions, 5 distance calculators, and 2 matrix matchers. They make for a total of 3*5*2=30 combinations, which I evaluate all on the public dataset from the first section. Here is the code:
I calculate the accuracy metric: how many matched identities are correct. I’ll further set a confidence threshold on the private dataset to trade recall for precision. For now, the 3D chart below summarizes the 30 combinations and helps to choose the optimal: unidecode normalization, max_ratio distance and, of course, lap solver peak at 0.78 accuracy.
Surprisingly, metaphones seriously drop the accuracy. Metaphones exclude the vowels and turn out to discard valuable information for the downstream matcher. unidecode does not provide a big advantage over simple, but is still beneficial. The max_ratio distance wins thanks to the aggregation of the unordered set of parts and the sorted concatenated sequence of parts, reminiscent of a primitive ensemble. The normalizations and the solvers being equal,distance_join_ratiois more powerful than distance_set_ratio. The greedy solution is always worse than the LAP, as expected.
We have to choose the fourth meta-parameter — the distance threshold. If the estimated distance between two identities is greater than this value, the algorithm concludes that there is no match, thus reducing recall but improving precision. The private dataset is 10 different pairs of lists with people’s identities, one or more names per identity. I fixed unidecode normalization, distance_max_ratio, and LAP solver, performed the matching and asked Waren Long, our head of product at Athenian, to evaluate the result using his rich domain knowledge. Waren deliberately examined each match and came back with a convincing average F1 chart.
This chart allows me to choose 0.5 as the optimal distance threshold. I achieve 0.93 precision @ 0.95 recall on average across the organizations.
Let’s summarize the previous sections.
I define the problem of fuzzy people names matching between two lists and highlight the pain points.I collect two datasets: a public one from GitHub data from 2016 and a private one from the clients’ data at Athenian.I define the fuzzy names matching algorithm as a sequence of three lower-level procedures: normalization, distance calculation, and the assignment problem solution.I consider several options on how to perform each step.I evaluate the accuracy on the public dataset to choose the best.I evaluate the F1 score on the private dataset to choose the distance threshold to trade recall for some extra precision.
I define the problem of fuzzy people names matching between two lists and highlight the pain points.
I collect two datasets: a public one from GitHub data from 2016 and a private one from the clients’ data at Athenian.
I define the fuzzy names matching algorithm as a sequence of three lower-level procedures: normalization, distance calculation, and the assignment problem solution.
I consider several options on how to perform each step.
I evaluate the accuracy on the public dataset to choose the best.
I evaluate the F1 score on the private dataset to choose the distance threshold to trade recall for some extra precision.
We at Athenian use the described algorithm in our production. I extracted the open-source core to a Python package called names-matcher. names-matcher boasts some additional tweaks to the main ideas presented in this blog post. Namely, the normalization step has become more complex; for example, I remove parts that are a concatenation of some other parts. Besides, I remove some stop words which are “trained” on the private dataset. The user can choose their own stop words list. Those tweaks give me further +0.5% accuracy improvement on the public dataset.
I assume that names-matcher can be improved even more. Please don’t hesitate to write your ideas in the issues! If you wish to be notified of my next posts, please subscribe to @vadimlearning on Twitter. I take the opportunity to thank the whole Athenian team for helping me create and Guillem Duran for the great 3D visualization of the accuracy.