Finding Similar Names Using Cosine Similarity | by Leon Lok | Towards Data Science

Humans are prone to making mistakes. With manually entered data, it’s only a matter of time before something goes wrong. And this is especially true for customer data. There’s usually two reasons for this:
the customer has to enter their information themselves through a user interface, or
somebody else will enter the information for them on their behalf.
Thus errors are bound to happen. These errors could be textual ones such as typos when entering a name. They could be misclicks such as selecting the wrong address from a dropdown menu after entering a postcode. As a result, multiple entries of the same customer could appear as two distinct customers especially if they’re a returning customer. Even if you analyse the data correctly, you could draw wrong and misleading conclusions from the analysis due to poor data.
Gathering incorrect customer information is a very common issue. It can be overwhelming trying to find textual errors and anomalies in a large dataset. That being said, there’s already a lot of techniques available for finding names/documents that look similar. Cosine similarity is one of such techniques. It’s relatively straight forward to implement, and provides a simple solution for finding similar text.
Because of how we usually gather customer data, incorrect customer information is a very common issue. It can be overwhelming trying to find textual errors and anomalies in a large dataset. Luckily, there’s already a lot of methods available for finding names/documents that look similar. One such method is called the cosine similarity. It’s relatively straight forward to implement, and provides a simple solution for finding similar text.
As with many natural language processing (NLP) techniques, this method calculates a numerical value. Hence this technique only works with vectors.
The idea is simple. Cosine similarity calculates a value known as the similarity by taking the cosine of the angle between two non-zero vectors. This ranges from 0 to 1, with 0 being the lowest (the least similar) and 1 being the highest (the most similar).
To demonstrate, if the angle between two vectors is 0°, then the similarity would be 1. Conversely, if the angle between two vectors is 90°, then the similarity would be 0. For two vectors with an angle greater than 90°, then we also consider those to be 0.
However, it’s important to note cosine similarity does not consider the magnitude of the vectors. Only the angle between the two vectors matter. This can be useful in certain contexts.
For example, let’s say we’re comparing two documents where each dimension of their respective vector represents a unique word. If one document is much larger in size and has a much higher frequency of a particular word, they could still be considered similar.
We can do the same thing when comparing names instead of words. We just need to have each vector dimension representing a character instead. Say we had two names with a similar spelling but are different in character length, like Conor and Connor, then this would still result in a high similarity score.
The mathematical definition of the cosine similarity is as follows:
where θ is the angle between two non-zero vectors A and B. Although the formula may look complicated, it’s simply a rearrangement of the definition of the dot product of two vectors. We take the dot product of vectors A and B and divide it by the product of the magnitude of both vectors. Each i represents a dimension of the vectors; in this context, it could be letters or words. If there’s no matching letter or word, then we disregard those (i.e. consider them as 0).
Our first task is to define a function that can take words as inputs and convert them into a vector representation. The code will be in Python 3, but feel free to use whichever language you feel most comfortable with — the underlying concept should be the same.
What we’re aiming to do is to create a function called word2vec() that’ll break down a word into its character components so that we can enter this information as input for the cosine_similarity() function later.
def word2vec(word):    # Count the number of characters in each word.    count_characters = Counter(word)    # Gets the set of characters and calculates the "length" of the vector.    set_characters = set(count_characters)    length = sqrt(sum(c*c for c in count_characters.values()))    return count_characters, set_characters, length, word
First, we use the Counter() method from the collections library to count the number of occurrences of each letter in our inputted word, as shown in the count_characters variable. This is just a dictionary where each key represents a character. The value of the key is the number of occurrences of that character.
We then get the set of characters used in our inputted word by using the set() function so that we can get the intersection of characters between two words, and we’ll name that variable set_characters . We need this for later when we’re calculating the cosine similarity.
The final part is to calculate the ‘length’ of the word. This is the magnitude of the word vector as shown previously in the cosine similarity formula. Remember that the reason we calculate the length this way is because we’re representing these words as vectors, where each used character represents a dimension of the vector.
These are all returned as a tuple so that we can easily extract the information. We also add the input word to the end of the output tuple to easily figure out what the original word was if we wanted to use the function to iterate a list of names.
We can now create the cosine_similarity() function, as shown below. The inputs here will be the outputs of the word2vec() function; that is, vector1 and vector2 are calculated by applying the word2vec() function.
def cosine_similarity(vector1, vector2, ndigits):        # Get the common characters between the two character sets    common_characters = vector1[1].intersection(vector2[1])    # Sum of the product of each intersection character.    product_summation = sum(vector1[0][character] * vector2[0]                  [character] for character in common_characters)    # Gets the length of each vector from the word2vec output.    length = vector1[2] * vector2[2]    # Calculates cosine similarity and rounds the value to ndigits decimal places.    if length == 0:        # Set value to 0 if word is empty.        similarity = 0    else:        similarity = round(product_summation/length, ndigits)    return similarity
First, we need to get a list of characters that are common to both words by using the intersection() function, and we call this variable common_characters .
Referring back to the formula, we then have to calculate the dot product. This is done by iterating through the list of common_characters and getting the count value of each iterated character, as seen in the product_summation variable.
Then, we calculate the product of the lengths of both vectors and call this variable length (looking back, I probably should have used a less confusing variable name).
We’re now ready to plug everything into the formula. However, cosine similarity only allows non-zero vectors due to the division involved. Hence, we need to add a condition that if length = 0 , then the similarity is 0; otherwise, calculate the cosine similarity as usual. The round() function is also used, and this is where we make use of the ndigits argument to round the similarity score to the desired number of decimal places.
With the necessary functions created, we can put everything together to create a function called find_similar() that scans through a list of names and returns a dataframe of results. The results dataframe will be a pandas one (so make sure pandas is installed and aliased as pd) and it’ll consist of three columns:
The names used in the names list will be the first column. These are the vectors used as the first argument of the cosine_similarity() function.
The second column will contain names used to compare to the first column. These are the vectors used as the second argument of the cosine_similarity() function.
The third column will contain the similarity scores calculated using the names in the first and second column.
Below is the Python code for this function. It takes a list of names as input and a ‘similarity threshold’. This threshold sets the minimum required for including pairs of names in the final results dataframe. Doing this removes the uninteresting comparisons from the results. Then, like the cosine_similarity() function, there’s an argument for the number of decimal places to calculate to called ndigits.
def find_similar(full_names_list, similarity_threshold, ndigits):    # Initiate an empty list to store results.    results_list = []    # Apply word2vec function to each name and store them in a list.    vector_list = [word2vec(str(i)) for i in full_names_list]    # Two loops to compare each vector with another vector only once.    for i in range(len(vector_list)):        # Get first vector        vector1 = vector_list[i]        for j in range(i+1, len(vector_list)):            # Get the next vector            vector2 = vector_list[j]            # Calculate cosine similarity            similarity_score = cosine_similarity(vector1, vector2, ndigits)            # Append to results list if similarity score is between 1 and the threshold.            # Note that scores of 1 can be ignored here if we want to exclude people with the same name.            if 1 >= similarity_score >= similarity_threshold:                results_list.append([vector1[3], vector2[3], similarity_score])            else:                pass    # Convert list to dataframe.    results_df = pd.DataFrame(results_list)    if len(results_df) != 0:        results_df.columns = ['full_name', 'comparison_name', 'similarity_score']    else:    # Can add error here if there's no results to return if desired.    pass    return results_df
First, we initialise a results list to store our results in. We then use the word2vec() function in a list comprehension for each name in the names list.
Now we need a way to compare each name with every other name only once; this can be done using two for loops, and we apply the cosine_similarity() function with each iteration. Since we don’t want comparisons that result in a lower similarity score than our specified input threshold, we include a if statement in the inside for loop. This basically says that we will append the similarity score to the results_list if it’s above the threshold.
We then convert the list into a pandas dataframe. At the end, there’s an if statement to check if the dataframe is empty. I personally haven’t added anything in the else statement for this, but an error could be added here for empty results if desired. It then returns the final dataframe as the output.
Say we had 4 names: Conor, Connor, Jon, and John. We can calculate the cosine similarity by comparing each name with every other name. Now that we’ve got the functions ready, it’s just a simple case of plugging everything in!
First, we’ll create a list of names.
names_list = ['Conor', 'Connor', 'Jon', 'John']
hen, we need to decide on the similarity threshold and the number of decimal places we want. Let’s use a similarity threshold of 0 for now. Doing this will give every comparison pair, and we’ll also set the desired decimal place to be 3.
similarity_threshold = 0ndigits = 3
Let’s put these in and see what we get.
results_df = find_similar(name_list, similarity_threshold, ndigits)print(results_df)
Immediately, we should notice that we’ve got 6 comparisons. This is what we’d expect since the number of unique pairs in a set of N items is N*(N-1)/2. We can also see that the pairs with high similarity scores are the Conor and Connor pair, as well as the Jon and John pair, which is also what we’d expect.
If we changed the similarity threshold to 0.8, then we should expect only the mentioned pairs to remain. Let’s give that a go.
Yep, as we expected. However, we can do an additional check by using names where we already know what the outcome will be. For example, we know that if we compared exactly the same names, then we’d expect a similarity score of 1. Conversely, if we compared names that were completely different (no common characters at all), then we’d expect a similarity score of 0.
Let’s do the same by using a different list of names. Say we had 2 names: Connor and Lee. Now we can create a new names list.
names_list_2 = ['Connor', 'Connor', 'Lee', 'Lee']
If we run the find_similar() function using this new names list with a similarity threshold of 0 and the same number of decimal places, then we get the following result.
Check passed! This is exactly the scores we’d expect. If we wanted to, we can use this as a check when unit testing, but I won’t delve into that. This just gives us some confidence that things are working as intended.
We’ve seen that it can be quite simple to find names that are similar by converting them into vectors and calculating the cosine similarity. If we applied this to real customer data, we could also apply additional checks on other attributes such as gender, date of birth, etc. for when we do find names that are similar. For example, Leon and Leona would achieve a high similarity score, but Leon would most likely be a male whereas Leona would most likely be a female; if this is the case, it’s unlikely that the similarity is due to an error like a typo. We want to exclude these. Having some additional checks like this would make it less likely to return such comparisons.
We also might not want to include those that achieve a similarity score of 1 since it’s possible for people to have the same name. Again, we could add some checks on other attributes like gender and date of birth. For example, if there are two Leon records, we can check whether they have the same gender and date of birth. If they do, then it’s likely a duplicate record. Regardless, whether or not you want to include similarity scores of 1 can easily be changed in the if statement in the find_similar() function.
The word2vec() function also doesn’t take into account anagrams, as some readers have pointed out. You could fix this by looking at pairs (or more) of letters across each word like a sliding window. If we’re comparing the inspector with inspectar and inceptors, the algorithm would consider inspector to be more similar to inspectar. This is because even though inceptors is an anagram of inspector, the order of the
Lastly, we might want to add a process before applying the find_similar() function to clean the names first, such as removing capital letters and hyphens, concatenating the first and last names of each customer, etc.
And that’s it! Hopefully, this provides a simple starting point for finding possible errors in a dataset based on name. If anybody has any other ideas on how to improve this, or want to discuss on how to implement it in a real setting or just have general queries about the code, please feel free to contact me.I’d love to have a chat!