Calculating String Similarity in Python | by Dario Radečić | Towards Data Science

Comparing strings in any way, shape or form is not a trivial task. Unless they are exactly equal, then the comparison is easy. But most of the time that won’t be the case — most likely you want to see if given strings are similar to a degree, and that’s a whole another animal.
Let’s go deeper into this with an example. You have a webpage, and it requires some user input — let’s say for some reason you decided to go with a text field instead of a dropdown menu for location information. There exists a possibility that the user will make a typo, and you don’t want typos stored in your database. You want the corrected, typo-free version.
But how would you go about correcting the string? I mean you can hardcode it, but do you really want to do so? There are just so many typo options, and hardcoding isn’t generally a good idea.
That’s the purpose of this article. I want to get you familiar with my top two string matching, or similarity calculation techniques:
Levenshtein distanceCosine similarity
Levenshtein distance
Cosine similarity
The first one is used mainly to address typos, and I find it pretty much useless if you want to compare two documents for example. That’s where the ladder comes in. It’s the exact opposite, useless for typo detection, but great for a whole sentence, or document similarity calculation.
After reading this article you’ll be able to do the following:
Determine which similarity algorithm is suitable for your situation
Implement it in Python
Okay, ready to dive in? Let’s go.
Let’s start with a basic definition:
In information theory, linguistics and computer science, the Levenshtein distance is a string metric for measuring the difference between two sequences. Informally, the Levenshtein distance between two words is the minimum number of single-character edits (insertions, deletions or substitutions) required to change one word into the other.[1]
In most simple words possible, you want to calculate how many transformations you need to perform on the string A to make it equal to string B. The algorithm is also known as Edit Distance, so maybe that’s the term more familiar to you.
To use it in Python you’ll need to install it, let’s say through pip:
pip install python-Levenshtein
And that’s it. Let’s now see how to use it.
Because ‘Levenshtein’ is so difficult to type correctly, and I’ve stated previously that I use it mainly for typo detection, it will be a great first example. But firstly, the imports:
Now you’re ready to calculate the distance:
The first one is typed correctly, and the second one is typed straight out of my head, you know, without googling it first. At first glance they look exactly the same, so you as the end-user maybe won’t even notice the typo. But it makes a huge difference if stored in some database which is used for analysis later on.
I’ve also stated previously that I wouldn’t use this algorithm for longer string, or entire documents, and here’s why:
As you can see, both sentences convey pretty much the same information, but 20 transformations are needed to get from one to another. So if you’ve set the strings to be auto-matched below some threshold, 20 probably won’t be that threshold, because it’s just too large for simple and short sentences like these two.
That’s where its cosine brother comes to rescue.
As before, let’s start with some basic definition:
Cosine similarity is a measure of similarity between two non-zero vectors of an inner product space that measures the cosine of the angle between them.[2]
I know, it’s not the cleanest of definitions, but I find it good enough. It requires some math knowledge, so if you don’t have it, please reference the following article (vectors in linear algebra):
towardsdatascience.com
Okay, let’s distill it even further.
The definition states that you should calculate the angle between two vectors first. But you can’t represent some sentence as a vector in n-dimensional space just out of the box.
You’ll want to construct a vector space from all the ‘sentences’ you want to calculate similarity for. That vector space will have as many dimensions as there are unique words in all sentences combined.
Okay, let’s do the import first and I’ll explain everything on the fly:
You’ll need the string module to remove punctuations from the string — ‘sentence’ and ‘sentence.’ are different by default, and you want to avoid that. CountVectorizer will take care of converting strings to numerical vectors, which is also neat. Finally, as this article is written in English, you’ll want to remove the most frequent words which give no meaning — they are called stopwords — words like ‘I’, ‘me’, ‘myself’, etc.
If you don’t remove the stopwords you’ll end up with a higher-dimensional space, and the angle between vectors will be greater — implying less similarity — even though the vectors convey pretty much the same information.
Now I’ll declare a list of some arbitrary, to a degree similar sentences. It’s a list because the vector space will be created from all unique words, and this will ensure that every vector has the same number of dimensions — as you cannot calculate the angle between vectors in different dimensional space:
Remember the imports? We imported quite a lot, and it’s time to make use of it. I’ll declare a function which will do the following:
Remove punctuations from a given stringLowercase the stringRemove stopwords
Remove punctuations from a given string
Lowercase the string
Remove stopwords
Now you don’t actually need a loop to apply those transformations for every sentence, you can use built-in map function:
Great, the preparation is almost done. Now you’ll utilize the power of CountVectorizer to perform some magic (not really). It will create k vectors in n-dimensional space, where k is the number of sentences, and n is the number of unique words in all sentences combined. Then, if a sentence contains a certain word, the value will be 1 and 0 otherwise:
And that’s it for the preparation. It was a lot (kind of), but it was worth it. If you now calculate the cosine similarity from obtained numeric vectors, you’ll get the following matrix:
The diagonal elements are 1 which makes sense, sentence X is perfectly ‘similar’ to sentence X. It is, however, inconvenient to slice this matrix, so I decided to define a function that will calculate the similarity for two given vectors. Keep in mind that cosine_similarity() expect 2D arrays, and the input vectors are 1D arrays by default, so that’s why we need reshaping:
And just to confirm everything works:
So cool. In the real-world example, this could be used to present some recommendations to the user, for example, based on the similarity of his reviews to every other user’s reviews.
Calculating similarities between numerical vectors is not difficult, the trick is to convert strings to numerical vectors first, and to discard everything irrelevant in the process.
As an exercise, it would be a good idea to find a dataset of some unlabeled emails or some other text, and try to use similarity metrics to group them somehow. It might also be a good idea for you to look at TF-IDF, as I didn’t explain it in this article.
Anyway, thanks for reading, I hope you managed to get something useful out of this.
Loved the article? Become a Medium member to continue learning without limits. I’ll receive a portion of your membership fee if you use the following link, with no extra cost to you.