Fuzzy matching with Levenshtein and PostgreSQL | by Dhanesh Budhrani | Towards Data Science

User experience is the result of a combination of factors: UI design, response time, tolerance of user errors, etc. The scope of this post falls into the third category. More specifically, I’ll be explaining how to handle user errors in search forms using PostgreSQL and Levenshtein’s distance algorithm. For instance, we may be looking for a user in our site called “Markus”, although we may be searching for him as “Marcus” (both of these names exist, and would be equivalent in different languages).
Let’s get started by getting familiarized with Levenshtein’s distance algorithm.
Levenshtein’s distance (we’ll call it LD from now on, just to be concise) intends to calculate the dissimilitude between two strings (hence, the higher the value, the less similar they are). This metric represents the number of operations that need to be applied to one of the strings to make it equal to the other one. Such operations include the following: inserting a character, deleting it or substituting it. It is worth mentioning that each operation has an associated cost, although it’s common to set the cost to 1 for all operations.
You’ll rarely need to implement the LD algorithm, given that it’s already present in many libraries of the major programming languages. However, it’s worth understanding how it works. We’ll work through an example to determine the LD between “Marcus” and “Markus”, setting an equivalent cost of 1 to all operations:
We create a matrix D of dimension m x n, where m and n represent the length of our two strings (referenced onwards as A and B).We iterate the matrix top-down and from left to right, computing the LD between the corresponding substrings.We compute the added cost (ac), being 0 if A[i] is equal to B[j] and 1 otherwise.We perform the following operation:
We create a matrix D of dimension m x n, where m and n represent the length of our two strings (referenced onwards as A and B).
We iterate the matrix top-down and from left to right, computing the LD between the corresponding substrings.
We compute the added cost (ac), being 0 if A[i] is equal to B[j] and 1 otherwise.
We perform the following operation:
D[i, j] = min(D[i-1, j] + 1, D[i, j-1] + 1, D[i-1, j-1] + ac)
which means that a cell (i, j) has a LD determined by the minimum of:
The cell right above plus one (due to an insertion operation).
The cell to the left plus one (due to an insertion operation).
The cell above and to the left plus ac (due to a substitution operation).
The following matrix represents the final result of this process:
The matrix above does not only offer us the LD between the full strings, but also the LD between any combination of substrings. The LD between the full strings can be found in position (m, n). In this case, we can see that “Marcus” and “Markus” have a LD of 1 (in red), which is caused by the substitution of “c” by “k”.
Now that you have an understanding of the algorithm, it’s time to get to the practical part. Applying the LD algorithm with PostgreSQL is very simple, all thanks to the fuzzystrmatch extension. This extension offers different algorithms for fuzzy string matching. The available options are the LD algorithm and a set of phonetic functions. Note that these phonetic functions (Soundex, Metaphone and Double Metaphone) may not perform optimally with non-English strings. For this reason, I consider LD to be the most robust option in international applications.
Nevertheless, let’s continue and add this extension to our PostgreSQL database:
CREATE EXTENSION fuzzystrmatch;
At this point, computing LD becomes as simple as running the following query:
SELECT levenshtein(str1, str2);
For instance:
SELECT levenshtein('Marcus', 'Markus');
Therefore, if you have a table of users and want to find all users that have a similar name to the user input (setting, for instance, a maximum LD of 1), your query may look like the following:
SELECT name FROM user WHERE levenshtein('Marcus', name) <= 1;
In this case, “Marcus” represents the user input. In this query, we’d be searching for user that are, either called Marcus, or have a name with a LD of 1 (e.g., “Markus”).
Note that fuzzystrmatch offers another interesting function that would be a bit more efficient in this scenario: levenshtein_less_equal. This function allows you to define a maximum distance threshold, after which PostgreSQL stops computing. In other words, if we set a maximum threshold of 1, and our pair of strings have a LD of 6, fuzzystrmatch will stop computing after reaching a LD of 2 considering that, anyway, we are not interested in the LD of such pair of strings, saving some computation resources (and yielding a LD of 2). Finally, it’s also worth noting that both of these functions have a version that allows you to define the cost of each operation (insertion, deletion and substitution).